Universality Phenomena in Machine Learning, and Their Applications

A canonical task in machine learning is to fit a model (from a certain class) to a dataset. In many settings there is little theoretical understanding of the algorithms used for this task, since they involve nonconvex optimization.

We have empirically observed that in many settings the models fitted to real-life datasets display randomlike properties — the model parameters behave like random numbers for various tests. This is somewhat reminiscent of the ‘universality’ phenomenon in mathematics and physics, whereby matrices in a host of settings turn out to display properties similar to those of the Gaussian ensemble.

In this talk, Sanjeev Arora will describe how these randomlike properties can be used to gain a new understanding in some settings — for example, they can offer insights into linear algebraic properties of word meanings in natural languages, and reversibility properties of fully connected deep nets. In some cases, they can lead us to provably efficient algorithms, such as algorithms for making inferences in a topic model.