Abstract: The recent success of machine learning suggests that neural networks may be capable of approximating high-dimensional functions with controllably small errors. As a result, they could outperform standard function interpolation methods that have been the workhorses of scientific computing but do not scale well with dimension. In support of this prospect, here I will review what is known about the trainability and accuracy of shallow neural networks, which offer the simplest instance of nonlinear learning in functional spaces that are fundamentally different from classic approximation spaces. The dynamics of training in these spaces can be analyzed using tools from optimal transport and statistical mechanics, which reveal when and how shallow neural networks can overcome the curse of dimensionality. I will also discuss how scientific computing problems in high-dimension once thought intractable can be revisited through the lens of these results. Finally, I will discuss open questions, including potential generalizations to deep architecture.
This talk is based on joint work with Grant Rotskoff, Joan Bruna, Zhengdao Chen, and Sammy Jelassi.