The NSF-Simons Research Collaborations on the MoDL initiative awarded two collaborations designed to support research activities focused on a particular set of topics involving some of the most challenging questions in the general area of mathematical and scientific foundations of deep learning.
Collaboration on the Theoretical Foundations of Deep Learning
Peter Bartlett — Director, University of California, Berkeley
Bin Yu — Co-Investigator, University of California, Berkeley
Emmanuel Abbé — PI, Ecole polytechnique fédérale de Lausanne
Mikhail Belkin — PI, University of California, San Diego
Amit Daniely — PI, Hebrew University
Andrea Montanari — PI, Stanford University
Alexander Rakhlin — PI, Massachusetts Institute of Technology
Elchanan Mossel — Co-Investigator, Massachusetts Institute of Technology
Nike Sun — Co-Investigator, Massachusetts Institute of Technology
Roman Vershynin — PI, University of California, Irvine
Nathan Srebro — PI, Toyota Technological Institute at Chicago
The success of deep learning has had a major impact across industry, commerce, science and society. But there are many aspects of this technology that are very different from classical methodology and that are poorly understood. Gaining a theoretical understanding will be crucial for overcoming its drawbacks. The Collaboration on the Theoretical Foundations of Deep Learning aims to address these challenges: understanding the mathematical mechanisms that underpin the practical success of deep learning, using this understanding to elucidate the limitations of current methods and extending them beyond the domains where they are currently applicable, and initiating the study of the array of mathematical problems that emerge. The team has planned a range of mechanisms to facilitate collaboration, including teleconference and in-person research meetings, a centrally organized postdoc program and a program for visits between institutions by postdocs and graduate students. The collaboration will have broad impacts through its education, human resource development and broadening participation programs, in particular through training a diverse cohort of graduate students and postdocs using an approach that emphasizes strong mentorship, flexibility and breadth of collaboration opportunities; through an annual summer school that will deliver curriculum in the theoretical foundations of deep learning to a diverse group of graduate students, postdocs and junior faculty; and through targeting broader participation in the collaboration’s research workshops and summer schools.
The collaboration’s research agenda is built on the following hypotheses: that over-parametrization allows efficient optimization; that interpolation with implicit regularization enables generalization; and that depth confers representational richness through compositionality. It aims to formulate and rigorously study these hypotheses as general mathematical phenomena, with the objective of understanding deep learning, extending its applicability and developing new methods. Beyond enabling the development of improved deep learning methods based on principled design techniques, understanding the mathematical mechanisms that underlie the success of deep learning will also have repercussions on statistics and mathematics, including a new point of view of classical statistical methods, such as reproducing kernel Hilbert spaces and decision forests, and new research directions in nonlinear matrix theory and in understanding random landscapes. In addition, the research workshops that the collaboration will organize will be open to the public and will serve the broader research community in addressing these key challenges.
Collaborative Research: Transferable, Hierarchical, Expressive, Optimal, Robust, Interpretable NETworks (THEORINET)
René Vidal — PI/Director, Johns Hopkins University
Mauro Maggioni — Co-Investigator, Johns Hopkins University
Joshua Vogelstein — Co-Investigator, Johns Hopkins University
Soledad Villar — Co-Investigator, Johns Hopkins University
Gitta Kutyniok — PI, Ludwig-Maximilians-Universität München
Guillermo Sapiro — PI, Duke University
Ingrid Daubechies — Co-Investigator, Duke University
Rong Ge — Co-Investigator, Duke University
Alejandro Ribeiro — PI, University of Pennsylvania
Edgar Dobriban — Co-Investigator, University of Pennsylvania
Robert Ghrist — Co-Investigator, University of Pennsylvania
George Pappas — Co-Investigator, University of Pennsylvania
Yi Ma — PI, University of California, Berkeley
S. Shankar Sastry — Co-Investigator, University of California, Berkeley
Jacob Steinhardt — Co-Investigator, University of California, Berkeley
Emmanuel Candès — PI, Stanford University
Recent advances in deep learning have led to many disruptive technologies: from automatic speech recognition systems to automated supermarkets, to self-driving cars. However, the complex and large-scale nature of deep networks makes them hard to analyze and, therefore, they are mostly used as black boxes without formal guarantees on their performance. Moreover, the design of deep networks remains an art and is largely driven by empirical performance on a data set. As deep learning systems are increasingly employed in our daily lives, it becomes critical to understand if their predictions satisfy certain desired properties. The goal of this collaboration is to develop a mathematical, statistical and computational framework that helps explain the success of current network architectures, understand its pitfalls and guide the design of novel architectures with guaranteed confidence, robustness, interpretability, optimality and transferability. This project will train a diverse STEM workforce with data science skills that are essential for the global competitiveness of the U.S. economy by creating new undergraduate and graduate programs in the foundations of data science and organizing a series of collaborative research events, including semester research programs and summer schools on the foundations of deep learning. This project will also impact women and underrepresented minorities by involving undergraduates in the foundations of data science.
Deep networks have led to dramatic improvements in the performance of pattern recognition systems. However, the mathematical reasons for this success remain elusive. This project brings together a multidisciplinary team of mathematicians, statisticians, theoretical computer scientists and electrical engineers to develop the mathematical and scientific foundations of deep learning. The project is divided into four main thrusts. The analysis thrust will use principles from approximation theory, information theory, statistical inference and robust control to analyze properties of deep networks such as expressivity, interpretability, confidence, fairness and robustness. The learning thrust will use principles from dynamical systems, non-convex and stochastic optimization, statistical learning theory, adaptive control, and high-dimensional statistics to design and analyze learning algorithms with guaranteed convergence, optimality and generalization properties. The design thrust will use principles from algebra, geometry, topology, graph theory and optimization to design and learn network architectures that capture algebraic, geometric and graph structures in both the data and the task. The transferability thrust will use principles from multiscale analysis and modeling, reinforcement learning and Markov decision processes to design and study data representations that are suitable for learning from and transferring to multiple tasks.