2026 Simons Collaboration on the Physics of Learning and Neural Computation Annual Meeting
Conference
Speakers
-
Yasman Bahri Google
-
Mike Douglas Harvard University
-
Surya Ganguli Stanford University
-
Andrea Montanari Stanford University
-
Cengiz Pehlevan Harvard University
-
Andrew Saxe Gatsby University College London
-
Eva Silverstein Stanford University
-
Matthieu Wyart Johns Hopkins University
When
-
All Day
-
All Day
Meeting Goals
Agenda
Thursday, September 24, 2026
| 9:30 AM | Surya Ganguli I Overview of the Physics of Learning Collaboration |
| 11:00 AM | Eva Silverstein I Learning Dynamics, Representations and Symmetry breaking |
| 1:00 PM | Cengiz Pehlevan I Toward a Science of Efficient AI Training |
| 2:30 PM | Yasaman Bahri I Representations in Neural Computation |
| 4:00 PM | Andrea Montanari I The Phase Diagram of 2-layer Neural Networks |
Friday, September 25, 2026
| 9:30 AM | Matthieu Wyart I How Much Data is Required to Learn the Structure of Language? |
| 11:00 AM | Michael Douglas I Reasoning: From Open Models to Solvable Models |
| 1:00 PM | Andrew Saxe I Understanding Multistage Training and Strategy Emergence |
Abstracts
Yasaman Bahri
Google
Representations in Neural Computation
Learned representations lie at the core of AI systems, connecting the statistics of data to nontrivial computation. What ingredients shape these representations, and what can we learn by studying their geometry? How do representations enable generalization and the emergence of new computational capabilities? Yasman Bahri will survey the collaboration’s efforts on these questions, including how representational structure directly affects generalization; how representational manifolds can yield key insights into model behavior; and how certain inductive biases can be crucial for learning useful representations. Bahri will then discuss a series of works in which we address open questions in our understanding of LLMs by considering representations that are learned only from two-token statistics in natural language. We show that representational geometry is directly shaped by various forms of hidden structure within these statistics, giving rise to nontrivial computation and explaining some universal features observed within LLMs.
Mike Douglas
Harvard University
Reasoning: From Open Models to Solvable Models
Over the past year, AI systems have crossed into research-level mathematics — solving competition and open problems whose solutions can be checked exactly. Mike Douglas will review the collaboration’s progress toward a physics of reasoning, organized around what is now measurable. Fully open math models (data, all checkpoints) let us treat reasoning as an experimental subject: solve rate versus distance to the training set, the emergence chronology of skills, and interventions on data and curricula. On the theory side, the collaboration now has its first solvable models of test-time reasoning — chain-of-thought in tractable attention models, with phase diagrams separating regimes where deeper reasoning helps, saturates, or hurts — and a unified view of test-time scaling laws spanning best-of-N search, LLM judges, and verifiers. Douglas will also survey progress on reinforcement learning and self-play at the edge of learnability, and on formalization: research-level mathematics and physics codified in Lean, where the proof checker provides a perfect reward signal. Finally, he will close with open problems: what these results predict for mathematical data, and what to measure next.
Surya Ganguli
Stanford University
Overview of the Physics of Learning Collaboration
Artificial intelligence has made tremendous engineering advances that stand poised to transform our society, yet alarmingly, we understand very little scientifically about how AI actually works. The time is now ripe to study AI as a complex system, by both exploiting and extending the power of physics and mathematics to open the black box of AI systems and improve them in a principled manner. In this talk, colloraboration director Surya Ganguli will present the overarching structure of the collaboration and frame the questions that the collaboration seeks to address concerning the physics of learning and neural computation, setting the stage for the other collaboration talks that explore how AI learns, scales, computes, and reasons, and how the structure of data itself makes all this possible. Ganguli will conclude with some new results on how phase transitions in the ability of generative AI to imagine new data can be derived by a mapping to spin glasses and information theory.
Andrea Montanari
Stanford University
The Phase Diagram of 2-Layer Neural Networks
Classical statistical models are constructed so that either the number of parameters is small with respect to the sample size, or the regularization is strong enough so that their “effective dimension” is small. In contrast, modern machine learning models are vastly miss-specified and overparametrized, and only very loosely regularized. These models are learnt using empirical risk minimization, but the empirical risk function has many near-global optima which are very far from each other in parameter space. How can we make sense of the good statistical properties of these models? Andrea Montanari will summarize what we learned over the years by studying a simple and yet extremely rich setting: learning in two-layer networks.
Cengiz Pehlevan
Harvard University
Toward a Science of Efficient AI Training
Training modern AI systems is increasingly expensive, yet many of the choices that determine this cost still depend on extensive empirical search. This talk will review how combining ideas from theoretical physics with targeted experiments make large-scale training more predictable, transferable, and efficient. Cengiz Pehlevan will begin by examining how parameterization and hyperparameter choices shape optimization dynamics. Pehlevan will then show how this understanding can guide the design of training schedules, emerging optimizers, and architectures. Throughout, Pehlevan will highlight the resulting empirical gains and the adoption of these ideas in industry.
Andrew Saxe
University College London
Understanding Multistage Training and Strategy Emergence
Modern foundation models are trained through multistage pipelines, yet basic questions remain poorly understood: how should limited compute be allocated across stages, and can reinforcement learning create new capabilities during post-training or only reweight existing primitives? In this talk, Andrew Saxe will address these questions with analytical and empirical methods. Saxe will introduce an analytically tractable model of multistage training, shedding light on the factors affecting optimal compute allocation to each stage. In the model, pretraining sets the ceiling on RL performance, with RL learning to reweight primitive strategies. Next, using a controlled rewrite-grammar environment where the pretraining distribution is known, Saxe will empirically show that transformers trained with reinforcement learning with verifiable rewards solve problems that remain nearly impossible after pretraining. By analyzing trajectories, RL first strengthens primitive reductions and then enters a chunking phase in which it forms valid compressed procedures that combine several primitive steps into one. Together, these results begin to shed light on the interactions across stages of modern training pipelines.
Eva Silverstein
Stanford University
Learning Dynamics, Representations and Symmetry Breaking
How do modern neural networks evolve from randomly initialized weights toward a useful representation of the data? How does this proceed layerwise, and what role does the structure of attention play in the case of language models?
Collaboration progress on this central problem draws from diverse methods including dynamical mean field theory, spectral methods, tractable toy models, symmetry analysis, and stochastic dynamics. Eva Silverstein will review several inter-related results exhibiting explicitly how interpretable representations emerge. These encompass multiple stages of learning: the escape from an initially uninformative regime, the emergence of structured directions during training, and, in simplified hierarchical models, the layer-by-layer construction of increasingly useful features. The structured directions that emerge during the learning dynamics include novel ones introduced via symmetry breaking in attention, enabling models to amplify or suppress meaningful token and representation classes.
Matthieu Wyart
Johns Hopkins University
How Much Data Is Required to Learn the Structure of Language?
Despite the fact that experimental neural scaling laws have substantially guided empirical progress in large-scale machine learning, no existing theory can quantitatively predict the exponents of these important laws for any modern LLM trained on natural languages. It is also unclear whether they can be beaten by training with other objectives. Matthieu Wyart will review the efforts of the collaboration in studying this phenomenon, including the first such theory in the case of data-limited scaling laws. Wyart will then introduce synthetic languages displaying a hidden hierarchy (a central notion in both linguistics and computer vision), and discuss how these languages are learnt by deep, but not shallow, nets. Scaling laws in these synthetic languages follow from the same principle as in natural languages when trained at the token level, but these languages can be learnt much more efficiently by machines predicting in an abstract space — justifying the success of joint embedding predictive architectures (JEPA) and other machines learning from their own latents.
More Events
November 11–12, 2026
Mathematics & Physical Sciences
2026 Simons Collaboration on New Frontiers in Superconductivity Annual Meeting
ConferenceNovember 12–13, 2026
Speakers
Daniel Agterberg, Erez Berg, Andrei Bernevig, Laura Classen, Dmitri Efetov, + 3 more
Mathematics & Physical Sciences
2026 Simons Collaboration on Global Categorical Symmetries Annual Meeting
ConferenceNovember 19–20, 2026