Publications

In this paper, we make the case that a scientific theory of deep learning is emerging. By this we mean a theory which
characterizes important properties and statistics of the training process, hidden representations, final weights, and
performance of neural networks. We pull together major strands of ongoing research in deep learning theory and identify
five growing bodies of work that point toward such a theory:
1. 2. 3. 4. solvable idealized settings that provide intuition for learning dynamics in realistic systems;
tractable limits that reveal insights into fundamental learning phenomena;
simple mathematical laws that capture important macroscopic observables;
theories of hyperparameters that disentangle them from the rest of the training process, leaving simpler systems
behind; and
5. universal behaviors shared across systems and settings which clarify which phenomena call for explanation.
Taken together, these bodies of work share certain broad traits: they are concerned with the dynamics of the training
process; they primarily seek to describe coarse aggregate statistics; and they emphasize falsifiable quantitative predictions.
We argue that the emerging theory is best thought of as a mechanics of the learning process, and suggest the name learning
mechanics. We assert that learning mechanics should be a mathematical theory, grounded in first-principles calculations
that closely predict empirics, reliant on well-tested approximations and assumptions, aiming for broad impact across the
machine learning stack once it reaches maturity.
We discuss the relationship between this mechanics perspective and other approaches for building a theory of deep
learning, including the statistical and information-theoretic perspectives. In particular, we anticipate a symbiotic and
mutually supportive relationship between learning mechanics and the developing discipline of mechanistic interpretability.
Where mechanistic interpretability aims to be the biology of deep learning, learning mechanics should aspire to be its
physics, mirroring the complementary relationship between biology and physics in the natural sciences.
We also review and address common arguments that fundamental theory will not be possible or is not important. We
conclude with a portrait of important open directions in learning mechanics and advice for beginners. We host further
introductory materials, perspectives, and open questions at learningmechanics.pub.

Neural networks exhibit a remarkable degree of representational convergence across diverse architectures, training objectives, and even data modalities. This convergence is predictive of alignment with brain representation. A recent hypothesis suggests this arises from learning the underlying structure in the environment in similar ways. However, it is unclear how individual stimuli elicit convergent representations across networks. An image can be perceived in multiple ways and expressed differently using words. Here, we introduce a methodology based on the Generalized Procrustes Algorithm to measure intra-modal representational convergence at the single-stimulus level. We applied this to vision models with distinct training objectives, selecting stimuli based on their degree of alignment (intra-modal dispersion). Crucially, we found that this intra-modal dispersion strongly modulates alignment between vision and language models (cross-modal convergence). Specifically, stimuli with low intra-modal dispersion (high agreement among vision models) elicited significantly higher cross-modal alignment than those with high dispersion, by up to a factor of two (e.g., in pairings of DINOv2 with language models). This effect was robust to stimulus selection criteria and generalized across different pairings of vision and language models. Measuring convergence at the single-stimulus level provides a path toward understanding the sources of convergence and divergence across modalities, and between neural networks and human neural representations.

The global dimensionality of a neural representation manifold provides rich insight into the computational process underlying both artificial and biological neural networks. However, all existing measures of global dimensionality are sensitive to the number of samples, i.e., the number of rows and columns of the sample matrix. We show that, in particular, the participation ratio of eigenvalues, a popular measure of global dimensionality, is highly biased with small sample sizes, and propose a bias-corrected estimator that is more accurate with finite samples and with noise. On synthetic data examples, we demonstrate that our estimator can recover the true known dimensionality. We apply our estimator to neural brain recordings, including calcium imaging, electrophysiological recordings, and fMRI data, and to the neural activations in a large language model and show our estimator is invariant to the sample size. Finally, our estimators can additionally be used to measure the local dimensionalities of curved neural manifolds by weighting the finite samples appropriately.

This paper introduces quasi-Monte Carlo latent variable models (QLVMs): a class of deep generative models that are specialized for finding extremely low-dimensional and interpretable embeddings of high-dimensional datasets. Unlike standard approaches, which rely on a learned encoder and variational lower bounds, QLVMs directly approximate the marginal likelihood by randomized quasi-Monte Carlo integration. While this brute force approach has drawbacks in higher-dimensional spaces, we find that it excels in fitting one, two, and three dimensional deep latent variable models. Empirical results on a range of datasets show that QLVMs consistently outperform conventional variational autoencoders (VAEs) and importance weighted autoencoders (IWAEs) with matched latent dimensionality. The resulting embeddings enable transparent visualization and post hoc analyses such as nonparametric density estimation, clustering, and geodesic path computation, which are nontrivial to validate in higher-dimensional spaces. While our approach is compute-intensive and struggles to generate fine-scale details in complex datasets, it offers a compelling solution for applications prioritizing interpretability and latent space analysis.

Representational similarity metrics typically force all units to be matched, making them susceptible to noise and outliers common in neural representations. We extend the soft-matching distance to a partial optimal transport setting that allows some neurons to remain unmatched, yielding rotation-sensitive but robust correspondences. This partial soft-matching distance provides theoretical advantages -- relaxing strict mass conservation while maintaining interpretable transport costs -- and practical benefits through efficient neuron ranking in terms of cross-network alignment without costly iterative recomputation. In simulations, it preserves correct matches under outliers and reliably selects the correct model in noise-corrupted identification tasks. On fMRI data, it automatically excludes low-reliability voxels and produces voxel rankings by alignment quality that closely match computationally expensive brute-force approaches. It achieves higher alignment precision across homologous brain areas than standard soft-matching, which is forced to match all units regardless of quality. In deep networks, highly matched units exhibit similar maximally exciting images, while unmatched units show divergent patterns. This ability to partition by match quality enables focused analyses, e.g., testing whether networks have privileged axes even within their most aligned subpopulations. Overall, partial soft-matching provides a principled and practical method for representational comparison under partial correspondence.

We present a new computational tool for the simulation of aligned assemblies of thin, bendable, but inextensible fibers immersed in a linear Stokes fluid. Such systems are of great importance in cellular mechanics because they arise in many intracellular (e.g., cytoskeleton-cytoplasm interactions) and extracellular (e.g., ciliary locomotion) microscale biological processes. The fiber bed is represented as an anisotropic poroelastic medium that obeys the Euler-Bernoulli beam theory and is hydrodynamically coupled to the viscous fluid through local-slender body theory. We develop two methodologies to solve the resulting fluid-structure interaction problem: (1) a classical approach where the solid is solved in the Lagrangian frame, and the fluid is solved using an Arbitrary-Lagrangian-Eulerian (ALE) method, and (2) a novel approach where the solid equations are expressed in the Eulerian frame and the fiber-fluid system is solved together using an ALE method. In both cases, the resulting set of equations is approximated using a Petrov-Galerkin stabilized finite element method specifically designed for the fiber-fluid interaction problem. Equal-order continuous finite elements are used for the spatial discretization of the deforming physical domain, and finite differences are used for temporal discretization. Both approaches are shown to be numerically stable and convergent at the expected order; and additionally, the pure ALE method can resolve extreme fiber deformations without the need for mesh reconstruction. Finally, our methods are validated by direct comparisons to discrete fiber simulations in two benchmark tests: (a) the shearing of an anchored fiber bed and (b) the emergence and evolution of cell-spanning vortices in cellular geometries.

Solid-phase peptide synthesis enables the study of countless peptides, but aggregation during synthesis hinders production of many sequences of interest. Now, machine learning models enable prior identification of aggregation-prone sequences, and highlight amino acid composition, rather than specific sequence, as the major determinant of aggregation.

We introduce a biologically inspired, multilayer neural architecture composed of Rectified Spectral Units (ReSUs). Each ReSU projects a recent window of its input history onto a canonical direction obtained via canonical correlation analysis (CCA) of previously observed past-future input pairs, and then rectifies either its positive or negative component. By encoding canonical directions in synaptic weights and temporal filters, ReSUs implement a local, self-supervised algorithm for progressively constructing increasingly complex features.
To evaluate both computational power and biological fidelity, we trained a two-layer ReSU network in a self-supervised regime on translating natural scenes. First-layer units, each driven by a single pixel, developed temporal filters resembling those of Drosophila post-photoreceptor neurons (L1/L2 and L3), including their empirically observed adaptation to signal-to-noise ratio (SNR). Second-layer units, which pooled spatially over the first layer, became direction-selective -- analogous to T4 motion-detecting cells -- with learned synaptic weight patterns approximating those derived from connectomic reconstructions.
Together, these results suggest that ReSUs offer (i) a principled framework for modeling sensory circuits and (ii) a biologically grounded, backpropagation-free paradigm for constructing deep self-supervised neural networks.

Brains and artificial neural networks compute with continuous variables such as object position or stimulus orientation. However, the complex variability in neural responses makes it difficult to link internal representational structure to task performance. We develop a statistical-mechanical theory of regression capacity that relates linear decoding efficiency of continuous variables to geometric properties of neural manifolds. Our theory handles complex neural variability and applies to real data, revealing increasing capacity for decoding object position and size along the monkey visual stream.

Generalization, the ability to perform well beyond the training context, is a hallmark of biological and artificial intelligence, yet anticipating unseen failures remains a central challenge. Conventional approaches often take a ``bottom-up'' mechanistic route by reverse-engineering interpretable features or circuits to build explanatory models. While insightful, these methods often struggle to provide the high-level, predictive signals for anticipating failure in real-world deployment. Here, we propose using a ``top-down'' approach to studying generalization failures inspired by medical biomarkers: identifying system-level measurements that serve as robust indicators of a model's future performance. Rather than mapping out detailed internal mechanisms, we systematically design and test network markers to probe structure, function links, identify prognostic indicators, and validate predictions in real-world settings. In image classification, we find that task-relevant geometric properties of in-distribution (ID) object manifolds consistently forecast poor out-of-distribution (OOD) generalization. In particular, reductions in two geometric measures, effective manifold dimensionality and utility, predict weaker OOD performance across diverse architectures, optimizers, and datasets. We apply this finding to transfer learning with ImageNet-pretrained models. We consistently find that the same geometric patterns predict OOD transfer performance more reliably than ID accuracy. This work demonstrates that representational geometry can expose hidden vulnerabilities, offering more robust guidance for model selection and AI interpretability.