Publications

We study level set teleportation, an optimization routine which tries to accelerate gradient descent (GD) by maximizing the gradient norm over a level set of the objective. While teleportation intuitively speeds-up GD via bigger steps, current work lacks convergence theory for convex functions, guarantees for solving the teleportation operator, and even clear empirical evidence showing this acceleration. We resolve these open questions. For convex functions satisfying Hessian stability, we prove that GD with teleportation obtains a combined sub-linear/linear convergence rate which is strictly faster than GD when the optimality gap is small. This is in sharp contrast to the standard (strongly) convex setting, where teleportation neither improves nor worsens convergence. To evaluate teleportation in practice, we develop a projected-gradient method requiring only Hessian-vector products. We use this to show that gradient methods with access to a teleportation oracle out-perform their standard versions on a variety of problems. We also find that GD with teleportation is faster than truncated Newton methods, particularly for non-convex optimization.

Understanding how neural systems efficiently process information through distributed representations is a fundamental challenge at the interface of neuroscience and machine learning. Recent approaches analyze the statistical and geometrical attributes of neural representations as population-level mechanistic descriptors of task implementation. In particular, manifold capacity has emerged as a promising framework linking population geometry to the separability of neural manifolds. However, this metric has been limited to linear readouts. Here, we propose a theoretical framework that overcomes this limitation by leveraging contextual input information. We derive an exact formula for the context-dependent capacity that depends on manifold geometry and context correlations, and validate it on synthetic and real data. Our framework's increased expressivity captures representation untanglement in deep networks at early stages of the layer hierarchy, previously inaccessible to analysis. As context-dependent nonlinearity is ubiquitous in neural systems, our data-driven and theoretically grounded approach promises to elucidate context-dependent computation across scales, datasets, and models.

We study emergent dynamics in a viscous drop subject to interfacial nematic activity. Using hydrodynamic simulations, we show how the interplay of nematodynamics, activity-driven flows in the fluid bulk, and surface deformations gives rise to a sequence of self-organized behaviors of increasing complexity, from periodic braiding motions of topological defects to chaotic defect dynamics and active turbulence, along with spontaneous shape changes and translation. Our findings recapitulate qualitative features of experiments and shed light on the mechanisms underpinning morphological dynamics in active interfaces.

Tissue deformations during morphogenesis can be active, driven by internal processes, or passive, resulting from stresses applied at their boundaries. Here, we introduce the Drosophila hindgut primordium as a model for studying boundary-driven tissue morphogenesis. We characterize its deformations and show that its complex shape changes can be a passive consequence of the deformations of the active regions of the embryo that surround it. First, we find an intermediate characteristic triangular shape in the 3D deformations of the hindgut. We construct a minimal model of the hindgut primordium as an elastic ring deformed by active midgut invagination and germ band extension on an ellipsoidal surface, which robustly captures the symmetry-breaking into this triangular shape. We then quantify the 3D kinematics of the tissue by a set of contours and discover that the hindgut deforms in two stages: an initial translation on the curved embryo surface followed by a rapid breaking of shape symmetry. We extend our model to show that the contour kinematics in both stages are consistent with our passive picture. Our results suggest that the role of in-plane deformations during hindgut morphogenesis is to translate the tissue to a region with anisotropic embryonic curvature and show that uniform boundary conditions are sufficient to generate the observed nonuniform shape change. Our work thus provides a possible explanation for the various characteristic shapes of blastopore-equivalents in different organisms and a framework for the mechanical emergence of global morphologies in complex developmental systems.

Using a state-of-the-art numerical scheme, we show that the Higgs mode under excitation exhibits chirped oscillations and exponential decay when fluctuations are included. This is in stark contrast to conventional BCS collisionless dynamics which predict power-law decay and the absence of chirping. The chirped amplitude mode enables us to determine the local modification of the effective potential even when the system is in a long-lived prethermal state. We then show that this chirped amplitude mode is an experimentally observable quantity since the photoinduced (super)current in pump-probe experiments serves as an efficient proxy for the order parameter dynamics, including the chirped dynamics. Our result is based on the attractive Hubbard model using dynamical mean-field theory within the symmetry-broken state after a excitation across the superconducting gap. Since the collective response involves long timescales, we extend the hierarchical low-rank compression method for nonequilibrium Green's functions to symmetry-broken states and show that it serves as an efficient representation despite long-lived memory kernels.

Biological and artificial neural systems form high-dimensional neural representations that underpin their computational capabilities. Methods for quantifying geometric similarity in neural representations have become a popular tool for identifying computational principles that are potentially shared across neural systems. These methods generally assume that neural responses are deterministic and static. However, responses of biological systems, and some artificial systems, are noisy and dynamically unfold over time. Furthermore, these characteristics can have substantial influence on a system’s computational capabilities. Here, we demonstrate that existing metrics can fail to capture key differences between neural systems with noisy dynamic responses. We then propose a metric for comparing the geometry of noisy neural trajectories, which can be derived as an optimal transport distance between Gaussian processes. We use the metric to compare models of neural responses in different regions of the motor system and to compare the dynamics of latent diffusion models for text-to-image synthesis.

Accurately placing macromolecular assemblies in the cellular context is important in understanding their mechanistic role inside the cell. Previously, we developed a 2D template-matching (2DTM) approach (Rickgauer et al., 2017[Rickgauer, J. P., Grigorieff, N. & Denk, W. (2017). eLife, 6, e25648.]; Lucas et al., 2021[Lucas, B. A., Himes, B. A., Xue, L., Grant, T., Mahamid, J. & Grigorieff, N. (2021). eLife, 10, e68946.]) in cisTEM (Grant et al., 2018[Grant, T., Rohou, A. & Grigorieff, N. (2018). eLife, 7, e35383.]) to detect targets in cellular cryo-EM images with high positional and orientational accuracy. 2DTM not only detects targets such as ribosomes in cryo-EM images but also provides data that enable the in situ classification and high-resolution reconstruction of these targets (Lucas et al., 2022[Lucas, B. A., Zhang, K., Loerch, S. & Grigorieff, N. (2022). eLife, 11, e79272.], 2023[Lucas, B. A., Himes, B. A. & Grigorieff, N. (2023). eLife, 12, 12RP90486.]; Elferich et al., 2022[Elferich, J., Schiroli, G., Scadden, D. T. & Grigorieff, N. (2022). eLife, 11, e80980.]). Building on these successes, this work aims to improve the 2DTM framework to detect more challenging targets in various environments.

A 2DTM search yields a signal-to-noise ratio (SNR) for every location in the cryo-EM image that depends on the cross-correlation between the template and the image (Rickgauer et al., 2017[Rickgauer, J. P., Grigorieff, N. & Denk, W. (2017). eLife, 6, e25648.]). A target is detected when the SNR value exceeds a statistically defined threshold that limits the average false positives to one per image, based on the assumption that the cryo-EM image is dominated by noise and cellular background and that the cross-correlation values observed across the image after whitening the noise/background follow a Gaussian distribution. The 2DTM SNR can be further normalized by subtracting the mean and dividing by the standard deviation of cross-correlations calculated across all sampled orientations at each location in the image (Rickgauer et al., 2017[Rickgauer, J. P., Grigorieff, N. & Denk, W. (2017). eLife, 6, e25648.]). This step is often referred to as `z-score' normalization (Spiegel & Stephens, 1999[Spiegel, M. R. & Stephens, L. J. (1999). Schaum's Outline of Theory and Problems of Statistics, 3rd ed. New York: McGraw-Hill.]). Using the z-score instead of the SNR improves the detection of capsomers in rotavirus double-layered particles (DLPs; Rickgauer et al., 2017[Rickgauer, J. P., Grigorieff, N. & Denk, W. (2017). eLife, 6, e25648.]) and ribosomes in a crowded cellular environment (Lucas et al., 2022[Lucas, B. A., Zhang, K., Loerch, S. & Grigorieff, N. (2022). eLife, 11, e79272.]). In the following, we will refer to the outputs of 2DTM as the 2DTM SNR and 2DTM z-score, respectively.

Previous applications of 2DTM have shown that the 2DTM SNR and z-score function differently depending on the characteristics of the sample and target. For example, when low-resolution features were suppressed by using a near-focus image setting (70 nm), the 2DTM SNR map showed a flat background with sharp peaks indicating the locations of apoferritins, even in a dense protein (bovine serum albumin) background (Rickgauer et al., 2017[Rickgauer, J. P., Grigorieff, N. & Denk, W. (2017). eLife, 6, e25648.]). On the other hand, low-resolution features from the target itself when strongly defocused (>2000 nm), or from the background structural noise, can result in broader peaks or an uneven background in the SNR map, complicating target detection (Rickgauer et al., 2017[Rickgauer, J. P., Grigorieff, N. & Denk, W. (2017). eLife, 6, e25648.]; Lucas et al., 2022[Lucas, B. A., Zhang, K., Loerch, S. & Grigorieff, N. (2022). eLife, 11, e79272.]). The misleading low-resolution background can be suppressed by calculating the 2DTM z-score (Rickgauer et al., 2017[Rickgauer, J. P., Grigorieff, N. & Denk, W. (2017). eLife, 6, e25648.]), which removes spurious correlations between the template and the structural noise in the image, thereby flattening the background and improving the detectability of targets in cellular environments (Rickgauer et al., 2020[Rickgauer, J. P., Choi, H., Lippincott-Schwartz, J. & Denk, W. (2020). bioRxiv, 2020.04.22.053868.]; Lucas et al., 2022[Lucas, B. A., Zhang, K., Loerch, S. & Grigorieff, N. (2022). eLife, 11, e79272.]). In Fig. 1[link](a), a segment of a previously published micrograph of a yeast lamella near the nucleus is presented (Lucas et al., 2022[Lucas, B. A., Zhang, K., Loerch, S. & Grigorieff, N. (2022). eLife, 11, e79272.]). This image section contains various cellular compartments located from left to right, including the vacuole, cytoplasm and nucleus. Using the mature 60S as a search template, 2DTM outputs a 2DTM SNR map and a 2DTM z-score map [Figs. 1[link](b) and 1[link](c)]. The bright spots in the 2DTM SNR map are locations with high correlation values, indicating 60S ribosomes. However, the peaks are surrounded by halos of increased SNR values extending to other low-resolution features in the image, such as membranes. The z-score map removes these halos and spurious matches of high-contrast features, thereby reducing the number of false detections (membranes or partial overlap with ribosomes) while preserving locations with high-resolution matches from the riboso

Quasi-2D Coulomb systems are of fundamental importance and have attracted much attention in many areas nowadays. Their reduced symmetry gives rise to interesting collective behaviors, but also brings great challenges for particle-based simulations. Here, we propose a novel algorithm framework to address the O(N2) simulation complexity associated with the long-range nature of Coulomb interactions. First, we introduce an efficient Sum-of-Exponentials (SOE) approximation for the long-range kernel associated with Ewald splitting, achieving uniform convergence in terms of inter-particle distance, which reduces the complexity to O(N7/5). We then introduce a random batch sampling method in the periodic dimensions, the stochastic approximation is proven to be both unbiased and with reduced variance via a tailored importance sampling strategy, further reducing the computational cost to O(N). The performance of our algorithm is demonstrated via various numerical examples. Notably, it achieves a speedup of 2∼3 orders of magnitude comparing with Ewald2D method, enabling molecular dynamics (MD) simulations with up to 106 particles on a single core. The present approach is therefore well-suited for large-scale particle-based simulations of Coulomb systems under confinement, making it possible to investigate the role of Coulomb interaction in many practical situations.

Large language models have demonstrated an impressive ability to perform factual recall. Prior work has found that transformers trained on factual recall tasks can store information at a rate proportional to their parameter count. In our work, we show that shallow transformers can use a combination of associative memories to obtain such near optimal storage capacity. We begin by proving that the storage capacities of both linear and MLP associative memories scale linearly with parameter count. We next introduce a synthetic factual recall task, and prove that a transformer with a single layer of self-attention followed by an MLP can obtain 100% accuracy on the task whenever either the total number of self-attention parameters or MLP parameters scales (up to log factors) linearly with the number of facts. In particular, the transformer can trade off between using the value matrices or the MLP as an associative memory to store the dataset of facts. We complement these expressivity results with an analysis of the gradient flow trajectory of a simplified linear attention model trained on our factual recall task, where we show that the model exhibits sequential learning behavior.

Macrocycles are a promising therapeutic class. The incorporation of heterochiral and non-natural chemical building-blocks presents challenges for rational design, however. With no existing machine learning methods tailored for heterochiral macrocycle design, we developed a novel convolutional autoencoder model to rapidly generate energetically favorable macrocycle backbones for heterochiral design and structure prediction. Our approach surpasses the current state-of-the-art method, Generalized Kinematic loop closure (GenKIC) in the Rosetta software suite. Given the absence of large, available macrocycle datasets, we created a custom dataset in-house and in silico. Our model, CyclicCAE, produces energetically stable backbones and designable structures more rapidly than GenKIC. It enables users to perform energy minimization, generate structurally similar or diverse inputs via MCMC, and conduct inpainting with fixed anchors or motifs. We propose that this novel method will accelerate the development of stable macrocycles, speeding up macrocycle drug design pipelines.