Publications

Prioritized experience replay, which improves sample efficiency by selecting relevant transitions to update parameter estimates, is a crucial component of contemporary value-based deep reinforcement learning models. Typically, transitions are prioritized based on their temporal difference error. However, this approach is prone to favoring noisy transitions, even when the value estimation closely approximates the target mean. This phenomenon resembles the noisy TV problem postulated in the exploration literature, in which exploration-guided agents get stuck by mistaking noise for novelty. To mitigate the disruptive effects of noise in value estimation, we propose using epistemic uncertainty estimation to guide the prioritization of transitions from the replay buffer. Epistemic uncertainty quantifies the uncertainty that can be reduced by learning, hence reducing transitions sampled from the buffer generated by unpredictable random processes. We first illustrate the benefits of epistemic uncertainty prioritized replay in two tabular toy models: a simple multi-arm bandit task, and a noisy gridworld. Subsequently, we evaluate our prioritization scheme on the Atari suite, outperforming quantile regression deep Q-learning benchmarks; thus forging a path for the use of uncertainty prioritized replay in reinforcement learning agents.

Understanding protein conformation is key to understanding their function. Importantly, most proteins adopt multiple conformations with nontrivial ensemble distributions that change depending on their environment to perform functions like catalysis, signaling, and transport. Recently, machine learning techniques, especially deep generative models, have been employed to develop protein conformation generators. These models, known as unified protein ensemble samplers, are trained on the Protein Data Bank (PDB) dataset and can generate diverse protein conformation ensembles given a protein sequence. However, their reliance solely on structural data from the PDB, which primarily captures folded protein states, restricts the diversity of the generated ensembles and can result in physically unrealistic conformations. In this paper, we overcome these challenges by introducing ExEnDiff, an experiment-guided diffusion model for protein conformation generation. ExEnDiff integrates experimental measurements as a physical prior, enabling the generation of protein conformations with desired properties. Our experiments on a variety of fast-folding and intrinsically disordered proteins demonstrate that ExEnDiff significantly advances the capabilities of current unified protein ensemble samplers. With little computational cost, ExEnDiff can capture important proteins' configuration properties and the underlying Boltzmann distribution, paving the way for a next-generation molecular dynamics engine. We further demonstrate the effectiveness of ExEnDiff to capture conformational changes in the presence of mutations and as an efficient tool for determining a reasonable collective variable space for protein ensembles. With these results, ExEnDiff is well poised to push the study of protein ensembles into a data-rich regime currently available to few problems in biology.

This paper presents a novel approach to learning free terminal time closed-loop control for robotic manipulation tasks, enabling dynamic adjustment of task duration and control inputs to enhance performance. We extend the supervised learning approach, namely solving selected optimal open-loop problems and utilizing them as training data for a policy network, to the free terminal time scenario. Three main challenges are addressed in this extension. First, we introduce a marching scheme that enhances the solution quality and increases the success rate of the open-loop solver by gradually refining time discretization. Second, we extend the QRnet in [1] to the free terminal time setting to address discontinuity and improve stability at the terminal state. Third, we present a more automated version of the initial value problem (IVP) enhanced sampling method from previous work [2] to adaptively update the training dataset, significantly improving its quality. By integrating these techniques, we develop a closed-loop policy that operates effectively over a broad domain with varying optimal time durations, achieving near globally optimal total costs. The appendix and videos are available at https://deepoptimalcontrol.github.io/FreeTimeManipulator.

Active nematics exhibit spontaneous flows through a well-known linear instability of the uniformly aligned quiescent state. Here, we show that even a linearly stable uniform state can experience a nonlinear instability, resulting in a discontinuous transition to spontaneous flows. In this case, quiescent and flowing states may coexist. Through a weakly nonlinear analysis and a numerical study, we trace the bifurcation diagram of striped patterns and show that the underlying pitchfork bifurcation switches from supercritical (continuous) to subcritical (discontinuous) by varying the flow-alignment parameter. We predict that the discontinuous spontaneous flow transition occurs for a wide range of parameters, including systems of contractile flow-aligning rods. Our predictions are relevant to active nematic turbulence and can potentially be tested in experiments on either cell layers or active cytoskeletal suspensions.

Characterizing the conformational ensemble of biomolecular systems is key to understand their functions. Cryoelectron microscopy (cryo-EM) captures two-dimensional snapshots of biomolecular ensembles, giving in principle access to thermodynamics. However, these images are very noisy and show projections of the molecule in unknown orientations, making it very difficult to identify the biomolecule’s conformation in each individual image. Here, we introduce cryo-EM simulation-based inference (cryoSBI) to infer the conformations of biomolecules and the uncertainties associated with the inference from individual cryo-EM images. CryoSBI builds on simulation-based inference, a merger of physics-based simulations and probabilistic deep learning, allowing us to use Bayesian inference even when likelihoods are too expensive to calculate. We begin with an ensemble of conformations, templates from experiments, and molecular modeling, serving as structural hypotheses. We train a neural network approximating the Bayesian posterior using simulated images from these templates and then use it to accurately infer the conformation of the biomolecule from each experimental image. Training is only done once on simulations, and after that, it takes just a few milliseconds to make inference on an image, making cryoSBI suitable for arbitrarily large datasets and direct analysis on micrographs. CryoSBI eliminates the need to estimate particle pose and imaging parameters, significantly enhancing the computational speed compared to explicit likelihood methods. Importantly, we obtain interpretable machine learning models by integrating physics-based approaches with deep neural networks, ensuring that our results are transparent and reliable. We illustrate and benchmark cryoSBI on synthetic data and showcase its promise on experimental single-particle cryo-EM data.

Studies of fate patterning during development typically emphasize cell-cell communication via diffusible chemical signals. Recent experiments on stem cell colonies, however, suggest that in some cases mechanical stresses, rather than secreted chemicals, enable long-ranged cell-cell interactions that specify positional information and pattern cell fates. These findings inspire a model of mechanical patterning: fate affects cell contractility, and pressure in the cell layer biases fate. Cells at the colony edge, more contractile than cells at the center, seed a pattern that propagates via force transmission. Strikingly, our model implies that the width of the outer fate domain varies nonmonotonically with substrate stiffness, a prediction that we confirm experimentally; we argue that a similar dependence on substrate stiffness can be achieved by a chemical morphogen only if strong constraints on the signaling pathway's mechanobiology are met. Our findings thus support the idea that mechanical stress can mediate patterning in the complete absence of chemical morphogens, even in nonmotile cell layers, thus expanding the repertoire of possible roles for mechanical signals in development and morphogenesis. Future tests of additional model predictions, like the effect of anisotropic substrate rigidity, will further broaden the range of achievable fate patterns.

Characterizing the conformational ensemble of biomolecular systems is key to understand their functions. Cryoelectron microscopy (cryo-EM) captures two-dimensional snapshots of biomolecular ensembles, giving in principle access to thermodynamics. However, these images are very noisy and show projections of the molecule in unknown orientations, making it very difficult to identify the biomolecule’s conformation in each individual image. Here, we introduce cryo-EM simulation-based inference (cryoSBI) to infer the conformations of biomolecules and the uncertainties associated with the inference from individual cryo-EM images. CryoSBI builds on simulation-based inference, a merger of physics-based simulations and probabilistic deep learning, allowing us to use Bayesian inference even when likelihoods are too expensive to calculate. We begin with an ensemble of conformations, templates from experiments, and molecular modeling, serving as structural hypotheses. We train a neural network approximating the Bayesian posterior using simulated images from these templates and then use it to accurately infer the conformation of the biomolecule from each experimental image. Training is only done once on simulations, and after that, it takes just a few milliseconds to make inference on an image, making cryoSBI suitable for arbitrarily large datasets and direct analysis on micrographs. CryoSBI eliminates the need to estimate particle pose and imaging parameters, significantly enhancing the computational speed compared to explicit likelihood methods. Importantly, we obtain interpretable machine learning models by integrating physics-based approaches with deep neural networks, ensuring that our results are transparent and reliable. We illustrate and benchmark cryoSBI on synthetic data and showcase its promise on experimental single-particle cryo-EM data.

Computational psychiatry studies suggest that individuals with autism spectrum disorder (ASD) inflexibly update their expectations. Here we leveraged high-yield rodent psychophysics, extensive behavioral modeling and brain-wide single-cell extracellular recordings to assess whether mice with different genetic perturbations associated with ASD show this same computational anomaly, and if so, what neurophysiological features are shared across genotypes. Mice harboring mutations in Fmr1, Cntnap2 or Shank3B show a blunted update of priors during decision-making. Compared with mice that flexibly updated their priors, inflexible updating of priors was associated with a shift in the weighting of prior encoding from sensory to frontal cortices. Furthermore, frontal areas in mouse models of ASD showed more units encoding deviations from the animals’ long-run prior, and sensory responses did not differentiate between expected and unexpected observations. These findings suggest that distinct genetic instantiations of ASD may yield common neurophysiological and behavioral phenotypes.

The remarkable cohesion and coordination observed in moving animal groups and their collective responsiveness to threats are thought to be mediated by scale-free correlations, where changes in the behavior of one animal influence others in the group, regardless of the distance between them. But are these features independent of group size? Here, we investigate group cohesiveness and collective responsiveness in computational models of massive schools of fish of up to 50,000 individuals. We show that as the number of swimmers increases, flow interactions destabilize the school, creating clusters that constantly fragment, disperse, and regroup, similar to their biological counterparts. We calculate the spatial correlation and speed of information propagation in these dynamic clusters. Spatial correlations in cohesive and polarized clusters are indeed scale free, much like in natural animal groups, but fragmentation events are preceded by a decrease in correlation length, thus diminishing the group's collective responsiveness, leaving it more vulnerable to predation events. Importantly, in groups undergoing collective turns, the information about the change in direction propagates linearly in time among group members, thanks to the non-reciprocal nature of the visual interactions between individuals. Merging speeds up the transfer of information within each cluster by several fold, while fragmentation slows it down. Our findings suggest that flow interactions may have played an important role in group size regulation, behavioral adaptations, and dispersion in living animal groups.

Dielectrically confined Coulomb systems are widely employed in molecular dynamics (MD) simulations. Despite extensive efforts in developing efficient and accurate algorithms for these systems, rigorous and accurate error estimates, which are crucial for optimal parameter selection for simulations, are still lacking. In this work, we present a rigorous error analysis in Ewald summation for electrostatic interactions in systems with two dielectric planar interfaces, where the polarization contribution is modeled by an infinitely reflected image charge series. An accurate error estimate is provided for the truncation error of image charge series, as well as decay rates of energy and force correction terms, as functions of system parameters such as vacuum layer thickness, dielectric permittivity, and image truncation levels. Extensive numerical tests conducted across several prototypical parameter settings validate our theoretical predictions. Additionally, our analysis elucidates the nonmonotonic error convergence behavior observed in previous numerical studies. Finally, we provide an optimal parameter selection strategy derived from our theoretical insights, offering practical guidance for efficient and accurate MD simulations of dielectric-confined systems.