2789 Publications

Detection of Moving Objects Using Self-motion Constraints on Optic Flow

H. Lutwak, Bas Rokers, E. P. Simoncelli

As we move through the world, the pattern of light projected on our eyes is complex and dynamic, yet we are still able to distinguish between moving and stationary objects. We propose that humans accomplish this by exploiting constraints that self-motion imposes on retinal velocities. When an eye translates and rotates in a stationary 3D scene, the velocity at each retinal location is constrained to a line segment in the 2D space of retinal velocities. The slope and intercept of this segment is determined by the eye's translation and rotation, and the position along the segment is determined by the local scene depth. Since all possible velocities arising from a stationary scene must lie on this segment, velocities that are not must correspond to objects moving within the scene. We hypothesize that humans make use of these constraints by using deviations of local velocity from these constraint lines to detect moving objects. To test this, we used a virtual reality headset to present rich wide-field stimuli, simulating the visual experience of translating forward in several virtual environments with varied precision of depth information. Participants had to determine if a cued object moved relative to the scene. Consistent with the hypothesis, we found that performance depended on the deviation of the object velocity from the constraint segment, rather than a difference between retinal velocities of the object and its local surround. We also found that the endpoints of the constraint segment reflected the precision of depth information available in the different virtual environments.

Show Abstract

Biomolecular condensates control and are defined by RNA-RNA interactions that arise in viral replication

Christine Roden, Dilimulati Aierken, R. Sealfon, et al.

Cells must limit RNA–RNA interactions to avoid irreversible RNA entanglement. Cells may prevent deleterious RNA-RNA interactions by genome organization to avoid complementarity however, RNA viruses generate long, perfectly complementary antisense RNA during replication. How do viral RNAs avoid irreversible entanglement? One possibility is RNA sequestration into biomolecular condensates. To test this, we reconstituted critical SARS-CoV-2 RNA–RNA interactions in Nucleocapsid condensates. We observed that RNAs with low propensity RNA–RNA interactions resulted in more round, liquid-like condensates while those with high sequence complementarity resulted in more heterogeneous networked morphology independent of RNA structure stability. Residue-resolution molecular simulations and direct sequencing-based detection of RNA–RNA interactions support that these properties arise from degree of trans RNA contacts. We propose that extensive RNA–RNA interactions in cell and viral replication are controlled via a combination of genome organization, timing, RNA sequence content, RNA production ratios, and emergent biomolecular condensate material properties.

Show Abstract

Detecting Moving Objects During Self-motion

H. Lutwak

s we move through the world, the pattern of light projected on our eyes is complex and dynamic. Even in a world that is completely stationary, our self-motion results in velocities on the retina. Added to this there exist independently moving objects, which also create evolving patterns of light on our eyes. Despite the fact that both induce retinal velocities, we are somehow able to accurately distinguish between stable parts of the environment and independently moving objects. One might hypothesize this is achieved by detecting discontinuities in the spatial pattern of velocities, however this computation is also sensitive to velocity discontinuities at the boundaries of stationary objects. We instead propose that humans make use of the specific constraints that self-motion imposes on retinal velocities. When an eye translates and rotates within a rigid 3D world, the velocity at each location on the retina is constrained to a line segment in the 2D space of retinal velocities (Longuet, Higgins, Prazdny 1980). The slope and intercept of this segment is determined by the eye's translation and rotation, and the position along the segment is determined by depth of the scene. Since all possible velocities arising from a rigid world must lie on this segment, velocities not on the segment must correspond to moving objects. We hypothesize that humans make use of these constraints, by partially inferring self-motion based on the global pattern of retinal velocities, and using deviations of local velocity from the resulting constraint lines to detect moving objects. We call this the depth constraint segment.
We first test if the depth constraint has an effect on 2D velocity discrimination using a simplified stimulus made with a collection of plaids that drifted according to a moving observer. Under these conditions, we failed to find convincing evidence that the constraint had on 2D velocity discrimination. We then tried to test the hypothesis with more naturalistic stimuli, viewed within a head-mounted virtual reality device, simulating a translation forward in different virtual environments. This time, consistent with the hypothesis, we found that performance depended on the deviation of the object velocity from the constraint segment, not on the difference between retinal velocities of the object and its surrounding velocities. Finally, we examine the effect of self-motion on detecting a specific kind of motion artifact (jitter) that occurs in an augmented reality display. We found that our ability to perceive the motion artifact depended on self-motion and the evoked eye movements.

Show Abstract

Geometric Effects in Large Scale Intracellular Flows

Olenka Jain, B. Chakrabarti, R. Farhadifar, Elizabeth R. Gavis, M. Shelley, S. Shvartsman

This work probes the role of cell geometry in orienting self-organized fluid flows in the late-stage Drosophila oocyte. Recent theoretical work has shown that a model, which relies only on hydrodynamic interactions of flexible, cortically anchored microtubules and the mechanical loads from molecular motors moving upon them, is sufficient to generate observed flows. While the emergence of flows has been studied in spheres, oocytes change shape during streaming, and it was unclear how robust these flows are to the geometry of the cell. Here we use biophysical theory and computational analysis to investigate the role of geometry and find that the axis of rotation is set by the shape of the domain and that the flow is robust to biologically relevant perturbations of the domain shape. Using live imaging and three-dimensional flow reconstruction, we test the predictions of the theory/simulation, finding consistency between the model and live experiments, further demonstrating a geometric dependence on flow direction in late-stage Drosophila oocytes.

Show Abstract

Investigating the membrane curvature sensing ability of the N-terminal domain of huntingtin

Shelli Frey, Jordyn Markle, A. Sahoo, et al.

Huntington's disease (HD) is an inherited neurodegenerative disorder associated with motor and cognitive decline, caused by a mutation in the poly-glutamine (polyQ) region near the N-terminus of the huntingtin (htt) protein. Expansion of the polyQ region results in the disease that is characterized by oligomeric and fibrillar aggregates of mutated protein. The first 17 amino acids (Nt17) of htt, which are adjacent to the polyQ tract, function as a lipid-binding domain, facilitated by the formation of an amphipathic α-helix. There is increasing evidence that lipid interactions may play a role in the toxic gain of function associated with the htt polyQ expansion, as membrane-related changes, including structural abnormalities of several organelles, are observed in HD. Given the uneven and curved shapes of organelles, it is important to examine the mechanistic preferences that drive the preferential partitioning of Nt17 to curved membranes. To better understand the role of the cell membrane environment in the interaction and aggregation of htt, circular dichroism, fluorescence microscopy, and coarse-grained molecular dynamics were employed to measure the association of Nt17 with phospholipid vesicles and subsequent effects throughout time. In zwitterionic curved membranes, sensing was driven by the bulky sidechains of phenylalanine residues, which are able to sense lipid packing defects in the curved regions of the membrane. However, in a mixture of zwitterionic and anionic lipids, curvature sensing is affected by the anionic lipid content, implying the surface charge of membranes affects the curvature sensing process. Salt screening experiments suggest a balance between the electrostatic and hydrophobic interactions that governs the extent to which Nt17 can sense physiologically relevant regions of curvature.

Show Abstract

Generalized Compressed Sensing for Image Reconstruction with Diffusion Probabilistic Models

Ling-Qi Zhang, Z. Kadkhodaie, E. P. Simoncelli, D. H. Brainard

We examine the problem of selecting a small set of linear measurements for reconstructing high-dimensional signals. Well-established methods for optimizing such measurements include principal component analysis (PCA), independent component analysis (ICA) and compressed sensing (CS) based on random projections, all of which rely on axis- or subspace-aligned statistical characterization of the signal source. However, many naturally occurring signals, including photographic images, contain richer statistical structure. To exploit such structure, we introduce a general method for obtaining an optimized set of linear measurements for efficient image reconstruction, where the signal statistics are expressed by the prior implicit in a neural network trained to perform denoising (known as a ``diffusion model''). We demonstrate that the optimal measurements derived for two natural image datasets differ from those of PCA, ICA, or CS, and result in substantially lower mean squared reconstruction error. Interestingly, the marginal distributions of the measurement values are asymmetrical (skewed), substantially more so than those of previous methods. We also find that optimizing with respect to perceptual loss, as quantified by structural similarity (SSIM), leads to measurements different from those obtained when optimizing for MSE. Our results highlight the importance of incorporating the specific statistical regularities of natural signals when designing effective linear measurements.

Show Abstract

Observation of Floquet–Bloch states in monolayer graphene

Floquet engineering is a novel method of manipulating quantum phases of matter via periodic driving [1, 2]. It has successfully been utilized in different platforms ranging from photonic systems [3] to optical lattice of ultracold atoms [4, 5]. In solids, light can be used as the periodic drive via coherent light-matter interaction. This leads to hybridization of Bloch electrons with photons resulting in replica bands known as Floquet-Bloch states. After the direct observation of Floquet-Bloch states in a topological insulator [6], their manifestations have been seen in a number of other experiments [7-14]. By engineering the electronic band structure using Floquet-Bloch states, various exotic phase transitions have been predicted [15-22] to occur. To realize these phases, it is necessary to better understand the nature of Floquet-Bloch states in different materials. However, direct energy and momentum resolved observation of these states is still limited to only few material systems [6, 10, 14, 23, 24]. Here, we report direct observation of Floquet-Bloch states in monolayer epitaxial graphene which was the first proposed material platform [15] for Floquet engineering. By using time- and angle-resolved photoemission spectroscopy (trARPES) with mid-infrared (mid-IR) pump excitation, we detected replicas of the Dirac cone. Pump polarization dependence of these replica bands unequivocally shows that they originate from the scattering between Floquet-Bloch states and photon-dressed free-electron-like photoemission final states, called Volkov states. Beyond graphene, our method can potentially be used to directly observe Floquet-Bloch states in other systems paving the way for Floquet engineering in a wide range of quantum materials.
Show Abstract

BAnG Bidirectional Anchored Generation for Conditional RNA Design

Roman Klypa, A. Bietti, Sergei Grudinin

Designing RNA molecules that interact with specific proteins is a critical challenge in experimental and computational biology. Existing computational approaches require a substantial amount of experimentally determined RNA sequences for each specific protein or a detailed knowledge of RNA structure, restricting their utility in practice. To address this limitation, we develop RNA-BAnG, a deep learning-based model designed to generate RNA sequences for protein interactions without these requirements. Central to our approach is a novel generative method, Bidirectional Anchored Generation (BAnG), which leverages the observation that protein-binding RNA sequences often contain functional binding motifs embedded within broader sequence contexts. We first validate our method on generic synthetic tasks involving similar localized motifs to those appearing in RNAs, demonstrating its benefits over existing generative approaches. We then evaluate our model on biological sequences, showing its effectiveness for conditional RNA sequence design given a binding protein.

Show Abstract

In-Context Denoising with One-Layer Transformers: Connections between Attention and Associative Memory Retrieval

We introduce in-context denoising, a task that refines the connection between attention-based architectures and dense associative memory (DAM) networks, also known as modern Hopfield networks. Using a Bayesian framework, we show theoretically and empirically that certain restricted denoising problems can be solved optimally even by a single-layer transformer. We demonstrate that a trained attention layer processes each denoising prompt by performing a single gradient descent update on a context-aware DAM energy landscape, where context tokens serve as associative memories and the query token acts as an initial state. This one-step update yields better solutions than exact retrieval of either a context token or a spurious local minimum, providing a concrete example of DAM networks extending beyond the standard retrieval paradigm. Overall, this work solidifies the link between associative memory and attention mechanisms first identified by Ramsauer et al., and demonstrates the relevance of associative memory models in the study of in-context learning.

Show Abstract

In-Context Denoising with One-Layer Transformers: Connections between Attention and Associative Memory Retrieval

We introduce in-context denoising, a task that refines the connection between attention-based architectures and dense associative memory (DAM) networks, also known as modern Hopfield networks. Using a Bayesian framework, we show theoretically and empirically that certain restricted denoising problems can be solved optimally even by a single-layer transformer. We demonstrate that a trained attention layer processes each denoising prompt by performing a single gradient descent update on a context-aware DAM energy landscape, where context tokens serve as associative memories and the query token acts as an initial state. This one-step update yields better solutions than exact retrieval of either a context token or a spurious local minimum, providing a concrete example of DAM networks extending beyond the standard retrieval paradigm. Overall, this work solidifies the link between associative memory and attention mechanisms first identified by Ramsauer et al., and demonstrates the relevance of associative memory models in the study of in-context learning.

Show Abstract
  • Previous Page
  • Viewing
  • Next Page
Advancing Research in Basic Science and MathematicsSubscribe to Flatiron Institute announcements and other foundation updates