Publications

In magic angle twisted bilayer graphene, transport, thermodynamic and spectroscopic experiments pinpoint at a competition between distinct low-energy states with and without electronic order. We use Dynamical Mean Field Theory (DMFT) on the topological heavy Fermion (THF) model of twisted bilayer graphene to investigate the emergence of electronic correlations and long-range order in the absence of strain. We contrast moment formation, Kondo screening and ordering on a temperature basis and explain the nature of emergent correlated states based on three central phenomena: (i) the formation of local spin and valley isospin moments around 100K, (ii) the ordering of the local isospin moments around 10K preempting Kondo screening, and (iii) a cascadic redistribution of charge between localized and delocalized electronic states upon doping. At integer fillings, we find that low energy spectral weight is depleted in the symmetric phase, while we find insulating states with gaps enhanced by exchange coupling in the zero-strain ordered phases. Doping away from integer filling results in distinct metallic states: a "bad metal" above the ordering temperature, where scattering off the disordered local moments suppresses electronic coherence, and a "good metal" in the ordered states with coherence of quasiparticles facilitated by isospin order. This finding reveals coherence from order as the microscopic mechanism behind the Pomeranchuk effect observed experimentally. Upon doping, there is a periodic charge reshuffling between localized and delocalized electronic orbitals leading to cascades of doping-induced Lifshitz transitions, local spectral weight redistributions and periodic variations of the electronic compressibility. Our findings provide a unified understanding of the most puzzling aspects of scanning tunneling spectroscopy, transport, and compressibility experiments.

The phase diagrams of quasi two-dimensional organic superconductors display a plethora of fundamental phenomena associated with strong electron correlations, such as unconventional superconductivity, metal-insulator transitions, frustrated magnetism and spin liquid behavior. We analyze a minimal model for these compounds, the Hubbard model on an anisotropic triangular lattice, using cutting-edge quantum embedding methods respecting the lattice symmetry. We demonstrate the existence of unconventional superconductivity by directly entering the symmetry-broken phase. We show that the crossover from the Fermi liquid metal to the Mott insulator is associated with the formation of a pseudogap. The predicted momentum-selective destruction of the Fermi surface into hot and cold regions provides motivation for further spectroscopic studies. Our results are in remarkable agreement with experimental phase diagrams of κ-BEDT organics.

We study the interplay between intrinsic topological order and superconductivity in a two-component Haldane bilayer, where the two layers are coupled by an attractive force. We obtain the phase diagram of the model with exact diagonalization in finite size, and develop arguments to assess the stability of the observed phases in the thermodynamic limit. Our main result is that a finite critical attraction strength is needed to pair fermions forming a fractional topological order. This behavior can be harnessed to create clean interfaces between a fractional topological insulator and a superconductor by gating, wherein non-Abelian parafermionic modes are trapped. We discuss realization of such interfaces in the bulk of double bilayers of transition metal dichalcogenides by inhomogenous electrostatic gating, which should mitigate the spurious effects of disorder and crystalline defects present on physical edges.

Parametric resonances and amplification have led to extraordinary photoinduced phenomena in pump-probe experiments. While these phenomena manifest themselves in out-of-equilibrium settings, here, we present the striking result of parametric amplification in equilibrium. In particular, we demonstrate that quantum and thermal fluctuations of a Raman-active mode amplifies light inside a cavity, at equilibrium, when the Raman mode frequency is twice the cavity mode frequency. This noise-driven amplification leads to the creation of an unusual parametric Raman polariton, intertwining the Raman mode with cavity squeezing fluctuations, with smoking gun signatures in Raman spectroscopy. In the resonant regime, we show the emergence of not only quantum light amplification but also localization and static shift of the Raman mode. Apart from the fundamental interest of equilibrium parametric amplification our study suggests a resonant mechanism for controlling Raman modes and thus matter properties by cavity fluctuations. We conclude by outlining how to compute the Raman-cavity coupling, and suggest possible experimental realization

It has been reported that upon doping a Mott insulator, there can be a crossover to a pseudogaped metallic phase followed by a first-order transition to another thermodynamically stable metallic phase. We call this first-order metal-metal transition the Sordi transition. It was argued that the initial reports of Sordi transitions at finite temperature could be explained by finite size effects and biases related to the model and method used. In this work, we report the Sordi transition on larger clusters at finite temperature on a triangular lattice, where long-range antiferromagnetic fluctuations are frustrated, using a different method, the dynamical cluster approximation instead of the cellular dynamical mean-field theory. This demonstrates that this first-order transition is a directly observable transition in doped Mott insulators and that it is relevant for experiments on candidate spin-liquid organic materials.

In magic angle twisted bilayer graphene, transport, thermodynamic and spectroscopic experiments pinpoint at a competition between distinct low-energy states with and without electronic order. We use Dynamical Mean Field Theory (DMFT) on the topological heavy Fermion (THF) model of twisted bilayer graphene to investigate the emergence of electronic correlations and long-range order in the absence of strain. We contrast moment formation, Kondo screening and ordering on a temperature basis and explain the nature of emergent correlated states based on three central phenomena: (i) the formation of local spin and valley isospin moments around 100K, (ii) the ordering of the local isospin moments around 10K preempting Kondo screening, and (iii) a cascadic redistribution of charge between localized and delocalized electronic states upon doping. At integer fillings, we find that low energy spectral weight is depleted in the symmetric phase, while we find insulating states with gaps enhanced by exchange coupling in the zero-strain ordered phases. Doping away from integer filling results in distinct metallic states: a "bad metal" above the ordering temperature, where scattering off the disordered local moments suppresses electronic coherence, and a "good metal" in the ordered states with coherence of quasiparticles facilitated by isospin order. This finding reveals coherence from order as the microscopic mechanism behind the Pomeranchuk effect observed experimentally. Upon doping, there is a periodic charge reshuffling between localized and delocalized electronic orbitals leading to cascades of doping-induced Lifshitz transitions, local spectral weight redistributions and periodic variations of the electronic compressibility. Our findings provide a unified understanding of the most puzzling aspects of scanning tunneling spectroscopy, transport, and compressibility experiments.

We investigate the application of matrix product state (MPS) representations of the influence functionals (IFs) for the calculation of real-time equilibrium correlation functions in open quantum systems. Focusing specifically on the unbiased spin-boson model, we explore the use of IF-MPSs for complex time propagation, as well as IF-MPSs for constructing correlation functions in the steady state. We examine three different IF approaches: one based on the Kadanoff–Baym contour targeting correlation functions at all times, one based on a complex contour targeting the correlation function at a single time, and a steady state formulation, which avoids imaginary or complex times, while providing access to correlation functions at all times. We show that within the IF language, the steady state formulation provides a powerful approach to evaluate equilibrium correlation functions.

Behavior changes over the course of learning a task. This behavioral change is due to shifts in neural responses that support improved performance. Here, we investigated how the underlying representational geometries in visual areas V4 and IT of the Macaque visual system change during a categorization learning task. Visual stimuli varied in two independent attributes, and monkeys learned to categorize them based on a category boundary in the stimulus space that was defined by a combination of the attributes. Chronic neural population recordings were obtained from V4 and IT over multiple days of training while a monkey learned the task through receiving correct/incorrect feedback. Additionally, we recorded from the same neural populations while a monkey performed a fixation task viewing the same sets of stimuli. In all eight analyzed tasks, the monkey’s performance on the categorization task improved with training. To link this behavioral improvement to the underlying population responses, we investigated how the geometry of neural population activity changed over the course of learning. We treated population responses to all stimuli in each of the two categories as manifold-like representations, and analyzed the geometric properties of these representations using mean-field theoretic manifold capacity analysis. As the monkey learned the task, we observed that the representations in both V4 and IT for the two classes became more separable as measured by an increase in manifold capacity. This increase in capacity was associated with a characteristic geometric change in the neural population response geometry. Our results suggest that both V4 and IT responses actively change during category learning in ways that directly lead to increased separability and improved readouts for downstream neural areas, and point towards future work linking these population-level geometric changes to local changes at the single-neuron level.

All organisms make temporal predictions, and their evolutionary fitness generally depends on the accuracy of these predictions. Understanding what structure enables computing temporal predictions in complex natural scenarios is central to computational neuroscience and machine learning. This thesis focuses on visual processing and describes a unified approach for the representation and estimation of dynamic visual signals, as computed with neural elements in brains or machines. We propose that temporal prediction can serve as a general objective function, both for unsupervised learning of visual representations and for estimat- ing probable future frames under uncertainty. Optimizing for next-frame prediction leverages the order of time, especially visual motion, and extracts predictive information from image sequences, without requiring labeled data. The architecture of a predictive system plays a critical role and we hypothesize that it should reflect the symmetry properties of the physical world. Specifically, a model that discovers the transformations acting in a visual scene should exploit these transformations to predict accurately and should remain agnostic when these transformations are ambiguous. Such interpretable models can serve as guides to explain the responses of neurons in sensory cortices and their functional role in visual perception.
We first describe the empirical structure of dynamic visual scenes and then develop a mathematical theory for exploiting that structure. The movement of observers and objects creates distinct temporal structures in visual signals, allowing for partial prediction of future signals based on past ones. Motivated by group representation theory, we propose a method to discover and utilize the transformation structures of image sequences and show that local phase measurements play a fundamental role. The proposed model extrapolates visual signals in a local polar representation, this representation is learned via next-frame prediction. This polar prediction model successfully recovers simple transformations in synthetic datasets and scales to natural image sequences. The architecture is simple yet effective: it contains a single hidden stage with one non-linearity that factorizes slow form and steady motion signals. We demonstrate that polar prediction achieves better prediction performance than traditional approaches based on motion compensation, and that it rivals conventional deep networks trained on prediction.

We then confront the inherent uncertainty of visual temporal prediction and develop a framework for learning and sampling the conditional density of the next frame given the past few observed frames. Casting prediction as a probabilistic inference problem is motivated by the need to cope with ambiguity in natural image sequences. We describe a regression- based framework that implicitly estimates the distribution of the next frame, effectively learning a conditional image density from high-dimensional signals. This is achieved with a simple resilience-to-noise objective function: a deep neural network is trained to map to past conditioning frames and a noisy observation of the next frame to an estimated denoised next frame. The network is trained over a range of noise levels without access to that noise level, i.e., it is blind and universal. This denoising objective has the desirable property of being local in the space of densities, and training across noise levels forces the network to extract information about the stable underlying distribution of probable next frame given past conditioning. We consider synthetic image sequences composed of moving disks that occlude each other and demonstrate that trained networks can handle challenging cases of bifurcating temporal trajectories -- effectively choosing one occlusion or another when the observation is ambiguous. Furthermore, local linear analysis of a network trained on natural image sequences reveals that the model automatically weights evidence by reliability: the model integrates information from past conditioning and noisy observation, adapting to the amount of predictive information in the conditioning, and the noise level in the observation.

Finally, we discuss the implications of this work for understanding biological vision. Starting from the polar prediction model, we derive a circuit algorithm composed of local neural computations for predicting natural image sequences. The circuit is composed of canonical computational elements that have received ample physiological evidence: its components resemble the normalized simple and direction-selective complex cell models of primate V1 neurons. Unlike the polar prediction model, this circuit algorithm does not impose a polar factorization, instead, it lets complex-cell-like units learn a combination of quadratic responses. Furthermore, we outline a method for gradually extracting slower and more abstract features by cascading this biologically plausible mechanism. These models offer a normative framework for understanding how the visual system represents sensory inputs in a form that simplifies temporal prediction. Together, our work on visual temporal prediction builds connections between computational modeling and brain sciences. These connections can guide the design and analysis of physiological and perceptual experiments, and can also motivate further developments in machine learning.

Many problems in image processing and computer vision rely, explicitly or implicitly, on statistical density models. Describing the full density of natural images, 𝑝(𝑥), is a daunting problem given the dimensionality of the signal space. Traditionally, models have been developed by combining assumed symmetry properties, with simple parametric forms, often within pre- specified transformed coordinate systems. While these models have led to steady advances in problems such as denoising, they are too simplistic to generate complex features that occur in our visual world.
Deep neural networks have provided state-of-the-art solutions for problems such as denoising, which implicitly rely on a prior probability model of natural images. Here, we first develop a robust and general methodology for extracting the prior. We rely on a statistical result due to Tweedie (1956) and Miyasawa (1961), who showed that the least-squares solution for removing additive Gaussian noise can be written directly in terms of the gradient of the log of the noisy signal density. We use this fact to develop a stochastic coarse-to-fine gradient ascent procedure for drawing high-probability samples from the implicit prior embedded within a neural network trained to perform blind (i.e., unknown noise level) least-squares denoising. This algorithm is similar to score-based diffusion framework, yet different in several ways.

Unlike the classical framework, we do not have direct access to the learned density, which gives rise to a crucial question: what is the prior? The rest of the thesis focuses on understanding and using this prior.

At the core of our coarse-to-fine gradient ascent sampling algorithm is a deep neural network (DNN) denoiser. Despite their success, we lack an understanding of the DNN denoiser mechanisms and more importantly what priors are being learned by these models. In order to make the DNN denoiser interpretable, we remove all network biases (i.e. additive constants), to enforce the denoising mapping to become locally linear. This architecture lends itself to local linear algebraic analysis through the Jacobian of the denoising map, which provides a high level interpretability. A desired side effect of locally linear models is that they generalize automatically across noise levels.

Next, we study the continuity of the implicit image prior. We design an experiment to investigate whether the prior interpolates between the training examples or consists of a discrete set of delta functions corresponding to a memorized set of training examples. We find that for small datasets, the latter is the case. But with large enough datasets, the network generalizes beyond training examples, evidenced by high quality novel generated samples. Surprisingly, we observe that, for large enough datasets, two models trained on non-overlapping subsets of a dataset learn nearly the same density. We analyze the learned denoising functions and show that the inductive biases give rise to a shrinkage operation in a basis adapted to the underlying image. Examination of these bases reveals oscillating harmonic structures along contours and in homogeneous regions. We demonstrate that trained denoisers are inductively biased towards these geometry-adaptive harmonic bases.

Having established that a DNN denoiser can generalize, we employ the learned image density to study the question of low-dimensionality of image priors. The goal is to exploit image properties to factorize the density into low dimensional densities, thereby reducing the number of parameters and training examples. To this end, we develop a low-dimensional probability model for images decomposed into multi-scale wavelet sub-bands. The image probability distribution is factorized as a product of conditional probabilities of its wavelet coefficients conditioned by coarser scale coefficients. We assume that these conditional probabilities are local and stationary, and hence can be captured with low-dimensional Markov models. Each conditional score can thus be estimated with a conditional CNN (cCNN) with a small receptive field (RF). The effective size of Markov neighborhoods (i.e. the size w.r.t to the grid size) grows from fine to coarser scales. The score of the coarse-scale low-pass band (a low-resolution version of the image) is modeled using a CNN with a global RF, enabling representation of large-scale image structures and organization. We evaluate our model and show that locality and stationarity assumptions hold for conditional RF sizes as small as 9 × 9 without harming performance. Thus, high-dimensional score estimation for images can be reduced to low-dimensional Markov conditional models, alleviating the curse of dimensionality.

Finally, we put the denoiser prior into use. A generalization of the coarse-to-fine gradient ascent sampling algorithm to constrained sampling provides a method for using the implicit prior to solve any linear inverse problem, with no additional training. We demonstrate the generality of the algorithm by using it to produce high-quality solutions in multiple applications, such as deblurring, colorization, compressive sensing, and super resolution.