Machine Learning at the Flatiron Institute Seminar: Giles Louppe

Title: Harness, don’t retrain: diffusion models as steerable priors for inverse problems

Abstract:
Among deep generative models, diffusion models stand out for a property that is easy to overlook: their sampling process can be steered at inference time. Because generation unfolds as a sequence of denoising steps, each step can be modified to account for observations, constraints, or other requirements, without retraining the model. A single pretrained diffusion model can thus serve as a reusable prior for a whole family of inverse problems, which is precisely what scientific applications call for: the forward model and the observations change, but the underlying physics, and hence the prior, do not. The catch is that exact steering, in the sense of sampling from the Bayesian posterior, requires the likelihood score along the diffusion path, which is intractable and typically approximated with crude assumptions. In this talk, I will present recent work from my group that makes this inference-time control both more accurate and more general: better approximations of the posterior score, sampling under hard constraints rather than soft guidance, and generating arbitrarily long trajectories of dynamical systems from a prior trained on short windows only. I will conclude with ongoing work on non-linear observation operators.