Entropic Forces From Noisy Training

Colloquium

Speaker

  • David Schwab headshot.
    David Schwab Northwestern University

When

  • – ET

Where

Virtual

160 Fifth Avenue

New York, NY 10010

Abstract

Stochastic gradient descent (SGD) forms the core optimization method for deep neural networks. While some theoretical progress has been made, it still remains unclear why SGD leads the learning dynamics in overparameterized networks to solutions that generalize well. Here we show that for overparameterized networks with a degenerate valley in their loss landscape, SGD on average decreases the trace of the Hessian of the loss. We also generalize this result to other noise structures and show that isotropic noise in the non-degenerate subspace of the Hessian decreases its determinant. Next, we turn to the question of characterizing and finding optimal representations for supervised learning. Traditionally, this question has been tackled using the Information Bottleneck, which compresses the inputs while retaining information about the targets, in a decoder-agnostic fashion. We propose the Decodable Information Bottleneck (DIB) that considers information retention and compression from the perspective of the desired predictive family. Empirically, we show that the framework can be used to enforce a small generalization gap on downstream classifiers and to predict the generalization ability of neural networks.

David Schwab: Entropic Forces From Noisy Training
A red, wireframe, spherical shape with wavy edges is centered on a light beige background, resembling a 3D mesh or abstract digital sculpture.

Event Series

CCA Colloquium

A weekly forum in computational astrophysics

Guest lectures on forefront astrophysical research topics aimed at non-expert professional physicists.

See events

More Events

Computational Astrophysics

Gaia Relay: NYC

Workshop

December 1–4, 2026

Computational Astrophysics

Fast Radio Bursts

Colloquium

December 1, 2026

3–4 p.m. ET

Speaker

Vicky Kaspi

December 11, 2026

3–4 p.m. ET

Speaker

Kyle Kremer