More Events
December 1–4, 2026
December 1, 2026
3–4 p.m. ET
Speaker
Vicky Kaspi
December 2, 2026
11 a.m.–12 p.m. ET
Speaker
Avi Flamholz, Ph.D.
Bringing people and ideas together to fund, conduct, and share transformative science.
Our grants advance knowledge and strengthen connections between science and society.
Discover the latest news, research, and updates.
We convene scientists, mathematicians, and thinkers to share ideas and explore new questions.
Founded by Jim and Marilyn Simons, we’re a private philanthropy advancing mathematics and basic science.
Work with us on the frontier of basic science research.
The Flatiron Institute is a research center that uses computational science to advance fields like physics, biology, and astronomy.
Search
Lecture
– ET
162 Fifth Avenue
New York, NY 10010
Title: Optimizing Data Use for Pre-training Language Models
Abstract: Modern language models are trained on massive, unstructured data consisting of trillions of tokens, typically obtained by crawling the web. In this talk, Danqui argues that we are still in the early stages of understanding pre-training data and unlocking its full potential, and that more effective use of data can lead to both more compute-efficient and more capable language models. Danqui will present several perspectives on improving data curation, focusing on three general techniques. First, quality filtering aims to train classifiers that can distinguish high-quality from low-quality documents at scale (QuRating). Second, domain curation focuses on developing taxonomies of web data and leveraging domain mixing strategies to enhance pre-training (WebOrganizer). Third, Danqui will introduce a simple pre-training approach that conditions on metadata, which both accelerates training and improves model steerability (MeCo). Together, these efforts highlight the importance of optimizing the use of pre-training data and point toward a more data-centric paradigm for training future language models.
December 1–4, 2026
December 1, 2026
3–4 p.m. ET
Speaker
Vicky Kaspi
December 2, 2026
11 a.m.–12 p.m. ET
Speaker
Avi Flamholz, Ph.D.