Machine Learning at the Flatiron Institute Seminar: Maximilian Nickel

Title: Epistemic Limits of Model Validation in Complex Social Systems

Abstract: AI has undergone a dramatic paradigm shift, not only in terms of the impressive capabilities of state-of-the-art models, but also in terms of how they are trained, deployed, and evaluated. Most importantly, AI systems do not exist in a controlled environment anymore (e.g., meticulously collected i.i.d. samples), but interact continuously with social systems, e.g., through training and evaluation data as well as their direct influence on social processes. Crucially, the validity of even our most basic machine learning methods is not guaranteed in this new context. Yet, without valid methodology we cannot ensure the intended outcomes of deployed AI systems nor continue to advance AI research in a scientifically sound way.

In this talk, I will therefore argue that we need new theoretical foundations for machine learning and AI that explicitly account for the complex social system with which an AI system interacts or in which it is situated. I will discuss this on the example of the ubiquitous train-test paradigm. While this form of model validation has arguably been one of the single most important contributors to the breathtaking progress in AI, I will show via rigorous impossibility results that it is not valid anymore for key tasks in modern AI under current data collection practices. Based on these insights, I will also introduce a novel cooperative approach to data collection with strong game-theoretical guarantees that can alleviate these issues. I will conclude this talk with a call for increased interdisciplinary work at the intersection of AI theory, methods, and society.