ZurichNLP #22

Tiago Pimentel (ETH Zurich): Promises and Limitations of Causality for Machine Learning Interpretability

How can we move from observing what a model does to understanding why it does it? In this talk, I argue that causality is necessary but not sufficient to uncover the mechanisms underlying model predictions. First, I examine a "macro" view of model analysis, showing how econometric tools—such as regression discontinuity or difference-in-differences—can isolate the causal impact of specific design choices, like tokenisation and training-data selection, on a model's outputs. Second, I turn to a "micro" view of mechanistic interpretability, focusing on causal abstraction as a method to establish whether a model implements a high-level algorithm. I demonstrate that this approach faces a critical limitation: without strict assumptions about how models encode information, the framework becomes vacuous, implying that any model implements any algorithm. This reveals that the ability to make counterfactual predictions about a model is not, on its own, sufficient to guarantee that we understand it. I conclude with a short discussion of how causality can be used to develop more principled interpretability methods, and with an open question about what interpretability is currently missing.

Amir Joudaki (ETH Zurich): What Must a Neural Network Remember to Learn?

Training a deep network is easy to write as an equation but hard to understand as a process: its parameters evolve together, changing both its predictions and the representations that shape future learning. Which parts of this evolving state are essential, and which can be compressed away? The question is more demanding than compressing a trained network, since two networks can agree on every input and still diverge the moment training resumes. Infinite-width theories inspired by statistical physics, such as Tensor Programs and dynamical mean-field theory, describe training through forward and backward responses and their correlations—at the price of an expanding history of past responses. The question here is: can that history be compressed while preserving depth, nonlinearity, and feature learning?

I will present a construction that replaces each neuron's response history with a finite collection of evolving moments, measured in a time coordinate tied to learning activity. These moments determine the learned interactions, while the initialized random matrices are retained exactly. We establish approximation guarantees over complete training intervals, including all-time guarantees for sufficiently small O(1) labels; for fixed depth, training data, and approximation order, the evolving state grows only linearly with width. Moreover, scaling the order jointly with the width yields a state subquadratic in width with accuracy comparable to that of the dense population predictor. Experiments show the compressed model can reproduce the function dense training selects on unseen inputs, not just the training labels. I will discuss the resulting trade-off between evolving state and accuracy, what these descriptions suggest about the variables that govern learning, and the prospect of an idealized model of deep learning.