ZurichNLP #20

Fabian Schaipp (Inria): All Quiet on the Optimization Front? Recent Trends in Training Algorithms for ML

Fine-tuned training recipes remain a core challenge for crafting state-of-the-art machine learning models. In this talk, we will give an overview of recent developments around optimization and training algorithms. In particular, this will cover (i) new optimization methods (Muon, SOAP, ...) challenging the reign of AdamW and (ii) hyperparameter tuning and scaling laws (learning-rates and schedules, muP, ...) for efficient training at scale.

Valentina Pyatkin (Allen Institute, ETH): Open LLM Post-Training: Lessons from Training Olmo and Tülu

Drawing on the OLMo and Tülu projects, which openly release not just model weights but data, code, and training decisions, this talk shares lessons about what works, what doesn't, and what we still don't understand about the post-training of language models.