ZurichNLP #14

Jonas Hübotter from ETH Zurich on Efficiently Learning at Test-Time with LLMs: The standard paradigm of machine learning separates training and testing. Training aims to learn a model by inductively extracting general rules from data, and testing applies this model to new, unseen data. We investigate an alternative transductive paradigm where the model is fine-tuned at test-time specifically to the given task. Our evaluation on the Pile dataset indicates that this paradigm can improve language modeling on a wide range of tasks. We identify the key challenge of deciding which data to select for test-time fine-tuning and show that the previously used Nearest Neighbor retrieval is ill-suited since it tends to select redundant data. To address this, we introduce SIFT, a data selection algorithm designed to reduce uncertainty about the model's response given a prompt, which unifies ideas from retrieval and active learning. Whereas Nearest Neighbor retrieval typically fails in the presence of information duplication, SIFT accounts for information duplication and optimizes the overall information gain of the selected examples.

Imanol Schlag from ETH Zurich on The Swiss AI LLM Effort: Building Transparent and Responsible AI for Switzerland and Beyond:

This talk will present the Swiss AI Large Language Model (LLM) Effort, a collaborative initiative led by ETH Zurich and EPFL. We aim to develop a 70B parameter LLM trained on 10+ trillion high-quality, multilingual tokens, with a focus on transparency, ethical alignment, and support for Swiss national languages. We'll share technical insights into our training approach on the Alps supercomputer, including our strategies for efficient scaling across thousands of GH200 GPUs. Our commitment to open science and legal compliance forms the foundation for disseminating AI technology across Swiss academia and industry. We'll discuss how this effort can drive innovation in various sectors, exemplified by projects like Meditron for healthcare and Ethel for education while providing a strong general-purpose AI foundation for Switzerland.