ZurichNLP #15

Matej Jusup: Mastering Board Games by External and Internal Planning with Language Models

This talk showcases the ability of Large Language Models (LLMs) to achieve advanced strategic planning in complex domains (Chess, Chess960, Hex, and Connect Four). We focus on reaching Grandmaster-level performance in chess while operating closer to the human search budget. To achieve this goal, we introduce, compare, and contrast two major approaches: In external search, our LLM guides Monte Carlo Tree Search rollouts and evaluations without calls to an external game engine, and in internal search, LLM is trained to generate in-context a linearized tree of search and a resulting final choice.

Bobby He: Outlier Features in LLMs: what are they, why do they matter, and how can we prevent them? Outlier Features (OFs) are widely documented to emerge in LLM training and have the undesirable effect of hindering quantisation, and ensuing efficiency gains, in afflicted models. Despite their practical importance, little is known about OFs: are they a regrettable necessity, or simply a consequence of suboptimal design choices? In this talk we'll show the latter, by identifying key architectural and optimiser choices that lead to OFs and presenting alternative options that mitigate them, without compromising training convergence.