ZurichNLP #23

Last Translation Benchmark

Vilém Zouhar (ETH Zurich)

For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. Standard benchmarks for machine translation are saturated and metrics unreliable, opaque, and vulnerable to reward-hacking. This prevents us from tracking progress in the field and identifying pathways for improvement. We introduce the Last Translation Benchmark, a collection of human-authored and peer-reviewed examples (texts, images, audio, videos) that break leading machine translation models. Each example comes with handcrafted verification rules describing concrete failure cases on that example, therefore allowing reliable and actionable future evaluation. The Last Translation Benchmark is a live dataset that accepts ongoing contributions.

Abstract: Last Translation Benchmark, Vilém Zouhar et al., CC BY 4.0.

Going Low-Bit Without Going Bananas: Pushing the Limits of Efficient Generative AI

Philippe Bich (Huawei Research)

How far can we push numerical precision down without compromising model quality? In this talk, we'll explore the challenges of low-bit inference across generative models, from LLMs to image, video, and audio generation. Drawing on our lab's latest research, we'll discuss how carefully designed transformations can make 4-bit weights and activations practical, and how to translate these advances into real inference speedups, bringing frontier AI models within reach of more accessible hardware.