ZurichCV #15

Said Gürbüz (ETH Zurich, IBM Research): DeskForge: Controllable Desktop Environments for Training Computer-Use Agents

Computer-use agents aim to carry out everyday tasks through the same applications and interfaces that people use. Doing this reliably requires understanding what is on screen and translating that understanding into actions as the interface changes.

This talk presents DeskForge, a framework that turns real applications into controllable desktop environments. It combines configurable scenes with dense annotations of what is actually visible and records how the interface changes after each action. The talk explores the design choices behind this combination: how to create diverse desktop situations, capture screen structure under occlusion, and turn interactions into useful supervision. Results and demonstrations show how better visual grounding translates into more successful long-horizon tasks.

Lorenzo Steno (Founding Researcher, Seldon Technologies): How to build RL environments for Computer-use?

Training computer-use agents requires infrastructure where agents can interact with real software, make mistakes safely, and run experiments at scale. This talk explores how to build that infrastructure: isolating execution in sandboxes, managing permissions and network access, and creating reproducible environments that reset reliably between rollouts. We cover the practical challenges of exposing desktop controls to agents and running parallel sessions without leaking state or compromising the sandbox.