OS-Helix: Unifying GUI Interaction and Code Execution in Native Computer-Using Agents
Abstract
OS-Helix is an end-to-end trained multimodal agent that integrates GUI actions and code execution via hybrid supervised and reinforcement learning, achieving substantial gains on desktop automation benchmarks.
Real desktop automation is neither purely visual nor purely programmatic. Agents must understand user-facing interfaces while efficiently manipulating files, scripts, and system state. However, native GUI/code coordination cannot be obtained by prompt-level tool composition. In our preliminary experiments, naively adding a code tool degrades a GUI agent, revealing that GUI/code routing must be internalized through training on high-quality hybrid data. We present OS-Helix, an end-to-end trained multimodal computer-using agent that unifies GUI actions and code execution in one action space. To construct the required supervision, we introduce application-artifact composition, pairing desktop applications with authentic file artifacts to instantiate executable Ubuntu states, and a decoupled instruction/evaluator generation pipeline that produces environment-grounded tasks whose final states can be verified regardless of the solution modality. We cold-start the policy with SFT on filtered hybrid data and further improve it through online RL in live desktop environments. For RL, we propose Step-balanced GSPO, which aggregates importance weight by action step rather than raw assistant tokens to prevent long code actions from dominating heterogeneous-action optimization. On OSWorld-Verified, OS-Helix-27B and OS-Helix-35B-A3B achieve relative improvements of 35.0% and 33.7% over their matched Qwen3.5 hybrid baselines, while OS-Helix-27B-RL further improves by 42.9% to reach 66.44% success. Ablations provide evidence consistent with contributions from learned GUI/code coordination, scalable verifiable supervision, and action step balanced optimization, while demonstrating cross-platform transfer on WindowsAgentArena. Our project homepage is at https://os-copilot.github.io/OS-Helix/.
Community
Hi everyone! We introduce OS-Helix, a computer-using agent trained to coordinate GUI interaction and code execution within a single policy. Desktop tasks often need both: the GUI exposes what users see, while code can work precisely with files and system state. Simply giving a GUI agent a code tool does not teach it when to use each one, so we train that coordination directly.
We build reproducible desktop environments by pairing applications with authentic artifacts, generate verifiable tasks, and use filtered GUI/code trajectories for supervised training. We then train online with Step-balanced GSPO, which addresses optimization imbalance between GUI and code actions. OS-Helix-27B-RL reaches 66.44% on OSWorld-Verified and 63.87% on WindowsAgentArena, without Windows-specific training data.
Project page · Models · Dataset. We’re organizing the training and evaluation code for release. We’d welcome your feedback and questions!
Models citing this paper 3
YYangzzzz/OS-Helix-35B-A3B
Datasets citing this paper 1
YYangzzzz/OS-Helix-Dataset
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper