arxiv:2610.31162

HiSTEVE 1.0: Harnessing Human-Interface VLA Agents for Open-World Games

Published on Oct 22

· Submitted by

Zecong Tang on Oct 26

#3 Paper of the day

Tencent

Upvote

Authors:

Zian Guan ,

Abstract

HiSTEVE is a hierarchical vision-language-action agent that combines long-horizon planning with native keyboard-and-mouse control to perform complex open-ended tasks in Minecraft.

Generated by thinkingmachines/Inkling-Small

Building virtual embodied agents that operate in open-ended games like human players is a long-standing goal requiring both long-horizon autonomy and reliable native-interface control. As a procedurally generated open world, Minecraft provides a natural testbed for studying the integration of these capabilities. Existing agents typically provide either native visuomotor control without long-horizon autonomy or high-level planning executed through environment APIs or predefined skills. To bridge this gap, we propose HiSTEVE, a hierarchical VLA agent with a capability-aware agent harness. At the orchestration level, we equip the harness with structured memory for tracking progress and a VLM planner for selecting subtasks within the VLA's capabilities. To reliably determine subtask completion, we further introduce a policy-coupled task completion state detector that conditions on VLA policy context to signal candidate completion. At the control level, a single VLA executes all subtasks with vision-only perception and native keyboard-and-mouse actions. To mitigate interference between heterogeneous GUI and non-GUI interactions, we introduce regime-specific RMSNorm branches and separate mouse-movement token embeddings. We also train the VLA through world knowledge injection, imitation learning, and reinforcement learning with environment feedback to improve reliability and recovery. HiSTEVE achieves state-of-the-art atomic-task performance, scoring 85.0% on MineStudio Bench and a 37.4% average success rate on OpenHA, while also completing long-horizon tasks such as crafting an iron pickaxe from scratch like human players.

View arXiv page View PDF Add to collection

Community

zecongtanghku

Paper submitter about 20 hours ago

🚀 We are excited to share our latest work: "HiSTEVE 1.0: Harnessing Human-Interface VLA Agents for Open-World Games"!

Can an AI agent autonomously complete long-horizon tasks in Minecraft using only visual observations and native keyboard-and-mouse controls, just like a human player?

Existing Minecraft agents typically focus on either native visuomotor control without long-horizon autonomy or high-level planning that relies on environment APIs and predefined skills. To this end, we introduce HiSTEVE 1.0, a hierarchical Vision-Language-Action (VLA) agent that integrates a capability-aware agent harness with a native-control VLA policy, enabling autonomous planning and execution in Minecraft without privileged game states or environment APIs.

🔑 Key Highlights

🧠 Capability-Aware Agent Harness: We develop a hierarchical orchestration framework that combines capability-constrained planning, structured memory, and evidence-driven execution recovery. A policy-coupled task completion detector further enables the agent to determine when to advance, recover, or replan during long-horizon execution.
🎮 Reliable Native-Interface Control: We design a VLA policy that handles both first-person gameplay and GUI interactions through native keyboard-and-mouse actions. To mitigate interference between these heterogeneous interaction regimes, we introduce regime-specific RMSNorm branches and separate mouse-movement token embeddings, together with a three-stage training strategy combining world knowledge injection, imitation learning, and reinforcement learning.
📈 Extensive Experimental Validation: HiSTEVE achieves state-of-the-art atomic-task performance, reaching 85.0% on MineStudio Bench and 37.4% average success on OpenHA. The complete agent also successfully executes long-horizon tasks such as crafting an iron pickaxe from scratch using only the human interface.

📄 Paper: https://arxiv.org/abs/2610.31162

We welcome feedback and discussions!