ZeroVLN: Learning to Navigate without Supervised Fine-Tuning of the VLM Brain
Abstract
ZeroVLN is a slow-fast vision-language navigation framework that avoids navigation-domain supervised fine-tuning of the VLM brain by using a semantic tool-calling interface and closed-loop reinforcement learning to coordinate a language-based brain with an action expert, improving VLN-CE performance while preserving general visual capabilities.
Representative vision-language navigation (VLN) methods typically couple a vision-language model (VLM) brain with a learnable action head and fine-tune the VLM brain on navigation-domain data to predict geometric goals. However, such adaptation can substantially alter base VLM weights and degrade general visual capabilities. We introduce ZeroVLN, a slow-fast navigation framework that achieves effective navigation with zero navigation-domain SFT of the VLM brain. Rather than teaching the brain to predict geometric goals, we equip it with a semantic tool-calling interface for conveying movement decisions to an action expert in natural language. We first adapt the action expert to carry out these movement requests, then jointly improve the brain's movement decisions and the action expert's continuous execution through closed-loop reinforcement learning (RL). Trained on the same navigation episodes, ZeroVLN substantially outperforms representative SFT-based hierarchical VLN methods on standard VLN-CE (continuous environments) benchmarks, while inducing only minor changes to the brain's weights and largely preserving its general capabilities. To our knowledge, ZeroVLN is the first slow-fast VLN framework with zero navigation-domain SFT of the VLM brain, and the first to jointly optimize its slow and fast policies through closed-loop agentic RL in VLN-CE.
Community
We introduce ZeroVLN,the first slow–fast vision-language navigation framework with zero navigation-domain supervised fine-tuning of the VLM brain, and the first to jointly optimize the brain and a continuous-action expert through closed-loop agentic reinforcement learning in continuous navigation environments.
Our central insight is to minimize what the brain must relearn to navigate. The brain conveys movement decisions through natural-language tool calls, and a trainable action expert translates them into continuous motion. We first construct reference trajectories from the base brain’s own movement requests and supervise only the action prefix executed before each replan.
We then develop a closed-loop agentic RL algorithm that jointly optimizes the brain’s semantic tool calls and the expert’s continuous actions. The algorithm combines complementary process and outcome rewards and uses a shared trajectory-level advantage to update both policies across all interaction rounds, with separate gradient paths for the brain and the expert.
With the same backbone and training episodes, ZeroVLN achieves higher success rates than NaVILA and DualVLN on all three benchmarks, with only 0.01497% relative VLM weight displacement, while largely preserving the base brain’s general visual capabilities.
Get this paper in your agent:
hf papers read 2611.26414 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 1
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper