VLX-Go
Vision-Language Short-Horizon Waypoint Prediction for Embodied Navigation
## Overview
VLX-Go is a lightweight vision-language waypoint planner for embodied navigation. Given recent monocular frames, the current observation, and a natural-language instruction, it predicts short-horizon local waypoints that can be executed by a downstream controller in simulation or on a robot.
Rather than relying on a general-purpose VLM to describe the scene or produce text-only actions, VLX-Go maps the visual-language state directly to a compact waypoint interface. This repository focuses on target following, local navigation, dynamic obstacle avoidance, and closed-loop evaluation.
VLX-Go builds on the technical direction of [OmTrackVLA](https://github.com/om-ai-lab/OmTrackVLA) and extends it toward lightweight waypoint prediction and closed-loop navigation research.
> [!TIP]
>
> **🚀 Try [VLX](https://om-agent.com/)** here and explore how it enables machine to perceive, reason, and act in real time.
## Community
Join the VLX community to connect with developers, explore applications, share feedback, and shape the future of multimodal AI.
Official WeChat |
Discord Community |
|
|
For technical support, partnerships, and community inquiries, contact us at **[marketing@hzlh.com](mailto:marketing@hzlh.com)**.
## Highlights
- **0.6B waypoint planner:** A lightweight architecture for lower inference cost and easier deployment.
- **Instruction-conditioned planning:** Natural-language instructions specify task intent, including following, reaching, and obstacle avoidance.
- **Short-horizon waypoint output:** The model predicts local motion targets instead of full global routes or text-only responses.
- **Temporal visual context:** Recent frames help capture target motion, occlusion changes, and scene dynamics.
- **Closed-loop evaluation:** Navigation is evaluated through repeated observation, prediction, execution, and feedback.
## Task
At each time step, VLX-Go solves a receding-horizon waypoint prediction problem.
**Input**
- Recent visual history: `H_t = {I_{t-k}, ..., I_{t-1}}`
- Current frame: `I_t`
- Instruction: `q`, for example, "follow the target person and avoid obstacles"
**Output**
- Short-horizon waypoint sequence: `W_t = {w_1, ..., w_T}`
Here, each `w_i` denotes a local motion target, such as position, heading, or another waypoint representation consumed by the controller. The exact dimensionality depends on the dataset and control interface.
```text
history frames + current frame + instruction
|
v
VLX-Go waypoint planner
|
v
short-horizon waypoints -> controller / simulator
```
## Method
VLX-Go separates waypoint planning from platform-specific low-level control. The planner predicts short-horizon local goals, while the downstream controller handles velocity commands, safety constraints, and dynamics.
| Stage | Description |
| --- | --- |
| Visual encoding | Encode the current frame and recent visual history into visual features |
| Language conditioning | Use the instruction as the task condition for planning |
| Waypoint prediction | Predict short-horizon local motion targets with a 0.6B planner |
| Closed-loop execution | Execute predicted waypoints, collect the next observation, and predict the next segment |
This rolling-horizon design is practical for dynamic scenes: targets may move, obstacles may enter the camera view, and earlier predictions can be corrected with new observations.
## Training
VLX-Go is first trained from offline trajectory data and can then be refined with online simulator feedback.
| Phase | Data / Signal | Objective |
| --- | --- | --- |
| Offline trajectory learning | Demonstration trajectories, video frames, language instructions | Learn target-following and local waypoint generation |
| Online optimization | Simulator feedback, collision signals, target state, reward signals | Improve robustness to occlusion, obstacles, and closed-loop drift |
Typical supervised objectives include waypoint regression, trajectory direction loss, optional velocity or action auxiliary loss, and smoothness regularization. The online stage complements supervised learning by exposing the policy to execution-time feedback that offline trajectory data may not cover.
## Evaluation
VLX-Go is evaluated on the STT task of EVT-Bench.
| Model | Parameters | STT SR ↑ | STT TR ↑ | STT CR ↓ |
| --- | --- | ---: | ---: | ---: |
| TrackVLA | 7B | 85.1 | 78.6 | **1.65** |
| NavFoM | 7B | 85.0 | 80.5 | - |
| Qwen-RobotNav-4B | 4B | 77.4 | 90.0 | 6.4 |
| Qwen-RobotNav-8B | 8B | 78.6 | 89.7 | 5.7 |
| **VLX-Go** | **0.6B** | **85.42** | **94.08** | 6.55 |
**Metrics:** SR is success rate, TR is tracking rate, and CR is collision rate. At the 0.6B scale, VLX-Go achieves strong success and tracking rates. Further reducing collision rate remains an important direction, especially through simulator, reward, controller, and safety-constraint tuning.
## Dataset
Coming soon.