---
license: cc-by-nc-4.0
base_model: nvidia/Cosmos3-Nano-Policy-DROID
pipeline_tag: robotics
datasets:
  - rooty2020/DROID-VP
tags:
  - robotics
  - world-model
  - video-generation
  - multi-view
  - point-tracking
  - droid
  - cosmos
---

<p align="center">
  <img src="https://huggingface.co/rooty2020/VPM-DROID/resolve/main/assets/vpm-icon.png" alt="VPM" width="120">
</p>

<p align="center">
  <a href="https://rootyjeon.github.io/video-point-model/"><img src="https://img.shields.io/badge/Project-Page-1f6feb.svg?logo=googlechrome&logoColor=white" alt="Project page"></a>
  <a href="https://arxiv.org/abs/2610.28594"><img src="https://img.shields.io/badge/arXiv-2610.28594-b31b1b.svg?logo=arxiv&logoColor=white" alt="arXiv"></a>
  <a href="https://github.com/rootyJeon/VPM"><img src="https://img.shields.io/badge/GitHub-Code-181717.svg?logo=github&logoColor=white" alt="GitHub"></a>
</p>

# VPM-DROID

**Video-Point Model (VPM)**, trained on [DROID](https://droid-dataset.github.io/). VPM is a multi-view video world
model built on [NVIDIA Cosmos3-Nano](https://huggingface.co/nvidia/Cosmos3-Nano). From the current observation,
this checkpoint generates, in one pass:

- 🎥 the next **16 frames of every camera**,
- 🦾 a **16-step robot action chunk** (7 joint positions + gripper),
- 📍 **point tracks** on the generated clip, from a point-track head that reads the model's own attention features
  and was trained on DROID robot point tracks.

## Model details

| | |
| :--- | :--- |
| Base model | Cosmos3-Nano |
| Setup | Video + Action + Point tracks on DROID, three cameras |
| Parameters | 15.20 B (`net.*`, 991 tensors, of which 181 / 28.6 M are the point-track head) |
| Weights | bf16 safetensors, 7 shards, 30.4 GB. Regular (non-EMA) weights, no adapters |
| Inputs | 1, 5 or 9 past frames of `exterior_1` (left), `exterior_2` (right) and `wrist` at 192 x 320, robot state, the clean past actions, language instruction. |
| Outputs | 16 generated frames per camera (15 fps), action chunk `[16, 8]`: 7 joint positions in rad + gripper; point tracks per camera with depth |


The video tokenizer (Wan2.2 VAE) and the Qwen3-VL text tokenizer are not part of this repository.


## Usage

With the VPM code, from the last 9 frames of the three cameras (`past.npz`: `frames` uint8 `[3, 9, H, W, 3]`), the
8 DROID joint actions between them (`past_actions.npy`), the robot state (7 joint angles in rad + gripper,
0 open … 1 closed) and the instruction:

```bash
bash inference/infer_droid.sh --ckpt rooty2020/VPM-DROID \
    --past-frames past.npz --past-actions past_actions.npy \
    --state q0,q1,q2,q3,q4,q5,q6,gripper --prompt "put the marker in the cup"
```

Or on an episode of [DROID-VP](https://huggingface.co/datasets/rooty2020/DROID-VP), built exactly as in training:

```bash
bash inference/infer_droid.sh --ckpt rooty2020/VPM-DROID \
    --data-root data/DROID-VP --episode 42 --frame 40
```

The outputs are the generated videos, `actions.npz` with the `[16, 8]` action chunk and a `meta.json`.
`--images ext1.png,ext2.png,wrist.png` uses the current frame only. The actions are much more accurate with the
9-frame context. `--tracks grid` also tracks a grid of points in every camera on the generated clip. The head was
trained with the camera intrinsics and extrinsics as input, so pass them with `--cameras cameras.json`
(intrinsics and world-to-camera extrinsics per view). The tracks are then also lifted to 3D. One GPU with about
40 GB of memory is enough.

To keep a local copy:

```bash
hf download rooty2020/VPM-DROID --local-dir checkpoints/VPM-DROID
```

## Files

```
model/model-0000X-of-00007.safetensors   weights (bf16, keys prefixed with net.)
model/model.safetensors.index.json       tensor -> shard map
release.json                             tensor and parameter counts, sha256 per shard, source iteration, gripper convention
LICENSE                                  CC BY-NC 4.0
LICENSE-OpenMDW-1.1.txt                  OpenMDW-1.1 agreement of the Cosmos3-Nano base model
assets/vpm-icon.png                      VPM logo shown on this card
```

## License and attribution

The fine-tuned weights in this repository are released under
[CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) (attribution: **[Byungwoo Jeon](https://rootyjeon.github.io/)**).
Non-commercial use only. For commercial licensing contact imbw2024@kaist.ac.kr.

**Built on NVIDIA Cosmos.** VPM-DROID is a derivative of
[nvidia/Cosmos3-Nano-Policy-DROID](https://huggingface.co/nvidia/Cosmos3-Nano-Policy-DROID) and its base model
[nvidia/Cosmos3-Nano](https://huggingface.co/nvidia/Cosmos3-Nano) (Copyright (c) 2026 NVIDIA CORPORATION &
AFFILIATES), which NVIDIA distributes under the OpenMDW License Agreement 1.1. As that agreement requires, a copy is
included here as [LICENSE-OpenMDW-1.1.txt](LICENSE-OpenMDW-1.1.txt) and NVIDIA's copyright notice is retained. Its
terms apply to the base-model portions of these weights, and the CC BY-NC 4.0 terms above apply to the fine-tuning
as released. The training data comes from [DROID](https://droid-dataset.github.io/), and its terms apply to the data.

## Citation

```bibtex
@article{jeon2026vpm,
  title={Learning the 4D World through Point Trajectory Forecasting},
  author={Jeon, Byungwoo and Hong, Sunghwan and Kim, Dohyeon and Lee, Jonghoon and Park, Jungwoo and Lee, Inhee and Baek, Doosan and Kim, Jinil and Lim, Taehyun and Joo, Hanbyul and Pollefeys, Marc and Xie, Saining and Shin, Jinwoo},
  journal={arXiv preprint arXiv:2610.28594},
  year={2026}
}
```