README.md

---
license: other
license_name: stability-ai-community-and-cc-by-nc-4.0
license_link: https://huggingface.co/jaibrdhn/depthworld#license
tags:
  - robotics
  - action_conditioned_video_model
  - video-generation
  - depth
  - world-model
base_model: stabilityai/stable-video-diffusion-img2vid
---

<div align="center">
  <h2>DepthWorld: 3D World Model for Robot Manipulation</h2>
  <p><b>Jai Bardhan, Josef Šivic, Vladimír Petrík</b></p>
  <p>Czech Institute of Informatics, Robotics and Cybernetics (CIIRC), Czech Technical University in Prague</p>
</div>

<div align="center" style="line-height: 1;">
  <a href="https://jaibardhan.com/depthworld/" target="_blank" style="margin: 2px;">
    <img alt="Project Page" src="https://img.shields.io/badge/Project-Page-green" style="display: inline-block; vertical-align: middle;"/>
  </a>
  <a href="https://github.com/Jai2500/depthworld" target="_blank" style="margin: 2px;">
    <img alt="GitHub" src="https://img.shields.io/badge/GitHub-depthworld-black" style="display: inline-block; vertical-align: middle;"/>
  </a>
  <a href="https://arxiv.org/abs/2610.08780" target="_blank" style="margin: 2px;">
    <img alt="ArXiv" src="https://img.shields.io/badge/ArXiv-2610.08780-red" style="display: inline-block; vertical-align: middle;"/>
  </a>
  <a href="https://huggingface.co/datasets/jaibrdhn/droid_3d_extrinsics" target="_blank" style="margin: 2px;">
    <img alt="DROID-3D Extrinsics" src="https://img.shields.io/badge/%F0%9F%A4%97-DROID--3D_Extrinsics-yellow" style="display: inline-block; vertical-align: middle;"/>
  </a>
  <a href="https://forms.gle/E84Kg4BedVpgMe436" target="_blank" style="margin: 2px;">
    <img alt="DROID-3D Access Form" src="https://img.shields.io/badge/DROID--3D-Access_Form-blue" style="display: inline-block; vertical-align: middle;"/>
  </a>
</div>

<div align="center">
  <img src="https://huggingface.co/jaibrdhn/depthworld/resolve/main/corl-2026-accepted.svg" alt="🎉 Accepted to CoRL 2026 🎉" width="520"/>
</div>

## TL;DR

DepthWorld is an action-conditioned world model for DROID-style robot
manipulation that jointly predicts multi-view RGB **and metric depth**,
producing rollouts that compose into a single, geometrically consistent 3D
world. RGB and depth are packed side-by-side along width per view and
modeled by a single Stable Video Diffusion UNet; an optional DPT head
additionally regresses per-pixel world-frame point maps.

![DepthWorld architecture](architecture.png)

## Checkpoints

Both are consolidated 90k-step EMA weights (fp32 `.pt` state dicts), trained
on the raw [DROID](https://droid-dataset.github.io/) dataset:

| file | contents | description | license |
|---|---|---|---|
| `horiz_90k_ema.pt` | UNet + action encoder | RGB+depth horiz world model | Stability AI Community |
| `horiz_dpt_vggt_90k_ema.pt` | + DPT pointmap head | same, plus a DPT head initialized from [`facebook/VGGT-1B`](https://huggingface.co/facebook/VGGT-1B) predicting world-frame XYZ + confidence, trained with a MapAnything-style robust loss | Stability AI Community + **CC-BY-NC 4.0 (non-commercial)** |

See [License](#license) for details.

## Usage

```bash
hf download jaibrdhn/depthworld horiz_90k_ema.pt config.json --local-dir checkpoints
hf download jaibrdhn/depthworld horiz_dpt_vggt_90k_ema.pt config.json --local-dir checkpoints
```

Refer to the [official GitHub repository](https://github.com/Jai2500/depthworld)
for setup, data preparation, training, and evaluation instructions. The eval
scripts consume these files directly, e.g.:

```bash
python scripts/eval_horiz_chunk.py \
    --ckpt_path checkpoints/horiz_dpt_vggt_90k_ema.pt --width 640 ...
```

## Acknowledgements

Built upon [Ctrl-World](https://github.com/Robert-gyj/Ctrl-World) and
[Stable Video Diffusion](https://github.com/Stability-AI/generative-models).
The pointmap head is warm-started from
[VGGT](https://github.com/facebookresearch/vggt) and trained with a loss
following [MapAnything](https://github.com/facebookresearch/map-anything).
Trained on the [DROID](https://droid-dataset.github.io/) dataset.

This work was supported by the European Union's Horizon Europe projects
AGIMUS (No. 101070165), euROBIN (No. 101070596), ERC FRONTIER (No.
101097822), ELIAS (No. 101120237), ELLIOT (No. 101214398), ČVUT Starting
grant "DREAM-ACT" (Project ID CVUT-StG-26-089), and CTU Future Fund (Project
ID: CVUT-BrF-26-22825M). This work was also supported by the EU’s Horizon
Europe Programme under the Grant agreement No. 101136607 (CLARA Project),
and was co-funded by the EU from the Operational Programme Jan Amos Komenský
(OP JAK) (project "Center for Artificial Intelligence and Quantum Computing
in System Brain Research", reg. no. CZ.02.01.01/00/23_029/0008437). Compute
resources and infrastructure were supported by the Ministry of Education,
Youth and Sports of the Czech Republic through the e-INFRA CZ (ID:90254).

## Citation

```bibtex
@inproceedings{bardhan2026depthworld,
  title     = {DepthWorld: 3D World Model for Robot Manipulation},
  author    = {Bardhan, Jai and \v{S}ivic, Josef and Petr\'{i}k, Vladim\'{i}r},
  booktitle = {Conference on Robot Learning (CoRL)},
  year      = {2026}
}
```

## License

Original code and contributions are released under the MIT license (see the
GitHub repository). The checkpoints themselves are derivative works of
upstream models and remain subject to their terms:

- Both checkpoints are fine-tuned from
  [`stabilityai/stable-video-diffusion-img2vid`](https://huggingface.co/stabilityai/stable-video-diffusion-img2vid)
  and are therefore subject to the
  [**Stability AI Community License**](https://stability.ai/community-license-agreement).
- The DPT pointmap head in `horiz_dpt_vggt_90k_ema.pt` was initialized from
  [`facebook/VGGT-1B`](https://huggingface.co/facebook/VGGT-1B)
  (**CC-BY-NC 4.0**); treat that checkpoint as **non-commercial research
  use**.