File size: 3,420 Bytes
ee1efcf
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
---
license: other
license_name: mit-and-nvidia-open-model-license
license_link: LICENSE.md
library_name: pytorch
tags:
- diffusion
- planning
- compositional-generation
- robotics
- video-generation
---

# Compositional Visual Planning: pretrained models

Pretrained models of [Compositional Visual Planning via Inference-Time Diffusion Scaling](https://arxiv.org/abs/2603.02646)
([project page](https://comp-visual-planning.github.io/), [code](https://github.com/yzhang4179/comp_visual_planning_release)) for the four SAPIEN tasks of the paper: cube, drawer, puzzle
and tool. Each task has a video diffusion planner over Cosmos image latents, an inverse dynamics model that predicts the
action between two consecutive latent frames, and the action range used to normalize the actions.

## Files

| File | Description | Size |
| --- | --- | --- |
| `cube/diffusion.ckpt` | DiT planner (16 blocks, width 1024, 16 heads) after 100k gradient steps | 2.5 GB |
| `drawer/diffusion.ckpt` | DiT planner (16 blocks, width 1024, 16 heads) after 460k gradient steps | 2.5 GB |
| `puzzle/diffusion.ckpt` | DiT planner (24 blocks, width 1024, 16 heads) after 60k gradient steps | 3.7 GB |
| `tool/diffusion.ckpt` | DiT planner (12 blocks, width 768, 12 heads) after 220k gradient steps | 1.0 GB |
| `<task>/invdyn.ckpt` | state dict of the inverse dynamics model (embed_dim 384), after 60k gradient steps (40k for tool) | 1.3 MB |
| `<task>/actions_minmax.json` | per-dimension min and max of the 8-dim actions | < 1 KB |
| `cosmos/Cosmos-Tokenizer-CI16x16/` | `encoder.jit` and `decoder.jit` of NVIDIA's [Cosmos-0.1-Tokenizer-CI16x16](https://huggingface.co/nvidia/Cosmos-0.1-Tokenizer-CI16x16), unmodified, with `NOTICE` and a copy of the license | 163 MB |

All planners are x0-predicting DDPMs with 500 cosine diffusion steps. They work on clips of 16 frames, and each frame
is the (16, 16, 16) Cosmos latent of a 256x256 image, scaled by 0.5. Each planner file holds the online and the EMA weights,
`{"model": ..., "model_ema": ...}`.

## Usage

The code downloads the files of a task when they are missing locally. To download them by hand, run from the root
of the [code repository](https://github.com/yzhang4179/comp_visual_planning_release):

```bash
pip install -U huggingface_hub
hf download comp-visual-planning/cvp-models --local-dir ckpts                                          # everything
hf download comp-visual-planning/cvp-models --include "cube/*" --include "cosmos/*" --local-dir ckpts  # one task
```

Then run `python eval.py env=cube method=cvp` (ours) or
`python eval.py env=cube method=gsc` (DiffCollage / Generative Skill Chaining baseline).

## License

The planners, inverse dynamics models and action ranges are released under the MIT license. The files in
`cosmos/Cosmos-Tokenizer-CI16x16/` are licensed by NVIDIA Corporation under the
[NVIDIA Open Model License](https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf).

## Citation

```bibtex
@misc{zhang2026compositionalvisualplanninginferencetime,
      title={Compositional Visual Planning via Inference-Time Diffusion Scaling},
      author={Yixin Zhang and Yunhao Luo and Utkarsh Aashu Mishra and Woo Chul Shin and Yongxin Chen and Danfei Xu},
      year={2026},
      eprint={2603.02646},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2603.02646},
}
```