File size: 3,420 Bytes
ee1efcf | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | ---
license: other
license_name: mit-and-nvidia-open-model-license
license_link: LICENSE.md
library_name: pytorch
tags:
- diffusion
- planning
- compositional-generation
- robotics
- video-generation
---
# Compositional Visual Planning: pretrained models
Pretrained models of [Compositional Visual Planning via Inference-Time Diffusion Scaling](https://arxiv.org/abs/2603.02646)
([project page](https://comp-visual-planning.github.io/), [code](https://github.com/yzhang4179/comp_visual_planning_release)) for the four SAPIEN tasks of the paper: cube, drawer, puzzle
and tool. Each task has a video diffusion planner over Cosmos image latents, an inverse dynamics model that predicts the
action between two consecutive latent frames, and the action range used to normalize the actions.
## Files
| File | Description | Size |
| --- | --- | --- |
| `cube/diffusion.ckpt` | DiT planner (16 blocks, width 1024, 16 heads) after 100k gradient steps | 2.5 GB |
| `drawer/diffusion.ckpt` | DiT planner (16 blocks, width 1024, 16 heads) after 460k gradient steps | 2.5 GB |
| `puzzle/diffusion.ckpt` | DiT planner (24 blocks, width 1024, 16 heads) after 60k gradient steps | 3.7 GB |
| `tool/diffusion.ckpt` | DiT planner (12 blocks, width 768, 12 heads) after 220k gradient steps | 1.0 GB |
| `<task>/invdyn.ckpt` | state dict of the inverse dynamics model (embed_dim 384), after 60k gradient steps (40k for tool) | 1.3 MB |
| `<task>/actions_minmax.json` | per-dimension min and max of the 8-dim actions | < 1 KB |
| `cosmos/Cosmos-Tokenizer-CI16x16/` | `encoder.jit` and `decoder.jit` of NVIDIA's [Cosmos-0.1-Tokenizer-CI16x16](https://huggingface.co/nvidia/Cosmos-0.1-Tokenizer-CI16x16), unmodified, with `NOTICE` and a copy of the license | 163 MB |
All planners are x0-predicting DDPMs with 500 cosine diffusion steps. They work on clips of 16 frames, and each frame
is the (16, 16, 16) Cosmos latent of a 256x256 image, scaled by 0.5. Each planner file holds the online and the EMA weights,
`{"model": ..., "model_ema": ...}`.
## Usage
The code downloads the files of a task when they are missing locally. To download them by hand, run from the root
of the [code repository](https://github.com/yzhang4179/comp_visual_planning_release):
```bash
pip install -U huggingface_hub
hf download comp-visual-planning/cvp-models --local-dir ckpts # everything
hf download comp-visual-planning/cvp-models --include "cube/*" --include "cosmos/*" --local-dir ckpts # one task
```
Then run `python eval.py env=cube method=cvp` (ours) or
`python eval.py env=cube method=gsc` (DiffCollage / Generative Skill Chaining baseline).
## License
The planners, inverse dynamics models and action ranges are released under the MIT license. The files in
`cosmos/Cosmos-Tokenizer-CI16x16/` are licensed by NVIDIA Corporation under the
[NVIDIA Open Model License](https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf).
## Citation
```bibtex
@misc{zhang2026compositionalvisualplanninginferencetime,
title={Compositional Visual Planning via Inference-Time Diffusion Scaling},
author={Yixin Zhang and Yunhao Luo and Utkarsh Aashu Mishra and Woo Chul Shin and Yongxin Chen and Danfei Xu},
year={2026},
eprint={2603.02646},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2603.02646},
}
```
|