|
Download README.md from comp-visual-planning/cvp-models: direct link, hf CLI and curl.
- Browser
- Download file 3.42 kB
-
https://huggingface.co/comp-visual-planning/cvp-models/resolve/main/README.md
- Command line
-
hf download hf://comp-visual-planning/cvp-models/README.md
-
curl -L -o README.md https://huggingface.co/comp-visual-planning/cvp-models/resolve/main/README.md
3.42 kB
| license: other | |
| license_name: mit-and-nvidia-open-model-license | |
| license_link: LICENSE.md | |
| library_name: pytorch | |
| tags: | |
| - diffusion | |
| - planning | |
| - compositional-generation | |
| - robotics | |
| - video-generation | |
| # Compositional Visual Planning: pretrained models | |
| Pretrained models of [Compositional Visual Planning via Inference-Time Diffusion Scaling](https://arxiv.org/abs/2603.02646) | |
| ([project page](https://comp-visual-planning.github.io/), [code](https://github.com/yzhang4179/comp_visual_planning_release)) for the four SAPIEN tasks of the paper: cube, drawer, puzzle | |
| and tool. Each task has a video diffusion planner over Cosmos image latents, an inverse dynamics model that predicts the | |
| action between two consecutive latent frames, and the action range used to normalize the actions. | |
| ## Files | |
| | File | Description | Size | | |
| | --- | --- | --- | | |
| | `cube/diffusion.ckpt` | DiT planner (16 blocks, width 1024, 16 heads) after 100k gradient steps | 2.5 GB | | |
| | `drawer/diffusion.ckpt` | DiT planner (16 blocks, width 1024, 16 heads) after 460k gradient steps | 2.5 GB | | |
| | `puzzle/diffusion.ckpt` | DiT planner (24 blocks, width 1024, 16 heads) after 60k gradient steps | 3.7 GB | | |
| | `tool/diffusion.ckpt` | DiT planner (12 blocks, width 768, 12 heads) after 220k gradient steps | 1.0 GB | | |
| | `<task>/invdyn.ckpt` | state dict of the inverse dynamics model (embed_dim 384), after 60k gradient steps (40k for tool) | 1.3 MB | | |
| | `<task>/actions_minmax.json` | per-dimension min and max of the 8-dim actions | < 1 KB | | |
| | `cosmos/Cosmos-Tokenizer-CI16x16/` | `encoder.jit` and `decoder.jit` of NVIDIA's [Cosmos-0.1-Tokenizer-CI16x16](https://huggingface.co/nvidia/Cosmos-0.1-Tokenizer-CI16x16), unmodified, with `NOTICE` and a copy of the license | 163 MB | | |
| All planners are x0-predicting DDPMs with 500 cosine diffusion steps. They work on clips of 16 frames, and each frame | |
| is the (16, 16, 16) Cosmos latent of a 256x256 image, scaled by 0.5. Each planner file holds the online and the EMA weights, | |
| `{"model": ..., "model_ema": ...}`. | |
| ## Usage | |
| The code downloads the files of a task when they are missing locally. To download them by hand, run from the root | |
| of the [code repository](https://github.com/yzhang4179/comp_visual_planning_release): | |
| ```bash | |
| pip install -U huggingface_hub | |
| hf download comp-visual-planning/cvp-models --local-dir ckpts # everything | |
| hf download comp-visual-planning/cvp-models --include "cube/*" --include "cosmos/*" --local-dir ckpts # one task | |
| ``` | |
| Then run `python eval.py env=cube method=cvp` (ours) or | |
| `python eval.py env=cube method=gsc` (DiffCollage / Generative Skill Chaining baseline). | |
| ## License | |
| The planners, inverse dynamics models and action ranges are released under the MIT license. The files in | |
| `cosmos/Cosmos-Tokenizer-CI16x16/` are licensed by NVIDIA Corporation under the | |
| [NVIDIA Open Model License](https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf). | |
| ## Citation | |
| ```bibtex | |
| @misc{zhang2026compositionalvisualplanninginferencetime, | |
| title={Compositional Visual Planning via Inference-Time Diffusion Scaling}, | |
| author={Yixin Zhang and Yunhao Luo and Utkarsh Aashu Mishra and Woo Chul Shin and Yongxin Chen and Danfei Xu}, | |
| year={2026}, | |
| eprint={2603.02646}, | |
| archivePrefix={arXiv}, | |
| primaryClass={cs.RO}, | |
| url={https://arxiv.org/abs/2603.02646}, | |
| } | |
| ``` | |