--- license: other license_name: mit-and-nvidia-open-model-license license_link: LICENSE.md library_name: pytorch tags: - diffusion - planning - compositional-generation - robotics - video-generation --- # Compositional Visual Planning: pretrained models Pretrained models of [Compositional Visual Planning via Inference-Time Diffusion Scaling](https://arxiv.org/abs/2603.02646) ([project page](https://comp-visual-planning.github.io/), [code](https://github.com/yzhang4179/comp_visual_planning_release)) for the four SAPIEN tasks of the paper: cube, drawer, puzzle and tool. Each task has a video diffusion planner over Cosmos image latents, an inverse dynamics model that predicts the action between two consecutive latent frames, and the action range used to normalize the actions. ## Files | File | Description | Size | | --- | --- | --- | | `cube/diffusion.ckpt` | DiT planner (16 blocks, width 1024, 16 heads) after 100k gradient steps | 2.5 GB | | `drawer/diffusion.ckpt` | DiT planner (16 blocks, width 1024, 16 heads) after 460k gradient steps | 2.5 GB | | `puzzle/diffusion.ckpt` | DiT planner (24 blocks, width 1024, 16 heads) after 60k gradient steps | 3.7 GB | | `tool/diffusion.ckpt` | DiT planner (12 blocks, width 768, 12 heads) after 220k gradient steps | 1.0 GB | | `/invdyn.ckpt` | state dict of the inverse dynamics model (embed_dim 384), after 60k gradient steps (40k for tool) | 1.3 MB | | `/actions_minmax.json` | per-dimension min and max of the 8-dim actions | < 1 KB | | `cosmos/Cosmos-Tokenizer-CI16x16/` | `encoder.jit` and `decoder.jit` of NVIDIA's [Cosmos-0.1-Tokenizer-CI16x16](https://huggingface.co/nvidia/Cosmos-0.1-Tokenizer-CI16x16), unmodified, with `NOTICE` and a copy of the license | 163 MB | All planners are x0-predicting DDPMs with 500 cosine diffusion steps. They work on clips of 16 frames, and each frame is the (16, 16, 16) Cosmos latent of a 256x256 image, scaled by 0.5. Each planner file holds the online and the EMA weights, `{"model": ..., "model_ema": ...}`. ## Usage The code downloads the files of a task when they are missing locally. To download them by hand, run from the root of the [code repository](https://github.com/yzhang4179/comp_visual_planning_release): ```bash pip install -U huggingface_hub hf download comp-visual-planning/cvp-models --local-dir ckpts # everything hf download comp-visual-planning/cvp-models --include "cube/*" --include "cosmos/*" --local-dir ckpts # one task ``` Then run `python eval.py env=cube method=cvp` (ours) or `python eval.py env=cube method=gsc` (DiffCollage / Generative Skill Chaining baseline). ## License The planners, inverse dynamics models and action ranges are released under the MIT license. The files in `cosmos/Cosmos-Tokenizer-CI16x16/` are licensed by NVIDIA Corporation under the [NVIDIA Open Model License](https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf). ## Citation ```bibtex @misc{zhang2026compositionalvisualplanninginferencetime, title={Compositional Visual Planning via Inference-Time Diffusion Scaling}, author={Yixin Zhang and Yunhao Luo and Utkarsh Aashu Mishra and Woo Chul Shin and Yongxin Chen and Danfei Xu}, year={2026}, eprint={2603.02646}, archivePrefix={arXiv}, primaryClass={cs.RO}, url={https://arxiv.org/abs/2603.02646}, } ```