Compositional Visual Planning: pretrained models

Pretrained models of Compositional Visual Planning via Inference-Time Diffusion Scaling (project page, code) for the four SAPIEN tasks of the paper: cube, drawer, puzzle and tool. Each task has a video diffusion planner over Cosmos image latents, an inverse dynamics model that predicts the action between two consecutive latent frames, and the action range used to normalize the actions.

Files

File Description Size
cube/diffusion.ckpt DiT planner (16 blocks, width 1024, 16 heads) after 100k gradient steps 2.5 GB
drawer/diffusion.ckpt DiT planner (16 blocks, width 1024, 16 heads) after 460k gradient steps 2.5 GB
puzzle/diffusion.ckpt DiT planner (24 blocks, width 1024, 16 heads) after 60k gradient steps 3.7 GB
tool/diffusion.ckpt DiT planner (12 blocks, width 768, 12 heads) after 220k gradient steps 1.0 GB
<task>/invdyn.ckpt state dict of the inverse dynamics model (embed_dim 384), after 60k gradient steps (40k for tool) 1.3 MB
<task>/actions_minmax.json per-dimension min and max of the 8-dim actions < 1 KB
cosmos/Cosmos-Tokenizer-CI16x16/ encoder.jit and decoder.jit of NVIDIA's Cosmos-0.1-Tokenizer-CI16x16, unmodified, with NOTICE and a copy of the license 163 MB

All planners are x0-predicting DDPMs with 500 cosine diffusion steps. They work on clips of 16 frames, and each frame is the (16, 16, 16) Cosmos latent of a 256x256 image, scaled by 0.5. Each planner file holds the online and the EMA weights, {"model": ..., "model_ema": ...}.

Usage

The code downloads the files of a task when they are missing locally. To download them by hand, run from the root of the code repository:

pip install -U huggingface_hub
hf download comp-visual-planning/cvp-models --local-dir ckpts                                          # everything
hf download comp-visual-planning/cvp-models --include "cube/*" --include "cosmos/*" --local-dir ckpts  # one task

Then run python eval.py env=cube method=cvp (ours) or python eval.py env=cube method=gsc (DiffCollage / Generative Skill Chaining baseline).

License

The planners, inverse dynamics models and action ranges are released under the MIT license. The files in cosmos/Cosmos-Tokenizer-CI16x16/ are licensed by NVIDIA Corporation under the NVIDIA Open Model License.

Citation

@misc{zhang2026compositionalvisualplanninginferencetime,
      title={Compositional Visual Planning via Inference-Time Diffusion Scaling},
      author={Yixin Zhang and Yunhao Luo and Utkarsh Aashu Mishra and Woo Chul Shin and Yongxin Chen and Danfei Xu},
      year={2026},
      eprint={2603.02646},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2603.02646},
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Video Preview
loading

Paper for comp-visual-planning/cvp-models