--- license: mit pipeline_tag: image-to-video tags: - robotics - video-generation - world-model - video-diffusion - action-conditioned - cross-embodiment ---

👏 CLAP: Cross-Embodiment Action-Conditioned Video World Models are Zero-Shot Physical Simulators

Paper Website Code License

CLAP is a cross-embodiment, action-conditioned video generation framework that unifies disparate human and robot action spaces via end-effector poses, language, and latent actions. This repo hosts the pretrained/post-trained/adapted CLAP checkpoints; training code, configs, and full usage instructions live in the [GitHub repo](https://github.com/omni-CLAP/clap). - **Two-Stage Curriculum:** *Learns unsupervised physical priors from unlabeled video, then grounds them in end-effector space for zero-shot deployment.* - **Sample-Efficient Adaptation:** *Facilitates few-shot transfer to target robots, matching or beating single-embodiment baselines like DROID (without post-training) and Bridge (with post-training).* - **Zero-Shot Real-World Generalization:** *Generalizes out-of-the-box to power real-world inference planning and RL finetuning with robot policies like π₀.₅ and MolmoAct-2.* ## ✨ Conditioning spaces | `conditioning` | Action representation | Typical use | |---|---|---| | 🦾 `"ee"` | 7-dim end-effector cartesian pose + gripper (or raw joint angles for joint-space embodiments) | Cross-embodiment, post-training, adaptation | | 🧬 `"lam"` | 32-dim latent-action-model embedding (learned, not physical) | Curriculum cross-embodiment training, including egocentric human video | | 💬 `"language"` | Per-frame CLIP-encoded text captions of the action | Language-conditioned, post-training, adaptation | ## 🗂️ Checkpoints Each subfolder below is a complete checkpoint (loadable directly via `CLAPModel.from_pretrained`, see the [GitHub repo](https://github.com/omni-CLAP/clap) for the `clap` package and CLI entrypoints). ### 🌐 Cross-embodiment models Trained for 100K steps on a mix of OXE datasets (`fractal`/`fmb`/`bc_z`/`taco_play`/`furniture_bench`/`bridge`/`droid`, `egodex` too for `lam`). All `ee`-conditioned models use absolute actions, while the `lang`-conditioned model uses relative-to-anchor-frame language captions. | Model name | Conditioning | Use case | |---|---|---| | [`clap-curr`](./cross_embodiment_oxe_curriculum_lam_ee) | `ee`, continued from a LAM-pretrained checkpoint | **Default** — getting-started default; base checkpoint for novel-embodiment adaptation | | [`clap-ee`](./cross_embodiment_oxe_ee) | `ee` — 7-dim end-effector cartesian | Cross-embodiment world model, EE-only (no LAM pretraining stage) | | [`clap-lam`](./cross_embodiment_oxe_lam_clap) | `lam` — 32-dim latent action | Curriculum LAM pretraining stage | | [`clap-lang`](./cross_embodiment_oxe_language_relative) | `language`, relative-to-anchor-frame captions | Language-conditioned generation/eval | ### 🎯 Post-trained on a single robot platform Each finetunes the matching cross-embodiment checkpoint above onto one dataset alone (still `ee`-conditioned regardless of the base's own conditioning). #### DROID | Model name | Conditioning | Use case | |---|---|---| | [`clap-curr-droid`](./post_train_oxe_curriculum_lam_ee_droid) | `ee` — 7-dim EE cartesian | Post-trained on `droid`, from the `clap-curr` base | | [`clap-ee-droid`](./post_train_oxe_ee_droid) | `ee` — 7-dim EE cartesian | Post-trained on `droid`, from the `clap-ee` base | | [`clap-lam-droid`](./post_train_oxe_lam_clap_droid) | `ee` — 7-dim EE cartesian | Post-trained on `droid`, from the `clap-lam` base | | [`clap-lang-droid`](./post_train_oxe_language_relative_droid) | `ee` — 7-dim EE cartesian | Post-trained on `droid`, from the `clap-lang` base | #### Bridge | Model name | Conditioning | Use case | |---|---|---| | [`clap-curr-bridge`](./post_train_oxe_curriculum_lam_ee_bridge) | `ee` — 7-dim EE cartesian | Post-trained on `bridge`, from the `clap-curr` base | | [`clap-ee-bridge`](./post_train_oxe_ee_bridge) | `ee` — 7-dim EE cartesian | Post-trained on `bridge`, from the `clap-ee` base | | [`clap-lam-bridge`](./post_train_oxe_lam_clap_bridge) | `ee` — 7-dim EE cartesian | Post-trained on `bridge`, from the `clap-lam` base | | [`clap-lang-bridge`](./post_train_oxe_language_relative_bridge) | `ee` — 7-dim EE cartesian | Post-trained on `bridge`, from the `clap-lang` base | ### 🆕 Novel-embodiment adaptation Finetuned on data for a new robot embodiment, building on physical priors from the cross-embodiment models (defaults to `clap-curr`). | Model name | Conditioning | Use case | |---|---|---| | [`adapt-yam`](./adapt_bimanual_yam) | `ee`, 14-dim joint-space | Novel-embodiment adaptation target: `bimanual_yam` | | [`adapt-g1`](./adapt_g1_humanoid) | `ee`, 26-dim joint-space | Novel-embodiment adaptation target: `g1_humanoid` | ## 🚀 Usage Install the `clap` package from GitHub, then point any checkpoint path below at a `--ckpt`/`checkpoint` arg (see the GitHub repo's [Getting started](https://github.com/omni-CLAP/clap#-getting-started) for `clap-rollout-replay`/`clap-teleop`/`clap-rollout-deploy`/`clap-eval`): ```bash git clone https://github.com/omni-CLAP/clap.git && cd clap uv venv .venv/clap --python 3.11 && source .venv/clap/bin/activate uv pip install -e . # download a checkpoint from this repo, e.g.: hf download omni-CLAP/CLAP --local-dir ./checkpoints --include "cross_embodiment_oxe_curriculum_lam_ee/*" ``` ## 📖 Glossary | Term | Meaning | |---|---| | **EE** | End-effector — the robot's gripper/hand; `ee`-conditioning uses its 7-dim cartesian pose (position + orientation) + gripper, as opposed to raw joint angles | | **LAM** | Latent Action Model — a learned, unsupervised action representation extracted directly from video (not a physical quantity), letting CLAP pretrain on video with no paired action labels, including egocentric human video | | **OXE** | Open X-Embodiment — the multi-robot, multi-dataset collection CLAP's cross-embodiment training draws from (`fractal`/`fmb`/`bc_z`/`taco_play`/`furniture_bench`/`bridge`/`droid`, `egodex` for `lam`) | | **DROID** | A large-scale real-world Franka-arm manipulation dataset/platform; one of CLAP's core cross-embodiment/post-training targets | | **bimanual YAM** / `bimanual_yam` | A dual-arm robot platform; one of CLAP's novel-embodiment adaptation targets (14-dim joint-space action) | | **G1** / `g1_humanoid` | Unitree G1, a bipedal humanoid robot (2 arms + 2 hands); the other novel-embodiment adaptation target (26-dim joint-space action) | | **SVD** | Stable Video Diffusion — the pretrained video diffusion backbone `CLAPModel`'s U-Net is built on | ## ⚖️ License MIT — see [LICENSE](https://github.com/omni-CLAP/clap/blob/main/LICENSE).