README.md

---
license: mit
pipeline_tag: image-to-video
tags:
- robotics
- video-generation
- world-model
- video-diffusion
- action-conditioned
- cross-embodiment
---

<div align="center">
  <h1>👏 CLAP: Cross-Embodiment Action-Conditioned Video World Models are Zero-Shot Physical Simulators</h1>
</div>

<p align="center">
  <img alt="Paper" src="https://img.shields.io/badge/Paper-coming%20soon-lightgrey?logo=arxiv">
  <a href="https://omni-clap.github.io"><img alt="Website" src="https://img.shields.io/badge/%F0%9F%8C%90%20Website-Live-d15619"></a>
  <a href="https://github.com/omni-CLAP/clap"><img alt="Code" src="https://img.shields.io/badge/GitHub-Code-181717?logo=github"></a>
  <a href="https://github.com/omni-CLAP/clap/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/badge/License-MIT-blue?logo=opensourceinitiative"></a>
</p>

CLAP is a cross-embodiment, action-conditioned video generation framework that unifies disparate human and robot action spaces via end-effector poses, language, and latent actions. This repo hosts the pretrained/post-trained/adapted CLAP checkpoints; training code, configs, and full usage instructions live in the [GitHub repo](https://github.com/omni-CLAP/clap).

- **Two-Stage Curriculum:** *Learns unsupervised physical priors from unlabeled video, then grounds them in end-effector space for zero-shot deployment.*
- **Sample-Efficient Adaptation:** *Facilitates few-shot transfer to target robots, matching or beating single-embodiment baselines like DROID (without post-training) and Bridge (with post-training).*
- **Zero-Shot Real-World Generalization:** *Generalizes out-of-the-box to power real-world inference planning and RL finetuning with robot policies like π₀.₅ and MolmoAct-2.*

## ✨ Conditioning spaces

| `conditioning` | Action representation | Typical use |
|---|---|---|
| 🦾 `"ee"` | 7-dim end-effector cartesian pose + gripper (or raw joint angles for joint-space embodiments) | Cross-embodiment, post-training, adaptation |
| 🧬 `"lam"` | 32-dim latent-action-model embedding (learned, not physical) | Curriculum cross-embodiment training, including egocentric human video |
| 💬 `"language"` | Per-frame CLIP-encoded text captions of the action | Language-conditioned, post-training, adaptation |

## 🗂️ Checkpoints

Each subfolder below is a complete checkpoint (loadable directly via `CLAPModel.from_pretrained`, see the [GitHub repo](https://github.com/omni-CLAP/clap) for the `clap` package and CLI entrypoints).

### 🌐 Cross-embodiment models

Trained for 100K steps on a mix of OXE datasets (`fractal`/`fmb`/`bc_z`/`taco_play`/`furniture_bench`/`bridge`/`droid`, `egodex` too for `lam`). All `ee`-conditioned models use absolute actions, while the `lang`-conditioned model uses relative-to-anchor-frame language captions.

| Model name | Conditioning | Use case |
|---|---|---|
| [`clap-curr`](./cross_embodiment_oxe_curriculum_lam_ee) | `ee`, continued from a LAM-pretrained checkpoint | **Default** — getting-started default; base checkpoint for novel-embodiment adaptation |
| [`clap-ee`](./cross_embodiment_oxe_ee) | `ee` — 7-dim end-effector cartesian | Cross-embodiment world model, EE-only (no LAM pretraining stage) |
| [`clap-lam`](./cross_embodiment_oxe_lam_clap) | `lam` — 32-dim latent action | Curriculum LAM pretraining stage |
| [`clap-lang`](./cross_embodiment_oxe_language_relative) | `language`, relative-to-anchor-frame captions | Language-conditioned generation/eval |

### 🎯 Post-trained on a single robot platform

Each finetunes the matching cross-embodiment checkpoint above onto one dataset alone (still `ee`-conditioned regardless of the base's own conditioning).

#### DROID

| Model name | Conditioning | Use case |
|---|---|---|
| [`clap-curr-droid`](./post_train_oxe_curriculum_lam_ee_droid) | `ee` — 7-dim EE cartesian | Post-trained on `droid`, from the `clap-curr` base |
| [`clap-ee-droid`](./post_train_oxe_ee_droid) | `ee` — 7-dim EE cartesian | Post-trained on `droid`, from the `clap-ee` base |
| [`clap-lam-droid`](./post_train_oxe_lam_clap_droid) | `ee` — 7-dim EE cartesian | Post-trained on `droid`, from the `clap-lam` base |
| [`clap-lang-droid`](./post_train_oxe_language_relative_droid) | `ee` — 7-dim EE cartesian | Post-trained on `droid`, from the `clap-lang` base |

#### Bridge

| Model name | Conditioning | Use case |
|---|---|---|
| [`clap-curr-bridge`](./post_train_oxe_curriculum_lam_ee_bridge) | `ee` — 7-dim EE cartesian | Post-trained on `bridge`, from the `clap-curr` base |
| [`clap-ee-bridge`](./post_train_oxe_ee_bridge) | `ee` — 7-dim EE cartesian | Post-trained on `bridge`, from the `clap-ee` base |
| [`clap-lam-bridge`](./post_train_oxe_lam_clap_bridge) | `ee` — 7-dim EE cartesian | Post-trained on `bridge`, from the `clap-lam` base |
| [`clap-lang-bridge`](./post_train_oxe_language_relative_bridge) | `ee` — 7-dim EE cartesian | Post-trained on `bridge`, from the `clap-lang` base |

### 🆕 Novel-embodiment adaptation

Finetuned on data for a new robot embodiment, building on physical priors from the cross-embodiment models (defaults to `clap-curr`).

| Model name | Conditioning | Use case |
|---|---|---|
| [`adapt-yam`](./adapt_bimanual_yam) | `ee`, 14-dim joint-space | Novel-embodiment adaptation target: `bimanual_yam` |
| [`adapt-g1`](./adapt_g1_humanoid) | `ee`, 26-dim joint-space | Novel-embodiment adaptation target: `g1_humanoid` |

## 🚀 Usage

Install the `clap` package from GitHub, then point any checkpoint path below at a `--ckpt`/`checkpoint` arg (see the GitHub repo's [Getting started](https://github.com/omni-CLAP/clap#-getting-started) for `clap-rollout-replay`/`clap-teleop`/`clap-rollout-deploy`/`clap-eval`):

```bash
git clone https://github.com/omni-CLAP/clap.git && cd clap
uv venv .venv/clap --python 3.11 && source .venv/clap/bin/activate
uv pip install -e .

# download a checkpoint from this repo, e.g.:
hf download omni-CLAP/CLAP --local-dir ./checkpoints --include "cross_embodiment_oxe_curriculum_lam_ee/*"
```

## 📖 Glossary

| Term | Meaning |
|---|---|
| **EE** | End-effector — the robot's gripper/hand; `ee`-conditioning uses its 7-dim cartesian pose (position + orientation) + gripper, as opposed to raw joint angles |
| **LAM** | Latent Action Model — a learned, unsupervised action representation extracted directly from video (not a physical quantity), letting CLAP pretrain on video with no paired action labels, including egocentric human video |
| **OXE** | Open X-Embodiment — the multi-robot, multi-dataset collection CLAP's cross-embodiment training draws from (`fractal`/`fmb`/`bc_z`/`taco_play`/`furniture_bench`/`bridge`/`droid`, `egodex` for `lam`) |
| **DROID** | A large-scale real-world Franka-arm manipulation dataset/platform; one of CLAP's core cross-embodiment/post-training targets |
| **bimanual YAM** / `bimanual_yam` | A dual-arm robot platform; one of CLAP's novel-embodiment adaptation targets (14-dim joint-space action) |
| **G1** / `g1_humanoid` | Unitree G1, a bipedal humanoid robot (2 arms + 2 hands); the other novel-embodiment adaptation target (26-dim joint-space action) |
| **SVD** | Stable Video Diffusion — the pretrained video diffusion backbone `CLAPModel`'s U-Net is built on |


## ⚖️ License

MIT — see [LICENSE](https://github.com/omni-CLAP/clap/blob/main/LICENSE).