Ethanliang99's picture
Cite the arXiv version
7548c9e verified
|
Raw History Blame Contribute Delete
3.2 kB
---
base_model:
- nvidia/Cosmos-Predict2-2B-Video2World
license: other
license_name: nvidia-base-model-license
license_link: https://huggingface.co/nvidia/Cosmos-Predict2-2B-Video2World
pipeline_tag: robotics
tags:
- robotics
- world-action-model
- video-diffusion
- franka
---
# MetaWAM-RealWorld-Predict2-2B
[**Paper**](https://arxiv.org/abs/2611.14365) | [**Code**](https://github.com/showlab/MetaWAM) | *From Video Frames to Robot Meta-Frames: A Unified Latent Interface for World Action Models* (CoRL 2026)
MetaWAM is a World Action Model fine-tuned from [Cosmos-Predict2-2B-Video2World](https://huggingface.co/nvidia/Cosmos-Predict2-2B-Video2World). Robot signals (task type, proprioception, action chunks, future proprioception and values) are packed into fixed coefficient slots of disjoint channel bands and lifted into compact *meta-frames* with a 2D inverse DCT. Slot-aware masks then train policy, world-model and value prediction in a single denoising model.
This checkpoint is a single policy for a **7-DoF Franka Panda**, trained jointly on three tabletop tasks for 7.2K steps. It sees one wrist camera and two third-person cameras, and predicts 16-step action chunks. The tasks and their instructions are:
| Task | Instruction |
|---|---|
| Sweep | `Sweep the green block into the dustpan` |
| Place Cup | `Place the cup on the green coaster` |
| Build Block | `Place the orange block on top of the gray block` |
The checkpoint is tied to our camera placement and robot setup. Treat it as a reference, or as an initialization for fine-tuning on your own data, rather than as a drop-in policy.
## Files
| File | Description |
|---|---|
| `MetaWAM-RealWorld-Predict2-2B.pt` | Model weights (bf16, consolidated) |
| `realworld_dataset_statistics.json` | Action / state normalization statistics of the training data |
| `realworld_t5_embeddings.pkl` | T5 embeddings of the three task instructions |
| `SHA256SUMS` | Checksums |
## Usage
Run the policy server with the inference config `metawam_realworld__inference` from the [MetaWAM repository](https://github.com/showlab/MetaWAM):
```bash
hf download Ethanliang99/MetaWAM-RealWorld-Predict2-2B --local-dir checkpoints/MetaWAM-RealWorld-Predict2-2B
export FRANKA_CKPT_PATH=checkpoints/MetaWAM-RealWorld-Predict2-2B/MetaWAM-RealWorld-Predict2-2B.pt
export FRANKA_DATASET_STATS_PATH=checkpoints/MetaWAM-RealWorld-Predict2-2B/realworld_dataset_statistics.json
export FRANKA_T5_EMBEDDINGS_PATH=checkpoints/MetaWAM-RealWorld-Predict2-2B/realworld_t5_embeddings.pkl
bash cosmos_policy/experiments/robot/franka/run_deploy_franka.sh
```
See [REALWORLD.md](https://github.com/showlab/MetaWAM/blob/main/REALWORLD.md) for the robot client, data collection and training.
## License
These weights are fine-tuned from `nvidia/Cosmos-Predict2-2B-Video2World`; use is subject to the license of the base model.
## Citation
```bibtex
@article{liang2026metawam,
title = {From Video Frames to Robot Meta-Frames: A Unified Latent Interface for World Action Models},
author = {Liang, Guoqiang and Lin, Yiqi and Chen, Yanzhe and Ma, Kevin Yuchen and Shou, Mike Zheng},
journal = {arXiv preprint arXiv:2611.14365},
year = {2026}
}
```