Create README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,76 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
tags:
|
| 3 |
+
- robotics
|
| 4 |
+
- world-model
|
| 5 |
+
- action-conditioned-video-generation
|
| 6 |
+
- robot-manipulation
|
| 7 |
+
- coachworld
|
| 8 |
+
- arxiv:2609.39685
|
| 9 |
+
---
|
| 10 |
+
|
| 11 |
+
# CoachWorld — Pretrained Robot World Model
|
| 12 |
+
|
| 13 |
+
[**Paper**](https://arxiv.org/abs/2609.39685) · [**Code**](https://github.com/RoboCoach-AI/CoachWorld) · [**Project Page & Videos**](https://robocoach-ai.github.io/)
|
| 14 |
+
|
| 15 |
+
**CoachWorld predicts future robot observations conditioned on visual history and robot actions.** It is the world-model component of **RoboCoach: World Models as Active Coaches for Compositional Robot Skills**.
|
| 16 |
+
|
| 17 |
+
This repository hosts the **CoachWorld pretrained checkpoint**, trained on heterogeneous single-arm and dual-arm robot data.
|
| 18 |
+
|
| 19 |
+
## Overview
|
| 20 |
+
|
| 21 |
+
CoachWorld is an action-conditioned video world model built on an adapted Wan2.2 TI backbone. It uses visual history and end-effector action conditioning to generate future observations, with camera and action conventions that support learning across robot datasets.
|
| 22 |
+
|
| 23 |
+
The model supports autoregressive rollout: generated observations become part of the context for subsequent predictions. In the full RoboCoach framework, skill policies interact with the world model, while a separate progress judge identifies subtask failures. These diagnoses guide targeted demonstration collection and policy updates.
|
| 24 |
+
|
| 25 |
+
## Model at a Glance
|
| 26 |
+
|
| 27 |
+
| Item | Description |
|
| 28 |
+
| --- | --- |
|
| 29 |
+
| Release | CoachWorld pretrain |
|
| 30 |
+
| Model type | Action-conditioned video world model |
|
| 31 |
+
| Backbone | Adapted Wan2.2 |
|
| 32 |
+
| Conditioning | Visual history and robot end-effector actions, using the prescribed camera/action representation |
|
| 33 |
+
| Output | Predicted future RGB observations |
|
| 34 |
+
| Rollout mode | Autoregressive video prediction |
|
| 35 |
+
| Training scope | Heterogeneous single-arm and dual-arm robot data |
|
| 36 |
+
| Implementation | Custom `coachworld` Python package |
|
| 37 |
+
|
| 38 |
+
## Getting Started
|
| 39 |
+
|
| 40 |
+
Download the weights from the [Files and versions](https://huggingface.co/JEdward/CoachWorld/tree/main) tab and use the [CoachWorld code repository](https://github.com/RoboCoach-AI/CoachWorld) for environment setup and model integration.
|
| 41 |
+
|
| 42 |
+
The implementation includes:
|
| 43 |
+
|
| 44 |
+
- Model conditioning, inference, and training components.
|
| 45 |
+
- Modified Wan2.2 model and VAE components.
|
| 46 |
+
- Video-latent data interfaces and action/camera conventions.
|
| 47 |
+
- Camera and robot geometry helpers.
|
| 48 |
+
- A distributed training entry point and example configuration.
|
| 49 |
+
|
| 50 |
+
**Input preparation matters.** Robot actions must follow the implementation’s coordinate-frame, normalization, and temporal-sampling conventions. Raw joint commands or pixel-space trajectories are not interchangeable with the model’s expected conditioning.
|
| 51 |
+
|
| 52 |
+
Use the CoachWorld implementation to load and run this checkpoint; this release does not claim compatibility with a generic Transformers or Diffusers loading pipeline.
|
| 53 |
+
|
| 54 |
+
## Citation
|
| 55 |
+
|
| 56 |
+
If you use CoachWorld in your research, please cite:
|
| 57 |
+
|
| 58 |
+
```bibtex
|
| 59 |
+
@article{liu2026robocoach,
|
| 60 |
+
title={RoboCoach: World Models as Active Coaches for Compositional Robot Skills},
|
| 61 |
+
author={Liu, Jiajun and Chen, Yifan and Liu, Yichao and Zhang, Jiayi
|
| 62 |
+
and Chen, Ruoqu and Xie, Shaoxuan and Yao, Guocai
|
| 63 |
+
and Xu, Mengdi and Cui, Sen and Zhang, Changshui},
|
| 64 |
+
journal={arXiv preprint arXiv:2609.39685},
|
| 65 |
+
year={2026},
|
| 66 |
+
url={https://arxiv.org/abs/2609.39685}
|
| 67 |
+
}
|
| 68 |
+
```
|
| 69 |
+
|
| 70 |
+
## Acknowledgments
|
| 71 |
+
|
| 72 |
+
CoachWorld builds on Wan2.2 and other open-source components. See the code repository’s [third-party notices](https://github.com/RoboCoach-AI/CoachWorld/blob/main/THIRD_PARTY.md) for component attribution and applicable terms.
|
| 73 |
+
|
| 74 |
+
## Contact
|
| 75 |
+
|
| 76 |
+
For implementation questions and reproducibility issues, please open an issue in the [CoachWorld repository](https://github.com/RoboCoach-AI/CoachWorld/issues).
|