Improve the WorldDiT model card layout (#2)
Browse files- Improve the WorldDiT model card layout (4b7fb954f1c92b19a043b1eaf4de8bf9b642ca09)
README.md
CHANGED
|
@@ -38,91 +38,81 @@ widget:
|
|
| 38 |
<img src="https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/paris2/model-card/v1/bagel_labs_logo.png" alt="Bagel Labs">
|
| 39 |
</p>
|
| 40 |
|
| 41 |
-
|
| 42 |
|
| 43 |
-
|
| 44 |
|
| 45 |
<p align="center">
|
| 46 |
-
<a href="https://huggingface.co/bageldotcom/worlddit" target="_blank">
|
| 47 |
-
<img src="https://img.shields.io/badge/馃_DOWNLOAD_WORLDDIT_WEIGHTS-FFD21E?style=for-the-badge&logoColor=000000" alt="Download WorldDiT Weights">
|
| 48 |
-
</a>
|
| 49 |
-
<a href="https://github.com/Lifelong-Robot-Learning/LIBERO" target="_blank">
|
| 50 |
-
<img src="https://img.shields.io/badge/馃_LIBERO_BENCHMARK-FF6B6B?style=for-the-badge&logoColor=white" alt="LIBERO Benchmark">
|
| 51 |
-
</a>
|
| 52 |
-
</p>
|
| 53 |
-
|
| 54 |
WorldDiT learns continuous robot action chunks and a future visual target
|
| 55 |
through one shared diffusion transformer. Deployment keeps only the action
|
| 56 |
path.
|
|
|
|
| 57 |
|
|
|
|
| 58 |
This release includes four LIBERO checkpoints, a self contained inference
|
| 59 |
runtime, and an evaluator for reproducing the reported suite results.
|
|
|
|
| 60 |
|
| 61 |
## See WorldDiT act
|
| 62 |
|
| 63 |
The four clips below show successful rollouts from the released checkpoints.
|
| 64 |
Each clip covers a different LIBERO suite and camera view.
|
| 65 |
|
| 66 |
-
<
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 67 |
|
| 68 |
-
|
|
| 69 |
|---|---|---|
|
| 70 |
-
|
|
| 71 |
-
| LIBERO Object | Agent view | [Open MP4](https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/worlddit/model-card/v1/worlddit_libero_object_agentview_task08_episode01.mp4) |
|
| 72 |
-
| LIBERO Goal | Side view | [Open MP4](https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/worlddit/model-card/v1/worlddit_libero_goal_sideview_task10_episode01.mp4) |
|
| 73 |
-
| LIBERO Long | Front view | [Open MP4](https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/worlddit/model-card/v1/worlddit_libero_10_frontview_task06_episode01.mp4) |
|
| 74 |
|
| 75 |
-
|
| 76 |
-
|
| 77 |
-
| Release component | Included artifact |
|
| 78 |
-
|---|---|
|
| 79 |
-
| LIBERO Spatial policy | SafeTensors checkpoint |
|
| 80 |
-
| LIBERO Object policy | SafeTensors checkpoint |
|
| 81 |
-
| LIBERO Goal policy | SafeTensors checkpoint |
|
| 82 |
-
| LIBERO Long policy | SafeTensors checkpoint |
|
| 83 |
-
| Model runtime | `inference.py` |
|
| 84 |
-
| Evaluation runtime | `eval.py` |
|
| 85 |
-
| Frozen encoders | CLIP ViT B 32 and MAE ViT B |
|
| 86 |
-
| Configuration | `config.json` |
|
| 87 |
-
| Environment | Pinned Python requirements |
|
| 88 |
-
|
| 89 |
-
The repository is self contained for WorldDiT inference. LIBERO still provides
|
| 90 |
-
the benchmark environments, assets, task definitions, and initial states.
|
| 91 |
-
|
| 92 |
-
## Reported LIBERO results
|
| 93 |
-
|
| 94 |
-
Across the four released suite checkpoints, WorldDiT records 1,898 successful
|
| 95 |
-
episodes out of 2,000 under the selection aware evaluation protocol.
|
| 96 |
-
|
| 97 |
-
| Suite | Successful episodes | Success rate |
|
| 98 |
-
|---|---:|---:|
|
| 99 |
-
| LIBERO Spatial | 490 of 500 | 98.0 percent |
|
| 100 |
-
| LIBERO Object | 485 of 500 | 97.0 percent |
|
| 101 |
-
| LIBERO Goal | 464 of 500 | 92.8 percent |
|
| 102 |
-
| LIBERO Long | 459 of 500 | 91.8 percent |
|
| 103 |
-
| Selection aware mean | 1,898 of 2,000 | 94.9 percent |
|
| 104 |
|
| 105 |
The released runtime and checkpoints were revalidated from a clean installation
|
| 106 |
-
on eight RTX Pro 6000 Blackwell GPUs.
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
staged checkpoint selection before the final five hundred episode score was
|
| 110 |
-
assembled.
|
| 111 |
-
|
| 112 |
-
## Model at a glance
|
| 113 |
-
|
| 114 |
-
| Property | Released configuration |
|
| 115 |
-
|---|---|
|
| 116 |
-
| Total parameters | 399.084 million |
|
| 117 |
-
| Trainable parameters | 135.107 million |
|
| 118 |
-
| Observation context | Three frames |
|
| 119 |
-
| Predicted action horizon | Seven actions |
|
| 120 |
-
| Executed before replanning | Three actions |
|
| 121 |
-
| Action dimension | Seven |
|
| 122 |
-
| Visual encoder | MAE ViT B |
|
| 123 |
-
| Language encoder | OpenAI CLIP ViT B 32 |
|
| 124 |
-
| Checkpoint format | SafeTensors |
|
| 125 |
-
| Evaluation environment | Headless LIBERO with EGL |
|
| 126 |
|
| 127 |
## Run a smoke test
|
| 128 |
|
|
@@ -179,7 +169,10 @@ and replans.
|
|
| 179 |
| Future visual supervision is present | No future visual output is requested |
|
| 180 |
| The complete training objective is active | Three actions execute before replanning |
|
| 181 |
|
| 182 |
-
##
|
|
|
|
|
|
|
|
|
|
| 183 |
|
| 184 |
### One GPU
|
| 185 |
|
|
@@ -213,7 +206,10 @@ libero_goal
|
|
| 213 |
libero_10
|
| 214 |
```
|
| 215 |
|
| 216 |
-
|
|
|
|
|
|
|
|
|
|
| 217 |
|
| 218 |
```text
|
| 219 |
.
|
|
@@ -234,7 +230,10 @@ libero_10
|
|
| 234 |
`dependencies/` contains the frozen visual and language encoder weights needed
|
| 235 |
by the released policy. No additional model downloads are required.
|
| 236 |
|
| 237 |
-
|
|
|
|
|
|
|
|
|
|
| 238 |
|
| 239 |
```python
|
| 240 |
from inference import load_model
|
|
@@ -253,7 +252,10 @@ actions = model(primary_images, wrist_images, robot_state, text_tokens)
|
|
| 253 |
|
| 254 |
Evaluation uses the final temporal slot of the predicted action tensor.
|
| 255 |
|
| 256 |
-
|
|
|
|
|
|
|
|
|
|
| 257 |
|
| 258 |
| Component | Specification |
|
| 259 |
|---|---|
|
|
@@ -267,21 +269,14 @@ Evaluation uses the final temporal slot of the predicted action tensor.
|
|
| 267 |
| Evaluation | Headless LIBERO with EGL |
|
| 268 |
| Checkpoint format | SafeTensors |
|
| 269 |
|
| 270 |
-
|
| 271 |
-
|
| 272 |
-
WorldDiT is intended for research on language conditioned robot manipulation in
|
| 273 |
-
the LIBERO simulator. The released checkpoints support reproduction,
|
| 274 |
-
evaluation, and architecture research across the four released suites.
|
| 275 |
|
| 276 |
-
##
|
| 277 |
|
| 278 |
-
|
| 279 |
-
|
| 280 |
-
across embodiments.
|
| 281 |
-
|
| 282 |
-
The present release does not isolate the causal contribution of the future
|
| 283 |
-
visual target. Total parameter count also does not measure training cost,
|
| 284 |
-
deployment latency, or runtime efficiency.
|
| 285 |
|
| 286 |
## Authors and contact
|
| 287 |
|
|
|
|
| 38 |
<img src="https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/paris2/model-card/v1/bagel_labs_logo.png" alt="Bagel Labs">
|
| 39 |
</p>
|
| 40 |
|
| 41 |
+
<h1 align="center">WorldDiT</h1>
|
| 42 |
|
| 43 |
+
<h2 align="center">One diffusion backbone learns what to do and what comes next.</h2>
|
| 44 |
|
| 45 |
<p align="center">
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 46 |
WorldDiT learns continuous robot action chunks and a future visual target
|
| 47 |
through one shared diffusion transformer. Deployment keeps only the action
|
| 48 |
path.
|
| 49 |
+
</p>
|
| 50 |
|
| 51 |
+
<p align="center">
|
| 52 |
This release includes four LIBERO checkpoints, a self contained inference
|
| 53 |
runtime, and an evaluator for reproducing the reported suite results.
|
| 54 |
+
</p>
|
| 55 |
|
| 56 |
## See WorldDiT act
|
| 57 |
|
| 58 |
The four clips below show successful rollouts from the released checkpoints.
|
| 59 |
Each clip covers a different LIBERO suite and camera view.
|
| 60 |
|
| 61 |
+
<table>
|
| 62 |
+
<tr>
|
| 63 |
+
<td width="50%" valign="top">
|
| 64 |
+
<video width="100%" controls muted loop playsinline preload="metadata"
|
| 65 |
+
src="https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/worlddit/model-card/v1/worlddit_libero_spatial_frontview_task05_episode01.mp4"></video>
|
| 66 |
+
<strong>LIBERO Spatial</strong><br>
|
| 67 |
+
Task 5, front view.
|
| 68 |
+
</td>
|
| 69 |
+
<td width="50%" valign="top">
|
| 70 |
+
<video width="100%" controls muted loop playsinline preload="metadata"
|
| 71 |
+
src="https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/worlddit/model-card/v1/worlddit_libero_object_agentview_task08_episode01.mp4"></video>
|
| 72 |
+
<strong>LIBERO Object</strong><br>
|
| 73 |
+
Task 8, agent view.
|
| 74 |
+
</td>
|
| 75 |
+
</tr>
|
| 76 |
+
<tr>
|
| 77 |
+
<td width="50%" valign="top">
|
| 78 |
+
<video width="100%" controls muted loop playsinline preload="metadata"
|
| 79 |
+
src="https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/worlddit/model-card/v1/worlddit_libero_goal_sideview_task10_episode01.mp4"></video>
|
| 80 |
+
<strong>LIBERO Goal</strong><br>
|
| 81 |
+
Task 10, side view.
|
| 82 |
+
</td>
|
| 83 |
+
<td width="50%" valign="top">
|
| 84 |
+
<video width="100%" controls muted loop playsinline preload="metadata"
|
| 85 |
+
src="https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/worlddit/model-card/v1/worlddit_libero_10_frontview_task06_episode01.mp4"></video>
|
| 86 |
+
<strong>LIBERO Long</strong><br>
|
| 87 |
+
Task 6, front view.
|
| 88 |
+
</td>
|
| 89 |
+
</tr>
|
| 90 |
+
</table>
|
| 91 |
+
|
| 92 |
+
[Spatial MP4](https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/worlddit/model-card/v1/worlddit_libero_spatial_frontview_task05_episode01.mp4)
|
| 93 |
+
路 [Object MP4](https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/worlddit/model-card/v1/worlddit_libero_object_agentview_task08_episode01.mp4)
|
| 94 |
+
路 [Goal MP4](https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/worlddit/model-card/v1/worlddit_libero_goal_sideview_task10_episode01.mp4)
|
| 95 |
+
路 [Long MP4](https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/worlddit/model-card/v1/worlddit_libero_10_frontview_task06_episode01.mp4)
|
| 96 |
+
|
| 97 |
+
## Release snapshot
|
| 98 |
+
|
| 99 |
+
| Reported LIBERO result | Released model |
|
| 100 |
+
|---|---|
|
| 101 |
+
| **94.9 percent** selection aware mean<br>**1,898 of 2,000** successful episodes | **399.084 million** total parameters<br>**135.107 million** trainable parameters |
|
| 102 |
+
| **98.0 percent** Spatial<br>**97.0 percent** Object | **Three** observation frames<br>**Seven** predicted actions |
|
| 103 |
+
| **92.8 percent** Goal<br>**91.8 percent** Long | **Three** actions executed before replanning<br>**Seven** action dimensions |
|
| 104 |
|
| 105 |
+
| Checkpoints | Runtime | Encoders and environment |
|
| 106 |
|---|---|---|
|
| 107 |
+
| Spatial<br>Object<br>Goal<br>Long | `inference.py`<br>`eval.py`<br>`config.json` | MAE ViT B<br>OpenAI CLIP ViT B 32<br>SafeTensors and pinned requirements |
|
|
|
|
|
|
|
|
|
|
| 108 |
|
| 109 |
+
The repository is self contained for WorldDiT inference. LIBERO provides the
|
| 110 |
+
benchmark environments, assets, task definitions, and initial states.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 111 |
|
| 112 |
The released runtime and checkpoints were revalidated from a clean installation
|
| 113 |
+
on eight RTX Pro 6000 Blackwell GPUs. The result is selection aware because
|
| 114 |
+
three hundred episodes per suite informed staged checkpoint selection before
|
| 115 |
+
the final five hundred episode score was assembled.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 116 |
|
| 117 |
## Run a smoke test
|
| 118 |
|
|
|
|
| 169 |
| Future visual supervision is present | No future visual output is requested |
|
| 170 |
| The complete training objective is active | Three actions execute before replanning |
|
| 171 |
|
| 172 |
+
## Reference
|
| 173 |
+
|
| 174 |
+
<details>
|
| 175 |
+
<summary><strong>Full evaluation commands</strong></summary>
|
| 176 |
|
| 177 |
### One GPU
|
| 178 |
|
|
|
|
| 206 |
libero_10
|
| 207 |
```
|
| 208 |
|
| 209 |
+
</details>
|
| 210 |
+
|
| 211 |
+
<details>
|
| 212 |
+
<summary><strong>Repository contents</strong></summary>
|
| 213 |
|
| 214 |
```text
|
| 215 |
.
|
|
|
|
| 230 |
`dependencies/` contains the frozen visual and language encoder weights needed
|
| 231 |
by the released policy. No additional model downloads are required.
|
| 232 |
|
| 233 |
+
</details>
|
| 234 |
+
|
| 235 |
+
<details>
|
| 236 |
+
<summary><strong>Inference API and tensor shapes</strong></summary>
|
| 237 |
|
| 238 |
```python
|
| 239 |
from inference import load_model
|
|
|
|
| 252 |
|
| 253 |
Evaluation uses the final temporal slot of the predicted action tensor.
|
| 254 |
|
| 255 |
+
</details>
|
| 256 |
+
|
| 257 |
+
<details>
|
| 258 |
+
<summary><strong>Architecture details</strong></summary>
|
| 259 |
|
| 260 |
| Component | Specification |
|
| 261 |
|---|---|
|
|
|
|
| 269 |
| Evaluation | Headless LIBERO with EGL |
|
| 270 |
| Checkpoint format | SafeTensors |
|
| 271 |
|
| 272 |
+
</details>
|
|
|
|
|
|
|
|
|
|
|
|
|
| 273 |
|
| 274 |
+
## Use and scope
|
| 275 |
|
| 276 |
+
| Intended use | Scope of the release |
|
| 277 |
+
|---|---|
|
| 278 |
+
| Research on language conditioned robot manipulation in the LIBERO simulator. The released checkpoints support reproduction, evaluation, and architecture research across the four released suites. | The results describe LIBERO simulation under the released evaluation protocol. They do not establish real robot reliability, safety, or transfer across embodiments. |
|
| 279 |
+
| The released checkpoints cover all four LIBERO suites. | The release does not isolate the causal contribution of the future visual target. Total parameter count does not measure training cost, deployment latency, or runtime efficiency. |
|
|
|
|
|
|
|
|
|
|
| 280 |
|
| 281 |
## Authors and contact
|
| 282 |
|