Describe the selected release candidate, usage, and validation limits
Browse files
README.md
ADDED
|
@@ -0,0 +1,111 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language: en
|
| 3 |
+
tags:
|
| 4 |
+
- robotics
|
| 5 |
+
- libero
|
| 6 |
+
- video-to-action
|
| 7 |
+
- openwam
|
| 8 |
+
pipeline_tag: robotics
|
| 9 |
+
---
|
| 10 |
+
|
| 11 |
+
# OpenWAM VTA LIBERO-10
|
| 12 |
+
|
| 13 |
+
A dual-expert video-then-action policy for LIBERO-10. This release contains
|
| 14 |
+
the complete video and action experts from the selected step-15,000 checkpoint.
|
| 15 |
+
It is not an inverse-dynamics-only model or a video-backbone-only export.
|
| 16 |
+
|
| 17 |
+
The policy first predicts future video latents from the task prompt and observed
|
| 18 |
+
scene, then predicts robot actions conditioned on those latents. During
|
| 19 |
+
closed-loop execution, new real observations update its recurrent history.
|
| 20 |
+
This is a single trained VTA policy, not a composition with an external IDM.
|
| 21 |
+
|
| 22 |
+
**Status:** private release candidate at
|
| 23 |
+
[`OpenWAM-Stanford/OpenWAM-VTA-LIBERO10`](https://huggingface.co/OpenWAM-Stanford/OpenWAM-VTA-LIBERO10).
|
| 24 |
+
The selected checkpoint is step **15,000**, replacing the earlier proposed
|
| 25 |
+
VTA release. The inference weights contain approximately 7.67 billion FP32
|
| 26 |
+
parameter elements (about 30.7 GB); this is not an inference-memory estimate.
|
| 27 |
+
|
| 28 |
+
Project: https://openwam.github.io/
|
| 29 |
+
|
| 30 |
+
Code: https://github.com/OpenWAM/OpenWAM
|
| 31 |
+
|
| 32 |
+
## Inference Contract
|
| 33 |
+
|
| 34 |
+
- Program: `video_then_action`, with `video_only` recurrent history visibility.
|
| 35 |
+
- Two 128x128 RGB camera views, concatenated into a 128x256 frontend input.
|
| 36 |
+
- Four predicted latent frames per chunk and sixteen raw 7D action commands.
|
| 37 |
+
- Attention window: 30; 20 video and 20 action denoising steps.
|
| 38 |
+
- Video guidance scale: 5; conditioned action generation with scale 1.
|
| 39 |
+
- Per-chunk proprioception, single-frame condition latents, and the
|
| 40 |
+
`legacy_prefix_single_frame_perchunk_proprio` sequence contract.
|
| 41 |
+
|
| 42 |
+
Use `resolved_config.yaml` with `model_state.pt` through the maintained
|
| 43 |
+
OpenWAM checkpoint loading path. Configure the external LingBot-VA VAE,
|
| 44 |
+
text encoder, and tokenizer assets using `paths.models.lingbot_va_base`.
|
| 45 |
+
Their source revision is recorded in `release_metadata.json`. The full
|
| 46 |
+
checkpoint supplies the policy weights; no private initialization checkpoint
|
| 47 |
+
is needed for inference.
|
| 48 |
+
|
| 49 |
+
## Download And Run
|
| 50 |
+
|
| 51 |
+
Authenticate with an account that can read this private repository, then download
|
| 52 |
+
the inference bundle:
|
| 53 |
+
|
| 54 |
+
```bash
|
| 55 |
+
hf download OpenWAM-Stanford/OpenWAM-VTA-LIBERO10 --local-dir vta-libero10
|
| 56 |
+
cd vta-libero10
|
| 57 |
+
sha256sum --check SHA256SUMS
|
| 58 |
+
```
|
| 59 |
+
|
| 60 |
+
Install OpenWAM and its LIBERO simulator dependencies following the code
|
| 61 |
+
repository instructions. Use the code revision recorded in
|
| 62 |
+
`release_metadata.json`, and set `paths.models.lingbot_va_base` in
|
| 63 |
+
`configs/local_paths.yaml` to the matching frontend assets. Run from that
|
| 64 |
+
checkout with `CKPT` set to the downloaded bundle's absolute path:
|
| 65 |
+
|
| 66 |
+
```bash
|
| 67 |
+
MUJOCO_GL=osmesa python scripts/run_libero_policy.py \
|
| 68 |
+
--cfg "$CKPT/resolved_config.yaml" --checkpoint "$CKPT/model_state.pt" \
|
| 69 |
+
--benchmark libero_10 --task-id 0 --episode-idx 0 --seed 0 \
|
| 70 |
+
--frontend-encode-mode lingbot_streaming_vae \
|
| 71 |
+
--inference-window-size 30 --execute-frame-chunk-size 4 \
|
| 72 |
+
--startup-model-obs-frames 1 --startup-env-init-steps 5 \
|
| 73 |
+
--max-timestep 800 --max-chunks 50 \
|
| 74 |
+
--runtime-device cuda:0 --action-device cuda:0 \
|
| 75 |
+
--frontend-device cuda:0 --decode-device cuda:0 \
|
| 76 |
+
--save-rollout-video --output-dir outputs/vta-libero10
|
| 77 |
+
```
|
| 78 |
+
|
| 79 |
+
This example uses the maintained CLI and preserves the checkpoint's inference
|
| 80 |
+
contract. A fresh GPU rollout of this packaged checkpoint remains to be run;
|
| 81 |
+
the example is not a new benchmark result. Do not use partial-checkpoint loading
|
| 82 |
+
or substitute another checkpoint's inference configuration.
|
| 83 |
+
|
| 84 |
+
## Contents And Evidence
|
| 85 |
+
|
| 86 |
+
| File | Contents |
|
| 87 |
+
| --- | --- |
|
| 88 |
+
| `model_state.pt` | Tensor-only policy checkpoint, including both experts |
|
| 89 |
+
| `resolved_config.yaml` | Portable architecture and inference settings |
|
| 90 |
+
| `release_metadata.json` | Code/assets revisions, tensor counts, source/export hashes |
|
| 91 |
+
| `SHA256SUMS` | Integrity checks for all release files |
|
| 92 |
+
| `README.md` | This model card |
|
| 93 |
+
|
| 94 |
+
The bundle contains tensor-only policy weights, a portable inference config,
|
| 95 |
+
checksums, and release metadata. It does not contain optimizer state, training
|
| 96 |
+
logs, or private dataset paths. Every active policy tensor retains its exact
|
| 97 |
+
original values and dtype. The export omits only the two tensors belonging to
|
| 98 |
+
the retired post-decoded feature projection (`visual_tower.decoder.proj`),
|
| 99 |
+
which is absent from the maintained VTA inference architecture. Neither the
|
| 100 |
+
VAE nor either expert is removed. Source and exported file hashes are recorded
|
| 101 |
+
separately; the exported file is not byte-identical to the training archive.
|
| 102 |
+
|
| 103 |
+
No benchmark success rate is assigned to this checkpoint by this release.
|
| 104 |
+
Results measured with other VTA checkpoints or other inference overrides must
|
| 105 |
+
not be attributed to these weights. Architecture/configuration checks establish
|
| 106 |
+
loadability, not closed-loop policy performance.
|
| 107 |
+
|
| 108 |
+
Weight-license confirmation and a rollout smoke test remain release gates.
|
| 109 |
+
The policy was developed for simulated LIBERO manipulation; no real-robot
|
| 110 |
+
safety or generalization guarantee is made. Private availability does not
|
| 111 |
+
constitute a redistribution license or approval for deployment.
|