Model card: results, download and launcher variables
Browse filesAdds the write-up link and reference results (Table 3), a Download section with `hf download` and the launcher variables, `hf download` for the Wan files, and links to the evaluation docs. Updates the README.md line in ASSET_SHA256SUMS. Weights, stats, videos, LICENSE and NOTICE are unchanged.
- ASSET_SHA256SUMS +1 -1
- README.md +36 -3
ASSET_SHA256SUMS
CHANGED
|
@@ -10,5 +10,5 @@ d2023cc6a5a33abd405c3a9f63b7fabc84df6c917798bd7080114bcbe9294d89 stats/libero_s
|
|
| 10 |
905ccc23cf99504e6f17f9fac2b5115896304be2d2dc0a9522f8886095c0548c videos/libero-object.mp4
|
| 11 |
b5393c1ae185b1d05ad79edde9b5dc7b10543694b5c6be0d44084362e962e51d videos/libero-goal.mp4
|
| 12 |
38e0b77c8ec5988f3357ad0e28b5456b42c95b5698e96961d5d0cb2f9433751a videos/libero-long.mp4
|
| 13 |
-
|
| 14 |
f8337593ad4c0174982bcd6219ba666128bf5b2bf587c338921b89e2dfd7d1ff replay.mp4
|
|
|
|
| 10 |
905ccc23cf99504e6f17f9fac2b5115896304be2d2dc0a9522f8886095c0548c videos/libero-object.mp4
|
| 11 |
b5393c1ae185b1d05ad79edde9b5dc7b10543694b5c6be0d44084362e962e51d videos/libero-goal.mp4
|
| 12 |
38e0b77c8ec5988f3357ad0e28b5456b42c95b5698e96961d5d0cb2f9433751a videos/libero-long.mp4
|
| 13 |
+
cf481fdf2c4e2dc6ceda595b933ca210c98465cc743bcff7155d434bb459b10d README.md
|
| 14 |
f8337593ad4c0174982bcd6219ba666128bf5b2bf587c338921b89e2dfd7d1ff replay.mp4
|
README.md
CHANGED
|
@@ -16,6 +16,7 @@ pipeline_tag: robotics
|
|
| 16 |
A latent world-action policy for the [LIBERO](https://github.com/Lifelong-Robot-Learning/LIBERO) benchmark, fine-tuned by Advanced Micro Devices, Inc. from [Wan2.2-TI2V-5B](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B).
|
| 17 |
|
| 18 |
Code: [AMD-AGI/faster-action-models](https://github.com/AMD-AGI/faster-action-models).
|
|
|
|
| 19 |
|
| 20 |
Part of the [Compact Action Expert (CAE) collection](https://huggingface.co/collections/amd/compact-action-expert-cae-6ace112e7bd665ff8c107ed2).
|
| 21 |
|
|
@@ -23,6 +24,19 @@ Part of the [Compact Action Expert (CAE) collection](https://huggingface.co/coll
|
|
| 23 |
|
| 24 |
The Compact Action Expert combines a 32-dimensional action expert (about 30M parameters) with a suite-conditioned K8 × d16 action VAE. The VAE encodes each 32 × 7 action chunk as 8 latent tokens of width 16, and the action expert denoises those latent chunks.
|
| 25 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
## Files
|
| 27 |
|
| 28 |
| File | Description |
|
|
@@ -38,22 +52,41 @@ Use the statistics file that matches the suite under evaluation. `policy.pt` con
|
|
| 38 |
|
| 39 |
SHA-256 checksums are in [`ASSET_SHA256SUMS`](ASSET_SHA256SUMS).
|
| 40 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 41 |
## Runtime weights
|
| 42 |
|
| 43 |
This repository does not include the Wan2.2 VAE, the umT5-XXL text encoder, or the umT5-XXL tokenizer. Download them from their upstream repositories:
|
| 44 |
|
| 45 |
```bash
|
| 46 |
-
|
| 47 |
Wan2.2_VAE.pth models_t5_umt5-xxl-enc-bf16.pth \
|
| 48 |
--local-dir checkpoints/Wan-AI/Wan2.2-TI2V-5B
|
| 49 |
-
|
| 50 |
--include "google/umt5-xxl/*" \
|
| 51 |
--local-dir checkpoints/Wan-AI/Wan2.1-T2V-1.3B
|
|
|
|
| 52 |
```
|
| 53 |
|
| 54 |
The full Wan video DiT is not required. `policy.pt` already contains the fine-tuned video and action experts.
|
| 55 |
|
| 56 |
-
LIBERO simulator assets, BDDL task files, and initial states are also required.
|
| 57 |
|
| 58 |
## Intended use
|
| 59 |
|
|
|
|
| 16 |
A latent world-action policy for the [LIBERO](https://github.com/Lifelong-Robot-Learning/LIBERO) benchmark, fine-tuned by Advanced Micro Devices, Inc. from [Wan2.2-TI2V-5B](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B).
|
| 17 |
|
| 18 |
Code: [AMD-AGI/faster-action-models](https://github.com/AMD-AGI/faster-action-models).
|
| 19 |
+
Write-up and reference results: [Optimizing World Action Models on AMD GPUs](https://rocm.blogs.amd.com/artificial-intelligence/faster-action-models/README.html).
|
| 20 |
|
| 21 |
Part of the [Compact Action Expert (CAE) collection](https://huggingface.co/collections/amd/compact-action-expert-cae-6ace112e7bd665ff8c107ed2).
|
| 22 |
|
|
|
|
| 24 |
|
| 25 |
The Compact Action Expert combines a 32-dimensional action expert (about 30M parameters) with a suite-conditioned K8 × d16 action VAE. The VAE encodes each 32 × 7 action chunk as 8 latent tokens of width 16, and the action expert denoises those latent chunks.
|
| 26 |
|
| 27 |
+
## Results
|
| 28 |
+
|
| 29 |
+
LIBERO (all four suites, 50 episodes per task) on one AMD Instinct™ MI355X GPU with ROCm 7.14.0 and PyTorch 2.11.0, simulated on the GPU (`--sim instinct`), with the GEAK-optimized inference path from the code repository. Reference results, from Table 3 of the [write-up](https://rocm.blogs.amd.com/artificial-intelligence/faster-action-models/README.html):
|
| 30 |
+
|
| 31 |
+
| Configuration | Success (%) | Latency per policy decision (ms) |
|
| 32 |
+
|---|---|---|
|
| 33 |
+
| FastWAM, no optimization (10 denoising steps) | 96.95 ± 0.22 | 222.55 |
|
| 34 |
+
| FastWAM + GEAK (10 steps) | 96.86 ± 0.30 | 24.17 |
|
| 35 |
+
| CAE-FastWAM + GEAK (10 steps) | 97.28 ± 0.20 | 12.82 |
|
| 36 |
+
| CAE-FastWAM + GEAK (4 steps) | 97.33 ± 0.26 | 8.83 |
|
| 37 |
+
|
| 38 |
+
Success is the mean ± standard deviation over 11 full LIBERO runs (5 for the 4-step row). The FastWAM rows use the authors' [`yuanty/fastwam`](https://huggingface.co/yuanty/fastwam) checkpoint. Reproduce the table with `scripts/reproduce_cae_fastwam.sh` in the code repository. Training took 22.7 hours on two AMD Instinct™ MI325X GPUs. Results are simulator measurements and may vary with configuration and software versions.
|
| 39 |
+
|
| 40 |
## Files
|
| 41 |
|
| 42 |
| File | Description |
|
|
|
|
| 52 |
|
| 53 |
SHA-256 checksums are in [`ASSET_SHA256SUMS`](ASSET_SHA256SUMS).
|
| 54 |
|
| 55 |
+
## Download
|
| 56 |
+
|
| 57 |
+
Run the downloads with a current `huggingface_hub` (`pip install -U huggingface_hub`), which provides the `hf` command. Releases before 0.34 only have the older spelling, `huggingface-cli download`. Download into a directory of its own, for example inside a clone of the code repository:
|
| 58 |
+
|
| 59 |
+
```bash
|
| 60 |
+
hf download amd/CAE-FastWAM --local-dir checkpoints/CAE-FastWAM
|
| 61 |
+
(cd checkpoints/CAE-FastWAM && grep -E '\.(pt|json)$' ASSET_SHA256SUMS | sha256sum -c -)
|
| 62 |
+
```
|
| 63 |
+
|
| 64 |
+
The evaluation launchers in the code repository take the files as:
|
| 65 |
+
|
| 66 |
+
```bash
|
| 67 |
+
export MODEL_CKPT=$PWD/checkpoints/CAE-FastWAM/policy.pt
|
| 68 |
+
export ACTION_VAE_CKPT=$PWD/checkpoints/CAE-FastWAM/action_vae.pt
|
| 69 |
+
export DATASET_STATS_TEMPLATE=$PWD/checkpoints/CAE-FastWAM/stats/{suite}.json
|
| 70 |
+
export TASK_CONFIG=libero_uncond_2cam224_1e-4_32hdim_latent_K8d16_freebits_pernorm_joint
|
| 71 |
+
```
|
| 72 |
+
|
| 73 |
## Runtime weights
|
| 74 |
|
| 75 |
This repository does not include the Wan2.2 VAE, the umT5-XXL text encoder, or the umT5-XXL tokenizer. Download them from their upstream repositories:
|
| 76 |
|
| 77 |
```bash
|
| 78 |
+
hf download Wan-AI/Wan2.2-TI2V-5B \
|
| 79 |
Wan2.2_VAE.pth models_t5_umt5-xxl-enc-bf16.pth \
|
| 80 |
--local-dir checkpoints/Wan-AI/Wan2.2-TI2V-5B
|
| 81 |
+
hf download Wan-AI/Wan2.1-T2V-1.3B \
|
| 82 |
--include "google/umt5-xxl/*" \
|
| 83 |
--local-dir checkpoints/Wan-AI/Wan2.1-T2V-1.3B
|
| 84 |
+
export DIFFSYNTH_MODEL_BASE_PATH=$PWD/checkpoints
|
| 85 |
```
|
| 86 |
|
| 87 |
The full Wan video DiT is not required. `policy.pt` already contains the fine-tuned video and action experts.
|
| 88 |
|
| 89 |
+
LIBERO simulator assets, BDDL task files, and initial states are also required. The evaluation commands are in [`MODEL_ZOO.md`](https://github.com/AMD-AGI/faster-action-models/blob/main/MODEL_ZOO.md) and [`docs/cae-fastwam-libero.md`](https://github.com/AMD-AGI/faster-action-models/blob/main/docs/cae-fastwam-libero.md).
|
| 90 |
|
| 91 |
## Intended use
|
| 92 |
|