Robotics
PyTorch
world-action-model
libero
Viani commited on
Commit
605cea6
·
verified ·
1 Parent(s): db8471f

Model card: results, download and launcher variables

Browse files

Adds the write-up link and reference results (Table 3), a Download section with `hf download` and the launcher variables, `hf download` for the Wan files, and links to the evaluation docs. Updates the README.md line in ASSET_SHA256SUMS. Weights, stats, videos, LICENSE and NOTICE are unchanged.

Files changed (2) hide show
  1. ASSET_SHA256SUMS +1 -1
  2. README.md +36 -3
ASSET_SHA256SUMS CHANGED
@@ -10,5 +10,5 @@ d2023cc6a5a33abd405c3a9f63b7fabc84df6c917798bd7080114bcbe9294d89 stats/libero_s
10
  905ccc23cf99504e6f17f9fac2b5115896304be2d2dc0a9522f8886095c0548c videos/libero-object.mp4
11
  b5393c1ae185b1d05ad79edde9b5dc7b10543694b5c6be0d44084362e962e51d videos/libero-goal.mp4
12
  38e0b77c8ec5988f3357ad0e28b5456b42c95b5698e96961d5d0cb2f9433751a videos/libero-long.mp4
13
- f0ae5036e9a6952a06c2dc7c91fe6fd4a791a79ec1fdc2ae0043da24c424c418 README.md
14
  f8337593ad4c0174982bcd6219ba666128bf5b2bf587c338921b89e2dfd7d1ff replay.mp4
 
10
  905ccc23cf99504e6f17f9fac2b5115896304be2d2dc0a9522f8886095c0548c videos/libero-object.mp4
11
  b5393c1ae185b1d05ad79edde9b5dc7b10543694b5c6be0d44084362e962e51d videos/libero-goal.mp4
12
  38e0b77c8ec5988f3357ad0e28b5456b42c95b5698e96961d5d0cb2f9433751a videos/libero-long.mp4
13
+ cf481fdf2c4e2dc6ceda595b933ca210c98465cc743bcff7155d434bb459b10d README.md
14
  f8337593ad4c0174982bcd6219ba666128bf5b2bf587c338921b89e2dfd7d1ff replay.mp4
README.md CHANGED
@@ -16,6 +16,7 @@ pipeline_tag: robotics
16
  A latent world-action policy for the [LIBERO](https://github.com/Lifelong-Robot-Learning/LIBERO) benchmark, fine-tuned by Advanced Micro Devices, Inc. from [Wan2.2-TI2V-5B](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B).
17
 
18
  Code: [AMD-AGI/faster-action-models](https://github.com/AMD-AGI/faster-action-models).
 
19
 
20
  Part of the [Compact Action Expert (CAE) collection](https://huggingface.co/collections/amd/compact-action-expert-cae-6ace112e7bd665ff8c107ed2).
21
 
@@ -23,6 +24,19 @@ Part of the [Compact Action Expert (CAE) collection](https://huggingface.co/coll
23
 
24
  The Compact Action Expert combines a 32-dimensional action expert (about 30M parameters) with a suite-conditioned K8 × d16 action VAE. The VAE encodes each 32 × 7 action chunk as 8 latent tokens of width 16, and the action expert denoises those latent chunks.
25
 
 
 
 
 
 
 
 
 
 
 
 
 
 
26
  ## Files
27
 
28
  | File | Description |
@@ -38,22 +52,41 @@ Use the statistics file that matches the suite under evaluation. `policy.pt` con
38
 
39
  SHA-256 checksums are in [`ASSET_SHA256SUMS`](ASSET_SHA256SUMS).
40
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
41
  ## Runtime weights
42
 
43
  This repository does not include the Wan2.2 VAE, the umT5-XXL text encoder, or the umT5-XXL tokenizer. Download them from their upstream repositories:
44
 
45
  ```bash
46
- huggingface-cli download Wan-AI/Wan2.2-TI2V-5B \
47
  Wan2.2_VAE.pth models_t5_umt5-xxl-enc-bf16.pth \
48
  --local-dir checkpoints/Wan-AI/Wan2.2-TI2V-5B
49
- huggingface-cli download Wan-AI/Wan2.1-T2V-1.3B \
50
  --include "google/umt5-xxl/*" \
51
  --local-dir checkpoints/Wan-AI/Wan2.1-T2V-1.3B
 
52
  ```
53
 
54
  The full Wan video DiT is not required. `policy.pt` already contains the fine-tuned video and action experts.
55
 
56
- LIBERO simulator assets, BDDL task files, and initial states are also required.
57
 
58
  ## Intended use
59
 
 
16
  A latent world-action policy for the [LIBERO](https://github.com/Lifelong-Robot-Learning/LIBERO) benchmark, fine-tuned by Advanced Micro Devices, Inc. from [Wan2.2-TI2V-5B](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B).
17
 
18
  Code: [AMD-AGI/faster-action-models](https://github.com/AMD-AGI/faster-action-models).
19
+ Write-up and reference results: [Optimizing World Action Models on AMD GPUs](https://rocm.blogs.amd.com/artificial-intelligence/faster-action-models/README.html).
20
 
21
  Part of the [Compact Action Expert (CAE) collection](https://huggingface.co/collections/amd/compact-action-expert-cae-6ace112e7bd665ff8c107ed2).
22
 
 
24
 
25
  The Compact Action Expert combines a 32-dimensional action expert (about 30M parameters) with a suite-conditioned K8 × d16 action VAE. The VAE encodes each 32 × 7 action chunk as 8 latent tokens of width 16, and the action expert denoises those latent chunks.
26
 
27
+ ## Results
28
+
29
+ LIBERO (all four suites, 50 episodes per task) on one AMD Instinct™ MI355X GPU with ROCm 7.14.0 and PyTorch 2.11.0, simulated on the GPU (`--sim instinct`), with the GEAK-optimized inference path from the code repository. Reference results, from Table 3 of the [write-up](https://rocm.blogs.amd.com/artificial-intelligence/faster-action-models/README.html):
30
+
31
+ | Configuration | Success (%) | Latency per policy decision (ms) |
32
+ |---|---|---|
33
+ | FastWAM, no optimization (10 denoising steps) | 96.95 ± 0.22 | 222.55 |
34
+ | FastWAM + GEAK (10 steps) | 96.86 ± 0.30 | 24.17 |
35
+ | CAE-FastWAM + GEAK (10 steps) | 97.28 ± 0.20 | 12.82 |
36
+ | CAE-FastWAM + GEAK (4 steps) | 97.33 ± 0.26 | 8.83 |
37
+
38
+ Success is the mean ± standard deviation over 11 full LIBERO runs (5 for the 4-step row). The FastWAM rows use the authors' [`yuanty/fastwam`](https://huggingface.co/yuanty/fastwam) checkpoint. Reproduce the table with `scripts/reproduce_cae_fastwam.sh` in the code repository. Training took 22.7 hours on two AMD Instinct™ MI325X GPUs. Results are simulator measurements and may vary with configuration and software versions.
39
+
40
  ## Files
41
 
42
  | File | Description |
 
52
 
53
  SHA-256 checksums are in [`ASSET_SHA256SUMS`](ASSET_SHA256SUMS).
54
 
55
+ ## Download
56
+
57
+ Run the downloads with a current `huggingface_hub` (`pip install -U huggingface_hub`), which provides the `hf` command. Releases before 0.34 only have the older spelling, `huggingface-cli download`. Download into a directory of its own, for example inside a clone of the code repository:
58
+
59
+ ```bash
60
+ hf download amd/CAE-FastWAM --local-dir checkpoints/CAE-FastWAM
61
+ (cd checkpoints/CAE-FastWAM && grep -E '\.(pt|json)$' ASSET_SHA256SUMS | sha256sum -c -)
62
+ ```
63
+
64
+ The evaluation launchers in the code repository take the files as:
65
+
66
+ ```bash
67
+ export MODEL_CKPT=$PWD/checkpoints/CAE-FastWAM/policy.pt
68
+ export ACTION_VAE_CKPT=$PWD/checkpoints/CAE-FastWAM/action_vae.pt
69
+ export DATASET_STATS_TEMPLATE=$PWD/checkpoints/CAE-FastWAM/stats/{suite}.json
70
+ export TASK_CONFIG=libero_uncond_2cam224_1e-4_32hdim_latent_K8d16_freebits_pernorm_joint
71
+ ```
72
+
73
  ## Runtime weights
74
 
75
  This repository does not include the Wan2.2 VAE, the umT5-XXL text encoder, or the umT5-XXL tokenizer. Download them from their upstream repositories:
76
 
77
  ```bash
78
+ hf download Wan-AI/Wan2.2-TI2V-5B \
79
  Wan2.2_VAE.pth models_t5_umt5-xxl-enc-bf16.pth \
80
  --local-dir checkpoints/Wan-AI/Wan2.2-TI2V-5B
81
+ hf download Wan-AI/Wan2.1-T2V-1.3B \
82
  --include "google/umt5-xxl/*" \
83
  --local-dir checkpoints/Wan-AI/Wan2.1-T2V-1.3B
84
+ export DIFFSYNTH_MODEL_BASE_PATH=$PWD/checkpoints
85
  ```
86
 
87
  The full Wan video DiT is not required. `policy.pt` already contains the fine-tuned video and action experts.
88
 
89
+ LIBERO simulator assets, BDDL task files, and initial states are also required. The evaluation commands are in [`MODEL_ZOO.md`](https://github.com/AMD-AGI/faster-action-models/blob/main/MODEL_ZOO.md) and [`docs/cae-fastwam-libero.md`](https://github.com/AMD-AGI/faster-action-models/blob/main/docs/cae-fastwam-libero.md).
90
 
91
  ## Intended use
92