Add Blockwise-OAT baseline artifacts (README.md)
Browse files
README.md
CHANGED
|
@@ -18,16 +18,36 @@ generation on a frozen [OAT](https://arxiv.org/abs/2602.04215) policy.
|
|
| 18 |
|
| 19 |
## Summary
|
| 20 |
|
| 21 |
-
|
| 22 |
-
|
|
|
|
|
|
|
| 23 |
| LIBERO-10 mean SR | **58.73% ± 0.18%** | **52.33% ± 1.04%** |
|
| 24 |
-
| Δ vs
|
| 25 |
-
|
|
| 26 |
-
| E2E `predict_action` (bs=1) | 36.4 ms | 30.1 ms (**1.21×**) |
|
| 27 |
| Tail train epochs | — | 15 (final CE 3.0607) |
|
| 28 |
|
| 29 |
-
AR
|
| 30 |
-
Blockwise
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
|
| 32 |
## Baseline artifacts (frozen)
|
| 33 |
|
|
|
|
| 18 |
|
| 19 |
## Summary
|
| 20 |
|
| 21 |
+
Primary SR reference: **OAT8 paper** on LIBERO-10 — **56.3%** ([OAT](https://arxiv.org/abs/2602.04215), external benchmark).
|
| 22 |
+
|
| 23 |
+
| Metric | AR (our eval) | Blockwise (P=4, r=1) |
|
| 24 |
+
|--------|---------------|----------------------|
|
| 25 |
| LIBERO-10 mean SR | **58.73% ± 0.18%** | **52.33% ± 1.04%** |
|
| 26 |
+
| **Δ vs OAT paper (56.3%)** | **+2.43 pp** | **-3.97 pp** |
|
| 27 |
+
| Paired Δ (BW − AR, same protocol) | — | **-6.40 pp** |
|
|
|
|
| 28 |
| Tail train epochs | — | 15 (final CE 3.0607) |
|
| 29 |
|
| 30 |
+
Our frozen AR checkpoint reproduces above the paper on this cluster stack (58.73% vs 56.3%).
|
| 31 |
+
Blockwise trades SR for faster token generation; tail training was only 15 epochs (resume planned).
|
| 32 |
+
|
| 33 |
+
### Inference speed (V100, cuda:0)
|
| 34 |
+
|
| 35 |
+
**Decoder-only** — 8 action tokens after `cond` is computed (`benchmark_blockwise_vs_ar`, warmup=10, 50 repeats):
|
| 36 |
+
|
| 37 |
+
| Batch | AR | Blockwise | Speedup |
|
| 38 |
+
|-------|-----|-----------|---------|
|
| 39 |
+
| bs=1 | 22.3 ms | 19.3 ms | **1.16×** |
|
| 40 |
+
| bs=8 | 31.4 ms | 26.6 ms | **1.18×** |
|
| 41 |
+
|
| 42 |
+
**End-to-end `predict_action`** — vision encoder + decoder + detokenize (warmup=20, 100 repeats):
|
| 43 |
+
|
| 44 |
+
| Batch | AR | Blockwise | Speedup |
|
| 45 |
+
|-------|-----|-----------|---------|
|
| 46 |
+
| bs=1 | 36.4 ms | 30.1 ms | **1.21×** |
|
| 47 |
+
| bs=8 | 37.0 ms | 34.8 ms | **1.06×** |
|
| 48 |
+
|
| 49 |
+
Decoder speedup is modest (~14–18% faster at bs=1) because the tail module is comparable in size to the AR stack;
|
| 50 |
+
e2e gain is smaller still when the vision encoder dominates latency.
|
| 51 |
|
| 52 |
## Baseline artifacts (frozen)
|
| 53 |
|