hackhackhack66666 commited on
Commit
1a83227
·
verified ·
1 Parent(s): 9357fed

Add Blockwise-OAT baseline artifacts (README.md)

Browse files
Files changed (1) hide show
  1. README.md +27 -7
README.md CHANGED
@@ -18,16 +18,36 @@ generation on a frozen [OAT](https://arxiv.org/abs/2602.04215) policy.
18
 
19
  ## Summary
20
 
21
- | Metric | AR baseline | Blockwise (P=4, r=1) |
22
- |--------|-------------|----------------------|
 
 
23
  | LIBERO-10 mean SR | **58.73% ± 0.18%** | **52.33% ± 1.04%** |
24
- | Δ vs AR | | **-6.40 pp** |
25
- | Decoder speedup (bs=1) | — | **1.16×** |
26
- | E2E `predict_action` (bs=1) | 36.4 ms | 30.1 ms (**1.21×**) |
27
  | Tail train epochs | — | 15 (final CE 3.0607) |
28
 
29
- AR baseline exceeds the published OAT8 reference (~56.3%) on our cluster stack.
30
- Blockwise achieves **>1× decoder speedup** but **−6.4 pp** SR at 15 tail epochs (resume training planned).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
31
 
32
  ## Baseline artifacts (frozen)
33
 
 
18
 
19
  ## Summary
20
 
21
+ Primary SR reference: **OAT8 paper** on LIBERO-10 — **56.3%** ([OAT](https://arxiv.org/abs/2602.04215), external benchmark).
22
+
23
+ | Metric | AR (our eval) | Blockwise (P=4, r=1) |
24
+ |--------|---------------|----------------------|
25
  | LIBERO-10 mean SR | **58.73% ± 0.18%** | **52.33% ± 1.04%** |
26
+ | **Δ vs OAT paper (56.3%)** | **+2.43 pp** | **-3.97 pp** |
27
+ | Paired Δ (BW − AR, same protocol) | — | **-6.40 pp** |
 
28
  | Tail train epochs | — | 15 (final CE 3.0607) |
29
 
30
+ Our frozen AR checkpoint reproduces above the paper on this cluster stack (58.73% vs 56.3%).
31
+ Blockwise trades SR for faster token generation; tail training was only 15 epochs (resume planned).
32
+
33
+ ### Inference speed (V100, cuda:0)
34
+
35
+ **Decoder-only** — 8 action tokens after `cond` is computed (`benchmark_blockwise_vs_ar`, warmup=10, 50 repeats):
36
+
37
+ | Batch | AR | Blockwise | Speedup |
38
+ |-------|-----|-----------|---------|
39
+ | bs=1 | 22.3 ms | 19.3 ms | **1.16×** |
40
+ | bs=8 | 31.4 ms | 26.6 ms | **1.18×** |
41
+
42
+ **End-to-end `predict_action`** — vision encoder + decoder + detokenize (warmup=20, 100 repeats):
43
+
44
+ | Batch | AR | Blockwise | Speedup |
45
+ |-------|-----|-----------|---------|
46
+ | bs=1 | 36.4 ms | 30.1 ms | **1.21×** |
47
+ | bs=8 | 37.0 ms | 34.8 ms | **1.06×** |
48
+
49
+ Decoder speedup is modest (~14–18% faster at bs=1) because the tail module is comparable in size to the AR stack;
50
+ e2e gain is smaller still when the vision encoder dominates latency.
51
 
52
  ## Baseline artifacts (frozen)
53