card: RoboCasa Kitchen success-rate table (arm x step), generated from metrics/eval_metrics.json

README.md

@@ -2,6 +2,60 @@
 license: apache-2.0
 tags: [video-world-model, robotics, robocasa, wan2.2, context-pooling, ibot]
 ---
+<!-- KITCHEN-SR:BEGIN -->
+### RoboCasa Kitchen — success rate (24 tasks x 50 episodes = 1200)
+
+| arm | pooling | 20k | 40k | 60k | 80k | 100k |
+|---|---|---|---|---|---|---|
+| `xattn_ffn_L3` | xattn + FFN refine @ L3 | — | — | **61.92%** | **64.50%** | **65.92%** |
+| `xattn_L3` | xattn (no FFN) @ L3 | — | — | **61.42%** | 77.78%<sup>†0/24</sup> | — |
+| `avg_L3` | avg @ L3 | — | — | **60.50%** | — | — |
+
+EMA weights (video DiT + action head), `seed42av`, `N_ENVS=1`, dropcur=1, budget 50 ep/task.
+Per-task numbers and the full protocol: [`metrics/eval_metrics.json`](metrics/eval_metrics.json).
+**†** = partial cell, fewer than 24 tasks at 50 episodes — not comparable with the bolded ones.
+
+Read the arms against each other at the SAME step only: these are single-seed cells.
+
+**Why `N_ENVS=1`.** Rendering several MuJoCo envs through one EGL context mis-serves frames —
+a camera is handed another camera's buffer (`wrist<-right`, `left<-right`, `right<-left`), with
+BGR channel swaps, persisting ~16 steps. Measured rate, per render call:
+
+| setting | mis-served | of |
+|---|---|---|
+| `N_ENVS=5` (n5free) | **5.9%** | 1,338 / 22,491 |
+| `N_ENVS=1` | **0.05%** | 11 / 22,485 |
+
+`make_current()` before `read_pixels` and a verify-retry both failed to move it in a 250-episode
+A/B, so the setting is the mitigation, not a code fix. Every cell above ran `N_ENVS=1`.
+
+**Audit, `xattn_L3` 60k (2026-09-15).** All 58 task logs report `N_ENVS=1`, `EVAL_WEIGHTS=ema`,
+zero `live`; 24/24 tasks hold exactly 50 finished episodes; no suffix-less (interrupted) videos,
+no truncated files, no exceptions or OOM. Note this is the *setting* argument — the verify-retry
+counter was not enabled in these runs, so mis-serves were not counted directly.
+
+### ⚠ These cells are not comparable with previously published Wan2.2 cells
+
+`N_ENVS` does two jobs at once, and dropping it to 1 gave up the second one:
+
+| | concurrency | scene seeds |
+|---|---|---|
+| protocol (`N_ENVS=5`) | 5 renderers at once | **5 scenes × 10 episodes**, `env_seed = SEED + idx` |
+| what ran here (`N_ENVS=1`) | 1 renderer | **1 scene × 50 episodes**, `env_seed = 42` only |
+
+`eval_wam_dit4dit_kitchen.sh:308` states the 5×10 grid *is* the measurement protocol and that
+lowering `N_ENVS` breaks comparability; its `seq-env` escape keeps the grid and only serialises
+the renderers (`N_ENVS itself never drops in either mode`). These runs took neither path — the
+local wrapper set `N_ENVS=1` on the parallel path, and every episode file is `env00`. The
+wrapper also exported an `ENV_LAYOUT=5` that the eval script never reads, so the banner printed
+`layout=5/1` and the grid looked intact while the seed set was a fifth of what it should be.
+
+So: **the arm-vs-arm ordering above holds** — all eight cells ran the identical single-scene
+setting — but each absolute number is "on this one scene", not a 5-scene average. Re-runs under
+the real protocol are in flight as separate `_env5` cells; until they land, do not place these
+numbers beside cells measured with the grid.
+<!-- KITCHEN-SR:END -->
+
 # WAM_DIT4DIT — context pooling on the **finalized B-method** base (RoboCasa kitchen)
 
 Wan2.2-TI2V-5B video DiT + 48-joint action head, `training_mode=joint`. The base is suhyeok's finalized