| # pi0.5 RoboCasa eval stack — negmesh8 160-episode replay benchmark |
|
|
| Everything needed to reproduce our evaluation on another cluster. Built and verified |
| 2026-09-22/23 on lab-gpu26. |
|
|
| ## What this evaluates |
|
|
| `PickPlaceCounterToCabinet`, 8 "hard mesh" objects × 20 exact-replay episodes = **160**. |
| Three augmentation arms fine-tuned from the same `pi05_ctc502_60k` warm start, plus the |
| base itself as reference. |
|
|
| ## Results so far (L40S, `per_query_epid`, action seed 42) |
|
|
| | checkpoint | all-160 | |
| |---|---| |
| | **base `pi05_ctc502_60k`** | **17/160 = 10.62%** | |
| | published reference for the same 160 | 7/160 = 4.38% — **not comparable, see below** | |
|
|
| Per object (base): AluminumFoil006 0/20 · BlenderJug023 3/20 · BlenderJug024 5/20 · |
| Jar025 4/20 · Juice008 0/20 · SyrupBottle006 2/20 · teapot_7 2/20 · wine_5 1/20 |
|
|
| ## THE THREE SERVING BUGS — read before running anything |
|
|
| The published 4.38% came from a harness that served the checkpoint wrongly. All three |
| must be right or your numbers are meaningless: |
|
|
| | | wrong (published) | correct | |
| |---|---|---| |
| | `action_horizon` | 10 | **50** | |
| | normalization | quantile (q01/q99) | **z-score** for the base | |
| | norm stats | `robocasa365_human300` | **the checkpoint's own ctc502 stats** | |
|
|
| Measured impact: **1/8 → 5/8** on identical seeded episodes; **7/160 → 17/160** on the |
| replay set. `jobs/robocasa_eval/assert_norm_config.py` gates every job against this — |
| run it, don't skip it. |
|
|
| **Fine-tuned arms differ from the base here**: VACE / MIMICGEN / POSE6DAUG were trained |
| with `use_quantile_norm=True` on `ctc502_qnorm`, so they must be served **quantile**, |
| not z-score. Each ships its own `assets/ctc502_qnorm/`. Crossing these silently corrupts |
| results. |
|
|
|
|
| ## TWO different "VACE" checkpoint sets exist — do not cross them |
|
|
| | HF repo | run | init | data | ships `assets/` | serve with | |
| |---|---|---|---|---|---| |
| | `pi05-VACE-negmesh8-243ep-ctc502-60k-30k-h100x2` | 19500 | `pi05_ctc502_60k` | 243 eps (drop13) | `ctc502_qnorm` | `..._vace_negmesh` (quantile) | |
| | `pi05-transfer-vace-negmesh256-allintra-bs64-30k-h100x4` | 18808 | **`pi05_base`** | **256 eps** | **`robocasa365_human300`** | a config with `asset_id="robocasa365_human300"` | |
|
|
| Only the first is the VACE arm of the three-way comparison in this package. The second is an |
| earlier run on a different initialization, different data and different normalization; its |
| numbers are not comparable to anything here. |
|
|
| Each checkpoint ships its own `assets/<asset_id>/`, so `policy_config.py` resolves from the |
| checkpoint itself — serving one with the other's config **fails loudly** (`norm stats not |
| found ... checkpoint assets/ contains: [...]`) rather than corrupting silently. Run |
| `jobs/robocasa_eval/assert_norm_config.py` and trust it. |
|
|
| ## GPU ARCHITECTURE IS A CONTROLLED VARIABLE |
|
|
| A100 vs L40S flips ~2 of 8 episodes. Every cross-architecture episode pair diverges at |
| **step 0, query 0** — the initial EGL rasterisation differs. Within one architecture, |
| two different physical GPUs agreed on 8/8 outcomes. |
|
|
| **Run every checkpoint you intend to compare on the same GPU model.** Our numbers are |
| L40S. `summarize_shards.py` refuses to aggregate across mixed `gpu_model` values. |
|
|
| ## Setup |
|
|
| 1. `openpi` from `Ronaldo-GOAT/transfer` (`transfer.tar`) — includes the **vendored |
| robosuite** with `load_model_on_init`. The PyPI robosuite 1.5.2 reports the same |
| version string but lacks it and will fail `gym.make`. Uninstall PyPI robosuite, then |
| `pip install -e third_party/robosuite`, and verify `robosuite.__file__` is inside the |
| bundle. |
| 2. RoboCasa assets (~16 GB extracted, 123,586 files) — `jobs/robocasa_eval/fetch_robocasa_assets.py`. |
| Not bundled. robocasa's own `download_kitchen_assets.py` calls `input()` and omits the |
| fixtures archive; use ours. |
| 3. **The 160-episode replay bank — get the right one.** It now ships **in this repo** at |
| `eval_stack/replay_bank_negmesh8/episodes/<object>/episode_0NN/` (968 files, 40.6 MB). |
| Mirror of `Ronaldo-GOAT/pi05-neg-mesh:episodes/`. 8 objects x 20 episodes, env seeds |
| 10000-10019, **policy/action seed 42**. Each episode ships `initial_model.xml.gz`, |
| `initial_sim_state.npz`, `initial_observation.npz`, `episode_metadata.pkl`, |
| `replay_state.json`, `result.json`. ~40 MB total. |
|
|
| **Verify you have the right bank** — `ls episodes/` must be exactly these 8: |
| ``` |
| aluminum_foil__AluminumFoil006 blender_jug__BlenderJug023 |
| blender_jug__BlenderJug024 jar__Jar025 |
| juice__Juice008 syrup_bottle__SyrupBottle006 |
| teapot__teapot_7 wine__wine_5 |
| ``` |
| ep_id -> object is contiguous blocks of 20 in that alphabetical order: 0-19 AluminumFoil006, |
| 20-39 BlenderJug023, 40-59 BlenderJug024, 60-79 Jar025, 80-99 Juice008, |
| 100-119 SyrupBottle006, 120-139 teapot_7, 140-159 wine_5. |
| |
| **There is a second, incompatible 160-episode bank** shipped in the same upstream repo: |
| `actaug/eval/table2_exact160/bank_pnpcountertocab_mimicgen8_exact160_*`. Its objects are |
| `donut_5, Jar023, MeasuringCup009, SoapDispenser010, steak_8, SyrupBottle005, teapot_6, |
| teapot_7` — **only `teapot_7` overlaps ours**, and its policy seed is **12345**, not 42. |
| It cannot score these checkpoints, and its `eval_exact160.sh` is a GR00T harness that |
| cannot even load them. Watch for the near-collisions: Jar**023**/Jar**025**, |
| SyrupBottle**005**/**006**, teapot_**6**/**7**. See `docs/EVAL_SETS.md`. |
| 4. Overlay `harness/` onto the openpi tree: |
| `main.py`, `get_eval_stats.py` → `examples/robocasa/` · `serve_policy.py` → `scripts/` · |
| `policy.py`, `policy_config.py` → `src/openpi/policies/` · `config.py` → `src/openpi/training/` |
| |
| ## Document precedence |
| |
| `README.md` (this file) > `docs/EVAL_SETS.md` > `docs/EVAL_SEEDS.md`. The last is the |
| upstream doc kept for provenance; it states `action_horizon=10`, which is **wrong** for |
| every checkpoint here. Its header now says so. |
| |
| ## Run — the 160-episode replay benchmark (what all our numbers use) |
| |
| ```bash |
| CKPT_DIR=<step_dir> # containing params/ and assets/ |
| CONFIG_NAME=<see below> |
| REPLAY_ROOT=eval_stack/replay_bank_negmesh8/episodes |
| ENV_SEED_OFFSET=0 NUM_TRIALS=160 |
| bash jobs/robocasa_eval.sbatch |
| python jobs/robocasa_eval/summarize_shards.py --dir <out> |
| ``` |
| |
| Configs: base → `pi05_robocasa_ctc502_60k_eval` (z-score) · VACE → |
| `..._vace_negmesh` · MIMICGEN → `..._mimicgen_aug256` · POSE6DAUG → `..._pose6daug_neg256` |
| (all quantile). |
| |
| Fixed protocol: `action_seed=42`, `replan_steps=5`, `resize_size=224`, camera 256, |
| `mp4_roundtrip=True`, `generative_textures=False`, noise |
| `default_rng(42 + ep_id*1_000_003 + query_idx)`, and |
| `XLA_FLAGS=--xla_gpu_deterministic_ops=true --xla_gpu_autotune_level=0`. |
|
|
| ## Two settings that cost us hours |
|
|
| - **`XLA_PYTHON_CLIENT_MEM_FRACTION=0.35`**, not 0.9. At 0.9 JAX takes 43 GB of a 48 GB |
| L40S and the MuJoCo/EGL contexts die with `GL_FRAMEBUFFER_UNSUPPORTED` (0x8cdd) once |
| you run more than ~2 workers. |
| - **mp4 round-trip belongs inside the `if not action_plan:` branch.** Outside it, the |
| encode runs every step and is discarded on 4 of 5 — **3.8× slower**, measured on |
| identical episodes with identical outcomes. |
| |
| Parallelism: one server per GPU, K=4 workers, sharded by `--args.env_seed_offset`. |
| Verified bit-identical noise and 100% outcome agreement vs serial — |
| `verify_parallel_seeding.py`. ~2.5× aggregate speedup, ~2.2 h per 160-episode target. |
| |
| ## Reproducibility |
| |
| Everything the code controls is exact. The residual is MuJoCo EGL rasterisation, which is |
| not bit-reproducible across processes (~1 pixel / 1 LSB). Measured: 4/8 episodes |
| bit-identical across cold processes, **8/8 agreeing on outcome, identical aggregate**. |
| Long rollouts diverge more often; one 740-step episode was bit-identical throughout. |
| |
| **Report success rates over the full set, never per-episode outcomes.** |
| |
| ## Comparability caveats |
| |
| - **MIMICGEN has 0 AluminumFoil006 training episodes** (VACE has 29) and only 5 Jar025 |
| (VACE 30). Those 40 episodes are effectively zero-shot for it. Use the **common-140** |
| subset as the head-to-head; report the 20 AluminumFoil006 episodes separately. |
| - **POSE6DAUG stopped at 5000 steps**; VACE/MIMICGEN ran to 30000. Compare only at |
| matched steps (1000/1500/5000), where all three saw the same LR schedule |
| (`decay_steps=30_000` deliberately unchanged). |
| - 20 episodes per object ⇒ one flip is 5 points. Per-object differences of one or two |
| successes are noise. |
| |
