Add RACE2 ablation nojitter H20 40k package (EMA overlay, materialized model, logs, eval)

ablation/abl_nojitter_h20_40k/README.md

@@ -0,0 +1,36 @@
+# RACE2 ablation A6 — no label jitter (VLABench, H=20, 40k EMA)
+
+Config `pi05_vlabench_race2abl_nojitter_h20` (model class `Race2NoJitterPI0`), experiment `abl_nojitter_h20_s42`.
+Full RACE2 recipe with `jitter_prob = 0`: the FM pass is teacher-forced with the exact b\* (no jittered changepoint labels). Loss, conditioning sites, auxiliary pass and serving are identical to full RACE2.
+Ablation definitions: `source/scripts/race2_ablation/README.md`.
+
+| | |
+|---|---|
+| Init | official VLABench pi0.5 teacher (`teacher_official/`, SHA256 `f6762a23…`) |
+| Training | 40,000 steps, seed 42, global batch 64 = 2 GPUs × micro 32 (no accumulation), warmup 500 → 5e-5 → cosine → 5e-6, AdamW (0.9, 0.95, wd 1e-10, clip 1.0), EMA 0.999, λ_timing 0.05, λ_base_fm 0.1, n_tau 10, VLM (PaliGemma + SigLIP) frozen |
+| Data | `SeonghoonYu/vlabench_primitive_ft_lerobot_224` rev `5772f03c…` (5,000 ep), changepoints `vlabench_cps.npz` SHA256 `6e31b274…` |
+| Hardware | 2 × A100-SXM4-80GB, 2026-09-16 07:58 → 2026-09-16 23:23 UTC (~15.4 h, ~1.37 s/step) |
+| Final train metrics (step 39,900) | fm 0.030, bce 0.137, peak±1 0.71, pres_acc 0.81 |
+
+## Evaluation (40k EMA, horizon = replan = 20, seed 7, tracks 1/2/3/4/6, 50 ep/task, 2,460 ep, physics errors excluded: 30)
+
+| Model | SR | IS | PS |
+|---|---:|---:|---:|
+| PlainFT H20 | 28.04 | 61.86 | 44.77 |
+| RACE2 full H20 (`h20_40k/`) | 32.86 | 61.48 | 47.84 |
+| A1 timing-only H20 (`abl_timing_only_h20_40k/`) | 31.06 | 61.01 | 45.23 |
+| **A6 no label jitter H20 (this)** | **32.80** | **62.72** | **48.71** |
+
+Per track SR (A6): T1 52.53 · T2 26.22 · T3 26.82 · T4 22.72 · T6 35.74. Full details in `logs/eval_summary.json`.
+Aggregation: task mean → equal track weight; PhysicsError episodes excluded from the task denominator.
+
+## Files
+
+- `model.safetensors` — full serving model: frozen teacher VLM tensors + EMA tensors (action expert, projections, RACE timing head, field). SHA256 in `SHA256SUMS`.
+- `overlay/ema.safetensors`, `overlay/metadata.pt` — the slim trainable-only EMA checkpoint written by the trainer (exactly what was evaluated).
+- `assets/vlabench/vlabench_ft_primitive/norm_stats.json` — normalization statistics (identical to the teacher's).
+- `training_config.json` — readable copy of the training config from `metadata.pt`.
+- `logs/train_metrics.log` (metric lines), `logs/train.log.gz` (full trainer log), `logs/history.jsonl`, `logs/status.json`, `logs/commands.json`, `logs/launch_validation.json`, `logs/driver.log` — training driver records; `logs/eval.log`, `logs/eval_summary.json`, `logs/eval_spec.json`, `logs/eval_pool_manifest.json` — pooled evaluation; `logs/smoke_nojitter_h20.log` — pre-flight smoke.
+- `source/` — the ablation model, heads, config and trainer files used (VLABench workspace repo `Seonghoon-Yu/vlabench` @ `9ac5495d`, `runtime/openpi/`).
+
+Load with `--policy.config pi05_vlabench_race2abl_nojitter_h20 --policy.dir <this dir>` in the matching fork; the horizon comes from the config. Optimizer/resume state is not included.

ablation/abl_nojitter_h20_40k/SHA256SUMS

@@ -0,0 +1,2 @@
+57808c86abc5f31158acba4283b2ec776e35eda7e8a9d07cb7546ffa1fd45bbe  model.safetensors
+38804fce0cf3d94eeefb2f120f0b7b955ef0e1a8711dee303ebbb14bba2d77a5  overlay/ema.safetensors

ablation/abl_nojitter_h20_40k/assets/vlabench/vlabench_ft_primitive/norm_stats.json

@@ -0,0 +1,280 @@
+{
+  "norm_stats": {
+    "state": {
+      "mean": [
+        0.02967369742691517,
+        0.39555251598358154,
+        0.25719010829925537,
+        0.4673101305961609,
+        -0.37718722224235535,
+        -0.11985507607460022,
+        0.44498270750045776,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0
+      ],
+      "std": [
+        0.1581457555294037,
+        0.13765743374824524,
+        0.1370578110218048,
+        2.873211145401001,
+        0.625612199306488,
+        1.7478904724121094,
+        0.49696388840675354,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0
+      ],
+      "q01": [
+        -0.2880732418596744,
+        0.16642616012468936,
+        0.016219121195189656,
+        -3.1415927410125732,
+        -1.5606922270774841,
+        -3.1378225915908815,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0
+      ],
+      "q99": [
+        0.33706892974972724,
+        0.7168794604014606,
+        0.4705753956319765,
+        3.1403361039161677,
+        0.029222321224212644,
+        3.1353093175888063,
+        0.9998,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0
+      ]
+    },
+    "actions": {
+      "mean": [
+        -0.00008998554403660819,
+        0.004090703558176756,
+        0.0006838542176410556,
+        0.5465309023857117,
+        -0.004646406974643469,
+        -0.019831420853734016,
+        0.5484480857849121,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0
+      ],
+      "std": [
+        0.012184085324406624,
+        0.013969186693429947,
+        0.021805619820952415,
+        3.0321147441864014,
+        0.028920045122504234,
+        0.7395531535148621,
+        0.497647225856781,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0
+      ],
+      "q01": [
+        -0.01572922980785374,
+        -0.011965612983703577,
+        -0.012324783468246447,
+        -6.28066982460022,
+        -0.07970612406730648,
+        -1.807010852813721,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0
+      ],
+      "q99": [
+        0.014919160246849028,
+        0.016237698739767104,
+        0.020379505300521905,
+        6.280669824600221,
+        0.06261572251319891,
+        1.7844429779052735,
+        0.9998,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0,
+        0.0
+      ]
+    }
+  }
+}

ablation/abl_nojitter_h20_40k/logs/commands.json

@@ -0,0 +1,57 @@
+{
+  "method": "race2abl_nojitter",
+  "config": "pi05_vlabench_race2abl_nojitter_h20",
+  "exp_name": "abl_nojitter_h20_s42",
+  "horizon": 20,
+  "micro_batch_size": 32,
+  "batch_size": 64,
+  "world_size": 2,
+  "per_gpu_batch_size": 32,
+  "accumulation_steps": 1,
+  "gpus": [
+    0,
+    1
+  ],
+  "train_command": [
+    "/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/torchrun",
+    "--standalone",
+    "--nnodes=1",
+    "--nproc_per_node=2",
+    "scripts/race2_ablation/train_race2abl_nojitter_vlabench.py",
+    "pi05_vlabench_race2abl_nojitter_h20",
+    "--exp_name",
+    "abl_nojitter_h20_s42",
+    "--model.micro-batch-size",
+    "32"
+  ],
+  "eval_command": [
+    "/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python",
+    "scripts/vlabench/eval_pool.py",
+    "--method",
+    "race2abl_nojitter",
+    "--horizon",
+    "20",
+    "--exp-name",
+    "abl_nojitter_h20_s42",
+    "--step",
+    "40000",
+    "--gpus",
+    "0,1",
+    "--jobs-per-gpu",
+    "2",
+    "--groups-per-job",
+    "2",
+    "--n-episode",
+    "50",
+    "--openpi-root",
+    "/home/vision/seonghoon/vlabench/runtime/openpi",
+    "--vlabench-root",
+    "/home/vision/seonghoon/vlabench/VLABench",
+    "--sim-python",
+    "/home/vision/seonghoon/vlabench/runtime/sim-env/bin/python"
+  ],
+  "checkpoint": "/home/vision/seonghoon/vlabench/runtime/openpi/checkpoints/pi05_vlabench_race2abl_nojitter_h20/abl_nojitter_h20_s42/40000/ema.safetensors",
+  "log_dir": "/home/vision/seonghoon/vlabench/runtime/openpi/logs/h20/race2abl_nojitter_abl_nojitter_h20_s42",
+  "openpi_root": "/home/vision/seonghoon/vlabench/runtime/openpi",
+  "setup_root": "/home/vision/seonghoon/vlabench/race_eval/logs/h30_h35_setup"
+}

ablation/abl_nojitter_h20_40k/logs/driver.log

@@ -0,0 +1,5 @@
+{"method": "race2abl_nojitter", "config": "pi05_vlabench_race2abl_nojitter_h20", "exp_name": "abl_nojitter_h20_s42", "horizon": 20, "micro_batch_size": 32, "batch_size": 64, "world_size": 2, "per_gpu_batch_size": 32, "accumulation_steps": 1, "gpus": [0, 1], "train_command": ["/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/torchrun", "--standalone", "--nnodes=1", "--nproc_per_node=2", "scripts/race2_ablation/train_race2abl_nojitter_vlabench.py", "pi05_vlabench_race2abl_nojitter_h20", "--exp_name", "abl_nojitter_h20_s42", "--model.micro-batch-size", "32"], "eval_command": ["/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python", "scripts/vlabench/eval_pool.py", "--method", "race2abl_nojitter", "--horizon", "20", "--exp-name", "abl_nojitter_h20_s42", "--step", "40000", "--gpus", "0,1", "--jobs-per-gpu", "2", "--groups-per-job", "2", "--n-episode", "50", "--openpi-root", "/home/vision/seonghoon/vlabench/runtime/openpi", "--vlabench-root", "/home/vision/seonghoon/vlabench/VLABench", "--sim-python", "/home/vision/seonghoon/vlabench/runtime/sim-env/bin/python"], "checkpoint": "/home/vision/seonghoon/vlabench/runtime/openpi/checkpoints/pi05_vlabench_race2abl_nojitter_h20/abl_nojitter_h20_s42/40000/ema.safetensors", "log_dir": "/home/vision/seonghoon/vlabench/runtime/openpi/logs/h20/race2abl_nojitter_abl_nojitter_h20_s42", "openpi_root": "/home/vision/seonghoon/vlabench/runtime/openpi", "setup_root": "/home/vision/seonghoon/vlabench/race_eval/logs/h30_h35_setup", "stage": "launch_validated", "time": "2026-09-16T07:58:58.077119+00:00", "pid": 1446134, "launch_validation": "/home/vision/seonghoon/vlabench/runtime/openpi/logs/h20/race2abl_nojitter_abl_nojitter_h20_s42/launch_validation.json"}
+{"method": "race2abl_nojitter", "config": "pi05_vlabench_race2abl_nojitter_h20", "exp_name": "abl_nojitter_h20_s42", "horizon": 20, "micro_batch_size": 32, "batch_size": 64, "world_size": 2, "per_gpu_batch_size": 32, "accumulation_steps": 1, "gpus": [0, 1], "train_command": ["/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/torchrun", "--standalone", "--nnodes=1", "--nproc_per_node=2", "scripts/race2_ablation/train_race2abl_nojitter_vlabench.py", "pi05_vlabench_race2abl_nojitter_h20", "--exp_name", "abl_nojitter_h20_s42", "--model.micro-batch-size", "32"], "eval_command": ["/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python", "scripts/vlabench/eval_pool.py", "--method", "race2abl_nojitter", "--horizon", "20", "--exp-name", "abl_nojitter_h20_s42", "--step", "40000", "--gpus", "0,1", "--jobs-per-gpu", "2", "--groups-per-job", "2", "--n-episode", "50", "--openpi-root", "/home/vision/seonghoon/vlabench/runtime/openpi", "--vlabench-root", "/home/vision/seonghoon/vlabench/VLABench", "--sim-python", "/home/vision/seonghoon/vlabench/runtime/sim-env/bin/python"], "checkpoint": "/home/vision/seonghoon/vlabench/runtime/openpi/checkpoints/pi05_vlabench_race2abl_nojitter_h20/abl_nojitter_h20_s42/40000/ema.safetensors", "log_dir": "/home/vision/seonghoon/vlabench/runtime/openpi/logs/h20/race2abl_nojitter_abl_nojitter_h20_s42", "openpi_root": "/home/vision/seonghoon/vlabench/runtime/openpi", "setup_root": "/home/vision/seonghoon/vlabench/race_eval/logs/h30_h35_setup", "stage": "training", "time": "2026-09-16T07:58:58.080047+00:00", "pid": 1446134, "child_pid": 1446169, "log": "/home/vision/seonghoon/vlabench/runtime/openpi/logs/h20/race2abl_nojitter_abl_nojitter_h20_s42/train.log"}
+{"method": "race2abl_nojitter", "config": "pi05_vlabench_race2abl_nojitter_h20", "exp_name": "abl_nojitter_h20_s42", "horizon": 20, "micro_batch_size": 32, "batch_size": 64, "world_size": 2, "per_gpu_batch_size": 32, "accumulation_steps": 1, "gpus": [0, 1], "train_command": ["/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/torchrun", "--standalone", "--nnodes=1", "--nproc_per_node=2", "scripts/race2_ablation/train_race2abl_nojitter_vlabench.py", "pi05_vlabench_race2abl_nojitter_h20", "--exp_name", "abl_nojitter_h20_s42", "--model.micro-batch-size", "32"], "eval_command": ["/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python", "scripts/vlabench/eval_pool.py", "--method", "race2abl_nojitter", "--horizon", "20", "--exp-name", "abl_nojitter_h20_s42", "--step", "40000", "--gpus", "0,1", "--jobs-per-gpu", "2", "--groups-per-job", "2", "--n-episode", "50", "--openpi-root", "/home/vision/seonghoon/vlabench/runtime/openpi", "--vlabench-root", "/home/vision/seonghoon/vlabench/VLABench", "--sim-python", "/home/vision/seonghoon/vlabench/runtime/sim-env/bin/python"], "checkpoint": "/home/vision/seonghoon/vlabench/runtime/openpi/checkpoints/pi05_vlabench_race2abl_nojitter_h20/abl_nojitter_h20_s42/40000/ema.safetensors", "log_dir": "/home/vision/seonghoon/vlabench/runtime/openpi/logs/h20/race2abl_nojitter_abl_nojitter_h20_s42", "openpi_root": "/home/vision/seonghoon/vlabench/runtime/openpi", "setup_root": "/home/vision/seonghoon/vlabench/race_eval/logs/h30_h35_setup", "stage": "trained", "time": "2026-09-16T23:23:12.762010+00:00", "pid": 1446134, "checkpoint_validation": {"step": 40000, "config": "pi05_vlabench_race2abl_nojitter_h20", "ema": "/home/vision/seonghoon/vlabench/runtime/openpi/checkpoints/pi05_vlabench_race2abl_nojitter_h20/abl_nojitter_h20_s42/40000/ema.safetensors", "n_tensors": 264, "horizon": 20, "micro_batch_size": 32}}
+{"method": "race2abl_nojitter", "config": "pi05_vlabench_race2abl_nojitter_h20", "exp_name": "abl_nojitter_h20_s42", "horizon": 20, "micro_batch_size": 32, "batch_size": 64, "world_size": 2, "per_gpu_batch_size": 32, "accumulation_steps": 1, "gpus": [0, 1], "train_command": ["/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/torchrun", "--standalone", "--nnodes=1", "--nproc_per_node=2", "scripts/race2_ablation/train_race2abl_nojitter_vlabench.py", "pi05_vlabench_race2abl_nojitter_h20", "--exp_name", "abl_nojitter_h20_s42", "--model.micro-batch-size", "32"], "eval_command": ["/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python", "scripts/vlabench/eval_pool.py", "--method", "race2abl_nojitter", "--horizon", "20", "--exp-name", "abl_nojitter_h20_s42", "--step", "40000", "--gpus", "0,1", "--jobs-per-gpu", "2", "--groups-per-job", "2", "--n-episode", "50", "--openpi-root", "/home/vision/seonghoon/vlabench/runtime/openpi", "--vlabench-root", "/home/vision/seonghoon/vlabench/VLABench", "--sim-python", "/home/vision/seonghoon/vlabench/runtime/sim-env/bin/python"], "checkpoint": "/home/vision/seonghoon/vlabench/runtime/openpi/checkpoints/pi05_vlabench_race2abl_nojitter_h20/abl_nojitter_h20_s42/40000/ema.safetensors", "log_dir": "/home/vision/seonghoon/vlabench/runtime/openpi/logs/h20/race2abl_nojitter_abl_nojitter_h20_s42", "openpi_root": "/home/vision/seonghoon/vlabench/runtime/openpi", "setup_root": "/home/vision/seonghoon/vlabench/race_eval/logs/h30_h35_setup", "stage": "evaluating", "time": "2026-09-16T23:23:12.764496+00:00", "pid": 1446134, "child_pid": 1871797, "log": "/home/vision/seonghoon/vlabench/runtime/openpi/logs/h20/race2abl_nojitter_abl_nojitter_h20_s42/eval.log"}
+{"method": "race2abl_nojitter", "config": "pi05_vlabench_race2abl_nojitter_h20", "exp_name": "abl_nojitter_h20_s42", "horizon": 20, "micro_batch_size": 32, "batch_size": 64, "world_size": 2, "per_gpu_batch_size": 32, "accumulation_steps": 1, "gpus": [0, 1], "train_command": ["/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/torchrun", "--standalone", "--nnodes=1", "--nproc_per_node=2", "scripts/race2_ablation/train_race2abl_nojitter_vlabench.py", "pi05_vlabench_race2abl_nojitter_h20", "--exp_name", "abl_nojitter_h20_s42", "--model.micro-batch-size", "32"], "eval_command": ["/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python", "scripts/vlabench/eval_pool.py", "--method", "race2abl_nojitter", "--horizon", "20", "--exp-name", "abl_nojitter_h20_s42", "--step", "40000", "--gpus", "0,1", "--jobs-per-gpu", "2", "--groups-per-job", "2", "--n-episode", "50", "--openpi-root", "/home/vision/seonghoon/vlabench/runtime/openpi", "--vlabench-root", "/home/vision/seonghoon/vlabench/VLABench", "--sim-python", "/home/vision/seonghoon/vlabench/runtime/sim-env/bin/python"], "checkpoint": "/home/vision/seonghoon/vlabench/runtime/openpi/checkpoints/pi05_vlabench_race2abl_nojitter_h20/abl_nojitter_h20_s42/40000/ema.safetensors", "log_dir": "/home/vision/seonghoon/vlabench/runtime/openpi/logs/h20/race2abl_nojitter_abl_nojitter_h20_s42", "openpi_root": "/home/vision/seonghoon/vlabench/runtime/openpi", "setup_root": "/home/vision/seonghoon/vlabench/race_eval/logs/h30_h35_setup", "stage": "complete", "time": "2026-09-17T06:22:33.052570+00:00", "pid": 1446134}

ablation/abl_nojitter_h20_40k/logs/eval.log

@@ -0,0 +1 @@
+[2026-09-17 15:22:33] POOL_EVALUATION_COMPLETE: /home/vision/seonghoon/vlabench/runtime/openpi/evaluate_results/vlabench_h20/race2abl_nojitter_h20_abl_nojitter_h20_s42_40000_ema_replan20_n50_seed7/summary.json

ablation/abl_nojitter_h20_40k/logs/eval_pool_manifest.json

@@ -0,0 +1,280 @@
+{
+  "attempt": "20260917_082314_73772fb6d0",
+  "jobs": {
+    "1": {
+      "gpu": 0,
+      "pid": 1872125,
+      "command": [
+        "/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python",
+        "/home/vision/seonghoon/vlabench/runtime/openpi/scripts/vlabench/eval_h25.py",
+        "--pool-worker",
+        "--method",
+        "race2abl_nojitter",
+        "--horizon",
+        "20",
+        "--exp-name",
+        "abl_nojitter_h20_s42",
+        "--step",
+        "40000",
+        "--gpu",
+        "0",
+        "--n-episode",
+        "50",
+        "--groups",
+        "2",
+        "--tracks",
+        "1",
+        "--replan-steps",
+        "20",
+        "--openpi-root",
+        "/home/vision/seonghoon/vlabench/runtime/openpi",
+        "--vlabench-root",
+        "/home/vision/seonghoon/vlabench/VLABench",
+        "--checkpoint-root",
+        "/home/vision/seonghoon/vlabench/runtime/openpi/checkpoints",
+        "--output-root",
+        "/home/vision/seonghoon/vlabench/runtime/openpi/evaluate_results/vlabench_h20/race2abl_nojitter_h20_abl_nojitter_h20_s42_40000_ema_replan20_n50_seed7/shards/track_1",
+        "--policy-python",
+        "/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python",
+        "--sim-python",
+        "/home/vision/seonghoon/vlabench/runtime/sim-env/bin/python",
+        "--startup-timeout",
+        "1200.0",
+        "--warmup-timeout",
+        "3600.0",
+        "--task-timeout",
+        "21600.0"
+      ],
+      "status": "complete",
+      "started_at": 1789600994.7304442,
+      "env_limits": {
+        "TORCHINDUCTOR_COMPILE_THREADS": "2",
+        "OMP_NUM_THREADS": "1",
+        "OPENBLAS_NUM_THREADS": "1",
+        "MKL_NUM_THREADS": "1"
+      },
+      "finished_at": 1789617126.4738705
+    },
+    "2": {
+      "gpu": 1,
+      "pid": 1872126,
+      "command": [
+        "/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python",
+        "/home/vision/seonghoon/vlabench/runtime/openpi/scripts/vlabench/eval_h25.py",
+        "--pool-worker",
+        "--method",
+        "race2abl_nojitter",
+        "--horizon",
+        "20",
+        "--exp-name",
+        "abl_nojitter_h20_s42",
+        "--step",
+        "40000",
+        "--gpu",
+        "1",
+        "--n-episode",
+        "50",
+        "--groups",
+        "2",
+        "--tracks",
+        "2",
+        "--replan-steps",
+        "20",
+        "--openpi-root",
+        "/home/vision/seonghoon/vlabench/runtime/openpi",
+        "--vlabench-root",
+        "/home/vision/seonghoon/vlabench/VLABench",
+        "--checkpoint-root",
+        "/home/vision/seonghoon/vlabench/runtime/openpi/checkpoints",
+        "--output-root",
+        "/home/vision/seonghoon/vlabench/runtime/openpi/evaluate_results/vlabench_h20/race2abl_nojitter_h20_abl_nojitter_h20_s42_40000_ema_replan20_n50_seed7/shards/track_2",
+        "--policy-python",
+        "/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python",
+        "--sim-python",
+        "/home/vision/seonghoon/vlabench/runtime/sim-env/bin/python",
+        "--startup-timeout",
+        "1200.0",
+        "--warmup-timeout",
+        "3600.0",
+        "--task-timeout",
+        "21600.0"
+      ],
+      "status": "complete",
+      "started_at": 1789600994.7326775,
+      "env_limits": {
+        "TORCHINDUCTOR_COMPILE_THREADS": "2",
+        "OMP_NUM_THREADS": "1",
+        "OPENBLAS_NUM_THREADS": "1",
+        "MKL_NUM_THREADS": "1"
+      },
+      "finished_at": 1789617740.1950321
+    },
+    "3": {
+      "gpu": 0,
+      "pid": 1872127,
+      "command": [
+        "/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python",
+        "/home/vision/seonghoon/vlabench/runtime/openpi/scripts/vlabench/eval_h25.py",
+        "--pool-worker",
+        "--method",
+        "race2abl_nojitter",
+        "--horizon",
+        "20",
+        "--exp-name",
+        "abl_nojitter_h20_s42",
+        "--step",
+        "40000",
+        "--gpu",
+        "0",
+        "--n-episode",
+        "50",
+        "--groups",
+        "2",
+        "--tracks",
+        "3",
+        "--replan-steps",
+        "20",
+        "--openpi-root",
+        "/home/vision/seonghoon/vlabench/runtime/openpi",
+        "--vlabench-root",
+        "/home/vision/seonghoon/vlabench/VLABench",
+        "--checkpoint-root",
+        "/home/vision/seonghoon/vlabench/runtime/openpi/checkpoints",
+        "--output-root",
+        "/home/vision/seonghoon/vlabench/runtime/openpi/evaluate_results/vlabench_h20/race2abl_nojitter_h20_abl_nojitter_h20_s42_40000_ema_replan20_n50_seed7/shards/track_3",
+        "--policy-python",
+        "/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python",
+        "--sim-python",
+        "/home/vision/seonghoon/vlabench/runtime/sim-env/bin/python",
+        "--startup-timeout",
+        "1200.0",
+        "--warmup-timeout",
+        "3600.0",
+        "--task-timeout",
+        "21600.0"
+      ],
+      "status": "complete",
+      "started_at": 1789600994.7343996,
+      "env_limits": {
+        "TORCHINDUCTOR_COMPILE_THREADS": "2",
+        "OMP_NUM_THREADS": "1",
+        "OPENBLAS_NUM_THREADS": "1",
+        "MKL_NUM_THREADS": "1"
+      },
+      "finished_at": 1789619455.8005862
+    },
+    "4": {
+      "gpu": 1,
+      "pid": 1872128,
+      "command": [
+        "/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python",
+        "/home/vision/seonghoon/vlabench/runtime/openpi/scripts/vlabench/eval_h25.py",
+        "--pool-worker",
+        "--method",
+        "race2abl_nojitter",
+        "--horizon",
+        "20",
+        "--exp-name",
+        "abl_nojitter_h20_s42",
+        "--step",
+        "40000",
+        "--gpu",
+        "1",
+        "--n-episode",
+        "50",
+        "--groups",
+        "2",
+        "--tracks",
+        "4",
+        "--replan-steps",
+        "20",
+        "--openpi-root",
+        "/home/vision/seonghoon/vlabench/runtime/openpi",
+        "--vlabench-root",
+        "/home/vision/seonghoon/vlabench/VLABench",
+        "--checkpoint-root",
+        "/home/vision/seonghoon/vlabench/runtime/openpi/checkpoints",
+        "--output-root",
+        "/home/vision/seonghoon/vlabench/runtime/openpi/evaluate_results/vlabench_h20/race2abl_nojitter_h20_abl_nojitter_h20_s42_40000_ema_replan20_n50_seed7/shards/track_4",
+        "--policy-python",
+        "/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python",
+        "--sim-python",
+        "/home/vision/seonghoon/vlabench/runtime/sim-env/bin/python",
+        "--startup-timeout",
+        "1200.0",
+        "--warmup-timeout",
+        "3600.0",
+        "--task-timeout",
+        "21600.0"
+      ],
+      "status": "complete",
+      "started_at": 1789600994.7361388,
+      "env_limits": {
+        "TORCHINDUCTOR_COMPILE_THREADS": "2",
+        "OMP_NUM_THREADS": "1",
+        "OPENBLAS_NUM_THREADS": "1",
+        "MKL_NUM_THREADS": "1"
+      },
+      "finished_at": 1789618535.9508383
+    },
+    "6": {
+      "gpu": 0,
+      "pid": 2654518,
+      "command": [
+        "/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python",
+        "/home/vision/seonghoon/vlabench/runtime/openpi/scripts/vlabench/eval_h25.py",
+        "--pool-worker",
+        "--method",
+        "race2abl_nojitter",
+        "--horizon",
+        "20",
+        "--exp-name",
+        "abl_nojitter_h20_s42",
+        "--step",
+        "40000",
+        "--gpu",
+        "0",
+        "--n-episode",
+        "50",
+        "--groups",
+        "2",
+        "--tracks",
+        "6",
+        "--replan-steps",
+        "20",
+        "--openpi-root",
+        "/home/vision/seonghoon/vlabench/runtime/openpi",
+        "--vlabench-root",
+        "/home/vision/seonghoon/vlabench/VLABench",
+        "--checkpoint-root",
+        "/home/vision/seonghoon/vlabench/runtime/openpi/checkpoints",
+        "--output-root",
+        "/home/vision/seonghoon/vlabench/runtime/openpi/evaluate_results/vlabench_h20/race2abl_nojitter_h20_abl_nojitter_h20_s42_40000_ema_replan20_n50_seed7/shards/track_6",
+        "--policy-python",
+        "/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python",
+        "--sim-python",
+        "/home/vision/seonghoon/vlabench/runtime/sim-env/bin/python",
+        "--startup-timeout",
+        "1200.0",
+        "--warmup-timeout",
+        "3600.0",
+        "--task-timeout",
+        "21600.0"
+      ],
+      "status": "complete",
+      "started_at": 1789617126.4909804,
+      "env_limits": {
+        "TORCHINDUCTOR_COMPILE_THREADS": "2",
+        "OMP_NUM_THREADS": "1",
+        "OPENBLAS_NUM_THREADS": "1",
+        "MKL_NUM_THREADS": "1"
+      },
+      "finished_at": 1789626151.255465
+    }
+  },
+  "prior_manifests": [],
+  "max_active_per_gpu": {
+    "0": 2,
+    "1": 2
+  }
+}

ablation/abl_nojitter_h20_40k/logs/eval_spec.json

@@ -0,0 +1,102 @@
+{
+  "config": "pi05_vlabench_race2abl_nojitter_h20",
+  "exp_name": "abl_nojitter_h20_s42",
+  "step": 40000,
+  "checkpoint": "/home/vision/seonghoon/vlabench/runtime/openpi/checkpoints/pi05_vlabench_race2abl_nojitter_h20/abl_nojitter_h20_s42/40000",
+  "ema_sha256": "38804fce0cf3d94eeefb2f120f0b7b955ef0e1a8711dee303ebbb14bba2d77a5",
+  "horizon": 20,
+  "replan_steps": 20,
+  "seed": 7,
+  "policy_seed": 7,
+  "policy_noise_rng": "independent_track_servers",
+  "requested_episodes": 50,
+  "tracks": {
+    "track_1_in_distribution": {
+      "select_painting": 50,
+      "select_book": 50,
+      "select_drink": 50,
+      "select_chemistry_tube": 50,
+      "select_poker": 50,
+      "select_mahjong": 50,
+      "select_toy": 50,
+      "select_fruit": 50,
+      "add_condiment": 50,
+      "insert_flower": 50
+    },
+    "track_2_cross_category": {
+      "select_painting": 50,
+      "select_fruit": 50,
+      "select_toy": 50,
+      "select_poker": 50,
+      "select_mahjong": 50,
+      "insert_flower": 10,
+      "select_chemistry_tube": 50,
+      "add_condiment": 50,
+      "select_drink": 50,
+      "select_book": 50
+    },
+    "track_3_common_sense": {
+      "select_painting": 50,
+      "select_book": 50,
+      "select_drink": 50,
+      "select_chemistry_tube": 50,
+      "select_toy": 50,
+      "select_fruit": 50,
+      "add_condiment": 50,
+      "insert_flower": 50,
+      "select_nth_largest_poker": 50,
+      "select_unique_type_mahjong": 50
+    },
+    "track_4_semantic_instruction": {
+      "select_painting": 50,
+      "select_book": 50,
+      "select_drink": 50,
+      "select_chemistry_tube": 50,
+      "select_poker": 50,
+      "select_mahjong": 50,
+      "select_toy": 50,
+      "select_fruit": 50,
+      "add_condiment": 50,
+      "insert_flower": 50
+    },
+    "track_6_unseen_texture": {
+      "select_painting": 50,
+      "select_book": 50,
+      "select_drink": 50,
+      "select_chemistry_tube": 50,
+      "select_poker": 50,
+      "select_mahjong": 50,
+      "select_toy": 50,
+      "select_fruit": 50,
+      "add_condiment": 50,
+      "insert_flower": 50
+    }
+  },
+  "track_sha256": {
+    "track_1_in_distribution": "61b07440c6f4272705bea0c8d721bc196e744232f48408075613cc9b6f7c8c04",
+    "track_2_cross_category": "cf004383de22d994f83126ca30a315cbfc6a92fd0dbb8d25fa4c185dd75a21fb",
+    "track_3_common_sense": "2aaa298580e74bec13555a6950b95241edf6c5264b498eeaf0e919837952ec6d",
+    "track_4_semantic_instruction": "9aff9dfc778ae35f299dce4c3aa07afbdc5275847490abd8037f3f5bdd9de411",
+    "track_6_unseen_texture": "249c1bca941b13267571b7641a91bcfbaffbc9c2b6b210d9ee2d2513324aa817"
+  },
+  "vlabench_root": "/home/vision/seonghoon/vlabench/VLABench",
+  "execution_mode": "pool",
+  "pool_protocol": {
+    "name": "track_pool_v1",
+    "policy_noise_rng": "independent_track_servers",
+    "policy_seed": 7,
+    "jobs_per_gpu": 2,
+    "groups_per_job": 2,
+    "note": "Each track has an independent seed-7 policy server with a shared noise RNG among its task clients. Client ordering and partial resumes can change draws; this is not bit-identical to a single-server or four-shard evaluation."
+  },
+  "gpus": [
+    0,
+    1
+  ],
+  "env_limits": {
+    "TORCHINDUCTOR_COMPILE_THREADS": "2",
+    "OMP_NUM_THREADS": "1",
+    "OPENBLAS_NUM_THREADS": "1",
+    "MKL_NUM_THREADS": "1"
+  }
+}

ablation/abl_nojitter_h20_40k/logs/eval_summary.json

@@ -0,0 +1,1249 @@
+{
+  "status": "complete",
+  "spec": {
+    "config": "pi05_vlabench_race2abl_nojitter_h20",
+    "exp_name": "abl_nojitter_h20_s42",
+    "step": 40000,
+    "checkpoint": "/home/vision/seonghoon/vlabench/runtime/openpi/checkpoints/pi05_vlabench_race2abl_nojitter_h20/abl_nojitter_h20_s42/40000",
+    "ema_sha256": "38804fce0cf3d94eeefb2f120f0b7b955ef0e1a8711dee303ebbb14bba2d77a5",
+    "horizon": 20,
+    "replan_steps": 20,
+    "seed": 7,
+    "policy_seed": 7,
+    "policy_noise_rng": "independent_track_servers",
+    "requested_episodes": 50,
+    "tracks": {
+      "track_1_in_distribution": {
+        "select_painting": 50,
+        "select_book": 50,
+        "select_drink": 50,
+        "select_chemistry_tube": 50,
+        "select_poker": 50,
+        "select_mahjong": 50,
+        "select_toy": 50,
+        "select_fruit": 50,
+        "add_condiment": 50,
+        "insert_flower": 50
+      },
+      "track_2_cross_category": {
+        "select_painting": 50,
+        "select_fruit": 50,
+        "select_toy": 50,
+        "select_poker": 50,
+        "select_mahjong": 50,
+        "insert_flower": 10,
+        "select_chemistry_tube": 50,
+        "add_condiment": 50,
+        "select_drink": 50,
+        "select_book": 50
+      },
+      "track_3_common_sense": {
+        "select_painting": 50,
+        "select_book": 50,
+        "select_drink": 50,
+        "select_chemistry_tube": 50,
+        "select_toy": 50,
+        "select_fruit": 50,
+        "add_condiment": 50,
+        "insert_flower": 50,
+        "select_nth_largest_poker": 50,
+        "select_unique_type_mahjong": 50
+      },
+      "track_4_semantic_instruction": {
+        "select_painting": 50,
+        "select_book": 50,
+        "select_drink": 50,
+        "select_chemistry_tube": 50,
+        "select_poker": 50,
+        "select_mahjong": 50,
+        "select_toy": 50,
+        "select_fruit": 50,
+        "add_condiment": 50,
+        "insert_flower": 50
+      },
+      "track_6_unseen_texture": {
+        "select_painting": 50,
+        "select_book": 50,
+        "select_drink": 50,
+        "select_chemistry_tube": 50,
+        "select_poker": 50,
+        "select_mahjong": 50,
+        "select_toy": 50,
+        "select_fruit": 50,
+        "add_condiment": 50,
+        "insert_flower": 50
+      }
+    },
+    "track_sha256": {
+      "track_1_in_distribution": "61b07440c6f4272705bea0c8d721bc196e744232f48408075613cc9b6f7c8c04",
+      "track_2_cross_category": "cf004383de22d994f83126ca30a315cbfc6a92fd0dbb8d25fa4c185dd75a21fb",
+      "track_3_common_sense": "2aaa298580e74bec13555a6950b95241edf6c5264b498eeaf0e919837952ec6d",
+      "track_4_semantic_instruction": "9aff9dfc778ae35f299dce4c3aa07afbdc5275847490abd8037f3f5bdd9de411",
+      "track_6_unseen_texture": "249c1bca941b13267571b7641a91bcfbaffbc9c2b6b210d9ee2d2513324aa817"
+    },
+    "vlabench_root": "/home/vision/seonghoon/vlabench/VLABench",
+    "execution_mode": "pool",
+    "pool_protocol": {
+      "name": "track_pool_v1",
+      "policy_noise_rng": "independent_track_servers",
+      "policy_seed": 7,
+      "jobs_per_gpu": 2,
+      "groups_per_job": 2,
+      "note": "Each track has an independent seed-7 policy server with a shared noise RNG among its task clients. Client ordering and partial resumes can change draws; this is not bit-identical to a single-server or four-shard evaluation."
+    },
+    "gpus": [
+      0,
+      1
+    ],
+    "env_limits": {
+      "TORCHINDUCTOR_COMPILE_THREADS": "2",
+      "OMP_NUM_THREADS": "1",
+      "OPENBLAS_NUM_THREADS": "1",
+      "MKL_NUM_THREADS": "1"
+    }
+  },
+  "tracks": {
+    "track_1_in_distribution": {
+      "status": "complete",
+      "track": "track_1_in_distribution",
+      "n_tasks": 10,
+      "episodes_per_task": {
+        "select_painting": 50,
+        "select_book": 50,
+        "select_drink": 50,
+        "select_chemistry_tube": 50,
+        "select_poker": 50,
+        "select_mahjong": 50,
+        "select_toy": 50,
+        "select_fruit": 50,
+        "add_condiment": 50,
+        "insert_flower": 50
+      },
+      "total_episodes": 500,
+      "per_task": {
+        "select_painting": {
+          "success_rate": 0.14,
+          "intention_score": 0.46,
+          "progress_score": 0.14
+        },
+        "select_book": {
+          "success_rate": 0.64,
+          "intention_score": 0.98,
+          "progress_score": 0.75
+        },
+        "select_drink": {
+          "success_rate": 0.42,
+          "intention_score": 1.0,
+          "progress_score": 0.6
+        },
+        "select_chemistry_tube": {
+          "success_rate": 0.7142857142857143,
+          "intention_score": 0.02040816326530612,
+          "progress_score": 0.8367346938775511
+        },
+        "select_poker": {
+          "success_rate": 0.54,
+          "intention_score": 0.84,
+          "progress_score": 0.74
+        },
+        "select_mahjong": {
+          "success_rate": 0.4782608695652174,
+          "intention_score": 0.8913043478260869,
+          "progress_score": 0.5434782608695652
+        },
+        "select_toy": {
+          "success_rate": 0.62,
+          "intention_score": 0.22,
+          "progress_score": 0.81
+        },
+        "select_fruit": {
+          "success_rate": 0.78,
+          "intention_score": 1.0,
+          "progress_score": 0.89
+        },
+        "add_condiment": {
+          "success_rate": 0.66,
+          "intention_score": 1.0,
+          "progress_score": 0.8066666666666668
+        },
+        "insert_flower": {
+          "success_rate": 0.26,
+          "intention_score": 1.0,
+          "progress_score": 0.62
+        }
+      },
+      "avg_success_rate": 0.5252546583850932,
+      "avg_intention_score": 0.7411712511091392,
+      "avg_progress_score": 0.6736879621413783,
+      "error_policy": "exclude_physics_errors_v1",
+      "per_task_episode_accounting": {
+        "select_painting": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_book": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_drink": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_chemistry_tube": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 49,
+          "excluded_episodes": 1,
+          "metric_denominator": 49,
+          "excluded_episode_ids": [
+            26
+          ]
+        },
+        "select_poker": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_mahjong": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 46,
+          "excluded_episodes": 4,
+          "metric_denominator": 46,
+          "excluded_episode_ids": [
+            20,
+            46,
+            47,
+            49
+          ]
+        },
+        "select_toy": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_fruit": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "add_condiment": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "insert_flower": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        }
+      },
+      "valid_episodes": 495,
+      "excluded_episodes": 5
+    },
+    "track_2_cross_category": {
+      "status": "complete",
+      "track": "track_2_cross_category",
+      "n_tasks": 10,
+      "episodes_per_task": {
+        "select_painting": 50,
+        "select_fruit": 50,
+        "select_toy": 50,
+        "select_poker": 50,
+        "select_mahjong": 50,
+        "insert_flower": 10,
+        "select_chemistry_tube": 50,
+        "add_condiment": 50,
+        "select_drink": 50,
+        "select_book": 50
+      },
+      "total_episodes": 460,
+      "per_task": {
+        "select_painting": {
+          "success_rate": 0.24,
+          "intention_score": 0.5,
+          "progress_score": 0.24
+        },
+        "select_fruit": {
+          "success_rate": 0.66,
+          "intention_score": 0.94,
+          "progress_score": 0.8
+        },
+        "select_toy": {
+          "success_rate": 0.52,
+          "intention_score": 0.1,
+          "progress_score": 0.71
+        },
+        "select_poker": {
+          "success_rate": 0.4,
+          "intention_score": 0.8,
+          "progress_score": 0.6400000000000001
+        },
+        "select_mahjong": {
+          "success_rate": 0.2558139534883721,
+          "intention_score": 0.6744186046511628,
+          "progress_score": 0.3372093023255814
+        },
+        "insert_flower": {
+          "success_rate": 0.2,
+          "intention_score": 1.0,
+          "progress_score": 0.45
+        },
+        "select_chemistry_tube": {
+          "success_rate": 0.14583333333333334,
+          "intention_score": 0.0,
+          "progress_score": 0.23958333333333334
+        },
+        "add_condiment": {
+          "success_rate": 0.06,
+          "intention_score": 0.28,
+          "progress_score": 0.16
+        },
+        "select_drink": {
+          "success_rate": 0.1,
+          "intention_score": 0.6,
+          "progress_score": 0.27
+        },
+        "select_book": {
+          "success_rate": 0.04,
+          "intention_score": 0.14,
+          "progress_score": 0.09
+        }
+      },
+      "avg_success_rate": 0.26216472868217056,
+      "avg_intention_score": 0.5034418604651162,
+      "avg_progress_score": 0.39367926356589156,
+      "error_policy": "exclude_physics_errors_v1",
+      "per_task_episode_accounting": {
+        "select_painting": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_fruit": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_toy": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_poker": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_mahjong": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 43,
+          "excluded_episodes": 7,
+          "metric_denominator": 43,
+          "excluded_episode_ids": [
+            0,
+            11,
+            25,
+            40,
+            41,
+            48,
+            49
+          ]
+        },
+        "insert_flower": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 10,
+          "valid_episodes": 10,
+          "excluded_episodes": 0,
+          "metric_denominator": 10,
+          "excluded_episode_ids": []
+        },
+        "select_chemistry_tube": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 48,
+          "excluded_episodes": 2,
+          "metric_denominator": 48,
+          "excluded_episode_ids": [
+            3,
+            30
+          ]
+        },
+        "add_condiment": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_drink": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_book": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        }
+      },
+      "valid_episodes": 451,
+      "excluded_episodes": 9
+    },
+    "track_3_common_sense": {
+      "status": "complete",
+      "track": "track_3_common_sense",
+      "n_tasks": 10,
+      "episodes_per_task": {
+        "select_painting": 50,
+        "select_book": 50,
+        "select_drink": 50,
+        "select_chemistry_tube": 50,
+        "select_toy": 50,
+        "select_fruit": 50,
+        "add_condiment": 50,
+        "insert_flower": 50,
+        "select_nth_largest_poker": 50,
+        "select_unique_type_mahjong": 50
+      },
+      "total_episodes": 500,
+      "per_task": {
+        "select_painting": {
+          "success_rate": 0.12,
+          "intention_score": 0.44,
+          "progress_score": 0.12
+        },
+        "select_book": {
+          "success_rate": 0.42,
+          "intention_score": 0.68,
+          "progress_score": 0.55
+        },
+        "select_drink": {
+          "success_rate": 0.10204081632653061,
+          "intention_score": 0.7959183673469388,
+          "progress_score": 0.30612244897959184
+        },
+        "select_chemistry_tube": {
+          "success_rate": 0.6,
+          "intention_score": 0.02,
+          "progress_score": 0.77
+        },
+        "select_toy": {
+          "success_rate": 0.24,
+          "intention_score": 0.4,
+          "progress_score": 0.57
+        },
+        "select_fruit": {
+          "success_rate": 0.58,
+          "intention_score": 0.96,
+          "progress_score": 0.75
+        },
+        "add_condiment": {
+          "success_rate": 0.16,
+          "intention_score": 0.68,
+          "progress_score": 0.39333333333333337
+        },
+        "insert_flower": {
+          "success_rate": 0.18,
+          "intention_score": 1.0,
+          "progress_score": 0.58
+        },
+        "select_nth_largest_poker": {
+          "success_rate": 0.14,
+          "intention_score": 0.24,
+          "progress_score": 0.16
+        },
+        "select_unique_type_mahjong": {
+          "success_rate": 0.13953488372093023,
+          "intention_score": 0.5581395348837209,
+          "progress_score": 0.1744186046511628
+        }
+      },
+      "avg_success_rate": 0.2681575700047461,
+      "avg_intention_score": 0.577405790223066,
+      "avg_progress_score": 0.43738743869640884,
+      "error_policy": "exclude_physics_errors_v1",
+      "per_task_episode_accounting": {
+        "select_painting": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_book": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_drink": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 49,
+          "excluded_episodes": 1,
+          "metric_denominator": 49,
+          "excluded_episode_ids": [
+            27
+          ]
+        },
+        "select_chemistry_tube": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_toy": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_fruit": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "add_condiment": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "insert_flower": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_nth_largest_poker": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_unique_type_mahjong": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 43,
+          "excluded_episodes": 7,
+          "metric_denominator": 43,
+          "excluded_episode_ids": [
+            0,
+            14,
+            25,
+            31,
+            32,
+            37,
+            49
+          ]
+        }
+      },
+      "valid_episodes": 492,
+      "excluded_episodes": 8
+    },
+    "track_4_semantic_instruction": {
+      "status": "complete",
+      "track": "track_4_semantic_instruction",
+      "n_tasks": 10,
+      "episodes_per_task": {
+        "select_painting": 50,
+        "select_book": 50,
+        "select_drink": 50,
+        "select_chemistry_tube": 50,
+        "select_poker": 50,
+        "select_mahjong": 50,
+        "select_toy": 50,
+        "select_fruit": 50,
+        "add_condiment": 50,
+        "insert_flower": 50
+      },
+      "total_episodes": 500,
+      "per_task": {
+        "select_painting": {
+          "success_rate": 0.1,
+          "intention_score": 0.42,
+          "progress_score": 0.1
+        },
+        "select_book": {
+          "success_rate": 0.24444444444444444,
+          "intention_score": 0.7111111111111111,
+          "progress_score": 0.4
+        },
+        "select_drink": {
+          "success_rate": 0.08,
+          "intention_score": 0.74,
+          "progress_score": 0.28
+        },
+        "select_chemistry_tube": {
+          "success_rate": 0.3673469387755102,
+          "intention_score": 0.04081632653061224,
+          "progress_score": 0.6224489795918368
+        },
+        "select_poker": {
+          "success_rate": 0.32,
+          "intention_score": 0.74,
+          "progress_score": 0.37333333333333335
+        },
+        "select_mahjong": {
+          "success_rate": 0.18,
+          "intention_score": 0.76,
+          "progress_score": 0.3
+        },
+        "select_toy": {
+          "success_rate": 0.16,
+          "intention_score": 0.36,
+          "progress_score": 0.5
+        },
+        "select_fruit": {
+          "success_rate": 0.46,
+          "intention_score": 0.88,
+          "progress_score": 0.65
+        },
+        "add_condiment": {
+          "success_rate": 0.16,
+          "intention_score": 0.64,
+          "progress_score": 0.36666666666666664
+        },
+        "insert_flower": {
+          "success_rate": 0.2,
+          "intention_score": 1.0,
+          "progress_score": 0.55
+        }
+      },
+      "avg_success_rate": 0.2271791383219955,
+      "avg_intention_score": 0.6291927437641723,
+      "avg_progress_score": 0.4142448979591837,
+      "error_policy": "exclude_physics_errors_v1",
+      "per_task_episode_accounting": {
+        "select_painting": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_book": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 45,
+          "excluded_episodes": 5,
+          "metric_denominator": 45,
+          "excluded_episode_ids": [
+            3,
+            4,
+            19,
+            22,
+            30
+          ]
+        },
+        "select_drink": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_chemistry_tube": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 49,
+          "excluded_episodes": 1,
+          "metric_denominator": 49,
+          "excluded_episode_ids": [
+            23
+          ]
+        },
+        "select_poker": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_mahjong": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_toy": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_fruit": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "add_condiment": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "insert_flower": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        }
+      },
+      "valid_episodes": 494,
+      "excluded_episodes": 6
+    },
+    "track_6_unseen_texture": {
+      "status": "complete",
+      "track": "track_6_unseen_texture",
+      "n_tasks": 10,
+      "episodes_per_task": {
+        "select_painting": 50,
+        "select_book": 50,
+        "select_drink": 50,
+        "select_chemistry_tube": 50,
+        "select_poker": 50,
+        "select_mahjong": 50,
+        "select_toy": 50,
+        "select_fruit": 50,
+        "add_condiment": 50,
+        "insert_flower": 50
+      },
+      "total_episodes": 500,
+      "per_task": {
+        "select_painting": {
+          "success_rate": 0.18,
+          "intention_score": 0.5,
+          "progress_score": 0.18
+        },
+        "select_book": {
+          "success_rate": 0.54,
+          "intention_score": 0.82,
+          "progress_score": 0.65
+        },
+        "select_drink": {
+          "success_rate": 0.36,
+          "intention_score": 0.94,
+          "progress_score": 0.56
+        },
+        "select_chemistry_tube": {
+          "success_rate": 0.38,
+          "intention_score": 0.02,
+          "progress_score": 0.61
+        },
+        "select_poker": {
+          "success_rate": 0.5,
+          "intention_score": 0.7,
+          "progress_score": 0.54
+        },
+        "select_mahjong": {
+          "success_rate": 0.3541666666666667,
+          "intention_score": 0.75,
+          "progress_score": 0.3541666666666667
+        },
+        "select_toy": {
+          "success_rate": 0.38,
+          "intention_score": 0.24,
+          "progress_score": 0.61
+        },
+        "select_fruit": {
+          "success_rate": 0.38,
+          "intention_score": 0.94,
+          "progress_score": 0.56
+        },
+        "add_condiment": {
+          "success_rate": 0.48,
+          "intention_score": 0.98,
+          "progress_score": 0.6933333333333331
+        },
+        "insert_flower": {
+          "success_rate": 0.02,
+          "intention_score": 0.96,
+          "progress_score": 0.41
+        }
+      },
+      "avg_success_rate": 0.3574166666666666,
+      "avg_intention_score": 0.685,
+      "avg_progress_score": 0.51675,
+      "error_policy": "exclude_physics_errors_v1",
+      "per_task_episode_accounting": {
+        "select_painting": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_book": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_drink": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_chemistry_tube": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_poker": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_mahjong": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 48,
+          "excluded_episodes": 2,
+          "metric_denominator": 48,
+          "excluded_episode_ids": [
+            25,
+            39
+          ]
+        },
+        "select_toy": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "select_fruit": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "add_condiment": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        },
+        "insert_flower": {
+          "error_policy": "exclude_physics_errors_v1",
+          "attempted_episodes": 50,
+          "valid_episodes": 50,
+          "excluded_episodes": 0,
+          "metric_denominator": 50,
+          "excluded_episode_ids": []
+        }
+      },
+      "valid_episodes": 498,
+      "excluded_episodes": 2
+    }
+  },
+  "n_tracks": 5,
+  "n_tasks": 50,
+  "total_episodes": 2460,
+  "shards": {
+    "track_1_in_distribution": {
+      "gpu": 0,
+      "output": "shards/track_1/race2abl_nojitter_h20_abl_nojitter_h20_s42_40000_ema_replan20_n50_seed7_subset_9d9b3e45",
+      "summary": "shards/track_1/race2abl_nojitter_h20_abl_nojitter_h20_s42_40000_ema_replan20_n50_seed7_subset_9d9b3e45/summary.json",
+      "completed_tasks": "shards/track_1/race2abl_nojitter_h20_abl_nojitter_h20_s42_40000_ema_replan20_n50_seed7_subset_9d9b3e45/track_1_in_distribution/completed_tasks",
+      "job": {
+        "gpu": 0,
+        "pid": 1872125,
+        "command": [
+          "/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python",
+          "/home/vision/seonghoon/vlabench/runtime/openpi/scripts/vlabench/eval_h25.py",
+          "--pool-worker",
+          "--method",
+          "race2abl_nojitter",
+          "--horizon",
+          "20",
+          "--exp-name",
+          "abl_nojitter_h20_s42",
+          "--step",
+          "40000",
+          "--gpu",
+          "0",
+          "--n-episode",
+          "50",
+          "--groups",
+          "2",
+          "--tracks",
+          "1",
+          "--replan-steps",
+          "20",
+          "--openpi-root",
+          "/home/vision/seonghoon/vlabench/runtime/openpi",
+          "--vlabench-root",
+          "/home/vision/seonghoon/vlabench/VLABench",
+          "--checkpoint-root",
+          "/home/vision/seonghoon/vlabench/runtime/openpi/checkpoints",
+          "--output-root",
+          "/home/vision/seonghoon/vlabench/runtime/openpi/evaluate_results/vlabench_h20/race2abl_nojitter_h20_abl_nojitter_h20_s42_40000_ema_replan20_n50_seed7/shards/track_1",
+          "--policy-python",
+          "/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python",
+          "--sim-python",
+          "/home/vision/seonghoon/vlabench/runtime/sim-env/bin/python",
+          "--startup-timeout",
+          "1200.0",
+          "--warmup-timeout",
+          "3600.0",
+          "--task-timeout",
+          "21600.0"
+        ],
+        "status": "complete",
+        "started_at": 1789600994.7304442,
+        "env_limits": {
+          "TORCHINDUCTOR_COMPILE_THREADS": "2",
+          "OMP_NUM_THREADS": "1",
+          "OPENBLAS_NUM_THREADS": "1",
+          "MKL_NUM_THREADS": "1"
+        },
+        "finished_at": 1789617126.4738705
+      }
+    },
+    "track_2_cross_category": {
+      "gpu": 1,
+      "output": "shards/track_2/race2abl_nojitter_h20_abl_nojitter_h20_s42_40000_ema_replan20_n50_seed7_subset_baa99750",
+      "summary": "shards/track_2/race2abl_nojitter_h20_abl_nojitter_h20_s42_40000_ema_replan20_n50_seed7_subset_baa99750/summary.json",
+      "completed_tasks": "shards/track_2/race2abl_nojitter_h20_abl_nojitter_h20_s42_40000_ema_replan20_n50_seed7_subset_baa99750/track_2_cross_category/completed_tasks",
+      "job": {
+        "gpu": 1,
+        "pid": 1872126,
+        "command": [
+          "/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python",
+          "/home/vision/seonghoon/vlabench/runtime/openpi/scripts/vlabench/eval_h25.py",
+          "--pool-worker",
+          "--method",
+          "race2abl_nojitter",
+          "--horizon",
+          "20",
+          "--exp-name",
+          "abl_nojitter_h20_s42",
+          "--step",
+          "40000",
+          "--gpu",
+          "1",
+          "--n-episode",
+          "50",
+          "--groups",
+          "2",
+          "--tracks",
+          "2",
+          "--replan-steps",
+          "20",
+          "--openpi-root",
+          "/home/vision/seonghoon/vlabench/runtime/openpi",
+          "--vlabench-root",
+          "/home/vision/seonghoon/vlabench/VLABench",
+          "--checkpoint-root",
+          "/home/vision/seonghoon/vlabench/runtime/openpi/checkpoints",
+          "--output-root",
+          "/home/vision/seonghoon/vlabench/runtime/openpi/evaluate_results/vlabench_h20/race2abl_nojitter_h20_abl_nojitter_h20_s42_40000_ema_replan20_n50_seed7/shards/track_2",
+          "--policy-python",
+          "/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python",
+          "--sim-python",
+          "/home/vision/seonghoon/vlabench/runtime/sim-env/bin/python",
+          "--startup-timeout",
+          "1200.0",
+          "--warmup-timeout",
+          "3600.0",
+          "--task-timeout",
+          "21600.0"
+        ],
+        "status": "complete",
+        "started_at": 1789600994.7326775,
+        "env_limits": {
+          "TORCHINDUCTOR_COMPILE_THREADS": "2",
+          "OMP_NUM_THREADS": "1",
+          "OPENBLAS_NUM_THREADS": "1",
+          "MKL_NUM_THREADS": "1"
+        },
+        "finished_at": 1789617740.1950321
+      }
+    },
+    "track_3_common_sense": {
+      "gpu": 0,
+      "output": "shards/track_3/race2abl_nojitter_h20_abl_nojitter_h20_s42_40000_ema_replan20_n50_seed7_subset_21b01caf",
+      "summary": "shards/track_3/race2abl_nojitter_h20_abl_nojitter_h20_s42_40000_ema_replan20_n50_seed7_subset_21b01caf/summary.json",
+      "completed_tasks": "shards/track_3/race2abl_nojitter_h20_abl_nojitter_h20_s42_40000_ema_replan20_n50_seed7_subset_21b01caf/track_3_common_sense/completed_tasks",
+      "job": {
+        "gpu": 0,
+        "pid": 1872127,
+        "command": [
+          "/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python",
+          "/home/vision/seonghoon/vlabench/runtime/openpi/scripts/vlabench/eval_h25.py",
+          "--pool-worker",
+          "--method",
+          "race2abl_nojitter",
+          "--horizon",
+          "20",
+          "--exp-name",
+          "abl_nojitter_h20_s42",
+          "--step",
+          "40000",
+          "--gpu",
+          "0",
+          "--n-episode",
+          "50",
+          "--groups",
+          "2",
+          "--tracks",
+          "3",
+          "--replan-steps",
+          "20",
+          "--openpi-root",
+          "/home/vision/seonghoon/vlabench/runtime/openpi",
+          "--vlabench-root",
+          "/home/vision/seonghoon/vlabench/VLABench",
+          "--checkpoint-root",
+          "/home/vision/seonghoon/vlabench/runtime/openpi/checkpoints",
+          "--output-root",
+          "/home/vision/seonghoon/vlabench/runtime/openpi/evaluate_results/vlabench_h20/race2abl_nojitter_h20_abl_nojitter_h20_s42_40000_ema_replan20_n50_seed7/shards/track_3",
+          "--policy-python",
+          "/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python",
+          "--sim-python",
+          "/home/vision/seonghoon/vlabench/runtime/sim-env/bin/python",
+          "--startup-timeout",
+          "1200.0",
+          "--warmup-timeout",
+          "3600.0",
+          "--task-timeout",
+          "21600.0"
+        ],
+        "status": "complete",
+        "started_at": 1789600994.7343996,
+        "env_limits": {
+          "TORCHINDUCTOR_COMPILE_THREADS": "2",
+          "OMP_NUM_THREADS": "1",
+          "OPENBLAS_NUM_THREADS": "1",
+          "MKL_NUM_THREADS": "1"
+        },
+        "finished_at": 1789619455.8005862
+      }
+    },
+    "track_4_semantic_instruction": {
+      "gpu": 1,
+      "output": "shards/track_4/race2abl_nojitter_h20_abl_nojitter_h20_s42_40000_ema_replan20_n50_seed7_subset_5fae63be",
+      "summary": "shards/track_4/race2abl_nojitter_h20_abl_nojitter_h20_s42_40000_ema_replan20_n50_seed7_subset_5fae63be/summary.json",
+      "completed_tasks": "shards/track_4/race2abl_nojitter_h20_abl_nojitter_h20_s42_40000_ema_replan20_n50_seed7_subset_5fae63be/track_4_semantic_instruction/completed_tasks",
+      "job": {
+        "gpu": 1,
+        "pid": 1872128,
+        "command": [
+          "/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python",
+          "/home/vision/seonghoon/vlabench/runtime/openpi/scripts/vlabench/eval_h25.py",
+          "--pool-worker",
+          "--method",
+          "race2abl_nojitter",
+          "--horizon",
+          "20",
+          "--exp-name",
+          "abl_nojitter_h20_s42",
+          "--step",
+          "40000",
+          "--gpu",
+          "1",
+          "--n-episode",
+          "50",
+          "--groups",
+          "2",
+          "--tracks",
+          "4",
+          "--replan-steps",
+          "20",
+          "--openpi-root",
+          "/home/vision/seonghoon/vlabench/runtime/openpi",
+          "--vlabench-root",
+          "/home/vision/seonghoon/vlabench/VLABench",
+          "--checkpoint-root",
+          "/home/vision/seonghoon/vlabench/runtime/openpi/checkpoints",
+          "--output-root",
+          "/home/vision/seonghoon/vlabench/runtime/openpi/evaluate_results/vlabench_h20/race2abl_nojitter_h20_abl_nojitter_h20_s42_40000_ema_replan20_n50_seed7/shards/track_4",
+          "--policy-python",
+          "/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python",
+          "--sim-python",
+          "/home/vision/seonghoon/vlabench/runtime/sim-env/bin/python",
+          "--startup-timeout",
+          "1200.0",
+          "--warmup-timeout",
+          "3600.0",
+          "--task-timeout",
+          "21600.0"
+        ],
+        "status": "complete",
+        "started_at": 1789600994.7361388,
+        "env_limits": {
+          "TORCHINDUCTOR_COMPILE_THREADS": "2",
+          "OMP_NUM_THREADS": "1",
+          "OPENBLAS_NUM_THREADS": "1",
+          "MKL_NUM_THREADS": "1"
+        },
+        "finished_at": 1789618535.9508383
+      }
+    },
+    "track_6_unseen_texture": {
+      "gpu": 0,
+      "output": "shards/track_6/race2abl_nojitter_h20_abl_nojitter_h20_s42_40000_ema_replan20_n50_seed7_subset_498302e6",
+      "summary": "shards/track_6/race2abl_nojitter_h20_abl_nojitter_h20_s42_40000_ema_replan20_n50_seed7_subset_498302e6/summary.json",
+      "completed_tasks": "shards/track_6/race2abl_nojitter_h20_abl_nojitter_h20_s42_40000_ema_replan20_n50_seed7_subset_498302e6/track_6_unseen_texture/completed_tasks",
+      "job": {
+        "gpu": 0,
+        "pid": 2654518,
+        "command": [
+          "/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python",
+          "/home/vision/seonghoon/vlabench/runtime/openpi/scripts/vlabench/eval_h25.py",
+          "--pool-worker",
+          "--method",
+          "race2abl_nojitter",
+          "--horizon",
+          "20",
+          "--exp-name",
+          "abl_nojitter_h20_s42",
+          "--step",
+          "40000",
+          "--gpu",
+          "0",
+          "--n-episode",
+          "50",
+          "--groups",
+          "2",
+          "--tracks",
+          "6",
+          "--replan-steps",
+          "20",
+          "--openpi-root",
+          "/home/vision/seonghoon/vlabench/runtime/openpi",
+          "--vlabench-root",
+          "/home/vision/seonghoon/vlabench/VLABench",
+          "--checkpoint-root",
+          "/home/vision/seonghoon/vlabench/runtime/openpi/checkpoints",
+          "--output-root",
+          "/home/vision/seonghoon/vlabench/runtime/openpi/evaluate_results/vlabench_h20/race2abl_nojitter_h20_abl_nojitter_h20_s42_40000_ema_replan20_n50_seed7/shards/track_6",
+          "--policy-python",
+          "/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python",
+          "--sim-python",
+          "/home/vision/seonghoon/vlabench/runtime/sim-env/bin/python",
+          "--startup-timeout",
+          "1200.0",
+          "--warmup-timeout",
+          "3600.0",
+          "--task-timeout",
+          "21600.0"
+        ],
+        "status": "complete",
+        "started_at": 1789617126.4909804,
+        "env_limits": {
+          "TORCHINDUCTOR_COMPILE_THREADS": "2",
+          "OMP_NUM_THREADS": "1",
+          "OPENBLAS_NUM_THREADS": "1",
+          "MKL_NUM_THREADS": "1"
+        },
+        "finished_at": 1789626151.255465
+      }
+    }
+  },
+  "pool_manifest": "pool_manifest.json",
+  "error_policy": "exclude_physics_errors_v1",
+  "valid_episodes": 2430,
+  "excluded_episodes": 30,
+  "aggregation": "Equal weight for five tracks; equal task weight within each track. Recorded PhysicsError episodes are excluded from each task's metric denominator; ordinary unsuccessful rollouts remain included.",
+  "avg_success_rate": 0.3280345524121344,
+  "avg_intention_score": 0.6272423291122988,
+  "avg_progress_score": 0.4871499124725725
+}

ablation/abl_nojitter_h20_40k/logs/history.jsonl

@@ -0,0 +1,5 @@
+{"method": "race2abl_nojitter", "config": "pi05_vlabench_race2abl_nojitter_h20", "exp_name": "abl_nojitter_h20_s42", "horizon": 20, "micro_batch_size": 32, "batch_size": 64, "world_size": 2, "per_gpu_batch_size": 32, "accumulation_steps": 1, "gpus": [0, 1], "train_command": ["/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/torchrun", "--standalone", "--nnodes=1", "--nproc_per_node=2", "scripts/race2_ablation/train_race2abl_nojitter_vlabench.py", "pi05_vlabench_race2abl_nojitter_h20", "--exp_name", "abl_nojitter_h20_s42", "--model.micro-batch-size", "32"], "eval_command": ["/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python", "scripts/vlabench/eval_pool.py", "--method", "race2abl_nojitter", "--horizon", "20", "--exp-name", "abl_nojitter_h20_s42", "--step", "40000", "--gpus", "0,1", "--jobs-per-gpu", "2", "--groups-per-job", "2", "--n-episode", "50", "--openpi-root", "/home/vision/seonghoon/vlabench/runtime/openpi", "--vlabench-root", "/home/vision/seonghoon/vlabench/VLABench", "--sim-python", "/home/vision/seonghoon/vlabench/runtime/sim-env/bin/python"], "checkpoint": "/home/vision/seonghoon/vlabench/runtime/openpi/checkpoints/pi05_vlabench_race2abl_nojitter_h20/abl_nojitter_h20_s42/40000/ema.safetensors", "log_dir": "/home/vision/seonghoon/vlabench/runtime/openpi/logs/h20/race2abl_nojitter_abl_nojitter_h20_s42", "openpi_root": "/home/vision/seonghoon/vlabench/runtime/openpi", "setup_root": "/home/vision/seonghoon/vlabench/race_eval/logs/h30_h35_setup", "stage": "launch_validated", "time": "2026-09-16T07:58:58.077119+00:00", "pid": 1446134, "launch_validation": "/home/vision/seonghoon/vlabench/runtime/openpi/logs/h20/race2abl_nojitter_abl_nojitter_h20_s42/launch_validation.json"}
+{"method": "race2abl_nojitter", "config": "pi05_vlabench_race2abl_nojitter_h20", "exp_name": "abl_nojitter_h20_s42", "horizon": 20, "micro_batch_size": 32, "batch_size": 64, "world_size": 2, "per_gpu_batch_size": 32, "accumulation_steps": 1, "gpus": [0, 1], "train_command": ["/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/torchrun", "--standalone", "--nnodes=1", "--nproc_per_node=2", "scripts/race2_ablation/train_race2abl_nojitter_vlabench.py", "pi05_vlabench_race2abl_nojitter_h20", "--exp_name", "abl_nojitter_h20_s42", "--model.micro-batch-size", "32"], "eval_command": ["/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python", "scripts/vlabench/eval_pool.py", "--method", "race2abl_nojitter", "--horizon", "20", "--exp-name", "abl_nojitter_h20_s42", "--step", "40000", "--gpus", "0,1", "--jobs-per-gpu", "2", "--groups-per-job", "2", "--n-episode", "50", "--openpi-root", "/home/vision/seonghoon/vlabench/runtime/openpi", "--vlabench-root", "/home/vision/seonghoon/vlabench/VLABench", "--sim-python", "/home/vision/seonghoon/vlabench/runtime/sim-env/bin/python"], "checkpoint": "/home/vision/seonghoon/vlabench/runtime/openpi/checkpoints/pi05_vlabench_race2abl_nojitter_h20/abl_nojitter_h20_s42/40000/ema.safetensors", "log_dir": "/home/vision/seonghoon/vlabench/runtime/openpi/logs/h20/race2abl_nojitter_abl_nojitter_h20_s42", "openpi_root": "/home/vision/seonghoon/vlabench/runtime/openpi", "setup_root": "/home/vision/seonghoon/vlabench/race_eval/logs/h30_h35_setup", "stage": "training", "time": "2026-09-16T07:58:58.080047+00:00", "pid": 1446134, "child_pid": 1446169, "log": "/home/vision/seonghoon/vlabench/runtime/openpi/logs/h20/race2abl_nojitter_abl_nojitter_h20_s42/train.log"}
+{"method": "race2abl_nojitter", "config": "pi05_vlabench_race2abl_nojitter_h20", "exp_name": "abl_nojitter_h20_s42", "horizon": 20, "micro_batch_size": 32, "batch_size": 64, "world_size": 2, "per_gpu_batch_size": 32, "accumulation_steps": 1, "gpus": [0, 1], "train_command": ["/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/torchrun", "--standalone", "--nnodes=1", "--nproc_per_node=2", "scripts/race2_ablation/train_race2abl_nojitter_vlabench.py", "pi05_vlabench_race2abl_nojitter_h20", "--exp_name", "abl_nojitter_h20_s42", "--model.micro-batch-size", "32"], "eval_command": ["/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python", "scripts/vlabench/eval_pool.py", "--method", "race2abl_nojitter", "--horizon", "20", "--exp-name", "abl_nojitter_h20_s42", "--step", "40000", "--gpus", "0,1", "--jobs-per-gpu", "2", "--groups-per-job", "2", "--n-episode", "50", "--openpi-root", "/home/vision/seonghoon/vlabench/runtime/openpi", "--vlabench-root", "/home/vision/seonghoon/vlabench/VLABench", "--sim-python", "/home/vision/seonghoon/vlabench/runtime/sim-env/bin/python"], "checkpoint": "/home/vision/seonghoon/vlabench/runtime/openpi/checkpoints/pi05_vlabench_race2abl_nojitter_h20/abl_nojitter_h20_s42/40000/ema.safetensors", "log_dir": "/home/vision/seonghoon/vlabench/runtime/openpi/logs/h20/race2abl_nojitter_abl_nojitter_h20_s42", "openpi_root": "/home/vision/seonghoon/vlabench/runtime/openpi", "setup_root": "/home/vision/seonghoon/vlabench/race_eval/logs/h30_h35_setup", "stage": "trained", "time": "2026-09-16T23:23:12.762010+00:00", "pid": 1446134, "checkpoint_validation": {"step": 40000, "config": "pi05_vlabench_race2abl_nojitter_h20", "ema": "/home/vision/seonghoon/vlabench/runtime/openpi/checkpoints/pi05_vlabench_race2abl_nojitter_h20/abl_nojitter_h20_s42/40000/ema.safetensors", "n_tensors": 264, "horizon": 20, "micro_batch_size": 32}}
+{"method": "race2abl_nojitter", "config": "pi05_vlabench_race2abl_nojitter_h20", "exp_name": "abl_nojitter_h20_s42", "horizon": 20, "micro_batch_size": 32, "batch_size": 64, "world_size": 2, "per_gpu_batch_size": 32, "accumulation_steps": 1, "gpus": [0, 1], "train_command": ["/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/torchrun", "--standalone", "--nnodes=1", "--nproc_per_node=2", "scripts/race2_ablation/train_race2abl_nojitter_vlabench.py", "pi05_vlabench_race2abl_nojitter_h20", "--exp_name", "abl_nojitter_h20_s42", "--model.micro-batch-size", "32"], "eval_command": ["/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python", "scripts/vlabench/eval_pool.py", "--method", "race2abl_nojitter", "--horizon", "20", "--exp-name", "abl_nojitter_h20_s42", "--step", "40000", "--gpus", "0,1", "--jobs-per-gpu", "2", "--groups-per-job", "2", "--n-episode", "50", "--openpi-root", "/home/vision/seonghoon/vlabench/runtime/openpi", "--vlabench-root", "/home/vision/seonghoon/vlabench/VLABench", "--sim-python", "/home/vision/seonghoon/vlabench/runtime/sim-env/bin/python"], "checkpoint": "/home/vision/seonghoon/vlabench/runtime/openpi/checkpoints/pi05_vlabench_race2abl_nojitter_h20/abl_nojitter_h20_s42/40000/ema.safetensors", "log_dir": "/home/vision/seonghoon/vlabench/runtime/openpi/logs/h20/race2abl_nojitter_abl_nojitter_h20_s42", "openpi_root": "/home/vision/seonghoon/vlabench/runtime/openpi", "setup_root": "/home/vision/seonghoon/vlabench/race_eval/logs/h30_h35_setup", "stage": "evaluating", "time": "2026-09-16T23:23:12.764496+00:00", "pid": 1446134, "child_pid": 1871797, "log": "/home/vision/seonghoon/vlabench/runtime/openpi/logs/h20/race2abl_nojitter_abl_nojitter_h20_s42/eval.log"}
+{"method": "race2abl_nojitter", "config": "pi05_vlabench_race2abl_nojitter_h20", "exp_name": "abl_nojitter_h20_s42", "horizon": 20, "micro_batch_size": 32, "batch_size": 64, "world_size": 2, "per_gpu_batch_size": 32, "accumulation_steps": 1, "gpus": [0, 1], "train_command": ["/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/torchrun", "--standalone", "--nnodes=1", "--nproc_per_node=2", "scripts/race2_ablation/train_race2abl_nojitter_vlabench.py", "pi05_vlabench_race2abl_nojitter_h20", "--exp_name", "abl_nojitter_h20_s42", "--model.micro-batch-size", "32"], "eval_command": ["/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python", "scripts/vlabench/eval_pool.py", "--method", "race2abl_nojitter", "--horizon", "20", "--exp-name", "abl_nojitter_h20_s42", "--step", "40000", "--gpus", "0,1", "--jobs-per-gpu", "2", "--groups-per-job", "2", "--n-episode", "50", "--openpi-root", "/home/vision/seonghoon/vlabench/runtime/openpi", "--vlabench-root", "/home/vision/seonghoon/vlabench/VLABench", "--sim-python", "/home/vision/seonghoon/vlabench/runtime/sim-env/bin/python"], "checkpoint": "/home/vision/seonghoon/vlabench/runtime/openpi/checkpoints/pi05_vlabench_race2abl_nojitter_h20/abl_nojitter_h20_s42/40000/ema.safetensors", "log_dir": "/home/vision/seonghoon/vlabench/runtime/openpi/logs/h20/race2abl_nojitter_abl_nojitter_h20_s42", "openpi_root": "/home/vision/seonghoon/vlabench/runtime/openpi", "setup_root": "/home/vision/seonghoon/vlabench/race_eval/logs/h30_h35_setup", "stage": "complete", "time": "2026-09-17T06:22:33.052570+00:00", "pid": 1446134}

ablation/abl_nojitter_h20_40k/logs/launch_validation.json

@@ -0,0 +1,149 @@
+{
+  "status": "verified",
+  "method": "race2abl_nojitter",
+  "config": "pi05_vlabench_race2abl_nojitter_h20",
+  "exp_name": "abl_nojitter_h20_s42",
+  "horizon": 20,
+  "micro_batch_size": 32,
+  "batch_size": 64,
+  "world_size": 2,
+  "per_gpu_batch_size": 32,
+  "accumulation_steps": 1,
+  "gpus": [
+    0,
+    1
+  ],
+  "train_command": [
+    "/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/torchrun",
+    "--standalone",
+    "--nnodes=1",
+    "--nproc_per_node=2",
+    "scripts/race2_ablation/train_race2abl_nojitter_vlabench.py",
+    "pi05_vlabench_race2abl_nojitter_h20",
+    "--exp_name",
+    "abl_nojitter_h20_s42",
+    "--model.micro-batch-size",
+    "32"
+  ],
+  "eval_command": [
+    "/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python",
+    "scripts/vlabench/eval_pool.py",
+    "--method",
+    "race2abl_nojitter",
+    "--horizon",
+    "20",
+    "--exp-name",
+    "abl_nojitter_h20_s42",
+    "--step",
+    "40000",
+    "--gpus",
+    "0,1",
+    "--jobs-per-gpu",
+    "2",
+    "--groups-per-job",
+    "2",
+    "--n-episode",
+    "50",
+    "--openpi-root",
+    "/home/vision/seonghoon/vlabench/runtime/openpi",
+    "--vlabench-root",
+    "/home/vision/seonghoon/vlabench/VLABench",
+    "--sim-python",
+    "/home/vision/seonghoon/vlabench/runtime/sim-env/bin/python"
+  ],
+  "checkpoint": "/home/vision/seonghoon/vlabench/runtime/openpi/checkpoints/pi05_vlabench_race2abl_nojitter_h20/abl_nojitter_h20_s42/40000/ema.safetensors",
+  "log_dir": "/home/vision/seonghoon/vlabench/runtime/openpi/logs/h20/race2abl_nojitter_abl_nojitter_h20_s42",
+  "openpi_root": "/home/vision/seonghoon/vlabench/runtime/openpi",
+  "setup_root": "/home/vision/seonghoon/vlabench/race_eval/logs/h30_h35_setup",
+  "setup_reports": {
+    "dataset_validation.json": {
+      "path": "/home/vision/seonghoon/vlabench/race_eval/logs/h30_h35_setup/dataset_validation.json",
+      "sha256": "756efafff23c16ca1d4efcc33088aee2c3a2446ac3ce99c2dc800788cf984358",
+      "status": "verified"
+    },
+    "teacher_validation.json": {
+      "path": "/home/vision/seonghoon/vlabench/race_eval/logs/h30_h35_setup/teacher_validation.json",
+      "sha256": "d8fa692b7ed306d32a36120b4c8069fc479c23a69034eba11aec111aebbe6feb",
+      "status": "verified"
+    },
+    "cache_ready.json": {
+      "path": "/home/vision/seonghoon/vlabench/race_eval/logs/h30_h35_setup/cache_ready.json",
+      "sha256": "481dcb9b8b880cfaa9104f2f9d4c2b27bda6c5fc6665aeee3c2b2453a0a33123",
+      "status": "verified"
+    }
+  },
+  "recipe": {
+    "name": "pi05_vlabench_race2abl_nojitter_h20",
+    "exp_name": "abl_nojitter_h20_s42",
+    "pytorch_weight_path": "checkpoints/pi05_vlabench_official/pytorch",
+    "seed": 42,
+    "batch_size": 64,
+    "pytorch_training_precision": "bfloat16",
+    "optimizer": {
+      "b1": 0.9,
+      "b2": 0.95,
+      "eps": 1e-08,
+      "weight_decay": 1e-10,
+      "clip_gradient_norm": 1.0
+    },
+    "lr_schedule": {
+      "warmup_steps": 500,
+      "peak_lr": 5e-05,
+      "decay_steps": 39500,
+      "decay_lr": 5e-06
+    },
+    "ema_decay": 0.999,
+    "model": {
+      "action_dim": 32,
+      "action_horizon": 20,
+      "max_token_len": 200,
+      "dtype": "bfloat16",
+      "paligemma_variant": "gemma_2b",
+      "action_expert_variant": "gemma_300m",
+      "pi05": true,
+      "discrete_state_input": false,
+      "gemma_depth": 18,
+      "sigma": 1.0,
+      "support": 3.0,
+      "jitter_prob": 0.0,
+      "lambda_timing": 0.05,
+      "lambda_base_fm": 0.1,
+      "read_tau": 1.0,
+      "n_tau": 10,
+      "tau_gate_init": 0.0,
+      "tau_gate_lr_mult": 20.0,
+      "timing_proj_dim": 256,
+      "timing_num_heads": 4,
+      "fast_serving": 0,
+      "kv_layer": -1
+    },
+    "data": {
+      "repo_id": "SeonghoonYu/vlabench_primitive_ft_lerobot_224",
+      "assets": {
+        "assets_dir": "./assets/pi05_vlabench",
+        "asset_id": "vlabench/vlabench_ft_primitive"
+      },
+      "phase_labels_path": null,
+      "phase_label_horizon": null,
+      "cps_path": "assets/gauss_timing/vlabench_cps.npz",
+      "gauss_sigma": 1.0,
+      "gauss_support": 3.0,
+      "base_config": {
+        "repo_id": null,
+        "asset_id": null,
+        "use_quantile_norm": false,
+        "action_sequence_keys": [
+          "actions"
+        ],
+        "prompt_from_task": true
+      }
+    }
+  },
+  "gpu_uuids": [
+    "GPU-04ba4c23-de58-e0b8-76b7-f74c2161d827",
+    "GPU-1b5eea96-3014-64c1-dd0e-f6a54d837101"
+  ],
+  "hf_lerobot_home": "/home/vision/seonghoon/vlabench/.cache/huggingface/lerobot",
+  "hf_datasets_cache": "/home/vision/seonghoon/vlabench/.cache/huggingface/datasets",
+  "openpi_data_home": "/home/vision/seonghoon/vlabench/.cache/openpi"
+}

ablation/abl_nojitter_h20_40k/logs/smoke_nojitter_h20.log

@@ -0,0 +1,26 @@
+Race2NoJitterPI0: conditioning ON | lambda_base_fm 0.1 lambda_timing 0.05 | head params=790,017 | dtype=bfloat16
+WARNING:root:
+The dataset you requested (SeonghoonYu/vlabench_primitive_ft_lerobot_224) is in 2.0 format.
+While current version of LeRobot is backward-compatible with it, the version of your dataset still uses global
+stats instead of per-episode stats. Update your dataset stats to the new format using this command:
+```
+python lerobot/common/datasets/v21/convert_dataset_v20_to_v21.py --repo-id=SeonghoonYu/vlabench_primitive_ft_lerobot_224
+```
+
+If you encounter a problem, contact LeRobot maintainers on [Discord](https://discord.com/invite/s3KuuzsPFb)
+or open an [issue on GitHub](https://github.com/huggingface/lerobot/issues/new/choose).
+
+WARNING:root:
+The dataset you requested (SeonghoonYu/vlabench_primitive_ft_lerobot_224) is in 2.0 format.
+While current version of LeRobot is backward-compatible with it, the version of your dataset still uses global
+stats instead of per-episode stats. Update your dataset stats to the new format using this command:
+```
+python lerobot/common/datasets/v21/convert_dataset_v20_to_v21.py --repo-id=SeonghoonYu/vlabench_primitive_ft_lerobot_224
+```
+
+If you encounter a problem, contact LeRobot maintainers on [Discord](https://discord.com/invite/s3KuuzsPFb)
+or open an [issue on GitHub](https://github.com/huggingface/lerobot/issues/new/choose).
+
+train: loss 0.4817 fm 0.3917 base_fm 0.52427 bce 0.7522 | grad race_head 1.11e+00 expert 1.38e+01 field 1.22e-02 VLM 0.00e+00
+serve: |a|max 6.824 finite=True | determinism 0.00e+00 | parity vs reference 0.00e+00 | per-call 402 ms (first 407 ms)
+SMOKE_OK

ablation/abl_nojitter_h20_40k/logs/status.json

@@ -0,0 +1,60 @@
+{
+  "method": "race2abl_nojitter",
+  "config": "pi05_vlabench_race2abl_nojitter_h20",
+  "exp_name": "abl_nojitter_h20_s42",
+  "horizon": 20,
+  "micro_batch_size": 32,
+  "batch_size": 64,
+  "world_size": 2,
+  "per_gpu_batch_size": 32,
+  "accumulation_steps": 1,
+  "gpus": [
+    0,
+    1
+  ],
+  "train_command": [
+    "/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/torchrun",
+    "--standalone",
+    "--nnodes=1",
+    "--nproc_per_node=2",
+    "scripts/race2_ablation/train_race2abl_nojitter_vlabench.py",
+    "pi05_vlabench_race2abl_nojitter_h20",
+    "--exp_name",
+    "abl_nojitter_h20_s42",
+    "--model.micro-batch-size",
+    "32"
+  ],
+  "eval_command": [
+    "/home/vision/seonghoon/vlabench/runtime/openpi/.venv/bin/python",
+    "scripts/vlabench/eval_pool.py",
+    "--method",
+    "race2abl_nojitter",
+    "--horizon",
+    "20",
+    "--exp-name",
+    "abl_nojitter_h20_s42",
+    "--step",
+    "40000",
+    "--gpus",
+    "0,1",
+    "--jobs-per-gpu",
+    "2",
+    "--groups-per-job",
+    "2",
+    "--n-episode",
+    "50",
+    "--openpi-root",
+    "/home/vision/seonghoon/vlabench/runtime/openpi",
+    "--vlabench-root",
+    "/home/vision/seonghoon/vlabench/VLABench",
+    "--sim-python",
+    "/home/vision/seonghoon/vlabench/runtime/sim-env/bin/python"
+  ],
+  "checkpoint": "/home/vision/seonghoon/vlabench/runtime/openpi/checkpoints/pi05_vlabench_race2abl_nojitter_h20/abl_nojitter_h20_s42/40000/ema.safetensors",
+  "log_dir": "/home/vision/seonghoon/vlabench/runtime/openpi/logs/h20/race2abl_nojitter_abl_nojitter_h20_s42",
+  "openpi_root": "/home/vision/seonghoon/vlabench/runtime/openpi",
+  "setup_root": "/home/vision/seonghoon/vlabench/race_eval/logs/h30_h35_setup",
+  "stage": "complete",
+  "time": "2026-09-17T06:22:33.052570+00:00",
+  "pid": 1446134
+}

ablation/abl_nojitter_h20_40k/logs/train.log.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4eddf3de8b507bc3e584291383540bc1639b2017ec9b2437c8853e95993b17b0
+size 1667120

ablation/abl_nojitter_h20_40k/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:57808c86abc5f31158acba4283b2ec776e35eda7e8a9d07cb7546ffa1fd45bbe
+size 7939730180

ablation/abl_nojitter_h20_40k/overlay/ema.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:38804fce0cf3d94eeefb2f120f0b7b955ef0e1a8711dee303ebbb14bba2d77a5
+size 2092955412

ablation/abl_nojitter_h20_40k/overlay/metadata.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d0b2208262b88fc3f7b9d5578d5daaf79af7a74e9ebe4ef514dacd060b4eff0d
+size 3699

ablation/abl_nojitter_h20_40k/provenance.json

@@ -0,0 +1,51 @@
+{
+  "method": "RACE2 ablation A6 (no label jitter)",
+  "config": "pi05_vlabench_race2abl_nojitter_h20",
+  "model_class": "Race2NoJitterPI0",
+  "exp_name": "abl_nojitter_h20_s42",
+  "step": 40000,
+  "weights": "EMA",
+  "ema_decay": 0.999,
+  "action_horizon": 20,
+  "training_seed": 42,
+  "world_size": 2,
+  "micro_batch_size": 32,
+  "accumulation_steps": 1,
+  "global_batch": 64,
+  "ema_overlay_sha256": "38804fce0cf3d94eeefb2f120f0b7b955ef0e1a8711dee303ebbb14bba2d77a5",
+  "materialized_model_sha256": "57808c86abc5f31158acba4283b2ec776e35eda7e8a9d07cb7546ffa1fd45bbe",
+  "teacher_sha256": "f6762a23e4374bcbd8f3a7297f159b02d253a335f125eb93595d431d37607172",
+  "dataset": {
+    "repo_id": "SeonghoonYu/vlabench_primitive_ft_lerobot_224",
+    "revision": "5772f03c3af8df38ee75d8f71cab0075563ed12a"
+  },
+  "changepoints_sha256": "6e31b27469d69fa65f2e3c41bcbf016b0f8b426048ca2d88d32367d1aa5aa4fc",
+  "code_repository": "https://github.com/Seonghoon-Yu/vlabench",
+  "code_revision": "9ac5495d1fae7a1dfc5404183bb46ac8ea16ce3f",
+  "hardware": "2 x NVIDIA A100-SXM4-80GB (host vision)",
+  "trained": "2026-09-16T07:58Z to 2026-09-16T23:23Z",
+  "evaluated": "2026-09-16T23:23Z to 2026-09-17T06:22Z",
+  "evaluation": {
+    "replan_steps": 20,
+    "seed": 7,
+    "tracks": [
+      1,
+      2,
+      3,
+      4,
+      6
+    ],
+    "n_episode": 2460,
+    "physics_errors_excluded": 30,
+    "success_rate_pct": 32.8,
+    "intention_score_pct": 62.72,
+    "progress_score_pct": 48.71,
+    "track_sr_pct": {
+      "T1": 52.53,
+      "T2": 26.22,
+      "T3": 26.82,
+      "T4": 22.72,
+      "T6": 35.74
+    }
+  }
+}

ablation/abl_nojitter_h20_40k/source/scripts/race2_ablation/README.md

@@ -0,0 +1,195 @@
+# RACE2 ablations (VLABench, H20)
+
+이 폴더는 full RACE2의 아홉 가지 ablation을 **기존 코드와 분리된 파일**로 구현한 것입니다.
+`scripts/race2/`, `openpi.models_pytorch.race{,2}`, full-method preset은 건드리지 않았습니다.
+
+## 1. Ablation 정의
+
+Full RACE2 (`RacePI0.forward`) 학습 step:
+```
+aux pass  τ=read_tau(=1, 순수 noise), 변조 없음
+            → timing head → bce(b*)                     timing loss     λ_timing  = 0.05
+            → action_out_proj → base_fm(u_t)            auxiliary loss  λ_base_fm = 0.1
+FM pass   τ~U(0,1), α(τ)·jitter(b*)·M(e_trans) 로 변조 (teacher forcing)   fm_loss
+total = fm_loss + 0.1·base_fm + 0.05·bce
+serving   aux pass → b̂ → α(τ)·b̂·M(e_trans) 를 expert norm site 37곳에 주입
+```
+
+| | preset (H20) | 모델 클래스 | 학습 loss | conditioning | serving |
+|---|---|---|---|---|---|
+| **A1 timing_only** | `pi05_vlabench_race2abl_timing_only_h20` | `Race2TimingOnlyPI0` | `fm_loss(변조 없음) + 0.05·bce` | **없음** — `e_trans`/`field_mod`/`tau_gate` 파라미터 자체가 없음, teacher forcing 없음 | 순수 pi0.5 sampler (aux pass 안 함) |
+| **A3 noaux** | `pi05_vlabench_race2abl_noaux_h20` | `Race2NoAuxPI0` | `fm_loss(변조, teacher-forced b*) + 0.05·bce` | full과 동일 | full과 동일 |
+| **A2 fmonly** | `pi05_vlabench_race2abl_fmonly_h20` | `Race2FmOnlyPI0` | `fm_loss(변조)` **만** — bce·base_fm은 no_grad 모니터링 | 파라미터·site는 full과 동일하지만 field에 **b\* 대신 head 출력 b̂** (teacher forcing·jitter 없음) | full과 동일 (b̂ → 변조) |
+| **A4 randcp** | `pi05_vlabench_race2abl_randcp_h20` | `Race2PI0` (full 그대로) | full과 동일 (`fm + 0.1·base_fm + 0.05·bce`) | full과 동일 — 단 **label b\*가 랜덤 changepoint**에서 계산됨 | full과 동일 |
+| **A5 speedcp** | `pi05_vlabench_race2abl_speedcp_h20` | `Race2PI0` (full 그대로) | full과 동일 | full과 동일 — 단 **label b\*가 PACE-style speed minima**에서 계산됨 | full과 동일 |
+| **A6 nojitter** | `pi05_vlabench_race2abl_nojitter_h20` | `Race2NoJitterPI0` | full과 동일 | full과 동일, 단 **jitter 없음** (`jitter_prob=0`, field에 b\* 그대로) | full과 동일 |
+| **A7 notf** (w/o teacher forcing) | `pi05_vlabench_race2abl_notf_h20` | `Race2NoTeacherForcingPI0` | full과 동일 (fm + 0.1·base_fm + 0.05·bce) | field에 b\* 대신 **`b̂.detach()`** (head는 bce로만 학습, field·expert는 예측 분포 아래서 학습; jitter 없음) | full과 동일 |
+| **A8 vlmhead** (head: VLM features only) | `pi05_vlabench_race2abl_vlmhead_h20` | `Race2VlmHeadPI0` | full과 동일 | full과 동일 | full과 동일 |
+| **A9 actionhead** (head: action features only) | `pi05_vlabench_race2abl_actionhead_h20` | `Race2ActionHeadPI0` | full과 동일 | full과 동일 | full과 동일 |
+
+- A1은 bce gradient가 aux pass의 `hidden_read`를 통해 **action expert로 흘러갑니다** (full과 같이 detach 안 함). 즉 "timing supervision을 multitask 신호로만 줬을 때 정책이 좋아지는가"를 봅니다.
+- A3는 `RacePI0.forward`가 `lambda_base_fm == 0`일 때 base_fm 항을 계산 자체를 안 하므로, 클래스는 그 설정을 고정만 합니다.
+- A2는 timing head·α(τ)·e_trans·field_mod가 **FM gradient만으로** 학습됩니다 (fm → Δ=α·b̂·M(e_trans) → b̂ → head → expert). auxiliary FM loss는 적용되지 않고(`base_fm`은 로그용), b̂이 changepoint 근접도를 배우는지는 로그의 `bce`/`pres_acc`/`peak_pm1`(모두 no_grad 모니터링)로 관찰합니다.
+  주의: `field_mod`가 zero-init이라 **step 0에서는 head/e_trans/tau_gate gradient가 0**이고, field_mod가 먼저 움직인 뒤(첫 optimizer step 이후) 경로가 열립니다 — full RACE2의 e_trans/tau_gate와 같은 동작이며 smoke가 이 2단계를 검사합니다.
+- A4는 **모델·loss·코드가 full과 완전히 같고 label 파일만 다릅니다.** `assets/gauss_timing/vlabench_cps_random_s2026.npz`는
+  `make_random_changepoints.py`가 원본 PELT 파일(`vlabench_cps.npz`, sha256 `6e31b274…`)에서 **episode마다 같은 개수**의 changepoint를
+  frame `[1, T−1)`에서 균등 랜덤(중복 없음, 간격 제약 없음, seed 2026)으로 뽑아 고정한 것입니다 (5,000 ep, 17,845 CP, 평균 3.57/ep 동일;
+  진짜 CP ±3 frame 안에 든 비율 25.3% ≈ 우연 수준 23.9%). 생성 결과는 `…_random_s2026.report.json`에 있고, trainer는 `randomized=True`
+  메타가 없는 파일(= 진짜 label)을 거부합니다. 로그의 `pres_acc`/`peak_pm1`은 랜덤 b\*에 대한 값이라 낮은 게 정상입니다.
+  다른 seed로 다시 뽑으려면 `.venv/bin/python scripts/race2_ablation/make_random_changepoints.py --seed <S>` 후 preset의 `cps_path`를 바꿉니다.
+- A5는 **label source를 PELT changepoint → PACE(arXiv 2606.00537)식 저속 valley로 바꾼 것**이고 모델·loss·코드는 full과 같습니다.
+  `make_speed_minima_changepoints.py`가 각 데모의 action(EE pose xyz)에서 translation speed $v_t=\|p_{t+1}-p_t\|$ → 이동평균(window 5) →
+  local minima(최소 간격 d_min=10) + prominence → task별로 500 데모의 prominence를 모아 ρ=5 percentile을 δ_T로 잡고 Φ ≥ δ_T만 채택
+  (PACE default d_min=10, ρ=5; PACE는 학습 데모에서 δ_T를 보정). PACE가 명시하지 않은 ψ/S/Φ는 위 표준 정의를 썼고 npz 메타에 기록됩니다.
+  두 가지 데이터 특성 대응: episode 처음/끝 5 frame의 minima 제외(정지 시작/종료), 그리고 **prominence < 1e-6 m/frame 제외** — VLABench 데모는
+  플래너가 만든 구간별 등속 궤적이라 평탄 구간의 float 잡음이 후보의 절반 이상을 차지해 이 하한 없이는 δ_T가 0이 됩니다.
+  결과 `vlabench_cps_speedmin_d10_r5.npz`: 5,000 ep, 14,466 CP, **2.89 ± 1.38 /ep** (PELT 3.57), CP 0개 episode 없음; task별 통계와
+  PELT와의 ±3 frame 일치율(5–54%)은 `.report.json`. 방향 전환은 속도에 안 나타나므로 PELT보다 CP가 적은 게 이 소스의 특성입니다.
+  inference는 필요 없습니다: PACE가 예측 chunk를 쓰는 건 test time에 데모가 없어서이고, label은 데모 action에서 직접 뽑는 것이 PELT와 같은 조건입니다.
+- A6는 `RacePI0._jitter_b`가 `jitter_prob<=0`이면 항등이라 config 고정만으로 정확히 "jitter 없음"이 됩니다.
+- A7은 `forward`만 교체: aux pass → head → bce, base_fm은 full 그대로이고 FM pass의 field만 `α(τ)·b̂.detach()·M(e_trans)`.
+  detach라서 head는 bce gradient만 받고(A2 fmonly와 달리 FM gradient는 head로 안 감), field(α·e_trans·field_mod)와 expert는 test time과 같은
+  "예측된 b̂" 분포 아래서 학습됩니다 — teacher forcing의 exposure bias만 분리합니다. jitter는 b\* 교란이라 여기선 없습니다.
+- A8/A9는 `_build_head`/`_timing_logits`만 교체 (`race2_ablation/heads.py`). 기준 head: `q=W_q·h_expert → +CrossAttn(q, V_cache) → +SelfAttn → Linear`.
+  A8 `Race2VlmOnlyHead`: `h_expert` 대신 **H개 학습 slot token**을 query로 (v1 `VlmOnlyHead`와 같은 방식 + v2 self-attn). **aux pass는 유지**해서 base_fm 등
+  다른 항은 full과 동일하고 head 입력만 바뀝니다 (구조상 bce gradient가 expert로 안 가는 것만 부수 효과).
+  A9 `Race2ActionOnlyHead`: cross-attention 블록 제거, `q → SelfAttn → Linear`만.
+- A1~A3 preset은 `pi05_vlabench_race2_h20`과 model class·`lambda_base_fm=0`(A2는 `lambda_timing=0`, `jitter_prob=0`도)·데이터 repo id만, A4·A5는 `cps_path`·데이터 repo id만, A6~A9는 model class(A6·A7은 `jitter_prob=0`도)·데이터 repo id만 다릅니다 (loss 가중치는 full과 동일) (teacher, LR schedule, EMA 0.999, λ_timing 0.05, n_tau 10, jitter 0.5, seed 42, 40k step, global batch 64 동일).
+  데이터 repo는 `VLABench/vlabench_primitive_ft_lerobot_224` 대신 같은 내용의 mirror `SeonghoonYu/vlabench_primitive_ft_lerobot_224`
+  (rev `5772f03c3af8df38ee75d8f71cab0075563ed12a`)를 가리킵니다.
+
+## 2. 가져오기
+
+`runtime/openpi/`는 git으로 추적됩니다 (`checkpoints/`, `.venv/`, `.serve/`, `.eval_compile_cache/`, `*.lock`, `runtime/sim-env/`만 제외;
+학습 로그 `logs/`와 평가 결과 `evaluate_results/`는 버전 관리됨). `git pull` 하면 아래 파일이 그대로 들어옵니다.
+같은 코드가 `third_party/openpi_race/`(Libero fork 없는 standalone)와 `race_eval/code/`(overlay)에도 동기화돼 있습니다.
+
+새 파일 (openpi 루트 기준):
+```
+src/openpi/models_pytorch/race2_ablation/__init__.py
+src/openpi/models_pytorch/race2_ablation/model.py          Race2TimingOnlyPI0, Race2NoAuxPI0, Race2FmOnlyPI0, Race2NoJitterPI0, Race2NoTeacherForcingPI0, Race2VlmHeadPI0, Race2ActionHeadPI0
+src/openpi/models_pytorch/race2_ablation/heads.py          Race2VlmOnlyHead, Race2ActionOnlyHead (A8/A9)
+src/openpi/models/race2_ablation_config.py                 위 클래스별 config 7개
+scripts/race2_ablation/train_race2abl_timing_only_vlabench.py   A1 trainer (A1 preset만 받음)
+scripts/race2_ablation/train_race2abl_noaux_vlabench.py         A3 trainer (A3 preset만 받음)
+scripts/race2_ablation/train_race2abl_fmonly_vlabench.py        A2 trainer (A2 preset만 받음)
+scripts/race2_ablation/train_race2abl_{nojitter,notf,vlmhead,actionhead}_vlabench.py   A6~A9 trainer (각자 preset만 받음)
+scripts/race2_ablation/train_race2abl_randcp_vlabench.py        A4 trainer (full trainer 복사본 + 랜덤 label 파일 검사)
+scripts/race2_ablation/make_random_changepoints.py              A4 label 생성기
+assets/gauss_timing/vlabench_cps_random_s2026.npz (+ .report.json)   A4 label (git 포함)
+scripts/race2_ablation/train_race2abl_speedcp_vlabench.py       A5 trainer (full trainer 복사본 + speed-minima label 파일 검사)
+scripts/race2_ablation/make_speed_minima_changepoints.py        A5 label 생성기 (PACE-style)
+assets/gauss_timing/vlabench_cps_speedmin_d10_r5.npz (+ .report.json)   A5 label (git 포함)
+scripts/race2_ablation/smoke_race2_ablation_vlabench.py         smoke test
+scripts/race2_ablation/env.sh                                   경로 설정 (파일 위치 기준, 변수 override 가능)
+scripts/race2_ablation/README.md
+```
+수정된 기존 파일 (method/horizon 이름 배관만):
+```
+src/openpi/training/config.py            끝부분 "# RACE2 ablations" 블록 (preset 54개, H10~H35)
+scripts/vlabench/run_h25_job.py          TRAINERS에 아홉 method 추가, RACE2_LIKE, --horizon 20 허용
+scripts/vlabench/eval_h25.py             --method / --horizon 20 choices
+scripts/vlabench/eval_pool.py            --method / --horizon 20 choices
+scripts/vlabench/checkpoint_validation.py  timing_only EMA는 field tensor가 없어야 통과
+```
+
+## 3. 준비물 체크리스트
+
+경로는 전부 `scripts/race2_ablation/env.sh`가 **파일 위치 기준으로** 계산합니다 (`OPENPI_ROOT` = 이 폴더의 두 단계 위,
+`VLABENCH_WORKSPACE` = 그 두 단계 위 = repo 루트). 워크스페이스 이름·위치가 달라도 그대로 동작하고, 데이터가 다른 곳에
+있으면 source 전에 해당 변수만 export 하면 됩니다. source 시 없는 항목을 `MISSING`으로 알려줍니다.
+
+| 항목 | 위치 | 없을 때 |
+|---|---|---|
+| 정책 venv | `<openpi_root>/.venv` (Python 3.11, torch 2.7.1+cu126, transformers 4.53.2 + `transformers_replace`) | 아래 (a) |
+| Teacher | `<openpi_root>/checkpoints/pi05_vlabench_official/pytorch/{model.safetensors,config.json,assets/}` — sha256 `f6762a23…` | 아래 (b): 로컬 사본 symlink 또는 HF `SeonghoonYu/RACE2_vlabench/teacher_official` |
+| 데이터셋 | `$HF_LEROBOT_HOME/SeonghoonYu/vlabench_primitive_ft_lerobot_224` (rev `5772f03c…`, 5,000 ep) | `~/.cache/huggingface/lerobot`에 이미 있으면 `export HF_HOME=$HOME/.cache/huggingface` 후 source; 없으면 HF에서 받기 |
+| Changepoints | `<openpi_root>/assets/gauss_timing/vlabench_cps.npz` (git 포함, sha256 `6e31b274…`) | — |
+| 시뮬 env | `<workspace>/runtime/sim-env` (Python 3.10, MuJoCo 3.2.2) + `VLABench/assets` | 평가 단계에만 필요; `third_party/openpi_race/setup/setup_sim_env.sh` 참고 |
+| driver 검증 기록 | `<workspace>/race_eval/logs/h30_h35_setup/{dataset_validation,teacher_validation,cache_ready}.json` (git 포함) | 다른 경로면 `--setup-root` |
+
+```bash
+cd <openpi_root>
+# (a) 정책 venv: pinned lock -> editable openpi + openpi-client -> transformers_replace
+uv venv --python 3.11.13 .venv
+uv pip install --python .venv/bin/python -r ../../third_party/openpi_race/setup/policy_requirements.lock.txt
+uv pip install --python .venv/bin/python --no-deps -e . -e packages/openpi-client
+cp -r src/openpi/models_pytorch/transformers_replace/* .venv/lib/python3.11/site-packages/transformers/
+.venv/bin/python -c "from transformers.models.siglip import check; assert check.check_whether_transformers_replace_is_installed_correctly(); print('ok')"
+
+# (b) teacher: 이미 받은 사본이 있으면 링크, 없으면 HF에서
+mkdir -p checkpoints && ln -s <path-to>/pi05_vlabench_official checkpoints/pi05_vlabench_official
+#   또는: .venv/bin/python -c "from huggingface_hub import snapshot_download as d; d('SeonghoonYu/RACE2_vlabench', allow_patterns='teacher_official/*', local_dir='checkpoints/_hf')" \
+#         && mkdir -p checkpoints/pi05_vlabench_official && ln -s ../_hf/teacher_official checkpoints/pi05_vlabench_official/pytorch
+
+# (c) 환경 (데이터가 다른 곳이면 그 변수를 먼저 export)
+source scripts/race2_ablation/env.sh        # MISSING 이 하나도 안 찍혀야 함
+```
+
+## 4. 실행 (H20)
+
+기준 구성은 **GPU 2장 × micro 32 = global batch 64, accumulation 없음** 입니다 (full RACE2 H25~H35와 동일 조건).
+driver `run_h25_job.py`는 `--gpus`에 1·2·4장을 받으며 global batch 64를 유지합니다 (4장이면 `--micro-batch-size 16`, 1장이면 32×accum 2).
+같은 global batch라도 GPU 수가 다르면 microbatch 단위 augmentation 난수가 달라 수치적으로 동일한 실행은 아니니, 비교군끼리는 같은 구성을 쓰세요.
+GPU가 4장이면 2장씩 두 ablation을 동시에 돌리는 것을 권합니다. 속도는 GPU·interconnect에 따라 크게 달라서 여기 적지 않습니다 —
+첫 100 step의 `t=`(초/100 step) 로그로 그 머신의 예상 시간을 계산하세요.
+
+```bash
+cd <openpi_root> && source scripts/race2_ablation/env.sh
+
+# (0) preset 로드 — GPU 불필요
+.venv/bin/python -c "import openpi.training.config as c; [c.get_config(f'pi05_vlabench_race2abl_{k}_h20') for k in ('timing_only','noaux','fmonly','randcp','speedcp','nojitter','notf','vlmhead','actionhead')]; print('presets ok')"
+
+# (1) smoke — 실제 데이터 배치로 loss 구성·gradient 경로·serving parity 검사. GPU 1장, eager 수 분
+TORCHDYNAMO_DISABLE=1 CUDA_VISIBLE_DEVICES=0 .venv/bin/python scripts/race2_ablation/smoke_race2_ablation_vlabench.py pi05_vlabench_race2abl_timing_only_h20 bfloat16
+TORCHDYNAMO_DISABLE=1 CUDA_VISIBLE_DEVICES=1 .venv/bin/python scripts/race2_ablation/smoke_race2_ablation_vlabench.py pi05_vlabench_race2abl_noaux_h20 bfloat16
+TORCHDYNAMO_DISABLE=1 CUDA_VISIBLE_DEVICES=2 .venv/bin/python scripts/race2_ablation/smoke_race2_ablation_vlabench.py pi05_vlabench_race2abl_fmonly_h20 bfloat16
+#   기대: SMOKE_OK. A1 "conditioning OFF", field grad 0, base_fm 0. A3 "conditioning ON", field grad > 0, parity vs reference 0.
+#   A2 loss == fm, step 0 head grad 0 -> field_mod 한 step 뒤 head/e_trans/tau_gate grad > 0.
+TORCHDYNAMO_DISABLE=1 CUDA_VISIBLE_DEVICES=3 .venv/bin/python scripts/race2/smoke_race2_vlabench.py pi05_vlabench_race2abl_randcp_h20 bfloat16
+#   A4는 full 모델이므로 full smoke 스크립트를 그대로 씀. 기대: SMOKE_OK, loss = fm + 0.1·base_fm + 0.05·bce.
+TORCHDYNAMO_DISABLE=1 CUDA_VISIBLE_DEVICES=3 .venv/bin/python scripts/race2/smoke_race2_vlabench.py pi05_vlabench_race2abl_speedcp_h20 bfloat16
+#   A5도 full 모델 → full smoke 스크립트. 기대: SMOKE_OK.
+for k in nojitter notf vlmhead actionhead; do TORCHDYNAMO_DISABLE=1 CUDA_VISIBLE_DEVICES=0 .venv/bin/python scripts/race2_ablation/smoke_race2_ablation_vlabench.py pi05_vlabench_race2abl_${k}_h20 bfloat16; done
+#   A6~A9: 기대 SMOKE_OK, loss = fm + 0.1·base_fm + 0.05·bce, head grad > 0, field grad > 0. A9는 head에 attn(cross) 블록이 없어야 함.
+
+# (2) dry-run — 커맨드/경로/검증기록만 출력, 실행 없음
+.venv/bin/python scripts/vlabench/run_h25_job.py --method race2abl_timing_only --horizon 20 --gpus 0,1 --exp-name abl_timing_only_h20_s42 --micro-batch-size 32 --dry-run
+
+# (3) 본 실행 — 학습 40k -> 최종 EMA 검증 -> track 1/2/3/4/6 pooled 평가(n=50)까지 자동. 각각 별도 persistent 프로세스.
+mkdir -p logs
+nohup .venv/bin/python scripts/vlabench/run_h25_job.py --method race2abl_timing_only --horizon 20 --gpus 0,1 --exp-name abl_timing_only_h20_s42 --micro-batch-size 32 > logs/abl_timing_only_h20.driver.log 2>&1 &
+nohup .venv/bin/python scripts/vlabench/run_h25_job.py --method race2abl_noaux       --horizon 20 --gpus 2,3 --exp-name abl_noaux_h20_s42       --micro-batch-size 32 > logs/abl_noaux_h20.driver.log 2>&1 &
+# 위 둘이 끝난 뒤 (같은 GPU 쌍으로):
+nohup .venv/bin/python scripts/vlabench/run_h25_job.py --method race2abl_fmonly      --horizon 20 --gpus 0,1 --exp-name abl_fmonly_h20_s42      --micro-batch-size 32 > logs/abl_fmonly_h20.driver.log 2>&1 &
+nohup .venv/bin/python scripts/vlabench/run_h25_job.py --method race2abl_randcp      --horizon 20 --gpus 2,3 --exp-name abl_randcp_h20_s42      --micro-batch-size 32 > logs/abl_randcp_h20.driver.log 2>&1 &
+nohup .venv/bin/python scripts/vlabench/run_h25_job.py --method race2abl_speedcp     --horizon 20 --gpus 0,1 --exp-name abl_speedcp_h20_s42     --micro-batch-size 32 > logs/abl_speedcp_h20.driver.log 2>&1 &
+# A6~A9 (method: race2abl_nojitter / race2abl_notf / race2abl_vlmhead / race2abl_actionhead), 같은 형식:
+nohup .venv/bin/python scripts/vlabench/run_h25_job.py --method race2abl_notf        --horizon 20 --gpus 2,3 --exp-name abl_notf_h20_s42        --micro-batch-size 32 > logs/abl_notf_h20.driver.log 2>&1 &
+```
+- 중단 시 같은 커맨드에 `--resume` (최신 `train_state.pt`에서 이어감; 데이터 순서/RNG는 bit-identical 아님).
+- driver 없이 trainer만: `torchrun --standalone --nnodes=1 --nproc_per_node=2 scripts/race2_ablation/train_race2abl_<id>_vlabench.py pi05_vlabench_race2abl_<id>_h20 --exp_name <exp> --model.micro-batch-size 32`
+- GPU 메모리: 2 × 32 구성에서 GPU당 약 31 GB (40 GB급 이상 필요).
+
+## 5. 산출물
+
+| | 경로 (openpi 루트 기준) |
+|---|---|
+| 체크포인트 | `checkpoints/pi05_vlabench_race2abl_<id>_h20/<exp>/<step>/` — `ema.safetensors`(1k마다, 20k부터), `raw.safetensors`(5k마다), `train_state.pt`(최신만) |
+| 학습 로그·상태 | `logs/h20/race2abl_<id>_<exp>/{train.log,status.json,history.jsonl,commands.json}` |
+| 평가 결과 | `evaluate_results/vlabench_h20/race2abl_<id>_h20_<exp>_40000_ema_replan20_n50_seed7/summary.json` (+ `spec.json`, `pool_manifest.json`, track별 shard) |
+
+학습 로그의 `fm`, `bce`, `pres_acc`, `peak_pm1`이 주요 지표입니다. A1은 `bfm`(base_fm)이 항상 0이고 `alpha_*`/`dmod`가 없으며,
+A3는 `bfm`만 0, A2는 `loss == fm`이고 `bfm`/`bce`/`pres_acc`는 모니터링 값이며 `b_hat_mean`이 추가로 찍힙니다. A4는 full과 같은 로그이며 시작 시 `RANDOM transition labels: …`, A5는 `SPEED-MINIMA transition labels: …` 한 줄이 찍힙니다. A6~A9는 full과 같은 로그(A7은 `b_hat_mean` 추가)이며 시작 시 `RACE2 ablation model: <class> | conditioning ON …`이 찍힙니다.
+실험이 끝나면 `.venv/bin/python scripts/vlabench/compact_train_log.py logs/h20/<run> --gz` 로 `train_metrics.log`(metric 줄만, ~120 KB)와 `train.log.gz`를 만든 뒤
+`git add runtime/openpi/logs runtime/openpi/evaluate_results && git commit && git push` 로 올립니다. 원본 `train.log`(tqdm 진행바로 ~30 MB)는 git-ignore됩니다.
+비교 대상 full RACE2 H20 / PlainFT H20은 `race_eval/tables/`(40k, 5 track)와 HF `SeonghoonYu/RACE2_vlabench/{h20_40k,plainft_h20_40k}`에 있습니다.
+
+## 6. 검증된 범위 (2026-09-14, 원 개발 머신)
+
+- smoke A1/A3(H30 preset), A2(H20 preset) 모두 `SMOKE_OK`: loss 구성이 정의와 정확히 일치, gradient 경로(A1 field 없음·expert 흐름, A3 field grad > 0, A2 field_mod 한 step 후 head grad > 0), serving determinism 0
+- A4: 랜덤 label 파일 생성·검증(개수 일치, 우연 수준 겹침), full smoke `SMOKE_OK`, trainer가 진짜 label 파일을 거부하는 것 확인
+- A5: label 파일 생성·통계, 데이터 파이프라인 로드·b\* 생성, trainer가 PELT/랜덤 파일을 거부하는 것, driver `--dry-run` 확인 (GPU smoke는 GPU 점유 중이라 미실행 — 다른 서버에서 (1)번으로)
+- A6~A9: preset이 full과 의도한 필드만 다름을 확인, driver `--dry-run` 통과, CPU에서 random-init forward/backward로 loss 구성·gradient 경로·head 구조 확인 (GPU smoke는 다른 서버에서 (1)번으로)
+- driver `--dry-run` 네 method 통과, 기존 unit test 45개 통과
+- 40k 본 학습·평가는 아직 어느 머신에서도 돌리지 않았습니다

ablation/abl_nojitter_h20_40k/source/scripts/race2_ablation/train_race2abl_nojitter_vlabench.py

@@ -0,0 +1,447 @@
+"""RACE2 ablation trainer: A6 — full RACE2 with the field jitter disabled (jitter_prob = 0).
+
+Runs ONLY pi05_vlabench_race2abl_nojitter_h{H} presets (model Race2NoJitterPI0). Direct use:
+
+  cd /root/vlabench/runtime/openpi && source /root/vlabench/race_eval/logs/h30_h35_setup/env.sh
+  .venv/bin/torchrun --standalone --nnodes=1 --nproc_per_node=2 \
+      scripts/race2_ablation/train_race2abl_nojitter_vlabench.py pi05_vlabench_race2abl_nojitter_h30 \
+      --exp_name nojitter_h30_s42 --model.micro-batch-size 32 [--resume]
+
+or through the driver (train -> final-EMA check -> pooled eval):
+  .venv/bin/python scripts/vlabench/run_h25_job.py --method race2abl_nojitter --horizon 30 --gpus 0,1 \
+      --exp-name nojitter_h30_s42 --micro-batch-size 32
+
+
+Copy of scripts/race2/train_race2_vlabench.py (same recipe machinery: fp32 master
+weights, EMA, cosine LR, slim overlay checkpoints) with the model checks relaxed to the
+ablation classes in openpi.models_pytorch.race2_ablation:
+  Race2TimingOnlyPI0  pi05_vlabench_race2abl_timing_only_h{H}  timing loss only, no field
+  Race2NoJitterPI0       pi05_vlabench_race2abl_nojitter_h{H}        full minus auxiliary loss
+The full-method trainer is left untouched.
+
+Original header follows.
+--------------------------------------------------------------------------------------
+Training entrypoint for RACE v2 on VLABench (Race2PI0).
+
+Standalone counterpart of scripts/race/train_race.py for the v2 timing head
+(cross-attention to the cached VLM values -> self-attention among the action slots ->
+linear -> sigmoid; openpi.models_pytorch.race2). Same data pipeline, objective and
+VLABench recipe machinery as the v1 trainer; kept as a separate file so the two methods
+never share (or break) each other's code.
+
+Recipe machinery (added for the VLABench campaign, usable for any RACE config):
+  * fp32 MASTER WEIGHTS for the trainable set. The model itself stays bf16 for compute,
+    exactly as before, but AdamW steps fp32 copies that are written back after every
+    step. Without this, the cosine tail (lr 5e-6 -> ~2.5e-4 relative updates) is rounded
+    away by the bf16 parameter spacing (2^-8).
+  * EMA of the master weights (config.ema_decay). The decay is warmed up as
+    min(decay, (1 + step) / (10 + step)) so the early 1k checkpoints are not dominated
+    by the initial weights.
+  * warmup + cosine LR straight from config.lr_schedule (CosineDecaySchedule).
+  * SLIM CHECKPOINTS. Only the trainable tensors are written (the VLM prefix is frozen
+    and identical to the teacher):
+        <step>/ema.safetensors    EMA weights, model dtype  (~1.6 GB)   every save
+        <step>/raw.safetensors    non-EMA weights            (~1.6 GB)   every RAW_EVERY-th save
+        <step>/train_state.pt     fp32 master + EMA + optimizer (~13 GB) newest step only
+        <step>/metadata.pt, <step>/assets/<asset_id>/norm_stats.json
+    A full serving checkpoint (teacher VLM + overlay) is materialised on demand with
+    scripts/race/materialize_ckpt.py (config-driven, shared with v1).
+
+Usage:
+  torchrun --standalone --nnodes=1 --nproc_per_node=<n> \
+      scripts/race2_ablation/train_race2abl_nojitter_vlabench.py pi05_vlabench_race2abl_nojitter_h30 --exp_name nojitter_h30_s42 [--resume]
+"""
+
+import contextlib
+import dataclasses
+import logging
+import os
+import pathlib
+import shutil
+import sys
+import time
+
+sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent.parent))  # scripts/ for train_pytorch
+
+import numpy as np
+import safetensors.torch
+import torch
+import torch.distributed as dist
+import tqdm
+
+import jax
+
+import openpi.models.race2_ablation_config as race2_ablation_config  # noqa: F401  (config registry)
+import openpi.models.model as _model
+import openpi.shared.normalize as _normalize
+import openpi.training.config as _config
+import openpi.training.data_loader as _data
+import openpi.training.gauss_timing_data as gauss_data
+
+import train_pytorch as base
+from vlabench.checkpoint_validation import validate_resume
+
+# module name fragments that are NEW in this model (absent from the teacher ckpt)
+FRESH_OK = ("race_head", "e_trans", "field_mod", "tau_gate")
+ABLATION_MODELS = ("Race2NoJitterPI0",)
+# a raw (non-EMA) overlay is written on every RAW_EVERY-th save (5 x 1k = every 5k steps)
+RAW_EVERY = 5
+TRAIN_STATE = "train_state.pt"
+
+
+def load_teacher(model, config):
+    model_path = os.path.join(config.pytorch_weight_path, "model.safetensors")
+    logging.info(f"Loading teacher weights from: {model_path}")
+    missing, unexpected = safetensors.torch.load_model(model, model_path, strict=False)
+    if unexpected:
+        raise ValueError(f"Unexpected keys in teacher checkpoint: {sorted(unexpected)[:5]}")
+    stale = [k for k in missing if not any(m in k for m in FRESH_OK)]
+    if stale:
+        raise ValueError(f"Teacher checkpoint missing expected policy keys: {sorted(stale)[:5]}")
+    logging.info(f"Loaded teacher; {len(missing)} new-module tensors keep fresh init")
+
+
+def make_loader(config, world_size):
+    """(observation, actions, b) triples; mirrors data_loader.create_torch_data_loader
+    but keeps the extra 'b' key instead of dropping it at Observation.from_dict."""
+    data_config = config.data.create(config.assets_dirs, config.model)
+    dataset = _data.create_torch_dataset(data_config, config.model.action_horizon, config.model)
+    dataset = _data.transform_dataset(dataset, data_config)
+    sampler = None
+    local_bs = config.batch_size
+    if dist.is_initialized():
+        sampler = torch.utils.data.distributed.DistributedSampler(
+            dataset, num_replicas=dist.get_world_size(), rank=dist.get_rank(), shuffle=True, drop_last=True
+        )
+        local_bs = config.batch_size // world_size
+    loader = _data.TorchDataLoader(
+        dataset,
+        local_batch_size=local_bs,
+        sharding=None,
+        shuffle=(sampler is None),
+        sampler=sampler,
+        num_batches=None,
+        num_workers=config.num_workers,
+        seed=config.seed,
+        framework="pytorch",
+    )
+    return loader, sampler, data_config
+
+
+class MasterState:
+    """fp32 master copies + EMA of the trainable parameters.
+
+    Every rank holds an identical copy: DDP averages the bf16 grads, so identical AdamW
+    steps run on identical fp32 masters. `sync_to_model` writes the masters back into the
+    live (bf16 / fp32) parameters after each optimizer step."""
+
+    def __init__(self, named_params, ema_decay: float | None, ema_warmup: bool = True):
+        self.names = [n for n, _ in named_params]
+        self.params = [p for _, p in named_params]
+        self.master = []
+        for p in self.params:
+            m = p.detach().clone().float().requires_grad_(True)
+            m.grad = torch.zeros_like(m)
+            self.master.append(m)
+        self.ema = [m.detach().clone() for m in self.master]
+        self.decay = 0.0 if ema_decay is None else float(ema_decay)
+        self.ema_warmup = ema_warmup
+
+    def grads_to_master(self):
+        for p, m in zip(self.params, self.master):
+            if p.grad is None:
+                m.grad.zero_()
+            else:
+                m.grad.copy_(p.grad)
+
+    @torch.no_grad()
+    def sync_to_model(self):
+        for p, m in zip(self.params, self.master):
+            p.copy_(m)
+
+    @torch.no_grad()
+    def update_ema(self, step: int) -> float:
+        d = self.decay
+        if self.ema_warmup:
+            d = min(d, (1.0 + step) / (10.0 + step))
+        torch._foreach_lerp_(self.ema, self.master, 1.0 - d)
+        return d
+
+    @torch.no_grad()
+    def overlay(self, tensors) -> dict[str, torch.Tensor]:
+        """Trainable tensors in the model's own dtypes (bf16 expert, fp32 norms/head), on CPU."""
+        return {n: t.detach().to(p.dtype).cpu().contiguous() for n, t, p in zip(self.names, tensors, self.params)}
+
+    @torch.no_grad()
+    def state_dict(self):
+        return {
+            "names": list(self.names),
+            "master": [m.detach().cpu() for m in self.master],
+            "ema": [e.detach().cpu() for e in self.ema],
+        }
+
+    @torch.no_grad()
+    def load_state_dict(self, sd):
+        if list(sd["names"]) != self.names:
+            raise ValueError("train_state parameter set differs from the model's trainable set")
+        for m, v in zip(self.master, sd["master"]):
+            m.copy_(v)
+        for e, v in zip(self.ema, sd["ema"]):
+            e.copy_(v)
+
+
+def latest_resumable_step(ckpt_dir: pathlib.Path):
+    if not ckpt_dir.exists():
+        return None
+    steps = [
+        int(d.name) for d in ckpt_dir.iterdir()
+        if d.is_dir() and d.name.isdigit() and (d / TRAIN_STATE).is_file()
+    ]
+    return max(steps) if steps else None
+
+
+def save_checkpoint(config, step, ms: MasterState, opt, data_config, is_main):
+    if not is_main or step == 0 or step % config.save_interval != 0:
+        return
+    final = config.checkpoint_dir / str(step)
+    tmp = config.checkpoint_dir / f"tmp_{step}"
+    shutil.rmtree(tmp, ignore_errors=True)
+    tmp.mkdir(parents=True)
+    t0 = time.time()
+    # evaluation weights only from save_overlays_from_step on; the resume state is always written
+    overlays = step >= int(getattr(config, "save_overlays_from_step", 0) or 0)
+    if overlays:
+        safetensors.torch.save_file(ms.overlay(ms.ema), str(tmp / "ema.safetensors"))
+        if (step // config.save_interval) % RAW_EVERY == 0 or step == config.num_train_steps:
+            safetensors.torch.save_file(ms.overlay(ms.master), str(tmp / "raw.safetensors"))
+        norm_stats = data_config.norm_stats
+        if norm_stats is not None and data_config.asset_id is not None:
+            _normalize.save(tmp / "assets" / data_config.asset_id, norm_stats)
+    torch.save(
+        {
+            "global_step": step,
+            "config": dataclasses.asdict(config),
+            "timestamp": time.time(),
+            "ema_decay": ms.decay,
+            "format": "trainable-only overlays; materialise with scripts/race/materialize_ckpt.py",
+        },
+        tmp / "metadata.pt",
+    )
+    torch.save({"step": step, **ms.state_dict(), "optimizer": opt.state_dict()}, tmp / TRAIN_STATE)
+    if final.exists():
+        shutil.rmtree(final)
+    tmp.rename(final)
+    logging.info(f"Saved checkpoint {final} ({'overlays + ' if overlays else ''}resume state, {time.time() - t0:.0f}s)")
+    # keep the full resume state (fp32 master + EMA + optimizer, ~13 GB) for the newest step only;
+    # a stale step directory that holds nothing but its resume state is removed altogether
+    for d in config.checkpoint_dir.iterdir():
+        if d.is_dir() and d.name.isdigit() and int(d.name) < step and (d / TRAIN_STATE).is_file():
+            (d / TRAIN_STATE).unlink()
+            if not (d / "ema.safetensors").is_file() and not (d / "raw.safetensors").is_file():
+                shutil.rmtree(d)
+                logging.info(f"Pruned resume-only checkpoint dir: {d}")
+            else:
+                logging.info(f"Pruned stale train state: {d / TRAIN_STATE}")
+
+
+def train_loop(config: _config.TrainConfig):
+    use_ddp, local_rank, device = base.setup_ddp()
+    is_main = (not use_ddp) or (dist.get_rank() == 0)
+    base.set_seed(config.seed, local_rank)
+    world_size = dist.get_world_size() if use_ddp else 1
+    logging.info(f"Batch size per GPU: {config.batch_size // world_size} (total: {config.batch_size})")
+
+    data_loader, sampler, data_config = make_loader(config, world_size)
+
+    model_cfg = config.model
+    object.__setattr__(model_cfg, "dtype", config.pytorch_training_precision or "float32")
+    model = model_cfg.build_model().to(device)
+    if type(model).__name__ not in ABLATION_MODELS:
+        raise ValueError(f"config built {type(model).__name__}, expected one of {ABLATION_MODELS}")
+    logging.info(f"RACE2 ablation model: {type(model).__name__} | conditioning "
+                 f"{'ON' if hasattr(model, 'field_mod') else 'OFF'} | lambda_base_fm {model_cfg.lambda_base_fm} "
+                 f"| lambda_timing {model_cfg.lambda_timing}")
+    logging.info(f"RACE v2 head: {type(model.race_head).__name__} "
+                 f"({sum(p.numel() for p in model.race_head.parameters()):,} params), kv_layer {model.kv_layer}")
+
+    resume_step = latest_resumable_step(config.checkpoint_dir) if config.resume else None
+    if resume_step is not None:
+        validate_resume(config, config.checkpoint_dir / str(resume_step), resume_step)
+    load_teacher(model, config)  # frozen VLM prefix always comes from the teacher
+    n_train, n_frozen = model.freeze_vlm()
+    logging.info(f"trainable {n_train/1e6:.1f}M / frozen {n_frozen/1e6:.0f}M (VLM prefix)")
+    # bias-free guarantee for the b=0 => zero-modulation claim (conditioned ablations only)
+    for lin in getattr(model, "field_mod", []):
+        assert lin.bias is None, "field_mod projections must be bias-free"
+    if not hasattr(model, "field_mod"):
+        assert not hasattr(model, "e_trans") and not hasattr(model, "tau_gate"), "timing-only model must carry no field tensors"
+    model.gradient_checkpointing_enable()
+
+    raw = model
+    if use_ddp:
+        model = torch.nn.parallel.DistributedDataParallel(
+            model, device_ids=[device.index] if device.type == "cuda" else None,
+            find_unused_parameters=True, gradient_as_bucket_view=True,
+        )
+
+    ms = MasterState([(n, p) for n, p in raw.named_parameters() if p.requires_grad], config.ema_decay)
+    gate_mult = float(getattr(model_cfg, "tau_gate_lr_mult", 1.0))
+    base_m = [m for n, m in zip(ms.names, ms.master) if "tau_gate" not in n]
+    gate_m = [m for n, m in zip(ms.names, ms.master) if "tau_gate" in n]
+    opt = torch.optim.AdamW(
+        [{"params": base_m, "lr_scale": 1.0},
+         {"params": gate_m, "lr_scale": gate_mult}],
+        lr=config.lr_schedule.peak_lr, betas=(config.optimizer.b1, config.optimizer.b2),
+        eps=config.optimizer.eps, weight_decay=config.optimizer.weight_decay,
+    )
+    if gate_m:
+        logging.info(f"tau_gate param group: {sum(p.numel() for p in gate_m)} scalars, lr x{gate_mult}")
+    logging.info(f"fp32 master weights + EMA(decay={ms.decay}, warmup) over {len(ms.master)} tensors")
+
+    warmup, peak, decay, end = (config.lr_schedule.warmup_steps, config.lr_schedule.peak_lr,
+                                config.lr_schedule.decay_steps, config.lr_schedule.decay_lr)
+
+    def lr_at(step):
+        if step < warmup:
+            return peak * (step + 1) / max(1, warmup)
+        t = min(1.0, (step - warmup) / max(1, decay))
+        return end + 0.5 * (peak - end) * (1 + np.cos(np.pi * t))
+
+    logging.info(f"lr: warmup {warmup} -> peak {peak:.1e} -> cosine over {decay} -> {end:.1e}")
+
+    global_step = 0
+    if resume_step is not None:
+        path = config.checkpoint_dir / str(resume_step) / TRAIN_STATE
+        logging.info(f"Resuming from {path}")
+        sd = torch.load(path, map_location="cpu", weights_only=False)
+        ms.load_state_dict(sd)
+        opt.load_state_dict(sd["optimizer"])
+        ms.sync_to_model()
+        global_step = int(sd["step"])
+        if global_step != resume_step:
+            raise ValueError(f"Checkpoint directory step {resume_step} differs from saved step {global_step}")
+        del sd
+        logging.info(f"Resumed at step {global_step}")
+    elif config.resume:
+        logging.info("--resume given but no resumable checkpoint found; starting from the teacher")
+    raw._step = global_step
+
+    per_gpu = config.batch_size // world_size
+    micro = getattr(model_cfg, "micro_batch_size", 0) or per_gpu
+    if per_gpu % micro != 0:
+        raise ValueError(f"per-GPU batch {per_gpu} not divisible by micro_batch_size {micro}")
+    accum = per_gpu // micro
+    if accum > 1:
+        logging.info(f"grad accumulation: micro {micro} x {accum} = per-GPU {per_gpu}")
+
+    infos = []
+    pbar = tqdm.tqdm(total=config.num_train_steps, initial=global_step, desc="Training",
+                     disable=not is_main, dynamic_ncols=True)
+    start = time.time()
+    epoch = global_step  # distinct sampler seeds across resumes
+    while global_step < config.num_train_steps:
+        # iterate the underlying torch loader per epoch so DistributedSampler
+        # reshuffles (TorchDataLoader's infinite __iter__ never calls set_epoch)
+        if sampler is not None:
+            sampler.set_epoch(epoch)
+        epoch += 1
+        for batch in data_loader.torch_loader:
+            if global_step >= config.num_train_steps:
+                break
+            batch = jax.tree.map(torch.as_tensor, batch)
+            b_star = batch["b"]
+            observation = _model.Observation.from_dict(batch)
+            actions = batch["actions"]
+            observation = jax.tree.map(lambda x: x.to(device), observation)
+            actions = actions.to(torch.float32).to(device)
+            b_star = b_star.to(torch.float32).to(device)
+            lr = lr_at(global_step)
+            for pg in opt.param_groups:
+                pg["lr"] = lr * pg.get("lr_scale", 1.0)
+            model.zero_grad(set_to_none=True)
+            loss_val = 0.0
+            metrics_acc = {}
+            for a in range(accum):
+                sl = slice(a * micro, (a + 1) * micro)
+                obs_m = jax.tree.map(lambda x: x[sl], observation)
+                # skip the DDP all-reduce on all but the last micro-step
+                ctx = model.no_sync() if (use_ddp and a < accum - 1) else contextlib.nullcontext()
+                with ctx:
+                    loss = model(obs_m, actions[sl], b_star=b_star[sl]) / accum
+                    loss.backward()
+                loss_val += loss.item()
+                for k, v in raw.last_metrics.items():
+                    metrics_acc[k] = metrics_acc.get(k, 0.0) + v / accum
+            raw.last_metrics = metrics_acc
+            ms.grads_to_master()
+            gn = torch.nn.utils.clip_grad_norm_(
+                ms.master, max_norm=config.optimizer.clip_gradient_norm, error_if_nonfinite=True
+            )
+            opt.step()
+            ms.sync_to_model()
+            ema_d = ms.update_ema(global_step)
+            if is_main:
+                m = {"loss": loss_val, "grad_norm": float(gn), **raw.last_metrics}
+                infos.append(m)
+                if global_step % config.log_interval == 0:
+                    avg = {k: sum(i.get(k, 0) for i in infos) / len(infos) for k in infos[0]}
+                    extra = " ".join(f"{k}={avg[k]:.4f}" for k in avg if k not in ("loss", "grad_norm"))
+                    mem = torch.cuda.max_memory_allocated(device) / 2**30 if device.type == "cuda" else 0.0
+                    logging.info(f"step={global_step} loss={avg['loss']:.4f} {extra} "
+                                 f"lr={lr:.2e} ema={ema_d:.4f} gn={avg['grad_norm']:.2f} "
+                                 f"mem={mem:.1f}G t={time.time()-start:.1f}s")
+                    infos = []
+                    start = time.time()
+                last = raw.last_metrics
+                pbar.update(1)
+                pbar.set_postfix({
+                    "loss": f"{loss_val:.3f}",
+                    "fm": f"{last.get('fm_loss', float('nan')):.3f}",
+                    "bfm": f"{last.get('base_fm', float('nan')):.3f}",
+                    "bce": f"{last.get('bce', float('nan')):.3f}",
+                    "pm1": f"{last.get('peak_pm1', float('nan')):.2f}",
+                    "pres": f"{last.get('pres_acc', float('nan')):.2f}",
+                    "b+": f"{last.get('b_pos', float('nan')):.2f}",
+                    "b-": f"{last.get('b_neg', float('nan')):.2f}",
+                    "g1": f"{last.get('g_first', float('nan')):.2f}",
+                    "g10": f"{last.get('g_last', float('nan')):.2f}",
+                    "dmod": f"{last.get('dmod', float('nan')):.2f}",
+                    "lr": f"{lr:.1e}",
+                })
+            global_step += 1
+            raw._step = global_step
+            save_checkpoint(config, global_step, ms, opt, data_config, is_main)
+            if use_ddp:
+                dist.barrier()  # rank 0 may have spent a minute writing the checkpoint
+    if is_main:
+        pbar.close()
+
+
+def main():
+    base.init_logging()
+    config = _config.cli()
+    if type(config.model) is not race2_ablation_config.Race2NoJitterConfig:
+        raise ValueError(f"this trainer runs race2abl_nojitter presets only (Race2NoJitterConfig); got {type(config.model).__name__}")
+    if "race2abl_nojitter_h" not in config.name:
+        raise ValueError(f"expected a pi05_vlabench_race2abl_nojitter_h<H> config, got {config.name!r}")
+    if config.resume and latest_resumable_step(config.checkpoint_dir) is None:
+        raise FileNotFoundError(f"No resumable checkpoint in {config.checkpoint_dir}")
+    if not config.resume and config.checkpoint_dir.exists():
+        saved_steps = [p for p in config.checkpoint_dir.iterdir() if p.is_dir() and p.name.isdigit()]
+        if saved_steps:
+            raise FileExistsError(
+                f"Checkpoints already exist in {config.checkpoint_dir}; use --resume or a new --exp_name"
+            )
+    cps = gauss_data.load_changepoints(config.data.cps_path)
+    n_cp = sum(len(v) for v in cps.values())
+    logging.info(f"changepoints: {config.data.cps_path} — {len(cps)} episodes, "
+                 f"{n_cp} CPs ({n_cp / len(cps):.2f}/episode)")
+    try:
+        train_loop(config)
+    finally:
+        if dist.is_initialized():
+            dist.destroy_process_group()
+
+
+if __name__ == "__main__":
+    main()

ablation/abl_nojitter_h20_40k/source/src/openpi/models/race2_ablation_config.py

@@ -0,0 +1,108 @@
+"""Configs for the RACE2 ablations (openpi.models_pytorch.race2_ablation).
+
+Both keep every RacePi0Config knob of the full method; lambda_base_fm is pinned to 0.
+  Race2TimingOnlyConfig -> Race2TimingOnlyPI0 (A1: timing loss only, no conditioning)
+  Race2NoAuxConfig      -> Race2NoAuxPI0      (A3: full method without the auxiliary loss)
+  Race2FmOnlyConfig     -> Race2FmOnlyPI0     (A2: conditioning by predicted b_hat, FM loss only)
+  Race2NoJitterConfig   -> Race2NoJitterPI0   (A6: full method, jitter_prob = 0)
+  Race2NoTFConfig       -> Race2NoTeacherForcingPI0 (A7: full losses, field from detached b_hat)
+  Race2VlmHeadConfig    -> Race2VlmHeadPI0    (A8: head sees VLM features only; aux pass kept)
+  Race2ActionHeadConfig -> Race2ActionHeadPI0 (A9: head sees action features only)
+"""
+
+import dataclasses
+
+from openpi.models import race2_config
+
+
+@dataclasses.dataclass(frozen=True)
+class Race2TimingOnlyConfig(race2_config.Race2Pi0Config):
+    lambda_base_fm: float = 0.0
+
+    def __post_init__(self):
+        super().__post_init__()
+        if float(self.lambda_base_fm) != 0.0:
+            raise ValueError("Race2TimingOnlyConfig: lambda_base_fm must be 0")
+
+    def build_model(self):
+        from openpi.models_pytorch.race2_ablation.model import Race2TimingOnlyPI0
+
+        return Race2TimingOnlyPI0(config=self)
+
+
+@dataclasses.dataclass(frozen=True)
+class Race2NoAuxConfig(race2_config.Race2Pi0Config):
+    lambda_base_fm: float = 0.0
+
+    def __post_init__(self):
+        super().__post_init__()
+        if float(self.lambda_base_fm) != 0.0:
+            raise ValueError("Race2NoAuxConfig: lambda_base_fm must be 0")
+
+    def build_model(self):
+        from openpi.models_pytorch.race2_ablation.model import Race2NoAuxPI0
+
+        return Race2NoAuxPI0(config=self)
+
+
+@dataclasses.dataclass(frozen=True)
+class Race2FmOnlyConfig(race2_config.Race2Pi0Config):
+    lambda_base_fm: float = 0.0
+    lambda_timing: float = 0.0
+    jitter_prob: float = 0.0  # unused: the field comes from b_hat, never from b*
+
+    def __post_init__(self):
+        super().__post_init__()
+        if float(self.lambda_base_fm) != 0.0 or float(self.lambda_timing) != 0.0:
+            raise ValueError("Race2FmOnlyConfig: lambda_base_fm and lambda_timing must be 0")
+
+    def build_model(self):
+        from openpi.models_pytorch.race2_ablation.model import Race2FmOnlyPI0
+
+        return Race2FmOnlyPI0(config=self)
+
+
+@dataclasses.dataclass(frozen=True)
+class Race2NoJitterConfig(race2_config.Race2Pi0Config):
+    jitter_prob: float = 0.0
+
+    def __post_init__(self):
+        super().__post_init__()
+        if float(self.jitter_prob) != 0.0:
+            raise ValueError("Race2NoJitterConfig: jitter_prob must be 0")
+
+    def build_model(self):
+        from openpi.models_pytorch.race2_ablation.model import Race2NoJitterPI0
+
+        return Race2NoJitterPI0(config=self)
+
+
+@dataclasses.dataclass(frozen=True)
+class Race2NoTFConfig(race2_config.Race2Pi0Config):
+    jitter_prob: float = 0.0  # unused: the field comes from the detached b_hat, never from b*
+
+    def __post_init__(self):
+        super().__post_init__()
+        if float(self.jitter_prob) != 0.0:
+            raise ValueError("Race2NoTFConfig: jitter_prob must be 0")
+
+    def build_model(self):
+        from openpi.models_pytorch.race2_ablation.model import Race2NoTeacherForcingPI0
+
+        return Race2NoTeacherForcingPI0(config=self)
+
+
+@dataclasses.dataclass(frozen=True)
+class Race2VlmHeadConfig(race2_config.Race2Pi0Config):
+    def build_model(self):
+        from openpi.models_pytorch.race2_ablation.model import Race2VlmHeadPI0
+
+        return Race2VlmHeadPI0(config=self)
+
+
+@dataclasses.dataclass(frozen=True)
+class Race2ActionHeadConfig(race2_config.Race2Pi0Config):
+    def build_model(self):
+        from openpi.models_pytorch.race2_ablation.model import Race2ActionHeadPI0
+
+        return Race2ActionHeadPI0(config=self)

ablation/abl_nojitter_h20_40k/source/src/openpi/models_pytorch/race2_ablation/__init__.py

@@ -0,0 +1,7 @@
+from openpi.models_pytorch.race2_ablation.model import (
+    Race2ActionHeadPI0, Race2FmOnlyPI0, Race2NoAuxPI0, Race2NoJitterPI0, Race2NoTeacherForcingPI0,
+    Race2TimingOnlyPI0, Race2VlmHeadPI0,
+)
+
+__all__ = ["Race2ActionHeadPI0", "Race2FmOnlyPI0", "Race2NoAuxPI0", "Race2NoJitterPI0", "Race2NoTeacherForcingPI0",
+           "Race2TimingOnlyPI0", "Race2VlmHeadPI0"]

ablation/abl_nojitter_h20_40k/source/src/openpi/models_pytorch/race2_ablation/heads.py

@@ -0,0 +1,61 @@
+"""Timing-head input ablations for RACE2 (openpi.models_pytorch.race2_ablation).
+
+Reference head (openpi.models_pytorch.race2.heads.Race2TimingHead):
+    q = W_q h_expert ; z = q + CrossAttn(q, V_cache) ; z = z + SelfAttn(LN z) ; logit = Linear(z)
+
+Race2VlmOnlyHead    "VLM features only": the auxiliary-pass expert features are NOT given to
+                    the head. H learned per-slot tokens replace h_expert as the queries; the
+                    rest (cross-attention to the cached VLM values, slot self-attention, linear)
+                    is identical to the reference head. Same pattern as the v1 ablation
+                    (openpi.models_pytorch.race.ablations.VlmOnlyHead) plus the v2 self-attention.
+Race2ActionOnlyHead "action features only": no access to the VLM cache. The queries from the
+                    auxiliary-pass expert features go straight to the slot self-attention and
+                    the linear; the cross-attention block is removed.
+
+Submodule names are kept aligned with the reference head where the blocks are the same
+(q_proj, ln, attn, ln_self, self_attn, out) so checkpoint keys stay readable.
+"""
+
+import torch
+from torch import nn
+
+
+class Race2VlmOnlyHead(nn.Module):
+    def __init__(self, expert_dim: int, kv_dim: int, horizon: int, proj_dim: int = 256, num_heads: int = 4):
+        super().__init__()
+        if kv_dim != proj_dim:
+            raise ValueError(f"cached values are {kv_dim}-d, head runs at {proj_dim}-d")
+        self.tokens = nn.Parameter(torch.randn(horizon, expert_dim) * 0.02)  # stands in for h_expert
+        self.q_proj = nn.Linear(expert_dim, proj_dim)
+        self.ln = nn.LayerNorm(proj_dim)
+        self.attn = nn.MultiheadAttention(proj_dim, num_heads, batch_first=True)   # cross-attention to VLM values
+        self.ln_self = nn.LayerNorm(proj_dim)
+        self.self_attn = nn.MultiheadAttention(proj_dim, num_heads, batch_first=True)
+        self.out = nn.Linear(proj_dim, 1)
+
+    def forward(self, prefix_values: torch.Tensor, prefix_pad_masks: torch.Tensor) -> torch.Tensor:
+        bsize = prefix_values.shape[0]
+        q = self.q_proj(self.tokens)[None].expand(bsize, -1, -1)        # (B, H, D) learned queries
+        kv = self.ln(prefix_values.squeeze(1).float())                  # (B, L, D)
+        a, _ = self.attn(q, kv, kv, key_padding_mask=~prefix_pad_masks.bool(), need_weights=False)
+        z = q + a
+        h = self.ln_self(z)
+        s, _ = self.self_attn(h, h, h, need_weights=False)
+        z = z + s
+        return self.out(z).squeeze(-1)                                  # (B, H) logits
+
+
+class Race2ActionOnlyHead(nn.Module):
+    def __init__(self, expert_dim: int, proj_dim: int = 256, num_heads: int = 4):
+        super().__init__()
+        self.q_proj = nn.Linear(expert_dim, proj_dim)
+        self.ln_self = nn.LayerNorm(proj_dim)
+        self.self_attn = nn.MultiheadAttention(proj_dim, num_heads, batch_first=True)
+        self.out = nn.Linear(proj_dim, 1)
+
+    def forward(self, expert_hidden: torch.Tensor) -> torch.Tensor:
+        z = self.q_proj(expert_hidden.float())                          # (B, H, D) auxiliary-pass slot features
+        h = self.ln_self(z)
+        s, _ = self.self_attn(h, h, h, need_weights=False)
+        z = z + s
+        return self.out(z).squeeze(-1)                                  # (B, H) logits

ablation/abl_nojitter_h20_40k/source/src/openpi/models_pytorch/race2_ablation/model.py

@@ -0,0 +1,384 @@
+"""RACE2 ablations (VLABench). Separate subclasses so the full method's code is untouched.
+
+Full RACE2 (openpi.models_pytorch.race.model.RacePI0.forward), per training step:
+    aux pass   tau = read_tau, unmodulated  -> timing head -> bce(b*)        [timing loss]
+                                            -> action_out_proj -> base_fm     [auxiliary loss]
+    FM pass    tau ~ U, modulated by alpha(tau) * jitter(b*) * M(e_trans)    [fm_loss]
+    total = fm_loss + lambda_base_fm * base_fm + lambda_timing * bce
+Serving: aux pass -> b_hat -> hook modulation of the denoising passes.
+
+A1  Race2TimingOnlyPI0  "timing loss only, no conditioning"
+    The head is trained with the timing BCE and its gradient reaches the action expert
+    through the auxiliary pass (hidden_read is NOT detached, as in the full method), but
+    there is no field: e_trans / field_mod / tau_gate are removed, the FM pass is the
+    plain pi0.5 flow-matching loss and serving is pi0.5's own sampler. This isolates the
+    effect of timing supervision as a multi-task signal.
+        total = fm_loss(unmodulated) + lambda_timing * bce
+
+A3  Race2NoAuxPI0       "full method without the auxiliary (base FM) loss"
+    Identical model, conditioning and serving; only the base FM term is dropped
+    (lambda_base_fm must be 0, which RacePI0.forward already honours exactly).
+        total = fm_loss(modulated, teacher-forced b*) + lambda_timing * bce
+
+A2  Race2FmOnlyPI0      "conditioning driven end-to-end by the FM loss"
+    Same parameters, conditioning sites and serving path as the full method, but the
+    field is built from the head's own prediction b_hat instead of the teacher-forced
+    b* (no jitter), and the only loss is the flow-matching loss. The timing head,
+    alpha(tau), e_trans and field_mod are therefore trained solely through the FM
+    gradient that flows back through the modulation; the timing BCE and the base FM
+    term are computed under no_grad for monitoring only.
+        total = fm_loss(modulated by alpha(tau) * b_hat * M(e_trans))
+
+A6  Race2NoJitterPI0    "w/o jittering": full method with jitter_prob pinned to 0, i.e. the
+    teacher-forced field uses b* exactly (RacePI0._jitter_b is the identity for p <= 0).
+
+A7  Race2NoTeacherForcingPI0  "w/o teacher forcing": full losses (fm + base_fm + bce), but the
+    FM pass is modulated by the head's own DETACHED prediction b_hat instead of b*. The head
+    keeps learning from the BCE only; the field (alpha, e_trans, field_mod) and the expert
+    learn under the prediction distribution they will see at test time. No jitter (it is a
+    b* perturbation). Isolates the exposure-bias effect of teacher forcing.
+
+A8  Race2VlmHeadPI0     "prediction head: VLM features only": the head never sees the
+    auxiliary-pass expert features; H learned slot tokens query the cached VLM values
+    (heads.Race2VlmOnlyHead). The auxiliary pass is still run so the base-FM loss and every
+    other term stay identical to the full method; only the head input changes (and, as a
+    consequence, the BCE gradient no longer reaches the expert).
+
+A9  Race2ActionHeadPI0  "prediction head: action features only": the head has no access to
+    the VLM cache; the auxiliary-pass slot features go through slot self-attention and the
+    linear only (heads.Race2ActionOnlyHead). Everything else is the full method.
+"""
+
+import torch
+import torch.nn.functional as F  # noqa: N812
+
+from openpi.models_pytorch.pi0_pytorch import PI0Pytorch
+from openpi.models_pytorch.race2.model import Race2PI0
+from openpi.models_pytorch.race2_ablation.heads import Race2ActionOnlyHead, Race2VlmOnlyHead
+
+
+class Race2TimingOnlyPI0(Race2PI0):
+    """A1: RACE2 timing head + BCE, no field modulation anywhere."""
+
+    def __init__(self, config):
+        super().__init__(config)
+        if float(config.lambda_base_fm) != 0.0:
+            raise ValueError("Race2TimingOnlyPI0 has no auxiliary loss; set lambda_base_fm=0")
+        # No conditioning: drop the field parameters so checkpoints carry no dead weights
+        # and nothing can modulate the expert. The norm hooks installed by RacePI0 stay
+        # inert because _mod_buf is never allocated.
+        del self.e_trans
+        del self.field_mod
+        del self.tau_gate
+        self._mod_consts = None
+        self._mod_buf = None
+
+    # ---- conditioning is structurally absent -----------------------------------
+    @torch.compiler.disable
+    def _refresh_mod_consts(self):
+        self._mod_consts = None
+        return None
+
+    def _prepare_mod_buf(self, bsize, horizon, device):  # never used; keep hooks inert
+        self._mod_buf = None
+
+    def _field(self, b, tau):
+        raise RuntimeError("Race2TimingOnlyPI0 has no field modulation")
+
+    # ---- training --------------------------------------------------------------
+    def forward(self, observation, actions, b_star=None, noise=None, time=None):
+        cfg = self.config
+        if b_star is None:
+            raise ValueError("Race2TimingOnlyPI0.forward requires b_star (B,H)")
+        images, img_masks, lang_tokens, lang_masks, state = self._preprocess_observation(observation, train=True)
+        device = actions.device
+        bsize = actions.shape[0]
+        b_star = b_star.to(device=device, dtype=torch.float32)
+
+        if noise is None:
+            noise = self.sample_noise(actions.shape, device)
+        u_t = noise - actions
+        use_bf16 = (
+            self.paligemma_with_expert.paligemma.language_model.layers[0].self_attn.q_proj.weight.dtype
+            == torch.bfloat16
+        )
+        prefix_pad_masks, prefix_keys, prefix_values, _prefix_hidden = self._prefix_h(
+            images, img_masks, lang_tokens, lang_masks, use_bf16
+        )
+
+        # ---- auxiliary pass (tau = read_tau, unmodulated) -> timing head only.
+        # hidden_read is not detached: the BCE gradient reaches the action expert.
+        read_time = torch.full((bsize,), cfg.read_tau, device=device)
+        x_read = cfg.read_tau * noise + (1 - cfg.read_tau) * actions
+        hidden_read = self._run_expert(
+            state, x_read, read_time, prefix_pad_masks, prefix_keys, prefix_values, field=None
+        )
+        logits = self._timing_logits(hidden_read, prefix_values[self.kv_layer], prefix_pad_masks)
+        timing_bce = F.binary_cross_entropy_with_logits(logits, b_star)
+
+        # ---- plain flow-matching pass: no field, no teacher forcing
+        if time is None:
+            time = self.sample_time(bsize, device)
+        x_t = time[:, None, None] * noise + (1 - time[:, None, None]) * actions
+        hidden = self._run_expert(state, x_t, time, prefix_pad_masks, prefix_keys, prefix_values, field=None)
+        fm_loss = F.mse_loss(u_t, self.action_out_proj(hidden))
+
+        total = fm_loss + cfg.lambda_timing * timing_bce
+
+        metrics = {"fm_loss": fm_loss.detach(), "bce": timing_bce.detach(),
+                   "base_fm": torch.zeros((), device=device)}
+        with torch.no_grad():
+            b_hat = torch.sigmoid(logits)
+            pos = b_star.max(dim=1).values >= 0.5
+            z = torch.tensor(0.0, device=device)
+            if pos.any():
+                peak_err = (b_hat[pos].argmax(dim=1) - b_star[pos].argmax(dim=1)).abs()
+                metrics["peak_pm1"] = (peak_err <= 1).float().mean()
+                metrics["b_pos"] = b_hat[pos].max(dim=1).values.mean()
+            else:
+                metrics["peak_pm1"] = z
+                metrics["b_pos"] = z
+            metrics["b_neg"] = b_hat[~pos].max(dim=1).values.mean() if (~pos).any() else z
+            metrics["pres_acc"] = ((b_hat.max(dim=1).values >= 0.5) == pos).float().mean()
+
+        self._step += 1
+        self.last_metrics = {k: float(v) for k, v in metrics.items()}
+        return total
+
+    # ---- serving: pi0.5's own sampler (hooks inert, no auxiliary pass) --------------
+    @torch.no_grad()
+    def sample_actions(self, device, observation, noise=None, num_steps=10):
+        return PI0Pytorch.sample_actions(self, device, observation, noise=noise, num_steps=num_steps)
+
+    @torch.no_grad()
+    def _sample_actions_reference(self, device, observation, noise=None, num_steps=10):
+        """Hand-rolled unmodulated rollout, parity target for tests only."""
+        cfg = self.config
+        bsize = observation.state.shape[0]
+        if noise is None:
+            noise = self.sample_noise((bsize, cfg.action_horizon, cfg.action_dim), device)
+        images, img_masks, lang_tokens, lang_masks, state = self._preprocess_observation(observation, train=False)
+        ppm, keys, values, _ = self._prefix_h(images, img_masks, lang_tokens, lang_masks, use_bf16=False)
+        dt = -1.0 / num_steps
+        x_t = noise.clone()
+        t = 1.0
+        for _ in range(num_steps):
+            tt = torch.full((bsize,), t, dtype=torch.float32, device=device)
+            hidden = self._run_expert(state, x_t, tt, ppm, keys, values, field=None)
+            x_t = x_t + dt * self.action_out_proj(hidden)
+            t += dt
+        return x_t
+
+
+class Race2FmOnlyPI0(Race2PI0):
+    """A2: RACE2 conditioning with b_hat in the loop, FM loss only (no timing / aux loss)."""
+
+    def __init__(self, config):
+        super().__init__(config)
+        if float(config.lambda_base_fm) != 0.0 or float(config.lambda_timing) != 0.0:
+            raise ValueError("Race2FmOnlyPI0 trains with the FM loss only; set lambda_base_fm=lambda_timing=0")
+
+    def forward(self, observation, actions, b_star=None, noise=None, time=None):
+        cfg = self.config
+        if b_star is None:
+            raise ValueError("Race2FmOnlyPI0.forward requires b_star (B,H) for monitoring metrics")
+        images, img_masks, lang_tokens, lang_masks, state = self._preprocess_observation(observation, train=True)
+        device = actions.device
+        bsize = actions.shape[0]
+        b_star = b_star.to(device=device, dtype=torch.float32)
+
+        if noise is None:
+            noise = self.sample_noise(actions.shape, device)
+        u_t = noise - actions
+        use_bf16 = (
+            self.paligemma_with_expert.paligemma.language_model.layers[0].self_attn.q_proj.weight.dtype
+            == torch.bfloat16
+        )
+        prefix_pad_masks, prefix_keys, prefix_values, _prefix_hidden = self._prefix_h(
+            images, img_masks, lang_tokens, lang_masks, use_bf16
+        )
+
+        # ---- auxiliary pass (tau = read_tau, unmodulated) -> head -> b_hat (kept in the graph)
+        read_time = torch.full((bsize,), cfg.read_tau, device=device)
+        x_read = cfg.read_tau * noise + (1 - cfg.read_tau) * actions
+        hidden_read = self._run_expert(
+            state, x_read, read_time, prefix_pad_masks, prefix_keys, prefix_values, field=None
+        )
+        logits = self._timing_logits(hidden_read, prefix_values[self.kv_layer], prefix_pad_masks)
+        b_hat = torch.sigmoid(logits)
+
+        # ---- modulated FM pass: field from the PREDICTED b_hat (no teacher forcing, no jitter)
+        if time is None:
+            time = self.sample_time(bsize, device)
+        x_t = time[:, None, None] * noise + (1 - time[:, None, None]) * actions
+        field = self._field(b_hat, time)
+        hidden = self._run_expert(state, x_t, time, prefix_pad_masks, prefix_keys, prefix_values, field=field)
+        fm_loss = F.mse_loss(u_t, self.action_out_proj(hidden))
+
+        total = fm_loss
+
+        metrics = {"fm_loss": fm_loss.detach()}
+        with torch.no_grad():
+            # monitoring only: how the FM-trained head relates to the changepoint target
+            metrics["bce"] = F.binary_cross_entropy_with_logits(logits.detach(), b_star)
+            metrics["base_fm"] = F.mse_loss(u_t, self.action_out_proj(hidden_read.detach()))
+            pos = b_star.max(dim=1).values >= 0.5
+            z = torch.tensor(0.0, device=device)
+            if pos.any():
+                peak_err = (b_hat[pos].argmax(dim=1) - b_star[pos].argmax(dim=1)).abs()
+                metrics["peak_pm1"] = (peak_err <= 1).float().mean()
+                metrics["b_pos"] = b_hat[pos].max(dim=1).values.mean()
+            else:
+                metrics["peak_pm1"] = z
+                metrics["b_pos"] = z
+            metrics["b_neg"] = b_hat[~pos].max(dim=1).values.mean() if (~pos).any() else z
+            metrics["pres_acc"] = ((b_hat.max(dim=1).values >= 0.5) == pos).float().mean()
+            metrics["b_hat_mean"] = b_hat.mean()
+            g = torch.sigmoid(self.tau_gate)
+            for i in range(cfg.n_tau):
+                metrics[f"alpha_{i}"] = g[i].detach()
+            metrics["g_first"] = g[0].detach()
+            metrics["g_last"] = g[-1].detach()
+            v = self.e_trans.float()
+            mods = torch.stack([lin(v) for lin in self.field_mod])
+            s_p, b_p, _ = mods.chunk(3, dim=-1)
+            metrics["dmod"] = (s_p.norm(dim=-1) + b_p.norm(dim=-1)).mean()
+
+        self._step += 1
+        self.last_metrics = {k: float(v) for k, v in metrics.items()}
+        return total
+
+
+class Race2NoAuxPI0(Race2PI0):
+    """A3: full RACE2 minus the auxiliary base-FM loss. RacePI0.forward skips the base FM
+    term entirely when lambda_base_fm == 0, so training, conditioning, serving and the
+    checkpoint layout are inherited unchanged; this class only pins that setting."""
+
+    def __init__(self, config):
+        super().__init__(config)
+        if float(config.lambda_base_fm) != 0.0:
+            raise ValueError("Race2NoAuxPI0 drops the auxiliary loss; set lambda_base_fm=0")
+
+
+class Race2NoJitterPI0(Race2PI0):
+    """A6: full RACE2 with the field-jitter disabled (jitter_prob must be 0)."""
+
+    def __init__(self, config):
+        super().__init__(config)
+        if float(config.jitter_prob) != 0.0:
+            raise ValueError("Race2NoJitterPI0 disables jitter; set jitter_prob=0")
+
+
+class Race2NoTeacherForcingPI0(Race2PI0):
+    """A7: full RACE2 losses, FM pass conditioned on the detached prediction b_hat (no b* teacher forcing)."""
+
+    def __init__(self, config):
+        super().__init__(config)
+        if float(config.jitter_prob) != 0.0:
+            raise ValueError("Race2NoTeacherForcingPI0 never sees b* in the field; set jitter_prob=0")
+
+    def forward(self, observation, actions, b_star=None, noise=None, time=None):
+        cfg = self.config
+        if b_star is None:
+            raise ValueError("Race2NoTeacherForcingPI0.forward requires b_star (B,H)")
+        images, img_masks, lang_tokens, lang_masks, state = self._preprocess_observation(observation, train=True)
+        device = actions.device
+        bsize = actions.shape[0]
+        b_star = b_star.to(device=device, dtype=torch.float32)
+
+        if noise is None:
+            noise = self.sample_noise(actions.shape, device)
+        u_t = noise - actions
+        use_bf16 = (
+            self.paligemma_with_expert.paligemma.language_model.layers[0].self_attn.q_proj.weight.dtype
+            == torch.bfloat16
+        )
+        prefix_pad_masks, prefix_keys, prefix_values, _prefix_hidden = self._prefix_h(
+            images, img_masks, lang_tokens, lang_masks, use_bf16
+        )
+
+        # ---- auxiliary pass (tau = read_tau, unmodulated) -> timing head + base FM (as in the full method)
+        read_time = torch.full((bsize,), cfg.read_tau, device=device)
+        x_read = cfg.read_tau * noise + (1 - cfg.read_tau) * actions
+        hidden_read = self._run_expert(
+            state, x_read, read_time, prefix_pad_masks, prefix_keys, prefix_values, field=None
+        )
+        logits = self._timing_logits(hidden_read, prefix_values[self.kv_layer], prefix_pad_masks)
+        timing_bce = F.binary_cross_entropy_with_logits(logits, b_star)
+        base_fm = (
+            F.mse_loss(u_t, self.action_out_proj(hidden_read))
+            if cfg.lambda_base_fm > 0
+            else torch.zeros((), device=device)
+        )
+
+        # ---- modulated pass: field from the DETACHED prediction (no teacher forcing, no jitter)
+        if time is None:
+            time = self.sample_time(bsize, device)
+        x_t = time[:, None, None] * noise + (1 - time[:, None, None]) * actions
+        b_field = torch.sigmoid(logits).detach()
+        field = self._field(b_field, time)
+        hidden = self._run_expert(state, x_t, time, prefix_pad_masks, prefix_keys, prefix_values, field=field)
+        fm_loss = F.mse_loss(u_t, self.action_out_proj(hidden))
+
+        total = fm_loss + cfg.lambda_base_fm * base_fm + cfg.lambda_timing * timing_bce
+
+        metrics = {"fm_loss": fm_loss.detach(), "base_fm": base_fm.detach(), "bce": timing_bce.detach()}
+        with torch.no_grad():
+            b_hat = torch.sigmoid(logits)
+            pos = b_star.max(dim=1).values >= 0.5
+            z = torch.tensor(0.0, device=device)
+            if pos.any():
+                peak_err = (b_hat[pos].argmax(dim=1) - b_star[pos].argmax(dim=1)).abs()
+                metrics["peak_pm1"] = (peak_err <= 1).float().mean()
+                metrics["b_pos"] = b_hat[pos].max(dim=1).values.mean()
+            else:
+                metrics["peak_pm1"] = z
+                metrics["b_pos"] = z
+            metrics["b_neg"] = b_hat[~pos].max(dim=1).values.mean() if (~pos).any() else z
+            metrics["pres_acc"] = ((b_hat.max(dim=1).values >= 0.5) == pos).float().mean()
+            metrics["b_hat_mean"] = b_hat.mean()
+            g = torch.sigmoid(self.tau_gate)
+            for i in range(cfg.n_tau):
+                metrics[f"alpha_{i}"] = g[i].detach()
+            metrics["g_first"] = g[0].detach()
+            metrics["g_last"] = g[-1].detach()
+            v = self.e_trans.float()
+            mods = torch.stack([lin(v) for lin in self.field_mod])
+            s_p, b_p, _ = mods.chunk(3, dim=-1)
+            metrics["dmod"] = (s_p.norm(dim=-1) + b_p.norm(dim=-1)).mean()
+
+        self._step += 1
+        self.last_metrics = {k: float(v) for k, v in metrics.items()}
+        return total
+
+
+class Race2VlmHeadPI0(Race2PI0):
+    """A8: timing head on the cached VLM values only (learned slot queries); auxiliary pass kept."""
+
+    def _build_head(self, config):
+        cache_dim = self.paligemma_with_expert.paligemma.language_model.layers[0].self_attn.head_dim
+        return Race2VlmOnlyHead(
+            expert_dim=self.action_in_proj.out_features,
+            kv_dim=cache_dim,
+            horizon=config.action_horizon,
+            proj_dim=config.timing_proj_dim,
+            num_heads=config.timing_num_heads,
+        )
+
+    # needs_aux stays True: the auxiliary pass still feeds the base-FM loss; the head ignores it.
+    def _timing_logits(self, hidden_read, cache_v, prefix_pad_masks):
+        return self.race_head(cache_v, prefix_pad_masks)
+
+
+class Race2ActionHeadPI0(Race2PI0):
+    """A9: timing head on the auxiliary-pass action features only (no VLM cross-attention)."""
+
+    def _build_head(self, config):
+        return Race2ActionOnlyHead(
+            expert_dim=self.action_in_proj.out_features,
+            proj_dim=config.timing_proj_dim,
+            num_heads=config.timing_num_heads,
+        )
+
+    def _timing_logits(self, hidden_read, cache_v, prefix_pad_masks):
+        return self.race_head(hidden_read)

ablation/abl_nojitter_h20_40k/training_config.json

@@ -0,0 +1,107 @@
+{
+  "global_step": 40000,
+  "config": {
+    "name": "pi05_vlabench_race2abl_nojitter_h20",
+    "project_name": "openpi",
+    "exp_name": "abl_nojitter_h20_s42",
+    "model": {
+      "action_dim": 32,
+      "action_horizon": 20,
+      "max_token_len": 200,
+      "dtype": "bfloat16",
+      "paligemma_variant": "gemma_2b",
+      "action_expert_variant": "gemma_300m",
+      "pi05": true,
+      "discrete_state_input": false,
+      "gemma_depth": 18,
+      "sigma": 1.0,
+      "support": 3.0,
+      "jitter_prob": 0.0,
+      "lambda_timing": 0.05,
+      "lambda_base_fm": 0.1,
+      "read_tau": 1.0,
+      "n_tau": 10,
+      "tau_gate_init": 0.0,
+      "tau_gate_lr_mult": 20.0,
+      "timing_proj_dim": 256,
+      "timing_num_heads": 4,
+      "micro_batch_size": 32,
+      "fast_serving": 0,
+      "kv_layer": -1
+    },
+    "weight_loader": {},
+    "pytorch_weight_path": "checkpoints/pi05_vlabench_official/pytorch",
+    "pytorch_weight_path_teacher": null,
+    "pytorch_training_precision": "bfloat16",
+    "lr_schedule": {
+      "warmup_steps": 500,
+      "peak_lr": 5e-05,
+      "decay_steps": 39500,
+      "decay_lr": 5e-06
+    },
+    "optimizer": {
+      "b1": 0.9,
+      "b2": 0.95,
+      "eps": 1e-08,
+      "weight_decay": 1e-10,
+      "clip_gradient_norm": 1.0
+    },
+    "ema_decay": 0.999,
+    "freeze_filter": "Nothing()",
+    "data": {
+      "repo_id": "SeonghoonYu/vlabench_primitive_ft_lerobot_224",
+      "assets": {
+        "assets_dir": "./assets/pi05_vlabench",
+        "asset_id": "vlabench/vlabench_ft_primitive"
+      },
+      "base_config": {
+        "repo_id": null,
+        "asset_id": null,
+        "norm_stats": null,
+        "repack_transforms": {
+          "inputs": [],
+          "outputs": []
+        },
+        "data_transforms": {
+          "inputs": [],
+          "outputs": []
+        },
+        "model_transforms": {
+          "inputs": [],
+          "outputs": []
+        },
+        "use_quantile_norm": false,
+        "action_sequence_keys": [
+          "actions"
+        ],
+        "prompt_from_task": true,
+        "rlds_data_dir": null,
+        "action_space": null,
+        "datasets": []
+      },
+      "phase_labels_path": null,
+      "phase_label_horizon": null,
+      "cps_path": "assets/gauss_timing/vlabench_cps.npz",
+      "gauss_sigma": 1.0,
+      "gauss_support": 3.0
+    },
+    "assets_base_dir": "./assets",
+    "checkpoint_base_dir": "./checkpoints",
+    "seed": 42,
+    "batch_size": 64,
+    "num_workers": 8,
+    "num_train_steps": 40000,
+    "log_interval": 100,
+    "save_interval": 1000,
+    "save_overlays_from_step": 20000,
+    "keep_period": 5000,
+    "overwrite": false,
+    "resume": false,
+    "wandb_enabled": false,
+    "policy_metadata": null,
+    "fsdp_devices": 1
+  },
+  "timestamp": 1789600972.2608142,
+  "ema_decay": 0.999,
+  "format": "trainable-only overlays; materialise with scripts/race/materialize_ckpt.py"
+}