AlexWortega commited on
Commit
59cf920
·
verified ·
1 Parent(s): 2e262f4

ablation.md: complete consistent-lag test (3 ages, fixes a filename-mapping bug that silently dropped the 1s row)

Browse files
Files changed (1) hide show
  1. ablation.md +13 -10
ablation.md CHANGED
@@ -218,12 +218,13 @@ problem, holding proprioception back by the same amount - keeping the observatio
218
  internally consistent, which is what caching the vision tower on a real robot would
219
  actually look like - should help.
220
 
221
- | pixel + proprio age | Success | 95% CI | delta | paired 95% CI | flips | McNemar p |
222
- |---|---|---|---|---|---|---|
223
- | 0.25 s | 21/150 = 14.0% | 9.3–20.5 | -70.0 | -78.0…-61.3 | +4/-109 | 0.0000*** |
224
- | 0.5 s | 0/150 = 0.0% | 0.0–2.5 | -84.0 | -89.3…-78.0 | +0/-126 | 0.0000*** |
 
225
 
226
- At 0.25 s, the internally-consistent version scores **14.0%** against **23.3%** for stale pixels with fresh proprioception — significantly *worse* (-9.3 points, 20 episodes lost against 6 won, McNemar p = 0.0094), not better. So the collapse is not explained by the observation being self-inconsistent — staling proprioception on top of the pixels makes things worse, not better. Holding proprioception back to make a cached vision tower's input "agree with itself" is not a viable mitigation: geometric staleness costs, and adding more of it costs more.
227
 
228
  ---
229
 
@@ -276,11 +277,13 @@ rather than at a matched frequency:
276
  ## 8. Not yet included
277
 
278
  Running or pending, and deliberately not summarized here until finished: the
279
- remaining consistent-lag cells (0.5 s, 1 s), the compute-matched gate comparison,
280
- the perturbation suite (instruction swap, paraphrase, mid-episode instruction
281
- change, occlusion, latency ramp), retrained staleness and latency-metadata
282
- variants, and same-harness baselines. The fresh/stale grid itself (section 5) is
283
- done: 12 cells plus two controls, 500 + 12x150 + 2x150 = 2,300 closed-loop episodes.
 
 
284
 
285
  `checkpoints/stage2_final` (SO-100/101 + Bridge mixture, pre-LIBERO) and
286
  `checkpoints/stage3_final` (+ spatial distillation) just finished downloading and
 
218
  internally consistent, which is what caching the vision tower on a real robot would
219
  actually look like - should help.
220
 
221
+ | age | pixels-only stale | pixels + proprio stale | delta | flips | McNemar p |
222
+ |---|---|---|---|---|---|
223
+ | 0.25 s | 23.3% | 14.0% | -9.3 | +6/-20 | 0.0094** |
224
+ | 0.5 s | 5.3% | 0.0% | -5.3 | +0/-8 | 0.0078** |
225
+ | 1.0 s | 2.7% | 0.7% | -2.0 | +1/-4 | 0.3750 |
226
 
227
+ All 3 ages tested, all in the same direction: making the observation internally consistent does not help, and at 2/3 of them it is significantly *worse*. So the collapse is not explained by the observation disagreeing with itself — staling proprioception on top of the pixels adds damage rather than removing it. Holding proprioception back to make a cached vision tower's input "agree with itself" is not a viable mitigation: geometric staleness costs, and adding more of it costs more.
228
 
229
  ---
230
 
 
277
  ## 8. Not yet included
278
 
279
  Running or pending, and deliberately not summarized here until finished: the
280
+ compute-matched gate comparison, the perturbation suite (instruction swap,
281
+ paraphrase, mid-episode instruction change, occlusion, latency ramp), retrained
282
+ staleness and latency-metadata variants, and same-harness baselines beyond the
283
+ three checkpoints in section 9. The fresh/stale grid (section 5) and the
284
+ consistent-lag check (section 5, three ages) are both done: 12 grid cells, two
285
+ controls, three consistency cells - 500 + 12x150 + 2x150 + 3x150 = 2,750
286
+ closed-loop episodes.
287
 
288
  `checkpoints/stage2_final` (SO-100/101 + Bridge mixture, pre-LIBERO) and
289
  `checkpoints/stage3_final` (+ spatial distillation) just finished downloading and