matCercola18 commited on
Commit
c67b25a
·
verified ·
1 Parent(s): 7b27e61

Add files using upload-large-folder tool

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. results/baselines/ablation_k.txt +10 -0
  2. results/baselines/ablation_k_run.log +17 -0
  3. results/baselines/ablation_steps_siso.txt +9 -0
  4. results/baselines/bank_density_siso.txt +68 -0
  5. results/baselines/distinct_retrieval_prereg.txt +27 -0
  6. results/baselines/fixed_controller_pendulum.txt +19 -0
  7. results/baselines/pendulum_bank_density.txt +36 -0
  8. results/baselines/pendulum_bank_density_diag_a.txt +15 -0
  9. results/baselines/pendulum_bank_density_diag_b_seed123.txt +8 -0
  10. results/baselines/retrieval_mimo_seed7777.txt +41 -0
  11. results/baselines/retrieval_objective_conditioning.txt +129 -0
  12. results/baselines/retrieval_pendulum.txt +28 -0
  13. results/baselines/retrieval_pendulum_seed321.txt +29 -0
  14. results/baselines/retrieval_siso.txt +36 -0
  15. results/baselines/siso_run.log +19 -0
  16. results/furuta_v3/deploy_sidebyside.txt +19 -0
  17. results/furuta_v3/ksweep_sidebyside.txt +9 -0
  18. results/furuta_v3/traj_gen_eval_v3_raw_clip1.txt +94 -0
  19. results/furuta_v3/traj_gen_eval_v3e2400s1_raw_clip1.txt +94 -0
  20. results/furuta_v3/traj_gen_v2ctl2400_raw_clip5.txt +30 -0
  21. results/furuta_v3/traj_gen_v3e2400_raw_clip1.txt +30 -0
  22. results/furuta_v3/traj_gen_warmstart_v2clip5_n200.txt +17 -0
  23. results/furuta_v3/traj_gen_warmstart_v2clip5_s1.txt +17 -0
  24. results/furuta_v3/traj_gen_warmstart_v2clip5_s2.txt +17 -0
  25. results/furuta_v3/traj_gen_warmstart_v3_n200.txt +17 -0
  26. results/furuta_v3/traj_gen_warmstart_v3_s2.txt +17 -0
  27. results/generalize/fig_generalize.png +0 -0
  28. results/generalize/summary.txt +5 -0
  29. results/metric_steering/mimo_summary.txt +12 -0
  30. results/metric_steering/refine_summary.txt +12 -0
  31. results/metric_steering/summary.txt +12 -0
  32. results/mimo/k_sweep_mimo.txt +11 -0
  33. results/mimo/target_ablation.md +6 -0
  34. results/pareto/fig_pareto.pdf +0 -0
  35. results/pareto/fig_pareto.png +0 -0
  36. results/pareto/table_pareto.md +9 -0
  37. results/pareto_specs/run_s0.log +28 -0
  38. results/pareto_specs/run_s1.log +26 -0
  39. results/pareto_specs/run_s2.log +29 -0
  40. results/pareto_specs/run_s3.log +29 -0
  41. results/pareto_specs/run_s4.log +28 -0
  42. results/pareto_specs/significance.txt +15 -0
  43. results/pareto_specs/summary.txt +15 -0
  44. results/pareto_specs/summary_s0.txt +15 -0
  45. results/pareto_specs/summary_s1.txt +15 -0
  46. results/pareto_specs/summary_s2.txt +15 -0
  47. results/pareto_specs/summary_s3.txt +15 -0
  48. results/pareto_specs/summary_s4.txt +15 -0
  49. results/physics_gen/pendulum_physics_gen.txt +720 -0
  50. results/physics_gen/pendulum_physics_gen_v2.txt +188 -0
results/baselines/ablation_k.txt ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ SISO K-sample ablation — pooled 5 seeds × 60 plants
2
+
3
+ K plant-stabilisable rate (≥1 of K) 95% CI
4
+ 1 0.633 [0.580, 0.687]
5
+ 2 0.713 [0.660, 0.763]
6
+ 4 0.747 [0.697, 0.793]
7
+ 8 0.770 [0.723, 0.817]
8
+ 16 0.787 [0.740, 0.833]
9
+
10
+ lift K=1 -> K=16: +0.153 (0.63 -> 0.79) — the generative distribution's value.
results/baselines/ablation_k_run.log ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [seed 0] done
2
+ [seed 1] done
3
+ [seed 2] done
4
+ [seed 3] done
5
+ [seed 4] done
6
+
7
+ === K ABLATION ===
8
+ SISO K-sample ablation — pooled 5 seeds × 60 plants
9
+
10
+ K plant-stabilisable rate (≥1 of K) 95% CI
11
+ 1 0.633 [0.580, 0.687]
12
+ 2 0.713 [0.660, 0.763]
13
+ 4 0.747 [0.697, 0.793]
14
+ 8 0.770 [0.723, 0.817]
15
+ 16 0.787 [0.740, 0.833]
16
+
17
+ lift K=1 -> K=16: +0.153 (0.63 -> 0.79) — the generative distribution's value.
results/baselines/ablation_steps_siso.txt ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ === SISO DDIM steps sweep (plants seed 0, N=60, K=8, sampler seeds (0, 1, 2), eta=0, device=cuda) ===
2
+ model fixed (full-data SISO); verifier/x0 = retrieval_baselines.py siso (x0 seed 777); oracle = 60-step Adam refinement of the DDPM@500 seed-0 K=8 winner (as bank_density; negative regret possible).
3
+ regret 'common' = median over plants covered by EVERY arm's seed-0 draw (n=46); 'own' = over each draw's own covered plants. ms/ctrl = wall time per sampled controller, batch 480.
4
+ arm cov per-samp reg common% reg own% ms/ctrl McNemar vs ddpm500 (n10,n01,p)
5
+ ddim5 0.817+-0.017 0.682+-0.009 1.25+-9.75 -0.46+-5.83 0.039+-0.029 (3, 0) p=0.25
6
+ ddim20 0.800+-0.017 0.658+-0.006 5.76+-13.42 5.71+-13.39 0.086+-0.000 (2, 1) p=1
7
+ ddim100 0.794+-0.025 0.634+-0.014 14.75+-9.27 13.04+-7.10 0.461+-0.005 (0, 1) p=1
8
+ ddim500 0.789+-0.010 0.622+-0.019 19.11+-10.09 16.61+-7.12 2.292+-0.010 (1, 1) p=1
9
+ ddpm500 0.794+-0.010 0.632+-0.013 25.27+-11.55 23.72+-10.49 1.997+-0.005 (0, 0) p=1
results/baselines/bank_density_siso.txt ADDED
@@ -0,0 +1,68 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ === BANK-DENSITY SWEEP, SISO (held-out draws: seeds [0, 4321], N=[60, 60], pooled N=120) ===
2
+ MODEL SIDE IS FIXED: full-data checkpoint, NOT retrained per bank fraction (the pendulum sweep retrained; this one does not).
3
+ bank = seeded random subset of training PLANTS (all stored controllers of the chosen plants); fractions (0.05, 0.1, 0.25, 0.5, 1.0); subset seeds (0, 1, 2) (f=1.0: single bank);
4
+ model draws: sampler seeds (0, 1, 2); McNemar/regret reference = model seed 0. Verifier/x0/N as retrieval_baselines.py siso; K=8 and K=1 (K=1 = first candidate).
5
+ oracle: shared per plant = 60-step Adam refinement (generalize.py protocol) of the model seed-0 K=8 winner (the SAME oracle yesterday's SISO model arm used), floored by that winner's J; arms that beat it show NEGATIVE regret; regret undefined where the model seed-0 draw is uncovered (never inside the common set)
6
+ regret % = (J_best - J_oracle)/J_oracle on the COMMON success set (plants where BOTH the arm and the model seed-0 draw are covered at that K); label 'cond.'.
7
+ coverage 95% CI: bootstrap (5000) over pooled plants of subset-seed 0. McNemar: exact, paired on pooled plants vs model seed 0, reported for each subset seed (min..max p).
8
+
9
+ ----- K=8 ----- MODEL coverage 0.719 +- 0.004 (3 draws), seed-0 CI [0.633,0.792], per-sample 0.566, cond. regret med 15.86%, gen 2.34 ms/ctrl
10
+ per-draw model coverage (seed 0/1/2): seed 0: 0.78/0.80/0.80; seed 4321: 0.65/0.63/0.65
11
+ arm=retrieval
12
+ f cov mean+-sd CI(ss0) per-samp dCov(ret-model) McNemar p(ss0..) reg_med% cond model reg% same set n_common NN dist
13
+ 0.05 0.531+-0.037 [0.408,0.592] 0.531 -0.189 3e-08..3.1e-05 86.89 16.55 63 0.0083
14
+ 0.10 0.461+-0.021 [0.375,0.558] 0.461 -0.258 1.2e-10..7.5e-09 85.70 16.68 55 0.0072
15
+ 0.25 0.478+-0.010 [0.383,0.558] 0.478 -0.242 1.9e-09..1.5e-08 77.99 14.06 57 0.0051
16
+ 0.50 0.456+-0.022 [0.375,0.550] 0.456 -0.264 5.8e-11..3.7e-09 80.47 15.09 54 0.0041
17
+ 1.00 0.450+-0.000 [0.358,0.542] 0.450 -0.269 4.7e-10..4.7e-10 73.24 14.42 54 0.0035
18
+ arm=blend
19
+ f cov mean+-sd CI(ss0) per-samp dCov(ret-model) McNemar p(ss0..) reg_med% cond model reg% same set n_common NN dist
20
+ 0.05 0.531+-0.037 [0.408,0.592] 0.531 -0.189 3e-08..3.1e-05 86.89 16.55 63 0.0083
21
+ 0.10 0.461+-0.021 [0.375,0.558] 0.461 -0.258 1.2e-10..7.5e-09 85.70 16.68 55 0.0072
22
+ 0.25 0.478+-0.010 [0.383,0.558] 0.478 -0.242 1.9e-09..1.5e-08 77.99 14.06 57 0.0051
23
+ 0.50 0.456+-0.022 [0.375,0.550] 0.456 -0.264 5.8e-11..3.7e-09 80.47 15.09 54 0.0041
24
+ 1.00 0.450+-0.000 [0.358,0.542] 0.450 -0.269 4.7e-10..4.7e-10 73.24 14.42 54 0.0035
25
+ arm=distinct
26
+ f cov mean+-sd CI(ss0) per-samp dCov(ret-model) McNemar p(ss0..) reg_med% cond model reg% same set n_common NN dist
27
+ 0.05 0.689+-0.004 [0.608,0.775] 0.484 -0.031 0.12..0.38 -4.80 15.32 82 0.0083
28
+ 0.10 0.689+-0.004 [0.608,0.767] 0.489 -0.031 0.12..0.38 -5.45 15.28 82 0.0072
29
+ 0.25 0.686+-0.004 [0.608,0.775] 0.474 -0.033 0.12..0.25 -10.99 15.32 82 0.0051
30
+ 0.50 0.686+-0.004 [0.608,0.775] 0.489 -0.033 0.12..0.25 -7.75 15.32 82 0.0041
31
+ 1.00 0.683+-0.000 [0.600,0.767] 0.483 -0.036 0.12..0.12 4.31 15.28 82 0.0035
32
+
33
+ ----- K=1 ----- MODEL coverage 0.600 +- 0.020 (3 draws), seed-0 CI [0.508,0.683], per-sample 0.600, cond. regret med 735.20%, gen 2.34 ms/ctrl
34
+ per-draw model coverage (seed 0/1/2): seed 0: 0.65/0.67/0.70; seed 4321: 0.55/0.48/0.55
35
+ arm=retrieval (blend K=1 == retrieval K=1 by construction)
36
+ f cov mean+-sd CI(ss0) per-samp dCov(ret-model) McNemar p(ss0..) reg_med% cond model reg% same set n_common NN dist
37
+ 0.05 0.531+-0.037 [0.408,0.592] 0.531 -0.069 0.023..0.84 119.81 955.04 55 0.0083
38
+ 0.10 0.461+-0.021 [0.375,0.558] 0.461 -0.139 0.00032..0.016 119.21 840.13 48 0.0072
39
+ 0.25 0.478+-0.010 [0.383,0.558] 0.478 -0.122 0.0059..0.024 88.08 1118.00 50 0.0051
40
+ 0.50 0.456+-0.022 [0.375,0.550] 0.456 -0.144 0.00019..0.0041 82.60 758.08 49 0.0041
41
+ 1.00 0.450+-0.000 [0.358,0.542] 0.450 -0.150 0.0014..0.0014 74.12 1131.46 48 0.0035
42
+
43
+ ----- retrieval coverage K=8 vs K=1 per f (mean over subset seeds; pooled draws) -----
44
+ f K=1 K=8 distinct K=8
45
+ 0.05 0.531 0.531 0.689
46
+ 0.10 0.461 0.461 0.689
47
+ 0.25 0.478 0.478 0.686
48
+ 0.50 0.456 0.456 0.686
49
+ 1.00 0.450 0.450 0.683
50
+
51
+ ----- smallest f at which retrieval MATCHES the model -----
52
+ rule (fixed before running): mean retrieval coverage >= model coverage - 0.05 AND pooled McNemar p >= 0.05 for >= 2 of 3 subset seeds (f=1.0: its single bank); 'never' if no f qualifies.
53
+ siso arm=retrieval K=8: never (none of the swept f)
54
+ siso arm=blend K=8: never (none of the swept f)
55
+ siso arm=distinct K=8: f=0.05
56
+ siso arm=retrieval K=1: never (none of the swept f)
57
+
58
+ ----- NN distance vs f (normalized conditioning space, mean over queries; mean over subset seeds) -----
59
+ f=0.05: draw0: 0.0098, draw4321: 0.0069
60
+ f=0.1: draw0: 0.0083, draw4321: 0.0061
61
+ f=0.25: draw0: 0.0060, draw4321: 0.0042
62
+ f=0.5: draw0: 0.0049, draw4321: 0.0033
63
+ f=1.0: draw0: 0.0042, draw4321: 0.0028
64
+
65
+ ADDENDUM: per-held-out-draw K=8 coverage (mean over subset seeds; model = seed-0 sampler / 3-seed mean)
66
+ draw0: model 0.794; retrieval f=0.05: 0.600 f=0.1: 0.533 f=0.25: 0.556 f=0.5: 0.522 f=1.0: 0.517 | distinct f=0.05: 0.772 f=0.1: 0.767 f=0.25: 0.767 f=0.5: 0.767 f=1.0: 0.767
67
+ draw1: model 0.644; retrieval f=0.05: 0.461 f=0.1: 0.389 f=0.25: 0.400 f=0.5: 0.389 f=1.0: 0.383 | distinct f=0.05: 0.606 f=0.1: 0.611 f=0.25: 0.606 f=0.5: 0.606 f=1.0: 0.600
68
+ retrieval f=1.0: plants where all 8 candidates share one verdict: 1.000 (pooled N=120)
results/baselines/distinct_retrieval_prereg.txt ADDED
@@ -0,0 +1,27 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ === PRE-REGISTERED distinct-plant retrieval vs diffusion, SISO, N=300 (plant seeds (100, 200, 300, 400, 500) x 60), full bank f=1.0, coverage only ===
2
+ model: full-data checkpoint, K<=16 samples/plant, sampler seeds 0/1/2 (seed 0 = paired reference; K-prefix of the 16 samples). verifier/x0 as bank_density sweep.
3
+
4
+ PRIMARY K=8: model(seed0) 0.773 distinct-first 0.750 diff +0.023 95% CI [+0.003,+0.047] McNemar model-only 9, retrieval-only 2, p=0.0654 => VERDICT: TIE
5
+ sampler seed 1: model 0.780 McNemar 12-3 p=0.0352
6
+ sampler seed 2: model 0.787 McNemar 13-2 p=0.00739
7
+ model 3-seed mean K=8: 0.780 +- 0.005
8
+
9
+ per plant-seed K=8 coverage (model seed0 / distinct-first):
10
+ 100: 0.800 / 0.783
11
+ 200: 0.800 / 0.783
12
+ 300: 0.833 / 0.800
13
+ 400: 0.733 / 0.717
14
+ 500: 0.700 / 0.667
15
+
16
+ SECONDARY coverage vs K (model seed0 [3-seed mean]; distinct-first; distinct-random mean of 3 seeds; McNemar model vs distinct-first)
17
+ K model s0 model mean dist-first dist-random McNemar (m-only,r-only,p)
18
+ 1 0.617 0.622 0.493 0.493+-0.000 (60,23,5.97e-05)
19
+ 2 0.707 0.706 0.660 0.660+-0.000 (24,10,0.0243)
20
+ 4 0.747 0.756 0.743 0.743+-0.000 (8,7,1)
21
+ 8 0.773 0.780 0.750 0.750+-0.000 (9,2,0.0654)
22
+ 16 0.787 0.793 0.753 0.753+-0.000 (12,2,0.0129)
23
+ random-stored K=8 (seed 0 draw) vs model: model-only 9, retrieval-only 2, p=0.0654
24
+
25
+ MIMO (distinct == plain retrieval, one controller per plant), f=1.0, K=8, from the sweep npz:
26
+ draw 9999: model 1.000 retrieval 0.992 McNemar 1-0 p=1
27
+ draw 4242: model 0.992 retrieval 0.992 McNemar 0-0 p=1
results/baselines/fixed_controller_pendulum.txt ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ === PENDULUM fixed dimensionless controller (N=40, seed=123 held-out) ===
2
+
3
+ Justification: theta'' = (g/l) sin(theta) - (b/(m l^2)) omega + u/(m l^2), controller u = -phi(y), y = c1 theta + c2 omega (c FIXED at (1.0, 0.35) for the whole family, as in the amortized model). The natural dimensionless rescaling of phi is by the holding torque m*g*l (the torque needed to hold the pendulum horizontal); a natural TIME scale is sqrt(l/g) (the small-oscillation period / sqrt(g/l) natural frequency), though this codebase does not rescale omega's contribution to y by it (c2 is fixed, not per-plant) -- so the fixed controller below rescales phi by m*g*l only, exactly mirroring how eval_pendulum.strong_oracle's own smart init (2.5*m*l*g*xp) and random-restart shapes ((U[0.5,2.5]*m*l*g)*xp*(|xp|/5+0.3)) are built.
4
+
5
+ Config space = SAME as the oracle's restarts: shape in {linear xp, nonlinear xp*(|xp|/5+0.3)}, K_hat in linspace(0.5,2.5,9). Tuned on a FRESH N=200 set (seed=555, disjoint from the held-out test set) by (certified_rate, success_rate); NOT re-tuned per plant (one fixed controller for the whole family). Cost = the paper's ABSOLUTE-weight J = integral (theta^2 + 0.1 omega^2) + u^2 dt (simulate_pendulum_batch default weights) -- not a scale-invariant metric.
6
+
7
+ Chosen fixed controller: shape=nonlinear K_hat=2.500 (phi(y) = 2.500 * m*g*l * shape(y))
8
+
9
+ success (empirical stability, N=40): 1.000
10
+ certified (regional LMI, N=40): 1.000
11
+ cost regret vs cached per-plant oracle: median +56.69% p90 +88.86% max +124.15% (n=40)
12
+ synthesis time: 81.2865 ms/plant (no per-plant solve -- table lookup + rescale)
13
+
14
+ top 5 tuning configs (shape, K_hat, certified_rate, success_rate) on the tuning set:
15
+ ('nonlinear', 2.5, 1.0, 1.0)
16
+ ('nonlinear', 2.25, 1.0, 1.0)
17
+ ('linear', 2.5, 1.0, 1.0)
18
+ ('linear', 2.25, 1.0, 1.0)
19
+ ('linear', 2.0, 1.0, 1.0)
results/baselines/pendulum_bank_density.txt ADDED
@@ -0,0 +1,36 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ === PENDULUM bank-density sweep ===
2
+ model = diffusion model TRAINED on the f-fraction subset (3 training seeds 42/43/44,
3
+ mean +/- sd across training seeds; ONE diffusion sampling seed=0 per model, K=8)
4
+ retrieval/interpolation = SAME f-fraction bank (built from the identical seeded subset
5
+ train_pendulum_ablation.py used), 3 seeds -> mean +/- sd, K=8
6
+ held-out sets: seed123 (cached oracle), seed321 (fresh oracle, run_pendulum_second_draw)
7
+
8
+ --- held-out set: seed123 ---
9
+ f arm coverage regret_med % regret_med 95%CI(seed0)
10
+ 0.25 model 1.000+-0.000 0.24+-0.22 [+0.41,+0.69]
11
+ 0.25 retrieval 1.000+-0.000 0.17+-0.01 [+0.01,+0.44]
12
+ 0.25 interpolation 1.000+-0.000 0.22+-0.05 [+0.02,+0.44]
13
+ 0.50 model 1.000+-0.000 0.98+-0.16 [+1.08,+1.33]
14
+ 0.50 retrieval 1.000+-0.000 0.30+-0.08 [+0.05,+0.41]
15
+ 0.50 interpolation 1.000+-0.000 0.30+-0.08 [+0.05,+0.41]
16
+ 1.00 model 1.000+-0.000 0.89+-0.21 [+0.93,+1.38]
17
+ 1.00 retrieval 1.000+-0.000 0.40+-0.00 [+0.22,+0.53]
18
+ 1.00 interpolation 1.000+-0.000 0.40+-0.00 [+0.22,+0.53]
19
+
20
+ --- held-out set: seed321 ---
21
+ f arm coverage regret_med % regret_med 95%CI(seed0)
22
+ 0.25 model 1.000+-0.000 0.26+-0.14 [+0.23,+0.72]
23
+ 0.25 retrieval 1.000+-0.000 0.24+-0.01 [+0.06,+0.33]
24
+ 0.25 interpolation 1.000+-0.000 0.25+-0.01 [+0.11,+0.37]
25
+ 0.50 model 1.000+-0.000 0.91+-0.16 [+0.85,+1.20]
26
+ 0.50 retrieval 1.000+-0.000 0.23+-0.06 [+0.04,+0.29]
27
+ 0.50 interpolation 1.000+-0.000 0.26+-0.05 [+0.04,+0.29]
28
+ 1.00 model 1.000+-0.000 0.87+-0.17 [+0.82,+1.24]
29
+ 1.00 retrieval 1.000+-0.000 0.29+-0.00 [+0.16,+0.36]
30
+ 1.00 interpolation 1.000+-0.000 0.30+-0.00 [+0.16,+0.42]
31
+
32
+ Crossover -- does the model's median regret ever drop BELOW retrieval's at the same f?
33
+ seed123: f=0.25: model +0.24% >= retrieval +0.17% (retrieval wins/ties); f=0.50: model +0.98% >= retrieval +0.30% (retrieval wins/ties); f=1.00: model +0.89% >= retrieval +0.40% (retrieval wins/ties)
34
+ seed321: f=0.25: model +0.26% >= retrieval +0.24% (retrieval wins/ties); f=0.50: model +0.91% >= retrieval +0.23% (retrieval wins/ties); f=1.00: model +0.87% >= retrieval +0.29% (retrieval wins/ties)
35
+ -> model overtakes retrieval's median regret at ANY swept f, on ANY held-out set: False
36
+ FINDING: the model never overtakes bank retrieval's cost on this sweep -- more data narrows the gap (higher f typically lowers both arms' regret) but does not flip the ordering within {0.25, 0.5, 1.0}.
results/baselines/pendulum_bank_density_diag_a.txt ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ === PENDULUM bank-density diagnostic (a): retrieval K=1 (single nearest neighbour, no best-of-K selection) vs f ===
2
+ If K=1 regret falls with f while best-of-8 (pendulum_bank_density.txt) does not, best-of-8's edge at low f is candidate DIVERSITY (a sparse bank's 8 nearest are spread out), not raw proximity -- a dense bank's 8 nearest become near-duplicates and best-of-8 stops gaining.
3
+
4
+ --- held-out set: seed123 ---
5
+ f K1 regret_med% K1 cov (3-seed mean +/- sd)
6
+ 0.25 1.71+-0.16 1.000
7
+ 0.50 1.63+-0.20 1.000
8
+ 1.00 1.84+-0.00 1.000
9
+
10
+ --- held-out set: seed321 ---
11
+ f K1 regret_med% K1 cov (3-seed mean +/- sd)
12
+ 0.25 1.19+-0.22 1.000
13
+ 0.50 1.42+-0.16 1.000
14
+ 1.00 1.36+-0.00 1.000
15
+
results/baselines/pendulum_bank_density_diag_b_seed123.txt ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ === PENDULUM bank-density diagnostic (b): native quality of stored targets (held-out seed=123, N=40) ===
2
+ For each held-out plant: find its nearest FULL-BANK (f=1.0, 2600 pts) training plant; evaluate that plant's OWN stored controller (spline_raw, as-stored, no re-decoding) ON THAT TRAINING PLANT'S OWN DYNAMICS (not the held-out query); regret vs a FRESH eval_pendulum.strong_oracle solve of that SAME training plant.
3
+ 39 unique nearest-neighbour training plants among the 40 held-out queries (some held-out plants share a nearest neighbour).
4
+
5
+ native target regret: median +1.40% p90 +2.67% max +3.58% (n=40/40)
6
+ nearest-neighbour distance (normalized space): median 0.1691 max 0.2201
7
+
8
+ Reading: if this native regret (~how good the DATASET's own targets are, solved once, reused everywhere) sits near retrieval's regret on the held-out query (bank_density.txt), retrieval is essentially reading back near-oracle targets with a small transfer penalty for the query-vs-neighbour distance; if the model's own regret (~0.9% at f=1.0) exceeds this native floor, the model is worse than its own training set accessed directly -- the same finding as Iteration 3 on the DP-target pendulum.
results/baselines/retrieval_mimo_seed7777.txt ADDED
@@ -0,0 +1,41 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ === MIMO retrieval/interpolation baselines (N=120, seed=7777) ===
2
+ verifier: simulate_closed_loop_mimo_batched rollout (n_ic=4, dt=0.03, duration=12.0, energy_thresh=0.3)
3
+ oracle: eval_mimo_regret.py's oracle protocol (fixed generic init 0.3*cap, 150-step batched Adam lr=0.08 through the differentiable MIMO sim, sector-projected) -- arm-independent, computed ONCE, shared by every arm (1187.0 ms/plant)
4
+ K_std=8, diffusion seeds=(0, 1, 2)
5
+
6
+ --- K=8 ---
7
+ arm coverage persamp reg_med% reg_p90% reg_max% ms/plant McNemar(n10,n01,p) Wilcoxon(n,p)
8
+ diffusion 0.99 0.94 2.31 9.23 30.13 153.77 (reference) (reference)
9
+ (sd across 3 seeds: coverage 0.004, per-samp 0.003)
10
+ retrieval 0.99 0.93 2.32 8.05 49.97 0.730 (1,0,1) (118,0.91)
11
+ (Wilcoxon n=118 = the COMMON success set: plants where BOTH this arm and diffusion have a finite/stable cost, not either arm's own coverage)
12
+ interpolation 0.99 0.93 2.95 15.27 74.15 0.730 (1,0,1) (118,0.049)
13
+ (Wilcoxon n=118 = the COMMON success set: plants where BOTH this arm and diffusion have a finite/stable cost, not either arm's own coverage)
14
+
15
+ --- K=1 ---
16
+ arm coverage persamp reg_med% reg_p90% reg_max% ms/plant McNemar(n10,n01,p) Wilcoxon(n,p)
17
+ diffusion 0.93 0.93 10.92 88.52 1045.64 153.77 (reference) (reference)
18
+ (sd across 3 seeds: coverage 0.014, per-samp 0.014)
19
+ retrieval 0.92 0.92 15.23 262.70 830.97 0.370 (5,5,1) (105,0.11)
20
+ (Wilcoxon n=105 = the COMMON success set: plants where BOTH this arm and diffusion have a finite/stable cost, not either arm's own coverage)
21
+ interpolation 0.92 0.92 15.23 262.70 830.97 0.370 (5,5,1) (105,0.11)
22
+ (Wilcoxon n=105 = the COMMON success set: plants where BOTH this arm and diffusion have a finite/stable cost, not either arm's own coverage)
23
+
24
+
25
+ --- addendum (post-hoc from npz, not re-simulated): CIs and pooled 2-draw (N=240) tests, diffusion seed[0] as reference ---
26
+ K=8 seed9999 diffusion: coverage 1.000 regret_med +2.32 [+1.63,+2.99]
27
+ K=8 seed9999 retrieval: coverage 0.992 regret_med +2.62 [+1.45,+3.45]
28
+ K=8 seed7777 diffusion: coverage 0.983 regret_med +2.30 [+1.90,+3.01]
29
+ K=8 seed7777 retrieval: coverage 0.992 regret_med +2.32 [+1.67,+3.06]
30
+ K=8 pooled McNemar coverage (retrieval-only, diffusion-only)=(1,1) p=1
31
+ K=8 pooled Wilcoxon cost diffusion vs retrieval on common set n=237: median J ratio 1.000 p=0.863
32
+ K=8 seed9999 diffusion coverage per draw: [1.0, 0.992, 1.0] retrieval 0.992
33
+ K=8 seed7777 diffusion coverage per draw: [0.983, 0.992, 0.983] retrieval 0.992
34
+ K=1 seed9999 diffusion: coverage 0.942 regret_med +8.87 [+6.14,+12.50]
35
+ K=1 seed9999 retrieval: coverage 0.858 regret_med +13.66 [+10.22,+19.86]
36
+ K=1 seed7777 diffusion: coverage 0.917 regret_med +9.85 [+7.63,+13.51]
37
+ K=1 seed7777 retrieval: coverage 0.917 regret_med +15.23 [+10.31,+20.11]
38
+ K=1 pooled McNemar coverage (retrieval-only, diffusion-only)=(5,15) p=0.0414
39
+ K=1 pooled Wilcoxon cost diffusion vs retrieval on common set n=208: median J ratio 0.982 p=0.0345
40
+ K=1 seed9999 diffusion coverage per draw: [0.942, 0.95, 0.942] retrieval 0.858
41
+ K=1 seed7777 diffusion coverage per draw: [0.917, 0.933, 0.95] retrieval 0.917
results/baselines/retrieval_objective_conditioning.txt ADDED
@@ -0,0 +1,129 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ RETRIEVAL vs OBJECTIVE-CONDITIONED GENERATION (pendulum)
2
+ N=40 plants/draw, rho=(0.7, 0.8, 0.9), K in (8,1), steer S=50
3
+ plant seeds: main=321 (exp_cbar_eval's), second=777; model torch seeds = seed*1000+rho_idx, seeds (0, 1, 2)
4
+ retr_b pool M=32. spec tolerance ise<=1.05*rho*ISE_ref (exp_cbar_eval satisfied_mask).
5
+
6
+ == Task 1: stream replica of exp_cbar_eval (seed 321) vs STORED results/pendulum/cbar_eval.json ==
7
+ rho=0.7: bestof8 replica met 1.000 cert 1.000 reg 0.84 | stored met 1.000 cert 1.000 reg 0.83 || single replica met 0.925 cert 0.925 reg 8.83 | stored met 0.950 cert 0.850 reg 8.25
8
+ rho=0.8: bestof8 replica met 1.000 cert 1.000 reg 0.61 | stored met 1.000 cert 1.000 reg 0.66 || single replica met 0.900 cert 0.850 reg 5.83 | stored met 0.900 cert 0.875 reg 6.17
9
+ rho=0.9: bestof8 replica met 1.000 cert 1.000 reg 0.90 | stored met 1.000 cert 1.000 reg 1.51 || single replica met 0.875 cert 0.850 reg 4.26 | stored met 0.850 cert 0.850 reg 5.22
10
+ oracle found per rho: [40, 40, 40]
11
+
12
+ --- MAIN DRAW seed 321: K=8 steer=none ---
13
+ rows: rho | arm | spec-met | certified | regret% median [95% CI] p90 n (model = mean±sd over 3 seeds; CI/p90 from seed 0)
14
+ [0.7 ] model met 1.000±0.000 cert 1.000±0.000 medreg 0.82±0.96 | seed0: 2.17 [ 0.66, 4.42] p90 11.80 n=40
15
+ [0.7 ] retr_a met 1.000 cert 1.000 medreg 1.05 [-0.13, 2.47] p90 9.54 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0793 | p over seeds (McN,Wil): 1,0.079; 1,0.29; 1,0.38
16
+ [0.7 ] retr_b met 1.000 cert 1.000 medreg 1.05 [ 0.16, 2.49] p90 8.38 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0548 | p over seeds (McN,Wil): 1,0.055; 1,0.83; 1,0.16
17
+ [0.7 ] retr_c met 1.000 cert 1.000 medreg 1.11 [-0.58, 2.07] p90 7.96 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.025 | p over seeds (McN,Wil): 1,0.025; 1,0.21; 1,0.17
18
+ [0.8 ] model met 1.000±0.000 cert 1.000±0.000 medreg 0.34±0.42 | seed0: -0.25 [-0.80, 0.98] p90 16.59 n=40
19
+ [0.8 ] retr_a met 1.000 cert 1.000 medreg 0.82 [-0.46, 2.05] p90 6.89 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.444 | p over seeds (McN,Wil): 1,0.44; 1,0.24; 1,0.082
20
+ [0.8 ] retr_b met 1.000 cert 1.000 medreg 0.38 [-0.79, 1.95] p90 9.84 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.334 | p over seeds (McN,Wil): 1,0.33; 1,0.23; 1,0.097
21
+ [0.8 ] retr_c met 1.000 cert 1.000 medreg 1.13 [-0.36, 1.78] p90 6.89 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.627 | p over seeds (McN,Wil): 1,0.63; 1,0.32; 1,0.11
22
+ [0.9 ] model met 1.000±0.000 cert 1.000±0.000 medreg 0.97±0.40 | seed0: 0.50 [ 0.23, 1.31] p90 14.98 n=40
23
+ [0.9 ] retr_a met 1.000 cert 1.000 medreg 0.60 [-0.28, 2.28] p90 9.58 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.347 | p over seeds (McN,Wil): 1,0.35; 1,0.034; 1,0.1
24
+ [0.9 ] retr_b met 1.000 cert 1.000 medreg 0.91 [-0.01, 2.72] p90 9.58 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.745 | p over seeds (McN,Wil): 1,0.74; 1,0.062; 1,0.28
25
+ [0.9 ] retr_c met 1.000 cert 1.000 medreg 0.14 [-0.69, 1.60] p90 8.28 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.109 | p over seeds (McN,Wil): 1,0.11; 1,0.022; 1,0.031
26
+ [pooled] model met 1.000±0.000 cert 1.000±0.000 medreg 0.73±0.11 | seed0: 0.65 [ 0.22, 1.19] p90 14.78 n=120
27
+ [pooled] retr_a met 1.000 cert 1.000 medreg 0.87 [ 0.21, 1.71] p90 9.56 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0556 | p over seeds (McN,Wil): 1,0.056; 1,0.015; 1,0.014
28
+ [pooled] retr_b met 1.000 cert 1.000 medreg 0.75 [ 0.21, 1.41] p90 9.56 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0608 | p over seeds (McN,Wil): 1,0.061; 1,0.066; 1,0.014
29
+ [pooled] retr_c met 1.000 cert 1.000 medreg 0.86 [-0.10, 1.56] p90 8.26 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0149 | p over seeds (McN,Wil): 1,0.015; 1,0.0092; 1,0.0024
30
+
31
+ --- MAIN DRAW seed 321: K=1 steer=none ---
32
+ rows: rho | arm | spec-met | certified | regret% median [95% CI] p90 n (model = mean±sd over 3 seeds; CI/p90 from seed 0)
33
+ [0.7 ] model met 0.908±0.012 cert 0.850±0.041 medreg 14.40±5.85 | seed0: 22.68 [ 8.02,36.94] p90 63.08 n=34
34
+ [0.7 ] retr_a met 0.700 cert 0.700 medreg 6.76 [ 2.03,12.26] p90 38.55 n=28 | vs model s0: McNemar 2-10 p=0.0386, Wilcoxon p=0.0395 | p over seeds (McN,Wil): 0.039,0.039; 0.057,0.037; 0.012,0.026
35
+ [0.7 ] retr_b met 0.675 cert 0.675 medreg 9.02 [ 2.90,13.68] p90 29.75 n=27 | vs model s0: McNemar 2-11 p=0.0225, Wilcoxon p=0.0384 | p over seeds (McN,Wil): 0.022,0.038; 0.022,0.052; 0.0063,0.038
36
+ [0.7 ] retr_c met 0.900 cert 0.900 medreg 8.51 [ 3.40,13.84] p90 31.05 n=36 | vs model s0: McNemar 3-3 p=1, Wilcoxon p=0.000906 | p over seeds (McN,Wil): 1,0.00091; 1,0.0013; 1,0.0052
37
+ [0.8 ] model met 0.883±0.051 cert 0.842±0.051 medreg 6.74±1.60 | seed0: 6.20 [ 3.96,15.82] p90 78.90 n=34
38
+ [0.8 ] retr_a met 0.775 cert 0.775 medreg 2.20 [ 0.87, 8.33] p90 24.33 n=31 | vs model s0: McNemar 4-8 p=0.388, Wilcoxon p=0.00247 | p over seeds (McN,Wil): 0.39,0.0025; 0.77,0.031; 0.065,0.06
39
+ [0.8 ] retr_b met 0.725 cert 0.725 medreg 9.86 [ 5.64,13.28] p90 29.73 n=29 | vs model s0: McNemar 4-10 p=0.18, Wilcoxon p=0.127 | p over seeds (McN,Wil): 0.18,0.13; 0.39,0.23; 0.012,0.69
40
+ [0.8 ] retr_c met 0.875 cert 0.875 medreg 5.78 [ 1.78,10.76] p90 27.94 n=35 | vs model s0: McNemar 3-3 p=1, Wilcoxon p=0.000398 | p over seeds (McN,Wil): 1,0.0004; 0.73,0.099; 0.38,0.074
41
+ [0.9 ] model met 0.892±0.031 cert 0.833±0.051 medreg 6.74±0.36 | seed0: 6.29 [ 4.54,11.19] p90 55.91 n=33
42
+ [0.9 ] retr_a met 0.650 cert 0.650 medreg 5.75 [ 4.02,17.38] p90 49.79 n=26 | vs model s0: McNemar 2-10 p=0.0386, Wilcoxon p=0.259 | p over seeds (McN,Wil): 0.039,0.26; 0.013,0.07; 0.0074,0.43
43
+ [0.9 ] retr_b met 0.700 cert 0.700 medreg 11.11 [ 4.49,17.55] p90 35.66 n=28 | vs model s0: McNemar 5-11 p=0.21, Wilcoxon p=0.427 | p over seeds (McN,Wil): 0.21,0.43; 0.057,0.14; 0.012,0.8
44
+ [0.9 ] retr_c met 0.825 cert 0.825 medreg 4.87 [ 1.64, 9.74] p90 31.94 n=33 | vs model s0: McNemar 4-5 p=1, Wilcoxon p=0.0564 | p over seeds (McN,Wil): 1,0.056; 0.55,0.015; 0.29,0.067
45
+ [pooled] model met 0.894±0.027 cert 0.842±0.048 medreg 8.03±0.52 | seed0: 8.62 [ 5.44,13.61] p90 64.20 n=101
46
+ [pooled] retr_a met 0.708 cert 0.708 medreg 6.09 [ 3.64, 8.44] p90 39.49 n=85 | vs model s0: McNemar 8-28 p=0.00119, Wilcoxon p=0.000397 | p over seeds (McN,Wil): 0.0012,0.0004; 0.0022,0.00066; 7.4e-06,0.004
47
+ [pooled] retr_b met 0.700 cert 0.700 medreg 9.73 [ 7.33,13.04] p90 32.05 n=84 | vs model s0: McNemar 11-32 p=0.00191, Wilcoxon p=0.00733 | p over seeds (McN,Wil): 0.0019,0.0073; 0.0011,0.0036; 7.7e-07,0.079
48
+ [pooled] retr_c met 0.867 cert 0.867 medreg 6.78 [ 3.92, 9.18] p90 30.74 n=104 | vs model s0: McNemar 10-11 p=1, Wilcoxon p=1.22e-06 | p over seeds (McN,Wil): 1,1.2e-06; 1,2.3e-05; 0.096,0.00013
49
+
50
+ --- MAIN DRAW seed 321: K=8 steer=S=50 ---
51
+ rows: rho | arm | spec-met | certified | regret% median [95% CI] p90 n (model = mean±sd over 3 seeds; CI/p90 from seed 0)
52
+ [0.7 ] model met 1.000±0.000 cert 1.000±0.000 medreg 0.17±0.05 | seed0: 0.23 [ 0.01, 0.75] p90 4.35 n=40
53
+ [0.7 ] retr_a met 1.000 cert 1.000 medreg -0.13 [-0.48, 0.15] p90 1.27 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0725 | p over seeds (McN,Wil): 1,0.073; 1,3.5e-05; 1,0.0056
54
+ [0.7 ] retr_b met 1.000 cert 1.000 medreg -0.22 [-1.04, 0.14] p90 1.56 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.00726 | p over seeds (McN,Wil): 1,0.0073; 1,0.0018; 1,1.7e-06
55
+ [0.7 ] retr_c met 1.000 cert 1.000 medreg -0.04 [-0.19, 0.23] p90 1.20 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.077 | p over seeds (McN,Wil): 1,0.077; 1,1.7e-05; 1,0.0067
56
+ [0.8 ] model met 1.000±0.000 cert 1.000±0.000 medreg -0.03±0.01 | seed0: -0.02 [-0.06, 0.00] p90 11.62 n=40
57
+ [0.8 ] retr_a met 1.000 cert 1.000 medreg -0.15 [-0.38,-0.08] p90 0.11 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=1.81e-05 | p over seeds (McN,Wil): 1,1.8e-05; 1,5.1e-05; 1,0.00014
58
+ [0.8 ] retr_b met 1.000 cert 1.000 medreg -0.14 [-0.42,-0.07] p90 0.09 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=8.15e-06 | p over seeds (McN,Wil): 1,8.2e-06; 1,3.3e-05; 1,0.0001
59
+ [0.8 ] retr_c met 1.000 cert 1.000 medreg -0.20 [-0.45,-0.10] p90 2.02 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.000603 | p over seeds (McN,Wil): 1,0.0006; 1,0.0013; 1,0.00025
60
+ [0.9 ] model met 1.000±0.000 cert 1.000±0.000 medreg -0.03±0.01 | seed0: -0.02 [-0.09, 0.03] p90 9.95 n=40
61
+ [0.9 ] retr_a met 1.000 cert 1.000 medreg -0.11 [-0.26,-0.06] p90 0.86 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.00558 | p over seeds (McN,Wil): 1,0.0056; 1,0.0012; 1,0.015
62
+ [0.9 ] retr_b met 1.000 cert 1.000 medreg -0.13 [-0.27,-0.04] p90 5.73 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.00305 | p over seeds (McN,Wil): 1,0.0031; 1,0.0016; 1,0.017
63
+ [0.9 ] retr_c met 1.000 cert 1.000 medreg -0.10 [-0.25,-0.07] p90 0.86 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.00726 | p over seeds (McN,Wil): 1,0.0073; 1,0.0014; 1,0.028
64
+ [pooled] model met 1.000±0.000 cert 1.000±0.000 medreg -0.01±0.00 | seed0: -0.00 [-0.03, 0.12] p90 10.09 n=120
65
+ [pooled] retr_a met 1.000 cert 1.000 medreg -0.13 [-0.21,-0.09] p90 1.20 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=4.26e-07 | p over seeds (McN,Wil): 1,4.3e-07; 1,2.3e-10; 1,3.4e-07
66
+ [pooled] retr_b met 1.000 cert 1.000 medreg -0.15 [-0.28,-0.09] p90 1.56 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=1.91e-08 | p over seeds (McN,Wil): 1,1.9e-08; 1,1.1e-08; 1,7.5e-10
67
+ [pooled] retr_c met 1.000 cert 1.000 medreg -0.12 [-0.20,-0.08] p90 1.33 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=4.94e-06 | p over seeds (McN,Wil): 1,4.9e-06; 1,1.4e-09; 1,9.8e-07
68
+
69
+ --- MAIN DRAW seed 321: K=1 steer=S=50 ---
70
+ rows: rho | arm | spec-met | certified | regret% median [95% CI] p90 n (model = mean±sd over 3 seeds; CI/p90 from seed 0)
71
+ [0.7 ] model met 0.992±0.012 cert 0.942±0.042 medreg 5.05±3.48 | seed0: 9.82 [ 0.63,28.76] p90 51.33 n=36
72
+ [0.7 ] retr_a met 0.925 cert 0.900 medreg 0.75 [ 0.18, 1.64] p90 26.67 n=36 | vs model s0: McNemar 1-3 p=0.625, Wilcoxon p=0.0105 | p over seeds (McN,Wil): 0.62,0.011; 0.25,0.023; 0.25,0.0063
73
+ [0.7 ] retr_b met 0.875 cert 0.850 medreg 1.03 [ 0.37, 4.53] p90 21.92 n=34 | vs model s0: McNemar 1-5 p=0.219, Wilcoxon p=0.0149 | p over seeds (McN,Wil): 0.22,0.015; 0.062,0.029; 0.062,0.015
74
+ [0.7 ] retr_c met 1.000 cert 1.000 medreg 1.13 [ 0.36, 4.11] p90 21.78 n=40 | vs model s0: McNemar 1-0 p=1, Wilcoxon p=0.00185 | p over seeds (McN,Wil): 1,0.0018; 1,0.0051; 1,0.0019
75
+ [0.8 ] model met 0.992±0.012 cert 0.933±0.012 medreg 0.37±0.22 | seed0: 0.48 [ 0.27, 5.73] p90 55.20 n=37
76
+ [0.8 ] retr_a met 0.975 cert 0.975 medreg 0.03 [ 0.00, 0.25] p90 18.99 n=39 | vs model s0: McNemar 1-1 p=1, Wilcoxon p=0.000159 | p over seeds (McN,Wil): 1,0.00016; 1,0.0031; 1,0.17
77
+ [0.8 ] retr_b met 0.850 cert 0.850 medreg 0.43 [-0.04, 1.84] p90 15.47 n=34 | vs model s0: McNemar 1-6 p=0.125, Wilcoxon p=0.0175 | p over seeds (McN,Wil): 0.12,0.017; 0.031,0.058; 0.031,0.66
78
+ [0.8 ] retr_c met 1.000 cert 1.000 medreg 0.14 [ 0.02, 1.23] p90 22.30 n=40 | vs model s0: McNemar 1-0 p=1, Wilcoxon p=3.36e-06 | p over seeds (McN,Wil): 1,3.4e-06; 1,0.051; 1,0.14
79
+ [0.9 ] model met 0.992±0.012 cert 0.950±0.054 medreg 0.54±0.31 | seed0: 0.89 [ 0.20, 1.57] p90 36.20 n=40
80
+ [0.9 ] retr_a met 0.975 cert 0.975 medreg 0.11 [-0.00, 0.61] p90 25.81 n=39 | vs model s0: McNemar 0-1 p=1, Wilcoxon p=0.0372 | p over seeds (McN,Wil): 1,0.037; 1,0.0091; 1,0.47
81
+ [0.9 ] retr_b met 0.925 cert 0.925 medreg 0.18 [ 0.03, 1.11] p90 26.47 n=37 | vs model s0: McNemar 0-3 p=0.25, Wilcoxon p=0.0363 | p over seeds (McN,Wil): 0.25,0.036; 0.62,0.037; 0.25,0.61
82
+ [0.9 ] retr_c met 1.000 cert 1.000 medreg 0.11 [ 0.04, 0.53] p90 26.30 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.00336 | p over seeds (McN,Wil): 1,0.0034; 1,0.012; 1,0.38
83
+ [pooled] model met 0.992±0.007 cert 0.942±0.027 medreg 0.70±0.23 | seed0: 0.97 [ 0.43, 1.96] p90 51.49 n=113
84
+ [pooled] retr_a met 0.958 cert 0.950 medreg 0.18 [ 0.06, 0.46] p90 24.60 n=114 | vs model s0: McNemar 2-5 p=0.453, Wilcoxon p=6.12e-06 | p over seeds (McN,Wil): 0.45,6.1e-06; 0.22,1.3e-05; 0.062,0.003
85
+ [pooled] retr_b met 0.883 cert 0.875 medreg 0.49 [ 0.18, 1.11] p90 23.23 n=105 | vs model s0: McNemar 2-14 p=0.00418, Wilcoxon p=6.01e-05 | p over seeds (McN,Wil): 0.0042,6e-05; 0.00098,0.00026; 0.00012,0.037
86
+ [pooled] retr_c met 1.000 cert 1.000 medreg 0.35 [ 0.14, 0.82] p90 23.85 n=120 | vs model s0: McNemar 2-0 p=0.5, Wilcoxon p=2.01e-08 | p over seeds (McN,Wil): 0.5,2e-08; 1,3.2e-05; 1,0.0015
87
+
88
+ synthesis time per plant (ms; mean over rho; verification NOT included):
89
+ model:s0 steer 258.21
90
+ model:s0 synth 89.52
91
+ model:s1 steer 270.89
92
+ model:s1 synth 91.89
93
+ model:s2 steer 267.04
94
+ model:s2 synth 89.57
95
+ retr_a steer 278.56
96
+ retr_a synth 0.08
97
+ retr_b steer 271.39
98
+ retr_b synth 0.09
99
+ retr_c steer 253.12
100
+ retr_c synth 0.14
101
+
102
+ oracle found per rho (draw 777): [40, 40, 40]
103
+
104
+ --- SECOND DRAW seed 777: K=8 steer=none ---
105
+ rows: rho | arm | spec-met | certified | regret% median [95% CI] p90 n (model = mean±sd over 3 seeds; CI/p90 from seed 0)
106
+ [0.7 ] model met 1.000±0.000 cert 1.000±0.000 medreg 1.13±0.34 | seed0: 1.23 [-0.22, 3.18] p90 11.81 n=40
107
+ [0.7 ] retr_a met 1.000 cert 1.000 medreg 0.94 [-0.93, 3.71] p90 12.46 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.952 | p over seeds (McN,Wil): 1,0.95; 1,0.045; 1,0.79
108
+ [0.7 ] retr_b met 1.000 cert 1.000 medreg 1.10 [-0.21, 2.94] p90 9.35 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.32 | p over seeds (McN,Wil): 1,0.32; 1,0.51; 1,0.25
109
+ [0.7 ] retr_c met 1.000 cert 1.000 medreg 0.50 [-0.93, 3.02] p90 11.93 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.847 | p over seeds (McN,Wil): 1,0.85; 1,0.12; 1,0.51
110
+ [0.8 ] model met 1.000±0.000 cert 1.000±0.000 medreg 0.97±0.29 | seed0: 0.62 [ 0.01, 1.74] p90 15.47 n=40
111
+ [0.8 ] retr_a met 1.000 cert 1.000 medreg 0.67 [-0.05, 2.42] p90 5.18 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.347 | p over seeds (McN,Wil): 1,0.35; 1,0.18; 1,0.024
112
+ [0.8 ] retr_b met 1.000 cert 1.000 medreg 1.31 [ 0.23, 2.61] p90 5.59 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.361 | p over seeds (McN,Wil): 1,0.36; 1,0.25; 1,0.15
113
+ [0.8 ] retr_c met 1.000 cert 1.000 medreg 0.44 [-0.16, 1.69] p90 5.43 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.277 | p over seeds (McN,Wil): 1,0.28; 1,0.14; 1,0.0094
114
+ [0.9 ] model met 1.000±0.000 cert 1.000±0.000 medreg 1.05±0.05 | seed0: 1.08 [ 0.46, 2.54] p90 8.63 n=40
115
+ [0.9 ] retr_a met 1.000 cert 1.000 medreg 0.64 [-0.44, 1.57] p90 7.26 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.125 | p over seeds (McN,Wil): 1,0.12; 1,0.094; 1,0.71
116
+ [0.9 ] retr_b met 1.000 cert 1.000 medreg 0.65 [-0.34, 1.66] p90 7.14 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.132 | p over seeds (McN,Wil): 1,0.13; 1,0.16; 1,0.53
117
+ [0.9 ] retr_c met 0.975 cert 0.975 medreg 0.50 [-0.35, 1.24] p90 5.51 n=39 | vs model s0: McNemar 0-1 p=1, Wilcoxon p=0.124 | p over seeds (McN,Wil): 1,0.12; 1,0.1; 1,0.69
118
+ [pooled] model met 1.000±0.000 cert 1.000±0.000 medreg 0.96±0.11 | seed0: 0.92 [ 0.43, 1.59] p90 11.90 n=120
119
+ [pooled] retr_a met 1.000 cert 1.000 medreg 0.80 [ 0.06, 1.32] p90 9.77 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.155 | p over seeds (McN,Wil): 1,0.16; 1,0.61; 1,0.097
120
+ [pooled] retr_b met 1.000 cert 1.000 medreg 0.99 [ 0.14, 1.57] p90 8.49 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0619 | p over seeds (McN,Wil): 1,0.062; 1,0.29; 1,0.062
121
+ [pooled] retr_c met 0.992 cert 0.992 medreg 0.50 [ 0.01, 1.01] p90 9.27 n=119 | vs model s0: McNemar 0-1 p=1, Wilcoxon p=0.102 | p over seeds (McN,Wil): 1,0.1; 1,0.29; 1,0.037
122
+
123
+ synthesis time per plant (ms; mean over rho; verification NOT included):
124
+ model:s0 synth 91.34
125
+ model:s1 synth 90.86
126
+ model:s2 synth 91.90
127
+ retr_a synth 0.08
128
+ retr_b synth 0.11
129
+ retr_c synth 0.11
results/baselines/retrieval_pendulum.txt ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ === PENDULUM retrieval/interpolation baselines (N=40, seed=123) ===
2
+ verifier: certify_controller (regional LMI) + simulate_pendulum_batch rollout (n_ic=3, dt=0.02, duration=6.0, energy_thresh=0.02, theta0=0.35)
3
+ oracle: CACHED results/pendulum/eval.npz (eval_pendulum.strong_oracle, 24 restarts x 100 steps), oracle_J backed out from best_costs/regrets
4
+ K_std=8, diffusion seeds=(0, 1, 2)
5
+
6
+ --- K=8 ---
7
+ arm cert_any stable_any persamp reg_med% reg_p90% reg_max% ms/plant McNemar(n10,n01,p) Wilcoxon(n,p)
8
+ diffusion 1.00 1.00 1.00 1.09 2.46 3.36 171.13 (reference) (reference)
9
+ (sd across 3 seeds: cert_any 0.000, stable_any 0.000, per-samp 0.001)
10
+ retrieval 1.00 1.00 1.00 0.40 0.86 1.70 0.375 (0,0,1) (40,2.8e-07)
11
+ interpolation 1.00 1.00 1.00 0.40 0.86 1.70 0.375 (0,0,1) (40,2.8e-07)
12
+
13
+ --- K=1 ---
14
+ arm cert_any stable_any persamp reg_med% reg_p90% reg_max% ms/plant McNemar(n10,n01,p) Wilcoxon(n,p)
15
+ diffusion 1.00 1.00 1.00 1.54 3.59 4.80 171.13 (reference) (reference)
16
+ (sd across 3 seeds: cert_any 0.000, stable_any 0.000, per-samp 0.000)
17
+ retrieval 1.00 1.00 1.00 1.84 3.70 5.85 0.127 (0,0,1) (40,0.13)
18
+ interpolation 1.00 1.00 1.00 1.84 3.70 5.85 0.127 (0,0,1) (40,0.13)
19
+
20
+
21
+ --- K=8 bootstrap 95% CIs (addendum, computed post-hoc from saved npz; not re-simulated) ---
22
+ diffusion (seed[0]): coverage 95% CI [1.000,1.000] regret_med 95% CI [+0.88,+1.35]%
23
+ retrieval: coverage 95% CI [1.000,1.000] regret_med 95% CI [+0.22,+0.53]%
24
+ interpolation: coverage 95% CI [1.000,1.000] regret_med 95% CI [+0.22,+0.53]%
25
+ --- K=1 bootstrap 95% CIs (addendum, computed post-hoc from saved npz; not re-simulated) ---
26
+ diffusion (seed[0]): coverage 95% CI [1.000,1.000] regret_med 95% CI [+1.33,+2.14]%
27
+ retrieval: coverage 95% CI [1.000,1.000] regret_med 95% CI [+1.37,+2.64]%
28
+ interpolation: coverage 95% CI [1.000,1.000] regret_med 95% CI [+1.37,+2.64]%
results/baselines/retrieval_pendulum_seed321.txt ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ === PENDULUM retrieval/interpolation baselines (N=40, seed=321) ===
2
+ verifier: certify_controller (regional LMI) + simulate_pendulum_batch rollout (n_ic=3, dt=0.02, duration=6.0, energy_thresh=0.02)
3
+ oracle: FRESH (not previously cached): eval_pendulum.strong_oracle (24 restarts x 100 steps), computed this run and cached to results/baselines/pendulum_oracle_seed321.npz
4
+ K_std=8, diffusion seeds=(0, 1, 2)
5
+
6
+ --- K=8 ---
7
+ arm cert_any stable_any persamp reg_med% reg_p90% reg_max% ms/plant McNemar(n10,n01,p) Wilcoxon(n,p)
8
+ diffusion 1.00 1.00 1.00 0.97 2.21 3.17 283.48 (reference) (reference)
9
+ (sd across 3 seeds: cert_any 0.000, stable_any 0.000, per-samp 0.000)
10
+ (seed[0] 95% CI: stable_any [1.00,1.00] regret_med [+0.75,+1.31]%)
11
+ retrieval 1.00 1.00 1.00 0.29 0.64 0.86 0.373 (0,0,1) (40,1e-07)
12
+ (Wilcoxon n=40 = plants where BOTH this arm and diffusion have a finite cost, i.e. the COMMON success set, not either arm's own coverage count)
13
+ (95% CI: stable_any [1.00,1.00] regret_med [+0.16,+0.36]%)
14
+ interpolation 1.00 1.00 1.00 0.30 0.67 2.01 0.373 (0,0,1) (40,1e-07)
15
+ (Wilcoxon n=40 = plants where BOTH this arm and diffusion have a finite cost, i.e. the COMMON success set, not either arm's own coverage count)
16
+ (95% CI: stable_any [1.00,1.00] regret_med [+0.16,+0.42]%)
17
+
18
+ --- K=1 ---
19
+ arm cert_any stable_any persamp reg_med% reg_p90% reg_max% ms/plant McNemar(n10,n01,p) Wilcoxon(n,p)
20
+ diffusion 1.00 1.00 1.00 1.55 3.37 5.05 283.48 (reference) (reference)
21
+ (sd across 3 seeds: cert_any 0.000, stable_any 0.000, per-samp 0.000)
22
+ (seed[0] 95% CI: stable_any [1.00,1.00] regret_med [+1.44,+1.94]%)
23
+ retrieval 1.00 1.00 1.00 1.36 2.68 5.20 0.128 (0,0,1) (40,0.055)
24
+ (Wilcoxon n=40 = plants where BOTH this arm and diffusion have a finite cost, i.e. the COMMON success set, not either arm's own coverage count)
25
+ (95% CI: stable_any [1.00,1.00] regret_med [+1.14,+1.81]%)
26
+ interpolation 1.00 1.00 1.00 1.36 2.68 5.20 0.128 (0,0,1) (40,0.055)
27
+ (Wilcoxon n=40 = plants where BOTH this arm and diffusion have a finite cost, i.e. the COMMON success set, not either arm's own coverage count)
28
+ (95% CI: stable_any [1.00,1.00] regret_med [+1.14,+1.81]%)
29
+
results/baselines/retrieval_siso.txt ADDED
@@ -0,0 +1,36 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ === SISO retrieval/interpolation baselines (N=60, seed=0) ===
2
+ verifier: batched RK4 rollout (nonlinear.costs.diff_realize machinery, family's actual q_state -- see module docstring), padded NSTATE=8, shared x0 (seed=777), n_ic=4, dt=0.03, duration=12.0, energy_thresh=0.3
3
+ oracle: NOT a single global oracle. generalize.py's SISO oracle protocol (60-step Adam, phi-only, sector-projected, lr=0.06) is warm-started from a candidate by construction, so we apply it IDENTICALLY to each arm's own best candidate, batched across plants (documented adaptation -- see docs/retrieval_baselines.md). Primary comparison is the direct paired cov/cost test vs the diffusion arm below, which needs no oracle.
4
+ K_std=8, diffusion seeds=(0, 1, 2)
5
+
6
+ --- K=8 ---
7
+ arm coverage persamp reg_med% reg_p90% reg_max% ms/plant McNemar(n10,n01,p) Wilcoxon(n,p)
8
+ diffusion 0.79 0.63 10.17 44.69 121.17 21.53 (reference) (reference)
9
+ (sd across 3 seeds: coverage 0.008, per-samp 0.010; seed[0] 95% CI: coverage [0.67,0.88] regret_med [+7.04,+20.00]%)
10
+ retrieval 0.52 0.52 0.16 4.42 61.94 6.665 (0,16,3.1e-05) (31,0.018)
11
+ (95% CI: coverage [0.40,0.65] regret_med [+0.05,+0.34]%)
12
+ interpolation 0.52 0.52 0.16 4.42 61.94 6.665 (0,16,3.1e-05) (31,0.018)
13
+ (95% CI: coverage [0.40,0.65] regret_med [+0.05,+0.34]%)
14
+ (regressor is a point estimate, budget-independent -- reported once, at K=1)
15
+
16
+ --- K=1 ---
17
+ arm coverage persamp reg_med% reg_p90% reg_max% ms/plant McNemar(n10,n01,p) Wilcoxon(n,p)
18
+ diffusion 0.67 0.67 9.02 69.47 161.30 21.53 (reference) (reference)
19
+ (sd across 3 seeds: coverage 0.021, per-samp 0.021; seed[0] 95% CI: coverage [0.53,0.77] regret_med [+6.18,+14.24]%)
20
+ retrieval 0.52 0.52 0.16 4.42 61.94 6.789 (3,11,0.057) (28,0.00058)
21
+ (95% CI: coverage [0.40,0.65] regret_med [+0.05,+0.34]%)
22
+ interpolation 0.52 0.52 0.16 4.42 61.94 6.789 (3,11,0.057) (28,0.00058)
23
+ (95% CI: coverage [0.40,0.65] regret_med [+0.05,+0.34]%)
24
+ regressor 0.75 0.75 63.96 88.20 125.65 0.299 (8,2,0.11) (37,2.2e-05)
25
+ (95% CI: coverage [0.63,0.85] regret_med [+54.57,+71.82]%)
26
+
27
+
28
+ --- addendum: confirming the success set used by the Wilcoxon cost test above (coordinator follow-up #3) ---
29
+ K=8: diffusion(seed0) covers 47/60 plants
30
+ retrieval: covers 31/60; common-with-diffusion=31 (= a strict subset of diffusion coverage)
31
+ interpolation: covers 31/60; common-with-diffusion=31 (= a strict subset of diffusion coverage)
32
+ K=1: diffusion(seed0) covers 39/60 plants
33
+ retrieval: covers 31/60; common-with-diffusion=28 (NOT a subset -- some plants only this arm covers)
34
+ interpolation: covers 31/60; common-with-diffusion=28 (NOT a subset -- some plants only this arm covers)
35
+ regressor: covers 45/60; common-with-diffusion=37 (NOT a subset -- some plants only this arm covers)
36
+ The Wilcoxon n reported in the table above IS this common-coverage count in every row (wilcoxon_test restricts to np.isfinite(a)&np.isfinite(b) before testing, i.e. plants where BOTH arms have a finite/stable cost) -- confirmed by direct recomputation from the saved per-plant arrays, not just re-reading the code. For K=8 retrieval/interpolation, n=31 exactly equals retrieval's own coverage count because every plant retrieval covers is ALSO covered by diffusion (a strict subset) -- so the reported regret numbers for retrieval/interpolation are simultaneously "conditional on own success set" AND "the common success set" at K=8.
results/baselines/siso_run.log ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [seed 0] done
2
+ [seed 1] done
3
+ [seed 2] done
4
+ [seed 3] done
5
+ [seed 4] done
6
+
7
+ === SISO BASELINES ===
8
+ SISO why-generative baseline — pooled over 5 seeds × 60 plants (N=300), K=8
9
+
10
+ regressor : plant-stabilisable 0.72 [0.67,0.77] per-sample-stable 0.72
11
+ diff_1shot : plant-stabilisable 0.61 [0.56,0.67] per-sample-stable 0.61
12
+ diff_Kshot : plant-stabilisable 0.75 [0.70,0.80] per-sample-stable 0.61
13
+
14
+ diffusion K-shot vs regressor (paired McNemar-ish sign test): p = 3.09e-02
15
+ controller diversity (per-point std across K diffusion samples): 3.4 (regressor = 0, a single point)
16
+
17
+ TAKEAWAY: the regressor (conditional mean) stabilises fewer plants than diffusion sampled
18
+ K times; the generative DISTRIBUTION is what lets test-time selection/repair find a
19
+ stabilising controller a point estimate misses.
results/furuta_v3/deploy_sidebyside.txt ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ deployment mismatch, K=8, nominal-conditioned generated+P (mean+-sd over 3 sampler seeds) vs retrieval+P and NMPC (reused, n=120 plants)
2
+ scenario retr NMPC v2 clipped v2 clip5 v3 v3 e2400 v3 vs retr (b-c,p) v3 vs v2clip (b-c,p) e2400 vs retr (b-c,p)
3
+ nominal 90.0% 95.8% 65.3+-0.4 85.8+-0.7 86.1+-1.6 91.4+-0.8 6-4 p=0.7539 29-1 p=0.0000 5-6 p=1.0000
4
+ fric0.6 86.7% 95.8% 62.5+-0.0 80.8+-2.7 84.4+-0.4 91.1+-1.7 10-8 p=0.8145 29-2 p=0.0000 8-11 p=0.6476
5
+ fric1.6 9.2% 64.2% 10.6+-0.8 11.1+-0.4 9.7+-1.4 10.3+-0.8 1-4 p=0.3750 4-4 p=1.0000 3-3 p=1.0000
6
+ fric2.5 0.0% 29.2% 0.0+-0.0 0.6+-0.4 0.3+-0.4 0.0+-0.0 0-1 p=1.0000 1-0 p=1.0000 0-0 p=1.0000
7
+ ml_mp-_Lp- 68.3% 94.2% 50.0+-4.5 63.6+-0.4 69.4+-2.4 70.8+-1.4 10-8 p=0.8145 29-8 p=0.0008 8-11 p=0.6476
8
+ ml_mp-_Lp+ 81.7% 95.8% 61.7+-0.0 77.2+-1.0 83.3+-0.7 89.2+-1.4 10-12 p=0.8318 31-5 p=0.0000 9-16 p=0.2295
9
+ ml_mp+_Lp- 48.3% 95.0% 39.7+-2.7 53.1+-3.5 47.8+-2.1 51.4+-2.0 15-15 p=1.0000 25-11 p=0.0288 14-16 p=0.8555
10
+ ml_mp+_Lp+ 75.8% 95.0% 57.5+-1.2 72.2+-2.1 78.3+-1.4 82.8+-0.4 15-18 p=0.7283 33-6 p=0.0000 11-19 p=0.2005
11
+ both_f1.6_mp-_Lp- 9.2% 55.0% 11.7+-0.7 13.9+-0.4 11.9+-2.1 11.9+-1.4 3-9 p=0.1460 7-3 p=0.3438 3-7 p=0.3438
12
+ both_f1.6_mp-_Lp+ 19.2% 75.0% 17.8+-1.0 17.8+-1.7 18.6+-1.4 18.9+-1.7 4-5 p=1.0000 6-2 p=0.2891 5-5 p=1.0000
13
+ both_f1.6_mp+_Lp- 0.8% 52.5% 0.8+-0.0 0.8+-0.0 0.8+-0.0 0.6+-0.8 1-1 p=1.0000 1-1 p=1.0000 1-0 p=1.0000
14
+ both_f1.6_mp+_Lp+ 1.7% 67.5% 2.5+-0.7 3.3+-0.7 1.7+-0.7 2.2+-1.4 0-1 p=1.0000 3-3 p=1.0000 0-3 p=0.2500
15
+ both_f2.5_mp-_Lp- 0.0% 21.7% 0.3+-0.4 0.0+-0.0 0.0+-0.0 0.3+-0.4 0-0 p=1.0000 0-0 p=1.0000 0-0 p=1.0000
16
+ both_f2.5_mp-_Lp+ 0.0% 33.3% 1.1+-0.4 0.6+-0.8 0.3+-0.4 0.0+-0.0 0-0 p=1.0000 0-1 p=1.0000 0-0 p=1.0000
17
+ both_f2.5_mp+_Lp- 0.0% 18.3% 0.0+-0.0 0.3+-0.4 0.3+-0.4 0.0+-0.0 0-0 p=1.0000 0-0 p=1.0000 0-0 p=1.0000
18
+ both_f2.5_mp+_Lp+ 0.0% 33.3% 0.0+-0.0 0.3+-0.4 0.0+-0.0 0.0+-0.0 0-0 p=1.0000 0-0 p=1.0000 0-0 p=1.0000
19
+ D5 (diffusion mean >= retrieval mean in some scenario): max over scenarios of (diffusion - retrieval), points: v2 clipped: +2.5, v2 clip5: +4.7, v3: +2.8, v3 e2400: +7.5
results/furuta_v3/ksweep_sidebyside.txt ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ generated+P pooled over both draws (mean of 3 sampler seeds) K=1 K=4 K=8 K=16
2
+ v2 clipped (shipped) 33.9% 58.9% 65.2% 68.0%
3
+ v2 clip +-5 (audit) 34.1% 75.0% 85.8% 90.0%
4
+ v3 s0 800ep 38.3% 75.8% 86.1% 88.6%
5
+ v3 s1 800ep 31.6% 71.4% 81.9% 86.4%
6
+ v3 s0 2400ep 53.6% 84.8% 91.3% 93.7%
7
+ v3 s1 2400ep 58.4% 87.5% 92.5% 93.3%
8
+ v2-data clip5 2400ep (control) 60.8% 87.0% 90.6% 91.9%
9
+ nearest+P (retrieval, same bank) 54.2% 82.5% 90.0% 93.3%
results/furuta_v3/traj_gen_eval_v3_raw_clip1.txt ADDED
@@ -0,0 +1,94 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ bank data/furuta_nmpc_bank3000.npz: 2801 entries
2
+ model models/Best_furuta_traj_cond_v3.pth: cond_dim=4 out_dim=116 hidden=256 layers=5 device=cpu
3
+ [seed4242] margin=0.3: resolving P for 227 unique bank source plants
4
+ [seed4242] sampler seed 0: 9.6s for 60x4 candidates (0 decode-invalid of 240)
5
+ [seed4242] sampler seed 1: 9.5s for 60x4 candidates (0 decode-invalid of 240)
6
+ [seed4242] sampler seed 2: 9.6s for 60x4 candidates (0 decode-invalid of 240)
7
+ [seed4242] 10/60 (31s)
8
+ [seed4242] 20/60 (46s)
9
+ [seed4242] 30/60 (72s)
10
+ [seed4242] 40/60 (88s)
11
+ [seed4242] 50/60 (108s)
12
+ [seed4242] 60/60 (137s)
13
+ [seed9001] margin=0.3: resolving P for 232 unique bank source plants
14
+ [seed9001] sampler seed 0: 10.5s for 60x4 candidates (0 decode-invalid of 240)
15
+ [seed9001] sampler seed 1: 10.7s for 60x4 candidates (0 decode-invalid of 240)
16
+ [seed9001] sampler seed 2: 10.2s for 60x4 candidates (0 decode-invalid of 240)
17
+ [seed9001] 10/60 (33s)
18
+ [seed9001] 20/60 (57s)
19
+ [seed9001] 30/60 (80s)
20
+ [seed9001] 40/60 (101s)
21
+ [seed9001] 50/60 (124s)
22
+ [seed9001] 60/60 (153s)
23
+
24
+ === traj_gen_eval (v3_raw_clip1): generated vs retrieval vs NMPC ===
25
+
26
+ --- plant set seed4242 (seed 4242, n=60) ---
27
+ arm succ 95% CI medJ med regret p90 synth
28
+ per-plant 93% [ 87, 98] 4.08 0.00% 0.00% 4.6s
29
+ nearest 62% [ 50, 73] 4.15 0.36% 1.51% 326ms
30
+ nearest+P 78% [ 67, 88] 4.09 0.69% 2.52% 638ms
31
+ nearest-margin 62% [ 50, 73] 4.15 0.36% 1.51% 306ms
32
+ NMPC 95% [ 88, 100] 4.08 0.14% 0.24% 26ms
33
+ generated 55.6%(sd3.9) [ 38, 63]* 4.05 1.05%(sd0.3) 3.14%(sd0.8) 406ms
34
+ generated+P 76.7%(sd1.4) [ 63, 85]* 4.09 1.04%(sd0.1) 3.58%(sd0.5) 1.1s
35
+ (* generated rows: mean (sd) over sampler seeds [0, 1, 2]; CI shown is for seed 0 only)
36
+ (certified hand-off rate == success rate for every arm except NMPC (a certificate is a precondition for success here); NMPC's own hand-off is n/a, it never switches to a static gain)
37
+
38
+ --- failure breakdown, primary seed 0 ---
39
+ per-plant NMPC never reached Omega:3 replay failed:1
40
+ nearest Lyapunov decrease fails at a vertex:13 rollout failed:9 true dynamics: 0.02% of the boundary has V-dot >= 0, 0.00% saturates:1
41
+ nearest+P rollout failed:13
42
+ nearest-margin Lyapunov decrease fails at a vertex:13 rollout failed:9 true dynamics: 0.02% of the boundary has V-dot >= 0, 0.00% saturates:1
43
+ NMPC :3
44
+ generated rollout failed:12 Lyapunov decrease fails at a vertex:9 peak |Kx| on Omega is 10.05 V > 10.0 V:2 peak |Kx| on Omega is 10.01 V > 10.0 V:2 peak |Kx| on Omega is 10.17 V > 10.0 V:1 peak |Kx| on Omega is 10.14 V > 10.0 V:1 peak |Kx| on Omega is 10.37 V > 10.0 V:1 peak |Kx| on Omega is 10.07 V > 10.0 V:1 peak |Kx| on Omega is 10.12 V > 10.0 V:1
45
+ generated+P rollout failed:15
46
+
47
+ --- paired McNemar on success (exact binomial on discordant pairs) ---
48
+ per-plant vs nearest beats on 20, loses on 1 p=0.0000 * cost ratio 1.004x p=0.0000
49
+ per-plant vs nearest+P beats on 10, loses on 1 p=0.0117 * cost ratio 1.007x p=0.0000
50
+ per-plant vs nearest-margin beats on 20, loses on 1 p=0.0000 * cost ratio 1.004x p=0.0000
51
+ per-plant vs NMPC beats on 0, loses on 1 p=1.0000 cost ratio 1.001x p=0.0002
52
+ per-plant vs generated beats on 26, loses on 0 p=0.0000 * cost ratio 1.010x p=0.0000
53
+ per-plant vs generated+P beats on 12, loses on 1 p=0.0034 * cost ratio 1.011x p=0.0000
54
+ nearest+P vs per-plant beats on 1, loses on 10 p=0.0117 * cost ratio 0.993x p=0.0000
55
+ nearest+P vs nearest beats on 10, loses on 0 p=0.0020 * cost ratio 1.000x p=0.8291
56
+ nearest+P vs nearest-margin beats on 10, loses on 0 p=0.0020 * cost ratio 1.000x p=0.8291
57
+ nearest+P vs NMPC beats on 0, loses on 10 p=0.0020 * cost ratio 0.995x p=0.0000
58
+ nearest+P vs generated beats on 20, loses on 3 p=0.0005 * cost ratio 1.005x p=0.0815
59
+ nearest+P vs generated+P beats on 8, loses on 6 p=0.7905 cost ratio 1.002x p=0.0414
60
+
61
+ --- plant set seed9001 (seed 9001, n=60) ---
62
+ arm succ 95% CI medJ med regret p90 synth
63
+ per-plant 93% [ 87, 98] 4.16 0.00% 0.00% 7.5s
64
+ nearest 65% [ 52, 77] 4.10 0.71% 2.11% 218ms
65
+ nearest+P 87% [ 78, 95] 4.14 0.59% 2.34% 567ms
66
+ nearest-margin 65% [ 52, 77] 4.10 0.71% 2.11% 260ms
67
+ NMPC 97% [ 92, 100] 4.17 0.07% 0.22% 34ms
68
+ generated 57.2%(sd3.4) [ 43, 68]* 4.16 0.66%(sd0.1) 2.85%(sd0.7) 448ms
69
+ generated+P 79.4%(sd3.4) [ 73, 93]* 4.15 0.76%(sd0.2) 3.50%(sd0.4) 1.0s
70
+ (* generated rows: mean (sd) over sampler seeds [0, 1, 2]; CI shown is for seed 0 only)
71
+ (certified hand-off rate == success rate for every arm except NMPC (a certificate is a precondition for success here); NMPC's own hand-off is n/a, it never switches to a static gain)
72
+
73
+ --- failure breakdown, primary seed 0 ---
74
+ per-plant replay failed:2 NMPC never reached Omega:2
75
+ nearest Lyapunov decrease fails at a vertex:16 rollout failed:5
76
+ nearest+P rollout failed:8
77
+ nearest-margin Lyapunov decrease fails at a vertex:16 rollout failed:5
78
+ NMPC :2
79
+ generated Lyapunov decrease fails at a vertex:19 rollout failed:2 peak |Kx| on Omega is 10.03 V > 10.0 V:1 peak |Kx| on Omega is 10.15 V > 10.0 V:1 true dynamics: 0.29% of the boundary has V-dot >= 0, 0.00% saturates:1 peak |Kx| on Omega is 10.06 V > 10.0 V:1 true dynamics: 0.60% of the boundary has V-dot >= 0, 0.00% saturates:1
80
+ generated+P rollout failed:10
81
+
82
+ --- paired McNemar on success (exact binomial on discordant pairs) ---
83
+ per-plant vs nearest beats on 18, loses on 1 p=0.0001 * cost ratio 1.007x p=0.0000
84
+ per-plant vs nearest+P beats on 5, loses on 1 p=0.2188 cost ratio 1.006x p=0.0000
85
+ per-plant vs nearest-margin beats on 18, loses on 1 p=0.0001 * cost ratio 1.007x p=0.0000
86
+ per-plant vs NMPC beats on 0, loses on 2 p=0.5000 cost ratio 1.001x p=0.0064
87
+ per-plant vs generated beats on 22, loses on 0 p=0.0000 * cost ratio 1.008x p=0.0000
88
+ per-plant vs generated+P beats on 6, loses on 0 p=0.0312 * cost ratio 1.010x p=0.0000
89
+ nearest+P vs per-plant beats on 1, loses on 5 p=0.2188 cost ratio 0.994x p=0.0000
90
+ nearest+P vs nearest beats on 13, loses on 0 p=0.0002 * cost ratio 1.000x p=0.1154
91
+ nearest+P vs nearest-margin beats on 13, loses on 0 p=0.0002 * cost ratio 1.000x p=0.1154
92
+ nearest+P vs NMPC beats on 0, loses on 6 p=0.0312 * cost ratio 0.995x p=0.0000
93
+ nearest+P vs generated beats on 20, loses on 2 p=0.0001 * cost ratio 1.003x p=0.1240
94
+ nearest+P vs generated+P beats on 5, loses on 3 p=0.7266 cost ratio 1.004x p=0.0165
results/furuta_v3/traj_gen_eval_v3e2400s1_raw_clip1.txt ADDED
@@ -0,0 +1,94 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ bank data/furuta_nmpc_bank3000.npz: 2801 entries
2
+ model models/Best_furuta_traj_cond_v3_e2400_s1.pth: cond_dim=4 out_dim=116 hidden=256 layers=5 device=cpu
3
+ [seed4242] margin=0.3: resolving P for 227 unique bank source plants
4
+ [seed4242] sampler seed 0: 9.6s for 60x4 candidates (0 decode-invalid of 240)
5
+ [seed4242] sampler seed 1: 9.5s for 60x4 candidates (0 decode-invalid of 240)
6
+ [seed4242] sampler seed 2: 9.6s for 60x4 candidates (0 decode-invalid of 240)
7
+ [seed4242] 10/60 (23s)
8
+ [seed4242] 20/60 (39s)
9
+ [seed4242] 30/60 (57s)
10
+ [seed4242] 40/60 (74s)
11
+ [seed4242] 50/60 (94s)
12
+ [seed4242] 60/60 (119s)
13
+ [seed9001] margin=0.3: resolving P for 232 unique bank source plants
14
+ [seed9001] sampler seed 0: 9.8s for 60x4 candidates (0 decode-invalid of 240)
15
+ [seed9001] sampler seed 1: 10.0s for 60x4 candidates (0 decode-invalid of 240)
16
+ [seed9001] sampler seed 2: 10.2s for 60x4 candidates (0 decode-invalid of 240)
17
+ [seed9001] 10/60 (29s)
18
+ [seed9001] 20/60 (49s)
19
+ [seed9001] 30/60 (78s)
20
+ [seed9001] 40/60 (100s)
21
+ [seed9001] 50/60 (120s)
22
+ [seed9001] 60/60 (146s)
23
+
24
+ === traj_gen_eval (v3e2400s1_raw_clip1): generated vs retrieval vs NMPC ===
25
+
26
+ --- plant set seed4242 (seed 4242, n=60) ---
27
+ arm succ 95% CI medJ med regret p90 synth
28
+ per-plant 93% [ 87, 98] 4.08 0.00% 0.00% 4.0s
29
+ nearest 62% [ 50, 73] 4.15 0.36% 1.51% 301ms
30
+ nearest+P 78% [ 67, 88] 4.09 0.69% 2.52% 583ms
31
+ nearest-margin 62% [ 50, 73] 4.15 0.36% 1.51% 305ms
32
+ NMPC 95% [ 88, 100] 4.08 0.14% 0.24% 23ms
33
+ generated 60.0%(sd4.1) [ 42, 68]* 4.09 0.65%(sd0.2) 2.27%(sd0.5) 300ms
34
+ generated+P 81.7%(sd2.4) [ 70, 90]* 4.09 0.77%(sd0.2) 2.82%(sd0.5) 686ms
35
+ (* generated rows: mean (sd) over sampler seeds [0, 1, 2]; CI shown is for seed 0 only)
36
+ (certified hand-off rate == success rate for every arm except NMPC (a certificate is a precondition for success here); NMPC's own hand-off is n/a, it never switches to a static gain)
37
+
38
+ --- failure breakdown, primary seed 0 ---
39
+ per-plant NMPC never reached Omega:3 replay failed:1
40
+ nearest Lyapunov decrease fails at a vertex:13 rollout failed:9 true dynamics: 0.02% of the boundary has V-dot >= 0, 0.00% saturates:1
41
+ nearest+P rollout failed:13
42
+ nearest-margin Lyapunov decrease fails at a vertex:13 rollout failed:9 true dynamics: 0.02% of the boundary has V-dot >= 0, 0.00% saturates:1
43
+ NMPC :3
44
+ generated Lyapunov decrease fails at a vertex:15 rollout failed:5 peak |Kx| on Omega is 10.02 V > 10.0 V:1 peak |Kx| on Omega is 10.04 V > 10.0 V:1 peak |Kx| on Omega is 10.16 V > 10.0 V:1 peak |Kx| on Omega is 10.30 V > 10.0 V:1 peak |Kx| on Omega is 10.12 V > 10.0 V:1 peak |Kx| on Omega is 10.03 V > 10.0 V:1 peak |Kx| on Omega is 10.28 V > 10.0 V:1
45
+ generated+P rollout failed:12
46
+
47
+ --- paired McNemar on success (exact binomial on discordant pairs) ---
48
+ per-plant vs nearest beats on 20, loses on 1 p=0.0000 * cost ratio 1.004x p=0.0000
49
+ per-plant vs nearest+P beats on 10, loses on 1 p=0.0117 * cost ratio 1.007x p=0.0000
50
+ per-plant vs nearest-margin beats on 20, loses on 1 p=0.0000 * cost ratio 1.004x p=0.0000
51
+ per-plant vs NMPC beats on 0, loses on 1 p=1.0000 cost ratio 1.001x p=0.0002
52
+ per-plant vs generated beats on 23, loses on 0 p=0.0000 * cost ratio 1.009x p=0.0000
53
+ per-plant vs generated+P beats on 8, loses on 0 p=0.0078 * cost ratio 1.011x p=0.0000
54
+ nearest+P vs per-plant beats on 1, loses on 10 p=0.0117 * cost ratio 0.993x p=0.0000
55
+ nearest+P vs nearest beats on 10, loses on 0 p=0.0020 * cost ratio 1.000x p=0.8291
56
+ nearest+P vs nearest-margin beats on 10, loses on 0 p=0.0020 * cost ratio 1.000x p=0.8291
57
+ nearest+P vs NMPC beats on 0, loses on 10 p=0.0020 * cost ratio 0.995x p=0.0000
58
+ nearest+P vs generated beats on 17, loses on 3 p=0.0026 * cost ratio 1.000x p=0.2988
59
+ nearest+P vs generated+P beats on 6, loses on 7 p=1.0000 cost ratio 1.003x p=0.0735
60
+
61
+ --- plant set seed9001 (seed 9001, n=60) ---
62
+ arm succ 95% CI medJ med regret p90 synth
63
+ per-plant 93% [ 87, 98] 4.16 0.00% 0.00% 7.8s
64
+ nearest 65% [ 52, 77] 4.10 0.71% 2.11% 265ms
65
+ nearest+P 87% [ 78, 95] 4.14 0.59% 2.34% 589ms
66
+ nearest-margin 65% [ 52, 77] 4.10 0.71% 2.11% 263ms
67
+ NMPC 97% [ 92, 100] 4.17 0.07% 0.22% 37ms
68
+ generated 63.3%(sd2.4) [ 48, 72]* 4.19 0.59%(sd0.1) 2.57%(sd0.3) 407ms
69
+ generated+P 90.6%(sd2.8) [ 87, 98]* 4.18 0.73%(sd0.0) 3.03%(sd0.3) 680ms
70
+ (* generated rows: mean (sd) over sampler seeds [0, 1, 2]; CI shown is for seed 0 only)
71
+ (certified hand-off rate == success rate for every arm except NMPC (a certificate is a precondition for success here); NMPC's own hand-off is n/a, it never switches to a static gain)
72
+
73
+ --- failure breakdown, primary seed 0 ---
74
+ per-plant replay failed:2 NMPC never reached Omega:2
75
+ nearest Lyapunov decrease fails at a vertex:16 rollout failed:5
76
+ nearest+P rollout failed:8
77
+ nearest-margin Lyapunov decrease fails at a vertex:16 rollout failed:5
78
+ NMPC :2
79
+ generated Lyapunov decrease fails at a vertex:15 peak |Kx| on Omega is 10.08 V > 10.0 V:1 true dynamics: 0.21% of the boundary has V-dot >= 0, 0.00% saturates:1 peak |Kx| on Omega is 10.01 V > 10.0 V:1 peak |Kx| on Omega is 10.31 V > 10.0 V:1 peak |Kx| on Omega is 10.17 V > 10.0 V:1 peak |Kx| on Omega is 10.03 V > 10.0 V:1 rollout failed:1 true dynamics: 0.26% of the boundary has V-dot >= 0, 0.00% saturates:1 true dynamics: 0.03% of the boundary has V-dot >= 0, 0.00% saturates:1
80
+ generated+P rollout failed:4
81
+
82
+ --- paired McNemar on success (exact binomial on discordant pairs) ---
83
+ per-plant vs nearest beats on 18, loses on 1 p=0.0001 * cost ratio 1.007x p=0.0000
84
+ per-plant vs nearest+P beats on 5, loses on 1 p=0.2188 cost ratio 1.006x p=0.0000
85
+ per-plant vs nearest-margin beats on 18, loses on 1 p=0.0001 * cost ratio 1.007x p=0.0000
86
+ per-plant vs NMPC beats on 0, loses on 2 p=0.5000 cost ratio 1.001x p=0.0064
87
+ per-plant vs generated beats on 20, loses on 0 p=0.0000 * cost ratio 1.006x p=0.0000
88
+ per-plant vs generated+P beats on 0, loses on 0 p=1.0000 cost ratio 1.007x p=0.0000
89
+ nearest+P vs per-plant beats on 1, loses on 5 p=0.2188 cost ratio 0.994x p=0.0000
90
+ nearest+P vs nearest beats on 13, loses on 0 p=0.0002 * cost ratio 1.000x p=0.1154
91
+ nearest+P vs nearest-margin beats on 13, loses on 0 p=0.0002 * cost ratio 1.000x p=0.1154
92
+ nearest+P vs NMPC beats on 0, loses on 6 p=0.0312 * cost ratio 0.995x p=0.0000
93
+ nearest+P vs generated beats on 18, loses on 2 p=0.0004 * cost ratio 1.002x p=0.3011
94
+ nearest+P vs generated+P beats on 1, loses on 5 p=0.2188 cost ratio 1.001x p=0.1153
results/furuta_v3/traj_gen_v2ctl2400_raw_clip5.txt ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ bank data/furuta_nmpc_bank3000.npz: 2801 entries
2
+ model models/Best_furuta_traj_cond_v2ctl_e2400.pth: cond_dim=4 out_dim=116
3
+ [seed4242] per-plant ceiling: 93%
4
+ [seed4242] nearest+P sequences done
5
+ [seed4242] generated+P seed 0 sequences done (36.5s sampling for 60x16)
6
+ [seed4242] generated+P seed 1 sequences done (37.0s sampling for 60x16)
7
+ [seed4242] generated+P seed 2 sequences done (36.6s sampling for 60x16)
8
+ [seed9001] per-plant ceiling: 93%
9
+ [seed9001] nearest+P sequences done
10
+ [seed9001] generated+P seed 0 sequences done (36.3s sampling for 60x16)
11
+ [seed9001] generated+P seed 1 sequences done (36.4s sampling for 60x16)
12
+ [seed9001] generated+P seed 2 sequences done (36.2s sampling for 60x16)
13
+
14
+ === candidate-budget sweep: generated+P vs nearest+P vs per-plant ceiling ===
15
+
16
+ --- plant set seed4242 (seed 4242, n=60) ---
17
+ K nearest+P generated+P per-plant
18
+ 1 51.7% t= 0.42s 55.6%(sd3.4) t= 0.95s 93.3% [McNemar nearest+P vs generated+P(seed0): 11-11, p=1.000]
19
+ 4 78.3% t= 0.66s 81.7%(sd2.4) t= 1.16s 93.3% [McNemar nearest+P vs generated+P(seed0): 6-7, p=1.000]
20
+ 8 88.3% t= 0.66s 86.7%(sd1.4) t= 1.16s 93.3% [McNemar nearest+P vs generated+P(seed0): 3-3, p=1.000]
21
+ 16 93.3% t= 0.66s 88.3%(sd1.4) t= 1.16s 93.3% [McNemar nearest+P vs generated+P(seed0): 4-1, p=0.375]
22
+ crossover K (generated+P mean >= nearest+P): 1
23
+
24
+ --- plant set seed9001 (seed 9001, n=60) ---
25
+ K nearest+P generated+P per-plant
26
+ 1 56.7% t= 0.50s 66.1%(sd1.6) t= 0.92s 93.3% [McNemar nearest+P vs generated+P(seed0): 13-18, p=0.473]
27
+ 4 86.7% t= 0.66s 92.2%(sd0.8) t= 1.05s 93.3% [McNemar nearest+P vs generated+P(seed0): 1-4, p=0.375]
28
+ 8 91.7% t= 0.66s 94.4%(sd0.8) t= 1.05s 93.3% [McNemar nearest+P vs generated+P(seed0): 0-2, p=0.500]
29
+ 16 93.3% t= 0.66s 95.6%(sd0.8) t= 1.05s 93.3% [McNemar nearest+P vs generated+P(seed0): 0-1, p=1.000]
30
+ crossover K (generated+P mean >= nearest+P): 1
results/furuta_v3/traj_gen_v3e2400_raw_clip1.txt ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ bank data/furuta_nmpc_bank3000.npz: 2801 entries
2
+ model models/Best_furuta_traj_cond_v3_e2400.pth: cond_dim=4 out_dim=116
3
+ [seed4242] per-plant ceiling: 93%
4
+ [seed4242] nearest+P sequences done
5
+ [seed4242] generated+P seed 0 sequences done (34.3s sampling for 60x16)
6
+ [seed4242] generated+P seed 1 sequences done (34.6s sampling for 60x16)
7
+ [seed4242] generated+P seed 2 sequences done (34.5s sampling for 60x16)
8
+ [seed9001] per-plant ceiling: 93%
9
+ [seed9001] nearest+P sequences done
10
+ [seed9001] generated+P seed 0 sequences done (34.1s sampling for 60x16)
11
+ [seed9001] generated+P seed 1 sequences done (34.3s sampling for 60x16)
12
+ [seed9001] generated+P seed 2 sequences done (34.1s sampling for 60x16)
13
+
14
+ === candidate-budget sweep: generated+P vs nearest+P vs per-plant ceiling ===
15
+
16
+ --- plant set seed4242 (seed 4242, n=60) ---
17
+ K nearest+P generated+P per-plant
18
+ 1 51.7% t= 0.29s 50.0%(sd7.2) t= 0.87s 93.3% [McNemar nearest+P vs generated+P(seed0): 14-9, p=0.405]
19
+ 4 78.3% t= 0.56s 82.8%(sd3.4) t= 1.13s 93.3% [McNemar nearest+P vs generated+P(seed0): 6-9, p=0.607]
20
+ 8 88.3% t= 0.56s 89.4%(sd2.1) t= 1.13s 93.3% [McNemar nearest+P vs generated+P(seed0): 3-4, p=1.000]
21
+ 16 93.3% t= 0.56s 91.7%(sd1.4) t= 1.13s 93.3% [McNemar nearest+P vs generated+P(seed0): 3-1, p=0.625]
22
+ crossover K (generated+P mean >= nearest+P): 4
23
+
24
+ --- plant set seed9001 (seed 9001, n=60) ---
25
+ K nearest+P generated+P per-plant
26
+ 1 56.7% t= 0.30s 57.2%(sd7.5) t= 0.86s 93.3% [McNemar nearest+P vs generated+P(seed0): 16-11, p=0.442]
27
+ 4 86.7% t= 0.45s 86.7%(sd1.4) t= 1.03s 93.3% [McNemar nearest+P vs generated+P(seed0): 5-4, p=1.000]
28
+ 8 91.7% t= 0.45s 93.3%(sd1.4) t= 1.03s 93.3% [McNemar nearest+P vs generated+P(seed0): 2-2, p=1.000]
29
+ 16 93.3% t= 0.45s 95.6%(sd0.8) t= 1.03s 93.3% [McNemar nearest+P vs generated+P(seed0): 1-2, p=1.000]
30
+ crossover K (generated+P mean >= nearest+P): 1
results/furuta_v3/traj_gen_warmstart_v2clip5_n200.txt ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ === warm-start test (n=200, seed 4242, max_iter=700) ===
2
+ target certificate: largest_certified_region per plant (unmargined); verifier: furuta_traj_bank.score_solution (identical to every other arm in this doc)
3
+
4
+ condition success med iter med t med J
5
+ cold 58.0% 290 2.01s 4.06
6
+ nearest-warm 78.0% 48 0.36s 4.10
7
+ generated-warm 79.0% 50 0.37s 4.10
8
+
9
+ --- paired tests ---
10
+ generated-warm vs cold success: beats 44, loses 2, McNemar p=0.0000 * time ratio 0.210x (n=114), Wilcoxon p=0.0000 *
11
+ generated-warm vs nearest-warm success: beats 4, loses 2, McNemar p=0.6875 time ratio 0.993x (n=154), Wilcoxon p=0.5074
12
+ nearest-warm vs cold success: beats 41, loses 1, McNemar p=0.0000 * time ratio 0.196x (n=115), Wilcoxon p=0.0000 *
13
+
14
+ --- failure breakdown ---
15
+ cold ipopt: Maximum_Iterations_Exceeded:36 ipopt: Infeasible_Problem_Detected:31 rollout failed:17
16
+ nearest-warm rollout failed:26 ipopt: Maximum_Iterations_Exceeded:10 ipopt: Infeasible_Problem_Detected:8
17
+ generated-warm rollout failed:24 ipopt: Maximum_Iterations_Exceeded:9 ipopt: Infeasible_Problem_Detected:9
results/furuta_v3/traj_gen_warmstart_v2clip5_s1.txt ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ === warm-start test (n=60, seed 4242, max_iter=700) ===
2
+ target certificate: largest_certified_region per plant (unmargined); verifier: furuta_traj_bank.score_solution (identical to every other arm in this doc)
3
+
4
+ condition success med iter med t med J
5
+ cold 53.3% 363 2.08s 4.06
6
+ nearest-warm 76.7% 45 0.28s 4.15
7
+ generated-warm 75.0% 46 0.27s 4.14
8
+
9
+ --- paired tests ---
10
+ generated-warm vs cold success: beats 13, loses 0, McNemar p=0.0002 * time ratio 0.147x (n=32), Wilcoxon p=0.0000 *
11
+ generated-warm vs nearest-warm success: beats 0, loses 1, McNemar p=1.0000 time ratio 1.056x (n=45), Wilcoxon p=0.6069
12
+ nearest-warm vs cold success: beats 14, loses 0, McNemar p=0.0001 * time ratio 0.159x (n=32), Wilcoxon p=0.0000 *
13
+
14
+ --- failure breakdown ---
15
+ cold ipopt: Maximum_Iterations_Exceeded:13 ipopt: Infeasible_Problem_Detected:11 rollout failed:4
16
+ nearest-warm ipopt: Infeasible_Problem_Detected:6 rollout failed:5 ipopt: Maximum_Iterations_Exceeded:3
17
+ generated-warm ipopt: Infeasible_Problem_Detected:7 ipopt: Maximum_Iterations_Exceeded:4 rollout failed:4
results/furuta_v3/traj_gen_warmstart_v2clip5_s2.txt ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ === warm-start test (n=60, seed 4242, max_iter=700) ===
2
+ target certificate: largest_certified_region per plant (unmargined); verifier: furuta_traj_bank.score_solution (identical to every other arm in this doc)
3
+
4
+ condition success med iter med t med J
5
+ cold 53.3% 363 2.07s 4.06
6
+ nearest-warm 76.7% 45 0.28s 4.15
7
+ generated-warm 75.0% 47 0.27s 4.14
8
+
9
+ --- paired tests ---
10
+ generated-warm vs cold success: beats 13, loses 0, McNemar p=0.0002 * time ratio 0.166x (n=32), Wilcoxon p=0.0000 *
11
+ generated-warm vs nearest-warm success: beats 0, loses 1, McNemar p=1.0000 time ratio 1.023x (n=45), Wilcoxon p=0.9199
12
+ nearest-warm vs cold success: beats 14, loses 0, McNemar p=0.0001 * time ratio 0.160x (n=32), Wilcoxon p=0.0000 *
13
+
14
+ --- failure breakdown ---
15
+ cold ipopt: Maximum_Iterations_Exceeded:13 ipopt: Infeasible_Problem_Detected:11 rollout failed:4
16
+ nearest-warm ipopt: Infeasible_Problem_Detected:6 rollout failed:5 ipopt: Maximum_Iterations_Exceeded:3
17
+ generated-warm ipopt: Infeasible_Problem_Detected:7 ipopt: Maximum_Iterations_Exceeded:4 rollout failed:4
results/furuta_v3/traj_gen_warmstart_v3_n200.txt ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ === warm-start test (n=200, seed 4242, max_iter=700) ===
2
+ target certificate: largest_certified_region per plant (unmargined); verifier: furuta_traj_bank.score_solution (identical to every other arm in this doc)
3
+
4
+ condition success med iter med t med J
5
+ cold 58.0% 290 2.08s 4.06
6
+ nearest-warm 78.0% 48 0.38s 4.10
7
+ generated-warm 79.5% 44 0.35s 4.10
8
+
9
+ --- paired tests ---
10
+ generated-warm vs cold success: beats 45, loses 2, McNemar p=0.0000 * time ratio 0.181x (n=114), Wilcoxon p=0.0000 *
11
+ generated-warm vs nearest-warm success: beats 4, loses 1, McNemar p=0.3750 time ratio 0.986x (n=155), Wilcoxon p=0.4041
12
+ nearest-warm vs cold success: beats 41, loses 1, McNemar p=0.0000 * time ratio 0.210x (n=115), Wilcoxon p=0.0000 *
13
+
14
+ --- failure breakdown ---
15
+ cold ipopt: Maximum_Iterations_Exceeded:36 ipopt: Infeasible_Problem_Detected:31 rollout failed:17
16
+ nearest-warm rollout failed:26 ipopt: Maximum_Iterations_Exceeded:10 ipopt: Infeasible_Problem_Detected:8
17
+ generated-warm rollout failed:21 ipopt: Maximum_Iterations_Exceeded:10 ipopt: Infeasible_Problem_Detected:10
results/furuta_v3/traj_gen_warmstart_v3_s2.txt ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ === warm-start test (n=60, seed 4242, max_iter=700) ===
2
+ target certificate: largest_certified_region per plant (unmargined); verifier: furuta_traj_bank.score_solution (identical to every other arm in this doc)
3
+
4
+ condition success med iter med t med J
5
+ cold 53.3% 363 1.94s 4.06
6
+ nearest-warm 76.7% 45 0.30s 4.15
7
+ generated-warm 75.0% 46 0.28s 4.14
8
+
9
+ --- paired tests ---
10
+ generated-warm vs cold success: beats 13, loses 0, McNemar p=0.0002 * time ratio 0.158x (n=32), Wilcoxon p=0.0000 *
11
+ generated-warm vs nearest-warm success: beats 0, loses 1, McNemar p=1.0000 time ratio 0.982x (n=45), Wilcoxon p=0.3581
12
+ nearest-warm vs cold success: beats 14, loses 0, McNemar p=0.0001 * time ratio 0.160x (n=32), Wilcoxon p=0.0000 *
13
+
14
+ --- failure breakdown ---
15
+ cold ipopt: Maximum_Iterations_Exceeded:13 ipopt: Infeasible_Problem_Detected:11 rollout failed:4
16
+ nearest-warm ipopt: Infeasible_Problem_Detected:6 rollout failed:5 ipopt: Maximum_Iterations_Exceeded:3
17
+ generated-warm ipopt: Infeasible_Problem_Detected:7 rollout failed:5 ipopt: Maximum_Iterations_Exceeded:3
results/generalize/fig_generalize.png ADDED
results/generalize/summary.txt ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ plants=120 controllers/plant=6 ICs/plant=4
2
+ per-sample stability rate (mean over plants): 0.64
3
+ plants with >=1 stabilizing controller: 0.81
4
+ regret of best controller vs per-plant oracle: median=+9.7% mean=+22.4% (n=97)
5
+ amortization: generation 6.5s for 720 controllers (9 ms each); per-plant oracle 24.1s each
results/metric_steering/mimo_summary.txt ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ MIMO 2x2 metric steering (held-out, N=40)
2
+ set ise iae control_ settling peak_con peak_y
3
+ prior 31.784 12.175 5.113 7.849 1.877 4.223
4
+ refine→control_effort 31.259 11.771 0.335* 7.543 0.922 4.277
5
+ refine→ise 28.752* 11.237 8.601 8.056 2.435 4.153
6
+ refine→settling 31.640 11.529 3.766 6.767* 2.080 4.324
7
+ refine→peak_control 31.223 11.754 0.298 7.547 0.853* 4.280
8
+
9
+ refine→control_effort: target -93.5% vs prior (5.113 → 0.335)
10
+ refine→ise : target -9.5% vs prior (31.784 → 28.752)
11
+ refine→settling : target -13.8% vs prior (7.849 → 6.767)
12
+ refine→peak_control : target -54.6% vs prior (1.877 → 0.853)
results/metric_steering/refine_summary.txt ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ reference plant G=1/(s+1) F=1 | refine 80 steps lr=0.05 | n=16 ICs=4
2
+ set ise iae itae control_ control_ peak_con peak_dev settling
3
+ prior 1.903 1.593 1.417 0.957 0.408 1.989 2.124 3.705
4
+ refine→control_effort 2.389 1.950 1.945 0.296* 0.051 1.214 2.147 4.259
5
+ refine→ise 0.841* 0.611 0.177 4.077 1.391 4.502 2.055 1.844
6
+ refine→settling 0.848 0.617 0.182 4.018 1.385 4.472 2.056 1.857*
7
+ refine→peak_control 2.381 1.943 1.935 0.403 0.057 1.205* 2.147 4.250
8
+
9
+ refine→control_effort: target -69.1% vs prior (0.957 → 0.296)
10
+ refine→ise : target -55.8% vs prior (1.903 → 0.841)
11
+ refine→settling : target -49.9% vs prior (3.705 → 1.857)
12
+ refine→peak_control : target -39.4% vs prior (1.989 → 1.205)
results/metric_steering/summary.txt ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ reference plant G=1/(s+1), F=1 | n=16 ICs=4 | steer λ=1.0
2
+ set ise iae itae control_ control_ peak_con peak_dev settling
3
+ unguided 1.906 1.593 1.417 0.992 0.410 1.932 2.082 3.553
4
+ steer→control_effort 1.904 1.592 1.414 0.979* 0.411 1.923 2.082 3.552
5
+ steer→ise 1.907* 1.593 1.418 1.007 0.409 1.943 2.082 3.553
6
+ steer→settling 1.907 1.594 1.419 0.998 0.408 1.936 2.082 3.556*
7
+ steer→peak_control 1.904 1.592 1.414 0.980 0.411 1.924* 2.082 3.552
8
+
9
+ steer→control_effort: target metric -1.4% vs unguided (0.992 → 0.979)
10
+ steer→ise : target metric +0.1% vs unguided (1.906 → 1.907)
11
+ steer→settling : target metric +0.1% vs unguided (3.553 → 3.556)
12
+ steer→peak_control : target metric -0.4% vs unguided (1.932 → 1.924)
results/mimo/k_sweep_mimo.txt ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ === K-sweep MIMO (N=60, seed 9999, gen 65.4 ms/ctrl, per-sample stab 0.963) ===
2
+ K plants>=1 stable gen time/plant
3
+ 1 0.950 0.07 s
4
+ 2 0.983 0.13 s
5
+ 4 1.000 0.26 s
6
+ 8 1.000 0.52 s
7
+ 16 1.000 1.05 s
8
+ 32 1.000 2.09 s
9
+
10
+ contrast: SISO needed K~32 to reach 1.0 (mixed random-in-sector dataset, per-sample ~0.66);
11
+ MIMO trained on cost-optimal CERTIFIED targets -> saturates at ~1.0 almost immediately.
results/mimo/target_ablation.md ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ **Table (§4.3 ablation).** MIMO training-target ablation: random-in-sector vs cost-optimal, same held-out plants (N=120, K=8, seed 9999), same per-plant oracle & phi=0 baseline.
2
+
3
+ | training targets | per-sample stability | plants stabilizable | regret vs oracle (med / mean) | cost vs phi=0 (med) |
4
+ |---|---|---|---|---|
5
+ | random (§6m) | 0.86 | 0.93 | +19.2% / +26.8% | +16.3% |
6
+ | cost-optimal (§6n) | 0.96 | 1.00 | +2.0% / +3.8% | +0.2% |
results/pareto/fig_pareto.pdf ADDED
Binary file (24.4 kB). View file
 
results/pareto/fig_pareto.png ADDED
results/pareto/table_pareto.md ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ **Table 2.** Cost-steered controller vs analytic optimum per cost weight rho (plant G=1/(s+1), 9 ICs). Steering = refine a fixed model controller on the differentiable J_rho. Regret = (achieved J_rho − V_rho)/V_rho.
2
+
3
+ | rho | V_rho (optimal) | achieved J_rho | regret % | state ∫q | control ∫u² |
4
+ |---|---|---|---|---|---|
5
+ | 0.1 | 1.044 | 1.063 | +1.83 | 0.731 | 3.321 |
6
+ | 0.3 | 1.499 | 1.503 | +0.25 | 1.078 | 1.416 |
7
+ | 1 | 1.980 | 1.982 | +0.13 | 1.635 | 0.347 |
8
+ | 3 | 2.276 | 2.278 | +0.09 | 2.088 | 0.064 |
9
+ | 10 | 2.431 | 2.433 | +0.10 | 2.352 | 0.008 |
results/pareto_specs/run_s0.log ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ plant 0: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.498 GD(prior)=0.917 warm=510ms cold=3572ms/pt
2
+ plant 1: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.000 GD(prior)=9.519 warm=490ms cold=3358ms/pt
3
+ plant 2: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=2.306 GD(prior)=0.963 warm=474ms cold=3398ms/pt
4
+ plant 3: prior_cloud=16 ours_front=10 oracle_front=9 GD(ours)=0.878 GD(prior)=3.064 warm=434ms cold=3358ms/pt
5
+ plant 4: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=7.361 GD(prior)=1.604 warm=443ms cold=3365ms/pt
6
+ plant 5: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=3.066 GD(prior)=1.507 warm=452ms cold=3383ms/pt
7
+ plant 6: prior_cloud=16 ours_front=10 oracle_front=2 GD(ours)=0.047 GD(prior)=1.535 warm=450ms cold=3222ms/pt
8
+ plant 8: prior_cloud=16 ours_front=10 oracle_front=9 GD(ours)=0.039 GD(prior)=0.481 warm=440ms cold=3041ms/pt
9
+ plant 9: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.007 GD(prior)=8.233 warm=454ms cold=3148ms/pt
10
+
11
+ === SPEC-PARETO SUMMARY ===
12
+ plants evaluated=9 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80
13
+ prior pass: 24.3s for 160 controllers (152 ms each)
14
+
15
+ Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer):
16
+ OURS (warm-steered front): median 0.498
17
+ PRIOR cloud front (no refine): median 1.535
18
+
19
+ Cost to trace ONE Pareto point (per plant):
20
+ OURS warm-start, 12 steps: 452 ms
21
+ CLASSICAL cold NLP, 80 steps: 3358 ms -> 7.4× slower per point
22
+
23
+ Certified (sector-valid) front points: median 100%
24
+
25
+ Binding-spec baseline (classical 'optimize only the most stringent req' = settling):
26
+ -> yields ONE corner controller, not the surface (see figure).
27
+
28
+ saved -> results/pareto_specs/pareto_specs_s0.npz
results/pareto_specs/run_s1.log ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ plant 0: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=2.325 GD(prior)=1.225 warm=537ms cold=3508ms/pt
2
+ plant 1: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=1.105 GD(prior)=0.985 warm=502ms cold=3496ms/pt
3
+ plant 2: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.050 GD(prior)=14.175 warm=491ms cold=3368ms/pt
4
+ plant 3: prior_cloud=15 ours_front=10 oracle_front=10 GD(ours)=0.015 GD(prior)=2.730 warm=482ms cold=3263ms/pt
5
+ plant 7: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.125 GD(prior)=2.600 warm=484ms cold=3530ms/pt
6
+ plant 8: prior_cloud=16 ours_front=10 oracle_front=6 GD(ours)=1.002 GD(prior)=1.098 warm=468ms cold=3416ms/pt
7
+ plant 9: prior_cloud=14 ours_front=10 oracle_front=7 GD(ours)=0.040 GD(prior)=1.691 warm=492ms cold=3489ms/pt
8
+
9
+ === SPEC-PARETO SUMMARY ===
10
+ plants evaluated=7 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80
11
+ prior pass: 25.4s for 160 controllers (159 ms each)
12
+
13
+ Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer):
14
+ OURS (warm-steered front): median 0.125
15
+ PRIOR cloud front (no refine): median 1.691
16
+
17
+ Cost to trace ONE Pareto point (per plant):
18
+ OURS warm-start, 12 steps: 491 ms
19
+ CLASSICAL cold NLP, 80 steps: 3489 ms -> 7.1× slower per point
20
+
21
+ Certified (sector-valid) front points: median 100%
22
+
23
+ Binding-spec baseline (classical 'optimize only the most stringent req' = settling):
24
+ -> yields ONE corner controller, not the surface (see figure).
25
+
26
+ saved -> results/pareto_specs/pareto_specs_s1.npz
results/pareto_specs/run_s2.log ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ plant 0: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=7.064 GD(prior)=2.149 warm=546ms cold=3524ms/pt
2
+ plant 1: prior_cloud=16 ours_front=10 oracle_front=5 GD(ours)=0.140 GD(prior)=1.773 warm=443ms cold=3508ms/pt
3
+ plant 2: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.069 GD(prior)=8.234 warm=456ms cold=3467ms/pt
4
+ plant 3: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=17.344 GD(prior)=4.264 warm=586ms cold=3411ms/pt
5
+ plant 4: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.135 GD(prior)=0.822 warm=487ms cold=3433ms/pt
6
+ plant 5: prior_cloud=14 ours_front=10 oracle_front=2 GD(ours)=0.003 GD(prior)=3.755 warm=514ms cold=3571ms/pt
7
+ plant 6: prior_cloud=16 ours_front=10 oracle_front=9 GD(ours)=0.864 GD(prior)=1.401 warm=550ms cold=3300ms/pt
8
+ plant 7: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.546 GD(prior)=41.553 warm=440ms cold=3012ms/pt
9
+ plant 8: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=389.233 GD(prior)=141.003 warm=571ms cold=3206ms/pt
10
+ plant 9: prior_cloud=12 ours_front=6 oracle_front=5 GD(ours)=0.218 GD(prior)=0.292 warm=442ms cold=2951ms/pt
11
+
12
+ === SPEC-PARETO SUMMARY ===
13
+ plants evaluated=10 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80
14
+ prior pass: 22.0s for 160 controllers (138 ms each)
15
+
16
+ Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer):
17
+ OURS (warm-steered front): median 0.382
18
+ PRIOR cloud front (no refine): median 2.952
19
+
20
+ Cost to trace ONE Pareto point (per plant):
21
+ OURS warm-start, 12 steps: 500 ms
22
+ CLASSICAL cold NLP, 80 steps: 3422 ms -> 6.8× slower per point
23
+
24
+ Certified (sector-valid) front points: median 100%
25
+
26
+ Binding-spec baseline (classical 'optimize only the most stringent req' = settling):
27
+ -> yields ONE corner controller, not the surface (see figure).
28
+
29
+ saved -> results/pareto_specs/pareto_specs_s2.npz
results/pareto_specs/run_s3.log ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ plant 0: prior_cloud=14 ours_front=10 oracle_front=10 GD(ours)=1.703 GD(prior)=1.502 warm=485ms cold=3322ms/pt
2
+ plant 1: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.724 GD(prior)=1.843 warm=473ms cold=3491ms/pt
3
+ plant 2: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.002 GD(prior)=20.795 warm=485ms cold=3509ms/pt
4
+ plant 3: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=5.185 GD(prior)=23.308 warm=459ms cold=3587ms/pt
5
+ plant 4: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=4.998 GD(prior)=1.749 warm=576ms cold=3684ms/pt
6
+ plant 5: prior_cloud=16 ours_front=10 oracle_front=2 GD(ours)=0.220 GD(prior)=1.960 warm=535ms cold=3218ms/pt
7
+ plant 6: prior_cloud=16 ours_front=10 oracle_front=1 GD(ours)=0.007 GD(prior)=125.069 warm=539ms cold=3641ms/pt
8
+ plant 7: prior_cloud=14 ours_front=10 oracle_front=10 GD(ours)=0.017 GD(prior)=27.143 warm=475ms cold=2982ms/pt
9
+ plant 8: prior_cloud=16 ours_front=10 oracle_front=2 GD(ours)=0.037 GD(prior)=1.681 warm=487ms cold=3164ms/pt
10
+ plant 9: prior_cloud=16 ours_front=10 oracle_front=9 GD(ours)=0.121 GD(prior)=2.394 warm=432ms cold=2892ms/pt
11
+
12
+ === SPEC-PARETO SUMMARY ===
13
+ plants evaluated=10 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80
14
+ prior pass: 24.5s for 160 controllers (153 ms each)
15
+
16
+ Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer):
17
+ OURS (warm-steered front): median 0.171
18
+ PRIOR cloud front (no refine): median 2.177
19
+
20
+ Cost to trace ONE Pareto point (per plant):
21
+ OURS warm-start, 12 steps: 485 ms
22
+ CLASSICAL cold NLP, 80 steps: 3407 ms -> 7.0× slower per point
23
+
24
+ Certified (sector-valid) front points: median 100%
25
+
26
+ Binding-spec baseline (classical 'optimize only the most stringent req' = settling):
27
+ -> yields ONE corner controller, not the surface (see figure).
28
+
29
+ saved -> results/pareto_specs/pareto_specs_s3.npz
results/pareto_specs/run_s4.log ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ plant 0: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.038 GD(prior)=530.974 warm=481ms cold=3474ms/pt
2
+ plant 2: prior_cloud=16 ours_front=10 oracle_front=2 GD(ours)=0.093 GD(prior)=2.589 warm=514ms cold=3507ms/pt
3
+ plant 3: prior_cloud=16 ours_front=10 oracle_front=2 GD(ours)=0.525 GD(prior)=5.005 warm=511ms cold=3623ms/pt
4
+ plant 4: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=9.910 GD(prior)=2.355 warm=485ms cold=3417ms/pt
5
+ plant 5: prior_cloud=16 ours_front=10 oracle_front=6 GD(ours)=8.178 GD(prior)=2.567 warm=433ms cold=3733ms/pt
6
+ plant 6: prior_cloud=10 ours_front=10 oracle_front=6 GD(ours)=1.339 GD(prior)=1.180 warm=449ms cold=3419ms/pt
7
+ plant 7: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=3.290 GD(prior)=1.603 warm=478ms cold=3313ms/pt
8
+ plant 8: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=15.436 GD(prior)=5.013 warm=462ms cold=3069ms/pt
9
+ plant 9: prior_cloud=15 ours_front=10 oracle_front=1 GD(ours)=0.085 GD(prior)=159.701 warm=463ms cold=3094ms/pt
10
+
11
+ === SPEC-PARETO SUMMARY ===
12
+ plants evaluated=9 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80
13
+ prior pass: 21.0s for 160 controllers (131 ms each)
14
+
15
+ Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer):
16
+ OURS (warm-steered front): median 1.339
17
+ PRIOR cloud front (no refine): median 2.589
18
+
19
+ Cost to trace ONE Pareto point (per plant):
20
+ OURS warm-start, 12 steps: 478 ms
21
+ CLASSICAL cold NLP, 80 steps: 3419 ms -> 7.2× slower per point
22
+
23
+ Certified (sector-valid) front points: median 100%
24
+
25
+ Binding-spec baseline (classical 'optimize only the most stringent req' = settling):
26
+ -> yields ONE corner controller, not the surface (see figure).
27
+
28
+ saved -> results/pareto_specs/pareto_specs_s4.npz
results/pareto_specs/significance.txt ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ pooled plants N = 45 (over 5 seeds)
2
+
3
+ CLAIM 1 — steering helps (ours_gd < prior_gd, paired):
4
+ win rate: 30/45 = 67% sign-test p = 1.78e-02
5
+ Wilcoxon signed-rank (ours<prior): p = 1.42e-02 (log-space p = 4.04e-05)
6
+ median GD ours = 0.498 (95% CI 0.121–1.002)
7
+ median GD prior = 2.355 (95% CI 1.691–3.064)
8
+
9
+ CLAIM 2 — lands on the oracle front (ours_gd < 0.5):
10
+ 23/45 = 51% 95% CI [36%, 66%]
11
+
12
+ per-plant speedup (cold/warm): median 6.9× (95% CI 6.8–7.1)
13
+ certified front points: median 100%
14
+
15
+ VERDICT: significant — steering improves GD (p<0.05 both tests) and reaches the front on a clear majority of plants.
results/pareto_specs/summary.txt ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ plants evaluated=11 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80
2
+ prior pass: 20.4s for 192 controllers (106 ms each)
3
+
4
+ Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer):
5
+ OURS (warm-steered front): median 0.145
6
+ PRIOR cloud front (no refine): median 2.927
7
+
8
+ Cost to trace ONE Pareto point (per plant):
9
+ OURS warm-start, 12 steps: 873 ms
10
+ CLASSICAL cold NLP, 80 steps: 5833 ms -> 6.7× slower per point
11
+
12
+ Certified (sector-valid) front points: median 100%
13
+
14
+ Binding-spec baseline (classical 'optimize only the most stringent req' = settling):
15
+ -> yields ONE corner controller, not the surface (see figure).
results/pareto_specs/summary_s0.txt ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ plants evaluated=9 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80
2
+ prior pass: 24.3s for 160 controllers (152 ms each)
3
+
4
+ Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer):
5
+ OURS (warm-steered front): median 0.498
6
+ PRIOR cloud front (no refine): median 1.535
7
+
8
+ Cost to trace ONE Pareto point (per plant):
9
+ OURS warm-start, 12 steps: 452 ms
10
+ CLASSICAL cold NLP, 80 steps: 3358 ms -> 7.4× slower per point
11
+
12
+ Certified (sector-valid) front points: median 100%
13
+
14
+ Binding-spec baseline (classical 'optimize only the most stringent req' = settling):
15
+ -> yields ONE corner controller, not the surface (see figure).
results/pareto_specs/summary_s1.txt ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ plants evaluated=7 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80
2
+ prior pass: 25.4s for 160 controllers (159 ms each)
3
+
4
+ Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer):
5
+ OURS (warm-steered front): median 0.125
6
+ PRIOR cloud front (no refine): median 1.691
7
+
8
+ Cost to trace ONE Pareto point (per plant):
9
+ OURS warm-start, 12 steps: 491 ms
10
+ CLASSICAL cold NLP, 80 steps: 3489 ms -> 7.1× slower per point
11
+
12
+ Certified (sector-valid) front points: median 100%
13
+
14
+ Binding-spec baseline (classical 'optimize only the most stringent req' = settling):
15
+ -> yields ONE corner controller, not the surface (see figure).
results/pareto_specs/summary_s2.txt ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ plants evaluated=10 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80
2
+ prior pass: 22.0s for 160 controllers (138 ms each)
3
+
4
+ Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer):
5
+ OURS (warm-steered front): median 0.382
6
+ PRIOR cloud front (no refine): median 2.952
7
+
8
+ Cost to trace ONE Pareto point (per plant):
9
+ OURS warm-start, 12 steps: 500 ms
10
+ CLASSICAL cold NLP, 80 steps: 3422 ms -> 6.8× slower per point
11
+
12
+ Certified (sector-valid) front points: median 100%
13
+
14
+ Binding-spec baseline (classical 'optimize only the most stringent req' = settling):
15
+ -> yields ONE corner controller, not the surface (see figure).
results/pareto_specs/summary_s3.txt ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ plants evaluated=10 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80
2
+ prior pass: 24.5s for 160 controllers (153 ms each)
3
+
4
+ Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer):
5
+ OURS (warm-steered front): median 0.171
6
+ PRIOR cloud front (no refine): median 2.177
7
+
8
+ Cost to trace ONE Pareto point (per plant):
9
+ OURS warm-start, 12 steps: 485 ms
10
+ CLASSICAL cold NLP, 80 steps: 3407 ms -> 7.0× slower per point
11
+
12
+ Certified (sector-valid) front points: median 100%
13
+
14
+ Binding-spec baseline (classical 'optimize only the most stringent req' = settling):
15
+ -> yields ONE corner controller, not the surface (see figure).
results/pareto_specs/summary_s4.txt ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ plants evaluated=9 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80
2
+ prior pass: 21.0s for 160 controllers (131 ms each)
3
+
4
+ Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer):
5
+ OURS (warm-steered front): median 1.339
6
+ PRIOR cloud front (no refine): median 2.589
7
+
8
+ Cost to trace ONE Pareto point (per plant):
9
+ OURS warm-start, 12 steps: 478 ms
10
+ CLASSICAL cold NLP, 80 steps: 3419 ms -> 7.2× slower per point
11
+
12
+ Certified (sector-valid) front points: median 100%
13
+
14
+ Binding-spec baseline (classical 'optimize only the most stringent req' = settling):
15
+ -> yields ONE corner controller, not the surface (see figure).
results/physics_gen/pendulum_physics_gen.txt ADDED
@@ -0,0 +1,720 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ === lvl0 K=8 oracle=old (median oracle J 1.352) ===
2
+ arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired
3
+ base_s0 1.000 1.000 1.000 1.09 | 1.000 1.000 1.09 16.0 0.00
4
+ base_s1 1.000 1.000 1.000 1.15 | 1.000 1.000 1.15 16.0 0.00
5
+ base_s2 1.000 1.000 1.000 1.07 | 1.000 1.000 1.07 16.0 0.00
6
+ lz_s42_g0 1.000 1.000 1.000 1.10 | 1.000 1.000 1.10 16.0 0.00
7
+ lz_s42_g1 1.000 1.000 1.000 1.05 | 1.000 1.000 1.05 16.0 0.00
8
+ lz_s42_g2 1.000 1.000 1.000 1.06 | 1.000 1.000 1.06 16.0 0.00
9
+ lz_s43_g0 1.000 1.000 1.000 1.18 | 1.000 1.000 1.18 16.0 0.00
10
+ lz_s44_g0 1.000 1.000 1.000 1.04 | 1.000 1.000 1.04 16.0 0.00
11
+ lzm_s42_g0 1.000 1.000 1.000 1.24 | 1.000 1.000 1.24 16.0 0.00
12
+ lzm_s43_g0 1.000 1.000 1.000 1.31 | 1.000 1.000 1.31 16.0 0.00
13
+ lzm_s44_g0 1.000 1.000 1.000 1.03 | 1.000 1.000 1.03 16.0 0.00
14
+ phys_s42_g0 1.000 1.000 1.000 1.28 | 1.000 1.000 1.28 16.0 0.00
15
+ phys_s43_g0 1.000 1.000 1.000 1.22 | 1.000 1.000 1.22 16.0 0.00
16
+ phys_s44_g0 1.000 1.000 1.000 1.06 | 1.000 1.000 1.06 16.0 0.00
17
+ nn 1.000 1.000 1.000 0.29 | 1.000 1.000 0.29 16.0 0.00
18
+ idw 1.000 1.000 1.000 0.29 | 1.000 1.000 0.29 16.0 0.00
19
+ nns 1.000 1.000 1.000 1.11 | 1.000 1.000 1.11 16.0 0.00
20
+ dls 1.000 1.000 1.000 1.14 | 1.000 1.000 1.14 16.0 0.00
21
+ lzres_s42_g0 1.000 1.000 1.000 1.18 | 1.000 1.000 1.18 16.0 0.00
22
+ lzres_s43_g0 1.000 1.000 1.000 1.18 | 1.000 1.000 1.18 16.0 0.00
23
+ lzres_s44_g0 1.000 1.000 1.000 1.19 | 1.000 1.000 1.19 16.0 0.00
24
+
25
+ -- paired Wilcoxon on regret, model minus best retrieval (B: best=nn; noB: best=nn); negative = model better; [95% CI of paired median]
26
+ base base_s0 B vs nn : +0.68 [+0.46,+0.84] p=2.21e-10 certified 1.000 vs 1.000
27
+ base base_s0 noB vs nn : +0.68 [+0.46,+0.84] p=2.21e-10 certified 1.000 vs 1.000
28
+ base base_s1 B vs nn : +0.76 [+0.56,+0.96] p=1.74e-10 certified 1.000 vs 1.000
29
+ base base_s1 noB vs nn : +0.76 [+0.56,+0.96] p=1.74e-10 certified 1.000 vs 1.000
30
+ base base_s2 B vs nn : +0.68 [+0.57,+0.80] p=1.58e-10 certified 1.000 vs 1.000
31
+ base base_s2 noB vs nn : +0.68 [+0.57,+0.80] p=1.58e-10 certified 1.000 vs 1.000
32
+ A_lz lz_s42_g0 B vs nn : +0.69 [+0.56,+0.89] p=1.9e-11 certified 1.000 vs 1.000
33
+ A_lz lz_s42_g0 noB vs nn : +0.69 [+0.56,+0.89] p=1.9e-11 certified 1.000 vs 1.000
34
+ A_lz lz_s42_g1 B vs nn : +0.67 [+0.60,+0.77] p=3.29e-11 certified 1.000 vs 1.000
35
+ A_lz lz_s42_g1 noB vs nn : +0.67 [+0.60,+0.77] p=3.29e-11 certified 1.000 vs 1.000
36
+ A_lz lz_s42_g2 B vs nn : +0.72 [+0.63,+0.83] p=6.59e-11 certified 1.000 vs 1.000
37
+ A_lz lz_s42_g2 noB vs nn : +0.72 [+0.63,+0.83] p=6.59e-11 certified 1.000 vs 1.000
38
+ A_lz lz_s43_g0 B vs nn : +0.76 [+0.65,+0.95] p=1.71e-11 certified 1.000 vs 1.000
39
+ A_lz lz_s43_g0 noB vs nn : +0.76 [+0.65,+0.95] p=1.71e-11 certified 1.000 vs 1.000
40
+ A_lz lz_s44_g0 B vs nn : +0.62 [+0.49,+0.78] p=2.84e-11 certified 1.000 vs 1.000
41
+ A_lz lz_s44_g0 noB vs nn : +0.62 [+0.49,+0.78] p=2.84e-11 certified 1.000 vs 1.000
42
+ A_lzm lzm_s42_g0 B vs nn : +0.82 [+0.73,+0.96] p=1.63e-11 certified 1.000 vs 1.000
43
+ A_lzm lzm_s42_g0 noB vs nn : +0.82 [+0.73,+0.96] p=1.63e-11 certified 1.000 vs 1.000
44
+ A_lzm lzm_s43_g0 B vs nn : +0.88 [+0.78,+1.00] p=1.63e-11 certified 1.000 vs 1.000
45
+ A_lzm lzm_s43_g0 noB vs nn : +0.88 [+0.78,+1.00] p=1.63e-11 certified 1.000 vs 1.000
46
+ A_lzm lzm_s44_g0 B vs nn : +0.59 [+0.46,+0.71] p=1.63e-11 certified 1.000 vs 1.000
47
+ A_lzm lzm_s44_g0 noB vs nn : +0.59 [+0.46,+0.71] p=1.63e-11 certified 1.000 vs 1.000
48
+ A_lzres lzres_s42_g0 B vs nn : +0.92 [+0.72,+0.98] p=1.9e-11 certified 1.000 vs 1.000
49
+ A_lzres lzres_s42_g0 noB vs nn : +0.92 [+0.72,+0.98] p=1.9e-11 certified 1.000 vs 1.000
50
+ A_lzres lzres_s43_g0 B vs nn : +0.92 [+0.76,+0.97] p=1.9e-11 certified 1.000 vs 1.000
51
+ A_lzres lzres_s43_g0 noB vs nn : +0.92 [+0.76,+0.97] p=1.9e-11 certified 1.000 vs 1.000
52
+ A_lzres lzres_s44_g0 B vs nn : +0.95 [+0.74,+1.03] p=1.71e-11 certified 1.000 vs 1.000
53
+ A_lzres lzres_s44_g0 noB vs nn : +0.95 [+0.74,+1.03] p=1.71e-11 certified 1.000 vs 1.000
54
+ A_phys phys_s42_g0 B vs nn : +0.93 [+0.83,+1.05] p=1.63e-11 certified 1.000 vs 1.000
55
+ A_phys phys_s42_g0 noB vs nn : +0.93 [+0.83,+1.05] p=1.63e-11 certified 1.000 vs 1.000
56
+ A_phys phys_s43_g0 B vs nn : +0.82 [+0.68,+0.95] p=1.63e-11 certified 1.000 vs 1.000
57
+ A_phys phys_s43_g0 noB vs nn : +0.82 [+0.68,+0.95] p=1.63e-11 certified 1.000 vs 1.000
58
+ A_phys phys_s44_g0 B vs nn : +0.66 [+0.51,+0.76] p=1.63e-11 certified 1.000 vs 1.000
59
+ A_phys phys_s44_g0 noB vs nn : +0.66 [+0.51,+0.76] p=1.63e-11 certified 1.000 vs 1.000
60
+
61
+ === lvl0 K=8 oracle=strong2 (median oracle J 1.351) ===
62
+ arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired
63
+ base_s0 1.000 1.000 1.000 1.11 | 1.000 1.000 1.11 16.0 0.00
64
+ base_s1 1.000 1.000 1.000 1.17 | 1.000 1.000 1.17 16.0 0.00
65
+ base_s2 1.000 1.000 1.000 1.12 | 1.000 1.000 1.12 16.0 0.00
66
+ lz_s42_g0 1.000 1.000 1.000 1.15 | 1.000 1.000 1.15 16.0 0.00
67
+ lz_s42_g1 1.000 1.000 1.000 1.14 | 1.000 1.000 1.14 16.0 0.00
68
+ lz_s42_g2 1.000 1.000 1.000 1.10 | 1.000 1.000 1.10 16.0 0.00
69
+ lz_s43_g0 1.000 1.000 1.000 1.23 | 1.000 1.000 1.23 16.0 0.00
70
+ lz_s44_g0 1.000 1.000 1.000 1.12 | 1.000 1.000 1.12 16.0 0.00
71
+ lzm_s42_g0 1.000 1.000 1.000 1.32 | 1.000 1.000 1.32 16.0 0.00
72
+ lzm_s43_g0 1.000 1.000 1.000 1.38 | 1.000 1.000 1.38 16.0 0.00
73
+ lzm_s44_g0 1.000 1.000 1.000 1.11 | 1.000 1.000 1.11 16.0 0.00
74
+ phys_s42_g0 1.000 1.000 1.000 1.37 | 1.000 1.000 1.37 16.0 0.00
75
+ phys_s43_g0 1.000 1.000 1.000 1.30 | 1.000 1.000 1.30 16.0 0.00
76
+ phys_s44_g0 1.000 1.000 1.000 1.15 | 1.000 1.000 1.15 16.0 0.00
77
+ nn 1.000 1.000 1.000 0.48 | 1.000 1.000 0.48 16.0 0.00
78
+ idw 1.000 1.000 1.000 0.48 | 1.000 1.000 0.48 16.0 0.00
79
+ nns 1.000 1.000 1.000 1.12 | 1.000 1.000 1.12 16.0 0.00
80
+ dls 1.000 1.000 1.000 1.23 | 1.000 1.000 1.23 16.0 0.00
81
+ lzres_s42_g0 1.000 1.000 1.000 1.29 | 1.000 1.000 1.29 16.0 0.00
82
+ lzres_s43_g0 1.000 1.000 1.000 1.28 | 1.000 1.000 1.28 16.0 0.00
83
+ lzres_s44_g0 1.000 1.000 1.000 1.30 | 1.000 1.000 1.30 16.0 0.00
84
+
85
+ -- paired Wilcoxon on regret, model minus best retrieval (B: best=nn; noB: best=nn); negative = model better; [95% CI of paired median]
86
+ base base_s0 B vs nn : +0.68 [+0.46,+0.84] p=2.21e-10 certified 1.000 vs 1.000
87
+ base base_s0 noB vs nn : +0.68 [+0.46,+0.84] p=2.21e-10 certified 1.000 vs 1.000
88
+ base base_s1 B vs nn : +0.76 [+0.56,+0.96] p=1.74e-10 certified 1.000 vs 1.000
89
+ base base_s1 noB vs nn : +0.76 [+0.56,+0.96] p=1.74e-10 certified 1.000 vs 1.000
90
+ base base_s2 B vs nn : +0.68 [+0.57,+0.80] p=1.58e-10 certified 1.000 vs 1.000
91
+ base base_s2 noB vs nn : +0.68 [+0.57,+0.80] p=1.58e-10 certified 1.000 vs 1.000
92
+ A_lz lz_s42_g0 B vs nn : +0.69 [+0.56,+0.89] p=1.9e-11 certified 1.000 vs 1.000
93
+ A_lz lz_s42_g0 noB vs nn : +0.69 [+0.56,+0.89] p=1.9e-11 certified 1.000 vs 1.000
94
+ A_lz lz_s42_g1 B vs nn : +0.68 [+0.60,+0.77] p=3.29e-11 certified 1.000 vs 1.000
95
+ A_lz lz_s42_g1 noB vs nn : +0.68 [+0.60,+0.77] p=3.29e-11 certified 1.000 vs 1.000
96
+ A_lz lz_s42_g2 B vs nn : +0.72 [+0.63,+0.83] p=6.92e-11 certified 1.000 vs 1.000
97
+ A_lz lz_s42_g2 noB vs nn : +0.72 [+0.63,+0.83] p=6.92e-11 certified 1.000 vs 1.000
98
+ A_lz lz_s43_g0 B vs nn : +0.76 [+0.65,+0.95] p=1.71e-11 certified 1.000 vs 1.000
99
+ A_lz lz_s43_g0 noB vs nn : +0.76 [+0.65,+0.95] p=1.71e-11 certified 1.000 vs 1.000
100
+ A_lz lz_s44_g0 B vs nn : +0.62 [+0.49,+0.78] p=2.84e-11 certified 1.000 vs 1.000
101
+ A_lz lz_s44_g0 noB vs nn : +0.62 [+0.49,+0.78] p=2.84e-11 certified 1.000 vs 1.000
102
+ A_lzm lzm_s42_g0 B vs nn : +0.82 [+0.73,+0.96] p=1.63e-11 certified 1.000 vs 1.000
103
+ A_lzm lzm_s42_g0 noB vs nn : +0.82 [+0.73,+0.96] p=1.63e-11 certified 1.000 vs 1.000
104
+ A_lzm lzm_s43_g0 B vs nn : +0.88 [+0.78,+1.00] p=1.63e-11 certified 1.000 vs 1.000
105
+ A_lzm lzm_s43_g0 noB vs nn : +0.88 [+0.78,+1.00] p=1.63e-11 certified 1.000 vs 1.000
106
+ A_lzm lzm_s44_g0 B vs nn : +0.59 [+0.46,+0.71] p=1.63e-11 certified 1.000 vs 1.000
107
+ A_lzm lzm_s44_g0 noB vs nn : +0.59 [+0.46,+0.71] p=1.63e-11 certified 1.000 vs 1.000
108
+ A_lzres lzres_s42_g0 B vs nn : +0.92 [+0.72,+0.98] p=1.9e-11 certified 1.000 vs 1.000
109
+ A_lzres lzres_s42_g0 noB vs nn : +0.92 [+0.72,+0.98] p=1.9e-11 certified 1.000 vs 1.000
110
+ A_lzres lzres_s43_g0 B vs nn : +0.92 [+0.78,+0.97] p=1.9e-11 certified 1.000 vs 1.000
111
+ A_lzres lzres_s43_g0 noB vs nn : +0.92 [+0.78,+0.97] p=1.9e-11 certified 1.000 vs 1.000
112
+ A_lzres lzres_s44_g0 B vs nn : +0.95 [+0.74,+1.03] p=1.71e-11 certified 1.000 vs 1.000
113
+ A_lzres lzres_s44_g0 noB vs nn : +0.95 [+0.74,+1.03] p=1.71e-11 certified 1.000 vs 1.000
114
+ A_phys phys_s42_g0 B vs nn : +0.93 [+0.83,+1.05] p=1.63e-11 certified 1.000 vs 1.000
115
+ A_phys phys_s42_g0 noB vs nn : +0.93 [+0.83,+1.05] p=1.63e-11 certified 1.000 vs 1.000
116
+ A_phys phys_s43_g0 B vs nn : +0.82 [+0.68,+0.95] p=1.63e-11 certified 1.000 vs 1.000
117
+ A_phys phys_s43_g0 noB vs nn : +0.82 [+0.68,+0.95] p=1.63e-11 certified 1.000 vs 1.000
118
+ A_phys phys_s44_g0 B vs nn : +0.66 [+0.51,+0.76] p=1.63e-11 certified 1.000 vs 1.000
119
+ A_phys phys_s44_g0 noB vs nn : +0.66 [+0.51,+0.76] p=1.63e-11 certified 1.000 vs 1.000
120
+
121
+ === lvl1.1 K=8 oracle=old (median oracle J 0.984) ===
122
+ arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired
123
+ base_s0 1.000 1.000 1.000 0.92 | 1.000 1.000 0.92 16.0 0.00
124
+ base_s1 1.000 1.000 1.000 0.92 | 1.000 1.000 0.92 16.0 0.00
125
+ base_s2 1.000 1.000 1.000 0.84 | 1.000 1.000 0.84 16.0 0.00
126
+ lz_s42_g0 1.000 1.000 1.000 1.04 | 1.000 1.000 1.04 16.0 0.00
127
+ lz_s42_g1 1.000 1.000 1.000 1.04 | 1.000 1.000 1.04 16.0 0.00
128
+ lz_s42_g2 1.000 1.000 1.000 1.03 | 1.000 1.000 1.03 16.0 0.00
129
+ lz_s43_g0 1.000 1.000 1.000 1.11 | 1.000 1.000 1.11 16.0 0.00
130
+ lz_s44_g0 1.000 1.000 1.000 0.98 | 1.000 1.000 0.98 16.0 0.00
131
+ lzm_s42_g0 1.000 1.000 1.000 1.24 | 1.000 1.000 1.24 16.0 0.00
132
+ lzm_s43_g0 1.000 1.000 1.000 1.28 | 1.000 1.000 1.28 16.0 0.00
133
+ lzm_s44_g0 1.000 1.000 1.000 1.03 | 1.000 1.000 1.03 16.0 0.00
134
+ phys_s42_g0 1.000 1.000 1.000 1.29 | 1.000 1.000 1.29 16.0 0.00
135
+ phys_s43_g0 1.000 1.000 1.000 1.27 | 1.000 1.000 1.27 16.0 0.00
136
+ phys_s44_g0 1.000 1.000 1.000 1.10 | 1.000 1.000 1.10 16.0 0.00
137
+ nn 1.000 1.000 1.000 0.53 | 1.000 1.000 0.53 16.0 0.00
138
+ idw 1.000 1.000 1.000 0.55 | 1.000 1.000 0.55 16.0 0.00
139
+ nns 1.000 1.000 1.000 0.95 | 1.000 1.000 0.95 16.0 0.00
140
+ dls 1.000 1.000 1.000 0.95 | 1.000 1.000 0.95 16.0 0.00
141
+ lzres_s42_g0 1.000 1.000 1.000 1.10 | 1.000 1.000 1.10 16.0 0.00
142
+ lzres_s43_g0 1.000 1.000 1.000 1.09 | 1.000 1.000 1.09 16.0 0.00
143
+ lzres_s44_g0 1.000 1.000 1.000 1.10 | 1.000 1.000 1.10 16.0 0.00
144
+
145
+ -- paired Wilcoxon on regret, model minus best retrieval (B: best=nn; noB: best=nn); negative = model better; [95% CI of paired median]
146
+ base base_s0 B vs nn : +0.28 [+0.02,+0.68] p=0.00419 certified 1.000 vs 1.000
147
+ base base_s0 noB vs nn : +0.28 [+0.02,+0.68] p=0.00419 certified 1.000 vs 1.000
148
+ base base_s1 B vs nn : +0.21 [+0.02,+0.69] p=0.00492 certified 1.000 vs 1.000
149
+ base base_s1 noB vs nn : +0.21 [+0.02,+0.69] p=0.00492 certified 1.000 vs 1.000
150
+ base base_s2 B vs nn : +0.31 [+0.01,+0.58] p=0.00753 certified 1.000 vs 1.000
151
+ base base_s2 noB vs nn : +0.31 [+0.01,+0.58] p=0.00753 certified 1.000 vs 1.000
152
+ A_lz lz_s42_g0 B vs nn : +0.63 [+0.35,+0.80] p=0.000179 certified 1.000 vs 1.000
153
+ A_lz lz_s42_g0 noB vs nn : +0.63 [+0.35,+0.80] p=0.000179 certified 1.000 vs 1.000
154
+ A_lz lz_s42_g1 B vs nn : +0.56 [+0.41,+0.80] p=0.000458 certified 1.000 vs 1.000
155
+ A_lz lz_s42_g1 noB vs nn : +0.56 [+0.41,+0.80] p=0.000458 certified 1.000 vs 1.000
156
+ A_lz lz_s42_g2 B vs nn : +0.62 [+0.36,+0.80] p=0.000239 certified 1.000 vs 1.000
157
+ A_lz lz_s42_g2 noB vs nn : +0.62 [+0.36,+0.80] p=0.000239 certified 1.000 vs 1.000
158
+ A_lz lz_s43_g0 B vs nn : +0.68 [+0.49,+0.87] p=6.21e-05 certified 1.000 vs 1.000
159
+ A_lz lz_s43_g0 noB vs nn : +0.68 [+0.49,+0.87] p=6.21e-05 certified 1.000 vs 1.000
160
+ A_lz lz_s44_g0 B vs nn : +0.52 [+0.27,+0.75] p=0.000446 certified 1.000 vs 1.000
161
+ A_lz lz_s44_g0 noB vs nn : +0.52 [+0.27,+0.75] p=0.000446 certified 1.000 vs 1.000
162
+ A_lzm lzm_s42_g0 B vs nn : +0.79 [+0.52,+0.92] p=3.78e-06 certified 1.000 vs 1.000
163
+ A_lzm lzm_s42_g0 noB vs nn : +0.79 [+0.52,+0.92] p=3.78e-06 certified 1.000 vs 1.000
164
+ A_lzm lzm_s43_g0 B vs nn : +0.84 [+0.59,+0.97] p=3.52e-06 certified 1.000 vs 1.000
165
+ A_lzm lzm_s43_g0 noB vs nn : +0.84 [+0.59,+0.97] p=3.52e-06 certified 1.000 vs 1.000
166
+ A_lzm lzm_s44_g0 B vs nn : +0.50 [+0.34,+0.63] p=5.83e-05 certified 1.000 vs 1.000
167
+ A_lzm lzm_s44_g0 noB vs nn : +0.50 [+0.34,+0.63] p=5.83e-05 certified 1.000 vs 1.000
168
+ A_lzres lzres_s42_g0 B vs nn : +0.76 [+0.40,+0.97] p=0.000125 certified 1.000 vs 1.000
169
+ A_lzres lzres_s42_g0 noB vs nn : +0.76 [+0.40,+0.97] p=0.000125 certified 1.000 vs 1.000
170
+ A_lzres lzres_s43_g0 B vs nn : +0.78 [+0.40,+0.97] p=0.000122 certified 1.000 vs 1.000
171
+ A_lzres lzres_s43_g0 noB vs nn : +0.78 [+0.40,+0.97] p=0.000122 certified 1.000 vs 1.000
172
+ A_lzres lzres_s44_g0 B vs nn : +0.79 [+0.39,+0.98] p=0.000118 certified 1.000 vs 1.000
173
+ A_lzres lzres_s44_g0 noB vs nn : +0.79 [+0.39,+0.98] p=0.000118 certified 1.000 vs 1.000
174
+ A_phys phys_s42_g0 B vs nn : +0.82 [+0.53,+1.06] p=3.4e-06 certified 1.000 vs 1.000
175
+ A_phys phys_s42_g0 noB vs nn : +0.82 [+0.53,+1.06] p=3.4e-06 certified 1.000 vs 1.000
176
+ A_phys phys_s43_g0 B vs nn : +0.84 [+0.49,+0.98] p=4.06e-06 certified 1.000 vs 1.000
177
+ A_phys phys_s43_g0 noB vs nn : +0.84 [+0.49,+0.98] p=4.06e-06 certified 1.000 vs 1.000
178
+ A_phys phys_s44_g0 B vs nn : +0.64 [+0.38,+0.80] p=2.23e-05 certified 1.000 vs 1.000
179
+ A_phys phys_s44_g0 noB vs nn : +0.64 [+0.38,+0.80] p=2.23e-05 certified 1.000 vs 1.000
180
+
181
+ === lvl1.1 K=8 oracle=strong2 (median oracle J 0.983) ===
182
+ arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired
183
+ base_s0 1.000 1.000 1.000 0.96 | 1.000 1.000 0.96 16.0 0.00
184
+ base_s1 1.000 1.000 1.000 1.02 | 1.000 1.000 1.02 16.0 0.00
185
+ base_s2 1.000 1.000 1.000 0.93 | 1.000 1.000 0.93 16.0 0.00
186
+ lz_s42_g0 1.000 1.000 1.000 1.18 | 1.000 1.000 1.18 16.0 0.00
187
+ lz_s42_g1 1.000 1.000 1.000 1.14 | 1.000 1.000 1.14 16.0 0.00
188
+ lz_s42_g2 1.000 1.000 1.000 1.14 | 1.000 1.000 1.14 16.0 0.00
189
+ lz_s43_g0 1.000 1.000 1.000 1.29 | 1.000 1.000 1.29 16.0 0.00
190
+ lz_s44_g0 1.000 1.000 1.000 1.10 | 1.000 1.000 1.10 16.0 0.00
191
+ lzm_s42_g0 1.000 1.000 1.000 1.30 | 1.000 1.000 1.30 16.0 0.00
192
+ lzm_s43_g0 1.000 1.000 1.000 1.35 | 1.000 1.000 1.35 16.0 0.00
193
+ lzm_s44_g0 1.000 1.000 1.000 1.12 | 1.000 1.000 1.12 16.0 0.00
194
+ phys_s42_g0 1.000 1.000 1.000 1.36 | 1.000 1.000 1.36 16.0 0.00
195
+ phys_s43_g0 1.000 1.000 1.000 1.30 | 1.000 1.000 1.30 16.0 0.00
196
+ phys_s44_g0 1.000 1.000 1.000 1.16 | 1.000 1.000 1.16 16.0 0.00
197
+ nn 1.000 1.000 1.000 0.80 | 1.000 1.000 0.80 16.0 0.00
198
+ idw 1.000 1.000 1.000 0.80 | 1.000 1.000 0.80 16.0 0.00
199
+ nns 1.000 1.000 1.000 1.02 | 1.000 1.000 1.02 16.0 0.00
200
+ dls 1.000 1.000 1.000 1.09 | 1.000 1.000 1.09 16.0 0.00
201
+ lzres_s42_g0 1.000 1.000 1.000 1.19 | 1.000 1.000 1.19 16.0 0.00
202
+ lzres_s43_g0 1.000 1.000 1.000 1.19 | 1.000 1.000 1.19 16.0 0.00
203
+ lzres_s44_g0 1.000 1.000 1.000 1.21 | 1.000 1.000 1.21 16.0 0.00
204
+
205
+ -- paired Wilcoxon on regret, model minus best retrieval (B: best=nn; noB: best=nn); negative = model better; [95% CI of paired median]
206
+ base base_s0 B vs nn : +0.28 [+0.02,+0.68] p=0.00419 certified 1.000 vs 1.000
207
+ base base_s0 noB vs nn : +0.28 [+0.02,+0.68] p=0.00419 certified 1.000 vs 1.000
208
+ base base_s1 B vs nn : +0.21 [+0.02,+0.69] p=0.00481 certified 1.000 vs 1.000
209
+ base base_s1 noB vs nn : +0.21 [+0.02,+0.69] p=0.00481 certified 1.000 vs 1.000
210
+ base base_s2 B vs nn : +0.31 [+0.01,+0.58] p=0.00737 certified 1.000 vs 1.000
211
+ base base_s2 noB vs nn : +0.31 [+0.01,+0.58] p=0.00737 certified 1.000 vs 1.000
212
+ A_lz lz_s42_g0 B vs nn : +0.63 [+0.35,+0.80] p=0.000174 certified 1.000 vs 1.000
213
+ A_lz lz_s42_g0 noB vs nn : +0.63 [+0.35,+0.80] p=0.000174 certified 1.000 vs 1.000
214
+ A_lz lz_s42_g1 B vs nn : +0.56 [+0.41,+0.80] p=0.000446 certified 1.000 vs 1.000
215
+ A_lz lz_s42_g1 noB vs nn : +0.56 [+0.41,+0.80] p=0.000446 certified 1.000 vs 1.000
216
+ A_lz lz_s42_g2 B vs nn : +0.62 [+0.36,+0.80] p=0.000239 certified 1.000 vs 1.000
217
+ A_lz lz_s42_g2 noB vs nn : +0.62 [+0.36,+0.80] p=0.000239 certified 1.000 vs 1.000
218
+ A_lz lz_s43_g0 B vs nn : +0.68 [+0.49,+0.87] p=6.21e-05 certified 1.000 vs 1.000
219
+ A_lz lz_s43_g0 noB vs nn : +0.68 [+0.49,+0.87] p=6.21e-05 certified 1.000 vs 1.000
220
+ A_lz lz_s44_g0 B vs nn : +0.52 [+0.27,+0.75] p=0.000446 certified 1.000 vs 1.000
221
+ A_lz lz_s44_g0 noB vs nn : +0.52 [+0.27,+0.75] p=0.000446 certified 1.000 vs 1.000
222
+ A_lzm lzm_s42_g0 B vs nn : +0.79 [+0.52,+0.92] p=3.65e-06 certified 1.000 vs 1.000
223
+ A_lzm lzm_s42_g0 noB vs nn : +0.79 [+0.52,+0.92] p=3.65e-06 certified 1.000 vs 1.000
224
+ A_lzm lzm_s43_g0 B vs nn : +0.84 [+0.59,+0.97] p=3.52e-06 certified 1.000 vs 1.000
225
+ A_lzm lzm_s43_g0 noB vs nn : +0.84 [+0.59,+0.97] p=3.52e-06 certified 1.000 vs 1.000
226
+ A_lzm lzm_s44_g0 B vs nn : +0.50 [+0.34,+0.63] p=5.83e-05 certified 1.000 vs 1.000
227
+ A_lzm lzm_s44_g0 noB vs nn : +0.50 [+0.34,+0.63] p=5.83e-05 certified 1.000 vs 1.000
228
+ A_lzres lzres_s42_g0 B vs nn : +0.76 [+0.40,+0.97] p=0.000122 certified 1.000 vs 1.000
229
+ A_lzres lzres_s42_g0 noB vs nn : +0.76 [+0.40,+0.97] p=0.000122 certified 1.000 vs 1.000
230
+ A_lzres lzres_s43_g0 B vs nn : +0.78 [+0.40,+0.97] p=0.000125 certified 1.000 vs 1.000
231
+ A_lzres lzres_s43_g0 noB vs nn : +0.78 [+0.40,+0.97] p=0.000125 certified 1.000 vs 1.000
232
+ A_lzres lzres_s44_g0 B vs nn : +0.79 [+0.39,+0.98] p=0.000111 certified 1.000 vs 1.000
233
+ A_lzres lzres_s44_g0 noB vs nn : +0.79 [+0.39,+0.98] p=0.000111 certified 1.000 vs 1.000
234
+ A_phys phys_s42_g0 B vs nn : +0.82 [+0.53,+1.06] p=3.4e-06 certified 1.000 vs 1.000
235
+ A_phys phys_s42_g0 noB vs nn : +0.82 [+0.53,+1.06] p=3.4e-06 certified 1.000 vs 1.000
236
+ A_phys phys_s43_g0 B vs nn : +0.84 [+0.49,+0.98] p=4.06e-06 certified 1.000 vs 1.000
237
+ A_phys phys_s43_g0 noB vs nn : +0.84 [+0.49,+0.98] p=4.06e-06 certified 1.000 vs 1.000
238
+ A_phys phys_s44_g0 B vs nn : +0.64 [+0.38,+0.80] p=2.23e-05 certified 1.000 vs 1.000
239
+ A_phys phys_s44_g0 noB vs nn : +0.64 [+0.38,+0.80] p=2.23e-05 certified 1.000 vs 1.000
240
+
241
+ === lvl1.25 K=8 oracle=old (median oracle J 0.318) ===
242
+ arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired
243
+ base_s0 1.000 1.000 1.000 0.70 | 1.000 1.000 0.70 16.0 0.00
244
+ base_s1 1.000 1.000 1.000 0.67 | 1.000 1.000 0.67 16.0 0.00
245
+ base_s2 1.000 1.000 1.000 0.72 | 1.000 1.000 0.72 16.0 0.00
246
+ lz_s42_g0 1.000 1.000 1.000 1.00 | 1.000 1.000 1.00 16.0 0.00
247
+ lz_s42_g1 1.000 1.000 1.000 0.98 | 1.000 1.000 0.98 16.0 0.00
248
+ lz_s42_g2 1.000 1.000 1.000 1.00 | 1.000 1.000 1.00 16.0 0.00
249
+ lz_s43_g0 1.000 1.000 1.000 1.07 | 1.000 1.000 1.07 16.0 0.00
250
+ lz_s44_g0 1.000 1.000 1.000 0.94 | 1.000 1.000 0.94 16.0 0.00
251
+ lzm_s42_g0 1.000 1.000 1.000 1.14 | 1.000 1.000 1.14 16.0 0.00
252
+ lzm_s43_g0 1.000 1.000 1.000 1.13 | 1.000 1.000 1.13 16.0 0.00
253
+ lzm_s44_g0 1.000 1.000 1.000 1.00 | 1.000 1.000 1.00 16.0 0.00
254
+ phys_s42_g0 1.000 1.000 1.000 1.11 | 1.000 1.000 1.11 16.0 0.00
255
+ phys_s43_g0 1.000 1.000 1.000 1.12 | 1.000 1.000 1.12 16.0 0.00
256
+ phys_s44_g0 1.000 1.000 1.000 1.03 | 1.000 1.000 1.03 16.0 0.00
257
+ nn 1.000 1.000 1.000 1.06 | 1.000 1.000 1.06 16.0 0.00
258
+ idw 1.000 1.000 1.000 1.00 | 1.000 1.000 1.00 16.0 0.00
259
+ nns 1.000 1.000 1.000 0.95 | 1.000 1.000 0.95 16.0 0.00
260
+ dls 1.000 1.000 1.000 0.97 | 1.000 1.000 0.97 16.0 0.00
261
+ lzres_s42_g0 1.000 1.000 1.000 1.00 | 1.000 1.000 1.00 16.0 0.00
262
+ lzres_s43_g0 1.000 1.000 1.000 0.95 | 1.000 1.000 0.95 16.0 0.00
263
+ lzres_s44_g0 1.000 1.000 1.000 0.98 | 1.000 1.000 0.98 16.0 0.00
264
+
265
+ -- paired Wilcoxon on regret, model minus best retrieval (B: best=nns; noB: best=nns); negative = model better; [95% CI of paired median]
266
+ base base_s0 B vs nns: -0.17 [-0.24,-0.10] p=0.0026 certified 1.000 vs 1.000
267
+ base base_s0 noB vs nns: -0.17 [-0.24,-0.10] p=0.0026 certified 1.000 vs 1.000
268
+ base base_s1 B vs nns: -0.17 [-0.21,-0.07] p=0.00117 certified 1.000 vs 1.000
269
+ base base_s1 noB vs nns: -0.17 [-0.21,-0.07] p=0.00117 certified 1.000 vs 1.000
270
+ base base_s2 B vs nns: -0.12 [-0.19,-0.03] p=0.00111 certified 1.000 vs 1.000
271
+ base base_s2 noB vs nns: -0.12 [-0.19,-0.03] p=0.00111 certified 1.000 vs 1.000
272
+ A_lz lz_s42_g0 B vs nns: +0.08 [+0.05,+0.12] p=0.00977 certified 1.000 vs 1.000
273
+ A_lz lz_s42_g0 noB vs nns: +0.08 [+0.05,+0.12] p=0.00977 certified 1.000 vs 1.000
274
+ A_lz lz_s42_g1 B vs nns: +0.09 [+0.05,+0.14] p=0.0109 certified 1.000 vs 1.000
275
+ A_lz lz_s42_g1 noB vs nns: +0.09 [+0.05,+0.14] p=0.0109 certified 1.000 vs 1.000
276
+ A_lz lz_s42_g2 B vs nns: +0.10 [+0.05,+0.18] p=0.00822 certified 1.000 vs 1.000
277
+ A_lz lz_s42_g2 noB vs nns: +0.10 [+0.05,+0.18] p=0.00822 certified 1.000 vs 1.000
278
+ A_lz lz_s43_g0 B vs nns: +0.15 [+0.11,+0.25] p=0.000399 certified 1.000 vs 1.000
279
+ A_lz lz_s43_g0 noB vs nns: +0.15 [+0.11,+0.25] p=0.000399 certified 1.000 vs 1.000
280
+ A_lz lz_s44_g0 B vs nns: +0.05 [-0.01,+0.09] p=0.126 certified 1.000 vs 1.000
281
+ A_lz lz_s44_g0 noB vs nns: +0.05 [-0.01,+0.09] p=0.126 certified 1.000 vs 1.000
282
+ A_lzm lzm_s42_g0 B vs nns: +0.41 [+0.32,+0.45] p=2.21e-11 certified 1.000 vs 1.000
283
+ A_lzm lzm_s42_g0 noB vs nns: +0.41 [+0.32,+0.45] p=2.21e-11 certified 1.000 vs 1.000
284
+ A_lzm lzm_s43_g0 B vs nns: +0.38 [+0.35,+0.44] p=2.56e-11 certified 1.000 vs 1.000
285
+ A_lzm lzm_s43_g0 noB vs nns: +0.38 [+0.35,+0.44] p=2.56e-11 certified 1.000 vs 1.000
286
+ A_lzm lzm_s44_g0 B vs nns: +0.11 [+0.07,+0.21] p=5.83e-05 certified 1.000 vs 1.000
287
+ A_lzm lzm_s44_g0 noB vs nns: +0.11 [+0.07,+0.21] p=5.83e-05 certified 1.000 vs 1.000
288
+ A_lzres lzres_s42_g0 B vs nns: +0.10 [+0.06,+0.17] p=0.0234 certified 1.000 vs 1.000
289
+ A_lzres lzres_s42_g0 noB vs nns: +0.10 [+0.06,+0.17] p=0.0234 certified 1.000 vs 1.000
290
+ A_lzres lzres_s43_g0 B vs nns: +0.11 [+0.06,+0.19] p=0.0277 certified 1.000 vs 1.000
291
+ A_lzres lzres_s43_g0 noB vs nns: +0.11 [+0.06,+0.19] p=0.0277 certified 1.000 vs 1.000
292
+ A_lzres lzres_s44_g0 B vs nns: +0.11 [+0.07,+0.22] p=0.0196 certified 1.000 vs 1.000
293
+ A_lzres lzres_s44_g0 noB vs nns: +0.11 [+0.07,+0.22] p=0.0196 certified 1.000 vs 1.000
294
+ A_phys phys_s42_g0 B vs nns: +0.37 [+0.31,+0.43] p=2.1e-11 certified 1.000 vs 1.000
295
+ A_phys phys_s42_g0 noB vs nns: +0.37 [+0.31,+0.43] p=2.1e-11 certified 1.000 vs 1.000
296
+ A_phys phys_s43_g0 B vs nns: +0.36 [+0.27,+0.48] p=2.98e-11 certified 1.000 vs 1.000
297
+ A_phys phys_s43_g0 noB vs nns: +0.36 [+0.27,+0.48] p=2.98e-11 certified 1.000 vs 1.000
298
+ A_phys phys_s44_g0 B vs nns: +0.15 [+0.09,+0.21] p=2.66e-07 certified 1.000 vs 1.000
299
+ A_phys phys_s44_g0 noB vs nns: +0.15 [+0.09,+0.21] p=2.66e-07 certified 1.000 vs 1.000
300
+
301
+ === lvl1.25 K=8 oracle=strong2 (median oracle J 0.318) ===
302
+ arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired
303
+ base_s0 1.000 1.000 1.000 0.86 | 1.000 1.000 0.86 16.0 0.00
304
+ base_s1 1.000 1.000 1.000 0.90 | 1.000 1.000 0.90 16.0 0.00
305
+ base_s2 1.000 1.000 1.000 0.91 | 1.000 1.000 0.91 16.0 0.00
306
+ lz_s42_g0 1.000 1.000 1.000 1.14 | 1.000 1.000 1.14 16.0 0.00
307
+ lz_s42_g1 1.000 1.000 1.000 1.13 | 1.000 1.000 1.13 16.0 0.00
308
+ lz_s42_g2 1.000 1.000 1.000 1.15 | 1.000 1.000 1.15 16.0 0.00
309
+ lz_s43_g0 1.000 1.000 1.000 1.23 | 1.000 1.000 1.23 16.0 0.00
310
+ lz_s44_g0 1.000 1.000 1.000 1.08 | 1.000 1.000 1.08 16.0 0.00
311
+ lzm_s42_g0 1.000 1.000 1.000 1.29 | 1.000 1.000 1.29 16.0 0.00
312
+ lzm_s43_g0 1.000 1.000 1.000 1.31 | 1.000 1.000 1.31 16.0 0.00
313
+ lzm_s44_g0 1.000 1.000 1.000 1.13 | 1.000 1.000 1.13 16.0 0.00
314
+ phys_s42_g0 1.000 1.000 1.000 1.26 | 1.000 1.000 1.26 16.0 0.00
315
+ phys_s43_g0 1.000 1.000 1.000 1.30 | 1.000 1.000 1.30 16.0 0.00
316
+ phys_s44_g0 1.000 1.000 1.000 1.16 | 1.000 1.000 1.16 16.0 0.00
317
+ nn 1.000 1.000 1.000 1.22 | 1.000 1.000 1.22 16.0 0.00
318
+ idw 1.000 1.000 1.000 1.17 | 1.000 1.000 1.17 16.0 0.00
319
+ nns 1.000 1.000 1.000 1.05 | 1.000 1.000 1.05 16.0 0.00
320
+ dls 1.000 1.000 1.000 1.08 | 1.000 1.000 1.08 16.0 0.00
321
+ lzres_s42_g0 1.000 1.000 1.000 1.17 | 1.000 1.000 1.17 16.0 0.00
322
+ lzres_s43_g0 1.000 1.000 1.000 1.17 | 1.000 1.000 1.17 16.0 0.00
323
+ lzres_s44_g0 1.000 1.000 1.000 1.17 | 1.000 1.000 1.17 16.0 0.00
324
+
325
+ -- paired Wilcoxon on regret, model minus best retrieval (B: best=nns; noB: best=nns); negative = model better; [95% CI of paired median]
326
+ base base_s0 B vs nns: -0.17 [-0.24,-0.10] p=0.00267 certified 1.000 vs 1.000
327
+ base base_s0 noB vs nns: -0.17 [-0.24,-0.10] p=0.00267 certified 1.000 vs 1.000
328
+ base base_s1 B vs nns: -0.17 [-0.21,-0.07] p=0.00117 certified 1.000 vs 1.000
329
+ base base_s1 noB vs nns: -0.17 [-0.21,-0.07] p=0.00117 certified 1.000 vs 1.000
330
+ base base_s2 B vs nns: -0.12 [-0.19,-0.03] p=0.00108 certified 1.000 vs 1.000
331
+ base base_s2 noB vs nns: -0.12 [-0.19,-0.03] p=0.00108 certified 1.000 vs 1.000
332
+ A_lz lz_s42_g0 B vs nns: +0.08 [+0.05,+0.12] p=0.0104 certified 1.000 vs 1.000
333
+ A_lz lz_s42_g0 noB vs nns: +0.08 [+0.05,+0.12] p=0.0104 certified 1.000 vs 1.000
334
+ A_lz lz_s42_g1 B vs nns: +0.09 [+0.05,+0.14] p=0.0111 certified 1.000 vs 1.000
335
+ A_lz lz_s42_g1 noB vs nns: +0.09 [+0.05,+0.14] p=0.0111 certified 1.000 vs 1.000
336
+ A_lz lz_s42_g2 B vs nns: +0.10 [+0.05,+0.18] p=0.00859 certified 1.000 vs 1.000
337
+ A_lz lz_s42_g2 noB vs nns: +0.10 [+0.05,+0.18] p=0.00859 certified 1.000 vs 1.000
338
+ A_lz lz_s43_g0 B vs nns: +0.15 [+0.11,+0.25] p=0.000446 certified 1.000 vs 1.000
339
+ A_lz lz_s43_g0 noB vs nns: +0.15 [+0.11,+0.25] p=0.000446 certified 1.000 vs 1.000
340
+ A_lz lz_s44_g0 B vs nns: +0.05 [-0.01,+0.09] p=0.126 certified 1.000 vs 1.000
341
+ A_lz lz_s44_g0 noB vs nns: +0.05 [-0.01,+0.09] p=0.126 certified 1.000 vs 1.000
342
+ A_lzm lzm_s42_g0 B vs nns: +0.41 [+0.32,+0.45] p=2.21e-11 certified 1.000 vs 1.000
343
+ A_lzm lzm_s42_g0 noB vs nns: +0.41 [+0.32,+0.45] p=2.21e-11 certified 1.000 vs 1.000
344
+ A_lzm lzm_s43_g0 B vs nns: +0.38 [+0.35,+0.44] p=2.56e-11 certified 1.000 vs 1.000
345
+ A_lzm lzm_s43_g0 noB vs nns: +0.38 [+0.35,+0.44] p=2.56e-11 certified 1.000 vs 1.000
346
+ A_lzm lzm_s44_g0 B vs nns: +0.11 [+0.07,+0.21] p=5.83e-05 certified 1.000 vs 1.000
347
+ A_lzm lzm_s44_g0 noB vs nns: +0.11 [+0.07,+0.21] p=5.83e-05 certified 1.000 vs 1.000
348
+ A_lzres lzres_s42_g0 B vs nns: +0.10 [+0.06,+0.17] p=0.0247 certified 1.000 vs 1.000
349
+ A_lzres lzres_s42_g0 noB vs nns: +0.10 [+0.06,+0.17] p=0.0247 certified 1.000 vs 1.000
350
+ A_lzres lzres_s43_g0 B vs nns: +0.11 [+0.06,+0.19] p=0.0283 certified 1.000 vs 1.000
351
+ A_lzres lzres_s43_g0 noB vs nns: +0.11 [+0.06,+0.19] p=0.0283 certified 1.000 vs 1.000
352
+ A_lzres lzres_s44_g0 B vs nns: +0.11 [+0.07,+0.22] p=0.0204 certified 1.000 vs 1.000
353
+ A_lzres lzres_s44_g0 noB vs nns: +0.11 [+0.07,+0.22] p=0.0204 certified 1.000 vs 1.000
354
+ A_phys phys_s42_g0 B vs nns: +0.37 [+0.31,+0.43] p=2.1e-11 certified 1.000 vs 1.000
355
+ A_phys phys_s42_g0 noB vs nns: +0.37 [+0.31,+0.43] p=2.1e-11 certified 1.000 vs 1.000
356
+ A_phys phys_s43_g0 B vs nns: +0.36 [+0.27,+0.49] p=2.98e-11 certified 1.000 vs 1.000
357
+ A_phys phys_s43_g0 noB vs nns: +0.36 [+0.27,+0.49] p=2.98e-11 certified 1.000 vs 1.000
358
+ A_phys phys_s44_g0 B vs nns: +0.15 [+0.09,+0.21] p=2.66e-07 certified 1.000 vs 1.000
359
+ A_phys phys_s44_g0 noB vs nns: +0.15 [+0.09,+0.21] p=2.66e-07 certified 1.000 vs 1.000
360
+
361
+ === lvl1.5 K=8 oracle=old (median oracle J 0.656) ===
362
+ arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired
363
+ base_s0 1.000 0.983 0.983 0.83 | 1.000 1.000 0.83 16.1 0.02
364
+ base_s1 1.000 0.983 0.967 0.92 | 1.000 1.000 0.92 16.1 0.02
365
+ base_s2 1.000 0.983 0.983 0.86 | 1.000 1.000 0.86 16.1 0.02
366
+ lz_s42_g0 1.000 1.000 1.000 1.26 | 1.000 1.000 1.26 16.0 0.00
367
+ lz_s42_g1 1.000 1.000 1.000 1.17 | 1.000 1.000 1.17 16.0 0.00
368
+ lz_s42_g2 1.000 1.000 1.000 1.20 | 1.000 1.000 1.20 16.0 0.00
369
+ lz_s43_g0 1.000 1.000 1.000 1.23 | 1.000 1.000 1.23 16.0 0.00
370
+ lz_s44_g0 1.000 1.000 1.000 1.09 | 1.000 1.000 1.09 16.0 0.00
371
+ lzm_s42_g0 1.000 1.000 1.000 1.26 | 1.000 1.000 1.26 16.0 0.00
372
+ lzm_s43_g0 1.000 1.000 1.000 1.22 | 1.000 1.000 1.22 16.0 0.00
373
+ lzm_s44_g0 1.000 1.000 1.000 1.03 | 1.000 1.000 1.03 16.0 0.00
374
+ phys_s42_g0 1.000 1.000 1.000 1.28 | 1.000 1.000 1.28 16.0 0.00
375
+ phys_s43_g0 1.000 1.000 1.000 1.29 | 1.000 1.000 1.29 16.0 0.00
376
+ phys_s44_g0 1.000 1.000 1.000 1.09 | 1.000 1.000 1.09 16.0 0.00
377
+ nn 1.000 1.000 1.000 1.08 | 1.000 1.000 1.08 16.0 0.00
378
+ idw 1.000 1.000 1.000 1.16 | 1.000 1.000 1.16 16.0 0.00
379
+ nns 1.000 1.000 1.000 0.98 | 1.000 1.000 0.98 16.0 0.00
380
+ dls 1.000 1.000 1.000 1.05 | 1.000 1.000 1.05 16.0 0.00
381
+ lzres_s42_g0 1.000 1.000 1.000 1.17 | 1.000 1.000 1.17 16.0 0.00
382
+ lzres_s43_g0 1.000 1.000 1.000 1.16 | 1.000 1.000 1.16 16.0 0.00
383
+ lzres_s44_g0 1.000 1.000 1.000 1.17 | 1.000 1.000 1.17 16.0 0.00
384
+
385
+ -- paired Wilcoxon on regret, model minus best retrieval (B: best=nns; noB: best=nns); negative = model better; [95% CI of paired median]
386
+ base base_s0 B vs nns: -0.03 [-0.18,+0.10] p=0.462 certified 1.000 vs 1.000
387
+ base base_s0 noB vs nns: -0.03 [-0.18,+0.10] p=0.475 certified 0.983 vs 1.000
388
+ base base_s1 B vs nns: +0.08 [-0.11,+0.37] p=0.069 certified 1.000 vs 1.000
389
+ base base_s1 noB vs nns: +0.08 [-0.11,+0.37] p=0.076 certified 0.967 vs 1.000
390
+ base base_s2 B vs nns: -0.03 [-0.14,+0.09] p=0.691 certified 1.000 vs 1.000
391
+ base base_s2 noB vs nns: -0.03 [-0.14,+0.09] p=0.691 certified 0.983 vs 1.000
392
+ A_lz lz_s42_g0 B vs nns: +0.27 [+0.13,+0.75] p=1.79e-08 certified 1.000 vs 1.000
393
+ A_lz lz_s42_g0 noB vs nns: +0.27 [+0.13,+0.75] p=1.79e-08 certified 1.000 vs 1.000
394
+ A_lz lz_s42_g1 B vs nns: +0.24 [+0.12,+0.69] p=1.32e-08 certified 1.000 vs 1.000
395
+ A_lz lz_s42_g1 noB vs nns: +0.24 [+0.12,+0.69] p=1.32e-08 certified 1.000 vs 1.000
396
+ A_lz lz_s42_g2 B vs nns: +0.22 [+0.11,+0.82] p=4.9e-08 certified 1.000 vs 1.000
397
+ A_lz lz_s42_g2 noB vs nns: +0.22 [+0.11,+0.82] p=4.9e-08 certified 1.000 vs 1.000
398
+ A_lz lz_s43_g0 B vs nns: +0.31 [+0.16,+0.64] p=1.02e-08 certified 1.000 vs 1.000
399
+ A_lz lz_s43_g0 noB vs nns: +0.31 [+0.16,+0.64] p=1.02e-08 certified 1.000 vs 1.000
400
+ A_lz lz_s44_g0 B vs nns: +0.21 [+0.12,+0.59] p=9.83e-08 certified 1.000 vs 1.000
401
+ A_lz lz_s44_g0 noB vs nns: +0.21 [+0.12,+0.59] p=9.83e-08 certified 1.000 vs 1.000
402
+ A_lzm lzm_s42_g0 B vs nns: +0.38 [+0.33,+0.51] p=2.56e-10 certified 1.000 vs 1.000
403
+ A_lzm lzm_s42_g0 noB vs nns: +0.38 [+0.33,+0.51] p=2.56e-10 certified 1.000 vs 1.000
404
+ A_lzm lzm_s43_g0 B vs nns: +0.39 [+0.33,+0.53] p=7.2e-10 certified 1.000 vs 1.000
405
+ A_lzm lzm_s43_g0 noB vs nns: +0.39 [+0.33,+0.53] p=7.2e-10 certified 1.000 vs 1.000
406
+ A_lzm lzm_s44_g0 B vs nns: +0.11 [+0.04,+0.24] p=1.5e-05 certified 1.000 vs 1.000
407
+ A_lzm lzm_s44_g0 noB vs nns: +0.11 [+0.04,+0.24] p=1.5e-05 certified 1.000 vs 1.000
408
+ A_lzres lzres_s42_g0 B vs nns: +0.16 [+0.08,+0.29] p=0.00527 certified 1.000 vs 1.000
409
+ A_lzres lzres_s42_g0 noB vs nns: +0.16 [+0.08,+0.29] p=0.00527 certified 1.000 vs 1.000
410
+ A_lzres lzres_s43_g0 B vs nns: +0.16 [+0.10,+0.32] p=0.00409 certified 1.000 vs 1.000
411
+ A_lzres lzres_s43_g0 noB vs nns: +0.16 [+0.10,+0.32] p=0.00409 certified 1.000 vs 1.000
412
+ A_lzres lzres_s44_g0 B vs nns: +0.14 [+0.07,+0.29] p=0.00936 certified 1.000 vs 1.000
413
+ A_lzres lzres_s44_g0 noB vs nns: +0.14 [+0.07,+0.29] p=0.00936 certified 1.000 vs 1.000
414
+ A_phys phys_s42_g0 B vs nns: +0.43 [+0.32,+0.64] p=1.92e-10 certified 1.000 vs 1.000
415
+ A_phys phys_s42_g0 noB vs nns: +0.43 [+0.32,+0.64] p=1.92e-10 certified 1.000 vs 1.000
416
+ A_phys phys_s43_g0 B vs nns: +0.43 [+0.30,+0.63] p=8.27e-10 certified 1.000 vs 1.000
417
+ A_phys phys_s43_g0 noB vs nns: +0.43 [+0.30,+0.63] p=8.27e-10 certified 1.000 vs 1.000
418
+ A_phys phys_s44_g0 B vs nns: +0.22 [+0.13,+0.45] p=3.51e-08 certified 1.000 vs 1.000
419
+ A_phys phys_s44_g0 noB vs nns: +0.22 [+0.13,+0.45] p=3.51e-08 certified 1.000 vs 1.000
420
+
421
+ === lvl1.5 K=8 oracle=strong2 (median oracle J 0.656) ===
422
+ arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired
423
+ base_s0 1.000 0.983 0.983 0.93 | 1.000 1.000 0.97 16.1 0.02
424
+ base_s1 1.000 0.983 0.967 1.01 | 1.000 1.000 1.10 16.1 0.02
425
+ base_s2 1.000 0.983 0.983 0.98 | 1.000 1.000 1.01 16.1 0.02
426
+ lz_s42_g0 1.000 1.000 1.000 1.33 | 1.000 1.000 1.33 16.0 0.00
427
+ lz_s42_g1 1.000 1.000 1.000 1.32 | 1.000 1.000 1.32 16.0 0.00
428
+ lz_s42_g2 1.000 1.000 1.000 1.33 | 1.000 1.000 1.33 16.0 0.00
429
+ lz_s43_g0 1.000 1.000 1.000 1.40 | 1.000 1.000 1.40 16.0 0.00
430
+ lz_s44_g0 1.000 1.000 1.000 1.27 | 1.000 1.000 1.27 16.0 0.00
431
+ lzm_s42_g0 1.000 1.000 1.000 1.35 | 1.000 1.000 1.35 16.0 0.00
432
+ lzm_s43_g0 1.000 1.000 1.000 1.31 | 1.000 1.000 1.31 16.0 0.00
433
+ lzm_s44_g0 1.000 1.000 1.000 1.15 | 1.000 1.000 1.15 16.0 0.00
434
+ phys_s42_g0 1.000 1.000 1.000 1.37 | 1.000 1.000 1.37 16.0 0.00
435
+ phys_s43_g0 1.000 1.000 1.000 1.35 | 1.000 1.000 1.35 16.0 0.00
436
+ phys_s44_g0 1.000 1.000 1.000 1.23 | 1.000 1.000 1.23 16.0 0.00
437
+ nn 1.000 1.000 1.000 1.18 | 1.000 1.000 1.18 16.0 0.00
438
+ idw 1.000 1.000 1.000 1.28 | 1.000 1.000 1.28 16.0 0.00
439
+ nns 1.000 1.000 1.000 1.12 | 1.000 1.000 1.12 16.0 0.00
440
+ dls 1.000 1.000 1.000 1.22 | 1.000 1.000 1.22 16.0 0.00
441
+ lzres_s42_g0 1.000 1.000 1.000 1.37 | 1.000 1.000 1.37 16.0 0.00
442
+ lzres_s43_g0 1.000 1.000 1.000 1.36 | 1.000 1.000 1.36 16.0 0.00
443
+ lzres_s44_g0 1.000 1.000 1.000 1.39 | 1.000 1.000 1.39 16.0 0.00
444
+
445
+ -- paired Wilcoxon on regret, model minus best retrieval (B: best=nns; noB: best=nns); negative = model better; [95% CI of paired median]
446
+ base base_s0 B vs nns: -0.03 [-0.18,+0.10] p=0.475 certified 1.000 vs 1.000
447
+ base base_s0 noB vs nns: -0.03 [-0.18,+0.10] p=0.484 certified 0.983 vs 1.000
448
+ base base_s1 B vs nns: +0.08 [-0.11,+0.37] p=0.0701 certified 1.000 vs 1.000
449
+ base base_s1 noB vs nns: +0.08 [-0.11,+0.37] p=0.0748 certified 0.967 vs 1.000
450
+ base base_s2 B vs nns: -0.03 [-0.14,+0.09] p=0.686 certified 1.000 vs 1.000
451
+ base base_s2 noB vs nns: -0.03 [-0.14,+0.09] p=0.686 certified 0.983 vs 1.000
452
+ A_lz lz_s42_g0 B vs nns: +0.27 [+0.13,+0.80] p=1.86e-08 certified 1.000 vs 1.000
453
+ A_lz lz_s42_g0 noB vs nns: +0.27 [+0.13,+0.80] p=1.86e-08 certified 1.000 vs 1.000
454
+ A_lz lz_s42_g1 B vs nns: +0.25 [+0.12,+0.69] p=1.32e-08 certified 1.000 vs 1.000
455
+ A_lz lz_s42_g1 noB vs nns: +0.25 [+0.12,+0.69] p=1.32e-08 certified 1.000 vs 1.000
456
+ A_lz lz_s42_g2 B vs nns: +0.22 [+0.11,+0.83] p=4.9e-08 certified 1.000 vs 1.000
457
+ A_lz lz_s42_g2 noB vs nns: +0.22 [+0.11,+0.83] p=4.9e-08 certified 1.000 vs 1.000
458
+ A_lz lz_s43_g0 B vs nns: +0.31 [+0.16,+0.64] p=9.77e-09 certified 1.000 vs 1.000
459
+ A_lz lz_s43_g0 noB vs nns: +0.31 [+0.16,+0.64] p=9.77e-09 certified 1.000 vs 1.000
460
+ A_lz lz_s44_g0 B vs nns: +0.21 [+0.12,+0.59] p=9.83e-08 certified 1.000 vs 1.000
461
+ A_lz lz_s44_g0 noB vs nns: +0.21 [+0.12,+0.59] p=9.83e-08 certified 1.000 vs 1.000
462
+ A_lzm lzm_s42_g0 B vs nns: +0.38 [+0.33,+0.51] p=2.56e-10 certified 1.000 vs 1.000
463
+ A_lzm lzm_s42_g0 noB vs nns: +0.38 [+0.33,+0.51] p=2.56e-10 certified 1.000 vs 1.000
464
+ A_lzm lzm_s43_g0 B vs nns: +0.39 [+0.33,+0.53] p=7.2e-10 certified 1.000 vs 1.000
465
+ A_lzm lzm_s43_g0 noB vs nns: +0.39 [+0.33,+0.53] p=7.2e-10 certified 1.000 vs 1.000
466
+ A_lzm lzm_s44_g0 B vs nns: +0.11 [+0.04,+0.24] p=1.55e-05 certified 1.000 vs 1.000
467
+ A_lzm lzm_s44_g0 noB vs nns: +0.11 [+0.04,+0.24] p=1.55e-05 certified 1.000 vs 1.000
468
+ A_lzres lzres_s42_g0 B vs nns: +0.16 [+0.08,+0.29] p=0.00539 certified 1.000 vs 1.000
469
+ A_lzres lzres_s42_g0 noB vs nns: +0.16 [+0.08,+0.29] p=0.00539 certified 1.000 vs 1.000
470
+ A_lzres lzres_s43_g0 B vs nns: +0.16 [+0.10,+0.32] p=0.00409 certified 1.000 vs 1.000
471
+ A_lzres lzres_s43_g0 noB vs nns: +0.16 [+0.10,+0.32] p=0.00409 certified 1.000 vs 1.000
472
+ A_lzres lzres_s44_g0 B vs nns: +0.14 [+0.07,+0.29] p=0.00936 certified 1.000 vs 1.000
473
+ A_lzres lzres_s44_g0 noB vs nns: +0.14 [+0.07,+0.29] p=0.00936 certified 1.000 vs 1.000
474
+ A_phys phys_s42_g0 B vs nns: +0.44 [+0.32,+0.64] p=1.92e-10 certified 1.000 vs 1.000
475
+ A_phys phys_s42_g0 noB vs nns: +0.44 [+0.32,+0.64] p=1.92e-10 certified 1.000 vs 1.000
476
+ A_phys phys_s43_g0 B vs nns: +0.43 [+0.30,+0.63] p=8.27e-10 certified 1.000 vs 1.000
477
+ A_phys phys_s43_g0 noB vs nns: +0.43 [+0.30,+0.63] p=8.27e-10 certified 1.000 vs 1.000
478
+ A_phys phys_s44_g0 B vs nns: +0.22 [+0.13,+0.45] p=3.51e-08 certified 1.000 vs 1.000
479
+ A_phys phys_s44_g0 noB vs nns: +0.22 [+0.13,+0.45] p=3.51e-08 certified 1.000 vs 1.000
480
+
481
+ === lvl2.0 K=8 oracle=old (median oracle J 0.557) ===
482
+ arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired
483
+ base_s0 1.000 0.933 0.833 0.73 | 1.000 1.000 0.75 16.5 0.07
484
+ base_s1 1.000 0.950 0.817 0.66 | 1.000 1.000 0.70 16.4 0.05
485
+ base_s2 1.000 0.950 0.833 0.65 | 1.000 1.000 0.72 16.4 0.05
486
+ lz_s42_g0 1.000 1.000 1.000 0.98 | 1.000 1.000 0.98 16.0 0.00
487
+ lz_s42_g1 1.000 1.000 1.000 0.97 | 1.000 1.000 0.97 16.0 0.00
488
+ lz_s42_g2 1.000 1.000 1.000 1.04 | 1.000 1.000 1.04 16.0 0.00
489
+ lz_s43_g0 1.000 1.000 1.000 1.13 | 1.000 1.000 1.13 16.0 0.00
490
+ lz_s44_g0 1.000 1.000 1.000 0.90 | 1.000 1.000 0.90 16.0 0.00
491
+ lzm_s42_g0 1.000 1.000 1.000 0.95 | 1.000 1.000 0.95 16.0 0.00
492
+ lzm_s43_g0 1.000 1.000 1.000 0.97 | 1.000 1.000 0.97 16.0 0.00
493
+ lzm_s44_g0 1.000 1.000 1.000 0.86 | 1.000 1.000 0.86 16.0 0.00
494
+ phys_s42_g0 1.000 1.000 1.000 1.11 | 1.000 1.000 1.11 16.0 0.00
495
+ phys_s43_g0 1.000 1.000 1.000 1.10 | 1.000 1.000 1.10 16.0 0.00
496
+ phys_s44_g0 1.000 1.000 1.000 0.97 | 1.000 1.000 0.97 16.0 0.00
497
+ nn 0.967 0.950 0.950 4.00 | 1.000 1.000 4.00 16.4 0.05
498
+ idw 0.950 0.933 0.933 4.31 | 1.000 1.000 4.31 16.6 0.07
499
+ nns 1.000 1.000 1.000 0.78 | 1.000 1.000 0.78 16.0 0.00
500
+ dls 1.000 1.000 1.000 0.81 | 1.000 1.000 0.81 16.0 0.00
501
+ lzres_s42_g0 1.000 1.000 1.000 0.94 | 1.000 1.000 0.94 16.0 0.00
502
+ lzres_s43_g0 1.000 1.000 1.000 0.94 | 1.000 1.000 0.94 16.0 0.00
503
+ lzres_s44_g0 1.000 1.000 1.000 0.92 | 1.000 1.000 0.92 16.0 0.00
504
+
505
+ -- paired Wilcoxon on regret, model minus best retrieval (B: best=nns; noB: best=nns); negative = model better; [95% CI of paired median]
506
+ base base_s0 B vs nns: -0.08 [-0.24,+0.21] p=0.768 certified 1.000 vs 1.000
507
+ base base_s0 noB vs nns: -0.08 [-0.20,+0.29] p=0.606 certified 0.833 vs 1.000
508
+ base base_s1 B vs nns: -0.10 [-0.26,+0.13] p=0.837 certified 1.000 vs 1.000
509
+ base base_s1 noB vs nns: -0.10 [-0.31,+0.13] p=0.819 certified 0.817 vs 1.000
510
+ base base_s2 B vs nns: -0.11 [-0.30,+0.22] p=0.729 certified 1.000 vs 1.000
511
+ base base_s2 noB vs nns: -0.12 [-0.30,+0.07] p=0.947 certified 0.833 vs 1.000
512
+ A_lz lz_s42_g0 B vs nns: +0.08 [+0.02,+0.33] p=0.0111 certified 1.000 vs 1.000
513
+ A_lz lz_s42_g0 noB vs nns: +0.08 [+0.02,+0.33] p=0.0111 certified 1.000 vs 1.000
514
+ A_lz lz_s42_g1 B vs nns: +0.07 [+0.02,+0.18] p=0.0174 certified 1.000 vs 1.000
515
+ A_lz lz_s42_g1 noB vs nns: +0.07 [+0.02,+0.18] p=0.0174 certified 1.000 vs 1.000
516
+ A_lz lz_s42_g2 B vs nns: +0.08 [+0.00,+0.28] p=0.0154 certified 1.000 vs 1.000
517
+ A_lz lz_s42_g2 noB vs nns: +0.08 [+0.00,+0.28] p=0.0154 certified 1.000 vs 1.000
518
+ A_lz lz_s43_g0 B vs nns: +0.21 [+0.07,+0.40] p=0.001 certified 1.000 vs 1.000
519
+ A_lz lz_s43_g0 noB vs nns: +0.21 [+0.07,+0.40] p=0.001 certified 1.000 vs 1.000
520
+ A_lz lz_s44_g0 B vs nns: +0.07 [-0.02,+0.35] p=0.0359 certified 1.000 vs 1.000
521
+ A_lz lz_s44_g0 noB vs nns: +0.07 [-0.02,+0.35] p=0.0359 certified 1.000 vs 1.000
522
+ A_lzm lzm_s42_g0 B vs nns: +0.09 [+0.02,+0.30] p=0.00254 certified 1.000 vs 1.000
523
+ A_lzm lzm_s42_g0 noB vs nns: +0.09 [+0.02,+0.30] p=0.00254 certified 1.000 vs 1.000
524
+ A_lzm lzm_s43_g0 B vs nns: +0.09 [+0.01,+0.21] p=0.0059 certified 1.000 vs 1.000
525
+ A_lzm lzm_s43_g0 noB vs nns: +0.09 [+0.01,+0.21] p=0.0059 certified 1.000 vs 1.000
526
+ A_lzm lzm_s44_g0 B vs nns: -0.01 [-0.06,+0.05] p=0.385 certified 1.000 vs 1.000
527
+ A_lzm lzm_s44_g0 noB vs nns: -0.01 [-0.06,+0.05] p=0.385 certified 1.000 vs 1.000
528
+ A_lzres lzres_s42_g0 B vs nns: +0.01 [-0.09,+0.09] p=0.381 certified 1.000 vs 1.000
529
+ A_lzres lzres_s42_g0 noB vs nns: +0.01 [-0.09,+0.09] p=0.381 certified 1.000 vs 1.000
530
+ A_lzres lzres_s43_g0 B vs nns: +0.01 [-0.09,+0.09] p=0.512 certified 1.000 vs 1.000
531
+ A_lzres lzres_s43_g0 noB vs nns: +0.01 [-0.09,+0.09] p=0.512 certified 1.000 vs 1.000
532
+ A_lzres lzres_s44_g0 B vs nns: +0.00 [-0.11,+0.09] p=0.32 certified 1.000 vs 1.000
533
+ A_lzres lzres_s44_g0 noB vs nns: +0.00 [-0.11,+0.09] p=0.32 certified 1.000 vs 1.000
534
+ A_phys phys_s42_g0 B vs nns: +0.21 [+0.03,+0.47] p=6.41e-05 certified 1.000 vs 1.000
535
+ A_phys phys_s42_g0 noB vs nns: +0.21 [+0.03,+0.47] p=6.41e-05 certified 1.000 vs 1.000
536
+ A_phys phys_s43_g0 B vs nns: +0.22 [+0.00,+0.40] p=0.000276 certified 1.000 vs 1.000
537
+ A_phys phys_s43_g0 noB vs nns: +0.22 [+0.00,+0.40] p=0.000276 certified 1.000 vs 1.000
538
+ A_phys phys_s44_g0 B vs nns: +0.07 [+0.00,+0.26] p=0.00539 certified 1.000 vs 1.000
539
+ A_phys phys_s44_g0 noB vs nns: +0.07 [+0.00,+0.26] p=0.00539 certified 1.000 vs 1.000
540
+
541
+ === lvl2.0 K=8 oracle=strong2 (median oracle J 0.555) ===
542
+ arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired
543
+ base_s0 1.000 0.933 0.833 0.87 | 1.000 1.000 0.96 16.5 0.07
544
+ base_s1 1.000 0.950 0.817 0.89 | 1.000 1.000 0.89 16.4 0.05
545
+ base_s2 1.000 0.950 0.833 0.87 | 1.000 1.000 0.97 16.4 0.05
546
+ lz_s42_g0 1.000 1.000 1.000 1.33 | 1.000 1.000 1.33 16.0 0.00
547
+ lz_s42_g1 1.000 1.000 1.000 1.33 | 1.000 1.000 1.33 16.0 0.00
548
+ lz_s42_g2 1.000 1.000 1.000 1.32 | 1.000 1.000 1.32 16.0 0.00
549
+ lz_s43_g0 1.000 1.000 1.000 1.39 | 1.000 1.000 1.39 16.0 0.00
550
+ lz_s44_g0 1.000 1.000 1.000 1.23 | 1.000 1.000 1.23 16.0 0.00
551
+ lzm_s42_g0 1.000 1.000 1.000 1.26 | 1.000 1.000 1.26 16.0 0.00
552
+ lzm_s43_g0 1.000 1.000 1.000 1.21 | 1.000 1.000 1.21 16.0 0.00
553
+ lzm_s44_g0 1.000 1.000 1.000 1.06 | 1.000 1.000 1.06 16.0 0.00
554
+ phys_s42_g0 1.000 1.000 1.000 1.35 | 1.000 1.000 1.35 16.0 0.00
555
+ phys_s43_g0 1.000 1.000 1.000 1.38 | 1.000 1.000 1.38 16.0 0.00
556
+ phys_s44_g0 1.000 1.000 1.000 1.28 | 1.000 1.000 1.28 16.0 0.00
557
+ nn 0.967 0.950 0.950 4.00 | 1.000 1.000 4.00 16.4 0.05
558
+ idw 0.950 0.933 0.933 4.40 | 1.000 1.000 4.40 16.6 0.07
559
+ nns 1.000 1.000 1.000 1.14 | 1.000 1.000 1.14 16.0 0.00
560
+ dls 1.000 1.000 1.000 1.26 | 1.000 1.000 1.26 16.0 0.00
561
+ lzres_s42_g0 1.000 1.000 1.000 1.36 | 1.000 1.000 1.36 16.0 0.00
562
+ lzres_s43_g0 1.000 1.000 1.000 1.36 | 1.000 1.000 1.36 16.0 0.00
563
+ lzres_s44_g0 1.000 1.000 1.000 1.38 | 1.000 1.000 1.38 16.0 0.00
564
+
565
+ -- paired Wilcoxon on regret, model minus best retrieval (B: best=nns; noB: best=nns); negative = model better; [95% CI of paired median]
566
+ base base_s0 B vs nns: -0.08 [-0.24,+0.21] p=0.768 certified 1.000 vs 1.000
567
+ base base_s0 noB vs nns: -0.08 [-0.20,+0.29] p=0.606 certified 0.833 vs 1.000
568
+ base base_s1 B vs nns: -0.10 [-0.26,+0.13] p=0.831 certified 1.000 vs 1.000
569
+ base base_s1 noB vs nns: -0.10 [-0.31,+0.13] p=0.814 certified 0.817 vs 1.000
570
+ base base_s2 B vs nns: -0.12 [-0.30,+0.22] p=0.729 certified 1.000 vs 1.000
571
+ base base_s2 noB vs nns: -0.12 [-0.30,+0.07] p=0.941 certified 0.833 vs 1.000
572
+ A_lz lz_s42_g0 B vs nns: +0.08 [+0.02,+0.33] p=0.0113 certified 1.000 vs 1.000
573
+ A_lz lz_s42_g0 noB vs nns: +0.08 [+0.02,+0.33] p=0.0113 certified 1.000 vs 1.000
574
+ A_lz lz_s42_g1 B vs nns: +0.07 [+0.02,+0.18] p=0.0178 certified 1.000 vs 1.000
575
+ A_lz lz_s42_g1 noB vs nns: +0.07 [+0.02,+0.18] p=0.0178 certified 1.000 vs 1.000
576
+ A_lz lz_s42_g2 B vs nns: +0.08 [+0.00,+0.28] p=0.0151 certified 1.000 vs 1.000
577
+ A_lz lz_s42_g2 noB vs nns: +0.08 [+0.00,+0.28] p=0.0151 certified 1.000 vs 1.000
578
+ A_lz lz_s43_g0 B vs nns: +0.22 [+0.07,+0.40] p=0.00103 certified 1.000 vs 1.000
579
+ A_lz lz_s43_g0 noB vs nns: +0.22 [+0.07,+0.40] p=0.00103 certified 1.000 vs 1.000
580
+ A_lz lz_s44_g0 B vs nns: +0.07 [-0.02,+0.35] p=0.0346 certified 1.000 vs 1.000
581
+ A_lz lz_s44_g0 noB vs nns: +0.07 [-0.02,+0.35] p=0.0346 certified 1.000 vs 1.000
582
+ A_lzm lzm_s42_g0 B vs nns: +0.09 [+0.02,+0.30] p=0.00254 certified 1.000 vs 1.000
583
+ A_lzm lzm_s42_g0 noB vs nns: +0.09 [+0.02,+0.30] p=0.00254 certified 1.000 vs 1.000
584
+ A_lzm lzm_s43_g0 B vs nns: +0.09 [+0.01,+0.21] p=0.0059 certified 1.000 vs 1.000
585
+ A_lzm lzm_s43_g0 noB vs nns: +0.09 [+0.01,+0.21] p=0.0059 certified 1.000 vs 1.000
586
+ A_lzm lzm_s44_g0 B vs nns: -0.01 [-0.06,+0.05] p=0.385 certified 1.000 vs 1.000
587
+ A_lzm lzm_s44_g0 noB vs nns: -0.01 [-0.06,+0.05] p=0.385 certified 1.000 vs 1.000
588
+ A_lzres lzres_s42_g0 B vs nns: +0.01 [-0.09,+0.09] p=0.377 certified 1.000 vs 1.000
589
+ A_lzres lzres_s42_g0 noB vs nns: +0.01 [-0.09,+0.09] p=0.377 certified 1.000 vs 1.000
590
+ A_lzres lzres_s43_g0 B vs nns: +0.01 [-0.09,+0.09] p=0.512 certified 1.000 vs 1.000
591
+ A_lzres lzres_s43_g0 noB vs nns: +0.01 [-0.09,+0.09] p=0.512 certified 1.000 vs 1.000
592
+ A_lzres lzres_s44_g0 B vs nns: +0.00 [-0.11,+0.09] p=0.317 certified 1.000 vs 1.000
593
+ A_lzres lzres_s44_g0 noB vs nns: +0.00 [-0.11,+0.09] p=0.317 certified 1.000 vs 1.000
594
+ A_phys phys_s42_g0 B vs nns: +0.21 [+0.03,+0.48] p=6.41e-05 certified 1.000 vs 1.000
595
+ A_phys phys_s42_g0 noB vs nns: +0.21 [+0.03,+0.48] p=6.41e-05 certified 1.000 vs 1.000
596
+ A_phys phys_s43_g0 B vs nns: +0.22 [+0.00,+0.40] p=0.000284 certified 1.000 vs 1.000
597
+ A_phys phys_s43_g0 noB vs nns: +0.22 [+0.00,+0.40] p=0.000284 certified 1.000 vs 1.000
598
+ A_phys phys_s44_g0 B vs nns: +0.07 [+0.00,+0.26] p=0.00527 certified 1.000 vs 1.000
599
+ A_phys phys_s44_g0 noB vs nns: +0.07 [+0.00,+0.26] p=0.00527 certified 1.000 vs 1.000
600
+
601
+ === pilotfar K=8 oracle=old (median oracle J 65.827) ===
602
+ arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired
603
+ base_s0 1.000 0.033 0.017 -6.49 | 1.000 1.000 -8.50 23.7 0.97
604
+ base_s1 1.000 0.000 0.000 -6.74 | 1.000 1.000 -8.82 24.0 1.00
605
+ base_s2 1.000 0.017 0.000 -6.68 | 1.000 1.000 -8.84 23.9 0.98
606
+ lz_s42_g0 1.000 1.000 1.000 4.97 | 1.000 1.000 4.97 16.0 0.00
607
+ lz_s42_g1 1.000 1.000 1.000 4.77 | 1.000 1.000 4.77 16.0 0.00
608
+ lz_s42_g2 1.000 1.000 1.000 4.82 | 1.000 1.000 4.82 16.0 0.00
609
+ lz_s43_g0 1.000 1.000 1.000 5.97 | 1.000 1.000 5.97 16.0 0.00
610
+ lz_s44_g0 1.000 1.000 1.000 5.17 | 1.000 1.000 5.17 16.0 0.00
611
+ lzm_s42_g0 1.000 1.000 1.000 9.28 | 1.000 1.000 9.28 16.0 0.00
612
+ lzm_s43_g0 1.000 1.000 1.000 10.32 | 1.000 1.000 10.32 16.0 0.00
613
+ lzm_s44_g0 1.000 1.000 1.000 7.79 | 1.000 1.000 7.79 16.0 0.00
614
+ phys_s42_g0 1.000 1.000 1.000 6.76 | 1.000 1.000 6.76 16.0 0.00
615
+ phys_s43_g0 1.000 1.000 1.000 5.36 | 1.000 1.000 5.36 16.0 0.00
616
+ phys_s44_g0 1.000 1.000 1.000 5.39 | 1.000 1.000 5.39 16.0 0.00
617
+ nn 0.717 0.517 0.517 69.64 | 1.000 1.000 55.28 19.9 0.48
618
+ idw 0.383 0.233 0.233 1000.00 | 1.000 1.000 58.52 22.1 0.77
619
+ nns 1.000 1.000 1.000 0.04 | 1.000 1.000 0.04 16.0 0.00
620
+ dls 1.000 1.000 1.000 -1.73 | 1.000 1.000 -1.73 16.0 0.00
621
+ lzres_s42_g0 1.000 1.000 1.000 -4.00 | 1.000 1.000 -4.00 16.0 0.00
622
+ lzres_s43_g0 1.000 1.000 1.000 -3.89 | 1.000 1.000 -3.89 16.0 0.00
623
+ lzres_s44_g0 1.000 1.000 1.000 -4.61 | 1.000 1.000 -4.61 16.0 0.00
624
+
625
+ -- paired Wilcoxon on regret, model minus best retrieval (B: best=dls; noB: best=dls); negative = model better; [95% CI of paired median]
626
+ base base_s0 B vs dls: -6.44 [-6.84,-6.01] p=1.63e-11 certified 1.000 vs 1.000
627
+ base base_s0 noB vs dls: -5.28 [-6.10,-4.50] p=2.95e-06 certified 0.017 vs 1.000
628
+ base base_s1 B vs dls: -6.37 [-6.85,-6.07] p=1.63e-11 certified 1.000 vs 1.000
629
+ base base_s1 noB vs dls: -5.14 [-5.94,-3.69] p=2.95e-06 certified 0.000 vs 1.000
630
+ base base_s2 B vs dls: -6.53 [-6.92,-6.25] p=1.63e-11 certified 1.000 vs 1.000
631
+ base base_s2 noB vs dls: -5.25 [-5.94,-3.91] p=1.38e-08 certified 0.000 vs 1.000
632
+ A_lz lz_s42_g0 B vs dls: +7.02 [+6.04,+8.04] p=1.63e-11 certified 1.000 vs 1.000
633
+ A_lz lz_s42_g0 noB vs dls: +7.02 [+6.04,+8.04] p=1.63e-11 certified 1.000 vs 1.000
634
+ A_lz lz_s42_g1 B vs dls: +6.55 [+5.94,+7.39] p=2.01e-10 certified 1.000 vs 1.000
635
+ A_lz lz_s42_g1 noB vs dls: +6.55 [+5.94,+7.39] p=2.01e-10 certified 1.000 vs 1.000
636
+ A_lz lz_s42_g2 B vs dls: +6.98 [+5.97,+8.41] p=1.63e-11 certified 1.000 vs 1.000
637
+ A_lz lz_s42_g2 noB vs dls: +6.98 [+5.97,+8.41] p=1.63e-11 certified 1.000 vs 1.000
638
+ A_lz lz_s43_g0 B vs dls: +7.87 [+6.93,+8.48] p=1.63e-11 certified 1.000 vs 1.000
639
+ A_lz lz_s43_g0 noB vs dls: +7.87 [+6.93,+8.48] p=1.63e-11 certified 1.000 vs 1.000
640
+ A_lz lz_s44_g0 B vs dls: +6.99 [+5.70,+7.82] p=1.63e-11 certified 1.000 vs 1.000
641
+ A_lz lz_s44_g0 noB vs dls: +6.99 [+5.70,+7.82] p=1.63e-11 certified 1.000 vs 1.000
642
+ A_lzm lzm_s42_g0 B vs dls: +11.32 [+10.43,+12.32] p=1.63e-11 certified 1.000 vs 1.000
643
+ A_lzm lzm_s42_g0 noB vs dls: +11.32 [+10.43,+12.32] p=1.63e-11 certified 1.000 vs 1.000
644
+ A_lzm lzm_s43_g0 B vs dls: +12.82 [+11.50,+13.35] p=1.63e-11 certified 1.000 vs 1.000
645
+ A_lzm lzm_s43_g0 noB vs dls: +12.82 [+11.50,+13.35] p=1.63e-11 certified 1.000 vs 1.000
646
+ A_lzm lzm_s44_g0 B vs dls: +10.05 [+8.95,+10.80] p=1.63e-11 certified 1.000 vs 1.000
647
+ A_lzm lzm_s44_g0 noB vs dls: +10.05 [+8.95,+10.80] p=1.63e-11 certified 1.000 vs 1.000
648
+ A_lzres lzres_s42_g0 B vs dls: -1.58 [-2.07,-1.26] p=1.99e-11 certified 1.000 vs 1.000
649
+ A_lzres lzres_s42_g0 noB vs dls: -1.58 [-2.07,-1.26] p=1.99e-11 certified 1.000 vs 1.000
650
+ A_lzres lzres_s43_g0 B vs dls: -1.64 [-1.95,-1.16] p=1.71e-11 certified 1.000 vs 1.000
651
+ A_lzres lzres_s43_g0 noB vs dls: -1.64 [-1.95,-1.16] p=1.71e-11 certified 1.000 vs 1.000
652
+ A_lzres lzres_s44_g0 B vs dls: -2.33 [-2.94,-1.61] p=1.63e-11 certified 1.000 vs 1.000
653
+ A_lzres lzres_s44_g0 noB vs dls: -2.33 [-2.94,-1.61] p=1.63e-11 certified 1.000 vs 1.000
654
+ A_phys phys_s42_g0 B vs dls: +8.78 [+8.12,+9.94] p=1.63e-11 certified 1.000 vs 1.000
655
+ A_phys phys_s42_g0 noB vs dls: +8.78 [+8.12,+9.94] p=1.63e-11 certified 1.000 vs 1.000
656
+ A_phys phys_s43_g0 B vs dls: +7.69 [+6.84,+8.44] p=1.63e-11 certified 1.000 vs 1.000
657
+ A_phys phys_s43_g0 noB vs dls: +7.69 [+6.84,+8.44] p=1.63e-11 certified 1.000 vs 1.000
658
+ A_phys phys_s44_g0 B vs dls: +7.51 [+6.74,+8.55] p=1.63e-11 certified 1.000 vs 1.000
659
+ A_phys phys_s44_g0 noB vs dls: +7.51 [+6.74,+8.55] p=1.63e-11 certified 1.000 vs 1.000
660
+
661
+ === pilotfar K=8 oracle=strong2 (median oracle J 59.001) ===
662
+ arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired
663
+ base_s0 1.000 0.033 0.017 2.90 | 1.000 1.000 1.03 23.7 0.97
664
+ base_s1 1.000 0.000 0.000 2.88 | 1.000 1.000 0.80 24.0 1.00
665
+ base_s2 1.000 0.017 0.000 2.89 | 1.000 1.000 0.86 23.9 0.98
666
+ lz_s42_g0 1.000 1.000 1.000 16.67 | 1.000 1.000 16.67 16.0 0.00
667
+ lz_s42_g1 1.000 1.000 1.000 16.33 | 1.000 1.000 16.33 16.0 0.00
668
+ lz_s42_g2 1.000 1.000 1.000 16.60 | 1.000 1.000 16.60 16.0 0.00
669
+ lz_s43_g0 1.000 1.000 1.000 17.50 | 1.000 1.000 17.50 16.0 0.00
670
+ lz_s44_g0 1.000 1.000 1.000 16.66 | 1.000 1.000 16.66 16.0 0.00
671
+ lzm_s42_g0 1.000 1.000 1.000 21.71 | 1.000 1.000 21.71 16.0 0.00
672
+ lzm_s43_g0 1.000 1.000 1.000 22.71 | 1.000 1.000 22.71 16.0 0.00
673
+ lzm_s44_g0 1.000 1.000 1.000 19.87 | 1.000 1.000 19.87 16.0 0.00
674
+ phys_s42_g0 1.000 1.000 1.000 18.81 | 1.000 1.000 18.81 16.0 0.00
675
+ phys_s43_g0 1.000 1.000 1.000 17.31 | 1.000 1.000 17.31 16.0 0.00
676
+ phys_s44_g0 1.000 1.000 1.000 17.47 | 1.000 1.000 17.47 16.0 0.00
677
+ nn 0.717 0.517 0.517 86.30 | 1.000 1.000 71.01 19.9 0.48
678
+ idw 0.383 0.233 0.233 1000.00 | 1.000 1.000 76.42 22.1 0.77
679
+ nns 1.000 1.000 1.000 11.35 | 1.000 1.000 11.35 16.0 0.00
680
+ dls 1.000 1.000 1.000 7.95 | 1.000 1.000 7.95 16.0 0.00
681
+ lzres_s42_g0 1.000 1.000 1.000 6.45 | 1.000 1.000 6.45 16.0 0.00
682
+ lzres_s43_g0 1.000 1.000 1.000 6.55 | 1.000 1.000 6.55 16.0 0.00
683
+ lzres_s44_g0 1.000 1.000 1.000 5.76 | 1.000 1.000 5.76 16.0 0.00
684
+
685
+ -- paired Wilcoxon on regret, model minus best retrieval (B: best=dls; noB: best=dls); negative = model better; [95% CI of paired median]
686
+ base base_s0 B vs dls: -7.02 [-7.55,-6.70] p=1.63e-11 certified 1.000 vs 1.000
687
+ base base_s0 noB vs dls: -5.73 [-6.77,-4.94] p=2.95e-06 certified 0.017 vs 1.000
688
+ base base_s1 B vs dls: -7.08 [-7.56,-6.66] p=1.63e-11 certified 1.000 vs 1.000
689
+ base base_s1 noB vs dls: -5.62 [-6.48,-4.07] p=2.95e-06 certified 0.000 vs 1.000
690
+ base base_s2 B vs dls: -7.26 [-7.65,-6.94] p=1.63e-11 certified 1.000 vs 1.000
691
+ base base_s2 noB vs dls: -5.85 [-6.45,-4.35] p=1.32e-08 certified 0.000 vs 1.000
692
+ A_lz lz_s42_g0 B vs dls: +7.78 [+6.68,+8.89] p=1.63e-11 certified 1.000 vs 1.000
693
+ A_lz lz_s42_g0 noB vs dls: +7.78 [+6.68,+8.89] p=1.63e-11 certified 1.000 vs 1.000
694
+ A_lz lz_s42_g1 B vs dls: +7.15 [+6.54,+8.15] p=2.01e-10 certified 1.000 vs 1.000
695
+ A_lz lz_s42_g1 noB vs dls: +7.15 [+6.54,+8.15] p=2.01e-10 certified 1.000 vs 1.000
696
+ A_lz lz_s42_g2 B vs dls: +7.76 [+6.52,+9.10] p=1.63e-11 certified 1.000 vs 1.000
697
+ A_lz lz_s42_g2 noB vs dls: +7.76 [+6.52,+9.10] p=1.63e-11 certified 1.000 vs 1.000
698
+ A_lz lz_s43_g0 B vs dls: +8.80 [+7.65,+9.50] p=1.63e-11 certified 1.000 vs 1.000
699
+ A_lz lz_s43_g0 noB vs dls: +8.80 [+7.65,+9.50] p=1.63e-11 certified 1.000 vs 1.000
700
+ A_lz lz_s44_g0 B vs dls: +7.70 [+6.37,+8.69] p=1.63e-11 certified 1.000 vs 1.000
701
+ A_lz lz_s44_g0 noB vs dls: +7.70 [+6.37,+8.69] p=1.63e-11 certified 1.000 vs 1.000
702
+ A_lzm lzm_s42_g0 B vs dls: +12.52 [+11.49,+13.81] p=1.63e-11 certified 1.000 vs 1.000
703
+ A_lzm lzm_s42_g0 noB vs dls: +12.52 [+11.49,+13.81] p=1.63e-11 certified 1.000 vs 1.000
704
+ A_lzm lzm_s43_g0 B vs dls: +14.22 [+12.55,+14.89] p=1.63e-11 certified 1.000 vs 1.000
705
+ A_lzm lzm_s43_g0 noB vs dls: +14.22 [+12.55,+14.89] p=1.63e-11 certified 1.000 vs 1.000
706
+ A_lzm lzm_s44_g0 B vs dls: +11.04 [+9.86,+12.05] p=1.63e-11 certified 1.000 vs 1.000
707
+ A_lzm lzm_s44_g0 noB vs dls: +11.04 [+9.86,+12.05] p=1.63e-11 certified 1.000 vs 1.000
708
+ A_lzres lzres_s42_g0 B vs dls: -1.77 [-2.27,-1.40] p=1.9e-11 certified 1.000 vs 1.000
709
+ A_lzres lzres_s42_g0 noB vs dls: -1.77 [-2.27,-1.40] p=1.9e-11 certified 1.000 vs 1.000
710
+ A_lzres lzres_s43_g0 B vs dls: -1.81 [-2.16,-1.27] p=1.71e-11 certified 1.000 vs 1.000
711
+ A_lzres lzres_s43_g0 noB vs dls: -1.81 [-2.16,-1.27] p=1.71e-11 certified 1.000 vs 1.000
712
+ A_lzres lzres_s44_g0 B vs dls: -2.58 [-3.23,-1.80] p=1.63e-11 certified 1.000 vs 1.000
713
+ A_lzres lzres_s44_g0 noB vs dls: -2.58 [-3.23,-1.80] p=1.63e-11 certified 1.000 vs 1.000
714
+ A_phys phys_s42_g0 B vs dls: +9.73 [+8.91,+10.99] p=1.63e-11 certified 1.000 vs 1.000
715
+ A_phys phys_s42_g0 noB vs dls: +9.73 [+8.91,+10.99] p=1.63e-11 certified 1.000 vs 1.000
716
+ A_phys phys_s43_g0 B vs dls: +8.56 [+7.48,+9.38] p=1.63e-11 certified 1.000 vs 1.000
717
+ A_phys phys_s43_g0 noB vs dls: +8.56 [+7.48,+9.38] p=1.63e-11 certified 1.000 vs 1.000
718
+ A_phys phys_s44_g0 B vs dls: +8.31 [+7.46,+9.46] p=1.63e-11 certified 1.000 vs 1.000
719
+ A_phys phys_s44_g0 noB vs dls: +8.31 [+7.46,+9.46] p=1.63e-11 certified 1.000 vs 1.000
720
+
results/physics_gen/pendulum_physics_gen_v2.txt ADDED
@@ -0,0 +1,188 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ === lvl0 K=8, oracle median J 1.351; value = mean over the group's runs of per-plant regret %, then median over plants ===
2
+ group n cov certified(no B) | with B: certified regret_med calls | no B regret_med
3
+ shipped 3 1.000 1.000 | 1.000 1.12 16.0 | 1.12
4
+ dimless-lz, orig labels 3 1.000 1.000 | 1.000 1.16 16.0 | 1.16
5
+ dimless-lzres, orig labels 3 1.000 1.000 | 1.000 1.29 16.0 | 1.29
6
+ tuned phys, multi-A (published) 3 1.000 1.000 | 1.000 0.34 16.0 | 0.34
7
+ tuned phys, multi-B (published) 3 1.000 0.894 | 1.000 0.20 16.0 | 0.19
8
+ A: phys, multi-A (new metric) 3 1.000 0.994 | 1.000 0.33 16.0 | 0.33
9
+ A: dimless-lz, multi-A 3 1.000 1.000 | 1.000 0.33 16.0 | 0.33
10
+ A: dimless-lzres, multi-A 3 1.000 1.000 | 1.000 0.34 16.0 | 0.34
11
+ A: dimless-lz, multi-B 3 1.000 0.783 | 1.000 0.23 16.0 | 0.21
12
+ nn (orig bank) 1 1.000 1.000 | 1.000 0.48 16.0 | 0.48
13
+ nns (orig) 1 1.000 1.000 | 1.000 1.12 16.0 | 1.12
14
+ dls (orig) 1 1.000 1.000 | 1.000 1.23 16.0 | 1.23
15
+ nn (multi-A) 1 1.000 0.983 | 1.000 0.30 16.0 | 0.30
16
+ nns (multi-A) 1 1.000 1.000 | 1.000 0.37 16.0 | 0.37
17
+ dls (multi-A) 1 1.000 1.000 | 1.000 0.30 16.0 | 0.30
18
+ nn (multi-B) 1 1.000 1.000 | 1.000 0.21 16.0 | 0.21
19
+ nns (multi-B) 1 1.000 1.000 | 1.000 0.11 16.0 | 0.11
20
+ dls (multi-B) 1 1.000 1.000 | 1.000 0.14 16.0 | 0.14
21
+ -- best retrieval arm with B: nns (multi-B); best original-bank arm: nn (orig bank). paired Wilcoxon, group minus retrieval (neg = model better) [95% CI of paired median], p
22
+ shipped B vs nns (multi-B) : +0.90 [+0.73,+1.14] p=5.41e-11
23
+ shipped B vs nn (orig bank): +0.66 [+0.58,+0.88] p=1.44e-10
24
+ dimless-lz, orig labels B vs nns (multi-B) : +0.89 [+0.71,+1.06] p=1.71e-11
25
+ dimless-lz, orig labels B vs nn (orig bank): +0.69 [+0.56,+0.84] p=1.9e-11
26
+ dimless-lzres, orig labels B vs nns (multi-B) : +1.10 [+1.02,+1.21] p=1.71e-11
27
+ dimless-lzres, orig labels B vs nn (orig bank): +0.93 [+0.77,+0.97] p=1.9e-11
28
+ tuned phys, multi-A (published) B vs nns (multi-B) : +0.10 [+0.05,+0.17] p=2.37e-06
29
+ tuned phys, multi-A (published) B vs nn (orig bank): -0.06 [-0.15,+0.01] p=0.0126
30
+ tuned phys, multi-B (published) B vs nns (multi-B) : +0.05 [+0.02,+0.07] p=6.21e-05
31
+ tuned phys, multi-B (published) B vs nn (orig bank): -0.09 [-0.14,-0.01] p=0.000276
32
+ A: phys, multi-A (new metric) B vs nns (multi-B) : +0.10 [+0.05,+0.15] p=4.09e-07
33
+ A: phys, multi-A (new metric) B vs nn (orig bank): -0.03 [-0.13,+0.03] p=0.0238
34
+ A: dimless-lz, multi-A B vs nns (multi-B) : +0.14 [+0.05,+0.18] p=1.42e-06
35
+ A: dimless-lz, multi-A B vs nn (orig bank): +0.02 [-0.14,+0.05] p=0.122
36
+ A: dimless-lzres, multi-A B vs nns (multi-B) : +0.11 [+0.06,+0.22] p=2.97e-09
37
+ A: dimless-lzres, multi-A B vs nn (orig bank): +0.02 [-0.08,+0.07] p=0.233
38
+ A: dimless-lz, multi-B B vs nns (multi-B) : +0.07 [+0.03,+0.11] p=0.000207
39
+ A: dimless-lz, multi-B B vs nn (orig bank): -0.02 [-0.17,+0.02] p=0.0069
40
+ -- factor decomposition (paired, with B): data = tuned-phys-multi-A vs shipped; recipe = A-phys(new metric) vs published tuned; physics = A-lz vs A-phys; trend = A-lzres vs A-lz; multiplicity = multi-B lz vs multi-A lz
41
+ tuned phys, multi-A (published) - shipped : -0.83 [-1.12,-0.57] p=8.42e-11
42
+ A: phys, multi-A (new metric) - tuned phys, multi-A (published) : +0.00 [-0.01,+0.02] p=0.401
43
+ A: dimless-lz, multi-A - A: phys, multi-A (new metric) : +0.00 [-0.03,+0.04] p=0.977
44
+ A: dimless-lzres, multi-A - A: dimless-lz, multi-A : +0.04 [+0.01,+0.06] p=0.354
45
+ A: dimless-lz, multi-B - A: dimless-lz, multi-A : -0.04 [-0.07,-0.02] p=0.000133
46
+ dimless-lz, orig labels - shipped : +0.08 [-0.07,+0.18] p=0.462
47
+
48
+ === lvl1.5 K=8, oracle median J 0.656; value = mean over the group's runs of per-plant regret %, then median over plants ===
49
+ group n cov certified(no B) | with B: certified regret_med calls | no B regret_med
50
+ shipped 3 1.000 0.978 | 1.000 1.00 16.1 | 0.95
51
+ dimless-lz, orig labels 3 1.000 1.000 | 1.000 1.33 16.0 | 1.33
52
+ dimless-lzres, orig labels 3 1.000 1.000 | 1.000 1.38 16.0 | 1.38
53
+ tuned phys, multi-A (published) 3 1.000 0.783 | 1.000 1.06 16.8 | 0.90
54
+ tuned phys, multi-B (published) 3 1.000 0.689 | 1.000 0.84 16.6 | 0.77
55
+ A: phys, multi-A (new metric) 3 1.000 0.806 | 1.000 1.12 16.7 | 0.66
56
+ A: dimless-lz, multi-A 3 1.000 1.000 | 1.000 1.26 16.0 | 1.26
57
+ A: dimless-lzres, multi-A 3 1.000 0.989 | 1.000 0.97 16.0 | 0.97
58
+ A: dimless-lz, multi-B 3 1.000 0.622 | 1.000 0.82 16.0 | 0.70
59
+ nn (orig bank) 1 1.000 1.000 | 1.000 1.18 16.0 | 1.18
60
+ nns (orig) 1 1.000 1.000 | 1.000 1.15 16.0 | 1.15
61
+ dls (orig) 1 1.000 1.000 | 1.000 1.22 16.0 | 1.22
62
+ nn (multi-A) 1 1.000 1.000 | 1.000 1.21 16.0 | 1.21
63
+ nns (multi-A) 1 1.000 1.000 | 1.000 0.92 16.0 | 0.92
64
+ dls (multi-A) 1 1.000 0.983 | 1.000 0.99 16.0 | 0.99
65
+ nn (multi-B) 1 1.000 0.933 | 1.000 1.25 16.0 | 1.22
66
+ nns (multi-B) 1 1.000 1.000 | 1.000 0.59 16.0 | 0.59
67
+ dls (multi-B) 1 1.000 0.983 | 1.000 0.56 16.0 | 0.56
68
+ -- best retrieval arm with B: dls (multi-B); best original-bank arm: nns (orig). paired Wilcoxon, group minus retrieval (neg = model better) [95% CI of paired median], p
69
+ shipped B vs dls (multi-B) : +0.10 [-0.04,+0.88] p=0.0113
70
+ shipped B vs nns (orig) : +0.05 [-0.09,+0.21] p=0.227
71
+ dimless-lz, orig labels B vs dls (multi-B) : +0.58 [+0.23,+1.69] p=3.75e-05
72
+ dimless-lz, orig labels B vs nns (orig) : +0.24 [+0.15,+0.70] p=1.79e-08
73
+ dimless-lzres, orig labels B vs dls (multi-B) : +0.86 [+0.43,+1.35] p=1.07e-05
74
+ dimless-lzres, orig labels B vs nns (orig) : +0.16 [+0.08,+0.31] p=0.0047
75
+ tuned phys, multi-A (published) B vs dls (multi-B) : +0.14 [+0.05,+0.81] p=0.000974
76
+ tuned phys, multi-A (published) B vs nns (orig) : -0.20 [-0.37,-0.01] p=0.149
77
+ tuned phys, multi-B (published) B vs dls (multi-B) : -0.01 [-0.11,+0.17] p=0.335
78
+ tuned phys, multi-B (published) B vs nns (orig) : -0.22 [-0.49,-0.07] p=0.000602
79
+ A: phys, multi-A (new metric) B vs dls (multi-B) : +0.10 [+0.02,+0.58] p=0.004
80
+ A: phys, multi-A (new metric) B vs nns (orig) : -0.24 [-0.48,+0.03] p=0.0701
81
+ A: dimless-lz, multi-A B vs dls (multi-B) : +0.07 [-0.02,+0.45] p=0.00267
82
+ A: dimless-lz, multi-A B vs nns (orig) : -0.33 [-0.71,+0.07] p=0.00242
83
+ A: dimless-lzres, multi-A B vs dls (multi-B) : +0.01 [-0.01,+0.18] p=0.0773
84
+ A: dimless-lzres, multi-A B vs nns (orig) : -0.05 [-0.60,+0.10] p=0.0059
85
+ A: dimless-lz, multi-B B vs dls (multi-B) : +0.03 [-0.05,+0.25] p=0.00998
86
+ A: dimless-lz, multi-B B vs nns (orig) : -0.33 [-0.80,+0.01] p=0.00254
87
+ -- factor decomposition (paired, with B): data = tuned-phys-multi-A vs shipped; recipe = A-phys(new metric) vs published tuned; physics = A-lz vs A-phys; trend = A-lzres vs A-lz; multiplicity = multi-B lz vs multi-A lz
88
+ tuned phys, multi-A (published) - shipped : -0.25 [-0.78,-0.10] p=0.00492
89
+ A: phys, multi-A (new metric) - tuned phys, multi-A (published) : -0.03 [-0.12,+0.03] p=0.466
90
+ A: dimless-lz, multi-A - A: phys, multi-A (new metric) : -0.01 [-0.11,+0.33] p=0.774
91
+ A: dimless-lzres, multi-A - A: dimless-lz, multi-A : -0.08 [-0.39,+0.00] p=0.000809
92
+ A: dimless-lz, multi-B - A: dimless-lz, multi-A : -0.04 [-0.08,+0.02] p=0.0393
93
+ dimless-lz, orig labels - shipped : +0.36 [+0.18,+0.58] p=7.88e-06
94
+
95
+ === lvl2.0 K=8, oracle median J 0.554; value = mean over the group's runs of per-plant regret %, then median over plants ===
96
+ group n cov certified(no B) | with B: certified regret_med calls | no B regret_med
97
+ shipped 3 1.000 0.828 | 1.000 1.01 16.4 | 0.94
98
+ dimless-lz, orig labels 3 1.000 1.000 | 1.000 1.37 16.0 | 1.37
99
+ dimless-lzres, orig labels 3 1.000 1.000 | 1.000 1.36 16.0 | 1.36
100
+ tuned phys, multi-A (published) 3 1.000 0.644 | 1.000 1.56 17.2 | 1.05
101
+ tuned phys, multi-B (published) 3 1.000 0.478 | 1.000 1.52 17.1 | 0.83
102
+ A: phys, multi-A (new metric) 3 1.000 0.650 | 1.000 1.92 17.4 | 1.20
103
+ A: dimless-lz, multi-A 3 1.000 1.000 | 1.000 1.25 16.0 | 1.25
104
+ A: dimless-lzres, multi-A 3 1.000 1.000 | 1.000 1.12 16.0 | 1.12
105
+ A: dimless-lz, multi-B 3 1.000 0.494 | 1.000 1.14 16.0 | 0.77
106
+ nn (orig bank) 1 0.967 0.950 | 1.000 4.00 16.4 | 4.00
107
+ nns (orig) 1 1.000 1.000 | 1.000 1.17 16.0 | 1.17
108
+ dls (orig) 1 1.000 1.000 | 1.000 1.31 16.0 | 1.31
109
+ nn (multi-A) 1 0.950 0.950 | 1.000 2.23 16.4 | 2.23
110
+ nns (multi-A) 1 1.000 1.000 | 1.000 1.29 16.0 | 1.29
111
+ dls (multi-A) 1 1.000 0.967 | 1.000 1.26 16.0 | 1.26
112
+ nn (multi-B) 1 0.950 0.883 | 1.000 4.04 16.5 | 4.04
113
+ nns (multi-B) 1 1.000 0.983 | 1.000 1.21 16.0 | 1.21
114
+ dls (multi-B) 1 1.000 1.000 | 1.000 1.04 16.0 | 1.04
115
+ -- best retrieval arm with B: dls (multi-B); best original-bank arm: nns (orig). paired Wilcoxon, group minus retrieval (neg = model better) [95% CI of paired median], p
116
+ shipped B vs dls (multi-B) : +0.01 [-0.24,+0.37] p=0.236
117
+ shipped B vs nns (orig) : -0.02 [-0.25,+0.24] p=0.527
118
+ dimless-lz, orig labels B vs dls (multi-B) : -0.19 [-0.43,+0.36] p=0.365
119
+ dimless-lz, orig labels B vs nns (orig) : +0.11 [+0.02,+0.37] p=0.0059
120
+ dimless-lzres, orig labels B vs dls (multi-B) : -0.19 [-0.42,+0.16] p=0.522
121
+ dimless-lzres, orig labels B vs nns (orig) : +0.01 [-0.09,+0.09] p=0.393
122
+ tuned phys, multi-A (published) B vs dls (multi-B) : +0.20 [-0.05,+0.55] p=0.0145
123
+ tuned phys, multi-A (published) B vs nns (orig) : +0.34 [-0.02,+0.78] p=0.0192
124
+ tuned phys, multi-B (published) B vs dls (multi-B) : +0.00 [-0.32,+0.36] p=0.139
125
+ tuned phys, multi-B (published) B vs nns (orig) : +0.15 [-0.09,+0.47] p=0.16
126
+ A: phys, multi-A (new metric) B vs dls (multi-B) : +0.21 [-0.06,+0.72] p=0.0047
127
+ A: phys, multi-A (new metric) B vs nns (orig) : +0.49 [+0.01,+0.97] p=0.0084
128
+ A: dimless-lz, multi-A B vs dls (multi-B) : +0.07 [-0.11,+0.17] p=0.245
129
+ A: dimless-lz, multi-A B vs nns (orig) : +0.11 [-0.10,+0.22] p=0.702
130
+ A: dimless-lzres, multi-A B vs dls (multi-B) : -0.01 [-0.18,+0.02] p=0.0773
131
+ A: dimless-lzres, multi-A B vs nns (orig) : +0.03 [-0.09,+0.18] p=0.566
132
+ A: dimless-lz, multi-B B vs dls (multi-B) : +0.05 [-0.01,+0.27] p=0.0283
133
+ A: dimless-lz, multi-B B vs nns (orig) : +0.19 [+0.07,+0.38] p=0.551
134
+ -- factor decomposition (paired, with B): data = tuned-phys-multi-A vs shipped; recipe = A-phys(new metric) vs published tuned; physics = A-lz vs A-phys; trend = A-lzres vs A-lz; multiplicity = multi-B lz vs multi-A lz
135
+ tuned phys, multi-A (published) - shipped : +0.05 [-0.33,+0.38] p=0.611
136
+ A: phys, multi-A (new metric) - tuned phys, multi-A (published) : +0.01 [-0.06,+0.28] p=0.306
137
+ A: dimless-lz, multi-A - A: phys, multi-A (new metric) : -0.13 [-0.96,+0.07] p=0.0167
138
+ A: dimless-lzres, multi-A - A: dimless-lz, multi-A : -0.06 [-0.32,+0.04] p=0.0164
139
+ A: dimless-lz, multi-B - A: dimless-lz, multi-A : +0.04 [-0.02,+0.16] p=0.101
140
+ dimless-lz, orig labels - shipped : +0.18 [+0.04,+0.49] p=0.448
141
+
142
+ === pilotfar K=8, oracle median J 58.949; value = mean over the group's runs of per-plant regret %, then median over plants ===
143
+ group n cov certified(no B) | with B: certified regret_med calls | no B regret_med
144
+ shipped 3 1.000 0.006 | 1.000 1.36 23.9 | 3.83
145
+ dimless-lz, orig labels 3 1.000 1.000 | 1.000 17.39 16.0 | 17.39
146
+ dimless-lzres, orig labels 3 1.000 1.000 | 1.000 6.42 16.0 | 6.42
147
+ tuned phys, multi-A (published) 3 0.994 0.078 | 1.000 1.28 22.3 | 5.56
148
+ tuned phys, multi-B (published) 3 0.967 0.056 | 1.000 1.68 23.2 | 21.58
149
+ A: phys, multi-A (new metric) 3 0.994 0.033 | 1.000 1.35 22.7 | 2.64
150
+ A: dimless-lz, multi-A 3 1.000 1.000 | 1.000 3.28 16.0 | 3.28
151
+ A: dimless-lzres, multi-A 3 1.000 0.983 | 1.000 0.76 16.0 | 0.74
152
+ A: dimless-lz, multi-B 3 1.000 0.372 | 1.000 0.84 16.0 | 0.42
153
+ nn (orig bank) 1 0.717 0.517 | 1.000 71.99 19.9 | 86.49
154
+ nns (orig) 1 1.000 1.000 | 1.000 11.67 16.0 | 11.67
155
+ dls (orig) 1 1.000 1.000 | 1.000 8.37 16.0 | 8.37
156
+ nn (multi-A) 1 0.233 0.150 | 1.000 77.29 22.8 | 1000.00
157
+ nns (multi-A) 1 1.000 1.000 | 1.000 8.65 16.0 | 8.65
158
+ dls (multi-A) 1 1.000 1.000 | 1.000 1.84 16.0 | 1.84
159
+ nn (multi-B) 1 0.233 0.150 | 1.000 11.75 22.8 | 1000.00
160
+ nns (multi-B) 1 1.000 0.067 | 1.000 1.71 16.0 | 0.36
161
+ dls (multi-B) 1 1.000 1.000 | 1.000 0.55 16.0 | 0.55
162
+ -- best retrieval arm with B: dls (multi-B); best original-bank arm: dls (orig). paired Wilcoxon, group minus retrieval (neg = model better) [95% CI of paired median], p
163
+ shipped B vs dls (multi-B) : +0.81 [+0.52,+0.99] p=4.43e-09
164
+ shipped B vs dls (orig) : -7.08 [-7.37,-6.67] p=1.63e-11
165
+ dimless-lz, orig labels B vs dls (multi-B) : +16.88 [+15.86,+18.34] p=1.63e-11
166
+ dimless-lz, orig labels B vs dls (orig) : +8.19 [+7.11,+8.95] p=1.63e-11
167
+ dimless-lzres, orig labels B vs dls (multi-B) : +5.87 [+5.56,+6.08] p=1.63e-11
168
+ dimless-lzres, orig labels B vs dls (orig) : -2.04 [-2.60,-1.47] p=1.71e-11
169
+ tuned phys, multi-A (published) B vs dls (multi-B) : +0.76 [+0.47,+1.36] p=2.85e-08
170
+ tuned phys, multi-A (published) B vs dls (orig) : -7.33 [-7.76,-6.14] p=2.62e-08
171
+ tuned phys, multi-B (published) B vs dls (multi-B) : +1.13 [+0.51,+1.77] p=2.71e-09
172
+ tuned phys, multi-B (published) B vs dls (orig) : -6.67 [-7.28,-6.03] p=9.07e-08
173
+ A: phys, multi-A (new metric) B vs dls (multi-B) : +0.78 [+0.32,+1.47] p=6.02e-08
174
+ A: phys, multi-A (new metric) B vs dls (orig) : -6.96 [-7.90,-6.41] p=2.59e-09
175
+ A: dimless-lz, multi-A B vs dls (multi-B) : +2.66 [+2.05,+3.10] p=1.63e-11
176
+ A: dimless-lz, multi-A B vs dls (orig) : -5.55 [-6.18,-5.21] p=1.63e-11
177
+ A: dimless-lzres, multi-A B vs dls (multi-B) : +0.11 [+0.02,+0.26] p=0.000636
178
+ A: dimless-lzres, multi-A B vs dls (orig) : -7.47 [-8.10,-7.04] p=1.63e-11
179
+ A: dimless-lz, multi-B B vs dls (multi-B) : +0.33 [+0.18,+0.52] p=1.06e-06
180
+ A: dimless-lz, multi-B B vs dls (orig) : -7.44 [-7.74,-6.97] p=1.71e-11
181
+ -- factor decomposition (paired, with B): data = tuned-phys-multi-A vs shipped; recipe = A-phys(new metric) vs published tuned; physics = A-lz vs A-phys; trend = A-lzres vs A-lz; multiplicity = multi-B lz vs multi-A lz
182
+ tuned phys, multi-A (published) - shipped : -0.03 [-0.34,+0.29] p=0.466
183
+ A: phys, multi-A (new metric) - tuned phys, multi-A (published) : +0.18 [-0.33,+0.55] p=0.918
184
+ A: dimless-lz, multi-A - A: phys, multi-A (new metric) : +1.88 [+0.98,+2.26] p=0.00204
185
+ A: dimless-lzres, multi-A - A: dimless-lz, multi-A : -2.14 [-2.69,-1.82] p=2.1e-11
186
+ A: dimless-lz, multi-B - A: dimless-lz, multi-A : -2.06 [-2.70,-1.13] p=1.07e-08
187
+ dimless-lz, orig labels - shipped : +15.83 [+15.19,+17.57] p=1.63e-11
188
+