Add files using upload-large-folder tool
Browse filesThis view is limited to 50 files because it contains too many changes. See raw diff
- results/baselines/ablation_k.txt +10 -0
- results/baselines/ablation_k_run.log +17 -0
- results/baselines/ablation_steps_siso.txt +9 -0
- results/baselines/bank_density_siso.txt +68 -0
- results/baselines/distinct_retrieval_prereg.txt +27 -0
- results/baselines/fixed_controller_pendulum.txt +19 -0
- results/baselines/pendulum_bank_density.txt +36 -0
- results/baselines/pendulum_bank_density_diag_a.txt +15 -0
- results/baselines/pendulum_bank_density_diag_b_seed123.txt +8 -0
- results/baselines/retrieval_mimo_seed7777.txt +41 -0
- results/baselines/retrieval_objective_conditioning.txt +129 -0
- results/baselines/retrieval_pendulum.txt +28 -0
- results/baselines/retrieval_pendulum_seed321.txt +29 -0
- results/baselines/retrieval_siso.txt +36 -0
- results/baselines/siso_run.log +19 -0
- results/furuta_v3/deploy_sidebyside.txt +19 -0
- results/furuta_v3/ksweep_sidebyside.txt +9 -0
- results/furuta_v3/traj_gen_eval_v3_raw_clip1.txt +94 -0
- results/furuta_v3/traj_gen_eval_v3e2400s1_raw_clip1.txt +94 -0
- results/furuta_v3/traj_gen_v2ctl2400_raw_clip5.txt +30 -0
- results/furuta_v3/traj_gen_v3e2400_raw_clip1.txt +30 -0
- results/furuta_v3/traj_gen_warmstart_v2clip5_n200.txt +17 -0
- results/furuta_v3/traj_gen_warmstart_v2clip5_s1.txt +17 -0
- results/furuta_v3/traj_gen_warmstart_v2clip5_s2.txt +17 -0
- results/furuta_v3/traj_gen_warmstart_v3_n200.txt +17 -0
- results/furuta_v3/traj_gen_warmstart_v3_s2.txt +17 -0
- results/generalize/fig_generalize.png +0 -0
- results/generalize/summary.txt +5 -0
- results/metric_steering/mimo_summary.txt +12 -0
- results/metric_steering/refine_summary.txt +12 -0
- results/metric_steering/summary.txt +12 -0
- results/mimo/k_sweep_mimo.txt +11 -0
- results/mimo/target_ablation.md +6 -0
- results/pareto/fig_pareto.pdf +0 -0
- results/pareto/fig_pareto.png +0 -0
- results/pareto/table_pareto.md +9 -0
- results/pareto_specs/run_s0.log +28 -0
- results/pareto_specs/run_s1.log +26 -0
- results/pareto_specs/run_s2.log +29 -0
- results/pareto_specs/run_s3.log +29 -0
- results/pareto_specs/run_s4.log +28 -0
- results/pareto_specs/significance.txt +15 -0
- results/pareto_specs/summary.txt +15 -0
- results/pareto_specs/summary_s0.txt +15 -0
- results/pareto_specs/summary_s1.txt +15 -0
- results/pareto_specs/summary_s2.txt +15 -0
- results/pareto_specs/summary_s3.txt +15 -0
- results/pareto_specs/summary_s4.txt +15 -0
- results/physics_gen/pendulum_physics_gen.txt +720 -0
- results/physics_gen/pendulum_physics_gen_v2.txt +188 -0
results/baselines/ablation_k.txt
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
SISO K-sample ablation — pooled 5 seeds × 60 plants
|
| 2 |
+
|
| 3 |
+
K plant-stabilisable rate (≥1 of K) 95% CI
|
| 4 |
+
1 0.633 [0.580, 0.687]
|
| 5 |
+
2 0.713 [0.660, 0.763]
|
| 6 |
+
4 0.747 [0.697, 0.793]
|
| 7 |
+
8 0.770 [0.723, 0.817]
|
| 8 |
+
16 0.787 [0.740, 0.833]
|
| 9 |
+
|
| 10 |
+
lift K=1 -> K=16: +0.153 (0.63 -> 0.79) — the generative distribution's value.
|
results/baselines/ablation_k_run.log
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[seed 0] done
|
| 2 |
+
[seed 1] done
|
| 3 |
+
[seed 2] done
|
| 4 |
+
[seed 3] done
|
| 5 |
+
[seed 4] done
|
| 6 |
+
|
| 7 |
+
=== K ABLATION ===
|
| 8 |
+
SISO K-sample ablation — pooled 5 seeds × 60 plants
|
| 9 |
+
|
| 10 |
+
K plant-stabilisable rate (≥1 of K) 95% CI
|
| 11 |
+
1 0.633 [0.580, 0.687]
|
| 12 |
+
2 0.713 [0.660, 0.763]
|
| 13 |
+
4 0.747 [0.697, 0.793]
|
| 14 |
+
8 0.770 [0.723, 0.817]
|
| 15 |
+
16 0.787 [0.740, 0.833]
|
| 16 |
+
|
| 17 |
+
lift K=1 -> K=16: +0.153 (0.63 -> 0.79) — the generative distribution's value.
|
results/baselines/ablation_steps_siso.txt
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== SISO DDIM steps sweep (plants seed 0, N=60, K=8, sampler seeds (0, 1, 2), eta=0, device=cuda) ===
|
| 2 |
+
model fixed (full-data SISO); verifier/x0 = retrieval_baselines.py siso (x0 seed 777); oracle = 60-step Adam refinement of the DDPM@500 seed-0 K=8 winner (as bank_density; negative regret possible).
|
| 3 |
+
regret 'common' = median over plants covered by EVERY arm's seed-0 draw (n=46); 'own' = over each draw's own covered plants. ms/ctrl = wall time per sampled controller, batch 480.
|
| 4 |
+
arm cov per-samp reg common% reg own% ms/ctrl McNemar vs ddpm500 (n10,n01,p)
|
| 5 |
+
ddim5 0.817+-0.017 0.682+-0.009 1.25+-9.75 -0.46+-5.83 0.039+-0.029 (3, 0) p=0.25
|
| 6 |
+
ddim20 0.800+-0.017 0.658+-0.006 5.76+-13.42 5.71+-13.39 0.086+-0.000 (2, 1) p=1
|
| 7 |
+
ddim100 0.794+-0.025 0.634+-0.014 14.75+-9.27 13.04+-7.10 0.461+-0.005 (0, 1) p=1
|
| 8 |
+
ddim500 0.789+-0.010 0.622+-0.019 19.11+-10.09 16.61+-7.12 2.292+-0.010 (1, 1) p=1
|
| 9 |
+
ddpm500 0.794+-0.010 0.632+-0.013 25.27+-11.55 23.72+-10.49 1.997+-0.005 (0, 0) p=1
|
results/baselines/bank_density_siso.txt
ADDED
|
@@ -0,0 +1,68 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== BANK-DENSITY SWEEP, SISO (held-out draws: seeds [0, 4321], N=[60, 60], pooled N=120) ===
|
| 2 |
+
MODEL SIDE IS FIXED: full-data checkpoint, NOT retrained per bank fraction (the pendulum sweep retrained; this one does not).
|
| 3 |
+
bank = seeded random subset of training PLANTS (all stored controllers of the chosen plants); fractions (0.05, 0.1, 0.25, 0.5, 1.0); subset seeds (0, 1, 2) (f=1.0: single bank);
|
| 4 |
+
model draws: sampler seeds (0, 1, 2); McNemar/regret reference = model seed 0. Verifier/x0/N as retrieval_baselines.py siso; K=8 and K=1 (K=1 = first candidate).
|
| 5 |
+
oracle: shared per plant = 60-step Adam refinement (generalize.py protocol) of the model seed-0 K=8 winner (the SAME oracle yesterday's SISO model arm used), floored by that winner's J; arms that beat it show NEGATIVE regret; regret undefined where the model seed-0 draw is uncovered (never inside the common set)
|
| 6 |
+
regret % = (J_best - J_oracle)/J_oracle on the COMMON success set (plants where BOTH the arm and the model seed-0 draw are covered at that K); label 'cond.'.
|
| 7 |
+
coverage 95% CI: bootstrap (5000) over pooled plants of subset-seed 0. McNemar: exact, paired on pooled plants vs model seed 0, reported for each subset seed (min..max p).
|
| 8 |
+
|
| 9 |
+
----- K=8 ----- MODEL coverage 0.719 +- 0.004 (3 draws), seed-0 CI [0.633,0.792], per-sample 0.566, cond. regret med 15.86%, gen 2.34 ms/ctrl
|
| 10 |
+
per-draw model coverage (seed 0/1/2): seed 0: 0.78/0.80/0.80; seed 4321: 0.65/0.63/0.65
|
| 11 |
+
arm=retrieval
|
| 12 |
+
f cov mean+-sd CI(ss0) per-samp dCov(ret-model) McNemar p(ss0..) reg_med% cond model reg% same set n_common NN dist
|
| 13 |
+
0.05 0.531+-0.037 [0.408,0.592] 0.531 -0.189 3e-08..3.1e-05 86.89 16.55 63 0.0083
|
| 14 |
+
0.10 0.461+-0.021 [0.375,0.558] 0.461 -0.258 1.2e-10..7.5e-09 85.70 16.68 55 0.0072
|
| 15 |
+
0.25 0.478+-0.010 [0.383,0.558] 0.478 -0.242 1.9e-09..1.5e-08 77.99 14.06 57 0.0051
|
| 16 |
+
0.50 0.456+-0.022 [0.375,0.550] 0.456 -0.264 5.8e-11..3.7e-09 80.47 15.09 54 0.0041
|
| 17 |
+
1.00 0.450+-0.000 [0.358,0.542] 0.450 -0.269 4.7e-10..4.7e-10 73.24 14.42 54 0.0035
|
| 18 |
+
arm=blend
|
| 19 |
+
f cov mean+-sd CI(ss0) per-samp dCov(ret-model) McNemar p(ss0..) reg_med% cond model reg% same set n_common NN dist
|
| 20 |
+
0.05 0.531+-0.037 [0.408,0.592] 0.531 -0.189 3e-08..3.1e-05 86.89 16.55 63 0.0083
|
| 21 |
+
0.10 0.461+-0.021 [0.375,0.558] 0.461 -0.258 1.2e-10..7.5e-09 85.70 16.68 55 0.0072
|
| 22 |
+
0.25 0.478+-0.010 [0.383,0.558] 0.478 -0.242 1.9e-09..1.5e-08 77.99 14.06 57 0.0051
|
| 23 |
+
0.50 0.456+-0.022 [0.375,0.550] 0.456 -0.264 5.8e-11..3.7e-09 80.47 15.09 54 0.0041
|
| 24 |
+
1.00 0.450+-0.000 [0.358,0.542] 0.450 -0.269 4.7e-10..4.7e-10 73.24 14.42 54 0.0035
|
| 25 |
+
arm=distinct
|
| 26 |
+
f cov mean+-sd CI(ss0) per-samp dCov(ret-model) McNemar p(ss0..) reg_med% cond model reg% same set n_common NN dist
|
| 27 |
+
0.05 0.689+-0.004 [0.608,0.775] 0.484 -0.031 0.12..0.38 -4.80 15.32 82 0.0083
|
| 28 |
+
0.10 0.689+-0.004 [0.608,0.767] 0.489 -0.031 0.12..0.38 -5.45 15.28 82 0.0072
|
| 29 |
+
0.25 0.686+-0.004 [0.608,0.775] 0.474 -0.033 0.12..0.25 -10.99 15.32 82 0.0051
|
| 30 |
+
0.50 0.686+-0.004 [0.608,0.775] 0.489 -0.033 0.12..0.25 -7.75 15.32 82 0.0041
|
| 31 |
+
1.00 0.683+-0.000 [0.600,0.767] 0.483 -0.036 0.12..0.12 4.31 15.28 82 0.0035
|
| 32 |
+
|
| 33 |
+
----- K=1 ----- MODEL coverage 0.600 +- 0.020 (3 draws), seed-0 CI [0.508,0.683], per-sample 0.600, cond. regret med 735.20%, gen 2.34 ms/ctrl
|
| 34 |
+
per-draw model coverage (seed 0/1/2): seed 0: 0.65/0.67/0.70; seed 4321: 0.55/0.48/0.55
|
| 35 |
+
arm=retrieval (blend K=1 == retrieval K=1 by construction)
|
| 36 |
+
f cov mean+-sd CI(ss0) per-samp dCov(ret-model) McNemar p(ss0..) reg_med% cond model reg% same set n_common NN dist
|
| 37 |
+
0.05 0.531+-0.037 [0.408,0.592] 0.531 -0.069 0.023..0.84 119.81 955.04 55 0.0083
|
| 38 |
+
0.10 0.461+-0.021 [0.375,0.558] 0.461 -0.139 0.00032..0.016 119.21 840.13 48 0.0072
|
| 39 |
+
0.25 0.478+-0.010 [0.383,0.558] 0.478 -0.122 0.0059..0.024 88.08 1118.00 50 0.0051
|
| 40 |
+
0.50 0.456+-0.022 [0.375,0.550] 0.456 -0.144 0.00019..0.0041 82.60 758.08 49 0.0041
|
| 41 |
+
1.00 0.450+-0.000 [0.358,0.542] 0.450 -0.150 0.0014..0.0014 74.12 1131.46 48 0.0035
|
| 42 |
+
|
| 43 |
+
----- retrieval coverage K=8 vs K=1 per f (mean over subset seeds; pooled draws) -----
|
| 44 |
+
f K=1 K=8 distinct K=8
|
| 45 |
+
0.05 0.531 0.531 0.689
|
| 46 |
+
0.10 0.461 0.461 0.689
|
| 47 |
+
0.25 0.478 0.478 0.686
|
| 48 |
+
0.50 0.456 0.456 0.686
|
| 49 |
+
1.00 0.450 0.450 0.683
|
| 50 |
+
|
| 51 |
+
----- smallest f at which retrieval MATCHES the model -----
|
| 52 |
+
rule (fixed before running): mean retrieval coverage >= model coverage - 0.05 AND pooled McNemar p >= 0.05 for >= 2 of 3 subset seeds (f=1.0: its single bank); 'never' if no f qualifies.
|
| 53 |
+
siso arm=retrieval K=8: never (none of the swept f)
|
| 54 |
+
siso arm=blend K=8: never (none of the swept f)
|
| 55 |
+
siso arm=distinct K=8: f=0.05
|
| 56 |
+
siso arm=retrieval K=1: never (none of the swept f)
|
| 57 |
+
|
| 58 |
+
----- NN distance vs f (normalized conditioning space, mean over queries; mean over subset seeds) -----
|
| 59 |
+
f=0.05: draw0: 0.0098, draw4321: 0.0069
|
| 60 |
+
f=0.1: draw0: 0.0083, draw4321: 0.0061
|
| 61 |
+
f=0.25: draw0: 0.0060, draw4321: 0.0042
|
| 62 |
+
f=0.5: draw0: 0.0049, draw4321: 0.0033
|
| 63 |
+
f=1.0: draw0: 0.0042, draw4321: 0.0028
|
| 64 |
+
|
| 65 |
+
ADDENDUM: per-held-out-draw K=8 coverage (mean over subset seeds; model = seed-0 sampler / 3-seed mean)
|
| 66 |
+
draw0: model 0.794; retrieval f=0.05: 0.600 f=0.1: 0.533 f=0.25: 0.556 f=0.5: 0.522 f=1.0: 0.517 | distinct f=0.05: 0.772 f=0.1: 0.767 f=0.25: 0.767 f=0.5: 0.767 f=1.0: 0.767
|
| 67 |
+
draw1: model 0.644; retrieval f=0.05: 0.461 f=0.1: 0.389 f=0.25: 0.400 f=0.5: 0.389 f=1.0: 0.383 | distinct f=0.05: 0.606 f=0.1: 0.611 f=0.25: 0.606 f=0.5: 0.606 f=1.0: 0.600
|
| 68 |
+
retrieval f=1.0: plants where all 8 candidates share one verdict: 1.000 (pooled N=120)
|
results/baselines/distinct_retrieval_prereg.txt
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== PRE-REGISTERED distinct-plant retrieval vs diffusion, SISO, N=300 (plant seeds (100, 200, 300, 400, 500) x 60), full bank f=1.0, coverage only ===
|
| 2 |
+
model: full-data checkpoint, K<=16 samples/plant, sampler seeds 0/1/2 (seed 0 = paired reference; K-prefix of the 16 samples). verifier/x0 as bank_density sweep.
|
| 3 |
+
|
| 4 |
+
PRIMARY K=8: model(seed0) 0.773 distinct-first 0.750 diff +0.023 95% CI [+0.003,+0.047] McNemar model-only 9, retrieval-only 2, p=0.0654 => VERDICT: TIE
|
| 5 |
+
sampler seed 1: model 0.780 McNemar 12-3 p=0.0352
|
| 6 |
+
sampler seed 2: model 0.787 McNemar 13-2 p=0.00739
|
| 7 |
+
model 3-seed mean K=8: 0.780 +- 0.005
|
| 8 |
+
|
| 9 |
+
per plant-seed K=8 coverage (model seed0 / distinct-first):
|
| 10 |
+
100: 0.800 / 0.783
|
| 11 |
+
200: 0.800 / 0.783
|
| 12 |
+
300: 0.833 / 0.800
|
| 13 |
+
400: 0.733 / 0.717
|
| 14 |
+
500: 0.700 / 0.667
|
| 15 |
+
|
| 16 |
+
SECONDARY coverage vs K (model seed0 [3-seed mean]; distinct-first; distinct-random mean of 3 seeds; McNemar model vs distinct-first)
|
| 17 |
+
K model s0 model mean dist-first dist-random McNemar (m-only,r-only,p)
|
| 18 |
+
1 0.617 0.622 0.493 0.493+-0.000 (60,23,5.97e-05)
|
| 19 |
+
2 0.707 0.706 0.660 0.660+-0.000 (24,10,0.0243)
|
| 20 |
+
4 0.747 0.756 0.743 0.743+-0.000 (8,7,1)
|
| 21 |
+
8 0.773 0.780 0.750 0.750+-0.000 (9,2,0.0654)
|
| 22 |
+
16 0.787 0.793 0.753 0.753+-0.000 (12,2,0.0129)
|
| 23 |
+
random-stored K=8 (seed 0 draw) vs model: model-only 9, retrieval-only 2, p=0.0654
|
| 24 |
+
|
| 25 |
+
MIMO (distinct == plain retrieval, one controller per plant), f=1.0, K=8, from the sweep npz:
|
| 26 |
+
draw 9999: model 1.000 retrieval 0.992 McNemar 1-0 p=1
|
| 27 |
+
draw 4242: model 0.992 retrieval 0.992 McNemar 0-0 p=1
|
results/baselines/fixed_controller_pendulum.txt
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== PENDULUM fixed dimensionless controller (N=40, seed=123 held-out) ===
|
| 2 |
+
|
| 3 |
+
Justification: theta'' = (g/l) sin(theta) - (b/(m l^2)) omega + u/(m l^2), controller u = -phi(y), y = c1 theta + c2 omega (c FIXED at (1.0, 0.35) for the whole family, as in the amortized model). The natural dimensionless rescaling of phi is by the holding torque m*g*l (the torque needed to hold the pendulum horizontal); a natural TIME scale is sqrt(l/g) (the small-oscillation period / sqrt(g/l) natural frequency), though this codebase does not rescale omega's contribution to y by it (c2 is fixed, not per-plant) -- so the fixed controller below rescales phi by m*g*l only, exactly mirroring how eval_pendulum.strong_oracle's own smart init (2.5*m*l*g*xp) and random-restart shapes ((U[0.5,2.5]*m*l*g)*xp*(|xp|/5+0.3)) are built.
|
| 4 |
+
|
| 5 |
+
Config space = SAME as the oracle's restarts: shape in {linear xp, nonlinear xp*(|xp|/5+0.3)}, K_hat in linspace(0.5,2.5,9). Tuned on a FRESH N=200 set (seed=555, disjoint from the held-out test set) by (certified_rate, success_rate); NOT re-tuned per plant (one fixed controller for the whole family). Cost = the paper's ABSOLUTE-weight J = integral (theta^2 + 0.1 omega^2) + u^2 dt (simulate_pendulum_batch default weights) -- not a scale-invariant metric.
|
| 6 |
+
|
| 7 |
+
Chosen fixed controller: shape=nonlinear K_hat=2.500 (phi(y) = 2.500 * m*g*l * shape(y))
|
| 8 |
+
|
| 9 |
+
success (empirical stability, N=40): 1.000
|
| 10 |
+
certified (regional LMI, N=40): 1.000
|
| 11 |
+
cost regret vs cached per-plant oracle: median +56.69% p90 +88.86% max +124.15% (n=40)
|
| 12 |
+
synthesis time: 81.2865 ms/plant (no per-plant solve -- table lookup + rescale)
|
| 13 |
+
|
| 14 |
+
top 5 tuning configs (shape, K_hat, certified_rate, success_rate) on the tuning set:
|
| 15 |
+
('nonlinear', 2.5, 1.0, 1.0)
|
| 16 |
+
('nonlinear', 2.25, 1.0, 1.0)
|
| 17 |
+
('linear', 2.5, 1.0, 1.0)
|
| 18 |
+
('linear', 2.25, 1.0, 1.0)
|
| 19 |
+
('linear', 2.0, 1.0, 1.0)
|
results/baselines/pendulum_bank_density.txt
ADDED
|
@@ -0,0 +1,36 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== PENDULUM bank-density sweep ===
|
| 2 |
+
model = diffusion model TRAINED on the f-fraction subset (3 training seeds 42/43/44,
|
| 3 |
+
mean +/- sd across training seeds; ONE diffusion sampling seed=0 per model, K=8)
|
| 4 |
+
retrieval/interpolation = SAME f-fraction bank (built from the identical seeded subset
|
| 5 |
+
train_pendulum_ablation.py used), 3 seeds -> mean +/- sd, K=8
|
| 6 |
+
held-out sets: seed123 (cached oracle), seed321 (fresh oracle, run_pendulum_second_draw)
|
| 7 |
+
|
| 8 |
+
--- held-out set: seed123 ---
|
| 9 |
+
f arm coverage regret_med % regret_med 95%CI(seed0)
|
| 10 |
+
0.25 model 1.000+-0.000 0.24+-0.22 [+0.41,+0.69]
|
| 11 |
+
0.25 retrieval 1.000+-0.000 0.17+-0.01 [+0.01,+0.44]
|
| 12 |
+
0.25 interpolation 1.000+-0.000 0.22+-0.05 [+0.02,+0.44]
|
| 13 |
+
0.50 model 1.000+-0.000 0.98+-0.16 [+1.08,+1.33]
|
| 14 |
+
0.50 retrieval 1.000+-0.000 0.30+-0.08 [+0.05,+0.41]
|
| 15 |
+
0.50 interpolation 1.000+-0.000 0.30+-0.08 [+0.05,+0.41]
|
| 16 |
+
1.00 model 1.000+-0.000 0.89+-0.21 [+0.93,+1.38]
|
| 17 |
+
1.00 retrieval 1.000+-0.000 0.40+-0.00 [+0.22,+0.53]
|
| 18 |
+
1.00 interpolation 1.000+-0.000 0.40+-0.00 [+0.22,+0.53]
|
| 19 |
+
|
| 20 |
+
--- held-out set: seed321 ---
|
| 21 |
+
f arm coverage regret_med % regret_med 95%CI(seed0)
|
| 22 |
+
0.25 model 1.000+-0.000 0.26+-0.14 [+0.23,+0.72]
|
| 23 |
+
0.25 retrieval 1.000+-0.000 0.24+-0.01 [+0.06,+0.33]
|
| 24 |
+
0.25 interpolation 1.000+-0.000 0.25+-0.01 [+0.11,+0.37]
|
| 25 |
+
0.50 model 1.000+-0.000 0.91+-0.16 [+0.85,+1.20]
|
| 26 |
+
0.50 retrieval 1.000+-0.000 0.23+-0.06 [+0.04,+0.29]
|
| 27 |
+
0.50 interpolation 1.000+-0.000 0.26+-0.05 [+0.04,+0.29]
|
| 28 |
+
1.00 model 1.000+-0.000 0.87+-0.17 [+0.82,+1.24]
|
| 29 |
+
1.00 retrieval 1.000+-0.000 0.29+-0.00 [+0.16,+0.36]
|
| 30 |
+
1.00 interpolation 1.000+-0.000 0.30+-0.00 [+0.16,+0.42]
|
| 31 |
+
|
| 32 |
+
Crossover -- does the model's median regret ever drop BELOW retrieval's at the same f?
|
| 33 |
+
seed123: f=0.25: model +0.24% >= retrieval +0.17% (retrieval wins/ties); f=0.50: model +0.98% >= retrieval +0.30% (retrieval wins/ties); f=1.00: model +0.89% >= retrieval +0.40% (retrieval wins/ties)
|
| 34 |
+
seed321: f=0.25: model +0.26% >= retrieval +0.24% (retrieval wins/ties); f=0.50: model +0.91% >= retrieval +0.23% (retrieval wins/ties); f=1.00: model +0.87% >= retrieval +0.29% (retrieval wins/ties)
|
| 35 |
+
-> model overtakes retrieval's median regret at ANY swept f, on ANY held-out set: False
|
| 36 |
+
FINDING: the model never overtakes bank retrieval's cost on this sweep -- more data narrows the gap (higher f typically lowers both arms' regret) but does not flip the ordering within {0.25, 0.5, 1.0}.
|
results/baselines/pendulum_bank_density_diag_a.txt
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== PENDULUM bank-density diagnostic (a): retrieval K=1 (single nearest neighbour, no best-of-K selection) vs f ===
|
| 2 |
+
If K=1 regret falls with f while best-of-8 (pendulum_bank_density.txt) does not, best-of-8's edge at low f is candidate DIVERSITY (a sparse bank's 8 nearest are spread out), not raw proximity -- a dense bank's 8 nearest become near-duplicates and best-of-8 stops gaining.
|
| 3 |
+
|
| 4 |
+
--- held-out set: seed123 ---
|
| 5 |
+
f K1 regret_med% K1 cov (3-seed mean +/- sd)
|
| 6 |
+
0.25 1.71+-0.16 1.000
|
| 7 |
+
0.50 1.63+-0.20 1.000
|
| 8 |
+
1.00 1.84+-0.00 1.000
|
| 9 |
+
|
| 10 |
+
--- held-out set: seed321 ---
|
| 11 |
+
f K1 regret_med% K1 cov (3-seed mean +/- sd)
|
| 12 |
+
0.25 1.19+-0.22 1.000
|
| 13 |
+
0.50 1.42+-0.16 1.000
|
| 14 |
+
1.00 1.36+-0.00 1.000
|
| 15 |
+
|
results/baselines/pendulum_bank_density_diag_b_seed123.txt
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== PENDULUM bank-density diagnostic (b): native quality of stored targets (held-out seed=123, N=40) ===
|
| 2 |
+
For each held-out plant: find its nearest FULL-BANK (f=1.0, 2600 pts) training plant; evaluate that plant's OWN stored controller (spline_raw, as-stored, no re-decoding) ON THAT TRAINING PLANT'S OWN DYNAMICS (not the held-out query); regret vs a FRESH eval_pendulum.strong_oracle solve of that SAME training plant.
|
| 3 |
+
39 unique nearest-neighbour training plants among the 40 held-out queries (some held-out plants share a nearest neighbour).
|
| 4 |
+
|
| 5 |
+
native target regret: median +1.40% p90 +2.67% max +3.58% (n=40/40)
|
| 6 |
+
nearest-neighbour distance (normalized space): median 0.1691 max 0.2201
|
| 7 |
+
|
| 8 |
+
Reading: if this native regret (~how good the DATASET's own targets are, solved once, reused everywhere) sits near retrieval's regret on the held-out query (bank_density.txt), retrieval is essentially reading back near-oracle targets with a small transfer penalty for the query-vs-neighbour distance; if the model's own regret (~0.9% at f=1.0) exceeds this native floor, the model is worse than its own training set accessed directly -- the same finding as Iteration 3 on the DP-target pendulum.
|
results/baselines/retrieval_mimo_seed7777.txt
ADDED
|
@@ -0,0 +1,41 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== MIMO retrieval/interpolation baselines (N=120, seed=7777) ===
|
| 2 |
+
verifier: simulate_closed_loop_mimo_batched rollout (n_ic=4, dt=0.03, duration=12.0, energy_thresh=0.3)
|
| 3 |
+
oracle: eval_mimo_regret.py's oracle protocol (fixed generic init 0.3*cap, 150-step batched Adam lr=0.08 through the differentiable MIMO sim, sector-projected) -- arm-independent, computed ONCE, shared by every arm (1187.0 ms/plant)
|
| 4 |
+
K_std=8, diffusion seeds=(0, 1, 2)
|
| 5 |
+
|
| 6 |
+
--- K=8 ---
|
| 7 |
+
arm coverage persamp reg_med% reg_p90% reg_max% ms/plant McNemar(n10,n01,p) Wilcoxon(n,p)
|
| 8 |
+
diffusion 0.99 0.94 2.31 9.23 30.13 153.77 (reference) (reference)
|
| 9 |
+
(sd across 3 seeds: coverage 0.004, per-samp 0.003)
|
| 10 |
+
retrieval 0.99 0.93 2.32 8.05 49.97 0.730 (1,0,1) (118,0.91)
|
| 11 |
+
(Wilcoxon n=118 = the COMMON success set: plants where BOTH this arm and diffusion have a finite/stable cost, not either arm's own coverage)
|
| 12 |
+
interpolation 0.99 0.93 2.95 15.27 74.15 0.730 (1,0,1) (118,0.049)
|
| 13 |
+
(Wilcoxon n=118 = the COMMON success set: plants where BOTH this arm and diffusion have a finite/stable cost, not either arm's own coverage)
|
| 14 |
+
|
| 15 |
+
--- K=1 ---
|
| 16 |
+
arm coverage persamp reg_med% reg_p90% reg_max% ms/plant McNemar(n10,n01,p) Wilcoxon(n,p)
|
| 17 |
+
diffusion 0.93 0.93 10.92 88.52 1045.64 153.77 (reference) (reference)
|
| 18 |
+
(sd across 3 seeds: coverage 0.014, per-samp 0.014)
|
| 19 |
+
retrieval 0.92 0.92 15.23 262.70 830.97 0.370 (5,5,1) (105,0.11)
|
| 20 |
+
(Wilcoxon n=105 = the COMMON success set: plants where BOTH this arm and diffusion have a finite/stable cost, not either arm's own coverage)
|
| 21 |
+
interpolation 0.92 0.92 15.23 262.70 830.97 0.370 (5,5,1) (105,0.11)
|
| 22 |
+
(Wilcoxon n=105 = the COMMON success set: plants where BOTH this arm and diffusion have a finite/stable cost, not either arm's own coverage)
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
--- addendum (post-hoc from npz, not re-simulated): CIs and pooled 2-draw (N=240) tests, diffusion seed[0] as reference ---
|
| 26 |
+
K=8 seed9999 diffusion: coverage 1.000 regret_med +2.32 [+1.63,+2.99]
|
| 27 |
+
K=8 seed9999 retrieval: coverage 0.992 regret_med +2.62 [+1.45,+3.45]
|
| 28 |
+
K=8 seed7777 diffusion: coverage 0.983 regret_med +2.30 [+1.90,+3.01]
|
| 29 |
+
K=8 seed7777 retrieval: coverage 0.992 regret_med +2.32 [+1.67,+3.06]
|
| 30 |
+
K=8 pooled McNemar coverage (retrieval-only, diffusion-only)=(1,1) p=1
|
| 31 |
+
K=8 pooled Wilcoxon cost diffusion vs retrieval on common set n=237: median J ratio 1.000 p=0.863
|
| 32 |
+
K=8 seed9999 diffusion coverage per draw: [1.0, 0.992, 1.0] retrieval 0.992
|
| 33 |
+
K=8 seed7777 diffusion coverage per draw: [0.983, 0.992, 0.983] retrieval 0.992
|
| 34 |
+
K=1 seed9999 diffusion: coverage 0.942 regret_med +8.87 [+6.14,+12.50]
|
| 35 |
+
K=1 seed9999 retrieval: coverage 0.858 regret_med +13.66 [+10.22,+19.86]
|
| 36 |
+
K=1 seed7777 diffusion: coverage 0.917 regret_med +9.85 [+7.63,+13.51]
|
| 37 |
+
K=1 seed7777 retrieval: coverage 0.917 regret_med +15.23 [+10.31,+20.11]
|
| 38 |
+
K=1 pooled McNemar coverage (retrieval-only, diffusion-only)=(5,15) p=0.0414
|
| 39 |
+
K=1 pooled Wilcoxon cost diffusion vs retrieval on common set n=208: median J ratio 0.982 p=0.0345
|
| 40 |
+
K=1 seed9999 diffusion coverage per draw: [0.942, 0.95, 0.942] retrieval 0.858
|
| 41 |
+
K=1 seed7777 diffusion coverage per draw: [0.917, 0.933, 0.95] retrieval 0.917
|
results/baselines/retrieval_objective_conditioning.txt
ADDED
|
@@ -0,0 +1,129 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
RETRIEVAL vs OBJECTIVE-CONDITIONED GENERATION (pendulum)
|
| 2 |
+
N=40 plants/draw, rho=(0.7, 0.8, 0.9), K in (8,1), steer S=50
|
| 3 |
+
plant seeds: main=321 (exp_cbar_eval's), second=777; model torch seeds = seed*1000+rho_idx, seeds (0, 1, 2)
|
| 4 |
+
retr_b pool M=32. spec tolerance ise<=1.05*rho*ISE_ref (exp_cbar_eval satisfied_mask).
|
| 5 |
+
|
| 6 |
+
== Task 1: stream replica of exp_cbar_eval (seed 321) vs STORED results/pendulum/cbar_eval.json ==
|
| 7 |
+
rho=0.7: bestof8 replica met 1.000 cert 1.000 reg 0.84 | stored met 1.000 cert 1.000 reg 0.83 || single replica met 0.925 cert 0.925 reg 8.83 | stored met 0.950 cert 0.850 reg 8.25
|
| 8 |
+
rho=0.8: bestof8 replica met 1.000 cert 1.000 reg 0.61 | stored met 1.000 cert 1.000 reg 0.66 || single replica met 0.900 cert 0.850 reg 5.83 | stored met 0.900 cert 0.875 reg 6.17
|
| 9 |
+
rho=0.9: bestof8 replica met 1.000 cert 1.000 reg 0.90 | stored met 1.000 cert 1.000 reg 1.51 || single replica met 0.875 cert 0.850 reg 4.26 | stored met 0.850 cert 0.850 reg 5.22
|
| 10 |
+
oracle found per rho: [40, 40, 40]
|
| 11 |
+
|
| 12 |
+
--- MAIN DRAW seed 321: K=8 steer=none ---
|
| 13 |
+
rows: rho | arm | spec-met | certified | regret% median [95% CI] p90 n (model = mean±sd over 3 seeds; CI/p90 from seed 0)
|
| 14 |
+
[0.7 ] model met 1.000±0.000 cert 1.000±0.000 medreg 0.82±0.96 | seed0: 2.17 [ 0.66, 4.42] p90 11.80 n=40
|
| 15 |
+
[0.7 ] retr_a met 1.000 cert 1.000 medreg 1.05 [-0.13, 2.47] p90 9.54 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0793 | p over seeds (McN,Wil): 1,0.079; 1,0.29; 1,0.38
|
| 16 |
+
[0.7 ] retr_b met 1.000 cert 1.000 medreg 1.05 [ 0.16, 2.49] p90 8.38 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0548 | p over seeds (McN,Wil): 1,0.055; 1,0.83; 1,0.16
|
| 17 |
+
[0.7 ] retr_c met 1.000 cert 1.000 medreg 1.11 [-0.58, 2.07] p90 7.96 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.025 | p over seeds (McN,Wil): 1,0.025; 1,0.21; 1,0.17
|
| 18 |
+
[0.8 ] model met 1.000±0.000 cert 1.000±0.000 medreg 0.34±0.42 | seed0: -0.25 [-0.80, 0.98] p90 16.59 n=40
|
| 19 |
+
[0.8 ] retr_a met 1.000 cert 1.000 medreg 0.82 [-0.46, 2.05] p90 6.89 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.444 | p over seeds (McN,Wil): 1,0.44; 1,0.24; 1,0.082
|
| 20 |
+
[0.8 ] retr_b met 1.000 cert 1.000 medreg 0.38 [-0.79, 1.95] p90 9.84 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.334 | p over seeds (McN,Wil): 1,0.33; 1,0.23; 1,0.097
|
| 21 |
+
[0.8 ] retr_c met 1.000 cert 1.000 medreg 1.13 [-0.36, 1.78] p90 6.89 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.627 | p over seeds (McN,Wil): 1,0.63; 1,0.32; 1,0.11
|
| 22 |
+
[0.9 ] model met 1.000±0.000 cert 1.000±0.000 medreg 0.97±0.40 | seed0: 0.50 [ 0.23, 1.31] p90 14.98 n=40
|
| 23 |
+
[0.9 ] retr_a met 1.000 cert 1.000 medreg 0.60 [-0.28, 2.28] p90 9.58 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.347 | p over seeds (McN,Wil): 1,0.35; 1,0.034; 1,0.1
|
| 24 |
+
[0.9 ] retr_b met 1.000 cert 1.000 medreg 0.91 [-0.01, 2.72] p90 9.58 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.745 | p over seeds (McN,Wil): 1,0.74; 1,0.062; 1,0.28
|
| 25 |
+
[0.9 ] retr_c met 1.000 cert 1.000 medreg 0.14 [-0.69, 1.60] p90 8.28 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.109 | p over seeds (McN,Wil): 1,0.11; 1,0.022; 1,0.031
|
| 26 |
+
[pooled] model met 1.000±0.000 cert 1.000±0.000 medreg 0.73±0.11 | seed0: 0.65 [ 0.22, 1.19] p90 14.78 n=120
|
| 27 |
+
[pooled] retr_a met 1.000 cert 1.000 medreg 0.87 [ 0.21, 1.71] p90 9.56 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0556 | p over seeds (McN,Wil): 1,0.056; 1,0.015; 1,0.014
|
| 28 |
+
[pooled] retr_b met 1.000 cert 1.000 medreg 0.75 [ 0.21, 1.41] p90 9.56 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0608 | p over seeds (McN,Wil): 1,0.061; 1,0.066; 1,0.014
|
| 29 |
+
[pooled] retr_c met 1.000 cert 1.000 medreg 0.86 [-0.10, 1.56] p90 8.26 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0149 | p over seeds (McN,Wil): 1,0.015; 1,0.0092; 1,0.0024
|
| 30 |
+
|
| 31 |
+
--- MAIN DRAW seed 321: K=1 steer=none ---
|
| 32 |
+
rows: rho | arm | spec-met | certified | regret% median [95% CI] p90 n (model = mean±sd over 3 seeds; CI/p90 from seed 0)
|
| 33 |
+
[0.7 ] model met 0.908±0.012 cert 0.850±0.041 medreg 14.40±5.85 | seed0: 22.68 [ 8.02,36.94] p90 63.08 n=34
|
| 34 |
+
[0.7 ] retr_a met 0.700 cert 0.700 medreg 6.76 [ 2.03,12.26] p90 38.55 n=28 | vs model s0: McNemar 2-10 p=0.0386, Wilcoxon p=0.0395 | p over seeds (McN,Wil): 0.039,0.039; 0.057,0.037; 0.012,0.026
|
| 35 |
+
[0.7 ] retr_b met 0.675 cert 0.675 medreg 9.02 [ 2.90,13.68] p90 29.75 n=27 | vs model s0: McNemar 2-11 p=0.0225, Wilcoxon p=0.0384 | p over seeds (McN,Wil): 0.022,0.038; 0.022,0.052; 0.0063,0.038
|
| 36 |
+
[0.7 ] retr_c met 0.900 cert 0.900 medreg 8.51 [ 3.40,13.84] p90 31.05 n=36 | vs model s0: McNemar 3-3 p=1, Wilcoxon p=0.000906 | p over seeds (McN,Wil): 1,0.00091; 1,0.0013; 1,0.0052
|
| 37 |
+
[0.8 ] model met 0.883±0.051 cert 0.842±0.051 medreg 6.74±1.60 | seed0: 6.20 [ 3.96,15.82] p90 78.90 n=34
|
| 38 |
+
[0.8 ] retr_a met 0.775 cert 0.775 medreg 2.20 [ 0.87, 8.33] p90 24.33 n=31 | vs model s0: McNemar 4-8 p=0.388, Wilcoxon p=0.00247 | p over seeds (McN,Wil): 0.39,0.0025; 0.77,0.031; 0.065,0.06
|
| 39 |
+
[0.8 ] retr_b met 0.725 cert 0.725 medreg 9.86 [ 5.64,13.28] p90 29.73 n=29 | vs model s0: McNemar 4-10 p=0.18, Wilcoxon p=0.127 | p over seeds (McN,Wil): 0.18,0.13; 0.39,0.23; 0.012,0.69
|
| 40 |
+
[0.8 ] retr_c met 0.875 cert 0.875 medreg 5.78 [ 1.78,10.76] p90 27.94 n=35 | vs model s0: McNemar 3-3 p=1, Wilcoxon p=0.000398 | p over seeds (McN,Wil): 1,0.0004; 0.73,0.099; 0.38,0.074
|
| 41 |
+
[0.9 ] model met 0.892±0.031 cert 0.833±0.051 medreg 6.74±0.36 | seed0: 6.29 [ 4.54,11.19] p90 55.91 n=33
|
| 42 |
+
[0.9 ] retr_a met 0.650 cert 0.650 medreg 5.75 [ 4.02,17.38] p90 49.79 n=26 | vs model s0: McNemar 2-10 p=0.0386, Wilcoxon p=0.259 | p over seeds (McN,Wil): 0.039,0.26; 0.013,0.07; 0.0074,0.43
|
| 43 |
+
[0.9 ] retr_b met 0.700 cert 0.700 medreg 11.11 [ 4.49,17.55] p90 35.66 n=28 | vs model s0: McNemar 5-11 p=0.21, Wilcoxon p=0.427 | p over seeds (McN,Wil): 0.21,0.43; 0.057,0.14; 0.012,0.8
|
| 44 |
+
[0.9 ] retr_c met 0.825 cert 0.825 medreg 4.87 [ 1.64, 9.74] p90 31.94 n=33 | vs model s0: McNemar 4-5 p=1, Wilcoxon p=0.0564 | p over seeds (McN,Wil): 1,0.056; 0.55,0.015; 0.29,0.067
|
| 45 |
+
[pooled] model met 0.894±0.027 cert 0.842±0.048 medreg 8.03±0.52 | seed0: 8.62 [ 5.44,13.61] p90 64.20 n=101
|
| 46 |
+
[pooled] retr_a met 0.708 cert 0.708 medreg 6.09 [ 3.64, 8.44] p90 39.49 n=85 | vs model s0: McNemar 8-28 p=0.00119, Wilcoxon p=0.000397 | p over seeds (McN,Wil): 0.0012,0.0004; 0.0022,0.00066; 7.4e-06,0.004
|
| 47 |
+
[pooled] retr_b met 0.700 cert 0.700 medreg 9.73 [ 7.33,13.04] p90 32.05 n=84 | vs model s0: McNemar 11-32 p=0.00191, Wilcoxon p=0.00733 | p over seeds (McN,Wil): 0.0019,0.0073; 0.0011,0.0036; 7.7e-07,0.079
|
| 48 |
+
[pooled] retr_c met 0.867 cert 0.867 medreg 6.78 [ 3.92, 9.18] p90 30.74 n=104 | vs model s0: McNemar 10-11 p=1, Wilcoxon p=1.22e-06 | p over seeds (McN,Wil): 1,1.2e-06; 1,2.3e-05; 0.096,0.00013
|
| 49 |
+
|
| 50 |
+
--- MAIN DRAW seed 321: K=8 steer=S=50 ---
|
| 51 |
+
rows: rho | arm | spec-met | certified | regret% median [95% CI] p90 n (model = mean±sd over 3 seeds; CI/p90 from seed 0)
|
| 52 |
+
[0.7 ] model met 1.000±0.000 cert 1.000±0.000 medreg 0.17±0.05 | seed0: 0.23 [ 0.01, 0.75] p90 4.35 n=40
|
| 53 |
+
[0.7 ] retr_a met 1.000 cert 1.000 medreg -0.13 [-0.48, 0.15] p90 1.27 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0725 | p over seeds (McN,Wil): 1,0.073; 1,3.5e-05; 1,0.0056
|
| 54 |
+
[0.7 ] retr_b met 1.000 cert 1.000 medreg -0.22 [-1.04, 0.14] p90 1.56 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.00726 | p over seeds (McN,Wil): 1,0.0073; 1,0.0018; 1,1.7e-06
|
| 55 |
+
[0.7 ] retr_c met 1.000 cert 1.000 medreg -0.04 [-0.19, 0.23] p90 1.20 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.077 | p over seeds (McN,Wil): 1,0.077; 1,1.7e-05; 1,0.0067
|
| 56 |
+
[0.8 ] model met 1.000±0.000 cert 1.000±0.000 medreg -0.03±0.01 | seed0: -0.02 [-0.06, 0.00] p90 11.62 n=40
|
| 57 |
+
[0.8 ] retr_a met 1.000 cert 1.000 medreg -0.15 [-0.38,-0.08] p90 0.11 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=1.81e-05 | p over seeds (McN,Wil): 1,1.8e-05; 1,5.1e-05; 1,0.00014
|
| 58 |
+
[0.8 ] retr_b met 1.000 cert 1.000 medreg -0.14 [-0.42,-0.07] p90 0.09 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=8.15e-06 | p over seeds (McN,Wil): 1,8.2e-06; 1,3.3e-05; 1,0.0001
|
| 59 |
+
[0.8 ] retr_c met 1.000 cert 1.000 medreg -0.20 [-0.45,-0.10] p90 2.02 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.000603 | p over seeds (McN,Wil): 1,0.0006; 1,0.0013; 1,0.00025
|
| 60 |
+
[0.9 ] model met 1.000±0.000 cert 1.000±0.000 medreg -0.03±0.01 | seed0: -0.02 [-0.09, 0.03] p90 9.95 n=40
|
| 61 |
+
[0.9 ] retr_a met 1.000 cert 1.000 medreg -0.11 [-0.26,-0.06] p90 0.86 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.00558 | p over seeds (McN,Wil): 1,0.0056; 1,0.0012; 1,0.015
|
| 62 |
+
[0.9 ] retr_b met 1.000 cert 1.000 medreg -0.13 [-0.27,-0.04] p90 5.73 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.00305 | p over seeds (McN,Wil): 1,0.0031; 1,0.0016; 1,0.017
|
| 63 |
+
[0.9 ] retr_c met 1.000 cert 1.000 medreg -0.10 [-0.25,-0.07] p90 0.86 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.00726 | p over seeds (McN,Wil): 1,0.0073; 1,0.0014; 1,0.028
|
| 64 |
+
[pooled] model met 1.000±0.000 cert 1.000±0.000 medreg -0.01±0.00 | seed0: -0.00 [-0.03, 0.12] p90 10.09 n=120
|
| 65 |
+
[pooled] retr_a met 1.000 cert 1.000 medreg -0.13 [-0.21,-0.09] p90 1.20 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=4.26e-07 | p over seeds (McN,Wil): 1,4.3e-07; 1,2.3e-10; 1,3.4e-07
|
| 66 |
+
[pooled] retr_b met 1.000 cert 1.000 medreg -0.15 [-0.28,-0.09] p90 1.56 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=1.91e-08 | p over seeds (McN,Wil): 1,1.9e-08; 1,1.1e-08; 1,7.5e-10
|
| 67 |
+
[pooled] retr_c met 1.000 cert 1.000 medreg -0.12 [-0.20,-0.08] p90 1.33 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=4.94e-06 | p over seeds (McN,Wil): 1,4.9e-06; 1,1.4e-09; 1,9.8e-07
|
| 68 |
+
|
| 69 |
+
--- MAIN DRAW seed 321: K=1 steer=S=50 ---
|
| 70 |
+
rows: rho | arm | spec-met | certified | regret% median [95% CI] p90 n (model = mean±sd over 3 seeds; CI/p90 from seed 0)
|
| 71 |
+
[0.7 ] model met 0.992±0.012 cert 0.942±0.042 medreg 5.05±3.48 | seed0: 9.82 [ 0.63,28.76] p90 51.33 n=36
|
| 72 |
+
[0.7 ] retr_a met 0.925 cert 0.900 medreg 0.75 [ 0.18, 1.64] p90 26.67 n=36 | vs model s0: McNemar 1-3 p=0.625, Wilcoxon p=0.0105 | p over seeds (McN,Wil): 0.62,0.011; 0.25,0.023; 0.25,0.0063
|
| 73 |
+
[0.7 ] retr_b met 0.875 cert 0.850 medreg 1.03 [ 0.37, 4.53] p90 21.92 n=34 | vs model s0: McNemar 1-5 p=0.219, Wilcoxon p=0.0149 | p over seeds (McN,Wil): 0.22,0.015; 0.062,0.029; 0.062,0.015
|
| 74 |
+
[0.7 ] retr_c met 1.000 cert 1.000 medreg 1.13 [ 0.36, 4.11] p90 21.78 n=40 | vs model s0: McNemar 1-0 p=1, Wilcoxon p=0.00185 | p over seeds (McN,Wil): 1,0.0018; 1,0.0051; 1,0.0019
|
| 75 |
+
[0.8 ] model met 0.992±0.012 cert 0.933±0.012 medreg 0.37±0.22 | seed0: 0.48 [ 0.27, 5.73] p90 55.20 n=37
|
| 76 |
+
[0.8 ] retr_a met 0.975 cert 0.975 medreg 0.03 [ 0.00, 0.25] p90 18.99 n=39 | vs model s0: McNemar 1-1 p=1, Wilcoxon p=0.000159 | p over seeds (McN,Wil): 1,0.00016; 1,0.0031; 1,0.17
|
| 77 |
+
[0.8 ] retr_b met 0.850 cert 0.850 medreg 0.43 [-0.04, 1.84] p90 15.47 n=34 | vs model s0: McNemar 1-6 p=0.125, Wilcoxon p=0.0175 | p over seeds (McN,Wil): 0.12,0.017; 0.031,0.058; 0.031,0.66
|
| 78 |
+
[0.8 ] retr_c met 1.000 cert 1.000 medreg 0.14 [ 0.02, 1.23] p90 22.30 n=40 | vs model s0: McNemar 1-0 p=1, Wilcoxon p=3.36e-06 | p over seeds (McN,Wil): 1,3.4e-06; 1,0.051; 1,0.14
|
| 79 |
+
[0.9 ] model met 0.992±0.012 cert 0.950±0.054 medreg 0.54±0.31 | seed0: 0.89 [ 0.20, 1.57] p90 36.20 n=40
|
| 80 |
+
[0.9 ] retr_a met 0.975 cert 0.975 medreg 0.11 [-0.00, 0.61] p90 25.81 n=39 | vs model s0: McNemar 0-1 p=1, Wilcoxon p=0.0372 | p over seeds (McN,Wil): 1,0.037; 1,0.0091; 1,0.47
|
| 81 |
+
[0.9 ] retr_b met 0.925 cert 0.925 medreg 0.18 [ 0.03, 1.11] p90 26.47 n=37 | vs model s0: McNemar 0-3 p=0.25, Wilcoxon p=0.0363 | p over seeds (McN,Wil): 0.25,0.036; 0.62,0.037; 0.25,0.61
|
| 82 |
+
[0.9 ] retr_c met 1.000 cert 1.000 medreg 0.11 [ 0.04, 0.53] p90 26.30 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.00336 | p over seeds (McN,Wil): 1,0.0034; 1,0.012; 1,0.38
|
| 83 |
+
[pooled] model met 0.992±0.007 cert 0.942±0.027 medreg 0.70±0.23 | seed0: 0.97 [ 0.43, 1.96] p90 51.49 n=113
|
| 84 |
+
[pooled] retr_a met 0.958 cert 0.950 medreg 0.18 [ 0.06, 0.46] p90 24.60 n=114 | vs model s0: McNemar 2-5 p=0.453, Wilcoxon p=6.12e-06 | p over seeds (McN,Wil): 0.45,6.1e-06; 0.22,1.3e-05; 0.062,0.003
|
| 85 |
+
[pooled] retr_b met 0.883 cert 0.875 medreg 0.49 [ 0.18, 1.11] p90 23.23 n=105 | vs model s0: McNemar 2-14 p=0.00418, Wilcoxon p=6.01e-05 | p over seeds (McN,Wil): 0.0042,6e-05; 0.00098,0.00026; 0.00012,0.037
|
| 86 |
+
[pooled] retr_c met 1.000 cert 1.000 medreg 0.35 [ 0.14, 0.82] p90 23.85 n=120 | vs model s0: McNemar 2-0 p=0.5, Wilcoxon p=2.01e-08 | p over seeds (McN,Wil): 0.5,2e-08; 1,3.2e-05; 1,0.0015
|
| 87 |
+
|
| 88 |
+
synthesis time per plant (ms; mean over rho; verification NOT included):
|
| 89 |
+
model:s0 steer 258.21
|
| 90 |
+
model:s0 synth 89.52
|
| 91 |
+
model:s1 steer 270.89
|
| 92 |
+
model:s1 synth 91.89
|
| 93 |
+
model:s2 steer 267.04
|
| 94 |
+
model:s2 synth 89.57
|
| 95 |
+
retr_a steer 278.56
|
| 96 |
+
retr_a synth 0.08
|
| 97 |
+
retr_b steer 271.39
|
| 98 |
+
retr_b synth 0.09
|
| 99 |
+
retr_c steer 253.12
|
| 100 |
+
retr_c synth 0.14
|
| 101 |
+
|
| 102 |
+
oracle found per rho (draw 777): [40, 40, 40]
|
| 103 |
+
|
| 104 |
+
--- SECOND DRAW seed 777: K=8 steer=none ---
|
| 105 |
+
rows: rho | arm | spec-met | certified | regret% median [95% CI] p90 n (model = mean±sd over 3 seeds; CI/p90 from seed 0)
|
| 106 |
+
[0.7 ] model met 1.000±0.000 cert 1.000±0.000 medreg 1.13±0.34 | seed0: 1.23 [-0.22, 3.18] p90 11.81 n=40
|
| 107 |
+
[0.7 ] retr_a met 1.000 cert 1.000 medreg 0.94 [-0.93, 3.71] p90 12.46 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.952 | p over seeds (McN,Wil): 1,0.95; 1,0.045; 1,0.79
|
| 108 |
+
[0.7 ] retr_b met 1.000 cert 1.000 medreg 1.10 [-0.21, 2.94] p90 9.35 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.32 | p over seeds (McN,Wil): 1,0.32; 1,0.51; 1,0.25
|
| 109 |
+
[0.7 ] retr_c met 1.000 cert 1.000 medreg 0.50 [-0.93, 3.02] p90 11.93 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.847 | p over seeds (McN,Wil): 1,0.85; 1,0.12; 1,0.51
|
| 110 |
+
[0.8 ] model met 1.000±0.000 cert 1.000±0.000 medreg 0.97±0.29 | seed0: 0.62 [ 0.01, 1.74] p90 15.47 n=40
|
| 111 |
+
[0.8 ] retr_a met 1.000 cert 1.000 medreg 0.67 [-0.05, 2.42] p90 5.18 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.347 | p over seeds (McN,Wil): 1,0.35; 1,0.18; 1,0.024
|
| 112 |
+
[0.8 ] retr_b met 1.000 cert 1.000 medreg 1.31 [ 0.23, 2.61] p90 5.59 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.361 | p over seeds (McN,Wil): 1,0.36; 1,0.25; 1,0.15
|
| 113 |
+
[0.8 ] retr_c met 1.000 cert 1.000 medreg 0.44 [-0.16, 1.69] p90 5.43 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.277 | p over seeds (McN,Wil): 1,0.28; 1,0.14; 1,0.0094
|
| 114 |
+
[0.9 ] model met 1.000±0.000 cert 1.000±0.000 medreg 1.05±0.05 | seed0: 1.08 [ 0.46, 2.54] p90 8.63 n=40
|
| 115 |
+
[0.9 ] retr_a met 1.000 cert 1.000 medreg 0.64 [-0.44, 1.57] p90 7.26 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.125 | p over seeds (McN,Wil): 1,0.12; 1,0.094; 1,0.71
|
| 116 |
+
[0.9 ] retr_b met 1.000 cert 1.000 medreg 0.65 [-0.34, 1.66] p90 7.14 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.132 | p over seeds (McN,Wil): 1,0.13; 1,0.16; 1,0.53
|
| 117 |
+
[0.9 ] retr_c met 0.975 cert 0.975 medreg 0.50 [-0.35, 1.24] p90 5.51 n=39 | vs model s0: McNemar 0-1 p=1, Wilcoxon p=0.124 | p over seeds (McN,Wil): 1,0.12; 1,0.1; 1,0.69
|
| 118 |
+
[pooled] model met 1.000±0.000 cert 1.000±0.000 medreg 0.96±0.11 | seed0: 0.92 [ 0.43, 1.59] p90 11.90 n=120
|
| 119 |
+
[pooled] retr_a met 1.000 cert 1.000 medreg 0.80 [ 0.06, 1.32] p90 9.77 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.155 | p over seeds (McN,Wil): 1,0.16; 1,0.61; 1,0.097
|
| 120 |
+
[pooled] retr_b met 1.000 cert 1.000 medreg 0.99 [ 0.14, 1.57] p90 8.49 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0619 | p over seeds (McN,Wil): 1,0.062; 1,0.29; 1,0.062
|
| 121 |
+
[pooled] retr_c met 0.992 cert 0.992 medreg 0.50 [ 0.01, 1.01] p90 9.27 n=119 | vs model s0: McNemar 0-1 p=1, Wilcoxon p=0.102 | p over seeds (McN,Wil): 1,0.1; 1,0.29; 1,0.037
|
| 122 |
+
|
| 123 |
+
synthesis time per plant (ms; mean over rho; verification NOT included):
|
| 124 |
+
model:s0 synth 91.34
|
| 125 |
+
model:s1 synth 90.86
|
| 126 |
+
model:s2 synth 91.90
|
| 127 |
+
retr_a synth 0.08
|
| 128 |
+
retr_b synth 0.11
|
| 129 |
+
retr_c synth 0.11
|
results/baselines/retrieval_pendulum.txt
ADDED
|
@@ -0,0 +1,28 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== PENDULUM retrieval/interpolation baselines (N=40, seed=123) ===
|
| 2 |
+
verifier: certify_controller (regional LMI) + simulate_pendulum_batch rollout (n_ic=3, dt=0.02, duration=6.0, energy_thresh=0.02, theta0=0.35)
|
| 3 |
+
oracle: CACHED results/pendulum/eval.npz (eval_pendulum.strong_oracle, 24 restarts x 100 steps), oracle_J backed out from best_costs/regrets
|
| 4 |
+
K_std=8, diffusion seeds=(0, 1, 2)
|
| 5 |
+
|
| 6 |
+
--- K=8 ---
|
| 7 |
+
arm cert_any stable_any persamp reg_med% reg_p90% reg_max% ms/plant McNemar(n10,n01,p) Wilcoxon(n,p)
|
| 8 |
+
diffusion 1.00 1.00 1.00 1.09 2.46 3.36 171.13 (reference) (reference)
|
| 9 |
+
(sd across 3 seeds: cert_any 0.000, stable_any 0.000, per-samp 0.001)
|
| 10 |
+
retrieval 1.00 1.00 1.00 0.40 0.86 1.70 0.375 (0,0,1) (40,2.8e-07)
|
| 11 |
+
interpolation 1.00 1.00 1.00 0.40 0.86 1.70 0.375 (0,0,1) (40,2.8e-07)
|
| 12 |
+
|
| 13 |
+
--- K=1 ---
|
| 14 |
+
arm cert_any stable_any persamp reg_med% reg_p90% reg_max% ms/plant McNemar(n10,n01,p) Wilcoxon(n,p)
|
| 15 |
+
diffusion 1.00 1.00 1.00 1.54 3.59 4.80 171.13 (reference) (reference)
|
| 16 |
+
(sd across 3 seeds: cert_any 0.000, stable_any 0.000, per-samp 0.000)
|
| 17 |
+
retrieval 1.00 1.00 1.00 1.84 3.70 5.85 0.127 (0,0,1) (40,0.13)
|
| 18 |
+
interpolation 1.00 1.00 1.00 1.84 3.70 5.85 0.127 (0,0,1) (40,0.13)
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
--- K=8 bootstrap 95% CIs (addendum, computed post-hoc from saved npz; not re-simulated) ---
|
| 22 |
+
diffusion (seed[0]): coverage 95% CI [1.000,1.000] regret_med 95% CI [+0.88,+1.35]%
|
| 23 |
+
retrieval: coverage 95% CI [1.000,1.000] regret_med 95% CI [+0.22,+0.53]%
|
| 24 |
+
interpolation: coverage 95% CI [1.000,1.000] regret_med 95% CI [+0.22,+0.53]%
|
| 25 |
+
--- K=1 bootstrap 95% CIs (addendum, computed post-hoc from saved npz; not re-simulated) ---
|
| 26 |
+
diffusion (seed[0]): coverage 95% CI [1.000,1.000] regret_med 95% CI [+1.33,+2.14]%
|
| 27 |
+
retrieval: coverage 95% CI [1.000,1.000] regret_med 95% CI [+1.37,+2.64]%
|
| 28 |
+
interpolation: coverage 95% CI [1.000,1.000] regret_med 95% CI [+1.37,+2.64]%
|
results/baselines/retrieval_pendulum_seed321.txt
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== PENDULUM retrieval/interpolation baselines (N=40, seed=321) ===
|
| 2 |
+
verifier: certify_controller (regional LMI) + simulate_pendulum_batch rollout (n_ic=3, dt=0.02, duration=6.0, energy_thresh=0.02)
|
| 3 |
+
oracle: FRESH (not previously cached): eval_pendulum.strong_oracle (24 restarts x 100 steps), computed this run and cached to results/baselines/pendulum_oracle_seed321.npz
|
| 4 |
+
K_std=8, diffusion seeds=(0, 1, 2)
|
| 5 |
+
|
| 6 |
+
--- K=8 ---
|
| 7 |
+
arm cert_any stable_any persamp reg_med% reg_p90% reg_max% ms/plant McNemar(n10,n01,p) Wilcoxon(n,p)
|
| 8 |
+
diffusion 1.00 1.00 1.00 0.97 2.21 3.17 283.48 (reference) (reference)
|
| 9 |
+
(sd across 3 seeds: cert_any 0.000, stable_any 0.000, per-samp 0.000)
|
| 10 |
+
(seed[0] 95% CI: stable_any [1.00,1.00] regret_med [+0.75,+1.31]%)
|
| 11 |
+
retrieval 1.00 1.00 1.00 0.29 0.64 0.86 0.373 (0,0,1) (40,1e-07)
|
| 12 |
+
(Wilcoxon n=40 = plants where BOTH this arm and diffusion have a finite cost, i.e. the COMMON success set, not either arm's own coverage count)
|
| 13 |
+
(95% CI: stable_any [1.00,1.00] regret_med [+0.16,+0.36]%)
|
| 14 |
+
interpolation 1.00 1.00 1.00 0.30 0.67 2.01 0.373 (0,0,1) (40,1e-07)
|
| 15 |
+
(Wilcoxon n=40 = plants where BOTH this arm and diffusion have a finite cost, i.e. the COMMON success set, not either arm's own coverage count)
|
| 16 |
+
(95% CI: stable_any [1.00,1.00] regret_med [+0.16,+0.42]%)
|
| 17 |
+
|
| 18 |
+
--- K=1 ---
|
| 19 |
+
arm cert_any stable_any persamp reg_med% reg_p90% reg_max% ms/plant McNemar(n10,n01,p) Wilcoxon(n,p)
|
| 20 |
+
diffusion 1.00 1.00 1.00 1.55 3.37 5.05 283.48 (reference) (reference)
|
| 21 |
+
(sd across 3 seeds: cert_any 0.000, stable_any 0.000, per-samp 0.000)
|
| 22 |
+
(seed[0] 95% CI: stable_any [1.00,1.00] regret_med [+1.44,+1.94]%)
|
| 23 |
+
retrieval 1.00 1.00 1.00 1.36 2.68 5.20 0.128 (0,0,1) (40,0.055)
|
| 24 |
+
(Wilcoxon n=40 = plants where BOTH this arm and diffusion have a finite cost, i.e. the COMMON success set, not either arm's own coverage count)
|
| 25 |
+
(95% CI: stable_any [1.00,1.00] regret_med [+1.14,+1.81]%)
|
| 26 |
+
interpolation 1.00 1.00 1.00 1.36 2.68 5.20 0.128 (0,0,1) (40,0.055)
|
| 27 |
+
(Wilcoxon n=40 = plants where BOTH this arm and diffusion have a finite cost, i.e. the COMMON success set, not either arm's own coverage count)
|
| 28 |
+
(95% CI: stable_any [1.00,1.00] regret_med [+1.14,+1.81]%)
|
| 29 |
+
|
results/baselines/retrieval_siso.txt
ADDED
|
@@ -0,0 +1,36 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== SISO retrieval/interpolation baselines (N=60, seed=0) ===
|
| 2 |
+
verifier: batched RK4 rollout (nonlinear.costs.diff_realize machinery, family's actual q_state -- see module docstring), padded NSTATE=8, shared x0 (seed=777), n_ic=4, dt=0.03, duration=12.0, energy_thresh=0.3
|
| 3 |
+
oracle: NOT a single global oracle. generalize.py's SISO oracle protocol (60-step Adam, phi-only, sector-projected, lr=0.06) is warm-started from a candidate by construction, so we apply it IDENTICALLY to each arm's own best candidate, batched across plants (documented adaptation -- see docs/retrieval_baselines.md). Primary comparison is the direct paired cov/cost test vs the diffusion arm below, which needs no oracle.
|
| 4 |
+
K_std=8, diffusion seeds=(0, 1, 2)
|
| 5 |
+
|
| 6 |
+
--- K=8 ---
|
| 7 |
+
arm coverage persamp reg_med% reg_p90% reg_max% ms/plant McNemar(n10,n01,p) Wilcoxon(n,p)
|
| 8 |
+
diffusion 0.79 0.63 10.17 44.69 121.17 21.53 (reference) (reference)
|
| 9 |
+
(sd across 3 seeds: coverage 0.008, per-samp 0.010; seed[0] 95% CI: coverage [0.67,0.88] regret_med [+7.04,+20.00]%)
|
| 10 |
+
retrieval 0.52 0.52 0.16 4.42 61.94 6.665 (0,16,3.1e-05) (31,0.018)
|
| 11 |
+
(95% CI: coverage [0.40,0.65] regret_med [+0.05,+0.34]%)
|
| 12 |
+
interpolation 0.52 0.52 0.16 4.42 61.94 6.665 (0,16,3.1e-05) (31,0.018)
|
| 13 |
+
(95% CI: coverage [0.40,0.65] regret_med [+0.05,+0.34]%)
|
| 14 |
+
(regressor is a point estimate, budget-independent -- reported once, at K=1)
|
| 15 |
+
|
| 16 |
+
--- K=1 ---
|
| 17 |
+
arm coverage persamp reg_med% reg_p90% reg_max% ms/plant McNemar(n10,n01,p) Wilcoxon(n,p)
|
| 18 |
+
diffusion 0.67 0.67 9.02 69.47 161.30 21.53 (reference) (reference)
|
| 19 |
+
(sd across 3 seeds: coverage 0.021, per-samp 0.021; seed[0] 95% CI: coverage [0.53,0.77] regret_med [+6.18,+14.24]%)
|
| 20 |
+
retrieval 0.52 0.52 0.16 4.42 61.94 6.789 (3,11,0.057) (28,0.00058)
|
| 21 |
+
(95% CI: coverage [0.40,0.65] regret_med [+0.05,+0.34]%)
|
| 22 |
+
interpolation 0.52 0.52 0.16 4.42 61.94 6.789 (3,11,0.057) (28,0.00058)
|
| 23 |
+
(95% CI: coverage [0.40,0.65] regret_med [+0.05,+0.34]%)
|
| 24 |
+
regressor 0.75 0.75 63.96 88.20 125.65 0.299 (8,2,0.11) (37,2.2e-05)
|
| 25 |
+
(95% CI: coverage [0.63,0.85] regret_med [+54.57,+71.82]%)
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
--- addendum: confirming the success set used by the Wilcoxon cost test above (coordinator follow-up #3) ---
|
| 29 |
+
K=8: diffusion(seed0) covers 47/60 plants
|
| 30 |
+
retrieval: covers 31/60; common-with-diffusion=31 (= a strict subset of diffusion coverage)
|
| 31 |
+
interpolation: covers 31/60; common-with-diffusion=31 (= a strict subset of diffusion coverage)
|
| 32 |
+
K=1: diffusion(seed0) covers 39/60 plants
|
| 33 |
+
retrieval: covers 31/60; common-with-diffusion=28 (NOT a subset -- some plants only this arm covers)
|
| 34 |
+
interpolation: covers 31/60; common-with-diffusion=28 (NOT a subset -- some plants only this arm covers)
|
| 35 |
+
regressor: covers 45/60; common-with-diffusion=37 (NOT a subset -- some plants only this arm covers)
|
| 36 |
+
The Wilcoxon n reported in the table above IS this common-coverage count in every row (wilcoxon_test restricts to np.isfinite(a)&np.isfinite(b) before testing, i.e. plants where BOTH arms have a finite/stable cost) -- confirmed by direct recomputation from the saved per-plant arrays, not just re-reading the code. For K=8 retrieval/interpolation, n=31 exactly equals retrieval's own coverage count because every plant retrieval covers is ALSO covered by diffusion (a strict subset) -- so the reported regret numbers for retrieval/interpolation are simultaneously "conditional on own success set" AND "the common success set" at K=8.
|
results/baselines/siso_run.log
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[seed 0] done
|
| 2 |
+
[seed 1] done
|
| 3 |
+
[seed 2] done
|
| 4 |
+
[seed 3] done
|
| 5 |
+
[seed 4] done
|
| 6 |
+
|
| 7 |
+
=== SISO BASELINES ===
|
| 8 |
+
SISO why-generative baseline — pooled over 5 seeds × 60 plants (N=300), K=8
|
| 9 |
+
|
| 10 |
+
regressor : plant-stabilisable 0.72 [0.67,0.77] per-sample-stable 0.72
|
| 11 |
+
diff_1shot : plant-stabilisable 0.61 [0.56,0.67] per-sample-stable 0.61
|
| 12 |
+
diff_Kshot : plant-stabilisable 0.75 [0.70,0.80] per-sample-stable 0.61
|
| 13 |
+
|
| 14 |
+
diffusion K-shot vs regressor (paired McNemar-ish sign test): p = 3.09e-02
|
| 15 |
+
controller diversity (per-point std across K diffusion samples): 3.4 (regressor = 0, a single point)
|
| 16 |
+
|
| 17 |
+
TAKEAWAY: the regressor (conditional mean) stabilises fewer plants than diffusion sampled
|
| 18 |
+
K times; the generative DISTRIBUTION is what lets test-time selection/repair find a
|
| 19 |
+
stabilising controller a point estimate misses.
|
results/furuta_v3/deploy_sidebyside.txt
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
deployment mismatch, K=8, nominal-conditioned generated+P (mean+-sd over 3 sampler seeds) vs retrieval+P and NMPC (reused, n=120 plants)
|
| 2 |
+
scenario retr NMPC v2 clipped v2 clip5 v3 v3 e2400 v3 vs retr (b-c,p) v3 vs v2clip (b-c,p) e2400 vs retr (b-c,p)
|
| 3 |
+
nominal 90.0% 95.8% 65.3+-0.4 85.8+-0.7 86.1+-1.6 91.4+-0.8 6-4 p=0.7539 29-1 p=0.0000 5-6 p=1.0000
|
| 4 |
+
fric0.6 86.7% 95.8% 62.5+-0.0 80.8+-2.7 84.4+-0.4 91.1+-1.7 10-8 p=0.8145 29-2 p=0.0000 8-11 p=0.6476
|
| 5 |
+
fric1.6 9.2% 64.2% 10.6+-0.8 11.1+-0.4 9.7+-1.4 10.3+-0.8 1-4 p=0.3750 4-4 p=1.0000 3-3 p=1.0000
|
| 6 |
+
fric2.5 0.0% 29.2% 0.0+-0.0 0.6+-0.4 0.3+-0.4 0.0+-0.0 0-1 p=1.0000 1-0 p=1.0000 0-0 p=1.0000
|
| 7 |
+
ml_mp-_Lp- 68.3% 94.2% 50.0+-4.5 63.6+-0.4 69.4+-2.4 70.8+-1.4 10-8 p=0.8145 29-8 p=0.0008 8-11 p=0.6476
|
| 8 |
+
ml_mp-_Lp+ 81.7% 95.8% 61.7+-0.0 77.2+-1.0 83.3+-0.7 89.2+-1.4 10-12 p=0.8318 31-5 p=0.0000 9-16 p=0.2295
|
| 9 |
+
ml_mp+_Lp- 48.3% 95.0% 39.7+-2.7 53.1+-3.5 47.8+-2.1 51.4+-2.0 15-15 p=1.0000 25-11 p=0.0288 14-16 p=0.8555
|
| 10 |
+
ml_mp+_Lp+ 75.8% 95.0% 57.5+-1.2 72.2+-2.1 78.3+-1.4 82.8+-0.4 15-18 p=0.7283 33-6 p=0.0000 11-19 p=0.2005
|
| 11 |
+
both_f1.6_mp-_Lp- 9.2% 55.0% 11.7+-0.7 13.9+-0.4 11.9+-2.1 11.9+-1.4 3-9 p=0.1460 7-3 p=0.3438 3-7 p=0.3438
|
| 12 |
+
both_f1.6_mp-_Lp+ 19.2% 75.0% 17.8+-1.0 17.8+-1.7 18.6+-1.4 18.9+-1.7 4-5 p=1.0000 6-2 p=0.2891 5-5 p=1.0000
|
| 13 |
+
both_f1.6_mp+_Lp- 0.8% 52.5% 0.8+-0.0 0.8+-0.0 0.8+-0.0 0.6+-0.8 1-1 p=1.0000 1-1 p=1.0000 1-0 p=1.0000
|
| 14 |
+
both_f1.6_mp+_Lp+ 1.7% 67.5% 2.5+-0.7 3.3+-0.7 1.7+-0.7 2.2+-1.4 0-1 p=1.0000 3-3 p=1.0000 0-3 p=0.2500
|
| 15 |
+
both_f2.5_mp-_Lp- 0.0% 21.7% 0.3+-0.4 0.0+-0.0 0.0+-0.0 0.3+-0.4 0-0 p=1.0000 0-0 p=1.0000 0-0 p=1.0000
|
| 16 |
+
both_f2.5_mp-_Lp+ 0.0% 33.3% 1.1+-0.4 0.6+-0.8 0.3+-0.4 0.0+-0.0 0-0 p=1.0000 0-1 p=1.0000 0-0 p=1.0000
|
| 17 |
+
both_f2.5_mp+_Lp- 0.0% 18.3% 0.0+-0.0 0.3+-0.4 0.3+-0.4 0.0+-0.0 0-0 p=1.0000 0-0 p=1.0000 0-0 p=1.0000
|
| 18 |
+
both_f2.5_mp+_Lp+ 0.0% 33.3% 0.0+-0.0 0.3+-0.4 0.0+-0.0 0.0+-0.0 0-0 p=1.0000 0-0 p=1.0000 0-0 p=1.0000
|
| 19 |
+
D5 (diffusion mean >= retrieval mean in some scenario): max over scenarios of (diffusion - retrieval), points: v2 clipped: +2.5, v2 clip5: +4.7, v3: +2.8, v3 e2400: +7.5
|
results/furuta_v3/ksweep_sidebyside.txt
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
generated+P pooled over both draws (mean of 3 sampler seeds) K=1 K=4 K=8 K=16
|
| 2 |
+
v2 clipped (shipped) 33.9% 58.9% 65.2% 68.0%
|
| 3 |
+
v2 clip +-5 (audit) 34.1% 75.0% 85.8% 90.0%
|
| 4 |
+
v3 s0 800ep 38.3% 75.8% 86.1% 88.6%
|
| 5 |
+
v3 s1 800ep 31.6% 71.4% 81.9% 86.4%
|
| 6 |
+
v3 s0 2400ep 53.6% 84.8% 91.3% 93.7%
|
| 7 |
+
v3 s1 2400ep 58.4% 87.5% 92.5% 93.3%
|
| 8 |
+
v2-data clip5 2400ep (control) 60.8% 87.0% 90.6% 91.9%
|
| 9 |
+
nearest+P (retrieval, same bank) 54.2% 82.5% 90.0% 93.3%
|
results/furuta_v3/traj_gen_eval_v3_raw_clip1.txt
ADDED
|
@@ -0,0 +1,94 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
bank data/furuta_nmpc_bank3000.npz: 2801 entries
|
| 2 |
+
model models/Best_furuta_traj_cond_v3.pth: cond_dim=4 out_dim=116 hidden=256 layers=5 device=cpu
|
| 3 |
+
[seed4242] margin=0.3: resolving P for 227 unique bank source plants
|
| 4 |
+
[seed4242] sampler seed 0: 9.6s for 60x4 candidates (0 decode-invalid of 240)
|
| 5 |
+
[seed4242] sampler seed 1: 9.5s for 60x4 candidates (0 decode-invalid of 240)
|
| 6 |
+
[seed4242] sampler seed 2: 9.6s for 60x4 candidates (0 decode-invalid of 240)
|
| 7 |
+
[seed4242] 10/60 (31s)
|
| 8 |
+
[seed4242] 20/60 (46s)
|
| 9 |
+
[seed4242] 30/60 (72s)
|
| 10 |
+
[seed4242] 40/60 (88s)
|
| 11 |
+
[seed4242] 50/60 (108s)
|
| 12 |
+
[seed4242] 60/60 (137s)
|
| 13 |
+
[seed9001] margin=0.3: resolving P for 232 unique bank source plants
|
| 14 |
+
[seed9001] sampler seed 0: 10.5s for 60x4 candidates (0 decode-invalid of 240)
|
| 15 |
+
[seed9001] sampler seed 1: 10.7s for 60x4 candidates (0 decode-invalid of 240)
|
| 16 |
+
[seed9001] sampler seed 2: 10.2s for 60x4 candidates (0 decode-invalid of 240)
|
| 17 |
+
[seed9001] 10/60 (33s)
|
| 18 |
+
[seed9001] 20/60 (57s)
|
| 19 |
+
[seed9001] 30/60 (80s)
|
| 20 |
+
[seed9001] 40/60 (101s)
|
| 21 |
+
[seed9001] 50/60 (124s)
|
| 22 |
+
[seed9001] 60/60 (153s)
|
| 23 |
+
|
| 24 |
+
=== traj_gen_eval (v3_raw_clip1): generated vs retrieval vs NMPC ===
|
| 25 |
+
|
| 26 |
+
--- plant set seed4242 (seed 4242, n=60) ---
|
| 27 |
+
arm succ 95% CI medJ med regret p90 synth
|
| 28 |
+
per-plant 93% [ 87, 98] 4.08 0.00% 0.00% 4.6s
|
| 29 |
+
nearest 62% [ 50, 73] 4.15 0.36% 1.51% 326ms
|
| 30 |
+
nearest+P 78% [ 67, 88] 4.09 0.69% 2.52% 638ms
|
| 31 |
+
nearest-margin 62% [ 50, 73] 4.15 0.36% 1.51% 306ms
|
| 32 |
+
NMPC 95% [ 88, 100] 4.08 0.14% 0.24% 26ms
|
| 33 |
+
generated 55.6%(sd3.9) [ 38, 63]* 4.05 1.05%(sd0.3) 3.14%(sd0.8) 406ms
|
| 34 |
+
generated+P 76.7%(sd1.4) [ 63, 85]* 4.09 1.04%(sd0.1) 3.58%(sd0.5) 1.1s
|
| 35 |
+
(* generated rows: mean (sd) over sampler seeds [0, 1, 2]; CI shown is for seed 0 only)
|
| 36 |
+
(certified hand-off rate == success rate for every arm except NMPC (a certificate is a precondition for success here); NMPC's own hand-off is n/a, it never switches to a static gain)
|
| 37 |
+
|
| 38 |
+
--- failure breakdown, primary seed 0 ---
|
| 39 |
+
per-plant NMPC never reached Omega:3 replay failed:1
|
| 40 |
+
nearest Lyapunov decrease fails at a vertex:13 rollout failed:9 true dynamics: 0.02% of the boundary has V-dot >= 0, 0.00% saturates:1
|
| 41 |
+
nearest+P rollout failed:13
|
| 42 |
+
nearest-margin Lyapunov decrease fails at a vertex:13 rollout failed:9 true dynamics: 0.02% of the boundary has V-dot >= 0, 0.00% saturates:1
|
| 43 |
+
NMPC :3
|
| 44 |
+
generated rollout failed:12 Lyapunov decrease fails at a vertex:9 peak |Kx| on Omega is 10.05 V > 10.0 V:2 peak |Kx| on Omega is 10.01 V > 10.0 V:2 peak |Kx| on Omega is 10.17 V > 10.0 V:1 peak |Kx| on Omega is 10.14 V > 10.0 V:1 peak |Kx| on Omega is 10.37 V > 10.0 V:1 peak |Kx| on Omega is 10.07 V > 10.0 V:1 peak |Kx| on Omega is 10.12 V > 10.0 V:1
|
| 45 |
+
generated+P rollout failed:15
|
| 46 |
+
|
| 47 |
+
--- paired McNemar on success (exact binomial on discordant pairs) ---
|
| 48 |
+
per-plant vs nearest beats on 20, loses on 1 p=0.0000 * cost ratio 1.004x p=0.0000
|
| 49 |
+
per-plant vs nearest+P beats on 10, loses on 1 p=0.0117 * cost ratio 1.007x p=0.0000
|
| 50 |
+
per-plant vs nearest-margin beats on 20, loses on 1 p=0.0000 * cost ratio 1.004x p=0.0000
|
| 51 |
+
per-plant vs NMPC beats on 0, loses on 1 p=1.0000 cost ratio 1.001x p=0.0002
|
| 52 |
+
per-plant vs generated beats on 26, loses on 0 p=0.0000 * cost ratio 1.010x p=0.0000
|
| 53 |
+
per-plant vs generated+P beats on 12, loses on 1 p=0.0034 * cost ratio 1.011x p=0.0000
|
| 54 |
+
nearest+P vs per-plant beats on 1, loses on 10 p=0.0117 * cost ratio 0.993x p=0.0000
|
| 55 |
+
nearest+P vs nearest beats on 10, loses on 0 p=0.0020 * cost ratio 1.000x p=0.8291
|
| 56 |
+
nearest+P vs nearest-margin beats on 10, loses on 0 p=0.0020 * cost ratio 1.000x p=0.8291
|
| 57 |
+
nearest+P vs NMPC beats on 0, loses on 10 p=0.0020 * cost ratio 0.995x p=0.0000
|
| 58 |
+
nearest+P vs generated beats on 20, loses on 3 p=0.0005 * cost ratio 1.005x p=0.0815
|
| 59 |
+
nearest+P vs generated+P beats on 8, loses on 6 p=0.7905 cost ratio 1.002x p=0.0414
|
| 60 |
+
|
| 61 |
+
--- plant set seed9001 (seed 9001, n=60) ---
|
| 62 |
+
arm succ 95% CI medJ med regret p90 synth
|
| 63 |
+
per-plant 93% [ 87, 98] 4.16 0.00% 0.00% 7.5s
|
| 64 |
+
nearest 65% [ 52, 77] 4.10 0.71% 2.11% 218ms
|
| 65 |
+
nearest+P 87% [ 78, 95] 4.14 0.59% 2.34% 567ms
|
| 66 |
+
nearest-margin 65% [ 52, 77] 4.10 0.71% 2.11% 260ms
|
| 67 |
+
NMPC 97% [ 92, 100] 4.17 0.07% 0.22% 34ms
|
| 68 |
+
generated 57.2%(sd3.4) [ 43, 68]* 4.16 0.66%(sd0.1) 2.85%(sd0.7) 448ms
|
| 69 |
+
generated+P 79.4%(sd3.4) [ 73, 93]* 4.15 0.76%(sd0.2) 3.50%(sd0.4) 1.0s
|
| 70 |
+
(* generated rows: mean (sd) over sampler seeds [0, 1, 2]; CI shown is for seed 0 only)
|
| 71 |
+
(certified hand-off rate == success rate for every arm except NMPC (a certificate is a precondition for success here); NMPC's own hand-off is n/a, it never switches to a static gain)
|
| 72 |
+
|
| 73 |
+
--- failure breakdown, primary seed 0 ---
|
| 74 |
+
per-plant replay failed:2 NMPC never reached Omega:2
|
| 75 |
+
nearest Lyapunov decrease fails at a vertex:16 rollout failed:5
|
| 76 |
+
nearest+P rollout failed:8
|
| 77 |
+
nearest-margin Lyapunov decrease fails at a vertex:16 rollout failed:5
|
| 78 |
+
NMPC :2
|
| 79 |
+
generated Lyapunov decrease fails at a vertex:19 rollout failed:2 peak |Kx| on Omega is 10.03 V > 10.0 V:1 peak |Kx| on Omega is 10.15 V > 10.0 V:1 true dynamics: 0.29% of the boundary has V-dot >= 0, 0.00% saturates:1 peak |Kx| on Omega is 10.06 V > 10.0 V:1 true dynamics: 0.60% of the boundary has V-dot >= 0, 0.00% saturates:1
|
| 80 |
+
generated+P rollout failed:10
|
| 81 |
+
|
| 82 |
+
--- paired McNemar on success (exact binomial on discordant pairs) ---
|
| 83 |
+
per-plant vs nearest beats on 18, loses on 1 p=0.0001 * cost ratio 1.007x p=0.0000
|
| 84 |
+
per-plant vs nearest+P beats on 5, loses on 1 p=0.2188 cost ratio 1.006x p=0.0000
|
| 85 |
+
per-plant vs nearest-margin beats on 18, loses on 1 p=0.0001 * cost ratio 1.007x p=0.0000
|
| 86 |
+
per-plant vs NMPC beats on 0, loses on 2 p=0.5000 cost ratio 1.001x p=0.0064
|
| 87 |
+
per-plant vs generated beats on 22, loses on 0 p=0.0000 * cost ratio 1.008x p=0.0000
|
| 88 |
+
per-plant vs generated+P beats on 6, loses on 0 p=0.0312 * cost ratio 1.010x p=0.0000
|
| 89 |
+
nearest+P vs per-plant beats on 1, loses on 5 p=0.2188 cost ratio 0.994x p=0.0000
|
| 90 |
+
nearest+P vs nearest beats on 13, loses on 0 p=0.0002 * cost ratio 1.000x p=0.1154
|
| 91 |
+
nearest+P vs nearest-margin beats on 13, loses on 0 p=0.0002 * cost ratio 1.000x p=0.1154
|
| 92 |
+
nearest+P vs NMPC beats on 0, loses on 6 p=0.0312 * cost ratio 0.995x p=0.0000
|
| 93 |
+
nearest+P vs generated beats on 20, loses on 2 p=0.0001 * cost ratio 1.003x p=0.1240
|
| 94 |
+
nearest+P vs generated+P beats on 5, loses on 3 p=0.7266 cost ratio 1.004x p=0.0165
|
results/furuta_v3/traj_gen_eval_v3e2400s1_raw_clip1.txt
ADDED
|
@@ -0,0 +1,94 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
bank data/furuta_nmpc_bank3000.npz: 2801 entries
|
| 2 |
+
model models/Best_furuta_traj_cond_v3_e2400_s1.pth: cond_dim=4 out_dim=116 hidden=256 layers=5 device=cpu
|
| 3 |
+
[seed4242] margin=0.3: resolving P for 227 unique bank source plants
|
| 4 |
+
[seed4242] sampler seed 0: 9.6s for 60x4 candidates (0 decode-invalid of 240)
|
| 5 |
+
[seed4242] sampler seed 1: 9.5s for 60x4 candidates (0 decode-invalid of 240)
|
| 6 |
+
[seed4242] sampler seed 2: 9.6s for 60x4 candidates (0 decode-invalid of 240)
|
| 7 |
+
[seed4242] 10/60 (23s)
|
| 8 |
+
[seed4242] 20/60 (39s)
|
| 9 |
+
[seed4242] 30/60 (57s)
|
| 10 |
+
[seed4242] 40/60 (74s)
|
| 11 |
+
[seed4242] 50/60 (94s)
|
| 12 |
+
[seed4242] 60/60 (119s)
|
| 13 |
+
[seed9001] margin=0.3: resolving P for 232 unique bank source plants
|
| 14 |
+
[seed9001] sampler seed 0: 9.8s for 60x4 candidates (0 decode-invalid of 240)
|
| 15 |
+
[seed9001] sampler seed 1: 10.0s for 60x4 candidates (0 decode-invalid of 240)
|
| 16 |
+
[seed9001] sampler seed 2: 10.2s for 60x4 candidates (0 decode-invalid of 240)
|
| 17 |
+
[seed9001] 10/60 (29s)
|
| 18 |
+
[seed9001] 20/60 (49s)
|
| 19 |
+
[seed9001] 30/60 (78s)
|
| 20 |
+
[seed9001] 40/60 (100s)
|
| 21 |
+
[seed9001] 50/60 (120s)
|
| 22 |
+
[seed9001] 60/60 (146s)
|
| 23 |
+
|
| 24 |
+
=== traj_gen_eval (v3e2400s1_raw_clip1): generated vs retrieval vs NMPC ===
|
| 25 |
+
|
| 26 |
+
--- plant set seed4242 (seed 4242, n=60) ---
|
| 27 |
+
arm succ 95% CI medJ med regret p90 synth
|
| 28 |
+
per-plant 93% [ 87, 98] 4.08 0.00% 0.00% 4.0s
|
| 29 |
+
nearest 62% [ 50, 73] 4.15 0.36% 1.51% 301ms
|
| 30 |
+
nearest+P 78% [ 67, 88] 4.09 0.69% 2.52% 583ms
|
| 31 |
+
nearest-margin 62% [ 50, 73] 4.15 0.36% 1.51% 305ms
|
| 32 |
+
NMPC 95% [ 88, 100] 4.08 0.14% 0.24% 23ms
|
| 33 |
+
generated 60.0%(sd4.1) [ 42, 68]* 4.09 0.65%(sd0.2) 2.27%(sd0.5) 300ms
|
| 34 |
+
generated+P 81.7%(sd2.4) [ 70, 90]* 4.09 0.77%(sd0.2) 2.82%(sd0.5) 686ms
|
| 35 |
+
(* generated rows: mean (sd) over sampler seeds [0, 1, 2]; CI shown is for seed 0 only)
|
| 36 |
+
(certified hand-off rate == success rate for every arm except NMPC (a certificate is a precondition for success here); NMPC's own hand-off is n/a, it never switches to a static gain)
|
| 37 |
+
|
| 38 |
+
--- failure breakdown, primary seed 0 ---
|
| 39 |
+
per-plant NMPC never reached Omega:3 replay failed:1
|
| 40 |
+
nearest Lyapunov decrease fails at a vertex:13 rollout failed:9 true dynamics: 0.02% of the boundary has V-dot >= 0, 0.00% saturates:1
|
| 41 |
+
nearest+P rollout failed:13
|
| 42 |
+
nearest-margin Lyapunov decrease fails at a vertex:13 rollout failed:9 true dynamics: 0.02% of the boundary has V-dot >= 0, 0.00% saturates:1
|
| 43 |
+
NMPC :3
|
| 44 |
+
generated Lyapunov decrease fails at a vertex:15 rollout failed:5 peak |Kx| on Omega is 10.02 V > 10.0 V:1 peak |Kx| on Omega is 10.04 V > 10.0 V:1 peak |Kx| on Omega is 10.16 V > 10.0 V:1 peak |Kx| on Omega is 10.30 V > 10.0 V:1 peak |Kx| on Omega is 10.12 V > 10.0 V:1 peak |Kx| on Omega is 10.03 V > 10.0 V:1 peak |Kx| on Omega is 10.28 V > 10.0 V:1
|
| 45 |
+
generated+P rollout failed:12
|
| 46 |
+
|
| 47 |
+
--- paired McNemar on success (exact binomial on discordant pairs) ---
|
| 48 |
+
per-plant vs nearest beats on 20, loses on 1 p=0.0000 * cost ratio 1.004x p=0.0000
|
| 49 |
+
per-plant vs nearest+P beats on 10, loses on 1 p=0.0117 * cost ratio 1.007x p=0.0000
|
| 50 |
+
per-plant vs nearest-margin beats on 20, loses on 1 p=0.0000 * cost ratio 1.004x p=0.0000
|
| 51 |
+
per-plant vs NMPC beats on 0, loses on 1 p=1.0000 cost ratio 1.001x p=0.0002
|
| 52 |
+
per-plant vs generated beats on 23, loses on 0 p=0.0000 * cost ratio 1.009x p=0.0000
|
| 53 |
+
per-plant vs generated+P beats on 8, loses on 0 p=0.0078 * cost ratio 1.011x p=0.0000
|
| 54 |
+
nearest+P vs per-plant beats on 1, loses on 10 p=0.0117 * cost ratio 0.993x p=0.0000
|
| 55 |
+
nearest+P vs nearest beats on 10, loses on 0 p=0.0020 * cost ratio 1.000x p=0.8291
|
| 56 |
+
nearest+P vs nearest-margin beats on 10, loses on 0 p=0.0020 * cost ratio 1.000x p=0.8291
|
| 57 |
+
nearest+P vs NMPC beats on 0, loses on 10 p=0.0020 * cost ratio 0.995x p=0.0000
|
| 58 |
+
nearest+P vs generated beats on 17, loses on 3 p=0.0026 * cost ratio 1.000x p=0.2988
|
| 59 |
+
nearest+P vs generated+P beats on 6, loses on 7 p=1.0000 cost ratio 1.003x p=0.0735
|
| 60 |
+
|
| 61 |
+
--- plant set seed9001 (seed 9001, n=60) ---
|
| 62 |
+
arm succ 95% CI medJ med regret p90 synth
|
| 63 |
+
per-plant 93% [ 87, 98] 4.16 0.00% 0.00% 7.8s
|
| 64 |
+
nearest 65% [ 52, 77] 4.10 0.71% 2.11% 265ms
|
| 65 |
+
nearest+P 87% [ 78, 95] 4.14 0.59% 2.34% 589ms
|
| 66 |
+
nearest-margin 65% [ 52, 77] 4.10 0.71% 2.11% 263ms
|
| 67 |
+
NMPC 97% [ 92, 100] 4.17 0.07% 0.22% 37ms
|
| 68 |
+
generated 63.3%(sd2.4) [ 48, 72]* 4.19 0.59%(sd0.1) 2.57%(sd0.3) 407ms
|
| 69 |
+
generated+P 90.6%(sd2.8) [ 87, 98]* 4.18 0.73%(sd0.0) 3.03%(sd0.3) 680ms
|
| 70 |
+
(* generated rows: mean (sd) over sampler seeds [0, 1, 2]; CI shown is for seed 0 only)
|
| 71 |
+
(certified hand-off rate == success rate for every arm except NMPC (a certificate is a precondition for success here); NMPC's own hand-off is n/a, it never switches to a static gain)
|
| 72 |
+
|
| 73 |
+
--- failure breakdown, primary seed 0 ---
|
| 74 |
+
per-plant replay failed:2 NMPC never reached Omega:2
|
| 75 |
+
nearest Lyapunov decrease fails at a vertex:16 rollout failed:5
|
| 76 |
+
nearest+P rollout failed:8
|
| 77 |
+
nearest-margin Lyapunov decrease fails at a vertex:16 rollout failed:5
|
| 78 |
+
NMPC :2
|
| 79 |
+
generated Lyapunov decrease fails at a vertex:15 peak |Kx| on Omega is 10.08 V > 10.0 V:1 true dynamics: 0.21% of the boundary has V-dot >= 0, 0.00% saturates:1 peak |Kx| on Omega is 10.01 V > 10.0 V:1 peak |Kx| on Omega is 10.31 V > 10.0 V:1 peak |Kx| on Omega is 10.17 V > 10.0 V:1 peak |Kx| on Omega is 10.03 V > 10.0 V:1 rollout failed:1 true dynamics: 0.26% of the boundary has V-dot >= 0, 0.00% saturates:1 true dynamics: 0.03% of the boundary has V-dot >= 0, 0.00% saturates:1
|
| 80 |
+
generated+P rollout failed:4
|
| 81 |
+
|
| 82 |
+
--- paired McNemar on success (exact binomial on discordant pairs) ---
|
| 83 |
+
per-plant vs nearest beats on 18, loses on 1 p=0.0001 * cost ratio 1.007x p=0.0000
|
| 84 |
+
per-plant vs nearest+P beats on 5, loses on 1 p=0.2188 cost ratio 1.006x p=0.0000
|
| 85 |
+
per-plant vs nearest-margin beats on 18, loses on 1 p=0.0001 * cost ratio 1.007x p=0.0000
|
| 86 |
+
per-plant vs NMPC beats on 0, loses on 2 p=0.5000 cost ratio 1.001x p=0.0064
|
| 87 |
+
per-plant vs generated beats on 20, loses on 0 p=0.0000 * cost ratio 1.006x p=0.0000
|
| 88 |
+
per-plant vs generated+P beats on 0, loses on 0 p=1.0000 cost ratio 1.007x p=0.0000
|
| 89 |
+
nearest+P vs per-plant beats on 1, loses on 5 p=0.2188 cost ratio 0.994x p=0.0000
|
| 90 |
+
nearest+P vs nearest beats on 13, loses on 0 p=0.0002 * cost ratio 1.000x p=0.1154
|
| 91 |
+
nearest+P vs nearest-margin beats on 13, loses on 0 p=0.0002 * cost ratio 1.000x p=0.1154
|
| 92 |
+
nearest+P vs NMPC beats on 0, loses on 6 p=0.0312 * cost ratio 0.995x p=0.0000
|
| 93 |
+
nearest+P vs generated beats on 18, loses on 2 p=0.0004 * cost ratio 1.002x p=0.3011
|
| 94 |
+
nearest+P vs generated+P beats on 1, loses on 5 p=0.2188 cost ratio 1.001x p=0.1153
|
results/furuta_v3/traj_gen_v2ctl2400_raw_clip5.txt
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
bank data/furuta_nmpc_bank3000.npz: 2801 entries
|
| 2 |
+
model models/Best_furuta_traj_cond_v2ctl_e2400.pth: cond_dim=4 out_dim=116
|
| 3 |
+
[seed4242] per-plant ceiling: 93%
|
| 4 |
+
[seed4242] nearest+P sequences done
|
| 5 |
+
[seed4242] generated+P seed 0 sequences done (36.5s sampling for 60x16)
|
| 6 |
+
[seed4242] generated+P seed 1 sequences done (37.0s sampling for 60x16)
|
| 7 |
+
[seed4242] generated+P seed 2 sequences done (36.6s sampling for 60x16)
|
| 8 |
+
[seed9001] per-plant ceiling: 93%
|
| 9 |
+
[seed9001] nearest+P sequences done
|
| 10 |
+
[seed9001] generated+P seed 0 sequences done (36.3s sampling for 60x16)
|
| 11 |
+
[seed9001] generated+P seed 1 sequences done (36.4s sampling for 60x16)
|
| 12 |
+
[seed9001] generated+P seed 2 sequences done (36.2s sampling for 60x16)
|
| 13 |
+
|
| 14 |
+
=== candidate-budget sweep: generated+P vs nearest+P vs per-plant ceiling ===
|
| 15 |
+
|
| 16 |
+
--- plant set seed4242 (seed 4242, n=60) ---
|
| 17 |
+
K nearest+P generated+P per-plant
|
| 18 |
+
1 51.7% t= 0.42s 55.6%(sd3.4) t= 0.95s 93.3% [McNemar nearest+P vs generated+P(seed0): 11-11, p=1.000]
|
| 19 |
+
4 78.3% t= 0.66s 81.7%(sd2.4) t= 1.16s 93.3% [McNemar nearest+P vs generated+P(seed0): 6-7, p=1.000]
|
| 20 |
+
8 88.3% t= 0.66s 86.7%(sd1.4) t= 1.16s 93.3% [McNemar nearest+P vs generated+P(seed0): 3-3, p=1.000]
|
| 21 |
+
16 93.3% t= 0.66s 88.3%(sd1.4) t= 1.16s 93.3% [McNemar nearest+P vs generated+P(seed0): 4-1, p=0.375]
|
| 22 |
+
crossover K (generated+P mean >= nearest+P): 1
|
| 23 |
+
|
| 24 |
+
--- plant set seed9001 (seed 9001, n=60) ---
|
| 25 |
+
K nearest+P generated+P per-plant
|
| 26 |
+
1 56.7% t= 0.50s 66.1%(sd1.6) t= 0.92s 93.3% [McNemar nearest+P vs generated+P(seed0): 13-18, p=0.473]
|
| 27 |
+
4 86.7% t= 0.66s 92.2%(sd0.8) t= 1.05s 93.3% [McNemar nearest+P vs generated+P(seed0): 1-4, p=0.375]
|
| 28 |
+
8 91.7% t= 0.66s 94.4%(sd0.8) t= 1.05s 93.3% [McNemar nearest+P vs generated+P(seed0): 0-2, p=0.500]
|
| 29 |
+
16 93.3% t= 0.66s 95.6%(sd0.8) t= 1.05s 93.3% [McNemar nearest+P vs generated+P(seed0): 0-1, p=1.000]
|
| 30 |
+
crossover K (generated+P mean >= nearest+P): 1
|
results/furuta_v3/traj_gen_v3e2400_raw_clip1.txt
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
bank data/furuta_nmpc_bank3000.npz: 2801 entries
|
| 2 |
+
model models/Best_furuta_traj_cond_v3_e2400.pth: cond_dim=4 out_dim=116
|
| 3 |
+
[seed4242] per-plant ceiling: 93%
|
| 4 |
+
[seed4242] nearest+P sequences done
|
| 5 |
+
[seed4242] generated+P seed 0 sequences done (34.3s sampling for 60x16)
|
| 6 |
+
[seed4242] generated+P seed 1 sequences done (34.6s sampling for 60x16)
|
| 7 |
+
[seed4242] generated+P seed 2 sequences done (34.5s sampling for 60x16)
|
| 8 |
+
[seed9001] per-plant ceiling: 93%
|
| 9 |
+
[seed9001] nearest+P sequences done
|
| 10 |
+
[seed9001] generated+P seed 0 sequences done (34.1s sampling for 60x16)
|
| 11 |
+
[seed9001] generated+P seed 1 sequences done (34.3s sampling for 60x16)
|
| 12 |
+
[seed9001] generated+P seed 2 sequences done (34.1s sampling for 60x16)
|
| 13 |
+
|
| 14 |
+
=== candidate-budget sweep: generated+P vs nearest+P vs per-plant ceiling ===
|
| 15 |
+
|
| 16 |
+
--- plant set seed4242 (seed 4242, n=60) ---
|
| 17 |
+
K nearest+P generated+P per-plant
|
| 18 |
+
1 51.7% t= 0.29s 50.0%(sd7.2) t= 0.87s 93.3% [McNemar nearest+P vs generated+P(seed0): 14-9, p=0.405]
|
| 19 |
+
4 78.3% t= 0.56s 82.8%(sd3.4) t= 1.13s 93.3% [McNemar nearest+P vs generated+P(seed0): 6-9, p=0.607]
|
| 20 |
+
8 88.3% t= 0.56s 89.4%(sd2.1) t= 1.13s 93.3% [McNemar nearest+P vs generated+P(seed0): 3-4, p=1.000]
|
| 21 |
+
16 93.3% t= 0.56s 91.7%(sd1.4) t= 1.13s 93.3% [McNemar nearest+P vs generated+P(seed0): 3-1, p=0.625]
|
| 22 |
+
crossover K (generated+P mean >= nearest+P): 4
|
| 23 |
+
|
| 24 |
+
--- plant set seed9001 (seed 9001, n=60) ---
|
| 25 |
+
K nearest+P generated+P per-plant
|
| 26 |
+
1 56.7% t= 0.30s 57.2%(sd7.5) t= 0.86s 93.3% [McNemar nearest+P vs generated+P(seed0): 16-11, p=0.442]
|
| 27 |
+
4 86.7% t= 0.45s 86.7%(sd1.4) t= 1.03s 93.3% [McNemar nearest+P vs generated+P(seed0): 5-4, p=1.000]
|
| 28 |
+
8 91.7% t= 0.45s 93.3%(sd1.4) t= 1.03s 93.3% [McNemar nearest+P vs generated+P(seed0): 2-2, p=1.000]
|
| 29 |
+
16 93.3% t= 0.45s 95.6%(sd0.8) t= 1.03s 93.3% [McNemar nearest+P vs generated+P(seed0): 1-2, p=1.000]
|
| 30 |
+
crossover K (generated+P mean >= nearest+P): 1
|
results/furuta_v3/traj_gen_warmstart_v2clip5_n200.txt
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== warm-start test (n=200, seed 4242, max_iter=700) ===
|
| 2 |
+
target certificate: largest_certified_region per plant (unmargined); verifier: furuta_traj_bank.score_solution (identical to every other arm in this doc)
|
| 3 |
+
|
| 4 |
+
condition success med iter med t med J
|
| 5 |
+
cold 58.0% 290 2.01s 4.06
|
| 6 |
+
nearest-warm 78.0% 48 0.36s 4.10
|
| 7 |
+
generated-warm 79.0% 50 0.37s 4.10
|
| 8 |
+
|
| 9 |
+
--- paired tests ---
|
| 10 |
+
generated-warm vs cold success: beats 44, loses 2, McNemar p=0.0000 * time ratio 0.210x (n=114), Wilcoxon p=0.0000 *
|
| 11 |
+
generated-warm vs nearest-warm success: beats 4, loses 2, McNemar p=0.6875 time ratio 0.993x (n=154), Wilcoxon p=0.5074
|
| 12 |
+
nearest-warm vs cold success: beats 41, loses 1, McNemar p=0.0000 * time ratio 0.196x (n=115), Wilcoxon p=0.0000 *
|
| 13 |
+
|
| 14 |
+
--- failure breakdown ---
|
| 15 |
+
cold ipopt: Maximum_Iterations_Exceeded:36 ipopt: Infeasible_Problem_Detected:31 rollout failed:17
|
| 16 |
+
nearest-warm rollout failed:26 ipopt: Maximum_Iterations_Exceeded:10 ipopt: Infeasible_Problem_Detected:8
|
| 17 |
+
generated-warm rollout failed:24 ipopt: Maximum_Iterations_Exceeded:9 ipopt: Infeasible_Problem_Detected:9
|
results/furuta_v3/traj_gen_warmstart_v2clip5_s1.txt
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== warm-start test (n=60, seed 4242, max_iter=700) ===
|
| 2 |
+
target certificate: largest_certified_region per plant (unmargined); verifier: furuta_traj_bank.score_solution (identical to every other arm in this doc)
|
| 3 |
+
|
| 4 |
+
condition success med iter med t med J
|
| 5 |
+
cold 53.3% 363 2.08s 4.06
|
| 6 |
+
nearest-warm 76.7% 45 0.28s 4.15
|
| 7 |
+
generated-warm 75.0% 46 0.27s 4.14
|
| 8 |
+
|
| 9 |
+
--- paired tests ---
|
| 10 |
+
generated-warm vs cold success: beats 13, loses 0, McNemar p=0.0002 * time ratio 0.147x (n=32), Wilcoxon p=0.0000 *
|
| 11 |
+
generated-warm vs nearest-warm success: beats 0, loses 1, McNemar p=1.0000 time ratio 1.056x (n=45), Wilcoxon p=0.6069
|
| 12 |
+
nearest-warm vs cold success: beats 14, loses 0, McNemar p=0.0001 * time ratio 0.159x (n=32), Wilcoxon p=0.0000 *
|
| 13 |
+
|
| 14 |
+
--- failure breakdown ---
|
| 15 |
+
cold ipopt: Maximum_Iterations_Exceeded:13 ipopt: Infeasible_Problem_Detected:11 rollout failed:4
|
| 16 |
+
nearest-warm ipopt: Infeasible_Problem_Detected:6 rollout failed:5 ipopt: Maximum_Iterations_Exceeded:3
|
| 17 |
+
generated-warm ipopt: Infeasible_Problem_Detected:7 ipopt: Maximum_Iterations_Exceeded:4 rollout failed:4
|
results/furuta_v3/traj_gen_warmstart_v2clip5_s2.txt
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== warm-start test (n=60, seed 4242, max_iter=700) ===
|
| 2 |
+
target certificate: largest_certified_region per plant (unmargined); verifier: furuta_traj_bank.score_solution (identical to every other arm in this doc)
|
| 3 |
+
|
| 4 |
+
condition success med iter med t med J
|
| 5 |
+
cold 53.3% 363 2.07s 4.06
|
| 6 |
+
nearest-warm 76.7% 45 0.28s 4.15
|
| 7 |
+
generated-warm 75.0% 47 0.27s 4.14
|
| 8 |
+
|
| 9 |
+
--- paired tests ---
|
| 10 |
+
generated-warm vs cold success: beats 13, loses 0, McNemar p=0.0002 * time ratio 0.166x (n=32), Wilcoxon p=0.0000 *
|
| 11 |
+
generated-warm vs nearest-warm success: beats 0, loses 1, McNemar p=1.0000 time ratio 1.023x (n=45), Wilcoxon p=0.9199
|
| 12 |
+
nearest-warm vs cold success: beats 14, loses 0, McNemar p=0.0001 * time ratio 0.160x (n=32), Wilcoxon p=0.0000 *
|
| 13 |
+
|
| 14 |
+
--- failure breakdown ---
|
| 15 |
+
cold ipopt: Maximum_Iterations_Exceeded:13 ipopt: Infeasible_Problem_Detected:11 rollout failed:4
|
| 16 |
+
nearest-warm ipopt: Infeasible_Problem_Detected:6 rollout failed:5 ipopt: Maximum_Iterations_Exceeded:3
|
| 17 |
+
generated-warm ipopt: Infeasible_Problem_Detected:7 ipopt: Maximum_Iterations_Exceeded:4 rollout failed:4
|
results/furuta_v3/traj_gen_warmstart_v3_n200.txt
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== warm-start test (n=200, seed 4242, max_iter=700) ===
|
| 2 |
+
target certificate: largest_certified_region per plant (unmargined); verifier: furuta_traj_bank.score_solution (identical to every other arm in this doc)
|
| 3 |
+
|
| 4 |
+
condition success med iter med t med J
|
| 5 |
+
cold 58.0% 290 2.08s 4.06
|
| 6 |
+
nearest-warm 78.0% 48 0.38s 4.10
|
| 7 |
+
generated-warm 79.5% 44 0.35s 4.10
|
| 8 |
+
|
| 9 |
+
--- paired tests ---
|
| 10 |
+
generated-warm vs cold success: beats 45, loses 2, McNemar p=0.0000 * time ratio 0.181x (n=114), Wilcoxon p=0.0000 *
|
| 11 |
+
generated-warm vs nearest-warm success: beats 4, loses 1, McNemar p=0.3750 time ratio 0.986x (n=155), Wilcoxon p=0.4041
|
| 12 |
+
nearest-warm vs cold success: beats 41, loses 1, McNemar p=0.0000 * time ratio 0.210x (n=115), Wilcoxon p=0.0000 *
|
| 13 |
+
|
| 14 |
+
--- failure breakdown ---
|
| 15 |
+
cold ipopt: Maximum_Iterations_Exceeded:36 ipopt: Infeasible_Problem_Detected:31 rollout failed:17
|
| 16 |
+
nearest-warm rollout failed:26 ipopt: Maximum_Iterations_Exceeded:10 ipopt: Infeasible_Problem_Detected:8
|
| 17 |
+
generated-warm rollout failed:21 ipopt: Maximum_Iterations_Exceeded:10 ipopt: Infeasible_Problem_Detected:10
|
results/furuta_v3/traj_gen_warmstart_v3_s2.txt
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== warm-start test (n=60, seed 4242, max_iter=700) ===
|
| 2 |
+
target certificate: largest_certified_region per plant (unmargined); verifier: furuta_traj_bank.score_solution (identical to every other arm in this doc)
|
| 3 |
+
|
| 4 |
+
condition success med iter med t med J
|
| 5 |
+
cold 53.3% 363 1.94s 4.06
|
| 6 |
+
nearest-warm 76.7% 45 0.30s 4.15
|
| 7 |
+
generated-warm 75.0% 46 0.28s 4.14
|
| 8 |
+
|
| 9 |
+
--- paired tests ---
|
| 10 |
+
generated-warm vs cold success: beats 13, loses 0, McNemar p=0.0002 * time ratio 0.158x (n=32), Wilcoxon p=0.0000 *
|
| 11 |
+
generated-warm vs nearest-warm success: beats 0, loses 1, McNemar p=1.0000 time ratio 0.982x (n=45), Wilcoxon p=0.3581
|
| 12 |
+
nearest-warm vs cold success: beats 14, loses 0, McNemar p=0.0001 * time ratio 0.160x (n=32), Wilcoxon p=0.0000 *
|
| 13 |
+
|
| 14 |
+
--- failure breakdown ---
|
| 15 |
+
cold ipopt: Maximum_Iterations_Exceeded:13 ipopt: Infeasible_Problem_Detected:11 rollout failed:4
|
| 16 |
+
nearest-warm ipopt: Infeasible_Problem_Detected:6 rollout failed:5 ipopt: Maximum_Iterations_Exceeded:3
|
| 17 |
+
generated-warm ipopt: Infeasible_Problem_Detected:7 rollout failed:5 ipopt: Maximum_Iterations_Exceeded:3
|
results/generalize/fig_generalize.png
ADDED
|
results/generalize/summary.txt
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
plants=120 controllers/plant=6 ICs/plant=4
|
| 2 |
+
per-sample stability rate (mean over plants): 0.64
|
| 3 |
+
plants with >=1 stabilizing controller: 0.81
|
| 4 |
+
regret of best controller vs per-plant oracle: median=+9.7% mean=+22.4% (n=97)
|
| 5 |
+
amortization: generation 6.5s for 720 controllers (9 ms each); per-plant oracle 24.1s each
|
results/metric_steering/mimo_summary.txt
ADDED
|
@@ -0,0 +1,12 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
MIMO 2x2 metric steering (held-out, N=40)
|
| 2 |
+
set ise iae control_ settling peak_con peak_y
|
| 3 |
+
prior 31.784 12.175 5.113 7.849 1.877 4.223
|
| 4 |
+
refine→control_effort 31.259 11.771 0.335* 7.543 0.922 4.277
|
| 5 |
+
refine→ise 28.752* 11.237 8.601 8.056 2.435 4.153
|
| 6 |
+
refine→settling 31.640 11.529 3.766 6.767* 2.080 4.324
|
| 7 |
+
refine→peak_control 31.223 11.754 0.298 7.547 0.853* 4.280
|
| 8 |
+
|
| 9 |
+
refine→control_effort: target -93.5% vs prior (5.113 → 0.335)
|
| 10 |
+
refine→ise : target -9.5% vs prior (31.784 → 28.752)
|
| 11 |
+
refine→settling : target -13.8% vs prior (7.849 → 6.767)
|
| 12 |
+
refine→peak_control : target -54.6% vs prior (1.877 → 0.853)
|
results/metric_steering/refine_summary.txt
ADDED
|
@@ -0,0 +1,12 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
reference plant G=1/(s+1) F=1 | refine 80 steps lr=0.05 | n=16 ICs=4
|
| 2 |
+
set ise iae itae control_ control_ peak_con peak_dev settling
|
| 3 |
+
prior 1.903 1.593 1.417 0.957 0.408 1.989 2.124 3.705
|
| 4 |
+
refine→control_effort 2.389 1.950 1.945 0.296* 0.051 1.214 2.147 4.259
|
| 5 |
+
refine→ise 0.841* 0.611 0.177 4.077 1.391 4.502 2.055 1.844
|
| 6 |
+
refine→settling 0.848 0.617 0.182 4.018 1.385 4.472 2.056 1.857*
|
| 7 |
+
refine→peak_control 2.381 1.943 1.935 0.403 0.057 1.205* 2.147 4.250
|
| 8 |
+
|
| 9 |
+
refine→control_effort: target -69.1% vs prior (0.957 → 0.296)
|
| 10 |
+
refine→ise : target -55.8% vs prior (1.903 → 0.841)
|
| 11 |
+
refine→settling : target -49.9% vs prior (3.705 → 1.857)
|
| 12 |
+
refine→peak_control : target -39.4% vs prior (1.989 → 1.205)
|
results/metric_steering/summary.txt
ADDED
|
@@ -0,0 +1,12 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
reference plant G=1/(s+1), F=1 | n=16 ICs=4 | steer λ=1.0
|
| 2 |
+
set ise iae itae control_ control_ peak_con peak_dev settling
|
| 3 |
+
unguided 1.906 1.593 1.417 0.992 0.410 1.932 2.082 3.553
|
| 4 |
+
steer→control_effort 1.904 1.592 1.414 0.979* 0.411 1.923 2.082 3.552
|
| 5 |
+
steer→ise 1.907* 1.593 1.418 1.007 0.409 1.943 2.082 3.553
|
| 6 |
+
steer→settling 1.907 1.594 1.419 0.998 0.408 1.936 2.082 3.556*
|
| 7 |
+
steer→peak_control 1.904 1.592 1.414 0.980 0.411 1.924* 2.082 3.552
|
| 8 |
+
|
| 9 |
+
steer→control_effort: target metric -1.4% vs unguided (0.992 → 0.979)
|
| 10 |
+
steer→ise : target metric +0.1% vs unguided (1.906 → 1.907)
|
| 11 |
+
steer→settling : target metric +0.1% vs unguided (3.553 → 3.556)
|
| 12 |
+
steer→peak_control : target metric -0.4% vs unguided (1.932 → 1.924)
|
results/mimo/k_sweep_mimo.txt
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== K-sweep MIMO (N=60, seed 9999, gen 65.4 ms/ctrl, per-sample stab 0.963) ===
|
| 2 |
+
K plants>=1 stable gen time/plant
|
| 3 |
+
1 0.950 0.07 s
|
| 4 |
+
2 0.983 0.13 s
|
| 5 |
+
4 1.000 0.26 s
|
| 6 |
+
8 1.000 0.52 s
|
| 7 |
+
16 1.000 1.05 s
|
| 8 |
+
32 1.000 2.09 s
|
| 9 |
+
|
| 10 |
+
contrast: SISO needed K~32 to reach 1.0 (mixed random-in-sector dataset, per-sample ~0.66);
|
| 11 |
+
MIMO trained on cost-optimal CERTIFIED targets -> saturates at ~1.0 almost immediately.
|
results/mimo/target_ablation.md
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
**Table (§4.3 ablation).** MIMO training-target ablation: random-in-sector vs cost-optimal, same held-out plants (N=120, K=8, seed 9999), same per-plant oracle & phi=0 baseline.
|
| 2 |
+
|
| 3 |
+
| training targets | per-sample stability | plants stabilizable | regret vs oracle (med / mean) | cost vs phi=0 (med) |
|
| 4 |
+
|---|---|---|---|---|
|
| 5 |
+
| random (§6m) | 0.86 | 0.93 | +19.2% / +26.8% | +16.3% |
|
| 6 |
+
| cost-optimal (§6n) | 0.96 | 1.00 | +2.0% / +3.8% | +0.2% |
|
results/pareto/fig_pareto.pdf
ADDED
|
Binary file (24.4 kB). View file
|
|
|
results/pareto/fig_pareto.png
ADDED
|
results/pareto/table_pareto.md
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
**Table 2.** Cost-steered controller vs analytic optimum per cost weight rho (plant G=1/(s+1), 9 ICs). Steering = refine a fixed model controller on the differentiable J_rho. Regret = (achieved J_rho − V_rho)/V_rho.
|
| 2 |
+
|
| 3 |
+
| rho | V_rho (optimal) | achieved J_rho | regret % | state ∫q | control ∫u² |
|
| 4 |
+
|---|---|---|---|---|---|
|
| 5 |
+
| 0.1 | 1.044 | 1.063 | +1.83 | 0.731 | 3.321 |
|
| 6 |
+
| 0.3 | 1.499 | 1.503 | +0.25 | 1.078 | 1.416 |
|
| 7 |
+
| 1 | 1.980 | 1.982 | +0.13 | 1.635 | 0.347 |
|
| 8 |
+
| 3 | 2.276 | 2.278 | +0.09 | 2.088 | 0.064 |
|
| 9 |
+
| 10 | 2.431 | 2.433 | +0.10 | 2.352 | 0.008 |
|
results/pareto_specs/run_s0.log
ADDED
|
@@ -0,0 +1,28 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
plant 0: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.498 GD(prior)=0.917 warm=510ms cold=3572ms/pt
|
| 2 |
+
plant 1: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.000 GD(prior)=9.519 warm=490ms cold=3358ms/pt
|
| 3 |
+
plant 2: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=2.306 GD(prior)=0.963 warm=474ms cold=3398ms/pt
|
| 4 |
+
plant 3: prior_cloud=16 ours_front=10 oracle_front=9 GD(ours)=0.878 GD(prior)=3.064 warm=434ms cold=3358ms/pt
|
| 5 |
+
plant 4: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=7.361 GD(prior)=1.604 warm=443ms cold=3365ms/pt
|
| 6 |
+
plant 5: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=3.066 GD(prior)=1.507 warm=452ms cold=3383ms/pt
|
| 7 |
+
plant 6: prior_cloud=16 ours_front=10 oracle_front=2 GD(ours)=0.047 GD(prior)=1.535 warm=450ms cold=3222ms/pt
|
| 8 |
+
plant 8: prior_cloud=16 ours_front=10 oracle_front=9 GD(ours)=0.039 GD(prior)=0.481 warm=440ms cold=3041ms/pt
|
| 9 |
+
plant 9: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.007 GD(prior)=8.233 warm=454ms cold=3148ms/pt
|
| 10 |
+
|
| 11 |
+
=== SPEC-PARETO SUMMARY ===
|
| 12 |
+
plants evaluated=9 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80
|
| 13 |
+
prior pass: 24.3s for 160 controllers (152 ms each)
|
| 14 |
+
|
| 15 |
+
Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer):
|
| 16 |
+
OURS (warm-steered front): median 0.498
|
| 17 |
+
PRIOR cloud front (no refine): median 1.535
|
| 18 |
+
|
| 19 |
+
Cost to trace ONE Pareto point (per plant):
|
| 20 |
+
OURS warm-start, 12 steps: 452 ms
|
| 21 |
+
CLASSICAL cold NLP, 80 steps: 3358 ms -> 7.4× slower per point
|
| 22 |
+
|
| 23 |
+
Certified (sector-valid) front points: median 100%
|
| 24 |
+
|
| 25 |
+
Binding-spec baseline (classical 'optimize only the most stringent req' = settling):
|
| 26 |
+
-> yields ONE corner controller, not the surface (see figure).
|
| 27 |
+
|
| 28 |
+
saved -> results/pareto_specs/pareto_specs_s0.npz
|
results/pareto_specs/run_s1.log
ADDED
|
@@ -0,0 +1,26 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
plant 0: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=2.325 GD(prior)=1.225 warm=537ms cold=3508ms/pt
|
| 2 |
+
plant 1: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=1.105 GD(prior)=0.985 warm=502ms cold=3496ms/pt
|
| 3 |
+
plant 2: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.050 GD(prior)=14.175 warm=491ms cold=3368ms/pt
|
| 4 |
+
plant 3: prior_cloud=15 ours_front=10 oracle_front=10 GD(ours)=0.015 GD(prior)=2.730 warm=482ms cold=3263ms/pt
|
| 5 |
+
plant 7: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.125 GD(prior)=2.600 warm=484ms cold=3530ms/pt
|
| 6 |
+
plant 8: prior_cloud=16 ours_front=10 oracle_front=6 GD(ours)=1.002 GD(prior)=1.098 warm=468ms cold=3416ms/pt
|
| 7 |
+
plant 9: prior_cloud=14 ours_front=10 oracle_front=7 GD(ours)=0.040 GD(prior)=1.691 warm=492ms cold=3489ms/pt
|
| 8 |
+
|
| 9 |
+
=== SPEC-PARETO SUMMARY ===
|
| 10 |
+
plants evaluated=7 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80
|
| 11 |
+
prior pass: 25.4s for 160 controllers (159 ms each)
|
| 12 |
+
|
| 13 |
+
Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer):
|
| 14 |
+
OURS (warm-steered front): median 0.125
|
| 15 |
+
PRIOR cloud front (no refine): median 1.691
|
| 16 |
+
|
| 17 |
+
Cost to trace ONE Pareto point (per plant):
|
| 18 |
+
OURS warm-start, 12 steps: 491 ms
|
| 19 |
+
CLASSICAL cold NLP, 80 steps: 3489 ms -> 7.1× slower per point
|
| 20 |
+
|
| 21 |
+
Certified (sector-valid) front points: median 100%
|
| 22 |
+
|
| 23 |
+
Binding-spec baseline (classical 'optimize only the most stringent req' = settling):
|
| 24 |
+
-> yields ONE corner controller, not the surface (see figure).
|
| 25 |
+
|
| 26 |
+
saved -> results/pareto_specs/pareto_specs_s1.npz
|
results/pareto_specs/run_s2.log
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
plant 0: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=7.064 GD(prior)=2.149 warm=546ms cold=3524ms/pt
|
| 2 |
+
plant 1: prior_cloud=16 ours_front=10 oracle_front=5 GD(ours)=0.140 GD(prior)=1.773 warm=443ms cold=3508ms/pt
|
| 3 |
+
plant 2: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.069 GD(prior)=8.234 warm=456ms cold=3467ms/pt
|
| 4 |
+
plant 3: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=17.344 GD(prior)=4.264 warm=586ms cold=3411ms/pt
|
| 5 |
+
plant 4: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.135 GD(prior)=0.822 warm=487ms cold=3433ms/pt
|
| 6 |
+
plant 5: prior_cloud=14 ours_front=10 oracle_front=2 GD(ours)=0.003 GD(prior)=3.755 warm=514ms cold=3571ms/pt
|
| 7 |
+
plant 6: prior_cloud=16 ours_front=10 oracle_front=9 GD(ours)=0.864 GD(prior)=1.401 warm=550ms cold=3300ms/pt
|
| 8 |
+
plant 7: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.546 GD(prior)=41.553 warm=440ms cold=3012ms/pt
|
| 9 |
+
plant 8: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=389.233 GD(prior)=141.003 warm=571ms cold=3206ms/pt
|
| 10 |
+
plant 9: prior_cloud=12 ours_front=6 oracle_front=5 GD(ours)=0.218 GD(prior)=0.292 warm=442ms cold=2951ms/pt
|
| 11 |
+
|
| 12 |
+
=== SPEC-PARETO SUMMARY ===
|
| 13 |
+
plants evaluated=10 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80
|
| 14 |
+
prior pass: 22.0s for 160 controllers (138 ms each)
|
| 15 |
+
|
| 16 |
+
Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer):
|
| 17 |
+
OURS (warm-steered front): median 0.382
|
| 18 |
+
PRIOR cloud front (no refine): median 2.952
|
| 19 |
+
|
| 20 |
+
Cost to trace ONE Pareto point (per plant):
|
| 21 |
+
OURS warm-start, 12 steps: 500 ms
|
| 22 |
+
CLASSICAL cold NLP, 80 steps: 3422 ms -> 6.8× slower per point
|
| 23 |
+
|
| 24 |
+
Certified (sector-valid) front points: median 100%
|
| 25 |
+
|
| 26 |
+
Binding-spec baseline (classical 'optimize only the most stringent req' = settling):
|
| 27 |
+
-> yields ONE corner controller, not the surface (see figure).
|
| 28 |
+
|
| 29 |
+
saved -> results/pareto_specs/pareto_specs_s2.npz
|
results/pareto_specs/run_s3.log
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
plant 0: prior_cloud=14 ours_front=10 oracle_front=10 GD(ours)=1.703 GD(prior)=1.502 warm=485ms cold=3322ms/pt
|
| 2 |
+
plant 1: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.724 GD(prior)=1.843 warm=473ms cold=3491ms/pt
|
| 3 |
+
plant 2: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.002 GD(prior)=20.795 warm=485ms cold=3509ms/pt
|
| 4 |
+
plant 3: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=5.185 GD(prior)=23.308 warm=459ms cold=3587ms/pt
|
| 5 |
+
plant 4: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=4.998 GD(prior)=1.749 warm=576ms cold=3684ms/pt
|
| 6 |
+
plant 5: prior_cloud=16 ours_front=10 oracle_front=2 GD(ours)=0.220 GD(prior)=1.960 warm=535ms cold=3218ms/pt
|
| 7 |
+
plant 6: prior_cloud=16 ours_front=10 oracle_front=1 GD(ours)=0.007 GD(prior)=125.069 warm=539ms cold=3641ms/pt
|
| 8 |
+
plant 7: prior_cloud=14 ours_front=10 oracle_front=10 GD(ours)=0.017 GD(prior)=27.143 warm=475ms cold=2982ms/pt
|
| 9 |
+
plant 8: prior_cloud=16 ours_front=10 oracle_front=2 GD(ours)=0.037 GD(prior)=1.681 warm=487ms cold=3164ms/pt
|
| 10 |
+
plant 9: prior_cloud=16 ours_front=10 oracle_front=9 GD(ours)=0.121 GD(prior)=2.394 warm=432ms cold=2892ms/pt
|
| 11 |
+
|
| 12 |
+
=== SPEC-PARETO SUMMARY ===
|
| 13 |
+
plants evaluated=10 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80
|
| 14 |
+
prior pass: 24.5s for 160 controllers (153 ms each)
|
| 15 |
+
|
| 16 |
+
Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer):
|
| 17 |
+
OURS (warm-steered front): median 0.171
|
| 18 |
+
PRIOR cloud front (no refine): median 2.177
|
| 19 |
+
|
| 20 |
+
Cost to trace ONE Pareto point (per plant):
|
| 21 |
+
OURS warm-start, 12 steps: 485 ms
|
| 22 |
+
CLASSICAL cold NLP, 80 steps: 3407 ms -> 7.0× slower per point
|
| 23 |
+
|
| 24 |
+
Certified (sector-valid) front points: median 100%
|
| 25 |
+
|
| 26 |
+
Binding-spec baseline (classical 'optimize only the most stringent req' = settling):
|
| 27 |
+
-> yields ONE corner controller, not the surface (see figure).
|
| 28 |
+
|
| 29 |
+
saved -> results/pareto_specs/pareto_specs_s3.npz
|
results/pareto_specs/run_s4.log
ADDED
|
@@ -0,0 +1,28 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
plant 0: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.038 GD(prior)=530.974 warm=481ms cold=3474ms/pt
|
| 2 |
+
plant 2: prior_cloud=16 ours_front=10 oracle_front=2 GD(ours)=0.093 GD(prior)=2.589 warm=514ms cold=3507ms/pt
|
| 3 |
+
plant 3: prior_cloud=16 ours_front=10 oracle_front=2 GD(ours)=0.525 GD(prior)=5.005 warm=511ms cold=3623ms/pt
|
| 4 |
+
plant 4: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=9.910 GD(prior)=2.355 warm=485ms cold=3417ms/pt
|
| 5 |
+
plant 5: prior_cloud=16 ours_front=10 oracle_front=6 GD(ours)=8.178 GD(prior)=2.567 warm=433ms cold=3733ms/pt
|
| 6 |
+
plant 6: prior_cloud=10 ours_front=10 oracle_front=6 GD(ours)=1.339 GD(prior)=1.180 warm=449ms cold=3419ms/pt
|
| 7 |
+
plant 7: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=3.290 GD(prior)=1.603 warm=478ms cold=3313ms/pt
|
| 8 |
+
plant 8: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=15.436 GD(prior)=5.013 warm=462ms cold=3069ms/pt
|
| 9 |
+
plant 9: prior_cloud=15 ours_front=10 oracle_front=1 GD(ours)=0.085 GD(prior)=159.701 warm=463ms cold=3094ms/pt
|
| 10 |
+
|
| 11 |
+
=== SPEC-PARETO SUMMARY ===
|
| 12 |
+
plants evaluated=9 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80
|
| 13 |
+
prior pass: 21.0s for 160 controllers (131 ms each)
|
| 14 |
+
|
| 15 |
+
Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer):
|
| 16 |
+
OURS (warm-steered front): median 1.339
|
| 17 |
+
PRIOR cloud front (no refine): median 2.589
|
| 18 |
+
|
| 19 |
+
Cost to trace ONE Pareto point (per plant):
|
| 20 |
+
OURS warm-start, 12 steps: 478 ms
|
| 21 |
+
CLASSICAL cold NLP, 80 steps: 3419 ms -> 7.2× slower per point
|
| 22 |
+
|
| 23 |
+
Certified (sector-valid) front points: median 100%
|
| 24 |
+
|
| 25 |
+
Binding-spec baseline (classical 'optimize only the most stringent req' = settling):
|
| 26 |
+
-> yields ONE corner controller, not the surface (see figure).
|
| 27 |
+
|
| 28 |
+
saved -> results/pareto_specs/pareto_specs_s4.npz
|
results/pareto_specs/significance.txt
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
pooled plants N = 45 (over 5 seeds)
|
| 2 |
+
|
| 3 |
+
CLAIM 1 — steering helps (ours_gd < prior_gd, paired):
|
| 4 |
+
win rate: 30/45 = 67% sign-test p = 1.78e-02
|
| 5 |
+
Wilcoxon signed-rank (ours<prior): p = 1.42e-02 (log-space p = 4.04e-05)
|
| 6 |
+
median GD ours = 0.498 (95% CI 0.121–1.002)
|
| 7 |
+
median GD prior = 2.355 (95% CI 1.691–3.064)
|
| 8 |
+
|
| 9 |
+
CLAIM 2 — lands on the oracle front (ours_gd < 0.5):
|
| 10 |
+
23/45 = 51% 95% CI [36%, 66%]
|
| 11 |
+
|
| 12 |
+
per-plant speedup (cold/warm): median 6.9× (95% CI 6.8–7.1)
|
| 13 |
+
certified front points: median 100%
|
| 14 |
+
|
| 15 |
+
VERDICT: significant — steering improves GD (p<0.05 both tests) and reaches the front on a clear majority of plants.
|
results/pareto_specs/summary.txt
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
plants evaluated=11 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80
|
| 2 |
+
prior pass: 20.4s for 192 controllers (106 ms each)
|
| 3 |
+
|
| 4 |
+
Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer):
|
| 5 |
+
OURS (warm-steered front): median 0.145
|
| 6 |
+
PRIOR cloud front (no refine): median 2.927
|
| 7 |
+
|
| 8 |
+
Cost to trace ONE Pareto point (per plant):
|
| 9 |
+
OURS warm-start, 12 steps: 873 ms
|
| 10 |
+
CLASSICAL cold NLP, 80 steps: 5833 ms -> 6.7× slower per point
|
| 11 |
+
|
| 12 |
+
Certified (sector-valid) front points: median 100%
|
| 13 |
+
|
| 14 |
+
Binding-spec baseline (classical 'optimize only the most stringent req' = settling):
|
| 15 |
+
-> yields ONE corner controller, not the surface (see figure).
|
results/pareto_specs/summary_s0.txt
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
plants evaluated=9 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80
|
| 2 |
+
prior pass: 24.3s for 160 controllers (152 ms each)
|
| 3 |
+
|
| 4 |
+
Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer):
|
| 5 |
+
OURS (warm-steered front): median 0.498
|
| 6 |
+
PRIOR cloud front (no refine): median 1.535
|
| 7 |
+
|
| 8 |
+
Cost to trace ONE Pareto point (per plant):
|
| 9 |
+
OURS warm-start, 12 steps: 452 ms
|
| 10 |
+
CLASSICAL cold NLP, 80 steps: 3358 ms -> 7.4× slower per point
|
| 11 |
+
|
| 12 |
+
Certified (sector-valid) front points: median 100%
|
| 13 |
+
|
| 14 |
+
Binding-spec baseline (classical 'optimize only the most stringent req' = settling):
|
| 15 |
+
-> yields ONE corner controller, not the surface (see figure).
|
results/pareto_specs/summary_s1.txt
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
plants evaluated=7 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80
|
| 2 |
+
prior pass: 25.4s for 160 controllers (159 ms each)
|
| 3 |
+
|
| 4 |
+
Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer):
|
| 5 |
+
OURS (warm-steered front): median 0.125
|
| 6 |
+
PRIOR cloud front (no refine): median 1.691
|
| 7 |
+
|
| 8 |
+
Cost to trace ONE Pareto point (per plant):
|
| 9 |
+
OURS warm-start, 12 steps: 491 ms
|
| 10 |
+
CLASSICAL cold NLP, 80 steps: 3489 ms -> 7.1× slower per point
|
| 11 |
+
|
| 12 |
+
Certified (sector-valid) front points: median 100%
|
| 13 |
+
|
| 14 |
+
Binding-spec baseline (classical 'optimize only the most stringent req' = settling):
|
| 15 |
+
-> yields ONE corner controller, not the surface (see figure).
|
results/pareto_specs/summary_s2.txt
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
plants evaluated=10 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80
|
| 2 |
+
prior pass: 22.0s for 160 controllers (138 ms each)
|
| 3 |
+
|
| 4 |
+
Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer):
|
| 5 |
+
OURS (warm-steered front): median 0.382
|
| 6 |
+
PRIOR cloud front (no refine): median 2.952
|
| 7 |
+
|
| 8 |
+
Cost to trace ONE Pareto point (per plant):
|
| 9 |
+
OURS warm-start, 12 steps: 500 ms
|
| 10 |
+
CLASSICAL cold NLP, 80 steps: 3422 ms -> 6.8× slower per point
|
| 11 |
+
|
| 12 |
+
Certified (sector-valid) front points: median 100%
|
| 13 |
+
|
| 14 |
+
Binding-spec baseline (classical 'optimize only the most stringent req' = settling):
|
| 15 |
+
-> yields ONE corner controller, not the surface (see figure).
|
results/pareto_specs/summary_s3.txt
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
plants evaluated=10 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80
|
| 2 |
+
prior pass: 24.5s for 160 controllers (153 ms each)
|
| 3 |
+
|
| 4 |
+
Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer):
|
| 5 |
+
OURS (warm-steered front): median 0.171
|
| 6 |
+
PRIOR cloud front (no refine): median 2.177
|
| 7 |
+
|
| 8 |
+
Cost to trace ONE Pareto point (per plant):
|
| 9 |
+
OURS warm-start, 12 steps: 485 ms
|
| 10 |
+
CLASSICAL cold NLP, 80 steps: 3407 ms -> 7.0× slower per point
|
| 11 |
+
|
| 12 |
+
Certified (sector-valid) front points: median 100%
|
| 13 |
+
|
| 14 |
+
Binding-spec baseline (classical 'optimize only the most stringent req' = settling):
|
| 15 |
+
-> yields ONE corner controller, not the surface (see figure).
|
results/pareto_specs/summary_s4.txt
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
plants evaluated=9 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80
|
| 2 |
+
prior pass: 21.0s for 160 controllers (131 ms each)
|
| 3 |
+
|
| 4 |
+
Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer):
|
| 5 |
+
OURS (warm-steered front): median 1.339
|
| 6 |
+
PRIOR cloud front (no refine): median 2.589
|
| 7 |
+
|
| 8 |
+
Cost to trace ONE Pareto point (per plant):
|
| 9 |
+
OURS warm-start, 12 steps: 478 ms
|
| 10 |
+
CLASSICAL cold NLP, 80 steps: 3419 ms -> 7.2× slower per point
|
| 11 |
+
|
| 12 |
+
Certified (sector-valid) front points: median 100%
|
| 13 |
+
|
| 14 |
+
Binding-spec baseline (classical 'optimize only the most stringent req' = settling):
|
| 15 |
+
-> yields ONE corner controller, not the surface (see figure).
|
results/physics_gen/pendulum_physics_gen.txt
ADDED
|
@@ -0,0 +1,720 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== lvl0 K=8 oracle=old (median oracle J 1.352) ===
|
| 2 |
+
arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired
|
| 3 |
+
base_s0 1.000 1.000 1.000 1.09 | 1.000 1.000 1.09 16.0 0.00
|
| 4 |
+
base_s1 1.000 1.000 1.000 1.15 | 1.000 1.000 1.15 16.0 0.00
|
| 5 |
+
base_s2 1.000 1.000 1.000 1.07 | 1.000 1.000 1.07 16.0 0.00
|
| 6 |
+
lz_s42_g0 1.000 1.000 1.000 1.10 | 1.000 1.000 1.10 16.0 0.00
|
| 7 |
+
lz_s42_g1 1.000 1.000 1.000 1.05 | 1.000 1.000 1.05 16.0 0.00
|
| 8 |
+
lz_s42_g2 1.000 1.000 1.000 1.06 | 1.000 1.000 1.06 16.0 0.00
|
| 9 |
+
lz_s43_g0 1.000 1.000 1.000 1.18 | 1.000 1.000 1.18 16.0 0.00
|
| 10 |
+
lz_s44_g0 1.000 1.000 1.000 1.04 | 1.000 1.000 1.04 16.0 0.00
|
| 11 |
+
lzm_s42_g0 1.000 1.000 1.000 1.24 | 1.000 1.000 1.24 16.0 0.00
|
| 12 |
+
lzm_s43_g0 1.000 1.000 1.000 1.31 | 1.000 1.000 1.31 16.0 0.00
|
| 13 |
+
lzm_s44_g0 1.000 1.000 1.000 1.03 | 1.000 1.000 1.03 16.0 0.00
|
| 14 |
+
phys_s42_g0 1.000 1.000 1.000 1.28 | 1.000 1.000 1.28 16.0 0.00
|
| 15 |
+
phys_s43_g0 1.000 1.000 1.000 1.22 | 1.000 1.000 1.22 16.0 0.00
|
| 16 |
+
phys_s44_g0 1.000 1.000 1.000 1.06 | 1.000 1.000 1.06 16.0 0.00
|
| 17 |
+
nn 1.000 1.000 1.000 0.29 | 1.000 1.000 0.29 16.0 0.00
|
| 18 |
+
idw 1.000 1.000 1.000 0.29 | 1.000 1.000 0.29 16.0 0.00
|
| 19 |
+
nns 1.000 1.000 1.000 1.11 | 1.000 1.000 1.11 16.0 0.00
|
| 20 |
+
dls 1.000 1.000 1.000 1.14 | 1.000 1.000 1.14 16.0 0.00
|
| 21 |
+
lzres_s42_g0 1.000 1.000 1.000 1.18 | 1.000 1.000 1.18 16.0 0.00
|
| 22 |
+
lzres_s43_g0 1.000 1.000 1.000 1.18 | 1.000 1.000 1.18 16.0 0.00
|
| 23 |
+
lzres_s44_g0 1.000 1.000 1.000 1.19 | 1.000 1.000 1.19 16.0 0.00
|
| 24 |
+
|
| 25 |
+
-- paired Wilcoxon on regret, model minus best retrieval (B: best=nn; noB: best=nn); negative = model better; [95% CI of paired median]
|
| 26 |
+
base base_s0 B vs nn : +0.68 [+0.46,+0.84] p=2.21e-10 certified 1.000 vs 1.000
|
| 27 |
+
base base_s0 noB vs nn : +0.68 [+0.46,+0.84] p=2.21e-10 certified 1.000 vs 1.000
|
| 28 |
+
base base_s1 B vs nn : +0.76 [+0.56,+0.96] p=1.74e-10 certified 1.000 vs 1.000
|
| 29 |
+
base base_s1 noB vs nn : +0.76 [+0.56,+0.96] p=1.74e-10 certified 1.000 vs 1.000
|
| 30 |
+
base base_s2 B vs nn : +0.68 [+0.57,+0.80] p=1.58e-10 certified 1.000 vs 1.000
|
| 31 |
+
base base_s2 noB vs nn : +0.68 [+0.57,+0.80] p=1.58e-10 certified 1.000 vs 1.000
|
| 32 |
+
A_lz lz_s42_g0 B vs nn : +0.69 [+0.56,+0.89] p=1.9e-11 certified 1.000 vs 1.000
|
| 33 |
+
A_lz lz_s42_g0 noB vs nn : +0.69 [+0.56,+0.89] p=1.9e-11 certified 1.000 vs 1.000
|
| 34 |
+
A_lz lz_s42_g1 B vs nn : +0.67 [+0.60,+0.77] p=3.29e-11 certified 1.000 vs 1.000
|
| 35 |
+
A_lz lz_s42_g1 noB vs nn : +0.67 [+0.60,+0.77] p=3.29e-11 certified 1.000 vs 1.000
|
| 36 |
+
A_lz lz_s42_g2 B vs nn : +0.72 [+0.63,+0.83] p=6.59e-11 certified 1.000 vs 1.000
|
| 37 |
+
A_lz lz_s42_g2 noB vs nn : +0.72 [+0.63,+0.83] p=6.59e-11 certified 1.000 vs 1.000
|
| 38 |
+
A_lz lz_s43_g0 B vs nn : +0.76 [+0.65,+0.95] p=1.71e-11 certified 1.000 vs 1.000
|
| 39 |
+
A_lz lz_s43_g0 noB vs nn : +0.76 [+0.65,+0.95] p=1.71e-11 certified 1.000 vs 1.000
|
| 40 |
+
A_lz lz_s44_g0 B vs nn : +0.62 [+0.49,+0.78] p=2.84e-11 certified 1.000 vs 1.000
|
| 41 |
+
A_lz lz_s44_g0 noB vs nn : +0.62 [+0.49,+0.78] p=2.84e-11 certified 1.000 vs 1.000
|
| 42 |
+
A_lzm lzm_s42_g0 B vs nn : +0.82 [+0.73,+0.96] p=1.63e-11 certified 1.000 vs 1.000
|
| 43 |
+
A_lzm lzm_s42_g0 noB vs nn : +0.82 [+0.73,+0.96] p=1.63e-11 certified 1.000 vs 1.000
|
| 44 |
+
A_lzm lzm_s43_g0 B vs nn : +0.88 [+0.78,+1.00] p=1.63e-11 certified 1.000 vs 1.000
|
| 45 |
+
A_lzm lzm_s43_g0 noB vs nn : +0.88 [+0.78,+1.00] p=1.63e-11 certified 1.000 vs 1.000
|
| 46 |
+
A_lzm lzm_s44_g0 B vs nn : +0.59 [+0.46,+0.71] p=1.63e-11 certified 1.000 vs 1.000
|
| 47 |
+
A_lzm lzm_s44_g0 noB vs nn : +0.59 [+0.46,+0.71] p=1.63e-11 certified 1.000 vs 1.000
|
| 48 |
+
A_lzres lzres_s42_g0 B vs nn : +0.92 [+0.72,+0.98] p=1.9e-11 certified 1.000 vs 1.000
|
| 49 |
+
A_lzres lzres_s42_g0 noB vs nn : +0.92 [+0.72,+0.98] p=1.9e-11 certified 1.000 vs 1.000
|
| 50 |
+
A_lzres lzres_s43_g0 B vs nn : +0.92 [+0.76,+0.97] p=1.9e-11 certified 1.000 vs 1.000
|
| 51 |
+
A_lzres lzres_s43_g0 noB vs nn : +0.92 [+0.76,+0.97] p=1.9e-11 certified 1.000 vs 1.000
|
| 52 |
+
A_lzres lzres_s44_g0 B vs nn : +0.95 [+0.74,+1.03] p=1.71e-11 certified 1.000 vs 1.000
|
| 53 |
+
A_lzres lzres_s44_g0 noB vs nn : +0.95 [+0.74,+1.03] p=1.71e-11 certified 1.000 vs 1.000
|
| 54 |
+
A_phys phys_s42_g0 B vs nn : +0.93 [+0.83,+1.05] p=1.63e-11 certified 1.000 vs 1.000
|
| 55 |
+
A_phys phys_s42_g0 noB vs nn : +0.93 [+0.83,+1.05] p=1.63e-11 certified 1.000 vs 1.000
|
| 56 |
+
A_phys phys_s43_g0 B vs nn : +0.82 [+0.68,+0.95] p=1.63e-11 certified 1.000 vs 1.000
|
| 57 |
+
A_phys phys_s43_g0 noB vs nn : +0.82 [+0.68,+0.95] p=1.63e-11 certified 1.000 vs 1.000
|
| 58 |
+
A_phys phys_s44_g0 B vs nn : +0.66 [+0.51,+0.76] p=1.63e-11 certified 1.000 vs 1.000
|
| 59 |
+
A_phys phys_s44_g0 noB vs nn : +0.66 [+0.51,+0.76] p=1.63e-11 certified 1.000 vs 1.000
|
| 60 |
+
|
| 61 |
+
=== lvl0 K=8 oracle=strong2 (median oracle J 1.351) ===
|
| 62 |
+
arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired
|
| 63 |
+
base_s0 1.000 1.000 1.000 1.11 | 1.000 1.000 1.11 16.0 0.00
|
| 64 |
+
base_s1 1.000 1.000 1.000 1.17 | 1.000 1.000 1.17 16.0 0.00
|
| 65 |
+
base_s2 1.000 1.000 1.000 1.12 | 1.000 1.000 1.12 16.0 0.00
|
| 66 |
+
lz_s42_g0 1.000 1.000 1.000 1.15 | 1.000 1.000 1.15 16.0 0.00
|
| 67 |
+
lz_s42_g1 1.000 1.000 1.000 1.14 | 1.000 1.000 1.14 16.0 0.00
|
| 68 |
+
lz_s42_g2 1.000 1.000 1.000 1.10 | 1.000 1.000 1.10 16.0 0.00
|
| 69 |
+
lz_s43_g0 1.000 1.000 1.000 1.23 | 1.000 1.000 1.23 16.0 0.00
|
| 70 |
+
lz_s44_g0 1.000 1.000 1.000 1.12 | 1.000 1.000 1.12 16.0 0.00
|
| 71 |
+
lzm_s42_g0 1.000 1.000 1.000 1.32 | 1.000 1.000 1.32 16.0 0.00
|
| 72 |
+
lzm_s43_g0 1.000 1.000 1.000 1.38 | 1.000 1.000 1.38 16.0 0.00
|
| 73 |
+
lzm_s44_g0 1.000 1.000 1.000 1.11 | 1.000 1.000 1.11 16.0 0.00
|
| 74 |
+
phys_s42_g0 1.000 1.000 1.000 1.37 | 1.000 1.000 1.37 16.0 0.00
|
| 75 |
+
phys_s43_g0 1.000 1.000 1.000 1.30 | 1.000 1.000 1.30 16.0 0.00
|
| 76 |
+
phys_s44_g0 1.000 1.000 1.000 1.15 | 1.000 1.000 1.15 16.0 0.00
|
| 77 |
+
nn 1.000 1.000 1.000 0.48 | 1.000 1.000 0.48 16.0 0.00
|
| 78 |
+
idw 1.000 1.000 1.000 0.48 | 1.000 1.000 0.48 16.0 0.00
|
| 79 |
+
nns 1.000 1.000 1.000 1.12 | 1.000 1.000 1.12 16.0 0.00
|
| 80 |
+
dls 1.000 1.000 1.000 1.23 | 1.000 1.000 1.23 16.0 0.00
|
| 81 |
+
lzres_s42_g0 1.000 1.000 1.000 1.29 | 1.000 1.000 1.29 16.0 0.00
|
| 82 |
+
lzres_s43_g0 1.000 1.000 1.000 1.28 | 1.000 1.000 1.28 16.0 0.00
|
| 83 |
+
lzres_s44_g0 1.000 1.000 1.000 1.30 | 1.000 1.000 1.30 16.0 0.00
|
| 84 |
+
|
| 85 |
+
-- paired Wilcoxon on regret, model minus best retrieval (B: best=nn; noB: best=nn); negative = model better; [95% CI of paired median]
|
| 86 |
+
base base_s0 B vs nn : +0.68 [+0.46,+0.84] p=2.21e-10 certified 1.000 vs 1.000
|
| 87 |
+
base base_s0 noB vs nn : +0.68 [+0.46,+0.84] p=2.21e-10 certified 1.000 vs 1.000
|
| 88 |
+
base base_s1 B vs nn : +0.76 [+0.56,+0.96] p=1.74e-10 certified 1.000 vs 1.000
|
| 89 |
+
base base_s1 noB vs nn : +0.76 [+0.56,+0.96] p=1.74e-10 certified 1.000 vs 1.000
|
| 90 |
+
base base_s2 B vs nn : +0.68 [+0.57,+0.80] p=1.58e-10 certified 1.000 vs 1.000
|
| 91 |
+
base base_s2 noB vs nn : +0.68 [+0.57,+0.80] p=1.58e-10 certified 1.000 vs 1.000
|
| 92 |
+
A_lz lz_s42_g0 B vs nn : +0.69 [+0.56,+0.89] p=1.9e-11 certified 1.000 vs 1.000
|
| 93 |
+
A_lz lz_s42_g0 noB vs nn : +0.69 [+0.56,+0.89] p=1.9e-11 certified 1.000 vs 1.000
|
| 94 |
+
A_lz lz_s42_g1 B vs nn : +0.68 [+0.60,+0.77] p=3.29e-11 certified 1.000 vs 1.000
|
| 95 |
+
A_lz lz_s42_g1 noB vs nn : +0.68 [+0.60,+0.77] p=3.29e-11 certified 1.000 vs 1.000
|
| 96 |
+
A_lz lz_s42_g2 B vs nn : +0.72 [+0.63,+0.83] p=6.92e-11 certified 1.000 vs 1.000
|
| 97 |
+
A_lz lz_s42_g2 noB vs nn : +0.72 [+0.63,+0.83] p=6.92e-11 certified 1.000 vs 1.000
|
| 98 |
+
A_lz lz_s43_g0 B vs nn : +0.76 [+0.65,+0.95] p=1.71e-11 certified 1.000 vs 1.000
|
| 99 |
+
A_lz lz_s43_g0 noB vs nn : +0.76 [+0.65,+0.95] p=1.71e-11 certified 1.000 vs 1.000
|
| 100 |
+
A_lz lz_s44_g0 B vs nn : +0.62 [+0.49,+0.78] p=2.84e-11 certified 1.000 vs 1.000
|
| 101 |
+
A_lz lz_s44_g0 noB vs nn : +0.62 [+0.49,+0.78] p=2.84e-11 certified 1.000 vs 1.000
|
| 102 |
+
A_lzm lzm_s42_g0 B vs nn : +0.82 [+0.73,+0.96] p=1.63e-11 certified 1.000 vs 1.000
|
| 103 |
+
A_lzm lzm_s42_g0 noB vs nn : +0.82 [+0.73,+0.96] p=1.63e-11 certified 1.000 vs 1.000
|
| 104 |
+
A_lzm lzm_s43_g0 B vs nn : +0.88 [+0.78,+1.00] p=1.63e-11 certified 1.000 vs 1.000
|
| 105 |
+
A_lzm lzm_s43_g0 noB vs nn : +0.88 [+0.78,+1.00] p=1.63e-11 certified 1.000 vs 1.000
|
| 106 |
+
A_lzm lzm_s44_g0 B vs nn : +0.59 [+0.46,+0.71] p=1.63e-11 certified 1.000 vs 1.000
|
| 107 |
+
A_lzm lzm_s44_g0 noB vs nn : +0.59 [+0.46,+0.71] p=1.63e-11 certified 1.000 vs 1.000
|
| 108 |
+
A_lzres lzres_s42_g0 B vs nn : +0.92 [+0.72,+0.98] p=1.9e-11 certified 1.000 vs 1.000
|
| 109 |
+
A_lzres lzres_s42_g0 noB vs nn : +0.92 [+0.72,+0.98] p=1.9e-11 certified 1.000 vs 1.000
|
| 110 |
+
A_lzres lzres_s43_g0 B vs nn : +0.92 [+0.78,+0.97] p=1.9e-11 certified 1.000 vs 1.000
|
| 111 |
+
A_lzres lzres_s43_g0 noB vs nn : +0.92 [+0.78,+0.97] p=1.9e-11 certified 1.000 vs 1.000
|
| 112 |
+
A_lzres lzres_s44_g0 B vs nn : +0.95 [+0.74,+1.03] p=1.71e-11 certified 1.000 vs 1.000
|
| 113 |
+
A_lzres lzres_s44_g0 noB vs nn : +0.95 [+0.74,+1.03] p=1.71e-11 certified 1.000 vs 1.000
|
| 114 |
+
A_phys phys_s42_g0 B vs nn : +0.93 [+0.83,+1.05] p=1.63e-11 certified 1.000 vs 1.000
|
| 115 |
+
A_phys phys_s42_g0 noB vs nn : +0.93 [+0.83,+1.05] p=1.63e-11 certified 1.000 vs 1.000
|
| 116 |
+
A_phys phys_s43_g0 B vs nn : +0.82 [+0.68,+0.95] p=1.63e-11 certified 1.000 vs 1.000
|
| 117 |
+
A_phys phys_s43_g0 noB vs nn : +0.82 [+0.68,+0.95] p=1.63e-11 certified 1.000 vs 1.000
|
| 118 |
+
A_phys phys_s44_g0 B vs nn : +0.66 [+0.51,+0.76] p=1.63e-11 certified 1.000 vs 1.000
|
| 119 |
+
A_phys phys_s44_g0 noB vs nn : +0.66 [+0.51,+0.76] p=1.63e-11 certified 1.000 vs 1.000
|
| 120 |
+
|
| 121 |
+
=== lvl1.1 K=8 oracle=old (median oracle J 0.984) ===
|
| 122 |
+
arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired
|
| 123 |
+
base_s0 1.000 1.000 1.000 0.92 | 1.000 1.000 0.92 16.0 0.00
|
| 124 |
+
base_s1 1.000 1.000 1.000 0.92 | 1.000 1.000 0.92 16.0 0.00
|
| 125 |
+
base_s2 1.000 1.000 1.000 0.84 | 1.000 1.000 0.84 16.0 0.00
|
| 126 |
+
lz_s42_g0 1.000 1.000 1.000 1.04 | 1.000 1.000 1.04 16.0 0.00
|
| 127 |
+
lz_s42_g1 1.000 1.000 1.000 1.04 | 1.000 1.000 1.04 16.0 0.00
|
| 128 |
+
lz_s42_g2 1.000 1.000 1.000 1.03 | 1.000 1.000 1.03 16.0 0.00
|
| 129 |
+
lz_s43_g0 1.000 1.000 1.000 1.11 | 1.000 1.000 1.11 16.0 0.00
|
| 130 |
+
lz_s44_g0 1.000 1.000 1.000 0.98 | 1.000 1.000 0.98 16.0 0.00
|
| 131 |
+
lzm_s42_g0 1.000 1.000 1.000 1.24 | 1.000 1.000 1.24 16.0 0.00
|
| 132 |
+
lzm_s43_g0 1.000 1.000 1.000 1.28 | 1.000 1.000 1.28 16.0 0.00
|
| 133 |
+
lzm_s44_g0 1.000 1.000 1.000 1.03 | 1.000 1.000 1.03 16.0 0.00
|
| 134 |
+
phys_s42_g0 1.000 1.000 1.000 1.29 | 1.000 1.000 1.29 16.0 0.00
|
| 135 |
+
phys_s43_g0 1.000 1.000 1.000 1.27 | 1.000 1.000 1.27 16.0 0.00
|
| 136 |
+
phys_s44_g0 1.000 1.000 1.000 1.10 | 1.000 1.000 1.10 16.0 0.00
|
| 137 |
+
nn 1.000 1.000 1.000 0.53 | 1.000 1.000 0.53 16.0 0.00
|
| 138 |
+
idw 1.000 1.000 1.000 0.55 | 1.000 1.000 0.55 16.0 0.00
|
| 139 |
+
nns 1.000 1.000 1.000 0.95 | 1.000 1.000 0.95 16.0 0.00
|
| 140 |
+
dls 1.000 1.000 1.000 0.95 | 1.000 1.000 0.95 16.0 0.00
|
| 141 |
+
lzres_s42_g0 1.000 1.000 1.000 1.10 | 1.000 1.000 1.10 16.0 0.00
|
| 142 |
+
lzres_s43_g0 1.000 1.000 1.000 1.09 | 1.000 1.000 1.09 16.0 0.00
|
| 143 |
+
lzres_s44_g0 1.000 1.000 1.000 1.10 | 1.000 1.000 1.10 16.0 0.00
|
| 144 |
+
|
| 145 |
+
-- paired Wilcoxon on regret, model minus best retrieval (B: best=nn; noB: best=nn); negative = model better; [95% CI of paired median]
|
| 146 |
+
base base_s0 B vs nn : +0.28 [+0.02,+0.68] p=0.00419 certified 1.000 vs 1.000
|
| 147 |
+
base base_s0 noB vs nn : +0.28 [+0.02,+0.68] p=0.00419 certified 1.000 vs 1.000
|
| 148 |
+
base base_s1 B vs nn : +0.21 [+0.02,+0.69] p=0.00492 certified 1.000 vs 1.000
|
| 149 |
+
base base_s1 noB vs nn : +0.21 [+0.02,+0.69] p=0.00492 certified 1.000 vs 1.000
|
| 150 |
+
base base_s2 B vs nn : +0.31 [+0.01,+0.58] p=0.00753 certified 1.000 vs 1.000
|
| 151 |
+
base base_s2 noB vs nn : +0.31 [+0.01,+0.58] p=0.00753 certified 1.000 vs 1.000
|
| 152 |
+
A_lz lz_s42_g0 B vs nn : +0.63 [+0.35,+0.80] p=0.000179 certified 1.000 vs 1.000
|
| 153 |
+
A_lz lz_s42_g0 noB vs nn : +0.63 [+0.35,+0.80] p=0.000179 certified 1.000 vs 1.000
|
| 154 |
+
A_lz lz_s42_g1 B vs nn : +0.56 [+0.41,+0.80] p=0.000458 certified 1.000 vs 1.000
|
| 155 |
+
A_lz lz_s42_g1 noB vs nn : +0.56 [+0.41,+0.80] p=0.000458 certified 1.000 vs 1.000
|
| 156 |
+
A_lz lz_s42_g2 B vs nn : +0.62 [+0.36,+0.80] p=0.000239 certified 1.000 vs 1.000
|
| 157 |
+
A_lz lz_s42_g2 noB vs nn : +0.62 [+0.36,+0.80] p=0.000239 certified 1.000 vs 1.000
|
| 158 |
+
A_lz lz_s43_g0 B vs nn : +0.68 [+0.49,+0.87] p=6.21e-05 certified 1.000 vs 1.000
|
| 159 |
+
A_lz lz_s43_g0 noB vs nn : +0.68 [+0.49,+0.87] p=6.21e-05 certified 1.000 vs 1.000
|
| 160 |
+
A_lz lz_s44_g0 B vs nn : +0.52 [+0.27,+0.75] p=0.000446 certified 1.000 vs 1.000
|
| 161 |
+
A_lz lz_s44_g0 noB vs nn : +0.52 [+0.27,+0.75] p=0.000446 certified 1.000 vs 1.000
|
| 162 |
+
A_lzm lzm_s42_g0 B vs nn : +0.79 [+0.52,+0.92] p=3.78e-06 certified 1.000 vs 1.000
|
| 163 |
+
A_lzm lzm_s42_g0 noB vs nn : +0.79 [+0.52,+0.92] p=3.78e-06 certified 1.000 vs 1.000
|
| 164 |
+
A_lzm lzm_s43_g0 B vs nn : +0.84 [+0.59,+0.97] p=3.52e-06 certified 1.000 vs 1.000
|
| 165 |
+
A_lzm lzm_s43_g0 noB vs nn : +0.84 [+0.59,+0.97] p=3.52e-06 certified 1.000 vs 1.000
|
| 166 |
+
A_lzm lzm_s44_g0 B vs nn : +0.50 [+0.34,+0.63] p=5.83e-05 certified 1.000 vs 1.000
|
| 167 |
+
A_lzm lzm_s44_g0 noB vs nn : +0.50 [+0.34,+0.63] p=5.83e-05 certified 1.000 vs 1.000
|
| 168 |
+
A_lzres lzres_s42_g0 B vs nn : +0.76 [+0.40,+0.97] p=0.000125 certified 1.000 vs 1.000
|
| 169 |
+
A_lzres lzres_s42_g0 noB vs nn : +0.76 [+0.40,+0.97] p=0.000125 certified 1.000 vs 1.000
|
| 170 |
+
A_lzres lzres_s43_g0 B vs nn : +0.78 [+0.40,+0.97] p=0.000122 certified 1.000 vs 1.000
|
| 171 |
+
A_lzres lzres_s43_g0 noB vs nn : +0.78 [+0.40,+0.97] p=0.000122 certified 1.000 vs 1.000
|
| 172 |
+
A_lzres lzres_s44_g0 B vs nn : +0.79 [+0.39,+0.98] p=0.000118 certified 1.000 vs 1.000
|
| 173 |
+
A_lzres lzres_s44_g0 noB vs nn : +0.79 [+0.39,+0.98] p=0.000118 certified 1.000 vs 1.000
|
| 174 |
+
A_phys phys_s42_g0 B vs nn : +0.82 [+0.53,+1.06] p=3.4e-06 certified 1.000 vs 1.000
|
| 175 |
+
A_phys phys_s42_g0 noB vs nn : +0.82 [+0.53,+1.06] p=3.4e-06 certified 1.000 vs 1.000
|
| 176 |
+
A_phys phys_s43_g0 B vs nn : +0.84 [+0.49,+0.98] p=4.06e-06 certified 1.000 vs 1.000
|
| 177 |
+
A_phys phys_s43_g0 noB vs nn : +0.84 [+0.49,+0.98] p=4.06e-06 certified 1.000 vs 1.000
|
| 178 |
+
A_phys phys_s44_g0 B vs nn : +0.64 [+0.38,+0.80] p=2.23e-05 certified 1.000 vs 1.000
|
| 179 |
+
A_phys phys_s44_g0 noB vs nn : +0.64 [+0.38,+0.80] p=2.23e-05 certified 1.000 vs 1.000
|
| 180 |
+
|
| 181 |
+
=== lvl1.1 K=8 oracle=strong2 (median oracle J 0.983) ===
|
| 182 |
+
arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired
|
| 183 |
+
base_s0 1.000 1.000 1.000 0.96 | 1.000 1.000 0.96 16.0 0.00
|
| 184 |
+
base_s1 1.000 1.000 1.000 1.02 | 1.000 1.000 1.02 16.0 0.00
|
| 185 |
+
base_s2 1.000 1.000 1.000 0.93 | 1.000 1.000 0.93 16.0 0.00
|
| 186 |
+
lz_s42_g0 1.000 1.000 1.000 1.18 | 1.000 1.000 1.18 16.0 0.00
|
| 187 |
+
lz_s42_g1 1.000 1.000 1.000 1.14 | 1.000 1.000 1.14 16.0 0.00
|
| 188 |
+
lz_s42_g2 1.000 1.000 1.000 1.14 | 1.000 1.000 1.14 16.0 0.00
|
| 189 |
+
lz_s43_g0 1.000 1.000 1.000 1.29 | 1.000 1.000 1.29 16.0 0.00
|
| 190 |
+
lz_s44_g0 1.000 1.000 1.000 1.10 | 1.000 1.000 1.10 16.0 0.00
|
| 191 |
+
lzm_s42_g0 1.000 1.000 1.000 1.30 | 1.000 1.000 1.30 16.0 0.00
|
| 192 |
+
lzm_s43_g0 1.000 1.000 1.000 1.35 | 1.000 1.000 1.35 16.0 0.00
|
| 193 |
+
lzm_s44_g0 1.000 1.000 1.000 1.12 | 1.000 1.000 1.12 16.0 0.00
|
| 194 |
+
phys_s42_g0 1.000 1.000 1.000 1.36 | 1.000 1.000 1.36 16.0 0.00
|
| 195 |
+
phys_s43_g0 1.000 1.000 1.000 1.30 | 1.000 1.000 1.30 16.0 0.00
|
| 196 |
+
phys_s44_g0 1.000 1.000 1.000 1.16 | 1.000 1.000 1.16 16.0 0.00
|
| 197 |
+
nn 1.000 1.000 1.000 0.80 | 1.000 1.000 0.80 16.0 0.00
|
| 198 |
+
idw 1.000 1.000 1.000 0.80 | 1.000 1.000 0.80 16.0 0.00
|
| 199 |
+
nns 1.000 1.000 1.000 1.02 | 1.000 1.000 1.02 16.0 0.00
|
| 200 |
+
dls 1.000 1.000 1.000 1.09 | 1.000 1.000 1.09 16.0 0.00
|
| 201 |
+
lzres_s42_g0 1.000 1.000 1.000 1.19 | 1.000 1.000 1.19 16.0 0.00
|
| 202 |
+
lzres_s43_g0 1.000 1.000 1.000 1.19 | 1.000 1.000 1.19 16.0 0.00
|
| 203 |
+
lzres_s44_g0 1.000 1.000 1.000 1.21 | 1.000 1.000 1.21 16.0 0.00
|
| 204 |
+
|
| 205 |
+
-- paired Wilcoxon on regret, model minus best retrieval (B: best=nn; noB: best=nn); negative = model better; [95% CI of paired median]
|
| 206 |
+
base base_s0 B vs nn : +0.28 [+0.02,+0.68] p=0.00419 certified 1.000 vs 1.000
|
| 207 |
+
base base_s0 noB vs nn : +0.28 [+0.02,+0.68] p=0.00419 certified 1.000 vs 1.000
|
| 208 |
+
base base_s1 B vs nn : +0.21 [+0.02,+0.69] p=0.00481 certified 1.000 vs 1.000
|
| 209 |
+
base base_s1 noB vs nn : +0.21 [+0.02,+0.69] p=0.00481 certified 1.000 vs 1.000
|
| 210 |
+
base base_s2 B vs nn : +0.31 [+0.01,+0.58] p=0.00737 certified 1.000 vs 1.000
|
| 211 |
+
base base_s2 noB vs nn : +0.31 [+0.01,+0.58] p=0.00737 certified 1.000 vs 1.000
|
| 212 |
+
A_lz lz_s42_g0 B vs nn : +0.63 [+0.35,+0.80] p=0.000174 certified 1.000 vs 1.000
|
| 213 |
+
A_lz lz_s42_g0 noB vs nn : +0.63 [+0.35,+0.80] p=0.000174 certified 1.000 vs 1.000
|
| 214 |
+
A_lz lz_s42_g1 B vs nn : +0.56 [+0.41,+0.80] p=0.000446 certified 1.000 vs 1.000
|
| 215 |
+
A_lz lz_s42_g1 noB vs nn : +0.56 [+0.41,+0.80] p=0.000446 certified 1.000 vs 1.000
|
| 216 |
+
A_lz lz_s42_g2 B vs nn : +0.62 [+0.36,+0.80] p=0.000239 certified 1.000 vs 1.000
|
| 217 |
+
A_lz lz_s42_g2 noB vs nn : +0.62 [+0.36,+0.80] p=0.000239 certified 1.000 vs 1.000
|
| 218 |
+
A_lz lz_s43_g0 B vs nn : +0.68 [+0.49,+0.87] p=6.21e-05 certified 1.000 vs 1.000
|
| 219 |
+
A_lz lz_s43_g0 noB vs nn : +0.68 [+0.49,+0.87] p=6.21e-05 certified 1.000 vs 1.000
|
| 220 |
+
A_lz lz_s44_g0 B vs nn : +0.52 [+0.27,+0.75] p=0.000446 certified 1.000 vs 1.000
|
| 221 |
+
A_lz lz_s44_g0 noB vs nn : +0.52 [+0.27,+0.75] p=0.000446 certified 1.000 vs 1.000
|
| 222 |
+
A_lzm lzm_s42_g0 B vs nn : +0.79 [+0.52,+0.92] p=3.65e-06 certified 1.000 vs 1.000
|
| 223 |
+
A_lzm lzm_s42_g0 noB vs nn : +0.79 [+0.52,+0.92] p=3.65e-06 certified 1.000 vs 1.000
|
| 224 |
+
A_lzm lzm_s43_g0 B vs nn : +0.84 [+0.59,+0.97] p=3.52e-06 certified 1.000 vs 1.000
|
| 225 |
+
A_lzm lzm_s43_g0 noB vs nn : +0.84 [+0.59,+0.97] p=3.52e-06 certified 1.000 vs 1.000
|
| 226 |
+
A_lzm lzm_s44_g0 B vs nn : +0.50 [+0.34,+0.63] p=5.83e-05 certified 1.000 vs 1.000
|
| 227 |
+
A_lzm lzm_s44_g0 noB vs nn : +0.50 [+0.34,+0.63] p=5.83e-05 certified 1.000 vs 1.000
|
| 228 |
+
A_lzres lzres_s42_g0 B vs nn : +0.76 [+0.40,+0.97] p=0.000122 certified 1.000 vs 1.000
|
| 229 |
+
A_lzres lzres_s42_g0 noB vs nn : +0.76 [+0.40,+0.97] p=0.000122 certified 1.000 vs 1.000
|
| 230 |
+
A_lzres lzres_s43_g0 B vs nn : +0.78 [+0.40,+0.97] p=0.000125 certified 1.000 vs 1.000
|
| 231 |
+
A_lzres lzres_s43_g0 noB vs nn : +0.78 [+0.40,+0.97] p=0.000125 certified 1.000 vs 1.000
|
| 232 |
+
A_lzres lzres_s44_g0 B vs nn : +0.79 [+0.39,+0.98] p=0.000111 certified 1.000 vs 1.000
|
| 233 |
+
A_lzres lzres_s44_g0 noB vs nn : +0.79 [+0.39,+0.98] p=0.000111 certified 1.000 vs 1.000
|
| 234 |
+
A_phys phys_s42_g0 B vs nn : +0.82 [+0.53,+1.06] p=3.4e-06 certified 1.000 vs 1.000
|
| 235 |
+
A_phys phys_s42_g0 noB vs nn : +0.82 [+0.53,+1.06] p=3.4e-06 certified 1.000 vs 1.000
|
| 236 |
+
A_phys phys_s43_g0 B vs nn : +0.84 [+0.49,+0.98] p=4.06e-06 certified 1.000 vs 1.000
|
| 237 |
+
A_phys phys_s43_g0 noB vs nn : +0.84 [+0.49,+0.98] p=4.06e-06 certified 1.000 vs 1.000
|
| 238 |
+
A_phys phys_s44_g0 B vs nn : +0.64 [+0.38,+0.80] p=2.23e-05 certified 1.000 vs 1.000
|
| 239 |
+
A_phys phys_s44_g0 noB vs nn : +0.64 [+0.38,+0.80] p=2.23e-05 certified 1.000 vs 1.000
|
| 240 |
+
|
| 241 |
+
=== lvl1.25 K=8 oracle=old (median oracle J 0.318) ===
|
| 242 |
+
arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired
|
| 243 |
+
base_s0 1.000 1.000 1.000 0.70 | 1.000 1.000 0.70 16.0 0.00
|
| 244 |
+
base_s1 1.000 1.000 1.000 0.67 | 1.000 1.000 0.67 16.0 0.00
|
| 245 |
+
base_s2 1.000 1.000 1.000 0.72 | 1.000 1.000 0.72 16.0 0.00
|
| 246 |
+
lz_s42_g0 1.000 1.000 1.000 1.00 | 1.000 1.000 1.00 16.0 0.00
|
| 247 |
+
lz_s42_g1 1.000 1.000 1.000 0.98 | 1.000 1.000 0.98 16.0 0.00
|
| 248 |
+
lz_s42_g2 1.000 1.000 1.000 1.00 | 1.000 1.000 1.00 16.0 0.00
|
| 249 |
+
lz_s43_g0 1.000 1.000 1.000 1.07 | 1.000 1.000 1.07 16.0 0.00
|
| 250 |
+
lz_s44_g0 1.000 1.000 1.000 0.94 | 1.000 1.000 0.94 16.0 0.00
|
| 251 |
+
lzm_s42_g0 1.000 1.000 1.000 1.14 | 1.000 1.000 1.14 16.0 0.00
|
| 252 |
+
lzm_s43_g0 1.000 1.000 1.000 1.13 | 1.000 1.000 1.13 16.0 0.00
|
| 253 |
+
lzm_s44_g0 1.000 1.000 1.000 1.00 | 1.000 1.000 1.00 16.0 0.00
|
| 254 |
+
phys_s42_g0 1.000 1.000 1.000 1.11 | 1.000 1.000 1.11 16.0 0.00
|
| 255 |
+
phys_s43_g0 1.000 1.000 1.000 1.12 | 1.000 1.000 1.12 16.0 0.00
|
| 256 |
+
phys_s44_g0 1.000 1.000 1.000 1.03 | 1.000 1.000 1.03 16.0 0.00
|
| 257 |
+
nn 1.000 1.000 1.000 1.06 | 1.000 1.000 1.06 16.0 0.00
|
| 258 |
+
idw 1.000 1.000 1.000 1.00 | 1.000 1.000 1.00 16.0 0.00
|
| 259 |
+
nns 1.000 1.000 1.000 0.95 | 1.000 1.000 0.95 16.0 0.00
|
| 260 |
+
dls 1.000 1.000 1.000 0.97 | 1.000 1.000 0.97 16.0 0.00
|
| 261 |
+
lzres_s42_g0 1.000 1.000 1.000 1.00 | 1.000 1.000 1.00 16.0 0.00
|
| 262 |
+
lzres_s43_g0 1.000 1.000 1.000 0.95 | 1.000 1.000 0.95 16.0 0.00
|
| 263 |
+
lzres_s44_g0 1.000 1.000 1.000 0.98 | 1.000 1.000 0.98 16.0 0.00
|
| 264 |
+
|
| 265 |
+
-- paired Wilcoxon on regret, model minus best retrieval (B: best=nns; noB: best=nns); negative = model better; [95% CI of paired median]
|
| 266 |
+
base base_s0 B vs nns: -0.17 [-0.24,-0.10] p=0.0026 certified 1.000 vs 1.000
|
| 267 |
+
base base_s0 noB vs nns: -0.17 [-0.24,-0.10] p=0.0026 certified 1.000 vs 1.000
|
| 268 |
+
base base_s1 B vs nns: -0.17 [-0.21,-0.07] p=0.00117 certified 1.000 vs 1.000
|
| 269 |
+
base base_s1 noB vs nns: -0.17 [-0.21,-0.07] p=0.00117 certified 1.000 vs 1.000
|
| 270 |
+
base base_s2 B vs nns: -0.12 [-0.19,-0.03] p=0.00111 certified 1.000 vs 1.000
|
| 271 |
+
base base_s2 noB vs nns: -0.12 [-0.19,-0.03] p=0.00111 certified 1.000 vs 1.000
|
| 272 |
+
A_lz lz_s42_g0 B vs nns: +0.08 [+0.05,+0.12] p=0.00977 certified 1.000 vs 1.000
|
| 273 |
+
A_lz lz_s42_g0 noB vs nns: +0.08 [+0.05,+0.12] p=0.00977 certified 1.000 vs 1.000
|
| 274 |
+
A_lz lz_s42_g1 B vs nns: +0.09 [+0.05,+0.14] p=0.0109 certified 1.000 vs 1.000
|
| 275 |
+
A_lz lz_s42_g1 noB vs nns: +0.09 [+0.05,+0.14] p=0.0109 certified 1.000 vs 1.000
|
| 276 |
+
A_lz lz_s42_g2 B vs nns: +0.10 [+0.05,+0.18] p=0.00822 certified 1.000 vs 1.000
|
| 277 |
+
A_lz lz_s42_g2 noB vs nns: +0.10 [+0.05,+0.18] p=0.00822 certified 1.000 vs 1.000
|
| 278 |
+
A_lz lz_s43_g0 B vs nns: +0.15 [+0.11,+0.25] p=0.000399 certified 1.000 vs 1.000
|
| 279 |
+
A_lz lz_s43_g0 noB vs nns: +0.15 [+0.11,+0.25] p=0.000399 certified 1.000 vs 1.000
|
| 280 |
+
A_lz lz_s44_g0 B vs nns: +0.05 [-0.01,+0.09] p=0.126 certified 1.000 vs 1.000
|
| 281 |
+
A_lz lz_s44_g0 noB vs nns: +0.05 [-0.01,+0.09] p=0.126 certified 1.000 vs 1.000
|
| 282 |
+
A_lzm lzm_s42_g0 B vs nns: +0.41 [+0.32,+0.45] p=2.21e-11 certified 1.000 vs 1.000
|
| 283 |
+
A_lzm lzm_s42_g0 noB vs nns: +0.41 [+0.32,+0.45] p=2.21e-11 certified 1.000 vs 1.000
|
| 284 |
+
A_lzm lzm_s43_g0 B vs nns: +0.38 [+0.35,+0.44] p=2.56e-11 certified 1.000 vs 1.000
|
| 285 |
+
A_lzm lzm_s43_g0 noB vs nns: +0.38 [+0.35,+0.44] p=2.56e-11 certified 1.000 vs 1.000
|
| 286 |
+
A_lzm lzm_s44_g0 B vs nns: +0.11 [+0.07,+0.21] p=5.83e-05 certified 1.000 vs 1.000
|
| 287 |
+
A_lzm lzm_s44_g0 noB vs nns: +0.11 [+0.07,+0.21] p=5.83e-05 certified 1.000 vs 1.000
|
| 288 |
+
A_lzres lzres_s42_g0 B vs nns: +0.10 [+0.06,+0.17] p=0.0234 certified 1.000 vs 1.000
|
| 289 |
+
A_lzres lzres_s42_g0 noB vs nns: +0.10 [+0.06,+0.17] p=0.0234 certified 1.000 vs 1.000
|
| 290 |
+
A_lzres lzres_s43_g0 B vs nns: +0.11 [+0.06,+0.19] p=0.0277 certified 1.000 vs 1.000
|
| 291 |
+
A_lzres lzres_s43_g0 noB vs nns: +0.11 [+0.06,+0.19] p=0.0277 certified 1.000 vs 1.000
|
| 292 |
+
A_lzres lzres_s44_g0 B vs nns: +0.11 [+0.07,+0.22] p=0.0196 certified 1.000 vs 1.000
|
| 293 |
+
A_lzres lzres_s44_g0 noB vs nns: +0.11 [+0.07,+0.22] p=0.0196 certified 1.000 vs 1.000
|
| 294 |
+
A_phys phys_s42_g0 B vs nns: +0.37 [+0.31,+0.43] p=2.1e-11 certified 1.000 vs 1.000
|
| 295 |
+
A_phys phys_s42_g0 noB vs nns: +0.37 [+0.31,+0.43] p=2.1e-11 certified 1.000 vs 1.000
|
| 296 |
+
A_phys phys_s43_g0 B vs nns: +0.36 [+0.27,+0.48] p=2.98e-11 certified 1.000 vs 1.000
|
| 297 |
+
A_phys phys_s43_g0 noB vs nns: +0.36 [+0.27,+0.48] p=2.98e-11 certified 1.000 vs 1.000
|
| 298 |
+
A_phys phys_s44_g0 B vs nns: +0.15 [+0.09,+0.21] p=2.66e-07 certified 1.000 vs 1.000
|
| 299 |
+
A_phys phys_s44_g0 noB vs nns: +0.15 [+0.09,+0.21] p=2.66e-07 certified 1.000 vs 1.000
|
| 300 |
+
|
| 301 |
+
=== lvl1.25 K=8 oracle=strong2 (median oracle J 0.318) ===
|
| 302 |
+
arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired
|
| 303 |
+
base_s0 1.000 1.000 1.000 0.86 | 1.000 1.000 0.86 16.0 0.00
|
| 304 |
+
base_s1 1.000 1.000 1.000 0.90 | 1.000 1.000 0.90 16.0 0.00
|
| 305 |
+
base_s2 1.000 1.000 1.000 0.91 | 1.000 1.000 0.91 16.0 0.00
|
| 306 |
+
lz_s42_g0 1.000 1.000 1.000 1.14 | 1.000 1.000 1.14 16.0 0.00
|
| 307 |
+
lz_s42_g1 1.000 1.000 1.000 1.13 | 1.000 1.000 1.13 16.0 0.00
|
| 308 |
+
lz_s42_g2 1.000 1.000 1.000 1.15 | 1.000 1.000 1.15 16.0 0.00
|
| 309 |
+
lz_s43_g0 1.000 1.000 1.000 1.23 | 1.000 1.000 1.23 16.0 0.00
|
| 310 |
+
lz_s44_g0 1.000 1.000 1.000 1.08 | 1.000 1.000 1.08 16.0 0.00
|
| 311 |
+
lzm_s42_g0 1.000 1.000 1.000 1.29 | 1.000 1.000 1.29 16.0 0.00
|
| 312 |
+
lzm_s43_g0 1.000 1.000 1.000 1.31 | 1.000 1.000 1.31 16.0 0.00
|
| 313 |
+
lzm_s44_g0 1.000 1.000 1.000 1.13 | 1.000 1.000 1.13 16.0 0.00
|
| 314 |
+
phys_s42_g0 1.000 1.000 1.000 1.26 | 1.000 1.000 1.26 16.0 0.00
|
| 315 |
+
phys_s43_g0 1.000 1.000 1.000 1.30 | 1.000 1.000 1.30 16.0 0.00
|
| 316 |
+
phys_s44_g0 1.000 1.000 1.000 1.16 | 1.000 1.000 1.16 16.0 0.00
|
| 317 |
+
nn 1.000 1.000 1.000 1.22 | 1.000 1.000 1.22 16.0 0.00
|
| 318 |
+
idw 1.000 1.000 1.000 1.17 | 1.000 1.000 1.17 16.0 0.00
|
| 319 |
+
nns 1.000 1.000 1.000 1.05 | 1.000 1.000 1.05 16.0 0.00
|
| 320 |
+
dls 1.000 1.000 1.000 1.08 | 1.000 1.000 1.08 16.0 0.00
|
| 321 |
+
lzres_s42_g0 1.000 1.000 1.000 1.17 | 1.000 1.000 1.17 16.0 0.00
|
| 322 |
+
lzres_s43_g0 1.000 1.000 1.000 1.17 | 1.000 1.000 1.17 16.0 0.00
|
| 323 |
+
lzres_s44_g0 1.000 1.000 1.000 1.17 | 1.000 1.000 1.17 16.0 0.00
|
| 324 |
+
|
| 325 |
+
-- paired Wilcoxon on regret, model minus best retrieval (B: best=nns; noB: best=nns); negative = model better; [95% CI of paired median]
|
| 326 |
+
base base_s0 B vs nns: -0.17 [-0.24,-0.10] p=0.00267 certified 1.000 vs 1.000
|
| 327 |
+
base base_s0 noB vs nns: -0.17 [-0.24,-0.10] p=0.00267 certified 1.000 vs 1.000
|
| 328 |
+
base base_s1 B vs nns: -0.17 [-0.21,-0.07] p=0.00117 certified 1.000 vs 1.000
|
| 329 |
+
base base_s1 noB vs nns: -0.17 [-0.21,-0.07] p=0.00117 certified 1.000 vs 1.000
|
| 330 |
+
base base_s2 B vs nns: -0.12 [-0.19,-0.03] p=0.00108 certified 1.000 vs 1.000
|
| 331 |
+
base base_s2 noB vs nns: -0.12 [-0.19,-0.03] p=0.00108 certified 1.000 vs 1.000
|
| 332 |
+
A_lz lz_s42_g0 B vs nns: +0.08 [+0.05,+0.12] p=0.0104 certified 1.000 vs 1.000
|
| 333 |
+
A_lz lz_s42_g0 noB vs nns: +0.08 [+0.05,+0.12] p=0.0104 certified 1.000 vs 1.000
|
| 334 |
+
A_lz lz_s42_g1 B vs nns: +0.09 [+0.05,+0.14] p=0.0111 certified 1.000 vs 1.000
|
| 335 |
+
A_lz lz_s42_g1 noB vs nns: +0.09 [+0.05,+0.14] p=0.0111 certified 1.000 vs 1.000
|
| 336 |
+
A_lz lz_s42_g2 B vs nns: +0.10 [+0.05,+0.18] p=0.00859 certified 1.000 vs 1.000
|
| 337 |
+
A_lz lz_s42_g2 noB vs nns: +0.10 [+0.05,+0.18] p=0.00859 certified 1.000 vs 1.000
|
| 338 |
+
A_lz lz_s43_g0 B vs nns: +0.15 [+0.11,+0.25] p=0.000446 certified 1.000 vs 1.000
|
| 339 |
+
A_lz lz_s43_g0 noB vs nns: +0.15 [+0.11,+0.25] p=0.000446 certified 1.000 vs 1.000
|
| 340 |
+
A_lz lz_s44_g0 B vs nns: +0.05 [-0.01,+0.09] p=0.126 certified 1.000 vs 1.000
|
| 341 |
+
A_lz lz_s44_g0 noB vs nns: +0.05 [-0.01,+0.09] p=0.126 certified 1.000 vs 1.000
|
| 342 |
+
A_lzm lzm_s42_g0 B vs nns: +0.41 [+0.32,+0.45] p=2.21e-11 certified 1.000 vs 1.000
|
| 343 |
+
A_lzm lzm_s42_g0 noB vs nns: +0.41 [+0.32,+0.45] p=2.21e-11 certified 1.000 vs 1.000
|
| 344 |
+
A_lzm lzm_s43_g0 B vs nns: +0.38 [+0.35,+0.44] p=2.56e-11 certified 1.000 vs 1.000
|
| 345 |
+
A_lzm lzm_s43_g0 noB vs nns: +0.38 [+0.35,+0.44] p=2.56e-11 certified 1.000 vs 1.000
|
| 346 |
+
A_lzm lzm_s44_g0 B vs nns: +0.11 [+0.07,+0.21] p=5.83e-05 certified 1.000 vs 1.000
|
| 347 |
+
A_lzm lzm_s44_g0 noB vs nns: +0.11 [+0.07,+0.21] p=5.83e-05 certified 1.000 vs 1.000
|
| 348 |
+
A_lzres lzres_s42_g0 B vs nns: +0.10 [+0.06,+0.17] p=0.0247 certified 1.000 vs 1.000
|
| 349 |
+
A_lzres lzres_s42_g0 noB vs nns: +0.10 [+0.06,+0.17] p=0.0247 certified 1.000 vs 1.000
|
| 350 |
+
A_lzres lzres_s43_g0 B vs nns: +0.11 [+0.06,+0.19] p=0.0283 certified 1.000 vs 1.000
|
| 351 |
+
A_lzres lzres_s43_g0 noB vs nns: +0.11 [+0.06,+0.19] p=0.0283 certified 1.000 vs 1.000
|
| 352 |
+
A_lzres lzres_s44_g0 B vs nns: +0.11 [+0.07,+0.22] p=0.0204 certified 1.000 vs 1.000
|
| 353 |
+
A_lzres lzres_s44_g0 noB vs nns: +0.11 [+0.07,+0.22] p=0.0204 certified 1.000 vs 1.000
|
| 354 |
+
A_phys phys_s42_g0 B vs nns: +0.37 [+0.31,+0.43] p=2.1e-11 certified 1.000 vs 1.000
|
| 355 |
+
A_phys phys_s42_g0 noB vs nns: +0.37 [+0.31,+0.43] p=2.1e-11 certified 1.000 vs 1.000
|
| 356 |
+
A_phys phys_s43_g0 B vs nns: +0.36 [+0.27,+0.49] p=2.98e-11 certified 1.000 vs 1.000
|
| 357 |
+
A_phys phys_s43_g0 noB vs nns: +0.36 [+0.27,+0.49] p=2.98e-11 certified 1.000 vs 1.000
|
| 358 |
+
A_phys phys_s44_g0 B vs nns: +0.15 [+0.09,+0.21] p=2.66e-07 certified 1.000 vs 1.000
|
| 359 |
+
A_phys phys_s44_g0 noB vs nns: +0.15 [+0.09,+0.21] p=2.66e-07 certified 1.000 vs 1.000
|
| 360 |
+
|
| 361 |
+
=== lvl1.5 K=8 oracle=old (median oracle J 0.656) ===
|
| 362 |
+
arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired
|
| 363 |
+
base_s0 1.000 0.983 0.983 0.83 | 1.000 1.000 0.83 16.1 0.02
|
| 364 |
+
base_s1 1.000 0.983 0.967 0.92 | 1.000 1.000 0.92 16.1 0.02
|
| 365 |
+
base_s2 1.000 0.983 0.983 0.86 | 1.000 1.000 0.86 16.1 0.02
|
| 366 |
+
lz_s42_g0 1.000 1.000 1.000 1.26 | 1.000 1.000 1.26 16.0 0.00
|
| 367 |
+
lz_s42_g1 1.000 1.000 1.000 1.17 | 1.000 1.000 1.17 16.0 0.00
|
| 368 |
+
lz_s42_g2 1.000 1.000 1.000 1.20 | 1.000 1.000 1.20 16.0 0.00
|
| 369 |
+
lz_s43_g0 1.000 1.000 1.000 1.23 | 1.000 1.000 1.23 16.0 0.00
|
| 370 |
+
lz_s44_g0 1.000 1.000 1.000 1.09 | 1.000 1.000 1.09 16.0 0.00
|
| 371 |
+
lzm_s42_g0 1.000 1.000 1.000 1.26 | 1.000 1.000 1.26 16.0 0.00
|
| 372 |
+
lzm_s43_g0 1.000 1.000 1.000 1.22 | 1.000 1.000 1.22 16.0 0.00
|
| 373 |
+
lzm_s44_g0 1.000 1.000 1.000 1.03 | 1.000 1.000 1.03 16.0 0.00
|
| 374 |
+
phys_s42_g0 1.000 1.000 1.000 1.28 | 1.000 1.000 1.28 16.0 0.00
|
| 375 |
+
phys_s43_g0 1.000 1.000 1.000 1.29 | 1.000 1.000 1.29 16.0 0.00
|
| 376 |
+
phys_s44_g0 1.000 1.000 1.000 1.09 | 1.000 1.000 1.09 16.0 0.00
|
| 377 |
+
nn 1.000 1.000 1.000 1.08 | 1.000 1.000 1.08 16.0 0.00
|
| 378 |
+
idw 1.000 1.000 1.000 1.16 | 1.000 1.000 1.16 16.0 0.00
|
| 379 |
+
nns 1.000 1.000 1.000 0.98 | 1.000 1.000 0.98 16.0 0.00
|
| 380 |
+
dls 1.000 1.000 1.000 1.05 | 1.000 1.000 1.05 16.0 0.00
|
| 381 |
+
lzres_s42_g0 1.000 1.000 1.000 1.17 | 1.000 1.000 1.17 16.0 0.00
|
| 382 |
+
lzres_s43_g0 1.000 1.000 1.000 1.16 | 1.000 1.000 1.16 16.0 0.00
|
| 383 |
+
lzres_s44_g0 1.000 1.000 1.000 1.17 | 1.000 1.000 1.17 16.0 0.00
|
| 384 |
+
|
| 385 |
+
-- paired Wilcoxon on regret, model minus best retrieval (B: best=nns; noB: best=nns); negative = model better; [95% CI of paired median]
|
| 386 |
+
base base_s0 B vs nns: -0.03 [-0.18,+0.10] p=0.462 certified 1.000 vs 1.000
|
| 387 |
+
base base_s0 noB vs nns: -0.03 [-0.18,+0.10] p=0.475 certified 0.983 vs 1.000
|
| 388 |
+
base base_s1 B vs nns: +0.08 [-0.11,+0.37] p=0.069 certified 1.000 vs 1.000
|
| 389 |
+
base base_s1 noB vs nns: +0.08 [-0.11,+0.37] p=0.076 certified 0.967 vs 1.000
|
| 390 |
+
base base_s2 B vs nns: -0.03 [-0.14,+0.09] p=0.691 certified 1.000 vs 1.000
|
| 391 |
+
base base_s2 noB vs nns: -0.03 [-0.14,+0.09] p=0.691 certified 0.983 vs 1.000
|
| 392 |
+
A_lz lz_s42_g0 B vs nns: +0.27 [+0.13,+0.75] p=1.79e-08 certified 1.000 vs 1.000
|
| 393 |
+
A_lz lz_s42_g0 noB vs nns: +0.27 [+0.13,+0.75] p=1.79e-08 certified 1.000 vs 1.000
|
| 394 |
+
A_lz lz_s42_g1 B vs nns: +0.24 [+0.12,+0.69] p=1.32e-08 certified 1.000 vs 1.000
|
| 395 |
+
A_lz lz_s42_g1 noB vs nns: +0.24 [+0.12,+0.69] p=1.32e-08 certified 1.000 vs 1.000
|
| 396 |
+
A_lz lz_s42_g2 B vs nns: +0.22 [+0.11,+0.82] p=4.9e-08 certified 1.000 vs 1.000
|
| 397 |
+
A_lz lz_s42_g2 noB vs nns: +0.22 [+0.11,+0.82] p=4.9e-08 certified 1.000 vs 1.000
|
| 398 |
+
A_lz lz_s43_g0 B vs nns: +0.31 [+0.16,+0.64] p=1.02e-08 certified 1.000 vs 1.000
|
| 399 |
+
A_lz lz_s43_g0 noB vs nns: +0.31 [+0.16,+0.64] p=1.02e-08 certified 1.000 vs 1.000
|
| 400 |
+
A_lz lz_s44_g0 B vs nns: +0.21 [+0.12,+0.59] p=9.83e-08 certified 1.000 vs 1.000
|
| 401 |
+
A_lz lz_s44_g0 noB vs nns: +0.21 [+0.12,+0.59] p=9.83e-08 certified 1.000 vs 1.000
|
| 402 |
+
A_lzm lzm_s42_g0 B vs nns: +0.38 [+0.33,+0.51] p=2.56e-10 certified 1.000 vs 1.000
|
| 403 |
+
A_lzm lzm_s42_g0 noB vs nns: +0.38 [+0.33,+0.51] p=2.56e-10 certified 1.000 vs 1.000
|
| 404 |
+
A_lzm lzm_s43_g0 B vs nns: +0.39 [+0.33,+0.53] p=7.2e-10 certified 1.000 vs 1.000
|
| 405 |
+
A_lzm lzm_s43_g0 noB vs nns: +0.39 [+0.33,+0.53] p=7.2e-10 certified 1.000 vs 1.000
|
| 406 |
+
A_lzm lzm_s44_g0 B vs nns: +0.11 [+0.04,+0.24] p=1.5e-05 certified 1.000 vs 1.000
|
| 407 |
+
A_lzm lzm_s44_g0 noB vs nns: +0.11 [+0.04,+0.24] p=1.5e-05 certified 1.000 vs 1.000
|
| 408 |
+
A_lzres lzres_s42_g0 B vs nns: +0.16 [+0.08,+0.29] p=0.00527 certified 1.000 vs 1.000
|
| 409 |
+
A_lzres lzres_s42_g0 noB vs nns: +0.16 [+0.08,+0.29] p=0.00527 certified 1.000 vs 1.000
|
| 410 |
+
A_lzres lzres_s43_g0 B vs nns: +0.16 [+0.10,+0.32] p=0.00409 certified 1.000 vs 1.000
|
| 411 |
+
A_lzres lzres_s43_g0 noB vs nns: +0.16 [+0.10,+0.32] p=0.00409 certified 1.000 vs 1.000
|
| 412 |
+
A_lzres lzres_s44_g0 B vs nns: +0.14 [+0.07,+0.29] p=0.00936 certified 1.000 vs 1.000
|
| 413 |
+
A_lzres lzres_s44_g0 noB vs nns: +0.14 [+0.07,+0.29] p=0.00936 certified 1.000 vs 1.000
|
| 414 |
+
A_phys phys_s42_g0 B vs nns: +0.43 [+0.32,+0.64] p=1.92e-10 certified 1.000 vs 1.000
|
| 415 |
+
A_phys phys_s42_g0 noB vs nns: +0.43 [+0.32,+0.64] p=1.92e-10 certified 1.000 vs 1.000
|
| 416 |
+
A_phys phys_s43_g0 B vs nns: +0.43 [+0.30,+0.63] p=8.27e-10 certified 1.000 vs 1.000
|
| 417 |
+
A_phys phys_s43_g0 noB vs nns: +0.43 [+0.30,+0.63] p=8.27e-10 certified 1.000 vs 1.000
|
| 418 |
+
A_phys phys_s44_g0 B vs nns: +0.22 [+0.13,+0.45] p=3.51e-08 certified 1.000 vs 1.000
|
| 419 |
+
A_phys phys_s44_g0 noB vs nns: +0.22 [+0.13,+0.45] p=3.51e-08 certified 1.000 vs 1.000
|
| 420 |
+
|
| 421 |
+
=== lvl1.5 K=8 oracle=strong2 (median oracle J 0.656) ===
|
| 422 |
+
arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired
|
| 423 |
+
base_s0 1.000 0.983 0.983 0.93 | 1.000 1.000 0.97 16.1 0.02
|
| 424 |
+
base_s1 1.000 0.983 0.967 1.01 | 1.000 1.000 1.10 16.1 0.02
|
| 425 |
+
base_s2 1.000 0.983 0.983 0.98 | 1.000 1.000 1.01 16.1 0.02
|
| 426 |
+
lz_s42_g0 1.000 1.000 1.000 1.33 | 1.000 1.000 1.33 16.0 0.00
|
| 427 |
+
lz_s42_g1 1.000 1.000 1.000 1.32 | 1.000 1.000 1.32 16.0 0.00
|
| 428 |
+
lz_s42_g2 1.000 1.000 1.000 1.33 | 1.000 1.000 1.33 16.0 0.00
|
| 429 |
+
lz_s43_g0 1.000 1.000 1.000 1.40 | 1.000 1.000 1.40 16.0 0.00
|
| 430 |
+
lz_s44_g0 1.000 1.000 1.000 1.27 | 1.000 1.000 1.27 16.0 0.00
|
| 431 |
+
lzm_s42_g0 1.000 1.000 1.000 1.35 | 1.000 1.000 1.35 16.0 0.00
|
| 432 |
+
lzm_s43_g0 1.000 1.000 1.000 1.31 | 1.000 1.000 1.31 16.0 0.00
|
| 433 |
+
lzm_s44_g0 1.000 1.000 1.000 1.15 | 1.000 1.000 1.15 16.0 0.00
|
| 434 |
+
phys_s42_g0 1.000 1.000 1.000 1.37 | 1.000 1.000 1.37 16.0 0.00
|
| 435 |
+
phys_s43_g0 1.000 1.000 1.000 1.35 | 1.000 1.000 1.35 16.0 0.00
|
| 436 |
+
phys_s44_g0 1.000 1.000 1.000 1.23 | 1.000 1.000 1.23 16.0 0.00
|
| 437 |
+
nn 1.000 1.000 1.000 1.18 | 1.000 1.000 1.18 16.0 0.00
|
| 438 |
+
idw 1.000 1.000 1.000 1.28 | 1.000 1.000 1.28 16.0 0.00
|
| 439 |
+
nns 1.000 1.000 1.000 1.12 | 1.000 1.000 1.12 16.0 0.00
|
| 440 |
+
dls 1.000 1.000 1.000 1.22 | 1.000 1.000 1.22 16.0 0.00
|
| 441 |
+
lzres_s42_g0 1.000 1.000 1.000 1.37 | 1.000 1.000 1.37 16.0 0.00
|
| 442 |
+
lzres_s43_g0 1.000 1.000 1.000 1.36 | 1.000 1.000 1.36 16.0 0.00
|
| 443 |
+
lzres_s44_g0 1.000 1.000 1.000 1.39 | 1.000 1.000 1.39 16.0 0.00
|
| 444 |
+
|
| 445 |
+
-- paired Wilcoxon on regret, model minus best retrieval (B: best=nns; noB: best=nns); negative = model better; [95% CI of paired median]
|
| 446 |
+
base base_s0 B vs nns: -0.03 [-0.18,+0.10] p=0.475 certified 1.000 vs 1.000
|
| 447 |
+
base base_s0 noB vs nns: -0.03 [-0.18,+0.10] p=0.484 certified 0.983 vs 1.000
|
| 448 |
+
base base_s1 B vs nns: +0.08 [-0.11,+0.37] p=0.0701 certified 1.000 vs 1.000
|
| 449 |
+
base base_s1 noB vs nns: +0.08 [-0.11,+0.37] p=0.0748 certified 0.967 vs 1.000
|
| 450 |
+
base base_s2 B vs nns: -0.03 [-0.14,+0.09] p=0.686 certified 1.000 vs 1.000
|
| 451 |
+
base base_s2 noB vs nns: -0.03 [-0.14,+0.09] p=0.686 certified 0.983 vs 1.000
|
| 452 |
+
A_lz lz_s42_g0 B vs nns: +0.27 [+0.13,+0.80] p=1.86e-08 certified 1.000 vs 1.000
|
| 453 |
+
A_lz lz_s42_g0 noB vs nns: +0.27 [+0.13,+0.80] p=1.86e-08 certified 1.000 vs 1.000
|
| 454 |
+
A_lz lz_s42_g1 B vs nns: +0.25 [+0.12,+0.69] p=1.32e-08 certified 1.000 vs 1.000
|
| 455 |
+
A_lz lz_s42_g1 noB vs nns: +0.25 [+0.12,+0.69] p=1.32e-08 certified 1.000 vs 1.000
|
| 456 |
+
A_lz lz_s42_g2 B vs nns: +0.22 [+0.11,+0.83] p=4.9e-08 certified 1.000 vs 1.000
|
| 457 |
+
A_lz lz_s42_g2 noB vs nns: +0.22 [+0.11,+0.83] p=4.9e-08 certified 1.000 vs 1.000
|
| 458 |
+
A_lz lz_s43_g0 B vs nns: +0.31 [+0.16,+0.64] p=9.77e-09 certified 1.000 vs 1.000
|
| 459 |
+
A_lz lz_s43_g0 noB vs nns: +0.31 [+0.16,+0.64] p=9.77e-09 certified 1.000 vs 1.000
|
| 460 |
+
A_lz lz_s44_g0 B vs nns: +0.21 [+0.12,+0.59] p=9.83e-08 certified 1.000 vs 1.000
|
| 461 |
+
A_lz lz_s44_g0 noB vs nns: +0.21 [+0.12,+0.59] p=9.83e-08 certified 1.000 vs 1.000
|
| 462 |
+
A_lzm lzm_s42_g0 B vs nns: +0.38 [+0.33,+0.51] p=2.56e-10 certified 1.000 vs 1.000
|
| 463 |
+
A_lzm lzm_s42_g0 noB vs nns: +0.38 [+0.33,+0.51] p=2.56e-10 certified 1.000 vs 1.000
|
| 464 |
+
A_lzm lzm_s43_g0 B vs nns: +0.39 [+0.33,+0.53] p=7.2e-10 certified 1.000 vs 1.000
|
| 465 |
+
A_lzm lzm_s43_g0 noB vs nns: +0.39 [+0.33,+0.53] p=7.2e-10 certified 1.000 vs 1.000
|
| 466 |
+
A_lzm lzm_s44_g0 B vs nns: +0.11 [+0.04,+0.24] p=1.55e-05 certified 1.000 vs 1.000
|
| 467 |
+
A_lzm lzm_s44_g0 noB vs nns: +0.11 [+0.04,+0.24] p=1.55e-05 certified 1.000 vs 1.000
|
| 468 |
+
A_lzres lzres_s42_g0 B vs nns: +0.16 [+0.08,+0.29] p=0.00539 certified 1.000 vs 1.000
|
| 469 |
+
A_lzres lzres_s42_g0 noB vs nns: +0.16 [+0.08,+0.29] p=0.00539 certified 1.000 vs 1.000
|
| 470 |
+
A_lzres lzres_s43_g0 B vs nns: +0.16 [+0.10,+0.32] p=0.00409 certified 1.000 vs 1.000
|
| 471 |
+
A_lzres lzres_s43_g0 noB vs nns: +0.16 [+0.10,+0.32] p=0.00409 certified 1.000 vs 1.000
|
| 472 |
+
A_lzres lzres_s44_g0 B vs nns: +0.14 [+0.07,+0.29] p=0.00936 certified 1.000 vs 1.000
|
| 473 |
+
A_lzres lzres_s44_g0 noB vs nns: +0.14 [+0.07,+0.29] p=0.00936 certified 1.000 vs 1.000
|
| 474 |
+
A_phys phys_s42_g0 B vs nns: +0.44 [+0.32,+0.64] p=1.92e-10 certified 1.000 vs 1.000
|
| 475 |
+
A_phys phys_s42_g0 noB vs nns: +0.44 [+0.32,+0.64] p=1.92e-10 certified 1.000 vs 1.000
|
| 476 |
+
A_phys phys_s43_g0 B vs nns: +0.43 [+0.30,+0.63] p=8.27e-10 certified 1.000 vs 1.000
|
| 477 |
+
A_phys phys_s43_g0 noB vs nns: +0.43 [+0.30,+0.63] p=8.27e-10 certified 1.000 vs 1.000
|
| 478 |
+
A_phys phys_s44_g0 B vs nns: +0.22 [+0.13,+0.45] p=3.51e-08 certified 1.000 vs 1.000
|
| 479 |
+
A_phys phys_s44_g0 noB vs nns: +0.22 [+0.13,+0.45] p=3.51e-08 certified 1.000 vs 1.000
|
| 480 |
+
|
| 481 |
+
=== lvl2.0 K=8 oracle=old (median oracle J 0.557) ===
|
| 482 |
+
arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired
|
| 483 |
+
base_s0 1.000 0.933 0.833 0.73 | 1.000 1.000 0.75 16.5 0.07
|
| 484 |
+
base_s1 1.000 0.950 0.817 0.66 | 1.000 1.000 0.70 16.4 0.05
|
| 485 |
+
base_s2 1.000 0.950 0.833 0.65 | 1.000 1.000 0.72 16.4 0.05
|
| 486 |
+
lz_s42_g0 1.000 1.000 1.000 0.98 | 1.000 1.000 0.98 16.0 0.00
|
| 487 |
+
lz_s42_g1 1.000 1.000 1.000 0.97 | 1.000 1.000 0.97 16.0 0.00
|
| 488 |
+
lz_s42_g2 1.000 1.000 1.000 1.04 | 1.000 1.000 1.04 16.0 0.00
|
| 489 |
+
lz_s43_g0 1.000 1.000 1.000 1.13 | 1.000 1.000 1.13 16.0 0.00
|
| 490 |
+
lz_s44_g0 1.000 1.000 1.000 0.90 | 1.000 1.000 0.90 16.0 0.00
|
| 491 |
+
lzm_s42_g0 1.000 1.000 1.000 0.95 | 1.000 1.000 0.95 16.0 0.00
|
| 492 |
+
lzm_s43_g0 1.000 1.000 1.000 0.97 | 1.000 1.000 0.97 16.0 0.00
|
| 493 |
+
lzm_s44_g0 1.000 1.000 1.000 0.86 | 1.000 1.000 0.86 16.0 0.00
|
| 494 |
+
phys_s42_g0 1.000 1.000 1.000 1.11 | 1.000 1.000 1.11 16.0 0.00
|
| 495 |
+
phys_s43_g0 1.000 1.000 1.000 1.10 | 1.000 1.000 1.10 16.0 0.00
|
| 496 |
+
phys_s44_g0 1.000 1.000 1.000 0.97 | 1.000 1.000 0.97 16.0 0.00
|
| 497 |
+
nn 0.967 0.950 0.950 4.00 | 1.000 1.000 4.00 16.4 0.05
|
| 498 |
+
idw 0.950 0.933 0.933 4.31 | 1.000 1.000 4.31 16.6 0.07
|
| 499 |
+
nns 1.000 1.000 1.000 0.78 | 1.000 1.000 0.78 16.0 0.00
|
| 500 |
+
dls 1.000 1.000 1.000 0.81 | 1.000 1.000 0.81 16.0 0.00
|
| 501 |
+
lzres_s42_g0 1.000 1.000 1.000 0.94 | 1.000 1.000 0.94 16.0 0.00
|
| 502 |
+
lzres_s43_g0 1.000 1.000 1.000 0.94 | 1.000 1.000 0.94 16.0 0.00
|
| 503 |
+
lzres_s44_g0 1.000 1.000 1.000 0.92 | 1.000 1.000 0.92 16.0 0.00
|
| 504 |
+
|
| 505 |
+
-- paired Wilcoxon on regret, model minus best retrieval (B: best=nns; noB: best=nns); negative = model better; [95% CI of paired median]
|
| 506 |
+
base base_s0 B vs nns: -0.08 [-0.24,+0.21] p=0.768 certified 1.000 vs 1.000
|
| 507 |
+
base base_s0 noB vs nns: -0.08 [-0.20,+0.29] p=0.606 certified 0.833 vs 1.000
|
| 508 |
+
base base_s1 B vs nns: -0.10 [-0.26,+0.13] p=0.837 certified 1.000 vs 1.000
|
| 509 |
+
base base_s1 noB vs nns: -0.10 [-0.31,+0.13] p=0.819 certified 0.817 vs 1.000
|
| 510 |
+
base base_s2 B vs nns: -0.11 [-0.30,+0.22] p=0.729 certified 1.000 vs 1.000
|
| 511 |
+
base base_s2 noB vs nns: -0.12 [-0.30,+0.07] p=0.947 certified 0.833 vs 1.000
|
| 512 |
+
A_lz lz_s42_g0 B vs nns: +0.08 [+0.02,+0.33] p=0.0111 certified 1.000 vs 1.000
|
| 513 |
+
A_lz lz_s42_g0 noB vs nns: +0.08 [+0.02,+0.33] p=0.0111 certified 1.000 vs 1.000
|
| 514 |
+
A_lz lz_s42_g1 B vs nns: +0.07 [+0.02,+0.18] p=0.0174 certified 1.000 vs 1.000
|
| 515 |
+
A_lz lz_s42_g1 noB vs nns: +0.07 [+0.02,+0.18] p=0.0174 certified 1.000 vs 1.000
|
| 516 |
+
A_lz lz_s42_g2 B vs nns: +0.08 [+0.00,+0.28] p=0.0154 certified 1.000 vs 1.000
|
| 517 |
+
A_lz lz_s42_g2 noB vs nns: +0.08 [+0.00,+0.28] p=0.0154 certified 1.000 vs 1.000
|
| 518 |
+
A_lz lz_s43_g0 B vs nns: +0.21 [+0.07,+0.40] p=0.001 certified 1.000 vs 1.000
|
| 519 |
+
A_lz lz_s43_g0 noB vs nns: +0.21 [+0.07,+0.40] p=0.001 certified 1.000 vs 1.000
|
| 520 |
+
A_lz lz_s44_g0 B vs nns: +0.07 [-0.02,+0.35] p=0.0359 certified 1.000 vs 1.000
|
| 521 |
+
A_lz lz_s44_g0 noB vs nns: +0.07 [-0.02,+0.35] p=0.0359 certified 1.000 vs 1.000
|
| 522 |
+
A_lzm lzm_s42_g0 B vs nns: +0.09 [+0.02,+0.30] p=0.00254 certified 1.000 vs 1.000
|
| 523 |
+
A_lzm lzm_s42_g0 noB vs nns: +0.09 [+0.02,+0.30] p=0.00254 certified 1.000 vs 1.000
|
| 524 |
+
A_lzm lzm_s43_g0 B vs nns: +0.09 [+0.01,+0.21] p=0.0059 certified 1.000 vs 1.000
|
| 525 |
+
A_lzm lzm_s43_g0 noB vs nns: +0.09 [+0.01,+0.21] p=0.0059 certified 1.000 vs 1.000
|
| 526 |
+
A_lzm lzm_s44_g0 B vs nns: -0.01 [-0.06,+0.05] p=0.385 certified 1.000 vs 1.000
|
| 527 |
+
A_lzm lzm_s44_g0 noB vs nns: -0.01 [-0.06,+0.05] p=0.385 certified 1.000 vs 1.000
|
| 528 |
+
A_lzres lzres_s42_g0 B vs nns: +0.01 [-0.09,+0.09] p=0.381 certified 1.000 vs 1.000
|
| 529 |
+
A_lzres lzres_s42_g0 noB vs nns: +0.01 [-0.09,+0.09] p=0.381 certified 1.000 vs 1.000
|
| 530 |
+
A_lzres lzres_s43_g0 B vs nns: +0.01 [-0.09,+0.09] p=0.512 certified 1.000 vs 1.000
|
| 531 |
+
A_lzres lzres_s43_g0 noB vs nns: +0.01 [-0.09,+0.09] p=0.512 certified 1.000 vs 1.000
|
| 532 |
+
A_lzres lzres_s44_g0 B vs nns: +0.00 [-0.11,+0.09] p=0.32 certified 1.000 vs 1.000
|
| 533 |
+
A_lzres lzres_s44_g0 noB vs nns: +0.00 [-0.11,+0.09] p=0.32 certified 1.000 vs 1.000
|
| 534 |
+
A_phys phys_s42_g0 B vs nns: +0.21 [+0.03,+0.47] p=6.41e-05 certified 1.000 vs 1.000
|
| 535 |
+
A_phys phys_s42_g0 noB vs nns: +0.21 [+0.03,+0.47] p=6.41e-05 certified 1.000 vs 1.000
|
| 536 |
+
A_phys phys_s43_g0 B vs nns: +0.22 [+0.00,+0.40] p=0.000276 certified 1.000 vs 1.000
|
| 537 |
+
A_phys phys_s43_g0 noB vs nns: +0.22 [+0.00,+0.40] p=0.000276 certified 1.000 vs 1.000
|
| 538 |
+
A_phys phys_s44_g0 B vs nns: +0.07 [+0.00,+0.26] p=0.00539 certified 1.000 vs 1.000
|
| 539 |
+
A_phys phys_s44_g0 noB vs nns: +0.07 [+0.00,+0.26] p=0.00539 certified 1.000 vs 1.000
|
| 540 |
+
|
| 541 |
+
=== lvl2.0 K=8 oracle=strong2 (median oracle J 0.555) ===
|
| 542 |
+
arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired
|
| 543 |
+
base_s0 1.000 0.933 0.833 0.87 | 1.000 1.000 0.96 16.5 0.07
|
| 544 |
+
base_s1 1.000 0.950 0.817 0.89 | 1.000 1.000 0.89 16.4 0.05
|
| 545 |
+
base_s2 1.000 0.950 0.833 0.87 | 1.000 1.000 0.97 16.4 0.05
|
| 546 |
+
lz_s42_g0 1.000 1.000 1.000 1.33 | 1.000 1.000 1.33 16.0 0.00
|
| 547 |
+
lz_s42_g1 1.000 1.000 1.000 1.33 | 1.000 1.000 1.33 16.0 0.00
|
| 548 |
+
lz_s42_g2 1.000 1.000 1.000 1.32 | 1.000 1.000 1.32 16.0 0.00
|
| 549 |
+
lz_s43_g0 1.000 1.000 1.000 1.39 | 1.000 1.000 1.39 16.0 0.00
|
| 550 |
+
lz_s44_g0 1.000 1.000 1.000 1.23 | 1.000 1.000 1.23 16.0 0.00
|
| 551 |
+
lzm_s42_g0 1.000 1.000 1.000 1.26 | 1.000 1.000 1.26 16.0 0.00
|
| 552 |
+
lzm_s43_g0 1.000 1.000 1.000 1.21 | 1.000 1.000 1.21 16.0 0.00
|
| 553 |
+
lzm_s44_g0 1.000 1.000 1.000 1.06 | 1.000 1.000 1.06 16.0 0.00
|
| 554 |
+
phys_s42_g0 1.000 1.000 1.000 1.35 | 1.000 1.000 1.35 16.0 0.00
|
| 555 |
+
phys_s43_g0 1.000 1.000 1.000 1.38 | 1.000 1.000 1.38 16.0 0.00
|
| 556 |
+
phys_s44_g0 1.000 1.000 1.000 1.28 | 1.000 1.000 1.28 16.0 0.00
|
| 557 |
+
nn 0.967 0.950 0.950 4.00 | 1.000 1.000 4.00 16.4 0.05
|
| 558 |
+
idw 0.950 0.933 0.933 4.40 | 1.000 1.000 4.40 16.6 0.07
|
| 559 |
+
nns 1.000 1.000 1.000 1.14 | 1.000 1.000 1.14 16.0 0.00
|
| 560 |
+
dls 1.000 1.000 1.000 1.26 | 1.000 1.000 1.26 16.0 0.00
|
| 561 |
+
lzres_s42_g0 1.000 1.000 1.000 1.36 | 1.000 1.000 1.36 16.0 0.00
|
| 562 |
+
lzres_s43_g0 1.000 1.000 1.000 1.36 | 1.000 1.000 1.36 16.0 0.00
|
| 563 |
+
lzres_s44_g0 1.000 1.000 1.000 1.38 | 1.000 1.000 1.38 16.0 0.00
|
| 564 |
+
|
| 565 |
+
-- paired Wilcoxon on regret, model minus best retrieval (B: best=nns; noB: best=nns); negative = model better; [95% CI of paired median]
|
| 566 |
+
base base_s0 B vs nns: -0.08 [-0.24,+0.21] p=0.768 certified 1.000 vs 1.000
|
| 567 |
+
base base_s0 noB vs nns: -0.08 [-0.20,+0.29] p=0.606 certified 0.833 vs 1.000
|
| 568 |
+
base base_s1 B vs nns: -0.10 [-0.26,+0.13] p=0.831 certified 1.000 vs 1.000
|
| 569 |
+
base base_s1 noB vs nns: -0.10 [-0.31,+0.13] p=0.814 certified 0.817 vs 1.000
|
| 570 |
+
base base_s2 B vs nns: -0.12 [-0.30,+0.22] p=0.729 certified 1.000 vs 1.000
|
| 571 |
+
base base_s2 noB vs nns: -0.12 [-0.30,+0.07] p=0.941 certified 0.833 vs 1.000
|
| 572 |
+
A_lz lz_s42_g0 B vs nns: +0.08 [+0.02,+0.33] p=0.0113 certified 1.000 vs 1.000
|
| 573 |
+
A_lz lz_s42_g0 noB vs nns: +0.08 [+0.02,+0.33] p=0.0113 certified 1.000 vs 1.000
|
| 574 |
+
A_lz lz_s42_g1 B vs nns: +0.07 [+0.02,+0.18] p=0.0178 certified 1.000 vs 1.000
|
| 575 |
+
A_lz lz_s42_g1 noB vs nns: +0.07 [+0.02,+0.18] p=0.0178 certified 1.000 vs 1.000
|
| 576 |
+
A_lz lz_s42_g2 B vs nns: +0.08 [+0.00,+0.28] p=0.0151 certified 1.000 vs 1.000
|
| 577 |
+
A_lz lz_s42_g2 noB vs nns: +0.08 [+0.00,+0.28] p=0.0151 certified 1.000 vs 1.000
|
| 578 |
+
A_lz lz_s43_g0 B vs nns: +0.22 [+0.07,+0.40] p=0.00103 certified 1.000 vs 1.000
|
| 579 |
+
A_lz lz_s43_g0 noB vs nns: +0.22 [+0.07,+0.40] p=0.00103 certified 1.000 vs 1.000
|
| 580 |
+
A_lz lz_s44_g0 B vs nns: +0.07 [-0.02,+0.35] p=0.0346 certified 1.000 vs 1.000
|
| 581 |
+
A_lz lz_s44_g0 noB vs nns: +0.07 [-0.02,+0.35] p=0.0346 certified 1.000 vs 1.000
|
| 582 |
+
A_lzm lzm_s42_g0 B vs nns: +0.09 [+0.02,+0.30] p=0.00254 certified 1.000 vs 1.000
|
| 583 |
+
A_lzm lzm_s42_g0 noB vs nns: +0.09 [+0.02,+0.30] p=0.00254 certified 1.000 vs 1.000
|
| 584 |
+
A_lzm lzm_s43_g0 B vs nns: +0.09 [+0.01,+0.21] p=0.0059 certified 1.000 vs 1.000
|
| 585 |
+
A_lzm lzm_s43_g0 noB vs nns: +0.09 [+0.01,+0.21] p=0.0059 certified 1.000 vs 1.000
|
| 586 |
+
A_lzm lzm_s44_g0 B vs nns: -0.01 [-0.06,+0.05] p=0.385 certified 1.000 vs 1.000
|
| 587 |
+
A_lzm lzm_s44_g0 noB vs nns: -0.01 [-0.06,+0.05] p=0.385 certified 1.000 vs 1.000
|
| 588 |
+
A_lzres lzres_s42_g0 B vs nns: +0.01 [-0.09,+0.09] p=0.377 certified 1.000 vs 1.000
|
| 589 |
+
A_lzres lzres_s42_g0 noB vs nns: +0.01 [-0.09,+0.09] p=0.377 certified 1.000 vs 1.000
|
| 590 |
+
A_lzres lzres_s43_g0 B vs nns: +0.01 [-0.09,+0.09] p=0.512 certified 1.000 vs 1.000
|
| 591 |
+
A_lzres lzres_s43_g0 noB vs nns: +0.01 [-0.09,+0.09] p=0.512 certified 1.000 vs 1.000
|
| 592 |
+
A_lzres lzres_s44_g0 B vs nns: +0.00 [-0.11,+0.09] p=0.317 certified 1.000 vs 1.000
|
| 593 |
+
A_lzres lzres_s44_g0 noB vs nns: +0.00 [-0.11,+0.09] p=0.317 certified 1.000 vs 1.000
|
| 594 |
+
A_phys phys_s42_g0 B vs nns: +0.21 [+0.03,+0.48] p=6.41e-05 certified 1.000 vs 1.000
|
| 595 |
+
A_phys phys_s42_g0 noB vs nns: +0.21 [+0.03,+0.48] p=6.41e-05 certified 1.000 vs 1.000
|
| 596 |
+
A_phys phys_s43_g0 B vs nns: +0.22 [+0.00,+0.40] p=0.000284 certified 1.000 vs 1.000
|
| 597 |
+
A_phys phys_s43_g0 noB vs nns: +0.22 [+0.00,+0.40] p=0.000284 certified 1.000 vs 1.000
|
| 598 |
+
A_phys phys_s44_g0 B vs nns: +0.07 [+0.00,+0.26] p=0.00527 certified 1.000 vs 1.000
|
| 599 |
+
A_phys phys_s44_g0 noB vs nns: +0.07 [+0.00,+0.26] p=0.00527 certified 1.000 vs 1.000
|
| 600 |
+
|
| 601 |
+
=== pilotfar K=8 oracle=old (median oracle J 65.827) ===
|
| 602 |
+
arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired
|
| 603 |
+
base_s0 1.000 0.033 0.017 -6.49 | 1.000 1.000 -8.50 23.7 0.97
|
| 604 |
+
base_s1 1.000 0.000 0.000 -6.74 | 1.000 1.000 -8.82 24.0 1.00
|
| 605 |
+
base_s2 1.000 0.017 0.000 -6.68 | 1.000 1.000 -8.84 23.9 0.98
|
| 606 |
+
lz_s42_g0 1.000 1.000 1.000 4.97 | 1.000 1.000 4.97 16.0 0.00
|
| 607 |
+
lz_s42_g1 1.000 1.000 1.000 4.77 | 1.000 1.000 4.77 16.0 0.00
|
| 608 |
+
lz_s42_g2 1.000 1.000 1.000 4.82 | 1.000 1.000 4.82 16.0 0.00
|
| 609 |
+
lz_s43_g0 1.000 1.000 1.000 5.97 | 1.000 1.000 5.97 16.0 0.00
|
| 610 |
+
lz_s44_g0 1.000 1.000 1.000 5.17 | 1.000 1.000 5.17 16.0 0.00
|
| 611 |
+
lzm_s42_g0 1.000 1.000 1.000 9.28 | 1.000 1.000 9.28 16.0 0.00
|
| 612 |
+
lzm_s43_g0 1.000 1.000 1.000 10.32 | 1.000 1.000 10.32 16.0 0.00
|
| 613 |
+
lzm_s44_g0 1.000 1.000 1.000 7.79 | 1.000 1.000 7.79 16.0 0.00
|
| 614 |
+
phys_s42_g0 1.000 1.000 1.000 6.76 | 1.000 1.000 6.76 16.0 0.00
|
| 615 |
+
phys_s43_g0 1.000 1.000 1.000 5.36 | 1.000 1.000 5.36 16.0 0.00
|
| 616 |
+
phys_s44_g0 1.000 1.000 1.000 5.39 | 1.000 1.000 5.39 16.0 0.00
|
| 617 |
+
nn 0.717 0.517 0.517 69.64 | 1.000 1.000 55.28 19.9 0.48
|
| 618 |
+
idw 0.383 0.233 0.233 1000.00 | 1.000 1.000 58.52 22.1 0.77
|
| 619 |
+
nns 1.000 1.000 1.000 0.04 | 1.000 1.000 0.04 16.0 0.00
|
| 620 |
+
dls 1.000 1.000 1.000 -1.73 | 1.000 1.000 -1.73 16.0 0.00
|
| 621 |
+
lzres_s42_g0 1.000 1.000 1.000 -4.00 | 1.000 1.000 -4.00 16.0 0.00
|
| 622 |
+
lzres_s43_g0 1.000 1.000 1.000 -3.89 | 1.000 1.000 -3.89 16.0 0.00
|
| 623 |
+
lzres_s44_g0 1.000 1.000 1.000 -4.61 | 1.000 1.000 -4.61 16.0 0.00
|
| 624 |
+
|
| 625 |
+
-- paired Wilcoxon on regret, model minus best retrieval (B: best=dls; noB: best=dls); negative = model better; [95% CI of paired median]
|
| 626 |
+
base base_s0 B vs dls: -6.44 [-6.84,-6.01] p=1.63e-11 certified 1.000 vs 1.000
|
| 627 |
+
base base_s0 noB vs dls: -5.28 [-6.10,-4.50] p=2.95e-06 certified 0.017 vs 1.000
|
| 628 |
+
base base_s1 B vs dls: -6.37 [-6.85,-6.07] p=1.63e-11 certified 1.000 vs 1.000
|
| 629 |
+
base base_s1 noB vs dls: -5.14 [-5.94,-3.69] p=2.95e-06 certified 0.000 vs 1.000
|
| 630 |
+
base base_s2 B vs dls: -6.53 [-6.92,-6.25] p=1.63e-11 certified 1.000 vs 1.000
|
| 631 |
+
base base_s2 noB vs dls: -5.25 [-5.94,-3.91] p=1.38e-08 certified 0.000 vs 1.000
|
| 632 |
+
A_lz lz_s42_g0 B vs dls: +7.02 [+6.04,+8.04] p=1.63e-11 certified 1.000 vs 1.000
|
| 633 |
+
A_lz lz_s42_g0 noB vs dls: +7.02 [+6.04,+8.04] p=1.63e-11 certified 1.000 vs 1.000
|
| 634 |
+
A_lz lz_s42_g1 B vs dls: +6.55 [+5.94,+7.39] p=2.01e-10 certified 1.000 vs 1.000
|
| 635 |
+
A_lz lz_s42_g1 noB vs dls: +6.55 [+5.94,+7.39] p=2.01e-10 certified 1.000 vs 1.000
|
| 636 |
+
A_lz lz_s42_g2 B vs dls: +6.98 [+5.97,+8.41] p=1.63e-11 certified 1.000 vs 1.000
|
| 637 |
+
A_lz lz_s42_g2 noB vs dls: +6.98 [+5.97,+8.41] p=1.63e-11 certified 1.000 vs 1.000
|
| 638 |
+
A_lz lz_s43_g0 B vs dls: +7.87 [+6.93,+8.48] p=1.63e-11 certified 1.000 vs 1.000
|
| 639 |
+
A_lz lz_s43_g0 noB vs dls: +7.87 [+6.93,+8.48] p=1.63e-11 certified 1.000 vs 1.000
|
| 640 |
+
A_lz lz_s44_g0 B vs dls: +6.99 [+5.70,+7.82] p=1.63e-11 certified 1.000 vs 1.000
|
| 641 |
+
A_lz lz_s44_g0 noB vs dls: +6.99 [+5.70,+7.82] p=1.63e-11 certified 1.000 vs 1.000
|
| 642 |
+
A_lzm lzm_s42_g0 B vs dls: +11.32 [+10.43,+12.32] p=1.63e-11 certified 1.000 vs 1.000
|
| 643 |
+
A_lzm lzm_s42_g0 noB vs dls: +11.32 [+10.43,+12.32] p=1.63e-11 certified 1.000 vs 1.000
|
| 644 |
+
A_lzm lzm_s43_g0 B vs dls: +12.82 [+11.50,+13.35] p=1.63e-11 certified 1.000 vs 1.000
|
| 645 |
+
A_lzm lzm_s43_g0 noB vs dls: +12.82 [+11.50,+13.35] p=1.63e-11 certified 1.000 vs 1.000
|
| 646 |
+
A_lzm lzm_s44_g0 B vs dls: +10.05 [+8.95,+10.80] p=1.63e-11 certified 1.000 vs 1.000
|
| 647 |
+
A_lzm lzm_s44_g0 noB vs dls: +10.05 [+8.95,+10.80] p=1.63e-11 certified 1.000 vs 1.000
|
| 648 |
+
A_lzres lzres_s42_g0 B vs dls: -1.58 [-2.07,-1.26] p=1.99e-11 certified 1.000 vs 1.000
|
| 649 |
+
A_lzres lzres_s42_g0 noB vs dls: -1.58 [-2.07,-1.26] p=1.99e-11 certified 1.000 vs 1.000
|
| 650 |
+
A_lzres lzres_s43_g0 B vs dls: -1.64 [-1.95,-1.16] p=1.71e-11 certified 1.000 vs 1.000
|
| 651 |
+
A_lzres lzres_s43_g0 noB vs dls: -1.64 [-1.95,-1.16] p=1.71e-11 certified 1.000 vs 1.000
|
| 652 |
+
A_lzres lzres_s44_g0 B vs dls: -2.33 [-2.94,-1.61] p=1.63e-11 certified 1.000 vs 1.000
|
| 653 |
+
A_lzres lzres_s44_g0 noB vs dls: -2.33 [-2.94,-1.61] p=1.63e-11 certified 1.000 vs 1.000
|
| 654 |
+
A_phys phys_s42_g0 B vs dls: +8.78 [+8.12,+9.94] p=1.63e-11 certified 1.000 vs 1.000
|
| 655 |
+
A_phys phys_s42_g0 noB vs dls: +8.78 [+8.12,+9.94] p=1.63e-11 certified 1.000 vs 1.000
|
| 656 |
+
A_phys phys_s43_g0 B vs dls: +7.69 [+6.84,+8.44] p=1.63e-11 certified 1.000 vs 1.000
|
| 657 |
+
A_phys phys_s43_g0 noB vs dls: +7.69 [+6.84,+8.44] p=1.63e-11 certified 1.000 vs 1.000
|
| 658 |
+
A_phys phys_s44_g0 B vs dls: +7.51 [+6.74,+8.55] p=1.63e-11 certified 1.000 vs 1.000
|
| 659 |
+
A_phys phys_s44_g0 noB vs dls: +7.51 [+6.74,+8.55] p=1.63e-11 certified 1.000 vs 1.000
|
| 660 |
+
|
| 661 |
+
=== pilotfar K=8 oracle=strong2 (median oracle J 59.001) ===
|
| 662 |
+
arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired
|
| 663 |
+
base_s0 1.000 0.033 0.017 2.90 | 1.000 1.000 1.03 23.7 0.97
|
| 664 |
+
base_s1 1.000 0.000 0.000 2.88 | 1.000 1.000 0.80 24.0 1.00
|
| 665 |
+
base_s2 1.000 0.017 0.000 2.89 | 1.000 1.000 0.86 23.9 0.98
|
| 666 |
+
lz_s42_g0 1.000 1.000 1.000 16.67 | 1.000 1.000 16.67 16.0 0.00
|
| 667 |
+
lz_s42_g1 1.000 1.000 1.000 16.33 | 1.000 1.000 16.33 16.0 0.00
|
| 668 |
+
lz_s42_g2 1.000 1.000 1.000 16.60 | 1.000 1.000 16.60 16.0 0.00
|
| 669 |
+
lz_s43_g0 1.000 1.000 1.000 17.50 | 1.000 1.000 17.50 16.0 0.00
|
| 670 |
+
lz_s44_g0 1.000 1.000 1.000 16.66 | 1.000 1.000 16.66 16.0 0.00
|
| 671 |
+
lzm_s42_g0 1.000 1.000 1.000 21.71 | 1.000 1.000 21.71 16.0 0.00
|
| 672 |
+
lzm_s43_g0 1.000 1.000 1.000 22.71 | 1.000 1.000 22.71 16.0 0.00
|
| 673 |
+
lzm_s44_g0 1.000 1.000 1.000 19.87 | 1.000 1.000 19.87 16.0 0.00
|
| 674 |
+
phys_s42_g0 1.000 1.000 1.000 18.81 | 1.000 1.000 18.81 16.0 0.00
|
| 675 |
+
phys_s43_g0 1.000 1.000 1.000 17.31 | 1.000 1.000 17.31 16.0 0.00
|
| 676 |
+
phys_s44_g0 1.000 1.000 1.000 17.47 | 1.000 1.000 17.47 16.0 0.00
|
| 677 |
+
nn 0.717 0.517 0.517 86.30 | 1.000 1.000 71.01 19.9 0.48
|
| 678 |
+
idw 0.383 0.233 0.233 1000.00 | 1.000 1.000 76.42 22.1 0.77
|
| 679 |
+
nns 1.000 1.000 1.000 11.35 | 1.000 1.000 11.35 16.0 0.00
|
| 680 |
+
dls 1.000 1.000 1.000 7.95 | 1.000 1.000 7.95 16.0 0.00
|
| 681 |
+
lzres_s42_g0 1.000 1.000 1.000 6.45 | 1.000 1.000 6.45 16.0 0.00
|
| 682 |
+
lzres_s43_g0 1.000 1.000 1.000 6.55 | 1.000 1.000 6.55 16.0 0.00
|
| 683 |
+
lzres_s44_g0 1.000 1.000 1.000 5.76 | 1.000 1.000 5.76 16.0 0.00
|
| 684 |
+
|
| 685 |
+
-- paired Wilcoxon on regret, model minus best retrieval (B: best=dls; noB: best=dls); negative = model better; [95% CI of paired median]
|
| 686 |
+
base base_s0 B vs dls: -7.02 [-7.55,-6.70] p=1.63e-11 certified 1.000 vs 1.000
|
| 687 |
+
base base_s0 noB vs dls: -5.73 [-6.77,-4.94] p=2.95e-06 certified 0.017 vs 1.000
|
| 688 |
+
base base_s1 B vs dls: -7.08 [-7.56,-6.66] p=1.63e-11 certified 1.000 vs 1.000
|
| 689 |
+
base base_s1 noB vs dls: -5.62 [-6.48,-4.07] p=2.95e-06 certified 0.000 vs 1.000
|
| 690 |
+
base base_s2 B vs dls: -7.26 [-7.65,-6.94] p=1.63e-11 certified 1.000 vs 1.000
|
| 691 |
+
base base_s2 noB vs dls: -5.85 [-6.45,-4.35] p=1.32e-08 certified 0.000 vs 1.000
|
| 692 |
+
A_lz lz_s42_g0 B vs dls: +7.78 [+6.68,+8.89] p=1.63e-11 certified 1.000 vs 1.000
|
| 693 |
+
A_lz lz_s42_g0 noB vs dls: +7.78 [+6.68,+8.89] p=1.63e-11 certified 1.000 vs 1.000
|
| 694 |
+
A_lz lz_s42_g1 B vs dls: +7.15 [+6.54,+8.15] p=2.01e-10 certified 1.000 vs 1.000
|
| 695 |
+
A_lz lz_s42_g1 noB vs dls: +7.15 [+6.54,+8.15] p=2.01e-10 certified 1.000 vs 1.000
|
| 696 |
+
A_lz lz_s42_g2 B vs dls: +7.76 [+6.52,+9.10] p=1.63e-11 certified 1.000 vs 1.000
|
| 697 |
+
A_lz lz_s42_g2 noB vs dls: +7.76 [+6.52,+9.10] p=1.63e-11 certified 1.000 vs 1.000
|
| 698 |
+
A_lz lz_s43_g0 B vs dls: +8.80 [+7.65,+9.50] p=1.63e-11 certified 1.000 vs 1.000
|
| 699 |
+
A_lz lz_s43_g0 noB vs dls: +8.80 [+7.65,+9.50] p=1.63e-11 certified 1.000 vs 1.000
|
| 700 |
+
A_lz lz_s44_g0 B vs dls: +7.70 [+6.37,+8.69] p=1.63e-11 certified 1.000 vs 1.000
|
| 701 |
+
A_lz lz_s44_g0 noB vs dls: +7.70 [+6.37,+8.69] p=1.63e-11 certified 1.000 vs 1.000
|
| 702 |
+
A_lzm lzm_s42_g0 B vs dls: +12.52 [+11.49,+13.81] p=1.63e-11 certified 1.000 vs 1.000
|
| 703 |
+
A_lzm lzm_s42_g0 noB vs dls: +12.52 [+11.49,+13.81] p=1.63e-11 certified 1.000 vs 1.000
|
| 704 |
+
A_lzm lzm_s43_g0 B vs dls: +14.22 [+12.55,+14.89] p=1.63e-11 certified 1.000 vs 1.000
|
| 705 |
+
A_lzm lzm_s43_g0 noB vs dls: +14.22 [+12.55,+14.89] p=1.63e-11 certified 1.000 vs 1.000
|
| 706 |
+
A_lzm lzm_s44_g0 B vs dls: +11.04 [+9.86,+12.05] p=1.63e-11 certified 1.000 vs 1.000
|
| 707 |
+
A_lzm lzm_s44_g0 noB vs dls: +11.04 [+9.86,+12.05] p=1.63e-11 certified 1.000 vs 1.000
|
| 708 |
+
A_lzres lzres_s42_g0 B vs dls: -1.77 [-2.27,-1.40] p=1.9e-11 certified 1.000 vs 1.000
|
| 709 |
+
A_lzres lzres_s42_g0 noB vs dls: -1.77 [-2.27,-1.40] p=1.9e-11 certified 1.000 vs 1.000
|
| 710 |
+
A_lzres lzres_s43_g0 B vs dls: -1.81 [-2.16,-1.27] p=1.71e-11 certified 1.000 vs 1.000
|
| 711 |
+
A_lzres lzres_s43_g0 noB vs dls: -1.81 [-2.16,-1.27] p=1.71e-11 certified 1.000 vs 1.000
|
| 712 |
+
A_lzres lzres_s44_g0 B vs dls: -2.58 [-3.23,-1.80] p=1.63e-11 certified 1.000 vs 1.000
|
| 713 |
+
A_lzres lzres_s44_g0 noB vs dls: -2.58 [-3.23,-1.80] p=1.63e-11 certified 1.000 vs 1.000
|
| 714 |
+
A_phys phys_s42_g0 B vs dls: +9.73 [+8.91,+10.99] p=1.63e-11 certified 1.000 vs 1.000
|
| 715 |
+
A_phys phys_s42_g0 noB vs dls: +9.73 [+8.91,+10.99] p=1.63e-11 certified 1.000 vs 1.000
|
| 716 |
+
A_phys phys_s43_g0 B vs dls: +8.56 [+7.48,+9.38] p=1.63e-11 certified 1.000 vs 1.000
|
| 717 |
+
A_phys phys_s43_g0 noB vs dls: +8.56 [+7.48,+9.38] p=1.63e-11 certified 1.000 vs 1.000
|
| 718 |
+
A_phys phys_s44_g0 B vs dls: +8.31 [+7.46,+9.46] p=1.63e-11 certified 1.000 vs 1.000
|
| 719 |
+
A_phys phys_s44_g0 noB vs dls: +8.31 [+7.46,+9.46] p=1.63e-11 certified 1.000 vs 1.000
|
| 720 |
+
|
results/physics_gen/pendulum_physics_gen_v2.txt
ADDED
|
@@ -0,0 +1,188 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== lvl0 K=8, oracle median J 1.351; value = mean over the group's runs of per-plant regret %, then median over plants ===
|
| 2 |
+
group n cov certified(no B) | with B: certified regret_med calls | no B regret_med
|
| 3 |
+
shipped 3 1.000 1.000 | 1.000 1.12 16.0 | 1.12
|
| 4 |
+
dimless-lz, orig labels 3 1.000 1.000 | 1.000 1.16 16.0 | 1.16
|
| 5 |
+
dimless-lzres, orig labels 3 1.000 1.000 | 1.000 1.29 16.0 | 1.29
|
| 6 |
+
tuned phys, multi-A (published) 3 1.000 1.000 | 1.000 0.34 16.0 | 0.34
|
| 7 |
+
tuned phys, multi-B (published) 3 1.000 0.894 | 1.000 0.20 16.0 | 0.19
|
| 8 |
+
A: phys, multi-A (new metric) 3 1.000 0.994 | 1.000 0.33 16.0 | 0.33
|
| 9 |
+
A: dimless-lz, multi-A 3 1.000 1.000 | 1.000 0.33 16.0 | 0.33
|
| 10 |
+
A: dimless-lzres, multi-A 3 1.000 1.000 | 1.000 0.34 16.0 | 0.34
|
| 11 |
+
A: dimless-lz, multi-B 3 1.000 0.783 | 1.000 0.23 16.0 | 0.21
|
| 12 |
+
nn (orig bank) 1 1.000 1.000 | 1.000 0.48 16.0 | 0.48
|
| 13 |
+
nns (orig) 1 1.000 1.000 | 1.000 1.12 16.0 | 1.12
|
| 14 |
+
dls (orig) 1 1.000 1.000 | 1.000 1.23 16.0 | 1.23
|
| 15 |
+
nn (multi-A) 1 1.000 0.983 | 1.000 0.30 16.0 | 0.30
|
| 16 |
+
nns (multi-A) 1 1.000 1.000 | 1.000 0.37 16.0 | 0.37
|
| 17 |
+
dls (multi-A) 1 1.000 1.000 | 1.000 0.30 16.0 | 0.30
|
| 18 |
+
nn (multi-B) 1 1.000 1.000 | 1.000 0.21 16.0 | 0.21
|
| 19 |
+
nns (multi-B) 1 1.000 1.000 | 1.000 0.11 16.0 | 0.11
|
| 20 |
+
dls (multi-B) 1 1.000 1.000 | 1.000 0.14 16.0 | 0.14
|
| 21 |
+
-- best retrieval arm with B: nns (multi-B); best original-bank arm: nn (orig bank). paired Wilcoxon, group minus retrieval (neg = model better) [95% CI of paired median], p
|
| 22 |
+
shipped B vs nns (multi-B) : +0.90 [+0.73,+1.14] p=5.41e-11
|
| 23 |
+
shipped B vs nn (orig bank): +0.66 [+0.58,+0.88] p=1.44e-10
|
| 24 |
+
dimless-lz, orig labels B vs nns (multi-B) : +0.89 [+0.71,+1.06] p=1.71e-11
|
| 25 |
+
dimless-lz, orig labels B vs nn (orig bank): +0.69 [+0.56,+0.84] p=1.9e-11
|
| 26 |
+
dimless-lzres, orig labels B vs nns (multi-B) : +1.10 [+1.02,+1.21] p=1.71e-11
|
| 27 |
+
dimless-lzres, orig labels B vs nn (orig bank): +0.93 [+0.77,+0.97] p=1.9e-11
|
| 28 |
+
tuned phys, multi-A (published) B vs nns (multi-B) : +0.10 [+0.05,+0.17] p=2.37e-06
|
| 29 |
+
tuned phys, multi-A (published) B vs nn (orig bank): -0.06 [-0.15,+0.01] p=0.0126
|
| 30 |
+
tuned phys, multi-B (published) B vs nns (multi-B) : +0.05 [+0.02,+0.07] p=6.21e-05
|
| 31 |
+
tuned phys, multi-B (published) B vs nn (orig bank): -0.09 [-0.14,-0.01] p=0.000276
|
| 32 |
+
A: phys, multi-A (new metric) B vs nns (multi-B) : +0.10 [+0.05,+0.15] p=4.09e-07
|
| 33 |
+
A: phys, multi-A (new metric) B vs nn (orig bank): -0.03 [-0.13,+0.03] p=0.0238
|
| 34 |
+
A: dimless-lz, multi-A B vs nns (multi-B) : +0.14 [+0.05,+0.18] p=1.42e-06
|
| 35 |
+
A: dimless-lz, multi-A B vs nn (orig bank): +0.02 [-0.14,+0.05] p=0.122
|
| 36 |
+
A: dimless-lzres, multi-A B vs nns (multi-B) : +0.11 [+0.06,+0.22] p=2.97e-09
|
| 37 |
+
A: dimless-lzres, multi-A B vs nn (orig bank): +0.02 [-0.08,+0.07] p=0.233
|
| 38 |
+
A: dimless-lz, multi-B B vs nns (multi-B) : +0.07 [+0.03,+0.11] p=0.000207
|
| 39 |
+
A: dimless-lz, multi-B B vs nn (orig bank): -0.02 [-0.17,+0.02] p=0.0069
|
| 40 |
+
-- factor decomposition (paired, with B): data = tuned-phys-multi-A vs shipped; recipe = A-phys(new metric) vs published tuned; physics = A-lz vs A-phys; trend = A-lzres vs A-lz; multiplicity = multi-B lz vs multi-A lz
|
| 41 |
+
tuned phys, multi-A (published) - shipped : -0.83 [-1.12,-0.57] p=8.42e-11
|
| 42 |
+
A: phys, multi-A (new metric) - tuned phys, multi-A (published) : +0.00 [-0.01,+0.02] p=0.401
|
| 43 |
+
A: dimless-lz, multi-A - A: phys, multi-A (new metric) : +0.00 [-0.03,+0.04] p=0.977
|
| 44 |
+
A: dimless-lzres, multi-A - A: dimless-lz, multi-A : +0.04 [+0.01,+0.06] p=0.354
|
| 45 |
+
A: dimless-lz, multi-B - A: dimless-lz, multi-A : -0.04 [-0.07,-0.02] p=0.000133
|
| 46 |
+
dimless-lz, orig labels - shipped : +0.08 [-0.07,+0.18] p=0.462
|
| 47 |
+
|
| 48 |
+
=== lvl1.5 K=8, oracle median J 0.656; value = mean over the group's runs of per-plant regret %, then median over plants ===
|
| 49 |
+
group n cov certified(no B) | with B: certified regret_med calls | no B regret_med
|
| 50 |
+
shipped 3 1.000 0.978 | 1.000 1.00 16.1 | 0.95
|
| 51 |
+
dimless-lz, orig labels 3 1.000 1.000 | 1.000 1.33 16.0 | 1.33
|
| 52 |
+
dimless-lzres, orig labels 3 1.000 1.000 | 1.000 1.38 16.0 | 1.38
|
| 53 |
+
tuned phys, multi-A (published) 3 1.000 0.783 | 1.000 1.06 16.8 | 0.90
|
| 54 |
+
tuned phys, multi-B (published) 3 1.000 0.689 | 1.000 0.84 16.6 | 0.77
|
| 55 |
+
A: phys, multi-A (new metric) 3 1.000 0.806 | 1.000 1.12 16.7 | 0.66
|
| 56 |
+
A: dimless-lz, multi-A 3 1.000 1.000 | 1.000 1.26 16.0 | 1.26
|
| 57 |
+
A: dimless-lzres, multi-A 3 1.000 0.989 | 1.000 0.97 16.0 | 0.97
|
| 58 |
+
A: dimless-lz, multi-B 3 1.000 0.622 | 1.000 0.82 16.0 | 0.70
|
| 59 |
+
nn (orig bank) 1 1.000 1.000 | 1.000 1.18 16.0 | 1.18
|
| 60 |
+
nns (orig) 1 1.000 1.000 | 1.000 1.15 16.0 | 1.15
|
| 61 |
+
dls (orig) 1 1.000 1.000 | 1.000 1.22 16.0 | 1.22
|
| 62 |
+
nn (multi-A) 1 1.000 1.000 | 1.000 1.21 16.0 | 1.21
|
| 63 |
+
nns (multi-A) 1 1.000 1.000 | 1.000 0.92 16.0 | 0.92
|
| 64 |
+
dls (multi-A) 1 1.000 0.983 | 1.000 0.99 16.0 | 0.99
|
| 65 |
+
nn (multi-B) 1 1.000 0.933 | 1.000 1.25 16.0 | 1.22
|
| 66 |
+
nns (multi-B) 1 1.000 1.000 | 1.000 0.59 16.0 | 0.59
|
| 67 |
+
dls (multi-B) 1 1.000 0.983 | 1.000 0.56 16.0 | 0.56
|
| 68 |
+
-- best retrieval arm with B: dls (multi-B); best original-bank arm: nns (orig). paired Wilcoxon, group minus retrieval (neg = model better) [95% CI of paired median], p
|
| 69 |
+
shipped B vs dls (multi-B) : +0.10 [-0.04,+0.88] p=0.0113
|
| 70 |
+
shipped B vs nns (orig) : +0.05 [-0.09,+0.21] p=0.227
|
| 71 |
+
dimless-lz, orig labels B vs dls (multi-B) : +0.58 [+0.23,+1.69] p=3.75e-05
|
| 72 |
+
dimless-lz, orig labels B vs nns (orig) : +0.24 [+0.15,+0.70] p=1.79e-08
|
| 73 |
+
dimless-lzres, orig labels B vs dls (multi-B) : +0.86 [+0.43,+1.35] p=1.07e-05
|
| 74 |
+
dimless-lzres, orig labels B vs nns (orig) : +0.16 [+0.08,+0.31] p=0.0047
|
| 75 |
+
tuned phys, multi-A (published) B vs dls (multi-B) : +0.14 [+0.05,+0.81] p=0.000974
|
| 76 |
+
tuned phys, multi-A (published) B vs nns (orig) : -0.20 [-0.37,-0.01] p=0.149
|
| 77 |
+
tuned phys, multi-B (published) B vs dls (multi-B) : -0.01 [-0.11,+0.17] p=0.335
|
| 78 |
+
tuned phys, multi-B (published) B vs nns (orig) : -0.22 [-0.49,-0.07] p=0.000602
|
| 79 |
+
A: phys, multi-A (new metric) B vs dls (multi-B) : +0.10 [+0.02,+0.58] p=0.004
|
| 80 |
+
A: phys, multi-A (new metric) B vs nns (orig) : -0.24 [-0.48,+0.03] p=0.0701
|
| 81 |
+
A: dimless-lz, multi-A B vs dls (multi-B) : +0.07 [-0.02,+0.45] p=0.00267
|
| 82 |
+
A: dimless-lz, multi-A B vs nns (orig) : -0.33 [-0.71,+0.07] p=0.00242
|
| 83 |
+
A: dimless-lzres, multi-A B vs dls (multi-B) : +0.01 [-0.01,+0.18] p=0.0773
|
| 84 |
+
A: dimless-lzres, multi-A B vs nns (orig) : -0.05 [-0.60,+0.10] p=0.0059
|
| 85 |
+
A: dimless-lz, multi-B B vs dls (multi-B) : +0.03 [-0.05,+0.25] p=0.00998
|
| 86 |
+
A: dimless-lz, multi-B B vs nns (orig) : -0.33 [-0.80,+0.01] p=0.00254
|
| 87 |
+
-- factor decomposition (paired, with B): data = tuned-phys-multi-A vs shipped; recipe = A-phys(new metric) vs published tuned; physics = A-lz vs A-phys; trend = A-lzres vs A-lz; multiplicity = multi-B lz vs multi-A lz
|
| 88 |
+
tuned phys, multi-A (published) - shipped : -0.25 [-0.78,-0.10] p=0.00492
|
| 89 |
+
A: phys, multi-A (new metric) - tuned phys, multi-A (published) : -0.03 [-0.12,+0.03] p=0.466
|
| 90 |
+
A: dimless-lz, multi-A - A: phys, multi-A (new metric) : -0.01 [-0.11,+0.33] p=0.774
|
| 91 |
+
A: dimless-lzres, multi-A - A: dimless-lz, multi-A : -0.08 [-0.39,+0.00] p=0.000809
|
| 92 |
+
A: dimless-lz, multi-B - A: dimless-lz, multi-A : -0.04 [-0.08,+0.02] p=0.0393
|
| 93 |
+
dimless-lz, orig labels - shipped : +0.36 [+0.18,+0.58] p=7.88e-06
|
| 94 |
+
|
| 95 |
+
=== lvl2.0 K=8, oracle median J 0.554; value = mean over the group's runs of per-plant regret %, then median over plants ===
|
| 96 |
+
group n cov certified(no B) | with B: certified regret_med calls | no B regret_med
|
| 97 |
+
shipped 3 1.000 0.828 | 1.000 1.01 16.4 | 0.94
|
| 98 |
+
dimless-lz, orig labels 3 1.000 1.000 | 1.000 1.37 16.0 | 1.37
|
| 99 |
+
dimless-lzres, orig labels 3 1.000 1.000 | 1.000 1.36 16.0 | 1.36
|
| 100 |
+
tuned phys, multi-A (published) 3 1.000 0.644 | 1.000 1.56 17.2 | 1.05
|
| 101 |
+
tuned phys, multi-B (published) 3 1.000 0.478 | 1.000 1.52 17.1 | 0.83
|
| 102 |
+
A: phys, multi-A (new metric) 3 1.000 0.650 | 1.000 1.92 17.4 | 1.20
|
| 103 |
+
A: dimless-lz, multi-A 3 1.000 1.000 | 1.000 1.25 16.0 | 1.25
|
| 104 |
+
A: dimless-lzres, multi-A 3 1.000 1.000 | 1.000 1.12 16.0 | 1.12
|
| 105 |
+
A: dimless-lz, multi-B 3 1.000 0.494 | 1.000 1.14 16.0 | 0.77
|
| 106 |
+
nn (orig bank) 1 0.967 0.950 | 1.000 4.00 16.4 | 4.00
|
| 107 |
+
nns (orig) 1 1.000 1.000 | 1.000 1.17 16.0 | 1.17
|
| 108 |
+
dls (orig) 1 1.000 1.000 | 1.000 1.31 16.0 | 1.31
|
| 109 |
+
nn (multi-A) 1 0.950 0.950 | 1.000 2.23 16.4 | 2.23
|
| 110 |
+
nns (multi-A) 1 1.000 1.000 | 1.000 1.29 16.0 | 1.29
|
| 111 |
+
dls (multi-A) 1 1.000 0.967 | 1.000 1.26 16.0 | 1.26
|
| 112 |
+
nn (multi-B) 1 0.950 0.883 | 1.000 4.04 16.5 | 4.04
|
| 113 |
+
nns (multi-B) 1 1.000 0.983 | 1.000 1.21 16.0 | 1.21
|
| 114 |
+
dls (multi-B) 1 1.000 1.000 | 1.000 1.04 16.0 | 1.04
|
| 115 |
+
-- best retrieval arm with B: dls (multi-B); best original-bank arm: nns (orig). paired Wilcoxon, group minus retrieval (neg = model better) [95% CI of paired median], p
|
| 116 |
+
shipped B vs dls (multi-B) : +0.01 [-0.24,+0.37] p=0.236
|
| 117 |
+
shipped B vs nns (orig) : -0.02 [-0.25,+0.24] p=0.527
|
| 118 |
+
dimless-lz, orig labels B vs dls (multi-B) : -0.19 [-0.43,+0.36] p=0.365
|
| 119 |
+
dimless-lz, orig labels B vs nns (orig) : +0.11 [+0.02,+0.37] p=0.0059
|
| 120 |
+
dimless-lzres, orig labels B vs dls (multi-B) : -0.19 [-0.42,+0.16] p=0.522
|
| 121 |
+
dimless-lzres, orig labels B vs nns (orig) : +0.01 [-0.09,+0.09] p=0.393
|
| 122 |
+
tuned phys, multi-A (published) B vs dls (multi-B) : +0.20 [-0.05,+0.55] p=0.0145
|
| 123 |
+
tuned phys, multi-A (published) B vs nns (orig) : +0.34 [-0.02,+0.78] p=0.0192
|
| 124 |
+
tuned phys, multi-B (published) B vs dls (multi-B) : +0.00 [-0.32,+0.36] p=0.139
|
| 125 |
+
tuned phys, multi-B (published) B vs nns (orig) : +0.15 [-0.09,+0.47] p=0.16
|
| 126 |
+
A: phys, multi-A (new metric) B vs dls (multi-B) : +0.21 [-0.06,+0.72] p=0.0047
|
| 127 |
+
A: phys, multi-A (new metric) B vs nns (orig) : +0.49 [+0.01,+0.97] p=0.0084
|
| 128 |
+
A: dimless-lz, multi-A B vs dls (multi-B) : +0.07 [-0.11,+0.17] p=0.245
|
| 129 |
+
A: dimless-lz, multi-A B vs nns (orig) : +0.11 [-0.10,+0.22] p=0.702
|
| 130 |
+
A: dimless-lzres, multi-A B vs dls (multi-B) : -0.01 [-0.18,+0.02] p=0.0773
|
| 131 |
+
A: dimless-lzres, multi-A B vs nns (orig) : +0.03 [-0.09,+0.18] p=0.566
|
| 132 |
+
A: dimless-lz, multi-B B vs dls (multi-B) : +0.05 [-0.01,+0.27] p=0.0283
|
| 133 |
+
A: dimless-lz, multi-B B vs nns (orig) : +0.19 [+0.07,+0.38] p=0.551
|
| 134 |
+
-- factor decomposition (paired, with B): data = tuned-phys-multi-A vs shipped; recipe = A-phys(new metric) vs published tuned; physics = A-lz vs A-phys; trend = A-lzres vs A-lz; multiplicity = multi-B lz vs multi-A lz
|
| 135 |
+
tuned phys, multi-A (published) - shipped : +0.05 [-0.33,+0.38] p=0.611
|
| 136 |
+
A: phys, multi-A (new metric) - tuned phys, multi-A (published) : +0.01 [-0.06,+0.28] p=0.306
|
| 137 |
+
A: dimless-lz, multi-A - A: phys, multi-A (new metric) : -0.13 [-0.96,+0.07] p=0.0167
|
| 138 |
+
A: dimless-lzres, multi-A - A: dimless-lz, multi-A : -0.06 [-0.32,+0.04] p=0.0164
|
| 139 |
+
A: dimless-lz, multi-B - A: dimless-lz, multi-A : +0.04 [-0.02,+0.16] p=0.101
|
| 140 |
+
dimless-lz, orig labels - shipped : +0.18 [+0.04,+0.49] p=0.448
|
| 141 |
+
|
| 142 |
+
=== pilotfar K=8, oracle median J 58.949; value = mean over the group's runs of per-plant regret %, then median over plants ===
|
| 143 |
+
group n cov certified(no B) | with B: certified regret_med calls | no B regret_med
|
| 144 |
+
shipped 3 1.000 0.006 | 1.000 1.36 23.9 | 3.83
|
| 145 |
+
dimless-lz, orig labels 3 1.000 1.000 | 1.000 17.39 16.0 | 17.39
|
| 146 |
+
dimless-lzres, orig labels 3 1.000 1.000 | 1.000 6.42 16.0 | 6.42
|
| 147 |
+
tuned phys, multi-A (published) 3 0.994 0.078 | 1.000 1.28 22.3 | 5.56
|
| 148 |
+
tuned phys, multi-B (published) 3 0.967 0.056 | 1.000 1.68 23.2 | 21.58
|
| 149 |
+
A: phys, multi-A (new metric) 3 0.994 0.033 | 1.000 1.35 22.7 | 2.64
|
| 150 |
+
A: dimless-lz, multi-A 3 1.000 1.000 | 1.000 3.28 16.0 | 3.28
|
| 151 |
+
A: dimless-lzres, multi-A 3 1.000 0.983 | 1.000 0.76 16.0 | 0.74
|
| 152 |
+
A: dimless-lz, multi-B 3 1.000 0.372 | 1.000 0.84 16.0 | 0.42
|
| 153 |
+
nn (orig bank) 1 0.717 0.517 | 1.000 71.99 19.9 | 86.49
|
| 154 |
+
nns (orig) 1 1.000 1.000 | 1.000 11.67 16.0 | 11.67
|
| 155 |
+
dls (orig) 1 1.000 1.000 | 1.000 8.37 16.0 | 8.37
|
| 156 |
+
nn (multi-A) 1 0.233 0.150 | 1.000 77.29 22.8 | 1000.00
|
| 157 |
+
nns (multi-A) 1 1.000 1.000 | 1.000 8.65 16.0 | 8.65
|
| 158 |
+
dls (multi-A) 1 1.000 1.000 | 1.000 1.84 16.0 | 1.84
|
| 159 |
+
nn (multi-B) 1 0.233 0.150 | 1.000 11.75 22.8 | 1000.00
|
| 160 |
+
nns (multi-B) 1 1.000 0.067 | 1.000 1.71 16.0 | 0.36
|
| 161 |
+
dls (multi-B) 1 1.000 1.000 | 1.000 0.55 16.0 | 0.55
|
| 162 |
+
-- best retrieval arm with B: dls (multi-B); best original-bank arm: dls (orig). paired Wilcoxon, group minus retrieval (neg = model better) [95% CI of paired median], p
|
| 163 |
+
shipped B vs dls (multi-B) : +0.81 [+0.52,+0.99] p=4.43e-09
|
| 164 |
+
shipped B vs dls (orig) : -7.08 [-7.37,-6.67] p=1.63e-11
|
| 165 |
+
dimless-lz, orig labels B vs dls (multi-B) : +16.88 [+15.86,+18.34] p=1.63e-11
|
| 166 |
+
dimless-lz, orig labels B vs dls (orig) : +8.19 [+7.11,+8.95] p=1.63e-11
|
| 167 |
+
dimless-lzres, orig labels B vs dls (multi-B) : +5.87 [+5.56,+6.08] p=1.63e-11
|
| 168 |
+
dimless-lzres, orig labels B vs dls (orig) : -2.04 [-2.60,-1.47] p=1.71e-11
|
| 169 |
+
tuned phys, multi-A (published) B vs dls (multi-B) : +0.76 [+0.47,+1.36] p=2.85e-08
|
| 170 |
+
tuned phys, multi-A (published) B vs dls (orig) : -7.33 [-7.76,-6.14] p=2.62e-08
|
| 171 |
+
tuned phys, multi-B (published) B vs dls (multi-B) : +1.13 [+0.51,+1.77] p=2.71e-09
|
| 172 |
+
tuned phys, multi-B (published) B vs dls (orig) : -6.67 [-7.28,-6.03] p=9.07e-08
|
| 173 |
+
A: phys, multi-A (new metric) B vs dls (multi-B) : +0.78 [+0.32,+1.47] p=6.02e-08
|
| 174 |
+
A: phys, multi-A (new metric) B vs dls (orig) : -6.96 [-7.90,-6.41] p=2.59e-09
|
| 175 |
+
A: dimless-lz, multi-A B vs dls (multi-B) : +2.66 [+2.05,+3.10] p=1.63e-11
|
| 176 |
+
A: dimless-lz, multi-A B vs dls (orig) : -5.55 [-6.18,-5.21] p=1.63e-11
|
| 177 |
+
A: dimless-lzres, multi-A B vs dls (multi-B) : +0.11 [+0.02,+0.26] p=0.000636
|
| 178 |
+
A: dimless-lzres, multi-A B vs dls (orig) : -7.47 [-8.10,-7.04] p=1.63e-11
|
| 179 |
+
A: dimless-lz, multi-B B vs dls (multi-B) : +0.33 [+0.18,+0.52] p=1.06e-06
|
| 180 |
+
A: dimless-lz, multi-B B vs dls (orig) : -7.44 [-7.74,-6.97] p=1.71e-11
|
| 181 |
+
-- factor decomposition (paired, with B): data = tuned-phys-multi-A vs shipped; recipe = A-phys(new metric) vs published tuned; physics = A-lz vs A-phys; trend = A-lzres vs A-lz; multiplicity = multi-B lz vs multi-A lz
|
| 182 |
+
tuned phys, multi-A (published) - shipped : -0.03 [-0.34,+0.29] p=0.466
|
| 183 |
+
A: phys, multi-A (new metric) - tuned phys, multi-A (published) : +0.18 [-0.33,+0.55] p=0.918
|
| 184 |
+
A: dimless-lz, multi-A - A: phys, multi-A (new metric) : +1.88 [+0.98,+2.26] p=0.00204
|
| 185 |
+
A: dimless-lzres, multi-A - A: dimless-lz, multi-A : -2.14 [-2.69,-1.82] p=2.1e-11
|
| 186 |
+
A: dimless-lz, multi-B - A: dimless-lz, multi-A : -2.06 [-2.70,-1.13] p=1.07e-08
|
| 187 |
+
dimless-lz, orig labels - shipped : +15.83 [+15.19,+17.57] p=1.63e-11
|
| 188 |
+
|