diff --git a/results/baselines/ablation_k.txt b/results/baselines/ablation_k.txt new file mode 100644 index 0000000000000000000000000000000000000000..aa3ae4f0ccda4274c4e6e001dbb15e7a9d61149c --- /dev/null +++ b/results/baselines/ablation_k.txt @@ -0,0 +1,10 @@ +SISO K-sample ablation — pooled 5 seeds × 60 plants + +K plant-stabilisable rate (≥1 of K) 95% CI +1 0.633 [0.580, 0.687] +2 0.713 [0.660, 0.763] +4 0.747 [0.697, 0.793] +8 0.770 [0.723, 0.817] +16 0.787 [0.740, 0.833] + +lift K=1 -> K=16: +0.153 (0.63 -> 0.79) — the generative distribution's value. diff --git a/results/baselines/ablation_k_run.log b/results/baselines/ablation_k_run.log new file mode 100644 index 0000000000000000000000000000000000000000..47d915974cd0858d9c7b809eac906eb2f9aa1c8b --- /dev/null +++ b/results/baselines/ablation_k_run.log @@ -0,0 +1,17 @@ +[seed 0] done +[seed 1] done +[seed 2] done +[seed 3] done +[seed 4] done + +=== K ABLATION === +SISO K-sample ablation — pooled 5 seeds × 60 plants + +K plant-stabilisable rate (≥1 of K) 95% CI +1 0.633 [0.580, 0.687] +2 0.713 [0.660, 0.763] +4 0.747 [0.697, 0.793] +8 0.770 [0.723, 0.817] +16 0.787 [0.740, 0.833] + +lift K=1 -> K=16: +0.153 (0.63 -> 0.79) — the generative distribution's value. diff --git a/results/baselines/ablation_steps_siso.txt b/results/baselines/ablation_steps_siso.txt new file mode 100644 index 0000000000000000000000000000000000000000..0e90276f2c55b6e397a7088537f9bb642d2002b1 --- /dev/null +++ b/results/baselines/ablation_steps_siso.txt @@ -0,0 +1,9 @@ +=== SISO DDIM steps sweep (plants seed 0, N=60, K=8, sampler seeds (0, 1, 2), eta=0, device=cuda) === +model fixed (full-data SISO); verifier/x0 = retrieval_baselines.py siso (x0 seed 777); oracle = 60-step Adam refinement of the DDPM@500 seed-0 K=8 winner (as bank_density; negative regret possible). +regret 'common' = median over plants covered by EVERY arm's seed-0 draw (n=46); 'own' = over each draw's own covered plants. ms/ctrl = wall time per sampled controller, batch 480. +arm cov per-samp reg common% reg own% ms/ctrl McNemar vs ddpm500 (n10,n01,p) +ddim5 0.817+-0.017 0.682+-0.009 1.25+-9.75 -0.46+-5.83 0.039+-0.029 (3, 0) p=0.25 +ddim20 0.800+-0.017 0.658+-0.006 5.76+-13.42 5.71+-13.39 0.086+-0.000 (2, 1) p=1 +ddim100 0.794+-0.025 0.634+-0.014 14.75+-9.27 13.04+-7.10 0.461+-0.005 (0, 1) p=1 +ddim500 0.789+-0.010 0.622+-0.019 19.11+-10.09 16.61+-7.12 2.292+-0.010 (1, 1) p=1 +ddpm500 0.794+-0.010 0.632+-0.013 25.27+-11.55 23.72+-10.49 1.997+-0.005 (0, 0) p=1 diff --git a/results/baselines/bank_density_siso.txt b/results/baselines/bank_density_siso.txt new file mode 100644 index 0000000000000000000000000000000000000000..d86ee0d715c51914ca029ab80719638696411f8a --- /dev/null +++ b/results/baselines/bank_density_siso.txt @@ -0,0 +1,68 @@ +=== BANK-DENSITY SWEEP, SISO (held-out draws: seeds [0, 4321], N=[60, 60], pooled N=120) === +MODEL SIDE IS FIXED: full-data checkpoint, NOT retrained per bank fraction (the pendulum sweep retrained; this one does not). +bank = seeded random subset of training PLANTS (all stored controllers of the chosen plants); fractions (0.05, 0.1, 0.25, 0.5, 1.0); subset seeds (0, 1, 2) (f=1.0: single bank); +model draws: sampler seeds (0, 1, 2); McNemar/regret reference = model seed 0. Verifier/x0/N as retrieval_baselines.py siso; K=8 and K=1 (K=1 = first candidate). +oracle: shared per plant = 60-step Adam refinement (generalize.py protocol) of the model seed-0 K=8 winner (the SAME oracle yesterday's SISO model arm used), floored by that winner's J; arms that beat it show NEGATIVE regret; regret undefined where the model seed-0 draw is uncovered (never inside the common set) +regret % = (J_best - J_oracle)/J_oracle on the COMMON success set (plants where BOTH the arm and the model seed-0 draw are covered at that K); label 'cond.'. +coverage 95% CI: bootstrap (5000) over pooled plants of subset-seed 0. McNemar: exact, paired on pooled plants vs model seed 0, reported for each subset seed (min..max p). + +----- K=8 ----- MODEL coverage 0.719 +- 0.004 (3 draws), seed-0 CI [0.633,0.792], per-sample 0.566, cond. regret med 15.86%, gen 2.34 ms/ctrl +per-draw model coverage (seed 0/1/2): seed 0: 0.78/0.80/0.80; seed 4321: 0.65/0.63/0.65 + arm=retrieval + f cov mean+-sd CI(ss0) per-samp dCov(ret-model) McNemar p(ss0..) reg_med% cond model reg% same set n_common NN dist + 0.05 0.531+-0.037 [0.408,0.592] 0.531 -0.189 3e-08..3.1e-05 86.89 16.55 63 0.0083 + 0.10 0.461+-0.021 [0.375,0.558] 0.461 -0.258 1.2e-10..7.5e-09 85.70 16.68 55 0.0072 + 0.25 0.478+-0.010 [0.383,0.558] 0.478 -0.242 1.9e-09..1.5e-08 77.99 14.06 57 0.0051 + 0.50 0.456+-0.022 [0.375,0.550] 0.456 -0.264 5.8e-11..3.7e-09 80.47 15.09 54 0.0041 + 1.00 0.450+-0.000 [0.358,0.542] 0.450 -0.269 4.7e-10..4.7e-10 73.24 14.42 54 0.0035 + arm=blend + f cov mean+-sd CI(ss0) per-samp dCov(ret-model) McNemar p(ss0..) reg_med% cond model reg% same set n_common NN dist + 0.05 0.531+-0.037 [0.408,0.592] 0.531 -0.189 3e-08..3.1e-05 86.89 16.55 63 0.0083 + 0.10 0.461+-0.021 [0.375,0.558] 0.461 -0.258 1.2e-10..7.5e-09 85.70 16.68 55 0.0072 + 0.25 0.478+-0.010 [0.383,0.558] 0.478 -0.242 1.9e-09..1.5e-08 77.99 14.06 57 0.0051 + 0.50 0.456+-0.022 [0.375,0.550] 0.456 -0.264 5.8e-11..3.7e-09 80.47 15.09 54 0.0041 + 1.00 0.450+-0.000 [0.358,0.542] 0.450 -0.269 4.7e-10..4.7e-10 73.24 14.42 54 0.0035 + arm=distinct + f cov mean+-sd CI(ss0) per-samp dCov(ret-model) McNemar p(ss0..) reg_med% cond model reg% same set n_common NN dist + 0.05 0.689+-0.004 [0.608,0.775] 0.484 -0.031 0.12..0.38 -4.80 15.32 82 0.0083 + 0.10 0.689+-0.004 [0.608,0.767] 0.489 -0.031 0.12..0.38 -5.45 15.28 82 0.0072 + 0.25 0.686+-0.004 [0.608,0.775] 0.474 -0.033 0.12..0.25 -10.99 15.32 82 0.0051 + 0.50 0.686+-0.004 [0.608,0.775] 0.489 -0.033 0.12..0.25 -7.75 15.32 82 0.0041 + 1.00 0.683+-0.000 [0.600,0.767] 0.483 -0.036 0.12..0.12 4.31 15.28 82 0.0035 + +----- K=1 ----- MODEL coverage 0.600 +- 0.020 (3 draws), seed-0 CI [0.508,0.683], per-sample 0.600, cond. regret med 735.20%, gen 2.34 ms/ctrl +per-draw model coverage (seed 0/1/2): seed 0: 0.65/0.67/0.70; seed 4321: 0.55/0.48/0.55 + arm=retrieval (blend K=1 == retrieval K=1 by construction) + f cov mean+-sd CI(ss0) per-samp dCov(ret-model) McNemar p(ss0..) reg_med% cond model reg% same set n_common NN dist + 0.05 0.531+-0.037 [0.408,0.592] 0.531 -0.069 0.023..0.84 119.81 955.04 55 0.0083 + 0.10 0.461+-0.021 [0.375,0.558] 0.461 -0.139 0.00032..0.016 119.21 840.13 48 0.0072 + 0.25 0.478+-0.010 [0.383,0.558] 0.478 -0.122 0.0059..0.024 88.08 1118.00 50 0.0051 + 0.50 0.456+-0.022 [0.375,0.550] 0.456 -0.144 0.00019..0.0041 82.60 758.08 49 0.0041 + 1.00 0.450+-0.000 [0.358,0.542] 0.450 -0.150 0.0014..0.0014 74.12 1131.46 48 0.0035 + +----- retrieval coverage K=8 vs K=1 per f (mean over subset seeds; pooled draws) ----- + f K=1 K=8 distinct K=8 + 0.05 0.531 0.531 0.689 + 0.10 0.461 0.461 0.689 + 0.25 0.478 0.478 0.686 + 0.50 0.456 0.456 0.686 + 1.00 0.450 0.450 0.683 + +----- smallest f at which retrieval MATCHES the model ----- +rule (fixed before running): mean retrieval coverage >= model coverage - 0.05 AND pooled McNemar p >= 0.05 for >= 2 of 3 subset seeds (f=1.0: its single bank); 'never' if no f qualifies. + siso arm=retrieval K=8: never (none of the swept f) + siso arm=blend K=8: never (none of the swept f) + siso arm=distinct K=8: f=0.05 + siso arm=retrieval K=1: never (none of the swept f) + +----- NN distance vs f (normalized conditioning space, mean over queries; mean over subset seeds) ----- + f=0.05: draw0: 0.0098, draw4321: 0.0069 + f=0.1: draw0: 0.0083, draw4321: 0.0061 + f=0.25: draw0: 0.0060, draw4321: 0.0042 + f=0.5: draw0: 0.0049, draw4321: 0.0033 + f=1.0: draw0: 0.0042, draw4321: 0.0028 + +ADDENDUM: per-held-out-draw K=8 coverage (mean over subset seeds; model = seed-0 sampler / 3-seed mean) + draw0: model 0.794; retrieval f=0.05: 0.600 f=0.1: 0.533 f=0.25: 0.556 f=0.5: 0.522 f=1.0: 0.517 | distinct f=0.05: 0.772 f=0.1: 0.767 f=0.25: 0.767 f=0.5: 0.767 f=1.0: 0.767 + draw1: model 0.644; retrieval f=0.05: 0.461 f=0.1: 0.389 f=0.25: 0.400 f=0.5: 0.389 f=1.0: 0.383 | distinct f=0.05: 0.606 f=0.1: 0.611 f=0.25: 0.606 f=0.5: 0.606 f=1.0: 0.600 + retrieval f=1.0: plants where all 8 candidates share one verdict: 1.000 (pooled N=120) diff --git a/results/baselines/distinct_retrieval_prereg.txt b/results/baselines/distinct_retrieval_prereg.txt new file mode 100644 index 0000000000000000000000000000000000000000..f9da7e0cda213bf25a2af007e60bf9a0e556c3fa --- /dev/null +++ b/results/baselines/distinct_retrieval_prereg.txt @@ -0,0 +1,27 @@ +=== PRE-REGISTERED distinct-plant retrieval vs diffusion, SISO, N=300 (plant seeds (100, 200, 300, 400, 500) x 60), full bank f=1.0, coverage only === +model: full-data checkpoint, K<=16 samples/plant, sampler seeds 0/1/2 (seed 0 = paired reference; K-prefix of the 16 samples). verifier/x0 as bank_density sweep. + +PRIMARY K=8: model(seed0) 0.773 distinct-first 0.750 diff +0.023 95% CI [+0.003,+0.047] McNemar model-only 9, retrieval-only 2, p=0.0654 => VERDICT: TIE + sampler seed 1: model 0.780 McNemar 12-3 p=0.0352 + sampler seed 2: model 0.787 McNemar 13-2 p=0.00739 + model 3-seed mean K=8: 0.780 +- 0.005 + +per plant-seed K=8 coverage (model seed0 / distinct-first): + 100: 0.800 / 0.783 + 200: 0.800 / 0.783 + 300: 0.833 / 0.800 + 400: 0.733 / 0.717 + 500: 0.700 / 0.667 + +SECONDARY coverage vs K (model seed0 [3-seed mean]; distinct-first; distinct-random mean of 3 seeds; McNemar model vs distinct-first) + K model s0 model mean dist-first dist-random McNemar (m-only,r-only,p) + 1 0.617 0.622 0.493 0.493+-0.000 (60,23,5.97e-05) + 2 0.707 0.706 0.660 0.660+-0.000 (24,10,0.0243) + 4 0.747 0.756 0.743 0.743+-0.000 (8,7,1) + 8 0.773 0.780 0.750 0.750+-0.000 (9,2,0.0654) + 16 0.787 0.793 0.753 0.753+-0.000 (12,2,0.0129) +random-stored K=8 (seed 0 draw) vs model: model-only 9, retrieval-only 2, p=0.0654 + +MIMO (distinct == plain retrieval, one controller per plant), f=1.0, K=8, from the sweep npz: + draw 9999: model 1.000 retrieval 0.992 McNemar 1-0 p=1 + draw 4242: model 0.992 retrieval 0.992 McNemar 0-0 p=1 diff --git a/results/baselines/fixed_controller_pendulum.txt b/results/baselines/fixed_controller_pendulum.txt new file mode 100644 index 0000000000000000000000000000000000000000..0be6218080d91301d4d53cd775d0265c165f5e77 --- /dev/null +++ b/results/baselines/fixed_controller_pendulum.txt @@ -0,0 +1,19 @@ +=== PENDULUM fixed dimensionless controller (N=40, seed=123 held-out) === + +Justification: theta'' = (g/l) sin(theta) - (b/(m l^2)) omega + u/(m l^2), controller u = -phi(y), y = c1 theta + c2 omega (c FIXED at (1.0, 0.35) for the whole family, as in the amortized model). The natural dimensionless rescaling of phi is by the holding torque m*g*l (the torque needed to hold the pendulum horizontal); a natural TIME scale is sqrt(l/g) (the small-oscillation period / sqrt(g/l) natural frequency), though this codebase does not rescale omega's contribution to y by it (c2 is fixed, not per-plant) -- so the fixed controller below rescales phi by m*g*l only, exactly mirroring how eval_pendulum.strong_oracle's own smart init (2.5*m*l*g*xp) and random-restart shapes ((U[0.5,2.5]*m*l*g)*xp*(|xp|/5+0.3)) are built. + +Config space = SAME as the oracle's restarts: shape in {linear xp, nonlinear xp*(|xp|/5+0.3)}, K_hat in linspace(0.5,2.5,9). Tuned on a FRESH N=200 set (seed=555, disjoint from the held-out test set) by (certified_rate, success_rate); NOT re-tuned per plant (one fixed controller for the whole family). Cost = the paper's ABSOLUTE-weight J = integral (theta^2 + 0.1 omega^2) + u^2 dt (simulate_pendulum_batch default weights) -- not a scale-invariant metric. + +Chosen fixed controller: shape=nonlinear K_hat=2.500 (phi(y) = 2.500 * m*g*l * shape(y)) + +success (empirical stability, N=40): 1.000 +certified (regional LMI, N=40): 1.000 +cost regret vs cached per-plant oracle: median +56.69% p90 +88.86% max +124.15% (n=40) +synthesis time: 81.2865 ms/plant (no per-plant solve -- table lookup + rescale) + +top 5 tuning configs (shape, K_hat, certified_rate, success_rate) on the tuning set: + ('nonlinear', 2.5, 1.0, 1.0) + ('nonlinear', 2.25, 1.0, 1.0) + ('linear', 2.5, 1.0, 1.0) + ('linear', 2.25, 1.0, 1.0) + ('linear', 2.0, 1.0, 1.0) diff --git a/results/baselines/pendulum_bank_density.txt b/results/baselines/pendulum_bank_density.txt new file mode 100644 index 0000000000000000000000000000000000000000..e5f636a245b36111bc8aeeb3b8b2e5145c656f7b --- /dev/null +++ b/results/baselines/pendulum_bank_density.txt @@ -0,0 +1,36 @@ +=== PENDULUM bank-density sweep === +model = diffusion model TRAINED on the f-fraction subset (3 training seeds 42/43/44, + mean +/- sd across training seeds; ONE diffusion sampling seed=0 per model, K=8) +retrieval/interpolation = SAME f-fraction bank (built from the identical seeded subset + train_pendulum_ablation.py used), 3 seeds -> mean +/- sd, K=8 +held-out sets: seed123 (cached oracle), seed321 (fresh oracle, run_pendulum_second_draw) + +--- held-out set: seed123 --- + f arm coverage regret_med % regret_med 95%CI(seed0) + 0.25 model 1.000+-0.000 0.24+-0.22 [+0.41,+0.69] + 0.25 retrieval 1.000+-0.000 0.17+-0.01 [+0.01,+0.44] + 0.25 interpolation 1.000+-0.000 0.22+-0.05 [+0.02,+0.44] + 0.50 model 1.000+-0.000 0.98+-0.16 [+1.08,+1.33] + 0.50 retrieval 1.000+-0.000 0.30+-0.08 [+0.05,+0.41] + 0.50 interpolation 1.000+-0.000 0.30+-0.08 [+0.05,+0.41] + 1.00 model 1.000+-0.000 0.89+-0.21 [+0.93,+1.38] + 1.00 retrieval 1.000+-0.000 0.40+-0.00 [+0.22,+0.53] + 1.00 interpolation 1.000+-0.000 0.40+-0.00 [+0.22,+0.53] + +--- held-out set: seed321 --- + f arm coverage regret_med % regret_med 95%CI(seed0) + 0.25 model 1.000+-0.000 0.26+-0.14 [+0.23,+0.72] + 0.25 retrieval 1.000+-0.000 0.24+-0.01 [+0.06,+0.33] + 0.25 interpolation 1.000+-0.000 0.25+-0.01 [+0.11,+0.37] + 0.50 model 1.000+-0.000 0.91+-0.16 [+0.85,+1.20] + 0.50 retrieval 1.000+-0.000 0.23+-0.06 [+0.04,+0.29] + 0.50 interpolation 1.000+-0.000 0.26+-0.05 [+0.04,+0.29] + 1.00 model 1.000+-0.000 0.87+-0.17 [+0.82,+1.24] + 1.00 retrieval 1.000+-0.000 0.29+-0.00 [+0.16,+0.36] + 1.00 interpolation 1.000+-0.000 0.30+-0.00 [+0.16,+0.42] + +Crossover -- does the model's median regret ever drop BELOW retrieval's at the same f? + seed123: f=0.25: model +0.24% >= retrieval +0.17% (retrieval wins/ties); f=0.50: model +0.98% >= retrieval +0.30% (retrieval wins/ties); f=1.00: model +0.89% >= retrieval +0.40% (retrieval wins/ties) + seed321: f=0.25: model +0.26% >= retrieval +0.24% (retrieval wins/ties); f=0.50: model +0.91% >= retrieval +0.23% (retrieval wins/ties); f=1.00: model +0.87% >= retrieval +0.29% (retrieval wins/ties) + -> model overtakes retrieval's median regret at ANY swept f, on ANY held-out set: False + FINDING: the model never overtakes bank retrieval's cost on this sweep -- more data narrows the gap (higher f typically lowers both arms' regret) but does not flip the ordering within {0.25, 0.5, 1.0}. diff --git a/results/baselines/pendulum_bank_density_diag_a.txt b/results/baselines/pendulum_bank_density_diag_a.txt new file mode 100644 index 0000000000000000000000000000000000000000..3d644bcc01da14eedbdbcdc6db84bb5ea7445c71 --- /dev/null +++ b/results/baselines/pendulum_bank_density_diag_a.txt @@ -0,0 +1,15 @@ +=== PENDULUM bank-density diagnostic (a): retrieval K=1 (single nearest neighbour, no best-of-K selection) vs f === +If K=1 regret falls with f while best-of-8 (pendulum_bank_density.txt) does not, best-of-8's edge at low f is candidate DIVERSITY (a sparse bank's 8 nearest are spread out), not raw proximity -- a dense bank's 8 nearest become near-duplicates and best-of-8 stops gaining. + +--- held-out set: seed123 --- + f K1 regret_med% K1 cov (3-seed mean +/- sd) + 0.25 1.71+-0.16 1.000 + 0.50 1.63+-0.20 1.000 + 1.00 1.84+-0.00 1.000 + +--- held-out set: seed321 --- + f K1 regret_med% K1 cov (3-seed mean +/- sd) + 0.25 1.19+-0.22 1.000 + 0.50 1.42+-0.16 1.000 + 1.00 1.36+-0.00 1.000 + diff --git a/results/baselines/pendulum_bank_density_diag_b_seed123.txt b/results/baselines/pendulum_bank_density_diag_b_seed123.txt new file mode 100644 index 0000000000000000000000000000000000000000..82f554823f27d0d08e07cd469d71e2158ca7f55b --- /dev/null +++ b/results/baselines/pendulum_bank_density_diag_b_seed123.txt @@ -0,0 +1,8 @@ +=== PENDULUM bank-density diagnostic (b): native quality of stored targets (held-out seed=123, N=40) === +For each held-out plant: find its nearest FULL-BANK (f=1.0, 2600 pts) training plant; evaluate that plant's OWN stored controller (spline_raw, as-stored, no re-decoding) ON THAT TRAINING PLANT'S OWN DYNAMICS (not the held-out query); regret vs a FRESH eval_pendulum.strong_oracle solve of that SAME training plant. +39 unique nearest-neighbour training plants among the 40 held-out queries (some held-out plants share a nearest neighbour). + +native target regret: median +1.40% p90 +2.67% max +3.58% (n=40/40) +nearest-neighbour distance (normalized space): median 0.1691 max 0.2201 + +Reading: if this native regret (~how good the DATASET's own targets are, solved once, reused everywhere) sits near retrieval's regret on the held-out query (bank_density.txt), retrieval is essentially reading back near-oracle targets with a small transfer penalty for the query-vs-neighbour distance; if the model's own regret (~0.9% at f=1.0) exceeds this native floor, the model is worse than its own training set accessed directly -- the same finding as Iteration 3 on the DP-target pendulum. diff --git a/results/baselines/retrieval_mimo_seed7777.txt b/results/baselines/retrieval_mimo_seed7777.txt new file mode 100644 index 0000000000000000000000000000000000000000..5aa7305241b284362de1baa7a6f8c2e0371f01c7 --- /dev/null +++ b/results/baselines/retrieval_mimo_seed7777.txt @@ -0,0 +1,41 @@ +=== MIMO retrieval/interpolation baselines (N=120, seed=7777) === +verifier: simulate_closed_loop_mimo_batched rollout (n_ic=4, dt=0.03, duration=12.0, energy_thresh=0.3) +oracle: eval_mimo_regret.py's oracle protocol (fixed generic init 0.3*cap, 150-step batched Adam lr=0.08 through the differentiable MIMO sim, sector-projected) -- arm-independent, computed ONCE, shared by every arm (1187.0 ms/plant) +K_std=8, diffusion seeds=(0, 1, 2) + +--- K=8 --- +arm coverage persamp reg_med% reg_p90% reg_max% ms/plant McNemar(n10,n01,p) Wilcoxon(n,p) +diffusion 0.99 0.94 2.31 9.23 30.13 153.77 (reference) (reference) + (sd across 3 seeds: coverage 0.004, per-samp 0.003) +retrieval 0.99 0.93 2.32 8.05 49.97 0.730 (1,0,1) (118,0.91) + (Wilcoxon n=118 = the COMMON success set: plants where BOTH this arm and diffusion have a finite/stable cost, not either arm's own coverage) +interpolation 0.99 0.93 2.95 15.27 74.15 0.730 (1,0,1) (118,0.049) + (Wilcoxon n=118 = the COMMON success set: plants where BOTH this arm and diffusion have a finite/stable cost, not either arm's own coverage) + +--- K=1 --- +arm coverage persamp reg_med% reg_p90% reg_max% ms/plant McNemar(n10,n01,p) Wilcoxon(n,p) +diffusion 0.93 0.93 10.92 88.52 1045.64 153.77 (reference) (reference) + (sd across 3 seeds: coverage 0.014, per-samp 0.014) +retrieval 0.92 0.92 15.23 262.70 830.97 0.370 (5,5,1) (105,0.11) + (Wilcoxon n=105 = the COMMON success set: plants where BOTH this arm and diffusion have a finite/stable cost, not either arm's own coverage) +interpolation 0.92 0.92 15.23 262.70 830.97 0.370 (5,5,1) (105,0.11) + (Wilcoxon n=105 = the COMMON success set: plants where BOTH this arm and diffusion have a finite/stable cost, not either arm's own coverage) + + +--- addendum (post-hoc from npz, not re-simulated): CIs and pooled 2-draw (N=240) tests, diffusion seed[0] as reference --- +K=8 seed9999 diffusion: coverage 1.000 regret_med +2.32 [+1.63,+2.99] +K=8 seed9999 retrieval: coverage 0.992 regret_med +2.62 [+1.45,+3.45] +K=8 seed7777 diffusion: coverage 0.983 regret_med +2.30 [+1.90,+3.01] +K=8 seed7777 retrieval: coverage 0.992 regret_med +2.32 [+1.67,+3.06] +K=8 pooled McNemar coverage (retrieval-only, diffusion-only)=(1,1) p=1 +K=8 pooled Wilcoxon cost diffusion vs retrieval on common set n=237: median J ratio 1.000 p=0.863 +K=8 seed9999 diffusion coverage per draw: [1.0, 0.992, 1.0] retrieval 0.992 +K=8 seed7777 diffusion coverage per draw: [0.983, 0.992, 0.983] retrieval 0.992 +K=1 seed9999 diffusion: coverage 0.942 regret_med +8.87 [+6.14,+12.50] +K=1 seed9999 retrieval: coverage 0.858 regret_med +13.66 [+10.22,+19.86] +K=1 seed7777 diffusion: coverage 0.917 regret_med +9.85 [+7.63,+13.51] +K=1 seed7777 retrieval: coverage 0.917 regret_med +15.23 [+10.31,+20.11] +K=1 pooled McNemar coverage (retrieval-only, diffusion-only)=(5,15) p=0.0414 +K=1 pooled Wilcoxon cost diffusion vs retrieval on common set n=208: median J ratio 0.982 p=0.0345 +K=1 seed9999 diffusion coverage per draw: [0.942, 0.95, 0.942] retrieval 0.858 +K=1 seed7777 diffusion coverage per draw: [0.917, 0.933, 0.95] retrieval 0.917 diff --git a/results/baselines/retrieval_objective_conditioning.txt b/results/baselines/retrieval_objective_conditioning.txt new file mode 100644 index 0000000000000000000000000000000000000000..5999f1471c2912c3bd47ab971e4f578c61550541 --- /dev/null +++ b/results/baselines/retrieval_objective_conditioning.txt @@ -0,0 +1,129 @@ +RETRIEVAL vs OBJECTIVE-CONDITIONED GENERATION (pendulum) +N=40 plants/draw, rho=(0.7, 0.8, 0.9), K in (8,1), steer S=50 +plant seeds: main=321 (exp_cbar_eval's), second=777; model torch seeds = seed*1000+rho_idx, seeds (0, 1, 2) +retr_b pool M=32. spec tolerance ise<=1.05*rho*ISE_ref (exp_cbar_eval satisfied_mask). + +== Task 1: stream replica of exp_cbar_eval (seed 321) vs STORED results/pendulum/cbar_eval.json == +rho=0.7: bestof8 replica met 1.000 cert 1.000 reg 0.84 | stored met 1.000 cert 1.000 reg 0.83 || single replica met 0.925 cert 0.925 reg 8.83 | stored met 0.950 cert 0.850 reg 8.25 +rho=0.8: bestof8 replica met 1.000 cert 1.000 reg 0.61 | stored met 1.000 cert 1.000 reg 0.66 || single replica met 0.900 cert 0.850 reg 5.83 | stored met 0.900 cert 0.875 reg 6.17 +rho=0.9: bestof8 replica met 1.000 cert 1.000 reg 0.90 | stored met 1.000 cert 1.000 reg 1.51 || single replica met 0.875 cert 0.850 reg 4.26 | stored met 0.850 cert 0.850 reg 5.22 +oracle found per rho: [40, 40, 40] + +--- MAIN DRAW seed 321: K=8 steer=none --- +rows: rho | arm | spec-met | certified | regret% median [95% CI] p90 n (model = mean±sd over 3 seeds; CI/p90 from seed 0) +[0.7 ] model met 1.000±0.000 cert 1.000±0.000 medreg 0.82±0.96 | seed0: 2.17 [ 0.66, 4.42] p90 11.80 n=40 +[0.7 ] retr_a met 1.000 cert 1.000 medreg 1.05 [-0.13, 2.47] p90 9.54 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0793 | p over seeds (McN,Wil): 1,0.079; 1,0.29; 1,0.38 +[0.7 ] retr_b met 1.000 cert 1.000 medreg 1.05 [ 0.16, 2.49] p90 8.38 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0548 | p over seeds (McN,Wil): 1,0.055; 1,0.83; 1,0.16 +[0.7 ] retr_c met 1.000 cert 1.000 medreg 1.11 [-0.58, 2.07] p90 7.96 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.025 | p over seeds (McN,Wil): 1,0.025; 1,0.21; 1,0.17 +[0.8 ] model met 1.000±0.000 cert 1.000±0.000 medreg 0.34±0.42 | seed0: -0.25 [-0.80, 0.98] p90 16.59 n=40 +[0.8 ] retr_a met 1.000 cert 1.000 medreg 0.82 [-0.46, 2.05] p90 6.89 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.444 | p over seeds (McN,Wil): 1,0.44; 1,0.24; 1,0.082 +[0.8 ] retr_b met 1.000 cert 1.000 medreg 0.38 [-0.79, 1.95] p90 9.84 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.334 | p over seeds (McN,Wil): 1,0.33; 1,0.23; 1,0.097 +[0.8 ] retr_c met 1.000 cert 1.000 medreg 1.13 [-0.36, 1.78] p90 6.89 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.627 | p over seeds (McN,Wil): 1,0.63; 1,0.32; 1,0.11 +[0.9 ] model met 1.000±0.000 cert 1.000±0.000 medreg 0.97±0.40 | seed0: 0.50 [ 0.23, 1.31] p90 14.98 n=40 +[0.9 ] retr_a met 1.000 cert 1.000 medreg 0.60 [-0.28, 2.28] p90 9.58 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.347 | p over seeds (McN,Wil): 1,0.35; 1,0.034; 1,0.1 +[0.9 ] retr_b met 1.000 cert 1.000 medreg 0.91 [-0.01, 2.72] p90 9.58 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.745 | p over seeds (McN,Wil): 1,0.74; 1,0.062; 1,0.28 +[0.9 ] retr_c met 1.000 cert 1.000 medreg 0.14 [-0.69, 1.60] p90 8.28 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.109 | p over seeds (McN,Wil): 1,0.11; 1,0.022; 1,0.031 +[pooled] model met 1.000±0.000 cert 1.000±0.000 medreg 0.73±0.11 | seed0: 0.65 [ 0.22, 1.19] p90 14.78 n=120 +[pooled] retr_a met 1.000 cert 1.000 medreg 0.87 [ 0.21, 1.71] p90 9.56 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0556 | p over seeds (McN,Wil): 1,0.056; 1,0.015; 1,0.014 +[pooled] retr_b met 1.000 cert 1.000 medreg 0.75 [ 0.21, 1.41] p90 9.56 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0608 | p over seeds (McN,Wil): 1,0.061; 1,0.066; 1,0.014 +[pooled] retr_c met 1.000 cert 1.000 medreg 0.86 [-0.10, 1.56] p90 8.26 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0149 | p over seeds (McN,Wil): 1,0.015; 1,0.0092; 1,0.0024 + +--- MAIN DRAW seed 321: K=1 steer=none --- +rows: rho | arm | spec-met | certified | regret% median [95% CI] p90 n (model = mean±sd over 3 seeds; CI/p90 from seed 0) +[0.7 ] model met 0.908±0.012 cert 0.850±0.041 medreg 14.40±5.85 | seed0: 22.68 [ 8.02,36.94] p90 63.08 n=34 +[0.7 ] retr_a met 0.700 cert 0.700 medreg 6.76 [ 2.03,12.26] p90 38.55 n=28 | vs model s0: McNemar 2-10 p=0.0386, Wilcoxon p=0.0395 | p over seeds (McN,Wil): 0.039,0.039; 0.057,0.037; 0.012,0.026 +[0.7 ] retr_b met 0.675 cert 0.675 medreg 9.02 [ 2.90,13.68] p90 29.75 n=27 | vs model s0: McNemar 2-11 p=0.0225, Wilcoxon p=0.0384 | p over seeds (McN,Wil): 0.022,0.038; 0.022,0.052; 0.0063,0.038 +[0.7 ] retr_c met 0.900 cert 0.900 medreg 8.51 [ 3.40,13.84] p90 31.05 n=36 | vs model s0: McNemar 3-3 p=1, Wilcoxon p=0.000906 | p over seeds (McN,Wil): 1,0.00091; 1,0.0013; 1,0.0052 +[0.8 ] model met 0.883±0.051 cert 0.842±0.051 medreg 6.74±1.60 | seed0: 6.20 [ 3.96,15.82] p90 78.90 n=34 +[0.8 ] retr_a met 0.775 cert 0.775 medreg 2.20 [ 0.87, 8.33] p90 24.33 n=31 | vs model s0: McNemar 4-8 p=0.388, Wilcoxon p=0.00247 | p over seeds (McN,Wil): 0.39,0.0025; 0.77,0.031; 0.065,0.06 +[0.8 ] retr_b met 0.725 cert 0.725 medreg 9.86 [ 5.64,13.28] p90 29.73 n=29 | vs model s0: McNemar 4-10 p=0.18, Wilcoxon p=0.127 | p over seeds (McN,Wil): 0.18,0.13; 0.39,0.23; 0.012,0.69 +[0.8 ] retr_c met 0.875 cert 0.875 medreg 5.78 [ 1.78,10.76] p90 27.94 n=35 | vs model s0: McNemar 3-3 p=1, Wilcoxon p=0.000398 | p over seeds (McN,Wil): 1,0.0004; 0.73,0.099; 0.38,0.074 +[0.9 ] model met 0.892±0.031 cert 0.833±0.051 medreg 6.74±0.36 | seed0: 6.29 [ 4.54,11.19] p90 55.91 n=33 +[0.9 ] retr_a met 0.650 cert 0.650 medreg 5.75 [ 4.02,17.38] p90 49.79 n=26 | vs model s0: McNemar 2-10 p=0.0386, Wilcoxon p=0.259 | p over seeds (McN,Wil): 0.039,0.26; 0.013,0.07; 0.0074,0.43 +[0.9 ] retr_b met 0.700 cert 0.700 medreg 11.11 [ 4.49,17.55] p90 35.66 n=28 | vs model s0: McNemar 5-11 p=0.21, Wilcoxon p=0.427 | p over seeds (McN,Wil): 0.21,0.43; 0.057,0.14; 0.012,0.8 +[0.9 ] retr_c met 0.825 cert 0.825 medreg 4.87 [ 1.64, 9.74] p90 31.94 n=33 | vs model s0: McNemar 4-5 p=1, Wilcoxon p=0.0564 | p over seeds (McN,Wil): 1,0.056; 0.55,0.015; 0.29,0.067 +[pooled] model met 0.894±0.027 cert 0.842±0.048 medreg 8.03±0.52 | seed0: 8.62 [ 5.44,13.61] p90 64.20 n=101 +[pooled] retr_a met 0.708 cert 0.708 medreg 6.09 [ 3.64, 8.44] p90 39.49 n=85 | vs model s0: McNemar 8-28 p=0.00119, Wilcoxon p=0.000397 | p over seeds (McN,Wil): 0.0012,0.0004; 0.0022,0.00066; 7.4e-06,0.004 +[pooled] retr_b met 0.700 cert 0.700 medreg 9.73 [ 7.33,13.04] p90 32.05 n=84 | vs model s0: McNemar 11-32 p=0.00191, Wilcoxon p=0.00733 | p over seeds (McN,Wil): 0.0019,0.0073; 0.0011,0.0036; 7.7e-07,0.079 +[pooled] retr_c met 0.867 cert 0.867 medreg 6.78 [ 3.92, 9.18] p90 30.74 n=104 | vs model s0: McNemar 10-11 p=1, Wilcoxon p=1.22e-06 | p over seeds (McN,Wil): 1,1.2e-06; 1,2.3e-05; 0.096,0.00013 + +--- MAIN DRAW seed 321: K=8 steer=S=50 --- +rows: rho | arm | spec-met | certified | regret% median [95% CI] p90 n (model = mean±sd over 3 seeds; CI/p90 from seed 0) +[0.7 ] model met 1.000±0.000 cert 1.000±0.000 medreg 0.17±0.05 | seed0: 0.23 [ 0.01, 0.75] p90 4.35 n=40 +[0.7 ] retr_a met 1.000 cert 1.000 medreg -0.13 [-0.48, 0.15] p90 1.27 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0725 | p over seeds (McN,Wil): 1,0.073; 1,3.5e-05; 1,0.0056 +[0.7 ] retr_b met 1.000 cert 1.000 medreg -0.22 [-1.04, 0.14] p90 1.56 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.00726 | p over seeds (McN,Wil): 1,0.0073; 1,0.0018; 1,1.7e-06 +[0.7 ] retr_c met 1.000 cert 1.000 medreg -0.04 [-0.19, 0.23] p90 1.20 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.077 | p over seeds (McN,Wil): 1,0.077; 1,1.7e-05; 1,0.0067 +[0.8 ] model met 1.000±0.000 cert 1.000±0.000 medreg -0.03±0.01 | seed0: -0.02 [-0.06, 0.00] p90 11.62 n=40 +[0.8 ] retr_a met 1.000 cert 1.000 medreg -0.15 [-0.38,-0.08] p90 0.11 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=1.81e-05 | p over seeds (McN,Wil): 1,1.8e-05; 1,5.1e-05; 1,0.00014 +[0.8 ] retr_b met 1.000 cert 1.000 medreg -0.14 [-0.42,-0.07] p90 0.09 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=8.15e-06 | p over seeds (McN,Wil): 1,8.2e-06; 1,3.3e-05; 1,0.0001 +[0.8 ] retr_c met 1.000 cert 1.000 medreg -0.20 [-0.45,-0.10] p90 2.02 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.000603 | p over seeds (McN,Wil): 1,0.0006; 1,0.0013; 1,0.00025 +[0.9 ] model met 1.000±0.000 cert 1.000±0.000 medreg -0.03±0.01 | seed0: -0.02 [-0.09, 0.03] p90 9.95 n=40 +[0.9 ] retr_a met 1.000 cert 1.000 medreg -0.11 [-0.26,-0.06] p90 0.86 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.00558 | p over seeds (McN,Wil): 1,0.0056; 1,0.0012; 1,0.015 +[0.9 ] retr_b met 1.000 cert 1.000 medreg -0.13 [-0.27,-0.04] p90 5.73 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.00305 | p over seeds (McN,Wil): 1,0.0031; 1,0.0016; 1,0.017 +[0.9 ] retr_c met 1.000 cert 1.000 medreg -0.10 [-0.25,-0.07] p90 0.86 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.00726 | p over seeds (McN,Wil): 1,0.0073; 1,0.0014; 1,0.028 +[pooled] model met 1.000±0.000 cert 1.000±0.000 medreg -0.01±0.00 | seed0: -0.00 [-0.03, 0.12] p90 10.09 n=120 +[pooled] retr_a met 1.000 cert 1.000 medreg -0.13 [-0.21,-0.09] p90 1.20 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=4.26e-07 | p over seeds (McN,Wil): 1,4.3e-07; 1,2.3e-10; 1,3.4e-07 +[pooled] retr_b met 1.000 cert 1.000 medreg -0.15 [-0.28,-0.09] p90 1.56 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=1.91e-08 | p over seeds (McN,Wil): 1,1.9e-08; 1,1.1e-08; 1,7.5e-10 +[pooled] retr_c met 1.000 cert 1.000 medreg -0.12 [-0.20,-0.08] p90 1.33 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=4.94e-06 | p over seeds (McN,Wil): 1,4.9e-06; 1,1.4e-09; 1,9.8e-07 + +--- MAIN DRAW seed 321: K=1 steer=S=50 --- +rows: rho | arm | spec-met | certified | regret% median [95% CI] p90 n (model = mean±sd over 3 seeds; CI/p90 from seed 0) +[0.7 ] model met 0.992±0.012 cert 0.942±0.042 medreg 5.05±3.48 | seed0: 9.82 [ 0.63,28.76] p90 51.33 n=36 +[0.7 ] retr_a met 0.925 cert 0.900 medreg 0.75 [ 0.18, 1.64] p90 26.67 n=36 | vs model s0: McNemar 1-3 p=0.625, Wilcoxon p=0.0105 | p over seeds (McN,Wil): 0.62,0.011; 0.25,0.023; 0.25,0.0063 +[0.7 ] retr_b met 0.875 cert 0.850 medreg 1.03 [ 0.37, 4.53] p90 21.92 n=34 | vs model s0: McNemar 1-5 p=0.219, Wilcoxon p=0.0149 | p over seeds (McN,Wil): 0.22,0.015; 0.062,0.029; 0.062,0.015 +[0.7 ] retr_c met 1.000 cert 1.000 medreg 1.13 [ 0.36, 4.11] p90 21.78 n=40 | vs model s0: McNemar 1-0 p=1, Wilcoxon p=0.00185 | p over seeds (McN,Wil): 1,0.0018; 1,0.0051; 1,0.0019 +[0.8 ] model met 0.992±0.012 cert 0.933±0.012 medreg 0.37±0.22 | seed0: 0.48 [ 0.27, 5.73] p90 55.20 n=37 +[0.8 ] retr_a met 0.975 cert 0.975 medreg 0.03 [ 0.00, 0.25] p90 18.99 n=39 | vs model s0: McNemar 1-1 p=1, Wilcoxon p=0.000159 | p over seeds (McN,Wil): 1,0.00016; 1,0.0031; 1,0.17 +[0.8 ] retr_b met 0.850 cert 0.850 medreg 0.43 [-0.04, 1.84] p90 15.47 n=34 | vs model s0: McNemar 1-6 p=0.125, Wilcoxon p=0.0175 | p over seeds (McN,Wil): 0.12,0.017; 0.031,0.058; 0.031,0.66 +[0.8 ] retr_c met 1.000 cert 1.000 medreg 0.14 [ 0.02, 1.23] p90 22.30 n=40 | vs model s0: McNemar 1-0 p=1, Wilcoxon p=3.36e-06 | p over seeds (McN,Wil): 1,3.4e-06; 1,0.051; 1,0.14 +[0.9 ] model met 0.992±0.012 cert 0.950±0.054 medreg 0.54±0.31 | seed0: 0.89 [ 0.20, 1.57] p90 36.20 n=40 +[0.9 ] retr_a met 0.975 cert 0.975 medreg 0.11 [-0.00, 0.61] p90 25.81 n=39 | vs model s0: McNemar 0-1 p=1, Wilcoxon p=0.0372 | p over seeds (McN,Wil): 1,0.037; 1,0.0091; 1,0.47 +[0.9 ] retr_b met 0.925 cert 0.925 medreg 0.18 [ 0.03, 1.11] p90 26.47 n=37 | vs model s0: McNemar 0-3 p=0.25, Wilcoxon p=0.0363 | p over seeds (McN,Wil): 0.25,0.036; 0.62,0.037; 0.25,0.61 +[0.9 ] retr_c met 1.000 cert 1.000 medreg 0.11 [ 0.04, 0.53] p90 26.30 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.00336 | p over seeds (McN,Wil): 1,0.0034; 1,0.012; 1,0.38 +[pooled] model met 0.992±0.007 cert 0.942±0.027 medreg 0.70±0.23 | seed0: 0.97 [ 0.43, 1.96] p90 51.49 n=113 +[pooled] retr_a met 0.958 cert 0.950 medreg 0.18 [ 0.06, 0.46] p90 24.60 n=114 | vs model s0: McNemar 2-5 p=0.453, Wilcoxon p=6.12e-06 | p over seeds (McN,Wil): 0.45,6.1e-06; 0.22,1.3e-05; 0.062,0.003 +[pooled] retr_b met 0.883 cert 0.875 medreg 0.49 [ 0.18, 1.11] p90 23.23 n=105 | vs model s0: McNemar 2-14 p=0.00418, Wilcoxon p=6.01e-05 | p over seeds (McN,Wil): 0.0042,6e-05; 0.00098,0.00026; 0.00012,0.037 +[pooled] retr_c met 1.000 cert 1.000 medreg 0.35 [ 0.14, 0.82] p90 23.85 n=120 | vs model s0: McNemar 2-0 p=0.5, Wilcoxon p=2.01e-08 | p over seeds (McN,Wil): 0.5,2e-08; 1,3.2e-05; 1,0.0015 + +synthesis time per plant (ms; mean over rho; verification NOT included): + model:s0 steer 258.21 + model:s0 synth 89.52 + model:s1 steer 270.89 + model:s1 synth 91.89 + model:s2 steer 267.04 + model:s2 synth 89.57 + retr_a steer 278.56 + retr_a synth 0.08 + retr_b steer 271.39 + retr_b synth 0.09 + retr_c steer 253.12 + retr_c synth 0.14 + +oracle found per rho (draw 777): [40, 40, 40] + +--- SECOND DRAW seed 777: K=8 steer=none --- +rows: rho | arm | spec-met | certified | regret% median [95% CI] p90 n (model = mean±sd over 3 seeds; CI/p90 from seed 0) +[0.7 ] model met 1.000±0.000 cert 1.000±0.000 medreg 1.13±0.34 | seed0: 1.23 [-0.22, 3.18] p90 11.81 n=40 +[0.7 ] retr_a met 1.000 cert 1.000 medreg 0.94 [-0.93, 3.71] p90 12.46 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.952 | p over seeds (McN,Wil): 1,0.95; 1,0.045; 1,0.79 +[0.7 ] retr_b met 1.000 cert 1.000 medreg 1.10 [-0.21, 2.94] p90 9.35 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.32 | p over seeds (McN,Wil): 1,0.32; 1,0.51; 1,0.25 +[0.7 ] retr_c met 1.000 cert 1.000 medreg 0.50 [-0.93, 3.02] p90 11.93 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.847 | p over seeds (McN,Wil): 1,0.85; 1,0.12; 1,0.51 +[0.8 ] model met 1.000±0.000 cert 1.000±0.000 medreg 0.97±0.29 | seed0: 0.62 [ 0.01, 1.74] p90 15.47 n=40 +[0.8 ] retr_a met 1.000 cert 1.000 medreg 0.67 [-0.05, 2.42] p90 5.18 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.347 | p over seeds (McN,Wil): 1,0.35; 1,0.18; 1,0.024 +[0.8 ] retr_b met 1.000 cert 1.000 medreg 1.31 [ 0.23, 2.61] p90 5.59 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.361 | p over seeds (McN,Wil): 1,0.36; 1,0.25; 1,0.15 +[0.8 ] retr_c met 1.000 cert 1.000 medreg 0.44 [-0.16, 1.69] p90 5.43 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.277 | p over seeds (McN,Wil): 1,0.28; 1,0.14; 1,0.0094 +[0.9 ] model met 1.000±0.000 cert 1.000±0.000 medreg 1.05±0.05 | seed0: 1.08 [ 0.46, 2.54] p90 8.63 n=40 +[0.9 ] retr_a met 1.000 cert 1.000 medreg 0.64 [-0.44, 1.57] p90 7.26 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.125 | p over seeds (McN,Wil): 1,0.12; 1,0.094; 1,0.71 +[0.9 ] retr_b met 1.000 cert 1.000 medreg 0.65 [-0.34, 1.66] p90 7.14 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.132 | p over seeds (McN,Wil): 1,0.13; 1,0.16; 1,0.53 +[0.9 ] retr_c met 0.975 cert 0.975 medreg 0.50 [-0.35, 1.24] p90 5.51 n=39 | vs model s0: McNemar 0-1 p=1, Wilcoxon p=0.124 | p over seeds (McN,Wil): 1,0.12; 1,0.1; 1,0.69 +[pooled] model met 1.000±0.000 cert 1.000±0.000 medreg 0.96±0.11 | seed0: 0.92 [ 0.43, 1.59] p90 11.90 n=120 +[pooled] retr_a met 1.000 cert 1.000 medreg 0.80 [ 0.06, 1.32] p90 9.77 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.155 | p over seeds (McN,Wil): 1,0.16; 1,0.61; 1,0.097 +[pooled] retr_b met 1.000 cert 1.000 medreg 0.99 [ 0.14, 1.57] p90 8.49 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0619 | p over seeds (McN,Wil): 1,0.062; 1,0.29; 1,0.062 +[pooled] retr_c met 0.992 cert 0.992 medreg 0.50 [ 0.01, 1.01] p90 9.27 n=119 | vs model s0: McNemar 0-1 p=1, Wilcoxon p=0.102 | p over seeds (McN,Wil): 1,0.1; 1,0.29; 1,0.037 + +synthesis time per plant (ms; mean over rho; verification NOT included): + model:s0 synth 91.34 + model:s1 synth 90.86 + model:s2 synth 91.90 + retr_a synth 0.08 + retr_b synth 0.11 + retr_c synth 0.11 diff --git a/results/baselines/retrieval_pendulum.txt b/results/baselines/retrieval_pendulum.txt new file mode 100644 index 0000000000000000000000000000000000000000..841e8d3b8a2996a0b667e395c343be720582a5bc --- /dev/null +++ b/results/baselines/retrieval_pendulum.txt @@ -0,0 +1,28 @@ +=== PENDULUM retrieval/interpolation baselines (N=40, seed=123) === +verifier: certify_controller (regional LMI) + simulate_pendulum_batch rollout (n_ic=3, dt=0.02, duration=6.0, energy_thresh=0.02, theta0=0.35) +oracle: CACHED results/pendulum/eval.npz (eval_pendulum.strong_oracle, 24 restarts x 100 steps), oracle_J backed out from best_costs/regrets +K_std=8, diffusion seeds=(0, 1, 2) + +--- K=8 --- +arm cert_any stable_any persamp reg_med% reg_p90% reg_max% ms/plant McNemar(n10,n01,p) Wilcoxon(n,p) +diffusion 1.00 1.00 1.00 1.09 2.46 3.36 171.13 (reference) (reference) + (sd across 3 seeds: cert_any 0.000, stable_any 0.000, per-samp 0.001) +retrieval 1.00 1.00 1.00 0.40 0.86 1.70 0.375 (0,0,1) (40,2.8e-07) +interpolation 1.00 1.00 1.00 0.40 0.86 1.70 0.375 (0,0,1) (40,2.8e-07) + +--- K=1 --- +arm cert_any stable_any persamp reg_med% reg_p90% reg_max% ms/plant McNemar(n10,n01,p) Wilcoxon(n,p) +diffusion 1.00 1.00 1.00 1.54 3.59 4.80 171.13 (reference) (reference) + (sd across 3 seeds: cert_any 0.000, stable_any 0.000, per-samp 0.000) +retrieval 1.00 1.00 1.00 1.84 3.70 5.85 0.127 (0,0,1) (40,0.13) +interpolation 1.00 1.00 1.00 1.84 3.70 5.85 0.127 (0,0,1) (40,0.13) + + +--- K=8 bootstrap 95% CIs (addendum, computed post-hoc from saved npz; not re-simulated) --- + diffusion (seed[0]): coverage 95% CI [1.000,1.000] regret_med 95% CI [+0.88,+1.35]% + retrieval: coverage 95% CI [1.000,1.000] regret_med 95% CI [+0.22,+0.53]% + interpolation: coverage 95% CI [1.000,1.000] regret_med 95% CI [+0.22,+0.53]% +--- K=1 bootstrap 95% CIs (addendum, computed post-hoc from saved npz; not re-simulated) --- + diffusion (seed[0]): coverage 95% CI [1.000,1.000] regret_med 95% CI [+1.33,+2.14]% + retrieval: coverage 95% CI [1.000,1.000] regret_med 95% CI [+1.37,+2.64]% + interpolation: coverage 95% CI [1.000,1.000] regret_med 95% CI [+1.37,+2.64]% diff --git a/results/baselines/retrieval_pendulum_seed321.txt b/results/baselines/retrieval_pendulum_seed321.txt new file mode 100644 index 0000000000000000000000000000000000000000..d6f2aa0fd77caad58fce97586fdef1810212f50b --- /dev/null +++ b/results/baselines/retrieval_pendulum_seed321.txt @@ -0,0 +1,29 @@ +=== PENDULUM retrieval/interpolation baselines (N=40, seed=321) === +verifier: certify_controller (regional LMI) + simulate_pendulum_batch rollout (n_ic=3, dt=0.02, duration=6.0, energy_thresh=0.02) +oracle: FRESH (not previously cached): eval_pendulum.strong_oracle (24 restarts x 100 steps), computed this run and cached to results/baselines/pendulum_oracle_seed321.npz +K_std=8, diffusion seeds=(0, 1, 2) + +--- K=8 --- +arm cert_any stable_any persamp reg_med% reg_p90% reg_max% ms/plant McNemar(n10,n01,p) Wilcoxon(n,p) +diffusion 1.00 1.00 1.00 0.97 2.21 3.17 283.48 (reference) (reference) + (sd across 3 seeds: cert_any 0.000, stable_any 0.000, per-samp 0.000) + (seed[0] 95% CI: stable_any [1.00,1.00] regret_med [+0.75,+1.31]%) +retrieval 1.00 1.00 1.00 0.29 0.64 0.86 0.373 (0,0,1) (40,1e-07) + (Wilcoxon n=40 = plants where BOTH this arm and diffusion have a finite cost, i.e. the COMMON success set, not either arm's own coverage count) + (95% CI: stable_any [1.00,1.00] regret_med [+0.16,+0.36]%) +interpolation 1.00 1.00 1.00 0.30 0.67 2.01 0.373 (0,0,1) (40,1e-07) + (Wilcoxon n=40 = plants where BOTH this arm and diffusion have a finite cost, i.e. the COMMON success set, not either arm's own coverage count) + (95% CI: stable_any [1.00,1.00] regret_med [+0.16,+0.42]%) + +--- K=1 --- +arm cert_any stable_any persamp reg_med% reg_p90% reg_max% ms/plant McNemar(n10,n01,p) Wilcoxon(n,p) +diffusion 1.00 1.00 1.00 1.55 3.37 5.05 283.48 (reference) (reference) + (sd across 3 seeds: cert_any 0.000, stable_any 0.000, per-samp 0.000) + (seed[0] 95% CI: stable_any [1.00,1.00] regret_med [+1.44,+1.94]%) +retrieval 1.00 1.00 1.00 1.36 2.68 5.20 0.128 (0,0,1) (40,0.055) + (Wilcoxon n=40 = plants where BOTH this arm and diffusion have a finite cost, i.e. the COMMON success set, not either arm's own coverage count) + (95% CI: stable_any [1.00,1.00] regret_med [+1.14,+1.81]%) +interpolation 1.00 1.00 1.00 1.36 2.68 5.20 0.128 (0,0,1) (40,0.055) + (Wilcoxon n=40 = plants where BOTH this arm and diffusion have a finite cost, i.e. the COMMON success set, not either arm's own coverage count) + (95% CI: stable_any [1.00,1.00] regret_med [+1.14,+1.81]%) + diff --git a/results/baselines/retrieval_siso.txt b/results/baselines/retrieval_siso.txt new file mode 100644 index 0000000000000000000000000000000000000000..cdfc293e810c416e883b34dd03dbe56d93e42696 --- /dev/null +++ b/results/baselines/retrieval_siso.txt @@ -0,0 +1,36 @@ +=== SISO retrieval/interpolation baselines (N=60, seed=0) === +verifier: batched RK4 rollout (nonlinear.costs.diff_realize machinery, family's actual q_state -- see module docstring), padded NSTATE=8, shared x0 (seed=777), n_ic=4, dt=0.03, duration=12.0, energy_thresh=0.3 +oracle: NOT a single global oracle. generalize.py's SISO oracle protocol (60-step Adam, phi-only, sector-projected, lr=0.06) is warm-started from a candidate by construction, so we apply it IDENTICALLY to each arm's own best candidate, batched across plants (documented adaptation -- see docs/retrieval_baselines.md). Primary comparison is the direct paired cov/cost test vs the diffusion arm below, which needs no oracle. +K_std=8, diffusion seeds=(0, 1, 2) + +--- K=8 --- +arm coverage persamp reg_med% reg_p90% reg_max% ms/plant McNemar(n10,n01,p) Wilcoxon(n,p) +diffusion 0.79 0.63 10.17 44.69 121.17 21.53 (reference) (reference) + (sd across 3 seeds: coverage 0.008, per-samp 0.010; seed[0] 95% CI: coverage [0.67,0.88] regret_med [+7.04,+20.00]%) +retrieval 0.52 0.52 0.16 4.42 61.94 6.665 (0,16,3.1e-05) (31,0.018) + (95% CI: coverage [0.40,0.65] regret_med [+0.05,+0.34]%) +interpolation 0.52 0.52 0.16 4.42 61.94 6.665 (0,16,3.1e-05) (31,0.018) + (95% CI: coverage [0.40,0.65] regret_med [+0.05,+0.34]%) + (regressor is a point estimate, budget-independent -- reported once, at K=1) + +--- K=1 --- +arm coverage persamp reg_med% reg_p90% reg_max% ms/plant McNemar(n10,n01,p) Wilcoxon(n,p) +diffusion 0.67 0.67 9.02 69.47 161.30 21.53 (reference) (reference) + (sd across 3 seeds: coverage 0.021, per-samp 0.021; seed[0] 95% CI: coverage [0.53,0.77] regret_med [+6.18,+14.24]%) +retrieval 0.52 0.52 0.16 4.42 61.94 6.789 (3,11,0.057) (28,0.00058) + (95% CI: coverage [0.40,0.65] regret_med [+0.05,+0.34]%) +interpolation 0.52 0.52 0.16 4.42 61.94 6.789 (3,11,0.057) (28,0.00058) + (95% CI: coverage [0.40,0.65] regret_med [+0.05,+0.34]%) +regressor 0.75 0.75 63.96 88.20 125.65 0.299 (8,2,0.11) (37,2.2e-05) + (95% CI: coverage [0.63,0.85] regret_med [+54.57,+71.82]%) + + +--- addendum: confirming the success set used by the Wilcoxon cost test above (coordinator follow-up #3) --- +K=8: diffusion(seed0) covers 47/60 plants + retrieval: covers 31/60; common-with-diffusion=31 (= a strict subset of diffusion coverage) + interpolation: covers 31/60; common-with-diffusion=31 (= a strict subset of diffusion coverage) +K=1: diffusion(seed0) covers 39/60 plants + retrieval: covers 31/60; common-with-diffusion=28 (NOT a subset -- some plants only this arm covers) + interpolation: covers 31/60; common-with-diffusion=28 (NOT a subset -- some plants only this arm covers) + regressor: covers 45/60; common-with-diffusion=37 (NOT a subset -- some plants only this arm covers) +The Wilcoxon n reported in the table above IS this common-coverage count in every row (wilcoxon_test restricts to np.isfinite(a)&np.isfinite(b) before testing, i.e. plants where BOTH arms have a finite/stable cost) -- confirmed by direct recomputation from the saved per-plant arrays, not just re-reading the code. For K=8 retrieval/interpolation, n=31 exactly equals retrieval's own coverage count because every plant retrieval covers is ALSO covered by diffusion (a strict subset) -- so the reported regret numbers for retrieval/interpolation are simultaneously "conditional on own success set" AND "the common success set" at K=8. diff --git a/results/baselines/siso_run.log b/results/baselines/siso_run.log new file mode 100644 index 0000000000000000000000000000000000000000..cc2d2bc0441bb08feb157d93b787e5b70e0a3c29 --- /dev/null +++ b/results/baselines/siso_run.log @@ -0,0 +1,19 @@ +[seed 0] done +[seed 1] done +[seed 2] done +[seed 3] done +[seed 4] done + +=== SISO BASELINES === +SISO why-generative baseline — pooled over 5 seeds × 60 plants (N=300), K=8 + + regressor : plant-stabilisable 0.72 [0.67,0.77] per-sample-stable 0.72 + diff_1shot : plant-stabilisable 0.61 [0.56,0.67] per-sample-stable 0.61 + diff_Kshot : plant-stabilisable 0.75 [0.70,0.80] per-sample-stable 0.61 + +diffusion K-shot vs regressor (paired McNemar-ish sign test): p = 3.09e-02 +controller diversity (per-point std across K diffusion samples): 3.4 (regressor = 0, a single point) + +TAKEAWAY: the regressor (conditional mean) stabilises fewer plants than diffusion sampled +K times; the generative DISTRIBUTION is what lets test-time selection/repair find a +stabilising controller a point estimate misses. diff --git a/results/furuta_v3/deploy_sidebyside.txt b/results/furuta_v3/deploy_sidebyside.txt new file mode 100644 index 0000000000000000000000000000000000000000..f0ec37bff7378d6b658ee4c578a41d298358ae56 --- /dev/null +++ b/results/furuta_v3/deploy_sidebyside.txt @@ -0,0 +1,19 @@ +deployment mismatch, K=8, nominal-conditioned generated+P (mean+-sd over 3 sampler seeds) vs retrieval+P and NMPC (reused, n=120 plants) +scenario retr NMPC v2 clipped v2 clip5 v3 v3 e2400 v3 vs retr (b-c,p) v3 vs v2clip (b-c,p) e2400 vs retr (b-c,p) +nominal 90.0% 95.8% 65.3+-0.4 85.8+-0.7 86.1+-1.6 91.4+-0.8 6-4 p=0.7539 29-1 p=0.0000 5-6 p=1.0000 +fric0.6 86.7% 95.8% 62.5+-0.0 80.8+-2.7 84.4+-0.4 91.1+-1.7 10-8 p=0.8145 29-2 p=0.0000 8-11 p=0.6476 +fric1.6 9.2% 64.2% 10.6+-0.8 11.1+-0.4 9.7+-1.4 10.3+-0.8 1-4 p=0.3750 4-4 p=1.0000 3-3 p=1.0000 +fric2.5 0.0% 29.2% 0.0+-0.0 0.6+-0.4 0.3+-0.4 0.0+-0.0 0-1 p=1.0000 1-0 p=1.0000 0-0 p=1.0000 +ml_mp-_Lp- 68.3% 94.2% 50.0+-4.5 63.6+-0.4 69.4+-2.4 70.8+-1.4 10-8 p=0.8145 29-8 p=0.0008 8-11 p=0.6476 +ml_mp-_Lp+ 81.7% 95.8% 61.7+-0.0 77.2+-1.0 83.3+-0.7 89.2+-1.4 10-12 p=0.8318 31-5 p=0.0000 9-16 p=0.2295 +ml_mp+_Lp- 48.3% 95.0% 39.7+-2.7 53.1+-3.5 47.8+-2.1 51.4+-2.0 15-15 p=1.0000 25-11 p=0.0288 14-16 p=0.8555 +ml_mp+_Lp+ 75.8% 95.0% 57.5+-1.2 72.2+-2.1 78.3+-1.4 82.8+-0.4 15-18 p=0.7283 33-6 p=0.0000 11-19 p=0.2005 +both_f1.6_mp-_Lp- 9.2% 55.0% 11.7+-0.7 13.9+-0.4 11.9+-2.1 11.9+-1.4 3-9 p=0.1460 7-3 p=0.3438 3-7 p=0.3438 +both_f1.6_mp-_Lp+ 19.2% 75.0% 17.8+-1.0 17.8+-1.7 18.6+-1.4 18.9+-1.7 4-5 p=1.0000 6-2 p=0.2891 5-5 p=1.0000 +both_f1.6_mp+_Lp- 0.8% 52.5% 0.8+-0.0 0.8+-0.0 0.8+-0.0 0.6+-0.8 1-1 p=1.0000 1-1 p=1.0000 1-0 p=1.0000 +both_f1.6_mp+_Lp+ 1.7% 67.5% 2.5+-0.7 3.3+-0.7 1.7+-0.7 2.2+-1.4 0-1 p=1.0000 3-3 p=1.0000 0-3 p=0.2500 +both_f2.5_mp-_Lp- 0.0% 21.7% 0.3+-0.4 0.0+-0.0 0.0+-0.0 0.3+-0.4 0-0 p=1.0000 0-0 p=1.0000 0-0 p=1.0000 +both_f2.5_mp-_Lp+ 0.0% 33.3% 1.1+-0.4 0.6+-0.8 0.3+-0.4 0.0+-0.0 0-0 p=1.0000 0-1 p=1.0000 0-0 p=1.0000 +both_f2.5_mp+_Lp- 0.0% 18.3% 0.0+-0.0 0.3+-0.4 0.3+-0.4 0.0+-0.0 0-0 p=1.0000 0-0 p=1.0000 0-0 p=1.0000 +both_f2.5_mp+_Lp+ 0.0% 33.3% 0.0+-0.0 0.3+-0.4 0.0+-0.0 0.0+-0.0 0-0 p=1.0000 0-0 p=1.0000 0-0 p=1.0000 +D5 (diffusion mean >= retrieval mean in some scenario): max over scenarios of (diffusion - retrieval), points: v2 clipped: +2.5, v2 clip5: +4.7, v3: +2.8, v3 e2400: +7.5 diff --git a/results/furuta_v3/ksweep_sidebyside.txt b/results/furuta_v3/ksweep_sidebyside.txt new file mode 100644 index 0000000000000000000000000000000000000000..9c8735e5f0e753ae7d482c77b22f7d1ce8336ff9 --- /dev/null +++ b/results/furuta_v3/ksweep_sidebyside.txt @@ -0,0 +1,9 @@ +generated+P pooled over both draws (mean of 3 sampler seeds) K=1 K=4 K=8 K=16 +v2 clipped (shipped) 33.9% 58.9% 65.2% 68.0% +v2 clip +-5 (audit) 34.1% 75.0% 85.8% 90.0% +v3 s0 800ep 38.3% 75.8% 86.1% 88.6% +v3 s1 800ep 31.6% 71.4% 81.9% 86.4% +v3 s0 2400ep 53.6% 84.8% 91.3% 93.7% +v3 s1 2400ep 58.4% 87.5% 92.5% 93.3% +v2-data clip5 2400ep (control) 60.8% 87.0% 90.6% 91.9% +nearest+P (retrieval, same bank) 54.2% 82.5% 90.0% 93.3% diff --git a/results/furuta_v3/traj_gen_eval_v3_raw_clip1.txt b/results/furuta_v3/traj_gen_eval_v3_raw_clip1.txt new file mode 100644 index 0000000000000000000000000000000000000000..074dd201f1c39c7e73684f0771f918d23ae920ee --- /dev/null +++ b/results/furuta_v3/traj_gen_eval_v3_raw_clip1.txt @@ -0,0 +1,94 @@ +bank data/furuta_nmpc_bank3000.npz: 2801 entries +model models/Best_furuta_traj_cond_v3.pth: cond_dim=4 out_dim=116 hidden=256 layers=5 device=cpu +[seed4242] margin=0.3: resolving P for 227 unique bank source plants +[seed4242] sampler seed 0: 9.6s for 60x4 candidates (0 decode-invalid of 240) +[seed4242] sampler seed 1: 9.5s for 60x4 candidates (0 decode-invalid of 240) +[seed4242] sampler seed 2: 9.6s for 60x4 candidates (0 decode-invalid of 240) +[seed4242] 10/60 (31s) +[seed4242] 20/60 (46s) +[seed4242] 30/60 (72s) +[seed4242] 40/60 (88s) +[seed4242] 50/60 (108s) +[seed4242] 60/60 (137s) +[seed9001] margin=0.3: resolving P for 232 unique bank source plants +[seed9001] sampler seed 0: 10.5s for 60x4 candidates (0 decode-invalid of 240) +[seed9001] sampler seed 1: 10.7s for 60x4 candidates (0 decode-invalid of 240) +[seed9001] sampler seed 2: 10.2s for 60x4 candidates (0 decode-invalid of 240) +[seed9001] 10/60 (33s) +[seed9001] 20/60 (57s) +[seed9001] 30/60 (80s) +[seed9001] 40/60 (101s) +[seed9001] 50/60 (124s) +[seed9001] 60/60 (153s) + +=== traj_gen_eval (v3_raw_clip1): generated vs retrieval vs NMPC === + +--- plant set seed4242 (seed 4242, n=60) --- + arm succ 95% CI medJ med regret p90 synth + per-plant 93% [ 87, 98] 4.08 0.00% 0.00% 4.6s + nearest 62% [ 50, 73] 4.15 0.36% 1.51% 326ms + nearest+P 78% [ 67, 88] 4.09 0.69% 2.52% 638ms + nearest-margin 62% [ 50, 73] 4.15 0.36% 1.51% 306ms + NMPC 95% [ 88, 100] 4.08 0.14% 0.24% 26ms + generated 55.6%(sd3.9) [ 38, 63]* 4.05 1.05%(sd0.3) 3.14%(sd0.8) 406ms + generated+P 76.7%(sd1.4) [ 63, 85]* 4.09 1.04%(sd0.1) 3.58%(sd0.5) 1.1s + (* generated rows: mean (sd) over sampler seeds [0, 1, 2]; CI shown is for seed 0 only) + (certified hand-off rate == success rate for every arm except NMPC (a certificate is a precondition for success here); NMPC's own hand-off is n/a, it never switches to a static gain) + + --- failure breakdown, primary seed 0 --- + per-plant NMPC never reached Omega:3 replay failed:1 + nearest Lyapunov decrease fails at a vertex:13 rollout failed:9 true dynamics: 0.02% of the boundary has V-dot >= 0, 0.00% saturates:1 + nearest+P rollout failed:13 + nearest-margin Lyapunov decrease fails at a vertex:13 rollout failed:9 true dynamics: 0.02% of the boundary has V-dot >= 0, 0.00% saturates:1 + NMPC :3 + generated rollout failed:12 Lyapunov decrease fails at a vertex:9 peak |Kx| on Omega is 10.05 V > 10.0 V:2 peak |Kx| on Omega is 10.01 V > 10.0 V:2 peak |Kx| on Omega is 10.17 V > 10.0 V:1 peak |Kx| on Omega is 10.14 V > 10.0 V:1 peak |Kx| on Omega is 10.37 V > 10.0 V:1 peak |Kx| on Omega is 10.07 V > 10.0 V:1 peak |Kx| on Omega is 10.12 V > 10.0 V:1 + generated+P rollout failed:15 + + --- paired McNemar on success (exact binomial on discordant pairs) --- + per-plant vs nearest beats on 20, loses on 1 p=0.0000 * cost ratio 1.004x p=0.0000 + per-plant vs nearest+P beats on 10, loses on 1 p=0.0117 * cost ratio 1.007x p=0.0000 + per-plant vs nearest-margin beats on 20, loses on 1 p=0.0000 * cost ratio 1.004x p=0.0000 + per-plant vs NMPC beats on 0, loses on 1 p=1.0000 cost ratio 1.001x p=0.0002 + per-plant vs generated beats on 26, loses on 0 p=0.0000 * cost ratio 1.010x p=0.0000 + per-plant vs generated+P beats on 12, loses on 1 p=0.0034 * cost ratio 1.011x p=0.0000 + nearest+P vs per-plant beats on 1, loses on 10 p=0.0117 * cost ratio 0.993x p=0.0000 + nearest+P vs nearest beats on 10, loses on 0 p=0.0020 * cost ratio 1.000x p=0.8291 + nearest+P vs nearest-margin beats on 10, loses on 0 p=0.0020 * cost ratio 1.000x p=0.8291 + nearest+P vs NMPC beats on 0, loses on 10 p=0.0020 * cost ratio 0.995x p=0.0000 + nearest+P vs generated beats on 20, loses on 3 p=0.0005 * cost ratio 1.005x p=0.0815 + nearest+P vs generated+P beats on 8, loses on 6 p=0.7905 cost ratio 1.002x p=0.0414 + +--- plant set seed9001 (seed 9001, n=60) --- + arm succ 95% CI medJ med regret p90 synth + per-plant 93% [ 87, 98] 4.16 0.00% 0.00% 7.5s + nearest 65% [ 52, 77] 4.10 0.71% 2.11% 218ms + nearest+P 87% [ 78, 95] 4.14 0.59% 2.34% 567ms + nearest-margin 65% [ 52, 77] 4.10 0.71% 2.11% 260ms + NMPC 97% [ 92, 100] 4.17 0.07% 0.22% 34ms + generated 57.2%(sd3.4) [ 43, 68]* 4.16 0.66%(sd0.1) 2.85%(sd0.7) 448ms + generated+P 79.4%(sd3.4) [ 73, 93]* 4.15 0.76%(sd0.2) 3.50%(sd0.4) 1.0s + (* generated rows: mean (sd) over sampler seeds [0, 1, 2]; CI shown is for seed 0 only) + (certified hand-off rate == success rate for every arm except NMPC (a certificate is a precondition for success here); NMPC's own hand-off is n/a, it never switches to a static gain) + + --- failure breakdown, primary seed 0 --- + per-plant replay failed:2 NMPC never reached Omega:2 + nearest Lyapunov decrease fails at a vertex:16 rollout failed:5 + nearest+P rollout failed:8 + nearest-margin Lyapunov decrease fails at a vertex:16 rollout failed:5 + NMPC :2 + generated Lyapunov decrease fails at a vertex:19 rollout failed:2 peak |Kx| on Omega is 10.03 V > 10.0 V:1 peak |Kx| on Omega is 10.15 V > 10.0 V:1 true dynamics: 0.29% of the boundary has V-dot >= 0, 0.00% saturates:1 peak |Kx| on Omega is 10.06 V > 10.0 V:1 true dynamics: 0.60% of the boundary has V-dot >= 0, 0.00% saturates:1 + generated+P rollout failed:10 + + --- paired McNemar on success (exact binomial on discordant pairs) --- + per-plant vs nearest beats on 18, loses on 1 p=0.0001 * cost ratio 1.007x p=0.0000 + per-plant vs nearest+P beats on 5, loses on 1 p=0.2188 cost ratio 1.006x p=0.0000 + per-plant vs nearest-margin beats on 18, loses on 1 p=0.0001 * cost ratio 1.007x p=0.0000 + per-plant vs NMPC beats on 0, loses on 2 p=0.5000 cost ratio 1.001x p=0.0064 + per-plant vs generated beats on 22, loses on 0 p=0.0000 * cost ratio 1.008x p=0.0000 + per-plant vs generated+P beats on 6, loses on 0 p=0.0312 * cost ratio 1.010x p=0.0000 + nearest+P vs per-plant beats on 1, loses on 5 p=0.2188 cost ratio 0.994x p=0.0000 + nearest+P vs nearest beats on 13, loses on 0 p=0.0002 * cost ratio 1.000x p=0.1154 + nearest+P vs nearest-margin beats on 13, loses on 0 p=0.0002 * cost ratio 1.000x p=0.1154 + nearest+P vs NMPC beats on 0, loses on 6 p=0.0312 * cost ratio 0.995x p=0.0000 + nearest+P vs generated beats on 20, loses on 2 p=0.0001 * cost ratio 1.003x p=0.1240 + nearest+P vs generated+P beats on 5, loses on 3 p=0.7266 cost ratio 1.004x p=0.0165 diff --git a/results/furuta_v3/traj_gen_eval_v3e2400s1_raw_clip1.txt b/results/furuta_v3/traj_gen_eval_v3e2400s1_raw_clip1.txt new file mode 100644 index 0000000000000000000000000000000000000000..87f0606091a2b871bf7ba29107b6d79e038b14ac --- /dev/null +++ b/results/furuta_v3/traj_gen_eval_v3e2400s1_raw_clip1.txt @@ -0,0 +1,94 @@ +bank data/furuta_nmpc_bank3000.npz: 2801 entries +model models/Best_furuta_traj_cond_v3_e2400_s1.pth: cond_dim=4 out_dim=116 hidden=256 layers=5 device=cpu +[seed4242] margin=0.3: resolving P for 227 unique bank source plants +[seed4242] sampler seed 0: 9.6s for 60x4 candidates (0 decode-invalid of 240) +[seed4242] sampler seed 1: 9.5s for 60x4 candidates (0 decode-invalid of 240) +[seed4242] sampler seed 2: 9.6s for 60x4 candidates (0 decode-invalid of 240) +[seed4242] 10/60 (23s) +[seed4242] 20/60 (39s) +[seed4242] 30/60 (57s) +[seed4242] 40/60 (74s) +[seed4242] 50/60 (94s) +[seed4242] 60/60 (119s) +[seed9001] margin=0.3: resolving P for 232 unique bank source plants +[seed9001] sampler seed 0: 9.8s for 60x4 candidates (0 decode-invalid of 240) +[seed9001] sampler seed 1: 10.0s for 60x4 candidates (0 decode-invalid of 240) +[seed9001] sampler seed 2: 10.2s for 60x4 candidates (0 decode-invalid of 240) +[seed9001] 10/60 (29s) +[seed9001] 20/60 (49s) +[seed9001] 30/60 (78s) +[seed9001] 40/60 (100s) +[seed9001] 50/60 (120s) +[seed9001] 60/60 (146s) + +=== traj_gen_eval (v3e2400s1_raw_clip1): generated vs retrieval vs NMPC === + +--- plant set seed4242 (seed 4242, n=60) --- + arm succ 95% CI medJ med regret p90 synth + per-plant 93% [ 87, 98] 4.08 0.00% 0.00% 4.0s + nearest 62% [ 50, 73] 4.15 0.36% 1.51% 301ms + nearest+P 78% [ 67, 88] 4.09 0.69% 2.52% 583ms + nearest-margin 62% [ 50, 73] 4.15 0.36% 1.51% 305ms + NMPC 95% [ 88, 100] 4.08 0.14% 0.24% 23ms + generated 60.0%(sd4.1) [ 42, 68]* 4.09 0.65%(sd0.2) 2.27%(sd0.5) 300ms + generated+P 81.7%(sd2.4) [ 70, 90]* 4.09 0.77%(sd0.2) 2.82%(sd0.5) 686ms + (* generated rows: mean (sd) over sampler seeds [0, 1, 2]; CI shown is for seed 0 only) + (certified hand-off rate == success rate for every arm except NMPC (a certificate is a precondition for success here); NMPC's own hand-off is n/a, it never switches to a static gain) + + --- failure breakdown, primary seed 0 --- + per-plant NMPC never reached Omega:3 replay failed:1 + nearest Lyapunov decrease fails at a vertex:13 rollout failed:9 true dynamics: 0.02% of the boundary has V-dot >= 0, 0.00% saturates:1 + nearest+P rollout failed:13 + nearest-margin Lyapunov decrease fails at a vertex:13 rollout failed:9 true dynamics: 0.02% of the boundary has V-dot >= 0, 0.00% saturates:1 + NMPC :3 + generated Lyapunov decrease fails at a vertex:15 rollout failed:5 peak |Kx| on Omega is 10.02 V > 10.0 V:1 peak |Kx| on Omega is 10.04 V > 10.0 V:1 peak |Kx| on Omega is 10.16 V > 10.0 V:1 peak |Kx| on Omega is 10.30 V > 10.0 V:1 peak |Kx| on Omega is 10.12 V > 10.0 V:1 peak |Kx| on Omega is 10.03 V > 10.0 V:1 peak |Kx| on Omega is 10.28 V > 10.0 V:1 + generated+P rollout failed:12 + + --- paired McNemar on success (exact binomial on discordant pairs) --- + per-plant vs nearest beats on 20, loses on 1 p=0.0000 * cost ratio 1.004x p=0.0000 + per-plant vs nearest+P beats on 10, loses on 1 p=0.0117 * cost ratio 1.007x p=0.0000 + per-plant vs nearest-margin beats on 20, loses on 1 p=0.0000 * cost ratio 1.004x p=0.0000 + per-plant vs NMPC beats on 0, loses on 1 p=1.0000 cost ratio 1.001x p=0.0002 + per-plant vs generated beats on 23, loses on 0 p=0.0000 * cost ratio 1.009x p=0.0000 + per-plant vs generated+P beats on 8, loses on 0 p=0.0078 * cost ratio 1.011x p=0.0000 + nearest+P vs per-plant beats on 1, loses on 10 p=0.0117 * cost ratio 0.993x p=0.0000 + nearest+P vs nearest beats on 10, loses on 0 p=0.0020 * cost ratio 1.000x p=0.8291 + nearest+P vs nearest-margin beats on 10, loses on 0 p=0.0020 * cost ratio 1.000x p=0.8291 + nearest+P vs NMPC beats on 0, loses on 10 p=0.0020 * cost ratio 0.995x p=0.0000 + nearest+P vs generated beats on 17, loses on 3 p=0.0026 * cost ratio 1.000x p=0.2988 + nearest+P vs generated+P beats on 6, loses on 7 p=1.0000 cost ratio 1.003x p=0.0735 + +--- plant set seed9001 (seed 9001, n=60) --- + arm succ 95% CI medJ med regret p90 synth + per-plant 93% [ 87, 98] 4.16 0.00% 0.00% 7.8s + nearest 65% [ 52, 77] 4.10 0.71% 2.11% 265ms + nearest+P 87% [ 78, 95] 4.14 0.59% 2.34% 589ms + nearest-margin 65% [ 52, 77] 4.10 0.71% 2.11% 263ms + NMPC 97% [ 92, 100] 4.17 0.07% 0.22% 37ms + generated 63.3%(sd2.4) [ 48, 72]* 4.19 0.59%(sd0.1) 2.57%(sd0.3) 407ms + generated+P 90.6%(sd2.8) [ 87, 98]* 4.18 0.73%(sd0.0) 3.03%(sd0.3) 680ms + (* generated rows: mean (sd) over sampler seeds [0, 1, 2]; CI shown is for seed 0 only) + (certified hand-off rate == success rate for every arm except NMPC (a certificate is a precondition for success here); NMPC's own hand-off is n/a, it never switches to a static gain) + + --- failure breakdown, primary seed 0 --- + per-plant replay failed:2 NMPC never reached Omega:2 + nearest Lyapunov decrease fails at a vertex:16 rollout failed:5 + nearest+P rollout failed:8 + nearest-margin Lyapunov decrease fails at a vertex:16 rollout failed:5 + NMPC :2 + generated Lyapunov decrease fails at a vertex:15 peak |Kx| on Omega is 10.08 V > 10.0 V:1 true dynamics: 0.21% of the boundary has V-dot >= 0, 0.00% saturates:1 peak |Kx| on Omega is 10.01 V > 10.0 V:1 peak |Kx| on Omega is 10.31 V > 10.0 V:1 peak |Kx| on Omega is 10.17 V > 10.0 V:1 peak |Kx| on Omega is 10.03 V > 10.0 V:1 rollout failed:1 true dynamics: 0.26% of the boundary has V-dot >= 0, 0.00% saturates:1 true dynamics: 0.03% of the boundary has V-dot >= 0, 0.00% saturates:1 + generated+P rollout failed:4 + + --- paired McNemar on success (exact binomial on discordant pairs) --- + per-plant vs nearest beats on 18, loses on 1 p=0.0001 * cost ratio 1.007x p=0.0000 + per-plant vs nearest+P beats on 5, loses on 1 p=0.2188 cost ratio 1.006x p=0.0000 + per-plant vs nearest-margin beats on 18, loses on 1 p=0.0001 * cost ratio 1.007x p=0.0000 + per-plant vs NMPC beats on 0, loses on 2 p=0.5000 cost ratio 1.001x p=0.0064 + per-plant vs generated beats on 20, loses on 0 p=0.0000 * cost ratio 1.006x p=0.0000 + per-plant vs generated+P beats on 0, loses on 0 p=1.0000 cost ratio 1.007x p=0.0000 + nearest+P vs per-plant beats on 1, loses on 5 p=0.2188 cost ratio 0.994x p=0.0000 + nearest+P vs nearest beats on 13, loses on 0 p=0.0002 * cost ratio 1.000x p=0.1154 + nearest+P vs nearest-margin beats on 13, loses on 0 p=0.0002 * cost ratio 1.000x p=0.1154 + nearest+P vs NMPC beats on 0, loses on 6 p=0.0312 * cost ratio 0.995x p=0.0000 + nearest+P vs generated beats on 18, loses on 2 p=0.0004 * cost ratio 1.002x p=0.3011 + nearest+P vs generated+P beats on 1, loses on 5 p=0.2188 cost ratio 1.001x p=0.1153 diff --git a/results/furuta_v3/traj_gen_v2ctl2400_raw_clip5.txt b/results/furuta_v3/traj_gen_v2ctl2400_raw_clip5.txt new file mode 100644 index 0000000000000000000000000000000000000000..6d5bfd7bb285b780b8a68bf821a9ee19f30cb037 --- /dev/null +++ b/results/furuta_v3/traj_gen_v2ctl2400_raw_clip5.txt @@ -0,0 +1,30 @@ +bank data/furuta_nmpc_bank3000.npz: 2801 entries +model models/Best_furuta_traj_cond_v2ctl_e2400.pth: cond_dim=4 out_dim=116 +[seed4242] per-plant ceiling: 93% +[seed4242] nearest+P sequences done +[seed4242] generated+P seed 0 sequences done (36.5s sampling for 60x16) +[seed4242] generated+P seed 1 sequences done (37.0s sampling for 60x16) +[seed4242] generated+P seed 2 sequences done (36.6s sampling for 60x16) +[seed9001] per-plant ceiling: 93% +[seed9001] nearest+P sequences done +[seed9001] generated+P seed 0 sequences done (36.3s sampling for 60x16) +[seed9001] generated+P seed 1 sequences done (36.4s sampling for 60x16) +[seed9001] generated+P seed 2 sequences done (36.2s sampling for 60x16) + +=== candidate-budget sweep: generated+P vs nearest+P vs per-plant ceiling === + +--- plant set seed4242 (seed 4242, n=60) --- + K nearest+P generated+P per-plant + 1 51.7% t= 0.42s 55.6%(sd3.4) t= 0.95s 93.3% [McNemar nearest+P vs generated+P(seed0): 11-11, p=1.000] + 4 78.3% t= 0.66s 81.7%(sd2.4) t= 1.16s 93.3% [McNemar nearest+P vs generated+P(seed0): 6-7, p=1.000] + 8 88.3% t= 0.66s 86.7%(sd1.4) t= 1.16s 93.3% [McNemar nearest+P vs generated+P(seed0): 3-3, p=1.000] + 16 93.3% t= 0.66s 88.3%(sd1.4) t= 1.16s 93.3% [McNemar nearest+P vs generated+P(seed0): 4-1, p=0.375] + crossover K (generated+P mean >= nearest+P): 1 + +--- plant set seed9001 (seed 9001, n=60) --- + K nearest+P generated+P per-plant + 1 56.7% t= 0.50s 66.1%(sd1.6) t= 0.92s 93.3% [McNemar nearest+P vs generated+P(seed0): 13-18, p=0.473] + 4 86.7% t= 0.66s 92.2%(sd0.8) t= 1.05s 93.3% [McNemar nearest+P vs generated+P(seed0): 1-4, p=0.375] + 8 91.7% t= 0.66s 94.4%(sd0.8) t= 1.05s 93.3% [McNemar nearest+P vs generated+P(seed0): 0-2, p=0.500] + 16 93.3% t= 0.66s 95.6%(sd0.8) t= 1.05s 93.3% [McNemar nearest+P vs generated+P(seed0): 0-1, p=1.000] + crossover K (generated+P mean >= nearest+P): 1 diff --git a/results/furuta_v3/traj_gen_v3e2400_raw_clip1.txt b/results/furuta_v3/traj_gen_v3e2400_raw_clip1.txt new file mode 100644 index 0000000000000000000000000000000000000000..42edeca511c01a2a98cf2b0e21b09800db9ed296 --- /dev/null +++ b/results/furuta_v3/traj_gen_v3e2400_raw_clip1.txt @@ -0,0 +1,30 @@ +bank data/furuta_nmpc_bank3000.npz: 2801 entries +model models/Best_furuta_traj_cond_v3_e2400.pth: cond_dim=4 out_dim=116 +[seed4242] per-plant ceiling: 93% +[seed4242] nearest+P sequences done +[seed4242] generated+P seed 0 sequences done (34.3s sampling for 60x16) +[seed4242] generated+P seed 1 sequences done (34.6s sampling for 60x16) +[seed4242] generated+P seed 2 sequences done (34.5s sampling for 60x16) +[seed9001] per-plant ceiling: 93% +[seed9001] nearest+P sequences done +[seed9001] generated+P seed 0 sequences done (34.1s sampling for 60x16) +[seed9001] generated+P seed 1 sequences done (34.3s sampling for 60x16) +[seed9001] generated+P seed 2 sequences done (34.1s sampling for 60x16) + +=== candidate-budget sweep: generated+P vs nearest+P vs per-plant ceiling === + +--- plant set seed4242 (seed 4242, n=60) --- + K nearest+P generated+P per-plant + 1 51.7% t= 0.29s 50.0%(sd7.2) t= 0.87s 93.3% [McNemar nearest+P vs generated+P(seed0): 14-9, p=0.405] + 4 78.3% t= 0.56s 82.8%(sd3.4) t= 1.13s 93.3% [McNemar nearest+P vs generated+P(seed0): 6-9, p=0.607] + 8 88.3% t= 0.56s 89.4%(sd2.1) t= 1.13s 93.3% [McNemar nearest+P vs generated+P(seed0): 3-4, p=1.000] + 16 93.3% t= 0.56s 91.7%(sd1.4) t= 1.13s 93.3% [McNemar nearest+P vs generated+P(seed0): 3-1, p=0.625] + crossover K (generated+P mean >= nearest+P): 4 + +--- plant set seed9001 (seed 9001, n=60) --- + K nearest+P generated+P per-plant + 1 56.7% t= 0.30s 57.2%(sd7.5) t= 0.86s 93.3% [McNemar nearest+P vs generated+P(seed0): 16-11, p=0.442] + 4 86.7% t= 0.45s 86.7%(sd1.4) t= 1.03s 93.3% [McNemar nearest+P vs generated+P(seed0): 5-4, p=1.000] + 8 91.7% t= 0.45s 93.3%(sd1.4) t= 1.03s 93.3% [McNemar nearest+P vs generated+P(seed0): 2-2, p=1.000] + 16 93.3% t= 0.45s 95.6%(sd0.8) t= 1.03s 93.3% [McNemar nearest+P vs generated+P(seed0): 1-2, p=1.000] + crossover K (generated+P mean >= nearest+P): 1 diff --git a/results/furuta_v3/traj_gen_warmstart_v2clip5_n200.txt b/results/furuta_v3/traj_gen_warmstart_v2clip5_n200.txt new file mode 100644 index 0000000000000000000000000000000000000000..83829180818d3e7d16369130a8478d800a409dc0 --- /dev/null +++ b/results/furuta_v3/traj_gen_warmstart_v2clip5_n200.txt @@ -0,0 +1,17 @@ +=== warm-start test (n=200, seed 4242, max_iter=700) === +target certificate: largest_certified_region per plant (unmargined); verifier: furuta_traj_bank.score_solution (identical to every other arm in this doc) + + condition success med iter med t med J + cold 58.0% 290 2.01s 4.06 + nearest-warm 78.0% 48 0.36s 4.10 + generated-warm 79.0% 50 0.37s 4.10 + +--- paired tests --- + generated-warm vs cold success: beats 44, loses 2, McNemar p=0.0000 * time ratio 0.210x (n=114), Wilcoxon p=0.0000 * + generated-warm vs nearest-warm success: beats 4, loses 2, McNemar p=0.6875 time ratio 0.993x (n=154), Wilcoxon p=0.5074 + nearest-warm vs cold success: beats 41, loses 1, McNemar p=0.0000 * time ratio 0.196x (n=115), Wilcoxon p=0.0000 * + +--- failure breakdown --- + cold ipopt: Maximum_Iterations_Exceeded:36 ipopt: Infeasible_Problem_Detected:31 rollout failed:17 + nearest-warm rollout failed:26 ipopt: Maximum_Iterations_Exceeded:10 ipopt: Infeasible_Problem_Detected:8 + generated-warm rollout failed:24 ipopt: Maximum_Iterations_Exceeded:9 ipopt: Infeasible_Problem_Detected:9 diff --git a/results/furuta_v3/traj_gen_warmstart_v2clip5_s1.txt b/results/furuta_v3/traj_gen_warmstart_v2clip5_s1.txt new file mode 100644 index 0000000000000000000000000000000000000000..ac0fd15814bbcc1d319eb78992281d71d1433e69 --- /dev/null +++ b/results/furuta_v3/traj_gen_warmstart_v2clip5_s1.txt @@ -0,0 +1,17 @@ +=== warm-start test (n=60, seed 4242, max_iter=700) === +target certificate: largest_certified_region per plant (unmargined); verifier: furuta_traj_bank.score_solution (identical to every other arm in this doc) + + condition success med iter med t med J + cold 53.3% 363 2.08s 4.06 + nearest-warm 76.7% 45 0.28s 4.15 + generated-warm 75.0% 46 0.27s 4.14 + +--- paired tests --- + generated-warm vs cold success: beats 13, loses 0, McNemar p=0.0002 * time ratio 0.147x (n=32), Wilcoxon p=0.0000 * + generated-warm vs nearest-warm success: beats 0, loses 1, McNemar p=1.0000 time ratio 1.056x (n=45), Wilcoxon p=0.6069 + nearest-warm vs cold success: beats 14, loses 0, McNemar p=0.0001 * time ratio 0.159x (n=32), Wilcoxon p=0.0000 * + +--- failure breakdown --- + cold ipopt: Maximum_Iterations_Exceeded:13 ipopt: Infeasible_Problem_Detected:11 rollout failed:4 + nearest-warm ipopt: Infeasible_Problem_Detected:6 rollout failed:5 ipopt: Maximum_Iterations_Exceeded:3 + generated-warm ipopt: Infeasible_Problem_Detected:7 ipopt: Maximum_Iterations_Exceeded:4 rollout failed:4 diff --git a/results/furuta_v3/traj_gen_warmstart_v2clip5_s2.txt b/results/furuta_v3/traj_gen_warmstart_v2clip5_s2.txt new file mode 100644 index 0000000000000000000000000000000000000000..ad4ba4ce3645b72a67c2de7fa2970381cc876727 --- /dev/null +++ b/results/furuta_v3/traj_gen_warmstart_v2clip5_s2.txt @@ -0,0 +1,17 @@ +=== warm-start test (n=60, seed 4242, max_iter=700) === +target certificate: largest_certified_region per plant (unmargined); verifier: furuta_traj_bank.score_solution (identical to every other arm in this doc) + + condition success med iter med t med J + cold 53.3% 363 2.07s 4.06 + nearest-warm 76.7% 45 0.28s 4.15 + generated-warm 75.0% 47 0.27s 4.14 + +--- paired tests --- + generated-warm vs cold success: beats 13, loses 0, McNemar p=0.0002 * time ratio 0.166x (n=32), Wilcoxon p=0.0000 * + generated-warm vs nearest-warm success: beats 0, loses 1, McNemar p=1.0000 time ratio 1.023x (n=45), Wilcoxon p=0.9199 + nearest-warm vs cold success: beats 14, loses 0, McNemar p=0.0001 * time ratio 0.160x (n=32), Wilcoxon p=0.0000 * + +--- failure breakdown --- + cold ipopt: Maximum_Iterations_Exceeded:13 ipopt: Infeasible_Problem_Detected:11 rollout failed:4 + nearest-warm ipopt: Infeasible_Problem_Detected:6 rollout failed:5 ipopt: Maximum_Iterations_Exceeded:3 + generated-warm ipopt: Infeasible_Problem_Detected:7 ipopt: Maximum_Iterations_Exceeded:4 rollout failed:4 diff --git a/results/furuta_v3/traj_gen_warmstart_v3_n200.txt b/results/furuta_v3/traj_gen_warmstart_v3_n200.txt new file mode 100644 index 0000000000000000000000000000000000000000..8d119ace4e644aa3701878b0d1bc121e96bf86c3 --- /dev/null +++ b/results/furuta_v3/traj_gen_warmstart_v3_n200.txt @@ -0,0 +1,17 @@ +=== warm-start test (n=200, seed 4242, max_iter=700) === +target certificate: largest_certified_region per plant (unmargined); verifier: furuta_traj_bank.score_solution (identical to every other arm in this doc) + + condition success med iter med t med J + cold 58.0% 290 2.08s 4.06 + nearest-warm 78.0% 48 0.38s 4.10 + generated-warm 79.5% 44 0.35s 4.10 + +--- paired tests --- + generated-warm vs cold success: beats 45, loses 2, McNemar p=0.0000 * time ratio 0.181x (n=114), Wilcoxon p=0.0000 * + generated-warm vs nearest-warm success: beats 4, loses 1, McNemar p=0.3750 time ratio 0.986x (n=155), Wilcoxon p=0.4041 + nearest-warm vs cold success: beats 41, loses 1, McNemar p=0.0000 * time ratio 0.210x (n=115), Wilcoxon p=0.0000 * + +--- failure breakdown --- + cold ipopt: Maximum_Iterations_Exceeded:36 ipopt: Infeasible_Problem_Detected:31 rollout failed:17 + nearest-warm rollout failed:26 ipopt: Maximum_Iterations_Exceeded:10 ipopt: Infeasible_Problem_Detected:8 + generated-warm rollout failed:21 ipopt: Maximum_Iterations_Exceeded:10 ipopt: Infeasible_Problem_Detected:10 diff --git a/results/furuta_v3/traj_gen_warmstart_v3_s2.txt b/results/furuta_v3/traj_gen_warmstart_v3_s2.txt new file mode 100644 index 0000000000000000000000000000000000000000..c924ee78ac0ba8ce84c3855fbdb33a11158ef8f5 --- /dev/null +++ b/results/furuta_v3/traj_gen_warmstart_v3_s2.txt @@ -0,0 +1,17 @@ +=== warm-start test (n=60, seed 4242, max_iter=700) === +target certificate: largest_certified_region per plant (unmargined); verifier: furuta_traj_bank.score_solution (identical to every other arm in this doc) + + condition success med iter med t med J + cold 53.3% 363 1.94s 4.06 + nearest-warm 76.7% 45 0.30s 4.15 + generated-warm 75.0% 46 0.28s 4.14 + +--- paired tests --- + generated-warm vs cold success: beats 13, loses 0, McNemar p=0.0002 * time ratio 0.158x (n=32), Wilcoxon p=0.0000 * + generated-warm vs nearest-warm success: beats 0, loses 1, McNemar p=1.0000 time ratio 0.982x (n=45), Wilcoxon p=0.3581 + nearest-warm vs cold success: beats 14, loses 0, McNemar p=0.0001 * time ratio 0.160x (n=32), Wilcoxon p=0.0000 * + +--- failure breakdown --- + cold ipopt: Maximum_Iterations_Exceeded:13 ipopt: Infeasible_Problem_Detected:11 rollout failed:4 + nearest-warm ipopt: Infeasible_Problem_Detected:6 rollout failed:5 ipopt: Maximum_Iterations_Exceeded:3 + generated-warm ipopt: Infeasible_Problem_Detected:7 rollout failed:5 ipopt: Maximum_Iterations_Exceeded:3 diff --git a/results/generalize/fig_generalize.png b/results/generalize/fig_generalize.png new file mode 100644 index 0000000000000000000000000000000000000000..f49ff54d50437ad21320575c345ef242ebb7ec2b Binary files /dev/null and b/results/generalize/fig_generalize.png differ diff --git a/results/generalize/summary.txt b/results/generalize/summary.txt new file mode 100644 index 0000000000000000000000000000000000000000..7f300e944c00fde4f9edfd0195b7b17f44e08e79 --- /dev/null +++ b/results/generalize/summary.txt @@ -0,0 +1,5 @@ +plants=120 controllers/plant=6 ICs/plant=4 +per-sample stability rate (mean over plants): 0.64 +plants with >=1 stabilizing controller: 0.81 +regret of best controller vs per-plant oracle: median=+9.7% mean=+22.4% (n=97) +amortization: generation 6.5s for 720 controllers (9 ms each); per-plant oracle 24.1s each diff --git a/results/metric_steering/mimo_summary.txt b/results/metric_steering/mimo_summary.txt new file mode 100644 index 0000000000000000000000000000000000000000..c9f900a66323920ee95bb3c41b94398018ff313a --- /dev/null +++ b/results/metric_steering/mimo_summary.txt @@ -0,0 +1,12 @@ +MIMO 2x2 metric steering (held-out, N=40) + set ise iae control_ settling peak_con peak_y + prior 31.784 12.175 5.113 7.849 1.877 4.223 +refine→control_effort 31.259 11.771 0.335* 7.543 0.922 4.277 + refine→ise 28.752* 11.237 8.601 8.056 2.435 4.153 + refine→settling 31.640 11.529 3.766 6.767* 2.080 4.324 +refine→peak_control 31.223 11.754 0.298 7.547 0.853* 4.280 + +refine→control_effort: target -93.5% vs prior (5.113 → 0.335) +refine→ise : target -9.5% vs prior (31.784 → 28.752) +refine→settling : target -13.8% vs prior (7.849 → 6.767) +refine→peak_control : target -54.6% vs prior (1.877 → 0.853) diff --git a/results/metric_steering/refine_summary.txt b/results/metric_steering/refine_summary.txt new file mode 100644 index 0000000000000000000000000000000000000000..f1e896b5804872f51e6356b41f3c26927610c3e1 --- /dev/null +++ b/results/metric_steering/refine_summary.txt @@ -0,0 +1,12 @@ +reference plant G=1/(s+1) F=1 | refine 80 steps lr=0.05 | n=16 ICs=4 + set ise iae itae control_ control_ peak_con peak_dev settling + prior 1.903 1.593 1.417 0.957 0.408 1.989 2.124 3.705 +refine→control_effort 2.389 1.950 1.945 0.296* 0.051 1.214 2.147 4.259 + refine→ise 0.841* 0.611 0.177 4.077 1.391 4.502 2.055 1.844 + refine→settling 0.848 0.617 0.182 4.018 1.385 4.472 2.056 1.857* +refine→peak_control 2.381 1.943 1.935 0.403 0.057 1.205* 2.147 4.250 + +refine→control_effort: target -69.1% vs prior (0.957 → 0.296) +refine→ise : target -55.8% vs prior (1.903 → 0.841) +refine→settling : target -49.9% vs prior (3.705 → 1.857) +refine→peak_control : target -39.4% vs prior (1.989 → 1.205) diff --git a/results/metric_steering/summary.txt b/results/metric_steering/summary.txt new file mode 100644 index 0000000000000000000000000000000000000000..7e45e913ec435f4cfea24eedbcc42ff8d4c218e3 --- /dev/null +++ b/results/metric_steering/summary.txt @@ -0,0 +1,12 @@ +reference plant G=1/(s+1), F=1 | n=16 ICs=4 | steer λ=1.0 + set ise iae itae control_ control_ peak_con peak_dev settling + unguided 1.906 1.593 1.417 0.992 0.410 1.932 2.082 3.553 +steer→control_effort 1.904 1.592 1.414 0.979* 0.411 1.923 2.082 3.552 + steer→ise 1.907* 1.593 1.418 1.007 0.409 1.943 2.082 3.553 + steer→settling 1.907 1.594 1.419 0.998 0.408 1.936 2.082 3.556* +steer→peak_control 1.904 1.592 1.414 0.980 0.411 1.924* 2.082 3.552 + +steer→control_effort: target metric -1.4% vs unguided (0.992 → 0.979) +steer→ise : target metric +0.1% vs unguided (1.906 → 1.907) +steer→settling : target metric +0.1% vs unguided (3.553 → 3.556) +steer→peak_control : target metric -0.4% vs unguided (1.932 → 1.924) diff --git a/results/mimo/k_sweep_mimo.txt b/results/mimo/k_sweep_mimo.txt new file mode 100644 index 0000000000000000000000000000000000000000..999c636c3f716c027c87b3cfb7f6b37bff37484f --- /dev/null +++ b/results/mimo/k_sweep_mimo.txt @@ -0,0 +1,11 @@ +=== K-sweep MIMO (N=60, seed 9999, gen 65.4 ms/ctrl, per-sample stab 0.963) === + K plants>=1 stable gen time/plant + 1 0.950 0.07 s + 2 0.983 0.13 s + 4 1.000 0.26 s + 8 1.000 0.52 s + 16 1.000 1.05 s + 32 1.000 2.09 s + +contrast: SISO needed K~32 to reach 1.0 (mixed random-in-sector dataset, per-sample ~0.66); + MIMO trained on cost-optimal CERTIFIED targets -> saturates at ~1.0 almost immediately. diff --git a/results/mimo/target_ablation.md b/results/mimo/target_ablation.md new file mode 100644 index 0000000000000000000000000000000000000000..29487b8fd4bc4cc4292ed5365b87bd52da4dd2e9 --- /dev/null +++ b/results/mimo/target_ablation.md @@ -0,0 +1,6 @@ +**Table (§4.3 ablation).** MIMO training-target ablation: random-in-sector vs cost-optimal, same held-out plants (N=120, K=8, seed 9999), same per-plant oracle & phi=0 baseline. + +| training targets | per-sample stability | plants stabilizable | regret vs oracle (med / mean) | cost vs phi=0 (med) | +|---|---|---|---|---| +| random (§6m) | 0.86 | 0.93 | +19.2% / +26.8% | +16.3% | +| cost-optimal (§6n) | 0.96 | 1.00 | +2.0% / +3.8% | +0.2% | diff --git a/results/pareto/fig_pareto.pdf b/results/pareto/fig_pareto.pdf new file mode 100644 index 0000000000000000000000000000000000000000..79b4b0b0948d28a992098c3b4e9b4c47537c88d9 Binary files /dev/null and b/results/pareto/fig_pareto.pdf differ diff --git a/results/pareto/fig_pareto.png b/results/pareto/fig_pareto.png new file mode 100644 index 0000000000000000000000000000000000000000..13e150931199e9c1e2b98b4ef331f76b07e1b9c5 Binary files /dev/null and b/results/pareto/fig_pareto.png differ diff --git a/results/pareto/table_pareto.md b/results/pareto/table_pareto.md new file mode 100644 index 0000000000000000000000000000000000000000..af3e2a2cb2250db48d1292faa21b116a49f01f28 --- /dev/null +++ b/results/pareto/table_pareto.md @@ -0,0 +1,9 @@ +**Table 2.** Cost-steered controller vs analytic optimum per cost weight rho (plant G=1/(s+1), 9 ICs). Steering = refine a fixed model controller on the differentiable J_rho. Regret = (achieved J_rho − V_rho)/V_rho. + +| rho | V_rho (optimal) | achieved J_rho | regret % | state ∫q | control ∫u² | +|---|---|---|---|---|---| +| 0.1 | 1.044 | 1.063 | +1.83 | 0.731 | 3.321 | +| 0.3 | 1.499 | 1.503 | +0.25 | 1.078 | 1.416 | +| 1 | 1.980 | 1.982 | +0.13 | 1.635 | 0.347 | +| 3 | 2.276 | 2.278 | +0.09 | 2.088 | 0.064 | +| 10 | 2.431 | 2.433 | +0.10 | 2.352 | 0.008 | diff --git a/results/pareto_specs/run_s0.log b/results/pareto_specs/run_s0.log new file mode 100644 index 0000000000000000000000000000000000000000..5e33536a651edda126178a23e42835d3b1bee007 --- /dev/null +++ b/results/pareto_specs/run_s0.log @@ -0,0 +1,28 @@ + plant 0: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.498 GD(prior)=0.917 warm=510ms cold=3572ms/pt + plant 1: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.000 GD(prior)=9.519 warm=490ms cold=3358ms/pt + plant 2: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=2.306 GD(prior)=0.963 warm=474ms cold=3398ms/pt + plant 3: prior_cloud=16 ours_front=10 oracle_front=9 GD(ours)=0.878 GD(prior)=3.064 warm=434ms cold=3358ms/pt + plant 4: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=7.361 GD(prior)=1.604 warm=443ms cold=3365ms/pt + plant 5: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=3.066 GD(prior)=1.507 warm=452ms cold=3383ms/pt + plant 6: prior_cloud=16 ours_front=10 oracle_front=2 GD(ours)=0.047 GD(prior)=1.535 warm=450ms cold=3222ms/pt + plant 8: prior_cloud=16 ours_front=10 oracle_front=9 GD(ours)=0.039 GD(prior)=0.481 warm=440ms cold=3041ms/pt + plant 9: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.007 GD(prior)=8.233 warm=454ms cold=3148ms/pt + +=== SPEC-PARETO SUMMARY === +plants evaluated=9 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80 +prior pass: 24.3s for 160 controllers (152 ms each) + +Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer): + OURS (warm-steered front): median 0.498 + PRIOR cloud front (no refine): median 1.535 + +Cost to trace ONE Pareto point (per plant): + OURS warm-start, 12 steps: 452 ms + CLASSICAL cold NLP, 80 steps: 3358 ms -> 7.4× slower per point + +Certified (sector-valid) front points: median 100% + +Binding-spec baseline (classical 'optimize only the most stringent req' = settling): + -> yields ONE corner controller, not the surface (see figure). + +saved -> results/pareto_specs/pareto_specs_s0.npz diff --git a/results/pareto_specs/run_s1.log b/results/pareto_specs/run_s1.log new file mode 100644 index 0000000000000000000000000000000000000000..3c8a4185047c93e93fccf8f01ee85a1e684a7abf --- /dev/null +++ b/results/pareto_specs/run_s1.log @@ -0,0 +1,26 @@ + plant 0: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=2.325 GD(prior)=1.225 warm=537ms cold=3508ms/pt + plant 1: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=1.105 GD(prior)=0.985 warm=502ms cold=3496ms/pt + plant 2: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.050 GD(prior)=14.175 warm=491ms cold=3368ms/pt + plant 3: prior_cloud=15 ours_front=10 oracle_front=10 GD(ours)=0.015 GD(prior)=2.730 warm=482ms cold=3263ms/pt + plant 7: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.125 GD(prior)=2.600 warm=484ms cold=3530ms/pt + plant 8: prior_cloud=16 ours_front=10 oracle_front=6 GD(ours)=1.002 GD(prior)=1.098 warm=468ms cold=3416ms/pt + plant 9: prior_cloud=14 ours_front=10 oracle_front=7 GD(ours)=0.040 GD(prior)=1.691 warm=492ms cold=3489ms/pt + +=== SPEC-PARETO SUMMARY === +plants evaluated=7 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80 +prior pass: 25.4s for 160 controllers (159 ms each) + +Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer): + OURS (warm-steered front): median 0.125 + PRIOR cloud front (no refine): median 1.691 + +Cost to trace ONE Pareto point (per plant): + OURS warm-start, 12 steps: 491 ms + CLASSICAL cold NLP, 80 steps: 3489 ms -> 7.1× slower per point + +Certified (sector-valid) front points: median 100% + +Binding-spec baseline (classical 'optimize only the most stringent req' = settling): + -> yields ONE corner controller, not the surface (see figure). + +saved -> results/pareto_specs/pareto_specs_s1.npz diff --git a/results/pareto_specs/run_s2.log b/results/pareto_specs/run_s2.log new file mode 100644 index 0000000000000000000000000000000000000000..355bded08f48c7294cc6ed4bf0c537345d9bb1d3 --- /dev/null +++ b/results/pareto_specs/run_s2.log @@ -0,0 +1,29 @@ + plant 0: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=7.064 GD(prior)=2.149 warm=546ms cold=3524ms/pt + plant 1: prior_cloud=16 ours_front=10 oracle_front=5 GD(ours)=0.140 GD(prior)=1.773 warm=443ms cold=3508ms/pt + plant 2: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.069 GD(prior)=8.234 warm=456ms cold=3467ms/pt + plant 3: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=17.344 GD(prior)=4.264 warm=586ms cold=3411ms/pt + plant 4: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.135 GD(prior)=0.822 warm=487ms cold=3433ms/pt + plant 5: prior_cloud=14 ours_front=10 oracle_front=2 GD(ours)=0.003 GD(prior)=3.755 warm=514ms cold=3571ms/pt + plant 6: prior_cloud=16 ours_front=10 oracle_front=9 GD(ours)=0.864 GD(prior)=1.401 warm=550ms cold=3300ms/pt + plant 7: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.546 GD(prior)=41.553 warm=440ms cold=3012ms/pt + plant 8: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=389.233 GD(prior)=141.003 warm=571ms cold=3206ms/pt + plant 9: prior_cloud=12 ours_front=6 oracle_front=5 GD(ours)=0.218 GD(prior)=0.292 warm=442ms cold=2951ms/pt + +=== SPEC-PARETO SUMMARY === +plants evaluated=10 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80 +prior pass: 22.0s for 160 controllers (138 ms each) + +Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer): + OURS (warm-steered front): median 0.382 + PRIOR cloud front (no refine): median 2.952 + +Cost to trace ONE Pareto point (per plant): + OURS warm-start, 12 steps: 500 ms + CLASSICAL cold NLP, 80 steps: 3422 ms -> 6.8× slower per point + +Certified (sector-valid) front points: median 100% + +Binding-spec baseline (classical 'optimize only the most stringent req' = settling): + -> yields ONE corner controller, not the surface (see figure). + +saved -> results/pareto_specs/pareto_specs_s2.npz diff --git a/results/pareto_specs/run_s3.log b/results/pareto_specs/run_s3.log new file mode 100644 index 0000000000000000000000000000000000000000..2f9d9c0cb1bd5f7b3aa287a38de69ad6476be2e2 --- /dev/null +++ b/results/pareto_specs/run_s3.log @@ -0,0 +1,29 @@ + plant 0: prior_cloud=14 ours_front=10 oracle_front=10 GD(ours)=1.703 GD(prior)=1.502 warm=485ms cold=3322ms/pt + plant 1: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.724 GD(prior)=1.843 warm=473ms cold=3491ms/pt + plant 2: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.002 GD(prior)=20.795 warm=485ms cold=3509ms/pt + plant 3: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=5.185 GD(prior)=23.308 warm=459ms cold=3587ms/pt + plant 4: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=4.998 GD(prior)=1.749 warm=576ms cold=3684ms/pt + plant 5: prior_cloud=16 ours_front=10 oracle_front=2 GD(ours)=0.220 GD(prior)=1.960 warm=535ms cold=3218ms/pt + plant 6: prior_cloud=16 ours_front=10 oracle_front=1 GD(ours)=0.007 GD(prior)=125.069 warm=539ms cold=3641ms/pt + plant 7: prior_cloud=14 ours_front=10 oracle_front=10 GD(ours)=0.017 GD(prior)=27.143 warm=475ms cold=2982ms/pt + plant 8: prior_cloud=16 ours_front=10 oracle_front=2 GD(ours)=0.037 GD(prior)=1.681 warm=487ms cold=3164ms/pt + plant 9: prior_cloud=16 ours_front=10 oracle_front=9 GD(ours)=0.121 GD(prior)=2.394 warm=432ms cold=2892ms/pt + +=== SPEC-PARETO SUMMARY === +plants evaluated=10 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80 +prior pass: 24.5s for 160 controllers (153 ms each) + +Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer): + OURS (warm-steered front): median 0.171 + PRIOR cloud front (no refine): median 2.177 + +Cost to trace ONE Pareto point (per plant): + OURS warm-start, 12 steps: 485 ms + CLASSICAL cold NLP, 80 steps: 3407 ms -> 7.0× slower per point + +Certified (sector-valid) front points: median 100% + +Binding-spec baseline (classical 'optimize only the most stringent req' = settling): + -> yields ONE corner controller, not the surface (see figure). + +saved -> results/pareto_specs/pareto_specs_s3.npz diff --git a/results/pareto_specs/run_s4.log b/results/pareto_specs/run_s4.log new file mode 100644 index 0000000000000000000000000000000000000000..64509fe2de0edcc92d27d464c2b741519eddc200 --- /dev/null +++ b/results/pareto_specs/run_s4.log @@ -0,0 +1,28 @@ + plant 0: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=0.038 GD(prior)=530.974 warm=481ms cold=3474ms/pt + plant 2: prior_cloud=16 ours_front=10 oracle_front=2 GD(ours)=0.093 GD(prior)=2.589 warm=514ms cold=3507ms/pt + plant 3: prior_cloud=16 ours_front=10 oracle_front=2 GD(ours)=0.525 GD(prior)=5.005 warm=511ms cold=3623ms/pt + plant 4: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=9.910 GD(prior)=2.355 warm=485ms cold=3417ms/pt + plant 5: prior_cloud=16 ours_front=10 oracle_front=6 GD(ours)=8.178 GD(prior)=2.567 warm=433ms cold=3733ms/pt + plant 6: prior_cloud=10 ours_front=10 oracle_front=6 GD(ours)=1.339 GD(prior)=1.180 warm=449ms cold=3419ms/pt + plant 7: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=3.290 GD(prior)=1.603 warm=478ms cold=3313ms/pt + plant 8: prior_cloud=16 ours_front=10 oracle_front=10 GD(ours)=15.436 GD(prior)=5.013 warm=462ms cold=3069ms/pt + plant 9: prior_cloud=15 ours_front=10 oracle_front=1 GD(ours)=0.085 GD(prior)=159.701 warm=463ms cold=3094ms/pt + +=== SPEC-PARETO SUMMARY === +plants evaluated=9 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80 +prior pass: 21.0s for 160 controllers (131 ms each) + +Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer): + OURS (warm-steered front): median 1.339 + PRIOR cloud front (no refine): median 2.589 + +Cost to trace ONE Pareto point (per plant): + OURS warm-start, 12 steps: 478 ms + CLASSICAL cold NLP, 80 steps: 3419 ms -> 7.2× slower per point + +Certified (sector-valid) front points: median 100% + +Binding-spec baseline (classical 'optimize only the most stringent req' = settling): + -> yields ONE corner controller, not the surface (see figure). + +saved -> results/pareto_specs/pareto_specs_s4.npz diff --git a/results/pareto_specs/significance.txt b/results/pareto_specs/significance.txt new file mode 100644 index 0000000000000000000000000000000000000000..5e8cdf9e7625d09f254d56949763725882f6ae44 --- /dev/null +++ b/results/pareto_specs/significance.txt @@ -0,0 +1,15 @@ +pooled plants N = 45 (over 5 seeds) + +CLAIM 1 — steering helps (ours_gd < prior_gd, paired): + win rate: 30/45 = 67% sign-test p = 1.78e-02 + Wilcoxon signed-rank (ours 6.7× slower per point + +Certified (sector-valid) front points: median 100% + +Binding-spec baseline (classical 'optimize only the most stringent req' = settling): + -> yields ONE corner controller, not the surface (see figure). diff --git a/results/pareto_specs/summary_s0.txt b/results/pareto_specs/summary_s0.txt new file mode 100644 index 0000000000000000000000000000000000000000..4ba10405dc2ce2d162edc3ad6d7c60340727d167 --- /dev/null +++ b/results/pareto_specs/summary_s0.txt @@ -0,0 +1,15 @@ +plants evaluated=9 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80 +prior pass: 24.3s for 160 controllers (152 ms each) + +Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer): + OURS (warm-steered front): median 0.498 + PRIOR cloud front (no refine): median 1.535 + +Cost to trace ONE Pareto point (per plant): + OURS warm-start, 12 steps: 452 ms + CLASSICAL cold NLP, 80 steps: 3358 ms -> 7.4× slower per point + +Certified (sector-valid) front points: median 100% + +Binding-spec baseline (classical 'optimize only the most stringent req' = settling): + -> yields ONE corner controller, not the surface (see figure). diff --git a/results/pareto_specs/summary_s1.txt b/results/pareto_specs/summary_s1.txt new file mode 100644 index 0000000000000000000000000000000000000000..77001af29be460b83e9ed69ebe412ab05c292154 --- /dev/null +++ b/results/pareto_specs/summary_s1.txt @@ -0,0 +1,15 @@ +plants evaluated=7 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80 +prior pass: 25.4s for 160 controllers (159 ms each) + +Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer): + OURS (warm-steered front): median 0.125 + PRIOR cloud front (no refine): median 1.691 + +Cost to trace ONE Pareto point (per plant): + OURS warm-start, 12 steps: 491 ms + CLASSICAL cold NLP, 80 steps: 3489 ms -> 7.1× slower per point + +Certified (sector-valid) front points: median 100% + +Binding-spec baseline (classical 'optimize only the most stringent req' = settling): + -> yields ONE corner controller, not the surface (see figure). diff --git a/results/pareto_specs/summary_s2.txt b/results/pareto_specs/summary_s2.txt new file mode 100644 index 0000000000000000000000000000000000000000..f6d4a823b3904bf13775dcda421b32592c2c38c4 --- /dev/null +++ b/results/pareto_specs/summary_s2.txt @@ -0,0 +1,15 @@ +plants evaluated=10 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80 +prior pass: 22.0s for 160 controllers (138 ms each) + +Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer): + OURS (warm-steered front): median 0.382 + PRIOR cloud front (no refine): median 2.952 + +Cost to trace ONE Pareto point (per plant): + OURS warm-start, 12 steps: 500 ms + CLASSICAL cold NLP, 80 steps: 3422 ms -> 6.8× slower per point + +Certified (sector-valid) front points: median 100% + +Binding-spec baseline (classical 'optimize only the most stringent req' = settling): + -> yields ONE corner controller, not the surface (see figure). diff --git a/results/pareto_specs/summary_s3.txt b/results/pareto_specs/summary_s3.txt new file mode 100644 index 0000000000000000000000000000000000000000..320b655383c8df23f30d0941c8818a329058b6b2 --- /dev/null +++ b/results/pareto_specs/summary_s3.txt @@ -0,0 +1,15 @@ +plants evaluated=10 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80 +prior pass: 24.5s for 160 controllers (153 ms each) + +Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer): + OURS (warm-steered front): median 0.171 + PRIOR cloud front (no refine): median 2.177 + +Cost to trace ONE Pareto point (per plant): + OURS warm-start, 12 steps: 485 ms + CLASSICAL cold NLP, 80 steps: 3407 ms -> 7.0× slower per point + +Certified (sector-valid) front points: median 100% + +Binding-spec baseline (classical 'optimize only the most stringent req' = settling): + -> yields ONE corner controller, not the surface (see figure). diff --git a/results/pareto_specs/summary_s4.txt b/results/pareto_specs/summary_s4.txt new file mode 100644 index 0000000000000000000000000000000000000000..343e5f723ed7550b28d3e7e596e1321afc9efa91 --- /dev/null +++ b/results/pareto_specs/summary_s4.txt @@ -0,0 +1,15 @@ +plants evaluated=9 prior K=16 weights/front=10 ICs=4 warm_steps=12 cold_steps=80 +prior pass: 21.0s for 160 controllers (131 ms each) + +Generational distance to the per-plant ORACLE Pareto front (normalized spec space, lower=closer): + OURS (warm-steered front): median 1.339 + PRIOR cloud front (no refine): median 2.589 + +Cost to trace ONE Pareto point (per plant): + OURS warm-start, 12 steps: 478 ms + CLASSICAL cold NLP, 80 steps: 3419 ms -> 7.2× slower per point + +Certified (sector-valid) front points: median 100% + +Binding-spec baseline (classical 'optimize only the most stringent req' = settling): + -> yields ONE corner controller, not the surface (see figure). diff --git a/results/physics_gen/pendulum_physics_gen.txt b/results/physics_gen/pendulum_physics_gen.txt new file mode 100644 index 0000000000000000000000000000000000000000..8b8b5094112cdfb82d4ad1905936612a87c4baa4 --- /dev/null +++ b/results/physics_gen/pendulum_physics_gen.txt @@ -0,0 +1,720 @@ +=== lvl0 K=8 oracle=old (median oracle J 1.352) === +arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired +base_s0 1.000 1.000 1.000 1.09 | 1.000 1.000 1.09 16.0 0.00 +base_s1 1.000 1.000 1.000 1.15 | 1.000 1.000 1.15 16.0 0.00 +base_s2 1.000 1.000 1.000 1.07 | 1.000 1.000 1.07 16.0 0.00 +lz_s42_g0 1.000 1.000 1.000 1.10 | 1.000 1.000 1.10 16.0 0.00 +lz_s42_g1 1.000 1.000 1.000 1.05 | 1.000 1.000 1.05 16.0 0.00 +lz_s42_g2 1.000 1.000 1.000 1.06 | 1.000 1.000 1.06 16.0 0.00 +lz_s43_g0 1.000 1.000 1.000 1.18 | 1.000 1.000 1.18 16.0 0.00 +lz_s44_g0 1.000 1.000 1.000 1.04 | 1.000 1.000 1.04 16.0 0.00 +lzm_s42_g0 1.000 1.000 1.000 1.24 | 1.000 1.000 1.24 16.0 0.00 +lzm_s43_g0 1.000 1.000 1.000 1.31 | 1.000 1.000 1.31 16.0 0.00 +lzm_s44_g0 1.000 1.000 1.000 1.03 | 1.000 1.000 1.03 16.0 0.00 +phys_s42_g0 1.000 1.000 1.000 1.28 | 1.000 1.000 1.28 16.0 0.00 +phys_s43_g0 1.000 1.000 1.000 1.22 | 1.000 1.000 1.22 16.0 0.00 +phys_s44_g0 1.000 1.000 1.000 1.06 | 1.000 1.000 1.06 16.0 0.00 +nn 1.000 1.000 1.000 0.29 | 1.000 1.000 0.29 16.0 0.00 +idw 1.000 1.000 1.000 0.29 | 1.000 1.000 0.29 16.0 0.00 +nns 1.000 1.000 1.000 1.11 | 1.000 1.000 1.11 16.0 0.00 +dls 1.000 1.000 1.000 1.14 | 1.000 1.000 1.14 16.0 0.00 +lzres_s42_g0 1.000 1.000 1.000 1.18 | 1.000 1.000 1.18 16.0 0.00 +lzres_s43_g0 1.000 1.000 1.000 1.18 | 1.000 1.000 1.18 16.0 0.00 +lzres_s44_g0 1.000 1.000 1.000 1.19 | 1.000 1.000 1.19 16.0 0.00 + +-- paired Wilcoxon on regret, model minus best retrieval (B: best=nn; noB: best=nn); negative = model better; [95% CI of paired median] +base base_s0 B vs nn : +0.68 [+0.46,+0.84] p=2.21e-10 certified 1.000 vs 1.000 +base base_s0 noB vs nn : +0.68 [+0.46,+0.84] p=2.21e-10 certified 1.000 vs 1.000 +base base_s1 B vs nn : +0.76 [+0.56,+0.96] p=1.74e-10 certified 1.000 vs 1.000 +base base_s1 noB vs nn : +0.76 [+0.56,+0.96] p=1.74e-10 certified 1.000 vs 1.000 +base base_s2 B vs nn : +0.68 [+0.57,+0.80] p=1.58e-10 certified 1.000 vs 1.000 +base base_s2 noB vs nn : +0.68 [+0.57,+0.80] p=1.58e-10 certified 1.000 vs 1.000 +A_lz lz_s42_g0 B vs nn : +0.69 [+0.56,+0.89] p=1.9e-11 certified 1.000 vs 1.000 +A_lz lz_s42_g0 noB vs nn : +0.69 [+0.56,+0.89] p=1.9e-11 certified 1.000 vs 1.000 +A_lz lz_s42_g1 B vs nn : +0.67 [+0.60,+0.77] p=3.29e-11 certified 1.000 vs 1.000 +A_lz lz_s42_g1 noB vs nn : +0.67 [+0.60,+0.77] p=3.29e-11 certified 1.000 vs 1.000 +A_lz lz_s42_g2 B vs nn : +0.72 [+0.63,+0.83] p=6.59e-11 certified 1.000 vs 1.000 +A_lz lz_s42_g2 noB vs nn : +0.72 [+0.63,+0.83] p=6.59e-11 certified 1.000 vs 1.000 +A_lz lz_s43_g0 B vs nn : +0.76 [+0.65,+0.95] p=1.71e-11 certified 1.000 vs 1.000 +A_lz lz_s43_g0 noB vs nn : +0.76 [+0.65,+0.95] p=1.71e-11 certified 1.000 vs 1.000 +A_lz lz_s44_g0 B vs nn : +0.62 [+0.49,+0.78] p=2.84e-11 certified 1.000 vs 1.000 +A_lz lz_s44_g0 noB vs nn : +0.62 [+0.49,+0.78] p=2.84e-11 certified 1.000 vs 1.000 +A_lzm lzm_s42_g0 B vs nn : +0.82 [+0.73,+0.96] p=1.63e-11 certified 1.000 vs 1.000 +A_lzm lzm_s42_g0 noB vs nn : +0.82 [+0.73,+0.96] p=1.63e-11 certified 1.000 vs 1.000 +A_lzm lzm_s43_g0 B vs nn : +0.88 [+0.78,+1.00] p=1.63e-11 certified 1.000 vs 1.000 +A_lzm lzm_s43_g0 noB vs nn : +0.88 [+0.78,+1.00] p=1.63e-11 certified 1.000 vs 1.000 +A_lzm lzm_s44_g0 B vs nn : +0.59 [+0.46,+0.71] p=1.63e-11 certified 1.000 vs 1.000 +A_lzm lzm_s44_g0 noB vs nn : +0.59 [+0.46,+0.71] p=1.63e-11 certified 1.000 vs 1.000 +A_lzres lzres_s42_g0 B vs nn : +0.92 [+0.72,+0.98] p=1.9e-11 certified 1.000 vs 1.000 +A_lzres lzres_s42_g0 noB vs nn : +0.92 [+0.72,+0.98] p=1.9e-11 certified 1.000 vs 1.000 +A_lzres lzres_s43_g0 B vs nn : +0.92 [+0.76,+0.97] p=1.9e-11 certified 1.000 vs 1.000 +A_lzres lzres_s43_g0 noB vs nn : +0.92 [+0.76,+0.97] p=1.9e-11 certified 1.000 vs 1.000 +A_lzres lzres_s44_g0 B vs nn : +0.95 [+0.74,+1.03] p=1.71e-11 certified 1.000 vs 1.000 +A_lzres lzres_s44_g0 noB vs nn : +0.95 [+0.74,+1.03] p=1.71e-11 certified 1.000 vs 1.000 +A_phys phys_s42_g0 B vs nn : +0.93 [+0.83,+1.05] p=1.63e-11 certified 1.000 vs 1.000 +A_phys phys_s42_g0 noB vs nn : +0.93 [+0.83,+1.05] p=1.63e-11 certified 1.000 vs 1.000 +A_phys phys_s43_g0 B vs nn : +0.82 [+0.68,+0.95] p=1.63e-11 certified 1.000 vs 1.000 +A_phys phys_s43_g0 noB vs nn : +0.82 [+0.68,+0.95] p=1.63e-11 certified 1.000 vs 1.000 +A_phys phys_s44_g0 B vs nn : +0.66 [+0.51,+0.76] p=1.63e-11 certified 1.000 vs 1.000 +A_phys phys_s44_g0 noB vs nn : +0.66 [+0.51,+0.76] p=1.63e-11 certified 1.000 vs 1.000 + +=== lvl0 K=8 oracle=strong2 (median oracle J 1.351) === +arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired +base_s0 1.000 1.000 1.000 1.11 | 1.000 1.000 1.11 16.0 0.00 +base_s1 1.000 1.000 1.000 1.17 | 1.000 1.000 1.17 16.0 0.00 +base_s2 1.000 1.000 1.000 1.12 | 1.000 1.000 1.12 16.0 0.00 +lz_s42_g0 1.000 1.000 1.000 1.15 | 1.000 1.000 1.15 16.0 0.00 +lz_s42_g1 1.000 1.000 1.000 1.14 | 1.000 1.000 1.14 16.0 0.00 +lz_s42_g2 1.000 1.000 1.000 1.10 | 1.000 1.000 1.10 16.0 0.00 +lz_s43_g0 1.000 1.000 1.000 1.23 | 1.000 1.000 1.23 16.0 0.00 +lz_s44_g0 1.000 1.000 1.000 1.12 | 1.000 1.000 1.12 16.0 0.00 +lzm_s42_g0 1.000 1.000 1.000 1.32 | 1.000 1.000 1.32 16.0 0.00 +lzm_s43_g0 1.000 1.000 1.000 1.38 | 1.000 1.000 1.38 16.0 0.00 +lzm_s44_g0 1.000 1.000 1.000 1.11 | 1.000 1.000 1.11 16.0 0.00 +phys_s42_g0 1.000 1.000 1.000 1.37 | 1.000 1.000 1.37 16.0 0.00 +phys_s43_g0 1.000 1.000 1.000 1.30 | 1.000 1.000 1.30 16.0 0.00 +phys_s44_g0 1.000 1.000 1.000 1.15 | 1.000 1.000 1.15 16.0 0.00 +nn 1.000 1.000 1.000 0.48 | 1.000 1.000 0.48 16.0 0.00 +idw 1.000 1.000 1.000 0.48 | 1.000 1.000 0.48 16.0 0.00 +nns 1.000 1.000 1.000 1.12 | 1.000 1.000 1.12 16.0 0.00 +dls 1.000 1.000 1.000 1.23 | 1.000 1.000 1.23 16.0 0.00 +lzres_s42_g0 1.000 1.000 1.000 1.29 | 1.000 1.000 1.29 16.0 0.00 +lzres_s43_g0 1.000 1.000 1.000 1.28 | 1.000 1.000 1.28 16.0 0.00 +lzres_s44_g0 1.000 1.000 1.000 1.30 | 1.000 1.000 1.30 16.0 0.00 + +-- paired Wilcoxon on regret, model minus best retrieval (B: best=nn; noB: best=nn); negative = model better; [95% CI of paired median] +base base_s0 B vs nn : +0.68 [+0.46,+0.84] p=2.21e-10 certified 1.000 vs 1.000 +base base_s0 noB vs nn : +0.68 [+0.46,+0.84] p=2.21e-10 certified 1.000 vs 1.000 +base base_s1 B vs nn : +0.76 [+0.56,+0.96] p=1.74e-10 certified 1.000 vs 1.000 +base base_s1 noB vs nn : +0.76 [+0.56,+0.96] p=1.74e-10 certified 1.000 vs 1.000 +base base_s2 B vs nn : +0.68 [+0.57,+0.80] p=1.58e-10 certified 1.000 vs 1.000 +base base_s2 noB vs nn : +0.68 [+0.57,+0.80] p=1.58e-10 certified 1.000 vs 1.000 +A_lz lz_s42_g0 B vs nn : +0.69 [+0.56,+0.89] p=1.9e-11 certified 1.000 vs 1.000 +A_lz lz_s42_g0 noB vs nn : +0.69 [+0.56,+0.89] p=1.9e-11 certified 1.000 vs 1.000 +A_lz lz_s42_g1 B vs nn : +0.68 [+0.60,+0.77] p=3.29e-11 certified 1.000 vs 1.000 +A_lz lz_s42_g1 noB vs nn : +0.68 [+0.60,+0.77] p=3.29e-11 certified 1.000 vs 1.000 +A_lz lz_s42_g2 B vs nn : +0.72 [+0.63,+0.83] p=6.92e-11 certified 1.000 vs 1.000 +A_lz lz_s42_g2 noB vs nn : +0.72 [+0.63,+0.83] p=6.92e-11 certified 1.000 vs 1.000 +A_lz lz_s43_g0 B vs nn : +0.76 [+0.65,+0.95] p=1.71e-11 certified 1.000 vs 1.000 +A_lz lz_s43_g0 noB vs nn : +0.76 [+0.65,+0.95] p=1.71e-11 certified 1.000 vs 1.000 +A_lz lz_s44_g0 B vs nn : +0.62 [+0.49,+0.78] p=2.84e-11 certified 1.000 vs 1.000 +A_lz lz_s44_g0 noB vs nn : +0.62 [+0.49,+0.78] p=2.84e-11 certified 1.000 vs 1.000 +A_lzm lzm_s42_g0 B vs nn : +0.82 [+0.73,+0.96] p=1.63e-11 certified 1.000 vs 1.000 +A_lzm lzm_s42_g0 noB vs nn : +0.82 [+0.73,+0.96] p=1.63e-11 certified 1.000 vs 1.000 +A_lzm lzm_s43_g0 B vs nn : +0.88 [+0.78,+1.00] p=1.63e-11 certified 1.000 vs 1.000 +A_lzm lzm_s43_g0 noB vs nn : +0.88 [+0.78,+1.00] p=1.63e-11 certified 1.000 vs 1.000 +A_lzm lzm_s44_g0 B vs nn : +0.59 [+0.46,+0.71] p=1.63e-11 certified 1.000 vs 1.000 +A_lzm lzm_s44_g0 noB vs nn : +0.59 [+0.46,+0.71] p=1.63e-11 certified 1.000 vs 1.000 +A_lzres lzres_s42_g0 B vs nn : +0.92 [+0.72,+0.98] p=1.9e-11 certified 1.000 vs 1.000 +A_lzres lzres_s42_g0 noB vs nn : +0.92 [+0.72,+0.98] p=1.9e-11 certified 1.000 vs 1.000 +A_lzres lzres_s43_g0 B vs nn : +0.92 [+0.78,+0.97] p=1.9e-11 certified 1.000 vs 1.000 +A_lzres lzres_s43_g0 noB vs nn : +0.92 [+0.78,+0.97] p=1.9e-11 certified 1.000 vs 1.000 +A_lzres lzres_s44_g0 B vs nn : +0.95 [+0.74,+1.03] p=1.71e-11 certified 1.000 vs 1.000 +A_lzres lzres_s44_g0 noB vs nn : +0.95 [+0.74,+1.03] p=1.71e-11 certified 1.000 vs 1.000 +A_phys phys_s42_g0 B vs nn : +0.93 [+0.83,+1.05] p=1.63e-11 certified 1.000 vs 1.000 +A_phys phys_s42_g0 noB vs nn : +0.93 [+0.83,+1.05] p=1.63e-11 certified 1.000 vs 1.000 +A_phys phys_s43_g0 B vs nn : +0.82 [+0.68,+0.95] p=1.63e-11 certified 1.000 vs 1.000 +A_phys phys_s43_g0 noB vs nn : +0.82 [+0.68,+0.95] p=1.63e-11 certified 1.000 vs 1.000 +A_phys phys_s44_g0 B vs nn : +0.66 [+0.51,+0.76] p=1.63e-11 certified 1.000 vs 1.000 +A_phys phys_s44_g0 noB vs nn : +0.66 [+0.51,+0.76] p=1.63e-11 certified 1.000 vs 1.000 + +=== lvl1.1 K=8 oracle=old (median oracle J 0.984) === +arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired +base_s0 1.000 1.000 1.000 0.92 | 1.000 1.000 0.92 16.0 0.00 +base_s1 1.000 1.000 1.000 0.92 | 1.000 1.000 0.92 16.0 0.00 +base_s2 1.000 1.000 1.000 0.84 | 1.000 1.000 0.84 16.0 0.00 +lz_s42_g0 1.000 1.000 1.000 1.04 | 1.000 1.000 1.04 16.0 0.00 +lz_s42_g1 1.000 1.000 1.000 1.04 | 1.000 1.000 1.04 16.0 0.00 +lz_s42_g2 1.000 1.000 1.000 1.03 | 1.000 1.000 1.03 16.0 0.00 +lz_s43_g0 1.000 1.000 1.000 1.11 | 1.000 1.000 1.11 16.0 0.00 +lz_s44_g0 1.000 1.000 1.000 0.98 | 1.000 1.000 0.98 16.0 0.00 +lzm_s42_g0 1.000 1.000 1.000 1.24 | 1.000 1.000 1.24 16.0 0.00 +lzm_s43_g0 1.000 1.000 1.000 1.28 | 1.000 1.000 1.28 16.0 0.00 +lzm_s44_g0 1.000 1.000 1.000 1.03 | 1.000 1.000 1.03 16.0 0.00 +phys_s42_g0 1.000 1.000 1.000 1.29 | 1.000 1.000 1.29 16.0 0.00 +phys_s43_g0 1.000 1.000 1.000 1.27 | 1.000 1.000 1.27 16.0 0.00 +phys_s44_g0 1.000 1.000 1.000 1.10 | 1.000 1.000 1.10 16.0 0.00 +nn 1.000 1.000 1.000 0.53 | 1.000 1.000 0.53 16.0 0.00 +idw 1.000 1.000 1.000 0.55 | 1.000 1.000 0.55 16.0 0.00 +nns 1.000 1.000 1.000 0.95 | 1.000 1.000 0.95 16.0 0.00 +dls 1.000 1.000 1.000 0.95 | 1.000 1.000 0.95 16.0 0.00 +lzres_s42_g0 1.000 1.000 1.000 1.10 | 1.000 1.000 1.10 16.0 0.00 +lzres_s43_g0 1.000 1.000 1.000 1.09 | 1.000 1.000 1.09 16.0 0.00 +lzres_s44_g0 1.000 1.000 1.000 1.10 | 1.000 1.000 1.10 16.0 0.00 + +-- paired Wilcoxon on regret, model minus best retrieval (B: best=nn; noB: best=nn); negative = model better; [95% CI of paired median] +base base_s0 B vs nn : +0.28 [+0.02,+0.68] p=0.00419 certified 1.000 vs 1.000 +base base_s0 noB vs nn : +0.28 [+0.02,+0.68] p=0.00419 certified 1.000 vs 1.000 +base base_s1 B vs nn : +0.21 [+0.02,+0.69] p=0.00492 certified 1.000 vs 1.000 +base base_s1 noB vs nn : +0.21 [+0.02,+0.69] p=0.00492 certified 1.000 vs 1.000 +base base_s2 B vs nn : +0.31 [+0.01,+0.58] p=0.00753 certified 1.000 vs 1.000 +base base_s2 noB vs nn : +0.31 [+0.01,+0.58] p=0.00753 certified 1.000 vs 1.000 +A_lz lz_s42_g0 B vs nn : +0.63 [+0.35,+0.80] p=0.000179 certified 1.000 vs 1.000 +A_lz lz_s42_g0 noB vs nn : +0.63 [+0.35,+0.80] p=0.000179 certified 1.000 vs 1.000 +A_lz lz_s42_g1 B vs nn : +0.56 [+0.41,+0.80] p=0.000458 certified 1.000 vs 1.000 +A_lz lz_s42_g1 noB vs nn : +0.56 [+0.41,+0.80] p=0.000458 certified 1.000 vs 1.000 +A_lz lz_s42_g2 B vs nn : +0.62 [+0.36,+0.80] p=0.000239 certified 1.000 vs 1.000 +A_lz lz_s42_g2 noB vs nn : +0.62 [+0.36,+0.80] p=0.000239 certified 1.000 vs 1.000 +A_lz lz_s43_g0 B vs nn : +0.68 [+0.49,+0.87] p=6.21e-05 certified 1.000 vs 1.000 +A_lz lz_s43_g0 noB vs nn : +0.68 [+0.49,+0.87] p=6.21e-05 certified 1.000 vs 1.000 +A_lz lz_s44_g0 B vs nn : +0.52 [+0.27,+0.75] p=0.000446 certified 1.000 vs 1.000 +A_lz lz_s44_g0 noB vs nn : +0.52 [+0.27,+0.75] p=0.000446 certified 1.000 vs 1.000 +A_lzm lzm_s42_g0 B vs nn : +0.79 [+0.52,+0.92] p=3.78e-06 certified 1.000 vs 1.000 +A_lzm lzm_s42_g0 noB vs nn : +0.79 [+0.52,+0.92] p=3.78e-06 certified 1.000 vs 1.000 +A_lzm lzm_s43_g0 B vs nn : +0.84 [+0.59,+0.97] p=3.52e-06 certified 1.000 vs 1.000 +A_lzm lzm_s43_g0 noB vs nn : +0.84 [+0.59,+0.97] p=3.52e-06 certified 1.000 vs 1.000 +A_lzm lzm_s44_g0 B vs nn : +0.50 [+0.34,+0.63] p=5.83e-05 certified 1.000 vs 1.000 +A_lzm lzm_s44_g0 noB vs nn : +0.50 [+0.34,+0.63] p=5.83e-05 certified 1.000 vs 1.000 +A_lzres lzres_s42_g0 B vs nn : +0.76 [+0.40,+0.97] p=0.000125 certified 1.000 vs 1.000 +A_lzres lzres_s42_g0 noB vs nn : +0.76 [+0.40,+0.97] p=0.000125 certified 1.000 vs 1.000 +A_lzres lzres_s43_g0 B vs nn : +0.78 [+0.40,+0.97] p=0.000122 certified 1.000 vs 1.000 +A_lzres lzres_s43_g0 noB vs nn : +0.78 [+0.40,+0.97] p=0.000122 certified 1.000 vs 1.000 +A_lzres lzres_s44_g0 B vs nn : +0.79 [+0.39,+0.98] p=0.000118 certified 1.000 vs 1.000 +A_lzres lzres_s44_g0 noB vs nn : +0.79 [+0.39,+0.98] p=0.000118 certified 1.000 vs 1.000 +A_phys phys_s42_g0 B vs nn : +0.82 [+0.53,+1.06] p=3.4e-06 certified 1.000 vs 1.000 +A_phys phys_s42_g0 noB vs nn : +0.82 [+0.53,+1.06] p=3.4e-06 certified 1.000 vs 1.000 +A_phys phys_s43_g0 B vs nn : +0.84 [+0.49,+0.98] p=4.06e-06 certified 1.000 vs 1.000 +A_phys phys_s43_g0 noB vs nn : +0.84 [+0.49,+0.98] p=4.06e-06 certified 1.000 vs 1.000 +A_phys phys_s44_g0 B vs nn : +0.64 [+0.38,+0.80] p=2.23e-05 certified 1.000 vs 1.000 +A_phys phys_s44_g0 noB vs nn : +0.64 [+0.38,+0.80] p=2.23e-05 certified 1.000 vs 1.000 + +=== lvl1.1 K=8 oracle=strong2 (median oracle J 0.983) === +arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired +base_s0 1.000 1.000 1.000 0.96 | 1.000 1.000 0.96 16.0 0.00 +base_s1 1.000 1.000 1.000 1.02 | 1.000 1.000 1.02 16.0 0.00 +base_s2 1.000 1.000 1.000 0.93 | 1.000 1.000 0.93 16.0 0.00 +lz_s42_g0 1.000 1.000 1.000 1.18 | 1.000 1.000 1.18 16.0 0.00 +lz_s42_g1 1.000 1.000 1.000 1.14 | 1.000 1.000 1.14 16.0 0.00 +lz_s42_g2 1.000 1.000 1.000 1.14 | 1.000 1.000 1.14 16.0 0.00 +lz_s43_g0 1.000 1.000 1.000 1.29 | 1.000 1.000 1.29 16.0 0.00 +lz_s44_g0 1.000 1.000 1.000 1.10 | 1.000 1.000 1.10 16.0 0.00 +lzm_s42_g0 1.000 1.000 1.000 1.30 | 1.000 1.000 1.30 16.0 0.00 +lzm_s43_g0 1.000 1.000 1.000 1.35 | 1.000 1.000 1.35 16.0 0.00 +lzm_s44_g0 1.000 1.000 1.000 1.12 | 1.000 1.000 1.12 16.0 0.00 +phys_s42_g0 1.000 1.000 1.000 1.36 | 1.000 1.000 1.36 16.0 0.00 +phys_s43_g0 1.000 1.000 1.000 1.30 | 1.000 1.000 1.30 16.0 0.00 +phys_s44_g0 1.000 1.000 1.000 1.16 | 1.000 1.000 1.16 16.0 0.00 +nn 1.000 1.000 1.000 0.80 | 1.000 1.000 0.80 16.0 0.00 +idw 1.000 1.000 1.000 0.80 | 1.000 1.000 0.80 16.0 0.00 +nns 1.000 1.000 1.000 1.02 | 1.000 1.000 1.02 16.0 0.00 +dls 1.000 1.000 1.000 1.09 | 1.000 1.000 1.09 16.0 0.00 +lzres_s42_g0 1.000 1.000 1.000 1.19 | 1.000 1.000 1.19 16.0 0.00 +lzres_s43_g0 1.000 1.000 1.000 1.19 | 1.000 1.000 1.19 16.0 0.00 +lzres_s44_g0 1.000 1.000 1.000 1.21 | 1.000 1.000 1.21 16.0 0.00 + +-- paired Wilcoxon on regret, model minus best retrieval (B: best=nn; noB: best=nn); negative = model better; [95% CI of paired median] +base base_s0 B vs nn : +0.28 [+0.02,+0.68] p=0.00419 certified 1.000 vs 1.000 +base base_s0 noB vs nn : +0.28 [+0.02,+0.68] p=0.00419 certified 1.000 vs 1.000 +base base_s1 B vs nn : +0.21 [+0.02,+0.69] p=0.00481 certified 1.000 vs 1.000 +base base_s1 noB vs nn : +0.21 [+0.02,+0.69] p=0.00481 certified 1.000 vs 1.000 +base base_s2 B vs nn : +0.31 [+0.01,+0.58] p=0.00737 certified 1.000 vs 1.000 +base base_s2 noB vs nn : +0.31 [+0.01,+0.58] p=0.00737 certified 1.000 vs 1.000 +A_lz lz_s42_g0 B vs nn : +0.63 [+0.35,+0.80] p=0.000174 certified 1.000 vs 1.000 +A_lz lz_s42_g0 noB vs nn : +0.63 [+0.35,+0.80] p=0.000174 certified 1.000 vs 1.000 +A_lz lz_s42_g1 B vs nn : +0.56 [+0.41,+0.80] p=0.000446 certified 1.000 vs 1.000 +A_lz lz_s42_g1 noB vs nn : +0.56 [+0.41,+0.80] p=0.000446 certified 1.000 vs 1.000 +A_lz lz_s42_g2 B vs nn : +0.62 [+0.36,+0.80] p=0.000239 certified 1.000 vs 1.000 +A_lz lz_s42_g2 noB vs nn : +0.62 [+0.36,+0.80] p=0.000239 certified 1.000 vs 1.000 +A_lz lz_s43_g0 B vs nn : +0.68 [+0.49,+0.87] p=6.21e-05 certified 1.000 vs 1.000 +A_lz lz_s43_g0 noB vs nn : +0.68 [+0.49,+0.87] p=6.21e-05 certified 1.000 vs 1.000 +A_lz lz_s44_g0 B vs nn : +0.52 [+0.27,+0.75] p=0.000446 certified 1.000 vs 1.000 +A_lz lz_s44_g0 noB vs nn : +0.52 [+0.27,+0.75] p=0.000446 certified 1.000 vs 1.000 +A_lzm lzm_s42_g0 B vs nn : +0.79 [+0.52,+0.92] p=3.65e-06 certified 1.000 vs 1.000 +A_lzm lzm_s42_g0 noB vs nn : +0.79 [+0.52,+0.92] p=3.65e-06 certified 1.000 vs 1.000 +A_lzm lzm_s43_g0 B vs nn : +0.84 [+0.59,+0.97] p=3.52e-06 certified 1.000 vs 1.000 +A_lzm lzm_s43_g0 noB vs nn : +0.84 [+0.59,+0.97] p=3.52e-06 certified 1.000 vs 1.000 +A_lzm lzm_s44_g0 B vs nn : +0.50 [+0.34,+0.63] p=5.83e-05 certified 1.000 vs 1.000 +A_lzm lzm_s44_g0 noB vs nn : +0.50 [+0.34,+0.63] p=5.83e-05 certified 1.000 vs 1.000 +A_lzres lzres_s42_g0 B vs nn : +0.76 [+0.40,+0.97] p=0.000122 certified 1.000 vs 1.000 +A_lzres lzres_s42_g0 noB vs nn : +0.76 [+0.40,+0.97] p=0.000122 certified 1.000 vs 1.000 +A_lzres lzres_s43_g0 B vs nn : +0.78 [+0.40,+0.97] p=0.000125 certified 1.000 vs 1.000 +A_lzres lzres_s43_g0 noB vs nn : +0.78 [+0.40,+0.97] p=0.000125 certified 1.000 vs 1.000 +A_lzres lzres_s44_g0 B vs nn : +0.79 [+0.39,+0.98] p=0.000111 certified 1.000 vs 1.000 +A_lzres lzres_s44_g0 noB vs nn : +0.79 [+0.39,+0.98] p=0.000111 certified 1.000 vs 1.000 +A_phys phys_s42_g0 B vs nn : +0.82 [+0.53,+1.06] p=3.4e-06 certified 1.000 vs 1.000 +A_phys phys_s42_g0 noB vs nn : +0.82 [+0.53,+1.06] p=3.4e-06 certified 1.000 vs 1.000 +A_phys phys_s43_g0 B vs nn : +0.84 [+0.49,+0.98] p=4.06e-06 certified 1.000 vs 1.000 +A_phys phys_s43_g0 noB vs nn : +0.84 [+0.49,+0.98] p=4.06e-06 certified 1.000 vs 1.000 +A_phys phys_s44_g0 B vs nn : +0.64 [+0.38,+0.80] p=2.23e-05 certified 1.000 vs 1.000 +A_phys phys_s44_g0 noB vs nn : +0.64 [+0.38,+0.80] p=2.23e-05 certified 1.000 vs 1.000 + +=== lvl1.25 K=8 oracle=old (median oracle J 0.318) === +arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired +base_s0 1.000 1.000 1.000 0.70 | 1.000 1.000 0.70 16.0 0.00 +base_s1 1.000 1.000 1.000 0.67 | 1.000 1.000 0.67 16.0 0.00 +base_s2 1.000 1.000 1.000 0.72 | 1.000 1.000 0.72 16.0 0.00 +lz_s42_g0 1.000 1.000 1.000 1.00 | 1.000 1.000 1.00 16.0 0.00 +lz_s42_g1 1.000 1.000 1.000 0.98 | 1.000 1.000 0.98 16.0 0.00 +lz_s42_g2 1.000 1.000 1.000 1.00 | 1.000 1.000 1.00 16.0 0.00 +lz_s43_g0 1.000 1.000 1.000 1.07 | 1.000 1.000 1.07 16.0 0.00 +lz_s44_g0 1.000 1.000 1.000 0.94 | 1.000 1.000 0.94 16.0 0.00 +lzm_s42_g0 1.000 1.000 1.000 1.14 | 1.000 1.000 1.14 16.0 0.00 +lzm_s43_g0 1.000 1.000 1.000 1.13 | 1.000 1.000 1.13 16.0 0.00 +lzm_s44_g0 1.000 1.000 1.000 1.00 | 1.000 1.000 1.00 16.0 0.00 +phys_s42_g0 1.000 1.000 1.000 1.11 | 1.000 1.000 1.11 16.0 0.00 +phys_s43_g0 1.000 1.000 1.000 1.12 | 1.000 1.000 1.12 16.0 0.00 +phys_s44_g0 1.000 1.000 1.000 1.03 | 1.000 1.000 1.03 16.0 0.00 +nn 1.000 1.000 1.000 1.06 | 1.000 1.000 1.06 16.0 0.00 +idw 1.000 1.000 1.000 1.00 | 1.000 1.000 1.00 16.0 0.00 +nns 1.000 1.000 1.000 0.95 | 1.000 1.000 0.95 16.0 0.00 +dls 1.000 1.000 1.000 0.97 | 1.000 1.000 0.97 16.0 0.00 +lzres_s42_g0 1.000 1.000 1.000 1.00 | 1.000 1.000 1.00 16.0 0.00 +lzres_s43_g0 1.000 1.000 1.000 0.95 | 1.000 1.000 0.95 16.0 0.00 +lzres_s44_g0 1.000 1.000 1.000 0.98 | 1.000 1.000 0.98 16.0 0.00 + +-- paired Wilcoxon on regret, model minus best retrieval (B: best=nns; noB: best=nns); negative = model better; [95% CI of paired median] +base base_s0 B vs nns: -0.17 [-0.24,-0.10] p=0.0026 certified 1.000 vs 1.000 +base base_s0 noB vs nns: -0.17 [-0.24,-0.10] p=0.0026 certified 1.000 vs 1.000 +base base_s1 B vs nns: -0.17 [-0.21,-0.07] p=0.00117 certified 1.000 vs 1.000 +base base_s1 noB vs nns: -0.17 [-0.21,-0.07] p=0.00117 certified 1.000 vs 1.000 +base base_s2 B vs nns: -0.12 [-0.19,-0.03] p=0.00111 certified 1.000 vs 1.000 +base base_s2 noB vs nns: -0.12 [-0.19,-0.03] p=0.00111 certified 1.000 vs 1.000 +A_lz lz_s42_g0 B vs nns: +0.08 [+0.05,+0.12] p=0.00977 certified 1.000 vs 1.000 +A_lz lz_s42_g0 noB vs nns: +0.08 [+0.05,+0.12] p=0.00977 certified 1.000 vs 1.000 +A_lz lz_s42_g1 B vs nns: +0.09 [+0.05,+0.14] p=0.0109 certified 1.000 vs 1.000 +A_lz lz_s42_g1 noB vs nns: +0.09 [+0.05,+0.14] p=0.0109 certified 1.000 vs 1.000 +A_lz lz_s42_g2 B vs nns: +0.10 [+0.05,+0.18] p=0.00822 certified 1.000 vs 1.000 +A_lz lz_s42_g2 noB vs nns: +0.10 [+0.05,+0.18] p=0.00822 certified 1.000 vs 1.000 +A_lz lz_s43_g0 B vs nns: +0.15 [+0.11,+0.25] p=0.000399 certified 1.000 vs 1.000 +A_lz lz_s43_g0 noB vs nns: +0.15 [+0.11,+0.25] p=0.000399 certified 1.000 vs 1.000 +A_lz lz_s44_g0 B vs nns: +0.05 [-0.01,+0.09] p=0.126 certified 1.000 vs 1.000 +A_lz lz_s44_g0 noB vs nns: +0.05 [-0.01,+0.09] p=0.126 certified 1.000 vs 1.000 +A_lzm lzm_s42_g0 B vs nns: +0.41 [+0.32,+0.45] p=2.21e-11 certified 1.000 vs 1.000 +A_lzm lzm_s42_g0 noB vs nns: +0.41 [+0.32,+0.45] p=2.21e-11 certified 1.000 vs 1.000 +A_lzm lzm_s43_g0 B vs nns: +0.38 [+0.35,+0.44] p=2.56e-11 certified 1.000 vs 1.000 +A_lzm lzm_s43_g0 noB vs nns: +0.38 [+0.35,+0.44] p=2.56e-11 certified 1.000 vs 1.000 +A_lzm lzm_s44_g0 B vs nns: +0.11 [+0.07,+0.21] p=5.83e-05 certified 1.000 vs 1.000 +A_lzm lzm_s44_g0 noB vs nns: +0.11 [+0.07,+0.21] p=5.83e-05 certified 1.000 vs 1.000 +A_lzres lzres_s42_g0 B vs nns: +0.10 [+0.06,+0.17] p=0.0234 certified 1.000 vs 1.000 +A_lzres lzres_s42_g0 noB vs nns: +0.10 [+0.06,+0.17] p=0.0234 certified 1.000 vs 1.000 +A_lzres lzres_s43_g0 B vs nns: +0.11 [+0.06,+0.19] p=0.0277 certified 1.000 vs 1.000 +A_lzres lzres_s43_g0 noB vs nns: +0.11 [+0.06,+0.19] p=0.0277 certified 1.000 vs 1.000 +A_lzres lzres_s44_g0 B vs nns: +0.11 [+0.07,+0.22] p=0.0196 certified 1.000 vs 1.000 +A_lzres lzres_s44_g0 noB vs nns: +0.11 [+0.07,+0.22] p=0.0196 certified 1.000 vs 1.000 +A_phys phys_s42_g0 B vs nns: +0.37 [+0.31,+0.43] p=2.1e-11 certified 1.000 vs 1.000 +A_phys phys_s42_g0 noB vs nns: +0.37 [+0.31,+0.43] p=2.1e-11 certified 1.000 vs 1.000 +A_phys phys_s43_g0 B vs nns: +0.36 [+0.27,+0.48] p=2.98e-11 certified 1.000 vs 1.000 +A_phys phys_s43_g0 noB vs nns: +0.36 [+0.27,+0.48] p=2.98e-11 certified 1.000 vs 1.000 +A_phys phys_s44_g0 B vs nns: +0.15 [+0.09,+0.21] p=2.66e-07 certified 1.000 vs 1.000 +A_phys phys_s44_g0 noB vs nns: +0.15 [+0.09,+0.21] p=2.66e-07 certified 1.000 vs 1.000 + +=== lvl1.25 K=8 oracle=strong2 (median oracle J 0.318) === +arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired +base_s0 1.000 1.000 1.000 0.86 | 1.000 1.000 0.86 16.0 0.00 +base_s1 1.000 1.000 1.000 0.90 | 1.000 1.000 0.90 16.0 0.00 +base_s2 1.000 1.000 1.000 0.91 | 1.000 1.000 0.91 16.0 0.00 +lz_s42_g0 1.000 1.000 1.000 1.14 | 1.000 1.000 1.14 16.0 0.00 +lz_s42_g1 1.000 1.000 1.000 1.13 | 1.000 1.000 1.13 16.0 0.00 +lz_s42_g2 1.000 1.000 1.000 1.15 | 1.000 1.000 1.15 16.0 0.00 +lz_s43_g0 1.000 1.000 1.000 1.23 | 1.000 1.000 1.23 16.0 0.00 +lz_s44_g0 1.000 1.000 1.000 1.08 | 1.000 1.000 1.08 16.0 0.00 +lzm_s42_g0 1.000 1.000 1.000 1.29 | 1.000 1.000 1.29 16.0 0.00 +lzm_s43_g0 1.000 1.000 1.000 1.31 | 1.000 1.000 1.31 16.0 0.00 +lzm_s44_g0 1.000 1.000 1.000 1.13 | 1.000 1.000 1.13 16.0 0.00 +phys_s42_g0 1.000 1.000 1.000 1.26 | 1.000 1.000 1.26 16.0 0.00 +phys_s43_g0 1.000 1.000 1.000 1.30 | 1.000 1.000 1.30 16.0 0.00 +phys_s44_g0 1.000 1.000 1.000 1.16 | 1.000 1.000 1.16 16.0 0.00 +nn 1.000 1.000 1.000 1.22 | 1.000 1.000 1.22 16.0 0.00 +idw 1.000 1.000 1.000 1.17 | 1.000 1.000 1.17 16.0 0.00 +nns 1.000 1.000 1.000 1.05 | 1.000 1.000 1.05 16.0 0.00 +dls 1.000 1.000 1.000 1.08 | 1.000 1.000 1.08 16.0 0.00 +lzres_s42_g0 1.000 1.000 1.000 1.17 | 1.000 1.000 1.17 16.0 0.00 +lzres_s43_g0 1.000 1.000 1.000 1.17 | 1.000 1.000 1.17 16.0 0.00 +lzres_s44_g0 1.000 1.000 1.000 1.17 | 1.000 1.000 1.17 16.0 0.00 + +-- paired Wilcoxon on regret, model minus best retrieval (B: best=nns; noB: best=nns); negative = model better; [95% CI of paired median] +base base_s0 B vs nns: -0.17 [-0.24,-0.10] p=0.00267 certified 1.000 vs 1.000 +base base_s0 noB vs nns: -0.17 [-0.24,-0.10] p=0.00267 certified 1.000 vs 1.000 +base base_s1 B vs nns: -0.17 [-0.21,-0.07] p=0.00117 certified 1.000 vs 1.000 +base base_s1 noB vs nns: -0.17 [-0.21,-0.07] p=0.00117 certified 1.000 vs 1.000 +base base_s2 B vs nns: -0.12 [-0.19,-0.03] p=0.00108 certified 1.000 vs 1.000 +base base_s2 noB vs nns: -0.12 [-0.19,-0.03] p=0.00108 certified 1.000 vs 1.000 +A_lz lz_s42_g0 B vs nns: +0.08 [+0.05,+0.12] p=0.0104 certified 1.000 vs 1.000 +A_lz lz_s42_g0 noB vs nns: +0.08 [+0.05,+0.12] p=0.0104 certified 1.000 vs 1.000 +A_lz lz_s42_g1 B vs nns: +0.09 [+0.05,+0.14] p=0.0111 certified 1.000 vs 1.000 +A_lz lz_s42_g1 noB vs nns: +0.09 [+0.05,+0.14] p=0.0111 certified 1.000 vs 1.000 +A_lz lz_s42_g2 B vs nns: +0.10 [+0.05,+0.18] p=0.00859 certified 1.000 vs 1.000 +A_lz lz_s42_g2 noB vs nns: +0.10 [+0.05,+0.18] p=0.00859 certified 1.000 vs 1.000 +A_lz lz_s43_g0 B vs nns: +0.15 [+0.11,+0.25] p=0.000446 certified 1.000 vs 1.000 +A_lz lz_s43_g0 noB vs nns: +0.15 [+0.11,+0.25] p=0.000446 certified 1.000 vs 1.000 +A_lz lz_s44_g0 B vs nns: +0.05 [-0.01,+0.09] p=0.126 certified 1.000 vs 1.000 +A_lz lz_s44_g0 noB vs nns: +0.05 [-0.01,+0.09] p=0.126 certified 1.000 vs 1.000 +A_lzm lzm_s42_g0 B vs nns: +0.41 [+0.32,+0.45] p=2.21e-11 certified 1.000 vs 1.000 +A_lzm lzm_s42_g0 noB vs nns: +0.41 [+0.32,+0.45] p=2.21e-11 certified 1.000 vs 1.000 +A_lzm lzm_s43_g0 B vs nns: +0.38 [+0.35,+0.44] p=2.56e-11 certified 1.000 vs 1.000 +A_lzm lzm_s43_g0 noB vs nns: +0.38 [+0.35,+0.44] p=2.56e-11 certified 1.000 vs 1.000 +A_lzm lzm_s44_g0 B vs nns: +0.11 [+0.07,+0.21] p=5.83e-05 certified 1.000 vs 1.000 +A_lzm lzm_s44_g0 noB vs nns: +0.11 [+0.07,+0.21] p=5.83e-05 certified 1.000 vs 1.000 +A_lzres lzres_s42_g0 B vs nns: +0.10 [+0.06,+0.17] p=0.0247 certified 1.000 vs 1.000 +A_lzres lzres_s42_g0 noB vs nns: +0.10 [+0.06,+0.17] p=0.0247 certified 1.000 vs 1.000 +A_lzres lzres_s43_g0 B vs nns: +0.11 [+0.06,+0.19] p=0.0283 certified 1.000 vs 1.000 +A_lzres lzres_s43_g0 noB vs nns: +0.11 [+0.06,+0.19] p=0.0283 certified 1.000 vs 1.000 +A_lzres lzres_s44_g0 B vs nns: +0.11 [+0.07,+0.22] p=0.0204 certified 1.000 vs 1.000 +A_lzres lzres_s44_g0 noB vs nns: +0.11 [+0.07,+0.22] p=0.0204 certified 1.000 vs 1.000 +A_phys phys_s42_g0 B vs nns: +0.37 [+0.31,+0.43] p=2.1e-11 certified 1.000 vs 1.000 +A_phys phys_s42_g0 noB vs nns: +0.37 [+0.31,+0.43] p=2.1e-11 certified 1.000 vs 1.000 +A_phys phys_s43_g0 B vs nns: +0.36 [+0.27,+0.49] p=2.98e-11 certified 1.000 vs 1.000 +A_phys phys_s43_g0 noB vs nns: +0.36 [+0.27,+0.49] p=2.98e-11 certified 1.000 vs 1.000 +A_phys phys_s44_g0 B vs nns: +0.15 [+0.09,+0.21] p=2.66e-07 certified 1.000 vs 1.000 +A_phys phys_s44_g0 noB vs nns: +0.15 [+0.09,+0.21] p=2.66e-07 certified 1.000 vs 1.000 + +=== lvl1.5 K=8 oracle=old (median oracle J 0.656) === +arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired +base_s0 1.000 0.983 0.983 0.83 | 1.000 1.000 0.83 16.1 0.02 +base_s1 1.000 0.983 0.967 0.92 | 1.000 1.000 0.92 16.1 0.02 +base_s2 1.000 0.983 0.983 0.86 | 1.000 1.000 0.86 16.1 0.02 +lz_s42_g0 1.000 1.000 1.000 1.26 | 1.000 1.000 1.26 16.0 0.00 +lz_s42_g1 1.000 1.000 1.000 1.17 | 1.000 1.000 1.17 16.0 0.00 +lz_s42_g2 1.000 1.000 1.000 1.20 | 1.000 1.000 1.20 16.0 0.00 +lz_s43_g0 1.000 1.000 1.000 1.23 | 1.000 1.000 1.23 16.0 0.00 +lz_s44_g0 1.000 1.000 1.000 1.09 | 1.000 1.000 1.09 16.0 0.00 +lzm_s42_g0 1.000 1.000 1.000 1.26 | 1.000 1.000 1.26 16.0 0.00 +lzm_s43_g0 1.000 1.000 1.000 1.22 | 1.000 1.000 1.22 16.0 0.00 +lzm_s44_g0 1.000 1.000 1.000 1.03 | 1.000 1.000 1.03 16.0 0.00 +phys_s42_g0 1.000 1.000 1.000 1.28 | 1.000 1.000 1.28 16.0 0.00 +phys_s43_g0 1.000 1.000 1.000 1.29 | 1.000 1.000 1.29 16.0 0.00 +phys_s44_g0 1.000 1.000 1.000 1.09 | 1.000 1.000 1.09 16.0 0.00 +nn 1.000 1.000 1.000 1.08 | 1.000 1.000 1.08 16.0 0.00 +idw 1.000 1.000 1.000 1.16 | 1.000 1.000 1.16 16.0 0.00 +nns 1.000 1.000 1.000 0.98 | 1.000 1.000 0.98 16.0 0.00 +dls 1.000 1.000 1.000 1.05 | 1.000 1.000 1.05 16.0 0.00 +lzres_s42_g0 1.000 1.000 1.000 1.17 | 1.000 1.000 1.17 16.0 0.00 +lzres_s43_g0 1.000 1.000 1.000 1.16 | 1.000 1.000 1.16 16.0 0.00 +lzres_s44_g0 1.000 1.000 1.000 1.17 | 1.000 1.000 1.17 16.0 0.00 + +-- paired Wilcoxon on regret, model minus best retrieval (B: best=nns; noB: best=nns); negative = model better; [95% CI of paired median] +base base_s0 B vs nns: -0.03 [-0.18,+0.10] p=0.462 certified 1.000 vs 1.000 +base base_s0 noB vs nns: -0.03 [-0.18,+0.10] p=0.475 certified 0.983 vs 1.000 +base base_s1 B vs nns: +0.08 [-0.11,+0.37] p=0.069 certified 1.000 vs 1.000 +base base_s1 noB vs nns: +0.08 [-0.11,+0.37] p=0.076 certified 0.967 vs 1.000 +base base_s2 B vs nns: -0.03 [-0.14,+0.09] p=0.691 certified 1.000 vs 1.000 +base base_s2 noB vs nns: -0.03 [-0.14,+0.09] p=0.691 certified 0.983 vs 1.000 +A_lz lz_s42_g0 B vs nns: +0.27 [+0.13,+0.75] p=1.79e-08 certified 1.000 vs 1.000 +A_lz lz_s42_g0 noB vs nns: +0.27 [+0.13,+0.75] p=1.79e-08 certified 1.000 vs 1.000 +A_lz lz_s42_g1 B vs nns: +0.24 [+0.12,+0.69] p=1.32e-08 certified 1.000 vs 1.000 +A_lz lz_s42_g1 noB vs nns: +0.24 [+0.12,+0.69] p=1.32e-08 certified 1.000 vs 1.000 +A_lz lz_s42_g2 B vs nns: +0.22 [+0.11,+0.82] p=4.9e-08 certified 1.000 vs 1.000 +A_lz lz_s42_g2 noB vs nns: +0.22 [+0.11,+0.82] p=4.9e-08 certified 1.000 vs 1.000 +A_lz lz_s43_g0 B vs nns: +0.31 [+0.16,+0.64] p=1.02e-08 certified 1.000 vs 1.000 +A_lz lz_s43_g0 noB vs nns: +0.31 [+0.16,+0.64] p=1.02e-08 certified 1.000 vs 1.000 +A_lz lz_s44_g0 B vs nns: +0.21 [+0.12,+0.59] p=9.83e-08 certified 1.000 vs 1.000 +A_lz lz_s44_g0 noB vs nns: +0.21 [+0.12,+0.59] p=9.83e-08 certified 1.000 vs 1.000 +A_lzm lzm_s42_g0 B vs nns: +0.38 [+0.33,+0.51] p=2.56e-10 certified 1.000 vs 1.000 +A_lzm lzm_s42_g0 noB vs nns: +0.38 [+0.33,+0.51] p=2.56e-10 certified 1.000 vs 1.000 +A_lzm lzm_s43_g0 B vs nns: +0.39 [+0.33,+0.53] p=7.2e-10 certified 1.000 vs 1.000 +A_lzm lzm_s43_g0 noB vs nns: +0.39 [+0.33,+0.53] p=7.2e-10 certified 1.000 vs 1.000 +A_lzm lzm_s44_g0 B vs nns: +0.11 [+0.04,+0.24] p=1.5e-05 certified 1.000 vs 1.000 +A_lzm lzm_s44_g0 noB vs nns: +0.11 [+0.04,+0.24] p=1.5e-05 certified 1.000 vs 1.000 +A_lzres lzres_s42_g0 B vs nns: +0.16 [+0.08,+0.29] p=0.00527 certified 1.000 vs 1.000 +A_lzres lzres_s42_g0 noB vs nns: +0.16 [+0.08,+0.29] p=0.00527 certified 1.000 vs 1.000 +A_lzres lzres_s43_g0 B vs nns: +0.16 [+0.10,+0.32] p=0.00409 certified 1.000 vs 1.000 +A_lzres lzres_s43_g0 noB vs nns: +0.16 [+0.10,+0.32] p=0.00409 certified 1.000 vs 1.000 +A_lzres lzres_s44_g0 B vs nns: +0.14 [+0.07,+0.29] p=0.00936 certified 1.000 vs 1.000 +A_lzres lzres_s44_g0 noB vs nns: +0.14 [+0.07,+0.29] p=0.00936 certified 1.000 vs 1.000 +A_phys phys_s42_g0 B vs nns: +0.43 [+0.32,+0.64] p=1.92e-10 certified 1.000 vs 1.000 +A_phys phys_s42_g0 noB vs nns: +0.43 [+0.32,+0.64] p=1.92e-10 certified 1.000 vs 1.000 +A_phys phys_s43_g0 B vs nns: +0.43 [+0.30,+0.63] p=8.27e-10 certified 1.000 vs 1.000 +A_phys phys_s43_g0 noB vs nns: +0.43 [+0.30,+0.63] p=8.27e-10 certified 1.000 vs 1.000 +A_phys phys_s44_g0 B vs nns: +0.22 [+0.13,+0.45] p=3.51e-08 certified 1.000 vs 1.000 +A_phys phys_s44_g0 noB vs nns: +0.22 [+0.13,+0.45] p=3.51e-08 certified 1.000 vs 1.000 + +=== lvl1.5 K=8 oracle=strong2 (median oracle J 0.656) === +arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired +base_s0 1.000 0.983 0.983 0.93 | 1.000 1.000 0.97 16.1 0.02 +base_s1 1.000 0.983 0.967 1.01 | 1.000 1.000 1.10 16.1 0.02 +base_s2 1.000 0.983 0.983 0.98 | 1.000 1.000 1.01 16.1 0.02 +lz_s42_g0 1.000 1.000 1.000 1.33 | 1.000 1.000 1.33 16.0 0.00 +lz_s42_g1 1.000 1.000 1.000 1.32 | 1.000 1.000 1.32 16.0 0.00 +lz_s42_g2 1.000 1.000 1.000 1.33 | 1.000 1.000 1.33 16.0 0.00 +lz_s43_g0 1.000 1.000 1.000 1.40 | 1.000 1.000 1.40 16.0 0.00 +lz_s44_g0 1.000 1.000 1.000 1.27 | 1.000 1.000 1.27 16.0 0.00 +lzm_s42_g0 1.000 1.000 1.000 1.35 | 1.000 1.000 1.35 16.0 0.00 +lzm_s43_g0 1.000 1.000 1.000 1.31 | 1.000 1.000 1.31 16.0 0.00 +lzm_s44_g0 1.000 1.000 1.000 1.15 | 1.000 1.000 1.15 16.0 0.00 +phys_s42_g0 1.000 1.000 1.000 1.37 | 1.000 1.000 1.37 16.0 0.00 +phys_s43_g0 1.000 1.000 1.000 1.35 | 1.000 1.000 1.35 16.0 0.00 +phys_s44_g0 1.000 1.000 1.000 1.23 | 1.000 1.000 1.23 16.0 0.00 +nn 1.000 1.000 1.000 1.18 | 1.000 1.000 1.18 16.0 0.00 +idw 1.000 1.000 1.000 1.28 | 1.000 1.000 1.28 16.0 0.00 +nns 1.000 1.000 1.000 1.12 | 1.000 1.000 1.12 16.0 0.00 +dls 1.000 1.000 1.000 1.22 | 1.000 1.000 1.22 16.0 0.00 +lzres_s42_g0 1.000 1.000 1.000 1.37 | 1.000 1.000 1.37 16.0 0.00 +lzres_s43_g0 1.000 1.000 1.000 1.36 | 1.000 1.000 1.36 16.0 0.00 +lzres_s44_g0 1.000 1.000 1.000 1.39 | 1.000 1.000 1.39 16.0 0.00 + +-- paired Wilcoxon on regret, model minus best retrieval (B: best=nns; noB: best=nns); negative = model better; [95% CI of paired median] +base base_s0 B vs nns: -0.03 [-0.18,+0.10] p=0.475 certified 1.000 vs 1.000 +base base_s0 noB vs nns: -0.03 [-0.18,+0.10] p=0.484 certified 0.983 vs 1.000 +base base_s1 B vs nns: +0.08 [-0.11,+0.37] p=0.0701 certified 1.000 vs 1.000 +base base_s1 noB vs nns: +0.08 [-0.11,+0.37] p=0.0748 certified 0.967 vs 1.000 +base base_s2 B vs nns: -0.03 [-0.14,+0.09] p=0.686 certified 1.000 vs 1.000 +base base_s2 noB vs nns: -0.03 [-0.14,+0.09] p=0.686 certified 0.983 vs 1.000 +A_lz lz_s42_g0 B vs nns: +0.27 [+0.13,+0.80] p=1.86e-08 certified 1.000 vs 1.000 +A_lz lz_s42_g0 noB vs nns: +0.27 [+0.13,+0.80] p=1.86e-08 certified 1.000 vs 1.000 +A_lz lz_s42_g1 B vs nns: +0.25 [+0.12,+0.69] p=1.32e-08 certified 1.000 vs 1.000 +A_lz lz_s42_g1 noB vs nns: +0.25 [+0.12,+0.69] p=1.32e-08 certified 1.000 vs 1.000 +A_lz lz_s42_g2 B vs nns: +0.22 [+0.11,+0.83] p=4.9e-08 certified 1.000 vs 1.000 +A_lz lz_s42_g2 noB vs nns: +0.22 [+0.11,+0.83] p=4.9e-08 certified 1.000 vs 1.000 +A_lz lz_s43_g0 B vs nns: +0.31 [+0.16,+0.64] p=9.77e-09 certified 1.000 vs 1.000 +A_lz lz_s43_g0 noB vs nns: +0.31 [+0.16,+0.64] p=9.77e-09 certified 1.000 vs 1.000 +A_lz lz_s44_g0 B vs nns: +0.21 [+0.12,+0.59] p=9.83e-08 certified 1.000 vs 1.000 +A_lz lz_s44_g0 noB vs nns: +0.21 [+0.12,+0.59] p=9.83e-08 certified 1.000 vs 1.000 +A_lzm lzm_s42_g0 B vs nns: +0.38 [+0.33,+0.51] p=2.56e-10 certified 1.000 vs 1.000 +A_lzm lzm_s42_g0 noB vs nns: +0.38 [+0.33,+0.51] p=2.56e-10 certified 1.000 vs 1.000 +A_lzm lzm_s43_g0 B vs nns: +0.39 [+0.33,+0.53] p=7.2e-10 certified 1.000 vs 1.000 +A_lzm lzm_s43_g0 noB vs nns: +0.39 [+0.33,+0.53] p=7.2e-10 certified 1.000 vs 1.000 +A_lzm lzm_s44_g0 B vs nns: +0.11 [+0.04,+0.24] p=1.55e-05 certified 1.000 vs 1.000 +A_lzm lzm_s44_g0 noB vs nns: +0.11 [+0.04,+0.24] p=1.55e-05 certified 1.000 vs 1.000 +A_lzres lzres_s42_g0 B vs nns: +0.16 [+0.08,+0.29] p=0.00539 certified 1.000 vs 1.000 +A_lzres lzres_s42_g0 noB vs nns: +0.16 [+0.08,+0.29] p=0.00539 certified 1.000 vs 1.000 +A_lzres lzres_s43_g0 B vs nns: +0.16 [+0.10,+0.32] p=0.00409 certified 1.000 vs 1.000 +A_lzres lzres_s43_g0 noB vs nns: +0.16 [+0.10,+0.32] p=0.00409 certified 1.000 vs 1.000 +A_lzres lzres_s44_g0 B vs nns: +0.14 [+0.07,+0.29] p=0.00936 certified 1.000 vs 1.000 +A_lzres lzres_s44_g0 noB vs nns: +0.14 [+0.07,+0.29] p=0.00936 certified 1.000 vs 1.000 +A_phys phys_s42_g0 B vs nns: +0.44 [+0.32,+0.64] p=1.92e-10 certified 1.000 vs 1.000 +A_phys phys_s42_g0 noB vs nns: +0.44 [+0.32,+0.64] p=1.92e-10 certified 1.000 vs 1.000 +A_phys phys_s43_g0 B vs nns: +0.43 [+0.30,+0.63] p=8.27e-10 certified 1.000 vs 1.000 +A_phys phys_s43_g0 noB vs nns: +0.43 [+0.30,+0.63] p=8.27e-10 certified 1.000 vs 1.000 +A_phys phys_s44_g0 B vs nns: +0.22 [+0.13,+0.45] p=3.51e-08 certified 1.000 vs 1.000 +A_phys phys_s44_g0 noB vs nns: +0.22 [+0.13,+0.45] p=3.51e-08 certified 1.000 vs 1.000 + +=== lvl2.0 K=8 oracle=old (median oracle J 0.557) === +arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired +base_s0 1.000 0.933 0.833 0.73 | 1.000 1.000 0.75 16.5 0.07 +base_s1 1.000 0.950 0.817 0.66 | 1.000 1.000 0.70 16.4 0.05 +base_s2 1.000 0.950 0.833 0.65 | 1.000 1.000 0.72 16.4 0.05 +lz_s42_g0 1.000 1.000 1.000 0.98 | 1.000 1.000 0.98 16.0 0.00 +lz_s42_g1 1.000 1.000 1.000 0.97 | 1.000 1.000 0.97 16.0 0.00 +lz_s42_g2 1.000 1.000 1.000 1.04 | 1.000 1.000 1.04 16.0 0.00 +lz_s43_g0 1.000 1.000 1.000 1.13 | 1.000 1.000 1.13 16.0 0.00 +lz_s44_g0 1.000 1.000 1.000 0.90 | 1.000 1.000 0.90 16.0 0.00 +lzm_s42_g0 1.000 1.000 1.000 0.95 | 1.000 1.000 0.95 16.0 0.00 +lzm_s43_g0 1.000 1.000 1.000 0.97 | 1.000 1.000 0.97 16.0 0.00 +lzm_s44_g0 1.000 1.000 1.000 0.86 | 1.000 1.000 0.86 16.0 0.00 +phys_s42_g0 1.000 1.000 1.000 1.11 | 1.000 1.000 1.11 16.0 0.00 +phys_s43_g0 1.000 1.000 1.000 1.10 | 1.000 1.000 1.10 16.0 0.00 +phys_s44_g0 1.000 1.000 1.000 0.97 | 1.000 1.000 0.97 16.0 0.00 +nn 0.967 0.950 0.950 4.00 | 1.000 1.000 4.00 16.4 0.05 +idw 0.950 0.933 0.933 4.31 | 1.000 1.000 4.31 16.6 0.07 +nns 1.000 1.000 1.000 0.78 | 1.000 1.000 0.78 16.0 0.00 +dls 1.000 1.000 1.000 0.81 | 1.000 1.000 0.81 16.0 0.00 +lzres_s42_g0 1.000 1.000 1.000 0.94 | 1.000 1.000 0.94 16.0 0.00 +lzres_s43_g0 1.000 1.000 1.000 0.94 | 1.000 1.000 0.94 16.0 0.00 +lzres_s44_g0 1.000 1.000 1.000 0.92 | 1.000 1.000 0.92 16.0 0.00 + +-- paired Wilcoxon on regret, model minus best retrieval (B: best=nns; noB: best=nns); negative = model better; [95% CI of paired median] +base base_s0 B vs nns: -0.08 [-0.24,+0.21] p=0.768 certified 1.000 vs 1.000 +base base_s0 noB vs nns: -0.08 [-0.20,+0.29] p=0.606 certified 0.833 vs 1.000 +base base_s1 B vs nns: -0.10 [-0.26,+0.13] p=0.837 certified 1.000 vs 1.000 +base base_s1 noB vs nns: -0.10 [-0.31,+0.13] p=0.819 certified 0.817 vs 1.000 +base base_s2 B vs nns: -0.11 [-0.30,+0.22] p=0.729 certified 1.000 vs 1.000 +base base_s2 noB vs nns: -0.12 [-0.30,+0.07] p=0.947 certified 0.833 vs 1.000 +A_lz lz_s42_g0 B vs nns: +0.08 [+0.02,+0.33] p=0.0111 certified 1.000 vs 1.000 +A_lz lz_s42_g0 noB vs nns: +0.08 [+0.02,+0.33] p=0.0111 certified 1.000 vs 1.000 +A_lz lz_s42_g1 B vs nns: +0.07 [+0.02,+0.18] p=0.0174 certified 1.000 vs 1.000 +A_lz lz_s42_g1 noB vs nns: +0.07 [+0.02,+0.18] p=0.0174 certified 1.000 vs 1.000 +A_lz lz_s42_g2 B vs nns: +0.08 [+0.00,+0.28] p=0.0154 certified 1.000 vs 1.000 +A_lz lz_s42_g2 noB vs nns: +0.08 [+0.00,+0.28] p=0.0154 certified 1.000 vs 1.000 +A_lz lz_s43_g0 B vs nns: +0.21 [+0.07,+0.40] p=0.001 certified 1.000 vs 1.000 +A_lz lz_s43_g0 noB vs nns: +0.21 [+0.07,+0.40] p=0.001 certified 1.000 vs 1.000 +A_lz lz_s44_g0 B vs nns: +0.07 [-0.02,+0.35] p=0.0359 certified 1.000 vs 1.000 +A_lz lz_s44_g0 noB vs nns: +0.07 [-0.02,+0.35] p=0.0359 certified 1.000 vs 1.000 +A_lzm lzm_s42_g0 B vs nns: +0.09 [+0.02,+0.30] p=0.00254 certified 1.000 vs 1.000 +A_lzm lzm_s42_g0 noB vs nns: +0.09 [+0.02,+0.30] p=0.00254 certified 1.000 vs 1.000 +A_lzm lzm_s43_g0 B vs nns: +0.09 [+0.01,+0.21] p=0.0059 certified 1.000 vs 1.000 +A_lzm lzm_s43_g0 noB vs nns: +0.09 [+0.01,+0.21] p=0.0059 certified 1.000 vs 1.000 +A_lzm lzm_s44_g0 B vs nns: -0.01 [-0.06,+0.05] p=0.385 certified 1.000 vs 1.000 +A_lzm lzm_s44_g0 noB vs nns: -0.01 [-0.06,+0.05] p=0.385 certified 1.000 vs 1.000 +A_lzres lzres_s42_g0 B vs nns: +0.01 [-0.09,+0.09] p=0.381 certified 1.000 vs 1.000 +A_lzres lzres_s42_g0 noB vs nns: +0.01 [-0.09,+0.09] p=0.381 certified 1.000 vs 1.000 +A_lzres lzres_s43_g0 B vs nns: +0.01 [-0.09,+0.09] p=0.512 certified 1.000 vs 1.000 +A_lzres lzres_s43_g0 noB vs nns: +0.01 [-0.09,+0.09] p=0.512 certified 1.000 vs 1.000 +A_lzres lzres_s44_g0 B vs nns: +0.00 [-0.11,+0.09] p=0.32 certified 1.000 vs 1.000 +A_lzres lzres_s44_g0 noB vs nns: +0.00 [-0.11,+0.09] p=0.32 certified 1.000 vs 1.000 +A_phys phys_s42_g0 B vs nns: +0.21 [+0.03,+0.47] p=6.41e-05 certified 1.000 vs 1.000 +A_phys phys_s42_g0 noB vs nns: +0.21 [+0.03,+0.47] p=6.41e-05 certified 1.000 vs 1.000 +A_phys phys_s43_g0 B vs nns: +0.22 [+0.00,+0.40] p=0.000276 certified 1.000 vs 1.000 +A_phys phys_s43_g0 noB vs nns: +0.22 [+0.00,+0.40] p=0.000276 certified 1.000 vs 1.000 +A_phys phys_s44_g0 B vs nns: +0.07 [+0.00,+0.26] p=0.00539 certified 1.000 vs 1.000 +A_phys phys_s44_g0 noB vs nns: +0.07 [+0.00,+0.26] p=0.00539 certified 1.000 vs 1.000 + +=== lvl2.0 K=8 oracle=strong2 (median oracle J 0.555) === +arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired +base_s0 1.000 0.933 0.833 0.87 | 1.000 1.000 0.96 16.5 0.07 +base_s1 1.000 0.950 0.817 0.89 | 1.000 1.000 0.89 16.4 0.05 +base_s2 1.000 0.950 0.833 0.87 | 1.000 1.000 0.97 16.4 0.05 +lz_s42_g0 1.000 1.000 1.000 1.33 | 1.000 1.000 1.33 16.0 0.00 +lz_s42_g1 1.000 1.000 1.000 1.33 | 1.000 1.000 1.33 16.0 0.00 +lz_s42_g2 1.000 1.000 1.000 1.32 | 1.000 1.000 1.32 16.0 0.00 +lz_s43_g0 1.000 1.000 1.000 1.39 | 1.000 1.000 1.39 16.0 0.00 +lz_s44_g0 1.000 1.000 1.000 1.23 | 1.000 1.000 1.23 16.0 0.00 +lzm_s42_g0 1.000 1.000 1.000 1.26 | 1.000 1.000 1.26 16.0 0.00 +lzm_s43_g0 1.000 1.000 1.000 1.21 | 1.000 1.000 1.21 16.0 0.00 +lzm_s44_g0 1.000 1.000 1.000 1.06 | 1.000 1.000 1.06 16.0 0.00 +phys_s42_g0 1.000 1.000 1.000 1.35 | 1.000 1.000 1.35 16.0 0.00 +phys_s43_g0 1.000 1.000 1.000 1.38 | 1.000 1.000 1.38 16.0 0.00 +phys_s44_g0 1.000 1.000 1.000 1.28 | 1.000 1.000 1.28 16.0 0.00 +nn 0.967 0.950 0.950 4.00 | 1.000 1.000 4.00 16.4 0.05 +idw 0.950 0.933 0.933 4.40 | 1.000 1.000 4.40 16.6 0.07 +nns 1.000 1.000 1.000 1.14 | 1.000 1.000 1.14 16.0 0.00 +dls 1.000 1.000 1.000 1.26 | 1.000 1.000 1.26 16.0 0.00 +lzres_s42_g0 1.000 1.000 1.000 1.36 | 1.000 1.000 1.36 16.0 0.00 +lzres_s43_g0 1.000 1.000 1.000 1.36 | 1.000 1.000 1.36 16.0 0.00 +lzres_s44_g0 1.000 1.000 1.000 1.38 | 1.000 1.000 1.38 16.0 0.00 + +-- paired Wilcoxon on regret, model minus best retrieval (B: best=nns; noB: best=nns); negative = model better; [95% CI of paired median] +base base_s0 B vs nns: -0.08 [-0.24,+0.21] p=0.768 certified 1.000 vs 1.000 +base base_s0 noB vs nns: -0.08 [-0.20,+0.29] p=0.606 certified 0.833 vs 1.000 +base base_s1 B vs nns: -0.10 [-0.26,+0.13] p=0.831 certified 1.000 vs 1.000 +base base_s1 noB vs nns: -0.10 [-0.31,+0.13] p=0.814 certified 0.817 vs 1.000 +base base_s2 B vs nns: -0.12 [-0.30,+0.22] p=0.729 certified 1.000 vs 1.000 +base base_s2 noB vs nns: -0.12 [-0.30,+0.07] p=0.941 certified 0.833 vs 1.000 +A_lz lz_s42_g0 B vs nns: +0.08 [+0.02,+0.33] p=0.0113 certified 1.000 vs 1.000 +A_lz lz_s42_g0 noB vs nns: +0.08 [+0.02,+0.33] p=0.0113 certified 1.000 vs 1.000 +A_lz lz_s42_g1 B vs nns: +0.07 [+0.02,+0.18] p=0.0178 certified 1.000 vs 1.000 +A_lz lz_s42_g1 noB vs nns: +0.07 [+0.02,+0.18] p=0.0178 certified 1.000 vs 1.000 +A_lz lz_s42_g2 B vs nns: +0.08 [+0.00,+0.28] p=0.0151 certified 1.000 vs 1.000 +A_lz lz_s42_g2 noB vs nns: +0.08 [+0.00,+0.28] p=0.0151 certified 1.000 vs 1.000 +A_lz lz_s43_g0 B vs nns: +0.22 [+0.07,+0.40] p=0.00103 certified 1.000 vs 1.000 +A_lz lz_s43_g0 noB vs nns: +0.22 [+0.07,+0.40] p=0.00103 certified 1.000 vs 1.000 +A_lz lz_s44_g0 B vs nns: +0.07 [-0.02,+0.35] p=0.0346 certified 1.000 vs 1.000 +A_lz lz_s44_g0 noB vs nns: +0.07 [-0.02,+0.35] p=0.0346 certified 1.000 vs 1.000 +A_lzm lzm_s42_g0 B vs nns: +0.09 [+0.02,+0.30] p=0.00254 certified 1.000 vs 1.000 +A_lzm lzm_s42_g0 noB vs nns: +0.09 [+0.02,+0.30] p=0.00254 certified 1.000 vs 1.000 +A_lzm lzm_s43_g0 B vs nns: +0.09 [+0.01,+0.21] p=0.0059 certified 1.000 vs 1.000 +A_lzm lzm_s43_g0 noB vs nns: +0.09 [+0.01,+0.21] p=0.0059 certified 1.000 vs 1.000 +A_lzm lzm_s44_g0 B vs nns: -0.01 [-0.06,+0.05] p=0.385 certified 1.000 vs 1.000 +A_lzm lzm_s44_g0 noB vs nns: -0.01 [-0.06,+0.05] p=0.385 certified 1.000 vs 1.000 +A_lzres lzres_s42_g0 B vs nns: +0.01 [-0.09,+0.09] p=0.377 certified 1.000 vs 1.000 +A_lzres lzres_s42_g0 noB vs nns: +0.01 [-0.09,+0.09] p=0.377 certified 1.000 vs 1.000 +A_lzres lzres_s43_g0 B vs nns: +0.01 [-0.09,+0.09] p=0.512 certified 1.000 vs 1.000 +A_lzres lzres_s43_g0 noB vs nns: +0.01 [-0.09,+0.09] p=0.512 certified 1.000 vs 1.000 +A_lzres lzres_s44_g0 B vs nns: +0.00 [-0.11,+0.09] p=0.317 certified 1.000 vs 1.000 +A_lzres lzres_s44_g0 noB vs nns: +0.00 [-0.11,+0.09] p=0.317 certified 1.000 vs 1.000 +A_phys phys_s42_g0 B vs nns: +0.21 [+0.03,+0.48] p=6.41e-05 certified 1.000 vs 1.000 +A_phys phys_s42_g0 noB vs nns: +0.21 [+0.03,+0.48] p=6.41e-05 certified 1.000 vs 1.000 +A_phys phys_s43_g0 B vs nns: +0.22 [+0.00,+0.40] p=0.000284 certified 1.000 vs 1.000 +A_phys phys_s43_g0 noB vs nns: +0.22 [+0.00,+0.40] p=0.000284 certified 1.000 vs 1.000 +A_phys phys_s44_g0 B vs nns: +0.07 [+0.00,+0.26] p=0.00527 certified 1.000 vs 1.000 +A_phys phys_s44_g0 noB vs nns: +0.07 [+0.00,+0.26] p=0.00527 certified 1.000 vs 1.000 + +=== pilotfar K=8 oracle=old (median oracle J 65.827) === +arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired +base_s0 1.000 0.033 0.017 -6.49 | 1.000 1.000 -8.50 23.7 0.97 +base_s1 1.000 0.000 0.000 -6.74 | 1.000 1.000 -8.82 24.0 1.00 +base_s2 1.000 0.017 0.000 -6.68 | 1.000 1.000 -8.84 23.9 0.98 +lz_s42_g0 1.000 1.000 1.000 4.97 | 1.000 1.000 4.97 16.0 0.00 +lz_s42_g1 1.000 1.000 1.000 4.77 | 1.000 1.000 4.77 16.0 0.00 +lz_s42_g2 1.000 1.000 1.000 4.82 | 1.000 1.000 4.82 16.0 0.00 +lz_s43_g0 1.000 1.000 1.000 5.97 | 1.000 1.000 5.97 16.0 0.00 +lz_s44_g0 1.000 1.000 1.000 5.17 | 1.000 1.000 5.17 16.0 0.00 +lzm_s42_g0 1.000 1.000 1.000 9.28 | 1.000 1.000 9.28 16.0 0.00 +lzm_s43_g0 1.000 1.000 1.000 10.32 | 1.000 1.000 10.32 16.0 0.00 +lzm_s44_g0 1.000 1.000 1.000 7.79 | 1.000 1.000 7.79 16.0 0.00 +phys_s42_g0 1.000 1.000 1.000 6.76 | 1.000 1.000 6.76 16.0 0.00 +phys_s43_g0 1.000 1.000 1.000 5.36 | 1.000 1.000 5.36 16.0 0.00 +phys_s44_g0 1.000 1.000 1.000 5.39 | 1.000 1.000 5.39 16.0 0.00 +nn 0.717 0.517 0.517 69.64 | 1.000 1.000 55.28 19.9 0.48 +idw 0.383 0.233 0.233 1000.00 | 1.000 1.000 58.52 22.1 0.77 +nns 1.000 1.000 1.000 0.04 | 1.000 1.000 0.04 16.0 0.00 +dls 1.000 1.000 1.000 -1.73 | 1.000 1.000 -1.73 16.0 0.00 +lzres_s42_g0 1.000 1.000 1.000 -4.00 | 1.000 1.000 -4.00 16.0 0.00 +lzres_s43_g0 1.000 1.000 1.000 -3.89 | 1.000 1.000 -3.89 16.0 0.00 +lzres_s44_g0 1.000 1.000 1.000 -4.61 | 1.000 1.000 -4.61 16.0 0.00 + +-- paired Wilcoxon on regret, model minus best retrieval (B: best=dls; noB: best=dls); negative = model better; [95% CI of paired median] +base base_s0 B vs dls: -6.44 [-6.84,-6.01] p=1.63e-11 certified 1.000 vs 1.000 +base base_s0 noB vs dls: -5.28 [-6.10,-4.50] p=2.95e-06 certified 0.017 vs 1.000 +base base_s1 B vs dls: -6.37 [-6.85,-6.07] p=1.63e-11 certified 1.000 vs 1.000 +base base_s1 noB vs dls: -5.14 [-5.94,-3.69] p=2.95e-06 certified 0.000 vs 1.000 +base base_s2 B vs dls: -6.53 [-6.92,-6.25] p=1.63e-11 certified 1.000 vs 1.000 +base base_s2 noB vs dls: -5.25 [-5.94,-3.91] p=1.38e-08 certified 0.000 vs 1.000 +A_lz lz_s42_g0 B vs dls: +7.02 [+6.04,+8.04] p=1.63e-11 certified 1.000 vs 1.000 +A_lz lz_s42_g0 noB vs dls: +7.02 [+6.04,+8.04] p=1.63e-11 certified 1.000 vs 1.000 +A_lz lz_s42_g1 B vs dls: +6.55 [+5.94,+7.39] p=2.01e-10 certified 1.000 vs 1.000 +A_lz lz_s42_g1 noB vs dls: +6.55 [+5.94,+7.39] p=2.01e-10 certified 1.000 vs 1.000 +A_lz lz_s42_g2 B vs dls: +6.98 [+5.97,+8.41] p=1.63e-11 certified 1.000 vs 1.000 +A_lz lz_s42_g2 noB vs dls: +6.98 [+5.97,+8.41] p=1.63e-11 certified 1.000 vs 1.000 +A_lz lz_s43_g0 B vs dls: +7.87 [+6.93,+8.48] p=1.63e-11 certified 1.000 vs 1.000 +A_lz lz_s43_g0 noB vs dls: +7.87 [+6.93,+8.48] p=1.63e-11 certified 1.000 vs 1.000 +A_lz lz_s44_g0 B vs dls: +6.99 [+5.70,+7.82] p=1.63e-11 certified 1.000 vs 1.000 +A_lz lz_s44_g0 noB vs dls: +6.99 [+5.70,+7.82] p=1.63e-11 certified 1.000 vs 1.000 +A_lzm lzm_s42_g0 B vs dls: +11.32 [+10.43,+12.32] p=1.63e-11 certified 1.000 vs 1.000 +A_lzm lzm_s42_g0 noB vs dls: +11.32 [+10.43,+12.32] p=1.63e-11 certified 1.000 vs 1.000 +A_lzm lzm_s43_g0 B vs dls: +12.82 [+11.50,+13.35] p=1.63e-11 certified 1.000 vs 1.000 +A_lzm lzm_s43_g0 noB vs dls: +12.82 [+11.50,+13.35] p=1.63e-11 certified 1.000 vs 1.000 +A_lzm lzm_s44_g0 B vs dls: +10.05 [+8.95,+10.80] p=1.63e-11 certified 1.000 vs 1.000 +A_lzm lzm_s44_g0 noB vs dls: +10.05 [+8.95,+10.80] p=1.63e-11 certified 1.000 vs 1.000 +A_lzres lzres_s42_g0 B vs dls: -1.58 [-2.07,-1.26] p=1.99e-11 certified 1.000 vs 1.000 +A_lzres lzres_s42_g0 noB vs dls: -1.58 [-2.07,-1.26] p=1.99e-11 certified 1.000 vs 1.000 +A_lzres lzres_s43_g0 B vs dls: -1.64 [-1.95,-1.16] p=1.71e-11 certified 1.000 vs 1.000 +A_lzres lzres_s43_g0 noB vs dls: -1.64 [-1.95,-1.16] p=1.71e-11 certified 1.000 vs 1.000 +A_lzres lzres_s44_g0 B vs dls: -2.33 [-2.94,-1.61] p=1.63e-11 certified 1.000 vs 1.000 +A_lzres lzres_s44_g0 noB vs dls: -2.33 [-2.94,-1.61] p=1.63e-11 certified 1.000 vs 1.000 +A_phys phys_s42_g0 B vs dls: +8.78 [+8.12,+9.94] p=1.63e-11 certified 1.000 vs 1.000 +A_phys phys_s42_g0 noB vs dls: +8.78 [+8.12,+9.94] p=1.63e-11 certified 1.000 vs 1.000 +A_phys phys_s43_g0 B vs dls: +7.69 [+6.84,+8.44] p=1.63e-11 certified 1.000 vs 1.000 +A_phys phys_s43_g0 noB vs dls: +7.69 [+6.84,+8.44] p=1.63e-11 certified 1.000 vs 1.000 +A_phys phys_s44_g0 B vs dls: +7.51 [+6.74,+8.55] p=1.63e-11 certified 1.000 vs 1.000 +A_phys phys_s44_g0 noB vs dls: +7.51 [+6.74,+8.55] p=1.63e-11 certified 1.000 vs 1.000 + +=== pilotfar K=8 oracle=strong2 (median oracle J 59.001) === +arm cov anyCert pickCert(noB) regret_noB% | B: cert stab regret% calls/plant repaired +base_s0 1.000 0.033 0.017 2.90 | 1.000 1.000 1.03 23.7 0.97 +base_s1 1.000 0.000 0.000 2.88 | 1.000 1.000 0.80 24.0 1.00 +base_s2 1.000 0.017 0.000 2.89 | 1.000 1.000 0.86 23.9 0.98 +lz_s42_g0 1.000 1.000 1.000 16.67 | 1.000 1.000 16.67 16.0 0.00 +lz_s42_g1 1.000 1.000 1.000 16.33 | 1.000 1.000 16.33 16.0 0.00 +lz_s42_g2 1.000 1.000 1.000 16.60 | 1.000 1.000 16.60 16.0 0.00 +lz_s43_g0 1.000 1.000 1.000 17.50 | 1.000 1.000 17.50 16.0 0.00 +lz_s44_g0 1.000 1.000 1.000 16.66 | 1.000 1.000 16.66 16.0 0.00 +lzm_s42_g0 1.000 1.000 1.000 21.71 | 1.000 1.000 21.71 16.0 0.00 +lzm_s43_g0 1.000 1.000 1.000 22.71 | 1.000 1.000 22.71 16.0 0.00 +lzm_s44_g0 1.000 1.000 1.000 19.87 | 1.000 1.000 19.87 16.0 0.00 +phys_s42_g0 1.000 1.000 1.000 18.81 | 1.000 1.000 18.81 16.0 0.00 +phys_s43_g0 1.000 1.000 1.000 17.31 | 1.000 1.000 17.31 16.0 0.00 +phys_s44_g0 1.000 1.000 1.000 17.47 | 1.000 1.000 17.47 16.0 0.00 +nn 0.717 0.517 0.517 86.30 | 1.000 1.000 71.01 19.9 0.48 +idw 0.383 0.233 0.233 1000.00 | 1.000 1.000 76.42 22.1 0.77 +nns 1.000 1.000 1.000 11.35 | 1.000 1.000 11.35 16.0 0.00 +dls 1.000 1.000 1.000 7.95 | 1.000 1.000 7.95 16.0 0.00 +lzres_s42_g0 1.000 1.000 1.000 6.45 | 1.000 1.000 6.45 16.0 0.00 +lzres_s43_g0 1.000 1.000 1.000 6.55 | 1.000 1.000 6.55 16.0 0.00 +lzres_s44_g0 1.000 1.000 1.000 5.76 | 1.000 1.000 5.76 16.0 0.00 + +-- paired Wilcoxon on regret, model minus best retrieval (B: best=dls; noB: best=dls); negative = model better; [95% CI of paired median] +base base_s0 B vs dls: -7.02 [-7.55,-6.70] p=1.63e-11 certified 1.000 vs 1.000 +base base_s0 noB vs dls: -5.73 [-6.77,-4.94] p=2.95e-06 certified 0.017 vs 1.000 +base base_s1 B vs dls: -7.08 [-7.56,-6.66] p=1.63e-11 certified 1.000 vs 1.000 +base base_s1 noB vs dls: -5.62 [-6.48,-4.07] p=2.95e-06 certified 0.000 vs 1.000 +base base_s2 B vs dls: -7.26 [-7.65,-6.94] p=1.63e-11 certified 1.000 vs 1.000 +base base_s2 noB vs dls: -5.85 [-6.45,-4.35] p=1.32e-08 certified 0.000 vs 1.000 +A_lz lz_s42_g0 B vs dls: +7.78 [+6.68,+8.89] p=1.63e-11 certified 1.000 vs 1.000 +A_lz lz_s42_g0 noB vs dls: +7.78 [+6.68,+8.89] p=1.63e-11 certified 1.000 vs 1.000 +A_lz lz_s42_g1 B vs dls: +7.15 [+6.54,+8.15] p=2.01e-10 certified 1.000 vs 1.000 +A_lz lz_s42_g1 noB vs dls: +7.15 [+6.54,+8.15] p=2.01e-10 certified 1.000 vs 1.000 +A_lz lz_s42_g2 B vs dls: +7.76 [+6.52,+9.10] p=1.63e-11 certified 1.000 vs 1.000 +A_lz lz_s42_g2 noB vs dls: +7.76 [+6.52,+9.10] p=1.63e-11 certified 1.000 vs 1.000 +A_lz lz_s43_g0 B vs dls: +8.80 [+7.65,+9.50] p=1.63e-11 certified 1.000 vs 1.000 +A_lz lz_s43_g0 noB vs dls: +8.80 [+7.65,+9.50] p=1.63e-11 certified 1.000 vs 1.000 +A_lz lz_s44_g0 B vs dls: +7.70 [+6.37,+8.69] p=1.63e-11 certified 1.000 vs 1.000 +A_lz lz_s44_g0 noB vs dls: +7.70 [+6.37,+8.69] p=1.63e-11 certified 1.000 vs 1.000 +A_lzm lzm_s42_g0 B vs dls: +12.52 [+11.49,+13.81] p=1.63e-11 certified 1.000 vs 1.000 +A_lzm lzm_s42_g0 noB vs dls: +12.52 [+11.49,+13.81] p=1.63e-11 certified 1.000 vs 1.000 +A_lzm lzm_s43_g0 B vs dls: +14.22 [+12.55,+14.89] p=1.63e-11 certified 1.000 vs 1.000 +A_lzm lzm_s43_g0 noB vs dls: +14.22 [+12.55,+14.89] p=1.63e-11 certified 1.000 vs 1.000 +A_lzm lzm_s44_g0 B vs dls: +11.04 [+9.86,+12.05] p=1.63e-11 certified 1.000 vs 1.000 +A_lzm lzm_s44_g0 noB vs dls: +11.04 [+9.86,+12.05] p=1.63e-11 certified 1.000 vs 1.000 +A_lzres lzres_s42_g0 B vs dls: -1.77 [-2.27,-1.40] p=1.9e-11 certified 1.000 vs 1.000 +A_lzres lzres_s42_g0 noB vs dls: -1.77 [-2.27,-1.40] p=1.9e-11 certified 1.000 vs 1.000 +A_lzres lzres_s43_g0 B vs dls: -1.81 [-2.16,-1.27] p=1.71e-11 certified 1.000 vs 1.000 +A_lzres lzres_s43_g0 noB vs dls: -1.81 [-2.16,-1.27] p=1.71e-11 certified 1.000 vs 1.000 +A_lzres lzres_s44_g0 B vs dls: -2.58 [-3.23,-1.80] p=1.63e-11 certified 1.000 vs 1.000 +A_lzres lzres_s44_g0 noB vs dls: -2.58 [-3.23,-1.80] p=1.63e-11 certified 1.000 vs 1.000 +A_phys phys_s42_g0 B vs dls: +9.73 [+8.91,+10.99] p=1.63e-11 certified 1.000 vs 1.000 +A_phys phys_s42_g0 noB vs dls: +9.73 [+8.91,+10.99] p=1.63e-11 certified 1.000 vs 1.000 +A_phys phys_s43_g0 B vs dls: +8.56 [+7.48,+9.38] p=1.63e-11 certified 1.000 vs 1.000 +A_phys phys_s43_g0 noB vs dls: +8.56 [+7.48,+9.38] p=1.63e-11 certified 1.000 vs 1.000 +A_phys phys_s44_g0 B vs dls: +8.31 [+7.46,+9.46] p=1.63e-11 certified 1.000 vs 1.000 +A_phys phys_s44_g0 noB vs dls: +8.31 [+7.46,+9.46] p=1.63e-11 certified 1.000 vs 1.000 + diff --git a/results/physics_gen/pendulum_physics_gen_v2.txt b/results/physics_gen/pendulum_physics_gen_v2.txt new file mode 100644 index 0000000000000000000000000000000000000000..67836b394baf42fba344939ab2bbc2e24c2962f9 --- /dev/null +++ b/results/physics_gen/pendulum_physics_gen_v2.txt @@ -0,0 +1,188 @@ +=== lvl0 K=8, oracle median J 1.351; value = mean over the group's runs of per-plant regret %, then median over plants === +group n cov certified(no B) | with B: certified regret_med calls | no B regret_med +shipped 3 1.000 1.000 | 1.000 1.12 16.0 | 1.12 +dimless-lz, orig labels 3 1.000 1.000 | 1.000 1.16 16.0 | 1.16 +dimless-lzres, orig labels 3 1.000 1.000 | 1.000 1.29 16.0 | 1.29 +tuned phys, multi-A (published) 3 1.000 1.000 | 1.000 0.34 16.0 | 0.34 +tuned phys, multi-B (published) 3 1.000 0.894 | 1.000 0.20 16.0 | 0.19 +A: phys, multi-A (new metric) 3 1.000 0.994 | 1.000 0.33 16.0 | 0.33 +A: dimless-lz, multi-A 3 1.000 1.000 | 1.000 0.33 16.0 | 0.33 +A: dimless-lzres, multi-A 3 1.000 1.000 | 1.000 0.34 16.0 | 0.34 +A: dimless-lz, multi-B 3 1.000 0.783 | 1.000 0.23 16.0 | 0.21 +nn (orig bank) 1 1.000 1.000 | 1.000 0.48 16.0 | 0.48 +nns (orig) 1 1.000 1.000 | 1.000 1.12 16.0 | 1.12 +dls (orig) 1 1.000 1.000 | 1.000 1.23 16.0 | 1.23 +nn (multi-A) 1 1.000 0.983 | 1.000 0.30 16.0 | 0.30 +nns (multi-A) 1 1.000 1.000 | 1.000 0.37 16.0 | 0.37 +dls (multi-A) 1 1.000 1.000 | 1.000 0.30 16.0 | 0.30 +nn (multi-B) 1 1.000 1.000 | 1.000 0.21 16.0 | 0.21 +nns (multi-B) 1 1.000 1.000 | 1.000 0.11 16.0 | 0.11 +dls (multi-B) 1 1.000 1.000 | 1.000 0.14 16.0 | 0.14 +-- best retrieval arm with B: nns (multi-B); best original-bank arm: nn (orig bank). paired Wilcoxon, group minus retrieval (neg = model better) [95% CI of paired median], p +shipped B vs nns (multi-B) : +0.90 [+0.73,+1.14] p=5.41e-11 +shipped B vs nn (orig bank): +0.66 [+0.58,+0.88] p=1.44e-10 +dimless-lz, orig labels B vs nns (multi-B) : +0.89 [+0.71,+1.06] p=1.71e-11 +dimless-lz, orig labels B vs nn (orig bank): +0.69 [+0.56,+0.84] p=1.9e-11 +dimless-lzres, orig labels B vs nns (multi-B) : +1.10 [+1.02,+1.21] p=1.71e-11 +dimless-lzres, orig labels B vs nn (orig bank): +0.93 [+0.77,+0.97] p=1.9e-11 +tuned phys, multi-A (published) B vs nns (multi-B) : +0.10 [+0.05,+0.17] p=2.37e-06 +tuned phys, multi-A (published) B vs nn (orig bank): -0.06 [-0.15,+0.01] p=0.0126 +tuned phys, multi-B (published) B vs nns (multi-B) : +0.05 [+0.02,+0.07] p=6.21e-05 +tuned phys, multi-B (published) B vs nn (orig bank): -0.09 [-0.14,-0.01] p=0.000276 +A: phys, multi-A (new metric) B vs nns (multi-B) : +0.10 [+0.05,+0.15] p=4.09e-07 +A: phys, multi-A (new metric) B vs nn (orig bank): -0.03 [-0.13,+0.03] p=0.0238 +A: dimless-lz, multi-A B vs nns (multi-B) : +0.14 [+0.05,+0.18] p=1.42e-06 +A: dimless-lz, multi-A B vs nn (orig bank): +0.02 [-0.14,+0.05] p=0.122 +A: dimless-lzres, multi-A B vs nns (multi-B) : +0.11 [+0.06,+0.22] p=2.97e-09 +A: dimless-lzres, multi-A B vs nn (orig bank): +0.02 [-0.08,+0.07] p=0.233 +A: dimless-lz, multi-B B vs nns (multi-B) : +0.07 [+0.03,+0.11] p=0.000207 +A: dimless-lz, multi-B B vs nn (orig bank): -0.02 [-0.17,+0.02] p=0.0069 +-- factor decomposition (paired, with B): data = tuned-phys-multi-A vs shipped; recipe = A-phys(new metric) vs published tuned; physics = A-lz vs A-phys; trend = A-lzres vs A-lz; multiplicity = multi-B lz vs multi-A lz +tuned phys, multi-A (published) - shipped : -0.83 [-1.12,-0.57] p=8.42e-11 +A: phys, multi-A (new metric) - tuned phys, multi-A (published) : +0.00 [-0.01,+0.02] p=0.401 +A: dimless-lz, multi-A - A: phys, multi-A (new metric) : +0.00 [-0.03,+0.04] p=0.977 +A: dimless-lzres, multi-A - A: dimless-lz, multi-A : +0.04 [+0.01,+0.06] p=0.354 +A: dimless-lz, multi-B - A: dimless-lz, multi-A : -0.04 [-0.07,-0.02] p=0.000133 +dimless-lz, orig labels - shipped : +0.08 [-0.07,+0.18] p=0.462 + +=== lvl1.5 K=8, oracle median J 0.656; value = mean over the group's runs of per-plant regret %, then median over plants === +group n cov certified(no B) | with B: certified regret_med calls | no B regret_med +shipped 3 1.000 0.978 | 1.000 1.00 16.1 | 0.95 +dimless-lz, orig labels 3 1.000 1.000 | 1.000 1.33 16.0 | 1.33 +dimless-lzres, orig labels 3 1.000 1.000 | 1.000 1.38 16.0 | 1.38 +tuned phys, multi-A (published) 3 1.000 0.783 | 1.000 1.06 16.8 | 0.90 +tuned phys, multi-B (published) 3 1.000 0.689 | 1.000 0.84 16.6 | 0.77 +A: phys, multi-A (new metric) 3 1.000 0.806 | 1.000 1.12 16.7 | 0.66 +A: dimless-lz, multi-A 3 1.000 1.000 | 1.000 1.26 16.0 | 1.26 +A: dimless-lzres, multi-A 3 1.000 0.989 | 1.000 0.97 16.0 | 0.97 +A: dimless-lz, multi-B 3 1.000 0.622 | 1.000 0.82 16.0 | 0.70 +nn (orig bank) 1 1.000 1.000 | 1.000 1.18 16.0 | 1.18 +nns (orig) 1 1.000 1.000 | 1.000 1.15 16.0 | 1.15 +dls (orig) 1 1.000 1.000 | 1.000 1.22 16.0 | 1.22 +nn (multi-A) 1 1.000 1.000 | 1.000 1.21 16.0 | 1.21 +nns (multi-A) 1 1.000 1.000 | 1.000 0.92 16.0 | 0.92 +dls (multi-A) 1 1.000 0.983 | 1.000 0.99 16.0 | 0.99 +nn (multi-B) 1 1.000 0.933 | 1.000 1.25 16.0 | 1.22 +nns (multi-B) 1 1.000 1.000 | 1.000 0.59 16.0 | 0.59 +dls (multi-B) 1 1.000 0.983 | 1.000 0.56 16.0 | 0.56 +-- best retrieval arm with B: dls (multi-B); best original-bank arm: nns (orig). paired Wilcoxon, group minus retrieval (neg = model better) [95% CI of paired median], p +shipped B vs dls (multi-B) : +0.10 [-0.04,+0.88] p=0.0113 +shipped B vs nns (orig) : +0.05 [-0.09,+0.21] p=0.227 +dimless-lz, orig labels B vs dls (multi-B) : +0.58 [+0.23,+1.69] p=3.75e-05 +dimless-lz, orig labels B vs nns (orig) : +0.24 [+0.15,+0.70] p=1.79e-08 +dimless-lzres, orig labels B vs dls (multi-B) : +0.86 [+0.43,+1.35] p=1.07e-05 +dimless-lzres, orig labels B vs nns (orig) : +0.16 [+0.08,+0.31] p=0.0047 +tuned phys, multi-A (published) B vs dls (multi-B) : +0.14 [+0.05,+0.81] p=0.000974 +tuned phys, multi-A (published) B vs nns (orig) : -0.20 [-0.37,-0.01] p=0.149 +tuned phys, multi-B (published) B vs dls (multi-B) : -0.01 [-0.11,+0.17] p=0.335 +tuned phys, multi-B (published) B vs nns (orig) : -0.22 [-0.49,-0.07] p=0.000602 +A: phys, multi-A (new metric) B vs dls (multi-B) : +0.10 [+0.02,+0.58] p=0.004 +A: phys, multi-A (new metric) B vs nns (orig) : -0.24 [-0.48,+0.03] p=0.0701 +A: dimless-lz, multi-A B vs dls (multi-B) : +0.07 [-0.02,+0.45] p=0.00267 +A: dimless-lz, multi-A B vs nns (orig) : -0.33 [-0.71,+0.07] p=0.00242 +A: dimless-lzres, multi-A B vs dls (multi-B) : +0.01 [-0.01,+0.18] p=0.0773 +A: dimless-lzres, multi-A B vs nns (orig) : -0.05 [-0.60,+0.10] p=0.0059 +A: dimless-lz, multi-B B vs dls (multi-B) : +0.03 [-0.05,+0.25] p=0.00998 +A: dimless-lz, multi-B B vs nns (orig) : -0.33 [-0.80,+0.01] p=0.00254 +-- factor decomposition (paired, with B): data = tuned-phys-multi-A vs shipped; recipe = A-phys(new metric) vs published tuned; physics = A-lz vs A-phys; trend = A-lzres vs A-lz; multiplicity = multi-B lz vs multi-A lz +tuned phys, multi-A (published) - shipped : -0.25 [-0.78,-0.10] p=0.00492 +A: phys, multi-A (new metric) - tuned phys, multi-A (published) : -0.03 [-0.12,+0.03] p=0.466 +A: dimless-lz, multi-A - A: phys, multi-A (new metric) : -0.01 [-0.11,+0.33] p=0.774 +A: dimless-lzres, multi-A - A: dimless-lz, multi-A : -0.08 [-0.39,+0.00] p=0.000809 +A: dimless-lz, multi-B - A: dimless-lz, multi-A : -0.04 [-0.08,+0.02] p=0.0393 +dimless-lz, orig labels - shipped : +0.36 [+0.18,+0.58] p=7.88e-06 + +=== lvl2.0 K=8, oracle median J 0.554; value = mean over the group's runs of per-plant regret %, then median over plants === +group n cov certified(no B) | with B: certified regret_med calls | no B regret_med +shipped 3 1.000 0.828 | 1.000 1.01 16.4 | 0.94 +dimless-lz, orig labels 3 1.000 1.000 | 1.000 1.37 16.0 | 1.37 +dimless-lzres, orig labels 3 1.000 1.000 | 1.000 1.36 16.0 | 1.36 +tuned phys, multi-A (published) 3 1.000 0.644 | 1.000 1.56 17.2 | 1.05 +tuned phys, multi-B (published) 3 1.000 0.478 | 1.000 1.52 17.1 | 0.83 +A: phys, multi-A (new metric) 3 1.000 0.650 | 1.000 1.92 17.4 | 1.20 +A: dimless-lz, multi-A 3 1.000 1.000 | 1.000 1.25 16.0 | 1.25 +A: dimless-lzres, multi-A 3 1.000 1.000 | 1.000 1.12 16.0 | 1.12 +A: dimless-lz, multi-B 3 1.000 0.494 | 1.000 1.14 16.0 | 0.77 +nn (orig bank) 1 0.967 0.950 | 1.000 4.00 16.4 | 4.00 +nns (orig) 1 1.000 1.000 | 1.000 1.17 16.0 | 1.17 +dls (orig) 1 1.000 1.000 | 1.000 1.31 16.0 | 1.31 +nn (multi-A) 1 0.950 0.950 | 1.000 2.23 16.4 | 2.23 +nns (multi-A) 1 1.000 1.000 | 1.000 1.29 16.0 | 1.29 +dls (multi-A) 1 1.000 0.967 | 1.000 1.26 16.0 | 1.26 +nn (multi-B) 1 0.950 0.883 | 1.000 4.04 16.5 | 4.04 +nns (multi-B) 1 1.000 0.983 | 1.000 1.21 16.0 | 1.21 +dls (multi-B) 1 1.000 1.000 | 1.000 1.04 16.0 | 1.04 +-- best retrieval arm with B: dls (multi-B); best original-bank arm: nns (orig). paired Wilcoxon, group minus retrieval (neg = model better) [95% CI of paired median], p +shipped B vs dls (multi-B) : +0.01 [-0.24,+0.37] p=0.236 +shipped B vs nns (orig) : -0.02 [-0.25,+0.24] p=0.527 +dimless-lz, orig labels B vs dls (multi-B) : -0.19 [-0.43,+0.36] p=0.365 +dimless-lz, orig labels B vs nns (orig) : +0.11 [+0.02,+0.37] p=0.0059 +dimless-lzres, orig labels B vs dls (multi-B) : -0.19 [-0.42,+0.16] p=0.522 +dimless-lzres, orig labels B vs nns (orig) : +0.01 [-0.09,+0.09] p=0.393 +tuned phys, multi-A (published) B vs dls (multi-B) : +0.20 [-0.05,+0.55] p=0.0145 +tuned phys, multi-A (published) B vs nns (orig) : +0.34 [-0.02,+0.78] p=0.0192 +tuned phys, multi-B (published) B vs dls (multi-B) : +0.00 [-0.32,+0.36] p=0.139 +tuned phys, multi-B (published) B vs nns (orig) : +0.15 [-0.09,+0.47] p=0.16 +A: phys, multi-A (new metric) B vs dls (multi-B) : +0.21 [-0.06,+0.72] p=0.0047 +A: phys, multi-A (new metric) B vs nns (orig) : +0.49 [+0.01,+0.97] p=0.0084 +A: dimless-lz, multi-A B vs dls (multi-B) : +0.07 [-0.11,+0.17] p=0.245 +A: dimless-lz, multi-A B vs nns (orig) : +0.11 [-0.10,+0.22] p=0.702 +A: dimless-lzres, multi-A B vs dls (multi-B) : -0.01 [-0.18,+0.02] p=0.0773 +A: dimless-lzres, multi-A B vs nns (orig) : +0.03 [-0.09,+0.18] p=0.566 +A: dimless-lz, multi-B B vs dls (multi-B) : +0.05 [-0.01,+0.27] p=0.0283 +A: dimless-lz, multi-B B vs nns (orig) : +0.19 [+0.07,+0.38] p=0.551 +-- factor decomposition (paired, with B): data = tuned-phys-multi-A vs shipped; recipe = A-phys(new metric) vs published tuned; physics = A-lz vs A-phys; trend = A-lzres vs A-lz; multiplicity = multi-B lz vs multi-A lz +tuned phys, multi-A (published) - shipped : +0.05 [-0.33,+0.38] p=0.611 +A: phys, multi-A (new metric) - tuned phys, multi-A (published) : +0.01 [-0.06,+0.28] p=0.306 +A: dimless-lz, multi-A - A: phys, multi-A (new metric) : -0.13 [-0.96,+0.07] p=0.0167 +A: dimless-lzres, multi-A - A: dimless-lz, multi-A : -0.06 [-0.32,+0.04] p=0.0164 +A: dimless-lz, multi-B - A: dimless-lz, multi-A : +0.04 [-0.02,+0.16] p=0.101 +dimless-lz, orig labels - shipped : +0.18 [+0.04,+0.49] p=0.448 + +=== pilotfar K=8, oracle median J 58.949; value = mean over the group's runs of per-plant regret %, then median over plants === +group n cov certified(no B) | with B: certified regret_med calls | no B regret_med +shipped 3 1.000 0.006 | 1.000 1.36 23.9 | 3.83 +dimless-lz, orig labels 3 1.000 1.000 | 1.000 17.39 16.0 | 17.39 +dimless-lzres, orig labels 3 1.000 1.000 | 1.000 6.42 16.0 | 6.42 +tuned phys, multi-A (published) 3 0.994 0.078 | 1.000 1.28 22.3 | 5.56 +tuned phys, multi-B (published) 3 0.967 0.056 | 1.000 1.68 23.2 | 21.58 +A: phys, multi-A (new metric) 3 0.994 0.033 | 1.000 1.35 22.7 | 2.64 +A: dimless-lz, multi-A 3 1.000 1.000 | 1.000 3.28 16.0 | 3.28 +A: dimless-lzres, multi-A 3 1.000 0.983 | 1.000 0.76 16.0 | 0.74 +A: dimless-lz, multi-B 3 1.000 0.372 | 1.000 0.84 16.0 | 0.42 +nn (orig bank) 1 0.717 0.517 | 1.000 71.99 19.9 | 86.49 +nns (orig) 1 1.000 1.000 | 1.000 11.67 16.0 | 11.67 +dls (orig) 1 1.000 1.000 | 1.000 8.37 16.0 | 8.37 +nn (multi-A) 1 0.233 0.150 | 1.000 77.29 22.8 | 1000.00 +nns (multi-A) 1 1.000 1.000 | 1.000 8.65 16.0 | 8.65 +dls (multi-A) 1 1.000 1.000 | 1.000 1.84 16.0 | 1.84 +nn (multi-B) 1 0.233 0.150 | 1.000 11.75 22.8 | 1000.00 +nns (multi-B) 1 1.000 0.067 | 1.000 1.71 16.0 | 0.36 +dls (multi-B) 1 1.000 1.000 | 1.000 0.55 16.0 | 0.55 +-- best retrieval arm with B: dls (multi-B); best original-bank arm: dls (orig). paired Wilcoxon, group minus retrieval (neg = model better) [95% CI of paired median], p +shipped B vs dls (multi-B) : +0.81 [+0.52,+0.99] p=4.43e-09 +shipped B vs dls (orig) : -7.08 [-7.37,-6.67] p=1.63e-11 +dimless-lz, orig labels B vs dls (multi-B) : +16.88 [+15.86,+18.34] p=1.63e-11 +dimless-lz, orig labels B vs dls (orig) : +8.19 [+7.11,+8.95] p=1.63e-11 +dimless-lzres, orig labels B vs dls (multi-B) : +5.87 [+5.56,+6.08] p=1.63e-11 +dimless-lzres, orig labels B vs dls (orig) : -2.04 [-2.60,-1.47] p=1.71e-11 +tuned phys, multi-A (published) B vs dls (multi-B) : +0.76 [+0.47,+1.36] p=2.85e-08 +tuned phys, multi-A (published) B vs dls (orig) : -7.33 [-7.76,-6.14] p=2.62e-08 +tuned phys, multi-B (published) B vs dls (multi-B) : +1.13 [+0.51,+1.77] p=2.71e-09 +tuned phys, multi-B (published) B vs dls (orig) : -6.67 [-7.28,-6.03] p=9.07e-08 +A: phys, multi-A (new metric) B vs dls (multi-B) : +0.78 [+0.32,+1.47] p=6.02e-08 +A: phys, multi-A (new metric) B vs dls (orig) : -6.96 [-7.90,-6.41] p=2.59e-09 +A: dimless-lz, multi-A B vs dls (multi-B) : +2.66 [+2.05,+3.10] p=1.63e-11 +A: dimless-lz, multi-A B vs dls (orig) : -5.55 [-6.18,-5.21] p=1.63e-11 +A: dimless-lzres, multi-A B vs dls (multi-B) : +0.11 [+0.02,+0.26] p=0.000636 +A: dimless-lzres, multi-A B vs dls (orig) : -7.47 [-8.10,-7.04] p=1.63e-11 +A: dimless-lz, multi-B B vs dls (multi-B) : +0.33 [+0.18,+0.52] p=1.06e-06 +A: dimless-lz, multi-B B vs dls (orig) : -7.44 [-7.74,-6.97] p=1.71e-11 +-- factor decomposition (paired, with B): data = tuned-phys-multi-A vs shipped; recipe = A-phys(new metric) vs published tuned; physics = A-lz vs A-phys; trend = A-lzres vs A-lz; multiplicity = multi-B lz vs multi-A lz +tuned phys, multi-A (published) - shipped : -0.03 [-0.34,+0.29] p=0.466 +A: phys, multi-A (new metric) - tuned phys, multi-A (published) : +0.18 [-0.33,+0.55] p=0.918 +A: dimless-lz, multi-A - A: phys, multi-A (new metric) : +1.88 [+0.98,+2.26] p=0.00204 +A: dimless-lzres, multi-A - A: dimless-lz, multi-A : -2.14 [-2.69,-1.82] p=2.1e-11 +A: dimless-lz, multi-B - A: dimless-lz, multi-A : -2.06 [-2.70,-1.13] p=1.07e-08 +dimless-lz, orig labels - shipped : +15.83 [+15.19,+17.57] p=1.63e-11 + diff --git a/results/physics_gen/run_lzres.sh b/results/physics_gen/run_lzres.sh new file mode 100644 index 0000000000000000000000000000000000000000..efb0d01653f722bf6054d90f2f06836cf613bf50 --- /dev/null +++ b/results/physics_gen/run_lzres.sh @@ -0,0 +1,7 @@ +#!/bin/bash +cd /home/cercola/matte_server/DiffusionControl +export OMP_NUM_THREADS=1 PYTHONPATH=. CUDA_VISIBLE_DEVICES="" +for t in pilotfar lvl2.0 lvl1.5 lvl0 lvl1.25 lvl1.1; do + while [ ! -f results/physics_gen/cands_$t.npz ]; do sleep 60; done + for st in gen ver; do myenv/bin/python -u -m nonlinear.experiments.physics_gen_eval --stage $st --tags $t --workers 3 --only lzres; done +done diff --git a/results/physics_gen/run_orc3.sh b/results/physics_gen/run_orc3.sh new file mode 100644 index 0000000000000000000000000000000000000000..ea5970d4546e26409a04f0624b608aa575b0bc5f --- /dev/null +++ b/results/physics_gen/run_orc3.sh @@ -0,0 +1,4 @@ +#!/bin/bash +cd /home/cercola/matte_server/DiffusionControl +export OMP_NUM_THREADS=1 PYTHONPATH=. CUDA_VISIBLE_DEVICES="" +myenv/bin/python -u -m nonlinear.experiments.physics_gen_eval2 --stage orc3 --tags $1 --workers 3 diff --git a/results/physics_gen/run_pend.sh b/results/physics_gen/run_pend.sh new file mode 100644 index 0000000000000000000000000000000000000000..129bba82ed18d64178a0e3459f4e74af4dbf5251 --- /dev/null +++ b/results/physics_gen/run_pend.sh @@ -0,0 +1,8 @@ +#!/bin/bash +cd /home/cercola/matte_server/DiffusionControl +export OMP_NUM_THREADS=1 PYTHONPATH=. CUDA_VISIBLE_DEVICES="" +for t in pilotfar lvl2.0 lvl1.5 lvl0 lvl1.25 lvl1.1; do + myenv/bin/python -u -m nonlinear.experiments.physics_gen_eval --stage gen --tags $t + myenv/bin/python -u -m nonlinear.experiments.physics_gen_eval --stage ver --tags $t --workers 8 + myenv/bin/python -u -m nonlinear.experiments.physics_gen_eval --stage orc2 --tags $t --workers 8 +done diff --git a/results/physics_gen/run_pend2.sh b/results/physics_gen/run_pend2.sh new file mode 100644 index 0000000000000000000000000000000000000000..aa344e8d4ac94f7d6233f6dfb2922f911cf0f85a --- /dev/null +++ b/results/physics_gen/run_pend2.sh @@ -0,0 +1,6 @@ +#!/bin/bash +cd /home/cercola/matte_server/DiffusionControl +export OMP_NUM_THREADS=1 PYTHONPATH=. CUDA_VISIBLE_DEVICES="" +for t in lvl1.1 lvl1.25 lvl0; do + for st in gen ver orc2; do myenv/bin/python -u -m nonlinear.experiments.physics_gen_eval --stage $st --tags $t --workers 4; done +done diff --git a/results/physics_gen/run_pend3.sh b/results/physics_gen/run_pend3.sh new file mode 100644 index 0000000000000000000000000000000000000000..739f413483574098fc56010b7c766dc8e3b52e21 --- /dev/null +++ b/results/physics_gen/run_pend3.sh @@ -0,0 +1,6 @@ +#!/bin/bash +cd /home/cercola/matte_server/DiffusionControl +export OMP_NUM_THREADS=1 PYTHONPATH=. CUDA_VISIBLE_DEVICES="" +for t in lvl0; do + for st in gen ver orc2; do myenv/bin/python -u -m nonlinear.experiments.physics_gen_eval --stage $st --tags $t --workers 4; done +done diff --git a/results/physics_gen/run_v2.sh b/results/physics_gen/run_v2.sh new file mode 100644 index 0000000000000000000000000000000000000000..1b39a5c00174760c11f7cc1fe03a4693eee5450b --- /dev/null +++ b/results/physics_gen/run_v2.sh @@ -0,0 +1,6 @@ +#!/bin/bash +cd /home/cercola/matte_server/DiffusionControl +export OMP_NUM_THREADS=1 PYTHONPATH=. CUDA_VISIBLE_DEVICES="" +t=$1 +myenv/bin/python -u -m nonlinear.experiments.physics_gen_eval2 --stage gen --tags $t --workers 3 +myenv/bin/python -u -m nonlinear.experiments.physics_gen_eval2 --stage ver --tags $t --workers 3 diff --git a/results/physics_gen/siso_physics_gen.txt b/results/physics_gen/siso_physics_gen.txt new file mode 100644 index 0000000000000000000000000000000000000000..896c35aa0a5237b490648020df156daefb92c88e --- /dev/null +++ b/results/physics_gen/siso_physics_gen.txt @@ -0,0 +1,75 @@ +=== SISO lvl0 N=120: best retrieval (K=8 coverage) = idw_8 === +arm K=1 K=8 per-cand | McNemar K=8 vs idw_8 (model-only/retr-only p) | K=1 vs nn +diff_s0 0.550 0.667 0.518 | 4-2 p=0.688 | 27-6 p=0.000324 +diff_s1 0.483 0.692 0.512 | 5-0 p=0.0625 | 25-12 p=0.047 +diff_s2 0.517 0.667 0.532 | 4-2 p=0.688 | 26-9 p=0.00599 +clip_s0 0.550 0.667 0.518 | 4-2 p=0.688 | 27-6 p=0.000324 +clip_s1 0.483 0.692 0.512 | 5-0 p=0.0625 | 25-12 p=0.047 +clip_s2 0.517 0.667 0.532 | 4-2 p=0.688 | 26-9 p=0.00599 +c1_s0 0.067 0.217 0.054 | 0-52 p=4.44e-16 | 3-40 p=3.02e-09 +c1_s1 0.025 0.217 0.052 | 1-53 p=6.11e-15 | 2-44 p=3.08e-11 +c1_s2 0.058 0.192 0.057 | 1-56 p=8.05e-16 | 2-40 p=4.11e-10 +c2_s0 0.200 0.442 0.152 | 0-25 p=5.96e-08 | 7-28 p=0.000508 +c2_s1 0.142 0.450 0.148 | 2-26 p=3.03e-06 | 7-35 p=1.51e-05 +c2_s2 0.150 0.492 0.161 | 0-19 p=3.81e-06 | 7-34 p=2.53e-05 +c3_s0 0.483 0.650 0.498 | 2-2 p=1 | 22-9 p=0.0294 +c3_s1 0.483 0.658 0.518 | 3-2 p=1 | 23-10 p=0.0351 +c3_s2 0.517 0.650 0.506 | 2-2 p=1 | 25-8 p=0.00455 +nn 0.375 0.642 0.401 | 0-1 p=1 | 0-0 p=1 +idw_8 0.558 0.650 0.424 | 0-0 p=1 | 0-0 p=1 +-- paired c3 minus deployed, K=1 (sampler seed i vs i): 4-12 p=0.0768; 6-6 p=1; 4-4 p=1 +-- paired c2 minus deployed, K=1 (sampler seed i vs i): 1-43 p=5.12e-12; 2-43 p=5.89e-11; 4-48 p=1.31e-10 +-- paired c3 minus deployed, K=8 (sampler seed i vs i): 1-3 p=0.625; 2-6 p=0.289; 1-3 p=0.625 +-- paired c2 minus deployed, K=8 (sampler seed i vs i): 0-27 p=1.49e-08; 0-29 p=3.73e-09; 1-22 p=5.72e-06 +-- 3-sampler-seed means K=8: diff=0.675 clip=0.675 c1=0.208 c2=0.461 c3=0.653 nn=0.642 + +=== SISO lvl1.5 N=120: best retrieval (K=8 coverage) = nn === +arm K=1 K=8 per-cand | McNemar K=8 vs nn (model-only/retr-only p) | K=1 vs nn +diff_s0 0.533 0.700 0.518 | 3-3 p=1 | 10-16 p=0.327 +diff_s1 0.508 0.683 0.508 | 3-5 p=0.727 | 7-16 p=0.0931 +diff_s2 0.517 0.692 0.506 | 2-3 p=1 | 8-16 p=0.152 +clip_s0 0.525 0.692 0.523 | 3-4 p=1 | 10-17 p=0.248 +clip_s1 0.525 0.692 0.527 | 3-4 p=1 | 7-14 p=0.189 +clip_s2 0.525 0.683 0.531 | 1-3 p=0.625 | 8-15 p=0.21 +c1_s0 0.208 0.450 0.182 | 5-35 p=1.38e-06 | 4-49 p=7.05e-11 +c1_s1 0.125 0.467 0.173 | 7-35 p=1.51e-05 | 3-58 p=3.29e-14 +c1_s2 0.142 0.458 0.188 | 6-35 p=4.87e-06 | 3-56 p=1.19e-13 +c2_s0 0.308 0.633 0.302 | 2-10 p=0.0386 | 3-36 p=3.61e-08 +c2_s1 0.342 0.625 0.302 | 1-10 p=0.0117 | 5-34 p=2.43e-06 +c2_s2 0.267 0.633 0.289 | 1-9 p=0.0215 | 3-41 p=1.62e-09 +c3_s0 0.608 0.700 0.621 | 1-1 p=1 | 12-9 p=0.664 +c3_s1 0.608 0.708 0.627 | 2-1 p=1 | 12-9 p=0.664 +c3_s2 0.608 0.725 0.637 | 3-0 p=0.25 | 10-7 p=0.629 +nn 0.583 0.700 0.549 | 0-0 p=1 | 0-0 p=1 +idw_8 0.667 0.700 0.559 | 0-0 p=1 | 0-0 p=1 +-- paired c3 minus deployed, K=1 (sampler seed i vs i): 17-8 p=0.108; 16-4 p=0.0118; 13-2 p=0.00739 +-- paired c2 minus deployed, K=1 (sampler seed i vs i): 7-34 p=2.53e-05; 7-27 p=0.000821; 8-38 p=9.25e-06 +-- paired c3 minus deployed, K=8 (sampler seed i vs i): 3-3 p=1; 6-3 p=0.508; 5-1 p=0.219 +-- paired c2 minus deployed, K=8 (sampler seed i vs i): 3-11 p=0.0574; 5-12 p=0.143; 4-11 p=0.118 +-- 3-sampler-seed means K=8: diff=0.692 clip=0.689 c1=0.458 c2=0.631 c3=0.711 nn=0.700 + +=== SISO lvl2.0 N=120: best retrieval (K=8 coverage) = nn === +arm K=1 K=8 per-cand | McNemar K=8 vs nn (model-only/retr-only p) | K=1 vs nn +diff_s0 0.408 0.667 0.438 | 1-23 p=2.98e-06 | 3-49 p=1.04e-11 +diff_s1 0.433 0.642 0.453 | 2-27 p=1.62e-06 | 4-47 p=2.42e-10 +diff_s2 0.450 0.692 0.448 | 1-20 p=2.1e-05 | 4-45 p=8.23e-10 +clip_s0 0.500 0.725 0.502 | 2-17 p=0.000729 | 3-38 p=1.05e-08 +clip_s1 0.542 0.725 0.524 | 3-18 p=0.00149 | 4-34 p=6.04e-07 +clip_s2 0.508 0.775 0.507 | 4-13 p=0.049 | 4-38 p=5.65e-08 +c1_s0 0.300 0.567 0.254 | 2-36 p=5.4e-09 | 2-61 p=4.37e-16 +c1_s1 0.225 0.600 0.273 | 3-33 p=2.27e-07 | 1-69 p=1.2e-19 +c1_s2 0.250 0.608 0.263 | 2-31 p=1.31e-07 | 2-67 p=8.19e-18 +c2_s0 0.383 0.775 0.433 | 0-9 p=0.00391 | 1-50 p=4.62e-14 +c2_s1 0.450 0.783 0.447 | 1-9 p=0.0215 | 1-42 p=1e-11 +c2_s2 0.442 0.817 0.445 | 1-5 p=0.219 | 1-43 p=5.12e-12 +c3_s0 0.750 0.858 0.758 | 1-0 p=1 | 5-10 p=0.302 +c3_s1 0.775 0.850 0.777 | 0-0 p=1 | 7-9 p=0.804 +c3_s2 0.767 0.858 0.757 | 1-0 p=1 | 6-9 p=0.607 +nn 0.792 0.850 0.717 | 0-0 p=1 | 0-0 p=1 +idw_8 0.850 0.850 0.724 | 0-0 p=1 | 0-0 p=1 +-- paired c3 minus deployed, K=1 (sampler seed i vs i): 46-5 p=2.33e-09; 43-2 p=5.89e-11; 40-2 p=4.11e-10 +-- paired c2 minus deployed, K=1 (sampler seed i vs i): 21-24 p=0.766; 26-24 p=0.888; 26-27 p=1 +-- paired c3 minus deployed, K=8 (sampler seed i vs i): 23-0 p=2.38e-07; 27-2 p=1.62e-06; 21-1 p=1.1e-05 +-- paired c2 minus deployed, K=8 (sampler seed i vs i): 22-9 p=0.0294; 26-9 p=0.00599; 21-6 p=0.00592 +-- 3-sampler-seed means K=8: diff=0.667 clip=0.742 c1=0.592 c2=0.792 c3=0.856 nn=0.850 + diff --git a/results/physics_gen/train_all.sh b/results/physics_gen/train_all.sh new file mode 100644 index 0000000000000000000000000000000000000000..ed65581bade9f6482248d186a154c6ff16ccd659 --- /dev/null +++ b/results/physics_gen/train_all.sh @@ -0,0 +1,8 @@ +#!/bin/bash +# 3 variants x 3 seeds, CPU, same recipe as size_ref (hidden 256, 5 layers, 800 epochs, batch 256, lr 2e-4) +cd /home/cercola/matte_server/DiffusionControl +export OMP_NUM_THREADS=1 WANDB_MODE=disabled PYTHONPATH=. CUDA_VISIBLE_DEVICES="" +for v in lzres; do for s in 42 43 44; do + sfx=""; [ $s != 42 ] && sfx="_s$s" + PEND_DATA=data/pendulum_dataset_dimless_$v.pt setsid nohup myenv/bin/python -u -m nonlinear.training.train_pendulum_ablation --tag dl_$v$sfx --seed $s --epochs 800 > results/physics_gen/logs/train_dl_${v}_$s.log 2>&1 < /dev/null & +done; done diff --git a/results/physics_gen/train_pg.sh b/results/physics_gen/train_pg.sh new file mode 100644 index 0000000000000000000000000000000000000000..611ef463ccaec4ad25039f5945fd1f3e62fff0b5 --- /dev/null +++ b/results/physics_gen/train_pg.sh @@ -0,0 +1,8 @@ +#!/bin/bash +cd /home/cercola/matte_server/DiffusionControl +export OMP_NUM_THREADS=1 PYTHONPATH=. CUDA_VISIBLE_DEVICES="" +for s in 1 2 3; do + for cfg in "phys A" "lz A" "lzres A" "lz B"; do set -- $cfg + setsid nohup myenv/bin/python -u -m nonlinear.pendulum_multi.train_multi_physgen --tag $2_$1_s$s --mode $1 --data $2 --seed $s > results/physics_gen/logs/trainpg_$2_$1_s$s.log 2>&1 < /dev/null & + done +done diff --git a/results/siso_coherent/costcond_siso.json b/results/siso_coherent/costcond_siso.json new file mode 100644 index 0000000000000000000000000000000000000000..151a0a37841b3df50a0d1a48ec612c17ca007573 --- /dev/null +++ b/results/siso_coherent/costcond_siso.json @@ -0,0 +1,33 @@ +{ + "0.1": { + "single_shot": 0.43282869792070633, + "best_of_K": 0.20829135170726656, + "rmse_median": 0.2400697487936606, + "K": 16 + }, + "0.3": { + "single_shot": 0.430463254232667, + "best_of_K": 0.18308652767418868, + "rmse_median": 0.16842255105935477, + "K": 16 + }, + "1.0": { + "single_shot": 1.7508217678023121, + "best_of_K": 0.25532378348100276, + "rmse_median": 0.1745144916129091, + "K": 16 + }, + "3.0": { + "single_shot": 3.117271487499664, + "best_of_K": 1.2944283702345893, + "rmse_median": 0.15652084001818967, + "K": 16 + }, + "10.0": { + "single_shot": 5.08596041211184, + "best_of_K": 1.620151557600361, + "rmse_median": 0.15025961185796094, + "K": 16 + }, + "heldout_rho1_singleshot_median": 1.480855930630882 +} \ No newline at end of file diff --git a/results/siso_coherent/dataset_stability.txt b/results/siso_coherent/dataset_stability.txt new file mode 100644 index 0000000000000000000000000000000000000000..0a9b3a88a8dba0752fd77db6bdd1a4b6b6c632c7 --- /dev/null +++ b/results/siso_coherent/dataset_stability.txt @@ -0,0 +1,9 @@ + +=== RAW SISO DATASET closed-loop stability (N=3000 sampled, crit final_energy<0.3, dt=0.03, dur=12.0, 4 ICs) === +stored controllers that ACTUALLY stabilize their own plant: 1550/3000 = 0.517 + (unrealizable loops skipped: 0) + beta in [ 0,1 ): stable 0.541 (n=2581) + beta in [ 1,5 ): stable 0.468 (n=284) + beta in [ 5,20 ): stable 0.220 (n=91) + beta in [ 20,1000000000.0): stable 0.000 (n=44) + median stored beta = 0.11, max = 191.0 diff --git a/results/siso_coherent/distill.log b/results/siso_coherent/distill.log new file mode 100644 index 0000000000000000000000000000000000000000..5367907d19be876e22fe4d91a7df7a219f8e1a18 --- /dev/null +++ b/results/siso_coherent/distill.log @@ -0,0 +1,9 @@ + +=== STABILIZE-REPAIR (Best_nonlinear_cond_popov_L_distill.pth, N=120, K=6) === +amortized prior plants-stabilizable: 96/120 = 0.800 +failing plants repaired: 24 + rescued keeping the model's own F: 10 + rescued by F=1 fallback: 8 + still hard: 6 +AFTER repair plants-stabilizable: 114/120 = 0.950 (+0.150) +cost: prior 9 ms/controller; repair 346.1s for 24 plants (14.42s/plant, gradient-free) diff --git a/results/siso_coherent/k_sweep.txt b/results/siso_coherent/k_sweep.txt new file mode 100644 index 0000000000000000000000000000000000000000..18b0ad516829f0e6f3202369a953c149dda03930 --- /dev/null +++ b/results/siso_coherent/k_sweep.txt @@ -0,0 +1,15 @@ +=== K-sweep: does more sampling beat repair? (SISO N=24, seed 0, gen 59.1 ms/ctrl) === + K plants>=1 stable gen time/plant (per-sample stab = 0.664) + 1 0.583 0.06 s + 2 0.792 0.12 s + 4 0.792 0.24 s + 8 0.875 0.47 s + 16 0.917 0.95 s + 32 1.000 1.89 s + 64 1.000 3.78 s + +reading: more sampling DOES help and reaches 1.0 at K=32 (1.89 s/plant of generation) -- but the + climb is SLOWER than i.i.d. 1-(1-p)^K (a heavy-tailed subset of plants needs many draws), it stays + UNCERTIFIED (rollout only), and it costs ~4700x more than the 0.4 ms/plant eigenvalue repair. + Trade-off: big-K keeps a GENERATIVE (nonlinear) controller; the eigenvalue repair is cheap+certified + but substitutes a classical linear gain. (N=24 sample; on larger N the tail may need K>64.) diff --git a/results/siso_coherent/prior_repair.log b/results/siso_coherent/prior_repair.log new file mode 100644 index 0000000000000000000000000000000000000000..98c8e1d336af1eb294d0010179c607c07f9b9c2d --- /dev/null +++ b/results/siso_coherent/prior_repair.log @@ -0,0 +1,9 @@ + +=== STABILIZE-REPAIR (Best_nonlinear_cond_popov_L2.pth, N=120, K=6) === +amortized prior plants-stabilizable: 97/120 = 0.808 +failing plants repaired: 23 + rescued keeping the model's own F: 9 + rescued by F=1 fallback: 8 + still hard: 6 +AFTER repair plants-stabilizable: 114/120 = 0.950 (+0.142) +cost: prior 9 ms/controller; repair 335.5s for 23 plants (14.59s/plant, gradient-free) diff --git a/results/siso_coherent/regret.log b/results/siso_coherent/regret.log new file mode 100644 index 0000000000000000000000000000000000000000..f31ca92948ce4d5d67e5faecc4c6c779288b7dc8 --- /dev/null +++ b/results/siso_coherent/regret.log @@ -0,0 +1,105 @@ + plant 0: stab=1.00 best_J=6.537 oracle_J=6.290 regret=+3.9% + plant 1: stab=0.83 best_J=2.574 oracle_J=2.417 regret=+6.5% + plant 2: stab=0.83 best_J=1.879 oracle_J=1.403 regret=+34.0% + plant 3: stab=0.67 best_J=2.215 oracle_J=1.770 regret=+25.2% + plant 4: stab=0.67 best_J=0.509 oracle_J=0.485 regret=+4.9% + plant 5: stab=1.00 best_J=0.687 oracle_J=0.619 regret=+11.0% + plant 6: stab=1.00 best_J=0.809 oracle_J=0.595 regret=+36.0% + plant 7: stab=0.17 best_J=283.717 oracle_J=273.466 regret=+3.7% + plant 8: stab=1.00 best_J=2.094 oracle_J=1.771 regret=+18.2% + plant 9: stab=1.00 best_J=1.071 oracle_J=0.786 regret=+36.2% + plant 10: stab=1.00 best_J=9.382 oracle_J=8.694 regret=+7.9% + plant 11: stab=0.83 best_J=2.526 oracle_J=2.387 regret=+5.8% + plant 12: stab=1.00 best_J=0.351 oracle_J=0.351 regret=+0.0% + plant 13: stab=0.67 best_J=1.483 oracle_J=1.424 regret=+4.1% + plant 14: stab=0.17 best_J=105.524 oracle_J=103.223 regret=+2.2% + plant 15: stab=0.83 best_J=1.038 oracle_J=1.003 regret=+3.5% + plant 17: stab=1.00 best_J=1.531 oracle_J=1.417 regret=+8.0% + plant 18: stab=0.83 best_J=8.341 oracle_J=7.222 regret=+15.5% + plant 19: stab=0.67 best_J=0.278 oracle_J=0.277 regret=+0.4% + plant 20: stab=0.83 best_J=3.226 oracle_J=3.164 regret=+2.0% + plant 22: stab=0.83 best_J=0.757 oracle_J=0.619 regret=+22.3% + plant 23: stab=1.00 best_J=1.384 oracle_J=1.181 regret=+17.2% + plant 24: stab=0.83 best_J=4.260 oracle_J=3.764 regret=+13.2% + plant 25: stab=0.83 best_J=0.990 oracle_J=0.986 regret=+0.4% + plant 26: stab=0.83 best_J=5.189 oracle_J=4.738 regret=+9.5% + plant 27: stab=0.83 best_J=2.985 oracle_J=2.595 regret=+15.0% + plant 28: stab=1.00 best_J=1.052 oracle_J=0.998 regret=+5.4% + plant 29: stab=0.67 best_J=3.359 oracle_J=3.297 regret=+1.9% + plant 31: stab=0.50 best_J=1.910 oracle_J=1.892 regret=+1.0% + plant 32: stab=1.00 best_J=0.502 oracle_J=0.352 regret=+42.6% + plant 33: stab=0.83 best_J=0.684 oracle_J=0.593 regret=+15.2% + plant 34: stab=1.00 best_J=5.260 oracle_J=4.887 regret=+7.6% + plant 35: stab=1.00 best_J=3.048 oracle_J=2.916 regret=+4.5% + plant 36: stab=0.83 best_J=0.277 oracle_J=0.198 regret=+39.9% + plant 37: stab=1.00 best_J=4.049 oracle_J=4.026 regret=+0.6% + plant 38: stab=1.00 best_J=0.269 oracle_J=0.216 regret=+24.5% + plant 42: stab=1.00 best_J=0.546 oracle_J=0.433 regret=+26.1% + plant 43: stab=1.00 best_J=2.052 oracle_J=1.958 regret=+4.8% + plant 44: stab=1.00 best_J=0.859 oracle_J=0.716 regret=+19.9% + plant 46: stab=0.83 best_J=1.361 oracle_J=0.735 regret=+85.2% + plant 47: stab=0.83 best_J=3.936 oracle_J=1.862 regret=+111.4% + plant 48: stab=1.00 best_J=0.337 oracle_J=0.266 regret=+26.6% + plant 50: stab=1.00 best_J=1.031 oracle_J=0.781 regret=+32.0% + plant 51: stab=0.83 best_J=0.025 oracle_J=0.023 regret=+7.7% + plant 54: stab=0.83 best_J=3.575 oracle_J=1.776 regret=+101.2% + plant 55: stab=0.33 best_J=10.067 oracle_J=9.275 regret=+8.5% + plant 56: stab=0.67 best_J=25.200 oracle_J=22.030 regret=+14.4% + plant 57: stab=1.00 best_J=1.892 oracle_J=1.776 regret=+6.5% + plant 58: stab=0.83 best_J=0.508 oracle_J=0.286 regret=+77.8% + plant 59: stab=0.83 best_J=2.120 oracle_J=1.601 regret=+32.4% + plant 60: stab=0.67 best_J=0.319 oracle_J=0.182 regret=+75.7% + plant 61: stab=0.67 best_J=1.896 oracle_J=1.728 regret=+9.7% + plant 62: stab=0.50 best_J=17.319 oracle_J=14.319 regret=+21.0% + plant 63: stab=0.83 best_J=2.028 oracle_J=1.705 regret=+19.0% + plant 64: stab=0.67 best_J=2.096 oracle_J=1.557 regret=+34.6% + plant 65: stab=0.33 best_J=6.734 oracle_J=6.216 regret=+8.3% + plant 67: stab=0.83 best_J=1.924 oracle_J=1.795 regret=+7.2% + plant 68: stab=0.50 best_J=22.505 oracle_J=22.152 regret=+1.6% + plant 69: stab=0.50 best_J=4.297 oracle_J=4.179 regret=+2.8% + plant 71: stab=1.00 best_J=2.505 oracle_J=1.959 regret=+27.9% + plant 72: stab=1.00 best_J=3.427 oracle_J=3.185 regret=+7.6% + plant 73: stab=0.17 best_J=17.639 oracle_J=17.222 regret=+2.4% + plant 74: stab=0.83 best_J=1.359 oracle_J=1.124 regret=+20.9% + plant 76: stab=0.33 best_J=2.660 oracle_J=2.407 regret=+10.5% + plant 77: stab=0.83 best_J=0.721 oracle_J=0.383 regret=+88.2% + plant 78: stab=1.00 best_J=1.724 oracle_J=1.713 regret=+0.6% + plant 80: stab=1.00 best_J=3.376 oracle_J=3.328 regret=+1.5% + plant 81: stab=0.33 best_J=4.273 oracle_J=4.160 regret=+2.7% + plant 82: stab=1.00 best_J=6.937 oracle_J=6.516 regret=+6.5% + plant 84: stab=1.00 best_J=4.128 oracle_J=3.767 regret=+9.6% + plant 85: stab=0.17 best_J=1242.474 oracle_J=1219.395 regret=+1.9% + plant 86: stab=1.00 best_J=0.682 oracle_J=0.451 regret=+51.1% + plant 87: stab=1.00 best_J=3.574 oracle_J=3.094 regret=+15.5% + plant 88: stab=1.00 best_J=1.647 oracle_J=1.609 regret=+2.4% + plant 89: stab=0.67 best_J=4.236 oracle_J=4.166 regret=+1.7% + plant 90: stab=0.83 best_J=8.468 oracle_J=4.034 regret=+109.9% + plant 91: stab=0.83 best_J=5.564 oracle_J=4.817 regret=+15.5% + plant 92: stab=0.83 best_J=3.322 oracle_J=2.365 regret=+40.5% + plant 93: stab=0.83 best_J=2.091 oracle_J=1.166 regret=+79.3% + plant 94: stab=0.17 best_J=14.134 oracle_J=14.134 regret=+0.0% + plant 95: stab=1.00 best_J=1.851 oracle_J=1.851 regret=+0.0% + plant 96: stab=0.83 best_J=0.415 oracle_J=0.392 regret=+5.8% + plant 97: stab=0.50 best_J=6.010 oracle_J=5.081 regret=+18.3% + plant 100: stab=1.00 best_J=0.266 oracle_J=0.255 regret=+4.6% + plant 102: stab=0.50 best_J=2.121 oracle_J=2.118 regret=+0.1% + plant 105: stab=0.83 best_J=2.381 oracle_J=1.402 regret=+69.9% + plant 106: stab=0.83 best_J=0.713 oracle_J=0.709 regret=+0.6% + plant 107: stab=0.67 best_J=1.631 oracle_J=0.966 regret=+68.9% + plant 109: stab=1.00 best_J=2.807 oracle_J=2.549 regret=+10.1% + plant 110: stab=1.00 best_J=0.850 oracle_J=0.807 regret=+5.3% + plant 111: stab=1.00 best_J=1.096 oracle_J=1.093 regret=+0.3% + plant 112: stab=0.17 best_J=21.331 oracle_J=18.581 regret=+14.8% + plant 114: stab=0.50 best_J=2.695 oracle_J=0.849 regret=+217.3% + plant 115: stab=0.83 best_J=1.604 oracle_J=1.305 regret=+22.9% + plant 117: stab=0.83 best_J=0.294 oracle_J=0.239 regret=+23.3% + plant 118: stab=1.00 best_J=2.423 oracle_J=1.961 regret=+23.5% + plant 119: stab=1.00 best_J=2.514 oracle_J=2.299 regret=+9.3% + +=== GENERALIZATION SUMMARY === +plants=120 controllers/plant=6 ICs/plant=4 +per-sample stability rate (mean over plants): 0.64 +plants with >=1 stabilizing controller: 0.81 +regret of best controller vs per-plant oracle: median=+9.7% mean=+22.4% (n=97) +amortization: generation 6.5s for 720 controllers (9 ms each); per-plant oracle 24.1s each +saved -> results/generalize/fig_generalize.png diff --git a/results/siso_coherent/repair_fast.txt b/results/siso_coherent/repair_fast.txt new file mode 100644 index 0000000000000000000000000000000000000000..4f9a729ab919b24c18402c4a2493ae951d4fb66c --- /dev/null +++ b/results/siso_coherent/repair_fast.txt @@ -0,0 +1,10 @@ +=== FAST eigenvalue-certified stabilize-repair (SISO, N=120, K=6, seed 0) === +base (prior) plants-stabilizable: 94/120 = 0.783 +failing plants: 26 +repaired (Hurwitz gain certified): 26 -> AFTER 120/120 = 1.000 + kept model's own filter F: 26 + fell back to F=1: 0 + repair gain was g=0 (revert to open loop; plant already open-loop stable): 26 + worst certified margin (max Re eig over repaired): -0.1014 (<0 = all certified stable) +cost: eigenvalue sweep 0.01s for 26 failing plants = 0.4 ms/plant + (vs ~14.6 s/plant rollout sweep -> ~33755x faster) diff --git a/results/siso_coherent/repair_scale.txt b/results/siso_coherent/repair_scale.txt new file mode 100644 index 0000000000000000000000000000000000000000..bf9d8956c8c5a1e901f4d492f50f939ab0d33fc3 --- /dev/null +++ b/results/siso_coherent/repair_scale.txt @@ -0,0 +1,9 @@ +=== REAL (scale) repair: temper the GENERATED controller phi_gamma=gamma*phi_gen (SISO N=120, K=6, seed 0) === +base (prior) plants-stabilizable: 94/120 = 0.783 +failing plants: 26 +repaired by scaling (rollout): 3 -> AFTER 97/120 = 0.808 +generated structure RETAINED (gamma*): median 1.00 mean 1.00 min 1.00 max 1.00 + rescued keeping >=50% of the generated controller (gamma*>=0.5): 3/3 + rescued with a NON-trivial generated shape (gamma*>0): 3/3 + degenerated to gamma*=0 (== the trivial fallback): 0/3 +cost: rollout scale-search 246.4s for 26 plants = 9.48 s/plant diff --git a/results/siso_coherent/sector.log b/results/siso_coherent/sector.log new file mode 100644 index 0000000000000000000000000000000000000000..c1dc9b5bccd0c078ff0b32368270090c4c957b7d --- /dev/null +++ b/results/siso_coherent/sector.log @@ -0,0 +1,9 @@ + +=== STABILIZE-REPAIR (Best_nonlinear_cond_popov_L_sector.pth, N=120, K=6) === +amortized prior plants-stabilizable: 92/120 = 0.767 +failing plants repaired: 28 + rescued keeping the model's own F: 14 + rescued by F=1 fallback: 8 + still hard: 6 +AFTER repair plants-stabilizable: 114/120 = 0.950 (+0.183) +cost: prior 9 ms/controller; repair 386.3s for 28 plants (13.80s/plant, gradient-free) diff --git a/results/witsenhausen/gate.txt b/results/witsenhausen/gate.txt new file mode 100644 index 0000000000000000000000000000000000000000..6b64e6af3ea3af9ca4d420fd203da4d1b4f6b7a3 --- /dev/null +++ b/results/witsenhausen/gate.txt @@ -0,0 +1,30 @@ +Witsenhausen W3 gate -- family: log k~U[log0.1,log1], log sigma~U[log1,log10] +bank: 150 training instances (seed 101); held-out: 60 instances (seed 4242); K=4 candidates/arm + +--- S=0 steering steps --- +arm median J med.regret% p90 regret% max regret% within1% p vs (b) +(a) per-instance multistart (ceiling) 0.49063 +(b) nearest-neighbour retrieval 0.47760 0.13% 3.69% 30.71% 68.3% n/a +(c) inverse-distance blend 0.48684 0.21% 8.80% 30.71% 61.7% 0.00069 +(d) affine optimum 0.61006 0.01% 68.32% 139.25% 53.3% 0.0074 +(e) K=4-start naive GD 0.83049 112.89% 732.46% 1796.74% 0.0% 2.1e-11 + (bootstrap 95% CI, median regret) nearest=(0.003966098854222078, 0.26555753064220783) blend=(0.08399576046968842, 1.1173934856850576) affine=(-0.008091904573284618, 7.953057598968764) naive=(68.25030190693533, 197.2386782981386) + (bootstrap 95% CI, within-1% rate, pts) nearest=(56.666666666666664, 80.0) blend=(49.958333333333336, 73.33333333333333) affine=(41.66666666666667, 66.66666666666666) naive=(0.0, 0.0) + at the benchmark point (k=0.2, sigma=5), absolute J (vs best-known 0.1669, Lee/Lau/Ho 2001 0.1670790): + ceiling(a)=0.416873 nearest(b)=0.388103 blend(c)=0.388103 affine(d)=0.959999 naive(e)=1.000011 + steering+rescore wall time (all queries x arms x candidates, batched) at this S: 154.0s total + +--- S=20 steering steps --- +arm median J med.regret% p90 regret% max regret% within1% p vs (b) +(a) per-instance multistart (ceiling) 0.48668 +(b) nearest-neighbour retrieval 0.46909 -0.00% 3.00% 30.42% 80.0% n/a +(c) inverse-distance blend 0.46909 -0.00% 3.45% 30.42% 76.7% 0.12 +(d) affine optimum 0.52158 0.05% 27.00% 59.49% 53.3% 7.3e-07 +(e) K=4-start naive GD 0.55688 15.49% 135.22% 1260.45% 11.7% 6.3e-10 + (bootstrap 95% CI, median regret) nearest=(-0.010938508456523353, 0.0063906583254214515) blend=(-0.006685963256623548, 0.027333948710640543) affine=(0.004397087281018256, 4.138011483786187) naive=(11.658543997478574, 27.20603585186857) + (bootstrap 95% CI, within-1% rate, pts) nearest=(70.0, 90.0) blend=(65.0, 86.66666666666667) affine=(41.66666666666667, 66.66666666666666) naive=(5.0, 20.0) + at the benchmark point (k=0.2, sigma=5), absolute J (vs best-known 0.1669, Lee/Lau/Ho 2001 0.1670790): + ceiling(a)=0.396640 nearest(b)=0.392105 blend(c)=0.392105 affine(d)=0.420832 naive(e)=0.619862 + steering+rescore wall time (all queries x arms x candidates, batched) at this S: 673.0s total + +GATE (pre-registered; decided at S=20 -- retrieval's best-repaired case): best retrieval arm = nearest (nearest-neighbour retrieval), median regret=-0.00% (pass needs >=5%), within-1% gap=20.0 pts (pass needs >=10). GATE PASSES (training a model is justified). diff --git a/results/witsenhausen/gate_v2.txt b/results/witsenhausen/gate_v2.txt new file mode 100644 index 0000000000000000000000000000000000000000..ac64b86bdba7324bdebad971580099688f6f2733 --- /dev/null +++ b/results/witsenhausen/gate_v2.txt @@ -0,0 +1,28 @@ +Witsenhausen gate v2 (cell class, strengthened solver) -- log k~U[log0.1,log1], log sigma~U[log1,log10] +bank: 150 instances (seed 101, 3/S starts, 1500 Adam steps, mean solve 7.3s); held-out ceiling: 60 instances (seed 4242, 6/S starts, 3000 steps, mean solve 29.1s); K=4 + +--- S=0 steering steps --- +arm median J med.regret% p90 regret% max regret% =5), within-1% gap 10.0 pts (needs >=10). GATE PASSES (training a model is justified). diff --git a/results/witsenhausen/landscape.txt b/results/witsenhausen/landscape.txt new file mode 100644 index 0000000000000000000000000000000000000000..5f50a08bda45a82f693cc9c7a701c61dc7294132 --- /dev/null +++ b/results/witsenhausen/landscape.txt @@ -0,0 +1,46 @@ +Witsenhausen landscape @ (k=0.2, sigma=5.0) +[60 smooth + 60 staircase starts, 250 Adam steps (search n_x0=n_w=100), rescored at n_x0=n_w=400, top 8 L-BFGS-polished] wall=1116.1s seed=0 +best found J=0.168387 (best known ~0.1669, Lee/Lau/Ho 2001 0.1670790; best affine 0.960000; Witsenhausen two-point ~0.4042) + +quadrature convergence at the best converged (non-affine) solution: + n_x0=n_w= 48 J=0.27990258 + n_x0=n_w= 80 J=0.36373100 (delta vs prev N: 8.38e-02) + n_x0=n_w= 128 J=0.42275739 (delta vs prev N: 5.90e-02) + n_x0=n_w= 200 J=0.29805564 (delta vs prev N: 1.25e-01) + n_x0=n_w= 256 J=0.42644757 (delta vs prev N: 1.28e-01) + n_x0=n_w= 320 J=0.27310045 (delta vs prev N: 1.53e-01) + +fraction of ALL 120 converged runs within 1% of 0.1669: 0.8% + smooth-origin only (n=60): 1.7% + staircase-origin only (n=60): 0.0% + +distinct basins (clustered by step count of converged phi): 26 +best/median cost within each basin: + 2-step basin: n= 3 best J=0.349566 median J=0.349567 + 7-step basin: n= 6 best J=0.441619 median J=0.595143 + 8-step basin: n= 2 best J=0.505500 median J=0.565070 + 9-step basin: n= 7 best J=0.658432 median J=0.756199 + 10-step basin: n= 1 best J=1.144861 median J=1.144861 + 11-step basin: n= 4 best J=0.522789 median J=0.667131 + 12-step basin: n= 4 best J=0.168387 median J=0.417650 + 13-step basin: n= 3 best J=0.473605 median J=0.608688 + 14-step basin: n= 3 best J=0.222993 median J=0.349575 + 15-step basin: n= 15 best J=0.630788 median J=0.963650 + 16-step basin: n= 2 best J=0.481733 median J=0.486208 + 17-step basin: n= 11 best J=0.586075 median J=1.055302 + 18-step basin: n= 2 best J=1.326456 median J=1.395630 + 19-step basin: n= 3 best J=0.638526 median J=1.063114 + 20-step basin: n= 2 best J=0.454145 median J=0.502972 + 21-step basin: n= 7 best J=1.050389 median J=1.326234 + 22-step basin: n= 1 best J=0.836890 median J=0.836890 + 23-step basin: n= 7 best J=0.830966 median J=1.089164 + 24-step basin: n= 2 best J=0.643229 median J=0.738592 + 25-step basin: n= 6 best J=0.585554 median J=0.896646 + 26-step basin: n= 3 best J=0.751279 median J=0.883918 + 27-step basin: n= 4 best J=0.665929 median J=1.030921 + 28-step basin: n= 5 best J=0.469345 median J=0.728369 + 29-step basin: n= 7 best J=0.206724 median J=0.580581 + 30-step basin: n= 9 best J=0.505608 median J=0.762360 + 33-step basin: n= 1 best J=0.539142 median J=0.539142 + +cost distribution over ALL 120 local minima: min=0.168387 p10=0.473338 median=0.832461 p90=1.323419 max=1.736215 diff --git a/results/witsenhausen/landscape_corrected.txt b/results/witsenhausen/landscape_corrected.txt new file mode 100644 index 0000000000000000000000000000000000000000..4a95aa23b09a2be5f1ac611f489f5430ff10c126 --- /dev/null +++ b/results/witsenhausen/landscape_corrected.txt @@ -0,0 +1,36 @@ +Witsenhausen landscape @ (k=0.2, sigma=5) -- CORRECTED with composite quadrature +(supersedes landscape.txt, which used the buggy global-GH report precision; see docs/witsenhausen.md 2.1/3.2) +runs=120 (60 smooth + 60 staircase origin) +best found J=0.415891 (7 steps, staircase-S=5 origin) +best smooth-origin=0.474142 best staircase-origin=0.415891 +distinct basins (by step count): 26 +fraction within 1%% of best-known 0.1669: 0.0% +cost distribution: min=0.4159 p10=0.6243 median=0.9275 p90=1.4286 max=2.0923 + +per-basin (step count): n, best J, median J + 2-step: n= 3 best=0.4652 median=0.4655 + 7-step: n= 6 best=0.4159 median=0.6959 + 8-step: n= 2 best=0.6263 median=0.6395 + 9-step: n= 7 best=0.6345 median=0.7669 + 10-step: n= 1 best=0.9848 median=0.9848 + 11-step: n= 4 best=0.5814 median=0.6835 + 12-step: n= 4 best=0.4741 median=0.6309 + 13-step: n= 3 best=0.6620 median=0.6871 + 14-step: n= 3 best=0.5262 median=0.6135 + 15-step: n= 15 best=0.6566 median=0.9658 + 16-step: n= 2 best=0.6715 median=0.6801 + 17-step: n= 11 best=0.6383 median=1.1571 + 18-step: n= 2 best=1.4027 median=1.5132 + 19-step: n= 3 best=0.7170 median=1.0464 + 20-step: n= 2 best=0.6255 median=0.6292 + 21-step: n= 7 best=1.0924 median=1.4453 + 22-step: n= 1 best=0.9775 median=0.9775 + 23-step: n= 7 best=0.8571 median=1.2169 + 24-step: n= 2 best=0.6784 median=0.7526 + 25-step: n= 6 best=0.5873 median=0.9077 + 26-step: n= 3 best=0.9248 median=0.9303 + 27-step: n= 4 best=0.6418 median=0.9568 + 28-step: n= 5 best=0.7633 median=0.9797 + 29-step: n= 7 best=0.5073 median=0.7102 + 30-step: n= 9 best=0.7462 median=0.9752 + 33-step: n= 1 best=0.6868 median=0.6868 diff --git a/results/witsenhausen/landscape_v2.txt b/results/witsenhausen/landscape_v2.txt new file mode 100644 index 0000000000000000000000000000000000000000..52186ea4c7c5cd3bf5ba02d0158d19619911a479 --- /dev/null +++ b/results/witsenhausen/landscape_v2.txt @@ -0,0 +1,11 @@ +Witsenhausen landscape v2 @ (k=0.2, sigma=5), cell class, 50 starts (S=2..8 x6 + 8 uniform), 3000 Adam + L-BFGS top-6 +best J=0.168463 (best known 0.16692), affine 0.9600 +within 1% of best known: 2.0% of 50 starts; distinct plateau counts: 8 + 3-plateau basin: n= 1 best J=0.208781 median J=0.208781 + 4-plateau basin: n= 14 best J=0.172714 median J=0.198351 + 5-plateau basin: n= 17 best J=0.213699 median J=0.296423 + 6-plateau basin: n= 10 best J=0.168463 median J=0.357794 + 7-plateau basin: n= 3 best J=0.456547 median J=0.515877 + 8-plateau basin: n= 3 best J=0.169041 median J=0.319848 + 10-plateau basin: n= 1 best J=0.267391 median J=0.267391 + 12-plateau basin: n= 1 best J=0.410489 median J=0.410489 diff --git a/results/witsenhausen/solver_benchmark.txt b/results/witsenhausen/solver_benchmark.txt new file mode 100644 index 0000000000000000000000000000000000000000..b9da16f6c23ed45b690d802c48d5b70c04cb8aee --- /dev/null +++ b/results/witsenhausen/solver_benchmark.txt @@ -0,0 +1,11 @@ +Witsenhausen solver benchmark @ (k=0.2, sigma=5.0); best known 0.16692 (Karlsson DA), 0.1670790 (Lee-Lau-Ho); Witsenhausen 2-point 0.4042; affine 0.9600. threads=6 +every J below is a fresh no-grad rescore at report precision; J < 0.1669 would be an error. +knot (a) Adam cosine 60 + LBFGS, 28 starts wall= 2.4s best J=0.377640 steps= 3 exact double-quad J=0.377640 +knot (a) Adam cosine 600 + LBFGS, 28 starts wall= 11.3s best J=0.377640 steps= 3 exact double-quad J=0.377640 +knot (a) Adam cosine 3000 + LBFGS, 28 starts wall= 55.2s best J=0.377640 steps= 3 exact double-quad J=0.377640 +knot (b+c) structured soft-step anneal + LBFGS wall= 7.9s best J=0.365601 steps= 11 exact double-quad J=0.365601 +cell (a) Adam cosine 60 + LBFGS, 29 starts wall= 0.5s best J=0.167768 steps= 6 independent brute-force J=0.167768 +cell (a) Adam cosine 600 + LBFGS, 29 starts wall= 0.8s best J=0.167768 steps= 6 independent brute-force J=0.167768 +cell (a) Adam cosine 3000 + LBFGS, 29 starts wall= 2.7s best J=0.167768 steps= 6 independent brute-force J=0.167768 + +NOT COMPLETED: cell (b) structured exact-staircase, cell (c) soft-step continuation and the exact-discontinuous reference rows of witsenhausen.solver_benchmark were OOM-killed (exit 137) on two attempts; they are not needed for the gate (cell (a) reaches 0.167768 = 1.0050x best known, brute-force confirmed). Independent confirmation of the cost itself: docs/witsenhausen.md 2.6. diff --git a/results/witsenhausen/w4_model_arm_v2.txt b/results/witsenhausen/w4_model_arm_v2.txt new file mode 100644 index 0000000000000000000000000000000000000000..eb9b103b211427689ee4f8d316a7d75cee6a2605 --- /dev/null +++ b/results/witsenhausen/w4_model_arm_v2.txt @@ -0,0 +1,3 @@ +W4 (v2, cell class): conditional diffusion model as arm (f), 3 seeded sampling draws, identical gate_v2 protocol (K=4, held-out 60 + benchmark point) + S=0: median_regret = 47.38 +- 4.72 % | within1% = 3.3 +- 0.0 % | median J = 0.57255 +- 0.02578 | bench_J = 0.496203 +- 0.028391 | p vs nearest = [0.0, 0.0, 0.0] [nearest: regret 0.11%, within1 71.7%] + S=20: median_regret = 3.06 +- 0.35 % | within1% = 34.4 +- 3.4 % | median J = 0.32016 +- 0.00526 | bench_J = 0.190055 +- 0.004812 | p vs nearest = [0.0, 0.0, 0.0] [nearest: regret -0.02%, within1 90.0%]