File size: 7,171 Bytes
68a167f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 | [seeds] held-out draws (9999, 4242), sampler (0, 1, 2), subset seeds (0, 1, 2) (rng 1000+ss), x0 seed 777
### mimo held-out draw seed=9999 N=120
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/mimo_lmi.py:29: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
prob.solve(solver=cp.SCS, verbose=False)
MIMO oracle (150-step Adam, arm-independent) ...
138s
bank: 5000 rows / 5000 unique plants; subset seeds (0, 1, 2); fractions (0.05, 0.1, 0.25, 0.5, 1.0)
f=0.05 subset-seed=0: 250 rows, nn-dist 0.1740 (4s)
f=0.05 subset-seed=1: 250 rows, nn-dist 0.1789 (4s)
f=0.05 subset-seed=2: 250 rows, nn-dist 0.1816 (4s)
f=0.1 subset-seed=0: 500 rows, nn-dist 0.1684 (4s)
f=0.1 subset-seed=1: 500 rows, nn-dist 0.1681 (4s)
f=0.1 subset-seed=2: 500 rows, nn-dist 0.1683 (4s)
f=0.25 subset-seed=0: 1250 rows, nn-dist 0.1555 (4s)
f=0.25 subset-seed=1: 1250 rows, nn-dist 0.1550 (4s)
f=0.25 subset-seed=2: 1250 rows, nn-dist 0.1578 (4s)
f=0.5 subset-seed=0: 2500 rows, nn-dist 0.1484 (4s)
f=0.5 subset-seed=1: 2500 rows, nn-dist 0.1491 (4s)
f=0.5 subset-seed=2: 2500 rows, nn-dist 0.1474 (4s)
f=1.0 subset-seed=0: 5000 rows, nn-dist 0.1411 (4s)
### mimo held-out draw seed=4242 N=120
MIMO oracle (150-step Adam, arm-independent) ...
133s
bank: 5000 rows / 5000 unique plants; subset seeds (0, 1, 2); fractions (0.05, 0.1, 0.25, 0.5, 1.0)
f=0.05 subset-seed=0: 250 rows, nn-dist 0.1704 (4s)
f=0.05 subset-seed=1: 250 rows, nn-dist 0.1732 (4s)
f=0.05 subset-seed=2: 250 rows, nn-dist 0.1703 (4s)
f=0.1 subset-seed=0: 500 rows, nn-dist 0.1636 (4s)
f=0.1 subset-seed=1: 500 rows, nn-dist 0.1628 (4s)
f=0.1 subset-seed=2: 500 rows, nn-dist 0.1628 (4s)
f=0.25 subset-seed=0: 1250 rows, nn-dist 0.1512 (4s)
f=0.25 subset-seed=1: 1250 rows, nn-dist 0.1530 (4s)
f=0.25 subset-seed=2: 1250 rows, nn-dist 0.1516 (4s)
f=0.5 subset-seed=0: 2500 rows, nn-dist 0.1438 (4s)
f=0.5 subset-seed=1: 2500 rows, nn-dist 0.1407 (4s)
f=0.5 subset-seed=2: 2500 rows, nn-dist 0.1427 (4s)
f=1.0 subset-seed=0: 5000 rows, nn-dist 0.1342 (4s)
=== BANK-DENSITY SWEEP, MIMO (held-out draws: seeds [9999, 4242], N=[120, 120], pooled N=240) ===
MODEL SIDE IS FIXED: full-data checkpoint, NOT retrained per bank fraction (the pendulum sweep retrained; this one does not).
bank = seeded random subset of training PLANTS (all stored controllers of the chosen plants); fractions (0.05, 0.1, 0.25, 0.5, 1.0); subset seeds (0, 1, 2) (f=1.0: single bank);
model draws: sampler seeds (0, 1, 2); McNemar/regret reference = model seed 0. Verifier/x0/N as retrieval_baselines.py mimo; K=8 and K=1 (K=1 = first candidate).
oracle: retrieval_baselines.mimo_oracle (150-step batched Adam, arm-independent)
regret % = (J_best - J_oracle)/J_oracle on the COMMON success set (plants where BOTH the arm and the model seed-0 draw are covered at that K); label 'cond.'.
coverage 95% CI: bootstrap (5000) over pooled plants of subset-seed 0. McNemar: exact, paired on pooled plants vs model seed 0, reported for each subset seed (min..max p).
----- K=8 ----- MODEL coverage 0.996 +- 0.000 (3 draws), seed-0 CI [0.988,1.000], per-sample 0.961, cond. regret med 2.27%, gen 54.28 ms/ctrl
per-draw model coverage (seed 0/1/2): seed 9999: 1.00/1.00/1.00; seed 4242: 0.99/0.99/0.99
arm=retrieval
f cov mean+-sd CI(ss0) per-samp dCov(ret-model) McNemar p(ss0..) reg_med% cond model reg% same set n_common NN dist
0.05 0.996+-0.003 [1.000,1.000] 0.914 +0.000 1..1 2.68 2.27 238 0.1747
0.10 0.994+-0.002 [0.988,1.000] 0.915 -0.001 1..1 2.76 2.27 238 0.1657
0.25 0.993+-0.002 [0.988,1.000] 0.927 -0.003 1..1 2.55 2.27 238 0.1540
0.50 0.993+-0.002 [0.988,1.000] 0.933 -0.003 1..1 2.59 2.27 238 0.1454
1.00 0.992+-0.000 [0.979,1.000] 0.934 -0.004 1..1 2.79 2.28 238 0.1376
arm=blend
f cov mean+-sd CI(ss0) per-samp dCov(ret-model) McNemar p(ss0..) reg_med% cond model reg% same set n_common NN dist
0.05 0.993+-0.004 [0.988,1.000] 0.916 -0.003 0.5..1 2.85 2.27 237 0.1747
0.10 0.993+-0.004 [0.988,1.000] 0.918 -0.003 0.5..1 2.85 2.27 238 0.1657
0.25 0.992+-0.000 [0.979,1.000] 0.930 -0.004 1..1 2.59 2.27 238 0.1540
0.50 0.993+-0.002 [0.988,1.000] 0.936 -0.003 1..1 2.75 2.27 238 0.1454
1.00 0.992+-0.000 [0.979,1.000] 0.940 -0.004 1..1 2.87 2.28 238 0.1376
----- K=1 ----- MODEL coverage 0.963 +- 0.000 (3 draws), seed-0 CI [0.938,0.983], per-sample 0.963, cond. regret med 11.62%, gen 54.28 ms/ctrl
per-draw model coverage (seed 0/1/2): seed 9999: 0.94/0.94/0.94; seed 4242: 0.98/0.98/0.98
arm=retrieval (blend K=1 == retrieval K=1 by construction)
f cov mean+-sd CI(ss0) per-samp dCov(ret-model) McNemar p(ss0..) reg_med% cond model reg% same set n_common NN dist
0.05 0.932+-0.002 [0.900,0.963] 0.932 -0.031 0.12..0.17 13.62 11.68 218 0.1747
0.10 0.922+-0.019 [0.896,0.958] 0.922 -0.040 0.00086..0.23 15.73 11.52 217 0.1657
0.25 0.925+-0.014 [0.892,0.958] 0.925 -0.037 0.015..0.23 15.29 11.61 217 0.1540
0.50 0.933+-0.009 [0.912,0.971] 0.933 -0.029 0.031..0.27 14.49 11.70 220 0.1454
1.00 0.908+-0.000 [0.871,0.942] 0.908 -0.054 0.0044..0.0044 13.81 11.48 215 0.1376
----- retrieval coverage K=8 vs K=1 per f (mean over subset seeds; pooled draws) -----
f K=1 K=8
0.05 0.932 0.996
0.10 0.922 0.994
0.25 0.925 0.993
0.50 0.933 0.993
1.00 0.908 0.992
----- smallest f at which retrieval MATCHES the model -----
rule (fixed before running): mean retrieval coverage >= model coverage - 0.05 AND pooled McNemar p >= 0.05 for >= 2 of 3 subset seeds (f=1.0: its single bank); 'never' if no f qualifies.
mimo arm=retrieval K=8: f=0.05
mimo arm=blend K=8: f=0.05
mimo arm=retrieval K=1: f=0.05
----- NN distance vs f (normalized conditioning space, mean over queries; mean over subset seeds) -----
f=0.05: draw9999: 0.1782, draw4242: 0.1713
f=0.1: draw9999: 0.1683, draw4242: 0.1631
f=0.25: draw9999: 0.1561, draw4242: 0.1519
f=0.5: draw9999: 0.1483, draw4242: 0.1424
f=1.0: draw9999: 0.1411, draw4242: 0.1342
|