File size: 7,171 Bytes
68a167f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
[seeds] held-out draws (9999, 4242), sampler (0, 1, 2), subset seeds (0, 1, 2) (rng 1000+ss), x0 seed 777

### mimo held-out draw seed=9999 N=120
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/mimo_lmi.py:29: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
  prob.solve(solver=cp.SCS, verbose=False)
MIMO oracle (150-step Adam, arm-independent) ...
  138s
bank: 5000 rows / 5000 unique plants; subset seeds (0, 1, 2); fractions (0.05, 0.1, 0.25, 0.5, 1.0)
  f=0.05 subset-seed=0: 250 rows, nn-dist 0.1740 (4s)
  f=0.05 subset-seed=1: 250 rows, nn-dist 0.1789 (4s)
  f=0.05 subset-seed=2: 250 rows, nn-dist 0.1816 (4s)
  f=0.1 subset-seed=0: 500 rows, nn-dist 0.1684 (4s)
  f=0.1 subset-seed=1: 500 rows, nn-dist 0.1681 (4s)
  f=0.1 subset-seed=2: 500 rows, nn-dist 0.1683 (4s)
  f=0.25 subset-seed=0: 1250 rows, nn-dist 0.1555 (4s)
  f=0.25 subset-seed=1: 1250 rows, nn-dist 0.1550 (4s)
  f=0.25 subset-seed=2: 1250 rows, nn-dist 0.1578 (4s)
  f=0.5 subset-seed=0: 2500 rows, nn-dist 0.1484 (4s)
  f=0.5 subset-seed=1: 2500 rows, nn-dist 0.1491 (4s)
  f=0.5 subset-seed=2: 2500 rows, nn-dist 0.1474 (4s)
  f=1.0 subset-seed=0: 5000 rows, nn-dist 0.1411 (4s)

### mimo held-out draw seed=4242 N=120
MIMO oracle (150-step Adam, arm-independent) ...
  133s
bank: 5000 rows / 5000 unique plants; subset seeds (0, 1, 2); fractions (0.05, 0.1, 0.25, 0.5, 1.0)
  f=0.05 subset-seed=0: 250 rows, nn-dist 0.1704 (4s)
  f=0.05 subset-seed=1: 250 rows, nn-dist 0.1732 (4s)
  f=0.05 subset-seed=2: 250 rows, nn-dist 0.1703 (4s)
  f=0.1 subset-seed=0: 500 rows, nn-dist 0.1636 (4s)
  f=0.1 subset-seed=1: 500 rows, nn-dist 0.1628 (4s)
  f=0.1 subset-seed=2: 500 rows, nn-dist 0.1628 (4s)
  f=0.25 subset-seed=0: 1250 rows, nn-dist 0.1512 (4s)
  f=0.25 subset-seed=1: 1250 rows, nn-dist 0.1530 (4s)
  f=0.25 subset-seed=2: 1250 rows, nn-dist 0.1516 (4s)
  f=0.5 subset-seed=0: 2500 rows, nn-dist 0.1438 (4s)
  f=0.5 subset-seed=1: 2500 rows, nn-dist 0.1407 (4s)
  f=0.5 subset-seed=2: 2500 rows, nn-dist 0.1427 (4s)
  f=1.0 subset-seed=0: 5000 rows, nn-dist 0.1342 (4s)
=== BANK-DENSITY SWEEP, MIMO (held-out draws: seeds [9999, 4242], N=[120, 120], pooled N=240) ===
MODEL SIDE IS FIXED: full-data checkpoint, NOT retrained per bank fraction (the pendulum sweep retrained; this one does not).
bank = seeded random subset of training PLANTS (all stored controllers of the chosen plants); fractions (0.05, 0.1, 0.25, 0.5, 1.0); subset seeds (0, 1, 2) (f=1.0: single bank);
model draws: sampler seeds (0, 1, 2); McNemar/regret reference = model seed 0. Verifier/x0/N as retrieval_baselines.py mimo; K=8 and K=1 (K=1 = first candidate).
oracle: retrieval_baselines.mimo_oracle (150-step batched Adam, arm-independent)
regret % = (J_best - J_oracle)/J_oracle on the COMMON success set (plants where BOTH the arm and the model seed-0 draw are covered at that K); label 'cond.'.
coverage 95% CI: bootstrap (5000) over pooled plants of subset-seed 0. McNemar: exact, paired on pooled plants vs model seed 0, reported for each subset seed (min..max p).

----- K=8 -----  MODEL coverage 0.996 +- 0.000 (3 draws), seed-0 CI [0.988,1.000], per-sample 0.961, cond. regret med 2.27%, gen 54.28 ms/ctrl
per-draw model coverage (seed 0/1/2): seed 9999: 1.00/1.00/1.00; seed 4242: 0.99/0.99/0.99
  arm=retrieval
      f  cov mean+-sd        CI(ss0)  per-samp  dCov(ret-model)       McNemar p(ss0..)  reg_med% cond  model reg% same set  n_common   NN dist
   0.05  0.996+-0.003 [1.000,1.000]     0.914           +0.000                   1..1           2.68                 2.27       238    0.1747
   0.10  0.994+-0.002 [0.988,1.000]     0.915           -0.001                   1..1           2.76                 2.27       238    0.1657
   0.25  0.993+-0.002 [0.988,1.000]     0.927           -0.003                   1..1           2.55                 2.27       238    0.1540
   0.50  0.993+-0.002 [0.988,1.000]     0.933           -0.003                   1..1           2.59                 2.27       238    0.1454
   1.00  0.992+-0.000 [0.979,1.000]     0.934           -0.004                   1..1           2.79                 2.28       238    0.1376
  arm=blend
      f  cov mean+-sd        CI(ss0)  per-samp  dCov(ret-model)       McNemar p(ss0..)  reg_med% cond  model reg% same set  n_common   NN dist
   0.05  0.993+-0.004 [0.988,1.000]     0.916           -0.003                 0.5..1           2.85                 2.27       237    0.1747
   0.10  0.993+-0.004 [0.988,1.000]     0.918           -0.003                 0.5..1           2.85                 2.27       238    0.1657
   0.25  0.992+-0.000 [0.979,1.000]     0.930           -0.004                   1..1           2.59                 2.27       238    0.1540
   0.50  0.993+-0.002 [0.988,1.000]     0.936           -0.003                   1..1           2.75                 2.27       238    0.1454
   1.00  0.992+-0.000 [0.979,1.000]     0.940           -0.004                   1..1           2.87                 2.28       238    0.1376

----- K=1 -----  MODEL coverage 0.963 +- 0.000 (3 draws), seed-0 CI [0.938,0.983], per-sample 0.963, cond. regret med 11.62%, gen 54.28 ms/ctrl
per-draw model coverage (seed 0/1/2): seed 9999: 0.94/0.94/0.94; seed 4242: 0.98/0.98/0.98
  arm=retrieval   (blend K=1 == retrieval K=1 by construction)
      f  cov mean+-sd        CI(ss0)  per-samp  dCov(ret-model)       McNemar p(ss0..)  reg_med% cond  model reg% same set  n_common   NN dist
   0.05  0.932+-0.002 [0.900,0.963]     0.932           -0.031             0.12..0.17          13.62                11.68       218    0.1747
   0.10  0.922+-0.019 [0.896,0.958]     0.922           -0.040          0.00086..0.23          15.73                11.52       217    0.1657
   0.25  0.925+-0.014 [0.892,0.958]     0.925           -0.037            0.015..0.23          15.29                11.61       217    0.1540
   0.50  0.933+-0.009 [0.912,0.971]     0.933           -0.029            0.031..0.27          14.49                11.70       220    0.1454
   1.00  0.908+-0.000 [0.871,0.942]     0.908           -0.054         0.0044..0.0044          13.81                11.48       215    0.1376

----- retrieval coverage K=8 vs K=1 per f (mean over subset seeds; pooled draws) -----
      f     K=1     K=8
   0.05   0.932   0.996
   0.10   0.922   0.994
   0.25   0.925   0.993
   0.50   0.933   0.993
   1.00   0.908   0.992

----- smallest f at which retrieval MATCHES the model -----
rule (fixed before running): mean retrieval coverage >= model coverage - 0.05 AND pooled McNemar p >= 0.05 for >= 2 of 3 subset seeds (f=1.0: its single bank); 'never' if no f qualifies.
  mimo arm=retrieval K=8: f=0.05
  mimo arm=blend K=8: f=0.05
  mimo arm=retrieval K=1: f=0.05

----- NN distance vs f (normalized conditioning space, mean over queries; mean over subset seeds) -----
  f=0.05: draw9999: 0.1782, draw4242: 0.1713
  f=0.1: draw9999: 0.1683, draw4242: 0.1631
  f=0.25: draw9999: 0.1561, draw4242: 0.1519
  f=0.5: draw9999: 0.1483, draw4242: 0.1424
  f=1.0: draw9999: 0.1411, draw4242: 0.1342