File size: 17,890 Bytes
68a167f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
seeds: plants 321 777 | model torch seeds seed*1000+rho_idx for (0, 1, 2) | pool M = 32
[draw] plants seed=321 model seeds=(0, 1, 2)
oracle ready (174s), found [40 40 40] of 40
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/pendulum_lmi.py:149: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
  prob.solve(solver=solver, verbose=False)
  rho=0.7 done (276s)
  rho=0.8 done (364s)
  rho=0.9 done (448s)
[draw] plants seed=777 model seeds=(0, 1, 2)
oracle ready (157s), found [40 40 40] of 40
  rho=0.7 done (176s)
  rho=0.8 done (190s)
  rho=0.9 done (204s)
RETRIEVAL vs OBJECTIVE-CONDITIONED GENERATION (pendulum)
N=40 plants/draw, rho=(0.7, 0.8, 0.9), K in (8,1), steer S=50
plant seeds: main=321 (exp_cbar_eval's), second=777; model torch seeds = seed*1000+rho_idx, seeds (0, 1, 2)
retr_b pool M=32. spec tolerance ise<=1.05*rho*ISE_ref (exp_cbar_eval satisfied_mask).

== Task 1: stream replica of exp_cbar_eval (seed 321) vs STORED results/pendulum/cbar_eval.json ==
rho=0.7: bestof8 replica met 1.000 cert 1.000 reg 0.84 | stored met 1.000 cert 1.000 reg 0.83 || single replica met 0.925 cert 0.925 reg 8.83 | stored met 0.950 cert 0.850 reg 8.25
rho=0.8: bestof8 replica met 1.000 cert 1.000 reg 0.61 | stored met 1.000 cert 1.000 reg 0.66 || single replica met 0.900 cert 0.850 reg 5.83 | stored met 0.900 cert 0.875 reg 6.17
rho=0.9: bestof8 replica met 1.000 cert 1.000 reg 0.90 | stored met 1.000 cert 1.000 reg 1.51 || single replica met 0.875 cert 0.850 reg 4.26 | stored met 0.850 cert 0.850 reg 5.22
oracle found per rho: [40, 40, 40]

--- MAIN DRAW seed 321: K=8 steer=none ---
rows: rho | arm | spec-met | certified | regret% median [95% CI] p90 n  (model = mean±sd over 3 seeds; CI/p90 from seed 0)
[0.7   ] model      met 1.000±0.000 cert 1.000±0.000 medreg 0.82±0.96 | seed0:   2.17 [ 0.66, 4.42] p90  11.80 n=40
[0.7   ] retr_a     met 1.000       cert 1.000       medreg   1.05 [-0.13, 2.47] p90   9.54 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0793 | p over seeds (McN,Wil): 1,0.079; 1,0.29; 1,0.38
[0.7   ] retr_b     met 1.000       cert 1.000       medreg   1.05 [ 0.16, 2.49] p90   8.38 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0548 | p over seeds (McN,Wil): 1,0.055; 1,0.83; 1,0.16
[0.7   ] retr_c     met 1.000       cert 1.000       medreg   1.11 [-0.58, 2.07] p90   7.96 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.025 | p over seeds (McN,Wil): 1,0.025; 1,0.21; 1,0.17
[0.8   ] model      met 1.000±0.000 cert 1.000±0.000 medreg 0.34±0.42 | seed0:  -0.25 [-0.80, 0.98] p90  16.59 n=40
[0.8   ] retr_a     met 1.000       cert 1.000       medreg   0.82 [-0.46, 2.05] p90   6.89 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.444 | p over seeds (McN,Wil): 1,0.44; 1,0.24; 1,0.082
[0.8   ] retr_b     met 1.000       cert 1.000       medreg   0.38 [-0.79, 1.95] p90   9.84 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.334 | p over seeds (McN,Wil): 1,0.33; 1,0.23; 1,0.097
[0.8   ] retr_c     met 1.000       cert 1.000       medreg   1.13 [-0.36, 1.78] p90   6.89 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.627 | p over seeds (McN,Wil): 1,0.63; 1,0.32; 1,0.11
[0.9   ] model      met 1.000±0.000 cert 1.000±0.000 medreg 0.97±0.40 | seed0:   0.50 [ 0.23, 1.31] p90  14.98 n=40
[0.9   ] retr_a     met 1.000       cert 1.000       medreg   0.60 [-0.28, 2.28] p90   9.58 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.347 | p over seeds (McN,Wil): 1,0.35; 1,0.034; 1,0.1
[0.9   ] retr_b     met 1.000       cert 1.000       medreg   0.91 [-0.01, 2.72] p90   9.58 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.745 | p over seeds (McN,Wil): 1,0.74; 1,0.062; 1,0.28
[0.9   ] retr_c     met 1.000       cert 1.000       medreg   0.14 [-0.69, 1.60] p90   8.28 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.109 | p over seeds (McN,Wil): 1,0.11; 1,0.022; 1,0.031
[pooled] model      met 1.000±0.000 cert 1.000±0.000 medreg 0.73±0.11 | seed0:   0.65 [ 0.22, 1.19] p90  14.78 n=120
[pooled] retr_a     met 1.000       cert 1.000       medreg   0.87 [ 0.21, 1.71] p90   9.56 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0556 | p over seeds (McN,Wil): 1,0.056; 1,0.015; 1,0.014
[pooled] retr_b     met 1.000       cert 1.000       medreg   0.75 [ 0.21, 1.41] p90   9.56 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0608 | p over seeds (McN,Wil): 1,0.061; 1,0.066; 1,0.014
[pooled] retr_c     met 1.000       cert 1.000       medreg   0.86 [-0.10, 1.56] p90   8.26 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0149 | p over seeds (McN,Wil): 1,0.015; 1,0.0092; 1,0.0024

--- MAIN DRAW seed 321: K=1 steer=none ---
rows: rho | arm | spec-met | certified | regret% median [95% CI] p90 n  (model = mean±sd over 3 seeds; CI/p90 from seed 0)
[0.7   ] model      met 0.908±0.012 cert 0.850±0.041 medreg 14.40±5.85 | seed0:  22.68 [ 8.02,36.94] p90  63.08 n=34
[0.7   ] retr_a     met 0.700       cert 0.700       medreg   6.76 [ 2.03,12.26] p90  38.55 n=28 | vs model s0: McNemar 2-10 p=0.0386, Wilcoxon p=0.0395 | p over seeds (McN,Wil): 0.039,0.039; 0.057,0.037; 0.012,0.026
[0.7   ] retr_b     met 0.675       cert 0.675       medreg   9.02 [ 2.90,13.68] p90  29.75 n=27 | vs model s0: McNemar 2-11 p=0.0225, Wilcoxon p=0.0384 | p over seeds (McN,Wil): 0.022,0.038; 0.022,0.052; 0.0063,0.038
[0.7   ] retr_c     met 0.900       cert 0.900       medreg   8.51 [ 3.40,13.84] p90  31.05 n=36 | vs model s0: McNemar 3-3 p=1, Wilcoxon p=0.000906 | p over seeds (McN,Wil): 1,0.00091; 1,0.0013; 1,0.0052
[0.8   ] model      met 0.883±0.051 cert 0.842±0.051 medreg 6.74±1.60 | seed0:   6.20 [ 3.96,15.82] p90  78.90 n=34
[0.8   ] retr_a     met 0.775       cert 0.775       medreg   2.20 [ 0.87, 8.33] p90  24.33 n=31 | vs model s0: McNemar 4-8 p=0.388, Wilcoxon p=0.00247 | p over seeds (McN,Wil): 0.39,0.0025; 0.77,0.031; 0.065,0.06
[0.8   ] retr_b     met 0.725       cert 0.725       medreg   9.86 [ 5.64,13.28] p90  29.73 n=29 | vs model s0: McNemar 4-10 p=0.18, Wilcoxon p=0.127 | p over seeds (McN,Wil): 0.18,0.13; 0.39,0.23; 0.012,0.69
[0.8   ] retr_c     met 0.875       cert 0.875       medreg   5.78 [ 1.78,10.76] p90  27.94 n=35 | vs model s0: McNemar 3-3 p=1, Wilcoxon p=0.000398 | p over seeds (McN,Wil): 1,0.0004; 0.73,0.099; 0.38,0.074
[0.9   ] model      met 0.892±0.031 cert 0.833±0.051 medreg 6.74±0.36 | seed0:   6.29 [ 4.54,11.19] p90  55.91 n=33
[0.9   ] retr_a     met 0.650       cert 0.650       medreg   5.75 [ 4.02,17.38] p90  49.79 n=26 | vs model s0: McNemar 2-10 p=0.0386, Wilcoxon p=0.259 | p over seeds (McN,Wil): 0.039,0.26; 0.013,0.07; 0.0074,0.43
[0.9   ] retr_b     met 0.700       cert 0.700       medreg  11.11 [ 4.49,17.55] p90  35.66 n=28 | vs model s0: McNemar 5-11 p=0.21, Wilcoxon p=0.427 | p over seeds (McN,Wil): 0.21,0.43; 0.057,0.14; 0.012,0.8
[0.9   ] retr_c     met 0.825       cert 0.825       medreg   4.87 [ 1.64, 9.74] p90  31.94 n=33 | vs model s0: McNemar 4-5 p=1, Wilcoxon p=0.0564 | p over seeds (McN,Wil): 1,0.056; 0.55,0.015; 0.29,0.067
[pooled] model      met 0.894±0.027 cert 0.842±0.048 medreg 8.03±0.52 | seed0:   8.62 [ 5.44,13.61] p90  64.20 n=101
[pooled] retr_a     met 0.708       cert 0.708       medreg   6.09 [ 3.64, 8.44] p90  39.49 n=85 | vs model s0: McNemar 8-28 p=0.00119, Wilcoxon p=0.000397 | p over seeds (McN,Wil): 0.0012,0.0004; 0.0022,0.00066; 7.4e-06,0.004
[pooled] retr_b     met 0.700       cert 0.700       medreg   9.73 [ 7.33,13.04] p90  32.05 n=84 | vs model s0: McNemar 11-32 p=0.00191, Wilcoxon p=0.00733 | p over seeds (McN,Wil): 0.0019,0.0073; 0.0011,0.0036; 7.7e-07,0.079
[pooled] retr_c     met 0.867       cert 0.867       medreg   6.78 [ 3.92, 9.18] p90  30.74 n=104 | vs model s0: McNemar 10-11 p=1, Wilcoxon p=1.22e-06 | p over seeds (McN,Wil): 1,1.2e-06; 1,2.3e-05; 0.096,0.00013

--- MAIN DRAW seed 321: K=8 steer=S=50 ---
rows: rho | arm | spec-met | certified | regret% median [95% CI] p90 n  (model = mean±sd over 3 seeds; CI/p90 from seed 0)
[0.7   ] model      met 1.000±0.000 cert 1.000±0.000 medreg 0.17±0.05 | seed0:   0.23 [ 0.01, 0.75] p90   4.35 n=40
[0.7   ] retr_a     met 1.000       cert 1.000       medreg  -0.13 [-0.48, 0.15] p90   1.27 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0725 | p over seeds (McN,Wil): 1,0.073; 1,3.5e-05; 1,0.0056
[0.7   ] retr_b     met 1.000       cert 1.000       medreg  -0.22 [-1.04, 0.14] p90   1.56 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.00726 | p over seeds (McN,Wil): 1,0.0073; 1,0.0018; 1,1.7e-06
[0.7   ] retr_c     met 1.000       cert 1.000       medreg  -0.04 [-0.19, 0.23] p90   1.20 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.077 | p over seeds (McN,Wil): 1,0.077; 1,1.7e-05; 1,0.0067
[0.8   ] model      met 1.000±0.000 cert 1.000±0.000 medreg -0.03±0.01 | seed0:  -0.02 [-0.06, 0.00] p90  11.62 n=40
[0.8   ] retr_a     met 1.000       cert 1.000       medreg  -0.15 [-0.38,-0.08] p90   0.11 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=1.81e-05 | p over seeds (McN,Wil): 1,1.8e-05; 1,5.1e-05; 1,0.00014
[0.8   ] retr_b     met 1.000       cert 1.000       medreg  -0.14 [-0.42,-0.07] p90   0.09 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=8.15e-06 | p over seeds (McN,Wil): 1,8.2e-06; 1,3.3e-05; 1,0.0001
[0.8   ] retr_c     met 1.000       cert 1.000       medreg  -0.20 [-0.45,-0.10] p90   2.02 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.000603 | p over seeds (McN,Wil): 1,0.0006; 1,0.0013; 1,0.00025
[0.9   ] model      met 1.000±0.000 cert 1.000±0.000 medreg -0.03±0.01 | seed0:  -0.02 [-0.09, 0.03] p90   9.95 n=40
[0.9   ] retr_a     met 1.000       cert 1.000       medreg  -0.11 [-0.26,-0.06] p90   0.86 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.00558 | p over seeds (McN,Wil): 1,0.0056; 1,0.0012; 1,0.015
[0.9   ] retr_b     met 1.000       cert 1.000       medreg  -0.13 [-0.27,-0.04] p90   5.73 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.00305 | p over seeds (McN,Wil): 1,0.0031; 1,0.0016; 1,0.017
[0.9   ] retr_c     met 1.000       cert 1.000       medreg  -0.10 [-0.25,-0.07] p90   0.86 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.00726 | p over seeds (McN,Wil): 1,0.0073; 1,0.0014; 1,0.028
[pooled] model      met 1.000±0.000 cert 1.000±0.000 medreg -0.01±0.00 | seed0:  -0.00 [-0.03, 0.12] p90  10.09 n=120
[pooled] retr_a     met 1.000       cert 1.000       medreg  -0.13 [-0.21,-0.09] p90   1.20 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=4.26e-07 | p over seeds (McN,Wil): 1,4.3e-07; 1,2.3e-10; 1,3.4e-07
[pooled] retr_b     met 1.000       cert 1.000       medreg  -0.15 [-0.28,-0.09] p90   1.56 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=1.91e-08 | p over seeds (McN,Wil): 1,1.9e-08; 1,1.1e-08; 1,7.5e-10
[pooled] retr_c     met 1.000       cert 1.000       medreg  -0.12 [-0.20,-0.08] p90   1.33 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=4.94e-06 | p over seeds (McN,Wil): 1,4.9e-06; 1,1.4e-09; 1,9.8e-07

--- MAIN DRAW seed 321: K=1 steer=S=50 ---
rows: rho | arm | spec-met | certified | regret% median [95% CI] p90 n  (model = mean±sd over 3 seeds; CI/p90 from seed 0)
[0.7   ] model      met 0.992±0.012 cert 0.942±0.042 medreg 5.05±3.48 | seed0:   9.82 [ 0.63,28.76] p90  51.33 n=36
[0.7   ] retr_a     met 0.925       cert 0.900       medreg   0.75 [ 0.18, 1.64] p90  26.67 n=36 | vs model s0: McNemar 1-3 p=0.625, Wilcoxon p=0.0105 | p over seeds (McN,Wil): 0.62,0.011; 0.25,0.023; 0.25,0.0063
[0.7   ] retr_b     met 0.875       cert 0.850       medreg   1.03 [ 0.37, 4.53] p90  21.92 n=34 | vs model s0: McNemar 1-5 p=0.219, Wilcoxon p=0.0149 | p over seeds (McN,Wil): 0.22,0.015; 0.062,0.029; 0.062,0.015
[0.7   ] retr_c     met 1.000       cert 1.000       medreg   1.13 [ 0.36, 4.11] p90  21.78 n=40 | vs model s0: McNemar 1-0 p=1, Wilcoxon p=0.00185 | p over seeds (McN,Wil): 1,0.0018; 1,0.0051; 1,0.0019
[0.8   ] model      met 0.992±0.012 cert 0.933±0.012 medreg 0.37±0.22 | seed0:   0.48 [ 0.27, 5.73] p90  55.20 n=37
[0.8   ] retr_a     met 0.975       cert 0.975       medreg   0.03 [ 0.00, 0.25] p90  18.99 n=39 | vs model s0: McNemar 1-1 p=1, Wilcoxon p=0.000159 | p over seeds (McN,Wil): 1,0.00016; 1,0.0031; 1,0.17
[0.8   ] retr_b     met 0.850       cert 0.850       medreg   0.43 [-0.04, 1.84] p90  15.47 n=34 | vs model s0: McNemar 1-6 p=0.125, Wilcoxon p=0.0175 | p over seeds (McN,Wil): 0.12,0.017; 0.031,0.058; 0.031,0.66
[0.8   ] retr_c     met 1.000       cert 1.000       medreg   0.14 [ 0.02, 1.23] p90  22.30 n=40 | vs model s0: McNemar 1-0 p=1, Wilcoxon p=3.36e-06 | p over seeds (McN,Wil): 1,3.4e-06; 1,0.051; 1,0.14
[0.9   ] model      met 0.992±0.012 cert 0.950±0.054 medreg 0.54±0.31 | seed0:   0.89 [ 0.20, 1.57] p90  36.20 n=40
[0.9   ] retr_a     met 0.975       cert 0.975       medreg   0.11 [-0.00, 0.61] p90  25.81 n=39 | vs model s0: McNemar 0-1 p=1, Wilcoxon p=0.0372 | p over seeds (McN,Wil): 1,0.037; 1,0.0091; 1,0.47
[0.9   ] retr_b     met 0.925       cert 0.925       medreg   0.18 [ 0.03, 1.11] p90  26.47 n=37 | vs model s0: McNemar 0-3 p=0.25, Wilcoxon p=0.0363 | p over seeds (McN,Wil): 0.25,0.036; 0.62,0.037; 0.25,0.61
[0.9   ] retr_c     met 1.000       cert 1.000       medreg   0.11 [ 0.04, 0.53] p90  26.30 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.00336 | p over seeds (McN,Wil): 1,0.0034; 1,0.012; 1,0.38
[pooled] model      met 0.992±0.007 cert 0.942±0.027 medreg 0.70±0.23 | seed0:   0.97 [ 0.43, 1.96] p90  51.49 n=113
[pooled] retr_a     met 0.958       cert 0.950       medreg   0.18 [ 0.06, 0.46] p90  24.60 n=114 | vs model s0: McNemar 2-5 p=0.453, Wilcoxon p=6.12e-06 | p over seeds (McN,Wil): 0.45,6.1e-06; 0.22,1.3e-05; 0.062,0.003
[pooled] retr_b     met 0.883       cert 0.875       medreg   0.49 [ 0.18, 1.11] p90  23.23 n=105 | vs model s0: McNemar 2-14 p=0.00418, Wilcoxon p=6.01e-05 | p over seeds (McN,Wil): 0.0042,6e-05; 0.00098,0.00026; 0.00012,0.037
[pooled] retr_c     met 1.000       cert 1.000       medreg   0.35 [ 0.14, 0.82] p90  23.85 n=120 | vs model s0: McNemar 2-0 p=0.5, Wilcoxon p=2.01e-08 | p over seeds (McN,Wil): 0.5,2e-08; 1,3.2e-05; 1,0.0015

synthesis time per plant (ms; mean over rho; verification NOT included):
  model:s0   steer     258.21
  model:s0   synth      89.52
  model:s1   steer     270.89
  model:s1   synth      91.89
  model:s2   steer     267.04
  model:s2   synth      89.57
  retr_a     steer     278.56
  retr_a     synth       0.08
  retr_b     steer     271.39
  retr_b     synth       0.09
  retr_c     steer     253.12
  retr_c     synth       0.14

oracle found per rho (draw 777): [40, 40, 40]

--- SECOND DRAW seed 777: K=8 steer=none ---
rows: rho | arm | spec-met | certified | regret% median [95% CI] p90 n  (model = mean±sd over 3 seeds; CI/p90 from seed 0)
[0.7   ] model      met 1.000±0.000 cert 1.000±0.000 medreg 1.13±0.34 | seed0:   1.23 [-0.22, 3.18] p90  11.81 n=40
[0.7   ] retr_a     met 1.000       cert 1.000       medreg   0.94 [-0.93, 3.71] p90  12.46 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.952 | p over seeds (McN,Wil): 1,0.95; 1,0.045; 1,0.79
[0.7   ] retr_b     met 1.000       cert 1.000       medreg   1.10 [-0.21, 2.94] p90   9.35 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.32 | p over seeds (McN,Wil): 1,0.32; 1,0.51; 1,0.25
[0.7   ] retr_c     met 1.000       cert 1.000       medreg   0.50 [-0.93, 3.02] p90  11.93 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.847 | p over seeds (McN,Wil): 1,0.85; 1,0.12; 1,0.51
[0.8   ] model      met 1.000±0.000 cert 1.000±0.000 medreg 0.97±0.29 | seed0:   0.62 [ 0.01, 1.74] p90  15.47 n=40
[0.8   ] retr_a     met 1.000       cert 1.000       medreg   0.67 [-0.05, 2.42] p90   5.18 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.347 | p over seeds (McN,Wil): 1,0.35; 1,0.18; 1,0.024
[0.8   ] retr_b     met 1.000       cert 1.000       medreg   1.31 [ 0.23, 2.61] p90   5.59 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.361 | p over seeds (McN,Wil): 1,0.36; 1,0.25; 1,0.15
[0.8   ] retr_c     met 1.000       cert 1.000       medreg   0.44 [-0.16, 1.69] p90   5.43 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.277 | p over seeds (McN,Wil): 1,0.28; 1,0.14; 1,0.0094
[0.9   ] model      met 1.000±0.000 cert 1.000±0.000 medreg 1.05±0.05 | seed0:   1.08 [ 0.46, 2.54] p90   8.63 n=40
[0.9   ] retr_a     met 1.000       cert 1.000       medreg   0.64 [-0.44, 1.57] p90   7.26 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.125 | p over seeds (McN,Wil): 1,0.12; 1,0.094; 1,0.71
[0.9   ] retr_b     met 1.000       cert 1.000       medreg   0.65 [-0.34, 1.66] p90   7.14 n=40 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.132 | p over seeds (McN,Wil): 1,0.13; 1,0.16; 1,0.53
[0.9   ] retr_c     met 0.975       cert 0.975       medreg   0.50 [-0.35, 1.24] p90   5.51 n=39 | vs model s0: McNemar 0-1 p=1, Wilcoxon p=0.124 | p over seeds (McN,Wil): 1,0.12; 1,0.1; 1,0.69
[pooled] model      met 1.000±0.000 cert 1.000±0.000 medreg 0.96±0.11 | seed0:   0.92 [ 0.43, 1.59] p90  11.90 n=120
[pooled] retr_a     met 1.000       cert 1.000       medreg   0.80 [ 0.06, 1.32] p90   9.77 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.155 | p over seeds (McN,Wil): 1,0.16; 1,0.61; 1,0.097
[pooled] retr_b     met 1.000       cert 1.000       medreg   0.99 [ 0.14, 1.57] p90   8.49 n=120 | vs model s0: McNemar 0-0 p=1, Wilcoxon p=0.0619 | p over seeds (McN,Wil): 1,0.062; 1,0.29; 1,0.062
[pooled] retr_c     met 0.992       cert 0.992       medreg   0.50 [ 0.01, 1.01] p90   9.27 n=119 | vs model s0: McNemar 0-1 p=1, Wilcoxon p=0.102 | p over seeds (McN,Wil): 1,0.1; 1,0.29; 1,0.037

synthesis time per plant (ms; mean over rho; verification NOT included):
  model:s0   synth      91.34
  model:s1   synth      90.86
  model:s2   synth      91.90
  retr_a     synth       0.08
  retr_b     synth       0.11
  retr_c     synth       0.11

saved