matCercola18 commited on
Commit
09f35d5
·
verified ·
1 Parent(s): 5d86adf

X29-A 3A: per-plant optimum in the actuator setting, X27 C-only arms re-scored on J*_A

Browse files
Files changed (49) hide show
  1. results/icml_r3/X29/PREREG.md +185 -0
  2. results/icml_r3/X29/act/heldout_m3/cert_id_c1.npz +3 -0
  3. results/icml_r3/X29/act/heldout_m3/cert_so3_c1.npz +3 -0
  4. results/icml_r3/X29/act/heldout_m3/cert_so3_c2.npz +3 -0
  5. results/icml_r3/X29/act/heldout_m3/cert_so3_c5.npz +3 -0
  6. results/icml_r3/X29/act/heldout_m3/fresh.npz +3 -0
  7. results/icml_r3/X29/act/heldout_m3/id_c1.npz +3 -0
  8. results/icml_r3/X29/act/heldout_m3/jstar.npz +3 -0
  9. results/icml_r3/X29/act/heldout_m3/so3_c1.npz +3 -0
  10. results/icml_r3/X29/act/heldout_m3/so3_c2.npz +3 -0
  11. results/icml_r3/X29/act/heldout_m3/so3_c5.npz +3 -0
  12. results/icml_r3/X29/act/heldout_m4/cert_id_c1.npz +3 -0
  13. results/icml_r3/X29/act/heldout_m4/cert_so3_c1.npz +3 -0
  14. results/icml_r3/X29/act/heldout_m4/cert_so3_c2.npz +3 -0
  15. results/icml_r3/X29/act/heldout_m4/cert_so3_c5.npz +3 -0
  16. results/icml_r3/X29/act/heldout_m4/fresh.npz +3 -0
  17. results/icml_r3/X29/act/heldout_m4/id_c1.npz +3 -0
  18. results/icml_r3/X29/act/heldout_m4/jstar.npz +3 -0
  19. results/icml_r3/X29/act/heldout_m4/so3_c1.npz +3 -0
  20. results/icml_r3/X29/act/heldout_m4/so3_c2.npz +3 -0
  21. results/icml_r3/X29/act/heldout_m4/so3_c5.npz +3 -0
  22. results/icml_r3/X29/act/heldout_m5/cert_id_c1.npz +3 -0
  23. results/icml_r3/X29/act/heldout_m5/cert_so3_c1.npz +3 -0
  24. results/icml_r3/X29/act/heldout_m5/cert_so3_c2.npz +3 -0
  25. results/icml_r3/X29/act/heldout_m5/cert_so3_c5.npz +3 -0
  26. results/icml_r3/X29/act/heldout_m5/fresh.npz +3 -0
  27. results/icml_r3/X29/act/heldout_m5/id_c1.npz +3 -0
  28. results/icml_r3/X29/act/heldout_m5/jstar.npz +3 -0
  29. results/icml_r3/X29/act/heldout_m5/so3_c1.npz +3 -0
  30. results/icml_r3/X29/act/heldout_m5/so3_c2.npz +3 -0
  31. results/icml_r3/X29/act/heldout_m5/so3_c5.npz +3 -0
  32. results/icml_r3/X29/act/report_3a_heldout_m3.txt +9 -0
  33. results/icml_r3/X29/act/report_3a_heldout_m4.txt +9 -0
  34. results/icml_r3/X29/act/report_3a_heldout_m5.txt +9 -0
  35. results/icml_r3/X29/act/selfcheck_cert_m3.json +7 -0
  36. results/icml_r3/X29/act/selfcheck_cert_m4.json +7 -0
  37. results/icml_r3/X29/act/selfcheck_cert_m5.json +7 -0
  38. results/icml_r3/X29/act/selfcheck_cost_m3.json +15 -0
  39. results/icml_r3/X29/act/selfcheck_cost_m4.json +15 -0
  40. results/icml_r3/X29/act/selfcheck_cost_m5.json +15 -0
  41. results/icml_r3/X29/act/x27rescore_d3fm3n8000.json +250 -0
  42. results/icml_r3/X29/act/x27rescore_d3fm3n8000.txt +23 -0
  43. results/icml_r3/X29/act/x27rescore_d3fm4n8000.json +250 -0
  44. results/icml_r3/X29/act/x27rescore_d3fm4n8000.txt +23 -0
  45. results/icml_r3/X29/act/x27rescore_d3fm5n16000.json +250 -0
  46. results/icml_r3/X29/act/x27rescore_d3fm5n16000.txt +23 -0
  47. results/icml_r3/X29/act/x27rows_d3fm3n8000.npy +3 -0
  48. results/icml_r3/X29/act/x27rows_d3fm4n8000.npy +3 -0
  49. results/icml_r3/X29/act/x27rows_d3fm5n16000.npy +3 -0
results/icml_r3/X29/PREREG.md ADDED
@@ -0,0 +1,185 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # X29 pre-registration: C+phi next to C-only, with the same baselines and KPIs
2
+
3
+ Written 2026-11-16, 18:30 UTC, before any X29 code ran or any X29 number existed.
4
+
5
+ Request (relayed by the presentation session dc_presentation, then confirmed by Matteo in the move-HPC session on 2026-11-16):
6
+ - "go", with the actuator setting in two steps (3A below, then he decides 3B);
7
+ - the actuator cost is X27's quadratic cost computed by simulation;
8
+ - m = 5 uses the 16k bank;
9
+ - then: "I need to have the final baseline comparison also for the spline methods".
10
+
11
+ Scope: the deck v10 slides listed by dc_presentation. SISO [35] [36] [37] [38] [73]; the MIMO spline setting (three new slides: certified vs b, regret vs b, CPU vs regret, plus [73]); the actuator setting [50] [51] [52] [84] [73] (after 3B).
12
+
13
+ Names:
14
+ - **C-only**: phi fixed (SISO sat_1; MIMO identity); the model generates the linear part only.
15
+ - **C+phi**: the model generates the linear part and the spline phi (X28).
16
+
17
+ Nothing published is replaced. C-only numbers are re-scored on the common reference J* of each setting, and the earlier numbers stay in their own reports.
18
+
19
+ ## 1. Conventions (all settings)
20
+
21
+ - **Regret** of a method on a plant at budget b: (J_b - J*) / J*.
22
+ - J_b is the cost of the cheapest certified candidate among its first b candidates.
23
+ - A plant with no certified candidate gets regret 1e3, i.e. it counts as a loss (X28 convention).
24
+ - **J\* per setting** = min over all stored labels of both variants:
25
+ - SISO: J_Fonly and the X28 test joint labels.
26
+ - MIMO spline: the T2 id and so3 labels.
27
+ - Actuator: the X27 stored best gain plus the 3A labels.
28
+ - **Reported per method and budget b:**
29
+ - plants with a certified candidate (%);
30
+ - median regret with a bootstrap 95% CI (10,000 resamples over plants);
31
+ - for methods with 3 training seeds: mean and range over seeds, plus seed 0 alone;
32
+ - b = 1, 4, 16, and 64 where it exists.
33
+ - **Slide [73] number:** certified plants at b = 1, the generator minus its own variant's kNN, plant-paired per seed, bootstrap 95% CI. Given for C+phi and for C-only.
34
+ - **No new verdict rule** in SISO and the MIMO spline setting. The registered X28 verdicts stand unchanged.
35
+ - **"Best baseline"** (summary slides): the non-generative method with the lowest b = 16 median regret in the setting; ties go to the higher certified share.
36
+
37
+ ## 2. SISO (X29-S): the 199 X21 test plants, J* as X28 S4
38
+
39
+ | arm | C-only | C+phi |
40
+ |---|---|---|
41
+ | generator | clL seeds 0-2, clip 1.5, t-repair (stored X28 `eval_test` npz; K64 seed 0) | clS seeds 0-2, same (stored; K64 seed 0) |
42
+ | kNN | X22 `knn_s2` on the X21 F-only bank | the same rule on the joint-label bank (below) |
43
+ | regression twin | X23 twins `x21_twin_clL{,_s1,_s2}/best.pth` | new twins of clS, seeds 0-2 (below) |
44
+ | per-plant optimizer | F-only, best of the first k X21 labels | joint, best of the first k joint rows (below) |
45
+ | optimizer from the generator's best sample | 1 F-only start (X24 refine) from clL seed 0 | 1 joint start from clS seed 0 |
46
+
47
+ Shared flat references, scored on J*:
48
+ - tuned saturated LQG or static gain (J_lin, plant files);
49
+ - fixed-order stabilization: best of the 16 stored HIFOO starts (`start_J`).
50
+
51
+ ### Details
52
+ - **kNN, C+phi.** Same ARD metric as X22 (`results/icml_r3/X21/ard.npz`) and the same rule: walk the nearest bank plants (all 1,971) and take the 2 cheapest rows of each, until 16 candidates (64 for K64).
53
+ - Rows are the c = 1 joint rows of `train_x28.rows()`.
54
+ - F is re-encoded on the neighbour at gain k = slope0(phi) (`cl_encode_k`) and decoded on the query plant at the same k (`cl_decode_k`); knots are copied.
55
+ - Then t-repair at k and `x28_eval.score`.
56
+ - Self-check: a bank plant queried as itself returns its cheapest row.
57
+ - **kNN, C-only.** Recomputed with the same X29 code (phi = sat_1, `x28_eval.score`). Its certified rate must agree with the stored X21 `bl_knn_s2_tr` within 1 point (self-check).
58
+ - **Twins.**
59
+ - The C+phi twin is `train_x28` with `X21_LOSS=reg`, `X21_CFG='{"cfg_drop": 0.0}'` and `X28_REP=cl` (the X23 recipe), seeds 0-2, 60,000 steps, best validation checkpoint.
60
+ - Both variants' twins are sampled like X21: candidate 1 deterministic, candidates 2-16 MC-dropout draws, seed 4242.
61
+ - C+phi twin output: knots clipped to [-1, 1], F decoded at k, t-repair at k.
62
+ - Both are scored with `x28_eval.score`. The C-only twin's certified rate must agree with the stored X21 result within 1 point (self-check).
63
+ - **Per-plant optimizers.**
64
+ - F-only with k starts (k in 1, 2, 4, 8, 16) = best of the first k X21 labels, as X24.
65
+ - Joint with k starts = best of min(o_J, J_x21) over the c = 1 rows with src < k (`labels_test`). The joint labels refine the F-only labels.
66
+ - "Joint, all" = J*, so its regret is 0 by construction.
67
+ - **Optimizer from the generator's best sample.**
68
+ - C-only: the X24 refine (ref_optimize, frozen config, freeze_ts) from clL seed 0's best certified b = 16 sample (clip 1.5).
69
+ - C+phi: the S2 warm arm (c = 1, F lr 0.02, knot lr 0.02, 400 steps, cheapest snapshot certified at s) from clS seed 0's best certified b = 16 sample.
70
+ - In both, J = min(start, result).
71
+ - The runs may batch plants with `spline_optimize_multi` (knot lr 0 and the frozen F lr for C-only), but only after an equivalence check against the single-plant function: 3 plants, 25 steps, relative J difference <= 1e-6. If the check fails, the single-plant function is used.
72
+ - **CPU (one timing job).**
73
+ - Plants: the 10 X24 timing plants (`ref_i[::20]`), one process per plant, one thread each, on a node with nothing else of this project running. The node's load average is reported.
74
+ - Timed per plant:
75
+ - one HIFOO start;
76
+ - the F-only optimizer at 1, 4, 16 starts;
77
+ - the joint optimizer at 1, 4, 16 starts (from the plant's first k labels, c = 1);
78
+ - generator sampling on CPU (16 samples) and scoring (16 candidates), for clL and clS seed 0;
79
+ - kNN retrieval plus scoring, both banks;
80
+ - twin sampling plus scoring, both variants;
81
+ - the two "from the best sample" optimizers.
82
+ - Medians over the timing plants. Start counts 2 and 8 are interpolated, as X24 did.
83
+ - CPU per point follows X24's formulas. The joint optimizer at k starts is charged k HIFOO starts + the F-only optimizer (k) + the joint optimizer (k).
84
+ - Batched generator sampling: 199 plants x 16 in one call on the node's GPU, wall time / 199, reported next to the one-plant CPU number.
85
+ - **Compute ratio ([38]).** CPU of the variant's own per-plant optimizer at the fewest starts k in {1, 2, 4, 8, 16} whose median regret is <= the generator's b = 16 median regret (seed 0), divided by the generator's b = 16 CPU. If no k qualifies, both numbers are reported as "not reached at 16 starts".
86
+
87
+ ## 3. MIMO spline setting (X29-T): T2, 200 fresh plants per m (m = 2, 3)
88
+
89
+ Plants, J*_T2, certificates, cost, repair R and the stored generator samples are as in the X28 addendum of 2026-11-15.
90
+
91
+ | arm | C-only | C+phi |
92
+ |---|---|---|
93
+ | generator | (a) seeds 0-2 (stored `gen_fb`) | (b) repaired, primary; (b) as drawn, secondary (stored) |
94
+ | kNN | label (a) of the 16 nearest bank plants | label (b) of the 16 nearest bank plants |
95
+ | regression twin | new, on dataset (a), seeds 0-2 | new, on dataset (b), seeds 0-2 |
96
+
97
+ Shared flat references:
98
+ - **48-start solver** (C-only): the T2 oracle's best gain K0 (`build_sofm`, 48 starts), with the Lyapunov test and the X28 cost.
99
+ - **Per-plant Adam optimizer** (flat lines): from the T2 label runs. C-only uses J_id. C+phi uses J_b = J*_T2, so its regret is 0 by construction.
100
+
101
+ ### Details
102
+ - **kNN.** Banks are the S2-M label plants (m = 3: [64, 8000) of `D2_scale/m3_bank.npz`; m = 2: [64, 4000) of `C_mimo/sof/bank.npz`).
103
+ - Features are scaled to the family box and then by the ARD length scales (x27_filter convention).
104
+ - m = 3 uses `results/icml_r3/X10/ard_m3.npz`. m = 2 uses `x10_retr.fit_ard` fitted on the m = 2 bank (output: label (a)), before any T2 kNN number.
105
+ - Candidates come nearest first, one label per plant.
106
+ - **Twins.**
107
+ - `mh_train` arm "reg" on the S3-M datasets `ds/` (a) and (b), seeds 0-2, with the S3-M steps, batch, lr and EMA; final EMA 0.999 weights.
108
+ - Candidates: 1 deterministic, then 2-16 MC-dropout draws (torch seed = model seed). This is the SISO convention, the one on slide [35].
109
+ - Decoding is as for the generators; ratios are projected to [0.1, 10].
110
+ - **Certificates.** C-only arms use the Lyapunov LMI (alpha = beta = 1). C+phi arms use the circle LMI with repair R (center r_i(1)). C+phi kNN and twin are also reported without repair.
111
+ - **CPU.**
112
+ - One plant at a time: one CPU thread per plant, covering sampling 16 on CPU, certificates (+ repair) and the cost rollouts the scoring code runs.
113
+ - Batched: all 200 plants in one call, with sampling and rollouts on the node's GPU and certificates on its CPU workers; total wall time / 200, with device and worker count stated.
114
+ - kNN and twins are timed the same way.
115
+ - 48-start solver: re-timed per plant on one thread, all 48 starts, with the early stop at the first certified start also reported.
116
+ - Adam optimizer: the time of 25 Adam steps, measured for one plant on one CPU thread and for 200 plants batched on the GPU, scaled to 400 steps plus the measured certification time.
117
+ - Timing plants: the first 10 T2 plants per m.
118
+ - **Budgets:** b = 1, 4, 16 (no 64).
119
+
120
+ ## 4. MIMO actuator setting (X29-A), m = 3, 4, 5
121
+
122
+ ### Definitions
123
+ - **Plants:** X27's 120 test plants per m (`results/icml_r2/D2_scale/m{m}_heldout_{4242,9001}.npz`). K_ref is the stored best gain, with cost J_ref.
124
+ - **Controller:** u = -Phi(phi(K y)).
125
+ - Phi = diag(Phi_i) is the unknown actuator, each Phi_i in the sector [0.4, 2.2] (X27).
126
+ - phi = diag(psi_i) is in the so3 class: odd, piecewise linear, knots at |x| = 1, 3, 5, knot ratios in [0.1, 10], ray extension.
127
+ - C-only is psi = identity.
128
+ - **Certificate:** the circle LMI with per-channel sector [0.4 alpha_i, 2.2 beta_i] (`x28_mimo.circle_lmi` with those ends), where alpha_i and beta_i are the min and max knot ratio.
129
+ - Sound: with alpha_i >= 0.1 > 0, Phi_i(psi_i(x)) / x = (Phi_i(v) / v)(v / x) lies in [0.4 alpha_i, 2.2 beta_i], and the composition is memoryless and diagonal, which is all the circle LMI with diagonal multipliers needs.
130
+ - Self-check: at identity knots it must agree with `family_sofm.verify` on 200 X27 bank gains per m, half of them perturbed (as X28's self-check).
131
+ - **Cost:** J = mean over the 2m unit state directions of the integral from 0 to T of |x|^2 + 0.1 |u|^2, with Phi = identity, RK4 at dt 0.01.
132
+ - Self-check: for linear K, J must equal `family_sofm.cost` (tr(P)/(2m)), with median relative difference <= 1e-3 and maximum <= 1e-2 on the 120 stored test gains per m.
133
+ - T starts at 20 s and doubles until the check passes.
134
+
135
+ ### 3A (now)
136
+ - **Label optimizer** on the 120 test plants per m, X28-M's procedure (`x28_mimo` optimize) with this section's certificate and cost:
137
+ - start K_ref;
138
+ - arm id (knots frozen at identity, c = 1) and arm so3 at gauges c = 1, 2, 5;
139
+ - K lr 0.02 and knot lr 0.02 (the X28-M values);
140
+ - 400 steps, a snapshot every 25 plus snapshot 0, keeping the cheapest certified snapshot.
141
+ - **Outputs:** J_id, J_so3 (the minimum of the so3 starts and id), and J*_A = min(J_ref, J_id, J_so3).
142
+ - **Reported, with no stop rule** (Matteo decides 3B):
143
+ - d = (J_id - J_so3) / J_so3: median with CI, and the share of plants with d >= 2%;
144
+ - (J_ref - J_id) / J_id, the gain from extra optimization alone;
145
+ - a new-direction diagnostic: each arm's best controller on 64 random unit state directions per plant (seed (8080, plant index)), as the median over plants of the per-plant median paired difference.
146
+ - **Expectation, written before the run:** d is small. The quadratic cost does not reward an amplitude-dependent gain, and the composed sector is 5.5 beta/alpha wide, so any nonlinear spline needs more robustness than a linear gain.
147
+ - **Also now:** X27's stored C-only pipeline arms are re-scored on J*_A. Their certified shares are unchanged; regret vs J*_A is reported next to the published regret vs J_ref.
148
+
149
+ ### 3B (only if Matteo says so after 3A; details by addendum before any 3B number)
150
+ - Labels on X27's banks: 8,000 plants at m = 3 and 4, and 16,000 at m = 5 (`m5_bank` + `m5_bank_b`).
151
+ - Generators, 3 seeds each:
152
+ - C+phi generators (so3 target, as X28-M);
153
+ - C-only generators trained on the id arm of the same label run, so both learn from the same optimizer.
154
+ - X27's published generators are reported next to them.
155
+ - Baselines: regression twins and kNN with the X10 ARD metric.
156
+ - X27's pipeline for C+phi: constant-gain filter over the composed sector, circle LMI on the composed sector, repair R, and solver iterations at the composed sector with phi fixed.
157
+ - Timing in one job.
158
+ - Verdict C+phi vs C-only: the X28 rule at b = 16. X27's YES rule is applied to each variant.
159
+
160
+ ## 5. Compute and reporting
161
+ - At most 2 GPU nodes, with CPU work inside them. Timing jobs run on a node with nothing else of this project running.
162
+ - Code:
163
+ - `nonlinear/experiments/x29_siso.py`
164
+ - `nonlinear/mimo_unstable/x29_t2.py`
165
+ - `nonlinear/mimo_unstable/x29_act.py`
166
+ - Results go in `results/icml_r3/X29/{siso,t2,act}/`; the write-up in `docs/icml_r3_x29.md`.
167
+ - Hugging Face: targeted upload of `results/icml_r3/X29/` and the new models (never the X7 paths).
168
+ - No paper edits.
169
+ - Any detail fixed after this file goes in a dated addendum (`PREREG.md`, or `PREREG_siso.md` / `PREREG_t2.md` for the SISO and T2 parts) before the affected number exists.
170
+
171
+ ## Addendum 2026-11-16, 18:20 UTC: X29-A cost quadrature (before any 3A number)
172
+
173
+ The first implementation of the section 4 cost integrated the stage cost with the trapezoid rule on the RK4 steps, the X28-M convention.
174
+
175
+ On the 120 m = 3 test gains, the cost self-check failed at every horizon:
176
+ - median relative difference to tr(P)/(2m) 0.25%, maximum 1.56%;
177
+ - unchanged at T = 20, 40, 80 and 160 s, with |x(T)| <= 8e-5 already at 20 s.
178
+
179
+ So the error is the quadrature on fast closed-loop modes, not the horizon, and doubling T cannot pass.
180
+
181
+ Change: the running cost is now integrated by RK4 as an extra state, with the stage cost evaluated at the four RK4 stages. dt stays 0.01 and the change applies to both arms. The self-check and its thresholds are unchanged and are re-run; if it still fails, dt is halved until it passes.
182
+
183
+ The composed-sector self-check at m = 3 passed (200 gains, 0 disagreements) and does not depend on the cost.
184
+
185
+ (Operational note: srun without `-n 1` started two copies of the self-check script on node 06. Both were killed. The only file they wrote, `selfcheck_cert_m3.json`, came out identical from both copies.)
results/icml_r3/X29/act/heldout_m3/cert_id_c1.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a1777bd6199d2995d11cd7cdfba69717ffa8ad782137833c9bdb524ab1da75c9
3
+ size 12108
results/icml_r3/X29/act/heldout_m3/cert_so3_c1.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ee99b6e4e4e49a53ea654573433cbca6ffe5faab8eb0577900c8570e6b7d4626
3
+ size 15459
results/icml_r3/X29/act/heldout_m3/cert_so3_c2.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f59d5df4e328147a511be0a033cc6203dbd398cdc42e82419b26a3c7e5aef5e2
3
+ size 13152
results/icml_r3/X29/act/heldout_m3/cert_so3_c5.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e38da7c59a5261a6edfa11da7c481d42185f0f8b2a3e8c4fb67e5fc56d4d32ec
3
+ size 12636
results/icml_r3/X29/act/heldout_m3/fresh.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:92aa9fe6421f8b04ee9ed8995e745d3b207df56fbdb65a3e7affb3f82a3ee467
3
+ size 118749
results/icml_r3/X29/act/heldout_m3/id_c1.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:97a56235334f16776869edc1cd6bc71b4e5e3ffe2e226cc2e3c2b2a99769ed79
3
+ size 191891
results/icml_r3/X29/act/heldout_m3/jstar.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:be496d7bca37e511f20fc8185d27376a8afe499475758b40146376e664142da9
3
+ size 5042
results/icml_r3/X29/act/heldout_m3/so3_c1.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:db9a9200b62cac5e66e5d580483c2aa384f88db71511e9c8f1c68aeb81d7357c
3
+ size 463656
results/icml_r3/X29/act/heldout_m3/so3_c2.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d3f56cd63a3d424f9ca8702dcd1a07bc7e05578454cd9553d6444fff52d2e233
3
+ size 460964
results/icml_r3/X29/act/heldout_m3/so3_c5.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1ae2a2fb127666fc4901714bd4b728a0359e5e9bb8a5833133caefa3d43c5456
3
+ size 440765
results/icml_r3/X29/act/heldout_m4/cert_id_c1.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:acf94787736a9b639973e154fa7af834894e21b72952603cd74c391fc976aee4
3
+ size 18609
results/icml_r3/X29/act/heldout_m4/cert_so3_c1.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ab9fe4e22c3f4cf60c47d6e5a9360a274c00052a9063bdfb91c29c25c8fc8732
3
+ size 21013
results/icml_r3/X29/act/heldout_m4/cert_so3_c2.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8bbeb57e6cef77584c141c1d560b1614bae831bcb843ef85031594864a72b79a
3
+ size 18897
results/icml_r3/X29/act/heldout_m4/cert_so3_c5.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2f402ca1207852391c9e222622b2faff7b682e2c8e678a6c63af3627f3e72939
3
+ size 18310
results/icml_r3/X29/act/heldout_m4/fresh.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7bd1c6a2a0f192c1c0e6f6b3d70c6be3bda5a8eadc5aaa315166868dbd408ab5
3
+ size 118679
results/icml_r3/X29/act/heldout_m4/id_c1.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e5da58ad83f6c2743e54be22d7305f430ba0bc58aac43608661be60454174d1f
3
+ size 321578
results/icml_r3/X29/act/heldout_m4/jstar.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:367caeebc244ece0dfdae026a6eb33ca4a798ace0921ee54abfd6927022dc5db
3
+ size 5043
results/icml_r3/X29/act/heldout_m4/so3_c1.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a93ee73d6b7100fea96dcea1f11bf95e7a61dfa1b1bf8179579b86fb49112431
3
+ size 681843
results/icml_r3/X29/act/heldout_m4/so3_c2.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2f7566a3df0c2bd9ec0d8f5974eb2d2e0222abd09a0423711a21e262994c5bfd
3
+ size 679078
results/icml_r3/X29/act/heldout_m4/so3_c5.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1032dd347c6a0e3ed1e46935f86e2921a33ceef3d54e05746249ac052c3f4525
3
+ size 656555
results/icml_r3/X29/act/heldout_m5/cert_id_c1.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0dbf6c4e6c8f46ff64eb7c0cd7719303a52e6a16befc210a1fae73a2ee30a8ab
3
+ size 27048
results/icml_r3/X29/act/heldout_m5/cert_so3_c1.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d4b7c43d54624e83877edca0c3eb96f855eb6ed8edc45c0ad428b10f48272e2e
3
+ size 30719
results/icml_r3/X29/act/heldout_m5/cert_so3_c2.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4e46ed35905aced34f4191ec7ee565c13c19a07fc6f983bb2d7cb3727fbdcd01
3
+ size 27687
results/icml_r3/X29/act/heldout_m5/cert_so3_c5.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:46a9251ee8898ae6e6b030bd26e452c9388c52c23be639a9679f96f2619eef01
3
+ size 27207
results/icml_r3/X29/act/heldout_m5/fresh.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7bedcca3bcd4cf193b3b4dca8fcfd39c030774cd68f0ef11285f10f16a195224
3
+ size 118625
results/icml_r3/X29/act/heldout_m5/id_c1.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:26ed018f08969cbdaffe6c90335212ab84fa33f6f13d8c4fe9dd93542b8d6009
3
+ size 489904
results/icml_r3/X29/act/heldout_m5/jstar.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2dd805bd86cca570e501009ab4ef7afd33a8194faeb884dac6bd67a1861ad916
3
+ size 5042
results/icml_r3/X29/act/heldout_m5/so3_c1.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:616715057316eeaf90e3501aff103c85a459ee3d8926756e3b0815203e6f0bd2
3
+ size 935972
results/icml_r3/X29/act/heldout_m5/so3_c2.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7cfc5582a822fee8375359d592911a33f09481673995b993ed22bb056733a135
3
+ size 932523
results/icml_r3/X29/act/heldout_m5/so3_c5.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a0e24942563def43043a2fcd3254b5bed13f5ae6cc171717b5079d0178095e0e
3
+ size 905631
results/icml_r3/X29/act/report_3a_heldout_m3.txt ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ X29-A 3A, m = 3, 120 X27 test plants; cost = simulated X27 cost (T = 20 s); certificate = circle LMI on [0.4 alpha, 2.2 beta]
2
+ certified: id 120, so3 starts so3_c1 120, so3_c2 120, so3_c5 120
3
+ d = (J_id - J_so3) / J_so3: median 0.00% [0.00, 0.00], >= 2% on 16 of 120, mean 0.68% -> no gain at the optimum by the 2% rule
4
+ extra optimization alone, (J_ref - J_id) / J_id: median -0.00% [-0.00, -0.00]
5
+ best so3 controller from: id_c1 69, so3_c1 22, so3_c2 15, so3_c5 14
6
+ composed-sector ratio max_i (2.2 beta_i) / (0.4 alpha_i) of the best so3 controller: median 5.50 (5.5 = linear), 90th pct 15.97
7
+ J*_A = min(J_ref, J_id, J_so3): median 2.098; J_ref is the J* on 75 plants, id on 18, so3 on 27
8
+ snapshot 0 kept: id 62; nonfinite gradient steps (all starts): 0
9
+ new directions (64 per plant, seed (8080, i)), median over plants of the per-plant median (J_id - J_so3) / J_so3: +0.00% [+0.00, +0.00], > 0 on 30 of 120
results/icml_r3/X29/act/report_3a_heldout_m4.txt ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ X29-A 3A, m = 4, 120 X27 test plants; cost = simulated X27 cost (T = 20 s); certificate = circle LMI on [0.4 alpha, 2.2 beta]
2
+ certified: id 120, so3 starts so3_c1 120, so3_c2 120, so3_c5 118
3
+ d = (J_id - J_so3) / J_so3: median 0.00% [0.00, 0.00], >= 2% on 9 of 120, mean 0.34% -> no gain at the optimum by the 2% rule
4
+ extra optimization alone, (J_ref - J_id) / J_id: median -0.00% [-0.00, -0.00]
5
+ best so3 controller from: id_c1 78, so3_c1 13, so3_c2 13, so3_c5 16
6
+ composed-sector ratio max_i (2.2 beta_i) / (0.4 alpha_i) of the best so3 controller: median 5.50 (5.5 = linear), 90th pct 11.37
7
+ J*_A = min(J_ref, J_id, J_so3): median 2.127; J_ref is the J* on 72 plants, id on 34, so3 on 14
8
+ snapshot 0 kept: id 71; nonfinite gradient steps (all starts): 0
9
+ new directions (64 per plant, seed (8080, i)), median over plants of the per-plant median (J_id - J_so3) / J_so3: +0.00% [+0.00, +0.00], > 0 on 13 of 120
results/icml_r3/X29/act/report_3a_heldout_m5.txt ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ X29-A 3A, m = 5, 120 X27 test plants; cost = simulated X27 cost (T = 20 s); certificate = circle LMI on [0.4 alpha, 2.2 beta]
2
+ certified: id 120, so3 starts so3_c1 120, so3_c2 120, so3_c5 120
3
+ d = (J_id - J_so3) / J_so3: median 0.00% [0.00, 0.00], >= 2% on 13 of 120, mean 0.44% -> no gain at the optimum by the 2% rule
4
+ extra optimization alone, (J_ref - J_id) / J_id: median -0.00% [-0.00, -0.00]
5
+ best so3 controller from: id_c1 76, so3_c1 15, so3_c2 15, so3_c5 14
6
+ composed-sector ratio max_i (2.2 beta_i) / (0.4 alpha_i) of the best so3 controller: median 5.50 (5.5 = linear), 90th pct 12.18
7
+ J*_A = min(J_ref, J_id, J_so3): median 1.985; J_ref is the J* on 77 plants, id on 26, so3 on 17
8
+ snapshot 0 kept: id 75; nonfinite gradient steps (all starts): 0
9
+ new directions (64 per plant, seed (8080, i)), median over plants of the per-plant median (J_id - J_so3) / J_so3: +0.00% [+0.00, +0.00], > 0 on 16 of 120
results/icml_r3/X29/act/selfcheck_cert_m3.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "m": 3,
3
+ "n": 200,
4
+ "verify_true": 121,
5
+ "lmi_true": 121,
6
+ "disagree": 0
7
+ }
results/icml_r3/X29/act/selfcheck_cert_m4.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "m": 4,
3
+ "n": 200,
4
+ "verify_true": 125,
5
+ "lmi_true": 125,
6
+ "disagree": 0
7
+ }
results/icml_r3/X29/act/selfcheck_cert_m5.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "m": 5,
3
+ "n": 200,
4
+ "verify_true": 115,
5
+ "lmi_true": 115,
6
+ "disagree": 0
7
+ }
results/icml_r3/X29/act/selfcheck_cost_m3.json ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "m": 3,
3
+ "n": 120,
4
+ "passed": true,
5
+ "T": 20.0,
6
+ "tries": [
7
+ {
8
+ "T": 20.0,
9
+ "rel_median": 1.2934926530061185e-05,
10
+ "rel_max": 0.0014178489529952417,
11
+ "E_max": 8.026504423256033e-05
12
+ }
13
+ ],
14
+ "J_lyap_median": 2.0978283107427393
15
+ }
results/icml_r3/X29/act/selfcheck_cost_m4.json ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "m": 4,
3
+ "n": 120,
4
+ "passed": true,
5
+ "T": 20.0,
6
+ "tries": [
7
+ {
8
+ "T": 20.0,
9
+ "rel_median": 1.5247918745172171e-05,
10
+ "rel_max": 0.0004239844159611186,
11
+ "E_max": 6.715112940831412e-05
12
+ }
13
+ ],
14
+ "J_lyap_median": 2.1280300984820215
15
+ }
results/icml_r3/X29/act/selfcheck_cost_m5.json ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "m": 5,
3
+ "n": 120,
4
+ "passed": true,
5
+ "T": 20.0,
6
+ "tries": [
7
+ {
8
+ "T": 20.0,
9
+ "rel_median": 2.2052955031149278e-05,
10
+ "rel_max": 0.00025976687111925803,
11
+ "E_max": 0.00014713529635589263
12
+ }
13
+ ],
14
+ "J_lyap_median": 2.009332559093612
15
+ }
results/icml_r3/X29/act/x27rescore_d3fm3n8000.json ADDED
@@ -0,0 +1,250 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "diff raw_b1": {
3
+ "rate": 0.5166666666666666,
4
+ "rate_per_seed": [
5
+ 0.5666666666666667,
6
+ 0.5083333333333333,
7
+ 0.475
8
+ ],
9
+ "regB_ref": 0.035166553853986704,
10
+ "regB_star": 0.04939973422469457,
11
+ "regA_ref": 0.035166553853986704,
12
+ "regA_star": 0.04939973422469457
13
+ },
14
+ "diff raw_b4": {
15
+ "rate": 0.7611111111111111,
16
+ "rate_per_seed": [
17
+ 0.7916666666666666,
18
+ 0.7666666666666667,
19
+ 0.725
20
+ ],
21
+ "regB_ref": 0.026566873792452816,
22
+ "regB_star": 0.036538855499143374,
23
+ "regA_ref": 0.03912745224114178,
24
+ "regA_star": 0.05271375933165312
25
+ },
26
+ "diff raw_b16": {
27
+ "rate": 0.9444444444444443,
28
+ "rate_per_seed": [
29
+ 0.9583333333333334,
30
+ 0.95,
31
+ 0.925
32
+ ],
33
+ "regB_ref": 0.017749766148115964,
34
+ "regB_star": 0.02536120788249141,
35
+ "regA_ref": 0.03975266005687195,
36
+ "regA_star": 0.051211998589972385
37
+ },
38
+ "diff filt_b1": {
39
+ "rate": 0.6861111111111112,
40
+ "rate_per_seed": [
41
+ 0.725,
42
+ 0.75,
43
+ 0.5833333333333334
44
+ ],
45
+ "regB_ref": 0.03864968435767513,
46
+ "regB_star": 0.051490162113829285,
47
+ "regA_ref": 0.03864968435767513,
48
+ "regA_star": 0.051490162113829285
49
+ },
50
+ "diff filt_b4": {
51
+ "rate": 0.9277777777777777,
52
+ "rate_per_seed": [
53
+ 0.9333333333333333,
54
+ 0.925,
55
+ 0.925
56
+ ],
57
+ "regB_ref": 0.024622161334723724,
58
+ "regB_star": 0.03283620184325242,
59
+ "regA_ref": 0.03892824451900456,
60
+ "regA_star": 0.05086767155048544
61
+ },
62
+ "diff filt_b16": {
63
+ "rate": 0.9861111111111112,
64
+ "rate_per_seed": [
65
+ 0.9833333333333333,
66
+ 1.0,
67
+ 0.975
68
+ ],
69
+ "regB_ref": 0.014977849993612624,
70
+ "regB_star": 0.0218213442998485,
71
+ "regA_ref": 0.03976742768706387,
72
+ "regA_star": 0.050933835066115485
73
+ },
74
+ "diff ref_b16_n3": {
75
+ "rate": 0.9944444444444445,
76
+ "rate_per_seed": [
77
+ 0.9916666666666667,
78
+ 1.0,
79
+ 0.9916666666666667
80
+ ],
81
+ "regB_ref": 0.015012072869242443,
82
+ "regB_star": 0.021944205370784453,
83
+ "regA_ref": 0.03991946776173949,
84
+ "regA_star": 0.051211998589972385
85
+ },
86
+ "diff ref_b16_n10": {
87
+ "rate": 0.9972222222222222,
88
+ "rate_per_seed": [
89
+ 1.0,
90
+ 1.0,
91
+ 0.9916666666666667
92
+ ],
93
+ "regB_ref": 0.015046295744872262,
94
+ "regB_star": 0.022067066441720407,
95
+ "regA_ref": 0.04007150783641511,
96
+ "regA_star": 0.051490162113829285
97
+ },
98
+ "twin raw_b1": {
99
+ "rate": 0.4916666666666667,
100
+ "rate_per_seed": [
101
+ 0.475,
102
+ 0.5,
103
+ 0.5
104
+ ],
105
+ "regB_ref": 0.034868968961000446,
106
+ "regB_star": 0.045860235685736495,
107
+ "regA_ref": 0.034868968961000446,
108
+ "regA_star": 0.045860235685736495
109
+ },
110
+ "twin raw_b4": {
111
+ "rate": 0.6222222222222222,
112
+ "rate_per_seed": [
113
+ 0.6333333333333333,
114
+ 0.6333333333333333,
115
+ 0.6
116
+ ],
117
+ "regB_ref": 0.031090565068918607,
118
+ "regB_star": 0.04009868135354511,
119
+ "regA_ref": 0.03602559875874334,
120
+ "regA_star": 0.04612835952042982
121
+ },
122
+ "twin raw_b16": {
123
+ "rate": 0.6888888888888888,
124
+ "rate_per_seed": [
125
+ 0.725,
126
+ 0.6583333333333333,
127
+ 0.6833333333333333
128
+ ],
129
+ "regB_ref": 0.024827462390698996,
130
+ "regB_star": 0.034008343863723534,
131
+ "regA_ref": 0.03665811970701527,
132
+ "regA_star": 0.04772048027118436
133
+ },
134
+ "twin filt_b1": {
135
+ "rate": 0.5722222222222222,
136
+ "rate_per_seed": [
137
+ 0.5583333333333333,
138
+ 0.5833333333333334,
139
+ 0.575
140
+ ],
141
+ "regB_ref": 0.035172887226527116,
142
+ "regB_star": 0.045343068252056584,
143
+ "regA_ref": 0.035172887226527116,
144
+ "regA_star": 0.045343068252056584
145
+ },
146
+ "twin filt_b4": {
147
+ "rate": 0.6805555555555557,
148
+ "rate_per_seed": [
149
+ 0.7,
150
+ 0.6416666666666667,
151
+ 0.7
152
+ ],
153
+ "regB_ref": 0.03138655281275926,
154
+ "regB_star": 0.040342528272814926,
155
+ "regA_ref": 0.036638098220030946,
156
+ "regA_star": 0.04700456890654414
157
+ },
158
+ "twin filt_b16": {
159
+ "rate": 0.7305555555555555,
160
+ "rate_per_seed": [
161
+ 0.75,
162
+ 0.7,
163
+ 0.7416666666666667
164
+ ],
165
+ "regB_ref": 0.02495787433019081,
166
+ "regB_star": 0.03388894221642125,
167
+ "regA_ref": 0.036800685855217674,
168
+ "regA_star": 0.04793953255697314
169
+ },
170
+ "knn raw_b1": {
171
+ "rate": 0.25833333333333336,
172
+ "rate_per_seed": [
173
+ 0.25833333333333336
174
+ ],
175
+ "regB_ref": 0.11371326220225075,
176
+ "regB_star": 0.11629817415072874,
177
+ "regA_ref": 0.11371326220225075,
178
+ "regA_star": 0.11629817415072874
179
+ },
180
+ "knn raw_b4": {
181
+ "rate": 0.44166666666666665,
182
+ "rate_per_seed": [
183
+ 0.44166666666666665
184
+ ],
185
+ "regB_ref": 0.09416819337804627,
186
+ "regB_star": 0.11250915295445885,
187
+ "regA_ref": 0.11635060647060991,
188
+ "regA_star": 0.13716816454409586
189
+ },
190
+ "knn raw_b16": {
191
+ "rate": 0.7083333333333334,
192
+ "rate_per_seed": [
193
+ 0.7083333333333334
194
+ ],
195
+ "regB_ref": 0.08590952578105893,
196
+ "regB_star": 0.09869180446652481,
197
+ "regA_ref": 0.13353007468957645,
198
+ "regA_star": 0.13956844190360695
199
+ },
200
+ "knn filt_b1": {
201
+ "rate": 0.5083333333333333,
202
+ "rate_per_seed": [
203
+ 0.5083333333333333
204
+ ],
205
+ "regB_ref": 0.134908830251387,
206
+ "regB_star": 0.14090533433695862,
207
+ "regA_ref": 0.134908830251387,
208
+ "regA_star": 0.14090533433695862
209
+ },
210
+ "knn filt_b4": {
211
+ "rate": 0.7833333333333333,
212
+ "rate_per_seed": [
213
+ 0.7833333333333333
214
+ ],
215
+ "regB_ref": 0.10139965849798827,
216
+ "regB_star": 0.10879331316422458,
217
+ "regA_ref": 0.13114353705654402,
218
+ "regA_star": 0.1383683032238514
219
+ },
220
+ "knn filt_b16": {
221
+ "rate": 0.8666666666666667,
222
+ "rate_per_seed": [
223
+ 0.8666666666666667
224
+ ],
225
+ "regB_ref": 0.07096639087530376,
226
+ "regB_star": 0.08244864896963622,
227
+ "regA_ref": 0.13353036702921328,
228
+ "regA_star": 0.1383683032238514
229
+ },
230
+ "knn ref_b16_n3": {
231
+ "rate": 0.9333333333333333,
232
+ "rate_per_seed": [
233
+ 0.9333333333333333
234
+ ],
235
+ "regB_ref": 0.07228077375089703,
236
+ "regB_star": 0.0830961623057771,
237
+ "regA_ref": 0.12499251210513573,
238
+ "regA_star": 0.13353036702921328
239
+ },
240
+ "knn ref_b16_n10": {
241
+ "rate": 0.9333333333333333,
242
+ "rate_per_seed": [
243
+ 0.9333333333333333
244
+ ],
245
+ "regB_ref": 0.07337219985527899,
246
+ "regB_star": 0.08575503047710231,
247
+ "regA_ref": 0.12802583666738943,
248
+ "regA_star": 0.13421974481011856
249
+ }
250
+ }
results/icml_r3/X29/act/x27rescore_d3fm3n8000.txt ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ X29-A: X27's C-only arms re-scored, m = 3, tag d3fm3n8000, 120 test plants. J*_A = min(J_ref, 3A id and so3 labels) is below J_ref on 45 plants (median (J_ref - J*_A) / J*_A 0.000%).
2
+ diff raw_b1 certified 51.7% (published 51.7%) | cheapest certified: regret vs J_ref 3.52% (published 3.52%), vs J*_A 4.94% | first certified: vs J_ref 3.52%, vs J*_A 4.94%
3
+ diff raw_b4 certified 76.1% (published 76.1%) | cheapest certified: regret vs J_ref 2.66% (published 2.66%), vs J*_A 3.65% | first certified: vs J_ref 3.91%, vs J*_A 5.27%
4
+ diff raw_b16 certified 94.4% (published 94.4%) | cheapest certified: regret vs J_ref 1.77% (published 1.77%), vs J*_A 2.54% | first certified: vs J_ref 3.98%, vs J*_A 5.12%
5
+ diff filt_b1 certified 68.6% (published 68.6%) | cheapest certified: regret vs J_ref 3.86% (published 3.86%), vs J*_A 5.15% | first certified: vs J_ref 3.86%, vs J*_A 5.15%
6
+ diff filt_b4 certified 92.8% (published 92.8%) | cheapest certified: regret vs J_ref 2.46% (published 2.46%), vs J*_A 3.28% | first certified: vs J_ref 3.89%, vs J*_A 5.09%
7
+ diff filt_b16 certified 98.6% (published 98.6%) | cheapest certified: regret vs J_ref 1.50% (published 1.50%), vs J*_A 2.18% | first certified: vs J_ref 3.98%, vs J*_A 5.09%
8
+ diff ref_b16_n3 certified 99.4% (published 99.4%) | cheapest certified: regret vs J_ref 1.50% (published 1.50%), vs J*_A 2.19% | first certified: vs J_ref 3.99%, vs J*_A 5.12%
9
+ diff ref_b16_n10 certified 99.7% (published 99.7%) | cheapest certified: regret vs J_ref 1.50% (published 1.50%), vs J*_A 2.21% | first certified: vs J_ref 4.01%, vs J*_A 5.15%
10
+ twin raw_b1 certified 49.2% (published 49.2%) | cheapest certified: regret vs J_ref 3.49% (published 3.49%), vs J*_A 4.59% | first certified: vs J_ref 3.49%, vs J*_A 4.59%
11
+ twin raw_b4 certified 62.2% (published 62.2%) | cheapest certified: regret vs J_ref 3.11% (published 3.11%), vs J*_A 4.01% | first certified: vs J_ref 3.60%, vs J*_A 4.61%
12
+ twin raw_b16 certified 68.9% (published 68.9%) | cheapest certified: regret vs J_ref 2.48% (published 2.48%), vs J*_A 3.40% | first certified: vs J_ref 3.67%, vs J*_A 4.77%
13
+ twin filt_b1 certified 57.2% (published 57.2%) | cheapest certified: regret vs J_ref 3.52% (published 3.52%), vs J*_A 4.53% | first certified: vs J_ref 3.52%, vs J*_A 4.53%
14
+ twin filt_b4 certified 68.1% (published 68.1%) | cheapest certified: regret vs J_ref 3.14% (published 3.14%), vs J*_A 4.03% | first certified: vs J_ref 3.66%, vs J*_A 4.70%
15
+ twin filt_b16 certified 73.1% (published 73.1%) | cheapest certified: regret vs J_ref 2.50% (published 2.50%), vs J*_A 3.39% | first certified: vs J_ref 3.68%, vs J*_A 4.79%
16
+ knn raw_b1 certified 25.8% (published 25.8%) | cheapest certified: regret vs J_ref 11.37% (published 11.37%), vs J*_A 11.63% | first certified: vs J_ref 11.37%, vs J*_A 11.63%
17
+ knn raw_b4 certified 44.2% (published 44.2%) | cheapest certified: regret vs J_ref 9.42% (published 9.42%), vs J*_A 11.25% | first certified: vs J_ref 11.64%, vs J*_A 13.72%
18
+ knn raw_b16 certified 70.8% (published 70.8%) | cheapest certified: regret vs J_ref 8.59% (published 8.59%), vs J*_A 9.87% | first certified: vs J_ref 13.35%, vs J*_A 13.96%
19
+ knn filt_b1 certified 50.8% (published 50.8%) | cheapest certified: regret vs J_ref 13.49% (published 13.49%), vs J*_A 14.09% | first certified: vs J_ref 13.49%, vs J*_A 14.09%
20
+ knn filt_b4 certified 78.3% (published 78.3%) | cheapest certified: regret vs J_ref 10.14% (published 10.14%), vs J*_A 10.88% | first certified: vs J_ref 13.11%, vs J*_A 13.84%
21
+ knn filt_b16 certified 86.7% (published 86.7%) | cheapest certified: regret vs J_ref 7.10% (published 7.10%), vs J*_A 8.24% | first certified: vs J_ref 13.35%, vs J*_A 13.84%
22
+ knn ref_b16_n3 certified 93.3% (published 93.3%) | cheapest certified: regret vs J_ref 7.23% (published 7.23%), vs J*_A 8.31% | first certified: vs J_ref 12.50%, vs J*_A 13.35%
23
+ knn ref_b16_n10 certified 93.3% (published 93.3%) | cheapest certified: regret vs J_ref 7.34% (published 7.34%), vs J*_A 8.58% | first certified: vs J_ref 12.80%, vs J*_A 13.42%
results/icml_r3/X29/act/x27rescore_d3fm4n8000.json ADDED
@@ -0,0 +1,250 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "diff raw_b1": {
3
+ "rate": 0.31666666666666665,
4
+ "rate_per_seed": [
5
+ 0.30833333333333335,
6
+ 0.325,
7
+ 0.31666666666666665
8
+ ],
9
+ "regB_ref": 0.04281187301679423,
10
+ "regB_star": 0.04962915968635695,
11
+ "regA_ref": 0.04281187301679423,
12
+ "regA_star": 0.04962915968635695
13
+ },
14
+ "diff raw_b4": {
15
+ "rate": 0.5722222222222222,
16
+ "rate_per_seed": [
17
+ 0.6,
18
+ 0.5666666666666667,
19
+ 0.55
20
+ ],
21
+ "regB_ref": 0.04023679269097036,
22
+ "regB_star": 0.047381757855627926,
23
+ "regA_ref": 0.04831828056771548,
24
+ "regA_star": 0.057019597285727364
25
+ },
26
+ "diff raw_b16": {
27
+ "rate": 0.7888888888888889,
28
+ "rate_per_seed": [
29
+ 0.8,
30
+ 0.775,
31
+ 0.7916666666666666
32
+ ],
33
+ "regB_ref": 0.03206114493999934,
34
+ "regB_star": 0.03878398370676894,
35
+ "regA_ref": 0.05338276992090041,
36
+ "regA_star": 0.05942518962649945
37
+ },
38
+ "diff filt_b1": {
39
+ "rate": 0.5249999999999999,
40
+ "rate_per_seed": [
41
+ 0.5083333333333333,
42
+ 0.5333333333333333,
43
+ 0.5333333333333333
44
+ ],
45
+ "regB_ref": 0.05256149004484412,
46
+ "regB_star": 0.05789482111431554,
47
+ "regA_ref": 0.05256149004484412,
48
+ "regA_star": 0.05789482111431554
49
+ },
50
+ "diff filt_b4": {
51
+ "rate": 0.8250000000000001,
52
+ "rate_per_seed": [
53
+ 0.8166666666666667,
54
+ 0.8166666666666667,
55
+ 0.8416666666666667
56
+ ],
57
+ "regB_ref": 0.04154891326937804,
58
+ "regB_star": 0.05096408010028841,
59
+ "regA_ref": 0.05679915606087227,
60
+ "regA_star": 0.06378147142694024
61
+ },
62
+ "diff filt_b16": {
63
+ "rate": 0.9194444444444444,
64
+ "rate_per_seed": [
65
+ 0.8916666666666667,
66
+ 0.9166666666666666,
67
+ 0.95
68
+ ],
69
+ "regB_ref": 0.030428223592535053,
70
+ "regB_star": 0.03729421986230963,
71
+ "regA_ref": 0.05678969283770341,
72
+ "regA_star": 0.06378147142694024
73
+ },
74
+ "diff ref_b16_n3": {
75
+ "rate": 0.9666666666666667,
76
+ "rate_per_seed": [
77
+ 0.975,
78
+ 0.95,
79
+ 0.975
80
+ ],
81
+ "regB_ref": 0.032163477351268455,
82
+ "regB_star": 0.03848414313894111,
83
+ "regA_ref": 0.05752555203944998,
84
+ "regA_star": 0.06417574626693279
85
+ },
86
+ "diff ref_b16_n10": {
87
+ "rate": 0.9777777777777777,
88
+ "rate_per_seed": [
89
+ 0.9833333333333333,
90
+ 0.9666666666666667,
91
+ 0.9833333333333333
92
+ ],
93
+ "regB_ref": 0.03239370322615842,
94
+ "regB_star": 0.03848414313894111,
95
+ "regA_ref": 0.0571452435546439,
96
+ "regA_star": 0.0639596586478054
97
+ },
98
+ "twin raw_b1": {
99
+ "rate": 0.3138888888888889,
100
+ "rate_per_seed": [
101
+ 0.31666666666666665,
102
+ 0.275,
103
+ 0.35
104
+ ],
105
+ "regB_ref": 0.05077043279796922,
106
+ "regB_star": 0.054942478154655694,
107
+ "regA_ref": 0.05077043279796922,
108
+ "regA_star": 0.054942478154655694
109
+ },
110
+ "twin raw_b4": {
111
+ "rate": 0.4305555555555555,
112
+ "rate_per_seed": [
113
+ 0.4083333333333333,
114
+ 0.44166666666666665,
115
+ 0.44166666666666665
116
+ ],
117
+ "regB_ref": 0.040770558505952126,
118
+ "regB_star": 0.04486400154999837,
119
+ "regA_ref": 0.044869776269671036,
120
+ "regA_star": 0.051729042499672806
121
+ },
122
+ "twin raw_b16": {
123
+ "rate": 0.48333333333333334,
124
+ "rate_per_seed": [
125
+ 0.4583333333333333,
126
+ 0.49166666666666664,
127
+ 0.5
128
+ ],
129
+ "regB_ref": 0.03419325650322769,
130
+ "regB_star": 0.03997382209205025,
131
+ "regA_ref": 0.04478727476519462,
132
+ "regA_star": 0.051812151699132114
133
+ },
134
+ "twin filt_b1": {
135
+ "rate": 0.3666666666666667,
136
+ "rate_per_seed": [
137
+ 0.38333333333333336,
138
+ 0.31666666666666665,
139
+ 0.4
140
+ ],
141
+ "regB_ref": 0.0509702614716806,
142
+ "regB_star": 0.05496224880262157,
143
+ "regA_ref": 0.0509702614716806,
144
+ "regA_star": 0.05496224880262157
145
+ },
146
+ "twin filt_b4": {
147
+ "rate": 0.4861111111111111,
148
+ "rate_per_seed": [
149
+ 0.475,
150
+ 0.48333333333333334,
151
+ 0.5
152
+ ],
153
+ "regB_ref": 0.039893127429255326,
154
+ "regB_star": 0.044859535745951584,
155
+ "regA_ref": 0.044869776269671036,
156
+ "regA_star": 0.051729042499672806
157
+ },
158
+ "twin filt_b16": {
159
+ "rate": 0.5222222222222221,
160
+ "rate_per_seed": [
161
+ 0.5083333333333333,
162
+ 0.525,
163
+ 0.5333333333333333
164
+ ],
165
+ "regB_ref": 0.03461930665839119,
166
+ "regB_star": 0.039952457345226344,
167
+ "regA_ref": 0.045169830011552614,
168
+ "regA_star": 0.05347942435820763
169
+ },
170
+ "knn raw_b1": {
171
+ "rate": 0.14166666666666666,
172
+ "rate_per_seed": [
173
+ 0.14166666666666666
174
+ ],
175
+ "regB_ref": 0.11972585447859752,
176
+ "regB_star": 0.1203804444936909,
177
+ "regA_ref": 0.11972585447859752,
178
+ "regA_star": 0.1203804444936909
179
+ },
180
+ "knn raw_b4": {
181
+ "rate": 0.39166666666666666,
182
+ "rate_per_seed": [
183
+ 0.39166666666666666
184
+ ],
185
+ "regB_ref": 0.11972585447859752,
186
+ "regB_star": 0.1249036336281455,
187
+ "regA_ref": 0.1249036336281455,
188
+ "regA_star": 0.13069438483572582
189
+ },
190
+ "knn raw_b16": {
191
+ "rate": 0.5916666666666667,
192
+ "rate_per_seed": [
193
+ 0.5916666666666667
194
+ ],
195
+ "regB_ref": 0.10947876745055107,
196
+ "regB_star": 0.11616937090964607,
197
+ "regA_ref": 0.13058469845406107,
198
+ "regA_star": 0.1400175969983004
199
+ },
200
+ "knn filt_b1": {
201
+ "rate": 0.45,
202
+ "rate_per_seed": [
203
+ 0.45
204
+ ],
205
+ "regB_ref": 0.11863223190950545,
206
+ "regB_star": 0.12336736317427233,
207
+ "regA_ref": 0.11863223190950545,
208
+ "regA_star": 0.12336736317427233
209
+ },
210
+ "knn filt_b4": {
211
+ "rate": 0.75,
212
+ "rate_per_seed": [
213
+ 0.75
214
+ ],
215
+ "regB_ref": 0.11100685037250602,
216
+ "regB_star": 0.11698292149647704,
217
+ "regA_ref": 0.13511513600466607,
218
+ "regA_star": 0.14419869136826147
219
+ },
220
+ "knn filt_b16": {
221
+ "rate": 0.7916666666666666,
222
+ "rate_per_seed": [
223
+ 0.7916666666666666
224
+ ],
225
+ "regB_ref": 0.0926797810653952,
226
+ "regB_star": 0.09917966684483925,
227
+ "regA_ref": 0.1400175969983004,
228
+ "regA_star": 0.14697604799704078
229
+ },
230
+ "knn ref_b16_n3": {
231
+ "rate": 0.9,
232
+ "rate_per_seed": [
233
+ 0.9
234
+ ],
235
+ "regB_ref": 0.09150018593164855,
236
+ "regB_star": 0.09801292912248993,
237
+ "regA_ref": 0.12780736488049338,
238
+ "regA_star": 0.13983158527678574
239
+ },
240
+ "knn ref_b16_n10": {
241
+ "rate": 0.9333333333333333,
242
+ "rate_per_seed": [
243
+ 0.9333333333333333
244
+ ],
245
+ "regB_ref": 0.09026618689333521,
246
+ "regB_star": 0.09677288836815534,
247
+ "regA_ref": 0.1258708743514112,
248
+ "regA_star": 0.1349668764575651
249
+ }
250
+ }
results/icml_r3/X29/act/x27rescore_d3fm4n8000.txt ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ X29-A: X27's C-only arms re-scored, m = 4, tag d3fm4n8000, 120 test plants. J*_A = min(J_ref, 3A id and so3 labels) is below J_ref on 48 plants (median (J_ref - J*_A) / J*_A 0.000%).
2
+ diff raw_b1 certified 31.7% (published 31.7%) | cheapest certified: regret vs J_ref 4.28% (published 4.28%), vs J*_A 4.96% | first certified: vs J_ref 4.28%, vs J*_A 4.96%
3
+ diff raw_b4 certified 57.2% (published 57.2%) | cheapest certified: regret vs J_ref 4.02% (published 4.02%), vs J*_A 4.74% | first certified: vs J_ref 4.83%, vs J*_A 5.70%
4
+ diff raw_b16 certified 78.9% (published 78.9%) | cheapest certified: regret vs J_ref 3.21% (published 3.21%), vs J*_A 3.88% | first certified: vs J_ref 5.34%, vs J*_A 5.94%
5
+ diff filt_b1 certified 52.5% (published 52.5%) | cheapest certified: regret vs J_ref 5.26% (published 5.26%), vs J*_A 5.79% | first certified: vs J_ref 5.26%, vs J*_A 5.79%
6
+ diff filt_b4 certified 82.5% (published 82.5%) | cheapest certified: regret vs J_ref 4.15% (published 4.15%), vs J*_A 5.10% | first certified: vs J_ref 5.68%, vs J*_A 6.38%
7
+ diff filt_b16 certified 91.9% (published 91.9%) | cheapest certified: regret vs J_ref 3.04% (published 3.04%), vs J*_A 3.73% | first certified: vs J_ref 5.68%, vs J*_A 6.38%
8
+ diff ref_b16_n3 certified 96.7% (published 96.7%) | cheapest certified: regret vs J_ref 3.22% (published 3.22%), vs J*_A 3.85% | first certified: vs J_ref 5.75%, vs J*_A 6.42%
9
+ diff ref_b16_n10 certified 97.8% (published 97.8%) | cheapest certified: regret vs J_ref 3.24% (published 3.24%), vs J*_A 3.85% | first certified: vs J_ref 5.71%, vs J*_A 6.40%
10
+ twin raw_b1 certified 31.4% (published 31.4%) | cheapest certified: regret vs J_ref 5.08% (published 5.08%), vs J*_A 5.49% | first certified: vs J_ref 5.08%, vs J*_A 5.49%
11
+ twin raw_b4 certified 43.1% (published 43.1%) | cheapest certified: regret vs J_ref 4.08% (published 4.08%), vs J*_A 4.49% | first certified: vs J_ref 4.49%, vs J*_A 5.17%
12
+ twin raw_b16 certified 48.3% (published 48.3%) | cheapest certified: regret vs J_ref 3.42% (published 3.42%), vs J*_A 4.00% | first certified: vs J_ref 4.48%, vs J*_A 5.18%
13
+ twin filt_b1 certified 36.7% (published 36.7%) | cheapest certified: regret vs J_ref 5.10% (published 5.10%), vs J*_A 5.50% | first certified: vs J_ref 5.10%, vs J*_A 5.50%
14
+ twin filt_b4 certified 48.6% (published 48.6%) | cheapest certified: regret vs J_ref 3.99% (published 3.99%), vs J*_A 4.49% | first certified: vs J_ref 4.49%, vs J*_A 5.17%
15
+ twin filt_b16 certified 52.2% (published 52.2%) | cheapest certified: regret vs J_ref 3.46% (published 3.46%), vs J*_A 4.00% | first certified: vs J_ref 4.52%, vs J*_A 5.35%
16
+ knn raw_b1 certified 14.2% (published 14.2%) | cheapest certified: regret vs J_ref 11.97% (published 11.97%), vs J*_A 12.04% | first certified: vs J_ref 11.97%, vs J*_A 12.04%
17
+ knn raw_b4 certified 39.2% (published 39.2%) | cheapest certified: regret vs J_ref 11.97% (published 11.97%), vs J*_A 12.49% | first certified: vs J_ref 12.49%, vs J*_A 13.07%
18
+ knn raw_b16 certified 59.2% (published 59.2%) | cheapest certified: regret vs J_ref 10.95% (published 10.95%), vs J*_A 11.62% | first certified: vs J_ref 13.06%, vs J*_A 14.00%
19
+ knn filt_b1 certified 45.0% (published 45.0%) | cheapest certified: regret vs J_ref 11.86% (published 11.86%), vs J*_A 12.34% | first certified: vs J_ref 11.86%, vs J*_A 12.34%
20
+ knn filt_b4 certified 75.0% (published 75.0%) | cheapest certified: regret vs J_ref 11.10% (published 11.10%), vs J*_A 11.70% | first certified: vs J_ref 13.51%, vs J*_A 14.42%
21
+ knn filt_b16 certified 79.2% (published 79.2%) | cheapest certified: regret vs J_ref 9.27% (published 9.27%), vs J*_A 9.92% | first certified: vs J_ref 14.00%, vs J*_A 14.70%
22
+ knn ref_b16_n3 certified 90.0% (published 90.0%) | cheapest certified: regret vs J_ref 9.15% (published 9.15%), vs J*_A 9.80% | first certified: vs J_ref 12.78%, vs J*_A 13.98%
23
+ knn ref_b16_n10 certified 93.3% (published 93.3%) | cheapest certified: regret vs J_ref 9.03% (published 9.03%), vs J*_A 9.68% | first certified: vs J_ref 12.59%, vs J*_A 13.50%
results/icml_r3/X29/act/x27rescore_d3fm5n16000.json ADDED
@@ -0,0 +1,250 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "diff raw_b1": {
3
+ "rate": 0.23611111111111113,
4
+ "rate_per_seed": [
5
+ 0.24166666666666667,
6
+ 0.225,
7
+ 0.24166666666666667
8
+ ],
9
+ "regB_ref": 0.054012242469847704,
10
+ "regB_star": 0.06580516710136042,
11
+ "regA_ref": 0.054012242469847704,
12
+ "regA_star": 0.06580516710136042
13
+ },
14
+ "diff raw_b4": {
15
+ "rate": 0.49166666666666664,
16
+ "rate_per_seed": [
17
+ 0.5083333333333333,
18
+ 0.525,
19
+ 0.44166666666666665
20
+ ],
21
+ "regB_ref": 0.046037766193027574,
22
+ "regB_star": 0.055668722380055824,
23
+ "regA_ref": 0.05123443031809827,
24
+ "regA_star": 0.06242150229911836
25
+ },
26
+ "diff raw_b16": {
27
+ "rate": 0.7916666666666666,
28
+ "rate_per_seed": [
29
+ 0.7833333333333333,
30
+ 0.8,
31
+ 0.7916666666666666
32
+ ],
33
+ "regB_ref": 0.0392393348275164,
34
+ "regB_star": 0.04843269982130982,
35
+ "regA_ref": 0.053670339354187746,
36
+ "regA_star": 0.0653423576378871
37
+ },
38
+ "diff filt_b1": {
39
+ "rate": 0.4583333333333333,
40
+ "rate_per_seed": [
41
+ 0.45,
42
+ 0.475,
43
+ 0.45
44
+ ],
45
+ "regB_ref": 0.05194845381582125,
46
+ "regB_star": 0.06513679765481717,
47
+ "regA_ref": 0.05194845381582125,
48
+ "regA_star": 0.06513679765481717
49
+ },
50
+ "diff filt_b4": {
51
+ "rate": 0.8138888888888888,
52
+ "rate_per_seed": [
53
+ 0.8416666666666667,
54
+ 0.8083333333333333,
55
+ 0.7916666666666666
56
+ ],
57
+ "regB_ref": 0.044054119434560945,
58
+ "regB_star": 0.05117955925012052,
59
+ "regA_ref": 0.053670339354187746,
60
+ "regA_star": 0.06526846423539756
61
+ },
62
+ "diff filt_b16": {
63
+ "rate": 0.9083333333333333,
64
+ "rate_per_seed": [
65
+ 0.9416666666666667,
66
+ 0.9083333333333333,
67
+ 0.875
68
+ ],
69
+ "regB_ref": 0.033760278025647406,
70
+ "regB_star": 0.0392393348275164,
71
+ "regA_ref": 0.05494152341582326,
72
+ "regA_star": 0.06614503852117859
73
+ },
74
+ "diff ref_b16_n3": {
75
+ "rate": 0.9638888888888889,
76
+ "rate_per_seed": [
77
+ 0.9833333333333333,
78
+ 0.95,
79
+ 0.9583333333333334
80
+ ],
81
+ "regB_ref": 0.034359315053998474,
82
+ "regB_star": 0.039602163709273386,
83
+ "regA_ref": 0.056236553327202676,
84
+ "regA_star": 0.06713280900832053
85
+ },
86
+ "diff ref_b16_n10": {
87
+ "rate": 0.9805555555555556,
88
+ "rate_per_seed": [
89
+ 0.9916666666666667,
90
+ 0.975,
91
+ 0.975
92
+ ],
93
+ "regB_ref": 0.03437394402248328,
94
+ "regB_star": 0.040009974164940454,
95
+ "regA_ref": 0.05649656778792678,
96
+ "regA_star": 0.06751706328759521
97
+ },
98
+ "twin raw_b1": {
99
+ "rate": 0.24722222222222223,
100
+ "rate_per_seed": [
101
+ 0.2833333333333333,
102
+ 0.24166666666666667,
103
+ 0.21666666666666667
104
+ ],
105
+ "regB_ref": 0.04588446769434418,
106
+ "regB_star": 0.05281368503715744,
107
+ "regA_ref": 0.04588446769434418,
108
+ "regA_star": 0.05281368503715744
109
+ },
110
+ "twin raw_b4": {
111
+ "rate": 0.37777777777777777,
112
+ "rate_per_seed": [
113
+ 0.43333333333333335,
114
+ 0.3333333333333333,
115
+ 0.36666666666666664
116
+ ],
117
+ "regB_ref": 0.03958246856186798,
118
+ "regB_star": 0.046969980502976294,
119
+ "regA_ref": 0.04453505836815008,
120
+ "regA_star": 0.05310045978676048
121
+ },
122
+ "twin raw_b16": {
123
+ "rate": 0.4916666666666667,
124
+ "rate_per_seed": [
125
+ 0.5416666666666666,
126
+ 0.4083333333333333,
127
+ 0.525
128
+ ],
129
+ "regB_ref": 0.03635521260826646,
130
+ "regB_star": 0.046204610388273615,
131
+ "regA_ref": 0.04846464655537219,
132
+ "regA_star": 0.05641944870545168
133
+ },
134
+ "twin filt_b1": {
135
+ "rate": 0.3416666666666666,
136
+ "rate_per_seed": [
137
+ 0.36666666666666664,
138
+ 0.30833333333333335,
139
+ 0.35
140
+ ],
141
+ "regB_ref": 0.04659602299391885,
142
+ "regB_star": 0.053387234536363515,
143
+ "regA_ref": 0.04659602299391885,
144
+ "regA_star": 0.053387234536363515
145
+ },
146
+ "twin filt_b4": {
147
+ "rate": 0.4833333333333334,
148
+ "rate_per_seed": [
149
+ 0.5583333333333333,
150
+ 0.4166666666666667,
151
+ 0.475
152
+ ],
153
+ "regB_ref": 0.04074265857517956,
154
+ "regB_star": 0.048658098886397605,
155
+ "regA_ref": 0.046240245344131514,
156
+ "regA_star": 0.056035186950899485
157
+ },
158
+ "twin filt_b16": {
159
+ "rate": 0.5555555555555556,
160
+ "rate_per_seed": [
161
+ 0.5833333333333334,
162
+ 0.5083333333333333,
163
+ 0.575
164
+ ],
165
+ "regB_ref": 0.03570862209728387,
166
+ "regB_star": 0.04591678068291327,
167
+ "regA_ref": 0.05002032485737595,
168
+ "regA_star": 0.057383941917565906
169
+ },
170
+ "knn raw_b1": {
171
+ "rate": 0.09166666666666666,
172
+ "rate_per_seed": [
173
+ 0.09166666666666666
174
+ ],
175
+ "regB_ref": 0.18029208813957154,
176
+ "regB_star": 0.18029208813957154,
177
+ "regA_ref": 0.18029208813957154,
178
+ "regA_star": 0.18029208813957154
179
+ },
180
+ "knn raw_b4": {
181
+ "rate": 0.24166666666666667,
182
+ "rate_per_seed": [
183
+ 0.24166666666666667
184
+ ],
185
+ "regB_ref": 0.16093589699487887,
186
+ "regB_star": 0.17212171903802753,
187
+ "regA_ref": 0.1725441277384172,
188
+ "regA_star": 0.18523628247042523
189
+ },
190
+ "knn raw_b16": {
191
+ "rate": 0.5166666666666667,
192
+ "rate_per_seed": [
193
+ 0.5166666666666667
194
+ ],
195
+ "regB_ref": 0.1327768182319501,
196
+ "regB_star": 0.14350847619758267,
197
+ "regA_ref": 0.14856354038966635,
198
+ "regA_star": 0.16313087614552568
199
+ },
200
+ "knn filt_b1": {
201
+ "rate": 0.3,
202
+ "rate_per_seed": [
203
+ 0.3
204
+ ],
205
+ "regB_ref": 0.17428865396751858,
206
+ "regB_star": 0.18276418530499838,
207
+ "regA_ref": 0.17428865396751858,
208
+ "regA_star": 0.18276418530499838
209
+ },
210
+ "knn filt_b4": {
211
+ "rate": 0.6666666666666666,
212
+ "rate_per_seed": [
213
+ 0.6666666666666666
214
+ ],
215
+ "regB_ref": 0.12985794400765538,
216
+ "regB_star": 0.13483477262840815,
217
+ "regA_ref": 0.1620537009612174,
218
+ "regA_star": 0.16564434998790312
219
+ },
220
+ "knn filt_b16": {
221
+ "rate": 0.725,
222
+ "rate_per_seed": [
223
+ 0.725
224
+ ],
225
+ "regB_ref": 0.10686761298831438,
226
+ "regB_star": 0.11477117848557294,
227
+ "regA_ref": 0.16317150492755594,
228
+ "regA_star": 0.16567695056159626
229
+ },
230
+ "knn ref_b16_n3": {
231
+ "rate": 0.9,
232
+ "rate_per_seed": [
233
+ 0.9
234
+ ],
235
+ "regB_ref": 0.10400741103859301,
236
+ "regB_star": 0.1113356454533998,
237
+ "regA_ref": 0.13930659114267874,
238
+ "regA_star": 0.15300463877630077
239
+ },
240
+ "knn ref_b16_n10": {
241
+ "rate": 0.925,
242
+ "rate_per_seed": [
243
+ 0.925
244
+ ],
245
+ "regB_ref": 0.10343356710695706,
246
+ "regB_star": 0.10973883506945237,
247
+ "regA_ref": 0.13558071904802294,
248
+ "regA_star": 0.14891194803545904
249
+ }
250
+ }
results/icml_r3/X29/act/x27rescore_d3fm5n16000.txt ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ X29-A: X27's C-only arms re-scored, m = 5, tag d3fm5n16000, 120 test plants. J*_A = min(J_ref, 3A id and so3 labels) is below J_ref on 43 plants (median (J_ref - J*_A) / J*_A 0.000%).
2
+ diff raw_b1 certified 23.6% (published 23.6%) | cheapest certified: regret vs J_ref 5.40% (published 5.40%), vs J*_A 6.58% | first certified: vs J_ref 5.40%, vs J*_A 6.58%
3
+ diff raw_b4 certified 49.2% (published 49.2%) | cheapest certified: regret vs J_ref 4.60% (published 4.60%), vs J*_A 5.57% | first certified: vs J_ref 5.12%, vs J*_A 6.24%
4
+ diff raw_b16 certified 79.2% (published 79.2%) | cheapest certified: regret vs J_ref 3.92% (published 3.92%), vs J*_A 4.84% | first certified: vs J_ref 5.37%, vs J*_A 6.53%
5
+ diff filt_b1 certified 45.8% (published 45.8%) | cheapest certified: regret vs J_ref 5.19% (published 5.19%), vs J*_A 6.51% | first certified: vs J_ref 5.19%, vs J*_A 6.51%
6
+ diff filt_b4 certified 81.4% (published 81.4%) | cheapest certified: regret vs J_ref 4.41% (published 4.41%), vs J*_A 5.12% | first certified: vs J_ref 5.37%, vs J*_A 6.53%
7
+ diff filt_b16 certified 90.8% (published 90.8%) | cheapest certified: regret vs J_ref 3.38% (published 3.38%), vs J*_A 3.92% | first certified: vs J_ref 5.49%, vs J*_A 6.61%
8
+ diff ref_b16_n3 certified 96.4% (published 96.4%) | cheapest certified: regret vs J_ref 3.44% (published 3.44%), vs J*_A 3.96% | first certified: vs J_ref 5.62%, vs J*_A 6.71%
9
+ diff ref_b16_n10 certified 98.1% (published 98.1%) | cheapest certified: regret vs J_ref 3.44% (published 3.44%), vs J*_A 4.00% | first certified: vs J_ref 5.65%, vs J*_A 6.75%
10
+ twin raw_b1 certified 24.7% (published 24.7%) | cheapest certified: regret vs J_ref 4.59% (published 4.59%), vs J*_A 5.28% | first certified: vs J_ref 4.59%, vs J*_A 5.28%
11
+ twin raw_b4 certified 37.8% (published 37.8%) | cheapest certified: regret vs J_ref 3.96% (published 3.96%), vs J*_A 4.70% | first certified: vs J_ref 4.45%, vs J*_A 5.31%
12
+ twin raw_b16 certified 49.2% (published 49.2%) | cheapest certified: regret vs J_ref 3.64% (published 3.64%), vs J*_A 4.62% | first certified: vs J_ref 4.85%, vs J*_A 5.64%
13
+ twin filt_b1 certified 34.2% (published 34.2%) | cheapest certified: regret vs J_ref 4.66% (published 4.66%), vs J*_A 5.34% | first certified: vs J_ref 4.66%, vs J*_A 5.34%
14
+ twin filt_b4 certified 48.3% (published 48.3%) | cheapest certified: regret vs J_ref 4.07% (published 4.07%), vs J*_A 4.87% | first certified: vs J_ref 4.62%, vs J*_A 5.60%
15
+ twin filt_b16 certified 55.6% (published 55.6%) | cheapest certified: regret vs J_ref 3.57% (published 3.57%), vs J*_A 4.59% | first certified: vs J_ref 5.00%, vs J*_A 5.74%
16
+ knn raw_b1 certified 9.2% (published 9.2%) | cheapest certified: regret vs J_ref 18.03% (published 18.03%), vs J*_A 18.03% | first certified: vs J_ref 18.03%, vs J*_A 18.03%
17
+ knn raw_b4 certified 24.2% (published 24.2%) | cheapest certified: regret vs J_ref 16.09% (published 16.09%), vs J*_A 17.21% | first certified: vs J_ref 17.25%, vs J*_A 18.52%
18
+ knn raw_b16 certified 51.7% (published 51.7%) | cheapest certified: regret vs J_ref 13.28% (published 13.28%), vs J*_A 14.35% | first certified: vs J_ref 14.86%, vs J*_A 16.31%
19
+ knn filt_b1 certified 30.0% (published 30.0%) | cheapest certified: regret vs J_ref 17.43% (published 17.43%), vs J*_A 18.28% | first certified: vs J_ref 17.43%, vs J*_A 18.28%
20
+ knn filt_b4 certified 66.7% (published 66.7%) | cheapest certified: regret vs J_ref 12.99% (published 12.99%), vs J*_A 13.48% | first certified: vs J_ref 16.21%, vs J*_A 16.56%
21
+ knn filt_b16 certified 72.5% (published 72.5%) | cheapest certified: regret vs J_ref 10.69% (published 10.69%), vs J*_A 11.48% | first certified: vs J_ref 16.32%, vs J*_A 16.57%
22
+ knn ref_b16_n3 certified 90.0% (published 90.0%) | cheapest certified: regret vs J_ref 10.40% (published 10.40%), vs J*_A 11.13% | first certified: vs J_ref 13.93%, vs J*_A 15.30%
23
+ knn ref_b16_n10 certified 92.5% (published 92.5%) | cheapest certified: regret vs J_ref 10.34% (published 10.34%), vs J*_A 10.97% | first certified: vs J_ref 13.56%, vs J*_A 14.89%
results/icml_r3/X29/act/x27rows_d3fm3n8000.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:624835be5f563c538921ea962a6c0dc7745508a99389b8c852050a3c50af1cdb
3
+ size 565807
results/icml_r3/X29/act/x27rows_d3fm4n8000.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c2eccb0a2ed06d5b6e29839ff88993bd8f9a6e6a722eb310c1b20af4e34ffd6c
3
+ size 536311
results/icml_r3/X29/act/x27rows_d3fm5n16000.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3c0f5b146a5d508573e732c9d6f1e3e2cdf38ca4902f234d05dd938e97900449
3
+ size 523822