matCercola18 commited on
Commit
374b6ec
·
verified ·
1 Parent(s): 8f92ad1

X29: SISO CPU and compute ratios, X27 re-score in the X29 convention, write-up

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. docs/icml_r3_x29.md +198 -0
  2. results/icml_r3/X29/act/x27rescore_d3fm3n8000.json +116 -22
  3. results/icml_r3/X29/act/x27rescore_d3fm3n8000.txt +22 -22
  4. results/icml_r3/X29/act/x27rescore_d3fm4n8000.json +116 -22
  5. results/icml_r3/X29/act/x27rescore_d3fm4n8000.txt +22 -22
  6. results/icml_r3/X29/act/x27rescore_d3fm5n16000.json +116 -22
  7. results/icml_r3/X29/act/x27rescore_d3fm5n16000.txt +22 -22
  8. results/icml_r3/X29/siso/report.txt +22 -1
  9. results/icml_r3/X29/siso/timing/plant_000_fopt1.json +1 -0
  10. results/icml_r3/X29/siso/timing/plant_000_fopt16.json +1 -0
  11. results/icml_r3/X29/siso/timing/plant_000_fopt4.json +1 -0
  12. results/icml_r3/X29/siso/timing/plant_000_gen_C+phi.json +1 -0
  13. results/icml_r3/X29/siso/timing/plant_000_gen_C-only.json +1 -0
  14. results/icml_r3/X29/siso/timing/plant_000_hifoo2.json +1 -0
  15. results/icml_r3/X29/siso/timing/plant_000_jopt1.json +1 -0
  16. results/icml_r3/X29/siso/timing/plant_000_jopt16.json +1 -0
  17. results/icml_r3/X29/siso/timing/plant_000_jopt4.json +1 -0
  18. results/icml_r3/X29/siso/timing/plant_000_knn_C+phi.json +1 -0
  19. results/icml_r3/X29/siso/timing/plant_000_knn_C-only.json +1 -0
  20. results/icml_r3/X29/siso/timing/plant_000_twin_C+phi.json +1 -0
  21. results/icml_r3/X29/siso/timing/plant_000_twin_C-only.json +1 -0
  22. results/icml_r3/X29/siso/timing/plant_021_fopt1.json +1 -0
  23. results/icml_r3/X29/siso/timing/plant_021_fopt16.json +1 -0
  24. results/icml_r3/X29/siso/timing/plant_021_fopt4.json +1 -0
  25. results/icml_r3/X29/siso/timing/plant_021_gen_C+phi.json +1 -0
  26. results/icml_r3/X29/siso/timing/plant_021_gen_C-only.json +1 -0
  27. results/icml_r3/X29/siso/timing/plant_021_hifoo2.json +1 -0
  28. results/icml_r3/X29/siso/timing/plant_021_jopt1.json +1 -0
  29. results/icml_r3/X29/siso/timing/plant_021_jopt16.json +1 -0
  30. results/icml_r3/X29/siso/timing/plant_021_jopt4.json +1 -0
  31. results/icml_r3/X29/siso/timing/plant_021_knn_C+phi.json +1 -0
  32. results/icml_r3/X29/siso/timing/plant_021_knn_C-only.json +1 -0
  33. results/icml_r3/X29/siso/timing/plant_021_twin_C+phi.json +1 -0
  34. results/icml_r3/X29/siso/timing/plant_021_twin_C-only.json +1 -0
  35. results/icml_r3/X29/siso/timing/plant_041_fopt1.json +1 -0
  36. results/icml_r3/X29/siso/timing/plant_041_fopt16.json +1 -0
  37. results/icml_r3/X29/siso/timing/plant_041_fopt4.json +1 -0
  38. results/icml_r3/X29/siso/timing/plant_041_gen_C+phi.json +1 -0
  39. results/icml_r3/X29/siso/timing/plant_041_gen_C-only.json +1 -0
  40. results/icml_r3/X29/siso/timing/plant_041_hifoo2.json +1 -0
  41. results/icml_r3/X29/siso/timing/plant_041_jopt1.json +1 -0
  42. results/icml_r3/X29/siso/timing/plant_041_jopt16.json +1 -0
  43. results/icml_r3/X29/siso/timing/plant_041_jopt4.json +1 -0
  44. results/icml_r3/X29/siso/timing/plant_041_knn_C+phi.json +1 -0
  45. results/icml_r3/X29/siso/timing/plant_041_knn_C-only.json +1 -0
  46. results/icml_r3/X29/siso/timing/plant_041_twin_C+phi.json +1 -0
  47. results/icml_r3/X29/siso/timing/plant_041_twin_C-only.json +1 -0
  48. results/icml_r3/X29/siso/timing/plant_061_fopt1.json +1 -0
  49. results/icml_r3/X29/siso/timing/plant_061_fopt16.json +1 -0
  50. results/icml_r3/X29/siso/timing/plant_061_fopt4.json +1 -0
docs/icml_r3_x29.md ADDED
@@ -0,0 +1,198 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ title: C+phi next to C-only with the same baselines and KPIs (X29). SISO, the two generators tie at 16 samples on the common J* and both have lower median regret than kNN, the regression twin and the flat references. MIMO spline setting (200 fresh plants per m), the repaired C+phi generator has the lowest median regret of the generators and baselines at 16 samples at both m, and C-only certifies more at 1 sample. Actuator setting, at the per-plant optimum C+phi equals C-only (median gain 0.00% at m = 3, 4, 5)
3
+ date: 2026-11-17
4
+ status: SISO (with CPU), MIMO spline setting (with CPU) and actuator step 3A complete; actuator step 3B (C+phi generators on X27's banks) not run, pending Matteo's decision
5
+ claim: X29 (Matteo 2026-11-16, via dc_presentation and in the move-HPC session): "I need to have the final baseline comparison also for the spline methods". C+phi (the model generates the linear part and the spline phi, X28) gets the same baselines and KPIs as C-only (phi fixed: SISO sat_1, MIMO identity) in every setting of the deck v10 slides.
6
+ maps_to: deck v10 slides [35] [36] [37] [38] [73] (SISO); the three new MIMO spline-setting slides and [73]; [50] [51] [52] [84] [73] (actuator); summaries [54] [72] [77]. No paper edit until Matteo agrees the claim.
7
+ scripts: nonlinear/experiments/x29_siso.py (SISO kNN, twins, optimizer from the best sample, timing, report), nonlinear/mimo_unstable/x29_t2.py (MIMO spline setting on T2: kNN, twins, 48-start solver, Adam references, timing, report), nonlinear/mimo_unstable/x29_act.py (actuator 3A: composed-sector certificate, simulated X27 cost, optimizer, new directions, report, X27 rows and re-score)
8
+ artifacts: results/icml_r3/X29/{PREREG.md, PREREG_siso.md, PREREG_t2.md}; results/icml_r3/X29/siso/{report.txt, eqcheck.txt, eval/, refine/, timing/, timing_batched.json}; results/icml_r3/X29/t2/{report.txt, numbers.json, cpu_report.txt, timing_m2.json, timing_m3.json, knn_m*_{a,b}.npz, twin_*.npz, solver_m*.npz, ard_m2.npz}; results/icml_r3/X29/act/{selfcheck_*.json, report_3a_heldout_m*.txt, heldout_m*/, x27rows_*.npy, x27rescore_*.txt, x27rescore_*.json}; models/x29_twin_clS{,_s1,_s2}/, models/x29t/; HF matCercola18/DiffusionControl commits 8f92ad1d (SISO), d7d0258 (MIMO spline setting), 09f35d5 (actuator 3A)
9
+ ---
10
+
11
+ # C+phi next to C-only with the same baselines and KPIs (X29)
12
+
13
+ ## TL;DR
14
+ - **Question:** The deck compares C-only (phi fixed; the model generates the linear part) with kNN, regression twins, per-plant optimizers and classical references. Matteo asked for the same comparison for C+phi (the model also generates the spline phi, X28) in every setting, on one regret reference per setting.
15
+ - **SISO (199 test plants):** at 16 samples the C+phi generator certifies 95.6% of plants with median regret 6.59%, and C-only 98.0% with 6.74% (3-seed means). Both have lower regret than their kNN (11.3%, 10.2%), their regression twins (31.8%, 20.1%), the tuned LQG or static gain (34.05%) and the best of 16 HIFOO starts (50.75%). At 1 sample C+phi certifies 50.6% of plants against 64.0%. The joint optimizer reaches the C+phi generator's regret with 2 starts; the F-only optimizer needs 8 to reach C-only's. On one CPU thread the C+phi generator costs 29.1 s per plant at 16 samples against 11.5 s (same sampling time, slower scoring), and the variant's own optimizer at the start count that reaches the generator's regret costs 106 times the C+phi generator's CPU and 138 times the C-only generator's (October: 140 times).
16
+ - **MIMO spline setting (200 fresh plants per m):** at 16 samples the repaired C+phi generator has median regret 6.56% at m = 3 and 3.59% at m = 2, against 10.22% and 6.42% for C-only, with certified shares within 0.5 points. Its kNN and twin also have lower regret than C-only's. The 48-start solver has 11.60% and 12.33%. At 1 sample C-only certifies more plants (95.7% against 79.7% at m = 3). Each generator costs about 3 s per plant on one CPU thread and about 0.05 s per plant batched on a GPU; the per-plant Adam optimizer costs about 4,200 to 5,100 s and 34 to 41 s.
17
+ - **Actuator setting (X27's 120 test plants per m):** at each plant's optimum C+phi costs the same as C-only (median gain 0.00% at m = 3, 4 and 5), and the best certified spline is linear on most plants. So a C+phi generator can at best tie here, and step 3B (C+phi generators on X27's banks, 2 to 3 GPU-days) waits for Matteo. X27's C-only pipeline, re-run on its stored candidates, reproduces every published number; on the common J*_A the regret of its filtered arms at 16 samples rises by 0.5 to 1.2 points in X27's convention (median over certified plants).
18
+ - **So what:** C+phi changes the comparison only in the MIMO spline setting, where it has lower regret than C-only for every method family at 16 samples. On SISO it ties, and in the actuator setting it has nothing to gain at the optimum. With 1 sample, C-only certifies more plants in SISO and in the MIMO spline setting.
19
+
20
+ ## 1. Problem / Goal
21
+ - **C-only:** SISO u = -sat_1(clip(F y, +-5)) with a generated dynamic filter F (X21); MIMO a generated gain K with phi = identity, in the X28-M spline setting (u = -K y) and in X27's actuator setting (u = -Phi(K y), Phi unknown in a sector).
22
+ - **C+phi:** the same model also generates phi. SISO: an 11-knot spline with |phi| <= 1, phi(0) = 0 (X28 S3). MIMO: per channel an odd piecewise-linear spline with knots at |x| = 1, 3, 5 and knot ratios in [0.1, 10] (the so3 class, X28-M).
23
+ - **Goal:** every deck KPI (certified share and median regret vs budget b, CPU vs regret, compute ratio, generator minus kNN at b = 1) for both variants, on one reference J* per setting, with the same baselines. Nothing published is replaced; C-only numbers are re-scored on the common J*.
24
+ - **No new verdict rule** in SISO and the MIMO spline setting: the registered X28 verdicts stand. The actuator setting had no C+phi yet; its first step (3A) measures whether C+phi can gain at all.
25
+
26
+ ## 2. Method
27
+ Pre-registration: `results/icml_r3/X29/PREREG.md` (2026-11-16 18:30 UTC, before any X29 code ran) and its addenda in `PREREG.md`, `PREREG_siso.md` and `PREREG_t2.md`, each filed before the number it affects.
28
+
29
+ - **Regret** at budget b: (J_b - J*) / J*, with J_b the cheapest certified of the first b candidates. A plant with no certified candidate counts 1e3. A cell where fewer than half the plants certify has no finite median ("n/a"). Median with a bootstrap 95% CI over plants (10,000 resamples). Seeded methods: 3-seed mean and range, plus seed 0.
30
+ - **J\* per setting:** SISO min(J_Fonly, X28 test joint labels) as X28 S4; MIMO spline setting min of the T2 id and so3 labels (gauges c = 1, 2, 5); actuator min(J_ref, J_id, J_so3) from step 3A.
31
+ - **SISO arms** (199 X21 test plants, clip 1.5, t-repair on every arm):
32
+ - generators: clL (C-only) and clS (C+phi), seeds 0 to 2, stored X28 evaluations; 64 samples for seed 0;
33
+ - kNN: X22's rule on the X21 F-only bank (C-only) and on the c = 1 joint rows of the 1,971 training plants (C+phi), ARD metric of X21;
34
+ - regression twins: X23's twins of clL (stored X21 outputs, see deviations) and new twins of clS (X23 recipe, 60,000 steps, best validation checkpoint), seeds 0 to 2;
35
+ - per-plant optimizers: F-only, best of the first k X21 labels (C-only's own); joint, best of the first k joint rows (C+phi's own); k = 1, 2, 4, 8, 16;
36
+ - optimizer from the generator's best b = 16 sample: 1 F-only start (X24 refine) for C-only, 1 joint start (S2 warm arm) for C+phi, seed 0;
37
+ - flat references: tuned saturated LQG or static gain (J_lin), best of 16 HIFOO starts.
38
+ - **MIMO spline setting** (T2: 200 fresh plants per m, m = 2, 3, as the X28 addendum of 2026-11-15):
39
+ - generators: C-only (a) and C+phi (b), seeds 0 to 2, stored; C+phi repaired (repair R: rejected samples' knots flattened toward r(1) until the circle LMI certifies) is primary, as drawn is secondary;
40
+ - kNN: label (a) or (b) of the 16 nearest S2-M bank plants, ARD metric (m = 3 X10's, m = 2 fitted on the m = 2 bank before any kNN number);
41
+ - regression twins: `mh_train` arm "reg" on the S3-M datasets (a) and (b), seeds 0 to 2;
42
+ - flat references: the 48-start solver's best gain (X27 oracle, Lyapunov test, X28 cost) and the per-plant Adam optimizer of each variant (C-only: id start; C+phi: J*_T2 itself, regret 0 by construction).
43
+ - **Actuator setting, step 3A** (X27's 120 test plants per m, m = 3, 4, 5):
44
+ - controller u = -Phi(phi(K y)), with Phi the unknown actuator (each channel in the sector [0.4, 2.2], X27) and phi in the so3 class; C-only is phi = identity;
45
+ - certificate: circle LMI on the composed sector [0.4 alpha_i, 2.2 beta_i], alpha_i and beta_i the min and max knot ratio of channel i. It is sound because each channel's composition Phi_i(psi_i(.)) is memoryless and lies in that sector; at identity knots it is X27's test;
46
+ - cost: X27's quadratic cost (mean over the 2m unit state directions of the integral of |x|^2 + 0.1 |u|^2, Phi = identity) by simulation, RK4 at dt 0.01, T = 20 s;
47
+ - label optimizer: X28-M's procedure from X27's stored best gain K_ref; arm id (knots frozen) and arm so3 at gauges c = 1, 2, 5; K lr and knot lr 0.02; 400 steps; the cheapest certified snapshot;
48
+ - reported: d = (J_id - J_so3) / J_so3; (J_ref - J_id) / J_id (extra optimization alone); the composed-sector ratio of the best so3 controller; d on 64 new random unit state directions per plant;
49
+ - X27's stored C-only candidates re-run through X27's pipeline per plant and re-scored on J*_A.
50
+ - **CPU:** one timing job per setting on a node with nothing else of this project running. "One" = one plant on one CPU thread, median over 10 timing plants (SISO: 6 such tasks at a time on 8 cores; MIMO: one at a time).
51
+ - MIMO (node move-hpc-08-gpu): "batched" = all 200 plants in one call, with sampling, rollouts and Adam on the node's NVIDIA L40S GPU and certificates on 6 CPU workers, wall time / 200.
52
+ - SISO (node move-hpc-06-gpu, also an L40S): "batched" = generator and twin sampling (199 plants x 16) and kNN retrieval in one call, wall time / 199, without scoring (PREREG section 2).
53
+ - Per-point CPU follows X24's formulas; the joint optimizer at k starts is charged k HIFOO starts + the F-only optimizer at k + the joint optimizer at k.
54
+ - **Self-checks** (two failed and were handled by addenda before the numbers they affect; see deviations):
55
+ - SISO: C-only kNN reproduces X21's stored kNN (K16 +0.16 points per candidate, +0.50 plants; K64 +0.07, +0.00); batched optimizers equal the single-plant functions (relative J difference 0 at 25 and 100 steps);
56
+ - MIMO spline setting: 5 bank plants per m retrieve themselves first; the solver's cost equals the stored label snapshot (maximum relative difference 0);
57
+ - actuator: the composed-sector certificate at identity knots agrees with `family_sofm.verify` on 200 gains per m (0 disagreements at m = 3, 4, 5); the simulated cost equals tr(P)/(2m) on the 120 stored gains per m (median relative difference 1.3e-5 to 2.2e-5, maximum 2.6e-4 to 1.4e-3).
58
+
59
+ ## 3. Results
60
+
61
+ ### 3.1 SISO, 199 test plants
62
+ Source: `results/icml_r3/X29/siso/report.txt`. Certified % / median regret %; generator and twin: 3-seed mean [range]; kNN: one run; b = 64: seed 0.
63
+
64
+ | method | b = 1 | b = 4 | b = 16 | b = 64 |
65
+ |---|---|---|---|---|
66
+ | C-only generator | 64.0 / 49.0 | 92.0 / 12.7 | 98.0 [97.5, 98.5] / 6.74 [6.62, 6.81] | 98.5 / 4.07 |
67
+ | C+phi generator | 50.6 / n/a on 2 of 3 seeds | 84.9 / 17.1 | 95.6 [95.0, 96.5] / 6.59 [6.30, 7.14] | 99.0 / 2.96 |
68
+ | C-only kNN | 57.3 / 81.3 | 86.9 / 19.4 | 96.5 / 10.19 | 98.5 / 6.49 |
69
+ | C+phi kNN | 46.2 / n/a | 77.9 / 22.7 | 93.5 / 11.32 | 98.0 / 6.26 |
70
+ | C-only twin | 68.5 / 48.1 | 77.2 / 30.3 | 85.6 [85.4, 85.9] / 20.1 [19.9, 20.3] | |
71
+ | C+phi twin | 42.2 / n/a | 61.8 / 66.1 | 71.0 [68.8, 72.9] / 31.8 [30.0, 32.9] | |
72
+ | LQG or static gain | | | 100 / 34.05 | |
73
+ | HIFOO, best of 16 | | | 100 / 50.75 | |
74
+
75
+ Per-plant optimizers, median regret % at k = 1, 2, 4, 8, 16 starts (certified 100%):
76
+ - F-only (C-only's own): 35.91, 13.55, 8.11, 4.19, 3.12.
77
+ - Joint (C+phi's own): 9.13, 4.39, 0.92, 0.00, 0.00. The joint labels define J*, so 0 at 8 and 16 starts is by construction.
78
+ - Generator at 16 plus 1 optimizer start (seed 0): C-only 98.5% / 2.60%, C+phi 95.0% / 0.01%. Only plants with a certified b = 16 sample get a start (196 C-only, 189 C+phi); the rest count as uncertified.
79
+
80
+ Compute ratio inputs ([38], seed 0): the C-only generator's 6.79% is first reached by the F-only optimizer at k = 8 (4.19%; k = 4 gives 8.11%). The C+phi generator's 7.14% is first reached by the joint optimizer at k = 2 (4.39%; k = 1 gives 9.13%). The paired verdict at b = 16 stays X28 S4's: no measurable difference on 3 of 3 seed pairs (gain -0.32 to +0.19 points).
81
+
82
+ [73], certified at b = 1, generator minus its own variant's kNN (points, plant-paired, 95% CI), seeds 0 / 1 / 2:
83
+ - C-only: +8.0 [-0.5, +16.6] / +8.0 [-0.5, +16.6] / +4.0 [-4.0, +12.1].
84
+ - C+phi: +3.0 [-6.0, +12.1] / +6.5 [-3.0, +16.1] / +3.5 [-5.5, +12.6].
85
+ - All six intervals include 0.
86
+
87
+ ### 3.2 MIMO spline setting, 200 fresh plants per m
88
+ Source: `results/icml_r3/X29/t2/report.txt`. Certified % / median regret %, 3-seed means (kNN one run).
89
+
90
+ | method | m = 3, b = 1 | m = 3, b = 16 | m = 2, b = 1 | m = 2, b = 16 |
91
+ |---|---|---|---|---|
92
+ | C+phi generator, repaired | 79.7 / 17.73 | 99.8 / 6.56 | 90.5 / 10.17 | 99.7 / 3.59 |
93
+ | C+phi generator, as drawn | 61.5 / 22.85 | 95.3 / 6.60 | 78.0 / 10.68 | 97.2 / 3.65 |
94
+ | C-only generator | 95.7 / 16.32 | 99.5 / 10.22 | 98.5 / 8.96 | 99.8 / 6.42 |
95
+ | C+phi kNN, repaired | 90.5 / 61.40 | 100 / 15.88 | 88.0 / 34.85 | 100 / 8.89 |
96
+ | C+phi kNN, as drawn | 48.0 / n/a | 99.5 / 17.55 | 60.0 / 76.27 | 100 / 9.83 |
97
+ | C-only kNN | 80.0 / 70.87 | 100 / 23.77 | 89.5 / 42.67 | 100 / 13.56 |
98
+ | C+phi twin, repaired | 76.3 / 22.16 | 95.0 / 10.84 | 88.3 / 12.14 | 98.0 / 6.23 |
99
+ | C+phi twin, as drawn | 60.3 / 28.50 | 83.2 / 11.26 | 77.7 / 12.74 | 90.8 / 6.31 |
100
+ | C-only twin | 94.2 / 18.21 | 98.7 / 13.81 | 98.7 / 9.22 | 99.7 / 7.71 |
101
+ | 48-start solver | | 100 / 11.60 | | 100 / 12.33 |
102
+ | Adam, C-only (id start) | | 100 / 4.33 | | 100 / 5.07 |
103
+ | Adam, C+phi (id + so3 c = 1, 2, 5) | | 100 / 0 (defines J*) | | 100 / 0 (defines J*) |
104
+
105
+ - At b = 4 (same file): repaired C+phi generator 93.8 / 9.70 (m = 3) and 97.8 / 5.47 (m = 2); C-only generator 99.2 / 12.73 and 99.3 / 7.30.
106
+ - [73], certified at b = 1, generator minus its own kNN (points, 95% CI), seeds 0 / 1 / 2:
107
+ - m = 3: C+phi repaired -10.0 [-17.0, -3.0] / -12.5 [-19.5, -5.5] / -10.0 [-16.5, -3.5]; as drawn +14.0 / +13.0 / +13.5 (CIs above 0); C-only +15.5 / +17.5 / +14.0 (CIs above 0).
108
+ - m = 2: C+phi repaired +3.0 [-3.0, +9.0] / +2.0 [-4.0, +8.0] / +2.5 [-3.0, +8.0]; as drawn +18.0 / +15.5 / +20.5 (CIs above 0); C-only +9.0 on all seeds (CIs above 0).
109
+ - With repair, the C+phi kNN certifies more plants at b = 1 than the C+phi generator at m = 3 (90.5% against 79.7%).
110
+
111
+ ### 3.3 Actuator setting, step 3A (per-plant optimum), 120 test plants per m
112
+ Source: `results/icml_r3/X29/act/report_3a_heldout_m{3,4,5}.txt`. All id and so3 starts certified on 118 to 120 of 120 plants.
113
+
114
+ | | m = 3 | m = 4 | m = 5 |
115
+ |---|---|---|---|
116
+ | d = (J_id - J_so3) / J_so3, median [CI] | 0.00% [0.00, 0.00] | 0.00% [0.00, 0.00] | 0.00% [0.00, 0.00] |
117
+ | plants with d >= 2% | 16 of 120 | 9 of 120 | 13 of 120 |
118
+ | mean d | 0.68% | 0.34% | 0.44% |
119
+ | extra optimization alone, (J_ref - J_id) / J_id, median | 0.00% | 0.00% | 0.00% |
120
+ | composed-sector ratio of the best so3 controller, median (90th pct); 5.5 = linear | 5.50 (15.97) | 5.50 (11.37) | 5.50 (12.18) |
121
+ | d on 64 new directions per plant, median of per-plant medians | +0.00% | +0.00% | +0.00% |
122
+
123
+ - The best so3 controller comes from the id start on 69, 78 and 76 of 120 plants: the spline starts found nothing cheaper there.
124
+ - The written expectation (PREREG section 4) was a small d: the composed sector is 5.5 beta/alpha wide, so a nonlinear spline needs a wider robust sector than a linear gain, and the quadratic cost does not reward an amplitude-dependent gain.
125
+
126
+ X27's C-only pipeline re-scored on J*_A (`results/icml_r3/X29/act/x27rescore_*.txt`). Re-running it on the stored candidates reproduces every published certified share and regret. Certified shares are unchanged. X27 publishes the median over certified plants (seeds pooled); in that convention, median regret %, published (vs J_ref) and on J*_A:
127
+
128
+ | X27 arm (keep the cheapest certified) | m = 3 | m = 4 | m = 5 (16k bank) |
129
+ |---|---|---|---|
130
+ | diffusion, filter, b = 16 | 1.50 -> 2.18 | 3.04 -> 3.73 | 3.38 -> 3.92 |
131
+ | diffusion, filter + 10 solver iterations | 1.50 -> 2.21 | 3.24 -> 3.85 | 3.44 -> 4.00 |
132
+ | twin, filter, b = 16 | 2.50 -> 3.39 | 3.46 -> 4.00 | 3.57 -> 4.59 |
133
+ | kNN, filter + 10 solver iterations | 7.34 -> 8.58 | 9.03 -> 9.68 | 10.34 -> 10.97 |
134
+
135
+ Stopping at the first certified candidate instead, the diffusion arm with iterations has 4.01 -> 5.15, 5.71 -> 6.40 and 5.65 -> 6.75.
136
+
137
+ In the X29 convention (all plants, uncertified = 1e3, median per seed, 3-seed mean), on J*_A, cheapest certified at b = 16, certified % / median regret %:
138
+
139
+ | X27 arm | m = 3 | m = 4 | m = 5 |
140
+ |---|---|---|---|
141
+ | diffusion, filter + 10 solver iterations | 99.7 / 2.22 | 97.8 / 3.96 | 98.1 / 4.04 |
142
+ | diffusion, filter | 98.6 / 2.24 | 91.9 / 4.05 | 90.8 / 4.30 |
143
+ | twin, filter | 73.1 / 5.25 | 52.2 / 12.06 | 55.6 / 9.61 |
144
+ | kNN, filter + 10 solver iterations (one run) | 93.3 / 8.84 | 93.3 / 10.01 | 92.5 / 11.51 |
145
+
146
+ The twin's low certified share counts against it here, unlike in the published convention.
147
+
148
+ ### 3.4 CPU
149
+ MIMO spline setting (`results/icml_r3/X29/t2/cpu_report.txt`; seconds per plant, one | batched; median regret on J*_T2):
150
+
151
+ | method | m = 3 | m = 2 |
152
+ |---|---|---|
153
+ | C+phi generator, b = 16 | 2.96 \| 0.055; 6.56% | 2.85 \| 0.054; 3.59% |
154
+ | C-only generator, b = 16 | 2.88 \| 0.050; 10.22% | 2.79 \| 0.049; 6.42% |
155
+ | C+phi kNN, b = 16 | 1.34 \| 0.054; 15.88% | 1.27 \| 0.053; 8.89% |
156
+ | C-only kNN, b = 16 | 1.25 \| 0.042; 23.77% | 1.18 \| 0.043; 13.56% |
157
+ | C+phi twin, b = 16 | 1.39 \| 0.050; 10.84% | 1.28 \| 0.051; 6.23% |
158
+ | C-only twin, b = 16 | 1.35 \| 0.045; 13.81% | 1.24 \| 0.044; 7.71% |
159
+ | 48-start solver, all starts | 6.01 \| not batched; 11.60% | 7.38 \| not batched; 12.33% |
160
+ | Adam, C+phi (4 starts, 400 steps + certify) | 5,131 \| 41.4; 0% | 5,102 \| 41.4; 0% |
161
+ | Adam, C-only (id start) | 4,177 \| 34.6; 4.33% | 4,170 \| 34.1; 5.07% |
162
+
163
+ At b = 1 a generator costs 1/16 of its b = 16 cost (X24's convention): 0.18 to 0.19 s one at a time.
164
+
165
+ SISO (`results/icml_r3/X29/siso/report.txt`, CPU section; seconds per plant on one CPU thread, median over the 10 timing plants; median regret on J*, seed 0):
166
+
167
+ | point | CPU | median regret |
168
+ |---|---|---|
169
+ | C-only generator, b = 1 / b = 16 | 0.7 s / 11.5 s | 54.90% / 6.79% |
170
+ | C+phi generator, b = 1 / b = 16 | 1.8 s / 29.1 s | n/a / 7.14% |
171
+ | F-only optimizer, 1 / 2 / 4 / 8 / 16 starts | 1,555 / 1,544 / 1,522 / 1,582 / 1,702 s | 35.91 / 13.55 / 8.11 / 4.19 / 3.12% |
172
+ | joint optimizer, 1 / 2 / 4 / 8 / 16 starts | 3,092 / 3,084 / 3,068 / 3,196 / 3,451 s | 9.13 / 4.39 / 0.92 / 0.00 / 0.00% |
173
+ | C-only generator 16 + 1 F-only start | 1,565 s | 2.60% |
174
+ | C+phi generator 16 + 1 joint start | 1,566 s | 0.01% |
175
+ | HIFOO, best of 16 | 15.3 s | 50.75% |
176
+ | C-only kNN / C+phi kNN, b = 16 | 9.5 s / 7.6 s | 10.19% / 11.32% |
177
+ | C-only twin / C+phi twin, b = 16 | 1.9 s / 17.2 s | 20.27% / 32.47% |
178
+
179
+ - **Compute ratio ([38]):** C-only, the F-only optimizer at 8 starts (1,582 s) against the generator's 11.5 s: 138 times. C+phi, the joint optimizer at 2 starts (3,084 s) against the generator's 29.1 s: 106 times.
180
+ - **Why the C+phi ratio is only somewhat smaller:** the joint optimizer needs 2 starts against 8, but the optimizer's cost is nearly flat in the number of starts (the k starts run in one vectorized call, X24's convention; the F-only optimizer at 1 start ranges from 1,503 to 1,610 s over the timing plants, wider than the differences between start counts). The joint optimizer is also charged the F-only optimizer it starts from, which doubles its cost. And the C+phi generator costs 2.5 times C-only's: sampling takes the same time (median 8.7 s for 16 samples), but scoring C+phi candidates takes a median 20.4 s against 2.8 s, from 10 to 110 s in total over the timing plants.
181
+ - **Batched on the GPU** (sampling only, 199 plants in one call, `timing_batched.json`): generator 0.019 s per plant for both variants, twin 0.003 s, kNN retrieval 0.013 s (C-only) and 0.005 s (C+phi).
182
+ - The optimizer tasks ran 6 at a time on 8 cores (load about 6); the MIMO timings ran one plant at a time (load 0.7 to 1.0). Ratios are within one setting, so this does not change them, but absolute seconds across the two settings are not measured under the same load.
183
+
184
+ ## 4. Conclusion
185
+ - **SISO:** C+phi and C-only generators tie at 16 samples on the common J*, as X28 S4 found, and both have lower median regret than kNN, the regression twins and the flat references. C+phi's own optimizer is stronger (2 joint starts reach the C+phi generator's regret; the F-only optimizer needs 8 to reach C-only's), but each optimizer costs about the same at any start count, so the compute ratio moves only from 138 times (C-only) to 106 times (C+phi). At 1 and 4 samples C+phi certifies fewer plants.
186
+ - **MIMO spline setting:** C+phi has lower median regret than C-only for each method family at 16 samples (difference of median regrets at m = 3 and m = 2: generator 3.7 and 2.8 points, kNN 7.9 and 4.7, twin 3.0 and 1.5), at equal certified share with repair. C-only certifies more at 1 sample. The C+phi generator at 16 samples has lower median regret than C-only's per-plant Adam optimizer at m = 2 (3.59% against 5.07%), not at m = 3 (6.56% against 4.33%), at about 1/1,500 of its CPU per plant on one thread.
187
+ - **Actuator setting:** generating phi gains nothing at the per-plant optimum, so the actuator slides can state "C+phi = C-only here" with the 3A numbers. Step 3B is not run unless Matteo asks for it.
188
+
189
+ **Deviations and caveats**
190
+ - SISO C-only twin rows are X21's stored outputs re-scored on J*. The re-sampled seed 0 missed its 1-point self-check on the certified share per candidate (+1.57) because the MC-dropout draws differ across machines; this was registered before any twin regret (`PREREG_siso.md`, 19:05 UTC). The re-sampled rows agree within 1 point at b = 16 (85.76% / 20.73%).
191
+ - The C+phi SISO twins reached their best validation loss early (step 4,000 of 60,000), so part of their gap to the generator may come from the training run.
192
+ - The actuator cost first used the trapezoid rule, which failed the tr(P) self-check at every horizon (median 0.25%); it was replaced by RK4 on an extra cost state before any 3A number (`PREREG.md` addendum 18:20 UTC).
193
+ - The joint optimizer and the C+phi Adam optimizer define J* in their settings, so their regret falls to 0 by construction.
194
+ - Regrets here are on each setting's common J* and are higher than earlier C-only numbers on J_Fonly (SISO) or J_ref (actuator); the two scales are not mixed.
195
+ - The MIMO spline J*_T2 uses id and so3 labels only, so T2 regrets are not on the scale of the held-out X28 regrets.
196
+ - The SISO and MIMO timings ran on two different nodes (move-hpc-06-gpu and move-hpc-08-gpu, 8 cores and an NVIDIA L40S each), each with nothing else of this project running. Ratios are computed within one setting's timing job.
197
+
198
+ **Sources:** the PREREG and addenda in `results/icml_r3/X29/`; the report files named in each section; HF commits 8f92ad1d (SISO), d7d0258 (MIMO spline setting) and 09f35d5 (actuator 3A).
results/icml_r3/X29/act/x27rescore_d3fm3n8000.json CHANGED
@@ -9,7 +9,12 @@
9
  "regB_ref": 0.035166553853986704,
10
  "regB_star": 0.04939973422469457,
11
  "regA_ref": 0.035166553853986704,
12
- "regA_star": 0.04939973422469457
 
 
 
 
 
13
  },
14
  "diff raw_b4": {
15
  "rate": 0.7611111111111111,
@@ -21,7 +26,12 @@
21
  "regB_ref": 0.026566873792452816,
22
  "regB_star": 0.036538855499143374,
23
  "regA_ref": 0.03912745224114178,
24
- "regA_star": 0.05271375933165312
 
 
 
 
 
25
  },
26
  "diff raw_b16": {
27
  "rate": 0.9444444444444443,
@@ -33,7 +43,12 @@
33
  "regB_ref": 0.017749766148115964,
34
  "regB_star": 0.02536120788249141,
35
  "regA_ref": 0.03975266005687195,
36
- "regA_star": 0.051211998589972385
 
 
 
 
 
37
  },
38
  "diff filt_b1": {
39
  "rate": 0.6861111111111112,
@@ -45,7 +60,12 @@
45
  "regB_ref": 0.03864968435767513,
46
  "regB_star": 0.051490162113829285,
47
  "regA_ref": 0.03864968435767513,
48
- "regA_star": 0.051490162113829285
 
 
 
 
 
49
  },
50
  "diff filt_b4": {
51
  "rate": 0.9277777777777777,
@@ -57,7 +77,12 @@
57
  "regB_ref": 0.024622161334723724,
58
  "regB_star": 0.03283620184325242,
59
  "regA_ref": 0.03892824451900456,
60
- "regA_star": 0.05086767155048544
 
 
 
 
 
61
  },
62
  "diff filt_b16": {
63
  "rate": 0.9861111111111112,
@@ -69,7 +94,12 @@
69
  "regB_ref": 0.014977849993612624,
70
  "regB_star": 0.0218213442998485,
71
  "regA_ref": 0.03976742768706387,
72
- "regA_star": 0.050933835066115485
 
 
 
 
 
73
  },
74
  "diff ref_b16_n3": {
75
  "rate": 0.9944444444444445,
@@ -81,7 +111,12 @@
81
  "regB_ref": 0.015012072869242443,
82
  "regB_star": 0.021944205370784453,
83
  "regA_ref": 0.03991946776173949,
84
- "regA_star": 0.051211998589972385
 
 
 
 
 
85
  },
86
  "diff ref_b16_n10": {
87
  "rate": 0.9972222222222222,
@@ -93,7 +128,12 @@
93
  "regB_ref": 0.015046295744872262,
94
  "regB_star": 0.022067066441720407,
95
  "regA_ref": 0.04007150783641511,
96
- "regA_star": 0.051490162113829285
 
 
 
 
 
97
  },
98
  "twin raw_b1": {
99
  "rate": 0.4916666666666667,
@@ -105,7 +145,12 @@
105
  "regB_ref": 0.034868968961000446,
106
  "regB_star": 0.045860235685736495,
107
  "regA_ref": 0.034868968961000446,
108
- "regA_star": 0.045860235685736495
 
 
 
 
 
109
  },
110
  "twin raw_b4": {
111
  "rate": 0.6222222222222222,
@@ -117,7 +162,12 @@
117
  "regB_ref": 0.031090565068918607,
118
  "regB_star": 0.04009868135354511,
119
  "regA_ref": 0.03602559875874334,
120
- "regA_star": 0.04612835952042982
 
 
 
 
 
121
  },
122
  "twin raw_b16": {
123
  "rate": 0.6888888888888888,
@@ -129,7 +179,12 @@
129
  "regB_ref": 0.024827462390698996,
130
  "regB_star": 0.034008343863723534,
131
  "regA_ref": 0.03665811970701527,
132
- "regA_star": 0.04772048027118436
 
 
 
 
 
133
  },
134
  "twin filt_b1": {
135
  "rate": 0.5722222222222222,
@@ -141,7 +196,12 @@
141
  "regB_ref": 0.035172887226527116,
142
  "regB_star": 0.045343068252056584,
143
  "regA_ref": 0.035172887226527116,
144
- "regA_star": 0.045343068252056584
 
 
 
 
 
145
  },
146
  "twin filt_b4": {
147
  "rate": 0.6805555555555557,
@@ -153,7 +213,12 @@
153
  "regB_ref": 0.03138655281275926,
154
  "regB_star": 0.040342528272814926,
155
  "regA_ref": 0.036638098220030946,
156
- "regA_star": 0.04700456890654414
 
 
 
 
 
157
  },
158
  "twin filt_b16": {
159
  "rate": 0.7305555555555555,
@@ -165,7 +230,12 @@
165
  "regB_ref": 0.02495787433019081,
166
  "regB_star": 0.03388894221642125,
167
  "regA_ref": 0.036800685855217674,
168
- "regA_star": 0.04793953255697314
 
 
 
 
 
169
  },
170
  "knn raw_b1": {
171
  "rate": 0.25833333333333336,
@@ -175,7 +245,10 @@
175
  "regB_ref": 0.11371326220225075,
176
  "regB_star": 0.11629817415072874,
177
  "regA_ref": 0.11371326220225075,
178
- "regA_star": 0.11629817415072874
 
 
 
179
  },
180
  "knn raw_b4": {
181
  "rate": 0.44166666666666665,
@@ -185,7 +258,10 @@
185
  "regB_ref": 0.09416819337804627,
186
  "regB_star": 0.11250915295445885,
187
  "regA_ref": 0.11635060647060991,
188
- "regA_star": 0.13716816454409586
 
 
 
189
  },
190
  "knn raw_b16": {
191
  "rate": 0.7083333333333334,
@@ -195,7 +271,10 @@
195
  "regB_ref": 0.08590952578105893,
196
  "regB_star": 0.09869180446652481,
197
  "regA_ref": 0.13353007468957645,
198
- "regA_star": 0.13956844190360695
 
 
 
199
  },
200
  "knn filt_b1": {
201
  "rate": 0.5083333333333333,
@@ -205,7 +284,10 @@
205
  "regB_ref": 0.134908830251387,
206
  "regB_star": 0.14090533433695862,
207
  "regA_ref": 0.134908830251387,
208
- "regA_star": 0.14090533433695862
 
 
 
209
  },
210
  "knn filt_b4": {
211
  "rate": 0.7833333333333333,
@@ -215,7 +297,10 @@
215
  "regB_ref": 0.10139965849798827,
216
  "regB_star": 0.10879331316422458,
217
  "regA_ref": 0.13114353705654402,
218
- "regA_star": 0.1383683032238514
 
 
 
219
  },
220
  "knn filt_b16": {
221
  "rate": 0.8666666666666667,
@@ -225,7 +310,10 @@
225
  "regB_ref": 0.07096639087530376,
226
  "regB_star": 0.08244864896963622,
227
  "regA_ref": 0.13353036702921328,
228
- "regA_star": 0.1383683032238514
 
 
 
229
  },
230
  "knn ref_b16_n3": {
231
  "rate": 0.9333333333333333,
@@ -235,7 +323,10 @@
235
  "regB_ref": 0.07228077375089703,
236
  "regB_star": 0.0830961623057771,
237
  "regA_ref": 0.12499251210513573,
238
- "regA_star": 0.13353036702921328
 
 
 
239
  },
240
  "knn ref_b16_n10": {
241
  "rate": 0.9333333333333333,
@@ -245,6 +336,9 @@
245
  "regB_ref": 0.07337219985527899,
246
  "regB_star": 0.08575503047710231,
247
  "regA_ref": 0.12802583666738943,
248
- "regA_star": 0.13421974481011856
 
 
 
249
  }
250
  }
 
9
  "regB_ref": 0.035166553853986704,
10
  "regB_star": 0.04939973422469457,
11
  "regA_ref": 0.035166553853986704,
12
+ "regA_star": 0.04939973422469457,
13
+ "reg29_star_per_seed": [
14
+ 0.1173585852888046,
15
+ 0.6048869266055182,
16
+ 1000.0
17
+ ]
18
  },
19
  "diff raw_b4": {
20
  "rate": 0.7611111111111111,
 
26
  "regB_ref": 0.026566873792452816,
27
  "regB_star": 0.036538855499143374,
28
  "regA_ref": 0.03912745224114178,
29
+ "regA_star": 0.05271375933165312,
30
+ "reg29_star_per_seed": [
31
+ 0.054677934415332974,
32
+ 0.05112708412480571,
33
+ 0.0515105084462818
34
+ ]
35
  },
36
  "diff raw_b16": {
37
  "rate": 0.9444444444444443,
 
43
  "regB_ref": 0.017749766148115964,
44
  "regB_star": 0.02536120788249141,
45
  "regA_ref": 0.03975266005687195,
46
+ "regA_star": 0.051211998589972385,
47
+ "reg29_star_per_seed": [
48
+ 0.025756009753485576,
49
+ 0.029513557045964256,
50
+ 0.02766465097867299
51
+ ]
52
  },
53
  "diff filt_b1": {
54
  "rate": 0.6861111111111112,
 
60
  "regB_ref": 0.03864968435767513,
61
  "regB_star": 0.051490162113829285,
62
  "regA_ref": 0.03864968435767513,
63
+ "regA_star": 0.051490162113829285,
64
+ "reg29_star_per_seed": [
65
+ 0.0763562167545252,
66
+ 0.07432971205930794,
67
+ 0.10765837454236571
68
+ ]
69
  },
70
  "diff filt_b4": {
71
  "rate": 0.9277777777777777,
 
77
  "regB_ref": 0.024622161334723724,
78
  "regB_star": 0.03283620184325242,
79
  "regA_ref": 0.03892824451900456,
80
+ "regA_star": 0.05086767155048544,
81
+ "reg29_star_per_seed": [
82
+ 0.03562885445803954,
83
+ 0.03860264335993191,
84
+ 0.032221163711480205
85
+ ]
86
  },
87
  "diff filt_b16": {
88
  "rate": 0.9861111111111112,
 
94
  "regB_ref": 0.014977849993612624,
95
  "regB_star": 0.0218213442998485,
96
  "regA_ref": 0.03976742768706387,
97
+ "regA_star": 0.050933835066115485,
98
+ "reg29_star_per_seed": [
99
+ 0.0235246180444032,
100
+ 0.023310251680270566,
101
+ 0.020264949687200495
102
+ ]
103
  },
104
  "diff ref_b16_n3": {
105
  "rate": 0.9944444444444445,
 
111
  "regB_ref": 0.015012072869242443,
112
  "regB_star": 0.021944205370784453,
113
  "regA_ref": 0.03991946776173949,
114
+ "regA_star": 0.051211998589972385,
115
+ "reg29_star_per_seed": [
116
+ 0.0235246180444032,
117
+ 0.023310251680270566,
118
+ 0.019837172273985404
119
+ ]
120
  },
121
  "diff ref_b16_n10": {
122
  "rate": 0.9972222222222222,
 
128
  "regB_ref": 0.015046295744872262,
129
  "regB_star": 0.022067066441720407,
130
  "regA_ref": 0.04007150783641511,
131
+ "regA_star": 0.051490162113829285,
132
+ "reg29_star_per_seed": [
133
+ 0.0235246180444032,
134
+ 0.023310251680270566,
135
+ 0.019837172273985404
136
+ ]
137
  },
138
  "twin raw_b1": {
139
  "rate": 0.4916666666666667,
 
145
  "regB_ref": 0.034868968961000446,
146
  "regB_star": 0.045860235685736495,
147
  "regA_ref": 0.034868968961000446,
148
+ "regA_star": 0.045860235685736495,
149
+ "reg29_star_per_seed": [
150
+ 1000.0,
151
+ 500.0670756532481,
152
+ 500.07857939015105
153
+ ]
154
  },
155
  "twin raw_b4": {
156
  "rate": 0.6222222222222222,
 
162
  "regB_ref": 0.031090565068918607,
163
  "regB_star": 0.04009868135354511,
164
  "regA_ref": 0.03602559875874334,
165
+ "regA_star": 0.04612835952042982,
166
+ "reg29_star_per_seed": [
167
+ 0.07311798148058501,
168
+ 0.07470892227613435,
169
+ 0.08088746958920823
170
+ ]
171
  },
172
  "twin raw_b16": {
173
  "rate": 0.6888888888888888,
 
179
  "regB_ref": 0.024827462390698996,
180
  "regB_star": 0.034008343863723534,
181
  "regA_ref": 0.03665811970701527,
182
+ "regA_star": 0.04772048027118436,
183
+ "reg29_star_per_seed": [
184
+ 0.05807689752517098,
185
+ 0.05422806624253773,
186
+ 0.05913398532637004
187
+ ]
188
  },
189
  "twin filt_b1": {
190
  "rate": 0.5722222222222222,
 
196
  "regB_ref": 0.035172887226527116,
197
  "regB_star": 0.045343068252056584,
198
  "regA_ref": 0.035172887226527116,
199
+ "regA_star": 0.045343068252056584,
200
+ "reg29_star_per_seed": [
201
+ 0.09196044154961569,
202
+ 0.10137589764619415,
203
+ 0.08672273104752559
204
+ ]
205
  },
206
  "twin filt_b4": {
207
  "rate": 0.6805555555555557,
 
213
  "regB_ref": 0.03138655281275926,
214
  "regB_star": 0.040342528272814926,
215
  "regA_ref": 0.036638098220030946,
216
+ "regA_star": 0.04700456890654414,
217
+ "reg29_star_per_seed": [
218
+ 0.06735180745496439,
219
+ 0.06821410044080367,
220
+ 0.06362610349884046
221
+ ]
222
  },
223
  "twin filt_b16": {
224
  "rate": 0.7305555555555555,
 
230
  "regB_ref": 0.02495787433019081,
231
  "regB_star": 0.03388894221642125,
232
  "regA_ref": 0.036800685855217674,
233
+ "regA_star": 0.04793953255697314,
234
+ "reg29_star_per_seed": [
235
+ 0.05279832218166869,
236
+ 0.05271963186300921,
237
+ 0.051832046588488945
238
+ ]
239
  },
240
  "knn raw_b1": {
241
  "rate": 0.25833333333333336,
 
245
  "regB_ref": 0.11371326220225075,
246
  "regB_star": 0.11629817415072874,
247
  "regA_ref": 0.11371326220225075,
248
+ "regA_star": 0.11629817415072874,
249
+ "reg29_star_per_seed": [
250
+ 1000.0
251
+ ]
252
  },
253
  "knn raw_b4": {
254
  "rate": 0.44166666666666665,
 
258
  "regB_ref": 0.09416819337804627,
259
  "regB_star": 0.11250915295445885,
260
  "regA_ref": 0.11635060647060991,
261
+ "regA_star": 0.13716816454409586,
262
+ "reg29_star_per_seed": [
263
+ 1000.0
264
+ ]
265
  },
266
  "knn raw_b16": {
267
  "rate": 0.7083333333333334,
 
271
  "regB_ref": 0.08590952578105893,
272
  "regB_star": 0.09869180446652481,
273
  "regA_ref": 0.13353007468957645,
274
+ "regA_star": 0.13956844190360695,
275
+ "reg29_star_per_seed": [
276
+ 0.1222650154352023
277
+ ]
278
  },
279
  "knn filt_b1": {
280
  "rate": 0.5083333333333333,
 
284
  "regB_ref": 0.134908830251387,
285
  "regB_star": 0.14090533433695862,
286
  "regA_ref": 0.134908830251387,
287
+ "regA_star": 0.14090533433695862,
288
+ "reg29_star_per_seed": [
289
+ 0.47128174901284803
290
+ ]
291
  },
292
  "knn filt_b4": {
293
  "rate": 0.7833333333333333,
 
297
  "regB_ref": 0.10139965849798827,
298
  "regB_star": 0.10879331316422458,
299
  "regA_ref": 0.13114353705654402,
300
+ "regA_star": 0.1383683032238514,
301
+ "reg29_star_per_seed": [
302
+ 0.12376717018326289
303
+ ]
304
  },
305
  "knn filt_b16": {
306
  "rate": 0.8666666666666667,
 
310
  "regB_ref": 0.07096639087530376,
311
  "regB_star": 0.08244864896963622,
312
  "regA_ref": 0.13353036702921328,
313
+ "regA_star": 0.1383683032238514,
314
+ "reg29_star_per_seed": [
315
+ 0.08899908416608437
316
+ ]
317
  },
318
  "knn ref_b16_n3": {
319
  "rate": 0.9333333333333333,
 
323
  "regB_ref": 0.07228077375089703,
324
  "regB_star": 0.0830961623057771,
325
  "regA_ref": 0.12499251210513573,
326
+ "regA_star": 0.13353036702921328,
327
+ "reg29_star_per_seed": [
328
+ 0.08746904759596996
329
+ ]
330
  },
331
  "knn ref_b16_n10": {
332
  "rate": 0.9333333333333333,
 
336
  "regB_ref": 0.07337219985527899,
337
  "regB_star": 0.08575503047710231,
338
  "regA_ref": 0.12802583666738943,
339
+ "regA_star": 0.13421974481011856,
340
+ "reg29_star_per_seed": [
341
+ 0.08835772743585302
342
+ ]
343
  }
344
  }
results/icml_r3/X29/act/x27rescore_d3fm3n8000.txt CHANGED
@@ -1,23 +1,23 @@
1
  X29-A: X27's C-only arms re-scored, m = 3, tag d3fm3n8000, 120 test plants. J*_A = min(J_ref, 3A id and so3 labels) is below J_ref on 45 plants (median (J_ref - J*_A) / J*_A 0.000%).
2
- diff raw_b1 certified 51.7% (published 51.7%) | cheapest certified: regret vs J_ref 3.52% (published 3.52%), vs J*_A 4.94% | first certified: vs J_ref 3.52%, vs J*_A 4.94%
3
- diff raw_b4 certified 76.1% (published 76.1%) | cheapest certified: regret vs J_ref 2.66% (published 2.66%), vs J*_A 3.65% | first certified: vs J_ref 3.91%, vs J*_A 5.27%
4
- diff raw_b16 certified 94.4% (published 94.4%) | cheapest certified: regret vs J_ref 1.77% (published 1.77%), vs J*_A 2.54% | first certified: vs J_ref 3.98%, vs J*_A 5.12%
5
- diff filt_b1 certified 68.6% (published 68.6%) | cheapest certified: regret vs J_ref 3.86% (published 3.86%), vs J*_A 5.15% | first certified: vs J_ref 3.86%, vs J*_A 5.15%
6
- diff filt_b4 certified 92.8% (published 92.8%) | cheapest certified: regret vs J_ref 2.46% (published 2.46%), vs J*_A 3.28% | first certified: vs J_ref 3.89%, vs J*_A 5.09%
7
- diff filt_b16 certified 98.6% (published 98.6%) | cheapest certified: regret vs J_ref 1.50% (published 1.50%), vs J*_A 2.18% | first certified: vs J_ref 3.98%, vs J*_A 5.09%
8
- diff ref_b16_n3 certified 99.4% (published 99.4%) | cheapest certified: regret vs J_ref 1.50% (published 1.50%), vs J*_A 2.19% | first certified: vs J_ref 3.99%, vs J*_A 5.12%
9
- diff ref_b16_n10 certified 99.7% (published 99.7%) | cheapest certified: regret vs J_ref 1.50% (published 1.50%), vs J*_A 2.21% | first certified: vs J_ref 4.01%, vs J*_A 5.15%
10
- twin raw_b1 certified 49.2% (published 49.2%) | cheapest certified: regret vs J_ref 3.49% (published 3.49%), vs J*_A 4.59% | first certified: vs J_ref 3.49%, vs J*_A 4.59%
11
- twin raw_b4 certified 62.2% (published 62.2%) | cheapest certified: regret vs J_ref 3.11% (published 3.11%), vs J*_A 4.01% | first certified: vs J_ref 3.60%, vs J*_A 4.61%
12
- twin raw_b16 certified 68.9% (published 68.9%) | cheapest certified: regret vs J_ref 2.48% (published 2.48%), vs J*_A 3.40% | first certified: vs J_ref 3.67%, vs J*_A 4.77%
13
- twin filt_b1 certified 57.2% (published 57.2%) | cheapest certified: regret vs J_ref 3.52% (published 3.52%), vs J*_A 4.53% | first certified: vs J_ref 3.52%, vs J*_A 4.53%
14
- twin filt_b4 certified 68.1% (published 68.1%) | cheapest certified: regret vs J_ref 3.14% (published 3.14%), vs J*_A 4.03% | first certified: vs J_ref 3.66%, vs J*_A 4.70%
15
- twin filt_b16 certified 73.1% (published 73.1%) | cheapest certified: regret vs J_ref 2.50% (published 2.50%), vs J*_A 3.39% | first certified: vs J_ref 3.68%, vs J*_A 4.79%
16
- knn raw_b1 certified 25.8% (published 25.8%) | cheapest certified: regret vs J_ref 11.37% (published 11.37%), vs J*_A 11.63% | first certified: vs J_ref 11.37%, vs J*_A 11.63%
17
- knn raw_b4 certified 44.2% (published 44.2%) | cheapest certified: regret vs J_ref 9.42% (published 9.42%), vs J*_A 11.25% | first certified: vs J_ref 11.64%, vs J*_A 13.72%
18
- knn raw_b16 certified 70.8% (published 70.8%) | cheapest certified: regret vs J_ref 8.59% (published 8.59%), vs J*_A 9.87% | first certified: vs J_ref 13.35%, vs J*_A 13.96%
19
- knn filt_b1 certified 50.8% (published 50.8%) | cheapest certified: regret vs J_ref 13.49% (published 13.49%), vs J*_A 14.09% | first certified: vs J_ref 13.49%, vs J*_A 14.09%
20
- knn filt_b4 certified 78.3% (published 78.3%) | cheapest certified: regret vs J_ref 10.14% (published 10.14%), vs J*_A 10.88% | first certified: vs J_ref 13.11%, vs J*_A 13.84%
21
- knn filt_b16 certified 86.7% (published 86.7%) | cheapest certified: regret vs J_ref 7.10% (published 7.10%), vs J*_A 8.24% | first certified: vs J_ref 13.35%, vs J*_A 13.84%
22
- knn ref_b16_n3 certified 93.3% (published 93.3%) | cheapest certified: regret vs J_ref 7.23% (published 7.23%), vs J*_A 8.31% | first certified: vs J_ref 12.50%, vs J*_A 13.35%
23
- knn ref_b16_n10 certified 93.3% (published 93.3%) | cheapest certified: regret vs J_ref 7.34% (published 7.34%), vs J*_A 8.58% | first certified: vs J_ref 12.80%, vs J*_A 13.42%
 
1
  X29-A: X27's C-only arms re-scored, m = 3, tag d3fm3n8000, 120 test plants. J*_A = min(J_ref, 3A id and so3 labels) is below J_ref on 45 plants (median (J_ref - J*_A) / J*_A 0.000%).
2
+ diff raw_b1 certified 51.7% (published 51.7%) | cheapest certified: regret vs J_ref 3.52% (published 3.52%), vs J*_A 4.94% | first certified: vs J_ref 3.52%, vs J*_A 4.94% | X29 convention vs J*_A, per seed: 11.74% / 60.49% / n/a
3
+ diff raw_b4 certified 76.1% (published 76.1%) | cheapest certified: regret vs J_ref 2.66% (published 2.66%), vs J*_A 3.65% | first certified: vs J_ref 3.91%, vs J*_A 5.27% | X29 convention vs J*_A, per seed: 5.47% / 5.11% / 5.15%
4
+ diff raw_b16 certified 94.4% (published 94.4%) | cheapest certified: regret vs J_ref 1.77% (published 1.77%), vs J*_A 2.54% | first certified: vs J_ref 3.98%, vs J*_A 5.12% | X29 convention vs J*_A, per seed: 2.58% / 2.95% / 2.77%
5
+ diff filt_b1 certified 68.6% (published 68.6%) | cheapest certified: regret vs J_ref 3.86% (published 3.86%), vs J*_A 5.15% | first certified: vs J_ref 3.86%, vs J*_A 5.15% | X29 convention vs J*_A, per seed: 7.64% / 7.43% / 10.77%
6
+ diff filt_b4 certified 92.8% (published 92.8%) | cheapest certified: regret vs J_ref 2.46% (published 2.46%), vs J*_A 3.28% | first certified: vs J_ref 3.89%, vs J*_A 5.09% | X29 convention vs J*_A, per seed: 3.56% / 3.86% / 3.22%
7
+ diff filt_b16 certified 98.6% (published 98.6%) | cheapest certified: regret vs J_ref 1.50% (published 1.50%), vs J*_A 2.18% | first certified: vs J_ref 3.98%, vs J*_A 5.09% | X29 convention vs J*_A, per seed: 2.35% / 2.33% / 2.03%
8
+ diff ref_b16_n3 certified 99.4% (published 99.4%) | cheapest certified: regret vs J_ref 1.50% (published 1.50%), vs J*_A 2.19% | first certified: vs J_ref 3.99%, vs J*_A 5.12% | X29 convention vs J*_A, per seed: 2.35% / 2.33% / 1.98%
9
+ diff ref_b16_n10 certified 99.7% (published 99.7%) | cheapest certified: regret vs J_ref 1.50% (published 1.50%), vs J*_A 2.21% | first certified: vs J_ref 4.01%, vs J*_A 5.15% | X29 convention vs J*_A, per seed: 2.35% / 2.33% / 1.98%
10
+ twin raw_b1 certified 49.2% (published 49.2%) | cheapest certified: regret vs J_ref 3.49% (published 3.49%), vs J*_A 4.59% | first certified: vs J_ref 3.49%, vs J*_A 4.59% | X29 convention vs J*_A, per seed: n/a / n/a / n/a
11
+ twin raw_b4 certified 62.2% (published 62.2%) | cheapest certified: regret vs J_ref 3.11% (published 3.11%), vs J*_A 4.01% | first certified: vs J_ref 3.60%, vs J*_A 4.61% | X29 convention vs J*_A, per seed: 7.31% / 7.47% / 8.09%
12
+ twin raw_b16 certified 68.9% (published 68.9%) | cheapest certified: regret vs J_ref 2.48% (published 2.48%), vs J*_A 3.40% | first certified: vs J_ref 3.67%, vs J*_A 4.77% | X29 convention vs J*_A, per seed: 5.81% / 5.42% / 5.91%
13
+ twin filt_b1 certified 57.2% (published 57.2%) | cheapest certified: regret vs J_ref 3.52% (published 3.52%), vs J*_A 4.53% | first certified: vs J_ref 3.52%, vs J*_A 4.53% | X29 convention vs J*_A, per seed: 9.20% / 10.14% / 8.67%
14
+ twin filt_b4 certified 68.1% (published 68.1%) | cheapest certified: regret vs J_ref 3.14% (published 3.14%), vs J*_A 4.03% | first certified: vs J_ref 3.66%, vs J*_A 4.70% | X29 convention vs J*_A, per seed: 6.74% / 6.82% / 6.36%
15
+ twin filt_b16 certified 73.1% (published 73.1%) | cheapest certified: regret vs J_ref 2.50% (published 2.50%), vs J*_A 3.39% | first certified: vs J_ref 3.68%, vs J*_A 4.79% | X29 convention vs J*_A, per seed: 5.28% / 5.27% / 5.18%
16
+ knn raw_b1 certified 25.8% (published 25.8%) | cheapest certified: regret vs J_ref 11.37% (published 11.37%), vs J*_A 11.63% | first certified: vs J_ref 11.37%, vs J*_A 11.63% | X29 convention vs J*_A, per seed: n/a
17
+ knn raw_b4 certified 44.2% (published 44.2%) | cheapest certified: regret vs J_ref 9.42% (published 9.42%), vs J*_A 11.25% | first certified: vs J_ref 11.64%, vs J*_A 13.72% | X29 convention vs J*_A, per seed: n/a
18
+ knn raw_b16 certified 70.8% (published 70.8%) | cheapest certified: regret vs J_ref 8.59% (published 8.59%), vs J*_A 9.87% | first certified: vs J_ref 13.35%, vs J*_A 13.96% | X29 convention vs J*_A, per seed: 12.23%
19
+ knn filt_b1 certified 50.8% (published 50.8%) | cheapest certified: regret vs J_ref 13.49% (published 13.49%), vs J*_A 14.09% | first certified: vs J_ref 13.49%, vs J*_A 14.09% | X29 convention vs J*_A, per seed: 47.13%
20
+ knn filt_b4 certified 78.3% (published 78.3%) | cheapest certified: regret vs J_ref 10.14% (published 10.14%), vs J*_A 10.88% | first certified: vs J_ref 13.11%, vs J*_A 13.84% | X29 convention vs J*_A, per seed: 12.38%
21
+ knn filt_b16 certified 86.7% (published 86.7%) | cheapest certified: regret vs J_ref 7.10% (published 7.10%), vs J*_A 8.24% | first certified: vs J_ref 13.35%, vs J*_A 13.84% | X29 convention vs J*_A, per seed: 8.90%
22
+ knn ref_b16_n3 certified 93.3% (published 93.3%) | cheapest certified: regret vs J_ref 7.23% (published 7.23%), vs J*_A 8.31% | first certified: vs J_ref 12.50%, vs J*_A 13.35% | X29 convention vs J*_A, per seed: 8.75%
23
+ knn ref_b16_n10 certified 93.3% (published 93.3%) | cheapest certified: regret vs J_ref 7.34% (published 7.34%), vs J*_A 8.58% | first certified: vs J_ref 12.80%, vs J*_A 13.42% | X29 convention vs J*_A, per seed: 8.84%
results/icml_r3/X29/act/x27rescore_d3fm4n8000.json CHANGED
@@ -9,7 +9,12 @@
9
  "regB_ref": 0.04281187301679423,
10
  "regB_star": 0.04962915968635695,
11
  "regA_ref": 0.04281187301679423,
12
- "regA_star": 0.04962915968635695
 
 
 
 
 
13
  },
14
  "diff raw_b4": {
15
  "rate": 0.5722222222222222,
@@ -21,7 +26,12 @@
21
  "regB_ref": 0.04023679269097036,
22
  "regB_star": 0.047381757855627926,
23
  "regA_ref": 0.04831828056771548,
24
- "regA_star": 0.057019597285727364
 
 
 
 
 
25
  },
26
  "diff raw_b16": {
27
  "rate": 0.7888888888888889,
@@ -33,7 +43,12 @@
33
  "regB_ref": 0.03206114493999934,
34
  "regB_star": 0.03878398370676894,
35
  "regA_ref": 0.05338276992090041,
36
- "regA_star": 0.05942518962649945
 
 
 
 
 
37
  },
38
  "diff filt_b1": {
39
  "rate": 0.5249999999999999,
@@ -45,7 +60,12 @@
45
  "regB_ref": 0.05256149004484412,
46
  "regB_star": 0.05789482111431554,
47
  "regA_ref": 0.05256149004484412,
48
- "regA_star": 0.05789482111431554
 
 
 
 
 
49
  },
50
  "diff filt_b4": {
51
  "rate": 0.8250000000000001,
@@ -57,7 +77,12 @@
57
  "regB_ref": 0.04154891326937804,
58
  "regB_star": 0.05096408010028841,
59
  "regA_ref": 0.05679915606087227,
60
- "regA_star": 0.06378147142694024
 
 
 
 
 
61
  },
62
  "diff filt_b16": {
63
  "rate": 0.9194444444444444,
@@ -69,7 +94,12 @@
69
  "regB_ref": 0.030428223592535053,
70
  "regB_star": 0.03729421986230963,
71
  "regA_ref": 0.05678969283770341,
72
- "regA_star": 0.06378147142694024
 
 
 
 
 
73
  },
74
  "diff ref_b16_n3": {
75
  "rate": 0.9666666666666667,
@@ -81,7 +111,12 @@
81
  "regB_ref": 0.032163477351268455,
82
  "regB_star": 0.03848414313894111,
83
  "regA_ref": 0.05752555203944998,
84
- "regA_star": 0.06417574626693279
 
 
 
 
 
85
  },
86
  "diff ref_b16_n10": {
87
  "rate": 0.9777777777777777,
@@ -93,7 +128,12 @@
93
  "regB_ref": 0.03239370322615842,
94
  "regB_star": 0.03848414313894111,
95
  "regA_ref": 0.0571452435546439,
96
- "regA_star": 0.0639596586478054
 
 
 
 
 
97
  },
98
  "twin raw_b1": {
99
  "rate": 0.3138888888888889,
@@ -105,7 +145,12 @@
105
  "regB_ref": 0.05077043279796922,
106
  "regB_star": 0.054942478154655694,
107
  "regA_ref": 0.05077043279796922,
108
- "regA_star": 0.054942478154655694
 
 
 
 
 
109
  },
110
  "twin raw_b4": {
111
  "rate": 0.4305555555555555,
@@ -117,7 +162,12 @@
117
  "regB_ref": 0.040770558505952126,
118
  "regB_star": 0.04486400154999837,
119
  "regA_ref": 0.044869776269671036,
120
- "regA_star": 0.051729042499672806
 
 
 
 
 
121
  },
122
  "twin raw_b16": {
123
  "rate": 0.48333333333333334,
@@ -129,7 +179,12 @@
129
  "regB_ref": 0.03419325650322769,
130
  "regB_star": 0.03997382209205025,
131
  "regA_ref": 0.04478727476519462,
132
- "regA_star": 0.051812151699132114
 
 
 
 
 
133
  },
134
  "twin filt_b1": {
135
  "rate": 0.3666666666666667,
@@ -141,7 +196,12 @@
141
  "regB_ref": 0.0509702614716806,
142
  "regB_star": 0.05496224880262157,
143
  "regA_ref": 0.0509702614716806,
144
- "regA_star": 0.05496224880262157
 
 
 
 
 
145
  },
146
  "twin filt_b4": {
147
  "rate": 0.4861111111111111,
@@ -153,7 +213,12 @@
153
  "regB_ref": 0.039893127429255326,
154
  "regB_star": 0.044859535745951584,
155
  "regA_ref": 0.044869776269671036,
156
- "regA_star": 0.051729042499672806
 
 
 
 
 
157
  },
158
  "twin filt_b16": {
159
  "rate": 0.5222222222222221,
@@ -165,7 +230,12 @@
165
  "regB_ref": 0.03461930665839119,
166
  "regB_star": 0.039952457345226344,
167
  "regA_ref": 0.045169830011552614,
168
- "regA_star": 0.05347942435820763
 
 
 
 
 
169
  },
170
  "knn raw_b1": {
171
  "rate": 0.14166666666666666,
@@ -175,7 +245,10 @@
175
  "regB_ref": 0.11972585447859752,
176
  "regB_star": 0.1203804444936909,
177
  "regA_ref": 0.11972585447859752,
178
- "regA_star": 0.1203804444936909
 
 
 
179
  },
180
  "knn raw_b4": {
181
  "rate": 0.39166666666666666,
@@ -185,7 +258,10 @@
185
  "regB_ref": 0.11972585447859752,
186
  "regB_star": 0.1249036336281455,
187
  "regA_ref": 0.1249036336281455,
188
- "regA_star": 0.13069438483572582
 
 
 
189
  },
190
  "knn raw_b16": {
191
  "rate": 0.5916666666666667,
@@ -195,7 +271,10 @@
195
  "regB_ref": 0.10947876745055107,
196
  "regB_star": 0.11616937090964607,
197
  "regA_ref": 0.13058469845406107,
198
- "regA_star": 0.1400175969983004
 
 
 
199
  },
200
  "knn filt_b1": {
201
  "rate": 0.45,
@@ -205,7 +284,10 @@
205
  "regB_ref": 0.11863223190950545,
206
  "regB_star": 0.12336736317427233,
207
  "regA_ref": 0.11863223190950545,
208
- "regA_star": 0.12336736317427233
 
 
 
209
  },
210
  "knn filt_b4": {
211
  "rate": 0.75,
@@ -215,7 +297,10 @@
215
  "regB_ref": 0.11100685037250602,
216
  "regB_star": 0.11698292149647704,
217
  "regA_ref": 0.13511513600466607,
218
- "regA_star": 0.14419869136826147
 
 
 
219
  },
220
  "knn filt_b16": {
221
  "rate": 0.7916666666666666,
@@ -225,7 +310,10 @@
225
  "regB_ref": 0.0926797810653952,
226
  "regB_star": 0.09917966684483925,
227
  "regA_ref": 0.1400175969983004,
228
- "regA_star": 0.14697604799704078
 
 
 
229
  },
230
  "knn ref_b16_n3": {
231
  "rate": 0.9,
@@ -235,7 +323,10 @@
235
  "regB_ref": 0.09150018593164855,
236
  "regB_star": 0.09801292912248993,
237
  "regA_ref": 0.12780736488049338,
238
- "regA_star": 0.13983158527678574
 
 
 
239
  },
240
  "knn ref_b16_n10": {
241
  "rate": 0.9333333333333333,
@@ -245,6 +336,9 @@
245
  "regB_ref": 0.09026618689333521,
246
  "regB_star": 0.09677288836815534,
247
  "regA_ref": 0.1258708743514112,
248
- "regA_star": 0.1349668764575651
 
 
 
249
  }
250
  }
 
9
  "regB_ref": 0.04281187301679423,
10
  "regB_star": 0.04962915968635695,
11
  "regA_ref": 0.04281187301679423,
12
+ "regA_star": 0.04962915968635695,
13
+ "reg29_star_per_seed": [
14
+ 1000.0,
15
+ 1000.0,
16
+ 1000.0
17
+ ]
18
  },
19
  "diff raw_b4": {
20
  "rate": 0.5722222222222222,
 
26
  "regB_ref": 0.04023679269097036,
27
  "regB_star": 0.047381757855627926,
28
  "regA_ref": 0.04831828056771548,
29
+ "regA_star": 0.057019597285727364,
30
+ "reg29_star_per_seed": [
31
+ 0.08495044681003916,
32
+ 0.0881141812966516,
33
+ 0.11122677860139785
34
+ ]
35
  },
36
  "diff raw_b16": {
37
  "rate": 0.7888888888888889,
 
43
  "regB_ref": 0.03206114493999934,
44
  "regB_star": 0.03878398370676894,
45
  "regA_ref": 0.05338276992090041,
46
+ "regA_star": 0.05942518962649945,
47
+ "reg29_star_per_seed": [
48
+ 0.047293419441651996,
49
+ 0.05764590082594878,
50
+ 0.05264291865089865
51
+ ]
52
  },
53
  "diff filt_b1": {
54
  "rate": 0.5249999999999999,
 
60
  "regB_ref": 0.05256149004484412,
61
  "regB_star": 0.05789482111431554,
62
  "regA_ref": 0.05256149004484412,
63
+ "regA_star": 0.05789482111431554,
64
+ "reg29_star_per_seed": [
65
+ 0.263211143468268,
66
+ 0.12940601631344728,
67
+ 0.14249857544129063
68
+ ]
69
  },
70
  "diff filt_b4": {
71
  "rate": 0.8250000000000001,
 
77
  "regB_ref": 0.04154891326937804,
78
  "regB_star": 0.05096408010028841,
79
  "regA_ref": 0.05679915606087227,
80
+ "regA_star": 0.06378147142694024,
81
+ "reg29_star_per_seed": [
82
+ 0.06236438872575767,
83
+ 0.06424515800692798,
84
+ 0.060460629694771506
85
+ ]
86
  },
87
  "diff filt_b16": {
88
  "rate": 0.9194444444444444,
 
94
  "regB_ref": 0.030428223592535053,
95
  "regB_star": 0.03729421986230963,
96
  "regA_ref": 0.05678969283770341,
97
+ "regA_star": 0.06378147142694024,
98
+ "reg29_star_per_seed": [
99
+ 0.038550378977868505,
100
+ 0.04405060528876292,
101
+ 0.03878398370676894
102
+ ]
103
  },
104
  "diff ref_b16_n3": {
105
  "rate": 0.9666666666666667,
 
111
  "regB_ref": 0.032163477351268455,
112
  "regB_star": 0.03848414313894111,
113
  "regA_ref": 0.05752555203944998,
114
+ "regA_star": 0.06417574626693279,
115
+ "reg29_star_per_seed": [
116
+ 0.038048491177176014,
117
+ 0.04358892940577752,
118
+ 0.03850457199311308
119
+ ]
120
  },
121
  "diff ref_b16_n10": {
122
  "rate": 0.9777777777777777,
 
128
  "regB_ref": 0.03239370322615842,
129
  "regB_star": 0.03848414313894111,
130
  "regA_ref": 0.0571452435546439,
131
+ "regA_star": 0.0639596586478054,
132
+ "reg29_star_per_seed": [
133
+ 0.0375075533219793,
134
+ 0.04358892940577752,
135
+ 0.037786990412120525
136
+ ]
137
  },
138
  "twin raw_b1": {
139
  "rate": 0.3138888888888889,
 
145
  "regB_ref": 0.05077043279796922,
146
  "regB_star": 0.054942478154655694,
147
  "regA_ref": 0.05077043279796922,
148
+ "regA_star": 0.054942478154655694,
149
+ "reg29_star_per_seed": [
150
+ 1000.0,
151
+ 1000.0,
152
+ 1000.0
153
+ ]
154
  },
155
  "twin raw_b4": {
156
  "rate": 0.4305555555555555,
 
162
  "regB_ref": 0.040770558505952126,
163
  "regB_star": 0.04486400154999837,
164
  "regA_ref": 0.044869776269671036,
165
+ "regA_star": 0.051729042499672806,
166
+ "reg29_star_per_seed": [
167
+ 1000.0,
168
+ 1000.0,
169
+ 1000.0
170
+ ]
171
  },
172
  "twin raw_b16": {
173
  "rate": 0.48333333333333334,
 
179
  "regB_ref": 0.03419325650322769,
180
  "regB_star": 0.03997382209205025,
181
  "regA_ref": 0.04478727476519462,
182
+ "regA_star": 0.051812151699132114,
183
+ "reg29_star_per_seed": [
184
+ 1000.0,
185
+ 1000.0,
186
+ 500.05918359518233
187
+ ]
188
  },
189
  "twin filt_b1": {
190
  "rate": 0.3666666666666667,
 
196
  "regB_ref": 0.0509702614716806,
197
  "regB_star": 0.05496224880262157,
198
  "regA_ref": 0.0509702614716806,
199
+ "regA_star": 0.05496224880262157,
200
+ "reg29_star_per_seed": [
201
+ 1000.0,
202
+ 1000.0,
203
+ 1000.0
204
+ ]
205
  },
206
  "twin filt_b4": {
207
  "rate": 0.4861111111111111,
 
213
  "regB_ref": 0.039893127429255326,
214
  "regB_star": 0.044859535745951584,
215
  "regA_ref": 0.044869776269671036,
216
+ "regA_star": 0.051729042499672806,
217
+ "reg29_star_per_seed": [
218
+ 1000.0,
219
+ 1000.0,
220
+ 500.06559232249737
221
+ ]
222
  },
223
  "twin filt_b16": {
224
  "rate": 0.5222222222222221,
 
230
  "regB_ref": 0.03461930665839119,
231
  "regB_star": 0.039952457345226344,
232
  "regA_ref": 0.045169830011552614,
233
+ "regA_star": 0.05347942435820763,
234
+ "reg29_star_per_seed": [
235
+ 0.13461082478981312,
236
+ 0.13603789452203452,
237
+ 0.09103587528874801
238
+ ]
239
  },
240
  "knn raw_b1": {
241
  "rate": 0.14166666666666666,
 
245
  "regB_ref": 0.11972585447859752,
246
  "regB_star": 0.1203804444936909,
247
  "regA_ref": 0.11972585447859752,
248
+ "regA_star": 0.1203804444936909,
249
+ "reg29_star_per_seed": [
250
+ 1000.0
251
+ ]
252
  },
253
  "knn raw_b4": {
254
  "rate": 0.39166666666666666,
 
258
  "regB_ref": 0.11972585447859752,
259
  "regB_star": 0.1249036336281455,
260
  "regA_ref": 0.1249036336281455,
261
+ "regA_star": 0.13069438483572582,
262
+ "reg29_star_per_seed": [
263
+ 1000.0
264
+ ]
265
  },
266
  "knn raw_b16": {
267
  "rate": 0.5916666666666667,
 
271
  "regB_ref": 0.10947876745055107,
272
  "regB_star": 0.11616937090964607,
273
  "regA_ref": 0.13058469845406107,
274
+ "regA_star": 0.1400175969983004,
275
+ "reg29_star_per_seed": [
276
+ 0.1783168905724013
277
+ ]
278
  },
279
  "knn filt_b1": {
280
  "rate": 0.45,
 
284
  "regB_ref": 0.11863223190950545,
285
  "regB_star": 0.12336736317427233,
286
  "regA_ref": 0.11863223190950545,
287
+ "regA_star": 0.12336736317427233,
288
+ "reg29_star_per_seed": [
289
+ 1000.0
290
+ ]
291
  },
292
  "knn filt_b4": {
293
  "rate": 0.75,
 
297
  "regB_ref": 0.11100685037250602,
298
  "regB_star": 0.11698292149647704,
299
  "regA_ref": 0.13511513600466607,
300
+ "regA_star": 0.14419869136826147,
301
+ "reg29_star_per_seed": [
302
+ 0.1489442082590664
303
+ ]
304
  },
305
  "knn filt_b16": {
306
  "rate": 0.7916666666666666,
 
310
  "regB_ref": 0.0926797810653952,
311
  "regB_star": 0.09917966684483925,
312
  "regA_ref": 0.1400175969983004,
313
+ "regA_star": 0.14697604799704078,
314
+ "reg29_star_per_seed": [
315
+ 0.11159647642534298
316
+ ]
317
  },
318
  "knn ref_b16_n3": {
319
  "rate": 0.9,
 
323
  "regB_ref": 0.09150018593164855,
324
  "regB_star": 0.09801292912248993,
325
  "regA_ref": 0.12780736488049338,
326
+ "regA_star": 0.13983158527678574,
327
+ "reg29_star_per_seed": [
328
+ 0.10582831324537034
329
+ ]
330
  },
331
  "knn ref_b16_n10": {
332
  "rate": 0.9333333333333333,
 
336
  "regB_ref": 0.09026618689333521,
337
  "regB_star": 0.09677288836815534,
338
  "regA_ref": 0.1258708743514112,
339
+ "regA_star": 0.1349668764575651,
340
+ "reg29_star_per_seed": [
341
+ 0.10014547921930417
342
+ ]
343
  }
344
  }
results/icml_r3/X29/act/x27rescore_d3fm4n8000.txt CHANGED
@@ -1,23 +1,23 @@
1
  X29-A: X27's C-only arms re-scored, m = 4, tag d3fm4n8000, 120 test plants. J*_A = min(J_ref, 3A id and so3 labels) is below J_ref on 48 plants (median (J_ref - J*_A) / J*_A 0.000%).
2
- diff raw_b1 certified 31.7% (published 31.7%) | cheapest certified: regret vs J_ref 4.28% (published 4.28%), vs J*_A 4.96% | first certified: vs J_ref 4.28%, vs J*_A 4.96%
3
- diff raw_b4 certified 57.2% (published 57.2%) | cheapest certified: regret vs J_ref 4.02% (published 4.02%), vs J*_A 4.74% | first certified: vs J_ref 4.83%, vs J*_A 5.70%
4
- diff raw_b16 certified 78.9% (published 78.9%) | cheapest certified: regret vs J_ref 3.21% (published 3.21%), vs J*_A 3.88% | first certified: vs J_ref 5.34%, vs J*_A 5.94%
5
- diff filt_b1 certified 52.5% (published 52.5%) | cheapest certified: regret vs J_ref 5.26% (published 5.26%), vs J*_A 5.79% | first certified: vs J_ref 5.26%, vs J*_A 5.79%
6
- diff filt_b4 certified 82.5% (published 82.5%) | cheapest certified: regret vs J_ref 4.15% (published 4.15%), vs J*_A 5.10% | first certified: vs J_ref 5.68%, vs J*_A 6.38%
7
- diff filt_b16 certified 91.9% (published 91.9%) | cheapest certified: regret vs J_ref 3.04% (published 3.04%), vs J*_A 3.73% | first certified: vs J_ref 5.68%, vs J*_A 6.38%
8
- diff ref_b16_n3 certified 96.7% (published 96.7%) | cheapest certified: regret vs J_ref 3.22% (published 3.22%), vs J*_A 3.85% | first certified: vs J_ref 5.75%, vs J*_A 6.42%
9
- diff ref_b16_n10 certified 97.8% (published 97.8%) | cheapest certified: regret vs J_ref 3.24% (published 3.24%), vs J*_A 3.85% | first certified: vs J_ref 5.71%, vs J*_A 6.40%
10
- twin raw_b1 certified 31.4% (published 31.4%) | cheapest certified: regret vs J_ref 5.08% (published 5.08%), vs J*_A 5.49% | first certified: vs J_ref 5.08%, vs J*_A 5.49%
11
- twin raw_b4 certified 43.1% (published 43.1%) | cheapest certified: regret vs J_ref 4.08% (published 4.08%), vs J*_A 4.49% | first certified: vs J_ref 4.49%, vs J*_A 5.17%
12
- twin raw_b16 certified 48.3% (published 48.3%) | cheapest certified: regret vs J_ref 3.42% (published 3.42%), vs J*_A 4.00% | first certified: vs J_ref 4.48%, vs J*_A 5.18%
13
- twin filt_b1 certified 36.7% (published 36.7%) | cheapest certified: regret vs J_ref 5.10% (published 5.10%), vs J*_A 5.50% | first certified: vs J_ref 5.10%, vs J*_A 5.50%
14
- twin filt_b4 certified 48.6% (published 48.6%) | cheapest certified: regret vs J_ref 3.99% (published 3.99%), vs J*_A 4.49% | first certified: vs J_ref 4.49%, vs J*_A 5.17%
15
- twin filt_b16 certified 52.2% (published 52.2%) | cheapest certified: regret vs J_ref 3.46% (published 3.46%), vs J*_A 4.00% | first certified: vs J_ref 4.52%, vs J*_A 5.35%
16
- knn raw_b1 certified 14.2% (published 14.2%) | cheapest certified: regret vs J_ref 11.97% (published 11.97%), vs J*_A 12.04% | first certified: vs J_ref 11.97%, vs J*_A 12.04%
17
- knn raw_b4 certified 39.2% (published 39.2%) | cheapest certified: regret vs J_ref 11.97% (published 11.97%), vs J*_A 12.49% | first certified: vs J_ref 12.49%, vs J*_A 13.07%
18
- knn raw_b16 certified 59.2% (published 59.2%) | cheapest certified: regret vs J_ref 10.95% (published 10.95%), vs J*_A 11.62% | first certified: vs J_ref 13.06%, vs J*_A 14.00%
19
- knn filt_b1 certified 45.0% (published 45.0%) | cheapest certified: regret vs J_ref 11.86% (published 11.86%), vs J*_A 12.34% | first certified: vs J_ref 11.86%, vs J*_A 12.34%
20
- knn filt_b4 certified 75.0% (published 75.0%) | cheapest certified: regret vs J_ref 11.10% (published 11.10%), vs J*_A 11.70% | first certified: vs J_ref 13.51%, vs J*_A 14.42%
21
- knn filt_b16 certified 79.2% (published 79.2%) | cheapest certified: regret vs J_ref 9.27% (published 9.27%), vs J*_A 9.92% | first certified: vs J_ref 14.00%, vs J*_A 14.70%
22
- knn ref_b16_n3 certified 90.0% (published 90.0%) | cheapest certified: regret vs J_ref 9.15% (published 9.15%), vs J*_A 9.80% | first certified: vs J_ref 12.78%, vs J*_A 13.98%
23
- knn ref_b16_n10 certified 93.3% (published 93.3%) | cheapest certified: regret vs J_ref 9.03% (published 9.03%), vs J*_A 9.68% | first certified: vs J_ref 12.59%, vs J*_A 13.50%
 
1
  X29-A: X27's C-only arms re-scored, m = 4, tag d3fm4n8000, 120 test plants. J*_A = min(J_ref, 3A id and so3 labels) is below J_ref on 48 plants (median (J_ref - J*_A) / J*_A 0.000%).
2
+ diff raw_b1 certified 31.7% (published 31.7%) | cheapest certified: regret vs J_ref 4.28% (published 4.28%), vs J*_A 4.96% | first certified: vs J_ref 4.28%, vs J*_A 4.96% | X29 convention vs J*_A, per seed: n/a / n/a / n/a
3
+ diff raw_b4 certified 57.2% (published 57.2%) | cheapest certified: regret vs J_ref 4.02% (published 4.02%), vs J*_A 4.74% | first certified: vs J_ref 4.83%, vs J*_A 5.70% | X29 convention vs J*_A, per seed: 8.50% / 8.81% / 11.12%
4
+ diff raw_b16 certified 78.9% (published 78.9%) | cheapest certified: regret vs J_ref 3.21% (published 3.21%), vs J*_A 3.88% | first certified: vs J_ref 5.34%, vs J*_A 5.94% | X29 convention vs J*_A, per seed: 4.73% / 5.76% / 5.26%
5
+ diff filt_b1 certified 52.5% (published 52.5%) | cheapest certified: regret vs J_ref 5.26% (published 5.26%), vs J*_A 5.79% | first certified: vs J_ref 5.26%, vs J*_A 5.79% | X29 convention vs J*_A, per seed: 26.32% / 12.94% / 14.25%
6
+ diff filt_b4 certified 82.5% (published 82.5%) | cheapest certified: regret vs J_ref 4.15% (published 4.15%), vs J*_A 5.10% | first certified: vs J_ref 5.68%, vs J*_A 6.38% | X29 convention vs J*_A, per seed: 6.24% / 6.42% / 6.05%
7
+ diff filt_b16 certified 91.9% (published 91.9%) | cheapest certified: regret vs J_ref 3.04% (published 3.04%), vs J*_A 3.73% | first certified: vs J_ref 5.68%, vs J*_A 6.38% | X29 convention vs J*_A, per seed: 3.86% / 4.41% / 3.88%
8
+ diff ref_b16_n3 certified 96.7% (published 96.7%) | cheapest certified: regret vs J_ref 3.22% (published 3.22%), vs J*_A 3.85% | first certified: vs J_ref 5.75%, vs J*_A 6.42% | X29 convention vs J*_A, per seed: 3.80% / 4.36% / 3.85%
9
+ diff ref_b16_n10 certified 97.8% (published 97.8%) | cheapest certified: regret vs J_ref 3.24% (published 3.24%), vs J*_A 3.85% | first certified: vs J_ref 5.71%, vs J*_A 6.40% | X29 convention vs J*_A, per seed: 3.75% / 4.36% / 3.78%
10
+ twin raw_b1 certified 31.4% (published 31.4%) | cheapest certified: regret vs J_ref 5.08% (published 5.08%), vs J*_A 5.49% | first certified: vs J_ref 5.08%, vs J*_A 5.49% | X29 convention vs J*_A, per seed: n/a / n/a / n/a
11
+ twin raw_b4 certified 43.1% (published 43.1%) | cheapest certified: regret vs J_ref 4.08% (published 4.08%), vs J*_A 4.49% | first certified: vs J_ref 4.49%, vs J*_A 5.17% | X29 convention vs J*_A, per seed: n/a / n/a / n/a
12
+ twin raw_b16 certified 48.3% (published 48.3%) | cheapest certified: regret vs J_ref 3.42% (published 3.42%), vs J*_A 4.00% | first certified: vs J_ref 4.48%, vs J*_A 5.18% | X29 convention vs J*_A, per seed: n/a / n/a / n/a
13
+ twin filt_b1 certified 36.7% (published 36.7%) | cheapest certified: regret vs J_ref 5.10% (published 5.10%), vs J*_A 5.50% | first certified: vs J_ref 5.10%, vs J*_A 5.50% | X29 convention vs J*_A, per seed: n/a / n/a / n/a
14
+ twin filt_b4 certified 48.6% (published 48.6%) | cheapest certified: regret vs J_ref 3.99% (published 3.99%), vs J*_A 4.49% | first certified: vs J_ref 4.49%, vs J*_A 5.17% | X29 convention vs J*_A, per seed: n/a / n/a / n/a
15
+ twin filt_b16 certified 52.2% (published 52.2%) | cheapest certified: regret vs J_ref 3.46% (published 3.46%), vs J*_A 4.00% | first certified: vs J_ref 4.52%, vs J*_A 5.35% | X29 convention vs J*_A, per seed: 13.46% / 13.60% / 9.10%
16
+ knn raw_b1 certified 14.2% (published 14.2%) | cheapest certified: regret vs J_ref 11.97% (published 11.97%), vs J*_A 12.04% | first certified: vs J_ref 11.97%, vs J*_A 12.04% | X29 convention vs J*_A, per seed: n/a
17
+ knn raw_b4 certified 39.2% (published 39.2%) | cheapest certified: regret vs J_ref 11.97% (published 11.97%), vs J*_A 12.49% | first certified: vs J_ref 12.49%, vs J*_A 13.07% | X29 convention vs J*_A, per seed: n/a
18
+ knn raw_b16 certified 59.2% (published 59.2%) | cheapest certified: regret vs J_ref 10.95% (published 10.95%), vs J*_A 11.62% | first certified: vs J_ref 13.06%, vs J*_A 14.00% | X29 convention vs J*_A, per seed: 17.83%
19
+ knn filt_b1 certified 45.0% (published 45.0%) | cheapest certified: regret vs J_ref 11.86% (published 11.86%), vs J*_A 12.34% | first certified: vs J_ref 11.86%, vs J*_A 12.34% | X29 convention vs J*_A, per seed: n/a
20
+ knn filt_b4 certified 75.0% (published 75.0%) | cheapest certified: regret vs J_ref 11.10% (published 11.10%), vs J*_A 11.70% | first certified: vs J_ref 13.51%, vs J*_A 14.42% | X29 convention vs J*_A, per seed: 14.89%
21
+ knn filt_b16 certified 79.2% (published 79.2%) | cheapest certified: regret vs J_ref 9.27% (published 9.27%), vs J*_A 9.92% | first certified: vs J_ref 14.00%, vs J*_A 14.70% | X29 convention vs J*_A, per seed: 11.16%
22
+ knn ref_b16_n3 certified 90.0% (published 90.0%) | cheapest certified: regret vs J_ref 9.15% (published 9.15%), vs J*_A 9.80% | first certified: vs J_ref 12.78%, vs J*_A 13.98% | X29 convention vs J*_A, per seed: 10.58%
23
+ knn ref_b16_n10 certified 93.3% (published 93.3%) | cheapest certified: regret vs J_ref 9.03% (published 9.03%), vs J*_A 9.68% | first certified: vs J_ref 12.59%, vs J*_A 13.50% | X29 convention vs J*_A, per seed: 10.01%
results/icml_r3/X29/act/x27rescore_d3fm5n16000.json CHANGED
@@ -9,7 +9,12 @@
9
  "regB_ref": 0.054012242469847704,
10
  "regB_star": 0.06580516710136042,
11
  "regA_ref": 0.054012242469847704,
12
- "regA_star": 0.06580516710136042
 
 
 
 
 
13
  },
14
  "diff raw_b4": {
15
  "rate": 0.49166666666666664,
@@ -21,7 +26,12 @@
21
  "regB_ref": 0.046037766193027574,
22
  "regB_star": 0.055668722380055824,
23
  "regA_ref": 0.05123443031809827,
24
- "regA_star": 0.06242150229911836
 
 
 
 
 
25
  },
26
  "diff raw_b16": {
27
  "rate": 0.7916666666666666,
@@ -33,7 +43,12 @@
33
  "regB_ref": 0.0392393348275164,
34
  "regB_star": 0.04843269982130982,
35
  "regA_ref": 0.053670339354187746,
36
- "regA_star": 0.0653423576378871
 
 
 
 
 
37
  },
38
  "diff filt_b1": {
39
  "rate": 0.4583333333333333,
@@ -45,7 +60,12 @@
45
  "regB_ref": 0.05194845381582125,
46
  "regB_star": 0.06513679765481717,
47
  "regA_ref": 0.05194845381582125,
48
- "regA_star": 0.06513679765481717
 
 
 
 
 
49
  },
50
  "diff filt_b4": {
51
  "rate": 0.8138888888888888,
@@ -57,7 +77,12 @@
57
  "regB_ref": 0.044054119434560945,
58
  "regB_star": 0.05117955925012052,
59
  "regA_ref": 0.053670339354187746,
60
- "regA_star": 0.06526846423539756
 
 
 
 
 
61
  },
62
  "diff filt_b16": {
63
  "rate": 0.9083333333333333,
@@ -69,7 +94,12 @@
69
  "regB_ref": 0.033760278025647406,
70
  "regB_star": 0.0392393348275164,
71
  "regA_ref": 0.05494152341582326,
72
- "regA_star": 0.06614503852117859
 
 
 
 
 
73
  },
74
  "diff ref_b16_n3": {
75
  "rate": 0.9638888888888889,
@@ -81,7 +111,12 @@
81
  "regB_ref": 0.034359315053998474,
82
  "regB_star": 0.039602163709273386,
83
  "regA_ref": 0.056236553327202676,
84
- "regA_star": 0.06713280900832053
 
 
 
 
 
85
  },
86
  "diff ref_b16_n10": {
87
  "rate": 0.9805555555555556,
@@ -93,7 +128,12 @@
93
  "regB_ref": 0.03437394402248328,
94
  "regB_star": 0.040009974164940454,
95
  "regA_ref": 0.05649656778792678,
96
- "regA_star": 0.06751706328759521
 
 
 
 
 
97
  },
98
  "twin raw_b1": {
99
  "rate": 0.24722222222222223,
@@ -105,7 +145,12 @@
105
  "regB_ref": 0.04588446769434418,
106
  "regB_star": 0.05281368503715744,
107
  "regA_ref": 0.04588446769434418,
108
- "regA_star": 0.05281368503715744
 
 
 
 
 
109
  },
110
  "twin raw_b4": {
111
  "rate": 0.37777777777777777,
@@ -117,7 +162,12 @@
117
  "regB_ref": 0.03958246856186798,
118
  "regB_star": 0.046969980502976294,
119
  "regA_ref": 0.04453505836815008,
120
- "regA_star": 0.05310045978676048
 
 
 
 
 
121
  },
122
  "twin raw_b16": {
123
  "rate": 0.4916666666666667,
@@ -129,7 +179,12 @@
129
  "regB_ref": 0.03635521260826646,
130
  "regB_star": 0.046204610388273615,
131
  "regA_ref": 0.04846464655537219,
132
- "regA_star": 0.05641944870545168
 
 
 
 
 
133
  },
134
  "twin filt_b1": {
135
  "rate": 0.3416666666666666,
@@ -141,7 +196,12 @@
141
  "regB_ref": 0.04659602299391885,
142
  "regB_star": 0.053387234536363515,
143
  "regA_ref": 0.04659602299391885,
144
- "regA_star": 0.053387234536363515
 
 
 
 
 
145
  },
146
  "twin filt_b4": {
147
  "rate": 0.4833333333333334,
@@ -153,7 +213,12 @@
153
  "regB_ref": 0.04074265857517956,
154
  "regB_star": 0.048658098886397605,
155
  "regA_ref": 0.046240245344131514,
156
- "regA_star": 0.056035186950899485
 
 
 
 
 
157
  },
158
  "twin filt_b16": {
159
  "rate": 0.5555555555555556,
@@ -165,7 +230,12 @@
165
  "regB_ref": 0.03570862209728387,
166
  "regB_star": 0.04591678068291327,
167
  "regA_ref": 0.05002032485737595,
168
- "regA_star": 0.057383941917565906
 
 
 
 
 
169
  },
170
  "knn raw_b1": {
171
  "rate": 0.09166666666666666,
@@ -175,7 +245,10 @@
175
  "regB_ref": 0.18029208813957154,
176
  "regB_star": 0.18029208813957154,
177
  "regA_ref": 0.18029208813957154,
178
- "regA_star": 0.18029208813957154
 
 
 
179
  },
180
  "knn raw_b4": {
181
  "rate": 0.24166666666666667,
@@ -185,7 +258,10 @@
185
  "regB_ref": 0.16093589699487887,
186
  "regB_star": 0.17212171903802753,
187
  "regA_ref": 0.1725441277384172,
188
- "regA_star": 0.18523628247042523
 
 
 
189
  },
190
  "knn raw_b16": {
191
  "rate": 0.5166666666666667,
@@ -195,7 +271,10 @@
195
  "regB_ref": 0.1327768182319501,
196
  "regB_star": 0.14350847619758267,
197
  "regA_ref": 0.14856354038966635,
198
- "regA_star": 0.16313087614552568
 
 
 
199
  },
200
  "knn filt_b1": {
201
  "rate": 0.3,
@@ -205,7 +284,10 @@
205
  "regB_ref": 0.17428865396751858,
206
  "regB_star": 0.18276418530499838,
207
  "regA_ref": 0.17428865396751858,
208
- "regA_star": 0.18276418530499838
 
 
 
209
  },
210
  "knn filt_b4": {
211
  "rate": 0.6666666666666666,
@@ -215,7 +297,10 @@
215
  "regB_ref": 0.12985794400765538,
216
  "regB_star": 0.13483477262840815,
217
  "regA_ref": 0.1620537009612174,
218
- "regA_star": 0.16564434998790312
 
 
 
219
  },
220
  "knn filt_b16": {
221
  "rate": 0.725,
@@ -225,7 +310,10 @@
225
  "regB_ref": 0.10686761298831438,
226
  "regB_star": 0.11477117848557294,
227
  "regA_ref": 0.16317150492755594,
228
- "regA_star": 0.16567695056159626
 
 
 
229
  },
230
  "knn ref_b16_n3": {
231
  "rate": 0.9,
@@ -235,7 +323,10 @@
235
  "regB_ref": 0.10400741103859301,
236
  "regB_star": 0.1113356454533998,
237
  "regA_ref": 0.13930659114267874,
238
- "regA_star": 0.15300463877630077
 
 
 
239
  },
240
  "knn ref_b16_n10": {
241
  "rate": 0.925,
@@ -245,6 +336,9 @@
245
  "regB_ref": 0.10343356710695706,
246
  "regB_star": 0.10973883506945237,
247
  "regA_ref": 0.13558071904802294,
248
- "regA_star": 0.14891194803545904
 
 
 
249
  }
250
  }
 
9
  "regB_ref": 0.054012242469847704,
10
  "regB_star": 0.06580516710136042,
11
  "regA_ref": 0.054012242469847704,
12
+ "regA_star": 0.06580516710136042,
13
+ "reg29_star_per_seed": [
14
+ 1000.0,
15
+ 1000.0,
16
+ 1000.0
17
+ ]
18
  },
19
  "diff raw_b4": {
20
  "rate": 0.49166666666666664,
 
26
  "regB_ref": 0.046037766193027574,
27
  "regB_star": 0.055668722380055824,
28
  "regA_ref": 0.05123443031809827,
29
+ "regA_star": 0.06242150229911836,
30
+ "reg29_star_per_seed": [
31
+ 0.19219722967026764,
32
+ 0.10473329948011612,
33
+ 1000.0
34
+ ]
35
  },
36
  "diff raw_b16": {
37
  "rate": 0.7916666666666666,
 
43
  "regB_ref": 0.0392393348275164,
44
  "regB_star": 0.04843269982130982,
45
  "regA_ref": 0.053670339354187746,
46
+ "regA_star": 0.0653423576378871,
47
+ "reg29_star_per_seed": [
48
+ 0.056011538175283704,
49
+ 0.05579918124137806,
50
+ 0.0638180447639215
51
+ ]
52
  },
53
  "diff filt_b1": {
54
  "rate": 0.4583333333333333,
 
60
  "regB_ref": 0.05194845381582125,
61
  "regB_star": 0.06513679765481717,
62
  "regA_ref": 0.05194845381582125,
63
+ "regA_star": 0.06513679765481717,
64
+ "reg29_star_per_seed": [
65
+ 1000.0,
66
+ 1000.0,
67
+ 1000.0
68
+ ]
69
  },
70
  "diff filt_b4": {
71
  "rate": 0.8138888888888888,
 
77
  "regB_ref": 0.044054119434560945,
78
  "regB_star": 0.05117955925012052,
79
  "regA_ref": 0.053670339354187746,
80
+ "regA_star": 0.06526846423539756,
81
+ "reg29_star_per_seed": [
82
+ 0.057999843375380555,
83
+ 0.06110192291621441,
84
+ 0.06608581358546517
85
+ ]
86
  },
87
  "diff filt_b16": {
88
  "rate": 0.9083333333333333,
 
94
  "regB_ref": 0.033760278025647406,
95
  "regB_star": 0.0392393348275164,
96
  "regA_ref": 0.05494152341582326,
97
+ "regA_star": 0.06614503852117859,
98
+ "reg29_star_per_seed": [
99
+ 0.041296215258291746,
100
+ 0.039501202588679285,
101
+ 0.04825939970362869
102
+ ]
103
  },
104
  "diff ref_b16_n3": {
105
  "rate": 0.9638888888888889,
 
111
  "regB_ref": 0.034359315053998474,
112
  "regB_star": 0.039602163709273386,
113
  "regA_ref": 0.056236553327202676,
114
+ "regA_star": 0.06713280900832053,
115
+ "reg29_star_per_seed": [
116
+ 0.041296215258291746,
117
+ 0.03844028224462015,
118
+ 0.04449522945084139
119
+ ]
120
  },
121
  "diff ref_b16_n10": {
122
  "rate": 0.9805555555555556,
 
128
  "regB_ref": 0.03437394402248328,
129
  "regB_star": 0.040009974164940454,
130
  "regA_ref": 0.05649656778792678,
131
+ "regA_star": 0.06751706328759521,
132
+ "reg29_star_per_seed": [
133
+ 0.041296215258291746,
134
+ 0.037411175602180435,
135
+ 0.04245451078867202
136
+ ]
137
  },
138
  "twin raw_b1": {
139
  "rate": 0.24722222222222223,
 
145
  "regB_ref": 0.04588446769434418,
146
  "regB_star": 0.05281368503715744,
147
  "regA_ref": 0.04588446769434418,
148
+ "regA_star": 0.05281368503715744,
149
+ "reg29_star_per_seed": [
150
+ 1000.0,
151
+ 1000.0,
152
+ 1000.0
153
+ ]
154
  },
155
  "twin raw_b4": {
156
  "rate": 0.37777777777777777,
 
162
  "regB_ref": 0.03958246856186798,
163
  "regB_star": 0.046969980502976294,
164
  "regA_ref": 0.04453505836815008,
165
+ "regA_star": 0.05310045978676048,
166
+ "reg29_star_per_seed": [
167
+ 1000.0,
168
+ 1000.0,
169
+ 1000.0
170
+ ]
171
  },
172
  "twin raw_b16": {
173
  "rate": 0.4916666666666667,
 
179
  "regB_ref": 0.03635521260826646,
180
  "regB_star": 0.046204610388273615,
181
  "regA_ref": 0.04846464655537219,
182
+ "regA_star": 0.05641944870545168,
183
+ "reg29_star_per_seed": [
184
+ 0.10244921887598657,
185
+ 1000.0,
186
+ 0.11581804604445223
187
+ ]
188
  },
189
  "twin filt_b1": {
190
  "rate": 0.3416666666666666,
 
196
  "regB_ref": 0.04659602299391885,
197
  "regB_star": 0.053387234536363515,
198
  "regA_ref": 0.04659602299391885,
199
+ "regA_star": 0.053387234536363515,
200
+ "reg29_star_per_seed": [
201
+ 1000.0,
202
+ 1000.0,
203
+ 1000.0
204
+ ]
205
  },
206
  "twin filt_b4": {
207
  "rate": 0.4833333333333334,
 
213
  "regB_ref": 0.04074265857517956,
214
  "regB_star": 0.048658098886397605,
215
  "regA_ref": 0.046240245344131514,
216
+ "regA_star": 0.056035186950899485,
217
+ "reg29_star_per_seed": [
218
+ 0.09849242387393675,
219
+ 1000.0,
220
+ 1000.0
221
+ ]
222
  },
223
  "twin filt_b16": {
224
  "rate": 0.5555555555555556,
 
230
  "regB_ref": 0.03570862209728387,
231
  "regB_star": 0.04591678068291327,
232
  "regA_ref": 0.05002032485737595,
233
+ "regA_star": 0.057383941917565906,
234
+ "reg29_star_per_seed": [
235
+ 0.085614514090114,
236
+ 0.10824800868369489,
237
+ 0.0945664020595226
238
+ ]
239
  },
240
  "knn raw_b1": {
241
  "rate": 0.09166666666666666,
 
245
  "regB_ref": 0.18029208813957154,
246
  "regB_star": 0.18029208813957154,
247
  "regA_ref": 0.18029208813957154,
248
+ "regA_star": 0.18029208813957154,
249
+ "reg29_star_per_seed": [
250
+ 1000.0
251
+ ]
252
  },
253
  "knn raw_b4": {
254
  "rate": 0.24166666666666667,
 
258
  "regB_ref": 0.16093589699487887,
259
  "regB_star": 0.17212171903802753,
260
  "regA_ref": 0.1725441277384172,
261
+ "regA_star": 0.18523628247042523,
262
+ "reg29_star_per_seed": [
263
+ 1000.0
264
+ ]
265
  },
266
  "knn raw_b16": {
267
  "rate": 0.5166666666666667,
 
271
  "regB_ref": 0.1327768182319501,
272
  "regB_star": 0.14350847619758267,
273
  "regA_ref": 0.14856354038966635,
274
+ "regA_star": 0.16313087614552568,
275
+ "reg29_star_per_seed": [
276
+ 0.3633192275090701
277
+ ]
278
  },
279
  "knn filt_b1": {
280
  "rate": 0.3,
 
284
  "regB_ref": 0.17428865396751858,
285
  "regB_star": 0.18276418530499838,
286
  "regA_ref": 0.17428865396751858,
287
+ "regA_star": 0.18276418530499838,
288
+ "reg29_star_per_seed": [
289
+ 1000.0
290
+ ]
291
  },
292
  "knn filt_b4": {
293
  "rate": 0.6666666666666666,
 
297
  "regB_ref": 0.12985794400765538,
298
  "regB_star": 0.13483477262840815,
299
  "regA_ref": 0.1620537009612174,
300
+ "regA_star": 0.16564434998790312,
301
+ "reg29_star_per_seed": [
302
+ 0.17692463714776696
303
+ ]
304
  },
305
  "knn filt_b16": {
306
  "rate": 0.725,
 
310
  "regB_ref": 0.10686761298831438,
311
  "regB_star": 0.11477117848557294,
312
  "regA_ref": 0.16317150492755594,
313
+ "regA_star": 0.16567695056159626,
314
+ "reg29_star_per_seed": [
315
+ 0.1354804464718561
316
+ ]
317
  },
318
  "knn ref_b16_n3": {
319
  "rate": 0.9,
 
323
  "regB_ref": 0.10400741103859301,
324
  "regB_star": 0.1113356454533998,
325
  "regA_ref": 0.13930659114267874,
326
+ "regA_star": 0.15300463877630077,
327
+ "reg29_star_per_seed": [
328
+ 0.1173305502662011
329
+ ]
330
  },
331
  "knn ref_b16_n10": {
332
  "rate": 0.925,
 
336
  "regB_ref": 0.10343356710695706,
337
  "regB_star": 0.10973883506945237,
338
  "regA_ref": 0.13558071904802294,
339
+ "regA_star": 0.14891194803545904,
340
+ "reg29_star_per_seed": [
341
+ 0.11506558365541786
342
+ ]
343
  }
344
  }
results/icml_r3/X29/act/x27rescore_d3fm5n16000.txt CHANGED
@@ -1,23 +1,23 @@
1
  X29-A: X27's C-only arms re-scored, m = 5, tag d3fm5n16000, 120 test plants. J*_A = min(J_ref, 3A id and so3 labels) is below J_ref on 43 plants (median (J_ref - J*_A) / J*_A 0.000%).
2
- diff raw_b1 certified 23.6% (published 23.6%) | cheapest certified: regret vs J_ref 5.40% (published 5.40%), vs J*_A 6.58% | first certified: vs J_ref 5.40%, vs J*_A 6.58%
3
- diff raw_b4 certified 49.2% (published 49.2%) | cheapest certified: regret vs J_ref 4.60% (published 4.60%), vs J*_A 5.57% | first certified: vs J_ref 5.12%, vs J*_A 6.24%
4
- diff raw_b16 certified 79.2% (published 79.2%) | cheapest certified: regret vs J_ref 3.92% (published 3.92%), vs J*_A 4.84% | first certified: vs J_ref 5.37%, vs J*_A 6.53%
5
- diff filt_b1 certified 45.8% (published 45.8%) | cheapest certified: regret vs J_ref 5.19% (published 5.19%), vs J*_A 6.51% | first certified: vs J_ref 5.19%, vs J*_A 6.51%
6
- diff filt_b4 certified 81.4% (published 81.4%) | cheapest certified: regret vs J_ref 4.41% (published 4.41%), vs J*_A 5.12% | first certified: vs J_ref 5.37%, vs J*_A 6.53%
7
- diff filt_b16 certified 90.8% (published 90.8%) | cheapest certified: regret vs J_ref 3.38% (published 3.38%), vs J*_A 3.92% | first certified: vs J_ref 5.49%, vs J*_A 6.61%
8
- diff ref_b16_n3 certified 96.4% (published 96.4%) | cheapest certified: regret vs J_ref 3.44% (published 3.44%), vs J*_A 3.96% | first certified: vs J_ref 5.62%, vs J*_A 6.71%
9
- diff ref_b16_n10 certified 98.1% (published 98.1%) | cheapest certified: regret vs J_ref 3.44% (published 3.44%), vs J*_A 4.00% | first certified: vs J_ref 5.65%, vs J*_A 6.75%
10
- twin raw_b1 certified 24.7% (published 24.7%) | cheapest certified: regret vs J_ref 4.59% (published 4.59%), vs J*_A 5.28% | first certified: vs J_ref 4.59%, vs J*_A 5.28%
11
- twin raw_b4 certified 37.8% (published 37.8%) | cheapest certified: regret vs J_ref 3.96% (published 3.96%), vs J*_A 4.70% | first certified: vs J_ref 4.45%, vs J*_A 5.31%
12
- twin raw_b16 certified 49.2% (published 49.2%) | cheapest certified: regret vs J_ref 3.64% (published 3.64%), vs J*_A 4.62% | first certified: vs J_ref 4.85%, vs J*_A 5.64%
13
- twin filt_b1 certified 34.2% (published 34.2%) | cheapest certified: regret vs J_ref 4.66% (published 4.66%), vs J*_A 5.34% | first certified: vs J_ref 4.66%, vs J*_A 5.34%
14
- twin filt_b4 certified 48.3% (published 48.3%) | cheapest certified: regret vs J_ref 4.07% (published 4.07%), vs J*_A 4.87% | first certified: vs J_ref 4.62%, vs J*_A 5.60%
15
- twin filt_b16 certified 55.6% (published 55.6%) | cheapest certified: regret vs J_ref 3.57% (published 3.57%), vs J*_A 4.59% | first certified: vs J_ref 5.00%, vs J*_A 5.74%
16
- knn raw_b1 certified 9.2% (published 9.2%) | cheapest certified: regret vs J_ref 18.03% (published 18.03%), vs J*_A 18.03% | first certified: vs J_ref 18.03%, vs J*_A 18.03%
17
- knn raw_b4 certified 24.2% (published 24.2%) | cheapest certified: regret vs J_ref 16.09% (published 16.09%), vs J*_A 17.21% | first certified: vs J_ref 17.25%, vs J*_A 18.52%
18
- knn raw_b16 certified 51.7% (published 51.7%) | cheapest certified: regret vs J_ref 13.28% (published 13.28%), vs J*_A 14.35% | first certified: vs J_ref 14.86%, vs J*_A 16.31%
19
- knn filt_b1 certified 30.0% (published 30.0%) | cheapest certified: regret vs J_ref 17.43% (published 17.43%), vs J*_A 18.28% | first certified: vs J_ref 17.43%, vs J*_A 18.28%
20
- knn filt_b4 certified 66.7% (published 66.7%) | cheapest certified: regret vs J_ref 12.99% (published 12.99%), vs J*_A 13.48% | first certified: vs J_ref 16.21%, vs J*_A 16.56%
21
- knn filt_b16 certified 72.5% (published 72.5%) | cheapest certified: regret vs J_ref 10.69% (published 10.69%), vs J*_A 11.48% | first certified: vs J_ref 16.32%, vs J*_A 16.57%
22
- knn ref_b16_n3 certified 90.0% (published 90.0%) | cheapest certified: regret vs J_ref 10.40% (published 10.40%), vs J*_A 11.13% | first certified: vs J_ref 13.93%, vs J*_A 15.30%
23
- knn ref_b16_n10 certified 92.5% (published 92.5%) | cheapest certified: regret vs J_ref 10.34% (published 10.34%), vs J*_A 10.97% | first certified: vs J_ref 13.56%, vs J*_A 14.89%
 
1
  X29-A: X27's C-only arms re-scored, m = 5, tag d3fm5n16000, 120 test plants. J*_A = min(J_ref, 3A id and so3 labels) is below J_ref on 43 plants (median (J_ref - J*_A) / J*_A 0.000%).
2
+ diff raw_b1 certified 23.6% (published 23.6%) | cheapest certified: regret vs J_ref 5.40% (published 5.40%), vs J*_A 6.58% | first certified: vs J_ref 5.40%, vs J*_A 6.58% | X29 convention vs J*_A, per seed: n/a / n/a / n/a
3
+ diff raw_b4 certified 49.2% (published 49.2%) | cheapest certified: regret vs J_ref 4.60% (published 4.60%), vs J*_A 5.57% | first certified: vs J_ref 5.12%, vs J*_A 6.24% | X29 convention vs J*_A, per seed: 19.22% / 10.47% / n/a
4
+ diff raw_b16 certified 79.2% (published 79.2%) | cheapest certified: regret vs J_ref 3.92% (published 3.92%), vs J*_A 4.84% | first certified: vs J_ref 5.37%, vs J*_A 6.53% | X29 convention vs J*_A, per seed: 5.60% / 5.58% / 6.38%
5
+ diff filt_b1 certified 45.8% (published 45.8%) | cheapest certified: regret vs J_ref 5.19% (published 5.19%), vs J*_A 6.51% | first certified: vs J_ref 5.19%, vs J*_A 6.51% | X29 convention vs J*_A, per seed: n/a / n/a / n/a
6
+ diff filt_b4 certified 81.4% (published 81.4%) | cheapest certified: regret vs J_ref 4.41% (published 4.41%), vs J*_A 5.12% | first certified: vs J_ref 5.37%, vs J*_A 6.53% | X29 convention vs J*_A, per seed: 5.80% / 6.11% / 6.61%
7
+ diff filt_b16 certified 90.8% (published 90.8%) | cheapest certified: regret vs J_ref 3.38% (published 3.38%), vs J*_A 3.92% | first certified: vs J_ref 5.49%, vs J*_A 6.61% | X29 convention vs J*_A, per seed: 4.13% / 3.95% / 4.83%
8
+ diff ref_b16_n3 certified 96.4% (published 96.4%) | cheapest certified: regret vs J_ref 3.44% (published 3.44%), vs J*_A 3.96% | first certified: vs J_ref 5.62%, vs J*_A 6.71% | X29 convention vs J*_A, per seed: 4.13% / 3.84% / 4.45%
9
+ diff ref_b16_n10 certified 98.1% (published 98.1%) | cheapest certified: regret vs J_ref 3.44% (published 3.44%), vs J*_A 4.00% | first certified: vs J_ref 5.65%, vs J*_A 6.75% | X29 convention vs J*_A, per seed: 4.13% / 3.74% / 4.25%
10
+ twin raw_b1 certified 24.7% (published 24.7%) | cheapest certified: regret vs J_ref 4.59% (published 4.59%), vs J*_A 5.28% | first certified: vs J_ref 4.59%, vs J*_A 5.28% | X29 convention vs J*_A, per seed: n/a / n/a / n/a
11
+ twin raw_b4 certified 37.8% (published 37.8%) | cheapest certified: regret vs J_ref 3.96% (published 3.96%), vs J*_A 4.70% | first certified: vs J_ref 4.45%, vs J*_A 5.31% | X29 convention vs J*_A, per seed: n/a / n/a / n/a
12
+ twin raw_b16 certified 49.2% (published 49.2%) | cheapest certified: regret vs J_ref 3.64% (published 3.64%), vs J*_A 4.62% | first certified: vs J_ref 4.85%, vs J*_A 5.64% | X29 convention vs J*_A, per seed: 10.24% / n/a / 11.58%
13
+ twin filt_b1 certified 34.2% (published 34.2%) | cheapest certified: regret vs J_ref 4.66% (published 4.66%), vs J*_A 5.34% | first certified: vs J_ref 4.66%, vs J*_A 5.34% | X29 convention vs J*_A, per seed: n/a / n/a / n/a
14
+ twin filt_b4 certified 48.3% (published 48.3%) | cheapest certified: regret vs J_ref 4.07% (published 4.07%), vs J*_A 4.87% | first certified: vs J_ref 4.62%, vs J*_A 5.60% | X29 convention vs J*_A, per seed: 9.85% / n/a / n/a
15
+ twin filt_b16 certified 55.6% (published 55.6%) | cheapest certified: regret vs J_ref 3.57% (published 3.57%), vs J*_A 4.59% | first certified: vs J_ref 5.00%, vs J*_A 5.74% | X29 convention vs J*_A, per seed: 8.56% / 10.82% / 9.46%
16
+ knn raw_b1 certified 9.2% (published 9.2%) | cheapest certified: regret vs J_ref 18.03% (published 18.03%), vs J*_A 18.03% | first certified: vs J_ref 18.03%, vs J*_A 18.03% | X29 convention vs J*_A, per seed: n/a
17
+ knn raw_b4 certified 24.2% (published 24.2%) | cheapest certified: regret vs J_ref 16.09% (published 16.09%), vs J*_A 17.21% | first certified: vs J_ref 17.25%, vs J*_A 18.52% | X29 convention vs J*_A, per seed: n/a
18
+ knn raw_b16 certified 51.7% (published 51.7%) | cheapest certified: regret vs J_ref 13.28% (published 13.28%), vs J*_A 14.35% | first certified: vs J_ref 14.86%, vs J*_A 16.31% | X29 convention vs J*_A, per seed: 36.33%
19
+ knn filt_b1 certified 30.0% (published 30.0%) | cheapest certified: regret vs J_ref 17.43% (published 17.43%), vs J*_A 18.28% | first certified: vs J_ref 17.43%, vs J*_A 18.28% | X29 convention vs J*_A, per seed: n/a
20
+ knn filt_b4 certified 66.7% (published 66.7%) | cheapest certified: regret vs J_ref 12.99% (published 12.99%), vs J*_A 13.48% | first certified: vs J_ref 16.21%, vs J*_A 16.56% | X29 convention vs J*_A, per seed: 17.69%
21
+ knn filt_b16 certified 72.5% (published 72.5%) | cheapest certified: regret vs J_ref 10.69% (published 10.69%), vs J*_A 11.48% | first certified: vs J_ref 16.32%, vs J*_A 16.57% | X29 convention vs J*_A, per seed: 13.55%
22
+ knn ref_b16_n3 certified 90.0% (published 90.0%) | cheapest certified: regret vs J_ref 10.40% (published 10.40%), vs J*_A 11.13% | first certified: vs J_ref 13.93%, vs J*_A 15.30% | X29 convention vs J*_A, per seed: 11.73%
23
+ knn ref_b16_n10 certified 92.5% (published 92.5%) | cheapest certified: regret vs J_ref 10.34% (published 10.34%), vs J*_A 10.97% | first certified: vs J_ref 13.56%, vs J*_A 14.89% | X29 convention vs J*_A, per seed: 11.51%
results/icml_r3/X29/siso/report.txt CHANGED
@@ -78,4 +78,25 @@ uncertified = 1e3; certified % / median regret [bootstrap 95% CI]; clip 1.5, t-r
78
  equivalence of the batched optimizers: C+phi 25 steps, plants [0, 1, 2]: J batched [0.089508 0.472122 0.018839] single [0.089508 0.472122 0.018839] used [1 1 0] [1 1 0] max rel diff 0.00e+00 -> ok
79
 
80
  == CPU (timing job)
81
- not run yet
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
78
  equivalence of the batched optimizers: C+phi 25 steps, plants [0, 1, 2]: J batched [0.089508 0.472122 0.018839] single [0.089508 0.472122 0.018839] used [1 1 0] [1 1 0] max rel diff 0.00e+00 -> ok
79
 
80
  == CPU (timing job)
81
+ medians over timing plants (one thread each): fopt1 1553.8 s (n=10), fopt16 1686.5 s (n=10), fopt4 1518.1 s (n=10), gen_C+phi 29.1 s (n=10), gen_C-only 11.5 s (n=10), hifoo2 1.0 s (n=10), jopt1 1537.3 s (n=10), jopt16 1749.6 s (n=10), jopt4 1545.7 s (n=10), knn_C+phi 7.6 s (n=10), knn_C-only 9.5 s (n=10), twin_C+phi 17.2 s (n=10), twin_C-only 1.9 s (n=10)
82
+ C-only generator b=16 CPU 11.5 s | median regret 6.79%
83
+ C+phi generator b=16 CPU 29.1 s | median regret 7.14%
84
+ C-only generator b=1 CPU 0.7 s | median regret 54.90%
85
+ C+phi generator b=1 CPU 1.8 s | median regret 100000.00%
86
+ F-only optimizer 1 starts CPU 1554.8 s | median regret 35.91%
87
+ joint optimizer 1 starts CPU 3092.1 s | median regret 9.13%
88
+ F-only optimizer 2 starts CPU 1543.8 s | median regret 13.55%
89
+ joint optimizer 2 starts CPU 3083.9 s | median regret 4.39%
90
+ F-only optimizer 4 starts CPU 1522.0 s | median regret 8.11%
91
+ joint optimizer 4 starts CPU 3067.7 s | median regret 0.92%
92
+ F-only optimizer 8 starts CPU 1581.9 s | median regret 4.19%
93
+ joint optimizer 8 starts CPU 3195.6 s | median regret 0.00%
94
+ F-only optimizer 16 starts CPU 1701.8 s | median regret 3.12%
95
+ joint optimizer 16 starts CPU 3451.4 s | median regret 0.00%
96
+ fixed-order stabilization (HIFOO best of 16) CPU 15.3 s | median regret 50.75%
97
+ C-only generator 16 + 1 optimizer start CPU 1565.3 s | median regret 2.60%
98
+ C+phi generator 16 + 1 optimizer start CPU 1566.4 s | median regret 0.01%
99
+ [38] C-only: F-only optimizer reaches the generator's 6.79% at 8 starts, 1582 s against 11.5 s: ratio 138x
100
+ [38] C+phi: joint optimizer reaches the generator's 7.14% at 2 starts, 3084 s against 29.1 s: ratio 106x
101
+ C-only kNN (retrieval + scoring, 16) 9.5 s; twin (sampling + scoring, 16) 1.9 s
102
+ C+phi kNN (retrieval + scoring, 16) 7.6 s; twin (sampling + scoring, 16) 17.2 s
results/icml_r3/X29/siso/timing/plant_000_fopt1.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 0, "method": "fopt1", "t": 1610.2393736839294, "k": 1, "load": [6.0107421875, 6.0, 5.39453125]}
results/icml_r3/X29/siso/timing/plant_000_fopt16.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 0, "method": "fopt16", "t": 1693.0486862659454, "k": 16, "load": [6.0, 6.0, 6.0]}
results/icml_r3/X29/siso/timing/plant_000_fopt4.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 0, "method": "fopt4", "t": 1529.6605348587036, "k": 4, "load": [6.0, 6.0, 5.9189453125]}
results/icml_r3/X29/siso/timing/plant_000_gen_C+phi.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 0, "method": "gen_C+phi", "t_sample": 8.641952991485596, "t_score": 20.20849061012268, "t": 28.850443601608276, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
results/icml_r3/X29/siso/timing/plant_000_gen_C-only.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 0, "method": "gen_C-only", "t_sample": 8.723515272140503, "t_score": 1.421351671218872, "t": 10.144866943359375, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
results/icml_r3/X29/siso/timing/plant_000_hifoo2.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 0, "method": "hifoo2", "t": 7.0185874700546265, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
results/icml_r3/X29/siso/timing/plant_000_jopt1.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 0, "method": "jopt1", "t": 1479.6743664741516, "k": 1, "load": [6.0, 6.0, 6.0]}
results/icml_r3/X29/siso/timing/plant_000_jopt16.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 0, "method": "jopt16", "t": 1720.920538187027, "k": 16, "load": [6.0, 6.0, 6.0]}
results/icml_r3/X29/siso/timing/plant_000_jopt4.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 0, "method": "jopt4", "t": 1511.4163365364075, "k": 4, "load": [6.0, 6.0, 6.0]}
results/icml_r3/X29/siso/timing/plant_000_knn_C+phi.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 0, "method": "knn_C+phi", "t_sample": 0.005944490432739258, "t_score": 7.443649768829346, "t": 7.449594259262085, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
results/icml_r3/X29/siso/timing/plant_000_knn_C-only.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 0, "method": "knn_C-only", "t_sample": 2.10442852973938, "t_score": 1.0252888202667236, "t": 3.1297173500061035, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
results/icml_r3/X29/siso/timing/plant_000_twin_C+phi.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 0, "method": "twin_C+phi", "t_sample": 0.23180317878723145, "t_score": 0.18750786781311035, "t": 0.4193110466003418, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
results/icml_r3/X29/siso/timing/plant_000_twin_C-only.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 0, "method": "twin_C-only", "t_sample": 0.25113391876220703, "t_score": 0.08510637283325195, "t": 0.336240291595459, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
results/icml_r3/X29/siso/timing/plant_021_fopt1.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 21, "method": "fopt1", "t": 1603.8419466018677, "k": 1, "load": [6.0, 6.0, 5.9150390625]}
results/icml_r3/X29/siso/timing/plant_021_fopt16.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 21, "method": "fopt16", "t": 1609.1883926391602, "k": 16, "load": [6.0, 6.0, 6.0]}
results/icml_r3/X29/siso/timing/plant_021_fopt4.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 21, "method": "fopt4", "t": 1512.834814786911, "k": 4, "load": [6.0, 6.0, 6.0]}
results/icml_r3/X29/siso/timing/plant_021_gen_C+phi.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 21, "method": "gen_C+phi", "t_sample": 8.728589296340942, "t_score": 21.289788722991943, "t": 30.018378019332886, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
results/icml_r3/X29/siso/timing/plant_021_gen_C-only.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 21, "method": "gen_C-only", "t_sample": 8.670607328414917, "t_score": 3.4282965660095215, "t": 12.098903894424438, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
results/icml_r3/X29/siso/timing/plant_021_hifoo2.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 21, "method": "hifoo2", "t": 0.7439807653427124, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
results/icml_r3/X29/siso/timing/plant_021_jopt1.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 21, "method": "jopt1", "t": 1616.4610531330109, "k": 1, "load": [6.0, 6.0, 6.0]}
results/icml_r3/X29/siso/timing/plant_021_jopt16.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 21, "method": "jopt16", "t": 1778.183555841446, "k": 16, "load": [6.0, 6.0, 6.0]}
results/icml_r3/X29/siso/timing/plant_021_jopt4.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 21, "method": "jopt4", "t": 1580.9241681098938, "k": 4, "load": [6.0, 6.0, 6.0]}
results/icml_r3/X29/siso/timing/plant_021_knn_C+phi.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 21, "method": "knn_C+phi", "t_sample": 0.005930423736572266, "t_score": 0.9871442317962646, "t": 0.9930746555328369, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
results/icml_r3/X29/siso/timing/plant_021_knn_C-only.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 21, "method": "knn_C-only", "t_sample": 2.1155502796173096, "t_score": 1.1700952053070068, "t": 3.2856454849243164, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
results/icml_r3/X29/siso/timing/plant_021_twin_C+phi.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 21, "method": "twin_C+phi", "t_sample": 0.2300739288330078, "t_score": 0.22470903396606445, "t": 0.45478296279907227, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
results/icml_r3/X29/siso/timing/plant_021_twin_C-only.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 21, "method": "twin_C-only", "t_sample": 0.22515177726745605, "t_score": 0.26130175590515137, "t": 0.4864535331726074, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
results/icml_r3/X29/siso/timing/plant_041_fopt1.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 41, "method": "fopt1", "t": 1533.378296136856, "k": 1, "load": [6.0, 6.0, 5.9150390625]}
results/icml_r3/X29/siso/timing/plant_041_fopt16.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 41, "method": "fopt16", "t": 1787.9273924827576, "k": 16, "load": [6.0, 6.0, 6.0]}
results/icml_r3/X29/siso/timing/plant_041_fopt4.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 41, "method": "fopt4", "t": 1572.8095934391022, "k": 4, "load": [6.0, 6.0, 6.0]}
results/icml_r3/X29/siso/timing/plant_041_gen_C+phi.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 41, "method": "gen_C+phi", "t_sample": 8.899398565292358, "t_score": 1.698326826095581, "t": 10.59772539138794, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
results/icml_r3/X29/siso/timing/plant_041_gen_C-only.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 41, "method": "gen_C-only", "t_sample": 8.66327452659607, "t_score": 1.2682888507843018, "t": 9.931563377380371, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
results/icml_r3/X29/siso/timing/plant_041_hifoo2.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 41, "method": "hifoo2", "t": 1.9034154415130615, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
results/icml_r3/X29/siso/timing/plant_041_jopt1.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 41, "method": "jopt1", "t": 1552.531121969223, "k": 1, "load": [6.0, 6.0, 6.0]}
results/icml_r3/X29/siso/timing/plant_041_jopt16.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 41, "method": "jopt16", "t": 1799.8491704463959, "k": 16, "load": [3.51025390625, 5.095703125, 5.66943359375]}
results/icml_r3/X29/siso/timing/plant_041_jopt4.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 41, "method": "jopt4", "t": 1499.8948347568512, "k": 4, "load": [6.0, 6.0, 6.0]}
results/icml_r3/X29/siso/timing/plant_041_knn_C+phi.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 41, "method": "knn_C+phi", "t_sample": 0.005980730056762695, "t_score": 0.11679935455322266, "t": 0.12278008460998535, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
results/icml_r3/X29/siso/timing/plant_041_knn_C-only.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 41, "method": "knn_C-only", "t_sample": 2.1139070987701416, "t_score": 0.5942299365997314, "t": 2.708137035369873, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
results/icml_r3/X29/siso/timing/plant_041_twin_C+phi.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 41, "method": "twin_C+phi", "t_sample": 0.20940399169921875, "t_score": 1.5725376605987549, "t": 1.7819416522979736, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
results/icml_r3/X29/siso/timing/plant_041_twin_C-only.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 41, "method": "twin_C-only", "t_sample": 0.1909935474395752, "t_score": 8.11414384841919, "t": 8.305137395858765, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
results/icml_r3/X29/siso/timing/plant_061_fopt1.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 61, "method": "fopt1", "t": 1546.6886041164398, "k": 1, "load": [6.0, 6.0, 5.9150390625]}
results/icml_r3/X29/siso/timing/plant_061_fopt16.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 61, "method": "fopt16", "t": 1680.0087144374847, "k": 16, "load": [6.0, 6.0, 6.0]}
results/icml_r3/X29/siso/timing/plant_061_fopt4.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"i": 61, "method": "fopt4", "t": 1523.4473977088928, "k": 4, "load": [6.0, 6.0, 6.0]}