X29-A 3A: per-plant optimum in the actuator setting, X27 C-only arms re-scored on J*_A
Browse files- results/icml_r3/X29/PREREG.md +185 -0
- results/icml_r3/X29/act/heldout_m3/cert_id_c1.npz +3 -0
- results/icml_r3/X29/act/heldout_m3/cert_so3_c1.npz +3 -0
- results/icml_r3/X29/act/heldout_m3/cert_so3_c2.npz +3 -0
- results/icml_r3/X29/act/heldout_m3/cert_so3_c5.npz +3 -0
- results/icml_r3/X29/act/heldout_m3/fresh.npz +3 -0
- results/icml_r3/X29/act/heldout_m3/id_c1.npz +3 -0
- results/icml_r3/X29/act/heldout_m3/jstar.npz +3 -0
- results/icml_r3/X29/act/heldout_m3/so3_c1.npz +3 -0
- results/icml_r3/X29/act/heldout_m3/so3_c2.npz +3 -0
- results/icml_r3/X29/act/heldout_m3/so3_c5.npz +3 -0
- results/icml_r3/X29/act/heldout_m4/cert_id_c1.npz +3 -0
- results/icml_r3/X29/act/heldout_m4/cert_so3_c1.npz +3 -0
- results/icml_r3/X29/act/heldout_m4/cert_so3_c2.npz +3 -0
- results/icml_r3/X29/act/heldout_m4/cert_so3_c5.npz +3 -0
- results/icml_r3/X29/act/heldout_m4/fresh.npz +3 -0
- results/icml_r3/X29/act/heldout_m4/id_c1.npz +3 -0
- results/icml_r3/X29/act/heldout_m4/jstar.npz +3 -0
- results/icml_r3/X29/act/heldout_m4/so3_c1.npz +3 -0
- results/icml_r3/X29/act/heldout_m4/so3_c2.npz +3 -0
- results/icml_r3/X29/act/heldout_m4/so3_c5.npz +3 -0
- results/icml_r3/X29/act/heldout_m5/cert_id_c1.npz +3 -0
- results/icml_r3/X29/act/heldout_m5/cert_so3_c1.npz +3 -0
- results/icml_r3/X29/act/heldout_m5/cert_so3_c2.npz +3 -0
- results/icml_r3/X29/act/heldout_m5/cert_so3_c5.npz +3 -0
- results/icml_r3/X29/act/heldout_m5/fresh.npz +3 -0
- results/icml_r3/X29/act/heldout_m5/id_c1.npz +3 -0
- results/icml_r3/X29/act/heldout_m5/jstar.npz +3 -0
- results/icml_r3/X29/act/heldout_m5/so3_c1.npz +3 -0
- results/icml_r3/X29/act/heldout_m5/so3_c2.npz +3 -0
- results/icml_r3/X29/act/heldout_m5/so3_c5.npz +3 -0
- results/icml_r3/X29/act/report_3a_heldout_m3.txt +9 -0
- results/icml_r3/X29/act/report_3a_heldout_m4.txt +9 -0
- results/icml_r3/X29/act/report_3a_heldout_m5.txt +9 -0
- results/icml_r3/X29/act/selfcheck_cert_m3.json +7 -0
- results/icml_r3/X29/act/selfcheck_cert_m4.json +7 -0
- results/icml_r3/X29/act/selfcheck_cert_m5.json +7 -0
- results/icml_r3/X29/act/selfcheck_cost_m3.json +15 -0
- results/icml_r3/X29/act/selfcheck_cost_m4.json +15 -0
- results/icml_r3/X29/act/selfcheck_cost_m5.json +15 -0
- results/icml_r3/X29/act/x27rescore_d3fm3n8000.json +250 -0
- results/icml_r3/X29/act/x27rescore_d3fm3n8000.txt +23 -0
- results/icml_r3/X29/act/x27rescore_d3fm4n8000.json +250 -0
- results/icml_r3/X29/act/x27rescore_d3fm4n8000.txt +23 -0
- results/icml_r3/X29/act/x27rescore_d3fm5n16000.json +250 -0
- results/icml_r3/X29/act/x27rescore_d3fm5n16000.txt +23 -0
- results/icml_r3/X29/act/x27rows_d3fm3n8000.npy +3 -0
- results/icml_r3/X29/act/x27rows_d3fm4n8000.npy +3 -0
- results/icml_r3/X29/act/x27rows_d3fm5n16000.npy +3 -0
results/icml_r3/X29/PREREG.md
ADDED
|
@@ -0,0 +1,185 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# X29 pre-registration: C+phi next to C-only, with the same baselines and KPIs
|
| 2 |
+
|
| 3 |
+
Written 2026-11-16, 18:30 UTC, before any X29 code ran or any X29 number existed.
|
| 4 |
+
|
| 5 |
+
Request (relayed by the presentation session dc_presentation, then confirmed by Matteo in the move-HPC session on 2026-11-16):
|
| 6 |
+
- "go", with the actuator setting in two steps (3A below, then he decides 3B);
|
| 7 |
+
- the actuator cost is X27's quadratic cost computed by simulation;
|
| 8 |
+
- m = 5 uses the 16k bank;
|
| 9 |
+
- then: "I need to have the final baseline comparison also for the spline methods".
|
| 10 |
+
|
| 11 |
+
Scope: the deck v10 slides listed by dc_presentation. SISO [35] [36] [37] [38] [73]; the MIMO spline setting (three new slides: certified vs b, regret vs b, CPU vs regret, plus [73]); the actuator setting [50] [51] [52] [84] [73] (after 3B).
|
| 12 |
+
|
| 13 |
+
Names:
|
| 14 |
+
- **C-only**: phi fixed (SISO sat_1; MIMO identity); the model generates the linear part only.
|
| 15 |
+
- **C+phi**: the model generates the linear part and the spline phi (X28).
|
| 16 |
+
|
| 17 |
+
Nothing published is replaced. C-only numbers are re-scored on the common reference J* of each setting, and the earlier numbers stay in their own reports.
|
| 18 |
+
|
| 19 |
+
## 1. Conventions (all settings)
|
| 20 |
+
|
| 21 |
+
- **Regret** of a method on a plant at budget b: (J_b - J*) / J*.
|
| 22 |
+
- J_b is the cost of the cheapest certified candidate among its first b candidates.
|
| 23 |
+
- A plant with no certified candidate gets regret 1e3, i.e. it counts as a loss (X28 convention).
|
| 24 |
+
- **J\* per setting** = min over all stored labels of both variants:
|
| 25 |
+
- SISO: J_Fonly and the X28 test joint labels.
|
| 26 |
+
- MIMO spline: the T2 id and so3 labels.
|
| 27 |
+
- Actuator: the X27 stored best gain plus the 3A labels.
|
| 28 |
+
- **Reported per method and budget b:**
|
| 29 |
+
- plants with a certified candidate (%);
|
| 30 |
+
- median regret with a bootstrap 95% CI (10,000 resamples over plants);
|
| 31 |
+
- for methods with 3 training seeds: mean and range over seeds, plus seed 0 alone;
|
| 32 |
+
- b = 1, 4, 16, and 64 where it exists.
|
| 33 |
+
- **Slide [73] number:** certified plants at b = 1, the generator minus its own variant's kNN, plant-paired per seed, bootstrap 95% CI. Given for C+phi and for C-only.
|
| 34 |
+
- **No new verdict rule** in SISO and the MIMO spline setting. The registered X28 verdicts stand unchanged.
|
| 35 |
+
- **"Best baseline"** (summary slides): the non-generative method with the lowest b = 16 median regret in the setting; ties go to the higher certified share.
|
| 36 |
+
|
| 37 |
+
## 2. SISO (X29-S): the 199 X21 test plants, J* as X28 S4
|
| 38 |
+
|
| 39 |
+
| arm | C-only | C+phi |
|
| 40 |
+
|---|---|---|
|
| 41 |
+
| generator | clL seeds 0-2, clip 1.5, t-repair (stored X28 `eval_test` npz; K64 seed 0) | clS seeds 0-2, same (stored; K64 seed 0) |
|
| 42 |
+
| kNN | X22 `knn_s2` on the X21 F-only bank | the same rule on the joint-label bank (below) |
|
| 43 |
+
| regression twin | X23 twins `x21_twin_clL{,_s1,_s2}/best.pth` | new twins of clS, seeds 0-2 (below) |
|
| 44 |
+
| per-plant optimizer | F-only, best of the first k X21 labels | joint, best of the first k joint rows (below) |
|
| 45 |
+
| optimizer from the generator's best sample | 1 F-only start (X24 refine) from clL seed 0 | 1 joint start from clS seed 0 |
|
| 46 |
+
|
| 47 |
+
Shared flat references, scored on J*:
|
| 48 |
+
- tuned saturated LQG or static gain (J_lin, plant files);
|
| 49 |
+
- fixed-order stabilization: best of the 16 stored HIFOO starts (`start_J`).
|
| 50 |
+
|
| 51 |
+
### Details
|
| 52 |
+
- **kNN, C+phi.** Same ARD metric as X22 (`results/icml_r3/X21/ard.npz`) and the same rule: walk the nearest bank plants (all 1,971) and take the 2 cheapest rows of each, until 16 candidates (64 for K64).
|
| 53 |
+
- Rows are the c = 1 joint rows of `train_x28.rows()`.
|
| 54 |
+
- F is re-encoded on the neighbour at gain k = slope0(phi) (`cl_encode_k`) and decoded on the query plant at the same k (`cl_decode_k`); knots are copied.
|
| 55 |
+
- Then t-repair at k and `x28_eval.score`.
|
| 56 |
+
- Self-check: a bank plant queried as itself returns its cheapest row.
|
| 57 |
+
- **kNN, C-only.** Recomputed with the same X29 code (phi = sat_1, `x28_eval.score`). Its certified rate must agree with the stored X21 `bl_knn_s2_tr` within 1 point (self-check).
|
| 58 |
+
- **Twins.**
|
| 59 |
+
- The C+phi twin is `train_x28` with `X21_LOSS=reg`, `X21_CFG='{"cfg_drop": 0.0}'` and `X28_REP=cl` (the X23 recipe), seeds 0-2, 60,000 steps, best validation checkpoint.
|
| 60 |
+
- Both variants' twins are sampled like X21: candidate 1 deterministic, candidates 2-16 MC-dropout draws, seed 4242.
|
| 61 |
+
- C+phi twin output: knots clipped to [-1, 1], F decoded at k, t-repair at k.
|
| 62 |
+
- Both are scored with `x28_eval.score`. The C-only twin's certified rate must agree with the stored X21 result within 1 point (self-check).
|
| 63 |
+
- **Per-plant optimizers.**
|
| 64 |
+
- F-only with k starts (k in 1, 2, 4, 8, 16) = best of the first k X21 labels, as X24.
|
| 65 |
+
- Joint with k starts = best of min(o_J, J_x21) over the c = 1 rows with src < k (`labels_test`). The joint labels refine the F-only labels.
|
| 66 |
+
- "Joint, all" = J*, so its regret is 0 by construction.
|
| 67 |
+
- **Optimizer from the generator's best sample.**
|
| 68 |
+
- C-only: the X24 refine (ref_optimize, frozen config, freeze_ts) from clL seed 0's best certified b = 16 sample (clip 1.5).
|
| 69 |
+
- C+phi: the S2 warm arm (c = 1, F lr 0.02, knot lr 0.02, 400 steps, cheapest snapshot certified at s) from clS seed 0's best certified b = 16 sample.
|
| 70 |
+
- In both, J = min(start, result).
|
| 71 |
+
- The runs may batch plants with `spline_optimize_multi` (knot lr 0 and the frozen F lr for C-only), but only after an equivalence check against the single-plant function: 3 plants, 25 steps, relative J difference <= 1e-6. If the check fails, the single-plant function is used.
|
| 72 |
+
- **CPU (one timing job).**
|
| 73 |
+
- Plants: the 10 X24 timing plants (`ref_i[::20]`), one process per plant, one thread each, on a node with nothing else of this project running. The node's load average is reported.
|
| 74 |
+
- Timed per plant:
|
| 75 |
+
- one HIFOO start;
|
| 76 |
+
- the F-only optimizer at 1, 4, 16 starts;
|
| 77 |
+
- the joint optimizer at 1, 4, 16 starts (from the plant's first k labels, c = 1);
|
| 78 |
+
- generator sampling on CPU (16 samples) and scoring (16 candidates), for clL and clS seed 0;
|
| 79 |
+
- kNN retrieval plus scoring, both banks;
|
| 80 |
+
- twin sampling plus scoring, both variants;
|
| 81 |
+
- the two "from the best sample" optimizers.
|
| 82 |
+
- Medians over the timing plants. Start counts 2 and 8 are interpolated, as X24 did.
|
| 83 |
+
- CPU per point follows X24's formulas. The joint optimizer at k starts is charged k HIFOO starts + the F-only optimizer (k) + the joint optimizer (k).
|
| 84 |
+
- Batched generator sampling: 199 plants x 16 in one call on the node's GPU, wall time / 199, reported next to the one-plant CPU number.
|
| 85 |
+
- **Compute ratio ([38]).** CPU of the variant's own per-plant optimizer at the fewest starts k in {1, 2, 4, 8, 16} whose median regret is <= the generator's b = 16 median regret (seed 0), divided by the generator's b = 16 CPU. If no k qualifies, both numbers are reported as "not reached at 16 starts".
|
| 86 |
+
|
| 87 |
+
## 3. MIMO spline setting (X29-T): T2, 200 fresh plants per m (m = 2, 3)
|
| 88 |
+
|
| 89 |
+
Plants, J*_T2, certificates, cost, repair R and the stored generator samples are as in the X28 addendum of 2026-11-15.
|
| 90 |
+
|
| 91 |
+
| arm | C-only | C+phi |
|
| 92 |
+
|---|---|---|
|
| 93 |
+
| generator | (a) seeds 0-2 (stored `gen_fb`) | (b) repaired, primary; (b) as drawn, secondary (stored) |
|
| 94 |
+
| kNN | label (a) of the 16 nearest bank plants | label (b) of the 16 nearest bank plants |
|
| 95 |
+
| regression twin | new, on dataset (a), seeds 0-2 | new, on dataset (b), seeds 0-2 |
|
| 96 |
+
|
| 97 |
+
Shared flat references:
|
| 98 |
+
- **48-start solver** (C-only): the T2 oracle's best gain K0 (`build_sofm`, 48 starts), with the Lyapunov test and the X28 cost.
|
| 99 |
+
- **Per-plant Adam optimizer** (flat lines): from the T2 label runs. C-only uses J_id. C+phi uses J_b = J*_T2, so its regret is 0 by construction.
|
| 100 |
+
|
| 101 |
+
### Details
|
| 102 |
+
- **kNN.** Banks are the S2-M label plants (m = 3: [64, 8000) of `D2_scale/m3_bank.npz`; m = 2: [64, 4000) of `C_mimo/sof/bank.npz`).
|
| 103 |
+
- Features are scaled to the family box and then by the ARD length scales (x27_filter convention).
|
| 104 |
+
- m = 3 uses `results/icml_r3/X10/ard_m3.npz`. m = 2 uses `x10_retr.fit_ard` fitted on the m = 2 bank (output: label (a)), before any T2 kNN number.
|
| 105 |
+
- Candidates come nearest first, one label per plant.
|
| 106 |
+
- **Twins.**
|
| 107 |
+
- `mh_train` arm "reg" on the S3-M datasets `ds/` (a) and (b), seeds 0-2, with the S3-M steps, batch, lr and EMA; final EMA 0.999 weights.
|
| 108 |
+
- Candidates: 1 deterministic, then 2-16 MC-dropout draws (torch seed = model seed). This is the SISO convention, the one on slide [35].
|
| 109 |
+
- Decoding is as for the generators; ratios are projected to [0.1, 10].
|
| 110 |
+
- **Certificates.** C-only arms use the Lyapunov LMI (alpha = beta = 1). C+phi arms use the circle LMI with repair R (center r_i(1)). C+phi kNN and twin are also reported without repair.
|
| 111 |
+
- **CPU.**
|
| 112 |
+
- One plant at a time: one CPU thread per plant, covering sampling 16 on CPU, certificates (+ repair) and the cost rollouts the scoring code runs.
|
| 113 |
+
- Batched: all 200 plants in one call, with sampling and rollouts on the node's GPU and certificates on its CPU workers; total wall time / 200, with device and worker count stated.
|
| 114 |
+
- kNN and twins are timed the same way.
|
| 115 |
+
- 48-start solver: re-timed per plant on one thread, all 48 starts, with the early stop at the first certified start also reported.
|
| 116 |
+
- Adam optimizer: the time of 25 Adam steps, measured for one plant on one CPU thread and for 200 plants batched on the GPU, scaled to 400 steps plus the measured certification time.
|
| 117 |
+
- Timing plants: the first 10 T2 plants per m.
|
| 118 |
+
- **Budgets:** b = 1, 4, 16 (no 64).
|
| 119 |
+
|
| 120 |
+
## 4. MIMO actuator setting (X29-A), m = 3, 4, 5
|
| 121 |
+
|
| 122 |
+
### Definitions
|
| 123 |
+
- **Plants:** X27's 120 test plants per m (`results/icml_r2/D2_scale/m{m}_heldout_{4242,9001}.npz`). K_ref is the stored best gain, with cost J_ref.
|
| 124 |
+
- **Controller:** u = -Phi(phi(K y)).
|
| 125 |
+
- Phi = diag(Phi_i) is the unknown actuator, each Phi_i in the sector [0.4, 2.2] (X27).
|
| 126 |
+
- phi = diag(psi_i) is in the so3 class: odd, piecewise linear, knots at |x| = 1, 3, 5, knot ratios in [0.1, 10], ray extension.
|
| 127 |
+
- C-only is psi = identity.
|
| 128 |
+
- **Certificate:** the circle LMI with per-channel sector [0.4 alpha_i, 2.2 beta_i] (`x28_mimo.circle_lmi` with those ends), where alpha_i and beta_i are the min and max knot ratio.
|
| 129 |
+
- Sound: with alpha_i >= 0.1 > 0, Phi_i(psi_i(x)) / x = (Phi_i(v) / v)(v / x) lies in [0.4 alpha_i, 2.2 beta_i], and the composition is memoryless and diagonal, which is all the circle LMI with diagonal multipliers needs.
|
| 130 |
+
- Self-check: at identity knots it must agree with `family_sofm.verify` on 200 X27 bank gains per m, half of them perturbed (as X28's self-check).
|
| 131 |
+
- **Cost:** J = mean over the 2m unit state directions of the integral from 0 to T of |x|^2 + 0.1 |u|^2, with Phi = identity, RK4 at dt 0.01.
|
| 132 |
+
- Self-check: for linear K, J must equal `family_sofm.cost` (tr(P)/(2m)), with median relative difference <= 1e-3 and maximum <= 1e-2 on the 120 stored test gains per m.
|
| 133 |
+
- T starts at 20 s and doubles until the check passes.
|
| 134 |
+
|
| 135 |
+
### 3A (now)
|
| 136 |
+
- **Label optimizer** on the 120 test plants per m, X28-M's procedure (`x28_mimo` optimize) with this section's certificate and cost:
|
| 137 |
+
- start K_ref;
|
| 138 |
+
- arm id (knots frozen at identity, c = 1) and arm so3 at gauges c = 1, 2, 5;
|
| 139 |
+
- K lr 0.02 and knot lr 0.02 (the X28-M values);
|
| 140 |
+
- 400 steps, a snapshot every 25 plus snapshot 0, keeping the cheapest certified snapshot.
|
| 141 |
+
- **Outputs:** J_id, J_so3 (the minimum of the so3 starts and id), and J*_A = min(J_ref, J_id, J_so3).
|
| 142 |
+
- **Reported, with no stop rule** (Matteo decides 3B):
|
| 143 |
+
- d = (J_id - J_so3) / J_so3: median with CI, and the share of plants with d >= 2%;
|
| 144 |
+
- (J_ref - J_id) / J_id, the gain from extra optimization alone;
|
| 145 |
+
- a new-direction diagnostic: each arm's best controller on 64 random unit state directions per plant (seed (8080, plant index)), as the median over plants of the per-plant median paired difference.
|
| 146 |
+
- **Expectation, written before the run:** d is small. The quadratic cost does not reward an amplitude-dependent gain, and the composed sector is 5.5 beta/alpha wide, so any nonlinear spline needs more robustness than a linear gain.
|
| 147 |
+
- **Also now:** X27's stored C-only pipeline arms are re-scored on J*_A. Their certified shares are unchanged; regret vs J*_A is reported next to the published regret vs J_ref.
|
| 148 |
+
|
| 149 |
+
### 3B (only if Matteo says so after 3A; details by addendum before any 3B number)
|
| 150 |
+
- Labels on X27's banks: 8,000 plants at m = 3 and 4, and 16,000 at m = 5 (`m5_bank` + `m5_bank_b`).
|
| 151 |
+
- Generators, 3 seeds each:
|
| 152 |
+
- C+phi generators (so3 target, as X28-M);
|
| 153 |
+
- C-only generators trained on the id arm of the same label run, so both learn from the same optimizer.
|
| 154 |
+
- X27's published generators are reported next to them.
|
| 155 |
+
- Baselines: regression twins and kNN with the X10 ARD metric.
|
| 156 |
+
- X27's pipeline for C+phi: constant-gain filter over the composed sector, circle LMI on the composed sector, repair R, and solver iterations at the composed sector with phi fixed.
|
| 157 |
+
- Timing in one job.
|
| 158 |
+
- Verdict C+phi vs C-only: the X28 rule at b = 16. X27's YES rule is applied to each variant.
|
| 159 |
+
|
| 160 |
+
## 5. Compute and reporting
|
| 161 |
+
- At most 2 GPU nodes, with CPU work inside them. Timing jobs run on a node with nothing else of this project running.
|
| 162 |
+
- Code:
|
| 163 |
+
- `nonlinear/experiments/x29_siso.py`
|
| 164 |
+
- `nonlinear/mimo_unstable/x29_t2.py`
|
| 165 |
+
- `nonlinear/mimo_unstable/x29_act.py`
|
| 166 |
+
- Results go in `results/icml_r3/X29/{siso,t2,act}/`; the write-up in `docs/icml_r3_x29.md`.
|
| 167 |
+
- Hugging Face: targeted upload of `results/icml_r3/X29/` and the new models (never the X7 paths).
|
| 168 |
+
- No paper edits.
|
| 169 |
+
- Any detail fixed after this file goes in a dated addendum (`PREREG.md`, or `PREREG_siso.md` / `PREREG_t2.md` for the SISO and T2 parts) before the affected number exists.
|
| 170 |
+
|
| 171 |
+
## Addendum 2026-11-16, 18:20 UTC: X29-A cost quadrature (before any 3A number)
|
| 172 |
+
|
| 173 |
+
The first implementation of the section 4 cost integrated the stage cost with the trapezoid rule on the RK4 steps, the X28-M convention.
|
| 174 |
+
|
| 175 |
+
On the 120 m = 3 test gains, the cost self-check failed at every horizon:
|
| 176 |
+
- median relative difference to tr(P)/(2m) 0.25%, maximum 1.56%;
|
| 177 |
+
- unchanged at T = 20, 40, 80 and 160 s, with |x(T)| <= 8e-5 already at 20 s.
|
| 178 |
+
|
| 179 |
+
So the error is the quadrature on fast closed-loop modes, not the horizon, and doubling T cannot pass.
|
| 180 |
+
|
| 181 |
+
Change: the running cost is now integrated by RK4 as an extra state, with the stage cost evaluated at the four RK4 stages. dt stays 0.01 and the change applies to both arms. The self-check and its thresholds are unchanged and are re-run; if it still fails, dt is halved until it passes.
|
| 182 |
+
|
| 183 |
+
The composed-sector self-check at m = 3 passed (200 gains, 0 disagreements) and does not depend on the cost.
|
| 184 |
+
|
| 185 |
+
(Operational note: srun without `-n 1` started two copies of the self-check script on node 06. Both were killed. The only file they wrote, `selfcheck_cert_m3.json`, came out identical from both copies.)
|
results/icml_r3/X29/act/heldout_m3/cert_id_c1.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a1777bd6199d2995d11cd7cdfba69717ffa8ad782137833c9bdb524ab1da75c9
|
| 3 |
+
size 12108
|
results/icml_r3/X29/act/heldout_m3/cert_so3_c1.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ee99b6e4e4e49a53ea654573433cbca6ffe5faab8eb0577900c8570e6b7d4626
|
| 3 |
+
size 15459
|
results/icml_r3/X29/act/heldout_m3/cert_so3_c2.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f59d5df4e328147a511be0a033cc6203dbd398cdc42e82419b26a3c7e5aef5e2
|
| 3 |
+
size 13152
|
results/icml_r3/X29/act/heldout_m3/cert_so3_c5.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e38da7c59a5261a6edfa11da7c481d42185f0f8b2a3e8c4fb67e5fc56d4d32ec
|
| 3 |
+
size 12636
|
results/icml_r3/X29/act/heldout_m3/fresh.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:92aa9fe6421f8b04ee9ed8995e745d3b207df56fbdb65a3e7affb3f82a3ee467
|
| 3 |
+
size 118749
|
results/icml_r3/X29/act/heldout_m3/id_c1.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:97a56235334f16776869edc1cd6bc71b4e5e3ffe2e226cc2e3c2b2a99769ed79
|
| 3 |
+
size 191891
|
results/icml_r3/X29/act/heldout_m3/jstar.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:be496d7bca37e511f20fc8185d27376a8afe499475758b40146376e664142da9
|
| 3 |
+
size 5042
|
results/icml_r3/X29/act/heldout_m3/so3_c1.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:db9a9200b62cac5e66e5d580483c2aa384f88db71511e9c8f1c68aeb81d7357c
|
| 3 |
+
size 463656
|
results/icml_r3/X29/act/heldout_m3/so3_c2.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d3f56cd63a3d424f9ca8702dcd1a07bc7e05578454cd9553d6444fff52d2e233
|
| 3 |
+
size 460964
|
results/icml_r3/X29/act/heldout_m3/so3_c5.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:1ae2a2fb127666fc4901714bd4b728a0359e5e9bb8a5833133caefa3d43c5456
|
| 3 |
+
size 440765
|
results/icml_r3/X29/act/heldout_m4/cert_id_c1.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:acf94787736a9b639973e154fa7af834894e21b72952603cd74c391fc976aee4
|
| 3 |
+
size 18609
|
results/icml_r3/X29/act/heldout_m4/cert_so3_c1.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ab9fe4e22c3f4cf60c47d6e5a9360a274c00052a9063bdfb91c29c25c8fc8732
|
| 3 |
+
size 21013
|
results/icml_r3/X29/act/heldout_m4/cert_so3_c2.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8bbeb57e6cef77584c141c1d560b1614bae831bcb843ef85031594864a72b79a
|
| 3 |
+
size 18897
|
results/icml_r3/X29/act/heldout_m4/cert_so3_c5.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2f402ca1207852391c9e222622b2faff7b682e2c8e678a6c63af3627f3e72939
|
| 3 |
+
size 18310
|
results/icml_r3/X29/act/heldout_m4/fresh.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7bd1c6a2a0f192c1c0e6f6b3d70c6be3bda5a8eadc5aaa315166868dbd408ab5
|
| 3 |
+
size 118679
|
results/icml_r3/X29/act/heldout_m4/id_c1.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e5da58ad83f6c2743e54be22d7305f430ba0bc58aac43608661be60454174d1f
|
| 3 |
+
size 321578
|
results/icml_r3/X29/act/heldout_m4/jstar.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:367caeebc244ece0dfdae026a6eb33ca4a798ace0921ee54abfd6927022dc5db
|
| 3 |
+
size 5043
|
results/icml_r3/X29/act/heldout_m4/so3_c1.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a93ee73d6b7100fea96dcea1f11bf95e7a61dfa1b1bf8179579b86fb49112431
|
| 3 |
+
size 681843
|
results/icml_r3/X29/act/heldout_m4/so3_c2.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2f7566a3df0c2bd9ec0d8f5974eb2d2e0222abd09a0423711a21e262994c5bfd
|
| 3 |
+
size 679078
|
results/icml_r3/X29/act/heldout_m4/so3_c5.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:1032dd347c6a0e3ed1e46935f86e2921a33ceef3d54e05746249ac052c3f4525
|
| 3 |
+
size 656555
|
results/icml_r3/X29/act/heldout_m5/cert_id_c1.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0dbf6c4e6c8f46ff64eb7c0cd7719303a52e6a16befc210a1fae73a2ee30a8ab
|
| 3 |
+
size 27048
|
results/icml_r3/X29/act/heldout_m5/cert_so3_c1.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d4b7c43d54624e83877edca0c3eb96f855eb6ed8edc45c0ad428b10f48272e2e
|
| 3 |
+
size 30719
|
results/icml_r3/X29/act/heldout_m5/cert_so3_c2.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4e46ed35905aced34f4191ec7ee565c13c19a07fc6f983bb2d7cb3727fbdcd01
|
| 3 |
+
size 27687
|
results/icml_r3/X29/act/heldout_m5/cert_so3_c5.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:46a9251ee8898ae6e6b030bd26e452c9388c52c23be639a9679f96f2619eef01
|
| 3 |
+
size 27207
|
results/icml_r3/X29/act/heldout_m5/fresh.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7bedcca3bcd4cf193b3b4dca8fcfd39c030774cd68f0ef11285f10f16a195224
|
| 3 |
+
size 118625
|
results/icml_r3/X29/act/heldout_m5/id_c1.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:26ed018f08969cbdaffe6c90335212ab84fa33f6f13d8c4fe9dd93542b8d6009
|
| 3 |
+
size 489904
|
results/icml_r3/X29/act/heldout_m5/jstar.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2dd805bd86cca570e501009ab4ef7afd33a8194faeb884dac6bd67a1861ad916
|
| 3 |
+
size 5042
|
results/icml_r3/X29/act/heldout_m5/so3_c1.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:616715057316eeaf90e3501aff103c85a459ee3d8926756e3b0815203e6f0bd2
|
| 3 |
+
size 935972
|
results/icml_r3/X29/act/heldout_m5/so3_c2.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7cfc5582a822fee8375359d592911a33f09481673995b993ed22bb056733a135
|
| 3 |
+
size 932523
|
results/icml_r3/X29/act/heldout_m5/so3_c5.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a0e24942563def43043a2fcd3254b5bed13f5ae6cc171717b5079d0178095e0e
|
| 3 |
+
size 905631
|
results/icml_r3/X29/act/report_3a_heldout_m3.txt
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
X29-A 3A, m = 3, 120 X27 test plants; cost = simulated X27 cost (T = 20 s); certificate = circle LMI on [0.4 alpha, 2.2 beta]
|
| 2 |
+
certified: id 120, so3 starts so3_c1 120, so3_c2 120, so3_c5 120
|
| 3 |
+
d = (J_id - J_so3) / J_so3: median 0.00% [0.00, 0.00], >= 2% on 16 of 120, mean 0.68% -> no gain at the optimum by the 2% rule
|
| 4 |
+
extra optimization alone, (J_ref - J_id) / J_id: median -0.00% [-0.00, -0.00]
|
| 5 |
+
best so3 controller from: id_c1 69, so3_c1 22, so3_c2 15, so3_c5 14
|
| 6 |
+
composed-sector ratio max_i (2.2 beta_i) / (0.4 alpha_i) of the best so3 controller: median 5.50 (5.5 = linear), 90th pct 15.97
|
| 7 |
+
J*_A = min(J_ref, J_id, J_so3): median 2.098; J_ref is the J* on 75 plants, id on 18, so3 on 27
|
| 8 |
+
snapshot 0 kept: id 62; nonfinite gradient steps (all starts): 0
|
| 9 |
+
new directions (64 per plant, seed (8080, i)), median over plants of the per-plant median (J_id - J_so3) / J_so3: +0.00% [+0.00, +0.00], > 0 on 30 of 120
|
results/icml_r3/X29/act/report_3a_heldout_m4.txt
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
X29-A 3A, m = 4, 120 X27 test plants; cost = simulated X27 cost (T = 20 s); certificate = circle LMI on [0.4 alpha, 2.2 beta]
|
| 2 |
+
certified: id 120, so3 starts so3_c1 120, so3_c2 120, so3_c5 118
|
| 3 |
+
d = (J_id - J_so3) / J_so3: median 0.00% [0.00, 0.00], >= 2% on 9 of 120, mean 0.34% -> no gain at the optimum by the 2% rule
|
| 4 |
+
extra optimization alone, (J_ref - J_id) / J_id: median -0.00% [-0.00, -0.00]
|
| 5 |
+
best so3 controller from: id_c1 78, so3_c1 13, so3_c2 13, so3_c5 16
|
| 6 |
+
composed-sector ratio max_i (2.2 beta_i) / (0.4 alpha_i) of the best so3 controller: median 5.50 (5.5 = linear), 90th pct 11.37
|
| 7 |
+
J*_A = min(J_ref, J_id, J_so3): median 2.127; J_ref is the J* on 72 plants, id on 34, so3 on 14
|
| 8 |
+
snapshot 0 kept: id 71; nonfinite gradient steps (all starts): 0
|
| 9 |
+
new directions (64 per plant, seed (8080, i)), median over plants of the per-plant median (J_id - J_so3) / J_so3: +0.00% [+0.00, +0.00], > 0 on 13 of 120
|
results/icml_r3/X29/act/report_3a_heldout_m5.txt
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
X29-A 3A, m = 5, 120 X27 test plants; cost = simulated X27 cost (T = 20 s); certificate = circle LMI on [0.4 alpha, 2.2 beta]
|
| 2 |
+
certified: id 120, so3 starts so3_c1 120, so3_c2 120, so3_c5 120
|
| 3 |
+
d = (J_id - J_so3) / J_so3: median 0.00% [0.00, 0.00], >= 2% on 13 of 120, mean 0.44% -> no gain at the optimum by the 2% rule
|
| 4 |
+
extra optimization alone, (J_ref - J_id) / J_id: median -0.00% [-0.00, -0.00]
|
| 5 |
+
best so3 controller from: id_c1 76, so3_c1 15, so3_c2 15, so3_c5 14
|
| 6 |
+
composed-sector ratio max_i (2.2 beta_i) / (0.4 alpha_i) of the best so3 controller: median 5.50 (5.5 = linear), 90th pct 12.18
|
| 7 |
+
J*_A = min(J_ref, J_id, J_so3): median 1.985; J_ref is the J* on 77 plants, id on 26, so3 on 17
|
| 8 |
+
snapshot 0 kept: id 75; nonfinite gradient steps (all starts): 0
|
| 9 |
+
new directions (64 per plant, seed (8080, i)), median over plants of the per-plant median (J_id - J_so3) / J_so3: +0.00% [+0.00, +0.00], > 0 on 16 of 120
|
results/icml_r3/X29/act/selfcheck_cert_m3.json
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"m": 3,
|
| 3 |
+
"n": 200,
|
| 4 |
+
"verify_true": 121,
|
| 5 |
+
"lmi_true": 121,
|
| 6 |
+
"disagree": 0
|
| 7 |
+
}
|
results/icml_r3/X29/act/selfcheck_cert_m4.json
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"m": 4,
|
| 3 |
+
"n": 200,
|
| 4 |
+
"verify_true": 125,
|
| 5 |
+
"lmi_true": 125,
|
| 6 |
+
"disagree": 0
|
| 7 |
+
}
|
results/icml_r3/X29/act/selfcheck_cert_m5.json
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"m": 5,
|
| 3 |
+
"n": 200,
|
| 4 |
+
"verify_true": 115,
|
| 5 |
+
"lmi_true": 115,
|
| 6 |
+
"disagree": 0
|
| 7 |
+
}
|
results/icml_r3/X29/act/selfcheck_cost_m3.json
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"m": 3,
|
| 3 |
+
"n": 120,
|
| 4 |
+
"passed": true,
|
| 5 |
+
"T": 20.0,
|
| 6 |
+
"tries": [
|
| 7 |
+
{
|
| 8 |
+
"T": 20.0,
|
| 9 |
+
"rel_median": 1.2934926530061185e-05,
|
| 10 |
+
"rel_max": 0.0014178489529952417,
|
| 11 |
+
"E_max": 8.026504423256033e-05
|
| 12 |
+
}
|
| 13 |
+
],
|
| 14 |
+
"J_lyap_median": 2.0978283107427393
|
| 15 |
+
}
|
results/icml_r3/X29/act/selfcheck_cost_m4.json
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"m": 4,
|
| 3 |
+
"n": 120,
|
| 4 |
+
"passed": true,
|
| 5 |
+
"T": 20.0,
|
| 6 |
+
"tries": [
|
| 7 |
+
{
|
| 8 |
+
"T": 20.0,
|
| 9 |
+
"rel_median": 1.5247918745172171e-05,
|
| 10 |
+
"rel_max": 0.0004239844159611186,
|
| 11 |
+
"E_max": 6.715112940831412e-05
|
| 12 |
+
}
|
| 13 |
+
],
|
| 14 |
+
"J_lyap_median": 2.1280300984820215
|
| 15 |
+
}
|
results/icml_r3/X29/act/selfcheck_cost_m5.json
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"m": 5,
|
| 3 |
+
"n": 120,
|
| 4 |
+
"passed": true,
|
| 5 |
+
"T": 20.0,
|
| 6 |
+
"tries": [
|
| 7 |
+
{
|
| 8 |
+
"T": 20.0,
|
| 9 |
+
"rel_median": 2.2052955031149278e-05,
|
| 10 |
+
"rel_max": 0.00025976687111925803,
|
| 11 |
+
"E_max": 0.00014713529635589263
|
| 12 |
+
}
|
| 13 |
+
],
|
| 14 |
+
"J_lyap_median": 2.009332559093612
|
| 15 |
+
}
|
results/icml_r3/X29/act/x27rescore_d3fm3n8000.json
ADDED
|
@@ -0,0 +1,250 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"diff raw_b1": {
|
| 3 |
+
"rate": 0.5166666666666666,
|
| 4 |
+
"rate_per_seed": [
|
| 5 |
+
0.5666666666666667,
|
| 6 |
+
0.5083333333333333,
|
| 7 |
+
0.475
|
| 8 |
+
],
|
| 9 |
+
"regB_ref": 0.035166553853986704,
|
| 10 |
+
"regB_star": 0.04939973422469457,
|
| 11 |
+
"regA_ref": 0.035166553853986704,
|
| 12 |
+
"regA_star": 0.04939973422469457
|
| 13 |
+
},
|
| 14 |
+
"diff raw_b4": {
|
| 15 |
+
"rate": 0.7611111111111111,
|
| 16 |
+
"rate_per_seed": [
|
| 17 |
+
0.7916666666666666,
|
| 18 |
+
0.7666666666666667,
|
| 19 |
+
0.725
|
| 20 |
+
],
|
| 21 |
+
"regB_ref": 0.026566873792452816,
|
| 22 |
+
"regB_star": 0.036538855499143374,
|
| 23 |
+
"regA_ref": 0.03912745224114178,
|
| 24 |
+
"regA_star": 0.05271375933165312
|
| 25 |
+
},
|
| 26 |
+
"diff raw_b16": {
|
| 27 |
+
"rate": 0.9444444444444443,
|
| 28 |
+
"rate_per_seed": [
|
| 29 |
+
0.9583333333333334,
|
| 30 |
+
0.95,
|
| 31 |
+
0.925
|
| 32 |
+
],
|
| 33 |
+
"regB_ref": 0.017749766148115964,
|
| 34 |
+
"regB_star": 0.02536120788249141,
|
| 35 |
+
"regA_ref": 0.03975266005687195,
|
| 36 |
+
"regA_star": 0.051211998589972385
|
| 37 |
+
},
|
| 38 |
+
"diff filt_b1": {
|
| 39 |
+
"rate": 0.6861111111111112,
|
| 40 |
+
"rate_per_seed": [
|
| 41 |
+
0.725,
|
| 42 |
+
0.75,
|
| 43 |
+
0.5833333333333334
|
| 44 |
+
],
|
| 45 |
+
"regB_ref": 0.03864968435767513,
|
| 46 |
+
"regB_star": 0.051490162113829285,
|
| 47 |
+
"regA_ref": 0.03864968435767513,
|
| 48 |
+
"regA_star": 0.051490162113829285
|
| 49 |
+
},
|
| 50 |
+
"diff filt_b4": {
|
| 51 |
+
"rate": 0.9277777777777777,
|
| 52 |
+
"rate_per_seed": [
|
| 53 |
+
0.9333333333333333,
|
| 54 |
+
0.925,
|
| 55 |
+
0.925
|
| 56 |
+
],
|
| 57 |
+
"regB_ref": 0.024622161334723724,
|
| 58 |
+
"regB_star": 0.03283620184325242,
|
| 59 |
+
"regA_ref": 0.03892824451900456,
|
| 60 |
+
"regA_star": 0.05086767155048544
|
| 61 |
+
},
|
| 62 |
+
"diff filt_b16": {
|
| 63 |
+
"rate": 0.9861111111111112,
|
| 64 |
+
"rate_per_seed": [
|
| 65 |
+
0.9833333333333333,
|
| 66 |
+
1.0,
|
| 67 |
+
0.975
|
| 68 |
+
],
|
| 69 |
+
"regB_ref": 0.014977849993612624,
|
| 70 |
+
"regB_star": 0.0218213442998485,
|
| 71 |
+
"regA_ref": 0.03976742768706387,
|
| 72 |
+
"regA_star": 0.050933835066115485
|
| 73 |
+
},
|
| 74 |
+
"diff ref_b16_n3": {
|
| 75 |
+
"rate": 0.9944444444444445,
|
| 76 |
+
"rate_per_seed": [
|
| 77 |
+
0.9916666666666667,
|
| 78 |
+
1.0,
|
| 79 |
+
0.9916666666666667
|
| 80 |
+
],
|
| 81 |
+
"regB_ref": 0.015012072869242443,
|
| 82 |
+
"regB_star": 0.021944205370784453,
|
| 83 |
+
"regA_ref": 0.03991946776173949,
|
| 84 |
+
"regA_star": 0.051211998589972385
|
| 85 |
+
},
|
| 86 |
+
"diff ref_b16_n10": {
|
| 87 |
+
"rate": 0.9972222222222222,
|
| 88 |
+
"rate_per_seed": [
|
| 89 |
+
1.0,
|
| 90 |
+
1.0,
|
| 91 |
+
0.9916666666666667
|
| 92 |
+
],
|
| 93 |
+
"regB_ref": 0.015046295744872262,
|
| 94 |
+
"regB_star": 0.022067066441720407,
|
| 95 |
+
"regA_ref": 0.04007150783641511,
|
| 96 |
+
"regA_star": 0.051490162113829285
|
| 97 |
+
},
|
| 98 |
+
"twin raw_b1": {
|
| 99 |
+
"rate": 0.4916666666666667,
|
| 100 |
+
"rate_per_seed": [
|
| 101 |
+
0.475,
|
| 102 |
+
0.5,
|
| 103 |
+
0.5
|
| 104 |
+
],
|
| 105 |
+
"regB_ref": 0.034868968961000446,
|
| 106 |
+
"regB_star": 0.045860235685736495,
|
| 107 |
+
"regA_ref": 0.034868968961000446,
|
| 108 |
+
"regA_star": 0.045860235685736495
|
| 109 |
+
},
|
| 110 |
+
"twin raw_b4": {
|
| 111 |
+
"rate": 0.6222222222222222,
|
| 112 |
+
"rate_per_seed": [
|
| 113 |
+
0.6333333333333333,
|
| 114 |
+
0.6333333333333333,
|
| 115 |
+
0.6
|
| 116 |
+
],
|
| 117 |
+
"regB_ref": 0.031090565068918607,
|
| 118 |
+
"regB_star": 0.04009868135354511,
|
| 119 |
+
"regA_ref": 0.03602559875874334,
|
| 120 |
+
"regA_star": 0.04612835952042982
|
| 121 |
+
},
|
| 122 |
+
"twin raw_b16": {
|
| 123 |
+
"rate": 0.6888888888888888,
|
| 124 |
+
"rate_per_seed": [
|
| 125 |
+
0.725,
|
| 126 |
+
0.6583333333333333,
|
| 127 |
+
0.6833333333333333
|
| 128 |
+
],
|
| 129 |
+
"regB_ref": 0.024827462390698996,
|
| 130 |
+
"regB_star": 0.034008343863723534,
|
| 131 |
+
"regA_ref": 0.03665811970701527,
|
| 132 |
+
"regA_star": 0.04772048027118436
|
| 133 |
+
},
|
| 134 |
+
"twin filt_b1": {
|
| 135 |
+
"rate": 0.5722222222222222,
|
| 136 |
+
"rate_per_seed": [
|
| 137 |
+
0.5583333333333333,
|
| 138 |
+
0.5833333333333334,
|
| 139 |
+
0.575
|
| 140 |
+
],
|
| 141 |
+
"regB_ref": 0.035172887226527116,
|
| 142 |
+
"regB_star": 0.045343068252056584,
|
| 143 |
+
"regA_ref": 0.035172887226527116,
|
| 144 |
+
"regA_star": 0.045343068252056584
|
| 145 |
+
},
|
| 146 |
+
"twin filt_b4": {
|
| 147 |
+
"rate": 0.6805555555555557,
|
| 148 |
+
"rate_per_seed": [
|
| 149 |
+
0.7,
|
| 150 |
+
0.6416666666666667,
|
| 151 |
+
0.7
|
| 152 |
+
],
|
| 153 |
+
"regB_ref": 0.03138655281275926,
|
| 154 |
+
"regB_star": 0.040342528272814926,
|
| 155 |
+
"regA_ref": 0.036638098220030946,
|
| 156 |
+
"regA_star": 0.04700456890654414
|
| 157 |
+
},
|
| 158 |
+
"twin filt_b16": {
|
| 159 |
+
"rate": 0.7305555555555555,
|
| 160 |
+
"rate_per_seed": [
|
| 161 |
+
0.75,
|
| 162 |
+
0.7,
|
| 163 |
+
0.7416666666666667
|
| 164 |
+
],
|
| 165 |
+
"regB_ref": 0.02495787433019081,
|
| 166 |
+
"regB_star": 0.03388894221642125,
|
| 167 |
+
"regA_ref": 0.036800685855217674,
|
| 168 |
+
"regA_star": 0.04793953255697314
|
| 169 |
+
},
|
| 170 |
+
"knn raw_b1": {
|
| 171 |
+
"rate": 0.25833333333333336,
|
| 172 |
+
"rate_per_seed": [
|
| 173 |
+
0.25833333333333336
|
| 174 |
+
],
|
| 175 |
+
"regB_ref": 0.11371326220225075,
|
| 176 |
+
"regB_star": 0.11629817415072874,
|
| 177 |
+
"regA_ref": 0.11371326220225075,
|
| 178 |
+
"regA_star": 0.11629817415072874
|
| 179 |
+
},
|
| 180 |
+
"knn raw_b4": {
|
| 181 |
+
"rate": 0.44166666666666665,
|
| 182 |
+
"rate_per_seed": [
|
| 183 |
+
0.44166666666666665
|
| 184 |
+
],
|
| 185 |
+
"regB_ref": 0.09416819337804627,
|
| 186 |
+
"regB_star": 0.11250915295445885,
|
| 187 |
+
"regA_ref": 0.11635060647060991,
|
| 188 |
+
"regA_star": 0.13716816454409586
|
| 189 |
+
},
|
| 190 |
+
"knn raw_b16": {
|
| 191 |
+
"rate": 0.7083333333333334,
|
| 192 |
+
"rate_per_seed": [
|
| 193 |
+
0.7083333333333334
|
| 194 |
+
],
|
| 195 |
+
"regB_ref": 0.08590952578105893,
|
| 196 |
+
"regB_star": 0.09869180446652481,
|
| 197 |
+
"regA_ref": 0.13353007468957645,
|
| 198 |
+
"regA_star": 0.13956844190360695
|
| 199 |
+
},
|
| 200 |
+
"knn filt_b1": {
|
| 201 |
+
"rate": 0.5083333333333333,
|
| 202 |
+
"rate_per_seed": [
|
| 203 |
+
0.5083333333333333
|
| 204 |
+
],
|
| 205 |
+
"regB_ref": 0.134908830251387,
|
| 206 |
+
"regB_star": 0.14090533433695862,
|
| 207 |
+
"regA_ref": 0.134908830251387,
|
| 208 |
+
"regA_star": 0.14090533433695862
|
| 209 |
+
},
|
| 210 |
+
"knn filt_b4": {
|
| 211 |
+
"rate": 0.7833333333333333,
|
| 212 |
+
"rate_per_seed": [
|
| 213 |
+
0.7833333333333333
|
| 214 |
+
],
|
| 215 |
+
"regB_ref": 0.10139965849798827,
|
| 216 |
+
"regB_star": 0.10879331316422458,
|
| 217 |
+
"regA_ref": 0.13114353705654402,
|
| 218 |
+
"regA_star": 0.1383683032238514
|
| 219 |
+
},
|
| 220 |
+
"knn filt_b16": {
|
| 221 |
+
"rate": 0.8666666666666667,
|
| 222 |
+
"rate_per_seed": [
|
| 223 |
+
0.8666666666666667
|
| 224 |
+
],
|
| 225 |
+
"regB_ref": 0.07096639087530376,
|
| 226 |
+
"regB_star": 0.08244864896963622,
|
| 227 |
+
"regA_ref": 0.13353036702921328,
|
| 228 |
+
"regA_star": 0.1383683032238514
|
| 229 |
+
},
|
| 230 |
+
"knn ref_b16_n3": {
|
| 231 |
+
"rate": 0.9333333333333333,
|
| 232 |
+
"rate_per_seed": [
|
| 233 |
+
0.9333333333333333
|
| 234 |
+
],
|
| 235 |
+
"regB_ref": 0.07228077375089703,
|
| 236 |
+
"regB_star": 0.0830961623057771,
|
| 237 |
+
"regA_ref": 0.12499251210513573,
|
| 238 |
+
"regA_star": 0.13353036702921328
|
| 239 |
+
},
|
| 240 |
+
"knn ref_b16_n10": {
|
| 241 |
+
"rate": 0.9333333333333333,
|
| 242 |
+
"rate_per_seed": [
|
| 243 |
+
0.9333333333333333
|
| 244 |
+
],
|
| 245 |
+
"regB_ref": 0.07337219985527899,
|
| 246 |
+
"regB_star": 0.08575503047710231,
|
| 247 |
+
"regA_ref": 0.12802583666738943,
|
| 248 |
+
"regA_star": 0.13421974481011856
|
| 249 |
+
}
|
| 250 |
+
}
|
results/icml_r3/X29/act/x27rescore_d3fm3n8000.txt
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
X29-A: X27's C-only arms re-scored, m = 3, tag d3fm3n8000, 120 test plants. J*_A = min(J_ref, 3A id and so3 labels) is below J_ref on 45 plants (median (J_ref - J*_A) / J*_A 0.000%).
|
| 2 |
+
diff raw_b1 certified 51.7% (published 51.7%) | cheapest certified: regret vs J_ref 3.52% (published 3.52%), vs J*_A 4.94% | first certified: vs J_ref 3.52%, vs J*_A 4.94%
|
| 3 |
+
diff raw_b4 certified 76.1% (published 76.1%) | cheapest certified: regret vs J_ref 2.66% (published 2.66%), vs J*_A 3.65% | first certified: vs J_ref 3.91%, vs J*_A 5.27%
|
| 4 |
+
diff raw_b16 certified 94.4% (published 94.4%) | cheapest certified: regret vs J_ref 1.77% (published 1.77%), vs J*_A 2.54% | first certified: vs J_ref 3.98%, vs J*_A 5.12%
|
| 5 |
+
diff filt_b1 certified 68.6% (published 68.6%) | cheapest certified: regret vs J_ref 3.86% (published 3.86%), vs J*_A 5.15% | first certified: vs J_ref 3.86%, vs J*_A 5.15%
|
| 6 |
+
diff filt_b4 certified 92.8% (published 92.8%) | cheapest certified: regret vs J_ref 2.46% (published 2.46%), vs J*_A 3.28% | first certified: vs J_ref 3.89%, vs J*_A 5.09%
|
| 7 |
+
diff filt_b16 certified 98.6% (published 98.6%) | cheapest certified: regret vs J_ref 1.50% (published 1.50%), vs J*_A 2.18% | first certified: vs J_ref 3.98%, vs J*_A 5.09%
|
| 8 |
+
diff ref_b16_n3 certified 99.4% (published 99.4%) | cheapest certified: regret vs J_ref 1.50% (published 1.50%), vs J*_A 2.19% | first certified: vs J_ref 3.99%, vs J*_A 5.12%
|
| 9 |
+
diff ref_b16_n10 certified 99.7% (published 99.7%) | cheapest certified: regret vs J_ref 1.50% (published 1.50%), vs J*_A 2.21% | first certified: vs J_ref 4.01%, vs J*_A 5.15%
|
| 10 |
+
twin raw_b1 certified 49.2% (published 49.2%) | cheapest certified: regret vs J_ref 3.49% (published 3.49%), vs J*_A 4.59% | first certified: vs J_ref 3.49%, vs J*_A 4.59%
|
| 11 |
+
twin raw_b4 certified 62.2% (published 62.2%) | cheapest certified: regret vs J_ref 3.11% (published 3.11%), vs J*_A 4.01% | first certified: vs J_ref 3.60%, vs J*_A 4.61%
|
| 12 |
+
twin raw_b16 certified 68.9% (published 68.9%) | cheapest certified: regret vs J_ref 2.48% (published 2.48%), vs J*_A 3.40% | first certified: vs J_ref 3.67%, vs J*_A 4.77%
|
| 13 |
+
twin filt_b1 certified 57.2% (published 57.2%) | cheapest certified: regret vs J_ref 3.52% (published 3.52%), vs J*_A 4.53% | first certified: vs J_ref 3.52%, vs J*_A 4.53%
|
| 14 |
+
twin filt_b4 certified 68.1% (published 68.1%) | cheapest certified: regret vs J_ref 3.14% (published 3.14%), vs J*_A 4.03% | first certified: vs J_ref 3.66%, vs J*_A 4.70%
|
| 15 |
+
twin filt_b16 certified 73.1% (published 73.1%) | cheapest certified: regret vs J_ref 2.50% (published 2.50%), vs J*_A 3.39% | first certified: vs J_ref 3.68%, vs J*_A 4.79%
|
| 16 |
+
knn raw_b1 certified 25.8% (published 25.8%) | cheapest certified: regret vs J_ref 11.37% (published 11.37%), vs J*_A 11.63% | first certified: vs J_ref 11.37%, vs J*_A 11.63%
|
| 17 |
+
knn raw_b4 certified 44.2% (published 44.2%) | cheapest certified: regret vs J_ref 9.42% (published 9.42%), vs J*_A 11.25% | first certified: vs J_ref 11.64%, vs J*_A 13.72%
|
| 18 |
+
knn raw_b16 certified 70.8% (published 70.8%) | cheapest certified: regret vs J_ref 8.59% (published 8.59%), vs J*_A 9.87% | first certified: vs J_ref 13.35%, vs J*_A 13.96%
|
| 19 |
+
knn filt_b1 certified 50.8% (published 50.8%) | cheapest certified: regret vs J_ref 13.49% (published 13.49%), vs J*_A 14.09% | first certified: vs J_ref 13.49%, vs J*_A 14.09%
|
| 20 |
+
knn filt_b4 certified 78.3% (published 78.3%) | cheapest certified: regret vs J_ref 10.14% (published 10.14%), vs J*_A 10.88% | first certified: vs J_ref 13.11%, vs J*_A 13.84%
|
| 21 |
+
knn filt_b16 certified 86.7% (published 86.7%) | cheapest certified: regret vs J_ref 7.10% (published 7.10%), vs J*_A 8.24% | first certified: vs J_ref 13.35%, vs J*_A 13.84%
|
| 22 |
+
knn ref_b16_n3 certified 93.3% (published 93.3%) | cheapest certified: regret vs J_ref 7.23% (published 7.23%), vs J*_A 8.31% | first certified: vs J_ref 12.50%, vs J*_A 13.35%
|
| 23 |
+
knn ref_b16_n10 certified 93.3% (published 93.3%) | cheapest certified: regret vs J_ref 7.34% (published 7.34%), vs J*_A 8.58% | first certified: vs J_ref 12.80%, vs J*_A 13.42%
|
results/icml_r3/X29/act/x27rescore_d3fm4n8000.json
ADDED
|
@@ -0,0 +1,250 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"diff raw_b1": {
|
| 3 |
+
"rate": 0.31666666666666665,
|
| 4 |
+
"rate_per_seed": [
|
| 5 |
+
0.30833333333333335,
|
| 6 |
+
0.325,
|
| 7 |
+
0.31666666666666665
|
| 8 |
+
],
|
| 9 |
+
"regB_ref": 0.04281187301679423,
|
| 10 |
+
"regB_star": 0.04962915968635695,
|
| 11 |
+
"regA_ref": 0.04281187301679423,
|
| 12 |
+
"regA_star": 0.04962915968635695
|
| 13 |
+
},
|
| 14 |
+
"diff raw_b4": {
|
| 15 |
+
"rate": 0.5722222222222222,
|
| 16 |
+
"rate_per_seed": [
|
| 17 |
+
0.6,
|
| 18 |
+
0.5666666666666667,
|
| 19 |
+
0.55
|
| 20 |
+
],
|
| 21 |
+
"regB_ref": 0.04023679269097036,
|
| 22 |
+
"regB_star": 0.047381757855627926,
|
| 23 |
+
"regA_ref": 0.04831828056771548,
|
| 24 |
+
"regA_star": 0.057019597285727364
|
| 25 |
+
},
|
| 26 |
+
"diff raw_b16": {
|
| 27 |
+
"rate": 0.7888888888888889,
|
| 28 |
+
"rate_per_seed": [
|
| 29 |
+
0.8,
|
| 30 |
+
0.775,
|
| 31 |
+
0.7916666666666666
|
| 32 |
+
],
|
| 33 |
+
"regB_ref": 0.03206114493999934,
|
| 34 |
+
"regB_star": 0.03878398370676894,
|
| 35 |
+
"regA_ref": 0.05338276992090041,
|
| 36 |
+
"regA_star": 0.05942518962649945
|
| 37 |
+
},
|
| 38 |
+
"diff filt_b1": {
|
| 39 |
+
"rate": 0.5249999999999999,
|
| 40 |
+
"rate_per_seed": [
|
| 41 |
+
0.5083333333333333,
|
| 42 |
+
0.5333333333333333,
|
| 43 |
+
0.5333333333333333
|
| 44 |
+
],
|
| 45 |
+
"regB_ref": 0.05256149004484412,
|
| 46 |
+
"regB_star": 0.05789482111431554,
|
| 47 |
+
"regA_ref": 0.05256149004484412,
|
| 48 |
+
"regA_star": 0.05789482111431554
|
| 49 |
+
},
|
| 50 |
+
"diff filt_b4": {
|
| 51 |
+
"rate": 0.8250000000000001,
|
| 52 |
+
"rate_per_seed": [
|
| 53 |
+
0.8166666666666667,
|
| 54 |
+
0.8166666666666667,
|
| 55 |
+
0.8416666666666667
|
| 56 |
+
],
|
| 57 |
+
"regB_ref": 0.04154891326937804,
|
| 58 |
+
"regB_star": 0.05096408010028841,
|
| 59 |
+
"regA_ref": 0.05679915606087227,
|
| 60 |
+
"regA_star": 0.06378147142694024
|
| 61 |
+
},
|
| 62 |
+
"diff filt_b16": {
|
| 63 |
+
"rate": 0.9194444444444444,
|
| 64 |
+
"rate_per_seed": [
|
| 65 |
+
0.8916666666666667,
|
| 66 |
+
0.9166666666666666,
|
| 67 |
+
0.95
|
| 68 |
+
],
|
| 69 |
+
"regB_ref": 0.030428223592535053,
|
| 70 |
+
"regB_star": 0.03729421986230963,
|
| 71 |
+
"regA_ref": 0.05678969283770341,
|
| 72 |
+
"regA_star": 0.06378147142694024
|
| 73 |
+
},
|
| 74 |
+
"diff ref_b16_n3": {
|
| 75 |
+
"rate": 0.9666666666666667,
|
| 76 |
+
"rate_per_seed": [
|
| 77 |
+
0.975,
|
| 78 |
+
0.95,
|
| 79 |
+
0.975
|
| 80 |
+
],
|
| 81 |
+
"regB_ref": 0.032163477351268455,
|
| 82 |
+
"regB_star": 0.03848414313894111,
|
| 83 |
+
"regA_ref": 0.05752555203944998,
|
| 84 |
+
"regA_star": 0.06417574626693279
|
| 85 |
+
},
|
| 86 |
+
"diff ref_b16_n10": {
|
| 87 |
+
"rate": 0.9777777777777777,
|
| 88 |
+
"rate_per_seed": [
|
| 89 |
+
0.9833333333333333,
|
| 90 |
+
0.9666666666666667,
|
| 91 |
+
0.9833333333333333
|
| 92 |
+
],
|
| 93 |
+
"regB_ref": 0.03239370322615842,
|
| 94 |
+
"regB_star": 0.03848414313894111,
|
| 95 |
+
"regA_ref": 0.0571452435546439,
|
| 96 |
+
"regA_star": 0.0639596586478054
|
| 97 |
+
},
|
| 98 |
+
"twin raw_b1": {
|
| 99 |
+
"rate": 0.3138888888888889,
|
| 100 |
+
"rate_per_seed": [
|
| 101 |
+
0.31666666666666665,
|
| 102 |
+
0.275,
|
| 103 |
+
0.35
|
| 104 |
+
],
|
| 105 |
+
"regB_ref": 0.05077043279796922,
|
| 106 |
+
"regB_star": 0.054942478154655694,
|
| 107 |
+
"regA_ref": 0.05077043279796922,
|
| 108 |
+
"regA_star": 0.054942478154655694
|
| 109 |
+
},
|
| 110 |
+
"twin raw_b4": {
|
| 111 |
+
"rate": 0.4305555555555555,
|
| 112 |
+
"rate_per_seed": [
|
| 113 |
+
0.4083333333333333,
|
| 114 |
+
0.44166666666666665,
|
| 115 |
+
0.44166666666666665
|
| 116 |
+
],
|
| 117 |
+
"regB_ref": 0.040770558505952126,
|
| 118 |
+
"regB_star": 0.04486400154999837,
|
| 119 |
+
"regA_ref": 0.044869776269671036,
|
| 120 |
+
"regA_star": 0.051729042499672806
|
| 121 |
+
},
|
| 122 |
+
"twin raw_b16": {
|
| 123 |
+
"rate": 0.48333333333333334,
|
| 124 |
+
"rate_per_seed": [
|
| 125 |
+
0.4583333333333333,
|
| 126 |
+
0.49166666666666664,
|
| 127 |
+
0.5
|
| 128 |
+
],
|
| 129 |
+
"regB_ref": 0.03419325650322769,
|
| 130 |
+
"regB_star": 0.03997382209205025,
|
| 131 |
+
"regA_ref": 0.04478727476519462,
|
| 132 |
+
"regA_star": 0.051812151699132114
|
| 133 |
+
},
|
| 134 |
+
"twin filt_b1": {
|
| 135 |
+
"rate": 0.3666666666666667,
|
| 136 |
+
"rate_per_seed": [
|
| 137 |
+
0.38333333333333336,
|
| 138 |
+
0.31666666666666665,
|
| 139 |
+
0.4
|
| 140 |
+
],
|
| 141 |
+
"regB_ref": 0.0509702614716806,
|
| 142 |
+
"regB_star": 0.05496224880262157,
|
| 143 |
+
"regA_ref": 0.0509702614716806,
|
| 144 |
+
"regA_star": 0.05496224880262157
|
| 145 |
+
},
|
| 146 |
+
"twin filt_b4": {
|
| 147 |
+
"rate": 0.4861111111111111,
|
| 148 |
+
"rate_per_seed": [
|
| 149 |
+
0.475,
|
| 150 |
+
0.48333333333333334,
|
| 151 |
+
0.5
|
| 152 |
+
],
|
| 153 |
+
"regB_ref": 0.039893127429255326,
|
| 154 |
+
"regB_star": 0.044859535745951584,
|
| 155 |
+
"regA_ref": 0.044869776269671036,
|
| 156 |
+
"regA_star": 0.051729042499672806
|
| 157 |
+
},
|
| 158 |
+
"twin filt_b16": {
|
| 159 |
+
"rate": 0.5222222222222221,
|
| 160 |
+
"rate_per_seed": [
|
| 161 |
+
0.5083333333333333,
|
| 162 |
+
0.525,
|
| 163 |
+
0.5333333333333333
|
| 164 |
+
],
|
| 165 |
+
"regB_ref": 0.03461930665839119,
|
| 166 |
+
"regB_star": 0.039952457345226344,
|
| 167 |
+
"regA_ref": 0.045169830011552614,
|
| 168 |
+
"regA_star": 0.05347942435820763
|
| 169 |
+
},
|
| 170 |
+
"knn raw_b1": {
|
| 171 |
+
"rate": 0.14166666666666666,
|
| 172 |
+
"rate_per_seed": [
|
| 173 |
+
0.14166666666666666
|
| 174 |
+
],
|
| 175 |
+
"regB_ref": 0.11972585447859752,
|
| 176 |
+
"regB_star": 0.1203804444936909,
|
| 177 |
+
"regA_ref": 0.11972585447859752,
|
| 178 |
+
"regA_star": 0.1203804444936909
|
| 179 |
+
},
|
| 180 |
+
"knn raw_b4": {
|
| 181 |
+
"rate": 0.39166666666666666,
|
| 182 |
+
"rate_per_seed": [
|
| 183 |
+
0.39166666666666666
|
| 184 |
+
],
|
| 185 |
+
"regB_ref": 0.11972585447859752,
|
| 186 |
+
"regB_star": 0.1249036336281455,
|
| 187 |
+
"regA_ref": 0.1249036336281455,
|
| 188 |
+
"regA_star": 0.13069438483572582
|
| 189 |
+
},
|
| 190 |
+
"knn raw_b16": {
|
| 191 |
+
"rate": 0.5916666666666667,
|
| 192 |
+
"rate_per_seed": [
|
| 193 |
+
0.5916666666666667
|
| 194 |
+
],
|
| 195 |
+
"regB_ref": 0.10947876745055107,
|
| 196 |
+
"regB_star": 0.11616937090964607,
|
| 197 |
+
"regA_ref": 0.13058469845406107,
|
| 198 |
+
"regA_star": 0.1400175969983004
|
| 199 |
+
},
|
| 200 |
+
"knn filt_b1": {
|
| 201 |
+
"rate": 0.45,
|
| 202 |
+
"rate_per_seed": [
|
| 203 |
+
0.45
|
| 204 |
+
],
|
| 205 |
+
"regB_ref": 0.11863223190950545,
|
| 206 |
+
"regB_star": 0.12336736317427233,
|
| 207 |
+
"regA_ref": 0.11863223190950545,
|
| 208 |
+
"regA_star": 0.12336736317427233
|
| 209 |
+
},
|
| 210 |
+
"knn filt_b4": {
|
| 211 |
+
"rate": 0.75,
|
| 212 |
+
"rate_per_seed": [
|
| 213 |
+
0.75
|
| 214 |
+
],
|
| 215 |
+
"regB_ref": 0.11100685037250602,
|
| 216 |
+
"regB_star": 0.11698292149647704,
|
| 217 |
+
"regA_ref": 0.13511513600466607,
|
| 218 |
+
"regA_star": 0.14419869136826147
|
| 219 |
+
},
|
| 220 |
+
"knn filt_b16": {
|
| 221 |
+
"rate": 0.7916666666666666,
|
| 222 |
+
"rate_per_seed": [
|
| 223 |
+
0.7916666666666666
|
| 224 |
+
],
|
| 225 |
+
"regB_ref": 0.0926797810653952,
|
| 226 |
+
"regB_star": 0.09917966684483925,
|
| 227 |
+
"regA_ref": 0.1400175969983004,
|
| 228 |
+
"regA_star": 0.14697604799704078
|
| 229 |
+
},
|
| 230 |
+
"knn ref_b16_n3": {
|
| 231 |
+
"rate": 0.9,
|
| 232 |
+
"rate_per_seed": [
|
| 233 |
+
0.9
|
| 234 |
+
],
|
| 235 |
+
"regB_ref": 0.09150018593164855,
|
| 236 |
+
"regB_star": 0.09801292912248993,
|
| 237 |
+
"regA_ref": 0.12780736488049338,
|
| 238 |
+
"regA_star": 0.13983158527678574
|
| 239 |
+
},
|
| 240 |
+
"knn ref_b16_n10": {
|
| 241 |
+
"rate": 0.9333333333333333,
|
| 242 |
+
"rate_per_seed": [
|
| 243 |
+
0.9333333333333333
|
| 244 |
+
],
|
| 245 |
+
"regB_ref": 0.09026618689333521,
|
| 246 |
+
"regB_star": 0.09677288836815534,
|
| 247 |
+
"regA_ref": 0.1258708743514112,
|
| 248 |
+
"regA_star": 0.1349668764575651
|
| 249 |
+
}
|
| 250 |
+
}
|
results/icml_r3/X29/act/x27rescore_d3fm4n8000.txt
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
X29-A: X27's C-only arms re-scored, m = 4, tag d3fm4n8000, 120 test plants. J*_A = min(J_ref, 3A id and so3 labels) is below J_ref on 48 plants (median (J_ref - J*_A) / J*_A 0.000%).
|
| 2 |
+
diff raw_b1 certified 31.7% (published 31.7%) | cheapest certified: regret vs J_ref 4.28% (published 4.28%), vs J*_A 4.96% | first certified: vs J_ref 4.28%, vs J*_A 4.96%
|
| 3 |
+
diff raw_b4 certified 57.2% (published 57.2%) | cheapest certified: regret vs J_ref 4.02% (published 4.02%), vs J*_A 4.74% | first certified: vs J_ref 4.83%, vs J*_A 5.70%
|
| 4 |
+
diff raw_b16 certified 78.9% (published 78.9%) | cheapest certified: regret vs J_ref 3.21% (published 3.21%), vs J*_A 3.88% | first certified: vs J_ref 5.34%, vs J*_A 5.94%
|
| 5 |
+
diff filt_b1 certified 52.5% (published 52.5%) | cheapest certified: regret vs J_ref 5.26% (published 5.26%), vs J*_A 5.79% | first certified: vs J_ref 5.26%, vs J*_A 5.79%
|
| 6 |
+
diff filt_b4 certified 82.5% (published 82.5%) | cheapest certified: regret vs J_ref 4.15% (published 4.15%), vs J*_A 5.10% | first certified: vs J_ref 5.68%, vs J*_A 6.38%
|
| 7 |
+
diff filt_b16 certified 91.9% (published 91.9%) | cheapest certified: regret vs J_ref 3.04% (published 3.04%), vs J*_A 3.73% | first certified: vs J_ref 5.68%, vs J*_A 6.38%
|
| 8 |
+
diff ref_b16_n3 certified 96.7% (published 96.7%) | cheapest certified: regret vs J_ref 3.22% (published 3.22%), vs J*_A 3.85% | first certified: vs J_ref 5.75%, vs J*_A 6.42%
|
| 9 |
+
diff ref_b16_n10 certified 97.8% (published 97.8%) | cheapest certified: regret vs J_ref 3.24% (published 3.24%), vs J*_A 3.85% | first certified: vs J_ref 5.71%, vs J*_A 6.40%
|
| 10 |
+
twin raw_b1 certified 31.4% (published 31.4%) | cheapest certified: regret vs J_ref 5.08% (published 5.08%), vs J*_A 5.49% | first certified: vs J_ref 5.08%, vs J*_A 5.49%
|
| 11 |
+
twin raw_b4 certified 43.1% (published 43.1%) | cheapest certified: regret vs J_ref 4.08% (published 4.08%), vs J*_A 4.49% | first certified: vs J_ref 4.49%, vs J*_A 5.17%
|
| 12 |
+
twin raw_b16 certified 48.3% (published 48.3%) | cheapest certified: regret vs J_ref 3.42% (published 3.42%), vs J*_A 4.00% | first certified: vs J_ref 4.48%, vs J*_A 5.18%
|
| 13 |
+
twin filt_b1 certified 36.7% (published 36.7%) | cheapest certified: regret vs J_ref 5.10% (published 5.10%), vs J*_A 5.50% | first certified: vs J_ref 5.10%, vs J*_A 5.50%
|
| 14 |
+
twin filt_b4 certified 48.6% (published 48.6%) | cheapest certified: regret vs J_ref 3.99% (published 3.99%), vs J*_A 4.49% | first certified: vs J_ref 4.49%, vs J*_A 5.17%
|
| 15 |
+
twin filt_b16 certified 52.2% (published 52.2%) | cheapest certified: regret vs J_ref 3.46% (published 3.46%), vs J*_A 4.00% | first certified: vs J_ref 4.52%, vs J*_A 5.35%
|
| 16 |
+
knn raw_b1 certified 14.2% (published 14.2%) | cheapest certified: regret vs J_ref 11.97% (published 11.97%), vs J*_A 12.04% | first certified: vs J_ref 11.97%, vs J*_A 12.04%
|
| 17 |
+
knn raw_b4 certified 39.2% (published 39.2%) | cheapest certified: regret vs J_ref 11.97% (published 11.97%), vs J*_A 12.49% | first certified: vs J_ref 12.49%, vs J*_A 13.07%
|
| 18 |
+
knn raw_b16 certified 59.2% (published 59.2%) | cheapest certified: regret vs J_ref 10.95% (published 10.95%), vs J*_A 11.62% | first certified: vs J_ref 13.06%, vs J*_A 14.00%
|
| 19 |
+
knn filt_b1 certified 45.0% (published 45.0%) | cheapest certified: regret vs J_ref 11.86% (published 11.86%), vs J*_A 12.34% | first certified: vs J_ref 11.86%, vs J*_A 12.34%
|
| 20 |
+
knn filt_b4 certified 75.0% (published 75.0%) | cheapest certified: regret vs J_ref 11.10% (published 11.10%), vs J*_A 11.70% | first certified: vs J_ref 13.51%, vs J*_A 14.42%
|
| 21 |
+
knn filt_b16 certified 79.2% (published 79.2%) | cheapest certified: regret vs J_ref 9.27% (published 9.27%), vs J*_A 9.92% | first certified: vs J_ref 14.00%, vs J*_A 14.70%
|
| 22 |
+
knn ref_b16_n3 certified 90.0% (published 90.0%) | cheapest certified: regret vs J_ref 9.15% (published 9.15%), vs J*_A 9.80% | first certified: vs J_ref 12.78%, vs J*_A 13.98%
|
| 23 |
+
knn ref_b16_n10 certified 93.3% (published 93.3%) | cheapest certified: regret vs J_ref 9.03% (published 9.03%), vs J*_A 9.68% | first certified: vs J_ref 12.59%, vs J*_A 13.50%
|
results/icml_r3/X29/act/x27rescore_d3fm5n16000.json
ADDED
|
@@ -0,0 +1,250 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"diff raw_b1": {
|
| 3 |
+
"rate": 0.23611111111111113,
|
| 4 |
+
"rate_per_seed": [
|
| 5 |
+
0.24166666666666667,
|
| 6 |
+
0.225,
|
| 7 |
+
0.24166666666666667
|
| 8 |
+
],
|
| 9 |
+
"regB_ref": 0.054012242469847704,
|
| 10 |
+
"regB_star": 0.06580516710136042,
|
| 11 |
+
"regA_ref": 0.054012242469847704,
|
| 12 |
+
"regA_star": 0.06580516710136042
|
| 13 |
+
},
|
| 14 |
+
"diff raw_b4": {
|
| 15 |
+
"rate": 0.49166666666666664,
|
| 16 |
+
"rate_per_seed": [
|
| 17 |
+
0.5083333333333333,
|
| 18 |
+
0.525,
|
| 19 |
+
0.44166666666666665
|
| 20 |
+
],
|
| 21 |
+
"regB_ref": 0.046037766193027574,
|
| 22 |
+
"regB_star": 0.055668722380055824,
|
| 23 |
+
"regA_ref": 0.05123443031809827,
|
| 24 |
+
"regA_star": 0.06242150229911836
|
| 25 |
+
},
|
| 26 |
+
"diff raw_b16": {
|
| 27 |
+
"rate": 0.7916666666666666,
|
| 28 |
+
"rate_per_seed": [
|
| 29 |
+
0.7833333333333333,
|
| 30 |
+
0.8,
|
| 31 |
+
0.7916666666666666
|
| 32 |
+
],
|
| 33 |
+
"regB_ref": 0.0392393348275164,
|
| 34 |
+
"regB_star": 0.04843269982130982,
|
| 35 |
+
"regA_ref": 0.053670339354187746,
|
| 36 |
+
"regA_star": 0.0653423576378871
|
| 37 |
+
},
|
| 38 |
+
"diff filt_b1": {
|
| 39 |
+
"rate": 0.4583333333333333,
|
| 40 |
+
"rate_per_seed": [
|
| 41 |
+
0.45,
|
| 42 |
+
0.475,
|
| 43 |
+
0.45
|
| 44 |
+
],
|
| 45 |
+
"regB_ref": 0.05194845381582125,
|
| 46 |
+
"regB_star": 0.06513679765481717,
|
| 47 |
+
"regA_ref": 0.05194845381582125,
|
| 48 |
+
"regA_star": 0.06513679765481717
|
| 49 |
+
},
|
| 50 |
+
"diff filt_b4": {
|
| 51 |
+
"rate": 0.8138888888888888,
|
| 52 |
+
"rate_per_seed": [
|
| 53 |
+
0.8416666666666667,
|
| 54 |
+
0.8083333333333333,
|
| 55 |
+
0.7916666666666666
|
| 56 |
+
],
|
| 57 |
+
"regB_ref": 0.044054119434560945,
|
| 58 |
+
"regB_star": 0.05117955925012052,
|
| 59 |
+
"regA_ref": 0.053670339354187746,
|
| 60 |
+
"regA_star": 0.06526846423539756
|
| 61 |
+
},
|
| 62 |
+
"diff filt_b16": {
|
| 63 |
+
"rate": 0.9083333333333333,
|
| 64 |
+
"rate_per_seed": [
|
| 65 |
+
0.9416666666666667,
|
| 66 |
+
0.9083333333333333,
|
| 67 |
+
0.875
|
| 68 |
+
],
|
| 69 |
+
"regB_ref": 0.033760278025647406,
|
| 70 |
+
"regB_star": 0.0392393348275164,
|
| 71 |
+
"regA_ref": 0.05494152341582326,
|
| 72 |
+
"regA_star": 0.06614503852117859
|
| 73 |
+
},
|
| 74 |
+
"diff ref_b16_n3": {
|
| 75 |
+
"rate": 0.9638888888888889,
|
| 76 |
+
"rate_per_seed": [
|
| 77 |
+
0.9833333333333333,
|
| 78 |
+
0.95,
|
| 79 |
+
0.9583333333333334
|
| 80 |
+
],
|
| 81 |
+
"regB_ref": 0.034359315053998474,
|
| 82 |
+
"regB_star": 0.039602163709273386,
|
| 83 |
+
"regA_ref": 0.056236553327202676,
|
| 84 |
+
"regA_star": 0.06713280900832053
|
| 85 |
+
},
|
| 86 |
+
"diff ref_b16_n10": {
|
| 87 |
+
"rate": 0.9805555555555556,
|
| 88 |
+
"rate_per_seed": [
|
| 89 |
+
0.9916666666666667,
|
| 90 |
+
0.975,
|
| 91 |
+
0.975
|
| 92 |
+
],
|
| 93 |
+
"regB_ref": 0.03437394402248328,
|
| 94 |
+
"regB_star": 0.040009974164940454,
|
| 95 |
+
"regA_ref": 0.05649656778792678,
|
| 96 |
+
"regA_star": 0.06751706328759521
|
| 97 |
+
},
|
| 98 |
+
"twin raw_b1": {
|
| 99 |
+
"rate": 0.24722222222222223,
|
| 100 |
+
"rate_per_seed": [
|
| 101 |
+
0.2833333333333333,
|
| 102 |
+
0.24166666666666667,
|
| 103 |
+
0.21666666666666667
|
| 104 |
+
],
|
| 105 |
+
"regB_ref": 0.04588446769434418,
|
| 106 |
+
"regB_star": 0.05281368503715744,
|
| 107 |
+
"regA_ref": 0.04588446769434418,
|
| 108 |
+
"regA_star": 0.05281368503715744
|
| 109 |
+
},
|
| 110 |
+
"twin raw_b4": {
|
| 111 |
+
"rate": 0.37777777777777777,
|
| 112 |
+
"rate_per_seed": [
|
| 113 |
+
0.43333333333333335,
|
| 114 |
+
0.3333333333333333,
|
| 115 |
+
0.36666666666666664
|
| 116 |
+
],
|
| 117 |
+
"regB_ref": 0.03958246856186798,
|
| 118 |
+
"regB_star": 0.046969980502976294,
|
| 119 |
+
"regA_ref": 0.04453505836815008,
|
| 120 |
+
"regA_star": 0.05310045978676048
|
| 121 |
+
},
|
| 122 |
+
"twin raw_b16": {
|
| 123 |
+
"rate": 0.4916666666666667,
|
| 124 |
+
"rate_per_seed": [
|
| 125 |
+
0.5416666666666666,
|
| 126 |
+
0.4083333333333333,
|
| 127 |
+
0.525
|
| 128 |
+
],
|
| 129 |
+
"regB_ref": 0.03635521260826646,
|
| 130 |
+
"regB_star": 0.046204610388273615,
|
| 131 |
+
"regA_ref": 0.04846464655537219,
|
| 132 |
+
"regA_star": 0.05641944870545168
|
| 133 |
+
},
|
| 134 |
+
"twin filt_b1": {
|
| 135 |
+
"rate": 0.3416666666666666,
|
| 136 |
+
"rate_per_seed": [
|
| 137 |
+
0.36666666666666664,
|
| 138 |
+
0.30833333333333335,
|
| 139 |
+
0.35
|
| 140 |
+
],
|
| 141 |
+
"regB_ref": 0.04659602299391885,
|
| 142 |
+
"regB_star": 0.053387234536363515,
|
| 143 |
+
"regA_ref": 0.04659602299391885,
|
| 144 |
+
"regA_star": 0.053387234536363515
|
| 145 |
+
},
|
| 146 |
+
"twin filt_b4": {
|
| 147 |
+
"rate": 0.4833333333333334,
|
| 148 |
+
"rate_per_seed": [
|
| 149 |
+
0.5583333333333333,
|
| 150 |
+
0.4166666666666667,
|
| 151 |
+
0.475
|
| 152 |
+
],
|
| 153 |
+
"regB_ref": 0.04074265857517956,
|
| 154 |
+
"regB_star": 0.048658098886397605,
|
| 155 |
+
"regA_ref": 0.046240245344131514,
|
| 156 |
+
"regA_star": 0.056035186950899485
|
| 157 |
+
},
|
| 158 |
+
"twin filt_b16": {
|
| 159 |
+
"rate": 0.5555555555555556,
|
| 160 |
+
"rate_per_seed": [
|
| 161 |
+
0.5833333333333334,
|
| 162 |
+
0.5083333333333333,
|
| 163 |
+
0.575
|
| 164 |
+
],
|
| 165 |
+
"regB_ref": 0.03570862209728387,
|
| 166 |
+
"regB_star": 0.04591678068291327,
|
| 167 |
+
"regA_ref": 0.05002032485737595,
|
| 168 |
+
"regA_star": 0.057383941917565906
|
| 169 |
+
},
|
| 170 |
+
"knn raw_b1": {
|
| 171 |
+
"rate": 0.09166666666666666,
|
| 172 |
+
"rate_per_seed": [
|
| 173 |
+
0.09166666666666666
|
| 174 |
+
],
|
| 175 |
+
"regB_ref": 0.18029208813957154,
|
| 176 |
+
"regB_star": 0.18029208813957154,
|
| 177 |
+
"regA_ref": 0.18029208813957154,
|
| 178 |
+
"regA_star": 0.18029208813957154
|
| 179 |
+
},
|
| 180 |
+
"knn raw_b4": {
|
| 181 |
+
"rate": 0.24166666666666667,
|
| 182 |
+
"rate_per_seed": [
|
| 183 |
+
0.24166666666666667
|
| 184 |
+
],
|
| 185 |
+
"regB_ref": 0.16093589699487887,
|
| 186 |
+
"regB_star": 0.17212171903802753,
|
| 187 |
+
"regA_ref": 0.1725441277384172,
|
| 188 |
+
"regA_star": 0.18523628247042523
|
| 189 |
+
},
|
| 190 |
+
"knn raw_b16": {
|
| 191 |
+
"rate": 0.5166666666666667,
|
| 192 |
+
"rate_per_seed": [
|
| 193 |
+
0.5166666666666667
|
| 194 |
+
],
|
| 195 |
+
"regB_ref": 0.1327768182319501,
|
| 196 |
+
"regB_star": 0.14350847619758267,
|
| 197 |
+
"regA_ref": 0.14856354038966635,
|
| 198 |
+
"regA_star": 0.16313087614552568
|
| 199 |
+
},
|
| 200 |
+
"knn filt_b1": {
|
| 201 |
+
"rate": 0.3,
|
| 202 |
+
"rate_per_seed": [
|
| 203 |
+
0.3
|
| 204 |
+
],
|
| 205 |
+
"regB_ref": 0.17428865396751858,
|
| 206 |
+
"regB_star": 0.18276418530499838,
|
| 207 |
+
"regA_ref": 0.17428865396751858,
|
| 208 |
+
"regA_star": 0.18276418530499838
|
| 209 |
+
},
|
| 210 |
+
"knn filt_b4": {
|
| 211 |
+
"rate": 0.6666666666666666,
|
| 212 |
+
"rate_per_seed": [
|
| 213 |
+
0.6666666666666666
|
| 214 |
+
],
|
| 215 |
+
"regB_ref": 0.12985794400765538,
|
| 216 |
+
"regB_star": 0.13483477262840815,
|
| 217 |
+
"regA_ref": 0.1620537009612174,
|
| 218 |
+
"regA_star": 0.16564434998790312
|
| 219 |
+
},
|
| 220 |
+
"knn filt_b16": {
|
| 221 |
+
"rate": 0.725,
|
| 222 |
+
"rate_per_seed": [
|
| 223 |
+
0.725
|
| 224 |
+
],
|
| 225 |
+
"regB_ref": 0.10686761298831438,
|
| 226 |
+
"regB_star": 0.11477117848557294,
|
| 227 |
+
"regA_ref": 0.16317150492755594,
|
| 228 |
+
"regA_star": 0.16567695056159626
|
| 229 |
+
},
|
| 230 |
+
"knn ref_b16_n3": {
|
| 231 |
+
"rate": 0.9,
|
| 232 |
+
"rate_per_seed": [
|
| 233 |
+
0.9
|
| 234 |
+
],
|
| 235 |
+
"regB_ref": 0.10400741103859301,
|
| 236 |
+
"regB_star": 0.1113356454533998,
|
| 237 |
+
"regA_ref": 0.13930659114267874,
|
| 238 |
+
"regA_star": 0.15300463877630077
|
| 239 |
+
},
|
| 240 |
+
"knn ref_b16_n10": {
|
| 241 |
+
"rate": 0.925,
|
| 242 |
+
"rate_per_seed": [
|
| 243 |
+
0.925
|
| 244 |
+
],
|
| 245 |
+
"regB_ref": 0.10343356710695706,
|
| 246 |
+
"regB_star": 0.10973883506945237,
|
| 247 |
+
"regA_ref": 0.13558071904802294,
|
| 248 |
+
"regA_star": 0.14891194803545904
|
| 249 |
+
}
|
| 250 |
+
}
|
results/icml_r3/X29/act/x27rescore_d3fm5n16000.txt
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
X29-A: X27's C-only arms re-scored, m = 5, tag d3fm5n16000, 120 test plants. J*_A = min(J_ref, 3A id and so3 labels) is below J_ref on 43 plants (median (J_ref - J*_A) / J*_A 0.000%).
|
| 2 |
+
diff raw_b1 certified 23.6% (published 23.6%) | cheapest certified: regret vs J_ref 5.40% (published 5.40%), vs J*_A 6.58% | first certified: vs J_ref 5.40%, vs J*_A 6.58%
|
| 3 |
+
diff raw_b4 certified 49.2% (published 49.2%) | cheapest certified: regret vs J_ref 4.60% (published 4.60%), vs J*_A 5.57% | first certified: vs J_ref 5.12%, vs J*_A 6.24%
|
| 4 |
+
diff raw_b16 certified 79.2% (published 79.2%) | cheapest certified: regret vs J_ref 3.92% (published 3.92%), vs J*_A 4.84% | first certified: vs J_ref 5.37%, vs J*_A 6.53%
|
| 5 |
+
diff filt_b1 certified 45.8% (published 45.8%) | cheapest certified: regret vs J_ref 5.19% (published 5.19%), vs J*_A 6.51% | first certified: vs J_ref 5.19%, vs J*_A 6.51%
|
| 6 |
+
diff filt_b4 certified 81.4% (published 81.4%) | cheapest certified: regret vs J_ref 4.41% (published 4.41%), vs J*_A 5.12% | first certified: vs J_ref 5.37%, vs J*_A 6.53%
|
| 7 |
+
diff filt_b16 certified 90.8% (published 90.8%) | cheapest certified: regret vs J_ref 3.38% (published 3.38%), vs J*_A 3.92% | first certified: vs J_ref 5.49%, vs J*_A 6.61%
|
| 8 |
+
diff ref_b16_n3 certified 96.4% (published 96.4%) | cheapest certified: regret vs J_ref 3.44% (published 3.44%), vs J*_A 3.96% | first certified: vs J_ref 5.62%, vs J*_A 6.71%
|
| 9 |
+
diff ref_b16_n10 certified 98.1% (published 98.1%) | cheapest certified: regret vs J_ref 3.44% (published 3.44%), vs J*_A 4.00% | first certified: vs J_ref 5.65%, vs J*_A 6.75%
|
| 10 |
+
twin raw_b1 certified 24.7% (published 24.7%) | cheapest certified: regret vs J_ref 4.59% (published 4.59%), vs J*_A 5.28% | first certified: vs J_ref 4.59%, vs J*_A 5.28%
|
| 11 |
+
twin raw_b4 certified 37.8% (published 37.8%) | cheapest certified: regret vs J_ref 3.96% (published 3.96%), vs J*_A 4.70% | first certified: vs J_ref 4.45%, vs J*_A 5.31%
|
| 12 |
+
twin raw_b16 certified 49.2% (published 49.2%) | cheapest certified: regret vs J_ref 3.64% (published 3.64%), vs J*_A 4.62% | first certified: vs J_ref 4.85%, vs J*_A 5.64%
|
| 13 |
+
twin filt_b1 certified 34.2% (published 34.2%) | cheapest certified: regret vs J_ref 4.66% (published 4.66%), vs J*_A 5.34% | first certified: vs J_ref 4.66%, vs J*_A 5.34%
|
| 14 |
+
twin filt_b4 certified 48.3% (published 48.3%) | cheapest certified: regret vs J_ref 4.07% (published 4.07%), vs J*_A 4.87% | first certified: vs J_ref 4.62%, vs J*_A 5.60%
|
| 15 |
+
twin filt_b16 certified 55.6% (published 55.6%) | cheapest certified: regret vs J_ref 3.57% (published 3.57%), vs J*_A 4.59% | first certified: vs J_ref 5.00%, vs J*_A 5.74%
|
| 16 |
+
knn raw_b1 certified 9.2% (published 9.2%) | cheapest certified: regret vs J_ref 18.03% (published 18.03%), vs J*_A 18.03% | first certified: vs J_ref 18.03%, vs J*_A 18.03%
|
| 17 |
+
knn raw_b4 certified 24.2% (published 24.2%) | cheapest certified: regret vs J_ref 16.09% (published 16.09%), vs J*_A 17.21% | first certified: vs J_ref 17.25%, vs J*_A 18.52%
|
| 18 |
+
knn raw_b16 certified 51.7% (published 51.7%) | cheapest certified: regret vs J_ref 13.28% (published 13.28%), vs J*_A 14.35% | first certified: vs J_ref 14.86%, vs J*_A 16.31%
|
| 19 |
+
knn filt_b1 certified 30.0% (published 30.0%) | cheapest certified: regret vs J_ref 17.43% (published 17.43%), vs J*_A 18.28% | first certified: vs J_ref 17.43%, vs J*_A 18.28%
|
| 20 |
+
knn filt_b4 certified 66.7% (published 66.7%) | cheapest certified: regret vs J_ref 12.99% (published 12.99%), vs J*_A 13.48% | first certified: vs J_ref 16.21%, vs J*_A 16.56%
|
| 21 |
+
knn filt_b16 certified 72.5% (published 72.5%) | cheapest certified: regret vs J_ref 10.69% (published 10.69%), vs J*_A 11.48% | first certified: vs J_ref 16.32%, vs J*_A 16.57%
|
| 22 |
+
knn ref_b16_n3 certified 90.0% (published 90.0%) | cheapest certified: regret vs J_ref 10.40% (published 10.40%), vs J*_A 11.13% | first certified: vs J_ref 13.93%, vs J*_A 15.30%
|
| 23 |
+
knn ref_b16_n10 certified 92.5% (published 92.5%) | cheapest certified: regret vs J_ref 10.34% (published 10.34%), vs J*_A 10.97% | first certified: vs J_ref 13.56%, vs J*_A 14.89%
|
results/icml_r3/X29/act/x27rows_d3fm3n8000.npy
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:624835be5f563c538921ea962a6c0dc7745508a99389b8c852050a3c50af1cdb
|
| 3 |
+
size 565807
|
results/icml_r3/X29/act/x27rows_d3fm4n8000.npy
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c2eccb0a2ed06d5b6e29839ff88993bd8f9a6e6a722eb310c1b20af4e34ffd6c
|
| 3 |
+
size 536311
|
results/icml_r3/X29/act/x27rows_d3fm5n16000.npy
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3c0f5b146a5d508573e732c9d6f1e3e2cdf38ca4902f234d05dd938e97900449
|
| 3 |
+
size 523822
|