X29: SISO CPU and compute ratios, X27 re-score in the X29 convention, write-up
Browse filesThis view is limited to 50 files because it contains too many changes. See raw diff
- docs/icml_r3_x29.md +198 -0
- results/icml_r3/X29/act/x27rescore_d3fm3n8000.json +116 -22
- results/icml_r3/X29/act/x27rescore_d3fm3n8000.txt +22 -22
- results/icml_r3/X29/act/x27rescore_d3fm4n8000.json +116 -22
- results/icml_r3/X29/act/x27rescore_d3fm4n8000.txt +22 -22
- results/icml_r3/X29/act/x27rescore_d3fm5n16000.json +116 -22
- results/icml_r3/X29/act/x27rescore_d3fm5n16000.txt +22 -22
- results/icml_r3/X29/siso/report.txt +22 -1
- results/icml_r3/X29/siso/timing/plant_000_fopt1.json +1 -0
- results/icml_r3/X29/siso/timing/plant_000_fopt16.json +1 -0
- results/icml_r3/X29/siso/timing/plant_000_fopt4.json +1 -0
- results/icml_r3/X29/siso/timing/plant_000_gen_C+phi.json +1 -0
- results/icml_r3/X29/siso/timing/plant_000_gen_C-only.json +1 -0
- results/icml_r3/X29/siso/timing/plant_000_hifoo2.json +1 -0
- results/icml_r3/X29/siso/timing/plant_000_jopt1.json +1 -0
- results/icml_r3/X29/siso/timing/plant_000_jopt16.json +1 -0
- results/icml_r3/X29/siso/timing/plant_000_jopt4.json +1 -0
- results/icml_r3/X29/siso/timing/plant_000_knn_C+phi.json +1 -0
- results/icml_r3/X29/siso/timing/plant_000_knn_C-only.json +1 -0
- results/icml_r3/X29/siso/timing/plant_000_twin_C+phi.json +1 -0
- results/icml_r3/X29/siso/timing/plant_000_twin_C-only.json +1 -0
- results/icml_r3/X29/siso/timing/plant_021_fopt1.json +1 -0
- results/icml_r3/X29/siso/timing/plant_021_fopt16.json +1 -0
- results/icml_r3/X29/siso/timing/plant_021_fopt4.json +1 -0
- results/icml_r3/X29/siso/timing/plant_021_gen_C+phi.json +1 -0
- results/icml_r3/X29/siso/timing/plant_021_gen_C-only.json +1 -0
- results/icml_r3/X29/siso/timing/plant_021_hifoo2.json +1 -0
- results/icml_r3/X29/siso/timing/plant_021_jopt1.json +1 -0
- results/icml_r3/X29/siso/timing/plant_021_jopt16.json +1 -0
- results/icml_r3/X29/siso/timing/plant_021_jopt4.json +1 -0
- results/icml_r3/X29/siso/timing/plant_021_knn_C+phi.json +1 -0
- results/icml_r3/X29/siso/timing/plant_021_knn_C-only.json +1 -0
- results/icml_r3/X29/siso/timing/plant_021_twin_C+phi.json +1 -0
- results/icml_r3/X29/siso/timing/plant_021_twin_C-only.json +1 -0
- results/icml_r3/X29/siso/timing/plant_041_fopt1.json +1 -0
- results/icml_r3/X29/siso/timing/plant_041_fopt16.json +1 -0
- results/icml_r3/X29/siso/timing/plant_041_fopt4.json +1 -0
- results/icml_r3/X29/siso/timing/plant_041_gen_C+phi.json +1 -0
- results/icml_r3/X29/siso/timing/plant_041_gen_C-only.json +1 -0
- results/icml_r3/X29/siso/timing/plant_041_hifoo2.json +1 -0
- results/icml_r3/X29/siso/timing/plant_041_jopt1.json +1 -0
- results/icml_r3/X29/siso/timing/plant_041_jopt16.json +1 -0
- results/icml_r3/X29/siso/timing/plant_041_jopt4.json +1 -0
- results/icml_r3/X29/siso/timing/plant_041_knn_C+phi.json +1 -0
- results/icml_r3/X29/siso/timing/plant_041_knn_C-only.json +1 -0
- results/icml_r3/X29/siso/timing/plant_041_twin_C+phi.json +1 -0
- results/icml_r3/X29/siso/timing/plant_041_twin_C-only.json +1 -0
- results/icml_r3/X29/siso/timing/plant_061_fopt1.json +1 -0
- results/icml_r3/X29/siso/timing/plant_061_fopt16.json +1 -0
- results/icml_r3/X29/siso/timing/plant_061_fopt4.json +1 -0
docs/icml_r3_x29.md
ADDED
|
@@ -0,0 +1,198 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
title: C+phi next to C-only with the same baselines and KPIs (X29). SISO, the two generators tie at 16 samples on the common J* and both have lower median regret than kNN, the regression twin and the flat references. MIMO spline setting (200 fresh plants per m), the repaired C+phi generator has the lowest median regret of the generators and baselines at 16 samples at both m, and C-only certifies more at 1 sample. Actuator setting, at the per-plant optimum C+phi equals C-only (median gain 0.00% at m = 3, 4, 5)
|
| 3 |
+
date: 2026-11-17
|
| 4 |
+
status: SISO (with CPU), MIMO spline setting (with CPU) and actuator step 3A complete; actuator step 3B (C+phi generators on X27's banks) not run, pending Matteo's decision
|
| 5 |
+
claim: X29 (Matteo 2026-11-16, via dc_presentation and in the move-HPC session): "I need to have the final baseline comparison also for the spline methods". C+phi (the model generates the linear part and the spline phi, X28) gets the same baselines and KPIs as C-only (phi fixed: SISO sat_1, MIMO identity) in every setting of the deck v10 slides.
|
| 6 |
+
maps_to: deck v10 slides [35] [36] [37] [38] [73] (SISO); the three new MIMO spline-setting slides and [73]; [50] [51] [52] [84] [73] (actuator); summaries [54] [72] [77]. No paper edit until Matteo agrees the claim.
|
| 7 |
+
scripts: nonlinear/experiments/x29_siso.py (SISO kNN, twins, optimizer from the best sample, timing, report), nonlinear/mimo_unstable/x29_t2.py (MIMO spline setting on T2: kNN, twins, 48-start solver, Adam references, timing, report), nonlinear/mimo_unstable/x29_act.py (actuator 3A: composed-sector certificate, simulated X27 cost, optimizer, new directions, report, X27 rows and re-score)
|
| 8 |
+
artifacts: results/icml_r3/X29/{PREREG.md, PREREG_siso.md, PREREG_t2.md}; results/icml_r3/X29/siso/{report.txt, eqcheck.txt, eval/, refine/, timing/, timing_batched.json}; results/icml_r3/X29/t2/{report.txt, numbers.json, cpu_report.txt, timing_m2.json, timing_m3.json, knn_m*_{a,b}.npz, twin_*.npz, solver_m*.npz, ard_m2.npz}; results/icml_r3/X29/act/{selfcheck_*.json, report_3a_heldout_m*.txt, heldout_m*/, x27rows_*.npy, x27rescore_*.txt, x27rescore_*.json}; models/x29_twin_clS{,_s1,_s2}/, models/x29t/; HF matCercola18/DiffusionControl commits 8f92ad1d (SISO), d7d0258 (MIMO spline setting), 09f35d5 (actuator 3A)
|
| 9 |
+
---
|
| 10 |
+
|
| 11 |
+
# C+phi next to C-only with the same baselines and KPIs (X29)
|
| 12 |
+
|
| 13 |
+
## TL;DR
|
| 14 |
+
- **Question:** The deck compares C-only (phi fixed; the model generates the linear part) with kNN, regression twins, per-plant optimizers and classical references. Matteo asked for the same comparison for C+phi (the model also generates the spline phi, X28) in every setting, on one regret reference per setting.
|
| 15 |
+
- **SISO (199 test plants):** at 16 samples the C+phi generator certifies 95.6% of plants with median regret 6.59%, and C-only 98.0% with 6.74% (3-seed means). Both have lower regret than their kNN (11.3%, 10.2%), their regression twins (31.8%, 20.1%), the tuned LQG or static gain (34.05%) and the best of 16 HIFOO starts (50.75%). At 1 sample C+phi certifies 50.6% of plants against 64.0%. The joint optimizer reaches the C+phi generator's regret with 2 starts; the F-only optimizer needs 8 to reach C-only's. On one CPU thread the C+phi generator costs 29.1 s per plant at 16 samples against 11.5 s (same sampling time, slower scoring), and the variant's own optimizer at the start count that reaches the generator's regret costs 106 times the C+phi generator's CPU and 138 times the C-only generator's (October: 140 times).
|
| 16 |
+
- **MIMO spline setting (200 fresh plants per m):** at 16 samples the repaired C+phi generator has median regret 6.56% at m = 3 and 3.59% at m = 2, against 10.22% and 6.42% for C-only, with certified shares within 0.5 points. Its kNN and twin also have lower regret than C-only's. The 48-start solver has 11.60% and 12.33%. At 1 sample C-only certifies more plants (95.7% against 79.7% at m = 3). Each generator costs about 3 s per plant on one CPU thread and about 0.05 s per plant batched on a GPU; the per-plant Adam optimizer costs about 4,200 to 5,100 s and 34 to 41 s.
|
| 17 |
+
- **Actuator setting (X27's 120 test plants per m):** at each plant's optimum C+phi costs the same as C-only (median gain 0.00% at m = 3, 4 and 5), and the best certified spline is linear on most plants. So a C+phi generator can at best tie here, and step 3B (C+phi generators on X27's banks, 2 to 3 GPU-days) waits for Matteo. X27's C-only pipeline, re-run on its stored candidates, reproduces every published number; on the common J*_A the regret of its filtered arms at 16 samples rises by 0.5 to 1.2 points in X27's convention (median over certified plants).
|
| 18 |
+
- **So what:** C+phi changes the comparison only in the MIMO spline setting, where it has lower regret than C-only for every method family at 16 samples. On SISO it ties, and in the actuator setting it has nothing to gain at the optimum. With 1 sample, C-only certifies more plants in SISO and in the MIMO spline setting.
|
| 19 |
+
|
| 20 |
+
## 1. Problem / Goal
|
| 21 |
+
- **C-only:** SISO u = -sat_1(clip(F y, +-5)) with a generated dynamic filter F (X21); MIMO a generated gain K with phi = identity, in the X28-M spline setting (u = -K y) and in X27's actuator setting (u = -Phi(K y), Phi unknown in a sector).
|
| 22 |
+
- **C+phi:** the same model also generates phi. SISO: an 11-knot spline with |phi| <= 1, phi(0) = 0 (X28 S3). MIMO: per channel an odd piecewise-linear spline with knots at |x| = 1, 3, 5 and knot ratios in [0.1, 10] (the so3 class, X28-M).
|
| 23 |
+
- **Goal:** every deck KPI (certified share and median regret vs budget b, CPU vs regret, compute ratio, generator minus kNN at b = 1) for both variants, on one reference J* per setting, with the same baselines. Nothing published is replaced; C-only numbers are re-scored on the common J*.
|
| 24 |
+
- **No new verdict rule** in SISO and the MIMO spline setting: the registered X28 verdicts stand. The actuator setting had no C+phi yet; its first step (3A) measures whether C+phi can gain at all.
|
| 25 |
+
|
| 26 |
+
## 2. Method
|
| 27 |
+
Pre-registration: `results/icml_r3/X29/PREREG.md` (2026-11-16 18:30 UTC, before any X29 code ran) and its addenda in `PREREG.md`, `PREREG_siso.md` and `PREREG_t2.md`, each filed before the number it affects.
|
| 28 |
+
|
| 29 |
+
- **Regret** at budget b: (J_b - J*) / J*, with J_b the cheapest certified of the first b candidates. A plant with no certified candidate counts 1e3. A cell where fewer than half the plants certify has no finite median ("n/a"). Median with a bootstrap 95% CI over plants (10,000 resamples). Seeded methods: 3-seed mean and range, plus seed 0.
|
| 30 |
+
- **J\* per setting:** SISO min(J_Fonly, X28 test joint labels) as X28 S4; MIMO spline setting min of the T2 id and so3 labels (gauges c = 1, 2, 5); actuator min(J_ref, J_id, J_so3) from step 3A.
|
| 31 |
+
- **SISO arms** (199 X21 test plants, clip 1.5, t-repair on every arm):
|
| 32 |
+
- generators: clL (C-only) and clS (C+phi), seeds 0 to 2, stored X28 evaluations; 64 samples for seed 0;
|
| 33 |
+
- kNN: X22's rule on the X21 F-only bank (C-only) and on the c = 1 joint rows of the 1,971 training plants (C+phi), ARD metric of X21;
|
| 34 |
+
- regression twins: X23's twins of clL (stored X21 outputs, see deviations) and new twins of clS (X23 recipe, 60,000 steps, best validation checkpoint), seeds 0 to 2;
|
| 35 |
+
- per-plant optimizers: F-only, best of the first k X21 labels (C-only's own); joint, best of the first k joint rows (C+phi's own); k = 1, 2, 4, 8, 16;
|
| 36 |
+
- optimizer from the generator's best b = 16 sample: 1 F-only start (X24 refine) for C-only, 1 joint start (S2 warm arm) for C+phi, seed 0;
|
| 37 |
+
- flat references: tuned saturated LQG or static gain (J_lin), best of 16 HIFOO starts.
|
| 38 |
+
- **MIMO spline setting** (T2: 200 fresh plants per m, m = 2, 3, as the X28 addendum of 2026-11-15):
|
| 39 |
+
- generators: C-only (a) and C+phi (b), seeds 0 to 2, stored; C+phi repaired (repair R: rejected samples' knots flattened toward r(1) until the circle LMI certifies) is primary, as drawn is secondary;
|
| 40 |
+
- kNN: label (a) or (b) of the 16 nearest S2-M bank plants, ARD metric (m = 3 X10's, m = 2 fitted on the m = 2 bank before any kNN number);
|
| 41 |
+
- regression twins: `mh_train` arm "reg" on the S3-M datasets (a) and (b), seeds 0 to 2;
|
| 42 |
+
- flat references: the 48-start solver's best gain (X27 oracle, Lyapunov test, X28 cost) and the per-plant Adam optimizer of each variant (C-only: id start; C+phi: J*_T2 itself, regret 0 by construction).
|
| 43 |
+
- **Actuator setting, step 3A** (X27's 120 test plants per m, m = 3, 4, 5):
|
| 44 |
+
- controller u = -Phi(phi(K y)), with Phi the unknown actuator (each channel in the sector [0.4, 2.2], X27) and phi in the so3 class; C-only is phi = identity;
|
| 45 |
+
- certificate: circle LMI on the composed sector [0.4 alpha_i, 2.2 beta_i], alpha_i and beta_i the min and max knot ratio of channel i. It is sound because each channel's composition Phi_i(psi_i(.)) is memoryless and lies in that sector; at identity knots it is X27's test;
|
| 46 |
+
- cost: X27's quadratic cost (mean over the 2m unit state directions of the integral of |x|^2 + 0.1 |u|^2, Phi = identity) by simulation, RK4 at dt 0.01, T = 20 s;
|
| 47 |
+
- label optimizer: X28-M's procedure from X27's stored best gain K_ref; arm id (knots frozen) and arm so3 at gauges c = 1, 2, 5; K lr and knot lr 0.02; 400 steps; the cheapest certified snapshot;
|
| 48 |
+
- reported: d = (J_id - J_so3) / J_so3; (J_ref - J_id) / J_id (extra optimization alone); the composed-sector ratio of the best so3 controller; d on 64 new random unit state directions per plant;
|
| 49 |
+
- X27's stored C-only candidates re-run through X27's pipeline per plant and re-scored on J*_A.
|
| 50 |
+
- **CPU:** one timing job per setting on a node with nothing else of this project running. "One" = one plant on one CPU thread, median over 10 timing plants (SISO: 6 such tasks at a time on 8 cores; MIMO: one at a time).
|
| 51 |
+
- MIMO (node move-hpc-08-gpu): "batched" = all 200 plants in one call, with sampling, rollouts and Adam on the node's NVIDIA L40S GPU and certificates on 6 CPU workers, wall time / 200.
|
| 52 |
+
- SISO (node move-hpc-06-gpu, also an L40S): "batched" = generator and twin sampling (199 plants x 16) and kNN retrieval in one call, wall time / 199, without scoring (PREREG section 2).
|
| 53 |
+
- Per-point CPU follows X24's formulas; the joint optimizer at k starts is charged k HIFOO starts + the F-only optimizer at k + the joint optimizer at k.
|
| 54 |
+
- **Self-checks** (two failed and were handled by addenda before the numbers they affect; see deviations):
|
| 55 |
+
- SISO: C-only kNN reproduces X21's stored kNN (K16 +0.16 points per candidate, +0.50 plants; K64 +0.07, +0.00); batched optimizers equal the single-plant functions (relative J difference 0 at 25 and 100 steps);
|
| 56 |
+
- MIMO spline setting: 5 bank plants per m retrieve themselves first; the solver's cost equals the stored label snapshot (maximum relative difference 0);
|
| 57 |
+
- actuator: the composed-sector certificate at identity knots agrees with `family_sofm.verify` on 200 gains per m (0 disagreements at m = 3, 4, 5); the simulated cost equals tr(P)/(2m) on the 120 stored gains per m (median relative difference 1.3e-5 to 2.2e-5, maximum 2.6e-4 to 1.4e-3).
|
| 58 |
+
|
| 59 |
+
## 3. Results
|
| 60 |
+
|
| 61 |
+
### 3.1 SISO, 199 test plants
|
| 62 |
+
Source: `results/icml_r3/X29/siso/report.txt`. Certified % / median regret %; generator and twin: 3-seed mean [range]; kNN: one run; b = 64: seed 0.
|
| 63 |
+
|
| 64 |
+
| method | b = 1 | b = 4 | b = 16 | b = 64 |
|
| 65 |
+
|---|---|---|---|---|
|
| 66 |
+
| C-only generator | 64.0 / 49.0 | 92.0 / 12.7 | 98.0 [97.5, 98.5] / 6.74 [6.62, 6.81] | 98.5 / 4.07 |
|
| 67 |
+
| C+phi generator | 50.6 / n/a on 2 of 3 seeds | 84.9 / 17.1 | 95.6 [95.0, 96.5] / 6.59 [6.30, 7.14] | 99.0 / 2.96 |
|
| 68 |
+
| C-only kNN | 57.3 / 81.3 | 86.9 / 19.4 | 96.5 / 10.19 | 98.5 / 6.49 |
|
| 69 |
+
| C+phi kNN | 46.2 / n/a | 77.9 / 22.7 | 93.5 / 11.32 | 98.0 / 6.26 |
|
| 70 |
+
| C-only twin | 68.5 / 48.1 | 77.2 / 30.3 | 85.6 [85.4, 85.9] / 20.1 [19.9, 20.3] | |
|
| 71 |
+
| C+phi twin | 42.2 / n/a | 61.8 / 66.1 | 71.0 [68.8, 72.9] / 31.8 [30.0, 32.9] | |
|
| 72 |
+
| LQG or static gain | | | 100 / 34.05 | |
|
| 73 |
+
| HIFOO, best of 16 | | | 100 / 50.75 | |
|
| 74 |
+
|
| 75 |
+
Per-plant optimizers, median regret % at k = 1, 2, 4, 8, 16 starts (certified 100%):
|
| 76 |
+
- F-only (C-only's own): 35.91, 13.55, 8.11, 4.19, 3.12.
|
| 77 |
+
- Joint (C+phi's own): 9.13, 4.39, 0.92, 0.00, 0.00. The joint labels define J*, so 0 at 8 and 16 starts is by construction.
|
| 78 |
+
- Generator at 16 plus 1 optimizer start (seed 0): C-only 98.5% / 2.60%, C+phi 95.0% / 0.01%. Only plants with a certified b = 16 sample get a start (196 C-only, 189 C+phi); the rest count as uncertified.
|
| 79 |
+
|
| 80 |
+
Compute ratio inputs ([38], seed 0): the C-only generator's 6.79% is first reached by the F-only optimizer at k = 8 (4.19%; k = 4 gives 8.11%). The C+phi generator's 7.14% is first reached by the joint optimizer at k = 2 (4.39%; k = 1 gives 9.13%). The paired verdict at b = 16 stays X28 S4's: no measurable difference on 3 of 3 seed pairs (gain -0.32 to +0.19 points).
|
| 81 |
+
|
| 82 |
+
[73], certified at b = 1, generator minus its own variant's kNN (points, plant-paired, 95% CI), seeds 0 / 1 / 2:
|
| 83 |
+
- C-only: +8.0 [-0.5, +16.6] / +8.0 [-0.5, +16.6] / +4.0 [-4.0, +12.1].
|
| 84 |
+
- C+phi: +3.0 [-6.0, +12.1] / +6.5 [-3.0, +16.1] / +3.5 [-5.5, +12.6].
|
| 85 |
+
- All six intervals include 0.
|
| 86 |
+
|
| 87 |
+
### 3.2 MIMO spline setting, 200 fresh plants per m
|
| 88 |
+
Source: `results/icml_r3/X29/t2/report.txt`. Certified % / median regret %, 3-seed means (kNN one run).
|
| 89 |
+
|
| 90 |
+
| method | m = 3, b = 1 | m = 3, b = 16 | m = 2, b = 1 | m = 2, b = 16 |
|
| 91 |
+
|---|---|---|---|---|
|
| 92 |
+
| C+phi generator, repaired | 79.7 / 17.73 | 99.8 / 6.56 | 90.5 / 10.17 | 99.7 / 3.59 |
|
| 93 |
+
| C+phi generator, as drawn | 61.5 / 22.85 | 95.3 / 6.60 | 78.0 / 10.68 | 97.2 / 3.65 |
|
| 94 |
+
| C-only generator | 95.7 / 16.32 | 99.5 / 10.22 | 98.5 / 8.96 | 99.8 / 6.42 |
|
| 95 |
+
| C+phi kNN, repaired | 90.5 / 61.40 | 100 / 15.88 | 88.0 / 34.85 | 100 / 8.89 |
|
| 96 |
+
| C+phi kNN, as drawn | 48.0 / n/a | 99.5 / 17.55 | 60.0 / 76.27 | 100 / 9.83 |
|
| 97 |
+
| C-only kNN | 80.0 / 70.87 | 100 / 23.77 | 89.5 / 42.67 | 100 / 13.56 |
|
| 98 |
+
| C+phi twin, repaired | 76.3 / 22.16 | 95.0 / 10.84 | 88.3 / 12.14 | 98.0 / 6.23 |
|
| 99 |
+
| C+phi twin, as drawn | 60.3 / 28.50 | 83.2 / 11.26 | 77.7 / 12.74 | 90.8 / 6.31 |
|
| 100 |
+
| C-only twin | 94.2 / 18.21 | 98.7 / 13.81 | 98.7 / 9.22 | 99.7 / 7.71 |
|
| 101 |
+
| 48-start solver | | 100 / 11.60 | | 100 / 12.33 |
|
| 102 |
+
| Adam, C-only (id start) | | 100 / 4.33 | | 100 / 5.07 |
|
| 103 |
+
| Adam, C+phi (id + so3 c = 1, 2, 5) | | 100 / 0 (defines J*) | | 100 / 0 (defines J*) |
|
| 104 |
+
|
| 105 |
+
- At b = 4 (same file): repaired C+phi generator 93.8 / 9.70 (m = 3) and 97.8 / 5.47 (m = 2); C-only generator 99.2 / 12.73 and 99.3 / 7.30.
|
| 106 |
+
- [73], certified at b = 1, generator minus its own kNN (points, 95% CI), seeds 0 / 1 / 2:
|
| 107 |
+
- m = 3: C+phi repaired -10.0 [-17.0, -3.0] / -12.5 [-19.5, -5.5] / -10.0 [-16.5, -3.5]; as drawn +14.0 / +13.0 / +13.5 (CIs above 0); C-only +15.5 / +17.5 / +14.0 (CIs above 0).
|
| 108 |
+
- m = 2: C+phi repaired +3.0 [-3.0, +9.0] / +2.0 [-4.0, +8.0] / +2.5 [-3.0, +8.0]; as drawn +18.0 / +15.5 / +20.5 (CIs above 0); C-only +9.0 on all seeds (CIs above 0).
|
| 109 |
+
- With repair, the C+phi kNN certifies more plants at b = 1 than the C+phi generator at m = 3 (90.5% against 79.7%).
|
| 110 |
+
|
| 111 |
+
### 3.3 Actuator setting, step 3A (per-plant optimum), 120 test plants per m
|
| 112 |
+
Source: `results/icml_r3/X29/act/report_3a_heldout_m{3,4,5}.txt`. All id and so3 starts certified on 118 to 120 of 120 plants.
|
| 113 |
+
|
| 114 |
+
| | m = 3 | m = 4 | m = 5 |
|
| 115 |
+
|---|---|---|---|
|
| 116 |
+
| d = (J_id - J_so3) / J_so3, median [CI] | 0.00% [0.00, 0.00] | 0.00% [0.00, 0.00] | 0.00% [0.00, 0.00] |
|
| 117 |
+
| plants with d >= 2% | 16 of 120 | 9 of 120 | 13 of 120 |
|
| 118 |
+
| mean d | 0.68% | 0.34% | 0.44% |
|
| 119 |
+
| extra optimization alone, (J_ref - J_id) / J_id, median | 0.00% | 0.00% | 0.00% |
|
| 120 |
+
| composed-sector ratio of the best so3 controller, median (90th pct); 5.5 = linear | 5.50 (15.97) | 5.50 (11.37) | 5.50 (12.18) |
|
| 121 |
+
| d on 64 new directions per plant, median of per-plant medians | +0.00% | +0.00% | +0.00% |
|
| 122 |
+
|
| 123 |
+
- The best so3 controller comes from the id start on 69, 78 and 76 of 120 plants: the spline starts found nothing cheaper there.
|
| 124 |
+
- The written expectation (PREREG section 4) was a small d: the composed sector is 5.5 beta/alpha wide, so a nonlinear spline needs a wider robust sector than a linear gain, and the quadratic cost does not reward an amplitude-dependent gain.
|
| 125 |
+
|
| 126 |
+
X27's C-only pipeline re-scored on J*_A (`results/icml_r3/X29/act/x27rescore_*.txt`). Re-running it on the stored candidates reproduces every published certified share and regret. Certified shares are unchanged. X27 publishes the median over certified plants (seeds pooled); in that convention, median regret %, published (vs J_ref) and on J*_A:
|
| 127 |
+
|
| 128 |
+
| X27 arm (keep the cheapest certified) | m = 3 | m = 4 | m = 5 (16k bank) |
|
| 129 |
+
|---|---|---|---|
|
| 130 |
+
| diffusion, filter, b = 16 | 1.50 -> 2.18 | 3.04 -> 3.73 | 3.38 -> 3.92 |
|
| 131 |
+
| diffusion, filter + 10 solver iterations | 1.50 -> 2.21 | 3.24 -> 3.85 | 3.44 -> 4.00 |
|
| 132 |
+
| twin, filter, b = 16 | 2.50 -> 3.39 | 3.46 -> 4.00 | 3.57 -> 4.59 |
|
| 133 |
+
| kNN, filter + 10 solver iterations | 7.34 -> 8.58 | 9.03 -> 9.68 | 10.34 -> 10.97 |
|
| 134 |
+
|
| 135 |
+
Stopping at the first certified candidate instead, the diffusion arm with iterations has 4.01 -> 5.15, 5.71 -> 6.40 and 5.65 -> 6.75.
|
| 136 |
+
|
| 137 |
+
In the X29 convention (all plants, uncertified = 1e3, median per seed, 3-seed mean), on J*_A, cheapest certified at b = 16, certified % / median regret %:
|
| 138 |
+
|
| 139 |
+
| X27 arm | m = 3 | m = 4 | m = 5 |
|
| 140 |
+
|---|---|---|---|
|
| 141 |
+
| diffusion, filter + 10 solver iterations | 99.7 / 2.22 | 97.8 / 3.96 | 98.1 / 4.04 |
|
| 142 |
+
| diffusion, filter | 98.6 / 2.24 | 91.9 / 4.05 | 90.8 / 4.30 |
|
| 143 |
+
| twin, filter | 73.1 / 5.25 | 52.2 / 12.06 | 55.6 / 9.61 |
|
| 144 |
+
| kNN, filter + 10 solver iterations (one run) | 93.3 / 8.84 | 93.3 / 10.01 | 92.5 / 11.51 |
|
| 145 |
+
|
| 146 |
+
The twin's low certified share counts against it here, unlike in the published convention.
|
| 147 |
+
|
| 148 |
+
### 3.4 CPU
|
| 149 |
+
MIMO spline setting (`results/icml_r3/X29/t2/cpu_report.txt`; seconds per plant, one | batched; median regret on J*_T2):
|
| 150 |
+
|
| 151 |
+
| method | m = 3 | m = 2 |
|
| 152 |
+
|---|---|---|
|
| 153 |
+
| C+phi generator, b = 16 | 2.96 \| 0.055; 6.56% | 2.85 \| 0.054; 3.59% |
|
| 154 |
+
| C-only generator, b = 16 | 2.88 \| 0.050; 10.22% | 2.79 \| 0.049; 6.42% |
|
| 155 |
+
| C+phi kNN, b = 16 | 1.34 \| 0.054; 15.88% | 1.27 \| 0.053; 8.89% |
|
| 156 |
+
| C-only kNN, b = 16 | 1.25 \| 0.042; 23.77% | 1.18 \| 0.043; 13.56% |
|
| 157 |
+
| C+phi twin, b = 16 | 1.39 \| 0.050; 10.84% | 1.28 \| 0.051; 6.23% |
|
| 158 |
+
| C-only twin, b = 16 | 1.35 \| 0.045; 13.81% | 1.24 \| 0.044; 7.71% |
|
| 159 |
+
| 48-start solver, all starts | 6.01 \| not batched; 11.60% | 7.38 \| not batched; 12.33% |
|
| 160 |
+
| Adam, C+phi (4 starts, 400 steps + certify) | 5,131 \| 41.4; 0% | 5,102 \| 41.4; 0% |
|
| 161 |
+
| Adam, C-only (id start) | 4,177 \| 34.6; 4.33% | 4,170 \| 34.1; 5.07% |
|
| 162 |
+
|
| 163 |
+
At b = 1 a generator costs 1/16 of its b = 16 cost (X24's convention): 0.18 to 0.19 s one at a time.
|
| 164 |
+
|
| 165 |
+
SISO (`results/icml_r3/X29/siso/report.txt`, CPU section; seconds per plant on one CPU thread, median over the 10 timing plants; median regret on J*, seed 0):
|
| 166 |
+
|
| 167 |
+
| point | CPU | median regret |
|
| 168 |
+
|---|---|---|
|
| 169 |
+
| C-only generator, b = 1 / b = 16 | 0.7 s / 11.5 s | 54.90% / 6.79% |
|
| 170 |
+
| C+phi generator, b = 1 / b = 16 | 1.8 s / 29.1 s | n/a / 7.14% |
|
| 171 |
+
| F-only optimizer, 1 / 2 / 4 / 8 / 16 starts | 1,555 / 1,544 / 1,522 / 1,582 / 1,702 s | 35.91 / 13.55 / 8.11 / 4.19 / 3.12% |
|
| 172 |
+
| joint optimizer, 1 / 2 / 4 / 8 / 16 starts | 3,092 / 3,084 / 3,068 / 3,196 / 3,451 s | 9.13 / 4.39 / 0.92 / 0.00 / 0.00% |
|
| 173 |
+
| C-only generator 16 + 1 F-only start | 1,565 s | 2.60% |
|
| 174 |
+
| C+phi generator 16 + 1 joint start | 1,566 s | 0.01% |
|
| 175 |
+
| HIFOO, best of 16 | 15.3 s | 50.75% |
|
| 176 |
+
| C-only kNN / C+phi kNN, b = 16 | 9.5 s / 7.6 s | 10.19% / 11.32% |
|
| 177 |
+
| C-only twin / C+phi twin, b = 16 | 1.9 s / 17.2 s | 20.27% / 32.47% |
|
| 178 |
+
|
| 179 |
+
- **Compute ratio ([38]):** C-only, the F-only optimizer at 8 starts (1,582 s) against the generator's 11.5 s: 138 times. C+phi, the joint optimizer at 2 starts (3,084 s) against the generator's 29.1 s: 106 times.
|
| 180 |
+
- **Why the C+phi ratio is only somewhat smaller:** the joint optimizer needs 2 starts against 8, but the optimizer's cost is nearly flat in the number of starts (the k starts run in one vectorized call, X24's convention; the F-only optimizer at 1 start ranges from 1,503 to 1,610 s over the timing plants, wider than the differences between start counts). The joint optimizer is also charged the F-only optimizer it starts from, which doubles its cost. And the C+phi generator costs 2.5 times C-only's: sampling takes the same time (median 8.7 s for 16 samples), but scoring C+phi candidates takes a median 20.4 s against 2.8 s, from 10 to 110 s in total over the timing plants.
|
| 181 |
+
- **Batched on the GPU** (sampling only, 199 plants in one call, `timing_batched.json`): generator 0.019 s per plant for both variants, twin 0.003 s, kNN retrieval 0.013 s (C-only) and 0.005 s (C+phi).
|
| 182 |
+
- The optimizer tasks ran 6 at a time on 8 cores (load about 6); the MIMO timings ran one plant at a time (load 0.7 to 1.0). Ratios are within one setting, so this does not change them, but absolute seconds across the two settings are not measured under the same load.
|
| 183 |
+
|
| 184 |
+
## 4. Conclusion
|
| 185 |
+
- **SISO:** C+phi and C-only generators tie at 16 samples on the common J*, as X28 S4 found, and both have lower median regret than kNN, the regression twins and the flat references. C+phi's own optimizer is stronger (2 joint starts reach the C+phi generator's regret; the F-only optimizer needs 8 to reach C-only's), but each optimizer costs about the same at any start count, so the compute ratio moves only from 138 times (C-only) to 106 times (C+phi). At 1 and 4 samples C+phi certifies fewer plants.
|
| 186 |
+
- **MIMO spline setting:** C+phi has lower median regret than C-only for each method family at 16 samples (difference of median regrets at m = 3 and m = 2: generator 3.7 and 2.8 points, kNN 7.9 and 4.7, twin 3.0 and 1.5), at equal certified share with repair. C-only certifies more at 1 sample. The C+phi generator at 16 samples has lower median regret than C-only's per-plant Adam optimizer at m = 2 (3.59% against 5.07%), not at m = 3 (6.56% against 4.33%), at about 1/1,500 of its CPU per plant on one thread.
|
| 187 |
+
- **Actuator setting:** generating phi gains nothing at the per-plant optimum, so the actuator slides can state "C+phi = C-only here" with the 3A numbers. Step 3B is not run unless Matteo asks for it.
|
| 188 |
+
|
| 189 |
+
**Deviations and caveats**
|
| 190 |
+
- SISO C-only twin rows are X21's stored outputs re-scored on J*. The re-sampled seed 0 missed its 1-point self-check on the certified share per candidate (+1.57) because the MC-dropout draws differ across machines; this was registered before any twin regret (`PREREG_siso.md`, 19:05 UTC). The re-sampled rows agree within 1 point at b = 16 (85.76% / 20.73%).
|
| 191 |
+
- The C+phi SISO twins reached their best validation loss early (step 4,000 of 60,000), so part of their gap to the generator may come from the training run.
|
| 192 |
+
- The actuator cost first used the trapezoid rule, which failed the tr(P) self-check at every horizon (median 0.25%); it was replaced by RK4 on an extra cost state before any 3A number (`PREREG.md` addendum 18:20 UTC).
|
| 193 |
+
- The joint optimizer and the C+phi Adam optimizer define J* in their settings, so their regret falls to 0 by construction.
|
| 194 |
+
- Regrets here are on each setting's common J* and are higher than earlier C-only numbers on J_Fonly (SISO) or J_ref (actuator); the two scales are not mixed.
|
| 195 |
+
- The MIMO spline J*_T2 uses id and so3 labels only, so T2 regrets are not on the scale of the held-out X28 regrets.
|
| 196 |
+
- The SISO and MIMO timings ran on two different nodes (move-hpc-06-gpu and move-hpc-08-gpu, 8 cores and an NVIDIA L40S each), each with nothing else of this project running. Ratios are computed within one setting's timing job.
|
| 197 |
+
|
| 198 |
+
**Sources:** the PREREG and addenda in `results/icml_r3/X29/`; the report files named in each section; HF commits 8f92ad1d (SISO), d7d0258 (MIMO spline setting) and 09f35d5 (actuator 3A).
|
results/icml_r3/X29/act/x27rescore_d3fm3n8000.json
CHANGED
|
@@ -9,7 +9,12 @@
|
|
| 9 |
"regB_ref": 0.035166553853986704,
|
| 10 |
"regB_star": 0.04939973422469457,
|
| 11 |
"regA_ref": 0.035166553853986704,
|
| 12 |
-
"regA_star": 0.04939973422469457
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
},
|
| 14 |
"diff raw_b4": {
|
| 15 |
"rate": 0.7611111111111111,
|
|
@@ -21,7 +26,12 @@
|
|
| 21 |
"regB_ref": 0.026566873792452816,
|
| 22 |
"regB_star": 0.036538855499143374,
|
| 23 |
"regA_ref": 0.03912745224114178,
|
| 24 |
-
"regA_star": 0.05271375933165312
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
},
|
| 26 |
"diff raw_b16": {
|
| 27 |
"rate": 0.9444444444444443,
|
|
@@ -33,7 +43,12 @@
|
|
| 33 |
"regB_ref": 0.017749766148115964,
|
| 34 |
"regB_star": 0.02536120788249141,
|
| 35 |
"regA_ref": 0.03975266005687195,
|
| 36 |
-
"regA_star": 0.051211998589972385
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
},
|
| 38 |
"diff filt_b1": {
|
| 39 |
"rate": 0.6861111111111112,
|
|
@@ -45,7 +60,12 @@
|
|
| 45 |
"regB_ref": 0.03864968435767513,
|
| 46 |
"regB_star": 0.051490162113829285,
|
| 47 |
"regA_ref": 0.03864968435767513,
|
| 48 |
-
"regA_star": 0.051490162113829285
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 49 |
},
|
| 50 |
"diff filt_b4": {
|
| 51 |
"rate": 0.9277777777777777,
|
|
@@ -57,7 +77,12 @@
|
|
| 57 |
"regB_ref": 0.024622161334723724,
|
| 58 |
"regB_star": 0.03283620184325242,
|
| 59 |
"regA_ref": 0.03892824451900456,
|
| 60 |
-
"regA_star": 0.05086767155048544
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 61 |
},
|
| 62 |
"diff filt_b16": {
|
| 63 |
"rate": 0.9861111111111112,
|
|
@@ -69,7 +94,12 @@
|
|
| 69 |
"regB_ref": 0.014977849993612624,
|
| 70 |
"regB_star": 0.0218213442998485,
|
| 71 |
"regA_ref": 0.03976742768706387,
|
| 72 |
-
"regA_star": 0.050933835066115485
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 73 |
},
|
| 74 |
"diff ref_b16_n3": {
|
| 75 |
"rate": 0.9944444444444445,
|
|
@@ -81,7 +111,12 @@
|
|
| 81 |
"regB_ref": 0.015012072869242443,
|
| 82 |
"regB_star": 0.021944205370784453,
|
| 83 |
"regA_ref": 0.03991946776173949,
|
| 84 |
-
"regA_star": 0.051211998589972385
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 85 |
},
|
| 86 |
"diff ref_b16_n10": {
|
| 87 |
"rate": 0.9972222222222222,
|
|
@@ -93,7 +128,12 @@
|
|
| 93 |
"regB_ref": 0.015046295744872262,
|
| 94 |
"regB_star": 0.022067066441720407,
|
| 95 |
"regA_ref": 0.04007150783641511,
|
| 96 |
-
"regA_star": 0.051490162113829285
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 97 |
},
|
| 98 |
"twin raw_b1": {
|
| 99 |
"rate": 0.4916666666666667,
|
|
@@ -105,7 +145,12 @@
|
|
| 105 |
"regB_ref": 0.034868968961000446,
|
| 106 |
"regB_star": 0.045860235685736495,
|
| 107 |
"regA_ref": 0.034868968961000446,
|
| 108 |
-
"regA_star": 0.045860235685736495
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 109 |
},
|
| 110 |
"twin raw_b4": {
|
| 111 |
"rate": 0.6222222222222222,
|
|
@@ -117,7 +162,12 @@
|
|
| 117 |
"regB_ref": 0.031090565068918607,
|
| 118 |
"regB_star": 0.04009868135354511,
|
| 119 |
"regA_ref": 0.03602559875874334,
|
| 120 |
-
"regA_star": 0.04612835952042982
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 121 |
},
|
| 122 |
"twin raw_b16": {
|
| 123 |
"rate": 0.6888888888888888,
|
|
@@ -129,7 +179,12 @@
|
|
| 129 |
"regB_ref": 0.024827462390698996,
|
| 130 |
"regB_star": 0.034008343863723534,
|
| 131 |
"regA_ref": 0.03665811970701527,
|
| 132 |
-
"regA_star": 0.04772048027118436
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 133 |
},
|
| 134 |
"twin filt_b1": {
|
| 135 |
"rate": 0.5722222222222222,
|
|
@@ -141,7 +196,12 @@
|
|
| 141 |
"regB_ref": 0.035172887226527116,
|
| 142 |
"regB_star": 0.045343068252056584,
|
| 143 |
"regA_ref": 0.035172887226527116,
|
| 144 |
-
"regA_star": 0.045343068252056584
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 145 |
},
|
| 146 |
"twin filt_b4": {
|
| 147 |
"rate": 0.6805555555555557,
|
|
@@ -153,7 +213,12 @@
|
|
| 153 |
"regB_ref": 0.03138655281275926,
|
| 154 |
"regB_star": 0.040342528272814926,
|
| 155 |
"regA_ref": 0.036638098220030946,
|
| 156 |
-
"regA_star": 0.04700456890654414
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 157 |
},
|
| 158 |
"twin filt_b16": {
|
| 159 |
"rate": 0.7305555555555555,
|
|
@@ -165,7 +230,12 @@
|
|
| 165 |
"regB_ref": 0.02495787433019081,
|
| 166 |
"regB_star": 0.03388894221642125,
|
| 167 |
"regA_ref": 0.036800685855217674,
|
| 168 |
-
"regA_star": 0.04793953255697314
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 169 |
},
|
| 170 |
"knn raw_b1": {
|
| 171 |
"rate": 0.25833333333333336,
|
|
@@ -175,7 +245,10 @@
|
|
| 175 |
"regB_ref": 0.11371326220225075,
|
| 176 |
"regB_star": 0.11629817415072874,
|
| 177 |
"regA_ref": 0.11371326220225075,
|
| 178 |
-
"regA_star": 0.11629817415072874
|
|
|
|
|
|
|
|
|
|
| 179 |
},
|
| 180 |
"knn raw_b4": {
|
| 181 |
"rate": 0.44166666666666665,
|
|
@@ -185,7 +258,10 @@
|
|
| 185 |
"regB_ref": 0.09416819337804627,
|
| 186 |
"regB_star": 0.11250915295445885,
|
| 187 |
"regA_ref": 0.11635060647060991,
|
| 188 |
-
"regA_star": 0.13716816454409586
|
|
|
|
|
|
|
|
|
|
| 189 |
},
|
| 190 |
"knn raw_b16": {
|
| 191 |
"rate": 0.7083333333333334,
|
|
@@ -195,7 +271,10 @@
|
|
| 195 |
"regB_ref": 0.08590952578105893,
|
| 196 |
"regB_star": 0.09869180446652481,
|
| 197 |
"regA_ref": 0.13353007468957645,
|
| 198 |
-
"regA_star": 0.13956844190360695
|
|
|
|
|
|
|
|
|
|
| 199 |
},
|
| 200 |
"knn filt_b1": {
|
| 201 |
"rate": 0.5083333333333333,
|
|
@@ -205,7 +284,10 @@
|
|
| 205 |
"regB_ref": 0.134908830251387,
|
| 206 |
"regB_star": 0.14090533433695862,
|
| 207 |
"regA_ref": 0.134908830251387,
|
| 208 |
-
"regA_star": 0.14090533433695862
|
|
|
|
|
|
|
|
|
|
| 209 |
},
|
| 210 |
"knn filt_b4": {
|
| 211 |
"rate": 0.7833333333333333,
|
|
@@ -215,7 +297,10 @@
|
|
| 215 |
"regB_ref": 0.10139965849798827,
|
| 216 |
"regB_star": 0.10879331316422458,
|
| 217 |
"regA_ref": 0.13114353705654402,
|
| 218 |
-
"regA_star": 0.1383683032238514
|
|
|
|
|
|
|
|
|
|
| 219 |
},
|
| 220 |
"knn filt_b16": {
|
| 221 |
"rate": 0.8666666666666667,
|
|
@@ -225,7 +310,10 @@
|
|
| 225 |
"regB_ref": 0.07096639087530376,
|
| 226 |
"regB_star": 0.08244864896963622,
|
| 227 |
"regA_ref": 0.13353036702921328,
|
| 228 |
-
"regA_star": 0.1383683032238514
|
|
|
|
|
|
|
|
|
|
| 229 |
},
|
| 230 |
"knn ref_b16_n3": {
|
| 231 |
"rate": 0.9333333333333333,
|
|
@@ -235,7 +323,10 @@
|
|
| 235 |
"regB_ref": 0.07228077375089703,
|
| 236 |
"regB_star": 0.0830961623057771,
|
| 237 |
"regA_ref": 0.12499251210513573,
|
| 238 |
-
"regA_star": 0.13353036702921328
|
|
|
|
|
|
|
|
|
|
| 239 |
},
|
| 240 |
"knn ref_b16_n10": {
|
| 241 |
"rate": 0.9333333333333333,
|
|
@@ -245,6 +336,9 @@
|
|
| 245 |
"regB_ref": 0.07337219985527899,
|
| 246 |
"regB_star": 0.08575503047710231,
|
| 247 |
"regA_ref": 0.12802583666738943,
|
| 248 |
-
"regA_star": 0.13421974481011856
|
|
|
|
|
|
|
|
|
|
| 249 |
}
|
| 250 |
}
|
|
|
|
| 9 |
"regB_ref": 0.035166553853986704,
|
| 10 |
"regB_star": 0.04939973422469457,
|
| 11 |
"regA_ref": 0.035166553853986704,
|
| 12 |
+
"regA_star": 0.04939973422469457,
|
| 13 |
+
"reg29_star_per_seed": [
|
| 14 |
+
0.1173585852888046,
|
| 15 |
+
0.6048869266055182,
|
| 16 |
+
1000.0
|
| 17 |
+
]
|
| 18 |
},
|
| 19 |
"diff raw_b4": {
|
| 20 |
"rate": 0.7611111111111111,
|
|
|
|
| 26 |
"regB_ref": 0.026566873792452816,
|
| 27 |
"regB_star": 0.036538855499143374,
|
| 28 |
"regA_ref": 0.03912745224114178,
|
| 29 |
+
"regA_star": 0.05271375933165312,
|
| 30 |
+
"reg29_star_per_seed": [
|
| 31 |
+
0.054677934415332974,
|
| 32 |
+
0.05112708412480571,
|
| 33 |
+
0.0515105084462818
|
| 34 |
+
]
|
| 35 |
},
|
| 36 |
"diff raw_b16": {
|
| 37 |
"rate": 0.9444444444444443,
|
|
|
|
| 43 |
"regB_ref": 0.017749766148115964,
|
| 44 |
"regB_star": 0.02536120788249141,
|
| 45 |
"regA_ref": 0.03975266005687195,
|
| 46 |
+
"regA_star": 0.051211998589972385,
|
| 47 |
+
"reg29_star_per_seed": [
|
| 48 |
+
0.025756009753485576,
|
| 49 |
+
0.029513557045964256,
|
| 50 |
+
0.02766465097867299
|
| 51 |
+
]
|
| 52 |
},
|
| 53 |
"diff filt_b1": {
|
| 54 |
"rate": 0.6861111111111112,
|
|
|
|
| 60 |
"regB_ref": 0.03864968435767513,
|
| 61 |
"regB_star": 0.051490162113829285,
|
| 62 |
"regA_ref": 0.03864968435767513,
|
| 63 |
+
"regA_star": 0.051490162113829285,
|
| 64 |
+
"reg29_star_per_seed": [
|
| 65 |
+
0.0763562167545252,
|
| 66 |
+
0.07432971205930794,
|
| 67 |
+
0.10765837454236571
|
| 68 |
+
]
|
| 69 |
},
|
| 70 |
"diff filt_b4": {
|
| 71 |
"rate": 0.9277777777777777,
|
|
|
|
| 77 |
"regB_ref": 0.024622161334723724,
|
| 78 |
"regB_star": 0.03283620184325242,
|
| 79 |
"regA_ref": 0.03892824451900456,
|
| 80 |
+
"regA_star": 0.05086767155048544,
|
| 81 |
+
"reg29_star_per_seed": [
|
| 82 |
+
0.03562885445803954,
|
| 83 |
+
0.03860264335993191,
|
| 84 |
+
0.032221163711480205
|
| 85 |
+
]
|
| 86 |
},
|
| 87 |
"diff filt_b16": {
|
| 88 |
"rate": 0.9861111111111112,
|
|
|
|
| 94 |
"regB_ref": 0.014977849993612624,
|
| 95 |
"regB_star": 0.0218213442998485,
|
| 96 |
"regA_ref": 0.03976742768706387,
|
| 97 |
+
"regA_star": 0.050933835066115485,
|
| 98 |
+
"reg29_star_per_seed": [
|
| 99 |
+
0.0235246180444032,
|
| 100 |
+
0.023310251680270566,
|
| 101 |
+
0.020264949687200495
|
| 102 |
+
]
|
| 103 |
},
|
| 104 |
"diff ref_b16_n3": {
|
| 105 |
"rate": 0.9944444444444445,
|
|
|
|
| 111 |
"regB_ref": 0.015012072869242443,
|
| 112 |
"regB_star": 0.021944205370784453,
|
| 113 |
"regA_ref": 0.03991946776173949,
|
| 114 |
+
"regA_star": 0.051211998589972385,
|
| 115 |
+
"reg29_star_per_seed": [
|
| 116 |
+
0.0235246180444032,
|
| 117 |
+
0.023310251680270566,
|
| 118 |
+
0.019837172273985404
|
| 119 |
+
]
|
| 120 |
},
|
| 121 |
"diff ref_b16_n10": {
|
| 122 |
"rate": 0.9972222222222222,
|
|
|
|
| 128 |
"regB_ref": 0.015046295744872262,
|
| 129 |
"regB_star": 0.022067066441720407,
|
| 130 |
"regA_ref": 0.04007150783641511,
|
| 131 |
+
"regA_star": 0.051490162113829285,
|
| 132 |
+
"reg29_star_per_seed": [
|
| 133 |
+
0.0235246180444032,
|
| 134 |
+
0.023310251680270566,
|
| 135 |
+
0.019837172273985404
|
| 136 |
+
]
|
| 137 |
},
|
| 138 |
"twin raw_b1": {
|
| 139 |
"rate": 0.4916666666666667,
|
|
|
|
| 145 |
"regB_ref": 0.034868968961000446,
|
| 146 |
"regB_star": 0.045860235685736495,
|
| 147 |
"regA_ref": 0.034868968961000446,
|
| 148 |
+
"regA_star": 0.045860235685736495,
|
| 149 |
+
"reg29_star_per_seed": [
|
| 150 |
+
1000.0,
|
| 151 |
+
500.0670756532481,
|
| 152 |
+
500.07857939015105
|
| 153 |
+
]
|
| 154 |
},
|
| 155 |
"twin raw_b4": {
|
| 156 |
"rate": 0.6222222222222222,
|
|
|
|
| 162 |
"regB_ref": 0.031090565068918607,
|
| 163 |
"regB_star": 0.04009868135354511,
|
| 164 |
"regA_ref": 0.03602559875874334,
|
| 165 |
+
"regA_star": 0.04612835952042982,
|
| 166 |
+
"reg29_star_per_seed": [
|
| 167 |
+
0.07311798148058501,
|
| 168 |
+
0.07470892227613435,
|
| 169 |
+
0.08088746958920823
|
| 170 |
+
]
|
| 171 |
},
|
| 172 |
"twin raw_b16": {
|
| 173 |
"rate": 0.6888888888888888,
|
|
|
|
| 179 |
"regB_ref": 0.024827462390698996,
|
| 180 |
"regB_star": 0.034008343863723534,
|
| 181 |
"regA_ref": 0.03665811970701527,
|
| 182 |
+
"regA_star": 0.04772048027118436,
|
| 183 |
+
"reg29_star_per_seed": [
|
| 184 |
+
0.05807689752517098,
|
| 185 |
+
0.05422806624253773,
|
| 186 |
+
0.05913398532637004
|
| 187 |
+
]
|
| 188 |
},
|
| 189 |
"twin filt_b1": {
|
| 190 |
"rate": 0.5722222222222222,
|
|
|
|
| 196 |
"regB_ref": 0.035172887226527116,
|
| 197 |
"regB_star": 0.045343068252056584,
|
| 198 |
"regA_ref": 0.035172887226527116,
|
| 199 |
+
"regA_star": 0.045343068252056584,
|
| 200 |
+
"reg29_star_per_seed": [
|
| 201 |
+
0.09196044154961569,
|
| 202 |
+
0.10137589764619415,
|
| 203 |
+
0.08672273104752559
|
| 204 |
+
]
|
| 205 |
},
|
| 206 |
"twin filt_b4": {
|
| 207 |
"rate": 0.6805555555555557,
|
|
|
|
| 213 |
"regB_ref": 0.03138655281275926,
|
| 214 |
"regB_star": 0.040342528272814926,
|
| 215 |
"regA_ref": 0.036638098220030946,
|
| 216 |
+
"regA_star": 0.04700456890654414,
|
| 217 |
+
"reg29_star_per_seed": [
|
| 218 |
+
0.06735180745496439,
|
| 219 |
+
0.06821410044080367,
|
| 220 |
+
0.06362610349884046
|
| 221 |
+
]
|
| 222 |
},
|
| 223 |
"twin filt_b16": {
|
| 224 |
"rate": 0.7305555555555555,
|
|
|
|
| 230 |
"regB_ref": 0.02495787433019081,
|
| 231 |
"regB_star": 0.03388894221642125,
|
| 232 |
"regA_ref": 0.036800685855217674,
|
| 233 |
+
"regA_star": 0.04793953255697314,
|
| 234 |
+
"reg29_star_per_seed": [
|
| 235 |
+
0.05279832218166869,
|
| 236 |
+
0.05271963186300921,
|
| 237 |
+
0.051832046588488945
|
| 238 |
+
]
|
| 239 |
},
|
| 240 |
"knn raw_b1": {
|
| 241 |
"rate": 0.25833333333333336,
|
|
|
|
| 245 |
"regB_ref": 0.11371326220225075,
|
| 246 |
"regB_star": 0.11629817415072874,
|
| 247 |
"regA_ref": 0.11371326220225075,
|
| 248 |
+
"regA_star": 0.11629817415072874,
|
| 249 |
+
"reg29_star_per_seed": [
|
| 250 |
+
1000.0
|
| 251 |
+
]
|
| 252 |
},
|
| 253 |
"knn raw_b4": {
|
| 254 |
"rate": 0.44166666666666665,
|
|
|
|
| 258 |
"regB_ref": 0.09416819337804627,
|
| 259 |
"regB_star": 0.11250915295445885,
|
| 260 |
"regA_ref": 0.11635060647060991,
|
| 261 |
+
"regA_star": 0.13716816454409586,
|
| 262 |
+
"reg29_star_per_seed": [
|
| 263 |
+
1000.0
|
| 264 |
+
]
|
| 265 |
},
|
| 266 |
"knn raw_b16": {
|
| 267 |
"rate": 0.7083333333333334,
|
|
|
|
| 271 |
"regB_ref": 0.08590952578105893,
|
| 272 |
"regB_star": 0.09869180446652481,
|
| 273 |
"regA_ref": 0.13353007468957645,
|
| 274 |
+
"regA_star": 0.13956844190360695,
|
| 275 |
+
"reg29_star_per_seed": [
|
| 276 |
+
0.1222650154352023
|
| 277 |
+
]
|
| 278 |
},
|
| 279 |
"knn filt_b1": {
|
| 280 |
"rate": 0.5083333333333333,
|
|
|
|
| 284 |
"regB_ref": 0.134908830251387,
|
| 285 |
"regB_star": 0.14090533433695862,
|
| 286 |
"regA_ref": 0.134908830251387,
|
| 287 |
+
"regA_star": 0.14090533433695862,
|
| 288 |
+
"reg29_star_per_seed": [
|
| 289 |
+
0.47128174901284803
|
| 290 |
+
]
|
| 291 |
},
|
| 292 |
"knn filt_b4": {
|
| 293 |
"rate": 0.7833333333333333,
|
|
|
|
| 297 |
"regB_ref": 0.10139965849798827,
|
| 298 |
"regB_star": 0.10879331316422458,
|
| 299 |
"regA_ref": 0.13114353705654402,
|
| 300 |
+
"regA_star": 0.1383683032238514,
|
| 301 |
+
"reg29_star_per_seed": [
|
| 302 |
+
0.12376717018326289
|
| 303 |
+
]
|
| 304 |
},
|
| 305 |
"knn filt_b16": {
|
| 306 |
"rate": 0.8666666666666667,
|
|
|
|
| 310 |
"regB_ref": 0.07096639087530376,
|
| 311 |
"regB_star": 0.08244864896963622,
|
| 312 |
"regA_ref": 0.13353036702921328,
|
| 313 |
+
"regA_star": 0.1383683032238514,
|
| 314 |
+
"reg29_star_per_seed": [
|
| 315 |
+
0.08899908416608437
|
| 316 |
+
]
|
| 317 |
},
|
| 318 |
"knn ref_b16_n3": {
|
| 319 |
"rate": 0.9333333333333333,
|
|
|
|
| 323 |
"regB_ref": 0.07228077375089703,
|
| 324 |
"regB_star": 0.0830961623057771,
|
| 325 |
"regA_ref": 0.12499251210513573,
|
| 326 |
+
"regA_star": 0.13353036702921328,
|
| 327 |
+
"reg29_star_per_seed": [
|
| 328 |
+
0.08746904759596996
|
| 329 |
+
]
|
| 330 |
},
|
| 331 |
"knn ref_b16_n10": {
|
| 332 |
"rate": 0.9333333333333333,
|
|
|
|
| 336 |
"regB_ref": 0.07337219985527899,
|
| 337 |
"regB_star": 0.08575503047710231,
|
| 338 |
"regA_ref": 0.12802583666738943,
|
| 339 |
+
"regA_star": 0.13421974481011856,
|
| 340 |
+
"reg29_star_per_seed": [
|
| 341 |
+
0.08835772743585302
|
| 342 |
+
]
|
| 343 |
}
|
| 344 |
}
|
results/icml_r3/X29/act/x27rescore_d3fm3n8000.txt
CHANGED
|
@@ -1,23 +1,23 @@
|
|
| 1 |
X29-A: X27's C-only arms re-scored, m = 3, tag d3fm3n8000, 120 test plants. J*_A = min(J_ref, 3A id and so3 labels) is below J_ref on 45 plants (median (J_ref - J*_A) / J*_A 0.000%).
|
| 2 |
-
diff raw_b1 certified 51.7% (published 51.7%) | cheapest certified: regret vs J_ref 3.52% (published 3.52%), vs J*_A 4.94% | first certified: vs J_ref 3.52%, vs J*_A 4.94%
|
| 3 |
-
diff raw_b4 certified 76.1% (published 76.1%) | cheapest certified: regret vs J_ref 2.66% (published 2.66%), vs J*_A 3.65% | first certified: vs J_ref 3.91%, vs J*_A 5.27%
|
| 4 |
-
diff raw_b16 certified 94.4% (published 94.4%) | cheapest certified: regret vs J_ref 1.77% (published 1.77%), vs J*_A 2.54% | first certified: vs J_ref 3.98%, vs J*_A 5.12%
|
| 5 |
-
diff filt_b1 certified 68.6% (published 68.6%) | cheapest certified: regret vs J_ref 3.86% (published 3.86%), vs J*_A 5.15% | first certified: vs J_ref 3.86%, vs J*_A 5.15%
|
| 6 |
-
diff filt_b4 certified 92.8% (published 92.8%) | cheapest certified: regret vs J_ref 2.46% (published 2.46%), vs J*_A 3.28% | first certified: vs J_ref 3.89%, vs J*_A 5.09%
|
| 7 |
-
diff filt_b16 certified 98.6% (published 98.6%) | cheapest certified: regret vs J_ref 1.50% (published 1.50%), vs J*_A 2.18% | first certified: vs J_ref 3.98%, vs J*_A 5.09%
|
| 8 |
-
diff ref_b16_n3 certified 99.4% (published 99.4%) | cheapest certified: regret vs J_ref 1.50% (published 1.50%), vs J*_A 2.19% | first certified: vs J_ref 3.99%, vs J*_A 5.12%
|
| 9 |
-
diff ref_b16_n10 certified 99.7% (published 99.7%) | cheapest certified: regret vs J_ref 1.50% (published 1.50%), vs J*_A 2.21% | first certified: vs J_ref 4.01%, vs J*_A 5.15%
|
| 10 |
-
twin raw_b1 certified 49.2% (published 49.2%) | cheapest certified: regret vs J_ref 3.49% (published 3.49%), vs J*_A 4.59% | first certified: vs J_ref 3.49%, vs J*_A 4.59%
|
| 11 |
-
twin raw_b4 certified 62.2% (published 62.2%) | cheapest certified: regret vs J_ref 3.11% (published 3.11%), vs J*_A 4.01% | first certified: vs J_ref 3.60%, vs J*_A 4.61%
|
| 12 |
-
twin raw_b16 certified 68.9% (published 68.9%) | cheapest certified: regret vs J_ref 2.48% (published 2.48%), vs J*_A 3.40% | first certified: vs J_ref 3.67%, vs J*_A 4.77%
|
| 13 |
-
twin filt_b1 certified 57.2% (published 57.2%) | cheapest certified: regret vs J_ref 3.52% (published 3.52%), vs J*_A 4.53% | first certified: vs J_ref 3.52%, vs J*_A 4.53%
|
| 14 |
-
twin filt_b4 certified 68.1% (published 68.1%) | cheapest certified: regret vs J_ref 3.14% (published 3.14%), vs J*_A 4.03% | first certified: vs J_ref 3.66%, vs J*_A 4.70%
|
| 15 |
-
twin filt_b16 certified 73.1% (published 73.1%) | cheapest certified: regret vs J_ref 2.50% (published 2.50%), vs J*_A 3.39% | first certified: vs J_ref 3.68%, vs J*_A 4.79%
|
| 16 |
-
knn raw_b1 certified 25.8% (published 25.8%) | cheapest certified: regret vs J_ref 11.37% (published 11.37%), vs J*_A 11.63% | first certified: vs J_ref 11.37%, vs J*_A 11.63%
|
| 17 |
-
knn raw_b4 certified 44.2% (published 44.2%) | cheapest certified: regret vs J_ref 9.42% (published 9.42%), vs J*_A 11.25% | first certified: vs J_ref 11.64%, vs J*_A 13.72%
|
| 18 |
-
knn raw_b16 certified 70.8% (published 70.8%) | cheapest certified: regret vs J_ref 8.59% (published 8.59%), vs J*_A 9.87% | first certified: vs J_ref 13.35%, vs J*_A 13.96%
|
| 19 |
-
knn filt_b1 certified 50.8% (published 50.8%) | cheapest certified: regret vs J_ref 13.49% (published 13.49%), vs J*_A 14.09% | first certified: vs J_ref 13.49%, vs J*_A 14.09%
|
| 20 |
-
knn filt_b4 certified 78.3% (published 78.3%) | cheapest certified: regret vs J_ref 10.14% (published 10.14%), vs J*_A 10.88% | first certified: vs J_ref 13.11%, vs J*_A 13.84%
|
| 21 |
-
knn filt_b16 certified 86.7% (published 86.7%) | cheapest certified: regret vs J_ref 7.10% (published 7.10%), vs J*_A 8.24% | first certified: vs J_ref 13.35%, vs J*_A 13.84%
|
| 22 |
-
knn ref_b16_n3 certified 93.3% (published 93.3%) | cheapest certified: regret vs J_ref 7.23% (published 7.23%), vs J*_A 8.31% | first certified: vs J_ref 12.50%, vs J*_A 13.35%
|
| 23 |
-
knn ref_b16_n10 certified 93.3% (published 93.3%) | cheapest certified: regret vs J_ref 7.34% (published 7.34%), vs J*_A 8.58% | first certified: vs J_ref 12.80%, vs J*_A 13.42%
|
|
|
|
| 1 |
X29-A: X27's C-only arms re-scored, m = 3, tag d3fm3n8000, 120 test plants. J*_A = min(J_ref, 3A id and so3 labels) is below J_ref on 45 plants (median (J_ref - J*_A) / J*_A 0.000%).
|
| 2 |
+
diff raw_b1 certified 51.7% (published 51.7%) | cheapest certified: regret vs J_ref 3.52% (published 3.52%), vs J*_A 4.94% | first certified: vs J_ref 3.52%, vs J*_A 4.94% | X29 convention vs J*_A, per seed: 11.74% / 60.49% / n/a
|
| 3 |
+
diff raw_b4 certified 76.1% (published 76.1%) | cheapest certified: regret vs J_ref 2.66% (published 2.66%), vs J*_A 3.65% | first certified: vs J_ref 3.91%, vs J*_A 5.27% | X29 convention vs J*_A, per seed: 5.47% / 5.11% / 5.15%
|
| 4 |
+
diff raw_b16 certified 94.4% (published 94.4%) | cheapest certified: regret vs J_ref 1.77% (published 1.77%), vs J*_A 2.54% | first certified: vs J_ref 3.98%, vs J*_A 5.12% | X29 convention vs J*_A, per seed: 2.58% / 2.95% / 2.77%
|
| 5 |
+
diff filt_b1 certified 68.6% (published 68.6%) | cheapest certified: regret vs J_ref 3.86% (published 3.86%), vs J*_A 5.15% | first certified: vs J_ref 3.86%, vs J*_A 5.15% | X29 convention vs J*_A, per seed: 7.64% / 7.43% / 10.77%
|
| 6 |
+
diff filt_b4 certified 92.8% (published 92.8%) | cheapest certified: regret vs J_ref 2.46% (published 2.46%), vs J*_A 3.28% | first certified: vs J_ref 3.89%, vs J*_A 5.09% | X29 convention vs J*_A, per seed: 3.56% / 3.86% / 3.22%
|
| 7 |
+
diff filt_b16 certified 98.6% (published 98.6%) | cheapest certified: regret vs J_ref 1.50% (published 1.50%), vs J*_A 2.18% | first certified: vs J_ref 3.98%, vs J*_A 5.09% | X29 convention vs J*_A, per seed: 2.35% / 2.33% / 2.03%
|
| 8 |
+
diff ref_b16_n3 certified 99.4% (published 99.4%) | cheapest certified: regret vs J_ref 1.50% (published 1.50%), vs J*_A 2.19% | first certified: vs J_ref 3.99%, vs J*_A 5.12% | X29 convention vs J*_A, per seed: 2.35% / 2.33% / 1.98%
|
| 9 |
+
diff ref_b16_n10 certified 99.7% (published 99.7%) | cheapest certified: regret vs J_ref 1.50% (published 1.50%), vs J*_A 2.21% | first certified: vs J_ref 4.01%, vs J*_A 5.15% | X29 convention vs J*_A, per seed: 2.35% / 2.33% / 1.98%
|
| 10 |
+
twin raw_b1 certified 49.2% (published 49.2%) | cheapest certified: regret vs J_ref 3.49% (published 3.49%), vs J*_A 4.59% | first certified: vs J_ref 3.49%, vs J*_A 4.59% | X29 convention vs J*_A, per seed: n/a / n/a / n/a
|
| 11 |
+
twin raw_b4 certified 62.2% (published 62.2%) | cheapest certified: regret vs J_ref 3.11% (published 3.11%), vs J*_A 4.01% | first certified: vs J_ref 3.60%, vs J*_A 4.61% | X29 convention vs J*_A, per seed: 7.31% / 7.47% / 8.09%
|
| 12 |
+
twin raw_b16 certified 68.9% (published 68.9%) | cheapest certified: regret vs J_ref 2.48% (published 2.48%), vs J*_A 3.40% | first certified: vs J_ref 3.67%, vs J*_A 4.77% | X29 convention vs J*_A, per seed: 5.81% / 5.42% / 5.91%
|
| 13 |
+
twin filt_b1 certified 57.2% (published 57.2%) | cheapest certified: regret vs J_ref 3.52% (published 3.52%), vs J*_A 4.53% | first certified: vs J_ref 3.52%, vs J*_A 4.53% | X29 convention vs J*_A, per seed: 9.20% / 10.14% / 8.67%
|
| 14 |
+
twin filt_b4 certified 68.1% (published 68.1%) | cheapest certified: regret vs J_ref 3.14% (published 3.14%), vs J*_A 4.03% | first certified: vs J_ref 3.66%, vs J*_A 4.70% | X29 convention vs J*_A, per seed: 6.74% / 6.82% / 6.36%
|
| 15 |
+
twin filt_b16 certified 73.1% (published 73.1%) | cheapest certified: regret vs J_ref 2.50% (published 2.50%), vs J*_A 3.39% | first certified: vs J_ref 3.68%, vs J*_A 4.79% | X29 convention vs J*_A, per seed: 5.28% / 5.27% / 5.18%
|
| 16 |
+
knn raw_b1 certified 25.8% (published 25.8%) | cheapest certified: regret vs J_ref 11.37% (published 11.37%), vs J*_A 11.63% | first certified: vs J_ref 11.37%, vs J*_A 11.63% | X29 convention vs J*_A, per seed: n/a
|
| 17 |
+
knn raw_b4 certified 44.2% (published 44.2%) | cheapest certified: regret vs J_ref 9.42% (published 9.42%), vs J*_A 11.25% | first certified: vs J_ref 11.64%, vs J*_A 13.72% | X29 convention vs J*_A, per seed: n/a
|
| 18 |
+
knn raw_b16 certified 70.8% (published 70.8%) | cheapest certified: regret vs J_ref 8.59% (published 8.59%), vs J*_A 9.87% | first certified: vs J_ref 13.35%, vs J*_A 13.96% | X29 convention vs J*_A, per seed: 12.23%
|
| 19 |
+
knn filt_b1 certified 50.8% (published 50.8%) | cheapest certified: regret vs J_ref 13.49% (published 13.49%), vs J*_A 14.09% | first certified: vs J_ref 13.49%, vs J*_A 14.09% | X29 convention vs J*_A, per seed: 47.13%
|
| 20 |
+
knn filt_b4 certified 78.3% (published 78.3%) | cheapest certified: regret vs J_ref 10.14% (published 10.14%), vs J*_A 10.88% | first certified: vs J_ref 13.11%, vs J*_A 13.84% | X29 convention vs J*_A, per seed: 12.38%
|
| 21 |
+
knn filt_b16 certified 86.7% (published 86.7%) | cheapest certified: regret vs J_ref 7.10% (published 7.10%), vs J*_A 8.24% | first certified: vs J_ref 13.35%, vs J*_A 13.84% | X29 convention vs J*_A, per seed: 8.90%
|
| 22 |
+
knn ref_b16_n3 certified 93.3% (published 93.3%) | cheapest certified: regret vs J_ref 7.23% (published 7.23%), vs J*_A 8.31% | first certified: vs J_ref 12.50%, vs J*_A 13.35% | X29 convention vs J*_A, per seed: 8.75%
|
| 23 |
+
knn ref_b16_n10 certified 93.3% (published 93.3%) | cheapest certified: regret vs J_ref 7.34% (published 7.34%), vs J*_A 8.58% | first certified: vs J_ref 12.80%, vs J*_A 13.42% | X29 convention vs J*_A, per seed: 8.84%
|
results/icml_r3/X29/act/x27rescore_d3fm4n8000.json
CHANGED
|
@@ -9,7 +9,12 @@
|
|
| 9 |
"regB_ref": 0.04281187301679423,
|
| 10 |
"regB_star": 0.04962915968635695,
|
| 11 |
"regA_ref": 0.04281187301679423,
|
| 12 |
-
"regA_star": 0.04962915968635695
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
},
|
| 14 |
"diff raw_b4": {
|
| 15 |
"rate": 0.5722222222222222,
|
|
@@ -21,7 +26,12 @@
|
|
| 21 |
"regB_ref": 0.04023679269097036,
|
| 22 |
"regB_star": 0.047381757855627926,
|
| 23 |
"regA_ref": 0.04831828056771548,
|
| 24 |
-
"regA_star": 0.057019597285727364
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
},
|
| 26 |
"diff raw_b16": {
|
| 27 |
"rate": 0.7888888888888889,
|
|
@@ -33,7 +43,12 @@
|
|
| 33 |
"regB_ref": 0.03206114493999934,
|
| 34 |
"regB_star": 0.03878398370676894,
|
| 35 |
"regA_ref": 0.05338276992090041,
|
| 36 |
-
"regA_star": 0.05942518962649945
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
},
|
| 38 |
"diff filt_b1": {
|
| 39 |
"rate": 0.5249999999999999,
|
|
@@ -45,7 +60,12 @@
|
|
| 45 |
"regB_ref": 0.05256149004484412,
|
| 46 |
"regB_star": 0.05789482111431554,
|
| 47 |
"regA_ref": 0.05256149004484412,
|
| 48 |
-
"regA_star": 0.05789482111431554
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 49 |
},
|
| 50 |
"diff filt_b4": {
|
| 51 |
"rate": 0.8250000000000001,
|
|
@@ -57,7 +77,12 @@
|
|
| 57 |
"regB_ref": 0.04154891326937804,
|
| 58 |
"regB_star": 0.05096408010028841,
|
| 59 |
"regA_ref": 0.05679915606087227,
|
| 60 |
-
"regA_star": 0.06378147142694024
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 61 |
},
|
| 62 |
"diff filt_b16": {
|
| 63 |
"rate": 0.9194444444444444,
|
|
@@ -69,7 +94,12 @@
|
|
| 69 |
"regB_ref": 0.030428223592535053,
|
| 70 |
"regB_star": 0.03729421986230963,
|
| 71 |
"regA_ref": 0.05678969283770341,
|
| 72 |
-
"regA_star": 0.06378147142694024
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 73 |
},
|
| 74 |
"diff ref_b16_n3": {
|
| 75 |
"rate": 0.9666666666666667,
|
|
@@ -81,7 +111,12 @@
|
|
| 81 |
"regB_ref": 0.032163477351268455,
|
| 82 |
"regB_star": 0.03848414313894111,
|
| 83 |
"regA_ref": 0.05752555203944998,
|
| 84 |
-
"regA_star": 0.06417574626693279
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 85 |
},
|
| 86 |
"diff ref_b16_n10": {
|
| 87 |
"rate": 0.9777777777777777,
|
|
@@ -93,7 +128,12 @@
|
|
| 93 |
"regB_ref": 0.03239370322615842,
|
| 94 |
"regB_star": 0.03848414313894111,
|
| 95 |
"regA_ref": 0.0571452435546439,
|
| 96 |
-
"regA_star": 0.0639596586478054
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 97 |
},
|
| 98 |
"twin raw_b1": {
|
| 99 |
"rate": 0.3138888888888889,
|
|
@@ -105,7 +145,12 @@
|
|
| 105 |
"regB_ref": 0.05077043279796922,
|
| 106 |
"regB_star": 0.054942478154655694,
|
| 107 |
"regA_ref": 0.05077043279796922,
|
| 108 |
-
"regA_star": 0.054942478154655694
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 109 |
},
|
| 110 |
"twin raw_b4": {
|
| 111 |
"rate": 0.4305555555555555,
|
|
@@ -117,7 +162,12 @@
|
|
| 117 |
"regB_ref": 0.040770558505952126,
|
| 118 |
"regB_star": 0.04486400154999837,
|
| 119 |
"regA_ref": 0.044869776269671036,
|
| 120 |
-
"regA_star": 0.051729042499672806
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 121 |
},
|
| 122 |
"twin raw_b16": {
|
| 123 |
"rate": 0.48333333333333334,
|
|
@@ -129,7 +179,12 @@
|
|
| 129 |
"regB_ref": 0.03419325650322769,
|
| 130 |
"regB_star": 0.03997382209205025,
|
| 131 |
"regA_ref": 0.04478727476519462,
|
| 132 |
-
"regA_star": 0.051812151699132114
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 133 |
},
|
| 134 |
"twin filt_b1": {
|
| 135 |
"rate": 0.3666666666666667,
|
|
@@ -141,7 +196,12 @@
|
|
| 141 |
"regB_ref": 0.0509702614716806,
|
| 142 |
"regB_star": 0.05496224880262157,
|
| 143 |
"regA_ref": 0.0509702614716806,
|
| 144 |
-
"regA_star": 0.05496224880262157
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 145 |
},
|
| 146 |
"twin filt_b4": {
|
| 147 |
"rate": 0.4861111111111111,
|
|
@@ -153,7 +213,12 @@
|
|
| 153 |
"regB_ref": 0.039893127429255326,
|
| 154 |
"regB_star": 0.044859535745951584,
|
| 155 |
"regA_ref": 0.044869776269671036,
|
| 156 |
-
"regA_star": 0.051729042499672806
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 157 |
},
|
| 158 |
"twin filt_b16": {
|
| 159 |
"rate": 0.5222222222222221,
|
|
@@ -165,7 +230,12 @@
|
|
| 165 |
"regB_ref": 0.03461930665839119,
|
| 166 |
"regB_star": 0.039952457345226344,
|
| 167 |
"regA_ref": 0.045169830011552614,
|
| 168 |
-
"regA_star": 0.05347942435820763
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 169 |
},
|
| 170 |
"knn raw_b1": {
|
| 171 |
"rate": 0.14166666666666666,
|
|
@@ -175,7 +245,10 @@
|
|
| 175 |
"regB_ref": 0.11972585447859752,
|
| 176 |
"regB_star": 0.1203804444936909,
|
| 177 |
"regA_ref": 0.11972585447859752,
|
| 178 |
-
"regA_star": 0.1203804444936909
|
|
|
|
|
|
|
|
|
|
| 179 |
},
|
| 180 |
"knn raw_b4": {
|
| 181 |
"rate": 0.39166666666666666,
|
|
@@ -185,7 +258,10 @@
|
|
| 185 |
"regB_ref": 0.11972585447859752,
|
| 186 |
"regB_star": 0.1249036336281455,
|
| 187 |
"regA_ref": 0.1249036336281455,
|
| 188 |
-
"regA_star": 0.13069438483572582
|
|
|
|
|
|
|
|
|
|
| 189 |
},
|
| 190 |
"knn raw_b16": {
|
| 191 |
"rate": 0.5916666666666667,
|
|
@@ -195,7 +271,10 @@
|
|
| 195 |
"regB_ref": 0.10947876745055107,
|
| 196 |
"regB_star": 0.11616937090964607,
|
| 197 |
"regA_ref": 0.13058469845406107,
|
| 198 |
-
"regA_star": 0.1400175969983004
|
|
|
|
|
|
|
|
|
|
| 199 |
},
|
| 200 |
"knn filt_b1": {
|
| 201 |
"rate": 0.45,
|
|
@@ -205,7 +284,10 @@
|
|
| 205 |
"regB_ref": 0.11863223190950545,
|
| 206 |
"regB_star": 0.12336736317427233,
|
| 207 |
"regA_ref": 0.11863223190950545,
|
| 208 |
-
"regA_star": 0.12336736317427233
|
|
|
|
|
|
|
|
|
|
| 209 |
},
|
| 210 |
"knn filt_b4": {
|
| 211 |
"rate": 0.75,
|
|
@@ -215,7 +297,10 @@
|
|
| 215 |
"regB_ref": 0.11100685037250602,
|
| 216 |
"regB_star": 0.11698292149647704,
|
| 217 |
"regA_ref": 0.13511513600466607,
|
| 218 |
-
"regA_star": 0.14419869136826147
|
|
|
|
|
|
|
|
|
|
| 219 |
},
|
| 220 |
"knn filt_b16": {
|
| 221 |
"rate": 0.7916666666666666,
|
|
@@ -225,7 +310,10 @@
|
|
| 225 |
"regB_ref": 0.0926797810653952,
|
| 226 |
"regB_star": 0.09917966684483925,
|
| 227 |
"regA_ref": 0.1400175969983004,
|
| 228 |
-
"regA_star": 0.14697604799704078
|
|
|
|
|
|
|
|
|
|
| 229 |
},
|
| 230 |
"knn ref_b16_n3": {
|
| 231 |
"rate": 0.9,
|
|
@@ -235,7 +323,10 @@
|
|
| 235 |
"regB_ref": 0.09150018593164855,
|
| 236 |
"regB_star": 0.09801292912248993,
|
| 237 |
"regA_ref": 0.12780736488049338,
|
| 238 |
-
"regA_star": 0.13983158527678574
|
|
|
|
|
|
|
|
|
|
| 239 |
},
|
| 240 |
"knn ref_b16_n10": {
|
| 241 |
"rate": 0.9333333333333333,
|
|
@@ -245,6 +336,9 @@
|
|
| 245 |
"regB_ref": 0.09026618689333521,
|
| 246 |
"regB_star": 0.09677288836815534,
|
| 247 |
"regA_ref": 0.1258708743514112,
|
| 248 |
-
"regA_star": 0.1349668764575651
|
|
|
|
|
|
|
|
|
|
| 249 |
}
|
| 250 |
}
|
|
|
|
| 9 |
"regB_ref": 0.04281187301679423,
|
| 10 |
"regB_star": 0.04962915968635695,
|
| 11 |
"regA_ref": 0.04281187301679423,
|
| 12 |
+
"regA_star": 0.04962915968635695,
|
| 13 |
+
"reg29_star_per_seed": [
|
| 14 |
+
1000.0,
|
| 15 |
+
1000.0,
|
| 16 |
+
1000.0
|
| 17 |
+
]
|
| 18 |
},
|
| 19 |
"diff raw_b4": {
|
| 20 |
"rate": 0.5722222222222222,
|
|
|
|
| 26 |
"regB_ref": 0.04023679269097036,
|
| 27 |
"regB_star": 0.047381757855627926,
|
| 28 |
"regA_ref": 0.04831828056771548,
|
| 29 |
+
"regA_star": 0.057019597285727364,
|
| 30 |
+
"reg29_star_per_seed": [
|
| 31 |
+
0.08495044681003916,
|
| 32 |
+
0.0881141812966516,
|
| 33 |
+
0.11122677860139785
|
| 34 |
+
]
|
| 35 |
},
|
| 36 |
"diff raw_b16": {
|
| 37 |
"rate": 0.7888888888888889,
|
|
|
|
| 43 |
"regB_ref": 0.03206114493999934,
|
| 44 |
"regB_star": 0.03878398370676894,
|
| 45 |
"regA_ref": 0.05338276992090041,
|
| 46 |
+
"regA_star": 0.05942518962649945,
|
| 47 |
+
"reg29_star_per_seed": [
|
| 48 |
+
0.047293419441651996,
|
| 49 |
+
0.05764590082594878,
|
| 50 |
+
0.05264291865089865
|
| 51 |
+
]
|
| 52 |
},
|
| 53 |
"diff filt_b1": {
|
| 54 |
"rate": 0.5249999999999999,
|
|
|
|
| 60 |
"regB_ref": 0.05256149004484412,
|
| 61 |
"regB_star": 0.05789482111431554,
|
| 62 |
"regA_ref": 0.05256149004484412,
|
| 63 |
+
"regA_star": 0.05789482111431554,
|
| 64 |
+
"reg29_star_per_seed": [
|
| 65 |
+
0.263211143468268,
|
| 66 |
+
0.12940601631344728,
|
| 67 |
+
0.14249857544129063
|
| 68 |
+
]
|
| 69 |
},
|
| 70 |
"diff filt_b4": {
|
| 71 |
"rate": 0.8250000000000001,
|
|
|
|
| 77 |
"regB_ref": 0.04154891326937804,
|
| 78 |
"regB_star": 0.05096408010028841,
|
| 79 |
"regA_ref": 0.05679915606087227,
|
| 80 |
+
"regA_star": 0.06378147142694024,
|
| 81 |
+
"reg29_star_per_seed": [
|
| 82 |
+
0.06236438872575767,
|
| 83 |
+
0.06424515800692798,
|
| 84 |
+
0.060460629694771506
|
| 85 |
+
]
|
| 86 |
},
|
| 87 |
"diff filt_b16": {
|
| 88 |
"rate": 0.9194444444444444,
|
|
|
|
| 94 |
"regB_ref": 0.030428223592535053,
|
| 95 |
"regB_star": 0.03729421986230963,
|
| 96 |
"regA_ref": 0.05678969283770341,
|
| 97 |
+
"regA_star": 0.06378147142694024,
|
| 98 |
+
"reg29_star_per_seed": [
|
| 99 |
+
0.038550378977868505,
|
| 100 |
+
0.04405060528876292,
|
| 101 |
+
0.03878398370676894
|
| 102 |
+
]
|
| 103 |
},
|
| 104 |
"diff ref_b16_n3": {
|
| 105 |
"rate": 0.9666666666666667,
|
|
|
|
| 111 |
"regB_ref": 0.032163477351268455,
|
| 112 |
"regB_star": 0.03848414313894111,
|
| 113 |
"regA_ref": 0.05752555203944998,
|
| 114 |
+
"regA_star": 0.06417574626693279,
|
| 115 |
+
"reg29_star_per_seed": [
|
| 116 |
+
0.038048491177176014,
|
| 117 |
+
0.04358892940577752,
|
| 118 |
+
0.03850457199311308
|
| 119 |
+
]
|
| 120 |
},
|
| 121 |
"diff ref_b16_n10": {
|
| 122 |
"rate": 0.9777777777777777,
|
|
|
|
| 128 |
"regB_ref": 0.03239370322615842,
|
| 129 |
"regB_star": 0.03848414313894111,
|
| 130 |
"regA_ref": 0.0571452435546439,
|
| 131 |
+
"regA_star": 0.0639596586478054,
|
| 132 |
+
"reg29_star_per_seed": [
|
| 133 |
+
0.0375075533219793,
|
| 134 |
+
0.04358892940577752,
|
| 135 |
+
0.037786990412120525
|
| 136 |
+
]
|
| 137 |
},
|
| 138 |
"twin raw_b1": {
|
| 139 |
"rate": 0.3138888888888889,
|
|
|
|
| 145 |
"regB_ref": 0.05077043279796922,
|
| 146 |
"regB_star": 0.054942478154655694,
|
| 147 |
"regA_ref": 0.05077043279796922,
|
| 148 |
+
"regA_star": 0.054942478154655694,
|
| 149 |
+
"reg29_star_per_seed": [
|
| 150 |
+
1000.0,
|
| 151 |
+
1000.0,
|
| 152 |
+
1000.0
|
| 153 |
+
]
|
| 154 |
},
|
| 155 |
"twin raw_b4": {
|
| 156 |
"rate": 0.4305555555555555,
|
|
|
|
| 162 |
"regB_ref": 0.040770558505952126,
|
| 163 |
"regB_star": 0.04486400154999837,
|
| 164 |
"regA_ref": 0.044869776269671036,
|
| 165 |
+
"regA_star": 0.051729042499672806,
|
| 166 |
+
"reg29_star_per_seed": [
|
| 167 |
+
1000.0,
|
| 168 |
+
1000.0,
|
| 169 |
+
1000.0
|
| 170 |
+
]
|
| 171 |
},
|
| 172 |
"twin raw_b16": {
|
| 173 |
"rate": 0.48333333333333334,
|
|
|
|
| 179 |
"regB_ref": 0.03419325650322769,
|
| 180 |
"regB_star": 0.03997382209205025,
|
| 181 |
"regA_ref": 0.04478727476519462,
|
| 182 |
+
"regA_star": 0.051812151699132114,
|
| 183 |
+
"reg29_star_per_seed": [
|
| 184 |
+
1000.0,
|
| 185 |
+
1000.0,
|
| 186 |
+
500.05918359518233
|
| 187 |
+
]
|
| 188 |
},
|
| 189 |
"twin filt_b1": {
|
| 190 |
"rate": 0.3666666666666667,
|
|
|
|
| 196 |
"regB_ref": 0.0509702614716806,
|
| 197 |
"regB_star": 0.05496224880262157,
|
| 198 |
"regA_ref": 0.0509702614716806,
|
| 199 |
+
"regA_star": 0.05496224880262157,
|
| 200 |
+
"reg29_star_per_seed": [
|
| 201 |
+
1000.0,
|
| 202 |
+
1000.0,
|
| 203 |
+
1000.0
|
| 204 |
+
]
|
| 205 |
},
|
| 206 |
"twin filt_b4": {
|
| 207 |
"rate": 0.4861111111111111,
|
|
|
|
| 213 |
"regB_ref": 0.039893127429255326,
|
| 214 |
"regB_star": 0.044859535745951584,
|
| 215 |
"regA_ref": 0.044869776269671036,
|
| 216 |
+
"regA_star": 0.051729042499672806,
|
| 217 |
+
"reg29_star_per_seed": [
|
| 218 |
+
1000.0,
|
| 219 |
+
1000.0,
|
| 220 |
+
500.06559232249737
|
| 221 |
+
]
|
| 222 |
},
|
| 223 |
"twin filt_b16": {
|
| 224 |
"rate": 0.5222222222222221,
|
|
|
|
| 230 |
"regB_ref": 0.03461930665839119,
|
| 231 |
"regB_star": 0.039952457345226344,
|
| 232 |
"regA_ref": 0.045169830011552614,
|
| 233 |
+
"regA_star": 0.05347942435820763,
|
| 234 |
+
"reg29_star_per_seed": [
|
| 235 |
+
0.13461082478981312,
|
| 236 |
+
0.13603789452203452,
|
| 237 |
+
0.09103587528874801
|
| 238 |
+
]
|
| 239 |
},
|
| 240 |
"knn raw_b1": {
|
| 241 |
"rate": 0.14166666666666666,
|
|
|
|
| 245 |
"regB_ref": 0.11972585447859752,
|
| 246 |
"regB_star": 0.1203804444936909,
|
| 247 |
"regA_ref": 0.11972585447859752,
|
| 248 |
+
"regA_star": 0.1203804444936909,
|
| 249 |
+
"reg29_star_per_seed": [
|
| 250 |
+
1000.0
|
| 251 |
+
]
|
| 252 |
},
|
| 253 |
"knn raw_b4": {
|
| 254 |
"rate": 0.39166666666666666,
|
|
|
|
| 258 |
"regB_ref": 0.11972585447859752,
|
| 259 |
"regB_star": 0.1249036336281455,
|
| 260 |
"regA_ref": 0.1249036336281455,
|
| 261 |
+
"regA_star": 0.13069438483572582,
|
| 262 |
+
"reg29_star_per_seed": [
|
| 263 |
+
1000.0
|
| 264 |
+
]
|
| 265 |
},
|
| 266 |
"knn raw_b16": {
|
| 267 |
"rate": 0.5916666666666667,
|
|
|
|
| 271 |
"regB_ref": 0.10947876745055107,
|
| 272 |
"regB_star": 0.11616937090964607,
|
| 273 |
"regA_ref": 0.13058469845406107,
|
| 274 |
+
"regA_star": 0.1400175969983004,
|
| 275 |
+
"reg29_star_per_seed": [
|
| 276 |
+
0.1783168905724013
|
| 277 |
+
]
|
| 278 |
},
|
| 279 |
"knn filt_b1": {
|
| 280 |
"rate": 0.45,
|
|
|
|
| 284 |
"regB_ref": 0.11863223190950545,
|
| 285 |
"regB_star": 0.12336736317427233,
|
| 286 |
"regA_ref": 0.11863223190950545,
|
| 287 |
+
"regA_star": 0.12336736317427233,
|
| 288 |
+
"reg29_star_per_seed": [
|
| 289 |
+
1000.0
|
| 290 |
+
]
|
| 291 |
},
|
| 292 |
"knn filt_b4": {
|
| 293 |
"rate": 0.75,
|
|
|
|
| 297 |
"regB_ref": 0.11100685037250602,
|
| 298 |
"regB_star": 0.11698292149647704,
|
| 299 |
"regA_ref": 0.13511513600466607,
|
| 300 |
+
"regA_star": 0.14419869136826147,
|
| 301 |
+
"reg29_star_per_seed": [
|
| 302 |
+
0.1489442082590664
|
| 303 |
+
]
|
| 304 |
},
|
| 305 |
"knn filt_b16": {
|
| 306 |
"rate": 0.7916666666666666,
|
|
|
|
| 310 |
"regB_ref": 0.0926797810653952,
|
| 311 |
"regB_star": 0.09917966684483925,
|
| 312 |
"regA_ref": 0.1400175969983004,
|
| 313 |
+
"regA_star": 0.14697604799704078,
|
| 314 |
+
"reg29_star_per_seed": [
|
| 315 |
+
0.11159647642534298
|
| 316 |
+
]
|
| 317 |
},
|
| 318 |
"knn ref_b16_n3": {
|
| 319 |
"rate": 0.9,
|
|
|
|
| 323 |
"regB_ref": 0.09150018593164855,
|
| 324 |
"regB_star": 0.09801292912248993,
|
| 325 |
"regA_ref": 0.12780736488049338,
|
| 326 |
+
"regA_star": 0.13983158527678574,
|
| 327 |
+
"reg29_star_per_seed": [
|
| 328 |
+
0.10582831324537034
|
| 329 |
+
]
|
| 330 |
},
|
| 331 |
"knn ref_b16_n10": {
|
| 332 |
"rate": 0.9333333333333333,
|
|
|
|
| 336 |
"regB_ref": 0.09026618689333521,
|
| 337 |
"regB_star": 0.09677288836815534,
|
| 338 |
"regA_ref": 0.1258708743514112,
|
| 339 |
+
"regA_star": 0.1349668764575651,
|
| 340 |
+
"reg29_star_per_seed": [
|
| 341 |
+
0.10014547921930417
|
| 342 |
+
]
|
| 343 |
}
|
| 344 |
}
|
results/icml_r3/X29/act/x27rescore_d3fm4n8000.txt
CHANGED
|
@@ -1,23 +1,23 @@
|
|
| 1 |
X29-A: X27's C-only arms re-scored, m = 4, tag d3fm4n8000, 120 test plants. J*_A = min(J_ref, 3A id and so3 labels) is below J_ref on 48 plants (median (J_ref - J*_A) / J*_A 0.000%).
|
| 2 |
-
diff raw_b1 certified 31.7% (published 31.7%) | cheapest certified: regret vs J_ref 4.28% (published 4.28%), vs J*_A 4.96% | first certified: vs J_ref 4.28%, vs J*_A 4.96%
|
| 3 |
-
diff raw_b4 certified 57.2% (published 57.2%) | cheapest certified: regret vs J_ref 4.02% (published 4.02%), vs J*_A 4.74% | first certified: vs J_ref 4.83%, vs J*_A 5.70%
|
| 4 |
-
diff raw_b16 certified 78.9% (published 78.9%) | cheapest certified: regret vs J_ref 3.21% (published 3.21%), vs J*_A 3.88% | first certified: vs J_ref 5.34%, vs J*_A 5.94%
|
| 5 |
-
diff filt_b1 certified 52.5% (published 52.5%) | cheapest certified: regret vs J_ref 5.26% (published 5.26%), vs J*_A 5.79% | first certified: vs J_ref 5.26%, vs J*_A 5.79%
|
| 6 |
-
diff filt_b4 certified 82.5% (published 82.5%) | cheapest certified: regret vs J_ref 4.15% (published 4.15%), vs J*_A 5.10% | first certified: vs J_ref 5.68%, vs J*_A 6.38%
|
| 7 |
-
diff filt_b16 certified 91.9% (published 91.9%) | cheapest certified: regret vs J_ref 3.04% (published 3.04%), vs J*_A 3.73% | first certified: vs J_ref 5.68%, vs J*_A 6.38%
|
| 8 |
-
diff ref_b16_n3 certified 96.7% (published 96.7%) | cheapest certified: regret vs J_ref 3.22% (published 3.22%), vs J*_A 3.85% | first certified: vs J_ref 5.75%, vs J*_A 6.42%
|
| 9 |
-
diff ref_b16_n10 certified 97.8% (published 97.8%) | cheapest certified: regret vs J_ref 3.24% (published 3.24%), vs J*_A 3.85% | first certified: vs J_ref 5.71%, vs J*_A 6.40%
|
| 10 |
-
twin raw_b1 certified 31.4% (published 31.4%) | cheapest certified: regret vs J_ref 5.08% (published 5.08%), vs J*_A 5.49% | first certified: vs J_ref 5.08%, vs J*_A 5.49%
|
| 11 |
-
twin raw_b4 certified 43.1% (published 43.1%) | cheapest certified: regret vs J_ref 4.08% (published 4.08%), vs J*_A 4.49% | first certified: vs J_ref 4.49%, vs J*_A 5.17%
|
| 12 |
-
twin raw_b16 certified 48.3% (published 48.3%) | cheapest certified: regret vs J_ref 3.42% (published 3.42%), vs J*_A 4.00% | first certified: vs J_ref 4.48%, vs J*_A 5.18%
|
| 13 |
-
twin filt_b1 certified 36.7% (published 36.7%) | cheapest certified: regret vs J_ref 5.10% (published 5.10%), vs J*_A 5.50% | first certified: vs J_ref 5.10%, vs J*_A 5.50%
|
| 14 |
-
twin filt_b4 certified 48.6% (published 48.6%) | cheapest certified: regret vs J_ref 3.99% (published 3.99%), vs J*_A 4.49% | first certified: vs J_ref 4.49%, vs J*_A 5.17%
|
| 15 |
-
twin filt_b16 certified 52.2% (published 52.2%) | cheapest certified: regret vs J_ref 3.46% (published 3.46%), vs J*_A 4.00% | first certified: vs J_ref 4.52%, vs J*_A 5.35%
|
| 16 |
-
knn raw_b1 certified 14.2% (published 14.2%) | cheapest certified: regret vs J_ref 11.97% (published 11.97%), vs J*_A 12.04% | first certified: vs J_ref 11.97%, vs J*_A 12.04%
|
| 17 |
-
knn raw_b4 certified 39.2% (published 39.2%) | cheapest certified: regret vs J_ref 11.97% (published 11.97%), vs J*_A 12.49% | first certified: vs J_ref 12.49%, vs J*_A 13.07%
|
| 18 |
-
knn raw_b16 certified 59.2% (published 59.2%) | cheapest certified: regret vs J_ref 10.95% (published 10.95%), vs J*_A 11.62% | first certified: vs J_ref 13.06%, vs J*_A 14.00%
|
| 19 |
-
knn filt_b1 certified 45.0% (published 45.0%) | cheapest certified: regret vs J_ref 11.86% (published 11.86%), vs J*_A 12.34% | first certified: vs J_ref 11.86%, vs J*_A 12.34%
|
| 20 |
-
knn filt_b4 certified 75.0% (published 75.0%) | cheapest certified: regret vs J_ref 11.10% (published 11.10%), vs J*_A 11.70% | first certified: vs J_ref 13.51%, vs J*_A 14.42%
|
| 21 |
-
knn filt_b16 certified 79.2% (published 79.2%) | cheapest certified: regret vs J_ref 9.27% (published 9.27%), vs J*_A 9.92% | first certified: vs J_ref 14.00%, vs J*_A 14.70%
|
| 22 |
-
knn ref_b16_n3 certified 90.0% (published 90.0%) | cheapest certified: regret vs J_ref 9.15% (published 9.15%), vs J*_A 9.80% | first certified: vs J_ref 12.78%, vs J*_A 13.98%
|
| 23 |
-
knn ref_b16_n10 certified 93.3% (published 93.3%) | cheapest certified: regret vs J_ref 9.03% (published 9.03%), vs J*_A 9.68% | first certified: vs J_ref 12.59%, vs J*_A 13.50%
|
|
|
|
| 1 |
X29-A: X27's C-only arms re-scored, m = 4, tag d3fm4n8000, 120 test plants. J*_A = min(J_ref, 3A id and so3 labels) is below J_ref on 48 plants (median (J_ref - J*_A) / J*_A 0.000%).
|
| 2 |
+
diff raw_b1 certified 31.7% (published 31.7%) | cheapest certified: regret vs J_ref 4.28% (published 4.28%), vs J*_A 4.96% | first certified: vs J_ref 4.28%, vs J*_A 4.96% | X29 convention vs J*_A, per seed: n/a / n/a / n/a
|
| 3 |
+
diff raw_b4 certified 57.2% (published 57.2%) | cheapest certified: regret vs J_ref 4.02% (published 4.02%), vs J*_A 4.74% | first certified: vs J_ref 4.83%, vs J*_A 5.70% | X29 convention vs J*_A, per seed: 8.50% / 8.81% / 11.12%
|
| 4 |
+
diff raw_b16 certified 78.9% (published 78.9%) | cheapest certified: regret vs J_ref 3.21% (published 3.21%), vs J*_A 3.88% | first certified: vs J_ref 5.34%, vs J*_A 5.94% | X29 convention vs J*_A, per seed: 4.73% / 5.76% / 5.26%
|
| 5 |
+
diff filt_b1 certified 52.5% (published 52.5%) | cheapest certified: regret vs J_ref 5.26% (published 5.26%), vs J*_A 5.79% | first certified: vs J_ref 5.26%, vs J*_A 5.79% | X29 convention vs J*_A, per seed: 26.32% / 12.94% / 14.25%
|
| 6 |
+
diff filt_b4 certified 82.5% (published 82.5%) | cheapest certified: regret vs J_ref 4.15% (published 4.15%), vs J*_A 5.10% | first certified: vs J_ref 5.68%, vs J*_A 6.38% | X29 convention vs J*_A, per seed: 6.24% / 6.42% / 6.05%
|
| 7 |
+
diff filt_b16 certified 91.9% (published 91.9%) | cheapest certified: regret vs J_ref 3.04% (published 3.04%), vs J*_A 3.73% | first certified: vs J_ref 5.68%, vs J*_A 6.38% | X29 convention vs J*_A, per seed: 3.86% / 4.41% / 3.88%
|
| 8 |
+
diff ref_b16_n3 certified 96.7% (published 96.7%) | cheapest certified: regret vs J_ref 3.22% (published 3.22%), vs J*_A 3.85% | first certified: vs J_ref 5.75%, vs J*_A 6.42% | X29 convention vs J*_A, per seed: 3.80% / 4.36% / 3.85%
|
| 9 |
+
diff ref_b16_n10 certified 97.8% (published 97.8%) | cheapest certified: regret vs J_ref 3.24% (published 3.24%), vs J*_A 3.85% | first certified: vs J_ref 5.71%, vs J*_A 6.40% | X29 convention vs J*_A, per seed: 3.75% / 4.36% / 3.78%
|
| 10 |
+
twin raw_b1 certified 31.4% (published 31.4%) | cheapest certified: regret vs J_ref 5.08% (published 5.08%), vs J*_A 5.49% | first certified: vs J_ref 5.08%, vs J*_A 5.49% | X29 convention vs J*_A, per seed: n/a / n/a / n/a
|
| 11 |
+
twin raw_b4 certified 43.1% (published 43.1%) | cheapest certified: regret vs J_ref 4.08% (published 4.08%), vs J*_A 4.49% | first certified: vs J_ref 4.49%, vs J*_A 5.17% | X29 convention vs J*_A, per seed: n/a / n/a / n/a
|
| 12 |
+
twin raw_b16 certified 48.3% (published 48.3%) | cheapest certified: regret vs J_ref 3.42% (published 3.42%), vs J*_A 4.00% | first certified: vs J_ref 4.48%, vs J*_A 5.18% | X29 convention vs J*_A, per seed: n/a / n/a / n/a
|
| 13 |
+
twin filt_b1 certified 36.7% (published 36.7%) | cheapest certified: regret vs J_ref 5.10% (published 5.10%), vs J*_A 5.50% | first certified: vs J_ref 5.10%, vs J*_A 5.50% | X29 convention vs J*_A, per seed: n/a / n/a / n/a
|
| 14 |
+
twin filt_b4 certified 48.6% (published 48.6%) | cheapest certified: regret vs J_ref 3.99% (published 3.99%), vs J*_A 4.49% | first certified: vs J_ref 4.49%, vs J*_A 5.17% | X29 convention vs J*_A, per seed: n/a / n/a / n/a
|
| 15 |
+
twin filt_b16 certified 52.2% (published 52.2%) | cheapest certified: regret vs J_ref 3.46% (published 3.46%), vs J*_A 4.00% | first certified: vs J_ref 4.52%, vs J*_A 5.35% | X29 convention vs J*_A, per seed: 13.46% / 13.60% / 9.10%
|
| 16 |
+
knn raw_b1 certified 14.2% (published 14.2%) | cheapest certified: regret vs J_ref 11.97% (published 11.97%), vs J*_A 12.04% | first certified: vs J_ref 11.97%, vs J*_A 12.04% | X29 convention vs J*_A, per seed: n/a
|
| 17 |
+
knn raw_b4 certified 39.2% (published 39.2%) | cheapest certified: regret vs J_ref 11.97% (published 11.97%), vs J*_A 12.49% | first certified: vs J_ref 12.49%, vs J*_A 13.07% | X29 convention vs J*_A, per seed: n/a
|
| 18 |
+
knn raw_b16 certified 59.2% (published 59.2%) | cheapest certified: regret vs J_ref 10.95% (published 10.95%), vs J*_A 11.62% | first certified: vs J_ref 13.06%, vs J*_A 14.00% | X29 convention vs J*_A, per seed: 17.83%
|
| 19 |
+
knn filt_b1 certified 45.0% (published 45.0%) | cheapest certified: regret vs J_ref 11.86% (published 11.86%), vs J*_A 12.34% | first certified: vs J_ref 11.86%, vs J*_A 12.34% | X29 convention vs J*_A, per seed: n/a
|
| 20 |
+
knn filt_b4 certified 75.0% (published 75.0%) | cheapest certified: regret vs J_ref 11.10% (published 11.10%), vs J*_A 11.70% | first certified: vs J_ref 13.51%, vs J*_A 14.42% | X29 convention vs J*_A, per seed: 14.89%
|
| 21 |
+
knn filt_b16 certified 79.2% (published 79.2%) | cheapest certified: regret vs J_ref 9.27% (published 9.27%), vs J*_A 9.92% | first certified: vs J_ref 14.00%, vs J*_A 14.70% | X29 convention vs J*_A, per seed: 11.16%
|
| 22 |
+
knn ref_b16_n3 certified 90.0% (published 90.0%) | cheapest certified: regret vs J_ref 9.15% (published 9.15%), vs J*_A 9.80% | first certified: vs J_ref 12.78%, vs J*_A 13.98% | X29 convention vs J*_A, per seed: 10.58%
|
| 23 |
+
knn ref_b16_n10 certified 93.3% (published 93.3%) | cheapest certified: regret vs J_ref 9.03% (published 9.03%), vs J*_A 9.68% | first certified: vs J_ref 12.59%, vs J*_A 13.50% | X29 convention vs J*_A, per seed: 10.01%
|
results/icml_r3/X29/act/x27rescore_d3fm5n16000.json
CHANGED
|
@@ -9,7 +9,12 @@
|
|
| 9 |
"regB_ref": 0.054012242469847704,
|
| 10 |
"regB_star": 0.06580516710136042,
|
| 11 |
"regA_ref": 0.054012242469847704,
|
| 12 |
-
"regA_star": 0.06580516710136042
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
},
|
| 14 |
"diff raw_b4": {
|
| 15 |
"rate": 0.49166666666666664,
|
|
@@ -21,7 +26,12 @@
|
|
| 21 |
"regB_ref": 0.046037766193027574,
|
| 22 |
"regB_star": 0.055668722380055824,
|
| 23 |
"regA_ref": 0.05123443031809827,
|
| 24 |
-
"regA_star": 0.06242150229911836
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
},
|
| 26 |
"diff raw_b16": {
|
| 27 |
"rate": 0.7916666666666666,
|
|
@@ -33,7 +43,12 @@
|
|
| 33 |
"regB_ref": 0.0392393348275164,
|
| 34 |
"regB_star": 0.04843269982130982,
|
| 35 |
"regA_ref": 0.053670339354187746,
|
| 36 |
-
"regA_star": 0.0653423576378871
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
},
|
| 38 |
"diff filt_b1": {
|
| 39 |
"rate": 0.4583333333333333,
|
|
@@ -45,7 +60,12 @@
|
|
| 45 |
"regB_ref": 0.05194845381582125,
|
| 46 |
"regB_star": 0.06513679765481717,
|
| 47 |
"regA_ref": 0.05194845381582125,
|
| 48 |
-
"regA_star": 0.06513679765481717
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 49 |
},
|
| 50 |
"diff filt_b4": {
|
| 51 |
"rate": 0.8138888888888888,
|
|
@@ -57,7 +77,12 @@
|
|
| 57 |
"regB_ref": 0.044054119434560945,
|
| 58 |
"regB_star": 0.05117955925012052,
|
| 59 |
"regA_ref": 0.053670339354187746,
|
| 60 |
-
"regA_star": 0.06526846423539756
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 61 |
},
|
| 62 |
"diff filt_b16": {
|
| 63 |
"rate": 0.9083333333333333,
|
|
@@ -69,7 +94,12 @@
|
|
| 69 |
"regB_ref": 0.033760278025647406,
|
| 70 |
"regB_star": 0.0392393348275164,
|
| 71 |
"regA_ref": 0.05494152341582326,
|
| 72 |
-
"regA_star": 0.06614503852117859
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 73 |
},
|
| 74 |
"diff ref_b16_n3": {
|
| 75 |
"rate": 0.9638888888888889,
|
|
@@ -81,7 +111,12 @@
|
|
| 81 |
"regB_ref": 0.034359315053998474,
|
| 82 |
"regB_star": 0.039602163709273386,
|
| 83 |
"regA_ref": 0.056236553327202676,
|
| 84 |
-
"regA_star": 0.06713280900832053
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 85 |
},
|
| 86 |
"diff ref_b16_n10": {
|
| 87 |
"rate": 0.9805555555555556,
|
|
@@ -93,7 +128,12 @@
|
|
| 93 |
"regB_ref": 0.03437394402248328,
|
| 94 |
"regB_star": 0.040009974164940454,
|
| 95 |
"regA_ref": 0.05649656778792678,
|
| 96 |
-
"regA_star": 0.06751706328759521
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 97 |
},
|
| 98 |
"twin raw_b1": {
|
| 99 |
"rate": 0.24722222222222223,
|
|
@@ -105,7 +145,12 @@
|
|
| 105 |
"regB_ref": 0.04588446769434418,
|
| 106 |
"regB_star": 0.05281368503715744,
|
| 107 |
"regA_ref": 0.04588446769434418,
|
| 108 |
-
"regA_star": 0.05281368503715744
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 109 |
},
|
| 110 |
"twin raw_b4": {
|
| 111 |
"rate": 0.37777777777777777,
|
|
@@ -117,7 +162,12 @@
|
|
| 117 |
"regB_ref": 0.03958246856186798,
|
| 118 |
"regB_star": 0.046969980502976294,
|
| 119 |
"regA_ref": 0.04453505836815008,
|
| 120 |
-
"regA_star": 0.05310045978676048
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 121 |
},
|
| 122 |
"twin raw_b16": {
|
| 123 |
"rate": 0.4916666666666667,
|
|
@@ -129,7 +179,12 @@
|
|
| 129 |
"regB_ref": 0.03635521260826646,
|
| 130 |
"regB_star": 0.046204610388273615,
|
| 131 |
"regA_ref": 0.04846464655537219,
|
| 132 |
-
"regA_star": 0.05641944870545168
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 133 |
},
|
| 134 |
"twin filt_b1": {
|
| 135 |
"rate": 0.3416666666666666,
|
|
@@ -141,7 +196,12 @@
|
|
| 141 |
"regB_ref": 0.04659602299391885,
|
| 142 |
"regB_star": 0.053387234536363515,
|
| 143 |
"regA_ref": 0.04659602299391885,
|
| 144 |
-
"regA_star": 0.053387234536363515
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 145 |
},
|
| 146 |
"twin filt_b4": {
|
| 147 |
"rate": 0.4833333333333334,
|
|
@@ -153,7 +213,12 @@
|
|
| 153 |
"regB_ref": 0.04074265857517956,
|
| 154 |
"regB_star": 0.048658098886397605,
|
| 155 |
"regA_ref": 0.046240245344131514,
|
| 156 |
-
"regA_star": 0.056035186950899485
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 157 |
},
|
| 158 |
"twin filt_b16": {
|
| 159 |
"rate": 0.5555555555555556,
|
|
@@ -165,7 +230,12 @@
|
|
| 165 |
"regB_ref": 0.03570862209728387,
|
| 166 |
"regB_star": 0.04591678068291327,
|
| 167 |
"regA_ref": 0.05002032485737595,
|
| 168 |
-
"regA_star": 0.057383941917565906
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 169 |
},
|
| 170 |
"knn raw_b1": {
|
| 171 |
"rate": 0.09166666666666666,
|
|
@@ -175,7 +245,10 @@
|
|
| 175 |
"regB_ref": 0.18029208813957154,
|
| 176 |
"regB_star": 0.18029208813957154,
|
| 177 |
"regA_ref": 0.18029208813957154,
|
| 178 |
-
"regA_star": 0.18029208813957154
|
|
|
|
|
|
|
|
|
|
| 179 |
},
|
| 180 |
"knn raw_b4": {
|
| 181 |
"rate": 0.24166666666666667,
|
|
@@ -185,7 +258,10 @@
|
|
| 185 |
"regB_ref": 0.16093589699487887,
|
| 186 |
"regB_star": 0.17212171903802753,
|
| 187 |
"regA_ref": 0.1725441277384172,
|
| 188 |
-
"regA_star": 0.18523628247042523
|
|
|
|
|
|
|
|
|
|
| 189 |
},
|
| 190 |
"knn raw_b16": {
|
| 191 |
"rate": 0.5166666666666667,
|
|
@@ -195,7 +271,10 @@
|
|
| 195 |
"regB_ref": 0.1327768182319501,
|
| 196 |
"regB_star": 0.14350847619758267,
|
| 197 |
"regA_ref": 0.14856354038966635,
|
| 198 |
-
"regA_star": 0.16313087614552568
|
|
|
|
|
|
|
|
|
|
| 199 |
},
|
| 200 |
"knn filt_b1": {
|
| 201 |
"rate": 0.3,
|
|
@@ -205,7 +284,10 @@
|
|
| 205 |
"regB_ref": 0.17428865396751858,
|
| 206 |
"regB_star": 0.18276418530499838,
|
| 207 |
"regA_ref": 0.17428865396751858,
|
| 208 |
-
"regA_star": 0.18276418530499838
|
|
|
|
|
|
|
|
|
|
| 209 |
},
|
| 210 |
"knn filt_b4": {
|
| 211 |
"rate": 0.6666666666666666,
|
|
@@ -215,7 +297,10 @@
|
|
| 215 |
"regB_ref": 0.12985794400765538,
|
| 216 |
"regB_star": 0.13483477262840815,
|
| 217 |
"regA_ref": 0.1620537009612174,
|
| 218 |
-
"regA_star": 0.16564434998790312
|
|
|
|
|
|
|
|
|
|
| 219 |
},
|
| 220 |
"knn filt_b16": {
|
| 221 |
"rate": 0.725,
|
|
@@ -225,7 +310,10 @@
|
|
| 225 |
"regB_ref": 0.10686761298831438,
|
| 226 |
"regB_star": 0.11477117848557294,
|
| 227 |
"regA_ref": 0.16317150492755594,
|
| 228 |
-
"regA_star": 0.16567695056159626
|
|
|
|
|
|
|
|
|
|
| 229 |
},
|
| 230 |
"knn ref_b16_n3": {
|
| 231 |
"rate": 0.9,
|
|
@@ -235,7 +323,10 @@
|
|
| 235 |
"regB_ref": 0.10400741103859301,
|
| 236 |
"regB_star": 0.1113356454533998,
|
| 237 |
"regA_ref": 0.13930659114267874,
|
| 238 |
-
"regA_star": 0.15300463877630077
|
|
|
|
|
|
|
|
|
|
| 239 |
},
|
| 240 |
"knn ref_b16_n10": {
|
| 241 |
"rate": 0.925,
|
|
@@ -245,6 +336,9 @@
|
|
| 245 |
"regB_ref": 0.10343356710695706,
|
| 246 |
"regB_star": 0.10973883506945237,
|
| 247 |
"regA_ref": 0.13558071904802294,
|
| 248 |
-
"regA_star": 0.14891194803545904
|
|
|
|
|
|
|
|
|
|
| 249 |
}
|
| 250 |
}
|
|
|
|
| 9 |
"regB_ref": 0.054012242469847704,
|
| 10 |
"regB_star": 0.06580516710136042,
|
| 11 |
"regA_ref": 0.054012242469847704,
|
| 12 |
+
"regA_star": 0.06580516710136042,
|
| 13 |
+
"reg29_star_per_seed": [
|
| 14 |
+
1000.0,
|
| 15 |
+
1000.0,
|
| 16 |
+
1000.0
|
| 17 |
+
]
|
| 18 |
},
|
| 19 |
"diff raw_b4": {
|
| 20 |
"rate": 0.49166666666666664,
|
|
|
|
| 26 |
"regB_ref": 0.046037766193027574,
|
| 27 |
"regB_star": 0.055668722380055824,
|
| 28 |
"regA_ref": 0.05123443031809827,
|
| 29 |
+
"regA_star": 0.06242150229911836,
|
| 30 |
+
"reg29_star_per_seed": [
|
| 31 |
+
0.19219722967026764,
|
| 32 |
+
0.10473329948011612,
|
| 33 |
+
1000.0
|
| 34 |
+
]
|
| 35 |
},
|
| 36 |
"diff raw_b16": {
|
| 37 |
"rate": 0.7916666666666666,
|
|
|
|
| 43 |
"regB_ref": 0.0392393348275164,
|
| 44 |
"regB_star": 0.04843269982130982,
|
| 45 |
"regA_ref": 0.053670339354187746,
|
| 46 |
+
"regA_star": 0.0653423576378871,
|
| 47 |
+
"reg29_star_per_seed": [
|
| 48 |
+
0.056011538175283704,
|
| 49 |
+
0.05579918124137806,
|
| 50 |
+
0.0638180447639215
|
| 51 |
+
]
|
| 52 |
},
|
| 53 |
"diff filt_b1": {
|
| 54 |
"rate": 0.4583333333333333,
|
|
|
|
| 60 |
"regB_ref": 0.05194845381582125,
|
| 61 |
"regB_star": 0.06513679765481717,
|
| 62 |
"regA_ref": 0.05194845381582125,
|
| 63 |
+
"regA_star": 0.06513679765481717,
|
| 64 |
+
"reg29_star_per_seed": [
|
| 65 |
+
1000.0,
|
| 66 |
+
1000.0,
|
| 67 |
+
1000.0
|
| 68 |
+
]
|
| 69 |
},
|
| 70 |
"diff filt_b4": {
|
| 71 |
"rate": 0.8138888888888888,
|
|
|
|
| 77 |
"regB_ref": 0.044054119434560945,
|
| 78 |
"regB_star": 0.05117955925012052,
|
| 79 |
"regA_ref": 0.053670339354187746,
|
| 80 |
+
"regA_star": 0.06526846423539756,
|
| 81 |
+
"reg29_star_per_seed": [
|
| 82 |
+
0.057999843375380555,
|
| 83 |
+
0.06110192291621441,
|
| 84 |
+
0.06608581358546517
|
| 85 |
+
]
|
| 86 |
},
|
| 87 |
"diff filt_b16": {
|
| 88 |
"rate": 0.9083333333333333,
|
|
|
|
| 94 |
"regB_ref": 0.033760278025647406,
|
| 95 |
"regB_star": 0.0392393348275164,
|
| 96 |
"regA_ref": 0.05494152341582326,
|
| 97 |
+
"regA_star": 0.06614503852117859,
|
| 98 |
+
"reg29_star_per_seed": [
|
| 99 |
+
0.041296215258291746,
|
| 100 |
+
0.039501202588679285,
|
| 101 |
+
0.04825939970362869
|
| 102 |
+
]
|
| 103 |
},
|
| 104 |
"diff ref_b16_n3": {
|
| 105 |
"rate": 0.9638888888888889,
|
|
|
|
| 111 |
"regB_ref": 0.034359315053998474,
|
| 112 |
"regB_star": 0.039602163709273386,
|
| 113 |
"regA_ref": 0.056236553327202676,
|
| 114 |
+
"regA_star": 0.06713280900832053,
|
| 115 |
+
"reg29_star_per_seed": [
|
| 116 |
+
0.041296215258291746,
|
| 117 |
+
0.03844028224462015,
|
| 118 |
+
0.04449522945084139
|
| 119 |
+
]
|
| 120 |
},
|
| 121 |
"diff ref_b16_n10": {
|
| 122 |
"rate": 0.9805555555555556,
|
|
|
|
| 128 |
"regB_ref": 0.03437394402248328,
|
| 129 |
"regB_star": 0.040009974164940454,
|
| 130 |
"regA_ref": 0.05649656778792678,
|
| 131 |
+
"regA_star": 0.06751706328759521,
|
| 132 |
+
"reg29_star_per_seed": [
|
| 133 |
+
0.041296215258291746,
|
| 134 |
+
0.037411175602180435,
|
| 135 |
+
0.04245451078867202
|
| 136 |
+
]
|
| 137 |
},
|
| 138 |
"twin raw_b1": {
|
| 139 |
"rate": 0.24722222222222223,
|
|
|
|
| 145 |
"regB_ref": 0.04588446769434418,
|
| 146 |
"regB_star": 0.05281368503715744,
|
| 147 |
"regA_ref": 0.04588446769434418,
|
| 148 |
+
"regA_star": 0.05281368503715744,
|
| 149 |
+
"reg29_star_per_seed": [
|
| 150 |
+
1000.0,
|
| 151 |
+
1000.0,
|
| 152 |
+
1000.0
|
| 153 |
+
]
|
| 154 |
},
|
| 155 |
"twin raw_b4": {
|
| 156 |
"rate": 0.37777777777777777,
|
|
|
|
| 162 |
"regB_ref": 0.03958246856186798,
|
| 163 |
"regB_star": 0.046969980502976294,
|
| 164 |
"regA_ref": 0.04453505836815008,
|
| 165 |
+
"regA_star": 0.05310045978676048,
|
| 166 |
+
"reg29_star_per_seed": [
|
| 167 |
+
1000.0,
|
| 168 |
+
1000.0,
|
| 169 |
+
1000.0
|
| 170 |
+
]
|
| 171 |
},
|
| 172 |
"twin raw_b16": {
|
| 173 |
"rate": 0.4916666666666667,
|
|
|
|
| 179 |
"regB_ref": 0.03635521260826646,
|
| 180 |
"regB_star": 0.046204610388273615,
|
| 181 |
"regA_ref": 0.04846464655537219,
|
| 182 |
+
"regA_star": 0.05641944870545168,
|
| 183 |
+
"reg29_star_per_seed": [
|
| 184 |
+
0.10244921887598657,
|
| 185 |
+
1000.0,
|
| 186 |
+
0.11581804604445223
|
| 187 |
+
]
|
| 188 |
},
|
| 189 |
"twin filt_b1": {
|
| 190 |
"rate": 0.3416666666666666,
|
|
|
|
| 196 |
"regB_ref": 0.04659602299391885,
|
| 197 |
"regB_star": 0.053387234536363515,
|
| 198 |
"regA_ref": 0.04659602299391885,
|
| 199 |
+
"regA_star": 0.053387234536363515,
|
| 200 |
+
"reg29_star_per_seed": [
|
| 201 |
+
1000.0,
|
| 202 |
+
1000.0,
|
| 203 |
+
1000.0
|
| 204 |
+
]
|
| 205 |
},
|
| 206 |
"twin filt_b4": {
|
| 207 |
"rate": 0.4833333333333334,
|
|
|
|
| 213 |
"regB_ref": 0.04074265857517956,
|
| 214 |
"regB_star": 0.048658098886397605,
|
| 215 |
"regA_ref": 0.046240245344131514,
|
| 216 |
+
"regA_star": 0.056035186950899485,
|
| 217 |
+
"reg29_star_per_seed": [
|
| 218 |
+
0.09849242387393675,
|
| 219 |
+
1000.0,
|
| 220 |
+
1000.0
|
| 221 |
+
]
|
| 222 |
},
|
| 223 |
"twin filt_b16": {
|
| 224 |
"rate": 0.5555555555555556,
|
|
|
|
| 230 |
"regB_ref": 0.03570862209728387,
|
| 231 |
"regB_star": 0.04591678068291327,
|
| 232 |
"regA_ref": 0.05002032485737595,
|
| 233 |
+
"regA_star": 0.057383941917565906,
|
| 234 |
+
"reg29_star_per_seed": [
|
| 235 |
+
0.085614514090114,
|
| 236 |
+
0.10824800868369489,
|
| 237 |
+
0.0945664020595226
|
| 238 |
+
]
|
| 239 |
},
|
| 240 |
"knn raw_b1": {
|
| 241 |
"rate": 0.09166666666666666,
|
|
|
|
| 245 |
"regB_ref": 0.18029208813957154,
|
| 246 |
"regB_star": 0.18029208813957154,
|
| 247 |
"regA_ref": 0.18029208813957154,
|
| 248 |
+
"regA_star": 0.18029208813957154,
|
| 249 |
+
"reg29_star_per_seed": [
|
| 250 |
+
1000.0
|
| 251 |
+
]
|
| 252 |
},
|
| 253 |
"knn raw_b4": {
|
| 254 |
"rate": 0.24166666666666667,
|
|
|
|
| 258 |
"regB_ref": 0.16093589699487887,
|
| 259 |
"regB_star": 0.17212171903802753,
|
| 260 |
"regA_ref": 0.1725441277384172,
|
| 261 |
+
"regA_star": 0.18523628247042523,
|
| 262 |
+
"reg29_star_per_seed": [
|
| 263 |
+
1000.0
|
| 264 |
+
]
|
| 265 |
},
|
| 266 |
"knn raw_b16": {
|
| 267 |
"rate": 0.5166666666666667,
|
|
|
|
| 271 |
"regB_ref": 0.1327768182319501,
|
| 272 |
"regB_star": 0.14350847619758267,
|
| 273 |
"regA_ref": 0.14856354038966635,
|
| 274 |
+
"regA_star": 0.16313087614552568,
|
| 275 |
+
"reg29_star_per_seed": [
|
| 276 |
+
0.3633192275090701
|
| 277 |
+
]
|
| 278 |
},
|
| 279 |
"knn filt_b1": {
|
| 280 |
"rate": 0.3,
|
|
|
|
| 284 |
"regB_ref": 0.17428865396751858,
|
| 285 |
"regB_star": 0.18276418530499838,
|
| 286 |
"regA_ref": 0.17428865396751858,
|
| 287 |
+
"regA_star": 0.18276418530499838,
|
| 288 |
+
"reg29_star_per_seed": [
|
| 289 |
+
1000.0
|
| 290 |
+
]
|
| 291 |
},
|
| 292 |
"knn filt_b4": {
|
| 293 |
"rate": 0.6666666666666666,
|
|
|
|
| 297 |
"regB_ref": 0.12985794400765538,
|
| 298 |
"regB_star": 0.13483477262840815,
|
| 299 |
"regA_ref": 0.1620537009612174,
|
| 300 |
+
"regA_star": 0.16564434998790312,
|
| 301 |
+
"reg29_star_per_seed": [
|
| 302 |
+
0.17692463714776696
|
| 303 |
+
]
|
| 304 |
},
|
| 305 |
"knn filt_b16": {
|
| 306 |
"rate": 0.725,
|
|
|
|
| 310 |
"regB_ref": 0.10686761298831438,
|
| 311 |
"regB_star": 0.11477117848557294,
|
| 312 |
"regA_ref": 0.16317150492755594,
|
| 313 |
+
"regA_star": 0.16567695056159626,
|
| 314 |
+
"reg29_star_per_seed": [
|
| 315 |
+
0.1354804464718561
|
| 316 |
+
]
|
| 317 |
},
|
| 318 |
"knn ref_b16_n3": {
|
| 319 |
"rate": 0.9,
|
|
|
|
| 323 |
"regB_ref": 0.10400741103859301,
|
| 324 |
"regB_star": 0.1113356454533998,
|
| 325 |
"regA_ref": 0.13930659114267874,
|
| 326 |
+
"regA_star": 0.15300463877630077,
|
| 327 |
+
"reg29_star_per_seed": [
|
| 328 |
+
0.1173305502662011
|
| 329 |
+
]
|
| 330 |
},
|
| 331 |
"knn ref_b16_n10": {
|
| 332 |
"rate": 0.925,
|
|
|
|
| 336 |
"regB_ref": 0.10343356710695706,
|
| 337 |
"regB_star": 0.10973883506945237,
|
| 338 |
"regA_ref": 0.13558071904802294,
|
| 339 |
+
"regA_star": 0.14891194803545904,
|
| 340 |
+
"reg29_star_per_seed": [
|
| 341 |
+
0.11506558365541786
|
| 342 |
+
]
|
| 343 |
}
|
| 344 |
}
|
results/icml_r3/X29/act/x27rescore_d3fm5n16000.txt
CHANGED
|
@@ -1,23 +1,23 @@
|
|
| 1 |
X29-A: X27's C-only arms re-scored, m = 5, tag d3fm5n16000, 120 test plants. J*_A = min(J_ref, 3A id and so3 labels) is below J_ref on 43 plants (median (J_ref - J*_A) / J*_A 0.000%).
|
| 2 |
-
diff raw_b1 certified 23.6% (published 23.6%) | cheapest certified: regret vs J_ref 5.40% (published 5.40%), vs J*_A 6.58% | first certified: vs J_ref 5.40%, vs J*_A 6.58%
|
| 3 |
-
diff raw_b4 certified 49.2% (published 49.2%) | cheapest certified: regret vs J_ref 4.60% (published 4.60%), vs J*_A 5.57% | first certified: vs J_ref 5.12%, vs J*_A 6.24%
|
| 4 |
-
diff raw_b16 certified 79.2% (published 79.2%) | cheapest certified: regret vs J_ref 3.92% (published 3.92%), vs J*_A 4.84% | first certified: vs J_ref 5.37%, vs J*_A 6.53%
|
| 5 |
-
diff filt_b1 certified 45.8% (published 45.8%) | cheapest certified: regret vs J_ref 5.19% (published 5.19%), vs J*_A 6.51% | first certified: vs J_ref 5.19%, vs J*_A 6.51%
|
| 6 |
-
diff filt_b4 certified 81.4% (published 81.4%) | cheapest certified: regret vs J_ref 4.41% (published 4.41%), vs J*_A 5.12% | first certified: vs J_ref 5.37%, vs J*_A 6.53%
|
| 7 |
-
diff filt_b16 certified 90.8% (published 90.8%) | cheapest certified: regret vs J_ref 3.38% (published 3.38%), vs J*_A 3.92% | first certified: vs J_ref 5.49%, vs J*_A 6.61%
|
| 8 |
-
diff ref_b16_n3 certified 96.4% (published 96.4%) | cheapest certified: regret vs J_ref 3.44% (published 3.44%), vs J*_A 3.96% | first certified: vs J_ref 5.62%, vs J*_A 6.71%
|
| 9 |
-
diff ref_b16_n10 certified 98.1% (published 98.1%) | cheapest certified: regret vs J_ref 3.44% (published 3.44%), vs J*_A 4.00% | first certified: vs J_ref 5.65%, vs J*_A 6.75%
|
| 10 |
-
twin raw_b1 certified 24.7% (published 24.7%) | cheapest certified: regret vs J_ref 4.59% (published 4.59%), vs J*_A 5.28% | first certified: vs J_ref 4.59%, vs J*_A 5.28%
|
| 11 |
-
twin raw_b4 certified 37.8% (published 37.8%) | cheapest certified: regret vs J_ref 3.96% (published 3.96%), vs J*_A 4.70% | first certified: vs J_ref 4.45%, vs J*_A 5.31%
|
| 12 |
-
twin raw_b16 certified 49.2% (published 49.2%) | cheapest certified: regret vs J_ref 3.64% (published 3.64%), vs J*_A 4.62% | first certified: vs J_ref 4.85%, vs J*_A 5.64%
|
| 13 |
-
twin filt_b1 certified 34.2% (published 34.2%) | cheapest certified: regret vs J_ref 4.66% (published 4.66%), vs J*_A 5.34% | first certified: vs J_ref 4.66%, vs J*_A 5.34%
|
| 14 |
-
twin filt_b4 certified 48.3% (published 48.3%) | cheapest certified: regret vs J_ref 4.07% (published 4.07%), vs J*_A 4.87% | first certified: vs J_ref 4.62%, vs J*_A 5.60%
|
| 15 |
-
twin filt_b16 certified 55.6% (published 55.6%) | cheapest certified: regret vs J_ref 3.57% (published 3.57%), vs J*_A 4.59% | first certified: vs J_ref 5.00%, vs J*_A 5.74%
|
| 16 |
-
knn raw_b1 certified 9.2% (published 9.2%) | cheapest certified: regret vs J_ref 18.03% (published 18.03%), vs J*_A 18.03% | first certified: vs J_ref 18.03%, vs J*_A 18.03%
|
| 17 |
-
knn raw_b4 certified 24.2% (published 24.2%) | cheapest certified: regret vs J_ref 16.09% (published 16.09%), vs J*_A 17.21% | first certified: vs J_ref 17.25%, vs J*_A 18.52%
|
| 18 |
-
knn raw_b16 certified 51.7% (published 51.7%) | cheapest certified: regret vs J_ref 13.28% (published 13.28%), vs J*_A 14.35% | first certified: vs J_ref 14.86%, vs J*_A 16.31%
|
| 19 |
-
knn filt_b1 certified 30.0% (published 30.0%) | cheapest certified: regret vs J_ref 17.43% (published 17.43%), vs J*_A 18.28% | first certified: vs J_ref 17.43%, vs J*_A 18.28%
|
| 20 |
-
knn filt_b4 certified 66.7% (published 66.7%) | cheapest certified: regret vs J_ref 12.99% (published 12.99%), vs J*_A 13.48% | first certified: vs J_ref 16.21%, vs J*_A 16.56%
|
| 21 |
-
knn filt_b16 certified 72.5% (published 72.5%) | cheapest certified: regret vs J_ref 10.69% (published 10.69%), vs J*_A 11.48% | first certified: vs J_ref 16.32%, vs J*_A 16.57%
|
| 22 |
-
knn ref_b16_n3 certified 90.0% (published 90.0%) | cheapest certified: regret vs J_ref 10.40% (published 10.40%), vs J*_A 11.13% | first certified: vs J_ref 13.93%, vs J*_A 15.30%
|
| 23 |
-
knn ref_b16_n10 certified 92.5% (published 92.5%) | cheapest certified: regret vs J_ref 10.34% (published 10.34%), vs J*_A 10.97% | first certified: vs J_ref 13.56%, vs J*_A 14.89%
|
|
|
|
| 1 |
X29-A: X27's C-only arms re-scored, m = 5, tag d3fm5n16000, 120 test plants. J*_A = min(J_ref, 3A id and so3 labels) is below J_ref on 43 plants (median (J_ref - J*_A) / J*_A 0.000%).
|
| 2 |
+
diff raw_b1 certified 23.6% (published 23.6%) | cheapest certified: regret vs J_ref 5.40% (published 5.40%), vs J*_A 6.58% | first certified: vs J_ref 5.40%, vs J*_A 6.58% | X29 convention vs J*_A, per seed: n/a / n/a / n/a
|
| 3 |
+
diff raw_b4 certified 49.2% (published 49.2%) | cheapest certified: regret vs J_ref 4.60% (published 4.60%), vs J*_A 5.57% | first certified: vs J_ref 5.12%, vs J*_A 6.24% | X29 convention vs J*_A, per seed: 19.22% / 10.47% / n/a
|
| 4 |
+
diff raw_b16 certified 79.2% (published 79.2%) | cheapest certified: regret vs J_ref 3.92% (published 3.92%), vs J*_A 4.84% | first certified: vs J_ref 5.37%, vs J*_A 6.53% | X29 convention vs J*_A, per seed: 5.60% / 5.58% / 6.38%
|
| 5 |
+
diff filt_b1 certified 45.8% (published 45.8%) | cheapest certified: regret vs J_ref 5.19% (published 5.19%), vs J*_A 6.51% | first certified: vs J_ref 5.19%, vs J*_A 6.51% | X29 convention vs J*_A, per seed: n/a / n/a / n/a
|
| 6 |
+
diff filt_b4 certified 81.4% (published 81.4%) | cheapest certified: regret vs J_ref 4.41% (published 4.41%), vs J*_A 5.12% | first certified: vs J_ref 5.37%, vs J*_A 6.53% | X29 convention vs J*_A, per seed: 5.80% / 6.11% / 6.61%
|
| 7 |
+
diff filt_b16 certified 90.8% (published 90.8%) | cheapest certified: regret vs J_ref 3.38% (published 3.38%), vs J*_A 3.92% | first certified: vs J_ref 5.49%, vs J*_A 6.61% | X29 convention vs J*_A, per seed: 4.13% / 3.95% / 4.83%
|
| 8 |
+
diff ref_b16_n3 certified 96.4% (published 96.4%) | cheapest certified: regret vs J_ref 3.44% (published 3.44%), vs J*_A 3.96% | first certified: vs J_ref 5.62%, vs J*_A 6.71% | X29 convention vs J*_A, per seed: 4.13% / 3.84% / 4.45%
|
| 9 |
+
diff ref_b16_n10 certified 98.1% (published 98.1%) | cheapest certified: regret vs J_ref 3.44% (published 3.44%), vs J*_A 4.00% | first certified: vs J_ref 5.65%, vs J*_A 6.75% | X29 convention vs J*_A, per seed: 4.13% / 3.74% / 4.25%
|
| 10 |
+
twin raw_b1 certified 24.7% (published 24.7%) | cheapest certified: regret vs J_ref 4.59% (published 4.59%), vs J*_A 5.28% | first certified: vs J_ref 4.59%, vs J*_A 5.28% | X29 convention vs J*_A, per seed: n/a / n/a / n/a
|
| 11 |
+
twin raw_b4 certified 37.8% (published 37.8%) | cheapest certified: regret vs J_ref 3.96% (published 3.96%), vs J*_A 4.70% | first certified: vs J_ref 4.45%, vs J*_A 5.31% | X29 convention vs J*_A, per seed: n/a / n/a / n/a
|
| 12 |
+
twin raw_b16 certified 49.2% (published 49.2%) | cheapest certified: regret vs J_ref 3.64% (published 3.64%), vs J*_A 4.62% | first certified: vs J_ref 4.85%, vs J*_A 5.64% | X29 convention vs J*_A, per seed: 10.24% / n/a / 11.58%
|
| 13 |
+
twin filt_b1 certified 34.2% (published 34.2%) | cheapest certified: regret vs J_ref 4.66% (published 4.66%), vs J*_A 5.34% | first certified: vs J_ref 4.66%, vs J*_A 5.34% | X29 convention vs J*_A, per seed: n/a / n/a / n/a
|
| 14 |
+
twin filt_b4 certified 48.3% (published 48.3%) | cheapest certified: regret vs J_ref 4.07% (published 4.07%), vs J*_A 4.87% | first certified: vs J_ref 4.62%, vs J*_A 5.60% | X29 convention vs J*_A, per seed: 9.85% / n/a / n/a
|
| 15 |
+
twin filt_b16 certified 55.6% (published 55.6%) | cheapest certified: regret vs J_ref 3.57% (published 3.57%), vs J*_A 4.59% | first certified: vs J_ref 5.00%, vs J*_A 5.74% | X29 convention vs J*_A, per seed: 8.56% / 10.82% / 9.46%
|
| 16 |
+
knn raw_b1 certified 9.2% (published 9.2%) | cheapest certified: regret vs J_ref 18.03% (published 18.03%), vs J*_A 18.03% | first certified: vs J_ref 18.03%, vs J*_A 18.03% | X29 convention vs J*_A, per seed: n/a
|
| 17 |
+
knn raw_b4 certified 24.2% (published 24.2%) | cheapest certified: regret vs J_ref 16.09% (published 16.09%), vs J*_A 17.21% | first certified: vs J_ref 17.25%, vs J*_A 18.52% | X29 convention vs J*_A, per seed: n/a
|
| 18 |
+
knn raw_b16 certified 51.7% (published 51.7%) | cheapest certified: regret vs J_ref 13.28% (published 13.28%), vs J*_A 14.35% | first certified: vs J_ref 14.86%, vs J*_A 16.31% | X29 convention vs J*_A, per seed: 36.33%
|
| 19 |
+
knn filt_b1 certified 30.0% (published 30.0%) | cheapest certified: regret vs J_ref 17.43% (published 17.43%), vs J*_A 18.28% | first certified: vs J_ref 17.43%, vs J*_A 18.28% | X29 convention vs J*_A, per seed: n/a
|
| 20 |
+
knn filt_b4 certified 66.7% (published 66.7%) | cheapest certified: regret vs J_ref 12.99% (published 12.99%), vs J*_A 13.48% | first certified: vs J_ref 16.21%, vs J*_A 16.56% | X29 convention vs J*_A, per seed: 17.69%
|
| 21 |
+
knn filt_b16 certified 72.5% (published 72.5%) | cheapest certified: regret vs J_ref 10.69% (published 10.69%), vs J*_A 11.48% | first certified: vs J_ref 16.32%, vs J*_A 16.57% | X29 convention vs J*_A, per seed: 13.55%
|
| 22 |
+
knn ref_b16_n3 certified 90.0% (published 90.0%) | cheapest certified: regret vs J_ref 10.40% (published 10.40%), vs J*_A 11.13% | first certified: vs J_ref 13.93%, vs J*_A 15.30% | X29 convention vs J*_A, per seed: 11.73%
|
| 23 |
+
knn ref_b16_n10 certified 92.5% (published 92.5%) | cheapest certified: regret vs J_ref 10.34% (published 10.34%), vs J*_A 10.97% | first certified: vs J_ref 13.56%, vs J*_A 14.89% | X29 convention vs J*_A, per seed: 11.51%
|
results/icml_r3/X29/siso/report.txt
CHANGED
|
@@ -78,4 +78,25 @@ uncertified = 1e3; certified % / median regret [bootstrap 95% CI]; clip 1.5, t-r
|
|
| 78 |
equivalence of the batched optimizers: C+phi 25 steps, plants [0, 1, 2]: J batched [0.089508 0.472122 0.018839] single [0.089508 0.472122 0.018839] used [1 1 0] [1 1 0] max rel diff 0.00e+00 -> ok
|
| 79 |
|
| 80 |
== CPU (timing job)
|
| 81 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 78 |
equivalence of the batched optimizers: C+phi 25 steps, plants [0, 1, 2]: J batched [0.089508 0.472122 0.018839] single [0.089508 0.472122 0.018839] used [1 1 0] [1 1 0] max rel diff 0.00e+00 -> ok
|
| 79 |
|
| 80 |
== CPU (timing job)
|
| 81 |
+
medians over timing plants (one thread each): fopt1 1553.8 s (n=10), fopt16 1686.5 s (n=10), fopt4 1518.1 s (n=10), gen_C+phi 29.1 s (n=10), gen_C-only 11.5 s (n=10), hifoo2 1.0 s (n=10), jopt1 1537.3 s (n=10), jopt16 1749.6 s (n=10), jopt4 1545.7 s (n=10), knn_C+phi 7.6 s (n=10), knn_C-only 9.5 s (n=10), twin_C+phi 17.2 s (n=10), twin_C-only 1.9 s (n=10)
|
| 82 |
+
C-only generator b=16 CPU 11.5 s | median regret 6.79%
|
| 83 |
+
C+phi generator b=16 CPU 29.1 s | median regret 7.14%
|
| 84 |
+
C-only generator b=1 CPU 0.7 s | median regret 54.90%
|
| 85 |
+
C+phi generator b=1 CPU 1.8 s | median regret 100000.00%
|
| 86 |
+
F-only optimizer 1 starts CPU 1554.8 s | median regret 35.91%
|
| 87 |
+
joint optimizer 1 starts CPU 3092.1 s | median regret 9.13%
|
| 88 |
+
F-only optimizer 2 starts CPU 1543.8 s | median regret 13.55%
|
| 89 |
+
joint optimizer 2 starts CPU 3083.9 s | median regret 4.39%
|
| 90 |
+
F-only optimizer 4 starts CPU 1522.0 s | median regret 8.11%
|
| 91 |
+
joint optimizer 4 starts CPU 3067.7 s | median regret 0.92%
|
| 92 |
+
F-only optimizer 8 starts CPU 1581.9 s | median regret 4.19%
|
| 93 |
+
joint optimizer 8 starts CPU 3195.6 s | median regret 0.00%
|
| 94 |
+
F-only optimizer 16 starts CPU 1701.8 s | median regret 3.12%
|
| 95 |
+
joint optimizer 16 starts CPU 3451.4 s | median regret 0.00%
|
| 96 |
+
fixed-order stabilization (HIFOO best of 16) CPU 15.3 s | median regret 50.75%
|
| 97 |
+
C-only generator 16 + 1 optimizer start CPU 1565.3 s | median regret 2.60%
|
| 98 |
+
C+phi generator 16 + 1 optimizer start CPU 1566.4 s | median regret 0.01%
|
| 99 |
+
[38] C-only: F-only optimizer reaches the generator's 6.79% at 8 starts, 1582 s against 11.5 s: ratio 138x
|
| 100 |
+
[38] C+phi: joint optimizer reaches the generator's 7.14% at 2 starts, 3084 s against 29.1 s: ratio 106x
|
| 101 |
+
C-only kNN (retrieval + scoring, 16) 9.5 s; twin (sampling + scoring, 16) 1.9 s
|
| 102 |
+
C+phi kNN (retrieval + scoring, 16) 7.6 s; twin (sampling + scoring, 16) 17.2 s
|
results/icml_r3/X29/siso/timing/plant_000_fopt1.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 0, "method": "fopt1", "t": 1610.2393736839294, "k": 1, "load": [6.0107421875, 6.0, 5.39453125]}
|
results/icml_r3/X29/siso/timing/plant_000_fopt16.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 0, "method": "fopt16", "t": 1693.0486862659454, "k": 16, "load": [6.0, 6.0, 6.0]}
|
results/icml_r3/X29/siso/timing/plant_000_fopt4.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 0, "method": "fopt4", "t": 1529.6605348587036, "k": 4, "load": [6.0, 6.0, 5.9189453125]}
|
results/icml_r3/X29/siso/timing/plant_000_gen_C+phi.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 0, "method": "gen_C+phi", "t_sample": 8.641952991485596, "t_score": 20.20849061012268, "t": 28.850443601608276, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
|
results/icml_r3/X29/siso/timing/plant_000_gen_C-only.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 0, "method": "gen_C-only", "t_sample": 8.723515272140503, "t_score": 1.421351671218872, "t": 10.144866943359375, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
|
results/icml_r3/X29/siso/timing/plant_000_hifoo2.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 0, "method": "hifoo2", "t": 7.0185874700546265, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
|
results/icml_r3/X29/siso/timing/plant_000_jopt1.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 0, "method": "jopt1", "t": 1479.6743664741516, "k": 1, "load": [6.0, 6.0, 6.0]}
|
results/icml_r3/X29/siso/timing/plant_000_jopt16.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 0, "method": "jopt16", "t": 1720.920538187027, "k": 16, "load": [6.0, 6.0, 6.0]}
|
results/icml_r3/X29/siso/timing/plant_000_jopt4.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 0, "method": "jopt4", "t": 1511.4163365364075, "k": 4, "load": [6.0, 6.0, 6.0]}
|
results/icml_r3/X29/siso/timing/plant_000_knn_C+phi.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 0, "method": "knn_C+phi", "t_sample": 0.005944490432739258, "t_score": 7.443649768829346, "t": 7.449594259262085, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
|
results/icml_r3/X29/siso/timing/plant_000_knn_C-only.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 0, "method": "knn_C-only", "t_sample": 2.10442852973938, "t_score": 1.0252888202667236, "t": 3.1297173500061035, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
|
results/icml_r3/X29/siso/timing/plant_000_twin_C+phi.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 0, "method": "twin_C+phi", "t_sample": 0.23180317878723145, "t_score": 0.18750786781311035, "t": 0.4193110466003418, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
|
results/icml_r3/X29/siso/timing/plant_000_twin_C-only.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 0, "method": "twin_C-only", "t_sample": 0.25113391876220703, "t_score": 0.08510637283325195, "t": 0.336240291595459, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
|
results/icml_r3/X29/siso/timing/plant_021_fopt1.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 21, "method": "fopt1", "t": 1603.8419466018677, "k": 1, "load": [6.0, 6.0, 5.9150390625]}
|
results/icml_r3/X29/siso/timing/plant_021_fopt16.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 21, "method": "fopt16", "t": 1609.1883926391602, "k": 16, "load": [6.0, 6.0, 6.0]}
|
results/icml_r3/X29/siso/timing/plant_021_fopt4.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 21, "method": "fopt4", "t": 1512.834814786911, "k": 4, "load": [6.0, 6.0, 6.0]}
|
results/icml_r3/X29/siso/timing/plant_021_gen_C+phi.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 21, "method": "gen_C+phi", "t_sample": 8.728589296340942, "t_score": 21.289788722991943, "t": 30.018378019332886, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
|
results/icml_r3/X29/siso/timing/plant_021_gen_C-only.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 21, "method": "gen_C-only", "t_sample": 8.670607328414917, "t_score": 3.4282965660095215, "t": 12.098903894424438, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
|
results/icml_r3/X29/siso/timing/plant_021_hifoo2.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 21, "method": "hifoo2", "t": 0.7439807653427124, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
|
results/icml_r3/X29/siso/timing/plant_021_jopt1.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 21, "method": "jopt1", "t": 1616.4610531330109, "k": 1, "load": [6.0, 6.0, 6.0]}
|
results/icml_r3/X29/siso/timing/plant_021_jopt16.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 21, "method": "jopt16", "t": 1778.183555841446, "k": 16, "load": [6.0, 6.0, 6.0]}
|
results/icml_r3/X29/siso/timing/plant_021_jopt4.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 21, "method": "jopt4", "t": 1580.9241681098938, "k": 4, "load": [6.0, 6.0, 6.0]}
|
results/icml_r3/X29/siso/timing/plant_021_knn_C+phi.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 21, "method": "knn_C+phi", "t_sample": 0.005930423736572266, "t_score": 0.9871442317962646, "t": 0.9930746555328369, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
|
results/icml_r3/X29/siso/timing/plant_021_knn_C-only.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 21, "method": "knn_C-only", "t_sample": 2.1155502796173096, "t_score": 1.1700952053070068, "t": 3.2856454849243164, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
|
results/icml_r3/X29/siso/timing/plant_021_twin_C+phi.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 21, "method": "twin_C+phi", "t_sample": 0.2300739288330078, "t_score": 0.22470903396606445, "t": 0.45478296279907227, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
|
results/icml_r3/X29/siso/timing/plant_021_twin_C-only.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 21, "method": "twin_C-only", "t_sample": 0.22515177726745605, "t_score": 0.26130175590515137, "t": 0.4864535331726074, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
|
results/icml_r3/X29/siso/timing/plant_041_fopt1.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 41, "method": "fopt1", "t": 1533.378296136856, "k": 1, "load": [6.0, 6.0, 5.9150390625]}
|
results/icml_r3/X29/siso/timing/plant_041_fopt16.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 41, "method": "fopt16", "t": 1787.9273924827576, "k": 16, "load": [6.0, 6.0, 6.0]}
|
results/icml_r3/X29/siso/timing/plant_041_fopt4.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 41, "method": "fopt4", "t": 1572.8095934391022, "k": 4, "load": [6.0, 6.0, 6.0]}
|
results/icml_r3/X29/siso/timing/plant_041_gen_C+phi.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 41, "method": "gen_C+phi", "t_sample": 8.899398565292358, "t_score": 1.698326826095581, "t": 10.59772539138794, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
|
results/icml_r3/X29/siso/timing/plant_041_gen_C-only.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 41, "method": "gen_C-only", "t_sample": 8.66327452659607, "t_score": 1.2682888507843018, "t": 9.931563377380371, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
|
results/icml_r3/X29/siso/timing/plant_041_hifoo2.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 41, "method": "hifoo2", "t": 1.9034154415130615, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
|
results/icml_r3/X29/siso/timing/plant_041_jopt1.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 41, "method": "jopt1", "t": 1552.531121969223, "k": 1, "load": [6.0, 6.0, 6.0]}
|
results/icml_r3/X29/siso/timing/plant_041_jopt16.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 41, "method": "jopt16", "t": 1799.8491704463959, "k": 16, "load": [3.51025390625, 5.095703125, 5.66943359375]}
|
results/icml_r3/X29/siso/timing/plant_041_jopt4.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 41, "method": "jopt4", "t": 1499.8948347568512, "k": 4, "load": [6.0, 6.0, 6.0]}
|
results/icml_r3/X29/siso/timing/plant_041_knn_C+phi.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 41, "method": "knn_C+phi", "t_sample": 0.005980730056762695, "t_score": 0.11679935455322266, "t": 0.12278008460998535, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
|
results/icml_r3/X29/siso/timing/plant_041_knn_C-only.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 41, "method": "knn_C-only", "t_sample": 2.1139070987701416, "t_score": 0.5942299365997314, "t": 2.708137035369873, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
|
results/icml_r3/X29/siso/timing/plant_041_twin_C+phi.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 41, "method": "twin_C+phi", "t_sample": 0.20940399169921875, "t_score": 1.5725376605987549, "t": 1.7819416522979736, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
|
results/icml_r3/X29/siso/timing/plant_041_twin_C-only.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 41, "method": "twin_C-only", "t_sample": 0.1909935474395752, "t_score": 8.11414384841919, "t": 8.305137395858765, "load": [1.01220703125, 2.14892578125, 4.05224609375]}
|
results/icml_r3/X29/siso/timing/plant_061_fopt1.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 61, "method": "fopt1", "t": 1546.6886041164398, "k": 1, "load": [6.0, 6.0, 5.9150390625]}
|
results/icml_r3/X29/siso/timing/plant_061_fopt16.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 61, "method": "fopt16", "t": 1680.0087144374847, "k": 16, "load": [6.0, 6.0, 6.0]}
|
results/icml_r3/X29/siso/timing/plant_061_fopt4.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"i": 61, "method": "fopt4", "t": 1523.4473977088928, "k": 4, "load": [6.0, 6.0, 6.0]}
|