results/icml_r3/X8
Browse files- results/icml_r3/X8/FROZEN.json +25 -0
- results/icml_r3/X8/PREREG.md +38 -0
- results/icml_r3/X8/cands/val_unetA0_ddim10_s0_k16.npy +3 -0
- results/icml_r3/X8/cands/val_x2_reg_mcd_seed0_div_s0.npy +3 -0
- results/icml_r3/X8/cands/val_x2_reg_mcd_seed1_div_s0.npy +3 -0
- results/icml_r3/X8/cands/val_x2_reg_mcd_seed2_div_s0.npy +3 -0
- results/icml_r3/X8/curves.json +1 -0
- results/icml_r3/X8/logs/report.log +154 -0
- results/icml_r3/X8/logs/test_main.log +88 -0
- results/icml_r3/X8/logs/test_seeds.log +117 -0
- results/icml_r3/X8/logs/val.log +181 -0
- results/icml_r3/X8/tables/test_gen0.npz +3 -0
- results/icml_r3/X8/tables/test_gen0_ref_it80_lr0.1_n200.npz +3 -0
- results/icml_r3/X8/tables/test_gen1.npz +3 -0
- results/icml_r3/X8/tables/test_gen1_ref_it80_lr0.1_n200.npz +3 -0
- results/icml_r3/X8/tables/test_gen2.npz +3 -0
- results/icml_r3/X8/tables/test_gen2_ref_it80_lr0.1_n200.npz +3 -0
- results/icml_r3/X8/tables/test_gen3.npz +3 -0
- results/icml_r3/X8/tables/test_gen3_ref_it80_lr0.1_n200.npz +3 -0
- results/icml_r3/X8/tables/test_gen4.npz +3 -0
- results/icml_r3/X8/tables/test_gen4_ref_it80_lr0.1_n200.npz +3 -0
- results/icml_r3/X8/tables/test_retr.npz +3 -0
- results/icml_r3/X8/tables/test_retr_ref_it80_lr0.1_n200.npz +3 -0
- results/icml_r3/X8/tables/test_twin.npz +3 -0
- results/icml_r3/X8/tables/test_twin_ref_it80_lr0.1_n200.npz +3 -0
- results/icml_r3/X8/tables/val_gen0.npz +3 -0
- results/icml_r3/X8/tables/val_gen0_ref_it40_lr0.1_n200.npz +3 -0
- results/icml_r3/X8/tables/val_gen0_ref_it40_lr0.1_n400.npz +3 -0
- results/icml_r3/X8/tables/val_gen0_ref_it40_lr0.2_n200.npz +3 -0
- results/icml_r3/X8/tables/val_gen0_ref_it80_lr0.1_n200.npz +3 -0
- results/icml_r3/X8/tables/val_gen0_ref_it80_lr0.2_n200.npz +3 -0
- results/icml_r3/X8/tables/val_retr.npz +3 -0
- results/icml_r3/X8/tables/val_retr_ref_it40_lr0.1_n200.npz +3 -0
- results/icml_r3/X8/tables/val_retr_ref_it40_lr0.1_n400.npz +3 -0
- results/icml_r3/X8/tables/val_retr_ref_it40_lr0.2_n200.npz +3 -0
- results/icml_r3/X8/tables/val_retr_ref_it80_lr0.1_n200.npz +3 -0
- results/icml_r3/X8/tables/val_retr_ref_it80_lr0.2_n200.npz +3 -0
- results/icml_r3/X8/tables/val_twin.npz +3 -0
- results/icml_r3/X8/tables/val_twin_ref_it40_lr0.1_n200.npz +3 -0
- results/icml_r3/X8/tables/val_twin_ref_it40_lr0.1_n400.npz +3 -0
- results/icml_r3/X8/tables/val_twin_ref_it40_lr0.2_n200.npz +3 -0
- results/icml_r3/X8/tables/val_twin_ref_it80_lr0.1_n200.npz +3 -0
- results/icml_r3/X8/tables/val_twin_ref_it80_lr0.2_n200.npz +3 -0
- results/icml_r3/X8/test_report.txt +154 -0
- results/icml_r3/X8/val_selection.txt +53 -0
results/icml_r3/X8/FROZEN.json
ADDED
|
@@ -0,0 +1,25 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"written": "2026-10-13 10:52 PDT",
|
| 3 |
+
"refine_cfg": {
|
| 4 |
+
"iters": 80,
|
| 5 |
+
"lr": 0.1,
|
| 6 |
+
"n": 200
|
| 7 |
+
},
|
| 8 |
+
"pipeline": "rank_refine",
|
| 9 |
+
"learned_metric": "xfer",
|
| 10 |
+
"val_rescue": {
|
| 11 |
+
"it40_lr0.1_n200": 0.9588477366255144,
|
| 12 |
+
"it80_lr0.1_n200": 0.9789666209419295,
|
| 13 |
+
"it40_lr0.2_n200": 0.9780521262002744,
|
| 14 |
+
"it80_lr0.2_n200": 0.9762231367169639,
|
| 15 |
+
"it40_lr0.1_n400": 0.9588477366255144
|
| 16 |
+
},
|
| 17 |
+
"val_pipe_score": {
|
| 18 |
+
"plain": 74.38271604938271,
|
| 19 |
+
"refine": 83.39506172839505,
|
| 20 |
+
"rank": 92.53086419753086,
|
| 21 |
+
"rank_refine": 96.29629629629629,
|
| 22 |
+
"refine_free": 100.0,
|
| 23 |
+
"rank_refine_free": 100.0
|
| 24 |
+
}
|
| 25 |
+
}
|
results/icml_r3/X8/PREREG.md
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# X8 PREREG: equal repair on the corrected arm (updates inventory row D7) (written 2026-10-13 09:06 PDT, before any validation or test scoring of this experiment)
|
| 2 |
+
|
| 3 |
+
virgo CPU only (CUDA_VISIBLE_DEVICES empty), <= 12 workers, RAYON_NUM_THREADS=1. Code: `nonlinear/experiments/furuta_x8.py`, `furuta_r3b_engine.py`, `furuta_r3b_arms.py`, `furuta_r3b_refine.py` (the v4 refinement re-pointed at the corrected arm; the torch RK4 matches the verifier's CasADi step to 7e-15, `python -m nonlinear.experiments.furuta_r3b_refine`).
|
| 4 |
+
Protocol = X2 (same plants val 51001 x60, test 71001+71002 x300, same verifier, bank `data/x2_bank3000.npz` = 2905 rows).
|
| 5 |
+
|
| 6 |
+
## Question
|
| 7 |
+
D7 (old arm, n = 120): with the SAME local refinement and ranking given to the generator and to retrieval the margin shrinks to <= 4 points. Does that hold on the corrected arm at n = 300 with refinement verifier calls COUNTED toward the budget, and for the regression twin as well?
|
| 8 |
+
|
| 9 |
+
## Arms and the repair procedure (identical for every arm)
|
| 10 |
+
- Each arm provides 16 ordered candidates per plant: **gen** = `unet_a` seed 0 (S1 config s96000 ema999, DDIM-10), the first 16 samples of each of the 3 cached X2 sampler seeds (3 replicas);
|
| 11 |
+
**retr_unit** = the 16 nearest bank rows in the X2 metric (deterministic); **retr_learned** = the 16 nearest rows under the learned metric of X10F (the metric, ard or xfer, with the higher mean X10F validation kNN rate over b = 1, 4, 16; ties -> ard);
|
| 12 |
+
**twin** = for twin member i in {0, 1, 2} and draw seed s in {0, 1, 2}: [deterministic prediction, 15 MC-dropout draws (jitter 0)] (9 replicas, the cached X2 draws).
|
| 13 |
+
- Verifier call = one verification of one candidate (`certified_catch_set` re-solving P for the candidate's K on the target plant, then `score_solution`, 8 s, RK4 2 ms, 10 V clip). A candidate that is certified but fails is **refined** once: 40 Adam steps (config frozen on validation) on the 101 voltage knots of u* through the differentiable open-loop simulator of the TARGET plant (corrected arm), objective soft-min over time of log(1 + x_w'Px_w) with the re-solved P; K, T, P unchanged; then verified again with the unchanged verifier. That re-verification **counts as one more verifier call**.
|
| 14 |
+
- Pipelines (success at budget b = a success within the first b verifier calls, b = 1..16): `plain` (given order); `refine` (given order, refinement, counted); `rank` (order by the open-loop depth min_t x_w'Px_w of each candidate's reference trajectory under the candidate's OWN P; no verifier call); `rank_refine` (rank, then refine, counted).
|
| 15 |
+
Secondary D7-style accounting `*_free` (the refined re-verification not counted) is reported as the upper bound of what repair can give. The refinement compute is not charged to b (it is not a verifier call); its wall-clock is reported descriptively.
|
| 16 |
+
|
| 17 |
+
## Selection on validation only (FROZEN.json written before any test scoring)
|
| 18 |
+
Validation arms: gen seed 0 sampler seed 0 (16 fresh samples per val plant, cached in X8/cands), retr_unit, retr_learned, twin members 0-2 draw seed 0.
|
| 19 |
+
1. Refinement configuration from {(iters, lr, n_substeps)} = {(40, 0.1, 200), (80, 0.1, 200), (40, 0.2, 200), (80, 0.2, 200), (40, 0.1, 400)}: the one with the highest pooled rescue rate (rescued / certified-failed candidates, all arms pooled); ties -> earlier in the list (cheaper).
|
| 20 |
+
2. Primary pipeline from {refine, rank_refine}: the one with the higher mean over the baseline arms (retr_unit, retr_learned, twin) of the success averaged over b = 1, 4, 16 (the pipeline that is strongest for the baselines); ties -> rank_refine.
|
| 21 |
+
All four counted/uncounted pipelines are reported on test regardless; the primary family below uses the frozen pipeline only.
|
| 22 |
+
|
| 23 |
+
## Test endpoints and tests
|
| 24 |
+
- Primary family, **9 tests**, Holm at alpha 0.05: (generator minus retr_unit, minus retr_learned, minus twin) x b in {1, 4, 16} under the frozen pipeline. Generator per plant = mean over its 3 sampler seeds; twin = mean over its 9 replicas; retrievals deterministic.
|
| 25 |
+
Statistic: mean paired per-plant difference (pp), percentile bootstrap CI over plants (10,000), two-sided sign-flip p (20,000). Holm is applied inside this family only.
|
| 26 |
+
- Secondary (unadjusted): the same differences for all pipelines and for b = 1..16 (budget curves); gen minus each arm under `plain` (reproduces X2 P1-P5 as a check); the D7 comparison (`*_free` at K = 1, 4); across training seeds: generator seeds 1-4 (X2 cached samples, 3 sampler seeds) under the frozen pipeline, difference to retr_unit / retr_learned / twin with the TRAINING SEED as the resampling unit (mean, t-interval over 5 seeds, seeds positive); no pooled-over-seeds McNemar.
|
| 27 |
+
- Falsification: the claim "the generator's lead survives equal repair" is refuted for a cell if its CI contains 0 (b = 1 and 4 matter for the low-budget claim; b = 16 for the ceiling claim). If retrieval or the twin catches up under repair the result is reported as the corrected-arm update of D7 and the claim is narrowed accordingly.
|
| 28 |
+
- Not covered: arms other than those listed (flow, MDN, WTA, cVAE are X1F on the old arm); repair beyond one refinement per candidate; compute-matched (wall-clock) accounting.
|
| 29 |
+
|
| 30 |
+
## Addendum 2026-10-13 10:53 PDT: validation done, configuration and pipeline frozen (no test scoring before this line)
|
| 31 |
+
`furuta_x8 val` (log results/icml_r3/X8/logs/val.log, val_selection.txt) finished at 10:52 and wrote FROZEN.json: refinement config (iters 80, lr 0.1, n 200), pooled validation rescue 97.9% (2141 of 2187 certified-failed candidates; the five configs span 95.9 to 97.9%), primary pipeline `rank_refine` (mean baseline-arm validation success over b = 1, 4, 16: 96.30 against 83.40 for `refine`), learned metric of retr_learned = `xfer` (X10F validation mean over b = 1, 4, 16: 66.7 against 65.6 for `ard`).
|
| 32 |
+
Validation also shows that with counted refinement every arm reaches 100% at b = 2 (`refine`) or b >= 2 (`rank_refine`) on the 60 validation plants; the budget cells b = 4 and b = 16 of the primary family are therefore expected to be saturated (difference 0, p = 1). A descriptive extra (not in the primary family) is added to the report: mean number of verifier calls to the first success (a failure counts as 17), per arm and pipeline, because it still discriminates when the success rates saturate.
|
| 33 |
+
|
| 34 |
+
## Addendum 2026-10-13 11:15 PDT (compute only, before any test result is read)
|
| 35 |
+
To fit the time budget the across-seed secondary (generator training seeds 1-4) uses ONE sampler seed (0) per seed instead of three. Seed 0 keeps its three sampler seeds. The seed is still the resampling unit. The primary family is unchanged.
|
| 36 |
+
|
| 37 |
+
## Addendum 2026-10-13 11:30 PDT (secondary sensitivity, added before the twin test tables existed and before any test statistic was computed; the generator and retrieval tables had been built, one unadjusted preview of their success rates was printed)
|
| 38 |
+
Refinement is not free in CPU time: on the test candidates a refinement batch costs 1.3 to 1.4 CPU-s per candidate (80 Adam steps, batches of 24, loaded machine) against 0.49 CPU-s for a plain verifier call (mean over the test run). Two extra pipelines `refine_c3` and `rank_refine_c3` charge one refinement attempt (refinement plus re-verification) 3 verifier calls, a compute-matched sensitivity. They are secondary and unadjusted; the primary family and its pipeline (`rank_refine`, one call per attempt) are unchanged.
|
results/icml_r3/X8/cands/val_unetA0_ddim10_s0_k16.npy
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:651d9fb81fe76d54518582292d3bdadd8cf6446496db9cc173b5af89eac59052
|
| 3 |
+
size 445568
|
results/icml_r3/X8/cands/val_x2_reg_mcd_seed0_div_s0.npy
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d0debbbcfb6e6f1d97ab5c41bcd037ff1e46e7dffcbe0953552a4c48e94b8880
|
| 3 |
+
size 445568
|
results/icml_r3/X8/cands/val_x2_reg_mcd_seed1_div_s0.npy
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0d96d5d3a2386ef667f1b55f1cc500de49e7dc0848233c1d6f7792f9b0067457
|
| 3 |
+
size 445568
|
results/icml_r3/X8/cands/val_x2_reg_mcd_seed2_div_s0.npy
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7aafc5c70c9d877428df8286cac03912eaaad0dd7dc98f99210317daf4eb4b8d
|
| 3 |
+
size 445568
|
results/icml_r3/X8/curves.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"plain": {"generator (seed 0, 3 sampler seeds)": {"mean": [94.0, 97.11111111111113, 97.66666666666669, 98.22222222222223, 98.77777777777777, 99.0, 99.0, 99.22222222222221, 99.22222222222221, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333], "ci": [[91.88888888888887, 95.44444444444444, 96.21944444444443, 96.88888888888889, 97.55555555555556, 97.88888888888889, 97.88888888888889, 98.11111111111113, 98.11111111111113, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333], [96.0, 98.55555555555554, 99.0, 99.33333333333333, 99.66666666666667, 99.88888888888887, 99.88888888888887, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_unit (X2 baseline)": {"mean": [40.0, 56.99999999999999, 66.33333333333333, 71.33333333333334, 74.66666666666667, 79.33333333333333, 82.0, 85.0, 86.33333333333333, 88.66666666666667, 89.0, 90.0, 91.0, 91.66666666666666, 92.66666666666666, 93.33333333333333], "ci": [[34.333333333333336, 51.33333333333333, 61.0, 66.0, 69.66666666666667, 74.66666666666667, 77.66666666666666, 81.0, 82.33333333333334, 85.0, 85.33333333333334, 86.66666666666667, 87.66666666666667, 88.33333333333333, 89.66666666666666, 90.33333333333333], [45.666666666666664, 62.66666666666667, 71.66666666666667, 76.33333333333333, 79.66666666666666, 84.0, 86.33333333333333, 89.0, 90.0, 92.0, 92.33333333333333, 93.33333333333333, 94.0, 94.66666666666667, 95.66666666666667, 96.0]]}, "retr_learned": {"mean": [42.0, 61.0, 71.33333333333334, 76.66666666666667, 82.0, 86.66666666666667, 89.66666666666666, 90.66666666666666, 92.0, 92.33333333333333, 93.0, 94.0, 94.33333333333334, 94.66666666666667, 95.0, 96.66666666666667], "ci": [[36.333333333333336, 55.333333333333336, 66.0, 72.0, 77.66666666666666, 82.66666666666667, 86.0, 87.33333333333333, 88.66666666666667, 89.33333333333333, 90.0, 91.0, 91.66666666666666, 92.0, 92.33333333333333, 94.33333333333334], [47.66666666666667, 66.66666666666666, 76.33333333333333, 81.33333333333333, 86.33333333333333, 90.33333333333333, 93.0, 93.66666666666667, 95.0, 95.33333333333334, 95.66666666666667, 96.66666666666667, 96.66666666666667, 97.0, 97.33333333333334, 98.66666666666667]]}, "twin (3 members x 3 draw seeds)": {"mean": [84.99999999999997, 88.40740740740736, 90.66666666666666, 91.88888888888886, 92.8518518518518, 93.96296296296295, 94.44444444444441, 94.88888888888887, 95.1111111111111, 95.37037037037038, 95.66666666666664, 95.7037037037037, 95.88888888888889, 96.0, 96.14814814814815, 96.22222222222223], "ci": [[81.8861111111111, 85.59259259259258, 88.07407407407408, 89.44444444444444, 90.5925925925926, 91.81481481481481, 92.4074074074074, 92.81481481481481, 93.1111111111111, 93.37037037037035, 93.74074074074072, 93.7037037037037, 93.96296296296295, 94.11111111111113, 94.33333333333334, 94.2962962962963], [88.0, 91.07407407407405, 93.00092592592591, 94.11203703703703, 94.88981481481481, 95.85185185185185, 96.2962962962963, 96.7037037037037, 96.88888888888889, 97.11111111111113, 97.37037037037037, 97.4074074074074, 97.59259259259258, 97.62962962962963, 97.81481481481482, 97.81481481481482]]}}, "refine": {"generator (seed 0, 3 sampler seeds)": {"mean": [94.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], "ci": [[91.77777777777779, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], [96.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_unit (X2 baseline)": {"mean": [40.0, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333], "ci": [[34.333333333333336, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333], [45.666666666666664, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_learned": {"mean": [42.0, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667], "ci": [[36.333333333333336, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 99.0, 99.0, 99.0, 99.0], [47.66666666666667, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "twin (3 members x 3 draw seeds)": {"mean": [84.99999999999997, 99.33333333333333, 99.33333333333333, 99.55555555555554, 99.55555555555554, 99.55555555555554, 99.55555555555554, 99.55555555555554, 99.55555555555554, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.66666666666667, 99.66666666666667, 99.66666666666667], "ci": [[81.88888888888887, 98.44444444444446, 98.44444444444446, 98.77777777777777, 98.77777777777777, 98.77777777777777, 98.77777777777777, 98.81481481481482, 98.77777777777777, 98.85185185185185, 98.85185185185185, 98.85185185185185, 98.88888888888889, 99.0, 99.0, 99.0], [88.0, 99.8888888888889, 99.8888888888889, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}}, "rank": {"generator (seed 0, 3 sampler seeds)": {"mean": [99.11111111111113, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333], "ci": [[98.0, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333], [99.8888888888889, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_unit (X2 baseline)": {"mean": [89.0, 91.66666666666666, 92.33333333333333, 92.66666666666666, 92.66666666666666, 92.66666666666666, 92.66666666666666, 92.66666666666666, 93.0, 93.0, 93.0, 93.0, 93.0, 93.0, 93.33333333333333, 93.33333333333333], "ci": [[85.33333333333334, 88.33333333333333, 89.33333333333333, 89.66666666666666, 89.66666666666666, 89.66666666666666, 89.66666666666666, 89.66666666666666, 90.0, 90.0, 90.0, 90.0, 90.0, 90.0, 90.33333333333333, 90.33333333333333], [92.33333333333333, 94.66666666666667, 95.0, 95.33333333333334, 95.33333333333334, 95.33333333333334, 95.33333333333334, 95.33333333333334, 95.66666666666667, 95.66666666666667, 95.66666666666667, 95.66666666666667, 95.66666666666667, 95.66666666666667, 96.0, 96.0]]}, "retr_learned": {"mean": [93.66666666666667, 95.0, 96.33333333333334, 96.66666666666667, 96.66666666666667, 96.66666666666667, 96.66666666666667, 96.66666666666667, 96.66666666666667, 96.66666666666667, 96.66666666666667, 96.66666666666667, 96.66666666666667, 96.66666666666667, 96.66666666666667, 96.66666666666667], "ci": [[90.66666666666666, 92.33333333333333, 94.0, 94.33333333333334, 94.33333333333334, 94.33333333333334, 94.66666666666667, 94.33333333333334, 94.66666666666667, 94.33333333333334, 94.66666666666667, 94.66666666666667, 94.66666666666667, 94.66666666666667, 94.33333333333334, 94.33333333333334], [96.33333333333334, 97.33333333333334, 98.33333333333333, 98.66666666666667, 98.66666666666667, 98.66666666666667, 98.66666666666667, 98.66666666666667, 98.66666666666667, 98.66666666666667, 98.66666666666667, 98.66666666666667, 98.66666666666667, 98.66666666666667, 98.66666666666667, 98.66666666666667]]}, "twin (3 members x 3 draw seeds)": {"mean": [96.0, 96.0, 96.03703703703704, 96.11111111111113, 96.11111111111113, 96.11111111111113, 96.11111111111113, 96.14814814814815, 96.14814814814815, 96.14814814814815, 96.14814814814815, 96.14814814814815, 96.14814814814815, 96.14814814814815, 96.18518518518519, 96.22222222222223], "ci": [[94.18518518518518, 94.14722222222221, 94.18518518518518, 94.18518518518518, 94.18518518518518, 94.22222222222223, 94.1851851851852, 94.25925925925925, 94.2962962962963, 94.18518518518518, 94.25925925925925, 94.22222222222223, 94.25925925925928, 94.25925925925925, 94.33333333333334, 94.37037037037037], [97.62962962962963, 97.7037037037037, 97.74074074074073, 97.77777777777777, 97.74074074074073, 97.77777777777777, 97.74074074074073, 97.77777777777779, 97.74074074074073, 97.77777777777777, 97.81481481481482, 97.7787037037037, 97.77777777777779, 97.85185185185185, 97.81481481481482, 97.85185185185185]]}}, "rank_refine": {"generator (seed 0, 3 sampler seeds)": {"mean": [99.11111111111113, 99.8888888888889, 99.8888888888889, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], "ci": [[98.0, 99.66666666666667, 99.66666666666667, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], [99.8888888888889, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_unit (X2 baseline)": {"mean": [89.0, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333], "ci": [[85.33333333333334, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333], [92.33333333333333, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_learned": {"mean": [93.66666666666667, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], "ci": [[90.66666666666666, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], [96.33333333333334, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "twin (3 members x 3 draw seeds)": {"mean": [96.0, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.62962962962963, 99.62962962962963, 99.62962962962963, 99.62962962962963, 99.62962962962963, 99.62962962962963, 99.62962962962963, 99.62962962962963, 99.66666666666667], "ci": [[94.11111111111113, 98.85185185185185, 98.85185185185185, 98.85185185185185, 98.85185185185185, 98.85185185185185, 98.85185185185185, 98.92592592592592, 98.92592592592592, 98.92592592592592, 98.92592592592592, 98.92592592592592, 98.92592592592592, 98.92592592592592, 98.88888888888889, 99.0], [97.7037037037037, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}}, "refine_free": {"generator (seed 0, 3 sampler seeds)": {"mean": [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], "ci": [[100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_unit (X2 baseline)": {"mean": [99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333], "ci": [[98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333], [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_learned": {"mean": [99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.66666666666667, 99.66666666666667, 99.66666666666667, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], "ci": [[98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 99.0, 99.0, 99.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "twin (3 members x 3 draw seeds)": {"mean": [99.33333333333333, 99.55555555555554, 99.55555555555554, 99.55555555555554, 99.59259259259258, 99.59259259259258, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667], "ci": [[98.44444444444446, 98.77777777777777, 98.77685185185186, 98.77777777777777, 98.85185185185185, 98.85185185185185, 99.0, 99.0, 99.0, 99.0, 99.0, 99.0, 99.0, 99.0, 99.0, 99.0], [99.8888888888889, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}}, "rank_refine_free": {"generator (seed 0, 3 sampler seeds)": {"mean": [99.8888888888889, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], "ci": [[99.66666666666667, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_unit (X2 baseline)": {"mean": [99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333], "ci": [[98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333], [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_learned": {"mean": [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], "ci": [[100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "twin (3 members x 3 draw seeds)": {"mean": [99.59259259259258, 99.59259259259258, 99.59259259259258, 99.62962962962963, 99.62962962962963, 99.62962962962963, 99.62962962962963, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667], "ci": [[98.85185185185185, 98.77777777777777, 98.77777777777777, 98.92592592592592, 98.92592592592592, 98.92592592592592, 98.92592592592592, 99.0, 99.0, 99.0, 99.0, 99.0, 99.0, 99.0, 99.0, 99.0], [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}}, "refine_c3": {"generator (seed 0, 3 sampler seeds)": {"mean": [94.0, 94.0, 94.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], "ci": [[91.77777777777779, 91.77777777777779, 91.77777777777779, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], [96.0, 96.0, 96.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_unit (X2 baseline)": {"mean": [40.0, 40.0, 40.0, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333], "ci": [[34.333333333333336, 34.333333333333336, 34.66666666666667, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333], [45.666666666666664, 45.666666666666664, 45.666666666666664, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_learned": {"mean": [42.0, 42.0, 42.0, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333], "ci": [[36.333333333333336, 36.333333333333336, 36.333333333333336, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333], [47.66666666666667, 47.66666666666667, 47.66666666666667, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "twin (3 members x 3 draw seeds)": {"mean": [84.99999999999997, 84.99999999999997, 84.99999999999997, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.55555555555554, 99.55555555555554, 99.55555555555554, 99.55555555555554, 99.55555555555554, 99.55555555555554, 99.55555555555554, 99.55555555555554, 99.55555555555554], "ci": [[81.77777777777777, 81.77777777777777, 81.77777777777777, 98.44444444444446, 98.44444444444446, 98.44444444444443, 98.44444444444446, 98.77777777777777, 98.77777777777777, 98.77777777777777, 98.77777777777777, 98.77777777777777, 98.77777777777777, 98.77777777777777, 98.77777777777777, 98.77777777777777], [88.1111111111111, 88.1111111111111, 88.1111111111111, 99.8888888888889, 99.8888888888889, 99.8888888888889, 99.8888888888889, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}}, "rank_refine_c3": {"generator (seed 0, 3 sampler seeds)": {"mean": [99.11111111111113, 99.11111111111113, 99.11111111111113, 99.8888888888889, 99.8888888888889, 99.8888888888889, 99.8888888888889, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], "ci": [[98.0, 98.0, 98.0, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], [99.8888888888889, 99.8888888888889, 99.8888888888889, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_unit (X2 baseline)": {"mean": [89.0, 89.0, 89.0, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333], "ci": [[85.33333333333334, 85.33333333333334, 85.33333333333334, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333], [92.33333333333333, 92.33333333333333, 92.33333333333333, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_learned": {"mean": [93.66666666666667, 93.66666666666667, 93.66666666666667, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], "ci": [[90.66666666666666, 90.66666666666666, 90.66666666666666, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], [96.33333333333334, 96.33333333333334, 96.33333333333334, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "twin (3 members x 3 draw seeds)": {"mean": [96.0, 96.0, 96.0, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.62962962962963], "ci": [[94.07407407407408, 94.11111111111113, 94.11111111111113, 98.85185185185185, 98.85185185185185, 98.85185185185185, 98.85185185185185, 98.85185185185185, 98.85185185185185, 98.77777777777777, 98.77777777777777, 98.85185185185185, 98.85185185185185, 98.85185185185185, 98.85185185185185, 98.92592592592592], [97.66666666666667, 97.62962962962962, 97.66666666666667, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}}, "ensemble": {"plain": {"ensemble": {"mean": [84.99999999999997, 92.55555555555553, 95.33333333333334, 95.74074074074076, 96.14814814814815, 96.55555555555554, 96.7037037037037, 96.85185185185185, 97.0, 97.0, 97.0, 97.0, 97.22222222222223, 97.44444444444443, 97.66666666666664, 97.74074074074073], "ci": [[81.88888888888887, 90.0, 92.66666666666666, 93.44444444444446, 93.92592592592595, 94.55555555555554, 94.7037037037037, 94.92592592592592, 95.1111111111111, 95.1111111111111, 95.1111111111111, 95.0, 95.37037037037037, 95.62962962962965, 96.0, 96.03703703703704], [88.1111111111111, 95.0, 97.66666666666667, 97.81481481481482, 98.03703703703704, 98.33333333333333, 98.44444444444446, 98.55555555555556, 98.66666666666667, 98.66666666666667, 98.66666666666667, 98.66666666666667, 98.81481481481482, 98.92592592592592, 99.1111111111111, 99.14814814814815]]}}}}
|
results/icml_r3/X8/logs/report.log
ADDED
|
@@ -0,0 +1,154 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# X8 / X18 test report (corrected arm, n=300; bank1 = 2905 rows); refine config {'iters': 80, 'lr': 0.1, 'n': 200}; frozen primary pipeline rank_refine; retr_learned metric = xfer
|
| 2 |
+
|
| 3 |
+
## Refinement statistics on the test candidates (per arm: valid, certified, certified-but-failed, rescued by refinement)
|
| 4 |
+
generator valid 14400 certified 14400 certified-failed 793 rescued 786 (99.1%)
|
| 5 |
+
retr_unit valid 4800 certified 4800 certified-failed 3947 rescued 3893 (98.6%)
|
| 6 |
+
retr_learned valid 4800 certified 4800 certified-failed 3787 rescued 3717 (98.2%)
|
| 7 |
+
twin valid 43200 certified 43200 certified-failed 8003 rescued 7758 (96.9%)
|
| 8 |
+
|
| 9 |
+
## Test success % [95% CI over plants], pipeline = plain
|
| 10 |
+
arm b=1 b=2 b=4 b=8 b=16
|
| 11 |
+
generator (seed 0, 3 sampler seeds) 94.0 [91.8,96.0] 97.1 [95.4,98.6] 98.2 [96.9,99.3] 99.2 [98.1,100.0] 99.3 [98.3,100.0]
|
| 12 |
+
retr_unit (X2 baseline) 40.0 [34.7,45.7] 57.0 [51.3,62.3] 71.3 [66.3,76.3] 85.0 [81.0,89.0] 93.3 [90.3,96.0]
|
| 13 |
+
retr_learned 42.0 [36.3,47.3] 61.0 [55.3,66.3] 76.7 [72.0,81.3] 90.7 [87.3,93.7] 96.7 [94.7,98.7]
|
| 14 |
+
twin (3 members x 3 draw seeds) 85.0 [81.9,88.0] 88.4 [85.6,91.0] 91.9 [89.5,94.1] 94.9 [92.9,96.7] 96.2 [94.3,97.9]
|
| 15 |
+
|
| 16 |
+
## Test success % [95% CI over plants], pipeline = refine
|
| 17 |
+
arm b=1 b=2 b=4 b=8 b=16
|
| 18 |
+
generator (seed 0, 3 sampler seeds) 94.0 [91.9,96.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
|
| 19 |
+
retr_unit (X2 baseline) 40.0 [34.7,45.3] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
|
| 20 |
+
retr_learned 42.0 [36.3,47.7] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.7 [99.0,100.0]
|
| 21 |
+
twin (3 members x 3 draw seeds) 85.0 [81.9,88.1] 99.3 [98.4,99.9] 99.6 [98.8,100.0] 99.6 [98.8,100.0] 99.7 [99.0,100.0]
|
| 22 |
+
|
| 23 |
+
## Test success % [95% CI over plants], pipeline = rank
|
| 24 |
+
arm b=1 b=2 b=4 b=8 b=16
|
| 25 |
+
generator (seed 0, 3 sampler seeds) 99.1 [98.0,99.9] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
|
| 26 |
+
retr_unit (X2 baseline) 89.0 [85.3,92.3] 91.7 [88.3,94.7] 92.7 [89.7,95.3] 92.7 [89.7,95.3] 93.3 [90.3,96.0]
|
| 27 |
+
retr_learned 93.7 [90.7,96.3] 95.0 [92.3,97.3] 96.7 [94.3,98.7] 96.7 [94.7,98.7] 96.7 [94.3,98.7]
|
| 28 |
+
twin (3 members x 3 draw seeds) 96.0 [94.1,97.6] 96.0 [94.1,97.7] 96.1 [94.2,97.7] 96.1 [94.3,97.8] 96.2 [94.4,97.9]
|
| 29 |
+
|
| 30 |
+
## Test success % [95% CI over plants], pipeline = rank_refine <- FROZEN PRIMARY
|
| 31 |
+
arm b=1 b=2 b=4 b=8 b=16
|
| 32 |
+
generator (seed 0, 3 sampler seeds) 99.1 [98.0,99.9] 99.9 [99.7,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
|
| 33 |
+
retr_unit (X2 baseline) 89.0 [85.3,92.3] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
|
| 34 |
+
retr_learned 93.7 [90.7,96.3] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
|
| 35 |
+
twin (3 members x 3 draw seeds) 96.0 [94.1,97.7] 99.6 [98.8,100.0] 99.6 [98.9,100.0] 99.6 [98.9,100.0] 99.7 [99.0,100.0]
|
| 36 |
+
|
| 37 |
+
## Test success % [95% CI over plants], pipeline = refine_free
|
| 38 |
+
arm b=1 b=2 b=4 b=8 b=16
|
| 39 |
+
generator (seed 0, 3 sampler seeds) 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
|
| 40 |
+
retr_unit (X2 baseline) 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
|
| 41 |
+
retr_learned 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.7 [99.0,100.0] 100.0 [100.0,100.0]
|
| 42 |
+
twin (3 members x 3 draw seeds) 99.3 [98.4,99.9] 99.6 [98.8,100.0] 99.6 [98.8,100.0] 99.7 [99.0,100.0] 99.7 [99.0,100.0]
|
| 43 |
+
|
| 44 |
+
## Test success % [95% CI over plants], pipeline = rank_refine_free
|
| 45 |
+
arm b=1 b=2 b=4 b=8 b=16
|
| 46 |
+
generator (seed 0, 3 sampler seeds) 99.9 [99.7,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
|
| 47 |
+
retr_unit (X2 baseline) 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
|
| 48 |
+
retr_learned 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
|
| 49 |
+
twin (3 members x 3 draw seeds) 99.6 [98.9,100.0] 99.6 [98.9,100.0] 99.6 [98.9,100.0] 99.7 [99.0,100.0] 99.7 [99.0,100.0]
|
| 50 |
+
|
| 51 |
+
## Test success % [95% CI over plants], pipeline = refine_c3
|
| 52 |
+
arm b=1 b=2 b=4 b=8 b=16
|
| 53 |
+
generator (seed 0, 3 sampler seeds) 94.0 [91.8,96.0] 94.0 [91.8,96.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
|
| 54 |
+
retr_unit (X2 baseline) 40.0 [34.7,45.7] 40.0 [34.7,45.7] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
|
| 55 |
+
retr_learned 42.0 [36.3,47.7] 42.0 [36.3,47.3] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
|
| 56 |
+
twin (3 members x 3 draw seeds) 85.0 [81.7,88.1] 85.0 [81.7,88.1] 99.3 [98.4,99.9] 99.6 [98.8,100.0] 99.6 [98.8,100.0]
|
| 57 |
+
|
| 58 |
+
## Test success % [95% CI over plants], pipeline = rank_refine_c3
|
| 59 |
+
arm b=1 b=2 b=4 b=8 b=16
|
| 60 |
+
generator (seed 0, 3 sampler seeds) 99.1 [98.0,99.9] 99.1 [98.0,99.9] 99.9 [99.7,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
|
| 61 |
+
retr_unit (X2 baseline) 89.0 [85.3,92.3] 89.0 [85.3,92.3] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
|
| 62 |
+
retr_learned 93.7 [90.7,96.3] 93.7 [90.7,96.3] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
|
| 63 |
+
twin (3 members x 3 draw seeds) 96.0 [94.1,97.6] 96.0 [94.1,97.7] 99.6 [98.9,100.0] 99.6 [98.9,100.0] 99.6 [98.9,100.0]
|
| 64 |
+
|
| 65 |
+
## Descriptive: mean number of verifier calls to the first success (a failure within 16 calls counts as 17), per arm; mean over replicas and plants
|
| 66 |
+
plain generator 1.22 retr_unit 4.22 retr_learned 3.48 twin 2.02
|
| 67 |
+
refine generator 1.06 retr_unit 1.70 retr_learned 1.67 twin 1.22
|
| 68 |
+
rank generator 1.11 retr_unit 2.19 retr_learned 1.58 twin 1.62
|
| 69 |
+
rank_refine generator 1.01 retr_unit 1.21 retr_learned 1.06 twin 1.10
|
| 70 |
+
refine_free generator 1.00 retr_unit 1.11 retr_learned 1.05 twin 1.06
|
| 71 |
+
rank_refine_free generator 1.00 retr_unit 1.11 retr_learned 1.00 twin 1.06
|
| 72 |
+
refine_c3 generator 1.18 retr_unit 2.89 retr_learned 2.83 twin 1.52
|
| 73 |
+
rank_refine_c3 generator 1.03 retr_unit 1.42 retr_learned 1.19 twin 1.17
|
| 74 |
+
|
| 75 |
+
## PRIMARY family (9 tests, Holm): generator minus comparator under rank_refine (counted refinement), pp [95% CI over plants], sign-flip p, Holm p, McNemar (gen-only/comp-only; sampler seed 0 vs twin member 0 draw 0) p
|
| 76 |
+
gen - retr_unit, b=1, pipeline rank_refine +10.1 [ +6.9,+13.7] p=0.0000 holm=0.0000 McNemar 31/1 p=0.0000
|
| 77 |
+
gen - retr_learned, b=1, pipeline rank_refine +5.4 [ +2.8, +8.4] p=0.0001 holm=0.0003 McNemar 18/2 p=0.0004
|
| 78 |
+
gen - twin, b=1, pipeline rank_refine +3.1 [ +1.6, +4.9] p=0.0000 holm=0.0000 McNemar 6/1 p=0.1250
|
| 79 |
+
gen - retr_unit, b=4, pipeline rank_refine +0.7 [ +0.0, +1.7] p=0.5009 holm=1.0000 McNemar 2/0 p=0.5000
|
| 80 |
+
gen - retr_learned, b=4, pipeline rank_refine +0.0 [ +0.0, +0.0] p=1.0000 holm=1.0000 McNemar 0/0 p=1.0000
|
| 81 |
+
gen - twin, b=4, pipeline rank_refine +0.4 [ +0.0, +1.2] p=0.5045 holm=1.0000 McNemar 1/0 p=1.0000
|
| 82 |
+
gen - retr_unit, b=16, pipeline rank_refine +0.7 [ +0.0, +1.7] p=0.4913 holm=1.0000 McNemar 2/0 p=0.5000
|
| 83 |
+
gen - retr_learned, b=16, pipeline rank_refine +0.0 [ +0.0, +0.0] p=1.0000 holm=1.0000 McNemar 0/0 p=1.0000
|
| 84 |
+
gen - twin, b=16, pipeline rank_refine +0.3 [ +0.0, +1.0] p=1.0000 holm=1.0000 McNemar 1/0 p=1.0000
|
| 85 |
+
|
| 86 |
+
## SECONDARY (unadjusted): generator minus comparator, every pipeline, pp [95% CI]
|
| 87 |
+
plain gen - retr_unit (X2 baseline) +54.0 [+48.0,+59.8] p=0.000 +40.1 [+34.2,+45.9] p=0.000 +26.9 [+21.8,+32.0] p=0.000 +14.2 [+10.3,+18.3] p=0.000 +6.0 [ +3.3, +9.0] p=0.000
|
| 88 |
+
plain gen - retr_learned +52.0 [+46.1,+57.8] p=0.000 +36.1 [+30.3,+41.9] p=0.000 +21.6 [+16.9,+26.6] p=0.000 +8.6 [ +5.3,+12.0] p=0.000 +2.7 [ +0.7, +4.7] p=0.021
|
| 89 |
+
plain gen - twin (3 members x 3 draw seeds) +9.0 [ +6.0,+12.1] p=0.000 +8.7 [ +6.1,+11.4] p=0.000 +6.3 [ +4.2, +8.6] p=0.000 +4.3 [ +2.7, +6.2] p=0.000 +3.1 [ +1.6, +4.8] p=0.000
|
| 90 |
+
refine gen - retr_unit (X2 baseline) +54.0 [+48.0,+59.8] p=0.000 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.501
|
| 91 |
+
refine gen - retr_learned +52.0 [+46.2,+58.0] p=0.000 +0.7 [ +0.0, +1.7] p=0.502 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.504 +0.3 [ +0.0, +1.0] p=1.000
|
| 92 |
+
refine gen - twin (3 members x 3 draw seeds) +9.0 [ +5.9,+12.1] p=0.000 +0.7 [ +0.1, +1.6] p=0.125 +0.4 [ +0.0, +1.2] p=0.256 +0.4 [ +0.0, +1.2] p=0.249 +0.3 [ +0.0, +1.0] p=1.000
|
| 93 |
+
rank gen - retr_unit (X2 baseline) +10.1 [ +6.8,+13.7] p=0.000 +7.7 [ +4.7,+11.0] p=0.000 +6.7 [ +4.0, +9.7] p=0.000 +6.7 [ +4.0, +9.7] p=0.000 +6.0 [ +3.3, +8.7] p=0.000
|
| 94 |
+
rank gen - retr_learned +5.4 [ +2.8, +8.3] p=0.000 +4.3 [ +2.0, +7.0] p=0.001 +2.7 [ +0.7, +5.0] p=0.021 +2.7 [ +0.7, +4.7] p=0.022 +2.7 [ +0.7, +5.0] p=0.023
|
| 95 |
+
rank gen - twin (3 members x 3 draw seeds) +3.1 [ +1.6, +4.8] p=0.000 +3.3 [ +1.9, +5.1] p=0.000 +3.2 [ +1.7, +5.0] p=0.000 +3.2 [ +1.7, +4.9] p=0.000 +3.1 [ +1.6, +4.8] p=0.000
|
| 96 |
+
rank_refine gen - retr_unit (X2 baseline) +10.1 [ +6.9,+13.7] p=0.000 +0.6 [ +0.0, +1.4] p=0.502 +0.7 [ +0.0, +1.7] p=0.501 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.500
|
| 97 |
+
rank_refine gen - retr_learned +5.4 [ +2.8, +8.3] p=0.000 -0.1 [ -0.3, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000
|
| 98 |
+
rank_refine gen - twin (3 members x 3 draw seeds) +3.1 [ +1.6, +4.8] p=0.000 +0.3 [ +0.0, +0.8] p=0.495 +0.4 [ +0.0, +1.1] p=0.495 +0.4 [ +0.0, +1.1] p=0.500 +0.3 [ +0.0, +1.0] p=1.000
|
| 99 |
+
refine_free gen - retr_unit (X2 baseline) +0.7 [ +0.0, +1.7] p=0.497 +0.7 [ +0.0, +1.7] p=0.504 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.502
|
| 100 |
+
refine_free gen - retr_learned +0.7 [ +0.0, +1.7] p=0.500 +0.7 [ +0.0, +1.7] p=0.504 +0.7 [ +0.0, +1.7] p=0.501 +0.3 [ +0.0, +1.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000
|
| 101 |
+
refine_free gen - twin (3 members x 3 draw seeds) +0.7 [ +0.1, +1.6] p=0.130 +0.4 [ +0.0, +1.2] p=0.252 +0.4 [ +0.0, +1.2] p=0.255 +0.3 [ +0.0, +1.0] p=1.000 +0.3 [ +0.0, +1.0] p=1.000
|
| 102 |
+
rank_refine_free gen - retr_unit (X2 baseline) +0.6 [ +0.0, +1.4] p=0.502 +0.7 [ +0.0, +1.7] p=0.496 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.506 +0.7 [ +0.0, +1.7] p=0.498
|
| 103 |
+
rank_refine_free gen - retr_learned -0.1 [ -0.3, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000
|
| 104 |
+
rank_refine_free gen - twin (3 members x 3 draw seeds) +0.3 [ +0.0, +0.8] p=0.505 +0.4 [ +0.0, +1.1] p=0.496 +0.4 [ +0.0, +1.1] p=0.498 +0.3 [ +0.0, +1.0] p=1.000 +0.3 [ +0.0, +1.0] p=1.000
|
| 105 |
+
refine_c3 gen - retr_unit (X2 baseline) +54.0 [+48.1,+59.9] p=0.000 +54.0 [+48.0,+60.0] p=0.000 +0.7 [ +0.0, +1.7] p=0.501 +0.7 [ +0.0, +1.7] p=0.497 +0.7 [ +0.0, +1.7] p=0.500
|
| 106 |
+
refine_c3 gen - retr_learned +52.0 [+46.1,+58.0] p=0.000 +52.0 [+46.0,+57.9] p=0.000 +0.7 [ +0.0, +1.7] p=0.496 +0.7 [ +0.0, +1.7] p=0.497 +0.7 [ +0.0, +1.7] p=0.499
|
| 107 |
+
refine_c3 gen - twin (3 members x 3 draw seeds) +9.0 [ +6.0,+12.0] p=0.000 +9.0 [ +6.0,+12.0] p=0.000 +0.7 [ +0.1, +1.6] p=0.121 +0.4 [ +0.0, +1.2] p=0.248 +0.4 [ +0.0, +1.2] p=0.248
|
| 108 |
+
rank_refine_c3 gen - retr_unit (X2 baseline) +10.1 [ +6.8,+13.6] p=0.000 +10.1 [ +6.8,+13.7] p=0.000 +0.6 [ +0.0, +1.4] p=0.498 +0.7 [ +0.0, +1.7] p=0.505 +0.7 [ +0.0, +1.7] p=0.503
|
| 109 |
+
rank_refine_c3 gen - retr_learned +5.4 [ +2.9, +8.3] p=0.000 +5.4 [ +2.8, +8.3] p=0.000 -0.1 [ -0.3, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000
|
| 110 |
+
rank_refine_c3 gen - twin (3 members x 3 draw seeds) +3.1 [ +1.6, +4.8] p=0.000 +3.1 [ +1.7, +4.9] p=0.000 +0.3 [ +0.0, +0.8] p=0.498 +0.4 [ +0.0, +1.1] p=0.503 +0.4 [ +0.0, +1.1] p=0.504
|
| 111 |
+
|
| 112 |
+
## Across training seeds (unit = seed; frozen pipeline rank_refine; seeds 1-4 use sampler seed 0 only, seed 0 is the mean over its 3 sampler seeds from the main table)
|
| 113 |
+
b=1: generator per seed [99.1, 99.7, 99.3, 99.7, 99.7]
|
| 114 |
+
minus retr_unit: per seed [10.1, 10.7, 10.3, 10.7, 10.7] mean +10.5 t95 [+10.2,+10.8] seeds positive 5/5
|
| 115 |
+
minus retr_learned: per seed [5.4, 6.0, 5.7, 6.0, 6.0] mean +5.8 t95 [+5.5,+6.1] seeds positive 5/5
|
| 116 |
+
minus twin: per seed [3.1, 3.7, 3.3, 3.7, 3.7] mean +3.5 t95 [+3.2,+3.8] seeds positive 5/5
|
| 117 |
+
b=4: generator per seed [100.0, 100.0, 100.0, 100.0, 100.0]
|
| 118 |
+
minus retr_unit: per seed [0.7, 0.7, 0.7, 0.7, 0.7] mean +0.7 t95 [+0.7,+0.7] seeds positive 5/5
|
| 119 |
+
minus retr_learned: per seed [0.0, 0.0, 0.0, 0.0, 0.0] mean +0.0 t95 [+0.0,+0.0] seeds positive 0/5
|
| 120 |
+
minus twin: per seed [0.4, 0.4, 0.4, 0.4, 0.4] mean +0.4 t95 [+0.4,+0.4] seeds positive 5/5
|
| 121 |
+
b=16: generator per seed [100.0, 100.0, 100.0, 100.0, 100.0]
|
| 122 |
+
minus retr_unit: per seed [0.7, 0.7, 0.7, 0.7, 0.7] mean +0.7 t95 [+0.7,+0.7] seeds positive 5/5
|
| 123 |
+
minus retr_learned: per seed [0.0, 0.0, 0.0, 0.0, 0.0] mean +0.0 t95 [+0.0,+0.0] seeds positive 0/5
|
| 124 |
+
minus twin: per seed [0.3, 0.3, 0.3, 0.3, 0.3] mean +0.3 t95 [+0.3,+0.3] seeds positive 5/5
|
| 125 |
+
|
| 126 |
+
# X18: twin ensemble (plain verification, no repair)
|
| 127 |
+
arm (plain) b=1 b=3 b=4 b=8 b=16
|
| 128 |
+
generator 94.0 [91.9,96.0] 97.7 [96.1,99.0] 98.2 [96.9,99.3] 99.2 [98.2,100.0] 99.3 [98.3,100.0]
|
| 129 |
+
twin ensemble (3 members, 9 replicas) 85.0 [81.8,88.1] 95.3 [92.7,97.7] 95.7 [93.4,97.8] 96.9 [94.9,98.6] 97.7 [96.1,99.1]
|
| 130 |
+
twin single member, MC dropout (9 replicas) 85.0 [81.8,88.0] 90.7 [88.1,93.1] 91.9 [89.4,94.1] 94.9 [92.9,96.7] 96.2 [94.3,97.9]
|
| 131 |
+
|
| 132 |
+
## X18 PRIMARY family (4 tests, Holm), pp [95% CI], sign-flip p, Holm p, McNemar (first replicas) p
|
| 133 |
+
ensemble - single-member twin, b=4 +3.9 [ +2.5, +5.3] p=0.0000 holm=0.0000 McNemar 6/1 p=0.1250
|
| 134 |
+
ensemble - single-member twin, b=16 +1.5 [ +0.8, +2.3] p=0.0000 holm=0.0000 McNemar 2/0 p=0.5000
|
| 135 |
+
generator - ensemble, b=4 +2.5 [ +0.6, +4.6] p=0.0176 holm=0.0353 McNemar 10/2 p=0.0386
|
| 136 |
+
generator - ensemble, b=16 +1.6 [ +0.4, +3.1] p=0.0213 holm=0.0353 McNemar 4/0 p=0.1250
|
| 137 |
+
|
| 138 |
+
## X18 SECONDARY (unadjusted), every b: ensemble - single twin | generator - ensemble | generator - single twin
|
| 139 |
+
b= 1: +0.0 [ +0.0, +0.0] p=1.000 | +9.0 [ +6.0,+12.1] p=0.000 | +9.0 [ +5.9,+12.1] p=0.000
|
| 140 |
+
b= 2: +4.1 [ +2.9, +5.4] p=0.000 | +4.6 [ +2.2, +7.1] p=0.000 | +8.7 [ +6.1,+11.3] p=0.000
|
| 141 |
+
b= 3: +4.7 [ +3.1, +6.3] p=0.000 | +2.3 [ +0.1, +4.8] p=0.059 | +7.0 [ +4.7, +9.4] p=0.000
|
| 142 |
+
b= 4: +3.9 [ +2.6, +5.3] p=0.000 | +2.5 [ +0.6, +4.6] p=0.016 | +6.3 [ +4.3, +8.7] p=0.000
|
| 143 |
+
b= 6: +2.6 [ +1.6, +3.7] p=0.000 | +2.4 [ +0.9, +4.2] p=0.006 | +5.0 [ +3.2, +7.0] p=0.000
|
| 144 |
+
b= 8: +2.0 [ +1.1, +2.9] p=0.000 | +2.4 [ +0.9, +4.1] p=0.003 | +4.3 [ +2.7, +6.2] p=0.000
|
| 145 |
+
b=12: +1.3 [ +0.5, +2.1] p=0.001 | +2.3 [ +0.9, +4.0] p=0.004 | +3.6 [ +2.1, +5.4] p=0.000
|
| 146 |
+
b=16: +1.5 [ +0.8, +2.3] p=0.000 | +1.6 [ +0.4, +3.1] p=0.024 | +3.1 [ +1.6, +4.8] p=0.000
|
| 147 |
+
|
| 148 |
+
## X18 with repair (secondary): ensemble vs single twin vs generator at b = 1, 4, 16, pipelines refine / rank_refine
|
| 149 |
+
refine: ens 85.0 99.7 99.7 | twin 85.0 99.6 99.7 | gen 94.0 100.0 100.0
|
| 150 |
+
rank_refine: ens 97.7 99.7 99.7 | twin 96.0 99.6 99.7 | gen 99.1 100.0 100.0
|
| 151 |
+
|
| 152 |
+
## Check: plain pipeline vs X2 (success % at b = 1, 4, 16; X2 values from results/icml_r3/X2/test_*.npz)
|
| 153 |
+
gen here 94.0 98.2 99.3 | X2 94.1 98.2 99.3
|
| 154 |
+
retr here 40.0 71.3 93.3 | X2 40.0 71.3 93.3
|
results/icml_r3/X8/logs/test_main.log
ADDED
|
@@ -0,0 +1,88 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[test] group gen0: ['gen0_s0', 'gen0_s1', 'gen0_s2']
|
| 2 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 3 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 4 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 5 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 6 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 7 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 8 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 9 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 10 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 11 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 12 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 13 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 14 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 15 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 16 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 17 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 18 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 19 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 20 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 21 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 22 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 23 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 24 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 25 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 26 |
+
verify 14400 candidates 591s: certified 14400, plain ok 13607 (mean 491 ms/call)
|
| 27 |
+
vmin 14400 candidates 2s
|
| 28 |
+
refine 793 failed-certified candidates 94s (median V_min 1.75 -> 0.26)
|
| 29 |
+
refined verifier ok 786 of 793 (105s)
|
| 30 |
+
[test] group retr: ['retr_unit', 'retr_learned']
|
| 31 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 32 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 33 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 34 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 35 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 36 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 37 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 38 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 39 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 40 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 41 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 42 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 43 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 44 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 45 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 46 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 47 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 48 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 49 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 50 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 51 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 52 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 53 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 54 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 55 |
+
verify 9600 candidates 401s: certified 9600, plain ok 1866 (mean 497 ms/call)
|
| 56 |
+
vmin 9600 candidates 2s
|
| 57 |
+
refine 7734 failed-certified candidates 840s (median V_min 3.88 -> 0.24)
|
| 58 |
+
refined verifier ok 7610 of 7734 (925s)
|
| 59 |
+
[test] group twin: ['twin0_d0', 'twin0_d1', 'twin0_d2', 'twin1_d0', 'twin1_d1', 'twin1_d2', 'twin2_d0', 'twin2_d1', 'twin2_d2']
|
| 60 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 61 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 62 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 63 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 64 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 65 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 66 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 67 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 68 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 69 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 70 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 71 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 72 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 73 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 74 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 75 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 76 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 77 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 78 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 79 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 80 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 81 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 82 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 83 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 84 |
+
verify 43200 candidates 1834s: certified 43200, plain ok 35197 (mean 509 ms/call)
|
| 85 |
+
vmin 43200 candidates 6s
|
| 86 |
+
refine 8003 failed-certified candidates 868s (median V_min 1.92 -> 0.24)
|
| 87 |
+
refined verifier ok 7758 of 8003 (956s)
|
| 88 |
+
test main done 4824s
|
results/icml_r3/X8/logs/test_seeds.log
ADDED
|
@@ -0,0 +1,117 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[test] group gen1: ['gen1_s0']
|
| 2 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 3 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 4 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 5 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 6 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 7 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 8 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 9 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 10 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 11 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 12 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 13 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 14 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 15 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 16 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 17 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 18 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 19 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 20 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 21 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 22 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 23 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 24 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 25 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 26 |
+
verify 4800 candidates 204s: certified 4800, plain ok 4589 (mean 499 ms/call)
|
| 27 |
+
vmin 4800 candidates 2s
|
| 28 |
+
refine 211 failed-certified candidates 32s (median V_min 1.64 -> 0.24)
|
| 29 |
+
refined verifier ok 205 of 211 (36s)
|
| 30 |
+
[test] group gen2: ['gen2_s0']
|
| 31 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 32 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 33 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 34 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 35 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 36 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 37 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 38 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 39 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 40 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 41 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 42 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 43 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 44 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 45 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 46 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 47 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 48 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 49 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 50 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 51 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 52 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 53 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 54 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 55 |
+
verify 4800 candidates 208s: certified 4800, plain ok 4573 (mean 509 ms/call)
|
| 56 |
+
vmin 4800 candidates 2s
|
| 57 |
+
refine 227 failed-certified candidates 32s (median V_min 1.79 -> 0.25)
|
| 58 |
+
refined verifier ok 226 of 227 (37s)
|
| 59 |
+
[test] group gen3: ['gen3_s0']
|
| 60 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 61 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 62 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 63 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 64 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 65 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 66 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 67 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 68 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 69 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 70 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 71 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 72 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 73 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 74 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 75 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 76 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 77 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 78 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 79 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 80 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 81 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 82 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 83 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 84 |
+
verify 4800 candidates 197s: certified 4800, plain ok 4603 (mean 478 ms/call)
|
| 85 |
+
vmin 4800 candidates 2s
|
| 86 |
+
refine 197 failed-certified candidates 32s (median V_min 1.69 -> 0.25)
|
| 87 |
+
refined verifier ok 190 of 197 (36s)
|
| 88 |
+
[test] group gen4: ['gen4_s0']
|
| 89 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 90 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 91 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 92 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 93 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 94 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 95 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 96 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 97 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 98 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 99 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 100 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 101 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 102 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 103 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 104 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 105 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 106 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 107 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 108 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 109 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 110 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 111 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 112 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 113 |
+
verify 4800 candidates 204s: certified 4800, plain ok 4606 (mean 493 ms/call)
|
| 114 |
+
vmin 4800 candidates 1s
|
| 115 |
+
refine 194 failed-certified candidates 32s (median V_min 1.71 -> 0.24)
|
| 116 |
+
refined verifier ok 191 of 194 (36s)
|
| 117 |
+
test seeds done 965s
|
results/icml_r3/X8/logs/val.log
ADDED
|
@@ -0,0 +1,181 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[val] group gen0: ['gen0_s0']
|
| 2 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 3 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 4 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 5 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 6 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 7 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 8 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 9 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 10 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 11 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 12 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 13 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 14 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 15 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 16 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 17 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 18 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 19 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 20 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 21 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 22 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 23 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 24 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 25 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 26 |
+
verify 960 candidates 37s: certified 960, plain ok 884 (mean 433 ms/call)
|
| 27 |
+
vmin 960 candidates 1s
|
| 28 |
+
refine 76 failed-certified candidates 16s (median V_min 1.78 -> 0.32)
|
| 29 |
+
refined verifier ok 70 of 76 (19s)
|
| 30 |
+
[val] group retr: ['retr_unit', 'retr_learned']
|
| 31 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 32 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 33 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 34 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 35 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 36 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 37 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 38 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 39 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 40 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 41 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 42 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 43 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 44 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 45 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 46 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 47 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 48 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 49 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 50 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 51 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 52 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 53 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 54 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 55 |
+
verify 1920 candidates 77s: certified 1920, plain ok 376 (mean 472 ms/call)
|
| 56 |
+
vmin 1920 candidates 1s
|
| 57 |
+
refine 1544 failed-certified candidates 93s (median V_min 4.06 -> 0.40)
|
| 58 |
+
refined verifier ok 1474 of 1544 (112s)
|
| 59 |
+
[val] group twin: ['twin0_d0', 'twin1_d0', 'twin2_d0']
|
| 60 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 61 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 62 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 63 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 64 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 65 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 66 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 67 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 68 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 69 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 70 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 71 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 72 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 73 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 74 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 75 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 76 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 77 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 78 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 79 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 80 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 81 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 82 |
+
/home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
|
| 83 |
+
cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
|
| 84 |
+
verify 2880 candidates 127s: certified 2880, plain ok 2313 (mean 520 ms/call)
|
| 85 |
+
vmin 2880 candidates 1s
|
| 86 |
+
refine 567 failed-certified candidates 33s (median V_min 2.00 -> 0.31)
|
| 87 |
+
refined verifier ok 553 of 567 (41s)
|
| 88 |
+
== refine config {'iters': 40, 'lr': 0.1, 'n': 200}: 415s
|
| 89 |
+
[val] group gen0: ['gen0_s0']
|
| 90 |
+
refine 76 failed-certified candidates 31s (median V_min 1.78 -> 0.28)
|
| 91 |
+
refined verifier ok 75 of 76 (33s)
|
| 92 |
+
[val] group retr: ['retr_unit', 'retr_learned']
|
| 93 |
+
refine 1544 failed-certified candidates 184s (median V_min 4.06 -> 0.25)
|
| 94 |
+
refined verifier ok 1499 of 1544 (203s)
|
| 95 |
+
[val] group twin: ['twin0_d0', 'twin1_d0', 'twin2_d0']
|
| 96 |
+
refine 567 failed-certified candidates 65s (median V_min 2.00 -> 0.25)
|
| 97 |
+
refined verifier ok 567 of 567 (73s)
|
| 98 |
+
== refine config {'iters': 80, 'lr': 0.1, 'n': 200}: 309s
|
| 99 |
+
[val] group gen0: ['gen0_s0']
|
| 100 |
+
refine 76 failed-certified candidates 16s (median V_min 1.78 -> 0.27)
|
| 101 |
+
refined verifier ok 76 of 76 (19s)
|
| 102 |
+
[val] group retr: ['retr_unit', 'retr_learned']
|
| 103 |
+
refine 1544 failed-certified candidates 92s (median V_min 4.06 -> 0.27)
|
| 104 |
+
refined verifier ok 1496 of 1544 (112s)
|
| 105 |
+
[val] group twin: ['twin0_d0', 'twin1_d0', 'twin2_d0']
|
| 106 |
+
refine 567 failed-certified candidates 33s (median V_min 2.00 -> 0.23)
|
| 107 |
+
refined verifier ok 567 of 567 (40s)
|
| 108 |
+
== refine config {'iters': 40, 'lr': 0.2, 'n': 200}: 171s
|
| 109 |
+
[val] group gen0: ['gen0_s0']
|
| 110 |
+
refine 76 failed-certified candidates 31s (median V_min 1.78 -> 0.22)
|
| 111 |
+
refined verifier ok 76 of 76 (33s)
|
| 112 |
+
[val] group retr: ['retr_unit', 'retr_learned']
|
| 113 |
+
refine 1544 failed-certified candidates 183s (median V_min 4.06 -> 0.20)
|
| 114 |
+
refined verifier ok 1492 of 1544 (202s)
|
| 115 |
+
[val] group twin: ['twin0_d0', 'twin1_d0', 'twin2_d0']
|
| 116 |
+
refine 567 failed-certified candidates 64s (median V_min 2.00 -> 0.18)
|
| 117 |
+
refined verifier ok 567 of 567 (72s)
|
| 118 |
+
== refine config {'iters': 80, 'lr': 0.2, 'n': 200}: 308s
|
| 119 |
+
[val] group gen0: ['gen0_s0']
|
| 120 |
+
refine 76 failed-certified candidates 32s (median V_min 2.29 -> 0.34)
|
| 121 |
+
refined verifier ok 70 of 76 (35s)
|
| 122 |
+
[val] group retr: ['retr_unit', 'retr_learned']
|
| 123 |
+
refine 1544 failed-certified candidates 196s (median V_min 4.23 -> 0.40)
|
| 124 |
+
refined verifier ok 1474 of 1544 (214s)
|
| 125 |
+
[val] group twin: ['twin0_d0', 'twin1_d0', 'twin2_d0']
|
| 126 |
+
refine 567 failed-certified candidates 64s (median V_min 2.49 -> 0.31)
|
| 127 |
+
refined verifier ok 553 of 567 (71s)
|
| 128 |
+
== refine config {'iters': 40, 'lr': 0.1, 'n': 400}: 320s
|
| 129 |
+
# X8 validation (60 plants; gen seed 0 sampler 0, retr_unit, retr_learned (metric xfer), twin members 0-2 draw seed 0)
|
| 130 |
+
|
| 131 |
+
refine config it40_lr0.1_n200: rescued 2097 of 2187 certified-failed candidates = 95.9% (per arm: gen0_s0 70/76, retr_unit 761/797, retr_learned 713/747, twin0_d0 195/195, twin1_d0 160/174, twin2_d0 198/198)
|
| 132 |
+
refine config it80_lr0.1_n200: rescued 2141 of 2187 certified-failed candidates = 97.9% (per arm: gen0_s0 75/76, retr_unit 777/797, retr_learned 722/747, twin0_d0 195/195, twin1_d0 174/174, twin2_d0 198/198)
|
| 133 |
+
refine config it40_lr0.2_n200: rescued 2139 of 2187 certified-failed candidates = 97.8% (per arm: gen0_s0 76/76, retr_unit 775/797, retr_learned 721/747, twin0_d0 195/195, twin1_d0 174/174, twin2_d0 198/198)
|
| 134 |
+
refine config it80_lr0.2_n200: rescued 2135 of 2187 certified-failed candidates = 97.6% (per arm: gen0_s0 76/76, retr_unit 775/797, retr_learned 717/747, twin0_d0 195/195, twin1_d0 174/174, twin2_d0 198/198)
|
| 135 |
+
refine config it40_lr0.1_n400: rescued 2097 of 2187 certified-failed candidates = 95.9% (per arm: gen0_s0 70/76, retr_unit 760/797, retr_learned 714/747, twin0_d0 195/195, twin1_d0 160/174, twin2_d0 198/198)
|
| 136 |
+
FROZEN refine config: {'iters': 80, 'lr': 0.1, 'n': 200} (highest pooled rescue rate; ties -> cheaper = earlier in grid)
|
| 137 |
+
|
| 138 |
+
# pipelines on validation, success % at b = 1/2/4/8/16 per arm (frozen refine config)
|
| 139 |
+
plain gen0_s0 88.3 93.3 98.3 98.3 98.3
|
| 140 |
+
plain retr_unit 31.7 50.0 73.3 85.0 93.3
|
| 141 |
+
plain retr_learned 35.0 55.0 70.0 88.3 95.0
|
| 142 |
+
plain twin0_d0 83.3 85.0 85.0 91.7 95.0
|
| 143 |
+
plain twin1_d0 86.7 95.0 98.3 98.3 98.3
|
| 144 |
+
plain twin2_d0 81.7 90.0 90.0 93.3 95.0
|
| 145 |
+
plain baseline mean over b=1,4,16 (retr_unit, retr_learned, twin): 74.38
|
| 146 |
+
refine gen0_s0 88.3 100.0 100.0 100.0 100.0
|
| 147 |
+
refine retr_unit 31.7 100.0 100.0 100.0 100.0
|
| 148 |
+
refine retr_learned 35.0 100.0 100.0 100.0 100.0
|
| 149 |
+
refine twin0_d0 83.3 100.0 100.0 100.0 100.0
|
| 150 |
+
refine twin1_d0 86.7 100.0 100.0 100.0 100.0
|
| 151 |
+
refine twin2_d0 81.7 100.0 100.0 100.0 100.0
|
| 152 |
+
refine baseline mean over b=1,4,16 (retr_unit, retr_learned, twin): 83.40
|
| 153 |
+
rank gen0_s0 98.3 98.3 98.3 98.3 98.3
|
| 154 |
+
rank retr_unit 83.3 88.3 93.3 93.3 93.3
|
| 155 |
+
rank retr_learned 88.3 93.3 93.3 95.0 95.0
|
| 156 |
+
rank twin0_d0 93.3 93.3 93.3 93.3 95.0
|
| 157 |
+
rank twin1_d0 98.3 98.3 98.3 98.3 98.3
|
| 158 |
+
rank twin2_d0 93.3 93.3 93.3 93.3 95.0
|
| 159 |
+
rank baseline mean over b=1,4,16 (retr_unit, retr_learned, twin): 92.53
|
| 160 |
+
rank_refine gen0_s0 98.3 100.0 100.0 100.0 100.0
|
| 161 |
+
rank_refine retr_unit 83.3 100.0 100.0 100.0 100.0
|
| 162 |
+
rank_refine retr_learned 88.3 100.0 100.0 100.0 100.0
|
| 163 |
+
rank_refine twin0_d0 93.3 100.0 100.0 100.0 100.0
|
| 164 |
+
rank_refine twin1_d0 98.3 100.0 100.0 100.0 100.0
|
| 165 |
+
rank_refine twin2_d0 93.3 100.0 100.0 100.0 100.0
|
| 166 |
+
rank_refine baseline mean over b=1,4,16 (retr_unit, retr_learned, twin): 96.30
|
| 167 |
+
refine_free gen0_s0 100.0 100.0 100.0 100.0 100.0
|
| 168 |
+
refine_free retr_unit 100.0 100.0 100.0 100.0 100.0
|
| 169 |
+
refine_free retr_learned 100.0 100.0 100.0 100.0 100.0
|
| 170 |
+
refine_free twin0_d0 100.0 100.0 100.0 100.0 100.0
|
| 171 |
+
refine_free twin1_d0 100.0 100.0 100.0 100.0 100.0
|
| 172 |
+
refine_free twin2_d0 100.0 100.0 100.0 100.0 100.0
|
| 173 |
+
refine_free baseline mean over b=1,4,16 (retr_unit, retr_learned, twin): 100.00
|
| 174 |
+
rank_refine_free gen0_s0 100.0 100.0 100.0 100.0 100.0
|
| 175 |
+
rank_refine_free retr_unit 100.0 100.0 100.0 100.0 100.0
|
| 176 |
+
rank_refine_free retr_learned 100.0 100.0 100.0 100.0 100.0
|
| 177 |
+
rank_refine_free twin0_d0 100.0 100.0 100.0 100.0 100.0
|
| 178 |
+
rank_refine_free twin1_d0 100.0 100.0 100.0 100.0 100.0
|
| 179 |
+
rank_refine_free twin2_d0 100.0 100.0 100.0 100.0 100.0
|
| 180 |
+
rank_refine_free baseline mean over b=1,4,16 (retr_unit, retr_learned, twin): 100.00
|
| 181 |
+
FROZEN primary pipeline: rank_refine (higher mean baseline-arm success over b = 1, 4, 16 among refine / rank_refine; ties -> rank_refine)
|
results/icml_r3/X8/tables/test_gen0.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5e3c141de6e2a43968f791da8dad58ed8400f488996bba72afbde69b9707dccf
|
| 3 |
+
size 2233694
|
results/icml_r3/X8/tables/test_gen0_ref_it80_lr0.1_n200.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:969f40b7576188e814c64c9e830ebf8d9223f5baa660198b1ffac8091c56c83d
|
| 3 |
+
size 245550
|
results/icml_r3/X8/tables/test_gen1.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5936baeb215e1a26c64e432da7b2fd69af444be7ac28754ff1580f409e107829
|
| 3 |
+
size 745694
|
results/icml_r3/X8/tables/test_gen1_ref_it80_lr0.1_n200.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f74b531e739ee9ee7c9b17d472c27264280cd93585c79d29c55f5dc70b36d9c4
|
| 3 |
+
size 82350
|
results/icml_r3/X8/tables/test_gen2.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f9c21f998ef7f08398d24b0e07b2032e890ccc3f9d28681de49513621c326e8b
|
| 3 |
+
size 745694
|
results/icml_r3/X8/tables/test_gen2_ref_it80_lr0.1_n200.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:babb6fa538e28efe57352effbcee684bede26a57b1ee54a092096b56f2ec8273
|
| 3 |
+
size 82350
|
results/icml_r3/X8/tables/test_gen3.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:cde14c2f8013dc4e8d0d6f1a28a8b7ca29b8789269c7a4652b3c279060328686
|
| 3 |
+
size 745694
|
results/icml_r3/X8/tables/test_gen3_ref_it80_lr0.1_n200.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:df8f99a1a02c97325f07d7ec16c5708bbaa0b5a2dfa4d25d4453beea7bc67b07
|
| 3 |
+
size 82350
|
results/icml_r3/X8/tables/test_gen4.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:724dea7ef87068820b39b1474300db22d904deac34ecf46edceafca0fba82ddd
|
| 3 |
+
size 745694
|
results/icml_r3/X8/tables/test_gen4_ref_it80_lr0.1_n200.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ab45481501f4e917936d35555aa5a5659462ca52b6697bbd40ad2392924bd418
|
| 3 |
+
size 82350
|
results/icml_r3/X8/tables/test_retr.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:99ecdbc7ac5746199a378543b598f005e62039429a242b05b1eb1c3b6f6e8489
|
| 3 |
+
size 1489694
|
results/icml_r3/X8/tables/test_retr_ref_it80_lr0.1_n200.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5995a5f4b3fdd8f54fdaa9937b30f1c7dd19345aba5a5c17c3b3a15a4e1478f7
|
| 3 |
+
size 163950
|
results/icml_r3/X8/tables/test_twin.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:97c80561cfd64ca2afbd72cffdae8b993a8543cd74a5a7d5fe97acccfe4f9be3
|
| 3 |
+
size 6697694
|
results/icml_r3/X8/tables/test_twin_ref_it80_lr0.1_n200.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7b7c043f12a6b93196aa709e44da724322fd066cf5d1c337f32aca729601a977
|
| 3 |
+
size 735150
|
results/icml_r3/X8/tables/val_gen0.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2ff1cfd2edbf44061c7e9c26c9cfa823dc85fdd98f17e1090510bac0e7271ec3
|
| 3 |
+
size 150494
|
results/icml_r3/X8/tables/val_gen0_ref_it40_lr0.1_n200.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:97ff2853ac9a85e4fa82e718c38822d4ef9a6e80ad41938ce48fb1c11c5b02d2
|
| 3 |
+
size 17070
|
results/icml_r3/X8/tables/val_gen0_ref_it40_lr0.1_n400.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2487d63ea9dd1ea496f84adc6322e08ba1883f153a448445ea57d2faddab2768
|
| 3 |
+
size 17070
|
results/icml_r3/X8/tables/val_gen0_ref_it40_lr0.2_n200.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:1817fe080258473e18bebdcea442c9e7d6eed882afccd54e045736f7241880c7
|
| 3 |
+
size 17070
|
results/icml_r3/X8/tables/val_gen0_ref_it80_lr0.1_n200.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3bcb676ee9abd32963e860920c9a3ffec3403dde4fad62f713ad9467f8c0f9c5
|
| 3 |
+
size 17070
|
results/icml_r3/X8/tables/val_gen0_ref_it80_lr0.2_n200.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ac17194fee8f496f27f85fee877bf8fe70ad85e49a11d897f82c3f105e5efc9e
|
| 3 |
+
size 17070
|
results/icml_r3/X8/tables/val_retr.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b134fe57f2911ef6a67edb51874de347752539f9610c755d09f694eb1c8f3756
|
| 3 |
+
size 299294
|
results/icml_r3/X8/tables/val_retr_ref_it40_lr0.1_n200.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3d30dbcd1bff4b099180aafc7b46a2d545e78aa5ecb387c5ce0462ea01d6dc4c
|
| 3 |
+
size 33390
|
results/icml_r3/X8/tables/val_retr_ref_it40_lr0.1_n400.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2eedf8179b5c543dce3cb5a11eb88c34aadd7acc897a9b0a281632bcea0d031a
|
| 3 |
+
size 33390
|
results/icml_r3/X8/tables/val_retr_ref_it40_lr0.2_n200.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c11b4534fb8befcc3992533c9d12d8798947fbe08091ecafb1b7b6e72adb0a72
|
| 3 |
+
size 33390
|
results/icml_r3/X8/tables/val_retr_ref_it80_lr0.1_n200.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:64ad1742d310f0ab456ab9e2a38ad2700aaa17952ecf8f664038f97be0887c9b
|
| 3 |
+
size 33390
|
results/icml_r3/X8/tables/val_retr_ref_it80_lr0.2_n200.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:937604109bb88fb469d9b605fdf3bf1c8b18c6da5a04264f6089a20a9487bcf2
|
| 3 |
+
size 33390
|
results/icml_r3/X8/tables/val_twin.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:fdf942f390c1fb8af0bacafebadfca4b3ef8ed15b3669580ec8533e39696eafe
|
| 3 |
+
size 448094
|
results/icml_r3/X8/tables/val_twin_ref_it40_lr0.1_n200.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:13b019039affff01ee7791c140c55cc424a665d7b3605ba1dbf2b96dd3722bf9
|
| 3 |
+
size 49710
|
results/icml_r3/X8/tables/val_twin_ref_it40_lr0.1_n400.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:932cf3ef0a9b378ec3a8f3f7e4eefd59e0c5977a8e1a8d05751a10ea60c703a9
|
| 3 |
+
size 49710
|
results/icml_r3/X8/tables/val_twin_ref_it40_lr0.2_n200.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:386d1465fda53f1137b5542951641f2c8c224ecbaa5853130f29547a712abf2d
|
| 3 |
+
size 49710
|
results/icml_r3/X8/tables/val_twin_ref_it80_lr0.1_n200.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:afab4f0be0f837cda27656378d71d132182c318077ea4a8d2cbd203edf0cb271
|
| 3 |
+
size 49710
|
results/icml_r3/X8/tables/val_twin_ref_it80_lr0.2_n200.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:bce8579bc72c2eab7ea3de7d494d6e058f8ba5b365a3ff5133ab27e4220d5cb8
|
| 3 |
+
size 49710
|
results/icml_r3/X8/test_report.txt
ADDED
|
@@ -0,0 +1,154 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# X8 / X18 test report (corrected arm, n=300; bank1 = 2905 rows); refine config {'iters': 80, 'lr': 0.1, 'n': 200}; frozen primary pipeline rank_refine; retr_learned metric = xfer
|
| 2 |
+
|
| 3 |
+
## Refinement statistics on the test candidates (per arm: valid, certified, certified-but-failed, rescued by refinement)
|
| 4 |
+
generator valid 14400 certified 14400 certified-failed 793 rescued 786 (99.1%)
|
| 5 |
+
retr_unit valid 4800 certified 4800 certified-failed 3947 rescued 3893 (98.6%)
|
| 6 |
+
retr_learned valid 4800 certified 4800 certified-failed 3787 rescued 3717 (98.2%)
|
| 7 |
+
twin valid 43200 certified 43200 certified-failed 8003 rescued 7758 (96.9%)
|
| 8 |
+
|
| 9 |
+
## Test success % [95% CI over plants], pipeline = plain
|
| 10 |
+
arm b=1 b=2 b=4 b=8 b=16
|
| 11 |
+
generator (seed 0, 3 sampler seeds) 94.0 [91.8,96.0] 97.1 [95.4,98.6] 98.2 [96.9,99.3] 99.2 [98.1,100.0] 99.3 [98.3,100.0]
|
| 12 |
+
retr_unit (X2 baseline) 40.0 [34.7,45.7] 57.0 [51.3,62.3] 71.3 [66.3,76.3] 85.0 [81.0,89.0] 93.3 [90.3,96.0]
|
| 13 |
+
retr_learned 42.0 [36.3,47.3] 61.0 [55.3,66.3] 76.7 [72.0,81.3] 90.7 [87.3,93.7] 96.7 [94.7,98.7]
|
| 14 |
+
twin (3 members x 3 draw seeds) 85.0 [81.9,88.0] 88.4 [85.6,91.0] 91.9 [89.5,94.1] 94.9 [92.9,96.7] 96.2 [94.3,97.9]
|
| 15 |
+
|
| 16 |
+
## Test success % [95% CI over plants], pipeline = refine
|
| 17 |
+
arm b=1 b=2 b=4 b=8 b=16
|
| 18 |
+
generator (seed 0, 3 sampler seeds) 94.0 [91.9,96.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
|
| 19 |
+
retr_unit (X2 baseline) 40.0 [34.7,45.3] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
|
| 20 |
+
retr_learned 42.0 [36.3,47.7] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.7 [99.0,100.0]
|
| 21 |
+
twin (3 members x 3 draw seeds) 85.0 [81.9,88.1] 99.3 [98.4,99.9] 99.6 [98.8,100.0] 99.6 [98.8,100.0] 99.7 [99.0,100.0]
|
| 22 |
+
|
| 23 |
+
## Test success % [95% CI over plants], pipeline = rank
|
| 24 |
+
arm b=1 b=2 b=4 b=8 b=16
|
| 25 |
+
generator (seed 0, 3 sampler seeds) 99.1 [98.0,99.9] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
|
| 26 |
+
retr_unit (X2 baseline) 89.0 [85.3,92.3] 91.7 [88.3,94.7] 92.7 [89.7,95.3] 92.7 [89.7,95.3] 93.3 [90.3,96.0]
|
| 27 |
+
retr_learned 93.7 [90.7,96.3] 95.0 [92.3,97.3] 96.7 [94.3,98.7] 96.7 [94.7,98.7] 96.7 [94.3,98.7]
|
| 28 |
+
twin (3 members x 3 draw seeds) 96.0 [94.1,97.6] 96.0 [94.1,97.7] 96.1 [94.2,97.7] 96.1 [94.3,97.8] 96.2 [94.4,97.9]
|
| 29 |
+
|
| 30 |
+
## Test success % [95% CI over plants], pipeline = rank_refine <- FROZEN PRIMARY
|
| 31 |
+
arm b=1 b=2 b=4 b=8 b=16
|
| 32 |
+
generator (seed 0, 3 sampler seeds) 99.1 [98.0,99.9] 99.9 [99.7,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
|
| 33 |
+
retr_unit (X2 baseline) 89.0 [85.3,92.3] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
|
| 34 |
+
retr_learned 93.7 [90.7,96.3] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
|
| 35 |
+
twin (3 members x 3 draw seeds) 96.0 [94.1,97.7] 99.6 [98.8,100.0] 99.6 [98.9,100.0] 99.6 [98.9,100.0] 99.7 [99.0,100.0]
|
| 36 |
+
|
| 37 |
+
## Test success % [95% CI over plants], pipeline = refine_free
|
| 38 |
+
arm b=1 b=2 b=4 b=8 b=16
|
| 39 |
+
generator (seed 0, 3 sampler seeds) 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
|
| 40 |
+
retr_unit (X2 baseline) 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
|
| 41 |
+
retr_learned 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.7 [99.0,100.0] 100.0 [100.0,100.0]
|
| 42 |
+
twin (3 members x 3 draw seeds) 99.3 [98.4,99.9] 99.6 [98.8,100.0] 99.6 [98.8,100.0] 99.7 [99.0,100.0] 99.7 [99.0,100.0]
|
| 43 |
+
|
| 44 |
+
## Test success % [95% CI over plants], pipeline = rank_refine_free
|
| 45 |
+
arm b=1 b=2 b=4 b=8 b=16
|
| 46 |
+
generator (seed 0, 3 sampler seeds) 99.9 [99.7,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
|
| 47 |
+
retr_unit (X2 baseline) 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
|
| 48 |
+
retr_learned 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
|
| 49 |
+
twin (3 members x 3 draw seeds) 99.6 [98.9,100.0] 99.6 [98.9,100.0] 99.6 [98.9,100.0] 99.7 [99.0,100.0] 99.7 [99.0,100.0]
|
| 50 |
+
|
| 51 |
+
## Test success % [95% CI over plants], pipeline = refine_c3
|
| 52 |
+
arm b=1 b=2 b=4 b=8 b=16
|
| 53 |
+
generator (seed 0, 3 sampler seeds) 94.0 [91.8,96.0] 94.0 [91.8,96.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
|
| 54 |
+
retr_unit (X2 baseline) 40.0 [34.7,45.7] 40.0 [34.7,45.7] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
|
| 55 |
+
retr_learned 42.0 [36.3,47.7] 42.0 [36.3,47.3] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
|
| 56 |
+
twin (3 members x 3 draw seeds) 85.0 [81.7,88.1] 85.0 [81.7,88.1] 99.3 [98.4,99.9] 99.6 [98.8,100.0] 99.6 [98.8,100.0]
|
| 57 |
+
|
| 58 |
+
## Test success % [95% CI over plants], pipeline = rank_refine_c3
|
| 59 |
+
arm b=1 b=2 b=4 b=8 b=16
|
| 60 |
+
generator (seed 0, 3 sampler seeds) 99.1 [98.0,99.9] 99.1 [98.0,99.9] 99.9 [99.7,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
|
| 61 |
+
retr_unit (X2 baseline) 89.0 [85.3,92.3] 89.0 [85.3,92.3] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
|
| 62 |
+
retr_learned 93.7 [90.7,96.3] 93.7 [90.7,96.3] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
|
| 63 |
+
twin (3 members x 3 draw seeds) 96.0 [94.1,97.6] 96.0 [94.1,97.7] 99.6 [98.9,100.0] 99.6 [98.9,100.0] 99.6 [98.9,100.0]
|
| 64 |
+
|
| 65 |
+
## Descriptive: mean number of verifier calls to the first success (a failure within 16 calls counts as 17), per arm; mean over replicas and plants
|
| 66 |
+
plain generator 1.22 retr_unit 4.22 retr_learned 3.48 twin 2.02
|
| 67 |
+
refine generator 1.06 retr_unit 1.70 retr_learned 1.67 twin 1.22
|
| 68 |
+
rank generator 1.11 retr_unit 2.19 retr_learned 1.58 twin 1.62
|
| 69 |
+
rank_refine generator 1.01 retr_unit 1.21 retr_learned 1.06 twin 1.10
|
| 70 |
+
refine_free generator 1.00 retr_unit 1.11 retr_learned 1.05 twin 1.06
|
| 71 |
+
rank_refine_free generator 1.00 retr_unit 1.11 retr_learned 1.00 twin 1.06
|
| 72 |
+
refine_c3 generator 1.18 retr_unit 2.89 retr_learned 2.83 twin 1.52
|
| 73 |
+
rank_refine_c3 generator 1.03 retr_unit 1.42 retr_learned 1.19 twin 1.17
|
| 74 |
+
|
| 75 |
+
## PRIMARY family (9 tests, Holm): generator minus comparator under rank_refine (counted refinement), pp [95% CI over plants], sign-flip p, Holm p, McNemar (gen-only/comp-only; sampler seed 0 vs twin member 0 draw 0) p
|
| 76 |
+
gen - retr_unit, b=1, pipeline rank_refine +10.1 [ +6.9,+13.7] p=0.0000 holm=0.0000 McNemar 31/1 p=0.0000
|
| 77 |
+
gen - retr_learned, b=1, pipeline rank_refine +5.4 [ +2.8, +8.4] p=0.0001 holm=0.0003 McNemar 18/2 p=0.0004
|
| 78 |
+
gen - twin, b=1, pipeline rank_refine +3.1 [ +1.6, +4.9] p=0.0000 holm=0.0000 McNemar 6/1 p=0.1250
|
| 79 |
+
gen - retr_unit, b=4, pipeline rank_refine +0.7 [ +0.0, +1.7] p=0.5009 holm=1.0000 McNemar 2/0 p=0.5000
|
| 80 |
+
gen - retr_learned, b=4, pipeline rank_refine +0.0 [ +0.0, +0.0] p=1.0000 holm=1.0000 McNemar 0/0 p=1.0000
|
| 81 |
+
gen - twin, b=4, pipeline rank_refine +0.4 [ +0.0, +1.2] p=0.5045 holm=1.0000 McNemar 1/0 p=1.0000
|
| 82 |
+
gen - retr_unit, b=16, pipeline rank_refine +0.7 [ +0.0, +1.7] p=0.4913 holm=1.0000 McNemar 2/0 p=0.5000
|
| 83 |
+
gen - retr_learned, b=16, pipeline rank_refine +0.0 [ +0.0, +0.0] p=1.0000 holm=1.0000 McNemar 0/0 p=1.0000
|
| 84 |
+
gen - twin, b=16, pipeline rank_refine +0.3 [ +0.0, +1.0] p=1.0000 holm=1.0000 McNemar 1/0 p=1.0000
|
| 85 |
+
|
| 86 |
+
## SECONDARY (unadjusted): generator minus comparator, every pipeline, pp [95% CI]
|
| 87 |
+
plain gen - retr_unit (X2 baseline) +54.0 [+48.0,+59.8] p=0.000 +40.1 [+34.2,+45.9] p=0.000 +26.9 [+21.8,+32.0] p=0.000 +14.2 [+10.3,+18.3] p=0.000 +6.0 [ +3.3, +9.0] p=0.000
|
| 88 |
+
plain gen - retr_learned +52.0 [+46.1,+57.8] p=0.000 +36.1 [+30.3,+41.9] p=0.000 +21.6 [+16.9,+26.6] p=0.000 +8.6 [ +5.3,+12.0] p=0.000 +2.7 [ +0.7, +4.7] p=0.021
|
| 89 |
+
plain gen - twin (3 members x 3 draw seeds) +9.0 [ +6.0,+12.1] p=0.000 +8.7 [ +6.1,+11.4] p=0.000 +6.3 [ +4.2, +8.6] p=0.000 +4.3 [ +2.7, +6.2] p=0.000 +3.1 [ +1.6, +4.8] p=0.000
|
| 90 |
+
refine gen - retr_unit (X2 baseline) +54.0 [+48.0,+59.8] p=0.000 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.501
|
| 91 |
+
refine gen - retr_learned +52.0 [+46.2,+58.0] p=0.000 +0.7 [ +0.0, +1.7] p=0.502 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.504 +0.3 [ +0.0, +1.0] p=1.000
|
| 92 |
+
refine gen - twin (3 members x 3 draw seeds) +9.0 [ +5.9,+12.1] p=0.000 +0.7 [ +0.1, +1.6] p=0.125 +0.4 [ +0.0, +1.2] p=0.256 +0.4 [ +0.0, +1.2] p=0.249 +0.3 [ +0.0, +1.0] p=1.000
|
| 93 |
+
rank gen - retr_unit (X2 baseline) +10.1 [ +6.8,+13.7] p=0.000 +7.7 [ +4.7,+11.0] p=0.000 +6.7 [ +4.0, +9.7] p=0.000 +6.7 [ +4.0, +9.7] p=0.000 +6.0 [ +3.3, +8.7] p=0.000
|
| 94 |
+
rank gen - retr_learned +5.4 [ +2.8, +8.3] p=0.000 +4.3 [ +2.0, +7.0] p=0.001 +2.7 [ +0.7, +5.0] p=0.021 +2.7 [ +0.7, +4.7] p=0.022 +2.7 [ +0.7, +5.0] p=0.023
|
| 95 |
+
rank gen - twin (3 members x 3 draw seeds) +3.1 [ +1.6, +4.8] p=0.000 +3.3 [ +1.9, +5.1] p=0.000 +3.2 [ +1.7, +5.0] p=0.000 +3.2 [ +1.7, +4.9] p=0.000 +3.1 [ +1.6, +4.8] p=0.000
|
| 96 |
+
rank_refine gen - retr_unit (X2 baseline) +10.1 [ +6.9,+13.7] p=0.000 +0.6 [ +0.0, +1.4] p=0.502 +0.7 [ +0.0, +1.7] p=0.501 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.500
|
| 97 |
+
rank_refine gen - retr_learned +5.4 [ +2.8, +8.3] p=0.000 -0.1 [ -0.3, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000
|
| 98 |
+
rank_refine gen - twin (3 members x 3 draw seeds) +3.1 [ +1.6, +4.8] p=0.000 +0.3 [ +0.0, +0.8] p=0.495 +0.4 [ +0.0, +1.1] p=0.495 +0.4 [ +0.0, +1.1] p=0.500 +0.3 [ +0.0, +1.0] p=1.000
|
| 99 |
+
refine_free gen - retr_unit (X2 baseline) +0.7 [ +0.0, +1.7] p=0.497 +0.7 [ +0.0, +1.7] p=0.504 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.502
|
| 100 |
+
refine_free gen - retr_learned +0.7 [ +0.0, +1.7] p=0.500 +0.7 [ +0.0, +1.7] p=0.504 +0.7 [ +0.0, +1.7] p=0.501 +0.3 [ +0.0, +1.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000
|
| 101 |
+
refine_free gen - twin (3 members x 3 draw seeds) +0.7 [ +0.1, +1.6] p=0.130 +0.4 [ +0.0, +1.2] p=0.252 +0.4 [ +0.0, +1.2] p=0.255 +0.3 [ +0.0, +1.0] p=1.000 +0.3 [ +0.0, +1.0] p=1.000
|
| 102 |
+
rank_refine_free gen - retr_unit (X2 baseline) +0.6 [ +0.0, +1.4] p=0.502 +0.7 [ +0.0, +1.7] p=0.496 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.506 +0.7 [ +0.0, +1.7] p=0.498
|
| 103 |
+
rank_refine_free gen - retr_learned -0.1 [ -0.3, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000
|
| 104 |
+
rank_refine_free gen - twin (3 members x 3 draw seeds) +0.3 [ +0.0, +0.8] p=0.505 +0.4 [ +0.0, +1.1] p=0.496 +0.4 [ +0.0, +1.1] p=0.498 +0.3 [ +0.0, +1.0] p=1.000 +0.3 [ +0.0, +1.0] p=1.000
|
| 105 |
+
refine_c3 gen - retr_unit (X2 baseline) +54.0 [+48.1,+59.9] p=0.000 +54.0 [+48.0,+60.0] p=0.000 +0.7 [ +0.0, +1.7] p=0.501 +0.7 [ +0.0, +1.7] p=0.497 +0.7 [ +0.0, +1.7] p=0.500
|
| 106 |
+
refine_c3 gen - retr_learned +52.0 [+46.1,+58.0] p=0.000 +52.0 [+46.0,+57.9] p=0.000 +0.7 [ +0.0, +1.7] p=0.496 +0.7 [ +0.0, +1.7] p=0.497 +0.7 [ +0.0, +1.7] p=0.499
|
| 107 |
+
refine_c3 gen - twin (3 members x 3 draw seeds) +9.0 [ +6.0,+12.0] p=0.000 +9.0 [ +6.0,+12.0] p=0.000 +0.7 [ +0.1, +1.6] p=0.121 +0.4 [ +0.0, +1.2] p=0.248 +0.4 [ +0.0, +1.2] p=0.248
|
| 108 |
+
rank_refine_c3 gen - retr_unit (X2 baseline) +10.1 [ +6.8,+13.6] p=0.000 +10.1 [ +6.8,+13.7] p=0.000 +0.6 [ +0.0, +1.4] p=0.498 +0.7 [ +0.0, +1.7] p=0.505 +0.7 [ +0.0, +1.7] p=0.503
|
| 109 |
+
rank_refine_c3 gen - retr_learned +5.4 [ +2.9, +8.3] p=0.000 +5.4 [ +2.8, +8.3] p=0.000 -0.1 [ -0.3, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000
|
| 110 |
+
rank_refine_c3 gen - twin (3 members x 3 draw seeds) +3.1 [ +1.6, +4.8] p=0.000 +3.1 [ +1.7, +4.9] p=0.000 +0.3 [ +0.0, +0.8] p=0.498 +0.4 [ +0.0, +1.1] p=0.503 +0.4 [ +0.0, +1.1] p=0.504
|
| 111 |
+
|
| 112 |
+
## Across training seeds (unit = seed; frozen pipeline rank_refine; seeds 1-4 use sampler seed 0 only, seed 0 is the mean over its 3 sampler seeds from the main table)
|
| 113 |
+
b=1: generator per seed [99.1, 99.7, 99.3, 99.7, 99.7]
|
| 114 |
+
minus retr_unit: per seed [10.1, 10.7, 10.3, 10.7, 10.7] mean +10.5 t95 [+10.2,+10.8] seeds positive 5/5
|
| 115 |
+
minus retr_learned: per seed [5.4, 6.0, 5.7, 6.0, 6.0] mean +5.8 t95 [+5.5,+6.1] seeds positive 5/5
|
| 116 |
+
minus twin: per seed [3.1, 3.7, 3.3, 3.7, 3.7] mean +3.5 t95 [+3.2,+3.8] seeds positive 5/5
|
| 117 |
+
b=4: generator per seed [100.0, 100.0, 100.0, 100.0, 100.0]
|
| 118 |
+
minus retr_unit: per seed [0.7, 0.7, 0.7, 0.7, 0.7] mean +0.7 t95 [+0.7,+0.7] seeds positive 5/5
|
| 119 |
+
minus retr_learned: per seed [0.0, 0.0, 0.0, 0.0, 0.0] mean +0.0 t95 [+0.0,+0.0] seeds positive 0/5
|
| 120 |
+
minus twin: per seed [0.4, 0.4, 0.4, 0.4, 0.4] mean +0.4 t95 [+0.4,+0.4] seeds positive 5/5
|
| 121 |
+
b=16: generator per seed [100.0, 100.0, 100.0, 100.0, 100.0]
|
| 122 |
+
minus retr_unit: per seed [0.7, 0.7, 0.7, 0.7, 0.7] mean +0.7 t95 [+0.7,+0.7] seeds positive 5/5
|
| 123 |
+
minus retr_learned: per seed [0.0, 0.0, 0.0, 0.0, 0.0] mean +0.0 t95 [+0.0,+0.0] seeds positive 0/5
|
| 124 |
+
minus twin: per seed [0.3, 0.3, 0.3, 0.3, 0.3] mean +0.3 t95 [+0.3,+0.3] seeds positive 5/5
|
| 125 |
+
|
| 126 |
+
# X18: twin ensemble (plain verification, no repair)
|
| 127 |
+
arm (plain) b=1 b=3 b=4 b=8 b=16
|
| 128 |
+
generator 94.0 [91.9,96.0] 97.7 [96.1,99.0] 98.2 [96.9,99.3] 99.2 [98.2,100.0] 99.3 [98.3,100.0]
|
| 129 |
+
twin ensemble (3 members, 9 replicas) 85.0 [81.8,88.1] 95.3 [92.7,97.7] 95.7 [93.4,97.8] 96.9 [94.9,98.6] 97.7 [96.1,99.1]
|
| 130 |
+
twin single member, MC dropout (9 replicas) 85.0 [81.8,88.0] 90.7 [88.1,93.1] 91.9 [89.4,94.1] 94.9 [92.9,96.7] 96.2 [94.3,97.9]
|
| 131 |
+
|
| 132 |
+
## X18 PRIMARY family (4 tests, Holm), pp [95% CI], sign-flip p, Holm p, McNemar (first replicas) p
|
| 133 |
+
ensemble - single-member twin, b=4 +3.9 [ +2.5, +5.3] p=0.0000 holm=0.0000 McNemar 6/1 p=0.1250
|
| 134 |
+
ensemble - single-member twin, b=16 +1.5 [ +0.8, +2.3] p=0.0000 holm=0.0000 McNemar 2/0 p=0.5000
|
| 135 |
+
generator - ensemble, b=4 +2.5 [ +0.6, +4.6] p=0.0176 holm=0.0353 McNemar 10/2 p=0.0386
|
| 136 |
+
generator - ensemble, b=16 +1.6 [ +0.4, +3.1] p=0.0213 holm=0.0353 McNemar 4/0 p=0.1250
|
| 137 |
+
|
| 138 |
+
## X18 SECONDARY (unadjusted), every b: ensemble - single twin | generator - ensemble | generator - single twin
|
| 139 |
+
b= 1: +0.0 [ +0.0, +0.0] p=1.000 | +9.0 [ +6.0,+12.1] p=0.000 | +9.0 [ +5.9,+12.1] p=0.000
|
| 140 |
+
b= 2: +4.1 [ +2.9, +5.4] p=0.000 | +4.6 [ +2.2, +7.1] p=0.000 | +8.7 [ +6.1,+11.3] p=0.000
|
| 141 |
+
b= 3: +4.7 [ +3.1, +6.3] p=0.000 | +2.3 [ +0.1, +4.8] p=0.059 | +7.0 [ +4.7, +9.4] p=0.000
|
| 142 |
+
b= 4: +3.9 [ +2.6, +5.3] p=0.000 | +2.5 [ +0.6, +4.6] p=0.016 | +6.3 [ +4.3, +8.7] p=0.000
|
| 143 |
+
b= 6: +2.6 [ +1.6, +3.7] p=0.000 | +2.4 [ +0.9, +4.2] p=0.006 | +5.0 [ +3.2, +7.0] p=0.000
|
| 144 |
+
b= 8: +2.0 [ +1.1, +2.9] p=0.000 | +2.4 [ +0.9, +4.1] p=0.003 | +4.3 [ +2.7, +6.2] p=0.000
|
| 145 |
+
b=12: +1.3 [ +0.5, +2.1] p=0.001 | +2.3 [ +0.9, +4.0] p=0.004 | +3.6 [ +2.1, +5.4] p=0.000
|
| 146 |
+
b=16: +1.5 [ +0.8, +2.3] p=0.000 | +1.6 [ +0.4, +3.1] p=0.024 | +3.1 [ +1.6, +4.8] p=0.000
|
| 147 |
+
|
| 148 |
+
## X18 with repair (secondary): ensemble vs single twin vs generator at b = 1, 4, 16, pipelines refine / rank_refine
|
| 149 |
+
refine: ens 85.0 99.7 99.7 | twin 85.0 99.6 99.7 | gen 94.0 100.0 100.0
|
| 150 |
+
rank_refine: ens 97.7 99.7 99.7 | twin 96.0 99.6 99.7 | gen 99.1 100.0 100.0
|
| 151 |
+
|
| 152 |
+
## Check: plain pipeline vs X2 (success % at b = 1, 4, 16; X2 values from results/icml_r3/X2/test_*.npz)
|
| 153 |
+
gen here 94.0 98.2 99.3 | X2 94.1 98.2 99.3
|
| 154 |
+
retr here 40.0 71.3 93.3 | X2 40.0 71.3 93.3
|
results/icml_r3/X8/val_selection.txt
ADDED
|
@@ -0,0 +1,53 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# X8 validation (60 plants; gen seed 0 sampler 0, retr_unit, retr_learned (metric xfer), twin members 0-2 draw seed 0)
|
| 2 |
+
|
| 3 |
+
refine config it40_lr0.1_n200: rescued 2097 of 2187 certified-failed candidates = 95.9% (per arm: gen0_s0 70/76, retr_unit 761/797, retr_learned 713/747, twin0_d0 195/195, twin1_d0 160/174, twin2_d0 198/198)
|
| 4 |
+
refine config it80_lr0.1_n200: rescued 2141 of 2187 certified-failed candidates = 97.9% (per arm: gen0_s0 75/76, retr_unit 777/797, retr_learned 722/747, twin0_d0 195/195, twin1_d0 174/174, twin2_d0 198/198)
|
| 5 |
+
refine config it40_lr0.2_n200: rescued 2139 of 2187 certified-failed candidates = 97.8% (per arm: gen0_s0 76/76, retr_unit 775/797, retr_learned 721/747, twin0_d0 195/195, twin1_d0 174/174, twin2_d0 198/198)
|
| 6 |
+
refine config it80_lr0.2_n200: rescued 2135 of 2187 certified-failed candidates = 97.6% (per arm: gen0_s0 76/76, retr_unit 775/797, retr_learned 717/747, twin0_d0 195/195, twin1_d0 174/174, twin2_d0 198/198)
|
| 7 |
+
refine config it40_lr0.1_n400: rescued 2097 of 2187 certified-failed candidates = 95.9% (per arm: gen0_s0 70/76, retr_unit 760/797, retr_learned 714/747, twin0_d0 195/195, twin1_d0 160/174, twin2_d0 198/198)
|
| 8 |
+
FROZEN refine config: {'iters': 80, 'lr': 0.1, 'n': 200} (highest pooled rescue rate; ties -> cheaper = earlier in grid)
|
| 9 |
+
|
| 10 |
+
# pipelines on validation, success % at b = 1/2/4/8/16 per arm (frozen refine config)
|
| 11 |
+
plain gen0_s0 88.3 93.3 98.3 98.3 98.3
|
| 12 |
+
plain retr_unit 31.7 50.0 73.3 85.0 93.3
|
| 13 |
+
plain retr_learned 35.0 55.0 70.0 88.3 95.0
|
| 14 |
+
plain twin0_d0 83.3 85.0 85.0 91.7 95.0
|
| 15 |
+
plain twin1_d0 86.7 95.0 98.3 98.3 98.3
|
| 16 |
+
plain twin2_d0 81.7 90.0 90.0 93.3 95.0
|
| 17 |
+
plain baseline mean over b=1,4,16 (retr_unit, retr_learned, twin): 74.38
|
| 18 |
+
refine gen0_s0 88.3 100.0 100.0 100.0 100.0
|
| 19 |
+
refine retr_unit 31.7 100.0 100.0 100.0 100.0
|
| 20 |
+
refine retr_learned 35.0 100.0 100.0 100.0 100.0
|
| 21 |
+
refine twin0_d0 83.3 100.0 100.0 100.0 100.0
|
| 22 |
+
refine twin1_d0 86.7 100.0 100.0 100.0 100.0
|
| 23 |
+
refine twin2_d0 81.7 100.0 100.0 100.0 100.0
|
| 24 |
+
refine baseline mean over b=1,4,16 (retr_unit, retr_learned, twin): 83.40
|
| 25 |
+
rank gen0_s0 98.3 98.3 98.3 98.3 98.3
|
| 26 |
+
rank retr_unit 83.3 88.3 93.3 93.3 93.3
|
| 27 |
+
rank retr_learned 88.3 93.3 93.3 95.0 95.0
|
| 28 |
+
rank twin0_d0 93.3 93.3 93.3 93.3 95.0
|
| 29 |
+
rank twin1_d0 98.3 98.3 98.3 98.3 98.3
|
| 30 |
+
rank twin2_d0 93.3 93.3 93.3 93.3 95.0
|
| 31 |
+
rank baseline mean over b=1,4,16 (retr_unit, retr_learned, twin): 92.53
|
| 32 |
+
rank_refine gen0_s0 98.3 100.0 100.0 100.0 100.0
|
| 33 |
+
rank_refine retr_unit 83.3 100.0 100.0 100.0 100.0
|
| 34 |
+
rank_refine retr_learned 88.3 100.0 100.0 100.0 100.0
|
| 35 |
+
rank_refine twin0_d0 93.3 100.0 100.0 100.0 100.0
|
| 36 |
+
rank_refine twin1_d0 98.3 100.0 100.0 100.0 100.0
|
| 37 |
+
rank_refine twin2_d0 93.3 100.0 100.0 100.0 100.0
|
| 38 |
+
rank_refine baseline mean over b=1,4,16 (retr_unit, retr_learned, twin): 96.30
|
| 39 |
+
refine_free gen0_s0 100.0 100.0 100.0 100.0 100.0
|
| 40 |
+
refine_free retr_unit 100.0 100.0 100.0 100.0 100.0
|
| 41 |
+
refine_free retr_learned 100.0 100.0 100.0 100.0 100.0
|
| 42 |
+
refine_free twin0_d0 100.0 100.0 100.0 100.0 100.0
|
| 43 |
+
refine_free twin1_d0 100.0 100.0 100.0 100.0 100.0
|
| 44 |
+
refine_free twin2_d0 100.0 100.0 100.0 100.0 100.0
|
| 45 |
+
refine_free baseline mean over b=1,4,16 (retr_unit, retr_learned, twin): 100.00
|
| 46 |
+
rank_refine_free gen0_s0 100.0 100.0 100.0 100.0 100.0
|
| 47 |
+
rank_refine_free retr_unit 100.0 100.0 100.0 100.0 100.0
|
| 48 |
+
rank_refine_free retr_learned 100.0 100.0 100.0 100.0 100.0
|
| 49 |
+
rank_refine_free twin0_d0 100.0 100.0 100.0 100.0 100.0
|
| 50 |
+
rank_refine_free twin1_d0 100.0 100.0 100.0 100.0 100.0
|
| 51 |
+
rank_refine_free twin2_d0 100.0 100.0 100.0 100.0 100.0
|
| 52 |
+
rank_refine_free baseline mean over b=1,4,16 (retr_unit, retr_learned, twin): 100.00
|
| 53 |
+
FROZEN primary pipeline: rank_refine (higher mean baseline-arm success over b = 1, 4, 16 among refine / rank_refine; ties -> rank_refine)
|