matCercola18 commited on
Commit
7823312
·
verified ·
1 Parent(s): af2076b

results/icml_r3/X8

Browse files
Files changed (45) hide show
  1. results/icml_r3/X8/FROZEN.json +25 -0
  2. results/icml_r3/X8/PREREG.md +38 -0
  3. results/icml_r3/X8/cands/val_unetA0_ddim10_s0_k16.npy +3 -0
  4. results/icml_r3/X8/cands/val_x2_reg_mcd_seed0_div_s0.npy +3 -0
  5. results/icml_r3/X8/cands/val_x2_reg_mcd_seed1_div_s0.npy +3 -0
  6. results/icml_r3/X8/cands/val_x2_reg_mcd_seed2_div_s0.npy +3 -0
  7. results/icml_r3/X8/curves.json +1 -0
  8. results/icml_r3/X8/logs/report.log +154 -0
  9. results/icml_r3/X8/logs/test_main.log +88 -0
  10. results/icml_r3/X8/logs/test_seeds.log +117 -0
  11. results/icml_r3/X8/logs/val.log +181 -0
  12. results/icml_r3/X8/tables/test_gen0.npz +3 -0
  13. results/icml_r3/X8/tables/test_gen0_ref_it80_lr0.1_n200.npz +3 -0
  14. results/icml_r3/X8/tables/test_gen1.npz +3 -0
  15. results/icml_r3/X8/tables/test_gen1_ref_it80_lr0.1_n200.npz +3 -0
  16. results/icml_r3/X8/tables/test_gen2.npz +3 -0
  17. results/icml_r3/X8/tables/test_gen2_ref_it80_lr0.1_n200.npz +3 -0
  18. results/icml_r3/X8/tables/test_gen3.npz +3 -0
  19. results/icml_r3/X8/tables/test_gen3_ref_it80_lr0.1_n200.npz +3 -0
  20. results/icml_r3/X8/tables/test_gen4.npz +3 -0
  21. results/icml_r3/X8/tables/test_gen4_ref_it80_lr0.1_n200.npz +3 -0
  22. results/icml_r3/X8/tables/test_retr.npz +3 -0
  23. results/icml_r3/X8/tables/test_retr_ref_it80_lr0.1_n200.npz +3 -0
  24. results/icml_r3/X8/tables/test_twin.npz +3 -0
  25. results/icml_r3/X8/tables/test_twin_ref_it80_lr0.1_n200.npz +3 -0
  26. results/icml_r3/X8/tables/val_gen0.npz +3 -0
  27. results/icml_r3/X8/tables/val_gen0_ref_it40_lr0.1_n200.npz +3 -0
  28. results/icml_r3/X8/tables/val_gen0_ref_it40_lr0.1_n400.npz +3 -0
  29. results/icml_r3/X8/tables/val_gen0_ref_it40_lr0.2_n200.npz +3 -0
  30. results/icml_r3/X8/tables/val_gen0_ref_it80_lr0.1_n200.npz +3 -0
  31. results/icml_r3/X8/tables/val_gen0_ref_it80_lr0.2_n200.npz +3 -0
  32. results/icml_r3/X8/tables/val_retr.npz +3 -0
  33. results/icml_r3/X8/tables/val_retr_ref_it40_lr0.1_n200.npz +3 -0
  34. results/icml_r3/X8/tables/val_retr_ref_it40_lr0.1_n400.npz +3 -0
  35. results/icml_r3/X8/tables/val_retr_ref_it40_lr0.2_n200.npz +3 -0
  36. results/icml_r3/X8/tables/val_retr_ref_it80_lr0.1_n200.npz +3 -0
  37. results/icml_r3/X8/tables/val_retr_ref_it80_lr0.2_n200.npz +3 -0
  38. results/icml_r3/X8/tables/val_twin.npz +3 -0
  39. results/icml_r3/X8/tables/val_twin_ref_it40_lr0.1_n200.npz +3 -0
  40. results/icml_r3/X8/tables/val_twin_ref_it40_lr0.1_n400.npz +3 -0
  41. results/icml_r3/X8/tables/val_twin_ref_it40_lr0.2_n200.npz +3 -0
  42. results/icml_r3/X8/tables/val_twin_ref_it80_lr0.1_n200.npz +3 -0
  43. results/icml_r3/X8/tables/val_twin_ref_it80_lr0.2_n200.npz +3 -0
  44. results/icml_r3/X8/test_report.txt +154 -0
  45. results/icml_r3/X8/val_selection.txt +53 -0
results/icml_r3/X8/FROZEN.json ADDED
@@ -0,0 +1,25 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "written": "2026-10-13 10:52 PDT",
3
+ "refine_cfg": {
4
+ "iters": 80,
5
+ "lr": 0.1,
6
+ "n": 200
7
+ },
8
+ "pipeline": "rank_refine",
9
+ "learned_metric": "xfer",
10
+ "val_rescue": {
11
+ "it40_lr0.1_n200": 0.9588477366255144,
12
+ "it80_lr0.1_n200": 0.9789666209419295,
13
+ "it40_lr0.2_n200": 0.9780521262002744,
14
+ "it80_lr0.2_n200": 0.9762231367169639,
15
+ "it40_lr0.1_n400": 0.9588477366255144
16
+ },
17
+ "val_pipe_score": {
18
+ "plain": 74.38271604938271,
19
+ "refine": 83.39506172839505,
20
+ "rank": 92.53086419753086,
21
+ "rank_refine": 96.29629629629629,
22
+ "refine_free": 100.0,
23
+ "rank_refine_free": 100.0
24
+ }
25
+ }
results/icml_r3/X8/PREREG.md ADDED
@@ -0,0 +1,38 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # X8 PREREG: equal repair on the corrected arm (updates inventory row D7) (written 2026-10-13 09:06 PDT, before any validation or test scoring of this experiment)
2
+
3
+ virgo CPU only (CUDA_VISIBLE_DEVICES empty), <= 12 workers, RAYON_NUM_THREADS=1. Code: `nonlinear/experiments/furuta_x8.py`, `furuta_r3b_engine.py`, `furuta_r3b_arms.py`, `furuta_r3b_refine.py` (the v4 refinement re-pointed at the corrected arm; the torch RK4 matches the verifier's CasADi step to 7e-15, `python -m nonlinear.experiments.furuta_r3b_refine`).
4
+ Protocol = X2 (same plants val 51001 x60, test 71001+71002 x300, same verifier, bank `data/x2_bank3000.npz` = 2905 rows).
5
+
6
+ ## Question
7
+ D7 (old arm, n = 120): with the SAME local refinement and ranking given to the generator and to retrieval the margin shrinks to <= 4 points. Does that hold on the corrected arm at n = 300 with refinement verifier calls COUNTED toward the budget, and for the regression twin as well?
8
+
9
+ ## Arms and the repair procedure (identical for every arm)
10
+ - Each arm provides 16 ordered candidates per plant: **gen** = `unet_a` seed 0 (S1 config s96000 ema999, DDIM-10), the first 16 samples of each of the 3 cached X2 sampler seeds (3 replicas);
11
+ **retr_unit** = the 16 nearest bank rows in the X2 metric (deterministic); **retr_learned** = the 16 nearest rows under the learned metric of X10F (the metric, ard or xfer, with the higher mean X10F validation kNN rate over b = 1, 4, 16; ties -> ard);
12
+ **twin** = for twin member i in {0, 1, 2} and draw seed s in {0, 1, 2}: [deterministic prediction, 15 MC-dropout draws (jitter 0)] (9 replicas, the cached X2 draws).
13
+ - Verifier call = one verification of one candidate (`certified_catch_set` re-solving P for the candidate's K on the target plant, then `score_solution`, 8 s, RK4 2 ms, 10 V clip). A candidate that is certified but fails is **refined** once: 40 Adam steps (config frozen on validation) on the 101 voltage knots of u* through the differentiable open-loop simulator of the TARGET plant (corrected arm), objective soft-min over time of log(1 + x_w'Px_w) with the re-solved P; K, T, P unchanged; then verified again with the unchanged verifier. That re-verification **counts as one more verifier call**.
14
+ - Pipelines (success at budget b = a success within the first b verifier calls, b = 1..16): `plain` (given order); `refine` (given order, refinement, counted); `rank` (order by the open-loop depth min_t x_w'Px_w of each candidate's reference trajectory under the candidate's OWN P; no verifier call); `rank_refine` (rank, then refine, counted).
15
+ Secondary D7-style accounting `*_free` (the refined re-verification not counted) is reported as the upper bound of what repair can give. The refinement compute is not charged to b (it is not a verifier call); its wall-clock is reported descriptively.
16
+
17
+ ## Selection on validation only (FROZEN.json written before any test scoring)
18
+ Validation arms: gen seed 0 sampler seed 0 (16 fresh samples per val plant, cached in X8/cands), retr_unit, retr_learned, twin members 0-2 draw seed 0.
19
+ 1. Refinement configuration from {(iters, lr, n_substeps)} = {(40, 0.1, 200), (80, 0.1, 200), (40, 0.2, 200), (80, 0.2, 200), (40, 0.1, 400)}: the one with the highest pooled rescue rate (rescued / certified-failed candidates, all arms pooled); ties -> earlier in the list (cheaper).
20
+ 2. Primary pipeline from {refine, rank_refine}: the one with the higher mean over the baseline arms (retr_unit, retr_learned, twin) of the success averaged over b = 1, 4, 16 (the pipeline that is strongest for the baselines); ties -> rank_refine.
21
+ All four counted/uncounted pipelines are reported on test regardless; the primary family below uses the frozen pipeline only.
22
+
23
+ ## Test endpoints and tests
24
+ - Primary family, **9 tests**, Holm at alpha 0.05: (generator minus retr_unit, minus retr_learned, minus twin) x b in {1, 4, 16} under the frozen pipeline. Generator per plant = mean over its 3 sampler seeds; twin = mean over its 9 replicas; retrievals deterministic.
25
+ Statistic: mean paired per-plant difference (pp), percentile bootstrap CI over plants (10,000), two-sided sign-flip p (20,000). Holm is applied inside this family only.
26
+ - Secondary (unadjusted): the same differences for all pipelines and for b = 1..16 (budget curves); gen minus each arm under `plain` (reproduces X2 P1-P5 as a check); the D7 comparison (`*_free` at K = 1, 4); across training seeds: generator seeds 1-4 (X2 cached samples, 3 sampler seeds) under the frozen pipeline, difference to retr_unit / retr_learned / twin with the TRAINING SEED as the resampling unit (mean, t-interval over 5 seeds, seeds positive); no pooled-over-seeds McNemar.
27
+ - Falsification: the claim "the generator's lead survives equal repair" is refuted for a cell if its CI contains 0 (b = 1 and 4 matter for the low-budget claim; b = 16 for the ceiling claim). If retrieval or the twin catches up under repair the result is reported as the corrected-arm update of D7 and the claim is narrowed accordingly.
28
+ - Not covered: arms other than those listed (flow, MDN, WTA, cVAE are X1F on the old arm); repair beyond one refinement per candidate; compute-matched (wall-clock) accounting.
29
+
30
+ ## Addendum 2026-10-13 10:53 PDT: validation done, configuration and pipeline frozen (no test scoring before this line)
31
+ `furuta_x8 val` (log results/icml_r3/X8/logs/val.log, val_selection.txt) finished at 10:52 and wrote FROZEN.json: refinement config (iters 80, lr 0.1, n 200), pooled validation rescue 97.9% (2141 of 2187 certified-failed candidates; the five configs span 95.9 to 97.9%), primary pipeline `rank_refine` (mean baseline-arm validation success over b = 1, 4, 16: 96.30 against 83.40 for `refine`), learned metric of retr_learned = `xfer` (X10F validation mean over b = 1, 4, 16: 66.7 against 65.6 for `ard`).
32
+ Validation also shows that with counted refinement every arm reaches 100% at b = 2 (`refine`) or b >= 2 (`rank_refine`) on the 60 validation plants; the budget cells b = 4 and b = 16 of the primary family are therefore expected to be saturated (difference 0, p = 1). A descriptive extra (not in the primary family) is added to the report: mean number of verifier calls to the first success (a failure counts as 17), per arm and pipeline, because it still discriminates when the success rates saturate.
33
+
34
+ ## Addendum 2026-10-13 11:15 PDT (compute only, before any test result is read)
35
+ To fit the time budget the across-seed secondary (generator training seeds 1-4) uses ONE sampler seed (0) per seed instead of three. Seed 0 keeps its three sampler seeds. The seed is still the resampling unit. The primary family is unchanged.
36
+
37
+ ## Addendum 2026-10-13 11:30 PDT (secondary sensitivity, added before the twin test tables existed and before any test statistic was computed; the generator and retrieval tables had been built, one unadjusted preview of their success rates was printed)
38
+ Refinement is not free in CPU time: on the test candidates a refinement batch costs 1.3 to 1.4 CPU-s per candidate (80 Adam steps, batches of 24, loaded machine) against 0.49 CPU-s for a plain verifier call (mean over the test run). Two extra pipelines `refine_c3` and `rank_refine_c3` charge one refinement attempt (refinement plus re-verification) 3 verifier calls, a compute-matched sensitivity. They are secondary and unadjusted; the primary family and its pipeline (`rank_refine`, one call per attempt) are unchanged.
results/icml_r3/X8/cands/val_unetA0_ddim10_s0_k16.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:651d9fb81fe76d54518582292d3bdadd8cf6446496db9cc173b5af89eac59052
3
+ size 445568
results/icml_r3/X8/cands/val_x2_reg_mcd_seed0_div_s0.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d0debbbcfb6e6f1d97ab5c41bcd037ff1e46e7dffcbe0953552a4c48e94b8880
3
+ size 445568
results/icml_r3/X8/cands/val_x2_reg_mcd_seed1_div_s0.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0d96d5d3a2386ef667f1b55f1cc500de49e7dc0848233c1d6f7792f9b0067457
3
+ size 445568
results/icml_r3/X8/cands/val_x2_reg_mcd_seed2_div_s0.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7aafc5c70c9d877428df8286cac03912eaaad0dd7dc98f99210317daf4eb4b8d
3
+ size 445568
results/icml_r3/X8/curves.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"plain": {"generator (seed 0, 3 sampler seeds)": {"mean": [94.0, 97.11111111111113, 97.66666666666669, 98.22222222222223, 98.77777777777777, 99.0, 99.0, 99.22222222222221, 99.22222222222221, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333], "ci": [[91.88888888888887, 95.44444444444444, 96.21944444444443, 96.88888888888889, 97.55555555555556, 97.88888888888889, 97.88888888888889, 98.11111111111113, 98.11111111111113, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333], [96.0, 98.55555555555554, 99.0, 99.33333333333333, 99.66666666666667, 99.88888888888887, 99.88888888888887, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_unit (X2 baseline)": {"mean": [40.0, 56.99999999999999, 66.33333333333333, 71.33333333333334, 74.66666666666667, 79.33333333333333, 82.0, 85.0, 86.33333333333333, 88.66666666666667, 89.0, 90.0, 91.0, 91.66666666666666, 92.66666666666666, 93.33333333333333], "ci": [[34.333333333333336, 51.33333333333333, 61.0, 66.0, 69.66666666666667, 74.66666666666667, 77.66666666666666, 81.0, 82.33333333333334, 85.0, 85.33333333333334, 86.66666666666667, 87.66666666666667, 88.33333333333333, 89.66666666666666, 90.33333333333333], [45.666666666666664, 62.66666666666667, 71.66666666666667, 76.33333333333333, 79.66666666666666, 84.0, 86.33333333333333, 89.0, 90.0, 92.0, 92.33333333333333, 93.33333333333333, 94.0, 94.66666666666667, 95.66666666666667, 96.0]]}, "retr_learned": {"mean": [42.0, 61.0, 71.33333333333334, 76.66666666666667, 82.0, 86.66666666666667, 89.66666666666666, 90.66666666666666, 92.0, 92.33333333333333, 93.0, 94.0, 94.33333333333334, 94.66666666666667, 95.0, 96.66666666666667], "ci": [[36.333333333333336, 55.333333333333336, 66.0, 72.0, 77.66666666666666, 82.66666666666667, 86.0, 87.33333333333333, 88.66666666666667, 89.33333333333333, 90.0, 91.0, 91.66666666666666, 92.0, 92.33333333333333, 94.33333333333334], [47.66666666666667, 66.66666666666666, 76.33333333333333, 81.33333333333333, 86.33333333333333, 90.33333333333333, 93.0, 93.66666666666667, 95.0, 95.33333333333334, 95.66666666666667, 96.66666666666667, 96.66666666666667, 97.0, 97.33333333333334, 98.66666666666667]]}, "twin (3 members x 3 draw seeds)": {"mean": [84.99999999999997, 88.40740740740736, 90.66666666666666, 91.88888888888886, 92.8518518518518, 93.96296296296295, 94.44444444444441, 94.88888888888887, 95.1111111111111, 95.37037037037038, 95.66666666666664, 95.7037037037037, 95.88888888888889, 96.0, 96.14814814814815, 96.22222222222223], "ci": [[81.8861111111111, 85.59259259259258, 88.07407407407408, 89.44444444444444, 90.5925925925926, 91.81481481481481, 92.4074074074074, 92.81481481481481, 93.1111111111111, 93.37037037037035, 93.74074074074072, 93.7037037037037, 93.96296296296295, 94.11111111111113, 94.33333333333334, 94.2962962962963], [88.0, 91.07407407407405, 93.00092592592591, 94.11203703703703, 94.88981481481481, 95.85185185185185, 96.2962962962963, 96.7037037037037, 96.88888888888889, 97.11111111111113, 97.37037037037037, 97.4074074074074, 97.59259259259258, 97.62962962962963, 97.81481481481482, 97.81481481481482]]}}, "refine": {"generator (seed 0, 3 sampler seeds)": {"mean": [94.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], "ci": [[91.77777777777779, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], [96.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_unit (X2 baseline)": {"mean": [40.0, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333], "ci": [[34.333333333333336, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333], [45.666666666666664, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_learned": {"mean": [42.0, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667], "ci": [[36.333333333333336, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 99.0, 99.0, 99.0, 99.0], [47.66666666666667, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "twin (3 members x 3 draw seeds)": {"mean": [84.99999999999997, 99.33333333333333, 99.33333333333333, 99.55555555555554, 99.55555555555554, 99.55555555555554, 99.55555555555554, 99.55555555555554, 99.55555555555554, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.66666666666667, 99.66666666666667, 99.66666666666667], "ci": [[81.88888888888887, 98.44444444444446, 98.44444444444446, 98.77777777777777, 98.77777777777777, 98.77777777777777, 98.77777777777777, 98.81481481481482, 98.77777777777777, 98.85185185185185, 98.85185185185185, 98.85185185185185, 98.88888888888889, 99.0, 99.0, 99.0], [88.0, 99.8888888888889, 99.8888888888889, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}}, "rank": {"generator (seed 0, 3 sampler seeds)": {"mean": [99.11111111111113, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333], "ci": [[98.0, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333], [99.8888888888889, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_unit (X2 baseline)": {"mean": [89.0, 91.66666666666666, 92.33333333333333, 92.66666666666666, 92.66666666666666, 92.66666666666666, 92.66666666666666, 92.66666666666666, 93.0, 93.0, 93.0, 93.0, 93.0, 93.0, 93.33333333333333, 93.33333333333333], "ci": [[85.33333333333334, 88.33333333333333, 89.33333333333333, 89.66666666666666, 89.66666666666666, 89.66666666666666, 89.66666666666666, 89.66666666666666, 90.0, 90.0, 90.0, 90.0, 90.0, 90.0, 90.33333333333333, 90.33333333333333], [92.33333333333333, 94.66666666666667, 95.0, 95.33333333333334, 95.33333333333334, 95.33333333333334, 95.33333333333334, 95.33333333333334, 95.66666666666667, 95.66666666666667, 95.66666666666667, 95.66666666666667, 95.66666666666667, 95.66666666666667, 96.0, 96.0]]}, "retr_learned": {"mean": [93.66666666666667, 95.0, 96.33333333333334, 96.66666666666667, 96.66666666666667, 96.66666666666667, 96.66666666666667, 96.66666666666667, 96.66666666666667, 96.66666666666667, 96.66666666666667, 96.66666666666667, 96.66666666666667, 96.66666666666667, 96.66666666666667, 96.66666666666667], "ci": [[90.66666666666666, 92.33333333333333, 94.0, 94.33333333333334, 94.33333333333334, 94.33333333333334, 94.66666666666667, 94.33333333333334, 94.66666666666667, 94.33333333333334, 94.66666666666667, 94.66666666666667, 94.66666666666667, 94.66666666666667, 94.33333333333334, 94.33333333333334], [96.33333333333334, 97.33333333333334, 98.33333333333333, 98.66666666666667, 98.66666666666667, 98.66666666666667, 98.66666666666667, 98.66666666666667, 98.66666666666667, 98.66666666666667, 98.66666666666667, 98.66666666666667, 98.66666666666667, 98.66666666666667, 98.66666666666667, 98.66666666666667]]}, "twin (3 members x 3 draw seeds)": {"mean": [96.0, 96.0, 96.03703703703704, 96.11111111111113, 96.11111111111113, 96.11111111111113, 96.11111111111113, 96.14814814814815, 96.14814814814815, 96.14814814814815, 96.14814814814815, 96.14814814814815, 96.14814814814815, 96.14814814814815, 96.18518518518519, 96.22222222222223], "ci": [[94.18518518518518, 94.14722222222221, 94.18518518518518, 94.18518518518518, 94.18518518518518, 94.22222222222223, 94.1851851851852, 94.25925925925925, 94.2962962962963, 94.18518518518518, 94.25925925925925, 94.22222222222223, 94.25925925925928, 94.25925925925925, 94.33333333333334, 94.37037037037037], [97.62962962962963, 97.7037037037037, 97.74074074074073, 97.77777777777777, 97.74074074074073, 97.77777777777777, 97.74074074074073, 97.77777777777779, 97.74074074074073, 97.77777777777777, 97.81481481481482, 97.7787037037037, 97.77777777777779, 97.85185185185185, 97.81481481481482, 97.85185185185185]]}}, "rank_refine": {"generator (seed 0, 3 sampler seeds)": {"mean": [99.11111111111113, 99.8888888888889, 99.8888888888889, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], "ci": [[98.0, 99.66666666666667, 99.66666666666667, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], [99.8888888888889, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_unit (X2 baseline)": {"mean": [89.0, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333], "ci": [[85.33333333333334, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333], [92.33333333333333, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_learned": {"mean": [93.66666666666667, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], "ci": [[90.66666666666666, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], [96.33333333333334, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "twin (3 members x 3 draw seeds)": {"mean": [96.0, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.62962962962963, 99.62962962962963, 99.62962962962963, 99.62962962962963, 99.62962962962963, 99.62962962962963, 99.62962962962963, 99.62962962962963, 99.66666666666667], "ci": [[94.11111111111113, 98.85185185185185, 98.85185185185185, 98.85185185185185, 98.85185185185185, 98.85185185185185, 98.85185185185185, 98.92592592592592, 98.92592592592592, 98.92592592592592, 98.92592592592592, 98.92592592592592, 98.92592592592592, 98.92592592592592, 98.88888888888889, 99.0], [97.7037037037037, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}}, "refine_free": {"generator (seed 0, 3 sampler seeds)": {"mean": [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], "ci": [[100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_unit (X2 baseline)": {"mean": [99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333], "ci": [[98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333], [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_learned": {"mean": [99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.66666666666667, 99.66666666666667, 99.66666666666667, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], "ci": [[98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 99.0, 99.0, 99.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "twin (3 members x 3 draw seeds)": {"mean": [99.33333333333333, 99.55555555555554, 99.55555555555554, 99.55555555555554, 99.59259259259258, 99.59259259259258, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667], "ci": [[98.44444444444446, 98.77777777777777, 98.77685185185186, 98.77777777777777, 98.85185185185185, 98.85185185185185, 99.0, 99.0, 99.0, 99.0, 99.0, 99.0, 99.0, 99.0, 99.0, 99.0], [99.8888888888889, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}}, "rank_refine_free": {"generator (seed 0, 3 sampler seeds)": {"mean": [99.8888888888889, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], "ci": [[99.66666666666667, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_unit (X2 baseline)": {"mean": [99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333], "ci": [[98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333], [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_learned": {"mean": [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], "ci": [[100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "twin (3 members x 3 draw seeds)": {"mean": [99.59259259259258, 99.59259259259258, 99.59259259259258, 99.62962962962963, 99.62962962962963, 99.62962962962963, 99.62962962962963, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667], "ci": [[98.85185185185185, 98.77777777777777, 98.77777777777777, 98.92592592592592, 98.92592592592592, 98.92592592592592, 98.92592592592592, 99.0, 99.0, 99.0, 99.0, 99.0, 99.0, 99.0, 99.0, 99.0], [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}}, "refine_c3": {"generator (seed 0, 3 sampler seeds)": {"mean": [94.0, 94.0, 94.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], "ci": [[91.77777777777779, 91.77777777777779, 91.77777777777779, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], [96.0, 96.0, 96.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_unit (X2 baseline)": {"mean": [40.0, 40.0, 40.0, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333], "ci": [[34.333333333333336, 34.333333333333336, 34.66666666666667, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333], [45.666666666666664, 45.666666666666664, 45.666666666666664, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_learned": {"mean": [42.0, 42.0, 42.0, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333], "ci": [[36.333333333333336, 36.333333333333336, 36.333333333333336, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333], [47.66666666666667, 47.66666666666667, 47.66666666666667, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "twin (3 members x 3 draw seeds)": {"mean": [84.99999999999997, 84.99999999999997, 84.99999999999997, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.55555555555554, 99.55555555555554, 99.55555555555554, 99.55555555555554, 99.55555555555554, 99.55555555555554, 99.55555555555554, 99.55555555555554, 99.55555555555554], "ci": [[81.77777777777777, 81.77777777777777, 81.77777777777777, 98.44444444444446, 98.44444444444446, 98.44444444444443, 98.44444444444446, 98.77777777777777, 98.77777777777777, 98.77777777777777, 98.77777777777777, 98.77777777777777, 98.77777777777777, 98.77777777777777, 98.77777777777777, 98.77777777777777], [88.1111111111111, 88.1111111111111, 88.1111111111111, 99.8888888888889, 99.8888888888889, 99.8888888888889, 99.8888888888889, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}}, "rank_refine_c3": {"generator (seed 0, 3 sampler seeds)": {"mean": [99.11111111111113, 99.11111111111113, 99.11111111111113, 99.8888888888889, 99.8888888888889, 99.8888888888889, 99.8888888888889, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], "ci": [[98.0, 98.0, 98.0, 99.66666666666667, 99.66666666666667, 99.66666666666667, 99.66666666666667, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], [99.8888888888889, 99.8888888888889, 99.8888888888889, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_unit (X2 baseline)": {"mean": [89.0, 89.0, 89.0, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333, 99.33333333333333], "ci": [[85.33333333333334, 85.33333333333334, 85.33333333333334, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333, 98.33333333333333], [92.33333333333333, 92.33333333333333, 92.33333333333333, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "retr_learned": {"mean": [93.66666666666667, 93.66666666666667, 93.66666666666667, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], "ci": [[90.66666666666666, 90.66666666666666, 90.66666666666666, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0], [96.33333333333334, 96.33333333333334, 96.33333333333334, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}, "twin (3 members x 3 draw seeds)": {"mean": [96.0, 96.0, 96.0, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.59259259259258, 99.62962962962963], "ci": [[94.07407407407408, 94.11111111111113, 94.11111111111113, 98.85185185185185, 98.85185185185185, 98.85185185185185, 98.85185185185185, 98.85185185185185, 98.85185185185185, 98.77777777777777, 98.77777777777777, 98.85185185185185, 98.85185185185185, 98.85185185185185, 98.85185185185185, 98.92592592592592], [97.66666666666667, 97.62962962962962, 97.66666666666667, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0]]}}, "ensemble": {"plain": {"ensemble": {"mean": [84.99999999999997, 92.55555555555553, 95.33333333333334, 95.74074074074076, 96.14814814814815, 96.55555555555554, 96.7037037037037, 96.85185185185185, 97.0, 97.0, 97.0, 97.0, 97.22222222222223, 97.44444444444443, 97.66666666666664, 97.74074074074073], "ci": [[81.88888888888887, 90.0, 92.66666666666666, 93.44444444444446, 93.92592592592595, 94.55555555555554, 94.7037037037037, 94.92592592592592, 95.1111111111111, 95.1111111111111, 95.1111111111111, 95.0, 95.37037037037037, 95.62962962962965, 96.0, 96.03703703703704], [88.1111111111111, 95.0, 97.66666666666667, 97.81481481481482, 98.03703703703704, 98.33333333333333, 98.44444444444446, 98.55555555555556, 98.66666666666667, 98.66666666666667, 98.66666666666667, 98.66666666666667, 98.81481481481482, 98.92592592592592, 99.1111111111111, 99.14814814814815]]}}}}
results/icml_r3/X8/logs/report.log ADDED
@@ -0,0 +1,154 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # X8 / X18 test report (corrected arm, n=300; bank1 = 2905 rows); refine config {'iters': 80, 'lr': 0.1, 'n': 200}; frozen primary pipeline rank_refine; retr_learned metric = xfer
2
+
3
+ ## Refinement statistics on the test candidates (per arm: valid, certified, certified-but-failed, rescued by refinement)
4
+ generator valid 14400 certified 14400 certified-failed 793 rescued 786 (99.1%)
5
+ retr_unit valid 4800 certified 4800 certified-failed 3947 rescued 3893 (98.6%)
6
+ retr_learned valid 4800 certified 4800 certified-failed 3787 rescued 3717 (98.2%)
7
+ twin valid 43200 certified 43200 certified-failed 8003 rescued 7758 (96.9%)
8
+
9
+ ## Test success % [95% CI over plants], pipeline = plain
10
+ arm b=1 b=2 b=4 b=8 b=16
11
+ generator (seed 0, 3 sampler seeds) 94.0 [91.8,96.0] 97.1 [95.4,98.6] 98.2 [96.9,99.3] 99.2 [98.1,100.0] 99.3 [98.3,100.0]
12
+ retr_unit (X2 baseline) 40.0 [34.7,45.7] 57.0 [51.3,62.3] 71.3 [66.3,76.3] 85.0 [81.0,89.0] 93.3 [90.3,96.0]
13
+ retr_learned 42.0 [36.3,47.3] 61.0 [55.3,66.3] 76.7 [72.0,81.3] 90.7 [87.3,93.7] 96.7 [94.7,98.7]
14
+ twin (3 members x 3 draw seeds) 85.0 [81.9,88.0] 88.4 [85.6,91.0] 91.9 [89.5,94.1] 94.9 [92.9,96.7] 96.2 [94.3,97.9]
15
+
16
+ ## Test success % [95% CI over plants], pipeline = refine
17
+ arm b=1 b=2 b=4 b=8 b=16
18
+ generator (seed 0, 3 sampler seeds) 94.0 [91.9,96.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
19
+ retr_unit (X2 baseline) 40.0 [34.7,45.3] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
20
+ retr_learned 42.0 [36.3,47.7] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.7 [99.0,100.0]
21
+ twin (3 members x 3 draw seeds) 85.0 [81.9,88.1] 99.3 [98.4,99.9] 99.6 [98.8,100.0] 99.6 [98.8,100.0] 99.7 [99.0,100.0]
22
+
23
+ ## Test success % [95% CI over plants], pipeline = rank
24
+ arm b=1 b=2 b=4 b=8 b=16
25
+ generator (seed 0, 3 sampler seeds) 99.1 [98.0,99.9] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
26
+ retr_unit (X2 baseline) 89.0 [85.3,92.3] 91.7 [88.3,94.7] 92.7 [89.7,95.3] 92.7 [89.7,95.3] 93.3 [90.3,96.0]
27
+ retr_learned 93.7 [90.7,96.3] 95.0 [92.3,97.3] 96.7 [94.3,98.7] 96.7 [94.7,98.7] 96.7 [94.3,98.7]
28
+ twin (3 members x 3 draw seeds) 96.0 [94.1,97.6] 96.0 [94.1,97.7] 96.1 [94.2,97.7] 96.1 [94.3,97.8] 96.2 [94.4,97.9]
29
+
30
+ ## Test success % [95% CI over plants], pipeline = rank_refine <- FROZEN PRIMARY
31
+ arm b=1 b=2 b=4 b=8 b=16
32
+ generator (seed 0, 3 sampler seeds) 99.1 [98.0,99.9] 99.9 [99.7,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
33
+ retr_unit (X2 baseline) 89.0 [85.3,92.3] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
34
+ retr_learned 93.7 [90.7,96.3] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
35
+ twin (3 members x 3 draw seeds) 96.0 [94.1,97.7] 99.6 [98.8,100.0] 99.6 [98.9,100.0] 99.6 [98.9,100.0] 99.7 [99.0,100.0]
36
+
37
+ ## Test success % [95% CI over plants], pipeline = refine_free
38
+ arm b=1 b=2 b=4 b=8 b=16
39
+ generator (seed 0, 3 sampler seeds) 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
40
+ retr_unit (X2 baseline) 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
41
+ retr_learned 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.7 [99.0,100.0] 100.0 [100.0,100.0]
42
+ twin (3 members x 3 draw seeds) 99.3 [98.4,99.9] 99.6 [98.8,100.0] 99.6 [98.8,100.0] 99.7 [99.0,100.0] 99.7 [99.0,100.0]
43
+
44
+ ## Test success % [95% CI over plants], pipeline = rank_refine_free
45
+ arm b=1 b=2 b=4 b=8 b=16
46
+ generator (seed 0, 3 sampler seeds) 99.9 [99.7,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
47
+ retr_unit (X2 baseline) 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
48
+ retr_learned 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
49
+ twin (3 members x 3 draw seeds) 99.6 [98.9,100.0] 99.6 [98.9,100.0] 99.6 [98.9,100.0] 99.7 [99.0,100.0] 99.7 [99.0,100.0]
50
+
51
+ ## Test success % [95% CI over plants], pipeline = refine_c3
52
+ arm b=1 b=2 b=4 b=8 b=16
53
+ generator (seed 0, 3 sampler seeds) 94.0 [91.8,96.0] 94.0 [91.8,96.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
54
+ retr_unit (X2 baseline) 40.0 [34.7,45.7] 40.0 [34.7,45.7] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
55
+ retr_learned 42.0 [36.3,47.7] 42.0 [36.3,47.3] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
56
+ twin (3 members x 3 draw seeds) 85.0 [81.7,88.1] 85.0 [81.7,88.1] 99.3 [98.4,99.9] 99.6 [98.8,100.0] 99.6 [98.8,100.0]
57
+
58
+ ## Test success % [95% CI over plants], pipeline = rank_refine_c3
59
+ arm b=1 b=2 b=4 b=8 b=16
60
+ generator (seed 0, 3 sampler seeds) 99.1 [98.0,99.9] 99.1 [98.0,99.9] 99.9 [99.7,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
61
+ retr_unit (X2 baseline) 89.0 [85.3,92.3] 89.0 [85.3,92.3] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
62
+ retr_learned 93.7 [90.7,96.3] 93.7 [90.7,96.3] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
63
+ twin (3 members x 3 draw seeds) 96.0 [94.1,97.6] 96.0 [94.1,97.7] 99.6 [98.9,100.0] 99.6 [98.9,100.0] 99.6 [98.9,100.0]
64
+
65
+ ## Descriptive: mean number of verifier calls to the first success (a failure within 16 calls counts as 17), per arm; mean over replicas and plants
66
+ plain generator 1.22 retr_unit 4.22 retr_learned 3.48 twin 2.02
67
+ refine generator 1.06 retr_unit 1.70 retr_learned 1.67 twin 1.22
68
+ rank generator 1.11 retr_unit 2.19 retr_learned 1.58 twin 1.62
69
+ rank_refine generator 1.01 retr_unit 1.21 retr_learned 1.06 twin 1.10
70
+ refine_free generator 1.00 retr_unit 1.11 retr_learned 1.05 twin 1.06
71
+ rank_refine_free generator 1.00 retr_unit 1.11 retr_learned 1.00 twin 1.06
72
+ refine_c3 generator 1.18 retr_unit 2.89 retr_learned 2.83 twin 1.52
73
+ rank_refine_c3 generator 1.03 retr_unit 1.42 retr_learned 1.19 twin 1.17
74
+
75
+ ## PRIMARY family (9 tests, Holm): generator minus comparator under rank_refine (counted refinement), pp [95% CI over plants], sign-flip p, Holm p, McNemar (gen-only/comp-only; sampler seed 0 vs twin member 0 draw 0) p
76
+ gen - retr_unit, b=1, pipeline rank_refine +10.1 [ +6.9,+13.7] p=0.0000 holm=0.0000 McNemar 31/1 p=0.0000
77
+ gen - retr_learned, b=1, pipeline rank_refine +5.4 [ +2.8, +8.4] p=0.0001 holm=0.0003 McNemar 18/2 p=0.0004
78
+ gen - twin, b=1, pipeline rank_refine +3.1 [ +1.6, +4.9] p=0.0000 holm=0.0000 McNemar 6/1 p=0.1250
79
+ gen - retr_unit, b=4, pipeline rank_refine +0.7 [ +0.0, +1.7] p=0.5009 holm=1.0000 McNemar 2/0 p=0.5000
80
+ gen - retr_learned, b=4, pipeline rank_refine +0.0 [ +0.0, +0.0] p=1.0000 holm=1.0000 McNemar 0/0 p=1.0000
81
+ gen - twin, b=4, pipeline rank_refine +0.4 [ +0.0, +1.2] p=0.5045 holm=1.0000 McNemar 1/0 p=1.0000
82
+ gen - retr_unit, b=16, pipeline rank_refine +0.7 [ +0.0, +1.7] p=0.4913 holm=1.0000 McNemar 2/0 p=0.5000
83
+ gen - retr_learned, b=16, pipeline rank_refine +0.0 [ +0.0, +0.0] p=1.0000 holm=1.0000 McNemar 0/0 p=1.0000
84
+ gen - twin, b=16, pipeline rank_refine +0.3 [ +0.0, +1.0] p=1.0000 holm=1.0000 McNemar 1/0 p=1.0000
85
+
86
+ ## SECONDARY (unadjusted): generator minus comparator, every pipeline, pp [95% CI]
87
+ plain gen - retr_unit (X2 baseline) +54.0 [+48.0,+59.8] p=0.000 +40.1 [+34.2,+45.9] p=0.000 +26.9 [+21.8,+32.0] p=0.000 +14.2 [+10.3,+18.3] p=0.000 +6.0 [ +3.3, +9.0] p=0.000
88
+ plain gen - retr_learned +52.0 [+46.1,+57.8] p=0.000 +36.1 [+30.3,+41.9] p=0.000 +21.6 [+16.9,+26.6] p=0.000 +8.6 [ +5.3,+12.0] p=0.000 +2.7 [ +0.7, +4.7] p=0.021
89
+ plain gen - twin (3 members x 3 draw seeds) +9.0 [ +6.0,+12.1] p=0.000 +8.7 [ +6.1,+11.4] p=0.000 +6.3 [ +4.2, +8.6] p=0.000 +4.3 [ +2.7, +6.2] p=0.000 +3.1 [ +1.6, +4.8] p=0.000
90
+ refine gen - retr_unit (X2 baseline) +54.0 [+48.0,+59.8] p=0.000 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.501
91
+ refine gen - retr_learned +52.0 [+46.2,+58.0] p=0.000 +0.7 [ +0.0, +1.7] p=0.502 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.504 +0.3 [ +0.0, +1.0] p=1.000
92
+ refine gen - twin (3 members x 3 draw seeds) +9.0 [ +5.9,+12.1] p=0.000 +0.7 [ +0.1, +1.6] p=0.125 +0.4 [ +0.0, +1.2] p=0.256 +0.4 [ +0.0, +1.2] p=0.249 +0.3 [ +0.0, +1.0] p=1.000
93
+ rank gen - retr_unit (X2 baseline) +10.1 [ +6.8,+13.7] p=0.000 +7.7 [ +4.7,+11.0] p=0.000 +6.7 [ +4.0, +9.7] p=0.000 +6.7 [ +4.0, +9.7] p=0.000 +6.0 [ +3.3, +8.7] p=0.000
94
+ rank gen - retr_learned +5.4 [ +2.8, +8.3] p=0.000 +4.3 [ +2.0, +7.0] p=0.001 +2.7 [ +0.7, +5.0] p=0.021 +2.7 [ +0.7, +4.7] p=0.022 +2.7 [ +0.7, +5.0] p=0.023
95
+ rank gen - twin (3 members x 3 draw seeds) +3.1 [ +1.6, +4.8] p=0.000 +3.3 [ +1.9, +5.1] p=0.000 +3.2 [ +1.7, +5.0] p=0.000 +3.2 [ +1.7, +4.9] p=0.000 +3.1 [ +1.6, +4.8] p=0.000
96
+ rank_refine gen - retr_unit (X2 baseline) +10.1 [ +6.9,+13.7] p=0.000 +0.6 [ +0.0, +1.4] p=0.502 +0.7 [ +0.0, +1.7] p=0.501 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.500
97
+ rank_refine gen - retr_learned +5.4 [ +2.8, +8.3] p=0.000 -0.1 [ -0.3, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000
98
+ rank_refine gen - twin (3 members x 3 draw seeds) +3.1 [ +1.6, +4.8] p=0.000 +0.3 [ +0.0, +0.8] p=0.495 +0.4 [ +0.0, +1.1] p=0.495 +0.4 [ +0.0, +1.1] p=0.500 +0.3 [ +0.0, +1.0] p=1.000
99
+ refine_free gen - retr_unit (X2 baseline) +0.7 [ +0.0, +1.7] p=0.497 +0.7 [ +0.0, +1.7] p=0.504 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.502
100
+ refine_free gen - retr_learned +0.7 [ +0.0, +1.7] p=0.500 +0.7 [ +0.0, +1.7] p=0.504 +0.7 [ +0.0, +1.7] p=0.501 +0.3 [ +0.0, +1.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000
101
+ refine_free gen - twin (3 members x 3 draw seeds) +0.7 [ +0.1, +1.6] p=0.130 +0.4 [ +0.0, +1.2] p=0.252 +0.4 [ +0.0, +1.2] p=0.255 +0.3 [ +0.0, +1.0] p=1.000 +0.3 [ +0.0, +1.0] p=1.000
102
+ rank_refine_free gen - retr_unit (X2 baseline) +0.6 [ +0.0, +1.4] p=0.502 +0.7 [ +0.0, +1.7] p=0.496 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.506 +0.7 [ +0.0, +1.7] p=0.498
103
+ rank_refine_free gen - retr_learned -0.1 [ -0.3, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000
104
+ rank_refine_free gen - twin (3 members x 3 draw seeds) +0.3 [ +0.0, +0.8] p=0.505 +0.4 [ +0.0, +1.1] p=0.496 +0.4 [ +0.0, +1.1] p=0.498 +0.3 [ +0.0, +1.0] p=1.000 +0.3 [ +0.0, +1.0] p=1.000
105
+ refine_c3 gen - retr_unit (X2 baseline) +54.0 [+48.1,+59.9] p=0.000 +54.0 [+48.0,+60.0] p=0.000 +0.7 [ +0.0, +1.7] p=0.501 +0.7 [ +0.0, +1.7] p=0.497 +0.7 [ +0.0, +1.7] p=0.500
106
+ refine_c3 gen - retr_learned +52.0 [+46.1,+58.0] p=0.000 +52.0 [+46.0,+57.9] p=0.000 +0.7 [ +0.0, +1.7] p=0.496 +0.7 [ +0.0, +1.7] p=0.497 +0.7 [ +0.0, +1.7] p=0.499
107
+ refine_c3 gen - twin (3 members x 3 draw seeds) +9.0 [ +6.0,+12.0] p=0.000 +9.0 [ +6.0,+12.0] p=0.000 +0.7 [ +0.1, +1.6] p=0.121 +0.4 [ +0.0, +1.2] p=0.248 +0.4 [ +0.0, +1.2] p=0.248
108
+ rank_refine_c3 gen - retr_unit (X2 baseline) +10.1 [ +6.8,+13.6] p=0.000 +10.1 [ +6.8,+13.7] p=0.000 +0.6 [ +0.0, +1.4] p=0.498 +0.7 [ +0.0, +1.7] p=0.505 +0.7 [ +0.0, +1.7] p=0.503
109
+ rank_refine_c3 gen - retr_learned +5.4 [ +2.9, +8.3] p=0.000 +5.4 [ +2.8, +8.3] p=0.000 -0.1 [ -0.3, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000
110
+ rank_refine_c3 gen - twin (3 members x 3 draw seeds) +3.1 [ +1.6, +4.8] p=0.000 +3.1 [ +1.7, +4.9] p=0.000 +0.3 [ +0.0, +0.8] p=0.498 +0.4 [ +0.0, +1.1] p=0.503 +0.4 [ +0.0, +1.1] p=0.504
111
+
112
+ ## Across training seeds (unit = seed; frozen pipeline rank_refine; seeds 1-4 use sampler seed 0 only, seed 0 is the mean over its 3 sampler seeds from the main table)
113
+ b=1: generator per seed [99.1, 99.7, 99.3, 99.7, 99.7]
114
+ minus retr_unit: per seed [10.1, 10.7, 10.3, 10.7, 10.7] mean +10.5 t95 [+10.2,+10.8] seeds positive 5/5
115
+ minus retr_learned: per seed [5.4, 6.0, 5.7, 6.0, 6.0] mean +5.8 t95 [+5.5,+6.1] seeds positive 5/5
116
+ minus twin: per seed [3.1, 3.7, 3.3, 3.7, 3.7] mean +3.5 t95 [+3.2,+3.8] seeds positive 5/5
117
+ b=4: generator per seed [100.0, 100.0, 100.0, 100.0, 100.0]
118
+ minus retr_unit: per seed [0.7, 0.7, 0.7, 0.7, 0.7] mean +0.7 t95 [+0.7,+0.7] seeds positive 5/5
119
+ minus retr_learned: per seed [0.0, 0.0, 0.0, 0.0, 0.0] mean +0.0 t95 [+0.0,+0.0] seeds positive 0/5
120
+ minus twin: per seed [0.4, 0.4, 0.4, 0.4, 0.4] mean +0.4 t95 [+0.4,+0.4] seeds positive 5/5
121
+ b=16: generator per seed [100.0, 100.0, 100.0, 100.0, 100.0]
122
+ minus retr_unit: per seed [0.7, 0.7, 0.7, 0.7, 0.7] mean +0.7 t95 [+0.7,+0.7] seeds positive 5/5
123
+ minus retr_learned: per seed [0.0, 0.0, 0.0, 0.0, 0.0] mean +0.0 t95 [+0.0,+0.0] seeds positive 0/5
124
+ minus twin: per seed [0.3, 0.3, 0.3, 0.3, 0.3] mean +0.3 t95 [+0.3,+0.3] seeds positive 5/5
125
+
126
+ # X18: twin ensemble (plain verification, no repair)
127
+ arm (plain) b=1 b=3 b=4 b=8 b=16
128
+ generator 94.0 [91.9,96.0] 97.7 [96.1,99.0] 98.2 [96.9,99.3] 99.2 [98.2,100.0] 99.3 [98.3,100.0]
129
+ twin ensemble (3 members, 9 replicas) 85.0 [81.8,88.1] 95.3 [92.7,97.7] 95.7 [93.4,97.8] 96.9 [94.9,98.6] 97.7 [96.1,99.1]
130
+ twin single member, MC dropout (9 replicas) 85.0 [81.8,88.0] 90.7 [88.1,93.1] 91.9 [89.4,94.1] 94.9 [92.9,96.7] 96.2 [94.3,97.9]
131
+
132
+ ## X18 PRIMARY family (4 tests, Holm), pp [95% CI], sign-flip p, Holm p, McNemar (first replicas) p
133
+ ensemble - single-member twin, b=4 +3.9 [ +2.5, +5.3] p=0.0000 holm=0.0000 McNemar 6/1 p=0.1250
134
+ ensemble - single-member twin, b=16 +1.5 [ +0.8, +2.3] p=0.0000 holm=0.0000 McNemar 2/0 p=0.5000
135
+ generator - ensemble, b=4 +2.5 [ +0.6, +4.6] p=0.0176 holm=0.0353 McNemar 10/2 p=0.0386
136
+ generator - ensemble, b=16 +1.6 [ +0.4, +3.1] p=0.0213 holm=0.0353 McNemar 4/0 p=0.1250
137
+
138
+ ## X18 SECONDARY (unadjusted), every b: ensemble - single twin | generator - ensemble | generator - single twin
139
+ b= 1: +0.0 [ +0.0, +0.0] p=1.000 | +9.0 [ +6.0,+12.1] p=0.000 | +9.0 [ +5.9,+12.1] p=0.000
140
+ b= 2: +4.1 [ +2.9, +5.4] p=0.000 | +4.6 [ +2.2, +7.1] p=0.000 | +8.7 [ +6.1,+11.3] p=0.000
141
+ b= 3: +4.7 [ +3.1, +6.3] p=0.000 | +2.3 [ +0.1, +4.8] p=0.059 | +7.0 [ +4.7, +9.4] p=0.000
142
+ b= 4: +3.9 [ +2.6, +5.3] p=0.000 | +2.5 [ +0.6, +4.6] p=0.016 | +6.3 [ +4.3, +8.7] p=0.000
143
+ b= 6: +2.6 [ +1.6, +3.7] p=0.000 | +2.4 [ +0.9, +4.2] p=0.006 | +5.0 [ +3.2, +7.0] p=0.000
144
+ b= 8: +2.0 [ +1.1, +2.9] p=0.000 | +2.4 [ +0.9, +4.1] p=0.003 | +4.3 [ +2.7, +6.2] p=0.000
145
+ b=12: +1.3 [ +0.5, +2.1] p=0.001 | +2.3 [ +0.9, +4.0] p=0.004 | +3.6 [ +2.1, +5.4] p=0.000
146
+ b=16: +1.5 [ +0.8, +2.3] p=0.000 | +1.6 [ +0.4, +3.1] p=0.024 | +3.1 [ +1.6, +4.8] p=0.000
147
+
148
+ ## X18 with repair (secondary): ensemble vs single twin vs generator at b = 1, 4, 16, pipelines refine / rank_refine
149
+ refine: ens 85.0 99.7 99.7 | twin 85.0 99.6 99.7 | gen 94.0 100.0 100.0
150
+ rank_refine: ens 97.7 99.7 99.7 | twin 96.0 99.6 99.7 | gen 99.1 100.0 100.0
151
+
152
+ ## Check: plain pipeline vs X2 (success % at b = 1, 4, 16; X2 values from results/icml_r3/X2/test_*.npz)
153
+ gen here 94.0 98.2 99.3 | X2 94.1 98.2 99.3
154
+ retr here 40.0 71.3 93.3 | X2 40.0 71.3 93.3
results/icml_r3/X8/logs/test_main.log ADDED
@@ -0,0 +1,88 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [test] group gen0: ['gen0_s0', 'gen0_s1', 'gen0_s2']
2
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
3
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
4
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
5
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
6
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
7
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
8
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
9
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
10
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
11
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
12
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
13
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
14
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
15
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
16
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
17
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
18
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
19
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
20
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
21
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
22
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
23
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
24
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
25
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
26
+ verify 14400 candidates 591s: certified 14400, plain ok 13607 (mean 491 ms/call)
27
+ vmin 14400 candidates 2s
28
+ refine 793 failed-certified candidates 94s (median V_min 1.75 -> 0.26)
29
+ refined verifier ok 786 of 793 (105s)
30
+ [test] group retr: ['retr_unit', 'retr_learned']
31
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
32
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
33
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
34
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
35
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
36
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
37
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
38
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
39
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
40
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
41
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
42
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
43
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
44
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
45
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
46
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
47
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
48
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
49
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
50
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
51
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
52
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
53
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
54
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
55
+ verify 9600 candidates 401s: certified 9600, plain ok 1866 (mean 497 ms/call)
56
+ vmin 9600 candidates 2s
57
+ refine 7734 failed-certified candidates 840s (median V_min 3.88 -> 0.24)
58
+ refined verifier ok 7610 of 7734 (925s)
59
+ [test] group twin: ['twin0_d0', 'twin0_d1', 'twin0_d2', 'twin1_d0', 'twin1_d1', 'twin1_d2', 'twin2_d0', 'twin2_d1', 'twin2_d2']
60
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
61
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
62
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
63
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
64
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
65
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
66
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
67
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
68
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
69
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
70
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
71
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
72
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
73
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
74
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
75
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
76
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
77
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
78
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
79
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
80
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
81
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
82
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
83
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
84
+ verify 43200 candidates 1834s: certified 43200, plain ok 35197 (mean 509 ms/call)
85
+ vmin 43200 candidates 6s
86
+ refine 8003 failed-certified candidates 868s (median V_min 1.92 -> 0.24)
87
+ refined verifier ok 7758 of 8003 (956s)
88
+ test main done 4824s
results/icml_r3/X8/logs/test_seeds.log ADDED
@@ -0,0 +1,117 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [test] group gen1: ['gen1_s0']
2
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
3
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
4
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
5
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
6
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
7
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
8
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
9
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
10
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
11
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
12
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
13
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
14
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
15
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
16
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
17
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
18
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
19
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
20
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
21
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
22
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
23
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
24
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
25
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
26
+ verify 4800 candidates 204s: certified 4800, plain ok 4589 (mean 499 ms/call)
27
+ vmin 4800 candidates 2s
28
+ refine 211 failed-certified candidates 32s (median V_min 1.64 -> 0.24)
29
+ refined verifier ok 205 of 211 (36s)
30
+ [test] group gen2: ['gen2_s0']
31
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
32
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
33
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
34
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
35
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
36
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
37
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
38
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
39
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
40
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
41
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
42
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
43
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
44
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
45
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
46
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
47
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
48
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
49
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
50
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
51
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
52
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
53
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
54
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
55
+ verify 4800 candidates 208s: certified 4800, plain ok 4573 (mean 509 ms/call)
56
+ vmin 4800 candidates 2s
57
+ refine 227 failed-certified candidates 32s (median V_min 1.79 -> 0.25)
58
+ refined verifier ok 226 of 227 (37s)
59
+ [test] group gen3: ['gen3_s0']
60
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
61
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
62
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
63
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
64
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
65
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
66
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
67
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
68
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
69
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
70
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
71
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
72
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
73
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
74
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
75
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
76
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
77
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
78
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
79
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
80
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
81
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
82
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
83
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
84
+ verify 4800 candidates 197s: certified 4800, plain ok 4603 (mean 478 ms/call)
85
+ vmin 4800 candidates 2s
86
+ refine 197 failed-certified candidates 32s (median V_min 1.69 -> 0.25)
87
+ refined verifier ok 190 of 197 (36s)
88
+ [test] group gen4: ['gen4_s0']
89
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
90
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
91
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
92
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
93
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
94
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
95
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
96
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
97
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
98
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
99
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
100
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
101
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
102
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
103
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
104
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
105
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
106
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
107
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
108
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
109
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
110
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
111
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
112
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
113
+ verify 4800 candidates 204s: certified 4800, plain ok 4606 (mean 493 ms/call)
114
+ vmin 4800 candidates 1s
115
+ refine 194 failed-certified candidates 32s (median V_min 1.71 -> 0.24)
116
+ refined verifier ok 191 of 194 (36s)
117
+ test seeds done 965s
results/icml_r3/X8/logs/val.log ADDED
@@ -0,0 +1,181 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [val] group gen0: ['gen0_s0']
2
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
3
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
4
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
5
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
6
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
7
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
8
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
9
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
10
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
11
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
12
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
13
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
14
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
15
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
16
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
17
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
18
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
19
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
20
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
21
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
22
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
23
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
24
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
25
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
26
+ verify 960 candidates 37s: certified 960, plain ok 884 (mean 433 ms/call)
27
+ vmin 960 candidates 1s
28
+ refine 76 failed-certified candidates 16s (median V_min 1.78 -> 0.32)
29
+ refined verifier ok 70 of 76 (19s)
30
+ [val] group retr: ['retr_unit', 'retr_learned']
31
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
32
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
33
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
34
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
35
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
36
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
37
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
38
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
39
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
40
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
41
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
42
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
43
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
44
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
45
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
46
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
47
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
48
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
49
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
50
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
51
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
52
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
53
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
54
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
55
+ verify 1920 candidates 77s: certified 1920, plain ok 376 (mean 472 ms/call)
56
+ vmin 1920 candidates 1s
57
+ refine 1544 failed-certified candidates 93s (median V_min 4.06 -> 0.40)
58
+ refined verifier ok 1474 of 1544 (112s)
59
+ [val] group twin: ['twin0_d0', 'twin1_d0', 'twin2_d0']
60
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
61
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
62
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
63
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
64
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
65
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
66
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
67
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
68
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
69
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
70
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
71
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
72
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
73
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
74
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
75
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
76
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
77
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
78
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
79
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
80
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
81
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
82
+ /home/cercola/matte_server/DiffusionControl/nonlinear/data_generation/furuta_roa.py:190: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
83
+ cp.Problem(cp.Maximize(cp.log_det(S)), cons).solve(solver=solver, verbose=False)
84
+ verify 2880 candidates 127s: certified 2880, plain ok 2313 (mean 520 ms/call)
85
+ vmin 2880 candidates 1s
86
+ refine 567 failed-certified candidates 33s (median V_min 2.00 -> 0.31)
87
+ refined verifier ok 553 of 567 (41s)
88
+ == refine config {'iters': 40, 'lr': 0.1, 'n': 200}: 415s
89
+ [val] group gen0: ['gen0_s0']
90
+ refine 76 failed-certified candidates 31s (median V_min 1.78 -> 0.28)
91
+ refined verifier ok 75 of 76 (33s)
92
+ [val] group retr: ['retr_unit', 'retr_learned']
93
+ refine 1544 failed-certified candidates 184s (median V_min 4.06 -> 0.25)
94
+ refined verifier ok 1499 of 1544 (203s)
95
+ [val] group twin: ['twin0_d0', 'twin1_d0', 'twin2_d0']
96
+ refine 567 failed-certified candidates 65s (median V_min 2.00 -> 0.25)
97
+ refined verifier ok 567 of 567 (73s)
98
+ == refine config {'iters': 80, 'lr': 0.1, 'n': 200}: 309s
99
+ [val] group gen0: ['gen0_s0']
100
+ refine 76 failed-certified candidates 16s (median V_min 1.78 -> 0.27)
101
+ refined verifier ok 76 of 76 (19s)
102
+ [val] group retr: ['retr_unit', 'retr_learned']
103
+ refine 1544 failed-certified candidates 92s (median V_min 4.06 -> 0.27)
104
+ refined verifier ok 1496 of 1544 (112s)
105
+ [val] group twin: ['twin0_d0', 'twin1_d0', 'twin2_d0']
106
+ refine 567 failed-certified candidates 33s (median V_min 2.00 -> 0.23)
107
+ refined verifier ok 567 of 567 (40s)
108
+ == refine config {'iters': 40, 'lr': 0.2, 'n': 200}: 171s
109
+ [val] group gen0: ['gen0_s0']
110
+ refine 76 failed-certified candidates 31s (median V_min 1.78 -> 0.22)
111
+ refined verifier ok 76 of 76 (33s)
112
+ [val] group retr: ['retr_unit', 'retr_learned']
113
+ refine 1544 failed-certified candidates 183s (median V_min 4.06 -> 0.20)
114
+ refined verifier ok 1492 of 1544 (202s)
115
+ [val] group twin: ['twin0_d0', 'twin1_d0', 'twin2_d0']
116
+ refine 567 failed-certified candidates 64s (median V_min 2.00 -> 0.18)
117
+ refined verifier ok 567 of 567 (72s)
118
+ == refine config {'iters': 80, 'lr': 0.2, 'n': 200}: 308s
119
+ [val] group gen0: ['gen0_s0']
120
+ refine 76 failed-certified candidates 32s (median V_min 2.29 -> 0.34)
121
+ refined verifier ok 70 of 76 (35s)
122
+ [val] group retr: ['retr_unit', 'retr_learned']
123
+ refine 1544 failed-certified candidates 196s (median V_min 4.23 -> 0.40)
124
+ refined verifier ok 1474 of 1544 (214s)
125
+ [val] group twin: ['twin0_d0', 'twin1_d0', 'twin2_d0']
126
+ refine 567 failed-certified candidates 64s (median V_min 2.49 -> 0.31)
127
+ refined verifier ok 553 of 567 (71s)
128
+ == refine config {'iters': 40, 'lr': 0.1, 'n': 400}: 320s
129
+ # X8 validation (60 plants; gen seed 0 sampler 0, retr_unit, retr_learned (metric xfer), twin members 0-2 draw seed 0)
130
+
131
+ refine config it40_lr0.1_n200: rescued 2097 of 2187 certified-failed candidates = 95.9% (per arm: gen0_s0 70/76, retr_unit 761/797, retr_learned 713/747, twin0_d0 195/195, twin1_d0 160/174, twin2_d0 198/198)
132
+ refine config it80_lr0.1_n200: rescued 2141 of 2187 certified-failed candidates = 97.9% (per arm: gen0_s0 75/76, retr_unit 777/797, retr_learned 722/747, twin0_d0 195/195, twin1_d0 174/174, twin2_d0 198/198)
133
+ refine config it40_lr0.2_n200: rescued 2139 of 2187 certified-failed candidates = 97.8% (per arm: gen0_s0 76/76, retr_unit 775/797, retr_learned 721/747, twin0_d0 195/195, twin1_d0 174/174, twin2_d0 198/198)
134
+ refine config it80_lr0.2_n200: rescued 2135 of 2187 certified-failed candidates = 97.6% (per arm: gen0_s0 76/76, retr_unit 775/797, retr_learned 717/747, twin0_d0 195/195, twin1_d0 174/174, twin2_d0 198/198)
135
+ refine config it40_lr0.1_n400: rescued 2097 of 2187 certified-failed candidates = 95.9% (per arm: gen0_s0 70/76, retr_unit 760/797, retr_learned 714/747, twin0_d0 195/195, twin1_d0 160/174, twin2_d0 198/198)
136
+ FROZEN refine config: {'iters': 80, 'lr': 0.1, 'n': 200} (highest pooled rescue rate; ties -> cheaper = earlier in grid)
137
+
138
+ # pipelines on validation, success % at b = 1/2/4/8/16 per arm (frozen refine config)
139
+ plain gen0_s0 88.3 93.3 98.3 98.3 98.3
140
+ plain retr_unit 31.7 50.0 73.3 85.0 93.3
141
+ plain retr_learned 35.0 55.0 70.0 88.3 95.0
142
+ plain twin0_d0 83.3 85.0 85.0 91.7 95.0
143
+ plain twin1_d0 86.7 95.0 98.3 98.3 98.3
144
+ plain twin2_d0 81.7 90.0 90.0 93.3 95.0
145
+ plain baseline mean over b=1,4,16 (retr_unit, retr_learned, twin): 74.38
146
+ refine gen0_s0 88.3 100.0 100.0 100.0 100.0
147
+ refine retr_unit 31.7 100.0 100.0 100.0 100.0
148
+ refine retr_learned 35.0 100.0 100.0 100.0 100.0
149
+ refine twin0_d0 83.3 100.0 100.0 100.0 100.0
150
+ refine twin1_d0 86.7 100.0 100.0 100.0 100.0
151
+ refine twin2_d0 81.7 100.0 100.0 100.0 100.0
152
+ refine baseline mean over b=1,4,16 (retr_unit, retr_learned, twin): 83.40
153
+ rank gen0_s0 98.3 98.3 98.3 98.3 98.3
154
+ rank retr_unit 83.3 88.3 93.3 93.3 93.3
155
+ rank retr_learned 88.3 93.3 93.3 95.0 95.0
156
+ rank twin0_d0 93.3 93.3 93.3 93.3 95.0
157
+ rank twin1_d0 98.3 98.3 98.3 98.3 98.3
158
+ rank twin2_d0 93.3 93.3 93.3 93.3 95.0
159
+ rank baseline mean over b=1,4,16 (retr_unit, retr_learned, twin): 92.53
160
+ rank_refine gen0_s0 98.3 100.0 100.0 100.0 100.0
161
+ rank_refine retr_unit 83.3 100.0 100.0 100.0 100.0
162
+ rank_refine retr_learned 88.3 100.0 100.0 100.0 100.0
163
+ rank_refine twin0_d0 93.3 100.0 100.0 100.0 100.0
164
+ rank_refine twin1_d0 98.3 100.0 100.0 100.0 100.0
165
+ rank_refine twin2_d0 93.3 100.0 100.0 100.0 100.0
166
+ rank_refine baseline mean over b=1,4,16 (retr_unit, retr_learned, twin): 96.30
167
+ refine_free gen0_s0 100.0 100.0 100.0 100.0 100.0
168
+ refine_free retr_unit 100.0 100.0 100.0 100.0 100.0
169
+ refine_free retr_learned 100.0 100.0 100.0 100.0 100.0
170
+ refine_free twin0_d0 100.0 100.0 100.0 100.0 100.0
171
+ refine_free twin1_d0 100.0 100.0 100.0 100.0 100.0
172
+ refine_free twin2_d0 100.0 100.0 100.0 100.0 100.0
173
+ refine_free baseline mean over b=1,4,16 (retr_unit, retr_learned, twin): 100.00
174
+ rank_refine_free gen0_s0 100.0 100.0 100.0 100.0 100.0
175
+ rank_refine_free retr_unit 100.0 100.0 100.0 100.0 100.0
176
+ rank_refine_free retr_learned 100.0 100.0 100.0 100.0 100.0
177
+ rank_refine_free twin0_d0 100.0 100.0 100.0 100.0 100.0
178
+ rank_refine_free twin1_d0 100.0 100.0 100.0 100.0 100.0
179
+ rank_refine_free twin2_d0 100.0 100.0 100.0 100.0 100.0
180
+ rank_refine_free baseline mean over b=1,4,16 (retr_unit, retr_learned, twin): 100.00
181
+ FROZEN primary pipeline: rank_refine (higher mean baseline-arm success over b = 1, 4, 16 among refine / rank_refine; ties -> rank_refine)
results/icml_r3/X8/tables/test_gen0.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5e3c141de6e2a43968f791da8dad58ed8400f488996bba72afbde69b9707dccf
3
+ size 2233694
results/icml_r3/X8/tables/test_gen0_ref_it80_lr0.1_n200.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:969f40b7576188e814c64c9e830ebf8d9223f5baa660198b1ffac8091c56c83d
3
+ size 245550
results/icml_r3/X8/tables/test_gen1.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5936baeb215e1a26c64e432da7b2fd69af444be7ac28754ff1580f409e107829
3
+ size 745694
results/icml_r3/X8/tables/test_gen1_ref_it80_lr0.1_n200.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f74b531e739ee9ee7c9b17d472c27264280cd93585c79d29c55f5dc70b36d9c4
3
+ size 82350
results/icml_r3/X8/tables/test_gen2.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f9c21f998ef7f08398d24b0e07b2032e890ccc3f9d28681de49513621c326e8b
3
+ size 745694
results/icml_r3/X8/tables/test_gen2_ref_it80_lr0.1_n200.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:babb6fa538e28efe57352effbcee684bede26a57b1ee54a092096b56f2ec8273
3
+ size 82350
results/icml_r3/X8/tables/test_gen3.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cde14c2f8013dc4e8d0d6f1a28a8b7ca29b8789269c7a4652b3c279060328686
3
+ size 745694
results/icml_r3/X8/tables/test_gen3_ref_it80_lr0.1_n200.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:df8f99a1a02c97325f07d7ec16c5708bbaa0b5a2dfa4d25d4453beea7bc67b07
3
+ size 82350
results/icml_r3/X8/tables/test_gen4.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:724dea7ef87068820b39b1474300db22d904deac34ecf46edceafca0fba82ddd
3
+ size 745694
results/icml_r3/X8/tables/test_gen4_ref_it80_lr0.1_n200.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ab45481501f4e917936d35555aa5a5659462ca52b6697bbd40ad2392924bd418
3
+ size 82350
results/icml_r3/X8/tables/test_retr.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:99ecdbc7ac5746199a378543b598f005e62039429a242b05b1eb1c3b6f6e8489
3
+ size 1489694
results/icml_r3/X8/tables/test_retr_ref_it80_lr0.1_n200.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5995a5f4b3fdd8f54fdaa9937b30f1c7dd19345aba5a5c17c3b3a15a4e1478f7
3
+ size 163950
results/icml_r3/X8/tables/test_twin.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:97c80561cfd64ca2afbd72cffdae8b993a8543cd74a5a7d5fe97acccfe4f9be3
3
+ size 6697694
results/icml_r3/X8/tables/test_twin_ref_it80_lr0.1_n200.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7b7c043f12a6b93196aa709e44da724322fd066cf5d1c337f32aca729601a977
3
+ size 735150
results/icml_r3/X8/tables/val_gen0.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2ff1cfd2edbf44061c7e9c26c9cfa823dc85fdd98f17e1090510bac0e7271ec3
3
+ size 150494
results/icml_r3/X8/tables/val_gen0_ref_it40_lr0.1_n200.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:97ff2853ac9a85e4fa82e718c38822d4ef9a6e80ad41938ce48fb1c11c5b02d2
3
+ size 17070
results/icml_r3/X8/tables/val_gen0_ref_it40_lr0.1_n400.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2487d63ea9dd1ea496f84adc6322e08ba1883f153a448445ea57d2faddab2768
3
+ size 17070
results/icml_r3/X8/tables/val_gen0_ref_it40_lr0.2_n200.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1817fe080258473e18bebdcea442c9e7d6eed882afccd54e045736f7241880c7
3
+ size 17070
results/icml_r3/X8/tables/val_gen0_ref_it80_lr0.1_n200.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3bcb676ee9abd32963e860920c9a3ffec3403dde4fad62f713ad9467f8c0f9c5
3
+ size 17070
results/icml_r3/X8/tables/val_gen0_ref_it80_lr0.2_n200.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ac17194fee8f496f27f85fee877bf8fe70ad85e49a11d897f82c3f105e5efc9e
3
+ size 17070
results/icml_r3/X8/tables/val_retr.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b134fe57f2911ef6a67edb51874de347752539f9610c755d09f694eb1c8f3756
3
+ size 299294
results/icml_r3/X8/tables/val_retr_ref_it40_lr0.1_n200.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3d30dbcd1bff4b099180aafc7b46a2d545e78aa5ecb387c5ce0462ea01d6dc4c
3
+ size 33390
results/icml_r3/X8/tables/val_retr_ref_it40_lr0.1_n400.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2eedf8179b5c543dce3cb5a11eb88c34aadd7acc897a9b0a281632bcea0d031a
3
+ size 33390
results/icml_r3/X8/tables/val_retr_ref_it40_lr0.2_n200.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c11b4534fb8befcc3992533c9d12d8798947fbe08091ecafb1b7b6e72adb0a72
3
+ size 33390
results/icml_r3/X8/tables/val_retr_ref_it80_lr0.1_n200.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:64ad1742d310f0ab456ab9e2a38ad2700aaa17952ecf8f664038f97be0887c9b
3
+ size 33390
results/icml_r3/X8/tables/val_retr_ref_it80_lr0.2_n200.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:937604109bb88fb469d9b605fdf3bf1c8b18c6da5a04264f6089a20a9487bcf2
3
+ size 33390
results/icml_r3/X8/tables/val_twin.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fdf942f390c1fb8af0bacafebadfca4b3ef8ed15b3669580ec8533e39696eafe
3
+ size 448094
results/icml_r3/X8/tables/val_twin_ref_it40_lr0.1_n200.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:13b019039affff01ee7791c140c55cc424a665d7b3605ba1dbf2b96dd3722bf9
3
+ size 49710
results/icml_r3/X8/tables/val_twin_ref_it40_lr0.1_n400.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:932cf3ef0a9b378ec3a8f3f7e4eefd59e0c5977a8e1a8d05751a10ea60c703a9
3
+ size 49710
results/icml_r3/X8/tables/val_twin_ref_it40_lr0.2_n200.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:386d1465fda53f1137b5542951641f2c8c224ecbaa5853130f29547a712abf2d
3
+ size 49710
results/icml_r3/X8/tables/val_twin_ref_it80_lr0.1_n200.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:afab4f0be0f837cda27656378d71d132182c318077ea4a8d2cbd203edf0cb271
3
+ size 49710
results/icml_r3/X8/tables/val_twin_ref_it80_lr0.2_n200.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bce8579bc72c2eab7ea3de7d494d6e058f8ba5b365a3ff5133ab27e4220d5cb8
3
+ size 49710
results/icml_r3/X8/test_report.txt ADDED
@@ -0,0 +1,154 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # X8 / X18 test report (corrected arm, n=300; bank1 = 2905 rows); refine config {'iters': 80, 'lr': 0.1, 'n': 200}; frozen primary pipeline rank_refine; retr_learned metric = xfer
2
+
3
+ ## Refinement statistics on the test candidates (per arm: valid, certified, certified-but-failed, rescued by refinement)
4
+ generator valid 14400 certified 14400 certified-failed 793 rescued 786 (99.1%)
5
+ retr_unit valid 4800 certified 4800 certified-failed 3947 rescued 3893 (98.6%)
6
+ retr_learned valid 4800 certified 4800 certified-failed 3787 rescued 3717 (98.2%)
7
+ twin valid 43200 certified 43200 certified-failed 8003 rescued 7758 (96.9%)
8
+
9
+ ## Test success % [95% CI over plants], pipeline = plain
10
+ arm b=1 b=2 b=4 b=8 b=16
11
+ generator (seed 0, 3 sampler seeds) 94.0 [91.8,96.0] 97.1 [95.4,98.6] 98.2 [96.9,99.3] 99.2 [98.1,100.0] 99.3 [98.3,100.0]
12
+ retr_unit (X2 baseline) 40.0 [34.7,45.7] 57.0 [51.3,62.3] 71.3 [66.3,76.3] 85.0 [81.0,89.0] 93.3 [90.3,96.0]
13
+ retr_learned 42.0 [36.3,47.3] 61.0 [55.3,66.3] 76.7 [72.0,81.3] 90.7 [87.3,93.7] 96.7 [94.7,98.7]
14
+ twin (3 members x 3 draw seeds) 85.0 [81.9,88.0] 88.4 [85.6,91.0] 91.9 [89.5,94.1] 94.9 [92.9,96.7] 96.2 [94.3,97.9]
15
+
16
+ ## Test success % [95% CI over plants], pipeline = refine
17
+ arm b=1 b=2 b=4 b=8 b=16
18
+ generator (seed 0, 3 sampler seeds) 94.0 [91.9,96.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
19
+ retr_unit (X2 baseline) 40.0 [34.7,45.3] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
20
+ retr_learned 42.0 [36.3,47.7] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.7 [99.0,100.0]
21
+ twin (3 members x 3 draw seeds) 85.0 [81.9,88.1] 99.3 [98.4,99.9] 99.6 [98.8,100.0] 99.6 [98.8,100.0] 99.7 [99.0,100.0]
22
+
23
+ ## Test success % [95% CI over plants], pipeline = rank
24
+ arm b=1 b=2 b=4 b=8 b=16
25
+ generator (seed 0, 3 sampler seeds) 99.1 [98.0,99.9] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
26
+ retr_unit (X2 baseline) 89.0 [85.3,92.3] 91.7 [88.3,94.7] 92.7 [89.7,95.3] 92.7 [89.7,95.3] 93.3 [90.3,96.0]
27
+ retr_learned 93.7 [90.7,96.3] 95.0 [92.3,97.3] 96.7 [94.3,98.7] 96.7 [94.7,98.7] 96.7 [94.3,98.7]
28
+ twin (3 members x 3 draw seeds) 96.0 [94.1,97.6] 96.0 [94.1,97.7] 96.1 [94.2,97.7] 96.1 [94.3,97.8] 96.2 [94.4,97.9]
29
+
30
+ ## Test success % [95% CI over plants], pipeline = rank_refine <- FROZEN PRIMARY
31
+ arm b=1 b=2 b=4 b=8 b=16
32
+ generator (seed 0, 3 sampler seeds) 99.1 [98.0,99.9] 99.9 [99.7,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
33
+ retr_unit (X2 baseline) 89.0 [85.3,92.3] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
34
+ retr_learned 93.7 [90.7,96.3] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
35
+ twin (3 members x 3 draw seeds) 96.0 [94.1,97.7] 99.6 [98.8,100.0] 99.6 [98.9,100.0] 99.6 [98.9,100.0] 99.7 [99.0,100.0]
36
+
37
+ ## Test success % [95% CI over plants], pipeline = refine_free
38
+ arm b=1 b=2 b=4 b=8 b=16
39
+ generator (seed 0, 3 sampler seeds) 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
40
+ retr_unit (X2 baseline) 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
41
+ retr_learned 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.7 [99.0,100.0] 100.0 [100.0,100.0]
42
+ twin (3 members x 3 draw seeds) 99.3 [98.4,99.9] 99.6 [98.8,100.0] 99.6 [98.8,100.0] 99.7 [99.0,100.0] 99.7 [99.0,100.0]
43
+
44
+ ## Test success % [95% CI over plants], pipeline = rank_refine_free
45
+ arm b=1 b=2 b=4 b=8 b=16
46
+ generator (seed 0, 3 sampler seeds) 99.9 [99.7,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
47
+ retr_unit (X2 baseline) 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
48
+ retr_learned 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
49
+ twin (3 members x 3 draw seeds) 99.6 [98.9,100.0] 99.6 [98.9,100.0] 99.6 [98.9,100.0] 99.7 [99.0,100.0] 99.7 [99.0,100.0]
50
+
51
+ ## Test success % [95% CI over plants], pipeline = refine_c3
52
+ arm b=1 b=2 b=4 b=8 b=16
53
+ generator (seed 0, 3 sampler seeds) 94.0 [91.8,96.0] 94.0 [91.8,96.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
54
+ retr_unit (X2 baseline) 40.0 [34.7,45.7] 40.0 [34.7,45.7] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
55
+ retr_learned 42.0 [36.3,47.7] 42.0 [36.3,47.3] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
56
+ twin (3 members x 3 draw seeds) 85.0 [81.7,88.1] 85.0 [81.7,88.1] 99.3 [98.4,99.9] 99.6 [98.8,100.0] 99.6 [98.8,100.0]
57
+
58
+ ## Test success % [95% CI over plants], pipeline = rank_refine_c3
59
+ arm b=1 b=2 b=4 b=8 b=16
60
+ generator (seed 0, 3 sampler seeds) 99.1 [98.0,99.9] 99.1 [98.0,99.9] 99.9 [99.7,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
61
+ retr_unit (X2 baseline) 89.0 [85.3,92.3] 89.0 [85.3,92.3] 99.3 [98.3,100.0] 99.3 [98.3,100.0] 99.3 [98.3,100.0]
62
+ retr_learned 93.7 [90.7,96.3] 93.7 [90.7,96.3] 100.0 [100.0,100.0] 100.0 [100.0,100.0] 100.0 [100.0,100.0]
63
+ twin (3 members x 3 draw seeds) 96.0 [94.1,97.6] 96.0 [94.1,97.7] 99.6 [98.9,100.0] 99.6 [98.9,100.0] 99.6 [98.9,100.0]
64
+
65
+ ## Descriptive: mean number of verifier calls to the first success (a failure within 16 calls counts as 17), per arm; mean over replicas and plants
66
+ plain generator 1.22 retr_unit 4.22 retr_learned 3.48 twin 2.02
67
+ refine generator 1.06 retr_unit 1.70 retr_learned 1.67 twin 1.22
68
+ rank generator 1.11 retr_unit 2.19 retr_learned 1.58 twin 1.62
69
+ rank_refine generator 1.01 retr_unit 1.21 retr_learned 1.06 twin 1.10
70
+ refine_free generator 1.00 retr_unit 1.11 retr_learned 1.05 twin 1.06
71
+ rank_refine_free generator 1.00 retr_unit 1.11 retr_learned 1.00 twin 1.06
72
+ refine_c3 generator 1.18 retr_unit 2.89 retr_learned 2.83 twin 1.52
73
+ rank_refine_c3 generator 1.03 retr_unit 1.42 retr_learned 1.19 twin 1.17
74
+
75
+ ## PRIMARY family (9 tests, Holm): generator minus comparator under rank_refine (counted refinement), pp [95% CI over plants], sign-flip p, Holm p, McNemar (gen-only/comp-only; sampler seed 0 vs twin member 0 draw 0) p
76
+ gen - retr_unit, b=1, pipeline rank_refine +10.1 [ +6.9,+13.7] p=0.0000 holm=0.0000 McNemar 31/1 p=0.0000
77
+ gen - retr_learned, b=1, pipeline rank_refine +5.4 [ +2.8, +8.4] p=0.0001 holm=0.0003 McNemar 18/2 p=0.0004
78
+ gen - twin, b=1, pipeline rank_refine +3.1 [ +1.6, +4.9] p=0.0000 holm=0.0000 McNemar 6/1 p=0.1250
79
+ gen - retr_unit, b=4, pipeline rank_refine +0.7 [ +0.0, +1.7] p=0.5009 holm=1.0000 McNemar 2/0 p=0.5000
80
+ gen - retr_learned, b=4, pipeline rank_refine +0.0 [ +0.0, +0.0] p=1.0000 holm=1.0000 McNemar 0/0 p=1.0000
81
+ gen - twin, b=4, pipeline rank_refine +0.4 [ +0.0, +1.2] p=0.5045 holm=1.0000 McNemar 1/0 p=1.0000
82
+ gen - retr_unit, b=16, pipeline rank_refine +0.7 [ +0.0, +1.7] p=0.4913 holm=1.0000 McNemar 2/0 p=0.5000
83
+ gen - retr_learned, b=16, pipeline rank_refine +0.0 [ +0.0, +0.0] p=1.0000 holm=1.0000 McNemar 0/0 p=1.0000
84
+ gen - twin, b=16, pipeline rank_refine +0.3 [ +0.0, +1.0] p=1.0000 holm=1.0000 McNemar 1/0 p=1.0000
85
+
86
+ ## SECONDARY (unadjusted): generator minus comparator, every pipeline, pp [95% CI]
87
+ plain gen - retr_unit (X2 baseline) +54.0 [+48.0,+59.8] p=0.000 +40.1 [+34.2,+45.9] p=0.000 +26.9 [+21.8,+32.0] p=0.000 +14.2 [+10.3,+18.3] p=0.000 +6.0 [ +3.3, +9.0] p=0.000
88
+ plain gen - retr_learned +52.0 [+46.1,+57.8] p=0.000 +36.1 [+30.3,+41.9] p=0.000 +21.6 [+16.9,+26.6] p=0.000 +8.6 [ +5.3,+12.0] p=0.000 +2.7 [ +0.7, +4.7] p=0.021
89
+ plain gen - twin (3 members x 3 draw seeds) +9.0 [ +6.0,+12.1] p=0.000 +8.7 [ +6.1,+11.4] p=0.000 +6.3 [ +4.2, +8.6] p=0.000 +4.3 [ +2.7, +6.2] p=0.000 +3.1 [ +1.6, +4.8] p=0.000
90
+ refine gen - retr_unit (X2 baseline) +54.0 [+48.0,+59.8] p=0.000 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.501
91
+ refine gen - retr_learned +52.0 [+46.2,+58.0] p=0.000 +0.7 [ +0.0, +1.7] p=0.502 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.504 +0.3 [ +0.0, +1.0] p=1.000
92
+ refine gen - twin (3 members x 3 draw seeds) +9.0 [ +5.9,+12.1] p=0.000 +0.7 [ +0.1, +1.6] p=0.125 +0.4 [ +0.0, +1.2] p=0.256 +0.4 [ +0.0, +1.2] p=0.249 +0.3 [ +0.0, +1.0] p=1.000
93
+ rank gen - retr_unit (X2 baseline) +10.1 [ +6.8,+13.7] p=0.000 +7.7 [ +4.7,+11.0] p=0.000 +6.7 [ +4.0, +9.7] p=0.000 +6.7 [ +4.0, +9.7] p=0.000 +6.0 [ +3.3, +8.7] p=0.000
94
+ rank gen - retr_learned +5.4 [ +2.8, +8.3] p=0.000 +4.3 [ +2.0, +7.0] p=0.001 +2.7 [ +0.7, +5.0] p=0.021 +2.7 [ +0.7, +4.7] p=0.022 +2.7 [ +0.7, +5.0] p=0.023
95
+ rank gen - twin (3 members x 3 draw seeds) +3.1 [ +1.6, +4.8] p=0.000 +3.3 [ +1.9, +5.1] p=0.000 +3.2 [ +1.7, +5.0] p=0.000 +3.2 [ +1.7, +4.9] p=0.000 +3.1 [ +1.6, +4.8] p=0.000
96
+ rank_refine gen - retr_unit (X2 baseline) +10.1 [ +6.9,+13.7] p=0.000 +0.6 [ +0.0, +1.4] p=0.502 +0.7 [ +0.0, +1.7] p=0.501 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.500
97
+ rank_refine gen - retr_learned +5.4 [ +2.8, +8.3] p=0.000 -0.1 [ -0.3, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000
98
+ rank_refine gen - twin (3 members x 3 draw seeds) +3.1 [ +1.6, +4.8] p=0.000 +0.3 [ +0.0, +0.8] p=0.495 +0.4 [ +0.0, +1.1] p=0.495 +0.4 [ +0.0, +1.1] p=0.500 +0.3 [ +0.0, +1.0] p=1.000
99
+ refine_free gen - retr_unit (X2 baseline) +0.7 [ +0.0, +1.7] p=0.497 +0.7 [ +0.0, +1.7] p=0.504 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.502
100
+ refine_free gen - retr_learned +0.7 [ +0.0, +1.7] p=0.500 +0.7 [ +0.0, +1.7] p=0.504 +0.7 [ +0.0, +1.7] p=0.501 +0.3 [ +0.0, +1.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000
101
+ refine_free gen - twin (3 members x 3 draw seeds) +0.7 [ +0.1, +1.6] p=0.130 +0.4 [ +0.0, +1.2] p=0.252 +0.4 [ +0.0, +1.2] p=0.255 +0.3 [ +0.0, +1.0] p=1.000 +0.3 [ +0.0, +1.0] p=1.000
102
+ rank_refine_free gen - retr_unit (X2 baseline) +0.6 [ +0.0, +1.4] p=0.502 +0.7 [ +0.0, +1.7] p=0.496 +0.7 [ +0.0, +1.7] p=0.498 +0.7 [ +0.0, +1.7] p=0.506 +0.7 [ +0.0, +1.7] p=0.498
103
+ rank_refine_free gen - retr_learned -0.1 [ -0.3, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000
104
+ rank_refine_free gen - twin (3 members x 3 draw seeds) +0.3 [ +0.0, +0.8] p=0.505 +0.4 [ +0.0, +1.1] p=0.496 +0.4 [ +0.0, +1.1] p=0.498 +0.3 [ +0.0, +1.0] p=1.000 +0.3 [ +0.0, +1.0] p=1.000
105
+ refine_c3 gen - retr_unit (X2 baseline) +54.0 [+48.1,+59.9] p=0.000 +54.0 [+48.0,+60.0] p=0.000 +0.7 [ +0.0, +1.7] p=0.501 +0.7 [ +0.0, +1.7] p=0.497 +0.7 [ +0.0, +1.7] p=0.500
106
+ refine_c3 gen - retr_learned +52.0 [+46.1,+58.0] p=0.000 +52.0 [+46.0,+57.9] p=0.000 +0.7 [ +0.0, +1.7] p=0.496 +0.7 [ +0.0, +1.7] p=0.497 +0.7 [ +0.0, +1.7] p=0.499
107
+ refine_c3 gen - twin (3 members x 3 draw seeds) +9.0 [ +6.0,+12.0] p=0.000 +9.0 [ +6.0,+12.0] p=0.000 +0.7 [ +0.1, +1.6] p=0.121 +0.4 [ +0.0, +1.2] p=0.248 +0.4 [ +0.0, +1.2] p=0.248
108
+ rank_refine_c3 gen - retr_unit (X2 baseline) +10.1 [ +6.8,+13.6] p=0.000 +10.1 [ +6.8,+13.7] p=0.000 +0.6 [ +0.0, +1.4] p=0.498 +0.7 [ +0.0, +1.7] p=0.505 +0.7 [ +0.0, +1.7] p=0.503
109
+ rank_refine_c3 gen - retr_learned +5.4 [ +2.9, +8.3] p=0.000 +5.4 [ +2.8, +8.3] p=0.000 -0.1 [ -0.3, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000 +0.0 [ +0.0, +0.0] p=1.000
110
+ rank_refine_c3 gen - twin (3 members x 3 draw seeds) +3.1 [ +1.6, +4.8] p=0.000 +3.1 [ +1.7, +4.9] p=0.000 +0.3 [ +0.0, +0.8] p=0.498 +0.4 [ +0.0, +1.1] p=0.503 +0.4 [ +0.0, +1.1] p=0.504
111
+
112
+ ## Across training seeds (unit = seed; frozen pipeline rank_refine; seeds 1-4 use sampler seed 0 only, seed 0 is the mean over its 3 sampler seeds from the main table)
113
+ b=1: generator per seed [99.1, 99.7, 99.3, 99.7, 99.7]
114
+ minus retr_unit: per seed [10.1, 10.7, 10.3, 10.7, 10.7] mean +10.5 t95 [+10.2,+10.8] seeds positive 5/5
115
+ minus retr_learned: per seed [5.4, 6.0, 5.7, 6.0, 6.0] mean +5.8 t95 [+5.5,+6.1] seeds positive 5/5
116
+ minus twin: per seed [3.1, 3.7, 3.3, 3.7, 3.7] mean +3.5 t95 [+3.2,+3.8] seeds positive 5/5
117
+ b=4: generator per seed [100.0, 100.0, 100.0, 100.0, 100.0]
118
+ minus retr_unit: per seed [0.7, 0.7, 0.7, 0.7, 0.7] mean +0.7 t95 [+0.7,+0.7] seeds positive 5/5
119
+ minus retr_learned: per seed [0.0, 0.0, 0.0, 0.0, 0.0] mean +0.0 t95 [+0.0,+0.0] seeds positive 0/5
120
+ minus twin: per seed [0.4, 0.4, 0.4, 0.4, 0.4] mean +0.4 t95 [+0.4,+0.4] seeds positive 5/5
121
+ b=16: generator per seed [100.0, 100.0, 100.0, 100.0, 100.0]
122
+ minus retr_unit: per seed [0.7, 0.7, 0.7, 0.7, 0.7] mean +0.7 t95 [+0.7,+0.7] seeds positive 5/5
123
+ minus retr_learned: per seed [0.0, 0.0, 0.0, 0.0, 0.0] mean +0.0 t95 [+0.0,+0.0] seeds positive 0/5
124
+ minus twin: per seed [0.3, 0.3, 0.3, 0.3, 0.3] mean +0.3 t95 [+0.3,+0.3] seeds positive 5/5
125
+
126
+ # X18: twin ensemble (plain verification, no repair)
127
+ arm (plain) b=1 b=3 b=4 b=8 b=16
128
+ generator 94.0 [91.9,96.0] 97.7 [96.1,99.0] 98.2 [96.9,99.3] 99.2 [98.2,100.0] 99.3 [98.3,100.0]
129
+ twin ensemble (3 members, 9 replicas) 85.0 [81.8,88.1] 95.3 [92.7,97.7] 95.7 [93.4,97.8] 96.9 [94.9,98.6] 97.7 [96.1,99.1]
130
+ twin single member, MC dropout (9 replicas) 85.0 [81.8,88.0] 90.7 [88.1,93.1] 91.9 [89.4,94.1] 94.9 [92.9,96.7] 96.2 [94.3,97.9]
131
+
132
+ ## X18 PRIMARY family (4 tests, Holm), pp [95% CI], sign-flip p, Holm p, McNemar (first replicas) p
133
+ ensemble - single-member twin, b=4 +3.9 [ +2.5, +5.3] p=0.0000 holm=0.0000 McNemar 6/1 p=0.1250
134
+ ensemble - single-member twin, b=16 +1.5 [ +0.8, +2.3] p=0.0000 holm=0.0000 McNemar 2/0 p=0.5000
135
+ generator - ensemble, b=4 +2.5 [ +0.6, +4.6] p=0.0176 holm=0.0353 McNemar 10/2 p=0.0386
136
+ generator - ensemble, b=16 +1.6 [ +0.4, +3.1] p=0.0213 holm=0.0353 McNemar 4/0 p=0.1250
137
+
138
+ ## X18 SECONDARY (unadjusted), every b: ensemble - single twin | generator - ensemble | generator - single twin
139
+ b= 1: +0.0 [ +0.0, +0.0] p=1.000 | +9.0 [ +6.0,+12.1] p=0.000 | +9.0 [ +5.9,+12.1] p=0.000
140
+ b= 2: +4.1 [ +2.9, +5.4] p=0.000 | +4.6 [ +2.2, +7.1] p=0.000 | +8.7 [ +6.1,+11.3] p=0.000
141
+ b= 3: +4.7 [ +3.1, +6.3] p=0.000 | +2.3 [ +0.1, +4.8] p=0.059 | +7.0 [ +4.7, +9.4] p=0.000
142
+ b= 4: +3.9 [ +2.6, +5.3] p=0.000 | +2.5 [ +0.6, +4.6] p=0.016 | +6.3 [ +4.3, +8.7] p=0.000
143
+ b= 6: +2.6 [ +1.6, +3.7] p=0.000 | +2.4 [ +0.9, +4.2] p=0.006 | +5.0 [ +3.2, +7.0] p=0.000
144
+ b= 8: +2.0 [ +1.1, +2.9] p=0.000 | +2.4 [ +0.9, +4.1] p=0.003 | +4.3 [ +2.7, +6.2] p=0.000
145
+ b=12: +1.3 [ +0.5, +2.1] p=0.001 | +2.3 [ +0.9, +4.0] p=0.004 | +3.6 [ +2.1, +5.4] p=0.000
146
+ b=16: +1.5 [ +0.8, +2.3] p=0.000 | +1.6 [ +0.4, +3.1] p=0.024 | +3.1 [ +1.6, +4.8] p=0.000
147
+
148
+ ## X18 with repair (secondary): ensemble vs single twin vs generator at b = 1, 4, 16, pipelines refine / rank_refine
149
+ refine: ens 85.0 99.7 99.7 | twin 85.0 99.6 99.7 | gen 94.0 100.0 100.0
150
+ rank_refine: ens 97.7 99.7 99.7 | twin 96.0 99.6 99.7 | gen 99.1 100.0 100.0
151
+
152
+ ## Check: plain pipeline vs X2 (success % at b = 1, 4, 16; X2 values from results/icml_r3/X2/test_*.npz)
153
+ gen here 94.0 98.2 99.3 | X2 94.1 98.2 99.3
154
+ retr here 40.0 71.3 93.3 | X2 40.0 71.3 93.3
results/icml_r3/X8/val_selection.txt ADDED
@@ -0,0 +1,53 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # X8 validation (60 plants; gen seed 0 sampler 0, retr_unit, retr_learned (metric xfer), twin members 0-2 draw seed 0)
2
+
3
+ refine config it40_lr0.1_n200: rescued 2097 of 2187 certified-failed candidates = 95.9% (per arm: gen0_s0 70/76, retr_unit 761/797, retr_learned 713/747, twin0_d0 195/195, twin1_d0 160/174, twin2_d0 198/198)
4
+ refine config it80_lr0.1_n200: rescued 2141 of 2187 certified-failed candidates = 97.9% (per arm: gen0_s0 75/76, retr_unit 777/797, retr_learned 722/747, twin0_d0 195/195, twin1_d0 174/174, twin2_d0 198/198)
5
+ refine config it40_lr0.2_n200: rescued 2139 of 2187 certified-failed candidates = 97.8% (per arm: gen0_s0 76/76, retr_unit 775/797, retr_learned 721/747, twin0_d0 195/195, twin1_d0 174/174, twin2_d0 198/198)
6
+ refine config it80_lr0.2_n200: rescued 2135 of 2187 certified-failed candidates = 97.6% (per arm: gen0_s0 76/76, retr_unit 775/797, retr_learned 717/747, twin0_d0 195/195, twin1_d0 174/174, twin2_d0 198/198)
7
+ refine config it40_lr0.1_n400: rescued 2097 of 2187 certified-failed candidates = 95.9% (per arm: gen0_s0 70/76, retr_unit 760/797, retr_learned 714/747, twin0_d0 195/195, twin1_d0 160/174, twin2_d0 198/198)
8
+ FROZEN refine config: {'iters': 80, 'lr': 0.1, 'n': 200} (highest pooled rescue rate; ties -> cheaper = earlier in grid)
9
+
10
+ # pipelines on validation, success % at b = 1/2/4/8/16 per arm (frozen refine config)
11
+ plain gen0_s0 88.3 93.3 98.3 98.3 98.3
12
+ plain retr_unit 31.7 50.0 73.3 85.0 93.3
13
+ plain retr_learned 35.0 55.0 70.0 88.3 95.0
14
+ plain twin0_d0 83.3 85.0 85.0 91.7 95.0
15
+ plain twin1_d0 86.7 95.0 98.3 98.3 98.3
16
+ plain twin2_d0 81.7 90.0 90.0 93.3 95.0
17
+ plain baseline mean over b=1,4,16 (retr_unit, retr_learned, twin): 74.38
18
+ refine gen0_s0 88.3 100.0 100.0 100.0 100.0
19
+ refine retr_unit 31.7 100.0 100.0 100.0 100.0
20
+ refine retr_learned 35.0 100.0 100.0 100.0 100.0
21
+ refine twin0_d0 83.3 100.0 100.0 100.0 100.0
22
+ refine twin1_d0 86.7 100.0 100.0 100.0 100.0
23
+ refine twin2_d0 81.7 100.0 100.0 100.0 100.0
24
+ refine baseline mean over b=1,4,16 (retr_unit, retr_learned, twin): 83.40
25
+ rank gen0_s0 98.3 98.3 98.3 98.3 98.3
26
+ rank retr_unit 83.3 88.3 93.3 93.3 93.3
27
+ rank retr_learned 88.3 93.3 93.3 95.0 95.0
28
+ rank twin0_d0 93.3 93.3 93.3 93.3 95.0
29
+ rank twin1_d0 98.3 98.3 98.3 98.3 98.3
30
+ rank twin2_d0 93.3 93.3 93.3 93.3 95.0
31
+ rank baseline mean over b=1,4,16 (retr_unit, retr_learned, twin): 92.53
32
+ rank_refine gen0_s0 98.3 100.0 100.0 100.0 100.0
33
+ rank_refine retr_unit 83.3 100.0 100.0 100.0 100.0
34
+ rank_refine retr_learned 88.3 100.0 100.0 100.0 100.0
35
+ rank_refine twin0_d0 93.3 100.0 100.0 100.0 100.0
36
+ rank_refine twin1_d0 98.3 100.0 100.0 100.0 100.0
37
+ rank_refine twin2_d0 93.3 100.0 100.0 100.0 100.0
38
+ rank_refine baseline mean over b=1,4,16 (retr_unit, retr_learned, twin): 96.30
39
+ refine_free gen0_s0 100.0 100.0 100.0 100.0 100.0
40
+ refine_free retr_unit 100.0 100.0 100.0 100.0 100.0
41
+ refine_free retr_learned 100.0 100.0 100.0 100.0 100.0
42
+ refine_free twin0_d0 100.0 100.0 100.0 100.0 100.0
43
+ refine_free twin1_d0 100.0 100.0 100.0 100.0 100.0
44
+ refine_free twin2_d0 100.0 100.0 100.0 100.0 100.0
45
+ refine_free baseline mean over b=1,4,16 (retr_unit, retr_learned, twin): 100.00
46
+ rank_refine_free gen0_s0 100.0 100.0 100.0 100.0 100.0
47
+ rank_refine_free retr_unit 100.0 100.0 100.0 100.0 100.0
48
+ rank_refine_free retr_learned 100.0 100.0 100.0 100.0 100.0
49
+ rank_refine_free twin0_d0 100.0 100.0 100.0 100.0 100.0
50
+ rank_refine_free twin1_d0 100.0 100.0 100.0 100.0 100.0
51
+ rank_refine_free twin2_d0 100.0 100.0 100.0 100.0 100.0
52
+ rank_refine_free baseline mean over b=1,4,16 (retr_unit, retr_learned, twin): 100.00
53
+ FROZEN primary pipeline: rank_refine (higher mean baseline-arm success over b = 1, 4, 16 among refine / rank_refine; ties -> rank_refine)