matCercola18 commited on
Commit
c00382a
·
verified ·
1 Parent(s): 28d44af

results/icml_r3/X18

Browse files
results/icml_r3/X18/PREREG.md ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # X18 PREREG: twin ensemble on the corrected arm (written 2026-10-13 09:06 PDT, before any test scoring of the ensemble)
2
+
3
+ virgo CPU only, <= 12 workers. Code: `nonlinear/experiments/furuta_x8.py` (ensemble sequences), `furuta_r3b_engine.py`. Protocol = X2 (plants, verifier, bank). Models: `models/v4/x2_reg_mcd_seed{0,1,2}_s96000.pth` (MC-dropout regression twin, the S2 family of X2).
4
+
5
+ ## Question
6
+ The X2 twin comparator (deterministic at b = 1, MC dropout beyond, one member) is a single regressor. Does using the 3 twin seeds as an ensemble (one candidate per member, MC dropout within members to fill the budget) close the gap to the generator?
7
+
8
+ ## Arms (plain verification, no repair; the repaired version is X8 with the same candidates)
9
+ - **Ensemble replicate (r, s)**, r in {0, 1, 2} (member rotation), s in {0, 1, 2} (MC-dropout draw seed): candidates 1-3 = the deterministic predictions of members r, r+1, r+2 (mod 3); candidates 4-16 = MC-dropout draws (jitter 0, draw seed s) taken round-robin over members r, r+1, r+2 (draw d of member m is entry 1+d of its [deterministic, draws] list); 16 candidates. 9 replicas.
10
+ - **MC-dropout-only twin (single member)**: member i in {0, 1, 2}, draw seed s: [deterministic prediction, 15 MC-dropout draws]; 9 replicas (3 members x 3 draw seeds). At b = 1 it equals the ensemble by construction (the same deterministic predictions, rotated).
11
+ - **Generator**: `unet_a` seed 0 (S1 config), 3 cached sampler seeds, first b samples (the X2 arm).
12
+ - Candidates are the cached X2 test samples; each is verified once with the unchanged verifier, in the stated order; success at b = any of the first b candidates passes.
13
+
14
+ ## Tests (no selection is made: everything is fixed above; validation is used only as a sanity check of the pipeline)
15
+ - Primary family, **4 tests**, Holm alpha 0.05: (ensemble minus MC-dropout-only twin) and (generator minus ensemble), each at b = 4 and b = 16. Statistic: mean paired per-plant difference (pp) over 300 plants, per-plant success averaged over the replicas, percentile bootstrap CI over plants (10,000), two-sided sign-flip p (20,000).
16
+ - Secondary (unadjusted): the same differences at b = 1..16 and at b = 3 (all members used once); the X2 draws-only twin ("test-time diversity") row; the ensemble with repair is read from X8 (not a separate test).
17
+ - The resampling unit for the ensemble is the plant: there is exactly one 3-member ensemble (3 twin seeds exist), so no across-ensemble seed replication is possible; the rotation and draw-seed replicas are within-ensemble. This is a stated limitation.
18
+ - Falsification: the ensemble "closes the gap" if generator minus ensemble has CI containing 0 at b = 4 and b = 16; the generator's lead over the single-member twin is "diversity-limited" if ensemble minus twin is significantly positive.
results/icml_r3/X18/test_report.txt ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # X18: twin ensemble (plain verification, no repair)
2
+ arm (plain) b=1 b=3 b=4 b=8 b=16
3
+ generator 94.0 [91.9,96.0] 97.7 [96.1,99.0] 98.2 [96.9,99.3] 99.2 [98.2,100.0] 99.3 [98.3,100.0]
4
+ twin ensemble (3 members, 9 replicas) 85.0 [81.8,88.1] 95.3 [92.7,97.7] 95.7 [93.4,97.8] 96.9 [94.9,98.6] 97.7 [96.1,99.1]
5
+ twin single member, MC dropout (9 replicas) 85.0 [81.8,88.0] 90.7 [88.1,93.1] 91.9 [89.4,94.1] 94.9 [92.9,96.7] 96.2 [94.3,97.9]
6
+
7
+ ## X18 PRIMARY family (4 tests, Holm), pp [95% CI], sign-flip p, Holm p, McNemar (first replicas) p
8
+ ensemble - single-member twin, b=4 +3.9 [ +2.5, +5.3] p=0.0000 holm=0.0000 McNemar 6/1 p=0.1250
9
+ ensemble - single-member twin, b=16 +1.5 [ +0.8, +2.3] p=0.0000 holm=0.0000 McNemar 2/0 p=0.5000
10
+ generator - ensemble, b=4 +2.5 [ +0.6, +4.6] p=0.0176 holm=0.0353 McNemar 10/2 p=0.0386
11
+ generator - ensemble, b=16 +1.6 [ +0.4, +3.1] p=0.0213 holm=0.0353 McNemar 4/0 p=0.1250
12
+
13
+ ## X18 SECONDARY (unadjusted), every b: ensemble - single twin | generator - ensemble | generator - single twin
14
+ b= 1: +0.0 [ +0.0, +0.0] p=1.000 | +9.0 [ +6.0,+12.1] p=0.000 | +9.0 [ +5.9,+12.1] p=0.000
15
+ b= 2: +4.1 [ +2.9, +5.4] p=0.000 | +4.6 [ +2.2, +7.1] p=0.000 | +8.7 [ +6.1,+11.3] p=0.000
16
+ b= 3: +4.7 [ +3.1, +6.3] p=0.000 | +2.3 [ +0.1, +4.8] p=0.059 | +7.0 [ +4.7, +9.4] p=0.000
17
+ b= 4: +3.9 [ +2.6, +5.3] p=0.000 | +2.5 [ +0.6, +4.6] p=0.016 | +6.3 [ +4.3, +8.7] p=0.000
18
+ b= 6: +2.6 [ +1.6, +3.7] p=0.000 | +2.4 [ +0.9, +4.2] p=0.006 | +5.0 [ +3.2, +7.0] p=0.000
19
+ b= 8: +2.0 [ +1.1, +2.9] p=0.000 | +2.4 [ +0.9, +4.1] p=0.003 | +4.3 [ +2.7, +6.2] p=0.000
20
+ b=12: +1.3 [ +0.5, +2.1] p=0.001 | +2.3 [ +0.9, +4.0] p=0.004 | +3.6 [ +2.1, +5.4] p=0.000
21
+ b=16: +1.5 [ +0.8, +2.3] p=0.000 | +1.6 [ +0.4, +3.1] p=0.024 | +3.1 [ +1.6, +4.8] p=0.000
22
+
23
+ ## X18 with repair (secondary): ensemble vs single twin vs generator at b = 1, 4, 16, pipelines refine / rank_refine
24
+ refine: ens 85.0 99.7 99.7 | twin 85.0 99.6 99.7 | gen 94.0 100.0 100.0
25
+ rank_refine: ens 97.7 99.7 99.7 | twin 96.0 99.6 99.7 | gen 99.1 100.0 100.0
26
+
27
+ ## Check: plain pipeline vs X2 (success % at b = 1, 4, 16; X2 values from results/icml_r3/X2/test_*.npz)
28
+ gen here 94.0 98.2 99.3 | X2 94.1 98.2 99.3
29
+ retr here 40.0 71.3 93.3 | X2 40.0 71.3 93.3