matCercola18's picture
Add files using upload-large-folder tool
929e74d verified
Raw History Blame Contribute Delete
10.7 kB
=== K=1 (N=80 plants; model rows = mean over runs; regret on the 24 fresh scoring ICs vs the fresh-IC oracle) ===
arm stable% cert% anyCert% reg_med [95% CI] reg_p90 reg_mean in-sample reg_med
retrieval, original bank 100.0 100.0 100.0 3.60 [+2.92,+4.03] 5.57 3.70 1.64
retrieval, multi bank (K nearest rows) 100.0 100.0 100.0 0.36 [+0.13,+0.53] 1.61 0.58 0.66
retrieval, multi bank best-per-plant 100.0 100.0 100.0 0.36 [+0.13,+0.53] 1.61 0.58 0.66
inverse-distance blend, original bank 100.0 100.0 100.0 3.29 [+3.00,+3.63] 5.01 3.42 1.54
tunedA 100.0 98.8 98.8 0.62 [+0.47,+0.81] 1.73 0.91 0.77
shipped 100.0 100.0 100.0 3.27 [+3.12,+3.50] 5.22 3.50 1.54
=== K=2 (N=80 plants; model rows = mean over runs; regret on the 24 fresh scoring ICs vs the fresh-IC oracle) ===
arm stable% cert% anyCert% reg_med [95% CI] reg_p90 reg_mean in-sample reg_med
retrieval, original bank 100.0 100.0 100.0 2.68 [+2.46,+2.99] 4.53 2.80 1.12
retrieval, multi bank (K nearest rows) 100.0 100.0 100.0 0.18 [+0.11,+0.31] 0.86 0.34 0.41
retrieval, multi bank best-per-plant 100.0 100.0 100.0 0.18 [+0.11,+0.31] 0.86 0.34 0.41
inverse-distance blend, original bank 100.0 100.0 100.0 2.94 [+2.68,+3.21] 4.38 2.86 1.30
tunedA 100.0 98.9 99.6 0.52 [+0.39,+0.69] 1.74 0.72 0.47
shipped 100.0 100.0 100.0 3.09 [+2.87,+3.26] 5.02 3.19 1.27
=== K=4 (N=80 plants; model rows = mean over runs; regret on the 24 fresh scoring ICs vs the fresh-IC oracle) ===
arm stable% cert% anyCert% reg_med [95% CI] reg_p90 reg_mean in-sample reg_med
retrieval, original bank 100.0 100.0 100.0 1.52 [+1.31,+1.85] 2.98 1.69 0.56
retrieval, multi bank (K nearest rows) 100.0 98.8 100.0 0.13 [+0.09,+0.20] 0.54 0.15 0.26
retrieval, multi bank best-per-plant 100.0 98.8 100.0 0.13 [+0.09,+0.20] 0.54 0.15 0.26
inverse-distance blend, original bank 100.0 100.0 100.0 1.57 [+1.34,+1.89] 3.39 1.83 0.59
tunedA 100.0 99.0 99.9 0.49 [+0.36,+0.56] 1.53 0.66 0.27
shipped 100.0 100.0 100.0 2.93 [+2.57,+3.12] 4.75 3.00 1.11
=== K=8 (N=80 plants; model rows = mean over runs; regret on the 24 fresh scoring ICs vs the fresh-IC oracle) ===
arm stable% cert% anyCert% reg_med [95% CI] reg_p90 reg_mean in-sample reg_med
retrieval, original bank 100.0 100.0 100.0 0.92 [+0.79,+1.15] 1.92 1.09 0.32
retrieval, multi bank (K nearest rows) 100.0 98.8 100.0 0.11 [+0.06,+0.17] 0.42 0.10 0.18
retrieval, multi bank best-per-plant 100.0 98.8 100.0 0.12 [+0.06,+0.17] 0.42 0.10 0.18
inverse-distance blend, original bank 100.0 100.0 100.0 0.92 [+0.80,+1.15] 1.93 1.12 0.32
tunedA 100.0 98.6 100.0 0.52 [+0.45,+0.63] 1.29 0.66 0.17
shipped 100.0 100.0 100.0 2.82 [+2.46,+3.00] 4.55 2.86 1.02
--- paired tests: tunedA minus other arm (negative = tunedA better); median paired diff in regret points [bootstrap 95% CI], Wilcoxon p, McNemar on majority-stable / picked-certified (n10 = tunedA only, n01 = other only) ---
K=1 vs retrieval, original bank -3.01 [-3.51,-2.36] p=1.5e-13 n=80 | stable McNemar 0-0 p=1 | cert 0-0 p=1 [PRIMARY]
K=1 vs retrieval, multi bank (K nearest rows) +0.19 [+0.11,+0.33] p=0.0037 n=80 | stable McNemar 0-0 p=1 | cert 0-0 p=1 [PRIMARY]
K=1 vs retrieval, multi bank best-per-plant +0.19 [+0.11,+0.33] p=0.0037 n=80 | stable McNemar 0-0 p=1 | cert 0-0 p=1 [exploratory]
K=1 vs inverse-distance blend, original bank -2.66 [-2.86,-2.44] p=2e-14 n=80 | stable McNemar 0-0 p=1 | cert 0-0 p=1 [exploratory]
K=1 vs shipped -2.76 [-3.03,-2.44] p=1.6e-14 n=80 | stable McNemar 0-0 p=1 | cert 0-0 p=1 [exploratory]
K=2 vs retrieval, original bank -2.05 [-2.36,-1.72] p=1.2e-13 n=80 | stable McNemar 0-0 p=1 | cert 0-1 p=1 [exploratory]
K=2 vs retrieval, multi bank (K nearest rows) +0.22 [+0.16,+0.42] p=9.3e-07 n=80 | stable McNemar 0-0 p=1 | cert 0-1 p=1 [exploratory]
K=2 vs retrieval, multi bank best-per-plant +0.22 [+0.16,+0.43] p=1.7e-06 n=80 | stable McNemar 0-0 p=1 | cert 0-1 p=1 [exploratory]
K=2 vs inverse-distance blend, original bank -2.16 [-2.62,-1.94] p=2.7e-14 n=80 | stable McNemar 0-0 p=1 | cert 0-1 p=1 [exploratory]
K=2 vs shipped -2.62 [-2.88,-2.25] p=1.2e-14 n=80 | stable McNemar 0-0 p=1 | cert 0-1 p=1 [exploratory]
K=4 vs retrieval, original bank -0.97 [-1.25,-0.58] p=7.8e-11 n=80 | stable McNemar 0-0 p=1 | cert 0-1 p=1 [exploratory]
K=4 vs retrieval, multi bank (K nearest rows) +0.33 [+0.23,+0.51] p=5.3e-10 n=80 | stable McNemar 0-0 p=1 | cert 1-1 p=1 [exploratory]
K=4 vs retrieval, multi bank best-per-plant +0.33 [+0.23,+0.51] p=5.3e-10 n=80 | stable McNemar 0-0 p=1 | cert 1-1 p=1 [exploratory]
K=4 vs inverse-distance blend, original bank -1.04 [-1.30,-0.61] p=1.2e-11 n=80 | stable McNemar 0-0 p=1 | cert 0-1 p=1 [exploratory]
K=4 vs shipped -2.57 [-2.81,-2.10] p=1.5e-14 n=80 | stable McNemar 0-0 p=1 | cert 0-1 p=1 [exploratory]
K=8 vs retrieval, original bank -0.25 [-0.48,-0.19] p=9.8e-06 n=80 | stable McNemar 0-0 p=1 | cert 0-1 p=1 [PRIMARY]
K=8 vs retrieval, multi bank (K nearest rows) +0.44 [+0.34,+0.53] p=4.9e-13 n=80 | stable McNemar 0-0 p=1 | cert 1-1 p=1 [PRIMARY]
K=8 vs retrieval, multi bank best-per-plant +0.43 [+0.33,+0.53] p=5.3e-13 n=80 | stable McNemar 0-0 p=1 | cert 1-1 p=1 [exploratory]
K=8 vs inverse-distance blend, original bank -0.25 [-0.48,-0.19] p=7.1e-06 n=80 | stable McNemar 0-0 p=1 | cert 0-1 p=1 [exploratory]
K=8 vs shipped -2.25 [-2.62,-1.99] p=1.9e-14 n=80 | stable McNemar 0-0 p=1 | cert 0-1 p=1 [exploratory]
--- paired tests: shipped minus other arm (negative = shipped better); median paired diff in regret points [bootstrap 95% CI], Wilcoxon p, McNemar on majority-stable / picked-certified (n10 = shipped only, n01 = other only) ---
K=1 vs retrieval, original bank -0.19 [-0.65,+0.42] p=0.28 n=80 | stable McNemar 0-0 p=1 | cert 0-0 p=1 [exploratory]
K=1 vs retrieval, multi bank (K nearest rows) +2.89 [+2.56,+3.13] p=9.1e-15 n=80 | stable McNemar 0-0 p=1 | cert 0-0 p=1 [exploratory]
K=1 vs retrieval, multi bank best-per-plant +2.89 [+2.56,+3.13] p=9.1e-15 n=80 | stable McNemar 0-0 p=1 | cert 0-0 p=1 [exploratory]
K=1 vs inverse-distance blend, original bank +0.24 [-0.28,+0.37] p=0.4 n=80 | stable McNemar 0-0 p=1 | cert 0-0 p=1 [exploratory]
K=1 vs tunedA +2.76 [+2.44,+3.03] p=1.6e-14 n=80 | stable McNemar 0-0 p=1 | cert 0-0 p=1 [exploratory]
K=2 vs retrieval, original bank +0.68 [+0.15,+0.93] p=0.015 n=80 | stable McNemar 0-0 p=1 | cert 0-0 p=1 [exploratory]
K=2 vs retrieval, multi bank (K nearest rows) +2.92 [+2.59,+3.12] p=7.8e-15 n=80 | stable McNemar 0-0 p=1 | cert 0-0 p=1 [exploratory]
K=2 vs retrieval, multi bank best-per-plant +2.88 [+2.53,+3.12] p=7.8e-15 n=80 | stable McNemar 0-0 p=1 | cert 0-0 p=1 [exploratory]
K=2 vs inverse-distance blend, original bank +0.34 [+0.06,+0.76] p=0.016 n=80 | stable McNemar 0-0 p=1 | cert 0-0 p=1 [exploratory]
K=2 vs tunedA +2.62 [+2.25,+2.88] p=1.2e-14 n=80 | stable McNemar 0-0 p=1 | cert 1-0 p=1 [exploratory]
K=4 vs retrieval, original bank +1.29 [+0.96,+1.62] p=1.4e-11 n=80 | stable McNemar 0-0 p=1 | cert 0-0 p=1 [exploratory]
K=4 vs retrieval, multi bank (K nearest rows) +2.70 [+2.32,+2.98] p=7.8e-15 n=80 | stable McNemar 0-0 p=1 | cert 1-0 p=1 [exploratory]
K=4 vs retrieval, multi bank best-per-plant +2.70 [+2.32,+2.98] p=7.8e-15 n=80 | stable McNemar 0-0 p=1 | cert 1-0 p=1 [exploratory]
K=4 vs inverse-distance blend, original bank +1.18 [+0.87,+1.58] p=2e-09 n=80 | stable McNemar 0-0 p=1 | cert 0-0 p=1 [exploratory]
K=4 vs tunedA +2.57 [+2.10,+2.81] p=1.5e-14 n=80 | stable McNemar 0-0 p=1 | cert 1-0 p=1 [exploratory]
K=8 vs retrieval, original bank +1.81 [+1.46,+1.95] p=1.8e-14 n=80 | stable McNemar 0-0 p=1 | cert 0-0 p=1 [exploratory]
K=8 vs retrieval, multi bank (K nearest rows) +2.59 [+2.25,+2.97] p=7.8e-15 n=80 | stable McNemar 0-0 p=1 | cert 1-0 p=1 [exploratory]
K=8 vs retrieval, multi bank best-per-plant +2.59 [+2.20,+2.97] p=7.8e-15 n=80 | stable McNemar 0-0 p=1 | cert 1-0 p=1 [exploratory]
K=8 vs inverse-distance blend, original bank +1.78 [+1.42,+1.94] p=1.9e-14 n=80 | stable McNemar 0-0 p=1 | cert 0-0 p=1 [exploratory]
K=8 vs tunedA +2.25 [+1.99,+2.62] p=1.9e-14 n=80 | stable McNemar 0-0 p=1 | cert 1-0 p=1 [exploratory]
Holm-corrected primary tests (alpha=0.05):
tunedA vs retr_orig K=1: p=1.46e-13, Holm threshold 0.0125 -> reject
tunedA vs retr_multi_rows K=8: p=4.9e-13, Holm threshold 0.0167 -> reject
tunedA vs retr_orig K=8: p=9.77e-06, Holm threshold 0.0250 -> reject
tunedA vs retr_multi_rows K=1: p=0.00371, Holm threshold 0.0500 -> reject