X28 write-up: fallback and fresh-plant replication
Browse files- docs/icml_r3_x28.md +40 -8
docs/icml_r3_x28.md
CHANGED
|
@@ -1,11 +1,11 @@
|
|
| 1 |
---
|
| 2 |
-
title: Generating the spline nonlinearity together with the controller (X28). SISO, no gain over phi = sat_1. MIMO, as registered no measurable difference at 16 samples; with the July-style projection of the knots the odd 3-knot spline generator is better at m = 3
|
| 3 |
-
date: 2026-11-
|
| 4 |
-
status: complete (SISO; MIMO optimum, spline-class study, labels, generators, paired test, new directions, test with repair)
|
| 5 |
claim: X28 (Matteo 2026-11-05): the July method (the diffusion model generates the 11-knot spline phi with the linear part) against the October setting (SISO phi = sat_1, X21; MIMO identity actuator, X27). If the spline loses, the measured loss documents why the setting changed.
|
| 6 |
maps_to: SISO section (why phi is fixed to sat_1), MIMO section (actuator class); no paper edit until Matteo agrees the claim
|
| 7 |
-
scripts: nonlinear/experiments/x28_siso.py (S1 gate, control, diagnostics, symmetric gate, S2 labels), nonlinear/training/train_x28.py (S3), nonlinear/experiments/x28_eval.py (S3 validation, S4 test), nonlinear/experiments/x28_paired.py (S4 paired test, fresh directions); nonlinear/mimo_unstable/x28_mimo.py (MIMO certificate, rollout, optimizer, gate, fresh directions, spline classes), nonlinear/mimo_unstable/x28_study_pick.py (class selection), nonlinear/mimo_unstable/x28_mimo_gen.py (MIMO datasets, generator jobs, sample, evaluate, paired test, new directions), nonlinear/mimo_unstable/mh_train.py (MIMO generator training, X27 trainer), nonlinear/mimo_unstable/x28_repair.py (MIMO test with repair: Popov LMI, knot flattening, validation choice, held-out test, loss tails)
|
| 8 |
-
artifacts: results/icml_r3/X28/{PREREG.md, gate_report.txt, control_report.txt, diag_report.txt, gate_sym_report.txt, labels_train/, labels_test/, eval_val_report.txt, val_clip_choice.json, eval_test/, eval_test_report.txt, eval_test_paired.txt, eval_test_fresh.txt}; results/icml_r3/X28/mimo/{selfcheck_m*.json, gate_*.txt, fresh_*.txt, study_m*/, study_choice.txt, lab0_m2/, lab0_m3/, lab1_m3/, ds/, gen/, gen_paired_m*.txt, gen_fresh_m*.txt, gen_fresh_m*.npz, gen_rep/, rep_selfcheck.txt, rep_val_choice.txt, gen_rep_paired_m*.txt, gen_rep_fresh_m*.txt, gen_rep_fresh_m*.npz, rep_tails.txt}; models/x28_{clS_s0,clS_s1,clS_s2,coefS_s0}/, models/x28m/; HF matCercola18/DiffusionControl commits 268d7a3 (labels, models, gate), c6dfe66 (S4 test, study), d339e9b (MIMO labels, generators, S4), cad6165 (S4 with repair)
|
| 9 |
---
|
| 10 |
|
| 11 |
# Generating the spline nonlinearity together with the controller (X28)
|
|
@@ -14,7 +14,8 @@ artifacts: results/icml_r3/X28/{PREREG.md, gate_report.txt, control_report.txt,
|
|
| 14 |
- **Question:** In July the diffusion model generated the nonlinearity phi (an 11-knot spline) together with the linear part. In October phi was fixed (SISO: sat_1; MIMO: identity). Does generating phi buy anything, and if not, what does it cost?
|
| 15 |
- **SISO answer:** No gain. At each plant's optimum the spline is cheaper than sat_1 by a median 2.84%, but all of it is knot asymmetry fitted to the 4 fixed initial directions of the cost: on 64 new directions it is no better (-0.36%), and an odd spline gains 0.00%. On the 199 test plants the spline generator ties the X21 sat_1 generator at 16 samples ("no measurable difference" on all 3 seed pairs), certifies 6 to 16 points fewer plants at 1 and 4 samples, and is 1.3% to 1.5% more expensive on new directions.
|
| 16 |
- **MIMO answer:** At the optimum every spline class beats identity by a median 4.5% to 6.6%, but only odd (sign-symmetric) classes keep a gain on new initial directions; free knots lose 2% to 6%. A pre-registered study over five classes picks odd piecewise-linear knots at |x| = 1, 3, 5 (3 numbers per channel). As registered, the generator of K and these knots ties the identity generator at 16 samples ("no measurable difference" at m = 2 and m = 3). Its paired median regret is 1.1 to 2.4 points lower, but it certifies only 57% to 76% of its samples against 93% to 99.5%. July certified every spline by projecting it into a safe sector, so a follow-up test, filed before any number, gives the spline generator that step: a rejected sample's knots are flattened until it certifies. With it, the spline generator certifies 99.2% to 100% of plants at 16 samples, like identity, and is better at m = 3 ("spline better" on 2 of 3 seed pairs: +1.3 and +1.8 points, Holm p 0.042). At m = 2 it is cheaper on all 3 pairs (+1.9 to +2.5 points, CIs above 0), but the rank test misses (Holm p 0.11), because it is worse by more than 10 points on 15 to 19 of 120 plants. At 1 sample it still certifies 12 to 20 points fewer plants. On new directions it is cheaper at m = 2 (+1.3% to +2.0%) and within the seed spread at m = 3.
|
| 17 |
-
- **
|
|
|
|
| 18 |
|
| 19 |
## 1. Problem / Goal
|
| 20 |
- **SISO setting (X21):** open-loop-unstable plants, bounded actuator, controller u = -phi(clip(F y, +-5)) with a dynamic filter F. X21 fixed phi = sat_1 and generated F. X28 generates F and the 10 nonzero knots of phi (|phi| <= 1, phi(0) = 0) with the same recipe.
|
|
@@ -32,6 +33,7 @@ artifacts: results/icml_r3/X28/{PREREG.md, gate_report.txt, control_report.txt,
|
|
| 32 |
- **MIMO S3 generators:** X27 recipe (DDPM-500, MLP 256 x 5, 96k steps, batch 64, lr 2e-4, EMA 0.999), 3 seeds per arm and m, 12 models.
|
| 33 |
- **MIMO S4 test:** the 120 held-out plants per m, 16 samples per plant (clip 1, seed = model seed). Arm (a) is certified by the Lyapunov LMI (alpha = beta = 1), arm (b) by the circle LMI. Cost on the 2m unit directions, J* = min of all held-out labels of the gate, and the SISO verdict rule on seed j against seed j. Diagnostic: the best certified 16-sample controller per plant on the 64 new directions of seed (7575, id), against (a) seed 0.
|
| 34 |
- **MIMO S4 with repair (PREREG addendum 2026-11-14; Matteo asked whether July was tested at its best, then "Yes, make it work"):** for a rejected arm (b) sample, the log knot ratios of each channel move toward a center g by a factor lam in {0.75, 0.5, 0.25, 0}, K is kept, and the largest certified lam wins. lam = 0 is the linear controller u = -diag(g) K y. Each sample yields one repaired controller, so the budget is unchanged. The center (geometric mean of r(1), r(3), r(5), or r(1)) was chosen on the 64 validation bank plants. A Popov certificate was planned next to the circle LMI, subject to two self-checks. Same samples, arm (a) results, J* and rule as S4-M.
|
|
|
|
| 35 |
|
| 36 |
## 3. Results
|
| 37 |
|
|
@@ -121,7 +123,37 @@ Source: `gen_rep_paired_m*.txt`; lam per sample in `gen_rep/eval_primary_*.npz`.
|
|
| 121 |
- **b = 4:** regret difference +0.32 to +1.48 points (Holm p >= 0.62); certified -5.8 to +1.7 points.
|
| 122 |
- **New directions** (`gen_rep_fresh_m*.txt`): at m = 2, (b) +2.03% [+0.40, +3.08], +1.69% [+0.51, +2.54], +1.29% [+0.49, +2.41], against -0.07% and -0.07% for (a) seeds 1 and 2. At m = 3, (b) +0.21% [-1.10, +2.12], +0.66% [-0.24, +2.72], +0.07% [-1.08, +1.67], against +0.26% and +0.33%. Non-convergent rollouts of the repaired arm: 42 to 123 of 7,680 at m = 2 and 0 to 58 at m = 3 (identity 0 to 64).
|
| 123 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 124 |
## 4. Conclusion
|
| 125 |
- **SISO:** generating phi does not pay. The optimum-level gain is a fit to the cost's initial directions, the generator ties sat_1 at 16 samples, loses certified plants at small budgets, and does worse on new directions. This is the measured reason for the October setting phi = sat_1.
|
| 126 |
-
- **MIMO:** a free spline repeats the SISO artifact. An odd spline with 3 knots per channel keeps a gain on new initial directions at the optimum (held-out: +2.96% at m = 2, +0.94% at m = 3). As registered, its generator ties identity at 16 samples, because it certifies far fewer samples (57% to 76% against 93% to 99.5%). Given July's projection step (flatten a rejected sample until it certifies), it certifies as many plants as identity at 16 samples and is cheaper by a paired median 1.3 to 2.5 points. That is "spline better" at m = 3; at m = 2 it is cheaper on all three pairs but not significant after Holm, because it loses big on the 15 to 19 plants where few of its samples certify. Identity stays better at 1 sample.
|
| 127 |
-
- **Caveats:** the SISO certificate is order dependent across calls (passes stand, fails prove nothing); the MIMO cost uses only positive unit directions, which is why the new-direction checks carry the conclusions; the 64-sample SISO result is one seed and was measured on the cost directions only; the paired rank test scores an uncertified plant as regret 1e3, so a few certification failures outweigh many small cost gains (the pre-registered rule, the same on SISO and MIMO); the MIMO new-direction comparison covers only plants where both generators certify; the repair test was filed after the registered S4-M result was known (it answers whether July was tested at its best) and is reported next to that verdict, not instead of it.
|
|
|
|
| 1 |
---
|
| 2 |
+
title: Generating the spline nonlinearity together with the controller (X28). SISO, no gain over phi = sat_1. MIMO, as registered no measurable difference at 16 samples; with the July-style projection of the knots the odd 3-knot spline generator is better at 16 samples (held-out m = 3; 200 fresh plants per m, m = 2 and 3) and worse at 1 sample; 8 spline plus 8 identity samples per plant is also better at both m on the fresh plants
|
| 3 |
+
date: 2026-11-15
|
| 4 |
+
status: complete (SISO; MIMO optimum, spline-class study, labels, generators, paired test, new directions, test with repair, identity fallback and replication on fresh plants)
|
| 5 |
claim: X28 (Matteo 2026-11-05): the July method (the diffusion model generates the 11-knot spline phi with the linear part) against the October setting (SISO phi = sat_1, X21; MIMO identity actuator, X27). If the spline loses, the measured loss documents why the setting changed.
|
| 6 |
maps_to: SISO section (why phi is fixed to sat_1), MIMO section (actuator class); no paper edit until Matteo agrees the claim
|
| 7 |
+
scripts: nonlinear/experiments/x28_siso.py (S1 gate, control, diagnostics, symmetric gate, S2 labels), nonlinear/training/train_x28.py (S3), nonlinear/experiments/x28_eval.py (S3 validation, S4 test), nonlinear/experiments/x28_paired.py (S4 paired test, fresh directions); nonlinear/mimo_unstable/x28_mimo.py (MIMO certificate, rollout, optimizer, gate, fresh directions, spline classes), nonlinear/mimo_unstable/x28_study_pick.py (class selection), nonlinear/mimo_unstable/x28_mimo_gen.py (MIMO datasets, generator jobs, sample, evaluate, paired test, new directions), nonlinear/mimo_unstable/mh_train.py (MIMO generator training, X27 trainer), nonlinear/mimo_unstable/x28_repair.py (MIMO test with repair: Popov LMI, knot flattening, validation choice, held-out test, loss tails), nonlinear/mimo_unstable/x28_fallback.py (MIMO identity fallback: fresh plants T2, their labels, validation choice, tests on T2 and the held-out plants)
|
| 8 |
+
artifacts: results/icml_r3/X28/{PREREG.md, gate_report.txt, control_report.txt, diag_report.txt, gate_sym_report.txt, labels_train/, labels_test/, eval_val_report.txt, val_clip_choice.json, eval_test/, eval_test_report.txt, eval_test_paired.txt, eval_test_fresh.txt}; results/icml_r3/X28/mimo/{selfcheck_m*.json, gate_*.txt, fresh_*.txt, study_m*/, study_choice.txt, lab0_m2/, lab0_m3/, lab1_m3/, ds/, gen/, gen_paired_m*.txt, gen_fresh_m*.txt, gen_fresh_m*.npz, gen_rep/, rep_selfcheck.txt, rep_val_choice.txt, gen_rep_paired_m*.txt, gen_rep_fresh_m*.txt, gen_rep_fresh_m*.npz, rep_tails.txt, t2/, t2_m2/, t2_m3/, gen_fb/, fb_val_choice.txt, fb_paired_t2_m*.txt, fb_fresh_t2_m*.txt, fb_fresh_t2_m*.npz, fb_heldout_m*.txt}; models/x28_{clS_s0,clS_s1,clS_s2,coefS_s0}/, models/x28m/; HF matCercola18/DiffusionControl commits 268d7a3 (labels, models, gate), c6dfe66 (S4 test, study), d339e9b (MIMO labels, generators, S4), cad6165 (S4 with repair), f7150e6 (fallback, fresh plants)
|
| 9 |
---
|
| 10 |
|
| 11 |
# Generating the spline nonlinearity together with the controller (X28)
|
|
|
|
| 14 |
- **Question:** In July the diffusion model generated the nonlinearity phi (an 11-knot spline) together with the linear part. In October phi was fixed (SISO: sat_1; MIMO: identity). Does generating phi buy anything, and if not, what does it cost?
|
| 15 |
- **SISO answer:** No gain. At each plant's optimum the spline is cheaper than sat_1 by a median 2.84%, but all of it is knot asymmetry fitted to the 4 fixed initial directions of the cost: on 64 new directions it is no better (-0.36%), and an odd spline gains 0.00%. On the 199 test plants the spline generator ties the X21 sat_1 generator at 16 samples ("no measurable difference" on all 3 seed pairs), certifies 6 to 16 points fewer plants at 1 and 4 samples, and is 1.3% to 1.5% more expensive on new directions.
|
| 16 |
- **MIMO answer:** At the optimum every spline class beats identity by a median 4.5% to 6.6%, but only odd (sign-symmetric) classes keep a gain on new initial directions; free knots lose 2% to 6%. A pre-registered study over five classes picks odd piecewise-linear knots at |x| = 1, 3, 5 (3 numbers per channel). As registered, the generator of K and these knots ties the identity generator at 16 samples ("no measurable difference" at m = 2 and m = 3). Its paired median regret is 1.1 to 2.4 points lower, but it certifies only 57% to 76% of its samples against 93% to 99.5%. July certified every spline by projecting it into a safe sector, so a follow-up test, filed before any number, gives the spline generator that step: a rejected sample's knots are flattened until it certifies. With it, the spline generator certifies 99.2% to 100% of plants at 16 samples, like identity, and is better at m = 3 ("spline better" on 2 of 3 seed pairs: +1.3 and +1.8 points, Holm p 0.042). At m = 2 it is cheaper on all 3 pairs (+1.9 to +2.5 points, CIs above 0), but the rank test misses (Holm p 0.11), because it is worse by more than 10 points on 15 to 19 of 120 plants. At 1 sample it still certifies 12 to 20 points fewer plants. On new directions it is cheaper at m = 2 (+1.3% to +2.0%) and within the seed spread at m = 3.
|
| 17 |
+
- **MIMO, fresh plants:** a third test, filed before any number (Matteo: "Can you try to improve the spline"), lets a few spline samples decide where to draw from the identity generator instead. It was designed from the held-out losses, so its verdict comes from 200 fresh plants per m that no model, rule or diagnostic had seen. Validation chose the simplest form: every plant gets 8 spline and 8 identity samples. On the fresh plants this mix is better than 16 identity samples at both m: +2.40 to +3.03 points on 3 of 3 seed pairs at m = 3, and +0.98 to +1.69 points at m = 2, "better" on 2 of 3 pairs because the third misses the 1-point bar by 0.02 (Holm p <= 2.7e-11 on all six). It is worse by more than 10 points on 1 to 6 plants of 200. The same fresh plants replicate the repaired spline generator alone: "spline better" at both m on 3 of 3 pairs (+3.11 to +3.72 points at m = 3, +1.76 to +2.33 at m = 2, Holm p <= 0.0019), cheaper on new directions at both m (+1.55% to +2.39%), certified like identity. The as-registered generator, without repair, again shows no measurable difference, and at 1 sample identity is still better.
|
| 18 |
+
- **So what:** On SISO, generating phi buys nothing, which supports phi = sat_1. On MIMO, generating the odd 3-knot shape pays at 16 samples once each sample is projected to a certified one, as in July: on fresh plants, equal certification and 1.8 to 3.7 points lower paired median regret at both m, and lower cost on new directions. Splitting the 16 samples between the spline and the identity generator also beats identity at both m, with fewer large losses but a smaller median gain. Identity stays better at 1 sample.
|
| 19 |
|
| 20 |
## 1. Problem / Goal
|
| 21 |
- **SISO setting (X21):** open-loop-unstable plants, bounded actuator, controller u = -phi(clip(F y, +-5)) with a dynamic filter F. X21 fixed phi = sat_1 and generated F. X28 generates F and the 10 nonzero knots of phi (|phi| <= 1, phi(0) = 0) with the same recipe.
|
|
|
|
| 33 |
- **MIMO S3 generators:** X27 recipe (DDPM-500, MLP 256 x 5, 96k steps, batch 64, lr 2e-4, EMA 0.999), 3 seeds per arm and m, 12 models.
|
| 34 |
- **MIMO S4 test:** the 120 held-out plants per m, 16 samples per plant (clip 1, seed = model seed). Arm (a) is certified by the Lyapunov LMI (alpha = beta = 1), arm (b) by the circle LMI. Cost on the 2m unit directions, J* = min of all held-out labels of the gate, and the SISO verdict rule on seed j against seed j. Diagnostic: the best certified 16-sample controller per plant on the 64 new directions of seed (7575, id), against (a) seed 0.
|
| 35 |
- **MIMO S4 with repair (PREREG addendum 2026-11-14; Matteo asked whether July was tested at its best, then "Yes, make it work"):** for a rejected arm (b) sample, the log knot ratios of each channel move toward a center g by a factor lam in {0.75, 0.5, 0.25, 0}, K is kept, and the largest certified lam wins. lam = 0 is the linear controller u = -diag(g) K y. Each sample yields one repaired controller, so the budget is unchanged. The center (geometric mean of r(1), r(3), r(5), or r(1)) was chosen on the 64 validation bank plants. A Popov certificate was planned next to the circle LMI, subject to two self-checks. Same samples, arm (a) results, J* and rule as S4-M.
|
| 36 |
+
- **MIMO identity fallback on fresh plants (PREREG addendum 2026-11-15; Matteo: "Can you try to improve the spline on decision 1"):** arm (c) draws the first q of its 16 samples from the spline generator; if at most c of them certify as sampled, the other 16 - q come from the identity generator of the same seed (sampler seed + 1000, independent of arm (a)'s draws), otherwise from the spline generator. Spline samples get the repair above (circle LMI, center r(1)), identity samples the Lyapunov test, and the best certified J is used. (q, c) with q in {4, 8} and c in {-1 (never), 0, ..., q (always)} was chosen per m on the 64 validation plants by the largest standardized Wilcoxon statistic of regret (a) - regret (c) at b = 16, the statistic that stopped m = 2. Test plants T2: 200 per m, built like the held-out sets (`build_sofm`, uniform in the family box, 48 oracle starts, kept iff the oracle certifies a gain, seeds 1115 and 1116). J*_T2 is the cheapest certified controller of the S2-M label optimizer on T2 (identity at c = 1, `so3` at c = 1, 2, 5, 400 Adam steps). The 12 S3-M generators sample T2 with seed = model seed. Primary: (c) against (a), the S4-M rule; replications: the repaired and the unrepaired spline generator against (a). New directions on T2: seed (7979, plant index). (c) is also reported on the 120 held-out plants, which shaped the variant.
|
| 37 |
|
| 38 |
## 3. Results
|
| 39 |
|
|
|
|
| 123 |
- **b = 4:** regret difference +0.32 to +1.48 points (Holm p >= 0.62); certified -5.8 to +1.7 points.
|
| 124 |
- **New directions** (`gen_rep_fresh_m*.txt`): at m = 2, (b) +2.03% [+0.40, +3.08], +1.69% [+0.51, +2.54], +1.29% [+0.49, +2.41], against -0.07% and -0.07% for (a) seeds 1 and 2. At m = 3, (b) +0.21% [-1.10, +2.12], +0.66% [-0.24, +2.72], +0.07% [-1.08, +1.67], against +0.26% and +0.33%. Non-convergent rollouts of the repaired arm: 42 to 123 of 7,680 at m = 2 and 0 to 58 at m = 3 (identity 0 to 64).
|
| 125 |
|
| 126 |
+
### 3.6 MIMO, spline with identity fallback, and replication on 200 fresh plants per m
|
| 127 |
+
Validation (`fb_val_choice.txt`, 64 plants per m, pooled over the 3 seed pairs; d = regret (a) - regret (c) at b = 16; every setting certifies 99.0% to 100% of plants, identity 100%):
|
| 128 |
+
| setting | m = 3: z, median d | m = 2: z, median d |
|
| 129 |
+
|---|---|---|
|
| 130 |
+
| never fall back (repaired spline alone) | +4.44, +2.82 pts | +3.91, +2.46 pts |
|
| 131 |
+
| q = 8, c = 3 (falls back on 34% / 16% of plants) | +7.24, +2.87 pts | +5.34, +2.52 pts |
|
| 132 |
+
| q = 8, c = 8: always 8 spline + 8 identity (chosen) | +7.63, +2.45 pts | +7.70, +1.64 pts |
|
| 133 |
+
|
| 134 |
+
The rule picks the fixed half-and-half split at both m: it removes the large losses best, at the price of a smaller median gain. So arm (c) is a mix of the two generators, not an adaptive fallback.
|
| 135 |
+
|
| 136 |
+
T2: each of the 4 builds kept 100 plants from 128 attempts; all four label runs certify 200 of 200 plants at both m. On T2 (`fb_paired_t2_m*.txt`):
|
| 137 |
+
| m | arm, seeds 0 / 1 / 2 | certified per sample | b = 1 certified | b = 16 certified | b = 16 regret vs J*_T2 |
|
| 138 |
+
|---|---|---|---|---|---|
|
| 139 |
+
| 3 | (a) identity | 95.4 / 95.8 / 93.7% | 95.5 / 97.5 / 94.0% | 100 / 100 / 98.5% | 10.05 / 10.24 / 10.36% |
|
| 140 |
+
| 3 | (b) `so3`, as sampled | 62.0 / 60.5 / 61.5% | 62.0 / 61.0 / 61.5% | 95.5 / 95.0 / 95.5% | 6.16 / 6.90 / 6.74% |
|
| 141 |
+
| 3 | (b) `so3`, repaired | 81.5 / 79.3 / 78.8% | 80.5 / 78.0 / 80.5% | 100 / 100 / 99.5% | 6.16 / 6.82 / 6.71% |
|
| 142 |
+
| 3 | (c) 8 spline + 8 identity | 88.0 / 87.7 / 86.2% | 80.5 / 78.0 / 80.5% | 100 / 100 / 99.5% | 6.20 / 6.00 / 6.25% |
|
| 143 |
+
| 2 | (a) identity | 98.6 / 98.7 / 98.4% | 98.5 / 98.5 / 98.5% | 99.5 / 100 / 100% | 6.39 / 6.47 / 6.39% |
|
| 144 |
+
| 2 | (b) `so3`, as sampled | 76.9 / 76.5 / 78.7% | 78.0 / 75.5 / 80.5% | 96.5 / 96.5 / 98.5% | 3.59 / 3.66 / 3.70% |
|
| 145 |
+
| 2 | (b) `so3`, repaired | 89.5 / 89.6 / 90.8% | 91.0 / 90.0 / 90.5% | 99.5 / 99.5 / 100% | 3.51 / 3.62 / 3.64% |
|
| 146 |
+
| 2 | (c) 8 spline + 8 identity | 94.3 / 94.1 / 94.3% | 91.0 / 90.0 / 90.5% | 100 / 100 / 100% | 3.33 / 3.54 / 3.40% |
|
| 147 |
+
|
| 148 |
+
Paired on T2 (positive means the spline arm is cheaper). In the `fb_*.txt` files each block prints its second arm under the letter b; in the primary blocks that arm is (c).
|
| 149 |
+
- **b = 16, primary, (c) against (a):** +3.03 [+1.80, +4.37], +2.40 [+1.47, +3.83], +2.43 [+1.13, +3.90] points at m = 3 (Holm p 1.8e-14, 7.5e-14, 2.7e-11) and +1.69 [+0.90, +2.93], +0.98 [+0.59, +2.14], +1.04 [+0.40, +2.25] at m = 2 (Holm p 4e-16, 5.5e-12, 1.4e-11); certified within +1.0 point. **Verdict: spline better at m = 3 (3 of 3 pairs) and at m = 2 (2 of 3; seed pair 1 misses the 1-point bar by 0.02).** (c) is worse by more than 10 points on 5, 5, 6 plants at m = 3 and 1, 3, 3 at m = 2, and better by more than 5 on 69 to 77 and 49 to 57.
|
| 150 |
+
- **Replication of S4-M with repair:** +3.40 [+2.20, +4.38], +3.11 [+1.94, +4.71], +3.72 [+1.55, +5.17] points at m = 3 (Holm p 6.8e-4, 7.4e-4, 7.4e-4) and +2.33 [+1.14, +3.27], +1.76 [+0.88, +3.25], +1.78 [+0.78, +3.16] at m = 2 (Holm p 2.9e-4, 0.0019, 0.0017); certified within 1 point. **Spline better at both m on 3 of 3 pairs.** The m = 3 result of section 3.5 holds on new plants, and m = 2, which missed on the 120 held-out plants, passes on 200 fresh ones.
|
| 151 |
+
- **Replication of S4-M as registered:** +3.00, +2.63, +2.86 points at m = 3 (Holm p 0.062 to 0.063, certified -4.5, -5.0, -3.0 points) and +1.74, +1.57, +1.63 at m = 2 (Holm p 0.0093 to 0.018, certified -3.0, -3.5, -1.5). **No measurable difference at both m** (1 of 3 pairs at m = 2), as in section 3.4.
|
| 152 |
+
- **b = 1** (the same for (b) repaired and (c), whose first samples are spline samples): 13.5 to 19.5 points fewer plants certified at m = 3 and 7.5 to 8.5 at m = 2 (McNemar p <= 7.3e-4); regret difference -1.35 to -0.65 points at m = 3 and -1.33 to +0.06 at m = 2 (Holm p <= 0.02). **b = 4:** regret difference +1.77 to +2.35 points at m = 3 and +0.18 to +1.07 at m = 2 (Holm p >= 0.76); certified -7.0 to -3.5 and -2.0 to -0.5 points.
|
| 153 |
+
- **New directions on T2** (`fb_fresh_t2_m*.txt`, against (a) seed 0): at m = 3, (c) +1.65% [+0.92, +2.62], +2.19% [+1.53, +2.83], +1.45% [+0.99, +2.13] and repaired (b) +1.72% [+0.86, +2.63], +2.39% [+1.42, +3.19], +1.55% [+0.39, +2.62], against +0.09% and +0.05% for (a) seeds 1 and 2. At m = 2, (c) +1.17% [+0.46, +2.24], +1.12% [+0.43, +1.99], +0.77% [+0.36, +1.35] and repaired (b) +1.92% [+0.98, +2.83], +2.08% [+1.09, +2.79], +2.06% [+0.62, +3.09], against -0.04% and -0.01%. Non-convergent rollouts per seed, of 12,608 to 12,800: (c) 46 to 117 at m = 3 and 0 at m = 2; repaired (b) 63 to 170 and 61 to 122; (a) 47 to 200 and 0 to 63.
|
| 154 |
+
- **Held-out plants** (`fb_heldout_m*.txt`; they shaped the variant, so this is secondary): (c) against (a) +1.18, +1.46, +1.33 points at m = 3 (Holm p <= 1.5e-5) and +2.05, +1.34, +1.38 at m = 2 (Holm p <= 1.5e-7), spline better on 3 of 3 pairs at both m; worse by more than 10 points on 0 or 1 plant at m = 3 and 1 or 2 at m = 2, against 12 to 19 for the repaired spline alone (`rep_tails.txt`).
|
| 155 |
+
|
| 156 |
## 4. Conclusion
|
| 157 |
- **SISO:** generating phi does not pay. The optimum-level gain is a fit to the cost's initial directions, the generator ties sat_1 at 16 samples, loses certified plants at small budgets, and does worse on new directions. This is the measured reason for the October setting phi = sat_1.
|
| 158 |
+
- **MIMO:** a free spline repeats the SISO artifact. An odd spline with 3 knots per channel keeps a gain on new initial directions at the optimum (held-out: +2.96% at m = 2, +0.94% at m = 3). As registered, its generator ties identity at 16 samples, because it certifies far fewer samples (57% to 76% against 93% to 99.5%). Given July's projection step (flatten a rejected sample until it certifies), it certifies as many plants as identity at 16 samples and is cheaper by a paired median 1.3 to 2.5 points. That is "spline better" at m = 3; at m = 2 it is cheaper on all three pairs but not significant after Holm, because it loses big on the 15 to 19 plants where few of its samples certify. Identity stays better at 1 sample. On 200 fresh plants per m the repaired spline generator is better at both m on all three pairs (1.8 to 3.7 points) and cheaper on new directions, so the m = 2 miss on the held-out plants did not hold up. Its weak point is the plants where the generator rarely produces a certifiable spline; giving half of the 16 samples to the identity generator cuts the losses there (worse by more than 10 points on 1 to 6 of 200 plants) and is also better than identity at both m, with a smaller median gain. So on MIMO the July idea pays at the 16-sample budget once each sample is projected to a certified one.
|
| 159 |
+
- **Caveats:** the SISO certificate is order dependent across calls (passes stand, fails prove nothing); the MIMO cost uses only positive unit directions, which is why the new-direction checks carry the conclusions; the 64-sample SISO result is one seed and was measured on the cost directions only; the paired rank test scores an uncertified plant as regret 1e3, so a few certification failures outweigh many small cost gains (the pre-registered rule, the same on SISO and MIMO); the MIMO new-direction comparison covers only plants where both generators certify; the repair test was filed after the registered S4-M result was known (it answers whether July was tested at its best) and is reported next to that verdict, not instead of it; the fallback test was designed from the held-out losses, which is why its verdict and the replications use fresh plants; J*_T2 comes from the identity and `so3` label runs only (the held-out J* also includes the `sp` and `so` classes), so T2 regrets are not on the same scale as held-out regrets; the chosen arm (c) is a fixed half-and-half mix of two generators, so it shows that adding the spline generator to the identity one pays at 16 samples, not that one generator replaces the other.
|