X28: write-up with the MIMO generator verdict
Browse files- docs/icml_r3_x28.md +27 -11
docs/icml_r3_x28.md
CHANGED
|
@@ -1,11 +1,11 @@
|
|
| 1 |
---
|
| 2 |
-
title: Generating the spline nonlinearity together with the controller (X28). SISO, no
|
| 3 |
-
date: 2026-11-
|
| 4 |
-
status:
|
| 5 |
claim: X28 (Matteo 2026-11-05): the July method (the diffusion model generates the 11-knot spline phi with the linear part) against the October setting (SISO phi = sat_1, X21; MIMO identity actuator, X27). If the spline loses, the measured loss documents why the setting changed.
|
| 6 |
maps_to: SISO section (why phi is fixed to sat_1), MIMO section (actuator class); no paper edit until Matteo agrees the claim
|
| 7 |
-
scripts: nonlinear/experiments/x28_siso.py (S1 gate, control, diagnostics, symmetric gate, S2 labels), nonlinear/training/train_x28.py (S3), nonlinear/experiments/x28_eval.py (S3 validation, S4 test), nonlinear/experiments/x28_paired.py (S4 paired test, fresh directions); nonlinear/mimo_unstable/x28_mimo.py (MIMO certificate, rollout, optimizer, gate, fresh directions, spline classes), nonlinear/mimo_unstable/x28_study_pick.py (class selection), nonlinear/mimo_unstable/x28_mimo_gen.py (MIMO
|
| 8 |
-
artifacts: results/icml_r3/X28/{PREREG.md, gate_report.txt, control_report.txt, diag_report.txt, gate_sym_report.txt, labels_train/, labels_test/, eval_val_report.txt, val_clip_choice.json, eval_test/, eval_test_report.txt, eval_test_paired.txt, eval_test_fresh.txt}; results/icml_r3/X28/mimo/{selfcheck_m*.json, gate_*.txt, fresh_*.txt, study_m*/, study_choice.txt}; models/x28_{clS_s0,clS_s1,clS_s2,coefS_s0}/; HF matCercola18/DiffusionControl commits 268d7a3 (labels, models, gate), c6dfe66 (S4 test, study)
|
| 9 |
---
|
| 10 |
|
| 11 |
# Generating the spline nonlinearity together with the controller (X28)
|
|
@@ -13,8 +13,8 @@ artifacts: results/icml_r3/X28/{PREREG.md, gate_report.txt, control_report.txt,
|
|
| 13 |
## TL;DR
|
| 14 |
- **Question:** In July the diffusion model generated the nonlinearity phi (an 11-knot spline) together with the linear part. In October phi was fixed (SISO: sat_1; MIMO: identity). Does generating phi buy anything, and if not, what does it cost?
|
| 15 |
- **SISO answer:** No gain. At each plant's optimum the spline is cheaper than sat_1 by a median 2.84%, but all of it is knot asymmetry fitted to the 4 fixed initial directions of the cost: on 64 new directions it is no better (-0.36%), and an odd spline gains 0.00%. On the 199 test plants the spline generator ties the X21 sat_1 generator at 16 samples ("no measurable difference" on all 3 seed pairs), certifies 6 to 16 points fewer plants at 1 and 4 samples, and is 1.3% to 1.5% more expensive on new directions.
|
| 16 |
-
- **MIMO answer
|
| 17 |
-
- **So what:** The October
|
| 18 |
|
| 19 |
## 1. Problem / Goal
|
| 20 |
- **SISO setting (X21):** open-loop-unstable plants, bounded actuator, controller u = -phi(clip(F y, +-5)) with a dynamic filter F. X21 fixed phi = sat_1 and generated F. X28 generates F and the 10 nonzero knots of phi (|phi| <= 1, phi(0) = 0) with the same recipe.
|
|
@@ -28,6 +28,9 @@ artifacts: results/icml_r3/X28/{PREREG.md, gate_report.txt, control_report.txt,
|
|
| 28 |
- **S4 test:** 199 X21 test plants, 16 samples per plant (64 for seed 0 as a secondary), J* = min of all F-only and joint labels, regret with uncertified = 1e3, paired against X21 clL scored by the same code.
|
| 29 |
- **MIMO gate:** 120 held-out plants per m (m = 2, 3), identity and spline arms from the same bank gain and the same Adam budget (400 steps, K lr 0.02 chosen on 64 validation bank plants), cost over the 2m positive unit directions (20 s, RK4 dt 0.01), circle LMI per channel (self-check against `family_sofm.verify`: 200 gains per m, 0 disagreements, `selfcheck_m*.json`). New directions: 64 random unit directions in R^(2m), both signs, seed (7575, id).
|
| 30 |
- **MIMO spline-class study (Matteo 2026-11-13):** five classes on the 64 validation bank plants per m: piecewise linear free (`sp`, 10 per channel), odd (`so`, 5), odd with knots at |x| = 1, 3, 5 (`so3`, 3), and cubic PCHIP on the ratios, free (`cf`, 10) and odd (`co`, 5); the PCHIP interpolant stays between its end values on each interval, so the sector stays exact. Rule filed before the run: eligible iff the new-direction gain over identity has CI lower bound > 0 at both m and at most 1% more non-convergent rollouts than identity; winner by mean gain, ties within 0.25 points to fewer parameters.
|
|
|
|
|
|
|
|
|
|
| 31 |
|
| 32 |
## 3. Results
|
| 33 |
|
|
@@ -84,10 +87,23 @@ Held-out check of `so3` (120 plants per m, its own Adam call next to the stored
|
|
| 84 |
|
| 85 |
Both pre-registered rules hold at both m (gate: median >= 2% and CI lower bound > 0; new directions: CI lower bound > 0).
|
| 86 |
|
| 87 |
-
### 3.4 MIMO,
|
| 88 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 89 |
|
| 90 |
## 4. Conclusion
|
| 91 |
- **SISO:** generating phi does not pay. The optimum-level gain is a fit to the cost's initial directions, the generator ties sat_1 at 16 samples, loses certified plants at small budgets, and does worse on new directions. This is the measured reason for the October setting phi = sat_1.
|
| 92 |
-
- **MIMO:** a free spline repeats the SISO artifact. An odd spline with 3 knots per channel keeps a gain on new initial directions at the optimum (held-out: +2.96% at m = 2, +0.94% at m = 3)
|
| 93 |
-
- **Caveats:** the SISO certificate is order dependent across calls (passes stand, fails prove nothing); the MIMO cost uses only positive unit directions, which is why the new-direction checks carry the conclusions; the 64-sample SISO result is one seed and was measured on the cost directions only.
|
|
|
|
| 1 |
---
|
| 2 |
+
title: Generating the spline nonlinearity together with the controller (X28). SISO and MIMO, no measurable difference from the October setting at 16 samples, and the spline generator certifies fewer plants at 1 and 4 samples; on MIMO the odd 3-knot spline lowers the cost but certifies fewer samples
|
| 3 |
+
date: 2026-11-14
|
| 4 |
+
status: complete (SISO; MIMO optimum, spline-class study, labels, generators, paired test, new directions)
|
| 5 |
claim: X28 (Matteo 2026-11-05): the July method (the diffusion model generates the 11-knot spline phi with the linear part) against the October setting (SISO phi = sat_1, X21; MIMO identity actuator, X27). If the spline loses, the measured loss documents why the setting changed.
|
| 6 |
maps_to: SISO section (why phi is fixed to sat_1), MIMO section (actuator class); no paper edit until Matteo agrees the claim
|
| 7 |
+
scripts: nonlinear/experiments/x28_siso.py (S1 gate, control, diagnostics, symmetric gate, S2 labels), nonlinear/training/train_x28.py (S3), nonlinear/experiments/x28_eval.py (S3 validation, S4 test), nonlinear/experiments/x28_paired.py (S4 paired test, fresh directions); nonlinear/mimo_unstable/x28_mimo.py (MIMO certificate, rollout, optimizer, gate, fresh directions, spline classes), nonlinear/mimo_unstable/x28_study_pick.py (class selection), nonlinear/mimo_unstable/x28_mimo_gen.py (MIMO datasets, generator jobs, sample, evaluate, paired test, new directions), nonlinear/mimo_unstable/mh_train.py (MIMO generator training, X27 trainer)
|
| 8 |
+
artifacts: results/icml_r3/X28/{PREREG.md, gate_report.txt, control_report.txt, diag_report.txt, gate_sym_report.txt, labels_train/, labels_test/, eval_val_report.txt, val_clip_choice.json, eval_test/, eval_test_report.txt, eval_test_paired.txt, eval_test_fresh.txt}; results/icml_r3/X28/mimo/{selfcheck_m*.json, gate_*.txt, fresh_*.txt, study_m*/, study_choice.txt, lab0_m2/, lab0_m3/, lab1_m3/, ds/, gen/, gen_paired_m*.txt, gen_fresh_m*.txt, gen_fresh_m*.npz}; models/x28_{clS_s0,clS_s1,clS_s2,coefS_s0}/, models/x28m/; HF matCercola18/DiffusionControl commits 268d7a3 (labels, models, gate), c6dfe66 (S4 test, study), d339e9b (MIMO labels, generators, S4)
|
| 9 |
---
|
| 10 |
|
| 11 |
# Generating the spline nonlinearity together with the controller (X28)
|
|
|
|
| 13 |
## TL;DR
|
| 14 |
- **Question:** In July the diffusion model generated the nonlinearity phi (an 11-knot spline) together with the linear part. In October phi was fixed (SISO: sat_1; MIMO: identity). Does generating phi buy anything, and if not, what does it cost?
|
| 15 |
- **SISO answer:** No gain. At each plant's optimum the spline is cheaper than sat_1 by a median 2.84%, but all of it is knot asymmetry fitted to the 4 fixed initial directions of the cost: on 64 new directions it is no better (-0.36%), and an odd spline gains 0.00%. On the 199 test plants the spline generator ties the X21 sat_1 generator at 16 samples ("no measurable difference" on all 3 seed pairs), certifies 6 to 16 points fewer plants at 1 and 4 samples, and is 1.3% to 1.5% more expensive on new directions.
|
| 16 |
+
- **MIMO answer:** At the optimum every spline class beats identity by a median 4.5% to 6.6%, but only odd (sign-symmetric) classes keep a gain on new initial directions; free knots lose 2% to 6%. A pre-registered study over five classes picks odd piecewise-linear knots at |x| = 1, 3, 5 (3 numbers per channel); on 120 held-out plants per size it keeps +2.96% (m = 2) and +0.94% (m = 3) on new directions at the optimum. The generator of K and these knots ties the identity generator at 16 samples ("no measurable difference" on all 3 seed pairs at m = 2 and m = 3). Its paired median regret is 1.1 to 2.4 points lower, but it certifies 57% to 76% of its samples against 93% to 99.5%, and at 16 samples it leaves 3 to 7 of 120 plants without a certified controller. At 1 sample it certifies 22 to 37 points fewer plants. On new directions it is cheaper at m = 2 (+1.6% to +2.1%) and within the seed spread at m = 3.
|
| 17 |
+
- **So what:** The October setting is supported on both. On SISO, generating phi buys nothing. On MIMO, the odd 3-knot shape lowers the paired median regret by 1.1 to 2.4 points and pays for it in certified samples; by the pre-registered rule the two generators do not differ at 16 samples, and identity is better at 1 and 4.
|
| 18 |
|
| 19 |
## 1. Problem / Goal
|
| 20 |
- **SISO setting (X21):** open-loop-unstable plants, bounded actuator, controller u = -phi(clip(F y, +-5)) with a dynamic filter F. X21 fixed phi = sat_1 and generated F. X28 generates F and the 10 nonzero knots of phi (|phi| <= 1, phi(0) = 0) with the same recipe.
|
|
|
|
| 28 |
- **S4 test:** 199 X21 test plants, 16 samples per plant (64 for seed 0 as a secondary), J* = min of all F-only and joint labels, regret with uncertified = 1e3, paired against X21 clL scored by the same code.
|
| 29 |
- **MIMO gate:** 120 held-out plants per m (m = 2, 3), identity and spline arms from the same bank gain and the same Adam budget (400 steps, K lr 0.02 chosen on 64 validation bank plants), cost over the 2m positive unit directions (20 s, RK4 dt 0.01), circle LMI per channel (self-check against `family_sofm.verify`: 200 gains per m, 0 disagreements, `selfcheck_m*.json`). New directions: 64 random unit directions in R^(2m), both signs, seed (7575, id).
|
| 30 |
- **MIMO spline-class study (Matteo 2026-11-13):** five classes on the 64 validation bank plants per m: piecewise linear free (`sp`, 10 per channel), odd (`so`, 5), odd with knots at |x| = 1, 3, 5 (`so3`, 3), and cubic PCHIP on the ratios, free (`cf`, 10) and odd (`co`, 5); the PCHIP interpolant stays between its end values on each interval, so the sector stays exact. Rule filed before the run: eligible iff the new-direction gain over identity has CI lower bound > 0 at both m and at most 1% more non-convergent rollouts than identity; winner by mean gain, ties within 0.25 points to fewer parameters.
|
| 31 |
+
- **MIMO S2 labels:** 7,936 bank plants at m = 3 (indices 64 to 8,000) and 3,936 at m = 2 (64 to 4,000), identity at c = 1 and `so3` at c = 1, 2, 5, 400 Adam steps each (rollout gradient checkpointing, same math). Label (a) is the identity gain K. Label (b) is the cheapest of the three `so3` starts and identity, stored as K plus log r_i(1), log r_i(3), log r_i(5) per channel with no gauge normalization (PREREG correction 2026-11-13: with knots at fixed positions, scaling a row of K is not a symmetry).
|
| 32 |
+
- **MIMO S3 generators:** X27 recipe (DDPM-500, MLP 256 x 5, 96k steps, batch 64, lr 2e-4, EMA 0.999), 3 seeds per arm and m, 12 models.
|
| 33 |
+
- **MIMO S4 test:** the 120 held-out plants per m, 16 samples per plant (clip 1, seed = model seed). Arm (a) is certified by the Lyapunov LMI (alpha = beta = 1), arm (b) by the circle LMI. Cost on the 2m unit directions, J* = min of all held-out labels of the gate, and the SISO verdict rule on seed j against seed j. Diagnostic: the best certified 16-sample controller per plant on the 64 new directions of seed (7575, id), against (a) seed 0.
|
| 34 |
|
| 35 |
## 3. Results
|
| 36 |
|
|
|
|
| 87 |
|
| 88 |
Both pre-registered rules hold at both m (gate: median >= 2% and CI lower bound > 0; new directions: CI lower bound > 0).
|
| 89 |
|
| 90 |
+
### 3.4 MIMO, generators on the 120 held-out plants
|
| 91 |
+
Labels (`ds/info_m*.json`): every plant is certified at every start (a start is the certified bank gain). Label (b) comes from `so3` c = 1 / 2 / 5 / identity on 4,146 / 3,029 / 740 / 21 plants at m = 3 and 1,949 / 1,470 / 517 / 0 at m = 2. It is cheaper than label (a) by a median 4.14% (m = 3) and 5.01% (m = 2). The target round trip reproduces J to 2e-16.
|
| 92 |
+
|
| 93 |
+
| m | arm, seeds 0 / 1 / 2 | certified per sample | b = 1 certified | b = 16 certified | b = 16 regret vs J* |
|
| 94 |
+
|---|---|---|---|---|---|
|
| 95 |
+
| 3 | (a) identity | 96.0 / 96.6 / 93.4% | 95.0 / 97.5 / 94.2% | 100 / 100 / 100% | 10.66 / 10.46 / 10.56% |
|
| 96 |
+
| 3 | (b) `so3` | 57.4 / 58.6 / 60.5% | 61.7 / 60.8 / 59.2% | 97.5 / 96.7 / 97.5% | 8.64 / 9.29 / 9.54% |
|
| 97 |
+
| 2 | (a) identity | 99.5 / 98.8 / 99.0% | 100.0 / 98.3 / 99.2% | 100 / 100 / 100% | 8.22 / 7.95 / 7.99% |
|
| 98 |
+
| 2 | (b) `so3` | 75.9 / 76.0 / 73.4% | 75.0 / 76.7 / 68.3% | 95.0 / 96.7 / 94.2% | 4.76 / 4.79 / 5.27% |
|
| 99 |
+
|
| 100 |
+
Source: `gen_paired_m*.txt`. Paired (seed j vs seed j; positive means the spline is cheaper):
|
| 101 |
+
- **b = 16 (primary):** (a) - (b) median regret difference +1.28 [-0.28, +2.51], +1.06 [+0.08, +2.61], +1.73 [+0.08, +3.45] points at m = 3 (Holm p 0.66, 0.47, 0.31) and +2.41 [+0.94, +3.39], +1.94 [+0.49, +2.66], +1.96 [+0.91, +2.93] at m = 2 (Holm p 0.31 on all three); certified -2.5, -3.3, -2.5 points (m = 3) and -5.0, -3.3, -5.8 (m = 2). **Verdict: no measurable difference at both m.** The bootstrap CI of the median excludes 0 on 5 of 6 pairs, but the rank test counts the 3 to 7 plants the spline leaves uncertified (regret 1e3) as its largest losses, and the certified drop exceeds the 2-point margin on all 6 pairs.
|
| 102 |
+
- **b = 1:** the spline generator certifies 33.3, 36.7, 35.0 points fewer plants at m = 3 and 25.0, 21.7, 30.8 at m = 2 (McNemar p <= 2.2e-07), and its regret is higher by 8.35, 8.34, 3.15 points (m = 3) and 2.51, 3.87, 4.44 (m = 2), Holm p <= 9e-05.
|
| 103 |
+
- **b = 4:** certified 7.5 to 14.2 points fewer (McNemar p 1.5e-05 to 0.035); regret difference between -0.42 and +1.29 points (Holm p >= 0.097).
|
| 104 |
+
- **New directions** (`gen_fresh_m*.txt`, best certified b = 16 sample per plant, 64 directions, against (a) seed 0, plants where both certify): at m = 2, (b) +2.08% [+0.55, +3.13], +1.69% [+0.60, +2.60], +1.57% [+0.68, +2.77], against -0.07% and -0.07% for (a) seeds 1 and 2. At m = 3, (b) +0.44% [-1.13, +1.97], +0.66% [-0.35, +2.52], +0.04% [-1.01, +1.67], against +0.26% and +0.33% for (a) seeds 1 and 2. Non-convergent rollouts: 0 to 41 per seed for (b), 0 to 64 for (a), of 7,232 to 7,680.
|
| 105 |
|
| 106 |
## 4. Conclusion
|
| 107 |
- **SISO:** generating phi does not pay. The optimum-level gain is a fit to the cost's initial directions, the generator ties sat_1 at 16 samples, loses certified plants at small budgets, and does worse on new directions. This is the measured reason for the October setting phi = sat_1.
|
| 108 |
+
- **MIMO:** a free spline repeats the SISO artifact. An odd spline with 3 knots per channel keeps a gain on new initial directions at the optimum (held-out: +2.96% at m = 2, +0.94% at m = 3). Its generator delivers part of it: paired median regret 1.1 to 2.4 points lower at 16 samples, and +1.6% to +2.1% on new directions at m = 2 (within the seed spread at m = 3). It certifies far fewer samples (57% to 76% against 93% to 99.5%). By the pre-registered rule the two generators do not differ at 16 samples, and the identity generator is better at 1 and 4 samples. This is the measured reason for the October identity actuator.
|
| 109 |
+
- **Caveats:** the SISO certificate is order dependent across calls (passes stand, fails prove nothing); the MIMO cost uses only positive unit directions, which is why the new-direction checks carry the conclusions; the 64-sample SISO result is one seed and was measured on the cost directions only; the paired rank test scores an uncertified plant as regret 1e3, so a few certification failures outweigh many small cost gains (the pre-registered rule, the same on SISO and MIMO); the MIMO new-direction comparison covers only plants where both generators certify.
|