Download docs/icml_r3_x28.md from matCercola18/DiffusionControl: direct link, hf CLI and curl.
- Browser
- Download file 31.7 kB
-
https://huggingface.co/matCercola18/DiffusionControl/resolve/main/docs/icml_r3_x28.md
- Command line
-
hf download hf://matCercola18/DiffusionControl/docs/icml_r3_x28.md
-
curl -L -o icml_r3_x28.md https://huggingface.co/matCercola18/DiffusionControl/resolve/main/docs/icml_r3_x28.md
Generating the spline nonlinearity together with the controller (X28)
TL;DR
- Question: In July the diffusion model generated the nonlinearity phi (an 11-knot spline) together with the linear part. In October phi was fixed (SISO: sat_1; MIMO: identity). Does generating phi buy anything, and if not, what does it cost?
- SISO answer: No gain. At each plant's optimum the spline is cheaper than sat_1 by a median 2.84%, but all of it is knot asymmetry fitted to the 4 fixed initial directions of the cost: on 64 new directions it is no better (-0.36%), and an odd spline gains 0.00%. On the 199 test plants the spline generator ties the X21 sat_1 generator at 16 samples ("no measurable difference" on all 3 seed pairs), certifies 6 to 16 points fewer plants at 1 and 4 samples, and is 1.3% to 1.5% more expensive on new directions.
- MIMO answer: At the optimum every spline class beats identity by a median 4.5% to 6.6%, but only odd (sign-symmetric) classes keep a gain on new initial directions; free knots lose 2% to 6%. A pre-registered study over five classes picks odd piecewise-linear knots at |x| = 1, 3, 5 (3 numbers per channel). As registered, the generator of K and these knots ties the identity generator at 16 samples ("no measurable difference" at m = 2 and m = 3). Its paired median regret is 1.1 to 2.4 points lower, but it certifies only 57% to 76% of its samples against 93% to 99.5%. July certified every spline by projecting it into a safe sector, so a follow-up test, filed before any number, gives the spline generator that step: a rejected sample's knots are flattened until it certifies. With it, the spline generator certifies 99.2% to 100% of plants at 16 samples, like identity, and is better at m = 3 ("spline better" on 2 of 3 seed pairs: +1.3 and +1.8 points, Holm p 0.042). At m = 2 it is cheaper on all 3 pairs (+1.9 to +2.5 points, CIs above 0), but the rank test misses (Holm p 0.11), because it is worse by more than 10 points on 15 to 19 of 120 plants. At 1 sample it still certifies 12 to 20 points fewer plants. On new directions it is cheaper at m = 2 (+1.3% to +2.0%) and within the seed spread at m = 3.
- MIMO, fresh plants: a third test, filed before any number (Matteo: "Can you try to improve the spline"), lets a few spline samples decide where to draw from the identity generator instead. It was designed from the held-out losses, so its verdict comes from 200 fresh plants per m that no model, rule or diagnostic had seen. Validation chose the simplest form: every plant gets 8 spline and 8 identity samples. On the fresh plants this mix is better than 16 identity samples at both m: +2.40 to +3.03 points on 3 of 3 seed pairs at m = 3, and +0.98 to +1.69 points at m = 2, "better" on 2 of 3 pairs because the third misses the 1-point bar by 0.02 (Holm p <= 2.7e-11 on all six). It is worse by more than 10 points on 1 to 6 plants of 200. The same fresh plants replicate the repaired spline generator alone: "spline better" at both m on 3 of 3 pairs (+3.11 to +3.72 points at m = 3, +1.76 to +2.33 at m = 2, Holm p <= 0.0019), cheaper on new directions at both m (+1.55% to +2.39%), certified like identity. The as-registered generator, without repair, again shows no measurable difference, and at 1 sample identity is still better.
- So what: On SISO, generating phi buys nothing, which supports phi = sat_1. On MIMO, generating the odd 3-knot shape pays at 16 samples once each sample is projected to a certified one, as in July: on fresh plants, equal certification and 1.8 to 3.7 points lower paired median regret at both m, and lower cost on new directions. Splitting the 16 samples between the spline and the identity generator also beats identity at both m, with fewer large losses but a smaller median gain. Identity stays better at 1 sample.
1. Problem / Goal
- SISO setting (X21): open-loop-unstable plants, bounded actuator, controller u = -phi(clip(F y, +-5)) with a dynamic filter F. X21 fixed phi = sat_1 and generated F. X28 generates F and the 10 nonzero knots of phi (|phi| <= 1, phi(0) = 0) with the same recipe.
- MIMO setting (X27 family): u = -Phi(K y), Phi = diag(psi_i), each psi_i a spline with knot ratios r = psi(x)/x in [0.1, 10], extended past +-5 along the ray, so the sector of each channel is exactly [min r, max r] and the circle criterion certifies the loop globally.
- Falsification (PREREG): the spline is "better" only if the paired median regret improves by at least 1 point with Holm p < 0.05 and the certified rate drops by at most 2 points, on at least 2 of 3 seed pairs; "worse" symmetrically; otherwise "no measurable difference".
2. Method
- S1 gate (per-plant optimum, 40 validation plants): projected Adam on F and the knots from the X20b sat_1 label, gauge starts c = 1, 2, 5 (sat(y/c) with F -> cF is the same controller), cheapest snapshot certified by the regional circle criterion at s1. A control arm with frozen knots and the same optimizer and budget separates the spline from extra optimization of F. Diagnostics: odd (symmetrized) knots, and 64 new unit initial directions per plant (seed (7474, i)).
- S2 labels: each X21 training label refined jointly at c = 1 (1,971 training plants, 30,655 rows, 22,954 with knots off sat_1; training log of
x28_clS_s0). Test plants also get c = 2, 5; these enter J* only. - S3 generators: X21 recipe (DDPM-500, MLP 512 x 8, EMA 0.999, cost-slot conditioning). clS: closed-loop coordinates of F at the linearized gain k = (phi(1) - phi(-1))/2 plus 10 knots, seeds 0 to 2; coefS: slog of the coefficients plus knots, seed 0. Sampler clip and t-repair fixed on validation (addendum 2026-11-13): t-repair on every arm, X21 clip rule (lowest regret among clips certifying all 40 validation plants).
- S4 test: 199 X21 test plants, 16 samples per plant (64 for seed 0 as a secondary), J* = min of all F-only and joint labels, regret with uncertified = 1e3, paired against X21 clL scored by the same code.
- MIMO gate: 120 held-out plants per m (m = 2, 3), identity and spline arms from the same bank gain and the same Adam budget (400 steps, K lr 0.02 chosen on 64 validation bank plants), cost over the 2m positive unit directions (20 s, RK4 dt 0.01), circle LMI per channel (self-check against
family_sofm.verify: 200 gains per m, 0 disagreements,selfcheck_m*.json). New directions: 64 random unit directions in R^(2m), both signs, seed (7575, id). - MIMO spline-class study (Matteo 2026-11-13): five classes on the 64 validation bank plants per m: piecewise linear free (
sp, 10 per channel), odd (so, 5), odd with knots at |x| = 1, 3, 5 (so3, 3), and cubic PCHIP on the ratios, free (cf, 10) and odd (co, 5); the PCHIP interpolant stays between its end values on each interval, so the sector stays exact. Rule filed before the run: eligible iff the new-direction gain over identity has CI lower bound > 0 at both m and at most 1% more non-convergent rollouts than identity; winner by mean gain, ties within 0.25 points to fewer parameters. - MIMO S2 labels: 7,936 bank plants at m = 3 (indices 64 to 8,000) and 3,936 at m = 2 (64 to 4,000), identity at c = 1 and
so3at c = 1, 2, 5, 400 Adam steps each (rollout gradient checkpointing, same math). Label (a) is the identity gain K. Label (b) is the cheapest of the threeso3starts and identity, stored as K plus log r_i(1), log r_i(3), log r_i(5) per channel with no gauge normalization (PREREG correction 2026-11-13: with knots at fixed positions, scaling a row of K is not a symmetry). - MIMO S3 generators: X27 recipe (DDPM-500, MLP 256 x 5, 96k steps, batch 64, lr 2e-4, EMA 0.999), 3 seeds per arm and m, 12 models.
- MIMO S4 test: the 120 held-out plants per m, 16 samples per plant (clip 1, seed = model seed). Arm (a) is certified by the Lyapunov LMI (alpha = beta = 1), arm (b) by the circle LMI. Cost on the 2m unit directions, J* = min of all held-out labels of the gate, and the SISO verdict rule on seed j against seed j. Diagnostic: the best certified 16-sample controller per plant on the 64 new directions of seed (7575, id), against (a) seed 0.
- MIMO S4 with repair (PREREG addendum 2026-11-14; Matteo asked whether July was tested at its best, then "Yes, make it work"): for a rejected arm (b) sample, the log knot ratios of each channel move toward a center g by a factor lam in {0.75, 0.5, 0.25, 0}, K is kept, and the largest certified lam wins. lam = 0 is the linear controller u = -diag(g) K y. Each sample yields one repaired controller, so the budget is unchanged. The center (geometric mean of r(1), r(3), r(5), or r(1)) was chosen on the 64 validation bank plants. A Popov certificate was planned next to the circle LMI, subject to two self-checks. Same samples, arm (a) results, J* and rule as S4-M.
- MIMO identity fallback on fresh plants (PREREG addendum 2026-11-15; Matteo: "Can you try to improve the spline on decision 1"): arm (c) draws the first q of its 16 samples from the spline generator; if at most c of them certify as sampled, the other 16 - q come from the identity generator of the same seed (sampler seed + 1000, independent of arm (a)'s draws), otherwise from the spline generator. Spline samples get the repair above (circle LMI, center r(1)), identity samples the Lyapunov test, and the best certified J is used. (q, c) with q in {4, 8} and c in {-1 (never), 0, ..., q (always)} was chosen per m on the 64 validation plants by the largest standardized Wilcoxon statistic of regret (a) - regret (c) at b = 16, the statistic that stopped m = 2. Test plants T2: 200 per m, built like the held-out sets (
build_sofm, uniform in the family box, 48 oracle starts, kept iff the oracle certifies a gain, seeds 1115 and 1116). J*_T2 is the cheapest certified controller of the S2-M label optimizer on T2 (identity at c = 1,so3at c = 1, 2, 5, 400 Adam steps). The 12 S3-M generators sample T2 with seed = model seed. Primary: (c) against (a), the S4-M rule; replications: the repaired and the unrepaired spline generator against (a). New directions on T2: seed (7979, plant index). (c) is also reported on the 120 held-out plants, which shaped the variant.
3. Results
3.1 SISO, per-plant optimum (40 validation plants)
| quantity | median [95% CI] | file |
|---|---|---|
| spline vs sat_1, gate d | 2.84% [1.06, 5.61], >= 2% on 23 of 40 | gate_report.txt |
| same optimizer and budget, knots frozen (control e) | 2.82% [1.01, 4.73] | control_report.txt |
| extra F optimization alone | 0.00% [0.00, 0.03] | control_report.txt |
| spline with its knots made odd, vs the spline | +3.85% [1.17, 5.78] dearer | diag_report.txt |
| spline vs frozen knots on 64 new directions | -0.36% [-4.87, 0.00] | diag_report.txt |
| odd-knot spline vs frozen knots (gate) | 0.00% [0.00, 0.00], >= 2% on 0 of 40 | gate_sym_report.txt |
The gain at the optimum is real for the cost as defined, and it is not extra F optimization. It comes from asymmetric knots that fit the 4 fixed initial directions; removing the asymmetry removes it, and it does not carry over to new directions. 3 of 40 best spline controllers did not re-certify in a fresh process (the certificate is order dependent); counting them as no gain gives e = 2.04% [0.43, 4.46] (PREREG addendum 2026-11-05).
3.2 SISO, generators on the 199 test plants
| arm (t-repair, validation clip) | certified per sample | b = 1 certified | b = 16 certified | b = 16 regret vs J* |
|---|---|---|---|---|
| clS spline, seeds 0 / 1 / 2 (clip 1.5) | 52.2 / 52.1 / 52.7% | 49.2 / 52.8 / 49.7% | 95.0 / 96.5 / 95.5% | 7.14 / 6.33 / 6.30% |
| X21 clL sat_1, seeds 0 / 1 / 2 (clip 1.5) | 65.9 / 65.1 / 65.5% | 65.3 / 65.3 / 61.3% | 98.5 / 98.0 / 97.5% | 6.79 / 6.62 / 6.81% |
| coefS spline, seed 0 (clip 3) | 49.7% | 48.7% | 95.0% | 10.58% |
Source: eval_test_report.txt. Paired (seed j vs seed j, eval_test_paired.txt):
- b = 16 (primary): clL - clS median regret difference -0.04 [-0.61, +0.48], +0.19 [-0.42, +0.91], -0.32 [-1.19, +0.19] points, Holm p = 1 on all three; certified -3.5, -1.5, -2.0 points. Verdict: no measurable difference.
- b = 1 and 4: the spline generator certifies fewer plants: -16.1, -12.6, -11.6 points at b = 1 (McNemar p 0.00045 to 0.0095) and -6.0, -8.5, -6.5 at b = 4.
- Without t-repair (secondary) the spline generator loses at b = 16 by 0.75 to 2.92 points (Holm p <= 0.0014). coefS loses by 3.23 points.
- 64 samples, seed 0 (secondary): at b = 64 the spline generator is cheaper on the cost directions, 2.96% vs 4.07%, paired +0.44 [0.16, 0.91] points (Holm p 7.5e-05 within that block); at b = 16 in the same runs 6.61% vs 6.60%.
- New directions (
eval_test_fresh.txt, best b = 16 sample per plant, 64 directions at s, against clL seed 0): clS -1.41% [-2.40, -0.30], -1.50% [-2.68, -0.30], -1.31% [-2.14, -0.30]; coefS -4.44%; clL seeds 1 and 2 +0.22% and +0.23% (seed-to-seed spread). - Knots: the best b = 16 sample moves the knots from sat_1 by a median 0.077 to 0.086 and is not odd (max |phi(x) + phi(-x)| median 0.055 to 0.066).
3.3 MIMO, per-plant optimum
Held-out gate, 120 plants per m (gate_heldout_m*.txt, fresh_heldout_m*.txt):
| class | m = 3 at optimum | m = 3 new directions | m = 2 at optimum | m = 2 new directions |
|---|---|---|---|---|
| free piecewise linear | 5.19% [4.55, 6.07] | -4.55% [-6.09, -3.55], 210 non-convergent | 6.36% [5.40, 7.05] | -1.58% [-2.64, -0.69], 131 non-convergent |
| odd piecewise linear | 4.36% [3.58, 5.14] | +0.54% [0.20, 1.27] | 5.52% [4.55, 6.36] | +2.71% [1.88, 3.39] |
Spline-class study, 64 validation plants per m (gate_study_m*.txt, study_choice.txt):
| class | parameters per channel | optimum m = 3 / m = 2 | new directions m = 3 | new directions m = 2 | eligible |
|---|---|---|---|---|---|
sp free PWL |
10 | 6.31% / 6.61% | -5.94% | -2.11% | no |
so odd PWL |
5 | 4.80% / 5.40% | +1.32% [0.64, 1.94] | +3.11% [2.48, 4.11] | yes |
so3 odd PWL, knots 1, 3, 5 |
3 | 4.54% / 5.22% | +1.82% [0.74, 2.57] | +3.99% [2.86, 5.13] | yes |
cf free cubic |
10 | 5.39% / 6.39% | -6.06% | -3.09% | no |
co odd cubic |
5 | 5.02% / 5.37% | +0.91% [0.45, 1.51] | +2.75% [2.13, 3.45] | yes |
Choice: so3. Interpolation (linear or cubic) does not matter; symmetry does. Fewer knots give the largest gain on new directions.
Held-out check of so3 (120 plants per m, its own Adam call next to the stored arms, which reproduce exactly; gate_heldout_m*.txt, fresh_heldout_m*.txt):
| m | at optimum | new directions | non-convergent |
|---|---|---|---|
| 2 | 4.83% [4.14, 6.34], >= 2% on 97 of 120 | +2.96% [2.29, 3.65], > 0 on 106 of 120 | 0 |
| 3 | 4.27% [3.26, 4.80], >= 2% on 93 of 120 | +0.94% [0.53, 2.00], > 0 on 90 of 120 | 0 |
Both pre-registered rules hold at both m (gate: median >= 2% and CI lower bound > 0; new directions: CI lower bound > 0).
3.4 MIMO, generators on the 120 held-out plants
Labels (ds/info_m*.json): every plant is certified at every start (a start is the certified bank gain). Label (b) comes from so3 c = 1 / 2 / 5 / identity on 4,146 / 3,029 / 740 / 21 plants at m = 3 and 1,949 / 1,470 / 517 / 0 at m = 2. It is cheaper than label (a) by a median 4.14% (m = 3) and 5.01% (m = 2). The target round trip reproduces J to 2e-16.
| m | arm, seeds 0 / 1 / 2 | certified per sample | b = 1 certified | b = 16 certified | b = 16 regret vs J* |
|---|---|---|---|---|---|
| 3 | (a) identity | 96.0 / 96.6 / 93.4% | 95.0 / 97.5 / 94.2% | 100 / 100 / 100% | 10.66 / 10.46 / 10.56% |
| 3 | (b) so3 |
57.4 / 58.6 / 60.5% | 61.7 / 60.8 / 59.2% | 97.5 / 96.7 / 97.5% | 8.64 / 9.29 / 9.54% |
| 2 | (a) identity | 99.5 / 98.8 / 99.0% | 100.0 / 98.3 / 99.2% | 100 / 100 / 100% | 8.22 / 7.95 / 7.99% |
| 2 | (b) so3 |
75.9 / 76.0 / 73.4% | 75.0 / 76.7 / 68.3% | 95.0 / 96.7 / 94.2% | 4.76 / 4.79 / 5.27% |
Source: gen_paired_m*.txt. Paired (seed j vs seed j; positive means the spline is cheaper):
- b = 16 (primary): (a) - (b) median regret difference +1.28 [-0.28, +2.51], +1.06 [+0.08, +2.61], +1.73 [+0.08, +3.45] points at m = 3 (Holm p 0.66, 0.47, 0.31) and +2.41 [+0.94, +3.39], +1.94 [+0.49, +2.66], +1.96 [+0.91, +2.93] at m = 2 (Holm p 0.31 on all three); certified -2.5, -3.3, -2.5 points (m = 3) and -5.0, -3.3, -5.8 (m = 2). Verdict: no measurable difference at both m. The bootstrap CI of the median excludes 0 on 5 of 6 pairs, but the rank test counts the 3 to 7 plants the spline leaves uncertified (regret 1e3) as its largest losses, and the certified drop exceeds the 2-point margin on all 6 pairs.
- b = 1: the spline generator certifies 33.3, 36.7, 35.0 points fewer plants at m = 3 and 25.0, 21.7, 30.8 at m = 2 (McNemar p <= 2.2e-07), and its regret is higher by 8.35, 8.34, 3.15 points (m = 3) and 2.51, 3.87, 4.44 (m = 2), Holm p <= 9e-05.
- b = 4: certified 7.5 to 14.2 points fewer (McNemar p 1.5e-05 to 0.035); regret difference between -0.42 and +1.29 points (Holm p >= 0.097).
- New directions (
gen_fresh_m*.txt, best certified b = 16 sample per plant, 64 directions, against (a) seed 0, plants where both certify): at m = 2, (b) +2.08% [+0.55, +3.13], +1.69% [+0.60, +2.60], +1.57% [+0.68, +2.77], against -0.07% and -0.07% for (a) seeds 1 and 2. At m = 3, (b) +0.44% [-1.13, +1.97], +0.66% [-0.35, +2.52], +0.04% [-1.01, +1.67], against +0.26% and +0.33% for (a) seeds 1 and 2. Non-convergent rollouts: 0 to 41 per seed for (b), 0 to 64 for (a), of 7,232 to 7,680.
3.5 MIMO, generators with the July projection (repair)
Self-checks (rep_selfcheck.txt): the Popov LMI accepts all 400 circle-certified validation samples and certifies 51 of 1,876 circle-rejected ones. Simulated for 60 s from 64 directions at radii 1 and 5, 24 of those 6,528 rollouts do not shrink (circle-certified controls: 0 of 25,600), so by the rule the test uses the circle LMI alone. The cause is not diagnosed: it could be a fault in the Popov code, or RK4 at dt = 0.01 on the wide-sector loops that Popov admits. Popov enters no result. Validation (rep_val_choice.txt): both centers certify 99.5% of plants at 16 samples (identity 100%); r(1) has the lower median regret (6.46% against 6.52%) and is used.
| m | arm, seeds 0 / 1 / 2 | certified per sample | b = 1 certified | b = 16 certified | b = 16 regret vs J* |
|---|---|---|---|---|---|
| 3 | (a) identity | 96.0 / 96.6 / 93.4% | 95.0 / 97.5 / 94.2% | 100 / 100 / 100% | 10.66 / 10.46 / 10.56% |
| 3 | (b) so3, repaired |
79.3 / 79.3 / 82.6% | 83.3 / 80.0 / 80.8% | 99.2 / 100 / 100% | 8.25 / 8.51 / 9.04% |
| 2 | (a) identity | 99.5 / 98.8 / 99.0% | 100.0 / 98.3 / 99.2% | 100 / 100 / 100% | 8.22 / 7.95 / 7.99% |
| 2 | (b) so3, repaired |
88.2 / 87.5 / 85.0% | 86.7 / 86.7 / 79.2% | 100 / 100 / 100% | 4.76 / 4.79 / 5.27% |
Source: gen_rep_paired_m*.txt; lam per sample in gen_rep/eval_primary_*.npz. Flattening certifies another 20.7% to 22.1% of the samples at m = 3 and 11.5% to 12.3% at m = 2 (1.4% to 2.4% only at lam = 0, the linear controller); 17.4% to 20.7% (m = 3) and 11.8% to 15.0% (m = 2) stay uncertified. Paired (positive means the spline is cheaper):
- b = 16 (primary): (a) - (b) median regret difference +1.91 [+0.45, +2.71], +1.33 [+0.29, +3.40], +1.80 [+0.30, +4.11] points at m = 3 (Holm p 0.11, 0.042, 0.042) and +2.52 [+1.56, +3.47], +1.94 [+0.63, +2.66], +1.96 [+0.91, +2.93] at m = 2 (Holm p 0.11 on all three); certified -0.8, 0.0, 0.0 points (m = 3) and 0.0 on all three (m = 2). Verdict: spline better at m = 3 (seed pairs 1 and 2), no measurable difference at m = 2.
- Why m = 2 misses (
rep_tails.txt): the spline is better by more than 5 points on 36 to 37 plants, but worse by more than 10 on 15 to 19. The rank test gives p 0.036, 0.050, 0.111 before Holm, so none passes after Holm. The large losses sit on plants where few samples certify as sampled (median 3 to 6 of 16, against 15.5 to 16 where the spline wins). There its best certified sample costs 35% to 41% regret, against 7% to 10% for identity. m = 3 shows the same pattern on 12 to 17 plants (median 2 to 3 of 16 certified as sampled). - b = 1: the spline generator still certifies 11.7 to 17.5 points fewer plants at m = 3 and 11.7 to 20.0 at m = 2 (McNemar p <= 0.0043), and its regret is higher by 2.14 to 4.18 points (m = 3) and 2.01 to 3.54 (m = 2), Holm p <= 0.019.
- b = 4: regret difference +0.32 to +1.48 points (Holm p >= 0.62); certified -5.8 to +1.7 points.
- New directions (
gen_rep_fresh_m*.txt): at m = 2, (b) +2.03% [+0.40, +3.08], +1.69% [+0.51, +2.54], +1.29% [+0.49, +2.41], against -0.07% and -0.07% for (a) seeds 1 and 2. At m = 3, (b) +0.21% [-1.10, +2.12], +0.66% [-0.24, +2.72], +0.07% [-1.08, +1.67], against +0.26% and +0.33%. Non-convergent rollouts of the repaired arm: 42 to 123 of 7,680 at m = 2 and 0 to 58 at m = 3 (identity 0 to 64).
3.6 MIMO, spline with identity fallback, and replication on 200 fresh plants per m
Validation (fb_val_choice.txt, 64 plants per m, pooled over the 3 seed pairs; d = regret (a) - regret (c) at b = 16; every setting certifies 99.0% to 100% of plants, identity 100%):
| setting | m = 3: z, median d | m = 2: z, median d |
|---|---|---|
| never fall back (repaired spline alone) | +4.44, +2.82 pts | +3.91, +2.46 pts |
| q = 8, c = 3 (falls back on 34% / 16% of plants) | +7.24, +2.87 pts | +5.34, +2.52 pts |
| q = 8, c = 8: always 8 spline + 8 identity (chosen) | +7.63, +2.45 pts | +7.70, +1.64 pts |
The rule picks the fixed half-and-half split at both m: it removes the large losses best, at the price of a smaller median gain. So arm (c) is a mix of the two generators, not an adaptive fallback.
T2: each of the 4 builds kept 100 plants from 128 attempts; all four label runs certify 200 of 200 plants at both m. On T2 (fb_paired_t2_m*.txt):
| m | arm, seeds 0 / 1 / 2 | certified per sample | b = 1 certified | b = 16 certified | b = 16 regret vs J*_T2 |
|---|---|---|---|---|---|
| 3 | (a) identity | 95.4 / 95.8 / 93.7% | 95.5 / 97.5 / 94.0% | 100 / 100 / 98.5% | 10.05 / 10.24 / 10.36% |
| 3 | (b) so3, as sampled |
62.0 / 60.5 / 61.5% | 62.0 / 61.0 / 61.5% | 95.5 / 95.0 / 95.5% | 6.16 / 6.90 / 6.74% |
| 3 | (b) so3, repaired |
81.5 / 79.3 / 78.8% | 80.5 / 78.0 / 80.5% | 100 / 100 / 99.5% | 6.16 / 6.82 / 6.71% |
| 3 | (c) 8 spline + 8 identity | 88.0 / 87.7 / 86.2% | 80.5 / 78.0 / 80.5% | 100 / 100 / 99.5% | 6.20 / 6.00 / 6.25% |
| 2 | (a) identity | 98.6 / 98.7 / 98.4% | 98.5 / 98.5 / 98.5% | 99.5 / 100 / 100% | 6.39 / 6.47 / 6.39% |
| 2 | (b) so3, as sampled |
76.9 / 76.5 / 78.7% | 78.0 / 75.5 / 80.5% | 96.5 / 96.5 / 98.5% | 3.59 / 3.66 / 3.70% |
| 2 | (b) so3, repaired |
89.5 / 89.6 / 90.8% | 91.0 / 90.0 / 90.5% | 99.5 / 99.5 / 100% | 3.51 / 3.62 / 3.64% |
| 2 | (c) 8 spline + 8 identity | 94.3 / 94.1 / 94.3% | 91.0 / 90.0 / 90.5% | 100 / 100 / 100% | 3.33 / 3.54 / 3.40% |
Paired on T2 (positive means the spline arm is cheaper). In the fb_*.txt files each block prints its second arm under the letter b; in the primary blocks that arm is (c).
- b = 16, primary, (c) against (a): +3.03 [+1.80, +4.37], +2.40 [+1.47, +3.83], +2.43 [+1.13, +3.90] points at m = 3 (Holm p 1.8e-14, 7.5e-14, 2.7e-11) and +1.69 [+0.90, +2.93], +0.98 [+0.59, +2.14], +1.04 [+0.40, +2.25] at m = 2 (Holm p 4e-16, 5.5e-12, 1.4e-11); certified within +1.0 point. Verdict: spline better at m = 3 (3 of 3 pairs) and at m = 2 (2 of 3; seed pair 1 misses the 1-point bar by 0.02). (c) is worse by more than 10 points on 5, 5, 6 plants at m = 3 and 1, 3, 3 at m = 2, and better by more than 5 on 69 to 77 and 49 to 57.
- Replication of S4-M with repair: +3.40 [+2.20, +4.38], +3.11 [+1.94, +4.71], +3.72 [+1.55, +5.17] points at m = 3 (Holm p 6.8e-4, 7.4e-4, 7.4e-4) and +2.33 [+1.14, +3.27], +1.76 [+0.88, +3.25], +1.78 [+0.78, +3.16] at m = 2 (Holm p 2.9e-4, 0.0019, 0.0017); certified within 1 point. Spline better at both m on 3 of 3 pairs. The m = 3 result of section 3.5 holds on new plants, and m = 2, which missed on the 120 held-out plants, passes on 200 fresh ones.
- Replication of S4-M as registered: +3.00, +2.63, +2.86 points at m = 3 (Holm p 0.062 to 0.063, certified -4.5, -5.0, -3.0 points) and +1.74, +1.57, +1.63 at m = 2 (Holm p 0.0093 to 0.018, certified -3.0, -3.5, -1.5). No measurable difference at both m (1 of 3 pairs at m = 2), as in section 3.4.
- b = 1 (the same for (b) repaired and (c), whose first samples are spline samples): 13.5 to 19.5 points fewer plants certified at m = 3 and 7.5 to 8.5 at m = 2 (McNemar p <= 7.3e-4); regret difference -1.35 to -0.65 points at m = 3 and -1.33 to +0.06 at m = 2 (Holm p <= 0.02). **b = 4:** regret difference +1.77 to +2.35 points at m = 3 and +0.18 to +1.07 at m = 2 (Holm p >= 0.76); certified -7.0 to -3.5 and -2.0 to -0.5 points.
- New directions on T2 (
fb_fresh_t2_m*.txt, against (a) seed 0): at m = 3, (c) +1.65% [+0.92, +2.62], +2.19% [+1.53, +2.83], +1.45% [+0.99, +2.13] and repaired (b) +1.72% [+0.86, +2.63], +2.39% [+1.42, +3.19], +1.55% [+0.39, +2.62], against +0.09% and +0.05% for (a) seeds 1 and 2. At m = 2, (c) +1.17% [+0.46, +2.24], +1.12% [+0.43, +1.99], +0.77% [+0.36, +1.35] and repaired (b) +1.92% [+0.98, +2.83], +2.08% [+1.09, +2.79], +2.06% [+0.62, +3.09], against -0.04% and -0.01%. Non-convergent rollouts per seed, of 12,608 to 12,800: (c) 46 to 117 at m = 3 and 0 at m = 2; repaired (b) 63 to 170 and 61 to 122; (a) 47 to 200 and 0 to 63. - Held-out plants (
fb_heldout_m*.txt; they shaped the variant, so this is secondary): (c) against (a) +1.18, +1.46, +1.33 points at m = 3 (Holm p <= 1.5e-5) and +2.05, +1.34, +1.38 at m = 2 (Holm p <= 1.5e-7), spline better on 3 of 3 pairs at both m; worse by more than 10 points on 0 or 1 plant at m = 3 and 1 or 2 at m = 2, against 12 to 19 for the repaired spline alone (rep_tails.txt).
4. Conclusion
- SISO: generating phi does not pay. The optimum-level gain is a fit to the cost's initial directions, the generator ties sat_1 at 16 samples, loses certified plants at small budgets, and does worse on new directions. This is the measured reason for the October setting phi = sat_1.
- MIMO: a free spline repeats the SISO artifact. An odd spline with 3 knots per channel keeps a gain on new initial directions at the optimum (held-out: +2.96% at m = 2, +0.94% at m = 3). As registered, its generator ties identity at 16 samples, because it certifies far fewer samples (57% to 76% against 93% to 99.5%). Given July's projection step (flatten a rejected sample until it certifies), it certifies as many plants as identity at 16 samples and is cheaper by a paired median 1.3 to 2.5 points. That is "spline better" at m = 3; at m = 2 it is cheaper on all three pairs but not significant after Holm, because it loses big on the 15 to 19 plants where few of its samples certify. Identity stays better at 1 sample. On 200 fresh plants per m the repaired spline generator is better at both m on all three pairs (1.8 to 3.7 points) and cheaper on new directions, so the m = 2 miss on the held-out plants did not hold up. Its weak point is the plants where the generator rarely produces a certifiable spline; giving half of the 16 samples to the identity generator cuts the losses there (worse by more than 10 points on 1 to 6 of 200 plants) and is also better than identity at both m, with a smaller median gain. So on MIMO the July idea pays at the 16-sample budget once each sample is projected to a certified one.
- Caveats: the SISO certificate is order dependent across calls (passes stand, fails prove nothing); the MIMO cost uses only positive unit directions, which is why the new-direction checks carry the conclusions; the 64-sample SISO result is one seed and was measured on the cost directions only; the paired rank test scores an uncertified plant as regret 1e3, so a few certification failures outweigh many small cost gains (the pre-registered rule, the same on SISO and MIMO); the MIMO new-direction comparison covers only plants where both generators certify; the repair test was filed after the registered S4-M result was known (it answers whether July was tested at its best) and is reported next to that verdict, not instead of it; the fallback test was designed from the held-out losses, which is why its verdict and the replications use fresh plants; J*_T2 comes from the identity and
so3label runs only (the held-out J* also includes thespandsoclasses), so T2 regrets are not on the same scale as held-out regrets; the chosen arm (c) is a fixed half-and-half mix of two generators, so it shows that adding the spline generator to the identity one pays at 16 samples, not that one generator replaces the other.