matCercola18 commited on
Commit
1778de1
·
verified ·
1 Parent(s): 10c5142

X29: actuator step 3B not run (Matteo's decision); PREREG addendum and write-up

Browse files
docs/icml_r3_x29.md CHANGED
@@ -1,7 +1,7 @@
1
  ---
2
  title: C+phi next to C-only with the same baselines and KPIs (X29). SISO, the two generators tie at 16 samples on the common J* and both have lower median regret than kNN, the regression twin and the flat references. MIMO spline setting (200 fresh plants per m), the repaired C+phi generator has the lowest median regret of the generators and baselines at 16 samples at both m, and C-only certifies more at 1 sample. Actuator setting, at the per-plant optimum C+phi equals C-only (median gain 0.00% at m = 3, 4, 5)
3
  date: 2026-11-17
4
- status: SISO (with CPU), MIMO spline setting (with CPU) and actuator step 3A complete; actuator step 3B (C+phi generators on X27's banks) not run, pending Matteo's decision
5
  claim: X29 (Matteo 2026-11-16, via dc_presentation and in the move-HPC session): "I need to have the final baseline comparison also for the spline methods". C+phi (the model generates the linear part and the spline phi, X28) gets the same baselines and KPIs as C-only (phi fixed: SISO sat_1, MIMO identity) in every setting of the deck v10 slides.
6
  maps_to: deck v10 slides [35] [36] [37] [38] [73] (SISO); the three new MIMO spline-setting slides and [73]; [50] [51] [52] [84] [73] (actuator); summaries [54] [72] [77]. No paper edit until Matteo agrees the claim.
7
  scripts: nonlinear/experiments/x29_siso.py (SISO kNN, twins, optimizer from the best sample, timing, report), nonlinear/mimo_unstable/x29_t2.py (MIMO spline setting on T2: kNN, twins, 48-start solver, Adam references, timing, report), nonlinear/mimo_unstable/x29_act.py (actuator 3A: composed-sector certificate, simulated X27 cost, optimizer, new directions, report, X27 rows and re-score)
@@ -14,7 +14,7 @@ artifacts: results/icml_r3/X29/{PREREG.md, PREREG_siso.md, PREREG_t2.md}; result
14
  - **Question:** The deck compares C-only (phi fixed; the model generates the linear part) with kNN, regression twins, per-plant optimizers and classical references. Matteo asked for the same comparison for C+phi (the model also generates the spline phi, X28) in every setting, on one regret reference per setting.
15
  - **SISO (199 test plants):** at 16 samples the C+phi generator certifies 95.6% of plants with median regret 6.59%, and C-only 98.0% with 6.74% (3-seed means). Both have lower regret than their kNN (11.3%, 10.2%), their regression twins (31.8%, 20.1%), the tuned LQG or static gain (34.05%) and the best of 16 HIFOO starts (50.75%). At 1 sample C+phi certifies 50.6% of plants against 64.0%. The joint optimizer reaches the C+phi generator's regret with 2 starts; the F-only optimizer needs 8 to reach C-only's. On one CPU thread the C+phi generator costs 29.1 s per plant at 16 samples against 11.5 s (same sampling time, slower scoring), and the variant's own optimizer at the start count that reaches the generator's regret costs 106 times the C+phi generator's CPU and 138 times the C-only generator's (October: 140 times).
16
  - **MIMO spline setting (200 fresh plants per m):** at 16 samples the repaired C+phi generator has median regret 6.56% at m = 3 and 3.59% at m = 2, against 10.22% and 6.42% for C-only, with certified shares within 0.5 points. Its kNN and twin also have lower regret than C-only's. The 48-start solver has 11.60% and 12.33%. At 1 sample C-only certifies more plants (95.7% against 79.7% at m = 3). Each generator costs about 3 s per plant on one CPU thread and about 0.05 s per plant batched on a GPU; the per-plant Adam optimizer costs about 4,200 to 5,100 s and 34 to 41 s.
17
- - **Actuator setting (X27's 120 test plants per m):** at each plant's optimum C+phi costs the same as C-only (median gain 0.00% at m = 3, 4 and 5), and the best certified spline is linear on most plants. So a C+phi generator can at best tie here, and step 3B (C+phi generators on X27's banks, 2 to 3 GPU-days) waits for Matteo. X27's C-only pipeline, re-run on its stored candidates, reproduces every published number; on the common J*_A the regret of its filtered arms at 16 samples rises by 0.5 to 1.2 points in X27's convention (median over certified plants).
18
  - **So what:** C+phi changes the comparison only in the MIMO spline setting, where it has lower regret than C-only for every method family at 16 samples. On SISO it ties, and in the actuator setting it has nothing to gain at the optimum. With 1 sample, C-only certifies more plants in SISO and in the MIMO spline setting.
19
 
20
  ## 1. Problem / Goal
@@ -184,12 +184,13 @@ SISO (`results/icml_r3/X29/siso/report.txt`, CPU section; seconds per plant on o
184
  ## 4. Conclusion
185
  - **SISO:** C+phi and C-only generators tie at 16 samples on the common J*, as X28 S4 found, and both have lower median regret than kNN, the regression twins and the flat references. C+phi's own optimizer is stronger (2 joint starts reach the C+phi generator's regret; the F-only optimizer needs 8 to reach C-only's), but each optimizer costs about the same at any start count, so the compute ratio moves only from 138 times (C-only) to 106 times (C+phi). At 1 and 4 samples C+phi certifies fewer plants.
186
  - **MIMO spline setting:** C+phi has lower median regret than C-only for each method family at 16 samples (difference of median regrets at m = 3 and m = 2: generator 3.7 and 2.8 points, kNN 7.9 and 4.7, twin 3.0 and 1.5), at equal certified share with repair. C-only certifies more at 1 sample. The C+phi generator at 16 samples has lower median regret than C-only's per-plant Adam optimizer at m = 2 (3.59% against 5.07%), not at m = 3 (6.56% against 4.33%), at about 1/1,500 of its CPU per plant on one thread.
187
- - **Actuator setting:** generating phi gains nothing at the per-plant optimum, so the actuator slides can state "C+phi = C-only here" with the 3A numbers. Step 3B is not run unless Matteo asks for it.
188
 
189
  **Deviations and caveats**
190
  - SISO C-only twin rows are X21's stored outputs re-scored on J*. The re-sampled seed 0 missed its 1-point self-check on the certified share per candidate (+1.57) because the MC-dropout draws differ across machines; this was registered before any twin regret (`PREREG_siso.md`, 19:05 UTC). The re-sampled rows agree within 1 point at b = 16 (85.76% / 20.73%).
191
  - The C+phi SISO twins reached their best validation loss early (step 4,000 of 60,000), so part of their gap to the generator may come from the training run.
192
  - The actuator cost first used the trapezoid rule, which failed the tr(P) self-check at every horizon (median 0.25%); it was replaced by RK4 on an extra cost state before any 3A number (`PREREG.md` addendum 18:20 UTC).
 
193
  - The joint optimizer and the C+phi Adam optimizer define J* in their settings, so their regret falls to 0 by construction.
194
  - Regrets here are on each setting's common J* and are higher than earlier C-only numbers on J_Fonly (SISO) or J_ref (actuator); the two scales are not mixed.
195
  - The MIMO spline J*_T2 uses id and so3 labels only, so T2 regrets are not on the scale of the held-out X28 regrets.
 
1
  ---
2
  title: C+phi next to C-only with the same baselines and KPIs (X29). SISO, the two generators tie at 16 samples on the common J* and both have lower median regret than kNN, the regression twin and the flat references. MIMO spline setting (200 fresh plants per m), the repaired C+phi generator has the lowest median regret of the generators and baselines at 16 samples at both m, and C-only certifies more at 1 sample. Actuator setting, at the per-plant optimum C+phi equals C-only (median gain 0.00% at m = 3, 4, 5)
3
  date: 2026-11-17
4
+ status: complete. SISO (with CPU), MIMO spline setting (with CPU) and actuator step 3A done; actuator step 3B (C+phi generators on X27's banks) not run, by Matteo's decision of 2026-11-17 (the deck draws C+phi at the C-only values in that setting, labeled)
5
  claim: X29 (Matteo 2026-11-16, via dc_presentation and in the move-HPC session): "I need to have the final baseline comparison also for the spline methods". C+phi (the model generates the linear part and the spline phi, X28) gets the same baselines and KPIs as C-only (phi fixed: SISO sat_1, MIMO identity) in every setting of the deck v10 slides.
6
  maps_to: deck v10 slides [35] [36] [37] [38] [73] (SISO); the three new MIMO spline-setting slides and [73]; [50] [51] [52] [84] [73] (actuator); summaries [54] [72] [77]. No paper edit until Matteo agrees the claim.
7
  scripts: nonlinear/experiments/x29_siso.py (SISO kNN, twins, optimizer from the best sample, timing, report), nonlinear/mimo_unstable/x29_t2.py (MIMO spline setting on T2: kNN, twins, 48-start solver, Adam references, timing, report), nonlinear/mimo_unstable/x29_act.py (actuator 3A: composed-sector certificate, simulated X27 cost, optimizer, new directions, report, X27 rows and re-score)
 
14
  - **Question:** The deck compares C-only (phi fixed; the model generates the linear part) with kNN, regression twins, per-plant optimizers and classical references. Matteo asked for the same comparison for C+phi (the model also generates the spline phi, X28) in every setting, on one regret reference per setting.
15
  - **SISO (199 test plants):** at 16 samples the C+phi generator certifies 95.6% of plants with median regret 6.59%, and C-only 98.0% with 6.74% (3-seed means). Both have lower regret than their kNN (11.3%, 10.2%), their regression twins (31.8%, 20.1%), the tuned LQG or static gain (34.05%) and the best of 16 HIFOO starts (50.75%). At 1 sample C+phi certifies 50.6% of plants against 64.0%. The joint optimizer reaches the C+phi generator's regret with 2 starts; the F-only optimizer needs 8 to reach C-only's. On one CPU thread the C+phi generator costs 29.1 s per plant at 16 samples against 11.5 s (same sampling time, slower scoring), and the variant's own optimizer at the start count that reaches the generator's regret costs 106 times the C+phi generator's CPU and 138 times the C-only generator's (October: 140 times).
16
  - **MIMO spline setting (200 fresh plants per m):** at 16 samples the repaired C+phi generator has median regret 6.56% at m = 3 and 3.59% at m = 2, against 10.22% and 6.42% for C-only, with certified shares within 0.5 points. Its kNN and twin also have lower regret than C-only's. The 48-start solver has 11.60% and 12.33%. At 1 sample C-only certifies more plants (95.7% against 79.7% at m = 3). Each generator costs about 3 s per plant on one CPU thread and about 0.05 s per plant batched on a GPU; the per-plant Adam optimizer costs about 4,200 to 5,100 s and 34 to 41 s.
17
+ - **Actuator setting (X27's 120 test plants per m):** at each plant's optimum C+phi costs the same as C-only (median gain 0.00% at m = 3, 4 and 5), and the best certified spline is linear on most plants. So a C+phi generator can at best tie here. Matteo decided not to run step 3B (C+phi generators on X27's banks, 2 to 3 GPU-days); the deck draws C+phi at the C-only values in this setting, labeled as not trained separately. X27's C-only pipeline, re-run on its stored candidates, reproduces every published number; on the common J*_A the regret of its filtered arms at 16 samples rises by 0.5 to 1.2 points in X27's convention (median over certified plants).
18
  - **So what:** C+phi changes the comparison only in the MIMO spline setting, where it has lower regret than C-only for every method family at 16 samples. On SISO it ties, and in the actuator setting it has nothing to gain at the optimum. With 1 sample, C-only certifies more plants in SISO and in the MIMO spline setting.
19
 
20
  ## 1. Problem / Goal
 
184
  ## 4. Conclusion
185
  - **SISO:** C+phi and C-only generators tie at 16 samples on the common J*, as X28 S4 found, and both have lower median regret than kNN, the regression twins and the flat references. C+phi's own optimizer is stronger (2 joint starts reach the C+phi generator's regret; the F-only optimizer needs 8 to reach C-only's), but each optimizer costs about the same at any start count, so the compute ratio moves only from 138 times (C-only) to 106 times (C+phi). At 1 and 4 samples C+phi certifies fewer plants.
186
  - **MIMO spline setting:** C+phi has lower median regret than C-only for each method family at 16 samples (difference of median regrets at m = 3 and m = 2: generator 3.7 and 2.8 points, kNN 7.9 and 4.7, twin 3.0 and 1.5), at equal certified share with repair. C-only certifies more at 1 sample. The C+phi generator at 16 samples has lower median regret than C-only's per-plant Adam optimizer at m = 2 (3.59% against 5.07%), not at m = 3 (6.56% against 4.33%), at about 1/1,500 of its CPU per plant on one thread.
187
+ - **Actuator setting:** generating phi gains nothing at the per-plant optimum. Step 3B is not run (Matteo's decision, 2026-11-17): the actuator slides draw C+phi at the C-only values, labeled "= C-only, not trained separately", with the 3A numbers as the reason.
188
 
189
  **Deviations and caveats**
190
  - SISO C-only twin rows are X21's stored outputs re-scored on J*. The re-sampled seed 0 missed its 1-point self-check on the certified share per candidate (+1.57) because the MC-dropout draws differ across machines; this was registered before any twin regret (`PREREG_siso.md`, 19:05 UTC). The re-sampled rows agree within 1 point at b = 16 (85.76% / 20.73%).
191
  - The C+phi SISO twins reached their best validation loss early (step 4,000 of 60,000), so part of their gap to the generator may come from the training run.
192
  - The actuator cost first used the trapezoid rule, which failed the tr(P) self-check at every horizon (median 0.25%); it was replaced by RK4 on an extra cost state before any 3A number (`PREREG.md` addendum 18:20 UTC).
193
+ - Actuator step 3B was planned only if Matteo asked for it after 3A, and was not run (`PREREG.md` addendum 2026-11-17). The C+phi actuator values on the deck are C-only's values, not a measurement.
194
  - The joint optimizer and the C+phi Adam optimizer define J* in their settings, so their regret falls to 0 by construction.
195
  - Regrets here are on each setting's common J* and are higher than earlier C-only numbers on J_Fonly (SISO) or J_ref (actuator); the two scales are not mixed.
196
  - The MIMO spline J*_T2 uses id and so3 labels only, so T2 regrets are not on the scale of the held-out X28 regrets.
results/icml_r3/X29/PREREG.md CHANGED
@@ -183,3 +183,11 @@ Change: the running cost is now integrated by RK4 as an extra state, with the st
183
  The composed-sector self-check at m = 3 passed (200 gains, 0 disagreements) and does not depend on the cost.
184
 
185
  (Operational note: srun without `-n 1` started two copies of the self-check script on node 06. Both were killed. The only file they wrote, `selfcheck_cert_m3.json`, came out identical from both copies.)
 
 
 
 
 
 
 
 
 
183
  The composed-sector self-check at m = 3 passed (200 gains, 0 disagreements) and does not depend on the cost.
184
 
185
  (Operational note: srun without `-n 1` started two copies of the self-check script on node 06. Both were killed. The only file they wrote, `selfcheck_cert_m3.json`, came out identical from both copies.)
186
+
187
+ ## Addendum 2026-11-17, 08:25 UTC: step 3B not run
188
+
189
+ After the 3A result (median d 0.00% at m = 3, 4 and 5), Matteo decided not to run 3B:
190
+ - "Yes, skip it" (his answer in the presentation session dc_presentation);
191
+ - "If they cost the same just put in the slide a c+spline line with the same cost" (in the move-HPC session).
192
+
193
+ So no C+phi actuator generator, twin, kNN or timing exists. In the actuator setting the deck draws C+phi at the C-only values, labeled as equal to C-only and not trained separately, with the 3A result as the reason. X29's actuator results are 3A and the re-score of X27's C-only arms.