Add files using upload-large-folder tool
Browse files- data/histograms.png +0 -0
- data/system_specs_distribution.png +0 -0
- models/arm_prov/arm_prov.done +1 -0
- models/arm_prov2b/arm2b.done +1 -0
- models/armprov/armprov.done +1 -0
- models/f3/f3_s0.done +1 -0
- models/f3/f3_s1.done +1 -0
- models/f3/f3_s2.done +1 -0
- models/rig1/rig1.done +1 -0
- models/rig1w/rig1w.done +1 -0
- models/rig1wg/rig1wg.done +1 -0
- results/audit/cem_prior_summary.txt +11 -0
- results/audit/furuta_clip_mechanism.txt +4 -0
- results/audit/furuta_failure_modes.txt +7 -0
- results/audit/furuta_hybrid_union.txt +8 -0
- results/audit/furuta_modes.txt +9 -0
- results/audit/guidance_B_heldout.txt +14 -0
- results/audit/mimo_agnostic.txt +11 -0
- results/audit/nn_distance.txt +52 -0
- results/audit/pendulum_ema.txt +18 -0
- results/audit/pendulum_eval_ckpts_relabelM4_e150.txt +16 -0
- results/audit/pendulum_ksweep.txt +76 -0
- results/audit/pendulum_labelquality.txt +4 -0
- results/audit/pendulum_ood.txt +24 -0
- results/audit/pendulum_relabel_retrieval.txt +27 -0
- results/audit/rederive.txt +28 -0
- results/audit/siso_prereg_pooled_seeds.txt +5 -0
- results/audit/siso_retrieval_metric.txt +10 -0
- results/audit/target_range_clip.txt +9 -0
- results/audit/traj_gen_eval_v2_clip5.0.txt +94 -0
- results/audit/traj_gen_ksweep_clip5.0.txt +30 -0
- results/audit/w4_model_arm_v2.txt +3 -0
- results/furuta_v3/gate_sidebyside.txt +9 -0
- results/furuta_v3/traj_gen_eval_v2ctl2400_raw_clip5.txt +94 -0
- results/furuta_v3/traj_gen_eval_v3_ema_clip1.txt +94 -0
- results/furuta_v3/traj_gen_eval_v3e2400_raw_clip1.txt +94 -0
- results/furuta_v3/traj_gen_eval_v3s1_raw_clip1.txt +94 -0
- results/furuta_v3/traj_gen_v3_raw_clip1.txt +30 -0
- results/furuta_v3/traj_gen_v3e2400s1_raw_clip1.txt +30 -0
- results/furuta_v3/traj_gen_v3s1_raw_clip1.txt +30 -0
- results/furuta_v3/traj_gen_warmstart_v2clip5_s0.txt +17 -0
- results/furuta_v3/traj_gen_warmstart_v3_s0.txt +17 -0
- results/furuta_v3/traj_gen_warmstart_v3_s1.txt +17 -0
- results/furuta_v3/traj_gen_warmstart_v3e2400_n200.txt +17 -0
- results/furuta_v3/traj_gen_warmstart_v3e2400_s0.txt +17 -0
- results/furuta_v3/traj_gen_warmstart_v3e2400_s1.txt +17 -0
- results/furuta_v3/traj_gen_warmstart_v3e2400_s2.txt +17 -0
- results/furuta_v3/valloss.txt +7 -0
- results/k_sweep_summary.md +81 -0
- results/regressor_siso_train.log +11 -0
data/histograms.png
ADDED
|
data/system_specs_distribution.png
ADDED
|
models/arm_prov/arm_prov.done
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
2500
|
models/arm_prov2b/arm2b.done
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
2500
|
models/armprov/armprov.done
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
6000
|
models/f3/f3_s0.done
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
24000
|
models/f3/f3_s1.done
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
24000
|
models/f3/f3_s2.done
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
24000
|
models/rig1/rig1.done
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
6000
|
models/rig1w/rig1w.done
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
6000
|
models/rig1wg/rig1wg.done
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
8000
|
results/audit/cem_prior_summary.txt
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
units 240
|
| 2 |
+
u_max=0.6 mgl coverage: {'e_retr': 0.5371, 'e_prior': 0.5363, 'a0': 0.5237, 'b0': 0.5236} diversity: {'e_retr': 0.201, 'e_prior': 0.162, 'a0': 0.068, 'b0': 0.019}
|
| 3 |
+
e_prior - e_retr: mean diff -0.0008 wins/ties/losses 9/12/19 Wilcoxon p=0.127
|
| 4 |
+
e_prior - b0: mean diff +0.0127 wins/ties/losses 13/22/5 Wilcoxon p=0.00568
|
| 5 |
+
e_retr - a0: mean diff +0.0135 wins/ties/losses 21/16/3 Wilcoxon p=6.66e-05
|
| 6 |
+
e_retr - b0: mean diff +0.0135 wins/ties/losses 21/16/3 Wilcoxon p=5.93e-05
|
| 7 |
+
u_max=0.9 mgl coverage: {'e_retr': 0.8833, 'e_prior': 0.8863, 'a0': 0.8749, 'b0': 0.8749} diversity: {'e_retr': 0.183, 'e_prior': 0.143, 'a0': 0.068, 'b0': 0.017}
|
| 8 |
+
e_prior - e_retr: mean diff +0.0029 wins/ties/losses 10/27/3 Wilcoxon p=0.0747
|
| 9 |
+
e_prior - b0: mean diff +0.0114 wins/ties/losses 11/29/0 Wilcoxon p=0.00335
|
| 10 |
+
e_retr - a0: mean diff +0.0085 wins/ties/losses 7/32/1 Wilcoxon p=0.0173
|
| 11 |
+
e_retr - b0: mean diff +0.0085 wins/ties/losses 7/32/1 Wilcoxon p=0.0173
|
results/audit/furuta_clip_mechanism.txt
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
T cap (z=+1) = 0.643 s; fraction of bank plants with T above cap: 0.112; max bank T 1.530
|
| 2 |
+
fraction of bank plants whose trajectory has any knot with |z|>1: 0.793
|
| 3 |
+
some of 8 nearest bank plants need T above the cap: n=33 generated+P 0.152 nearest+P 0.515
|
| 4 |
+
none do: n=87 generated+P 0.793 nearest+P 0.943
|
results/audit/furuta_failure_modes.txt
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
bank minority-mode (>1 pump) fraction: 0.087
|
| 2 |
+
N plants 120 mean gen+P success 0.617 nearest+P 0.825 (K=4)
|
| 3 |
+
local neighbourhood all-majority-mode: n=75 gen+P success 0.756 nearest+P 0.947
|
| 4 |
+
locally mixed (some minority-mode neighbours): n=45 gen+P success 0.385 nearest+P 0.622
|
| 5 |
+
Spearman(local latch-time CV, gen+P success): rho -0.502 p=5.28e-09
|
| 6 |
+
Spearman(local latch-time CV, nearest+P success): rho -0.361 p=5.22e-05
|
| 7 |
+
high-dispersion half: gen+P 0.439 nearest+P 0.700 | low half: gen+P 0.794 nearest+P 0.950
|
results/audit/furuta_hybrid_union.txt
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
seed4242 K=4: nearest 0.783 generated 0.600 union(K nearest + K generated) 0.833 model-only plants 3 ceiling 0.933; nearest K/2 + gen K/2 not computable from stored prefixes
|
| 2 |
+
seed4242 K=8: nearest 0.883 generated 0.650 union(K nearest + K generated) 0.883 model-only plants 0 ceiling 0.933; nearest K/2 + gen K/2 not computable from stored prefixes
|
| 3 |
+
seed4242 K=16: nearest 0.933 generated 0.667 union(K nearest + K generated) 0.933 model-only plants 0 ceiling 0.933; nearest K/2 + gen K/2 not computable from stored prefixes
|
| 4 |
+
plants the model solves that nearest K=16 does not: 0; plants nearest K=4 misses that generated K=4 solves: 3
|
| 5 |
+
seed9001 K=4: nearest 0.867 generated 0.550 union(K nearest + K generated) 0.900 model-only plants 2 ceiling 0.933; nearest K/2 + gen K/2 not computable from stored prefixes
|
| 6 |
+
seed9001 K=8: nearest 0.917 generated 0.650 union(K nearest + K generated) 0.917 model-only plants 0 ceiling 0.933; nearest K/2 + gen K/2 not computable from stored prefixes
|
| 7 |
+
seed9001 K=16: nearest 0.933 generated 0.683 union(K nearest + K generated) 0.933 model-only plants 0 ceiling 0.933; nearest K/2 + gen K/2 not computable from stored prefixes
|
| 8 |
+
plants the model solves that nearest K=16 does not: 0; plants nearest K=4 misses that generated K=4 solves: 2
|
results/audit/furuta_modes.txt
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
N 2801 T range 0.41 1.53 T quantiles [0.44 0.48 0.52 0.57 0.73]
|
| 2 |
+
u(tau) sign changes (swing count proxy): histogram [ 0 0 2435 88 196 62 6 6 6 1 1]
|
| 3 |
+
smoothed sign-change hist [ 0 0 2556 117 81 36 9 0 1 1]
|
| 4 |
+
NN (in plant space) mean dist 0.308
|
| 5 |
+
neighbour |dT|/std(T): median 0.081 p75 0.161 p90 0.323 p99 2.903
|
| 6 |
+
frac of plants whose NN has smoothed-sign-change count differing by >=1: 0.091 ; by >=2: 0.062
|
| 7 |
+
||u_i-u_nn||: median 13.2 ; ||u_i-u_random||: median 40.4 ; ratio 0.33
|
| 8 |
+
kNN(8)-mean regressor of u: R^2 = 0.640
|
| 9 |
+
best-of-8 single neighbour trajectory distance / kNN-mean distance (median): 7.09 / 10.40
|
results/audit/guidance_B_heldout.txt
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
held-out J_bb (16 fresh ICs), means over 3 seeds, n=60 plants: {'r0': 16.75, 'rc': 16.85, 'rm': 17.19, 'p0': 18.83, 'pm': 15.49, 'ps': 17.45}
|
| 2 |
+
search J_bb: {'r0': 16.05, 'rc': 15.18, 'rm': 16.42, 'p0': 17.02, 'pm': 11.31, 'ps': 13.97}
|
| 3 |
+
search
|
| 4 |
+
pm-rm: mean -5.11 med -5.70 p=1.1e-09 (a better on 54/60)
|
| 5 |
+
pm-rc: mean -3.87 med -3.38 p=2.7e-08 (a better on 50/60)
|
| 6 |
+
pm-r0: mean -4.74 med -4.78 p=2.7e-09 (a better on 52/60)
|
| 7 |
+
ps-pm: mean +2.66 med +2.68 p=3.3e-11 (a better on 2/60)
|
| 8 |
+
p0-r0: mean +0.97 med +0.55 p=0.14 (a better on 26/60)
|
| 9 |
+
held-out
|
| 10 |
+
pm-rm: mean -1.70 med -0.96 p=0.018 (a better on 37/60)
|
| 11 |
+
pm-rc: mean -1.36 med -0.62 p=0.074 (a better on 36/60)
|
| 12 |
+
pm-r0: mean -1.26 med -0.58 p=0.087 (a better on 36/60)
|
| 13 |
+
ps-pm: mean +1.96 med +1.82 p=8.7e-06 (a better on 13/60)
|
| 14 |
+
p0-r0: mean +2.08 med +1.54 p=0.0021 (a better on 18/60)
|
results/audit/mimo_agnostic.txt
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
random bank rows K=1 cov 0.833 per-sample 0.833 median regret 13.96% mean 45.66% p90 132.94%
|
| 2 |
+
random bank rows K=8 cov 0.983 per-sample 0.869 median regret 2.66% mean 4.83% p90 9.57%
|
| 3 |
+
bank-mean controller K=1 cov 0.892 per-sample 0.892 median regret 15.50% mean 25.15% p90 56.70%
|
| 4 |
+
bank-median controller K=1 cov 0.958 per-sample 0.958 median regret 5.99% mean 10.40% p90 23.42%
|
| 5 |
+
phi=0 (no controller) K=1 cov 1.000 per-sample 1.000 median regret 1.26% mean 5.48% p90 17.72%
|
| 6 |
+
bank-mean clipped to plant cap K=1 cov 0.983 per-sample 0.983 median regret 15.22% mean 22.16% p90 53.87%
|
| 7 |
+
paired rand8 vs retrieval K=8: n=117 median diff +0.11pp mean diff +0.06pp Wilcoxon p=0.772
|
| 8 |
+
paired phi0 vs retrieval K=8: n=119 median diff +0.00pp mean diff +1.21pp Wilcoxon p=0.513
|
| 9 |
+
paired rand8 vs model K=8 (seed0): n=118 median diff +0.02pp mean diff +0.84pp Wilcoxon p=0.639
|
| 10 |
+
paired phi0 vs model K=8 (seed0): n=120 median diff -0.34pp mean diff +1.54pp Wilcoxon p=0.602
|
| 11 |
+
oracle beaten (regret<0) -- retrieval K=8 npz: 0.03333333333333333 model: 0.03333333333333333
|
results/audit/nn_distance.txt
ADDED
|
@@ -0,0 +1,52 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
== PENDULUM (standardized (l,m,b)); box l[0.5,1.5] m[0.3,1.2] b[0.02,0.3]; bank raw min/max: [0.5 0.3 0.02] [1.5 1.2 0.3]
|
| 2 |
+
unique bank rows: 2600 of 2600
|
| 3 |
+
[pend seed123] bank N=2600 dim=3 test->bank NN: mean 0.1570 med 0.1691 min 0.07331 max 0.2201 | bank LOO NN: mean 0.1424 med 0.1416 min 0.00943 ratio test/LOO(mean) 1.10
|
| 4 |
+
frac test with NN < 1e-6 (exact dup): 0.000; frac below bank-LOO 10th pct: 0.025
|
| 5 |
+
[pend seed321] bank N=2600 dim=3 test->bank NN: mean 0.1398 med 0.1312 min 0.04187 max 0.2631 | bank LOO NN: mean 0.1424 med 0.1416 min 0.00943 ratio test/LOO(mean) 0.98
|
| 6 |
+
frac test with NN < 1e-6 (exact dup): 0.000; frac below bank-LOO 10th pct: 0.050
|
| 7 |
+
[pend seed777] bank N=2600 dim=3 test->bank NN: mean 0.1333 med 0.1376 min 0.04364 max 0.2281 | bank LOO NN: mean 0.1424 med 0.1416 min 0.00943 ratio test/LOO(mean) 0.94
|
| 8 |
+
frac test with NN < 1e-6 (exact dup): 0.000; frac below bank-LOO 10th pct: 0.150
|
| 9 |
+
[pend] NN dist vs bank size: N=130:0.3998, N=260:0.3148, N=650:0.2300, N=1300:0.1780, N=2600:0.1434 | slope -0.345 -> d_eff ~ 2.9 (ambient dim 3)
|
| 10 |
+
[pend iid3000 (null of 'dense bank')] bank N=2600 dim=3 test->bank NN: mean 0.1426 med 0.1408 min 0.01501 max 0.3298 | bank LOO NN: mean 0.1424 med 0.1416 min 0.00943 ratio test/LOO(mean) 1.00
|
| 11 |
+
frac test with NN < 1e-6 (exact dup): 0.000; frac below bank-LOO 10th pct: 0.098
|
| 12 |
+
pend: NN-dist percentiles over iid test (5,50,95,99): [0.058 0.141 0.236 0.272]
|
| 13 |
+
== PENDULUM objective-cond bank (l,m,b,rho) std; rho_range (0.7, 1.0) rho min/max 0.7000994682312012 0.9999604821205139
|
| 14 |
+
cbar-cond seed321 rho0.7: test->bank NN mean 0.371
|
| 15 |
+
cbar-cond seed321 rho0.8: test->bank NN mean 0.307
|
| 16 |
+
cbar-cond seed321 rho0.9: test->bank NN mean 0.288
|
| 17 |
+
cbar-cond seed777 rho0.7: test->bank NN mean 0.372
|
| 18 |
+
cbar-cond seed777 rho0.8: test->bank NN mean 0.305
|
| 19 |
+
cbar-cond seed777 rho0.9: test->bank NN mean 0.304
|
| 20 |
+
cbar-cond bank LOO NN mean 0.31
|
| 21 |
+
== SISO bank rows 180160 unique plants 5635; G_scale 321.9
|
| 22 |
+
[siso seed0] bank N=5635 dim=10 test->bank NN: mean 0.0042 med 0.0026 min 0.00040 max 0.0267 | bank LOO NN: mean 0.0039 med 0.0023 min 0.00015 ratio test/LOO(mean) 1.08
|
| 23 |
+
frac test with NN < 1e-6 (exact dup): 0.000; frac below bank-LOO 10th pct: 0.067
|
| 24 |
+
[siso seed100] bank N=5635 dim=10 test->bank NN: mean 0.0030 med 0.0023 min 0.00027 max 0.0092 | bank LOO NN: mean 0.0039 med 0.0023 min 0.00015 ratio test/LOO(mean) 0.77
|
| 25 |
+
frac test with NN < 1e-6 (exact dup): 0.000; frac below bank-LOO 10th pct: 0.100
|
| 26 |
+
[siso seed200] bank N=5635 dim=10 test->bank NN: mean 0.0051 med 0.0024 min 0.00021 max 0.0566 | bank LOO NN: mean 0.0039 med 0.0023 min 0.00015 ratio test/LOO(mean) 1.33
|
| 27 |
+
frac test with NN < 1e-6 (exact dup): 0.000; frac below bank-LOO 10th pct: 0.117
|
| 28 |
+
[siso seed300] bank N=5635 dim=10 test->bank NN: mean 0.0077 med 0.0020 min 0.00044 max 0.1921 | bank LOO NN: mean 0.0039 med 0.0023 min 0.00015 ratio test/LOO(mean) 2.00
|
| 29 |
+
frac test with NN < 1e-6 (exact dup): 0.000; frac below bank-LOO 10th pct: 0.083
|
| 30 |
+
[siso seed400] bank N=5635 dim=10 test->bank NN: mean 0.0089 med 0.0028 min 0.00037 max 0.2771 | bank LOO NN: mean 0.0039 med 0.0023 min 0.00015 ratio test/LOO(mean) 2.29
|
| 31 |
+
frac test with NN < 1e-6 (exact dup): 0.000; frac below bank-LOO 10th pct: 0.050
|
| 32 |
+
[siso seed500] bank N=5635 dim=10 test->bank NN: mean 0.0037 med 0.0019 min 0.00020 max 0.0406 | bank LOO NN: mean 0.0039 med 0.0023 min 0.00015 ratio test/LOO(mean) 0.96
|
| 33 |
+
frac test with NN < 1e-6 (exact dup): 0.000; frac below bank-LOO 10th pct: 0.217
|
| 34 |
+
[siso seed4321] bank N=5635 dim=10 test->bank NN: mean 0.0028 med 0.0017 min 0.00023 max 0.0151 | bank LOO NN: mean 0.0039 med 0.0023 min 0.00015 ratio test/LOO(mean) 0.72
|
| 35 |
+
frac test with NN < 1e-6 (exact dup): 0.000; frac below bank-LOO 10th pct: 0.100
|
| 36 |
+
[siso] NN dist vs bank size: N=281:0.0124, N=563:0.0105, N=1408:0.0074, N=2817:0.0062, N=5635:0.0051 | slope -0.308 -> d_eff ~ 3.2 (ambient dim 10)
|
| 37 |
+
per-dim std of bank (normalized): [0. 0. 0.0015 0.0016 0.0004 0. 0.0022 0.099 0.1279 0.047 ]
|
| 38 |
+
per-dim std of held-out (normalized): [0. 0. 0.0016 0.0016 0.0005 0. 0.0022 0.1119 0.1402 0.0447]
|
| 39 |
+
frac held-out coords outside bank [min,max] per-dim: [0. 1. 0.002 0.002 0.002 1. 0. 0.005 0.002 0. ]
|
| 40 |
+
== MIMO bank 5000 dim 20, unique 5000
|
| 41 |
+
[mimo seed9999] bank N=5000 dim=20 test->bank NN: mean 0.1411 med 0.1388 min 0.08064 max 0.2122 | bank LOO NN: mean 0.1382 med 0.1359 min 0.07568 ratio test/LOO(mean) 1.02
|
| 42 |
+
frac test with NN < 1e-6 (exact dup): 0.000; frac below bank-LOO 10th pct: 0.075
|
| 43 |
+
[mimo seed4242] bank N=5000 dim=20 test->bank NN: mean 0.1342 med 0.1316 min 0.00000 max 0.2308 | bank LOO NN: mean 0.1382 med 0.1359 min 0.07568 ratio test/LOO(mean) 0.97
|
| 44 |
+
frac test with NN < 1e-6 (exact dup): 0.008; frac below bank-LOO 10th pct: 0.133
|
| 45 |
+
[mimo seed7777] bank N=5000 dim=20 test->bank NN: mean 0.1402 med 0.1391 min 0.08869 max 0.1996 | bank LOO NN: mean 0.1382 med 0.1359 min 0.07568 ratio test/LOO(mean) 1.01
|
| 46 |
+
frac test with NN < 1e-6 (exact dup): 0.000; frac below bank-LOO 10th pct: 0.083
|
| 47 |
+
[mimo] NN dist vs bank size: N=250:0.1764, N=500:0.1669, N=1250:0.1546, N=2500:0.1458, N=5000:0.1385 | slope -0.082 -> d_eff ~ 12.3 (ambient dim 20)
|
| 48 |
+
[furuta seed4242] bank N=2801 dim=4 test->bank NN: mean 0.3167 med 0.3222 min 0.07650 max 0.7878 | bank LOO NN: mean 0.3082 med 0.3104 min 0.02732 ratio test/LOO(mean) 1.03
|
| 49 |
+
frac test with NN < 1e-6 (exact dup): 0.000; frac below bank-LOO 10th pct: 0.150
|
| 50 |
+
[furuta seed9001] bank N=2801 dim=4 test->bank NN: mean 0.3149 med 0.3076 min 0.14310 max 0.5247 | bank LOO NN: mean 0.3082 med 0.3104 min 0.02732 ratio test/LOO(mean) 1.02
|
| 51 |
+
frac test with NN < 1e-6 (exact dup): 0.000; frac below bank-LOO 10th pct: 0.083
|
| 52 |
+
[furuta] NN dist vs bank size: N=140:0.6783, N=280:0.5651, N=700:0.4410, N=1400:0.3802, N=2801:0.3158 | slope -0.253 -> d_eff ~ 3.9 (ambient dim 4)
|
results/audit/pendulum_ema.txt
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
deployed raw K= 1 in-sample regret med 1.56% fresh-IC J vs retrieval: median -0.21% (p=1.0e-01)
|
| 2 |
+
deployed raw K= 8 in-sample regret med 1.12% fresh-IC J vs retrieval: median +1.94% (p=1.5e-14)
|
| 3 |
+
deployed raw K=32 in-sample regret med 0.89% fresh-IC J vs retrieval: median +1.88% (p=9.8e-15)
|
| 4 |
+
deployed EMA(avg) K= 1 in-sample regret med 1.20% fresh-IC J vs retrieval: median -0.87% (p=1.2e-03)
|
| 5 |
+
deployed EMA(avg) K= 8 in-sample regret med 0.33% fresh-IC J vs retrieval: median +0.80% (p=1.9e-09)
|
| 6 |
+
deployed EMA(avg) K=32 in-sample regret med 0.22% fresh-IC J vs retrieval: median +1.56% (p=3.0e-12)
|
| 7 |
+
size_ref_s42 raw K= 1 in-sample regret med 1.46% fresh-IC J vs retrieval: median -0.47% (p=2.3e-02)
|
| 8 |
+
size_ref_s42 raw K= 8 in-sample regret med 1.10% fresh-IC J vs retrieval: median +1.50% (p=4.4e-14)
|
| 9 |
+
size_ref_s42 raw K=32 in-sample regret med 0.89% fresh-IC J vs retrieval: median +1.60% (p=3.1e-14)
|
| 10 |
+
size_ref_s42 EMA(avg) K= 1 in-sample regret med 1.06% fresh-IC J vs retrieval: median -0.24% (p=9.3e-01)
|
| 11 |
+
size_ref_s42 EMA(avg) K= 8 in-sample regret med 0.26% fresh-IC J vs retrieval: median +1.74% (p=3.9e-14)
|
| 12 |
+
size_ref_s42 EMA(avg) K=32 in-sample regret med 0.14% fresh-IC J vs retrieval: median +2.47% (p=8.2e-15)
|
| 13 |
+
e200_s42 raw K= 1 in-sample regret med 1.33% fresh-IC J vs retrieval: median +0.11% (p=8.6e-01)
|
| 14 |
+
e200_s42 raw K= 8 in-sample regret med 0.72% fresh-IC J vs retrieval: median +1.52% (p=4.0e-12)
|
| 15 |
+
e200_s42 raw K=32 in-sample regret med 0.44% fresh-IC J vs retrieval: median +1.76% (p=1.3e-13)
|
| 16 |
+
e200_s42 EMA(avg) K= 1 in-sample regret med 8.69% fresh-IC J vs retrieval: median +6.55% (p=1.0e-10)
|
| 17 |
+
e200_s42 EMA(avg) K= 8 in-sample regret med 0.42% fresh-IC J vs retrieval: median +2.53% (p=5.7e-11)
|
| 18 |
+
e200_s42 EMA(avg) K=32 in-sample regret med 0.09% fresh-IC J vs retrieval: median +2.16% (p=3.5e-12)
|
results/audit/pendulum_eval_ckpts_relabelM4_e150.txt
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
retrieval(full bank) K= 1 cov 1.000 regret_sel med 1.64 Jfresh(selpick) med 1.1526 Jcv med 1.0808
|
| 2 |
+
retrieval(full bank) K= 2 cov 1.000 regret_sel med 1.12 Jfresh(selpick) med 1.1526 Jcv med 1.0808
|
| 3 |
+
retrieval(full bank) K= 4 cov 1.000 regret_sel med 0.56 Jfresh(selpick) med 1.1465 Jcv med 1.0758
|
| 4 |
+
retrieval(full bank) K= 8 cov 1.000 regret_sel med 0.32 Jfresh(selpick) med 1.1374 Jcv med 1.0681
|
| 5 |
+
retrieval(full bank) K=16 cov 1.000 regret_sel med 0.14 Jfresh(selpick) med 1.1374 Jcv med 1.0627
|
| 6 |
+
retrieval(full bank) K=32 cov 1.000 regret_sel med 0.10 Jfresh(selpick) med 1.1330 Jcv med 1.0620
|
| 7 |
+
relabelM4_e150 K= 1 cov 1.000 regret_sel med 0.23 Jfresh(selpick) med 1.1375 Jcv med 1.0680
|
| 8 |
+
relabelM4_e150 K= 2 cov 1.000 regret_sel med 0.13 Jfresh(selpick) med 1.1383 Jcv med 1.0671
|
| 9 |
+
relabelM4_e150 K= 4 cov 1.000 regret_sel med 0.09 Jfresh(selpick) med 1.1355 Jcv med 1.0640
|
| 10 |
+
relabelM4_e150 K= 8 cov 1.000 regret_sel med 0.07 Jfresh(selpick) med 1.1369 Jcv med 1.0640
|
| 11 |
+
relabelM4_e150 K=16 cov 1.000 regret_sel med 0.04 Jfresh(selpick) med 1.1369 Jcv med 1.0634
|
| 12 |
+
relabelM4_e150 K=32 cov 1.000 regret_sel med 0.04 Jfresh(selpick) med 1.1386 Jcv med 1.0634
|
| 13 |
+
(22s)
|
| 14 |
+
K=1 relabelM4_e150 - retrieval: in-sample regret diff med -1.44pp p=1.7e-14 | fresh-IC J rel diff med -2.81% p=1.1e-14
|
| 15 |
+
K=8 relabelM4_e150 - retrieval: in-sample regret diff med -0.14pp p=5.3e-14 | fresh-IC J rel diff med -0.17% p=6.5e-04
|
| 16 |
+
K=32 relabelM4_e150 - retrieval: in-sample regret diff med -0.04pp p=3.2e-12 | fresh-IC J rel diff med +0.06% p=2.3e-02
|
results/audit/pendulum_ksweep.txt
ADDED
|
@@ -0,0 +1,76 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
retrieval(full bank) K= 1 cov 1.000 regret_sel med 1.64 Jfresh(selpick) med 1.1526 Jcv med 1.0808
|
| 2 |
+
retrieval(full bank) K= 2 cov 1.000 regret_sel med 1.12 Jfresh(selpick) med 1.1526 Jcv med 1.0808
|
| 3 |
+
retrieval(full bank) K= 4 cov 1.000 regret_sel med 0.56 Jfresh(selpick) med 1.1465 Jcv med 1.0758
|
| 4 |
+
retrieval(full bank) K= 8 cov 1.000 regret_sel med 0.32 Jfresh(selpick) med 1.1374 Jcv med 1.0681
|
| 5 |
+
retrieval(full bank) K=16 cov 1.000 regret_sel med 0.14 Jfresh(selpick) med 1.1374 Jcv med 1.0627
|
| 6 |
+
retrieval(full bank) K=32 cov 1.000 regret_sel med 0.10 Jfresh(selpick) med 1.1330 Jcv med 1.0620
|
| 7 |
+
retrieval(full bank) K=64 cov 1.000 regret_sel med 0.05 Jfresh(selpick) med 1.1341 Jcv med 1.0621
|
| 8 |
+
retrieval(25% bank) K= 1 cov 1.000 regret_sel med 1.36 Jfresh(selpick) med 1.1662 Jcv med 1.0925
|
| 9 |
+
retrieval(25% bank) K= 2 cov 1.000 regret_sel med 0.67 Jfresh(selpick) med 1.1512 Jcv med 1.0797
|
| 10 |
+
retrieval(25% bank) K= 4 cov 1.000 regret_sel med 0.41 Jfresh(selpick) med 1.1462 Jcv med 1.0756
|
| 11 |
+
retrieval(25% bank) K= 8 cov 1.000 regret_sel med 0.22 Jfresh(selpick) med 1.1347 Jcv med 1.0660
|
| 12 |
+
retrieval(25% bank) K=16 cov 1.000 regret_sel med 0.12 Jfresh(selpick) med 1.1347 Jcv med 1.0658
|
| 13 |
+
retrieval(25% bank) K=32 cov 1.000 regret_sel med 0.05 Jfresh(selpick) med 1.1347 Jcv med 1.0633
|
| 14 |
+
retrieval(25% bank) K=64 cov 1.000 regret_sel med 0.05 Jfresh(selpick) med 1.1359 Jcv med 1.0628
|
| 15 |
+
deployed K= 1 cov 1.000 regret_sel med 1.56 Jfresh(selpick) med 1.1614 Jcv med 1.0878
|
| 16 |
+
deployed K= 2 cov 1.000 regret_sel med 1.30 Jfresh(selpick) med 1.1609 Jcv med 1.0873
|
| 17 |
+
deployed K= 4 cov 1.000 regret_sel med 1.18 Jfresh(selpick) med 1.1609 Jcv med 1.0874
|
| 18 |
+
deployed K= 8 cov 1.000 regret_sel med 1.04 Jfresh(selpick) med 1.1592 Jcv med 1.0852
|
| 19 |
+
deployed K=16 cov 1.000 regret_sel med 0.98 Jfresh(selpick) med 1.1570 Jcv med 1.0838
|
| 20 |
+
deployed K=32 cov 1.000 regret_sel med 0.91 Jfresh(selpick) med 1.1533 Jcv med 1.0807
|
| 21 |
+
deployed K=64 cov 1.000 regret_sel med 0.86 Jfresh(selpick) med 1.1533 Jcv med 1.0807
|
| 22 |
+
(49s)
|
| 23 |
+
size_ref_s42 K= 1 cov 1.000 regret_sel med 1.51 Jfresh(selpick) med 1.1546 Jcv med 1.0830
|
| 24 |
+
size_ref_s42 K= 2 cov 1.000 regret_sel med 1.23 Jfresh(selpick) med 1.1544 Jcv med 1.0827
|
| 25 |
+
size_ref_s42 K= 4 cov 1.000 regret_sel med 1.15 Jfresh(selpick) med 1.1544 Jcv med 1.0827
|
| 26 |
+
size_ref_s42 K= 8 cov 1.000 regret_sel med 1.04 Jfresh(selpick) med 1.1526 Jcv med 1.0806
|
| 27 |
+
size_ref_s42 K=16 cov 1.000 regret_sel med 0.94 Jfresh(selpick) med 1.1510 Jcv med 1.0796
|
| 28 |
+
size_ref_s42 K=32 cov 1.000 regret_sel med 0.86 Jfresh(selpick) med 1.1481 Jcv med 1.0767
|
| 29 |
+
size_ref_s42 K=64 cov 1.000 regret_sel med 0.81 Jfresh(selpick) med 1.1481 Jcv med 1.0767
|
| 30 |
+
(170s)
|
| 31 |
+
size_ref_s43 K= 1 cov 1.000 regret_sel med 0.97 Jfresh(selpick) med 1.1571 Jcv med 1.0835
|
| 32 |
+
size_ref_s43 K= 2 cov 1.000 regret_sel med 0.84 Jfresh(selpick) med 1.1550 Jcv med 1.0818
|
| 33 |
+
size_ref_s43 K= 4 cov 1.000 regret_sel med 0.76 Jfresh(selpick) med 1.1550 Jcv med 1.0818
|
| 34 |
+
size_ref_s43 K= 8 cov 1.000 regret_sel med 0.58 Jfresh(selpick) med 1.1531 Jcv med 1.0804
|
| 35 |
+
size_ref_s43 K=16 cov 1.000 regret_sel med 0.49 Jfresh(selpick) med 1.1521 Jcv med 1.0795
|
| 36 |
+
size_ref_s43 K=32 cov 1.000 regret_sel med 0.42 Jfresh(selpick) med 1.1493 Jcv med 1.0756
|
| 37 |
+
size_ref_s43 K=64 cov 1.000 regret_sel med 0.37 Jfresh(selpick) med 1.1549 Jcv med 1.0756
|
| 38 |
+
(217s)
|
| 39 |
+
e200_s42 K= 1 cov 1.000 regret_sel med 1.45 Jfresh(selpick) med 1.1624 Jcv med 1.0882
|
| 40 |
+
e200_s42 K= 2 cov 1.000 regret_sel med 1.13 Jfresh(selpick) med 1.1591 Jcv med 1.0850
|
| 41 |
+
e200_s42 K= 4 cov 1.000 regret_sel med 0.95 Jfresh(selpick) med 1.1611 Jcv med 1.0850
|
| 42 |
+
e200_s42 K= 8 cov 1.000 regret_sel med 0.69 Jfresh(selpick) med 1.1650 Jcv med 1.0825
|
| 43 |
+
e200_s42 K=16 cov 1.000 regret_sel med 0.57 Jfresh(selpick) med 1.1728 Jcv med 1.0819
|
| 44 |
+
e200_s42 K=32 cov 1.000 regret_sel med 0.49 Jfresh(selpick) med 1.1571 Jcv med 1.0751
|
| 45 |
+
e200_s42 K=64 cov 1.000 regret_sel med 0.36 Jfresh(selpick) med 1.1555 Jcv med 1.0751
|
| 46 |
+
(258s)
|
| 47 |
+
e200_s43 K= 1 cov 1.000 regret_sel med 1.01 Jfresh(selpick) med 1.1675 Jcv med 1.0891
|
| 48 |
+
e200_s43 K= 2 cov 1.000 regret_sel med 0.56 Jfresh(selpick) med 1.1536 Jcv med 1.0775
|
| 49 |
+
e200_s43 K= 4 cov 1.000 regret_sel med 0.35 Jfresh(selpick) med 1.1536 Jcv med 1.0753
|
| 50 |
+
e200_s43 K= 8 cov 1.000 regret_sel med 0.21 Jfresh(selpick) med 1.1758 Jcv med 1.0756
|
| 51 |
+
e200_s43 K=16 cov 1.000 regret_sel med 0.12 Jfresh(selpick) med 1.1758 Jcv med 1.0749
|
| 52 |
+
e200_s43 K=32 cov 1.000 regret_sel med 0.07 Jfresh(selpick) med 1.1736 Jcv med 1.0677
|
| 53 |
+
e200_s43 K=64 cov 1.000 regret_sel med 0.04 Jfresh(selpick) med 1.1856 Jcv med 1.0710
|
| 54 |
+
(299s)
|
| 55 |
+
p25_s42 K= 1 cov 1.000 regret_sel med 1.19 Jfresh(selpick) med 1.1526 Jcv med 1.0810
|
| 56 |
+
p25_s42 K= 2 cov 1.000 regret_sel med 0.96 Jfresh(selpick) med 1.1500 Jcv med 1.0775
|
| 57 |
+
p25_s42 K= 4 cov 1.000 regret_sel med 0.75 Jfresh(selpick) med 1.1485 Jcv med 1.0763
|
| 58 |
+
p25_s42 K= 8 cov 1.000 regret_sel med 0.47 Jfresh(selpick) med 1.1520 Jcv med 1.0751
|
| 59 |
+
p25_s42 K=16 cov 1.000 regret_sel med 0.36 Jfresh(selpick) med 1.1495 Jcv med 1.0749
|
| 60 |
+
p25_s42 K=32 cov 1.000 regret_sel med 0.22 Jfresh(selpick) med 1.1451 Jcv med 1.0699
|
| 61 |
+
p25_s42 K=64 cov 1.000 regret_sel med 0.18 Jfresh(selpick) med 1.1482 Jcv med 1.0696
|
| 62 |
+
(346s)
|
| 63 |
+
p25_s43 K= 1 cov 1.000 regret_sel med 0.77 Jfresh(selpick) med 1.1716 Jcv med 1.0915
|
| 64 |
+
p25_s43 K= 2 cov 1.000 regret_sel med 0.36 Jfresh(selpick) med 1.1616 Jcv med 1.0826
|
| 65 |
+
p25_s43 K= 4 cov 1.000 regret_sel med 0.24 Jfresh(selpick) med 1.1616 Jcv med 1.0826
|
| 66 |
+
p25_s43 K= 8 cov 1.000 regret_sel med 0.13 Jfresh(selpick) med 1.1684 Jcv med 1.0822
|
| 67 |
+
p25_s43 K=16 cov 1.000 regret_sel med 0.10 Jfresh(selpick) med 1.1615 Jcv med 1.0846
|
| 68 |
+
p25_s43 K=32 cov 1.000 regret_sel med 0.07 Jfresh(selpick) med 1.1640 Jcv med 1.0742
|
| 69 |
+
p25_s43 K=64 cov 1.000 regret_sel med 0.05 Jfresh(selpick) med 1.1640 Jcv med 1.0742
|
| 70 |
+
(498s)
|
| 71 |
+
K=8 deployed - retrieval: in-sample regret diff med +0.59pp p=1.1e-11 | fresh-IC J rel diff med +1.68% p=4.1e-14 | CV-selected rel diff med +1.46% p=8.8e-14
|
| 72 |
+
K=8 e200_s42 - retrieval: in-sample regret diff med +0.26pp p=3.2e-06 | fresh-IC J rel diff med +1.64% p=1.2e-11 | CV-selected rel diff med +1.05% p=2.1e-10
|
| 73 |
+
K=8 p25_s42 - retrieval: in-sample regret diff med +0.17pp p=3.9e-04 | fresh-IC J rel diff med +1.43% p=2.1e-10 | CV-selected rel diff med +0.82% p=9.4e-10
|
| 74 |
+
K=32 deployed - retrieval: in-sample regret diff med +0.52pp p=3.4e-11 | fresh-IC J rel diff med +2.00% p=1.7e-14 | CV-selected rel diff med +1.92% p=2.6e-14
|
| 75 |
+
K=32 e200_s42 - retrieval: in-sample regret diff med +0.27pp p=4.6e-08 | fresh-IC J rel diff med +2.14% p=8.0e-13 | CV-selected rel diff med +1.28% p=3.7e-13
|
| 76 |
+
K=32 p25_s42 - retrieval: in-sample regret diff med +0.05pp p=1.6e-03 | fresh-IC J rel diff med +1.53% p=6.7e-13 | CV-selected rel diff med +0.81% p=1.8e-12
|
results/audit/pendulum_labelquality.txt
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
own label stable: 1.0
|
| 2 |
+
own-label excess over best of 33 (own + 32 neighbours): median 1.14% mean 1.66% p90 4.12%
|
| 3 |
+
own-label excess over best of own+8 neighbours: median 1.00%
|
| 4 |
+
rank of own label among 33 (0=best): median 15.0 ; fraction own is the best: 0.003
|
results/audit/pendulum_ood.txt
ADDED
|
@@ -0,0 +1,24 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
== near-OOD (edge x1.25): N=24 NN-dist test->bank mean 1.429 (in-box iid null 0.143)
|
| 2 |
+
oracle finite: 24 / 24
|
| 3 |
+
K=1 model s0 : stable 1.000 cert 0.833 median regret -3.27% p90 -0.54%
|
| 4 |
+
K=1 model s1 : stable 1.000 cert 0.833 median regret -4.24% p90 -2.01%
|
| 5 |
+
K=1 model s2 : stable 1.000 cert 0.833 median regret -4.03% p90 -1.41%
|
| 6 |
+
K=1 retrieval: stable 1.000 cert 1.000 median regret -4.95% p90 -1.71%
|
| 7 |
+
K=1 paired Wilcoxon model(s0) vs retrieval on common: n=24 p=0.0491; median diff +0.58pp
|
| 8 |
+
K=8 model s0 : stable 1.000 cert 1.000 median regret -5.09% p90 -2.94%
|
| 9 |
+
K=8 model s1 : stable 1.000 cert 0.958 median regret -5.35% p90 -2.60%
|
| 10 |
+
K=8 model s2 : stable 1.000 cert 1.000 median regret -5.42% p90 -3.14%
|
| 11 |
+
K=8 retrieval: stable 1.000 cert 1.000 median regret -5.85% p90 -4.72%
|
| 12 |
+
K=8 paired Wilcoxon model(s0) vs retrieval on common: n=24 p=0.0059; median diff +0.86pp
|
| 13 |
+
== far-OOD (edge x1.67): N=24 NN-dist test->bank mean 3.940 (in-box iid null 0.143)
|
| 14 |
+
oracle finite: 24 / 24
|
| 15 |
+
K=1 model s0 : stable 0.958 cert 0.000 median regret 0.38% p90 44.23%
|
| 16 |
+
K=1 model s1 : stable 0.917 cert 0.000 median regret -0.67% p90 15.96%
|
| 17 |
+
K=1 model s2 : stable 0.833 cert 0.000 median regret 1.51% p90 27.89%
|
| 18 |
+
K=1 retrieval: stable 0.750 cert 0.625 median regret 18.48% p90 111.50%
|
| 19 |
+
K=1 paired Wilcoxon model(s0) vs retrieval on common: n=18 p=7.63e-06; median diff -21.03pp
|
| 20 |
+
K=8 model s0 : stable 1.000 cert 0.042 median regret -6.71% p90 25.09%
|
| 21 |
+
K=8 model s1 : stable 1.000 cert 0.000 median regret -6.14% p90 12.72%
|
| 22 |
+
K=8 model s2 : stable 1.000 cert 0.042 median regret -6.90% p90 6.89%
|
| 23 |
+
K=8 retrieval: stable 0.750 cert 0.625 median regret 18.48% p90 111.50%
|
| 24 |
+
K=8 paired Wilcoxon model(s0) vs retrieval on common: n=18 p=7.63e-06; median diff -26.63pp
|
results/audit/pendulum_relabel_retrieval.txt
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
retrieval(orig bank) K= 1 cov 1.000 regret_sel med 1.64 Jfresh(selpick) med 1.1526 Jcv med 1.0808
|
| 2 |
+
retrieval(orig bank) K= 2 cov 1.000 regret_sel med 1.12 Jfresh(selpick) med 1.1526 Jcv med 1.0808
|
| 3 |
+
retrieval(orig bank) K= 4 cov 1.000 regret_sel med 0.56 Jfresh(selpick) med 1.1465 Jcv med 1.0758
|
| 4 |
+
retrieval(orig bank) K= 8 cov 1.000 regret_sel med 0.32 Jfresh(selpick) med 1.1374 Jcv med 1.0681
|
| 5 |
+
retrieval(orig bank) K=16 cov 1.000 regret_sel med 0.14 Jfresh(selpick) med 1.1374 Jcv med 1.0627
|
| 6 |
+
retrieval(orig bank) K=32 cov 1.000 regret_sel med 0.10 Jfresh(selpick) med 1.1330 Jcv med 1.0620
|
| 7 |
+
retrieval(relabelled bank) K= 1 cov 1.000 regret_sel med 0.25 Jfresh(selpick) med 1.1302 Jcv med 1.0627
|
| 8 |
+
retrieval(relabelled bank) K= 2 cov 1.000 regret_sel med 0.20 Jfresh(selpick) med 1.1306 Jcv med 1.0627
|
| 9 |
+
retrieval(relabelled bank) K= 4 cov 1.000 regret_sel med 0.17 Jfresh(selpick) med 1.1317 Jcv med 1.0623
|
| 10 |
+
retrieval(relabelled bank) K= 8 cov 1.000 regret_sel med 0.10 Jfresh(selpick) med 1.1356 Jcv med 1.0623
|
| 11 |
+
retrieval(relabelled bank) K=16 cov 1.000 regret_sel med 0.10 Jfresh(selpick) med 1.1381 Jcv med 1.0621
|
| 12 |
+
retrieval(relabelled bank) K=32 cov 1.000 regret_sel med 0.05 Jfresh(selpick) med 1.1365 Jcv med 1.0615
|
| 13 |
+
K=1 relabelled-bank retrieval - original-bank retrieval: fresh-IC J rel diff median -2.38% p=8.5e-15
|
| 14 |
+
K=8 relabelled-bank retrieval - original-bank retrieval: fresh-IC J rel diff median -0.23% p=4.1e-06
|
| 15 |
+
K=32 relabelled-bank retrieval - original-bank retrieval: fresh-IC J rel diff median +0.00% p=2.0e-01
|
| 16 |
+
model trained on relabelled targets K= 1 cov 1.000 regret_sel med 0.23 Jfresh(selpick) med 1.1375 Jcv med 1.0680
|
| 17 |
+
model trained on relabelled targets K= 2 cov 1.000 regret_sel med 0.13 Jfresh(selpick) med 1.1383 Jcv med 1.0671
|
| 18 |
+
model trained on relabelled targets K= 4 cov 1.000 regret_sel med 0.09 Jfresh(selpick) med 1.1355 Jcv med 1.0640
|
| 19 |
+
model trained on relabelled targets K= 8 cov 1.000 regret_sel med 0.07 Jfresh(selpick) med 1.1369 Jcv med 1.0640
|
| 20 |
+
model trained on relabelled targets K=16 cov 1.000 regret_sel med 0.04 Jfresh(selpick) med 1.1369 Jcv med 1.0634
|
| 21 |
+
model trained on relabelled targets K=32 cov 1.000 regret_sel med 0.04 Jfresh(selpick) med 1.1386 Jcv med 1.0634
|
| 22 |
+
K=1 relabel-trained model - original-bank retrieval: fresh-IC J rel diff median -2.81% p=1.1e-14; in-sample regret diff median -1.44pp
|
| 23 |
+
K=1 relabel-trained model - relabelled-bank retrieval: fresh-IC J rel diff median -0.20% p=2.7e-02; in-sample regret diff median -0.08pp
|
| 24 |
+
K=8 relabel-trained model - original-bank retrieval: fresh-IC J rel diff median -0.17% p=6.5e-04; in-sample regret diff median -0.14pp
|
| 25 |
+
K=8 relabel-trained model - relabelled-bank retrieval: fresh-IC J rel diff median +0.08% p=2.3e-02; in-sample regret diff median -0.03pp
|
| 26 |
+
K=32 relabel-trained model - original-bank retrieval: fresh-IC J rel diff median +0.06% p=2.3e-02; in-sample regret diff median -0.04pp
|
| 27 |
+
K=32 relabel-trained model - relabelled-bank retrieval: fresh-IC J rel diff median +0.06% p=2.1e-04; in-sample regret diff median -0.02pp
|
results/audit/rederive.txt
ADDED
|
@@ -0,0 +1,28 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[1] pendulum K=8 seed123: model median regret per seed [1.172 1.085 1.008] mean 1.089 | retrieval 0.396 | Wilcoxon(seed0) n,p (40, np.float64(7.880189514253289e-07))
|
| 2 |
+
cov model [1. 1. 1.] retr 1.0
|
| 3 |
+
[2] SISO prereg K=8: model 0.7733 distinct 0.75 McNemar(m-only,r-only,p) (9, 2, np.float64(0.0654296875))
|
| 4 |
+
seed 1 (12, 3, np.float64(0.03515625)) 0.78
|
| 5 |
+
seed 2 (13, 2, np.float64(0.00738525390625)) 0.787
|
| 6 |
+
K=1: 0.617 0.493 (60, 23, np.float64(5.973595638605142e-05))
|
| 7 |
+
[3] MIMO K=8: model cov [1. 0.992 1. ] retr 0.992 McNemar (1, 0, np.float64(1.0))
|
| 8 |
+
median regret model per seed [2.32 2.35 2.28] retr 2.62 Wilcoxon (119, np.float32(0.90291226))
|
| 9 |
+
MEAN regret model [3.94 3.44 3.48] retr 4.31 p90 model [9.6 8.29 8.78] retr 11.04
|
| 10 |
+
frac of plants with negative regret (model beats 'oracle') model s0: 0.033 retr: 0.033 | min regret model -29.58
|
| 11 |
+
[4] Furuta seed4242 K=1: nearest+P 0.517 generated+P 0.333 McNemar(near-only? ) (17, 6, np.float64(0.03468966484069824))
|
| 12 |
+
[4] Furuta seed4242 K=4: nearest+P 0.783 generated+P 0.600 McNemar(near-only? ) (14, 3, np.float64(0.012725830078125))
|
| 13 |
+
[4] Furuta seed4242 K=8: nearest+P 0.883 generated+P 0.650 McNemar(near-only? ) (14, 0, np.float64(0.0001220703125))
|
| 14 |
+
[4] Furuta seed4242 K=16: nearest+P 0.933 generated+P 0.667 McNemar(near-only? ) (16, 0, np.float64(3.0517578125e-05))
|
| 15 |
+
[4] Furuta seed9001 K=1: nearest+P 0.567 generated+P 0.300 McNemar(near-only? ) (21, 5, np.float64(0.002493917942047119))
|
| 16 |
+
[4] Furuta seed9001 K=4: nearest+P 0.867 generated+P 0.550 McNemar(near-only? ) (21, 2, np.float64(6.604194641113281e-05))
|
| 17 |
+
[4] Furuta seed9001 K=8: nearest+P 0.917 generated+P 0.650 McNemar(near-only? ) (16, 0, np.float64(3.0517578125e-05))
|
| 18 |
+
[4] Furuta seed9001 K=16: nearest+P 0.933 generated+P 0.683 McNemar(near-only? ) (15, 0, np.float64(6.103515625e-05))
|
| 19 |
+
[5] Furuta deploy nominal K=8 retr 0.9 K=16 0.933 NMPC 0.958 gen seeds [np.float64(0.65), np.float64(0.658), np.float64(0.65)]
|
| 20 |
+
retr vs NMPC nominal McNemar (0, 7, np.float64(0.015625))
|
| 21 |
+
fric0.6: retr 0.867 nmpc 0.958 trueID 0.992
|
| 22 |
+
fric1.6: retr 0.092 nmpc 0.642 trueID 0.583
|
| 23 |
+
fric2.5: retr 0.000 nmpc 0.292 trueID 0.250
|
| 24 |
+
[6] guidance A mu=0.6: a0=0.5237, a=0.5236, b0=0.5236, b=0.5236, c=0.5229, d=0.5236, e=0.5378, f=0.5237
|
| 25 |
+
[6] guidance A mu=0.9: a0=0.8749, a=0.8749, b0=0.8749, b=0.8749, c=0.8738, d=0.8749, e=0.8809, f=0.8749
|
| 26 |
+
[7] Witsenhausen S20 nearest: median regret -0.017% within1 90.0% (n=60; 54) gate gap 10.0
|
| 27 |
+
nearest strictly below ceiling: 58.3 pct within 1.5pct of ceiling: 91.7 pct
|
| 28 |
+
within-1pct relative to min(ceiling,nearest): 90.0
|
results/audit/siso_prereg_pooled_seeds.txt
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
K= 1: model 3-seed mean 0.622 distinct 0.493 diff +0.129 CI[+0.078,+0.181] sign-flip permutation p=0.0000
|
| 2 |
+
K= 2: model 3-seed mean 0.706 distinct 0.660 diff +0.046 CI[+0.011,+0.080] sign-flip permutation p=0.0123
|
| 3 |
+
K= 4: model 3-seed mean 0.756 distinct 0.743 diff +0.012 CI[-0.008,+0.032] sign-flip permutation p=0.2675
|
| 4 |
+
K= 8: model 3-seed mean 0.780 distinct 0.750 diff +0.030 CI[+0.009,+0.051] sign-flip permutation p=0.0064
|
| 5 |
+
K=16: model 3-seed mean 0.793 distinct 0.753 diff +0.040 CI[+0.018,+0.064] sign-flip permutation p=0.0009
|
results/audit/siso_retrieval_metric.txt
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
SISO N=300 coverage vs K (any of K candidates verified stable)
|
| 2 |
+
K model(DDIM20) retr global retr zscore retr den3 McNemar model vs zscore (m-only,r-only,p)
|
| 3 |
+
1 0.630 0.493 0.580 0.490 (41,26,p=0.0864)
|
| 4 |
+
2 0.717 0.660 0.667 0.660 (22,7,p=0.00813)
|
| 5 |
+
4 0.757 0.743 0.733 0.737 (10,3,p=0.0923)
|
| 6 |
+
8 0.767 0.750 0.747 0.747 (9,3,p=0.146)
|
| 7 |
+
16 0.780 0.753 0.760 0.750 (8,2,p=0.109)
|
| 8 |
+
32 0.797 0.757 0.767 0.753 (10,1,p=0.0117)
|
| 9 |
+
64 0.807 0.763 0.773 0.770 (11,1,p=0.00635)
|
| 10 |
+
128 - 0.790 0.780 0.780
|
results/audit/target_range_clip.txt
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
pendulum_dataset.spline: shape (2600, 11) frac|x|>1 0.0000 max|x| 1.00 frac rows with any >1: 0.000
|
| 2 |
+
pendulum_cbar_cond.spline: shape (2533, 11) frac|x|>1 0.0000 max|x| 1.00 frac rows with any >1: 0.000
|
| 3 |
+
popov_dataset_normalized_L.spline: shape (180160, 11) frac|x|>1 0.0000 max|x| 1.00 frac rows with any >1: 0.000
|
| 4 |
+
popov_dataset_normalized_L.F_vec: shape (180160, 10) frac|x|>1 0.0000 max|x| 1.00 frac rows with any >1: 0.000
|
| 5 |
+
mimo_dataset_2x2_opt.spline: shape (5000, 2, 11) frac|x|>1 0.0000 max|x| 1.00 frac rows with any >1: 0.000
|
| 6 |
+
furuta_traj_dataset_v2.spline: shape (2801, 116) frac|x|>1 0.1922 max|x| 37.41 frac rows with any >1: 0.954
|
| 7 |
+
furuta_dataset.spline: shape (1678, 4) frac|x|>1 0.2934 max|x| 2.76 frac rows with any >1: 0.672
|
| 8 |
+
furuta_handoff_dataset.spline: shape (1414, 5) frac|x|>1 0.3151 max|x| 4.81 frac rows with any >1: 0.798
|
| 9 |
+
furuta_full_dataset.spline: shape (1500, 7) frac|x|>1 0.1936 max|x| 3.85 frac rows with any >1: 0.663
|
results/audit/traj_gen_eval_v2_clip5.0.txt
ADDED
|
@@ -0,0 +1,94 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
bank data/furuta_nmpc_bank3000.npz: 2801 entries
|
| 2 |
+
model models/Best_furuta_traj_cond_v2.pth: cond_dim=4 out_dim=116 hidden=256 layers=5 device=cpu
|
| 3 |
+
[seed4242] margin=0.3: resolving P for 227 unique bank source plants
|
| 4 |
+
[seed4242] sampler seed 0: 8.9s for 60x4 candidates (0 decode-invalid of 240)
|
| 5 |
+
[seed4242] sampler seed 1: 8.9s for 60x4 candidates (0 decode-invalid of 240)
|
| 6 |
+
[seed4242] sampler seed 2: 8.9s for 60x4 candidates (0 decode-invalid of 240)
|
| 7 |
+
[seed4242] 10/60 (27s)
|
| 8 |
+
[seed4242] 20/60 (46s)
|
| 9 |
+
[seed4242] 30/60 (69s)
|
| 10 |
+
[seed4242] 40/60 (87s)
|
| 11 |
+
[seed4242] 50/60 (111s)
|
| 12 |
+
[seed4242] 60/60 (138s)
|
| 13 |
+
[seed9001] margin=0.3: resolving P for 232 unique bank source plants
|
| 14 |
+
[seed9001] sampler seed 0: 8.8s for 60x4 candidates (0 decode-invalid of 240)
|
| 15 |
+
[seed9001] sampler seed 1: 8.8s for 60x4 candidates (0 decode-invalid of 240)
|
| 16 |
+
[seed9001] sampler seed 2: 8.9s for 60x4 candidates (0 decode-invalid of 240)
|
| 17 |
+
[seed9001] 10/60 (23s)
|
| 18 |
+
[seed9001] 20/60 (46s)
|
| 19 |
+
[seed9001] 30/60 (68s)
|
| 20 |
+
[seed9001] 40/60 (91s)
|
| 21 |
+
[seed9001] 50/60 (108s)
|
| 22 |
+
[seed9001] 60/60 (135s)
|
| 23 |
+
|
| 24 |
+
=== traj_gen_eval (v2_clip5.0): generated vs retrieval vs NMPC ===
|
| 25 |
+
|
| 26 |
+
--- plant set seed4242 (seed 4242, n=60) ---
|
| 27 |
+
arm succ 95% CI medJ med regret p90 synth
|
| 28 |
+
per-plant 93% [ 87, 98] 4.08 0.00% 0.00% 3.3s
|
| 29 |
+
nearest 62% [ 50, 73] 4.15 0.36% 1.51% 272ms
|
| 30 |
+
nearest+P 78% [ 67, 88] 4.09 0.69% 2.52% 551ms
|
| 31 |
+
nearest-margin 62% [ 50, 73] 4.15 0.36% 1.51% 275ms
|
| 32 |
+
NMPC 95% [ 88, 100] 4.08 0.14% 0.24% 21ms
|
| 33 |
+
generated 50.0%(sd6.2) [ 45, 72]* 4.02 0.97%(sd0.2) 2.58%(sd0.5) 301ms
|
| 34 |
+
generated+P 68.9%(sd3.4) [ 62, 83]* 4.08 1.02%(sd0.2) 3.43%(sd0.7) 979ms
|
| 35 |
+
(* generated rows: mean (sd) over sampler seeds [0, 1, 2]; CI shown is for seed 0 only)
|
| 36 |
+
(certified hand-off rate == success rate for every arm except NMPC (a certificate is a precondition for success here); NMPC's own hand-off is n/a, it never switches to a static gain)
|
| 37 |
+
|
| 38 |
+
--- failure breakdown, primary seed 0 ---
|
| 39 |
+
per-plant NMPC never reached Omega:3 replay failed:1
|
| 40 |
+
nearest Lyapunov decrease fails at a vertex:13 rollout failed:9 true dynamics: 0.02% of the boundary has V-dot >= 0, 0.00% saturates:1
|
| 41 |
+
nearest+P rollout failed:13
|
| 42 |
+
nearest-margin Lyapunov decrease fails at a vertex:13 rollout failed:9 true dynamics: 0.02% of the boundary has V-dot >= 0, 0.00% saturates:1
|
| 43 |
+
NMPC :3
|
| 44 |
+
generated Lyapunov decrease fails at a vertex:17 rollout failed:6 peak |Kx| on Omega is 10.26 V > 10.0 V:1 peak |Kx| on Omega is 10.23 V > 10.0 V:1
|
| 45 |
+
generated+P rollout failed:16
|
| 46 |
+
|
| 47 |
+
--- paired McNemar on success (exact binomial on discordant pairs) ---
|
| 48 |
+
per-plant vs nearest beats on 20, loses on 1 p=0.0000 * cost ratio 1.004x p=0.0000
|
| 49 |
+
per-plant vs nearest+P beats on 10, loses on 1 p=0.0117 * cost ratio 1.007x p=0.0000
|
| 50 |
+
per-plant vs nearest-margin beats on 20, loses on 1 p=0.0000 * cost ratio 1.004x p=0.0000
|
| 51 |
+
per-plant vs NMPC beats on 0, loses on 1 p=1.0000 cost ratio 1.001x p=0.0002
|
| 52 |
+
per-plant vs generated beats on 21, loses on 0 p=0.0000 * cost ratio 1.006x p=0.0000
|
| 53 |
+
per-plant vs generated+P beats on 12, loses on 0 p=0.0005 * cost ratio 1.008x p=0.0000
|
| 54 |
+
nearest+P vs per-plant beats on 1, loses on 10 p=0.0117 * cost ratio 0.993x p=0.0000
|
| 55 |
+
nearest+P vs nearest beats on 10, loses on 0 p=0.0020 * cost ratio 1.000x p=0.8291
|
| 56 |
+
nearest+P vs nearest-margin beats on 10, loses on 0 p=0.0020 * cost ratio 1.000x p=0.8291
|
| 57 |
+
nearest+P vs NMPC beats on 0, loses on 10 p=0.0020 * cost ratio 0.995x p=0.0000
|
| 58 |
+
nearest+P vs generated beats on 15, loses on 3 p=0.0075 * cost ratio 1.002x p=0.7891
|
| 59 |
+
nearest+P vs generated+P beats on 9, loses on 6 p=0.6072 cost ratio 1.001x p=0.5183
|
| 60 |
+
|
| 61 |
+
--- plant set seed9001 (seed 9001, n=60) ---
|
| 62 |
+
arm succ 95% CI medJ med regret p90 synth
|
| 63 |
+
per-plant 93% [ 87, 98] 4.16 0.00% 0.00% 5.2s
|
| 64 |
+
nearest 65% [ 52, 77] 4.10 0.71% 2.11% 154ms
|
| 65 |
+
nearest+P 87% [ 78, 95] 4.14 0.59% 2.34% 276ms
|
| 66 |
+
nearest-margin 65% [ 52, 77] 4.10 0.71% 2.11% 148ms
|
| 67 |
+
NMPC 97% [ 92, 100] 4.17 0.07% 0.22% 21ms
|
| 68 |
+
generated 50.6%(sd1.6) [ 40, 63]* 4.14 0.69%(sd0.1) 2.46%(sd0.2) 310ms
|
| 69 |
+
generated+P 75.6%(sd3.9) [ 68, 88]* 4.17 0.92%(sd0.3) 3.06%(sd0.6) 719ms
|
| 70 |
+
(* generated rows: mean (sd) over sampler seeds [0, 1, 2]; CI shown is for seed 0 only)
|
| 71 |
+
(certified hand-off rate == success rate for every arm except NMPC (a certificate is a precondition for success here); NMPC's own hand-off is n/a, it never switches to a static gain)
|
| 72 |
+
|
| 73 |
+
--- failure breakdown, primary seed 0 ---
|
| 74 |
+
per-plant replay failed:2 NMPC never reached Omega:2
|
| 75 |
+
nearest Lyapunov decrease fails at a vertex:16 rollout failed:5
|
| 76 |
+
nearest+P rollout failed:8
|
| 77 |
+
nearest-margin Lyapunov decrease fails at a vertex:16 rollout failed:5
|
| 78 |
+
NMPC :2
|
| 79 |
+
generated Lyapunov decrease fails at a vertex:20 rollout failed:6 peak |Kx| on Omega is 10.02 V > 10.0 V:1 true dynamics: 0.37% of the boundary has V-dot >= 0, 0.00% saturates:1 peak |Kx| on Omega is 10.24 V > 10.0 V:1
|
| 80 |
+
generated+P rollout failed:13
|
| 81 |
+
|
| 82 |
+
--- paired McNemar on success (exact binomial on discordant pairs) ---
|
| 83 |
+
per-plant vs nearest beats on 18, loses on 1 p=0.0001 * cost ratio 1.007x p=0.0000
|
| 84 |
+
per-plant vs nearest+P beats on 5, loses on 1 p=0.2188 cost ratio 1.006x p=0.0000
|
| 85 |
+
per-plant vs nearest-margin beats on 18, loses on 1 p=0.0001 * cost ratio 1.007x p=0.0000
|
| 86 |
+
per-plant vs NMPC beats on 0, loses on 2 p=0.5000 cost ratio 1.001x p=0.0064
|
| 87 |
+
per-plant vs generated beats on 25, loses on 0 p=0.0000 * cost ratio 1.006x p=0.0000
|
| 88 |
+
per-plant vs generated+P beats on 10, loses on 1 p=0.0117 * cost ratio 1.007x p=0.0000
|
| 89 |
+
nearest+P vs per-plant beats on 1, loses on 5 p=0.2188 cost ratio 0.994x p=0.0000
|
| 90 |
+
nearest+P vs nearest beats on 13, loses on 0 p=0.0002 * cost ratio 1.000x p=0.1154
|
| 91 |
+
nearest+P vs nearest-margin beats on 13, loses on 0 p=0.0002 * cost ratio 1.000x p=0.1154
|
| 92 |
+
nearest+P vs NMPC beats on 0, loses on 6 p=0.0312 * cost ratio 0.995x p=0.0000
|
| 93 |
+
nearest+P vs generated beats on 23, loses on 2 p=0.0000 * cost ratio 1.003x p=0.1207
|
| 94 |
+
nearest+P vs generated+P beats on 10, loses on 5 p=0.3018 cost ratio 1.002x p=0.1870
|
results/audit/traj_gen_ksweep_clip5.0.txt
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
bank data/furuta_nmpc_bank3000.npz: 2801 entries
|
| 2 |
+
model models/Best_furuta_traj_cond_v2.pth: cond_dim=4 out_dim=116
|
| 3 |
+
[seed4242] per-plant ceiling: 93%
|
| 4 |
+
[seed4242] nearest+P sequences done
|
| 5 |
+
[seed4242] generated+P seed 0 sequences done (33.9s sampling for 60x16)
|
| 6 |
+
[seed4242] generated+P seed 1 sequences done (33.8s sampling for 60x16)
|
| 7 |
+
[seed4242] generated+P seed 2 sequences done (33.7s sampling for 60x16)
|
| 8 |
+
[seed9001] per-plant ceiling: 93%
|
| 9 |
+
[seed9001] nearest+P sequences done
|
| 10 |
+
[seed9001] generated+P seed 0 sequences done (33.5s sampling for 60x16)
|
| 11 |
+
[seed9001] generated+P seed 1 sequences done (33.8s sampling for 60x16)
|
| 12 |
+
[seed9001] generated+P seed 2 sequences done (34.4s sampling for 60x16)
|
| 13 |
+
|
| 14 |
+
=== candidate-budget sweep: generated+P vs nearest+P vs per-plant ceiling ===
|
| 15 |
+
|
| 16 |
+
--- plant set seed4242 (seed 4242, n=60) ---
|
| 17 |
+
K nearest+P generated+P per-plant
|
| 18 |
+
1 51.7% t= 0.29s 33.9%(sd1.6) t= 0.86s 93.3% [McNemar nearest+P vs generated+P(seed0): 18-8, p=0.076]
|
| 19 |
+
4 78.3% t= 0.54s 73.9%(sd2.8) t= 1.40s 93.3% [McNemar nearest+P vs generated+P(seed0): 9-4, p=0.267]
|
| 20 |
+
8 88.3% t= 0.54s 83.3%(sd1.4) t= 1.40s 93.3% [McNemar nearest+P vs generated+P(seed0): 5-3, p=0.727]
|
| 21 |
+
16 93.3% t= 0.54s 86.7%(sd0.0) t= 1.40s 93.3% [McNemar nearest+P vs generated+P(seed0): 5-1, p=0.219]
|
| 22 |
+
crossover K (generated+P mean >= nearest+P): never within tested range
|
| 23 |
+
|
| 24 |
+
--- plant set seed9001 (seed 9001, n=60) ---
|
| 25 |
+
K nearest+P generated+P per-plant
|
| 26 |
+
1 56.7% t= 0.29s 34.4%(sd9.8) t= 0.86s 93.3% [McNemar nearest+P vs generated+P(seed0): 20-2, p=0.000]
|
| 27 |
+
4 86.7% t= 0.45s 76.1%(sd2.8) t= 1.16s 93.3% [McNemar nearest+P vs generated+P(seed0): 11-3, p=0.057]
|
| 28 |
+
8 91.7% t= 0.45s 88.3%(sd2.4) t= 1.16s 93.3% [McNemar nearest+P vs generated+P(seed0): 5-1, p=0.219]
|
| 29 |
+
16 93.3% t= 0.45s 93.3%(sd1.4) t= 1.16s 93.3% [McNemar nearest+P vs generated+P(seed0): 1-0, p=1.000]
|
| 30 |
+
crossover K (generated+P mean >= nearest+P): never within tested range
|
results/audit/w4_model_arm_v2.txt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
W4 (v2, cell class): conditional diffusion model as arm (f), 3 seeded sampling draws, identical gate_v2 protocol (K=4, held-out 60 + benchmark point)
|
| 2 |
+
S=0: median_regret = 51.00 +- 6.91 % | within1% = 3.3 +- 1.4 % | median J = 0.56195 +- 0.02642 | bench_J = 0.454890 +- 0.056261 | p vs nearest = [0.0, 0.0, 0.0] [nearest: regret 0.11%, within1 71.7%]
|
| 3 |
+
S=20: median_regret = 2.90 +- 0.81 % | within1% = 34.4 +- 3.9 % | median J = 0.32561 +- 0.00326 | bench_J = 0.191351 +- 0.003120 | p vs nearest = [0.0, 0.0, 0.0] [nearest: regret -0.02%, within1 90.0%]
|
results/furuta_v3/gate_sidebyside.txt
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
config gen+P d4242 gen+P d9001 pooled gen raw pooled nearest+P diff pts [95% CI] McNemar s0 (n+P-only,gen-only,p)
|
| 2 |
+
v2 clipped (shipped) 63.3% 60.0% 61.7% 52.2% 82.5% -20.8 [-29.4,-12.5] 30-5 p=0.000
|
| 3 |
+
v2 clip +-5 (audit) 68.9% 75.6% 72.2% 50.3% 82.5% -10.3 [-16.9, -3.6] 19-11 p=0.200
|
| 4 |
+
v3 raw s0 (800ep) 76.7% 79.4% 78.1% 56.4% 82.5% -4.4 [-10.8, +1.9] 13-9 p=0.523
|
| 5 |
+
v3 raw s1 (800ep) 72.8% 77.2% 75.0% 54.7% 82.5% -7.5 [-13.6, -1.1] 13-7 p=0.263
|
| 6 |
+
v3 raw s0 (2400ep) 80.0% 88.3% 84.2% 59.7% 82.5% +1.7 [ -4.4, +8.6] 8-10 p=0.815
|
| 7 |
+
v3 raw s1 (2400ep) 81.7% 90.6% 86.1% 61.7% 82.5% +3.6 [ -2.5,+10.0] 7-12 p=0.359
|
| 8 |
+
v2-data clip5 raw (2400ep control) 82.8% 88.9% 85.8% 70.6% 82.5% +3.3 [ -2.5, +9.2] 5-9 p=0.424
|
| 9 |
+
v3 averaged-wts s0 (800ep) 38.3% 35.6% 36.9% 13.9% 82.5% -45.6 [-52.8,-38.3] 58-5 p=0.000
|
results/furuta_v3/traj_gen_eval_v2ctl2400_raw_clip5.txt
ADDED
|
@@ -0,0 +1,94 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
bank data/furuta_nmpc_bank3000.npz: 2801 entries
|
| 2 |
+
model models/Best_furuta_traj_cond_v2ctl_e2400.pth: cond_dim=4 out_dim=116 hidden=256 layers=5 device=cpu
|
| 3 |
+
[seed4242] margin=0.3: resolving P for 227 unique bank source plants
|
| 4 |
+
[seed4242] sampler seed 0: 9.3s for 60x4 candidates (0 decode-invalid of 240)
|
| 5 |
+
[seed4242] sampler seed 1: 9.4s for 60x4 candidates (0 decode-invalid of 240)
|
| 6 |
+
[seed4242] sampler seed 2: 9.5s for 60x4 candidates (0 decode-invalid of 240)
|
| 7 |
+
[seed4242] 10/60 (24s)
|
| 8 |
+
[seed4242] 20/60 (37s)
|
| 9 |
+
[seed4242] 30/60 (58s)
|
| 10 |
+
[seed4242] 40/60 (71s)
|
| 11 |
+
[seed4242] 50/60 (92s)
|
| 12 |
+
[seed4242] 60/60 (111s)
|
| 13 |
+
[seed9001] margin=0.3: resolving P for 232 unique bank source plants
|
| 14 |
+
[seed9001] sampler seed 0: 10.1s for 60x4 candidates (0 decode-invalid of 240)
|
| 15 |
+
[seed9001] sampler seed 1: 9.7s for 60x4 candidates (0 decode-invalid of 240)
|
| 16 |
+
[seed9001] sampler seed 2: 9.9s for 60x4 candidates (0 decode-invalid of 240)
|
| 17 |
+
[seed9001] 10/60 (29s)
|
| 18 |
+
[seed9001] 20/60 (47s)
|
| 19 |
+
[seed9001] 30/60 (76s)
|
| 20 |
+
[seed9001] 40/60 (94s)
|
| 21 |
+
[seed9001] 50/60 (118s)
|
| 22 |
+
[seed9001] 60/60 (143s)
|
| 23 |
+
|
| 24 |
+
=== traj_gen_eval (v2ctl2400_raw_clip5): generated vs retrieval vs NMPC ===
|
| 25 |
+
|
| 26 |
+
--- plant set seed4242 (seed 4242, n=60) ---
|
| 27 |
+
arm succ 95% CI medJ med regret p90 synth
|
| 28 |
+
per-plant 93% [ 87, 98] 4.08 0.00% 0.00% 4.8s
|
| 29 |
+
nearest 62% [ 50, 73] 4.15 0.36% 1.51% 285ms
|
| 30 |
+
nearest+P 78% [ 67, 88] 4.09 0.69% 2.52% 585ms
|
| 31 |
+
nearest-margin 62% [ 50, 73] 4.15 0.36% 1.51% 288ms
|
| 32 |
+
NMPC 95% [ 88, 100] 4.08 0.14% 0.24% 23ms
|
| 33 |
+
generated 68.9%(sd1.6) [ 58, 80]* 4.05 0.62%(sd0.2) 2.54%(sd0.2) 299ms
|
| 34 |
+
generated+P 82.8%(sd0.8) [ 72, 90]* 4.08 0.66%(sd0.2) 2.89%(sd0.3) 617ms
|
| 35 |
+
(* generated rows: mean (sd) over sampler seeds [0, 1, 2]; CI shown is for seed 0 only)
|
| 36 |
+
(certified hand-off rate == success rate for every arm except NMPC (a certificate is a precondition for success here); NMPC's own hand-off is n/a, it never switches to a static gain)
|
| 37 |
+
|
| 38 |
+
--- failure breakdown, primary seed 0 ---
|
| 39 |
+
per-plant NMPC never reached Omega:3 replay failed:1
|
| 40 |
+
nearest Lyapunov decrease fails at a vertex:13 rollout failed:9 true dynamics: 0.02% of the boundary has V-dot >= 0, 0.00% saturates:1
|
| 41 |
+
nearest+P rollout failed:13
|
| 42 |
+
nearest-margin Lyapunov decrease fails at a vertex:13 rollout failed:9 true dynamics: 0.02% of the boundary has V-dot >= 0, 0.00% saturates:1
|
| 43 |
+
NMPC :3
|
| 44 |
+
generated Lyapunov decrease fails at a vertex:7 rollout failed:5 peak |Kx| on Omega is 10.13 V > 10.0 V:1 peak |Kx| on Omega is 10.12 V > 10.0 V:1 peak |Kx| on Omega is 10.53 V > 10.0 V:1 peak |Kx| on Omega is 10.36 V > 10.0 V:1 peak |Kx| on Omega is 10.19 V > 10.0 V:1 peak |Kx| on Omega is 10.00 V > 10.0 V:1
|
| 45 |
+
generated+P rollout failed:11
|
| 46 |
+
|
| 47 |
+
--- paired McNemar on success (exact binomial on discordant pairs) ---
|
| 48 |
+
per-plant vs nearest beats on 20, loses on 1 p=0.0000 * cost ratio 1.004x p=0.0000
|
| 49 |
+
per-plant vs nearest+P beats on 10, loses on 1 p=0.0117 * cost ratio 1.007x p=0.0000
|
| 50 |
+
per-plant vs nearest-margin beats on 20, loses on 1 p=0.0000 * cost ratio 1.004x p=0.0000
|
| 51 |
+
per-plant vs NMPC beats on 0, loses on 1 p=1.0000 cost ratio 1.001x p=0.0002
|
| 52 |
+
per-plant vs generated beats on 14, loses on 0 p=0.0001 * cost ratio 1.008x p=0.0000
|
| 53 |
+
per-plant vs generated+P beats on 7, loses on 0 p=0.0156 * cost ratio 1.009x p=0.0000
|
| 54 |
+
nearest+P vs per-plant beats on 1, loses on 10 p=0.0117 * cost ratio 0.993x p=0.0000
|
| 55 |
+
nearest+P vs nearest beats on 10, loses on 0 p=0.0020 * cost ratio 1.000x p=0.8291
|
| 56 |
+
nearest+P vs nearest-margin beats on 10, loses on 0 p=0.0020 * cost ratio 1.000x p=0.8291
|
| 57 |
+
nearest+P vs NMPC beats on 0, loses on 10 p=0.0020 * cost ratio 0.995x p=0.0000
|
| 58 |
+
nearest+P vs generated beats on 9, loses on 4 p=0.2668 cost ratio 1.001x p=0.2675
|
| 59 |
+
nearest+P vs generated+P beats on 4, loses on 6 p=0.7539 cost ratio 1.001x p=0.2467
|
| 60 |
+
|
| 61 |
+
--- plant set seed9001 (seed 9001, n=60) ---
|
| 62 |
+
arm succ 95% CI medJ med regret p90 synth
|
| 63 |
+
per-plant 93% [ 87, 98] 4.16 0.00% 0.00% 7.9s
|
| 64 |
+
nearest 65% [ 52, 77] 4.10 0.71% 2.11% 305ms
|
| 65 |
+
nearest+P 87% [ 78, 95] 4.14 0.59% 2.34% 566ms
|
| 66 |
+
nearest-margin 65% [ 52, 77] 4.10 0.71% 2.11% 269ms
|
| 67 |
+
NMPC 97% [ 92, 100] 4.17 0.07% 0.22% 37ms
|
| 68 |
+
generated 72.2%(sd2.8) [ 57, 80]* 4.13 0.47%(sd0.1) 2.46%(sd0.7) 404ms
|
| 69 |
+
generated+P 88.9%(sd1.6) [ 82, 97]* 4.15 0.65%(sd0.2) 2.46%(sd0.2) 660ms
|
| 70 |
+
(* generated rows: mean (sd) over sampler seeds [0, 1, 2]; CI shown is for seed 0 only)
|
| 71 |
+
(certified hand-off rate == success rate for every arm except NMPC (a certificate is a precondition for success here); NMPC's own hand-off is n/a, it never switches to a static gain)
|
| 72 |
+
|
| 73 |
+
--- failure breakdown, primary seed 0 ---
|
| 74 |
+
per-plant replay failed:2 NMPC never reached Omega:2
|
| 75 |
+
nearest Lyapunov decrease fails at a vertex:16 rollout failed:5
|
| 76 |
+
nearest+P rollout failed:8
|
| 77 |
+
nearest-margin Lyapunov decrease fails at a vertex:16 rollout failed:5
|
| 78 |
+
NMPC :2
|
| 79 |
+
generated Lyapunov decrease fails at a vertex:10 rollout failed:6 peak |Kx| on Omega is 10.52 V > 10.0 V:1 peak |Kx| on Omega is 10.17 V > 10.0 V:1 peak |Kx| on Omega is 10.71 V > 10.0 V:1
|
| 80 |
+
generated+P rollout failed:6
|
| 81 |
+
|
| 82 |
+
--- paired McNemar on success (exact binomial on discordant pairs) ---
|
| 83 |
+
per-plant vs nearest beats on 18, loses on 1 p=0.0001 * cost ratio 1.007x p=0.0000
|
| 84 |
+
per-plant vs nearest+P beats on 5, loses on 1 p=0.2188 cost ratio 1.006x p=0.0000
|
| 85 |
+
per-plant vs nearest-margin beats on 18, loses on 1 p=0.0001 * cost ratio 1.007x p=0.0000
|
| 86 |
+
per-plant vs NMPC beats on 0, loses on 2 p=0.5000 cost ratio 1.001x p=0.0064
|
| 87 |
+
per-plant vs generated beats on 15, loses on 0 p=0.0001 * cost ratio 1.005x p=0.0000
|
| 88 |
+
per-plant vs generated+P beats on 2, loses on 0 p=0.5000 cost ratio 1.009x p=0.0000
|
| 89 |
+
nearest+P vs per-plant beats on 1, loses on 5 p=0.2188 cost ratio 0.994x p=0.0000
|
| 90 |
+
nearest+P vs nearest beats on 13, loses on 0 p=0.0002 * cost ratio 1.000x p=0.1154
|
| 91 |
+
nearest+P vs nearest-margin beats on 13, loses on 0 p=0.0002 * cost ratio 1.000x p=0.1154
|
| 92 |
+
nearest+P vs NMPC beats on 0, loses on 6 p=0.0312 * cost ratio 0.995x p=0.0000
|
| 93 |
+
nearest+P vs generated beats on 13, loses on 2 p=0.0074 * cost ratio 1.001x p=0.5116
|
| 94 |
+
nearest+P vs generated+P beats on 1, loses on 3 p=0.6250 cost ratio 1.002x p=0.1682
|
results/furuta_v3/traj_gen_eval_v3_ema_clip1.txt
ADDED
|
@@ -0,0 +1,94 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
bank data/furuta_nmpc_bank3000.npz: 2801 entries
|
| 2 |
+
model models/Best_furuta_traj_cond_v3.pth: cond_dim=4 out_dim=116 hidden=256 layers=5 device=cpu
|
| 3 |
+
[seed4242] margin=0.3: resolving P for 227 unique bank source plants
|
| 4 |
+
[seed4242] sampler seed 0: 9.4s for 60x4 candidates (0 decode-invalid of 240)
|
| 5 |
+
[seed4242] sampler seed 1: 9.5s for 60x4 candidates (0 decode-invalid of 240)
|
| 6 |
+
[seed4242] sampler seed 2: 9.5s for 60x4 candidates (0 decode-invalid of 240)
|
| 7 |
+
[seed4242] 10/60 (32s)
|
| 8 |
+
[seed4242] 20/60 (53s)
|
| 9 |
+
[seed4242] 30/60 (77s)
|
| 10 |
+
[seed4242] 40/60 (96s)
|
| 11 |
+
[seed4242] 50/60 (118s)
|
| 12 |
+
[seed4242] 60/60 (141s)
|
| 13 |
+
[seed9001] margin=0.3: resolving P for 232 unique bank source plants
|
| 14 |
+
[seed9001] sampler seed 0: 10.6s for 60x4 candidates (0 decode-invalid of 240)
|
| 15 |
+
[seed9001] sampler seed 1: 10.8s for 60x4 candidates (0 decode-invalid of 240)
|
| 16 |
+
[seed9001] sampler seed 2: 11.3s for 60x4 candidates (0 decode-invalid of 240)
|
| 17 |
+
[seed9001] 10/60 (39s)
|
| 18 |
+
[seed9001] 20/60 (62s)
|
| 19 |
+
[seed9001] 30/60 (89s)
|
| 20 |
+
[seed9001] 40/60 (115s)
|
| 21 |
+
[seed9001] 50/60 (139s)
|
| 22 |
+
[seed9001] 60/60 (167s)
|
| 23 |
+
|
| 24 |
+
=== traj_gen_eval (v3_ema_clip1): generated vs retrieval vs NMPC ===
|
| 25 |
+
|
| 26 |
+
--- plant set seed4242 (seed 4242, n=60) ---
|
| 27 |
+
arm succ 95% CI medJ med regret p90 synth
|
| 28 |
+
per-plant 93% [ 87, 98] 4.08 0.00% 0.00% 4.5s
|
| 29 |
+
nearest 62% [ 50, 73] 4.15 0.36% 1.51% 310ms
|
| 30 |
+
nearest+P 78% [ 67, 88] 4.09 0.69% 2.52% 637ms
|
| 31 |
+
nearest-margin 62% [ 50, 73] 4.15 0.36% 1.51% 310ms
|
| 32 |
+
NMPC 95% [ 88, 100] 4.08 0.14% 0.24% 25ms
|
| 33 |
+
generated 18.9%(sd3.4) [ 8, 28]* 4.00 2.04%(sd0.6) 28.63%(sd22.1) 422ms
|
| 34 |
+
generated+P 38.3%(sd2.7) [ 23, 47]* 4.15 1.97%(sd0.6) 45.55%(sd28.3) 1.7s
|
| 35 |
+
(* generated rows: mean (sd) over sampler seeds [0, 1, 2]; CI shown is for seed 0 only)
|
| 36 |
+
(certified hand-off rate == success rate for every arm except NMPC (a certificate is a precondition for success here); NMPC's own hand-off is n/a, it never switches to a static gain)
|
| 37 |
+
|
| 38 |
+
--- failure breakdown, primary seed 0 ---
|
| 39 |
+
per-plant NMPC never reached Omega:3 replay failed:1
|
| 40 |
+
nearest Lyapunov decrease fails at a vertex:13 rollout failed:9 true dynamics: 0.02% of the boundary has V-dot >= 0, 0.00% saturates:1
|
| 41 |
+
nearest+P rollout failed:13
|
| 42 |
+
nearest-margin Lyapunov decrease fails at a vertex:13 rollout failed:9 true dynamics: 0.02% of the boundary has V-dot >= 0, 0.00% saturates:1
|
| 43 |
+
NMPC :3
|
| 44 |
+
generated Lyapunov decrease fails at a vertex:29 rollout failed:19 peak |Kx| on Omega is 10.47 V > 10.0 V:1
|
| 45 |
+
generated+P rollout failed:39
|
| 46 |
+
|
| 47 |
+
--- paired McNemar on success (exact binomial on discordant pairs) ---
|
| 48 |
+
per-plant vs nearest beats on 20, loses on 1 p=0.0000 * cost ratio 1.004x p=0.0000
|
| 49 |
+
per-plant vs nearest+P beats on 10, loses on 1 p=0.0117 * cost ratio 1.007x p=0.0000
|
| 50 |
+
per-plant vs nearest-margin beats on 20, loses on 1 p=0.0000 * cost ratio 1.004x p=0.0000
|
| 51 |
+
per-plant vs NMPC beats on 0, loses on 1 p=1.0000 cost ratio 1.001x p=0.0002
|
| 52 |
+
per-plant vs generated beats on 45, loses on 0 p=0.0000 * cost ratio 1.028x p=0.0010
|
| 53 |
+
per-plant vs generated+P beats on 35, loses on 0 p=0.0000 * cost ratio 1.029x p=0.0000
|
| 54 |
+
nearest+P vs per-plant beats on 1, loses on 10 p=0.0117 * cost ratio 0.993x p=0.0000
|
| 55 |
+
nearest+P vs nearest beats on 10, loses on 0 p=0.0020 * cost ratio 1.000x p=0.8291
|
| 56 |
+
nearest+P vs nearest-margin beats on 10, loses on 0 p=0.0020 * cost ratio 1.000x p=0.8291
|
| 57 |
+
nearest+P vs NMPC beats on 0, loses on 10 p=0.0020 * cost ratio 0.995x p=0.0000
|
| 58 |
+
nearest+P vs generated beats on 38, loses on 2 p=0.0000 * cost ratio 1.008x p=0.0547
|
| 59 |
+
nearest+P vs generated+P beats on 30, loses on 4 p=0.0000 * cost ratio 1.013x p=0.0005
|
| 60 |
+
|
| 61 |
+
--- plant set seed9001 (seed 9001, n=60) ---
|
| 62 |
+
arm succ 95% CI medJ med regret p90 synth
|
| 63 |
+
per-plant 93% [ 87, 98] 4.16 0.00% 0.00% 7.6s
|
| 64 |
+
nearest 65% [ 52, 77] 4.10 0.71% 2.11% 226ms
|
| 65 |
+
nearest+P 87% [ 78, 95] 4.14 0.59% 2.34% 524ms
|
| 66 |
+
nearest-margin 65% [ 52, 77] 4.10 0.71% 2.11% 244ms
|
| 67 |
+
NMPC 97% [ 92, 100] 4.17 0.07% 0.22% 30ms
|
| 68 |
+
generated 8.9%(sd0.8) [ 2, 15]* 4.15 1.40%(sd0.6) 2.81%(sd1.2) 562ms
|
| 69 |
+
generated+P 35.6%(sd4.4) [ 30, 55]* 4.18 1.98%(sd0.2) 4.47%(sd0.8) 2.0s
|
| 70 |
+
(* generated rows: mean (sd) over sampler seeds [0, 1, 2]; CI shown is for seed 0 only)
|
| 71 |
+
(certified hand-off rate == success rate for every arm except NMPC (a certificate is a precondition for success here); NMPC's own hand-off is n/a, it never switches to a static gain)
|
| 72 |
+
|
| 73 |
+
--- failure breakdown, primary seed 0 ---
|
| 74 |
+
per-plant replay failed:2 NMPC never reached Omega:2
|
| 75 |
+
nearest Lyapunov decrease fails at a vertex:16 rollout failed:5
|
| 76 |
+
nearest+P rollout failed:8
|
| 77 |
+
nearest-margin Lyapunov decrease fails at a vertex:16 rollout failed:5
|
| 78 |
+
NMPC :2
|
| 79 |
+
generated Lyapunov decrease fails at a vertex:31 rollout failed:23 peak |Kx| on Omega is 10.04 V > 10.0 V:1
|
| 80 |
+
generated+P rollout failed:35
|
| 81 |
+
|
| 82 |
+
--- paired McNemar on success (exact binomial on discordant pairs) ---
|
| 83 |
+
per-plant vs nearest beats on 18, loses on 1 p=0.0001 * cost ratio 1.007x p=0.0000
|
| 84 |
+
per-plant vs nearest+P beats on 5, loses on 1 p=0.2188 cost ratio 1.006x p=0.0000
|
| 85 |
+
per-plant vs nearest-margin beats on 18, loses on 1 p=0.0001 * cost ratio 1.007x p=0.0000
|
| 86 |
+
per-plant vs NMPC beats on 0, loses on 2 p=0.5000 cost ratio 1.001x p=0.0064
|
| 87 |
+
per-plant vs generated beats on 51, loses on 0 p=0.0000 * cost ratio nanx p=nan
|
| 88 |
+
per-plant vs generated+P beats on 31, loses on 0 p=0.0000 * cost ratio 1.022x p=0.0000
|
| 89 |
+
nearest+P vs per-plant beats on 1, loses on 5 p=0.2188 cost ratio 0.994x p=0.0000
|
| 90 |
+
nearest+P vs nearest beats on 13, loses on 0 p=0.0002 * cost ratio 1.000x p=0.1154
|
| 91 |
+
nearest+P vs nearest-margin beats on 13, loses on 0 p=0.0002 * cost ratio 1.000x p=0.1154
|
| 92 |
+
nearest+P vs NMPC beats on 0, loses on 6 p=0.0312 * cost ratio 0.995x p=0.0000
|
| 93 |
+
nearest+P vs generated beats on 48, loses on 1 p=0.0000 * cost ratio nanx p=nan
|
| 94 |
+
nearest+P vs generated+P beats on 28, loses on 1 p=0.0000 * cost ratio 1.018x p=0.0001
|
results/furuta_v3/traj_gen_eval_v3e2400_raw_clip1.txt
ADDED
|
@@ -0,0 +1,94 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
bank data/furuta_nmpc_bank3000.npz: 2801 entries
|
| 2 |
+
model models/Best_furuta_traj_cond_v3_e2400.pth: cond_dim=4 out_dim=116 hidden=256 layers=5 device=cpu
|
| 3 |
+
[seed4242] margin=0.3: resolving P for 227 unique bank source plants
|
| 4 |
+
[seed4242] sampler seed 0: 11.5s for 60x4 candidates (0 decode-invalid of 240)
|
| 5 |
+
[seed4242] sampler seed 1: 10.2s for 60x4 candidates (0 decode-invalid of 240)
|
| 6 |
+
[seed4242] sampler seed 2: 10.2s for 60x4 candidates (0 decode-invalid of 240)
|
| 7 |
+
[seed4242] 10/60 (31s)
|
| 8 |
+
[seed4242] 20/60 (47s)
|
| 9 |
+
[seed4242] 30/60 (66s)
|
| 10 |
+
[seed4242] 40/60 (83s)
|
| 11 |
+
[seed4242] 50/60 (105s)
|
| 12 |
+
[seed4242] 60/60 (128s)
|
| 13 |
+
[seed9001] margin=0.3: resolving P for 232 unique bank source plants
|
| 14 |
+
[seed9001] sampler seed 0: 9.4s for 60x4 candidates (0 decode-invalid of 240)
|
| 15 |
+
[seed9001] sampler seed 1: 9.6s for 60x4 candidates (0 decode-invalid of 240)
|
| 16 |
+
[seed9001] sampler seed 2: 9.4s for 60x4 candidates (0 decode-invalid of 240)
|
| 17 |
+
[seed9001] 10/60 (22s)
|
| 18 |
+
[seed9001] 20/60 (35s)
|
| 19 |
+
[seed9001] 30/60 (55s)
|
| 20 |
+
[seed9001] 40/60 (67s)
|
| 21 |
+
[seed9001] 50/60 (85s)
|
| 22 |
+
[seed9001] 60/60 (110s)
|
| 23 |
+
|
| 24 |
+
=== traj_gen_eval (v3e2400_raw_clip1): generated vs retrieval vs NMPC ===
|
| 25 |
+
|
| 26 |
+
--- plant set seed4242 (seed 4242, n=60) ---
|
| 27 |
+
arm succ 95% CI medJ med regret p90 synth
|
| 28 |
+
per-plant 93% [ 87, 98] 4.08 0.00% 0.00% 5.1s
|
| 29 |
+
nearest 62% [ 50, 73] 4.15 0.36% 1.51% 317ms
|
| 30 |
+
nearest+P 78% [ 67, 88] 4.09 0.69% 2.52% 729ms
|
| 31 |
+
nearest-margin 62% [ 50, 73] 4.15 0.36% 1.51% 291ms
|
| 32 |
+
NMPC 95% [ 88, 100] 4.08 0.14% 0.24% 25ms
|
| 33 |
+
generated 55.6%(sd4.8) [ 37, 63]* 4.07 0.88%(sd0.2) 2.70%(sd0.4) 340ms
|
| 34 |
+
generated+P 80.0%(sd1.4) [ 68, 88]* 4.09 0.85%(sd0.1) 2.52%(sd0.6) 671ms
|
| 35 |
+
(* generated rows: mean (sd) over sampler seeds [0, 1, 2]; CI shown is for seed 0 only)
|
| 36 |
+
(certified hand-off rate == success rate for every arm except NMPC (a certificate is a precondition for success here); NMPC's own hand-off is n/a, it never switches to a static gain)
|
| 37 |
+
|
| 38 |
+
--- failure breakdown, primary seed 0 ---
|
| 39 |
+
per-plant NMPC never reached Omega:3 replay failed:1
|
| 40 |
+
nearest Lyapunov decrease fails at a vertex:13 rollout failed:9 true dynamics: 0.02% of the boundary has V-dot >= 0, 0.00% saturates:1
|
| 41 |
+
nearest+P rollout failed:13
|
| 42 |
+
nearest-margin Lyapunov decrease fails at a vertex:13 rollout failed:9 true dynamics: 0.02% of the boundary has V-dot >= 0, 0.00% saturates:1
|
| 43 |
+
NMPC :3
|
| 44 |
+
generated Lyapunov decrease fails at a vertex:18 rollout failed:6 peak |Kx| on Omega is 10.26 V > 10.0 V:1 peak |Kx| on Omega is 10.18 V > 10.0 V:1 peak |Kx| on Omega is 10.04 V > 10.0 V:1 peak |Kx| on Omega is 10.25 V > 10.0 V:1 peak |Kx| on Omega is 10.10 V > 10.0 V:1 peak |Kx| on Omega is 10.02 V > 10.0 V:1
|
| 45 |
+
generated+P rollout failed:13
|
| 46 |
+
|
| 47 |
+
--- paired McNemar on success (exact binomial on discordant pairs) ---
|
| 48 |
+
per-plant vs nearest beats on 20, loses on 1 p=0.0000 * cost ratio 1.004x p=0.0000
|
| 49 |
+
per-plant vs nearest+P beats on 10, loses on 1 p=0.0117 * cost ratio 1.007x p=0.0000
|
| 50 |
+
per-plant vs nearest-margin beats on 20, loses on 1 p=0.0000 * cost ratio 1.004x p=0.0000
|
| 51 |
+
per-plant vs NMPC beats on 0, loses on 1 p=1.0000 cost ratio 1.001x p=0.0002
|
| 52 |
+
per-plant vs generated beats on 26, loses on 0 p=0.0000 * cost ratio 1.007x p=0.0000
|
| 53 |
+
per-plant vs generated+P beats on 9, loses on 0 p=0.0039 * cost ratio 1.007x p=0.0000
|
| 54 |
+
nearest+P vs per-plant beats on 1, loses on 10 p=0.0117 * cost ratio 0.993x p=0.0000
|
| 55 |
+
nearest+P vs nearest beats on 10, loses on 0 p=0.0020 * cost ratio 1.000x p=0.8291
|
| 56 |
+
nearest+P vs nearest-margin beats on 10, loses on 0 p=0.0020 * cost ratio 1.000x p=0.8291
|
| 57 |
+
nearest+P vs NMPC beats on 0, loses on 10 p=0.0020 * cost ratio 0.995x p=0.0000
|
| 58 |
+
nearest+P vs generated beats on 19, loses on 2 p=0.0002 * cost ratio 1.002x p=0.4515
|
| 59 |
+
nearest+P vs generated+P beats on 6, loses on 6 p=1.0000 cost ratio 1.002x p=0.2411
|
| 60 |
+
|
| 61 |
+
--- plant set seed9001 (seed 9001, n=60) ---
|
| 62 |
+
arm succ 95% CI medJ med regret p90 synth
|
| 63 |
+
per-plant 93% [ 87, 98] 4.16 0.00% 0.00% 5.6s
|
| 64 |
+
nearest 65% [ 52, 77] 4.10 0.71% 2.11% 163ms
|
| 65 |
+
nearest+P 87% [ 78, 95] 4.14 0.59% 2.34% 423ms
|
| 66 |
+
nearest-margin 65% [ 52, 77] 4.10 0.71% 2.11% 163ms
|
| 67 |
+
NMPC 97% [ 92, 100] 4.17 0.07% 0.22% 22ms
|
| 68 |
+
generated 63.9%(sd2.8) [ 48, 72]* 4.21 0.82%(sd0.0) 3.52%(sd0.1) 307ms
|
| 69 |
+
generated+P 88.3%(sd2.4) [ 82, 97]* 4.20 0.83%(sd0.1) 3.50%(sd0.4) 615ms
|
| 70 |
+
(* generated rows: mean (sd) over sampler seeds [0, 1, 2]; CI shown is for seed 0 only)
|
| 71 |
+
(certified hand-off rate == success rate for every arm except NMPC (a certificate is a precondition for success here); NMPC's own hand-off is n/a, it never switches to a static gain)
|
| 72 |
+
|
| 73 |
+
--- failure breakdown, primary seed 0 ---
|
| 74 |
+
per-plant replay failed:2 NMPC never reached Omega:2
|
| 75 |
+
nearest Lyapunov decrease fails at a vertex:16 rollout failed:5
|
| 76 |
+
nearest+P rollout failed:8
|
| 77 |
+
nearest-margin Lyapunov decrease fails at a vertex:16 rollout failed:5
|
| 78 |
+
NMPC :2
|
| 79 |
+
generated Lyapunov decrease fails at a vertex:12 rollout failed:4 peak |Kx| on Omega is 10.10 V > 10.0 V:1 true dynamics: 0.38% of the boundary has V-dot >= 0, 0.00% saturates:1 peak |Kx| on Omega is 10.35 V > 10.0 V:1 peak |Kx| on Omega is 10.11 V > 10.0 V:1 peak |Kx| on Omega is 10.04 V > 10.0 V:1 true dynamics: 0.20% of the boundary has V-dot >= 0, 0.00% saturates:1 true dynamics: 0.03% of the boundary has V-dot >= 0, 0.00% saturates:1 peak |Kx| on Omega is 10.76 V > 10.0 V:1
|
| 80 |
+
generated+P rollout failed:6
|
| 81 |
+
|
| 82 |
+
--- paired McNemar on success (exact binomial on discordant pairs) ---
|
| 83 |
+
per-plant vs nearest beats on 18, loses on 1 p=0.0001 * cost ratio 1.007x p=0.0000
|
| 84 |
+
per-plant vs nearest+P beats on 5, loses on 1 p=0.2188 cost ratio 1.006x p=0.0000
|
| 85 |
+
per-plant vs nearest-margin beats on 18, loses on 1 p=0.0001 * cost ratio 1.007x p=0.0000
|
| 86 |
+
per-plant vs NMPC beats on 0, loses on 2 p=0.5000 cost ratio 1.001x p=0.0064
|
| 87 |
+
per-plant vs generated beats on 20, loses on 0 p=0.0000 * cost ratio 1.009x p=0.0000
|
| 88 |
+
per-plant vs generated+P beats on 2, loses on 0 p=0.5000 cost ratio 1.009x p=0.0000
|
| 89 |
+
nearest+P vs per-plant beats on 1, loses on 5 p=0.2188 cost ratio 0.994x p=0.0000
|
| 90 |
+
nearest+P vs nearest beats on 13, loses on 0 p=0.0002 * cost ratio 1.000x p=0.1154
|
| 91 |
+
nearest+P vs nearest-margin beats on 13, loses on 0 p=0.0002 * cost ratio 1.000x p=0.1154
|
| 92 |
+
nearest+P vs NMPC beats on 0, loses on 6 p=0.0312 * cost ratio 0.995x p=0.0000
|
| 93 |
+
nearest+P vs generated beats on 18, loses on 2 p=0.0004 * cost ratio 1.005x p=0.0036
|
| 94 |
+
nearest+P vs generated+P beats on 2, loses on 4 p=0.6875 cost ratio 1.003x p=0.0005
|
results/furuta_v3/traj_gen_eval_v3s1_raw_clip1.txt
ADDED
|
@@ -0,0 +1,94 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
bank data/furuta_nmpc_bank3000.npz: 2801 entries
|
| 2 |
+
model models/Best_furuta_traj_cond_v3_s1.pth: cond_dim=4 out_dim=116 hidden=256 layers=5 device=cpu
|
| 3 |
+
[seed4242] margin=0.3: resolving P for 227 unique bank source plants
|
| 4 |
+
[seed4242] sampler seed 0: 9.7s for 60x4 candidates (0 decode-invalid of 240)
|
| 5 |
+
[seed4242] sampler seed 1: 9.7s for 60x4 candidates (0 decode-invalid of 240)
|
| 6 |
+
[seed4242] sampler seed 2: 9.5s for 60x4 candidates (0 decode-invalid of 240)
|
| 7 |
+
[seed4242] 10/60 (29s)
|
| 8 |
+
[seed4242] 20/60 (47s)
|
| 9 |
+
[seed4242] 30/60 (67s)
|
| 10 |
+
[seed4242] 40/60 (84s)
|
| 11 |
+
[seed4242] 50/60 (102s)
|
| 12 |
+
[seed4242] 60/60 (122s)
|
| 13 |
+
[seed9001] margin=0.3: resolving P for 232 unique bank source plants
|
| 14 |
+
[seed9001] sampler seed 0: 9.4s for 60x4 candidates (0 decode-invalid of 240)
|
| 15 |
+
[seed9001] sampler seed 1: 9.3s for 60x4 candidates (0 decode-invalid of 240)
|
| 16 |
+
[seed9001] sampler seed 2: 9.4s for 60x4 candidates (0 decode-invalid of 240)
|
| 17 |
+
[seed9001] 10/60 (31s)
|
| 18 |
+
[seed9001] 20/60 (49s)
|
| 19 |
+
[seed9001] 30/60 (70s)
|
| 20 |
+
[seed9001] 40/60 (88s)
|
| 21 |
+
[seed9001] 50/60 (110s)
|
| 22 |
+
[seed9001] 60/60 (148s)
|
| 23 |
+
|
| 24 |
+
=== traj_gen_eval (v3s1_raw_clip1): generated vs retrieval vs NMPC ===
|
| 25 |
+
|
| 26 |
+
--- plant set seed4242 (seed 4242, n=60) ---
|
| 27 |
+
arm succ 95% CI medJ med regret p90 synth
|
| 28 |
+
per-plant 93% [ 87, 98] 4.08 0.00% 0.00% 3.9s
|
| 29 |
+
nearest 62% [ 50, 73] 4.15 0.36% 1.51% 321ms
|
| 30 |
+
nearest+P 78% [ 67, 88] 4.09 0.69% 2.52% 671ms
|
| 31 |
+
nearest-margin 62% [ 50, 73] 4.15 0.36% 1.51% 316ms
|
| 32 |
+
NMPC 95% [ 88, 100] 4.08 0.14% 0.24% 25ms
|
| 33 |
+
generated 52.8%(sd0.8) [ 40, 67]* 4.07 0.94%(sd0.1) 3.18%(sd0.2) 354ms
|
| 34 |
+
generated+P 72.8%(sd2.1) [ 63, 85]* 4.07 0.96%(sd0.1) 3.13%(sd0.2) 1.0s
|
| 35 |
+
(* generated rows: mean (sd) over sampler seeds [0, 1, 2]; CI shown is for seed 0 only)
|
| 36 |
+
(certified hand-off rate == success rate for every arm except NMPC (a certificate is a precondition for success here); NMPC's own hand-off is n/a, it never switches to a static gain)
|
| 37 |
+
|
| 38 |
+
--- failure breakdown, primary seed 0 ---
|
| 39 |
+
per-plant NMPC never reached Omega:3 replay failed:1
|
| 40 |
+
nearest Lyapunov decrease fails at a vertex:13 rollout failed:9 true dynamics: 0.02% of the boundary has V-dot >= 0, 0.00% saturates:1
|
| 41 |
+
nearest+P rollout failed:13
|
| 42 |
+
nearest-margin Lyapunov decrease fails at a vertex:13 rollout failed:9 true dynamics: 0.02% of the boundary has V-dot >= 0, 0.00% saturates:1
|
| 43 |
+
NMPC :3
|
| 44 |
+
generated rollout failed:12 Lyapunov decrease fails at a vertex:12 peak |Kx| on Omega is 10.15 V > 10.0 V:1 peak |Kx| on Omega is 10.43 V > 10.0 V:1 peak |Kx| on Omega is 10.07 V > 10.0 V:1 peak |Kx| on Omega is 10.11 V > 10.0 V:1
|
| 45 |
+
generated+P rollout failed:15
|
| 46 |
+
|
| 47 |
+
--- paired McNemar on success (exact binomial on discordant pairs) ---
|
| 48 |
+
per-plant vs nearest beats on 20, loses on 1 p=0.0000 * cost ratio 1.004x p=0.0000
|
| 49 |
+
per-plant vs nearest+P beats on 10, loses on 1 p=0.0117 * cost ratio 1.007x p=0.0000
|
| 50 |
+
per-plant vs nearest-margin beats on 20, loses on 1 p=0.0000 * cost ratio 1.004x p=0.0000
|
| 51 |
+
per-plant vs NMPC beats on 0, loses on 1 p=1.0000 cost ratio 1.001x p=0.0002
|
| 52 |
+
per-plant vs generated beats on 24, loses on 0 p=0.0000 * cost ratio 1.010x p=0.0000
|
| 53 |
+
per-plant vs generated+P beats on 11, loses on 0 p=0.0010 * cost ratio 1.010x p=0.0000
|
| 54 |
+
nearest+P vs per-plant beats on 1, loses on 10 p=0.0117 * cost ratio 0.993x p=0.0000
|
| 55 |
+
nearest+P vs nearest beats on 10, loses on 0 p=0.0020 * cost ratio 1.000x p=0.8291
|
| 56 |
+
nearest+P vs nearest-margin beats on 10, loses on 0 p=0.0020 * cost ratio 1.000x p=0.8291
|
| 57 |
+
nearest+P vs NMPC beats on 0, loses on 10 p=0.0020 * cost ratio 0.995x p=0.0000
|
| 58 |
+
nearest+P vs generated beats on 18, loses on 3 p=0.0015 * cost ratio 1.005x p=0.1316
|
| 59 |
+
nearest+P vs generated+P beats on 6, loses on 4 p=0.7539 cost ratio 1.004x p=0.0562
|
| 60 |
+
|
| 61 |
+
--- plant set seed9001 (seed 9001, n=60) ---
|
| 62 |
+
arm succ 95% CI medJ med regret p90 synth
|
| 63 |
+
per-plant 93% [ 87, 98] 4.16 0.00% 0.00% 7.0s
|
| 64 |
+
nearest 65% [ 52, 77] 4.10 0.71% 2.11% 229ms
|
| 65 |
+
nearest+P 87% [ 78, 95] 4.14 0.59% 2.34% 519ms
|
| 66 |
+
nearest-margin 65% [ 52, 77] 4.10 0.71% 2.11% 223ms
|
| 67 |
+
NMPC 97% [ 92, 100] 4.17 0.07% 0.22% 23ms
|
| 68 |
+
generated 56.7%(sd5.9) [ 47, 73]* 4.15 0.93%(sd0.1) 3.01%(sd0.1) 458ms
|
| 69 |
+
generated+P 77.2%(sd2.1) [ 70, 90]* 4.19 0.92%(sd0.1) 2.93%(sd0.1) 1.2s
|
| 70 |
+
(* generated rows: mean (sd) over sampler seeds [0, 1, 2]; CI shown is for seed 0 only)
|
| 71 |
+
(certified hand-off rate == success rate for every arm except NMPC (a certificate is a precondition for success here); NMPC's own hand-off is n/a, it never switches to a static gain)
|
| 72 |
+
|
| 73 |
+
--- failure breakdown, primary seed 0 ---
|
| 74 |
+
per-plant replay failed:2 NMPC never reached Omega:2
|
| 75 |
+
nearest Lyapunov decrease fails at a vertex:16 rollout failed:5
|
| 76 |
+
nearest+P rollout failed:8
|
| 77 |
+
nearest-margin Lyapunov decrease fails at a vertex:16 rollout failed:5
|
| 78 |
+
NMPC :2
|
| 79 |
+
generated Lyapunov decrease fails at a vertex:14 rollout failed:7 peak |Kx| on Omega is 10.18 V > 10.0 V:1 peak |Kx| on Omega is 10.06 V > 10.0 V:1 true dynamics: 0.57% of the boundary has V-dot >= 0, 0.00% saturates:1
|
| 80 |
+
generated+P rollout failed:12
|
| 81 |
+
|
| 82 |
+
--- paired McNemar on success (exact binomial on discordant pairs) ---
|
| 83 |
+
per-plant vs nearest beats on 18, loses on 1 p=0.0001 * cost ratio 1.007x p=0.0000
|
| 84 |
+
per-plant vs nearest+P beats on 5, loses on 1 p=0.2188 cost ratio 1.006x p=0.0000
|
| 85 |
+
per-plant vs nearest-margin beats on 18, loses on 1 p=0.0001 * cost ratio 1.007x p=0.0000
|
| 86 |
+
per-plant vs NMPC beats on 0, loses on 2 p=0.5000 cost ratio 1.001x p=0.0064
|
| 87 |
+
per-plant vs generated beats on 20, loses on 0 p=0.0000 * cost ratio 1.009x p=0.0000
|
| 88 |
+
per-plant vs generated+P beats on 8, loses on 0 p=0.0078 * cost ratio 1.010x p=0.0000
|
| 89 |
+
nearest+P vs per-plant beats on 1, loses on 5 p=0.2188 cost ratio 0.994x p=0.0000
|
| 90 |
+
nearest+P vs nearest beats on 13, loses on 0 p=0.0002 * cost ratio 1.000x p=0.1154
|
| 91 |
+
nearest+P vs nearest-margin beats on 13, loses on 0 p=0.0002 * cost ratio 1.000x p=0.1154
|
| 92 |
+
nearest+P vs NMPC beats on 0, loses on 6 p=0.0312 * cost ratio 0.995x p=0.0000
|
| 93 |
+
nearest+P vs generated beats on 18, loses on 2 p=0.0004 * cost ratio 1.004x p=0.0081
|
| 94 |
+
nearest+P vs generated+P beats on 7, loses on 3 p=0.3438 cost ratio 1.005x p=0.0008
|
results/furuta_v3/traj_gen_v3_raw_clip1.txt
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
bank data/furuta_nmpc_bank3000.npz: 2801 entries
|
| 2 |
+
model models/Best_furuta_traj_cond_v3.pth: cond_dim=4 out_dim=116
|
| 3 |
+
[seed4242] per-plant ceiling: 93%
|
| 4 |
+
[seed4242] nearest+P sequences done
|
| 5 |
+
[seed4242] generated+P seed 0 sequences done (36.5s sampling for 60x16)
|
| 6 |
+
[seed4242] generated+P seed 1 sequences done (36.6s sampling for 60x16)
|
| 7 |
+
[seed4242] generated+P seed 2 sequences done (36.3s sampling for 60x16)
|
| 8 |
+
[seed9001] per-plant ceiling: 93%
|
| 9 |
+
[seed9001] nearest+P sequences done
|
| 10 |
+
[seed9001] generated+P seed 0 sequences done (38.4s sampling for 60x16)
|
| 11 |
+
[seed9001] generated+P seed 1 sequences done (36.7s sampling for 60x16)
|
| 12 |
+
[seed9001] generated+P seed 2 sequences done (35.7s sampling for 60x16)
|
| 13 |
+
|
| 14 |
+
=== candidate-budget sweep: generated+P vs nearest+P vs per-plant ceiling ===
|
| 15 |
+
|
| 16 |
+
--- plant set seed4242 (seed 4242, n=60) ---
|
| 17 |
+
K nearest+P generated+P per-plant
|
| 18 |
+
1 51.7% t= 0.44s 38.3%(sd1.4) t= 1.00s 93.3% [McNemar nearest+P vs generated+P(seed0): 16-7, p=0.093]
|
| 19 |
+
4 78.3% t= 0.70s 73.9%(sd2.1) t= 1.61s 93.3% [McNemar nearest+P vs generated+P(seed0): 9-8, p=1.000]
|
| 20 |
+
8 88.3% t= 0.70s 83.3%(sd3.6) t= 1.61s 93.3% [McNemar nearest+P vs generated+P(seed0): 3-3, p=1.000]
|
| 21 |
+
16 93.3% t= 0.70s 85.0%(sd2.7) t= 1.61s 93.3% [McNemar nearest+P vs generated+P(seed0): 4-1, p=0.375]
|
| 22 |
+
crossover K (generated+P mean >= nearest+P): never within tested range
|
| 23 |
+
|
| 24 |
+
--- plant set seed9001 (seed 9001, n=60) ---
|
| 25 |
+
K nearest+P generated+P per-plant
|
| 26 |
+
1 56.7% t= 0.33s 38.3%(sd6.2) t= 0.99s 93.3% [McNemar nearest+P vs generated+P(seed0): 22-6, p=0.004]
|
| 27 |
+
4 86.7% t= 0.49s 77.8%(sd0.8) t= 1.50s 93.3% [McNemar nearest+P vs generated+P(seed0): 8-3, p=0.227]
|
| 28 |
+
8 91.7% t= 0.49s 88.9%(sd0.8) t= 1.50s 93.3% [McNemar nearest+P vs generated+P(seed0): 3-1, p=0.625]
|
| 29 |
+
16 93.3% t= 0.49s 92.2%(sd1.6) t= 1.50s 93.3% [McNemar nearest+P vs generated+P(seed0): 1-1, p=1.000]
|
| 30 |
+
crossover K (generated+P mean >= nearest+P): never within tested range
|
results/furuta_v3/traj_gen_v3e2400s1_raw_clip1.txt
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
bank data/furuta_nmpc_bank3000.npz: 2801 entries
|
| 2 |
+
model models/Best_furuta_traj_cond_v3_e2400_s1.pth: cond_dim=4 out_dim=116
|
| 3 |
+
[seed4242] per-plant ceiling: 93%
|
| 4 |
+
[seed4242] nearest+P sequences done
|
| 5 |
+
[seed4242] generated+P seed 0 sequences done (36.6s sampling for 60x16)
|
| 6 |
+
[seed4242] generated+P seed 1 sequences done (37.9s sampling for 60x16)
|
| 7 |
+
[seed4242] generated+P seed 2 sequences done (37.1s sampling for 60x16)
|
| 8 |
+
[seed9001] per-plant ceiling: 93%
|
| 9 |
+
[seed9001] nearest+P sequences done
|
| 10 |
+
[seed9001] generated+P seed 0 sequences done (36.5s sampling for 60x16)
|
| 11 |
+
[seed9001] generated+P seed 1 sequences done (36.2s sampling for 60x16)
|
| 12 |
+
[seed9001] generated+P seed 2 sequences done (36.8s sampling for 60x16)
|
| 13 |
+
|
| 14 |
+
=== candidate-budget sweep: generated+P vs nearest+P vs per-plant ceiling ===
|
| 15 |
+
|
| 16 |
+
--- plant set seed4242 (seed 4242, n=60) ---
|
| 17 |
+
K nearest+P generated+P per-plant
|
| 18 |
+
1 51.7% t= 0.32s 55.6%(sd9.1) t= 1.10s 93.3% [McNemar nearest+P vs generated+P(seed0): 14-9, p=0.405]
|
| 19 |
+
4 78.3% t= 0.61s 85.0%(sd2.7) t= 1.22s 93.3% [McNemar nearest+P vs generated+P(seed0): 3-7, p=0.344]
|
| 20 |
+
8 88.3% t= 0.61s 90.0%(sd1.4) t= 1.22s 93.3% [McNemar nearest+P vs generated+P(seed0): 3-4, p=1.000]
|
| 21 |
+
16 93.3% t= 0.61s 91.1%(sd1.6) t= 1.22s 93.3% [McNemar nearest+P vs generated+P(seed0): 3-1, p=0.625]
|
| 22 |
+
crossover K (generated+P mean >= nearest+P): 1
|
| 23 |
+
|
| 24 |
+
--- plant set seed9001 (seed 9001, n=60) ---
|
| 25 |
+
K nearest+P generated+P per-plant
|
| 26 |
+
1 56.7% t= 0.48s 61.1%(sd5.7) t= 0.99s 93.3% [McNemar nearest+P vs generated+P(seed0): 15-13, p=0.851]
|
| 27 |
+
4 86.7% t= 0.68s 90.0%(sd0.0) t= 1.17s 93.3% [McNemar nearest+P vs generated+P(seed0): 2-4, p=0.688]
|
| 28 |
+
8 91.7% t= 0.68s 95.0%(sd1.4) t= 1.17s 93.3% [McNemar nearest+P vs generated+P(seed0): 1-2, p=1.000]
|
| 29 |
+
16 93.3% t= 0.68s 95.6%(sd0.8) t= 1.17s 93.3% [McNemar nearest+P vs generated+P(seed0): 0-1, p=1.000]
|
| 30 |
+
crossover K (generated+P mean >= nearest+P): 1
|
results/furuta_v3/traj_gen_v3s1_raw_clip1.txt
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
bank data/furuta_nmpc_bank3000.npz: 2801 entries
|
| 2 |
+
model models/Best_furuta_traj_cond_v3_s1.pth: cond_dim=4 out_dim=116
|
| 3 |
+
[seed4242] per-plant ceiling: 93%
|
| 4 |
+
[seed4242] nearest+P sequences done
|
| 5 |
+
[seed4242] generated+P seed 0 sequences done (39.8s sampling for 60x16)
|
| 6 |
+
[seed4242] generated+P seed 1 sequences done (35.2s sampling for 60x16)
|
| 7 |
+
[seed4242] generated+P seed 2 sequences done (35.2s sampling for 60x16)
|
| 8 |
+
[seed9001] per-plant ceiling: 93%
|
| 9 |
+
[seed9001] nearest+P sequences done
|
| 10 |
+
[seed9001] generated+P seed 0 sequences done (35.6s sampling for 60x16)
|
| 11 |
+
[seed9001] generated+P seed 1 sequences done (35.4s sampling for 60x16)
|
| 12 |
+
[seed9001] generated+P seed 2 sequences done (35.2s sampling for 60x16)
|
| 13 |
+
|
| 14 |
+
=== candidate-budget sweep: generated+P vs nearest+P vs per-plant ceiling ===
|
| 15 |
+
|
| 16 |
+
--- plant set seed4242 (seed 4242, n=60) ---
|
| 17 |
+
K nearest+P generated+P per-plant
|
| 18 |
+
1 51.7% t= 0.42s 29.4%(sd4.2) t= 0.91s 93.3% [McNemar nearest+P vs generated+P(seed0): 20-4, p=0.002]
|
| 19 |
+
4 78.3% t= 0.76s 70.6%(sd5.5) t= 1.48s 93.3% [McNemar nearest+P vs generated+P(seed0): 9-5, p=0.424]
|
| 20 |
+
8 88.3% t= 0.76s 81.1%(sd1.6) t= 1.48s 93.3% [McNemar nearest+P vs generated+P(seed0): 5-2, p=0.453]
|
| 21 |
+
16 93.3% t= 0.76s 82.2%(sd0.8) t= 1.48s 93.3% [McNemar nearest+P vs generated+P(seed0): 7-1, p=0.070]
|
| 22 |
+
crossover K (generated+P mean >= nearest+P): never within tested range
|
| 23 |
+
|
| 24 |
+
--- plant set seed9001 (seed 9001, n=60) ---
|
| 25 |
+
K nearest+P generated+P per-plant
|
| 26 |
+
1 56.7% t= 0.30s 33.9%(sd9.6) t= 0.93s 93.3% [McNemar nearest+P vs generated+P(seed0): 25-4, p=0.000]
|
| 27 |
+
4 86.7% t= 0.47s 72.2%(sd2.1) t= 1.70s 93.3% [McNemar nearest+P vs generated+P(seed0): 10-1, p=0.012]
|
| 28 |
+
8 91.7% t= 0.47s 82.8%(sd1.6) t= 1.70s 93.3% [McNemar nearest+P vs generated+P(seed0): 7-1, p=0.070]
|
| 29 |
+
16 93.3% t= 0.47s 90.6%(sd2.1) t= 1.70s 93.3% [McNemar nearest+P vs generated+P(seed0): 2-0, p=0.500]
|
| 30 |
+
crossover K (generated+P mean >= nearest+P): never within tested range
|
results/furuta_v3/traj_gen_warmstart_v2clip5_s0.txt
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== warm-start test (n=60, seed 4242, max_iter=700) ===
|
| 2 |
+
target certificate: largest_certified_region per plant (unmargined); verifier: furuta_traj_bank.score_solution (identical to every other arm in this doc)
|
| 3 |
+
|
| 4 |
+
condition success med iter med t med J
|
| 5 |
+
cold 53.3% 363 2.03s 4.06
|
| 6 |
+
nearest-warm 76.7% 45 0.27s 4.15
|
| 7 |
+
generated-warm 75.0% 50 0.30s 4.14
|
| 8 |
+
|
| 9 |
+
--- paired tests ---
|
| 10 |
+
generated-warm vs cold success: beats 13, loses 0, McNemar p=0.0002 * time ratio 0.175x (n=32), Wilcoxon p=0.0000 *
|
| 11 |
+
generated-warm vs nearest-warm success: beats 0, loses 1, McNemar p=1.0000 time ratio 1.127x (n=45), Wilcoxon p=0.0707
|
| 12 |
+
nearest-warm vs cold success: beats 14, loses 0, McNemar p=0.0001 * time ratio 0.147x (n=32), Wilcoxon p=0.0000 *
|
| 13 |
+
|
| 14 |
+
--- failure breakdown ---
|
| 15 |
+
cold ipopt: Maximum_Iterations_Exceeded:13 ipopt: Infeasible_Problem_Detected:11 rollout failed:4
|
| 16 |
+
nearest-warm ipopt: Infeasible_Problem_Detected:6 rollout failed:5 ipopt: Maximum_Iterations_Exceeded:3
|
| 17 |
+
generated-warm ipopt: Maximum_Iterations_Exceeded:7 ipopt: Infeasible_Problem_Detected:4 rollout failed:4
|
results/furuta_v3/traj_gen_warmstart_v3_s0.txt
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== warm-start test (n=60, seed 4242, max_iter=700) ===
|
| 2 |
+
target certificate: largest_certified_region per plant (unmargined); verifier: furuta_traj_bank.score_solution (identical to every other arm in this doc)
|
| 3 |
+
|
| 4 |
+
condition success med iter med t med J
|
| 5 |
+
cold 53.3% 363 2.03s 4.06
|
| 6 |
+
nearest-warm 76.7% 45 0.31s 4.15
|
| 7 |
+
generated-warm 76.7% 50 0.32s 4.15
|
| 8 |
+
|
| 9 |
+
--- paired tests ---
|
| 10 |
+
generated-warm vs cold success: beats 14, loses 0, McNemar p=0.0001 * time ratio 0.181x (n=32), Wilcoxon p=0.0000 *
|
| 11 |
+
generated-warm vs nearest-warm success: beats 0, loses 0, McNemar p=1.0000 time ratio 1.061x (n=46), Wilcoxon p=0.1474
|
| 12 |
+
nearest-warm vs cold success: beats 14, loses 0, McNemar p=0.0001 * time ratio 0.161x (n=32), Wilcoxon p=0.0000 *
|
| 13 |
+
|
| 14 |
+
--- failure breakdown ---
|
| 15 |
+
cold ipopt: Maximum_Iterations_Exceeded:13 ipopt: Infeasible_Problem_Detected:11 rollout failed:4
|
| 16 |
+
nearest-warm ipopt: Infeasible_Problem_Detected:6 rollout failed:5 ipopt: Maximum_Iterations_Exceeded:3
|
| 17 |
+
generated-warm ipopt: Infeasible_Problem_Detected:6 rollout failed:5 ipopt: Maximum_Iterations_Exceeded:3
|
results/furuta_v3/traj_gen_warmstart_v3_s1.txt
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== warm-start test (n=60, seed 4242, max_iter=700) ===
|
| 2 |
+
target certificate: largest_certified_region per plant (unmargined); verifier: furuta_traj_bank.score_solution (identical to every other arm in this doc)
|
| 3 |
+
|
| 4 |
+
condition success med iter med t med J
|
| 5 |
+
cold 53.3% 363 2.04s 4.06
|
| 6 |
+
nearest-warm 76.7% 45 0.29s 4.15
|
| 7 |
+
generated-warm 75.0% 48 0.30s 4.14
|
| 8 |
+
|
| 9 |
+
--- paired tests ---
|
| 10 |
+
generated-warm vs cold success: beats 13, loses 0, McNemar p=0.0002 * time ratio 0.155x (n=32), Wilcoxon p=0.0000 *
|
| 11 |
+
generated-warm vs nearest-warm success: beats 0, loses 1, McNemar p=1.0000 time ratio 1.107x (n=45), Wilcoxon p=0.3238
|
| 12 |
+
nearest-warm vs cold success: beats 14, loses 0, McNemar p=0.0001 * time ratio 0.149x (n=32), Wilcoxon p=0.0000 *
|
| 13 |
+
|
| 14 |
+
--- failure breakdown ---
|
| 15 |
+
cold ipopt: Maximum_Iterations_Exceeded:13 ipopt: Infeasible_Problem_Detected:11 rollout failed:4
|
| 16 |
+
nearest-warm ipopt: Infeasible_Problem_Detected:6 rollout failed:5 ipopt: Maximum_Iterations_Exceeded:3
|
| 17 |
+
generated-warm ipopt: Infeasible_Problem_Detected:8 ipopt: Maximum_Iterations_Exceeded:4 rollout failed:3
|
results/furuta_v3/traj_gen_warmstart_v3e2400_n200.txt
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== warm-start test (n=200, seed 4242, max_iter=700) ===
|
| 2 |
+
target certificate: largest_certified_region per plant (unmargined); verifier: furuta_traj_bank.score_solution (identical to every other arm in this doc)
|
| 3 |
+
|
| 4 |
+
condition success med iter med t med J
|
| 5 |
+
cold 58.0% 290 1.54s 4.06
|
| 6 |
+
nearest-warm 78.0% 48 0.28s 4.10
|
| 7 |
+
generated-warm 78.0% 48 0.28s 4.10
|
| 8 |
+
|
| 9 |
+
--- paired tests ---
|
| 10 |
+
generated-warm vs cold success: beats 41, loses 1, McNemar p=0.0000 * time ratio 0.213x (n=115), Wilcoxon p=0.0000 *
|
| 11 |
+
generated-warm vs nearest-warm success: beats 1, loses 1, McNemar p=1.0000 time ratio 1.020x (n=155), Wilcoxon p=0.5271
|
| 12 |
+
nearest-warm vs cold success: beats 41, loses 1, McNemar p=0.0000 * time ratio 0.205x (n=115), Wilcoxon p=0.0000 *
|
| 13 |
+
|
| 14 |
+
--- failure breakdown ---
|
| 15 |
+
cold ipopt: Maximum_Iterations_Exceeded:36 ipopt: Infeasible_Problem_Detected:31 rollout failed:17
|
| 16 |
+
nearest-warm rollout failed:26 ipopt: Maximum_Iterations_Exceeded:10 ipopt: Infeasible_Problem_Detected:8
|
| 17 |
+
generated-warm rollout failed:24 ipopt: Infeasible_Problem_Detected:11 ipopt: Maximum_Iterations_Exceeded:9
|
results/furuta_v3/traj_gen_warmstart_v3e2400_s0.txt
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== warm-start test (n=60, seed 4242, max_iter=700) ===
|
| 2 |
+
target certificate: largest_certified_region per plant (unmargined); verifier: furuta_traj_bank.score_solution (identical to every other arm in this doc)
|
| 3 |
+
|
| 4 |
+
condition success med iter med t med J
|
| 5 |
+
cold 53.3% 363 1.97s 4.06
|
| 6 |
+
nearest-warm 76.7% 45 0.26s 4.15
|
| 7 |
+
generated-warm 75.0% 43 0.25s 4.14
|
| 8 |
+
|
| 9 |
+
--- paired tests ---
|
| 10 |
+
generated-warm vs cold success: beats 13, loses 0, McNemar p=0.0002 * time ratio 0.152x (n=32), Wilcoxon p=0.0000 *
|
| 11 |
+
generated-warm vs nearest-warm success: beats 0, loses 1, McNemar p=1.0000 time ratio 1.056x (n=45), Wilcoxon p=0.7713
|
| 12 |
+
nearest-warm vs cold success: beats 14, loses 0, McNemar p=0.0001 * time ratio 0.156x (n=32), Wilcoxon p=0.0000 *
|
| 13 |
+
|
| 14 |
+
--- failure breakdown ---
|
| 15 |
+
cold ipopt: Maximum_Iterations_Exceeded:13 ipopt: Infeasible_Problem_Detected:11 rollout failed:4
|
| 16 |
+
nearest-warm ipopt: Infeasible_Problem_Detected:6 rollout failed:5 ipopt: Maximum_Iterations_Exceeded:3
|
| 17 |
+
generated-warm ipopt: Infeasible_Problem_Detected:6 rollout failed:5 ipopt: Maximum_Iterations_Exceeded:4
|
results/furuta_v3/traj_gen_warmstart_v3e2400_s1.txt
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== warm-start test (n=60, seed 4242, max_iter=700) ===
|
| 2 |
+
target certificate: largest_certified_region per plant (unmargined); verifier: furuta_traj_bank.score_solution (identical to every other arm in this doc)
|
| 3 |
+
|
| 4 |
+
condition success med iter med t med J
|
| 5 |
+
cold 53.3% 363 2.03s 4.06
|
| 6 |
+
nearest-warm 76.7% 45 0.27s 4.15
|
| 7 |
+
generated-warm 75.0% 44 0.26s 4.14
|
| 8 |
+
|
| 9 |
+
--- paired tests ---
|
| 10 |
+
generated-warm vs cold success: beats 13, loses 0, McNemar p=0.0002 * time ratio 0.157x (n=32), Wilcoxon p=0.0000 *
|
| 11 |
+
generated-warm vs nearest-warm success: beats 0, loses 1, McNemar p=1.0000 time ratio 1.008x (n=45), Wilcoxon p=0.9643
|
| 12 |
+
nearest-warm vs cold success: beats 14, loses 0, McNemar p=0.0001 * time ratio 0.160x (n=32), Wilcoxon p=0.0000 *
|
| 13 |
+
|
| 14 |
+
--- failure breakdown ---
|
| 15 |
+
cold ipopt: Maximum_Iterations_Exceeded:13 ipopt: Infeasible_Problem_Detected:11 rollout failed:4
|
| 16 |
+
nearest-warm ipopt: Infeasible_Problem_Detected:6 rollout failed:5 ipopt: Maximum_Iterations_Exceeded:3
|
| 17 |
+
generated-warm ipopt: Infeasible_Problem_Detected:6 rollout failed:5 ipopt: Maximum_Iterations_Exceeded:4
|
results/furuta_v3/traj_gen_warmstart_v3e2400_s2.txt
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== warm-start test (n=60, seed 4242, max_iter=700) ===
|
| 2 |
+
target certificate: largest_certified_region per plant (unmargined); verifier: furuta_traj_bank.score_solution (identical to every other arm in this doc)
|
| 3 |
+
|
| 4 |
+
condition success med iter med t med J
|
| 5 |
+
cold 53.3% 363 2.04s 4.06
|
| 6 |
+
nearest-warm 76.7% 45 0.27s 4.15
|
| 7 |
+
generated-warm 75.0% 47 0.28s 4.14
|
| 8 |
+
|
| 9 |
+
--- paired tests ---
|
| 10 |
+
generated-warm vs cold success: beats 13, loses 0, McNemar p=0.0002 * time ratio 0.160x (n=32), Wilcoxon p=0.0000 *
|
| 11 |
+
generated-warm vs nearest-warm success: beats 0, loses 1, McNemar p=1.0000 time ratio 1.038x (n=45), Wilcoxon p=0.7799
|
| 12 |
+
nearest-warm vs cold success: beats 14, loses 0, McNemar p=0.0001 * time ratio 0.151x (n=32), Wilcoxon p=0.0000 *
|
| 13 |
+
|
| 14 |
+
--- failure breakdown ---
|
| 15 |
+
cold ipopt: Maximum_Iterations_Exceeded:13 ipopt: Infeasible_Problem_Detected:11 rollout failed:4
|
| 16 |
+
nearest-warm ipopt: Infeasible_Problem_Detected:6 rollout failed:5 ipopt: Maximum_Iterations_Exceeded:3
|
| 17 |
+
generated-warm ipopt: Infeasible_Problem_Detected:7 ipopt: Maximum_Iterations_Exceeded:4 rollout failed:4
|
results/furuta_v3/valloss.txt
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
models/Best_furuta_traj_cond_v3.pth: val eps-MSE raw 0.0560 averaged 0.0777 (trainer best_val 0.0704, ckpt time 14.9 min)
|
| 2 |
+
models/Last_furuta_traj_cond_v3.pth: val eps-MSE raw 0.0560 averaged 0.0777 (trainer best_val 0.0704, ckpt time 14.9 min)
|
| 3 |
+
models/Best_furuta_traj_cond_v3_s1.pth: val eps-MSE raw 0.0566 averaged 0.0808 (trainer best_val 0.0642, ckpt time 13.3 min)
|
| 4 |
+
models/Best_furuta_traj_cond_v2.pth: val eps-MSE raw 0.0997 averaged 0.1348 (trainer best_val 0.1208, ckpt time 8.6 min)
|
| 5 |
+
models/Best_furuta_traj_cond_v3_e2400.pth: val eps-MSE raw 0.0366 averaged 0.0665 (trainer best_val 0.0566, ckpt time 42.5 min)
|
| 6 |
+
models/Best_furuta_traj_cond_v3_e2400_s1.pth: val eps-MSE raw 0.0367 averaged 0.0632 (trainer best_val 0.0538, ckpt time 39.2 min)
|
| 7 |
+
models/Best_furuta_traj_cond_v2ctl_e2400.pth: val eps-MSE raw 0.0776 averaged 0.0993 (trainer best_val 0.0816, ckpt time 37.6 min)
|
results/k_sweep_summary.md
ADDED
|
@@ -0,0 +1,81 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# K-sweep summary β coverage vs sampling budget K (SISO / MIMO / pendulum)
|
| 2 |
+
|
| 3 |
+
"Does drawing more controllers from the prior close the stabilizable gap?" β measured per family.
|
| 4 |
+
Coverage = fraction of held-out plants with β₯1 stabilizing controller among K draws. Scripts:
|
| 5 |
+
`k_sweep.py` (SISO), `k_sweep_mimo.py` (MIMO), `k_sweep_pendulum.py` (pendulum).
|
| 6 |
+
|
| 7 |
+
## Per-plant oracle (what "stabilizing" means, per family)
|
| 8 |
+
|
| 9 |
+
Coverage counts a plant as covered if **β₯1 of its K drawn controllers stabilizes it**. The decision
|
| 10 |
+
procedure that labels a single (plant, controller) pair β the *per-plant oracle* β is **not the same
|
| 11 |
+
across families**, and only the pendulum uses a genuine certificate:
|
| 12 |
+
|
| 13 |
+
| family | script | oracle used in the sweep | parameters | certificate? |
|
| 14 |
+
|---|---|---|---|---|
|
| 15 |
+
| SISO | `k_sweep.py` | **rollout only** β `simulate_closed_loop_batch`; stable iff mean `final_energy` finite and `< 0.3` | n_ic=4 (`x0=randn`), dt=0.03, T=12 s | none β rel-deg-3 loop, not LMI-certifiable |
|
| 16 |
+
| MIMO | `k_sweep_mimo.py` | **rollout only** β `simulate_closed_loop_mimo_batched`; stable iff `final_energy < 0.3` and `J_integral` finite | n_ic=4, dt=0.03, T=12 s | exists (`certified_beta`) but **not called** in the sweep |
|
| 17 |
+
| Pendulum | `k_sweep_pendulum.py` | **rollout AND certificate** β `simulate_pendulum_batch` (stable iff `final_energy < 0.02`) **plus** `certify_controller` (regional LMI), reported as a separate curve | rollout: n_ic=3, dt=0.02, T=6 s | **yes** β regional LMI per plant |
|
| 18 |
+
|
| 19 |
+
Reading this precisely:
|
| 20 |
+
|
| 21 |
+
- **SISO** β the oracle is a **finite-horizon energy-decay rollout**, no certificate. SISO loops are
|
| 22 |
+
relative degree 3, outside what the quadratic Popov/circle criterion can certify, so there is no
|
| 23 |
+
per-plant certificate to call; "stable" means the state energy has decayed below 0.3 by T=12 s from
|
| 24 |
+
4 random initial conditions.
|
| 25 |
+
- **MIMO** β the sweep oracle is **also rollout only** (energy + finite cost). A valid MIMO certificate
|
| 26 |
+
(`certified_beta`, the circle/diagonal-multiplier LMI) exists and certifies the *dataset targets*,
|
| 27 |
+
but it is **not invoked inside the K-sweep** β so the MIMO coverage numbers here are rollout-validated,
|
| 28 |
+
not per-draw-certified. (The "100% certified" MIMO claim elsewhere refers to the dataset targets and to
|
| 29 |
+
the eigenvalue-certified repair, not to this coverage sweep.)
|
| 30 |
+
- **Pendulum** β the **only** family whose sweep runs a true per-plant certificate oracle. Each draw is
|
| 31 |
+
both simulated (`final_energy < 0.02`) and passed through the regional LMI `certify_controller`; both
|
| 32 |
+
the "β₯1 stable" and "β₯1 certified" curves are reported, and both sit at 1.000 for all K.
|
| 33 |
+
|
| 34 |
+
So: SISO/MIMO coverage = empirical (rollout) oracle; pendulum coverage = empirical **and** certificate
|
| 35 |
+
oracle. This is the honest scope β the certified-of-K story is pendulum-only.
|
| 36 |
+
|
| 37 |
+
## Cross-task comparison
|
| 38 |
+
|
| 39 |
+
The needed sampling budget is a direct read-out of dataset quality: cost-optimal + certified targets
|
| 40 |
+
(MIMO, pendulum) need essentially no sampling; SISO's unfiltered random dataset needs Kβ32.
|
| 41 |
+
|
| 42 |
+
| task | dataset targets | per-sample stability | K to reach β₯1-stable = 1.0 |
|
| 43 |
+
|---|---|---|---|
|
| 44 |
+
| SISO | random-in-sector (not filtered, ~52% stable) | 0.66 | ~32 |
|
| 45 |
+
| MIMO | cost-optimal, certified | 0.96 | 4 |
|
| 46 |
+
| Pendulum | cost-optimal, certified | 1.00 | 1 (and certified at K=1) |
|
| 47 |
+
|
| 48 |
+
## Per-task curves
|
| 49 |
+
|
| 50 |
+
**SISO** (N=24, seed 0, per-sample stability 0.66):
|
| 51 |
+
|
| 52 |
+
| K | 1 | 2 | 4 | 8 | 16 | 32 | 64 |
|
| 53 |
+
|---|---|---|---|---|---|---|---|
|
| 54 |
+
| plants β₯1 stable | 0.583 | 0.792 | 0.792 | 0.875 | 0.917 | 1.000 | 1.000 |
|
| 55 |
+
|
| 56 |
+
**MIMO** (N=60, seed 9999, per-sample stability 0.96):
|
| 57 |
+
|
| 58 |
+
| K | 1 | 2 | 4 | 8 | 16 | 32 |
|
| 59 |
+
|---|---|---|---|---|---|---|
|
| 60 |
+
| plants β₯1 stable | 0.950 | 0.983 | 1.000 | 1.000 | 1.000 | 1.000 |
|
| 61 |
+
|
| 62 |
+
**Pendulum** (N=24, seed 123, per-sample stability 1.00; has a valid certificate, so also reported):
|
| 63 |
+
|
| 64 |
+
| K | 1 | 2 | 4 | 8 | 16 |
|
| 65 |
+
|---|---|---|---|---|---|
|
| 66 |
+
| plants β₯1 stable | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
|
| 67 |
+
| plants β₯1 **certified** | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
|
| 68 |
+
|
| 69 |
+
## Takeaways
|
| 70 |
+
|
| 71 |
+
- The SISO "gap" is specifically an artifact of its unfiltered random dataset (only ~52% of stored
|
| 72 |
+
controllers are actually stable). The prior imitates that mixed distribution, so more draws are
|
| 73 |
+
needed to land a stable one.
|
| 74 |
+
- MIMO and the pendulum have no stabilizable gap: one to four draws already stabilize every plant,
|
| 75 |
+
and the pendulum is regionally **certified** from a single draw.
|
| 76 |
+
- So the sampling budget K to stabilize scales inversely with dataset quality β the cleanest single
|
| 77 |
+
read-out of "the prior imitates its targets."
|
| 78 |
+
|
| 79 |
+
Figures: `figures/fig1b_ksweep.png` (cross-task coverage vs K, + SISO coverage vs inference time).
|
| 80 |
+
Also reproduced in `RESULTS.md` Β§4.2 ("Cross-task K-sweep"). Raw per-task logs:
|
| 81 |
+
`results/siso_coherent/k_sweep.txt`, `results/mimo/k_sweep_mimo.txt`, `results/pendulum/k_sweep_pendulum.txt`.
|
results/regressor_siso_train.log
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
device cuda | N=180160 in=10 out=21
|
| 2 |
+
epoch 0 | val MSE 0.00175 (best 0.00175)
|
| 3 |
+
epoch 5 | val MSE 0.00175 (best 0.00174)
|
| 4 |
+
epoch 10 | val MSE 0.00174 (best 0.00174)
|
| 5 |
+
epoch 15 | val MSE 0.00174 (best 0.00174)
|
| 6 |
+
epoch 20 | val MSE 0.00174 (best 0.00174)
|
| 7 |
+
epoch 25 | val MSE 0.00174 (best 0.00174)
|
| 8 |
+
epoch 30 | val MSE 0.00174 (best 0.00174)
|
| 9 |
+
epoch 35 | val MSE 0.00174 (best 0.00174)
|
| 10 |
+
epoch 39 | val MSE 0.00174 (best 0.00174)
|
| 11 |
+
saved -> models/regressor_siso.pt (best val MSE 0.00174)
|