Thibaut Forest Claude Opus 5 (1M context) commited on
Commit ·
f5f3fda
1
Parent(s): 86f3f93
Phase 56 addendum 2: frontier purge gain = hidden per-scenario appetite (corr 0.76); head default reverted to legacy champion
Browse filesAnchor decomposition: order-only worth -37..-47 +/- 43 n.s.; the -342 was
count reallocation via calibration (era hazard leaks through both-sides
fits - purge limitation #3). BATTERYSWAP_FRONTIER now defaults OFF; dials
kept for explicit A/B. Shipping bar tightened to count-matched evaluation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- EXPERIMENTS.md +31 -0
- batteryswap_example/submit.py +33 -5
EXPERIMENTS.md
CHANGED
|
@@ -3079,3 +3079,34 @@ tie-aware rank comparison zero mismatches). What the anchored head tests
|
|
| 3079 |
publicly, cleanly for the first time: is the frontier ORDER better on the
|
| 3080 |
scored fleet? Downside is bounded to ordering (count machinery is the
|
| 3081 |
measured champion's).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3079 |
publicly, cleanly for the first time: is the frontier ORDER better on the
|
| 3080 |
scored fleet? Downside is bounded to ordering (count machinery is the
|
| 3081 |
measured champion's).
|
| 3082 |
+
|
| 3083 |
+
### Phase 56 addendum 2 - full decomposition: the frontier's purge gain was a hidden per-scenario appetite dial; head reverted to legacy default
|
| 3084 |
+
|
| 3085 |
+
Anchor decomposition on the purge (all paired, 48 scenarios):
|
| 3086 |
+
| arm | values (count source) | order | SCORE | vs old champ |
|
| 3087 |
+
|---|---|---|---|---|
|
| 3088 |
+
| unanchored frontier | logbag_rank | frontier | 1630.3 | -341.7 (publicly +215.6) |
|
| 3089 |
+
| scale-anchored | lbr shape, donor sums | frontier | 1925.1 | -46.9 +/- 42.6 (33W/14L, n.s.) |
|
| 3090 |
+
| value-anchored | plain logbag | frontier | 1934.8 | -37.2 +/- 44.1 (n.s.) |
|
| 3091 |
+
|
| 3092 |
+
The smoking gun: mean swaps are FLAT (9.21 vs 9.23) but per-scenario counts
|
| 3093 |
+
swing |d| = 2.56 on average (23/48 scenarios differ by >= 2), and
|
| 3094 |
+
corr(count delta, cost gain) = 0.759. The 15 scenarios where logbag_rank's
|
| 3095 |
+
sum(p42) swelled the budget carry +956 gain each (~= the entire -342); the
|
| 3096 |
+
25 where it shrank carry -2. MECHANISM: in both-sides purge fits, future-side
|
| 3097 |
+
pairs from a deadly era raise the fitted base rate exactly in deadly
|
| 3098 |
+
scenarios - era-level hazard leaking through CALIBRATION. The label-window
|
| 3099 |
+
purge cannot block it; deployment's past-only fits cannot replicate it
|
| 3100 |
+
(measured collapse 0.78-0.80x at late origins); the public board refuted it
|
| 3101 |
+
(-63 swaps, +215.6). PURGE LIMITATION #3, and it retroactively weakens every
|
| 3102 |
+
"swaps flat at the mean" clearance in this campaign: the shipping bar is now
|
| 3103 |
+
COUNT-MATCHED (value-anchored) evaluation, where the frontier's true
|
| 3104 |
+
ordering value is -37..-47 +/- 43 - real-looking W/L (33/14) but sub-bar.
|
| 3105 |
+
|
| 3106 |
+
ACTIONS: BATTERYSWAP_FRONTIER default OFF (head = legacy logbag champion,
|
| 3107 |
+
the measured-1661.45 behavior, plus all hardening); frontier dials retained
|
| 3108 |
+
for explicit A/B (ANCHOR=scale|value|off, W2). No model-side campaign
|
| 3109 |
+
candidate survives count-matched adjudication. Instruments going forward:
|
| 3110 |
+
deployment-mirror walk-forward (past-only fits) as the public predictor;
|
| 3111 |
+
building-GroupKFold with test-group-planned-as-fleet (user proposal,
|
| 3112 |
+
2026-08-22) as the held-out/private-fleet predictor.
|
batteryswap_example/submit.py
CHANGED
|
@@ -106,7 +106,13 @@ MODEL_KIND = 'logbag'
|
|
| 106 |
# partner destroys the increment, so the public score is the arbiter).
|
| 107 |
# Confirmed local optimum by 5 lanes (hyperparams, features, weights, neural
|
| 108 |
# rankers, physics blends all subsumed or worse head-to-head).
|
| 109 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 110 |
# PUBLIC REFUTATION of the UNANCHORED frontier (698401a: 1877.08 vs 1661.45,
|
| 111 |
# swaps 517 vs 580): under deployment's past-only fits, logbag_rank's
|
| 112 |
# sum(p42) collapses vs plain logbag as the pool grows (measured ratio 1.02
|
|
@@ -116,7 +122,19 @@ FRONTIER = os.environ.get('BATTERYSWAP_FRONTIER', '1') != '0'
|
|
| 116 |
# VALUE DONOR (count/pricing/timing = the measured-580-swap calibration,
|
| 117 |
# byte-identical to the old champion's machinery) and the frontier supplies
|
| 118 |
# ONLY the rank order. Ordering-only in deployment, not just in CV.
|
| 119 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 120 |
# Blend weight dial for public A/B: 0.30 = full frontier (purge 1630.3);
|
| 121 |
# 0 = plain logbag_rank, no gbm (purge 1746.4 - the clean linear component,
|
| 122 |
# fallback if the gbm increment turns out to be purge flattery on public);
|
|
@@ -142,8 +160,9 @@ class FrontierRUL:
|
|
| 142 |
t['r_tau'] = t.groupby('origin')['tau_projected'].rank(pct=True)
|
| 143 |
self.primary = LogBagFeats(['r_vlast', 'r_tau']).fit(t)
|
| 144 |
self.partner = SklearnRUL('gbm').fit(t)
|
| 145 |
-
#
|
| 146 |
-
self.donor = SklearnRUL(MODEL_KIND).fit(train)
|
|
|
|
| 147 |
return self
|
| 148 |
|
| 149 |
def predict(self, rows):
|
|
@@ -157,12 +176,21 @@ class FrontierRUL:
|
|
| 157 |
if FRONTIER_W2 > 0:
|
| 158 |
p2 = self.partner.predict(r)
|
| 159 |
rk = (1 - FRONTIER_W2) * rk + FRONTIER_W2 * p2['p42'].rank(pct=True)
|
| 160 |
-
|
|
|
|
|
|
|
|
|
|
| 161 |
vals = np.sort(base['p42'].values)[::-1]
|
| 162 |
out = base.copy()
|
| 163 |
out['p42'] = pd.Series(
|
| 164 |
vals, index=rk.sort_values(ascending=False).index
|
| 165 |
).reindex(out.index)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 166 |
return out
|
| 167 |
|
| 168 |
|
|
|
|
| 106 |
# partner destroys the increment, so the public score is the arbiter).
|
| 107 |
# Confirmed local optimum by 5 lanes (hyperparams, features, weights, neural
|
| 108 |
# rankers, physics blends all subsumed or worse head-to-head).
|
| 109 |
+
# DEFAULT OFF (phase 56 final decomposition): the frontier's purge gain was
|
| 110 |
+
# a hidden per-scenario appetite dial - corr(count delta, cost gain) 0.76;
|
| 111 |
+
# at matched counts the ordering is worth only -37..-47 +/- 43 (n.s.).
|
| 112 |
+
# Era-hazard leaks through CALIBRATION in both-sides purge fits; deployment
|
| 113 |
+
# (past-only) cannot replicate it and public refuted it (1877.08). The dials
|
| 114 |
+
# below remain for explicit A/B only.
|
| 115 |
+
FRONTIER = os.environ.get('BATTERYSWAP_FRONTIER', '0') != '0'
|
| 116 |
# PUBLIC REFUTATION of the UNANCHORED frontier (698401a: 1877.08 vs 1661.45,
|
| 117 |
# swaps 517 vs 580): under deployment's past-only fits, logbag_rank's
|
| 118 |
# sum(p42) collapses vs plain logbag as the pool grows (measured ratio 1.02
|
|
|
|
| 122 |
# VALUE DONOR (count/pricing/timing = the measured-580-swap calibration,
|
| 123 |
# byte-identical to the old champion's machinery) and the frontier supplies
|
| 124 |
# ONLY the rank order. Ordering-only in deployment, not just in CV.
|
| 125 |
+
# ANCHOR MODES (purge decomposition, phase 56 addendum 2): the frontier's
|
| 126 |
+
# -341.7 CV gain is mostly the VALUE SHAPE, not the order - value-anchoring
|
| 127 |
+
# to plain logbag keeps only the order and scores 1934.8 on the purge
|
| 128 |
+
# (-37.2 +/- 44.1 vs old champion, n.s.). 'scale' keeps logbag_rank's value
|
| 129 |
+
# SHAPE (the gain carrier) but rescales sum(p42) to the plain-logbag donor's
|
| 130 |
+
# sum, so the swap count is the measured champion calibration by construction
|
| 131 |
+
# while shape and order stay the frontier's. 'value' = donor values (order
|
| 132 |
+
# only). 'off' = unanchored (publicly refuted 1877.08).
|
| 133 |
+
FRONTIER_ANCHOR = os.environ.get('BATTERYSWAP_FRONTIER_ANCHOR', 'scale').lower()
|
| 134 |
+
if FRONTIER_ANCHOR in ('1', 'true'):
|
| 135 |
+
FRONTIER_ANCHOR = 'value'
|
| 136 |
+
if FRONTIER_ANCHOR in ('0', 'false', 'none'):
|
| 137 |
+
FRONTIER_ANCHOR = 'off'
|
| 138 |
# Blend weight dial for public A/B: 0.30 = full frontier (purge 1630.3);
|
| 139 |
# 0 = plain logbag_rank, no gbm (purge 1746.4 - the clean linear component,
|
| 140 |
# fallback if the gbm increment turns out to be purge flattery on public);
|
|
|
|
| 160 |
t['r_tau'] = t.groupby('origin')['tau_projected'].rank(pct=True)
|
| 161 |
self.primary = LogBagFeats(['r_vlast', 'r_tau']).fit(t)
|
| 162 |
self.partner = SklearnRUL('gbm').fit(t)
|
| 163 |
+
# donor: the old champion's calibration owns the swap count
|
| 164 |
+
self.donor = (SklearnRUL(MODEL_KIND).fit(train)
|
| 165 |
+
if FRONTIER_ANCHOR != 'off' else None)
|
| 166 |
return self
|
| 167 |
|
| 168 |
def predict(self, rows):
|
|
|
|
| 176 |
if FRONTIER_W2 > 0:
|
| 177 |
p2 = self.partner.predict(r)
|
| 178 |
rk = (1 - FRONTIER_W2) * rk + FRONTIER_W2 * p2['p42'].rank(pct=True)
|
| 179 |
+
if FRONTIER_ANCHOR == 'value':
|
| 180 |
+
base = self.donor.predict(rows) # donor values, frontier order
|
| 181 |
+
else:
|
| 182 |
+
base = pred # frontier values (and shape)
|
| 183 |
vals = np.sort(base['p42'].values)[::-1]
|
| 184 |
out = base.copy()
|
| 185 |
out['p42'] = pd.Series(
|
| 186 |
vals, index=rk.sort_values(ascending=False).index
|
| 187 |
).reindex(out.index)
|
| 188 |
+
if FRONTIER_ANCHOR == 'scale':
|
| 189 |
+
# calibration correction: frontier shape, donor COUNT - fixes the
|
| 190 |
+
# measured past-only sum(p42) collapse (0.78-0.80x at late origins)
|
| 191 |
+
dsum, psum = self.donor.predict(rows)['p42'].sum(), out['p42'].sum()
|
| 192 |
+
if psum > 0:
|
| 193 |
+
out['p42'] *= dsum / psum
|
| 194 |
return out
|
| 195 |
|
| 196 |
|