Thibaut Forest Claude Opus 5 (1M context) commited on
Commit
f5f3fda
·
1 Parent(s): 86f3f93

Phase 56 addendum 2: frontier purge gain = hidden per-scenario appetite (corr 0.76); head default reverted to legacy champion

Browse files

Anchor decomposition: order-only worth -37..-47 +/- 43 n.s.; the -342 was
count reallocation via calibration (era hazard leaks through both-sides
fits - purge limitation #3). BATTERYSWAP_FRONTIER now defaults OFF; dials
kept for explicit A/B. Shipping bar tightened to count-matched evaluation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Files changed (2) hide show
  1. EXPERIMENTS.md +31 -0
  2. batteryswap_example/submit.py +33 -5
EXPERIMENTS.md CHANGED
@@ -3079,3 +3079,34 @@ tie-aware rank comparison zero mismatches). What the anchored head tests
3079
  publicly, cleanly for the first time: is the frontier ORDER better on the
3080
  scored fleet? Downside is bounded to ordering (count machinery is the
3081
  measured champion's).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3079
  publicly, cleanly for the first time: is the frontier ORDER better on the
3080
  scored fleet? Downside is bounded to ordering (count machinery is the
3081
  measured champion's).
3082
+
3083
+ ### Phase 56 addendum 2 - full decomposition: the frontier's purge gain was a hidden per-scenario appetite dial; head reverted to legacy default
3084
+
3085
+ Anchor decomposition on the purge (all paired, 48 scenarios):
3086
+ | arm | values (count source) | order | SCORE | vs old champ |
3087
+ |---|---|---|---|---|
3088
+ | unanchored frontier | logbag_rank | frontier | 1630.3 | -341.7 (publicly +215.6) |
3089
+ | scale-anchored | lbr shape, donor sums | frontier | 1925.1 | -46.9 +/- 42.6 (33W/14L, n.s.) |
3090
+ | value-anchored | plain logbag | frontier | 1934.8 | -37.2 +/- 44.1 (n.s.) |
3091
+
3092
+ The smoking gun: mean swaps are FLAT (9.21 vs 9.23) but per-scenario counts
3093
+ swing |d| = 2.56 on average (23/48 scenarios differ by >= 2), and
3094
+ corr(count delta, cost gain) = 0.759. The 15 scenarios where logbag_rank's
3095
+ sum(p42) swelled the budget carry +956 gain each (~= the entire -342); the
3096
+ 25 where it shrank carry -2. MECHANISM: in both-sides purge fits, future-side
3097
+ pairs from a deadly era raise the fitted base rate exactly in deadly
3098
+ scenarios - era-level hazard leaking through CALIBRATION. The label-window
3099
+ purge cannot block it; deployment's past-only fits cannot replicate it
3100
+ (measured collapse 0.78-0.80x at late origins); the public board refuted it
3101
+ (-63 swaps, +215.6). PURGE LIMITATION #3, and it retroactively weakens every
3102
+ "swaps flat at the mean" clearance in this campaign: the shipping bar is now
3103
+ COUNT-MATCHED (value-anchored) evaluation, where the frontier's true
3104
+ ordering value is -37..-47 +/- 43 - real-looking W/L (33/14) but sub-bar.
3105
+
3106
+ ACTIONS: BATTERYSWAP_FRONTIER default OFF (head = legacy logbag champion,
3107
+ the measured-1661.45 behavior, plus all hardening); frontier dials retained
3108
+ for explicit A/B (ANCHOR=scale|value|off, W2). No model-side campaign
3109
+ candidate survives count-matched adjudication. Instruments going forward:
3110
+ deployment-mirror walk-forward (past-only fits) as the public predictor;
3111
+ building-GroupKFold with test-group-planned-as-fleet (user proposal,
3112
+ 2026-08-22) as the held-out/private-fleet predictor.
batteryswap_example/submit.py CHANGED
@@ -106,7 +106,13 @@ MODEL_KIND = 'logbag'
106
  # partner destroys the increment, so the public score is the arbiter).
107
  # Confirmed local optimum by 5 lanes (hyperparams, features, weights, neural
108
  # rankers, physics blends all subsumed or worse head-to-head).
109
- FRONTIER = os.environ.get('BATTERYSWAP_FRONTIER', '1') != '0'
 
 
 
 
 
 
110
  # PUBLIC REFUTATION of the UNANCHORED frontier (698401a: 1877.08 vs 1661.45,
111
  # swaps 517 vs 580): under deployment's past-only fits, logbag_rank's
112
  # sum(p42) collapses vs plain logbag as the pool grows (measured ratio 1.02
@@ -116,7 +122,19 @@ FRONTIER = os.environ.get('BATTERYSWAP_FRONTIER', '1') != '0'
116
  # VALUE DONOR (count/pricing/timing = the measured-580-swap calibration,
117
  # byte-identical to the old champion's machinery) and the frontier supplies
118
  # ONLY the rank order. Ordering-only in deployment, not just in CV.
119
- FRONTIER_ANCHOR = os.environ.get('BATTERYSWAP_FRONTIER_ANCHOR', '1') != '0'
 
 
 
 
 
 
 
 
 
 
 
 
120
  # Blend weight dial for public A/B: 0.30 = full frontier (purge 1630.3);
121
  # 0 = plain logbag_rank, no gbm (purge 1746.4 - the clean linear component,
122
  # fallback if the gbm increment turns out to be purge flattery on public);
@@ -142,8 +160,9 @@ class FrontierRUL:
142
  t['r_tau'] = t.groupby('origin')['tau_projected'].rank(pct=True)
143
  self.primary = LogBagFeats(['r_vlast', 'r_tau']).fit(t)
144
  self.partner = SklearnRUL('gbm').fit(t)
145
- # value donor: the old champion's calibration owns the swap count
146
- self.donor = SklearnRUL(MODEL_KIND).fit(train) if FRONTIER_ANCHOR else None
 
147
  return self
148
 
149
  def predict(self, rows):
@@ -157,12 +176,21 @@ class FrontierRUL:
157
  if FRONTIER_W2 > 0:
158
  p2 = self.partner.predict(r)
159
  rk = (1 - FRONTIER_W2) * rk + FRONTIER_W2 * p2['p42'].rank(pct=True)
160
- base = self.donor.predict(rows) if self.donor is not None else pred
 
 
 
161
  vals = np.sort(base['p42'].values)[::-1]
162
  out = base.copy()
163
  out['p42'] = pd.Series(
164
  vals, index=rk.sort_values(ascending=False).index
165
  ).reindex(out.index)
 
 
 
 
 
 
166
  return out
167
 
168
 
 
106
  # partner destroys the increment, so the public score is the arbiter).
107
  # Confirmed local optimum by 5 lanes (hyperparams, features, weights, neural
108
  # rankers, physics blends all subsumed or worse head-to-head).
109
+ # DEFAULT OFF (phase 56 final decomposition): the frontier's purge gain was
110
+ # a hidden per-scenario appetite dial - corr(count delta, cost gain) 0.76;
111
+ # at matched counts the ordering is worth only -37..-47 +/- 43 (n.s.).
112
+ # Era-hazard leaks through CALIBRATION in both-sides purge fits; deployment
113
+ # (past-only) cannot replicate it and public refuted it (1877.08). The dials
114
+ # below remain for explicit A/B only.
115
+ FRONTIER = os.environ.get('BATTERYSWAP_FRONTIER', '0') != '0'
116
  # PUBLIC REFUTATION of the UNANCHORED frontier (698401a: 1877.08 vs 1661.45,
117
  # swaps 517 vs 580): under deployment's past-only fits, logbag_rank's
118
  # sum(p42) collapses vs plain logbag as the pool grows (measured ratio 1.02
 
122
  # VALUE DONOR (count/pricing/timing = the measured-580-swap calibration,
123
  # byte-identical to the old champion's machinery) and the frontier supplies
124
  # ONLY the rank order. Ordering-only in deployment, not just in CV.
125
+ # ANCHOR MODES (purge decomposition, phase 56 addendum 2): the frontier's
126
+ # -341.7 CV gain is mostly the VALUE SHAPE, not the order - value-anchoring
127
+ # to plain logbag keeps only the order and scores 1934.8 on the purge
128
+ # (-37.2 +/- 44.1 vs old champion, n.s.). 'scale' keeps logbag_rank's value
129
+ # SHAPE (the gain carrier) but rescales sum(p42) to the plain-logbag donor's
130
+ # sum, so the swap count is the measured champion calibration by construction
131
+ # while shape and order stay the frontier's. 'value' = donor values (order
132
+ # only). 'off' = unanchored (publicly refuted 1877.08).
133
+ FRONTIER_ANCHOR = os.environ.get('BATTERYSWAP_FRONTIER_ANCHOR', 'scale').lower()
134
+ if FRONTIER_ANCHOR in ('1', 'true'):
135
+ FRONTIER_ANCHOR = 'value'
136
+ if FRONTIER_ANCHOR in ('0', 'false', 'none'):
137
+ FRONTIER_ANCHOR = 'off'
138
  # Blend weight dial for public A/B: 0.30 = full frontier (purge 1630.3);
139
  # 0 = plain logbag_rank, no gbm (purge 1746.4 - the clean linear component,
140
  # fallback if the gbm increment turns out to be purge flattery on public);
 
160
  t['r_tau'] = t.groupby('origin')['tau_projected'].rank(pct=True)
161
  self.primary = LogBagFeats(['r_vlast', 'r_tau']).fit(t)
162
  self.partner = SklearnRUL('gbm').fit(t)
163
+ # donor: the old champion's calibration owns the swap count
164
+ self.donor = (SklearnRUL(MODEL_KIND).fit(train)
165
+ if FRONTIER_ANCHOR != 'off' else None)
166
  return self
167
 
168
  def predict(self, rows):
 
176
  if FRONTIER_W2 > 0:
177
  p2 = self.partner.predict(r)
178
  rk = (1 - FRONTIER_W2) * rk + FRONTIER_W2 * p2['p42'].rank(pct=True)
179
+ if FRONTIER_ANCHOR == 'value':
180
+ base = self.donor.predict(rows) # donor values, frontier order
181
+ else:
182
+ base = pred # frontier values (and shape)
183
  vals = np.sort(base['p42'].values)[::-1]
184
  out = base.copy()
185
  out['p42'] = pd.Series(
186
  vals, index=rk.sort_values(ascending=False).index
187
  ).reindex(out.index)
188
+ if FRONTIER_ANCHOR == 'scale':
189
+ # calibration correction: frontier shape, donor COUNT - fixes the
190
+ # measured past-only sum(p42) collapse (0.78-0.80x at late origins)
191
+ dsum, psum = self.donor.predict(rows)['p42'].sum(), out['p42'].sum()
192
+ if psum > 0:
193
+ out['p42'] *= dsum / psum
194
  return out
195
 
196