Thibaut Forest Claude Opus 5 (1M context) commited on
Commit
2a6a86a
·
1 Parent(s): 9d114fd

CV v5 fidelity pass (user): GroupKFold with deployment-exact held-out semantics

Browse files

Fold fits on TRAIN buildings' full closed history (= the static foreign fit
deployment-legal for held-out fleets); planner state = submit.py static-path
knobs exactly, incl. the cold-start floor gate (structurally inert at fold
sizes >= 60 deaths vs gate 20); in-fold purge dropped (stricter than the
deployment it represents, bought no public coherence). Both public and
private splits are ~2x held-out fleets (Docker rehearsal) -> ONE CV for both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Files changed (1) hide show
  1. batteryswap_example/cv.py +41 -25
batteryswap_example/cv.py CHANGED
@@ -1,19 +1,24 @@
1
- """THE CV (user directive 2026-08-22 v4): building-GroupKFold x temporal purge.
2
 
3
  ONE protocol lives here, deliberately. Buildings are assigned to K folds,
4
  stratified on battery-failure count and location (travel centrality). For
5
- each fold, for each of the 48 scenario times: fit on the TRAIN buildings'
6
- pairs whose 42d label windows lie ENTIRELY outside the scenario window
7
- (origin + W <= T or origin >= T + W - the label-overlap purge), then plan
8
- the TEST buildings as a STANDALONE fleet and evaluate its cost.
 
 
 
 
9
  THE SCORE is the mean over folds of the fold's mean scenario cost.
10
 
11
- Double holdout, no tuning: the fit never sees the test buildings (fleet
12
- axis) AND never sees the scenario's own window (time axis). Coherence with
13
- the public board is CHECKED against the recorded public deltas, never
14
- fitted - a calibrated pricing layer was explicitly rejected (user,
15
- 2026-08-22): an instrument tuned to reproduce past public deltas can only
16
- confirm what it was tuned to.
 
17
 
18
  Why this protocol (2026-08-22, the day the temporal purge fell):
19
 
@@ -71,7 +76,8 @@ BASE_FEATURES = list(mc.FEATURES)
71
  # frontier): plain logbag / tsroute alpha 0.8 + break-even top-up, censor-
72
  # aware pricing. Frontier dials exist in submit.py for explicit A/B only.
73
  CHAMPION = dict(model='logbag', planner='tsroute', p_star=0.8, day_q=0.05,
74
- ec_margin=0.8, ec_pmax=25.0, split_days=7)
 
75
 
76
  N_FOLDS = 5
77
 
@@ -160,10 +166,10 @@ VARIANTS = {'base': _base, 'ranks_add': _ranks_add, 'vrel': _vrel}
160
 
161
  def run_bgkf(dataset_path=Path('dataset'), kind=None, variant='base',
162
  blend=None, n_folds=N_FOLDS, origins=range(48)):
163
- """THE SCORE. For each fold, per scenario: fit on the TRAIN buildings'
164
- purged pairs (label windows fully outside the scenario window), then
165
- plan the TEST buildings as a standalone fleet and evaluate. Floor off
166
- (harness minimalism; the deployed floor is a submit.py concern).
167
  Candidate hooks: `kind` swaps the model family (mc.MODELS key or
168
  SklearnRUL spec), `variant` the feature set, `blend`=(kind2, w2) a
169
  value-preserving rank blend (order changes, the primary's p42 values
@@ -178,11 +184,14 @@ def run_bgkf(dataset_path=Path('dataset'), kind=None, variant='base',
178
  folds, dev_b = build_folds(t, ctx, dataset_path, k=n_folds)
179
  t_bldg = t['device_id'].map(dev_b)
180
 
 
 
 
181
  p = mc.plan_tsroute
182
- p.alpha, p.p_be, p.floor_day, p.p_conf = CHAMPION['p_star'], 0.0, 0, 0.5
183
  p.ec_margin, p.ec_pmax = CHAMPION['ec_margin'], CHAMPION['ec_pmax']
184
  p.split_days = CHAMPION['split_days']
185
- p.floor_scale, p.recent_deaths = 0.0, 0.0
186
 
187
  def _fit(k_, train):
188
  hazard = k_ in ('aft', 'dthaz', 'dthaz_aux', 'logbag_open',
@@ -198,16 +207,13 @@ def run_bgkf(dataset_path=Path('dataset'), kind=None, variant='base',
198
  recs = []
199
  for f in range(n_folds):
200
  test_b = set(folds.index[folds == f])
201
- fold_train = closed[~t_bldg.loc[closed.index].isin(test_b)]
 
 
 
202
  for idx in origins:
203
  c = ctx[idx]
204
  T = c['start_time']
205
- # the purge, inside the fold: train pairs' label windows must lie
206
- # entirely before or after the scenario window
207
- train = fold_train[(fold_train['origin'] + W <= T)
208
- | (fold_train['origin'] >= T + W)]
209
- model = _fit(kind, train)
210
- partner = _fit(blend[0], train) if blend is not None else None
211
  fleet = c['locs'][c['locs']['building'].isin(test_b)]
212
  rows = (t[t['origin_idx'] == idx].set_index('device_id')
213
  .reindex(fleet['battery']).dropna(subset=['v_last']))
@@ -225,6 +231,16 @@ def run_bgkf(dataset_path=Path('dataset'), kind=None, variant='base',
225
  pred['p42'] = pd.Series(
226
  vals, index=r.sort_values(ascending=False).index
227
  ).reindex(pred.index)
 
 
 
 
 
 
 
 
 
 
228
  plan = mc.plan_tsroute(pred, None, locs, T, CHAMPION['day_q'],
229
  travel_costs=c['travel_costs'],
230
  settings=c['settings'])
 
1
+ """THE CV (user directive 2026-08-22 v5): building-GroupKFold, deployment-faithful.
2
 
3
  ONE protocol lives here, deliberately. Buildings are assigned to K folds,
4
  stratified on battery-failure count and location (travel centrality). For
5
+ each fold: ONE fit on the TRAIN buildings' full closed-pair history (that is
6
+ exactly what the deployed held-out path does - a foreign fleet's labels were
7
+ all known when the submission was built, so "before and after" is
8
+ deployment-legal, not a leak), then for each of the 48 scenario times the
9
+ TEST buildings are planned as a STANDALONE fleet with the DEPLOYMENT-EXACT
10
+ planner state (submit.py's static-path knobs, including the cold-start
11
+ floor gate, which is structurally inert here because every fold trains on
12
+ 60+ unique deaths vs the gate's 20) and costed by the real evaluator.
13
  THE SCORE is the mean over folds of the fold's mean scenario cost.
14
 
15
+ Fidelity pass (v5): this harness is the closest possible local twin of the
16
+ held-out deployment path - both public and private splits are ~2x unseen
17
+ fleets served by that path (Docker dress rehearsal, phase 27). Coherence
18
+ with the public board is CHECKED against recorded public deltas, never
19
+ fitted (a calibrated pricing layer was explicitly rejected). The in-fold
20
+ purge (v4) was dropped: it made the CV stricter than the deployment it
21
+ represents and bought no public coherence (identical sign flips).
22
 
23
  Why this protocol (2026-08-22, the day the temporal purge fell):
24
 
 
76
  # frontier): plain logbag / tsroute alpha 0.8 + break-even top-up, censor-
77
  # aware pricing. Frontier dials exist in submit.py for explicit A/B only.
78
  CHAMPION = dict(model='logbag', planner='tsroute', p_star=0.8, day_q=0.05,
79
+ ec_margin=0.8, ec_pmax=25.0, split_days=7,
80
+ floor_scale=2.5, floor_gate=20) # submit.py FLOOR_* defaults
81
 
82
  N_FOLDS = 5
83
 
 
166
 
167
  def run_bgkf(dataset_path=Path('dataset'), kind=None, variant='base',
168
  blend=None, n_folds=N_FOLDS, origins=range(48)):
169
+ """THE SCORE. Per fold: one fit on the TRAIN buildings' full closed-pair
170
+ history (deployment-exact for a held-out fleet), then per scenario the
171
+ TEST buildings are planned as a standalone fleet under submit.py's
172
+ static-path planner state (incl. the cold-start floor gate) and costed.
173
  Candidate hooks: `kind` swaps the model family (mc.MODELS key or
174
  SklearnRUL spec), `variant` the feature set, `blend`=(kind2, w2) a
175
  value-preserving rank blend (order changes, the primary's p42 values
 
184
  folds, dev_b = build_folds(t, ctx, dataset_path, k=n_folds)
185
  t_bldg = t['device_id'].map(dev_b)
186
 
187
+ # deployment-exact planner state: exactly the attributes submit.py's
188
+ # static path sets, everything else at models_cv module defaults
189
+ # (p_be=0, floor_day=0, p_conf=0.5 - deployment does not touch them)
190
  p = mc.plan_tsroute
191
+ p.alpha = CHAMPION['p_star']
192
  p.ec_margin, p.ec_pmax = CHAMPION['ec_margin'], CHAMPION['ec_pmax']
193
  p.split_days = CHAMPION['split_days']
194
+ p.floor_scale, p.recent_deaths = CHAMPION['floor_scale'], 0.0
195
 
196
  def _fit(k_, train):
197
  hazard = k_ in ('aft', 'dthaz', 'dthaz_aux', 'logbag_open',
 
207
  recs = []
208
  for f in range(n_folds):
209
  test_b = set(folds.index[folds == f])
210
+ train = closed[~t_bldg.loc[closed.index].isin(test_b)]
211
+ model = _fit(kind, train)
212
+ partner = _fit(blend[0], train) if blend is not None else None
213
+ train_uniq = int(train.loc[train['dies42'] == 1, 'device_id'].nunique())
214
  for idx in origins:
215
  c = ctx[idx]
216
  T = c['start_time']
 
 
 
 
 
 
217
  fleet = c['locs'][c['locs']['building'].isin(test_b)]
218
  rows = (t[t['origin_idx'] == idx].set_index('device_id')
219
  .reindex(fleet['battery']).dropna(subset=['v_last']))
 
231
  pred['p42'] = pd.Series(
232
  vals, index=r.sort_values(ascending=False).index
233
  ).reindex(pred.index)
234
+ # cold-start floor, gated exactly as submit.py: observable
235
+ # recent deaths / voltage-zone census, armed only when the fit
236
+ # saw < floor_gate unique deaths (inert at fold sizes >= 60)
237
+ p.recent_deaths = 0.0
238
+ if CHAMPION['floor_scale'] > 0 and train_uniq < CHAMPION['floor_gate']:
239
+ recent = int(((eol > T - W) & (eol <= T)).sum())
240
+ zone = int(((rows['days_below_245'] >= 0)
241
+ & (rows['days_below_245'] <= WINDOW_DAYS)
242
+ & (rows['staleness'] <= 7)).sum())
243
+ p.recent_deaths = float(max(recent, zone))
244
  plan = mc.plan_tsroute(pred, None, locs, T, CHAMPION['day_q'],
245
  travel_costs=c['travel_costs'],
246
  settings=c['settings'])