Thibaut Forest Claude Opus 5 (1M context) commited on
Commit
9d114fd
·
1 Parent(s): 24b7f13

CV v4 (user directive): building-GroupKFold x temporal purge - per fold, per scenario, fit on train buildings' pairs with label windows entirely outside the scenario window; test scenario on the test buildings as a standalone fleet

Browse files

Double holdout by construction (fleet axis + time axis), no tuning: the
public-calibrated pricing layer was explicitly rejected. Coherence with
recorded public deltas is checked, never fitted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Files changed (1) hide show
  1. batteryswap_example/cv.py +41 -27
batteryswap_example/cv.py CHANGED
@@ -1,13 +1,20 @@
1
- """THE CV (user directive 2026-08-22 v3): building-GroupKFold cost harness.
2
 
3
  ONE protocol lives here, deliberately. Buildings are assigned to K folds,
4
  stratified on battery-failure count and location (travel centrality). For
5
- each fold: fit ONE model on ALL closed training pairs of the TRAIN buildings
6
- (their full history - deployment-legal for foreign fleets, whose labels were
7
- all known when the submission was built), then for each of the 48 scenario
8
- times plan the TEST buildings as a STANDALONE fleet and evaluate its cost.
9
  THE SCORE is the mean over folds of the fold's mean scenario cost.
10
 
 
 
 
 
 
 
 
11
  Why this protocol (2026-08-22, the day the temporal purge fell):
12
 
13
  * It simulates the HELD-OUT deployment path: the competition's private split
@@ -20,14 +27,16 @@ Why this protocol (2026-08-22, the day the temporal purge fell):
20
  count bias (phase 26): the count rule operates on the test fleet's own
21
  sum(p42), exactly as deployment does on a held-out fleet.
22
 
23
- * The retired temporal purge was refuted as a deployment predictor three
24
- ways in one day (phase 56): count-coupled changes (purge loved alpha 1.5,
25
- public +527), label-process exploits (cert-first re-derived at its exact
26
- pre-refutation signature), and era-hazard calibration leak through
27
- future-side fits (the frontier: purge -341.7 -> public +215.6; the gain
28
- was a hidden per-scenario appetite dial, corr(count delta, cost gain)
29
- 0.76). The common thread: its fits saw the future. This harness's fits
30
- never see the test buildings at all.
 
 
31
 
32
  * Diagnostics are PLAN-DERIVED only (cost blocks + in-window swap count), so
33
  ANY method that yields a plan can be scored. Candidates must win at ~flat
@@ -151,13 +160,14 @@ VARIANTS = {'base': _base, 'ranks_add': _ranks_add, 'vrel': _vrel}
151
 
152
  def run_bgkf(dataset_path=Path('dataset'), kind=None, variant='base',
153
  blend=None, n_folds=N_FOLDS, origins=range(48)):
154
- """THE SCORE. For each fold: one fit on the TRAIN buildings' closed pairs
155
- (full history), then per scenario time plan the TEST buildings as a
156
- standalone fleet and evaluate. Floor off (harness minimalism; the
157
- deployed floor is a submit.py concern). Candidate hooks: `kind` swaps the
158
- model family (mc.MODELS key or SklearnRUL spec), `variant` the feature
159
- set, `blend`=(kind2, w2) a value-preserving rank blend (order changes,
160
- the primary's p42 values and hence the count machinery do not).
 
161
  """
162
  kind = kind or CHAMPION['model']
163
  t = load_table(dataset_path)
@@ -188,12 +198,16 @@ def run_bgkf(dataset_path=Path('dataset'), kind=None, variant='base',
188
  recs = []
189
  for f in range(n_folds):
190
  test_b = set(folds.index[folds == f])
191
- train = closed[~t_bldg.loc[closed.index].isin(test_b)]
192
- model = _fit(kind, train)
193
- partner = _fit(blend[0], train) if blend is not None else None
194
  for idx in origins:
195
  c = ctx[idx]
196
  T = c['start_time']
 
 
 
 
 
 
197
  fleet = c['locs'][c['locs']['building'].isin(test_b)]
198
  rows = (t[t['origin_idx'] == idx].set_index('device_id')
199
  .reindex(fleet['battery']).dropna(subset=['v_last']))
@@ -312,10 +326,10 @@ def main():
312
  candidate_report()
313
  else:
314
  raise SystemExit(
315
- f'unknown protocol {proto!r} - the building-GroupKFold harness '
316
- f'is the only CV (user directive 2026-08-22 v3). The temporal '
317
- f'purge lives in git at f5f3fda; the older walk-forward family '
318
- f'at 2000281.')
319
  print(f'\n[{time.time() - t0:.0f}s]')
320
 
321
 
 
1
+ """THE CV (user directive 2026-08-22 v4): building-GroupKFold x temporal purge.
2
 
3
  ONE protocol lives here, deliberately. Buildings are assigned to K folds,
4
  stratified on battery-failure count and location (travel centrality). For
5
+ each fold, for each of the 48 scenario times: fit on the TRAIN buildings'
6
+ pairs whose 42d label windows lie ENTIRELY outside the scenario window
7
+ (origin + W <= T or origin >= T + W - the label-overlap purge), then plan
8
+ the TEST buildings as a STANDALONE fleet and evaluate its cost.
9
  THE SCORE is the mean over folds of the fold's mean scenario cost.
10
 
11
+ Double holdout, no tuning: the fit never sees the test buildings (fleet
12
+ axis) AND never sees the scenario's own window (time axis). Coherence with
13
+ the public board is CHECKED against the recorded public deltas, never
14
+ fitted - a calibrated pricing layer was explicitly rejected (user,
15
+ 2026-08-22): an instrument tuned to reproduce past public deltas can only
16
+ confirm what it was tuned to.
17
+
18
  Why this protocol (2026-08-22, the day the temporal purge fell):
19
 
20
  * It simulates the HELD-OUT deployment path: the competition's private split
 
27
  count bias (phase 26): the count rule operates on the test fleet's own
28
  sum(p42), exactly as deployment does on a held-out fleet.
29
 
30
+ * Instrument history: the pure temporal purge (fleet-wide) fell to three
31
+ public refutations in one day (phase 56: count-coupling, label-process
32
+ ghosts, era-calibration leak - its fits saw the future of the same fleet).
33
+ The pure building split (v3) fell the same evening: it still sign-flipped
34
+ all three public killers (phase 57 alignment check) because its fits saw
35
+ the scenario's own era through other buildings. v4 removes both channels
36
+ structurally. What NO local split can price (public-measured, 4x): the
37
+ public fleet's uncatchable tail and its train-absent ghost devices -
38
+ count-coupled and label-process candidates stay report-only regardless of
39
+ what this harness says.
40
 
41
  * Diagnostics are PLAN-DERIVED only (cost blocks + in-window swap count), so
42
  ANY method that yields a plan can be scored. Candidates must win at ~flat
 
160
 
161
  def run_bgkf(dataset_path=Path('dataset'), kind=None, variant='base',
162
  blend=None, n_folds=N_FOLDS, origins=range(48)):
163
+ """THE SCORE. For each fold, per scenario: fit on the TRAIN buildings'
164
+ purged pairs (label windows fully outside the scenario window), then
165
+ plan the TEST buildings as a standalone fleet and evaluate. Floor off
166
+ (harness minimalism; the deployed floor is a submit.py concern).
167
+ Candidate hooks: `kind` swaps the model family (mc.MODELS key or
168
+ SklearnRUL spec), `variant` the feature set, `blend`=(kind2, w2) a
169
+ value-preserving rank blend (order changes, the primary's p42 values
170
+ and hence the count machinery do not).
171
  """
172
  kind = kind or CHAMPION['model']
173
  t = load_table(dataset_path)
 
198
  recs = []
199
  for f in range(n_folds):
200
  test_b = set(folds.index[folds == f])
201
+ fold_train = closed[~t_bldg.loc[closed.index].isin(test_b)]
 
 
202
  for idx in origins:
203
  c = ctx[idx]
204
  T = c['start_time']
205
+ # the purge, inside the fold: train pairs' label windows must lie
206
+ # entirely before or after the scenario window
207
+ train = fold_train[(fold_train['origin'] + W <= T)
208
+ | (fold_train['origin'] >= T + W)]
209
+ model = _fit(kind, train)
210
+ partner = _fit(blend[0], train) if blend is not None else None
211
  fleet = c['locs'][c['locs']['building'].isin(test_b)]
212
  rows = (t[t['origin_idx'] == idx].set_index('device_id')
213
  .reindex(fleet['battery']).dropna(subset=['v_last']))
 
326
  candidate_report()
327
  else:
328
  raise SystemExit(
329
+ f'unknown protocol {proto!r} - the building-GroupKFold x purge '
330
+ f'harness is the only CV (user directive 2026-08-22 v4). The '
331
+ f'fleet-wide purge lives in git at f5f3fda; the pure building '
332
+ f'split at 4be2283; the walk-forward family at 2000281.')
333
  print(f'\n[{time.time() - t0:.0f}s]')
334
 
335