Thibaut Forest Claude Opus 5 (1M context) commited on
Commit
a354f2b
·
1 Parent(s): 2de326a

Phase 17: label-arrival hazard, domain-split, cost-weighting, backtest-alpha all refuted; count channel closed

Browse files
Files changed (2) hide show
  1. EXPERIMENTS.md +29 -0
  2. batteryswap_example/models_cv.py +117 -0
EXPERIMENTS.md CHANGED
@@ -536,6 +536,35 @@ prohibitively expensive to the break-even top-up as well (two-sided fix,
536
  phase 11). Example decoded: d_57e99ff00802 - EOL 2026-02-02 (scored death),
537
  replacement battery reports until 2026-05-13, admin end_time 2026-08-01.
538
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
539
  ## Test-look ledger
540
 
541
  8 looks total (now historical - see protocol change above); look 9 = the
 
536
  phase 11). Example decoded: d_57e99ff00802 - EOL 2026-02-02 (scored death),
537
  replacement battery reports until 2026-05-13, admin end_time 2026-08-01.
538
 
539
+ ## Phase 17 - label-process modeling & count correction (all five directions closed)
540
+
541
+ A structured attack on the 1679 -> 1400 gap along the label-process and
542
+ count-error channels (user-provided plan). Every direction adjudicated:
543
+
544
+ | candidate | panel | verdict |
545
+ |---|---|---|
546
+ | label-arrival hazard (two-regime: logbag for not-crossed, KM P(EOL recorded in window \| crossed a days ago) for the death zone) | **2837.7 (+1158, 1 win/33)** | refuted: the classifier's p42 for crossed devices is ALREADY the correctly calibrated mixture - dies42 labels contain the arrival process; overriding ~0.15 with KM arrival ~0.4-0.9 inflates sum(p42), explodes N |
547
+ | + domain-split training (classifier never trained/queried in death zone) | **2938.1 (+1259)** | refuted: most positives ARE crossed near death - splitting starves the classifier of deaths |
548
+ | cost-weighted training (miss ~200 vs era FP price) | **2693.7 (+1015)** | refuted; worst exactly at early origins where FP weights are largest |
549
+ | oracle-N diagnostic (per-origin N sweep, real evaluator) | mean regret **222.6**/origin, median 185 | the pool is real but the oracle direction is era-inconsistent (winner's curse around a shallow optimum) |
550
+ | backtest-alpha corrector (pick alpha from last-K closed origins' replayed curves - simulated offline from the saved sweeps) | K=4: +113, K=6: +89, K=8: +36 vs fixed 0.8 | refuted: converges to picking 0.8 (18/26 at K=8); per-origin optimal alpha is unpredictable noise |
551
+
552
+ The label-process channel has now been attacked five ways across phases 15
553
+ and 17 (binary zeroing, hard quarantine, domain split, KM arrival hazard,
554
+ cost weighting) - each loses by hundreds. Conclusion: the bagged logistic's
555
+ implicit calibration of the label-recording process is better than every
556
+ explicit model of it we can impose, and the fixed CV-chosen count (alpha=0.8)
557
+ is better than every adaptive corrector tested.
558
+
559
+ Per the plan's own decision rule ("if the first two deliver under ~120
560
+ combined, 1400 on the panel likely isn't reachable"): they delivered negative
561
+ amounts; **1400 is not reachable on this panel without new information
562
+ channels**. The honest target remains deployment-era origins (ghost-dense,
563
+ cheap-FP), which the shipped mechanisms - adaptive per-origin refits,
564
+ break-even top-up, censoring-aware pricing, building day-splitting - already
565
+ serve, and whose panel deltas systematically understate their deployment
566
+ value. Champion unchanged: panel 1679.2, all-48 Docker 2146.9.
567
+
568
  ## Test-look ledger
569
 
570
  8 looks total (now historical - see protocol change above); look 9 = the
batteryswap_example/models_cv.py CHANGED
@@ -198,6 +198,77 @@ class SklearnRUL:
198
  GHOST_ZERO = False # phase-15 candidate: exclude label-process ghosts from plans
199
  QUARANTINE = False # phase-15 candidate: drop looks-dead-labeled-alive training rows
200
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
201
  SILENT_V = 2.55
202
  SILENT_FEATURES = ['rng_30', 'vstd_30', 'tau_projected', 'cohort_rel_age_m',
203
  'age_days', 'v_corr_last']
@@ -456,6 +527,49 @@ class BlendDHAux:
456
  return out
457
 
458
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
459
  class WeightedBagLogistic:
460
  """Bagged logistic with recency-weighted samples: pairs from recent origins
461
  count more, on the theory that fleet composition and degradation mix drift."""
@@ -867,6 +981,9 @@ MODELS = {
867
  'cg_all': lambda: CohortGate('all'),
868
  'cg_silent': lambda: CohortGate('silent'),
869
  'logbag_dh': LogbagDH,
 
 
 
870
  }
871
 
872
 
 
198
  GHOST_ZERO = False # phase-15 candidate: exclude label-process ghosts from plans
199
  QUARANTINE = False # phase-15 candidate: drop looks-dead-labeled-alive training rows
200
 
201
+
202
+ class LogbagLAH:
203
+ """
204
+ Two-regime scorer for the label process (phase 17). The metric charges
205
+ RECORDED deaths, and EOL recording lags the physical 2.45-crossing by a
206
+ median ~34d (a third of recorded deaths lag >42d; ghosts never get one).
207
+ For devices already in the death zone the uncertainty is not "will it
208
+ die" but "will the label arrive in this window" - a pure function of
209
+ time-since-crossing. So:
210
+ not crossed -> champion logbag p42 (its real domain)
211
+ crossed and still low -> p42 = A(a) = P(EOL recorded in (a, a+42] | none
212
+ by a), a Kaplan-Meier on observed crossing->EOL lags (uncensored:
213
+ recorded deaths; censored: crossed devices with no EOL yet).
214
+ Unlike the phase-15 binary zero rule this keeps mid-lag devices included
215
+ (A ~ 0.3-0.6) and lets true ghosts fade gradually as their censored mass
216
+ flattens the curve. domain_split=True additionally trains the classifier
217
+ only on not-crossed rows, since it is never queried in the death zone.
218
+ """
219
+
220
+ BUCKET = 14.0 # KM lag buckets (days); 82 events cannot support finer
221
+
222
+ def __init__(self, domain_split=False):
223
+ self.domain_split = domain_split
224
+
225
+ def fit(self, train):
226
+ crossed = train['days_below_245'].notna() & (train['days_below_245'] >= 0)
227
+ self.who = SklearnRUL('logbag').fit(train[~crossed] if self.domain_split
228
+ else train)
229
+
230
+ # one lag observation per device: crossing -> EOL (or censored at the
231
+ # last origin it appears). days_below_245 is leak-free per row.
232
+ sub = train[crossed]
233
+ per_dev = sub.sort_values('origin').groupby('device_id').tail(1)
234
+ cross_date = per_dev['origin'] - pd.to_timedelta(per_dev['days_below_245'], unit='D')
235
+ has_eol = per_dev['eol'].notna()
236
+ lag = np.where(has_eol,
237
+ (per_dev['eol'] - cross_date).dt.days,
238
+ per_dev['days_below_245'])
239
+ lag = np.clip(lag, 0, None)
240
+ event = has_eol.values
241
+
242
+ # discrete KM over lag buckets
243
+ k = (lag // self.BUCKET).astype(int)
244
+ kmax = int(k.max()) + 1 if len(k) else 1
245
+ surv, s = [], 1.0
246
+ for b in range(kmax):
247
+ at_risk = (k >= b).sum()
248
+ died = ((k == b) & event).sum()
249
+ s *= 1.0 - (died / at_risk if at_risk > 0 else 0.0)
250
+ surv.append(s)
251
+ self.surv = np.array(surv)
252
+ return self
253
+
254
+ def _arrival(self, a):
255
+ """P(EOL recorded within the next 42d | crossed a days ago, none yet)."""
256
+ b0 = np.clip((a // self.BUCKET).astype(int), 0, len(self.surv) - 1)
257
+ b1 = np.clip(((a + WINDOW_DAYS) // self.BUCKET).astype(int), 0, len(self.surv) - 1)
258
+ s0 = self.surv[b0]
259
+ s1 = self.surv[b1]
260
+ return np.where(s0 > 1e-9, 1.0 - s1 / s0, 0.0)
261
+
262
+ def predict(self, rows):
263
+ out = self.who.predict(rows)
264
+ crossed = (rows['days_below_245'].notna()
265
+ & (rows['days_below_245'] >= 0)
266
+ & (rows['v_corr_last'] < 2.50)).values
267
+ if crossed.any():
268
+ a = rows.loc[crossed, 'days_below_245'].values
269
+ out.loc[crossed, 'p42'] = self._arrival(a)
270
+ return out
271
+
272
  SILENT_V = 2.55
273
  SILENT_FEATURES = ['rng_30', 'vstd_30', 'tau_projected', 'cohort_rel_age_m',
274
  'age_days', 'v_corr_last']
 
527
  return out
528
 
529
 
530
+ class CostWeightBag:
531
+ """
532
+ Bagged logistic with DECISION-STAKE sample weights (phase 17): a missed
533
+ positive costs ~emergency (~200); a false inclusion costs its era's
534
+ false-swap price (0.5/day against the end_time+30 imputation, so early
535
+ origins' negatives matter ~10x more than near-data-end ones). Weights
536
+ normalized to mean 1 so the L2 scale (C=1) stays comparable to logbag.
537
+ """
538
+
539
+ MISS_W = 200.0
540
+
541
+ def __init__(self, n_estimators=15):
542
+ self.n_estimators = n_estimators
543
+
544
+ def fit(self, train):
545
+ X, y = train[FEATURES], train['dies42']
546
+ data_end = train['origin'].max()
547
+ fp_price = (0.5 * ((data_end - train['origin']).dt.days + 30 - 21)
548
+ ).clip(lower=1.0).values
549
+ w = np.where(y.values == 1, self.MISS_W, fp_price)
550
+ w = w / w.mean()
551
+ rng = np.random.RandomState(0)
552
+ self.models = []
553
+ n = len(X)
554
+ for _ in range(self.n_estimators):
555
+ idx = rng.randint(0, n, n)
556
+ m = make_pipeline(SimpleImputer(strategy='median'),
557
+ StandardScaler(),
558
+ LogisticRegression(max_iter=2000, C=1.0))
559
+ m.fit(X.iloc[idx], y.iloc[idx],
560
+ logisticregression__sample_weight=w[idx])
561
+ self.models.append(m)
562
+ dying = train[train['dies42'] == 1]
563
+ self.days = DayQuantiles().fit(dying[FEATURES], dying['days_to_eol'])
564
+ return self
565
+
566
+ def predict(self, rows):
567
+ X = rows[FEATURES]
568
+ out = self.days.predict(X)
569
+ out['p42'] = np.mean([m.predict_proba(X)[:, 1] for m in self.models], axis=0)
570
+ return out
571
+
572
+
573
  class WeightedBagLogistic:
574
  """Bagged logistic with recency-weighted samples: pairs from recent origins
575
  count more, on the theory that fleet composition and degradation mix drift."""
 
981
  'cg_all': lambda: CohortGate('all'),
982
  'cg_silent': lambda: CohortGate('silent'),
983
  'logbag_dh': LogbagDH,
984
+ 'logbag_lah': LogbagLAH,
985
+ 'logbag_lah_ds': lambda: LogbagLAH(domain_split=True),
986
+ 'logbag_cw': CostWeightBag,
987
  }
988
 
989