Thibaut Forest Claude Opus 5 (1M context) commited on
Commit ·
a354f2b
1
Parent(s): 2de326a
Phase 17: label-arrival hazard, domain-split, cost-weighting, backtest-alpha all refuted; count channel closed
Browse files- EXPERIMENTS.md +29 -0
- batteryswap_example/models_cv.py +117 -0
EXPERIMENTS.md
CHANGED
|
@@ -536,6 +536,35 @@ prohibitively expensive to the break-even top-up as well (two-sided fix,
|
|
| 536 |
phase 11). Example decoded: d_57e99ff00802 - EOL 2026-02-02 (scored death),
|
| 537 |
replacement battery reports until 2026-05-13, admin end_time 2026-08-01.
|
| 538 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 539 |
## Test-look ledger
|
| 540 |
|
| 541 |
8 looks total (now historical - see protocol change above); look 9 = the
|
|
|
|
| 536 |
phase 11). Example decoded: d_57e99ff00802 - EOL 2026-02-02 (scored death),
|
| 537 |
replacement battery reports until 2026-05-13, admin end_time 2026-08-01.
|
| 538 |
|
| 539 |
+
## Phase 17 - label-process modeling & count correction (all five directions closed)
|
| 540 |
+
|
| 541 |
+
A structured attack on the 1679 -> 1400 gap along the label-process and
|
| 542 |
+
count-error channels (user-provided plan). Every direction adjudicated:
|
| 543 |
+
|
| 544 |
+
| candidate | panel | verdict |
|
| 545 |
+
|---|---|---|
|
| 546 |
+
| label-arrival hazard (two-regime: logbag for not-crossed, KM P(EOL recorded in window \| crossed a days ago) for the death zone) | **2837.7 (+1158, 1 win/33)** | refuted: the classifier's p42 for crossed devices is ALREADY the correctly calibrated mixture - dies42 labels contain the arrival process; overriding ~0.15 with KM arrival ~0.4-0.9 inflates sum(p42), explodes N |
|
| 547 |
+
| + domain-split training (classifier never trained/queried in death zone) | **2938.1 (+1259)** | refuted: most positives ARE crossed near death - splitting starves the classifier of deaths |
|
| 548 |
+
| cost-weighted training (miss ~200 vs era FP price) | **2693.7 (+1015)** | refuted; worst exactly at early origins where FP weights are largest |
|
| 549 |
+
| oracle-N diagnostic (per-origin N sweep, real evaluator) | mean regret **222.6**/origin, median 185 | the pool is real but the oracle direction is era-inconsistent (winner's curse around a shallow optimum) |
|
| 550 |
+
| backtest-alpha corrector (pick alpha from last-K closed origins' replayed curves - simulated offline from the saved sweeps) | K=4: +113, K=6: +89, K=8: +36 vs fixed 0.8 | refuted: converges to picking 0.8 (18/26 at K=8); per-origin optimal alpha is unpredictable noise |
|
| 551 |
+
|
| 552 |
+
The label-process channel has now been attacked five ways across phases 15
|
| 553 |
+
and 17 (binary zeroing, hard quarantine, domain split, KM arrival hazard,
|
| 554 |
+
cost weighting) - each loses by hundreds. Conclusion: the bagged logistic's
|
| 555 |
+
implicit calibration of the label-recording process is better than every
|
| 556 |
+
explicit model of it we can impose, and the fixed CV-chosen count (alpha=0.8)
|
| 557 |
+
is better than every adaptive corrector tested.
|
| 558 |
+
|
| 559 |
+
Per the plan's own decision rule ("if the first two deliver under ~120
|
| 560 |
+
combined, 1400 on the panel likely isn't reachable"): they delivered negative
|
| 561 |
+
amounts; **1400 is not reachable on this panel without new information
|
| 562 |
+
channels**. The honest target remains deployment-era origins (ghost-dense,
|
| 563 |
+
cheap-FP), which the shipped mechanisms - adaptive per-origin refits,
|
| 564 |
+
break-even top-up, censoring-aware pricing, building day-splitting - already
|
| 565 |
+
serve, and whose panel deltas systematically understate their deployment
|
| 566 |
+
value. Champion unchanged: panel 1679.2, all-48 Docker 2146.9.
|
| 567 |
+
|
| 568 |
## Test-look ledger
|
| 569 |
|
| 570 |
8 looks total (now historical - see protocol change above); look 9 = the
|
batteryswap_example/models_cv.py
CHANGED
|
@@ -198,6 +198,77 @@ class SklearnRUL:
|
|
| 198 |
GHOST_ZERO = False # phase-15 candidate: exclude label-process ghosts from plans
|
| 199 |
QUARANTINE = False # phase-15 candidate: drop looks-dead-labeled-alive training rows
|
| 200 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 201 |
SILENT_V = 2.55
|
| 202 |
SILENT_FEATURES = ['rng_30', 'vstd_30', 'tau_projected', 'cohort_rel_age_m',
|
| 203 |
'age_days', 'v_corr_last']
|
|
@@ -456,6 +527,49 @@ class BlendDHAux:
|
|
| 456 |
return out
|
| 457 |
|
| 458 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 459 |
class WeightedBagLogistic:
|
| 460 |
"""Bagged logistic with recency-weighted samples: pairs from recent origins
|
| 461 |
count more, on the theory that fleet composition and degradation mix drift."""
|
|
@@ -867,6 +981,9 @@ MODELS = {
|
|
| 867 |
'cg_all': lambda: CohortGate('all'),
|
| 868 |
'cg_silent': lambda: CohortGate('silent'),
|
| 869 |
'logbag_dh': LogbagDH,
|
|
|
|
|
|
|
|
|
|
| 870 |
}
|
| 871 |
|
| 872 |
|
|
|
|
| 198 |
GHOST_ZERO = False # phase-15 candidate: exclude label-process ghosts from plans
|
| 199 |
QUARANTINE = False # phase-15 candidate: drop looks-dead-labeled-alive training rows
|
| 200 |
|
| 201 |
+
|
| 202 |
+
class LogbagLAH:
|
| 203 |
+
"""
|
| 204 |
+
Two-regime scorer for the label process (phase 17). The metric charges
|
| 205 |
+
RECORDED deaths, and EOL recording lags the physical 2.45-crossing by a
|
| 206 |
+
median ~34d (a third of recorded deaths lag >42d; ghosts never get one).
|
| 207 |
+
For devices already in the death zone the uncertainty is not "will it
|
| 208 |
+
die" but "will the label arrive in this window" - a pure function of
|
| 209 |
+
time-since-crossing. So:
|
| 210 |
+
not crossed -> champion logbag p42 (its real domain)
|
| 211 |
+
crossed and still low -> p42 = A(a) = P(EOL recorded in (a, a+42] | none
|
| 212 |
+
by a), a Kaplan-Meier on observed crossing->EOL lags (uncensored:
|
| 213 |
+
recorded deaths; censored: crossed devices with no EOL yet).
|
| 214 |
+
Unlike the phase-15 binary zero rule this keeps mid-lag devices included
|
| 215 |
+
(A ~ 0.3-0.6) and lets true ghosts fade gradually as their censored mass
|
| 216 |
+
flattens the curve. domain_split=True additionally trains the classifier
|
| 217 |
+
only on not-crossed rows, since it is never queried in the death zone.
|
| 218 |
+
"""
|
| 219 |
+
|
| 220 |
+
BUCKET = 14.0 # KM lag buckets (days); 82 events cannot support finer
|
| 221 |
+
|
| 222 |
+
def __init__(self, domain_split=False):
|
| 223 |
+
self.domain_split = domain_split
|
| 224 |
+
|
| 225 |
+
def fit(self, train):
|
| 226 |
+
crossed = train['days_below_245'].notna() & (train['days_below_245'] >= 0)
|
| 227 |
+
self.who = SklearnRUL('logbag').fit(train[~crossed] if self.domain_split
|
| 228 |
+
else train)
|
| 229 |
+
|
| 230 |
+
# one lag observation per device: crossing -> EOL (or censored at the
|
| 231 |
+
# last origin it appears). days_below_245 is leak-free per row.
|
| 232 |
+
sub = train[crossed]
|
| 233 |
+
per_dev = sub.sort_values('origin').groupby('device_id').tail(1)
|
| 234 |
+
cross_date = per_dev['origin'] - pd.to_timedelta(per_dev['days_below_245'], unit='D')
|
| 235 |
+
has_eol = per_dev['eol'].notna()
|
| 236 |
+
lag = np.where(has_eol,
|
| 237 |
+
(per_dev['eol'] - cross_date).dt.days,
|
| 238 |
+
per_dev['days_below_245'])
|
| 239 |
+
lag = np.clip(lag, 0, None)
|
| 240 |
+
event = has_eol.values
|
| 241 |
+
|
| 242 |
+
# discrete KM over lag buckets
|
| 243 |
+
k = (lag // self.BUCKET).astype(int)
|
| 244 |
+
kmax = int(k.max()) + 1 if len(k) else 1
|
| 245 |
+
surv, s = [], 1.0
|
| 246 |
+
for b in range(kmax):
|
| 247 |
+
at_risk = (k >= b).sum()
|
| 248 |
+
died = ((k == b) & event).sum()
|
| 249 |
+
s *= 1.0 - (died / at_risk if at_risk > 0 else 0.0)
|
| 250 |
+
surv.append(s)
|
| 251 |
+
self.surv = np.array(surv)
|
| 252 |
+
return self
|
| 253 |
+
|
| 254 |
+
def _arrival(self, a):
|
| 255 |
+
"""P(EOL recorded within the next 42d | crossed a days ago, none yet)."""
|
| 256 |
+
b0 = np.clip((a // self.BUCKET).astype(int), 0, len(self.surv) - 1)
|
| 257 |
+
b1 = np.clip(((a + WINDOW_DAYS) // self.BUCKET).astype(int), 0, len(self.surv) - 1)
|
| 258 |
+
s0 = self.surv[b0]
|
| 259 |
+
s1 = self.surv[b1]
|
| 260 |
+
return np.where(s0 > 1e-9, 1.0 - s1 / s0, 0.0)
|
| 261 |
+
|
| 262 |
+
def predict(self, rows):
|
| 263 |
+
out = self.who.predict(rows)
|
| 264 |
+
crossed = (rows['days_below_245'].notna()
|
| 265 |
+
& (rows['days_below_245'] >= 0)
|
| 266 |
+
& (rows['v_corr_last'] < 2.50)).values
|
| 267 |
+
if crossed.any():
|
| 268 |
+
a = rows.loc[crossed, 'days_below_245'].values
|
| 269 |
+
out.loc[crossed, 'p42'] = self._arrival(a)
|
| 270 |
+
return out
|
| 271 |
+
|
| 272 |
SILENT_V = 2.55
|
| 273 |
SILENT_FEATURES = ['rng_30', 'vstd_30', 'tau_projected', 'cohort_rel_age_m',
|
| 274 |
'age_days', 'v_corr_last']
|
|
|
|
| 527 |
return out
|
| 528 |
|
| 529 |
|
| 530 |
+
class CostWeightBag:
|
| 531 |
+
"""
|
| 532 |
+
Bagged logistic with DECISION-STAKE sample weights (phase 17): a missed
|
| 533 |
+
positive costs ~emergency (~200); a false inclusion costs its era's
|
| 534 |
+
false-swap price (0.5/day against the end_time+30 imputation, so early
|
| 535 |
+
origins' negatives matter ~10x more than near-data-end ones). Weights
|
| 536 |
+
normalized to mean 1 so the L2 scale (C=1) stays comparable to logbag.
|
| 537 |
+
"""
|
| 538 |
+
|
| 539 |
+
MISS_W = 200.0
|
| 540 |
+
|
| 541 |
+
def __init__(self, n_estimators=15):
|
| 542 |
+
self.n_estimators = n_estimators
|
| 543 |
+
|
| 544 |
+
def fit(self, train):
|
| 545 |
+
X, y = train[FEATURES], train['dies42']
|
| 546 |
+
data_end = train['origin'].max()
|
| 547 |
+
fp_price = (0.5 * ((data_end - train['origin']).dt.days + 30 - 21)
|
| 548 |
+
).clip(lower=1.0).values
|
| 549 |
+
w = np.where(y.values == 1, self.MISS_W, fp_price)
|
| 550 |
+
w = w / w.mean()
|
| 551 |
+
rng = np.random.RandomState(0)
|
| 552 |
+
self.models = []
|
| 553 |
+
n = len(X)
|
| 554 |
+
for _ in range(self.n_estimators):
|
| 555 |
+
idx = rng.randint(0, n, n)
|
| 556 |
+
m = make_pipeline(SimpleImputer(strategy='median'),
|
| 557 |
+
StandardScaler(),
|
| 558 |
+
LogisticRegression(max_iter=2000, C=1.0))
|
| 559 |
+
m.fit(X.iloc[idx], y.iloc[idx],
|
| 560 |
+
logisticregression__sample_weight=w[idx])
|
| 561 |
+
self.models.append(m)
|
| 562 |
+
dying = train[train['dies42'] == 1]
|
| 563 |
+
self.days = DayQuantiles().fit(dying[FEATURES], dying['days_to_eol'])
|
| 564 |
+
return self
|
| 565 |
+
|
| 566 |
+
def predict(self, rows):
|
| 567 |
+
X = rows[FEATURES]
|
| 568 |
+
out = self.days.predict(X)
|
| 569 |
+
out['p42'] = np.mean([m.predict_proba(X)[:, 1] for m in self.models], axis=0)
|
| 570 |
+
return out
|
| 571 |
+
|
| 572 |
+
|
| 573 |
class WeightedBagLogistic:
|
| 574 |
"""Bagged logistic with recency-weighted samples: pairs from recent origins
|
| 575 |
count more, on the theory that fleet composition and degradation mix drift."""
|
|
|
|
| 981 |
'cg_all': lambda: CohortGate('all'),
|
| 982 |
'cg_silent': lambda: CohortGate('silent'),
|
| 983 |
'logbag_dh': LogbagDH,
|
| 984 |
+
'logbag_lah': LogbagLAH,
|
| 985 |
+
'logbag_lah_ds': lambda: LogbagLAH(domain_split=True),
|
| 986 |
+
'logbag_cw': CostWeightBag,
|
| 987 |
}
|
| 988 |
|
| 989 |
|