Thibaut Forest
phase 68: T5 global-optimum probe (6 lanes) - all rejected with mechanisms; 60d65da stands as final
3704c62
|
Raw History Blame Contribute Delete
6.01 kB

T5 BASE-REBUILD lane: wide-feature stacked nonlinear base vs the linear order-primary — REJECTED (wash, not capacity-limited)

Date: 2026-08-23. Lane: BASE-REBUILD (T5 global-optimum probe). Baseline: head config 60d65da, CV 1541.3 on bgkf (/tmp/ref_head_1541.csv; parity re-verified through this lane's own driver: 1541.3, +0.0, swaps 2.27). Champion order = .50rank(logbag vm7) + .30rank(WienerFHT) + .20*rank(gapHGB on FEATURES+vmin7_rank+GAP4); values/counts always champion logbag (order-only via blend w2=1.0).

Hypothesis

The champion's order primary is a bagged LOGISTIC on ~14 hand-picked snapshot features. The rank-4 public competitor's base is a stacked nonlinear ensemble over a wide multi-scale feature space. Grafts onto our pipeline all failed (saturated local optimum) — this lane REBUILT the base as a unit: wide table + stacked ensemble, and tested it as order primary / full order / diversity partner.

Build

  1. WIDE TABLE (/tmp/c29/build_wide.py -> /tmp/c29/wide.parquet): 78 feature cols per (device_id, origin_idx), all 168 origins (real+synthetic), strictly as-of-origin from the daily corrected-voltage series (beta 0.005 fixed): rolling {3,7,14,30,60,90,180,365}d x {mean,min,max,std,coverage, masked-OLS slope}; window contrasts (mean/min/slope 7-30, 14-90, 30-180, 60-365); days-since-min {30,90,180,365}; temp means {7,30,90}; v_last_c/data_gap_days; within-snapshot pct-ranks of v_last_c and 7d-min globally/per-building/per-room; building/room device counts. Leak-freedom verified by recomputing 25 random (device,origin) rows from the raw series truncated at origin — 0 mismatches beyond float32 precision (2e-5). Coverage >= 86% on every column.

  2. STACKED BASE (/tmp/c29/run.py StackBase): learners = HGB(depth3, lr.06, 300it), HGB(depth2, lr.03, 600it), RF(350, min_leaf 5) on wide+FEATURES+ vmin7_rank+GAP4 (~96 cols), LogisticRegression on the champion's scaled feature set; building-GroupKFold(4) OOF probabilities (deaths-sorted snake, deterministic — the established contamination rule) -> LogisticRegression (C=0.03) calibrator; learners refit on full fold-train. Fold-safe inside .fit(train); memoized per outer fold. cv.py/models_cv.py md5-pinned per arm (race guard); no repo files touched.

Results (paired vs head 1541.3, 48 scenarios; bgkf, p_star=1.0, split 3, tw 2.0)

arm order SCORE delta±SE sig W/L wilcoxon swaps
parity champion .50/.30/.20 rebuilt 1541.3 +0.0±0.0 0.4σ 1W/1L – 2.27
arm_a stack replaces logbag at .50 (+.30 wn +.20 gap) 1549.4 +8.1±48.0 0.2σ 25W/19L p0.441 2.28
arm_b stack as FULL order (w 1.0) 1533.1 -8.2±55.1 0.1σ 30W/17L p0.295 2.26
arm_c .50rank(champ composite) + .50rank(stack) 1532.2 -9.2±36.6 0.3σ 24W/19L p0.583 2.28

No arm reaches 1σ (adoption bar for a new mechanism: 2σ). Weight mini-sweep not triggered (rule: only if a/c < 0 at >= 1σ). Swaps flat 2.26-2.28 everywhere — count-flat as designed.

Block structure (delta means): every arm WINS warmup s0-7 (arm_a -78, arm_b -262, arm_c -104) and LOSES the panel s8-41 (+27 / +48 / +9); nearend ~+13. The tiny negative means are entirely the 8 warmup scenarios; the 34-scenario panel prefers the champion. Deltas are heavy-tailed (arm_b: s21 +1545, s0 -810, s12 +794, s28 -757) — hence SE 37-55 and no resolvable effect.

AUC comparison (dies42, building-OOF, closed rows at real origins)

fold n_test deaths logbag STACK wiener gapHGB
0 3056 123 0.962 0.974 0.929 0.969
1 1987 91 0.963 0.979 0.930 0.970
2 3661 80 0.968 0.962 0.939 0.974
3 967 71 0.837 0.930 0.833 0.893
4 7917 61 0.958 0.978 0.965 0.963

Stack beats logbag in 4/5 folds (mean per-fold 0.965 vs 0.938; +9.3 pts on the hard fold 3, exactly where logbag collapses). Pooled-across-folds AUC is lower for the stack (0.948 vs 0.957) only because its probability scale varies more across fold fits — per-fold is the fair capacity read. Inner-OOF AUCs per learner: h3 .947-.977, h2 .955-.977, rf .960-.977, lr .956-.972, meta .972-.981 — the linear learner is competitive with the trees even inside the wide space.

Verdict

REJECTED — hypothesis answered, both halves:

  1. The capacity is real. The rebuilt wide-space stacked base is a genuinely better building-OOF dies42 classifier than the champion logbag (4/5 folds, large margin where training deaths are scarce). The competitor-style base CAN be built on this data and it does classify better. This is not a feature-engineering or model-capacity failure.

  2. The capacity does not price. At every mixing weight (0.5 primary, 1.0 full, 0.5 diversity ensemble) plan cost is a wash (|delta| <= 9.2, <= 0.3σ, wilcoxon p >= 0.29). Its classification edge concentrates in warmup/sparse- label regimes (fold 3, s0-7) where scenario costs are small; in the panel block that dominates THE SCORE, logbag's calibrated top ranks near the inclusion cutoff are already sufficient, and the stack's reorderings there are net-slightly-negative. Same mechanism as the t4 stacking lane (t4_stack.md): dies42-AUC gains live in separating obvious cases, cost lives in the fine ordering near the cutoff.

The linear order-primary is genuinely sufficient at this data size (461 devices, ~1130 death-rows): base-model capacity is NOT the binding constraint on bgkf cost. Do not revisit base-model family swaps as a cost lever; any future gain must come from a signal that reorders the marginal band specifically (or from count/timing machinery, which is closed/tiered separately).

Artifacts: /tmp/c29/build_wide.py, /tmp/c29/wide.parquet, /tmp/c29/run.py, /tmp/c29/arms.out (incl. full AUC diag), /tmp/cv_bgkfcand_c29_{parity,arm_a, arm_b,arm_c}.csv.