Thibaut Forest
Claude Fable 5
phase 68: T5 global-optimum probe (6 lanes) - all rejected with mechanisms; 60d65da stands as final
3704c62
|
Raw History Blame Contribute Delete
6.01 kB
# T5 BASE-REBUILD lane: wide-feature stacked nonlinear base vs the linear order-primary — REJECTED (wash, not capacity-limited)
Date: 2026-08-23. Lane: BASE-REBUILD (T5 global-optimum probe).
Baseline: head config 60d65da, CV 1541.3 on bgkf (`/tmp/ref_head_1541.csv`;
parity re-verified through this lane's own driver: 1541.3, +0.0, swaps 2.27).
Champion order = .50*rank(logbag vm7) + .30*rank(WienerFHT) + .20*rank(gapHGB
on FEATURES+vmin7_rank+GAP4); values/counts always champion logbag (order-only
via blend w2=1.0).
## Hypothesis
The champion's order primary is a bagged LOGISTIC on ~14 hand-picked snapshot
features. The rank-4 public competitor's base is a stacked nonlinear ensemble
over a wide multi-scale feature space. Grafts onto our pipeline all failed
(saturated local optimum) — this lane REBUILT the base as a unit: wide table +
stacked ensemble, and tested it as order primary / full order / diversity
partner.
## Build
1. WIDE TABLE (`/tmp/c29/build_wide.py` -> `/tmp/c29/wide.parquet`): 78
feature cols per (device_id, origin_idx), all 168 origins (real+synthetic),
strictly as-of-origin from the daily corrected-voltage series (beta 0.005
fixed): rolling {3,7,14,30,60,90,180,365}d x {mean,min,max,std,coverage,
masked-OLS slope}; window contrasts (mean/min/slope 7-30, 14-90, 30-180,
60-365); days-since-min {30,90,180,365}; temp means {7,30,90};
v_last_c/data_gap_days; within-snapshot pct-ranks of v_last_c and 7d-min
globally/per-building/per-room; building/room device counts. Leak-freedom
verified by recomputing 25 random (device,origin) rows from the raw series
truncated at origin — 0 mismatches beyond float32 precision (2e-5).
Coverage >= 86% on every column.
2. STACKED BASE (`/tmp/c29/run.py` StackBase): learners = HGB(depth3, lr.06,
300it), HGB(depth2, lr.03, 600it), RF(350, min_leaf 5) on wide+FEATURES+
vmin7_rank+GAP4 (~96 cols), LogisticRegression on the champion's scaled
feature set; building-GroupKFold(4) OOF probabilities (deaths-sorted snake,
deterministic — the established contamination rule) -> LogisticRegression
(C=0.03) calibrator; learners refit on full fold-train. Fold-safe inside
.fit(train); memoized per outer fold. cv.py/models_cv.py md5-pinned per arm
(race guard); no repo files touched.
## Results (paired vs head 1541.3, 48 scenarios; bgkf, p_star=1.0, split 3, tw 2.0)
| arm | order | SCORE | delta±SE | sig | W/L | wilcoxon | swaps |
|---|---|---|---|---|---|---|---|
| parity | champion .50/.30/.20 rebuilt | 1541.3 | +0.0±0.0 | 0.4σ | 1W/1L | – | 2.27 |
| arm_a | stack replaces logbag at .50 (+.30 wn +.20 gap) | 1549.4 | +8.1±48.0 | 0.2σ | 25W/19L | p0.441 | 2.28 |
| arm_b | stack as FULL order (w 1.0) | 1533.1 | -8.2±55.1 | 0.1σ | 30W/17L | p0.295 | 2.26 |
| arm_c | .50*rank(champ composite) + .50*rank(stack) | 1532.2 | -9.2±36.6 | 0.3σ | 24W/19L | p0.583 | 2.28 |
No arm reaches 1σ (adoption bar for a new mechanism: 2σ). Weight mini-sweep
not triggered (rule: only if a/c < 0 at >= 1σ). Swaps flat 2.26-2.28
everywhere — count-flat as designed.
Block structure (delta means): every arm WINS warmup s0-7 (arm_a -78, arm_b
-262, arm_c -104) and LOSES the panel s8-41 (+27 / +48 / +9); nearend ~+13.
The tiny negative means are entirely the 8 warmup scenarios; the 34-scenario
panel prefers the champion. Deltas are heavy-tailed (arm_b: s21 +1545, s0
-810, s12 +794, s28 -757) — hence SE 37-55 and no resolvable effect.
## AUC comparison (dies42, building-OOF, closed rows at real origins)
| fold | n_test | deaths | logbag | STACK | wiener | gapHGB |
|---|---|---|---|---|---|---|
| 0 | 3056 | 123 | 0.962 | **0.974** | 0.929 | 0.969 |
| 1 | 1987 | 91 | 0.963 | **0.979** | 0.930 | 0.970 |
| 2 | 3661 | 80 | **0.968** | 0.962 | 0.939 | 0.974 |
| 3 | 967 | 71 | 0.837 | **0.930** | 0.833 | 0.893 |
| 4 | 7917 | 61 | 0.958 | **0.978** | 0.965 | 0.963 |
Stack beats logbag in 4/5 folds (mean per-fold 0.965 vs 0.938; +9.3 pts on
the hard fold 3, exactly where logbag collapses). Pooled-across-folds AUC is
lower for the stack (0.948 vs 0.957) only because its probability scale varies
more across fold fits — per-fold is the fair capacity read. Inner-OOF AUCs per
learner: h3 .947-.977, h2 .955-.977, rf .960-.977, lr .956-.972, meta .972-.981
— the linear learner is competitive with the trees even inside the wide space.
## Verdict
REJECTED — hypothesis answered, both halves:
1. **The capacity is real.** The rebuilt wide-space stacked base is a genuinely
better building-OOF dies42 classifier than the champion logbag (4/5 folds,
large margin where training deaths are scarce). The competitor-style base
CAN be built on this data and it does classify better. This is not a
feature-engineering or model-capacity failure.
2. **The capacity does not price.** At every mixing weight (0.5 primary, 1.0
full, 0.5 diversity ensemble) plan cost is a wash (|delta| <= 9.2, <= 0.3σ,
wilcoxon p >= 0.29). Its classification edge concentrates in warmup/sparse-
label regimes (fold 3, s0-7) where scenario costs are small; in the panel
block that dominates THE SCORE, logbag's calibrated top ranks near the
inclusion cutoff are already sufficient, and the stack's reorderings there
are net-slightly-negative. Same mechanism as the t4 stacking lane
(`t4_stack.md`): dies42-AUC gains live in separating obvious cases, cost
lives in the fine ordering near the cutoff.
The linear order-primary is genuinely sufficient at this data size (461
devices, ~1130 death-rows): base-model capacity is NOT the binding constraint
on bgkf cost. Do not revisit base-model family swaps as a cost lever; any
future gain must come from a signal that reorders the marginal band
specifically (or from count/timing machinery, which is closed/tiered
separately).
Artifacts: /tmp/c29/build_wide.py, /tmp/c29/wide.parquet, /tmp/c29/run.py,
/tmp/c29/arms.out (incl. full AUC diag), /tmp/cv_bgkfcand_c29_{parity,arm_a,
arm_b,arm_c}.csv.