Phase 36: fit-protocol axis - warm-up mechanism validated (-49.8+/-6.0 across 6 seeds) but inert in deployment (static_uniq=82 vs gate 20)
Browse filesquar (drop 26 'looks dead, labeled survives' ghosts) and devw (device-equal
resampling) each improve warm-up substantially and wreck the panel; gated at
train_deaths<20 they leave panel/near-end byte-identical. Stacked (dq_g20):
all-48 1619.1 vs 1661.7, every one of 6 resampling seeds improves.
Not adopted. submit.py scores public,private - both held-out, both static
branch, train_uniq = static_uniq = 82 unique dead devices in the foreign
pool. Every gate tested is far below that, so the branch never runs where it
is scored. The whole warm-up regime (23.9% of the checksum) is absent from
the scored splits, which is why the cold-start floor's huge local gain
(2146.9->1851.0) never showed on public.
Deployment-faithful window is s_8-47, base 1518.0. Re-ranked all 86 screened
configs on it: no hidden winner, only the known s_39-46 tail artifacts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- EXPERIMENTS.md +73 -0
|
@@ -1663,3 +1663,76 @@ and the ONLY fields the organizers vary are `base_location` (16 values) and
|
|
| 1663 |
unobserved_eol_days, limits, overtime - is constant. So the hole is real but
|
| 1664 |
unlikely to fire. Left unwired pending a decision; models_cv.py is hash-pinned
|
| 1665 |
(exp_x28.py PINS) and the fix is not a measured gain.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1663 |
unobserved_eol_days, limits, overtime - is constant. So the hole is real but
|
| 1664 |
unlikely to fire. Left unwired pending a decision; models_cv.py is hash-pinned
|
| 1665 |
(exp_x28.py PINS) and the fix is not a measured gain.
|
| 1666 |
+
|
| 1667 |
+
## Phase 36 - the fit-protocol axis: a real warm-up mechanism that deployment never runs
|
| 1668 |
+
|
| 1669 |
+
Features, model families, hyperparameters and planner knobs are all swept out.
|
| 1670 |
+
The untouched axis was **how the training set is built and weighted**.
|
| 1671 |
+
Motivation: at s_0 the training set is ~10k rows holding 3-20 positives, each
|
| 1672 |
+
device contributes ~168 autocorrelated grid rows (so long-lived devices
|
| 1673 |
+
dominate by negative count), and `tau_projected` is entirely NaN there - the
|
| 1674 |
+
imputer silently drops it. `/tmp/screenfit.py`.
|
| 1675 |
+
|
| 1676 |
+
Ungated arms, full-fleet all-48 (base 1661.7):
|
| 1677 |
+
|
| 1678 |
+
| arm | warm-up | panel | all-48 |
|
| 1679 |
+
|---|---|---|---|
|
| 1680 |
+
| base | 2379.7 | 1679.2 | 1661.7 |
|
| 1681 |
+
| quar (QUARANTINE on) | **2202.4 (-177.3)** | 2172.7 | +395.0 |
|
| 1682 |
+
| devw (device-equal weight) | **2211.0 (-168.7)** | 2053.0 | +254.7 |
|
| 1683 |
+
| bal (class_weight balanced) | 2467.2 | 4709.4 | +2442.9 |
|
| 1684 |
+
| recw180 / recw365 | 2465.6 / 2505.1 | 1748.4 / 1747.0 | +51.5 / +64.8 |
|
| 1685 |
+
| thin2 / thin4 | 2401.4 / 2533.6 | 1697.4 / 1906.5 | +20.3 / +191.1 |
|
| 1686 |
+
| logcal | 2756.1 | 1772.6 | +139.6 |
|
| 1687 |
+
| ddepth3 | 2499.7 | 1680.9 | +16.7 |
|
| 1688 |
+
|
| 1689 |
+
Clean regime split: `quar` and `devw` help warm-up and wreck the panel. Mechanism
|
| 1690 |
+
is coherent - dropping the 26 "looks dead, labeled survives" ghosts and
|
| 1691 |
+
de-weighting long-lived devices removes label noise/imbalance that only bites
|
| 1692 |
+
when positives are scarce; with hundreds of positives those same rows are
|
| 1693 |
+
informative hard negatives.
|
| 1694 |
+
|
| 1695 |
+
**Gate them the way the floor is gated** (`train_deaths < 20`). Verified the
|
| 1696 |
+
gate splits exactly at s_0-7: gated arms leave panel (1679.2) and near-end
|
| 1697 |
+
(604.8) byte-identical.
|
| 1698 |
+
|
| 1699 |
+
| gated arm | warm-up | all-48 | delta | W/L/T |
|
| 1700 |
+
|---|---|---|---|---|
|
| 1701 |
+
| devw_g12 / g20 / g30 | 2318.1 / 2211.0 / 2211.0 | 1651.4 / 1633.5 / 1653.2 | -10.3 / **-28.1** / -8.4 | 6/2/40 at g20 |
|
| 1702 |
+
| quar_g12 / g20 / g30 | 2249.8 / 2202.4 / 2202.4 | 1640.0 / 1632.1 / 1663.9 | -21.6 / **-29.6** / +2.3 | 7/1/40 at g20 |
|
| 1703 |
+
| **dq_g20** (both stacked) | **2124.7** | **1619.1** | **-42.5** | 7/1/40 |
|
| 1704 |
+
| dq_g30 | 2124.7 | 1659.4 | -2.3 | 7/5/36 |
|
| 1705 |
+
|
| 1706 |
+
Gate 30 leaks into the panel (1736.0) and destroys the gain; 20 is the optimum,
|
| 1707 |
+
matching FLOOR_GATE. Seed stability for the `devw` resample (`/tmp/seedcheck.py`,
|
| 1708 |
+
6 seeds): warm-up 2081.2 +/- 36.0, all-48 delta **-49.8 +/- 6.0**, range
|
| 1709 |
+
[-58.8, -42.5]. **Every seed improves**, and the reported seed 0 is the worst
|
| 1710 |
+
of the six. The mechanism is real and not a resampling artifact.
|
| 1711 |
+
|
| 1712 |
+
### ...and it is inert in deployment
|
| 1713 |
+
|
| 1714 |
+
`submit.py` scores `public,private` only; both are held-out, so both take the
|
| 1715 |
+
static branch where `train_uniq = static_uniq` = unique dead devices in the
|
| 1716 |
+
full foreign train pool. **That number is 82.** Every gate tested (12/20/30/40)
|
| 1717 |
+
is far below it, so the branch never executes on a scored split.
|
| 1718 |
+
|
| 1719 |
+
This generalises beyond the candidate: **the whole warm-up regime - 23.9% of
|
| 1720 |
+
the all-48 checksum - is structurally absent from the scored splits**, and the
|
| 1721 |
+
existing cold-start floor has the same property. It retro-explains the record:
|
| 1722 |
+
the floor moved all-48 2146.9 -> 1851.0 (a huge local gain) while public did
|
| 1723 |
+
not move until phase 23's unmasked pooling landed independently.
|
| 1724 |
+
|
| 1725 |
+
**Consequence for measurement**: s_8-47 is the deployment-faithful window - the
|
| 1726 |
+
gate is shut there locally exactly as it is in deployment. Base on that metric
|
| 1727 |
+
is **1518.0**. dq_g20 is byte-identical on it.
|
| 1728 |
+
|
| 1729 |
+
Not adopted: inert where it is scored, and it would reset the checksum baseline
|
| 1730 |
+
and break comparability with every prior phase.
|
| 1731 |
+
|
| 1732 |
+
### Free re-analysis: all 86 screened configs re-ranked on s_8-47
|
| 1733 |
+
|
| 1734 |
+
If warm-up had been dragging a good config down, re-scoring on the deployment
|
| 1735 |
+
window would surface it. It does not. The only negatives are the same
|
| 1736 |
+
tail artifacts already rejected in phase 35 - ec0.5 -13.0 (still hostage to
|
| 1737 |
+
s_40 +389), ec0.7 -7.1, parkmob -6.2 (one scenario), ddepth3 -4.0 (21W/18L,
|
| 1738 |
+
noise). No hidden winner.
|