Thibaut Forest Claude Fable 5 commited on
Commit
c64ca4a
·
1 Parent(s): 7d93fb8

Phase 36: fit-protocol axis - warm-up mechanism validated (-49.8+/-6.0 across 6 seeds) but inert in deployment (static_uniq=82 vs gate 20)

Browse files

quar (drop 26 'looks dead, labeled survives' ghosts) and devw (device-equal
resampling) each improve warm-up substantially and wreck the panel; gated at
train_deaths<20 they leave panel/near-end byte-identical. Stacked (dq_g20):
all-48 1619.1 vs 1661.7, every one of 6 resampling seeds improves.

Not adopted. submit.py scores public,private - both held-out, both static
branch, train_uniq = static_uniq = 82 unique dead devices in the foreign
pool. Every gate tested is far below that, so the branch never runs where it
is scored. The whole warm-up regime (23.9% of the checksum) is absent from
the scored splits, which is why the cold-start floor's huge local gain
(2146.9->1851.0) never showed on public.

Deployment-faithful window is s_8-47, base 1518.0. Re-ranked all 86 screened
configs on it: no hidden winner, only the known s_39-46 tail artifacts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Files changed (1) hide show
  1. EXPERIMENTS.md +73 -0
EXPERIMENTS.md CHANGED
@@ -1663,3 +1663,76 @@ and the ONLY fields the organizers vary are `base_location` (16 values) and
1663
  unobserved_eol_days, limits, overtime - is constant. So the hole is real but
1664
  unlikely to fire. Left unwired pending a decision; models_cv.py is hash-pinned
1665
  (exp_x28.py PINS) and the fix is not a measured gain.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1663
  unobserved_eol_days, limits, overtime - is constant. So the hole is real but
1664
  unlikely to fire. Left unwired pending a decision; models_cv.py is hash-pinned
1665
  (exp_x28.py PINS) and the fix is not a measured gain.
1666
+
1667
+ ## Phase 36 - the fit-protocol axis: a real warm-up mechanism that deployment never runs
1668
+
1669
+ Features, model families, hyperparameters and planner knobs are all swept out.
1670
+ The untouched axis was **how the training set is built and weighted**.
1671
+ Motivation: at s_0 the training set is ~10k rows holding 3-20 positives, each
1672
+ device contributes ~168 autocorrelated grid rows (so long-lived devices
1673
+ dominate by negative count), and `tau_projected` is entirely NaN there - the
1674
+ imputer silently drops it. `/tmp/screenfit.py`.
1675
+
1676
+ Ungated arms, full-fleet all-48 (base 1661.7):
1677
+
1678
+ | arm | warm-up | panel | all-48 |
1679
+ |---|---|---|---|
1680
+ | base | 2379.7 | 1679.2 | 1661.7 |
1681
+ | quar (QUARANTINE on) | **2202.4 (-177.3)** | 2172.7 | +395.0 |
1682
+ | devw (device-equal weight) | **2211.0 (-168.7)** | 2053.0 | +254.7 |
1683
+ | bal (class_weight balanced) | 2467.2 | 4709.4 | +2442.9 |
1684
+ | recw180 / recw365 | 2465.6 / 2505.1 | 1748.4 / 1747.0 | +51.5 / +64.8 |
1685
+ | thin2 / thin4 | 2401.4 / 2533.6 | 1697.4 / 1906.5 | +20.3 / +191.1 |
1686
+ | logcal | 2756.1 | 1772.6 | +139.6 |
1687
+ | ddepth3 | 2499.7 | 1680.9 | +16.7 |
1688
+
1689
+ Clean regime split: `quar` and `devw` help warm-up and wreck the panel. Mechanism
1690
+ is coherent - dropping the 26 "looks dead, labeled survives" ghosts and
1691
+ de-weighting long-lived devices removes label noise/imbalance that only bites
1692
+ when positives are scarce; with hundreds of positives those same rows are
1693
+ informative hard negatives.
1694
+
1695
+ **Gate them the way the floor is gated** (`train_deaths < 20`). Verified the
1696
+ gate splits exactly at s_0-7: gated arms leave panel (1679.2) and near-end
1697
+ (604.8) byte-identical.
1698
+
1699
+ | gated arm | warm-up | all-48 | delta | W/L/T |
1700
+ |---|---|---|---|---|
1701
+ | devw_g12 / g20 / g30 | 2318.1 / 2211.0 / 2211.0 | 1651.4 / 1633.5 / 1653.2 | -10.3 / **-28.1** / -8.4 | 6/2/40 at g20 |
1702
+ | quar_g12 / g20 / g30 | 2249.8 / 2202.4 / 2202.4 | 1640.0 / 1632.1 / 1663.9 | -21.6 / **-29.6** / +2.3 | 7/1/40 at g20 |
1703
+ | **dq_g20** (both stacked) | **2124.7** | **1619.1** | **-42.5** | 7/1/40 |
1704
+ | dq_g30 | 2124.7 | 1659.4 | -2.3 | 7/5/36 |
1705
+
1706
+ Gate 30 leaks into the panel (1736.0) and destroys the gain; 20 is the optimum,
1707
+ matching FLOOR_GATE. Seed stability for the `devw` resample (`/tmp/seedcheck.py`,
1708
+ 6 seeds): warm-up 2081.2 +/- 36.0, all-48 delta **-49.8 +/- 6.0**, range
1709
+ [-58.8, -42.5]. **Every seed improves**, and the reported seed 0 is the worst
1710
+ of the six. The mechanism is real and not a resampling artifact.
1711
+
1712
+ ### ...and it is inert in deployment
1713
+
1714
+ `submit.py` scores `public,private` only; both are held-out, so both take the
1715
+ static branch where `train_uniq = static_uniq` = unique dead devices in the
1716
+ full foreign train pool. **That number is 82.** Every gate tested (12/20/30/40)
1717
+ is far below it, so the branch never executes on a scored split.
1718
+
1719
+ This generalises beyond the candidate: **the whole warm-up regime - 23.9% of
1720
+ the all-48 checksum - is structurally absent from the scored splits**, and the
1721
+ existing cold-start floor has the same property. It retro-explains the record:
1722
+ the floor moved all-48 2146.9 -> 1851.0 (a huge local gain) while public did
1723
+ not move until phase 23's unmasked pooling landed independently.
1724
+
1725
+ **Consequence for measurement**: s_8-47 is the deployment-faithful window - the
1726
+ gate is shut there locally exactly as it is in deployment. Base on that metric
1727
+ is **1518.0**. dq_g20 is byte-identical on it.
1728
+
1729
+ Not adopted: inert where it is scored, and it would reset the checksum baseline
1730
+ and break comparability with every prior phase.
1731
+
1732
+ ### Free re-analysis: all 86 screened configs re-ranked on s_8-47
1733
+
1734
+ If warm-up had been dragging a good config down, re-scoring on the deployment
1735
+ window would surface it. It does not. The only negatives are the same
1736
+ tail artifacts already rejected in phase 35 - ec0.5 -13.0 (still hostage to
1737
+ s_40 +389), ec0.7 -7.1, parkmob -6.2 (one scenario), ddepth3 -4.0 (21W/18L,
1738
+ noise). No hidden winner.