File size: 8,490 Bytes
3704c62 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 | # T5 DATA-WALL: is label scarcity the binding constraint?
Baseline 60d65da champion, CV 1541.3 on bgkf (parity verified cell-exact vs
/tmp/ref_head_1541.csv: max per-origin diff 0.0). Champion = values
logbag(vm7), order .50 rank(logbag) + .30 rank(WienerFHT) + .20 rank(gapHGB),
planner p_star=1.0, split_days=3, timing_weight=2.0. All work in /tmp/c32/
(run.py driver, analyze.py + extrap.py analysis, cv_*.csv per-arm results,
arms.csv per-draw table, fold_arms.csv fold-level table). No repo file
touched; monkeypatch only; nothing committed.
## Design
Learning curve: a model-class wrapper drops fold-TRAIN buildings inside
fit() before refitting BOTH the values primary and all three order sources;
test folds, planner state, and evaluator identical to the champion run.
frac=1.0 through the wrapper reproduces 1541.3 exactly (bit-faithful, every
origin 0.0). Two samplers, both stratified on per-building train death
count:
* v1 (`f*`): death-sorted list split into n_keep contiguous strata, one
building drawn per stratum. Structural quirk found post-hoc: at frac .8
the multi-building strata all sit at the HIGH-death end, so drops
concentrate on high-death buildings (keeps only ~63% of deaths at 79% of
buildings). Kept as an informative composition contrast, not discarded.
* v2 (`g*`): drops evenly spaced across the death-sorted list (random
offset), so kept-death fraction tracks the building fraction. This is the
clean volume axis. 5 seeds at .4/.6/.8/.9.
Count-flat caveat (by design, per the tasking): subsampling changes the
primary's p42 values, so the count machinery moves within each arm (swaps
2.09-2.88 vs ref 2.27). The curve measures the WHOLE champion's data
response including that channel, not a values-frozen ordering response.
## 1. Learning curve
Per-draw deltas vs 1541.3 (SCORE units; x = actual fraction of the 320
fold-train unique deaths kept, aggregated over folds):
| nominal | n draws | mean SCORE | delta | draw scatter (sd) | per-draw deltas |
|---|---|---|---|---|---|
| 40% | 8 | 1684.6 | +143.3 | 131.4 | +379, -19, +246 (v1); +171, +98, +173, +108, -10 (v2) |
| 60% | 8 | 1626.3 | +85.0 | 66.4 | +136, +225, +11 (v1); +72, +63, +60, +51, +62 (v2) |
| 80% | 8 | 1657.1 | +115.8 | 66.6 | +146, +198, +211 (v1); +57, +138, +40, +72, +65 (v2) |
| 90% | 5 | 1612.0 | +70.6 | 47.3 | +45, +93, +11, +136, +68 (v2) |
| 100% | 1 | 1541.3 | 0.0 | - | parity, exact |
v2-only (clean volume axis): x=0.42 -> +108; x=0.62 -> +62; x=0.76 -> +74;
x=0.89 -> +71; x=1.00 -> 0.
The shape is NOT a smooth data-volume law. It is a shelf plus a cliff:
any subsample from 60% to 90% costs a roughly constant +60..+75, and the
local slope on the v2 shelf 0.6->0.9 is -29.9 +/- 70.1 per unit
death-fraction (p=0.68) - i.e. per +10% data: -3.0 +/- 7.0, statistically
zero. The drop to 0 happens only at exactly 100%, which is the
zero-perturbation parity point (the alpha=1.0 count appetite and every dial
were tuned on the full-data model); the shelf offset is perturbation +
composition cost, not marginal data value.
Composition dominates volume, three independent lines:
* Matched death-fraction ~0.63-0.65: v1 draws (drops hit high-death
buildings) +185.0 mean vs v2 draws (even drops) +80.7 mean. Same label
count, 2.3x the damage when the lost labels come from high-death
buildings.
* Two 40%-of-corpus draws MATCH full data (f40s1 -18.7, g40s4 -10.5): the
model family already saturates on a favorable 40% subset (~32-37% of
rows, ~39-42% of deaths).
* Fold-level paired regression (150 fold x arm points, delta fold cost vs
deaths dropped): slope +24.0 per dropped unique death (~ +0.5 SCORE per
death; p=0.009) but r=0.21 - deaths-dropped explains only ~4% of the
variance; which buildings dropped explains the rest.
## Extrapolation to 150% / 200% data (honest bounds)
Fits on all draws + the exact 100% anchor, x = kept-death fraction;
bootstrap resampled within levels, 3000 reps; gains measured against each
fit's OWN y(1) so the fit's anchor miss does not inflate them:
| fit (v2-only + anchor) | y(1.0) | gain at x=1.5 | gain at x=2.0 |
|---|---|---|---|
| power y = c + a*x^-b | 1586.8 (misses true 1541.3 by +45.5) | -25.7 [-47..0] | -43.7 [-80..0] |
| log y = b0 + b1*ln x | 1586.4 (miss +45.1) | -26.6 [-58..+3] | -45.5 [-100..+5] |
(All-draws fits: gain200 -35 [-103..-9] power, -90 [-162..-10] log; the log
fit is dragged steep by the v1 composition-skewed deep cuts and misses the
known anchor by +38 - treat it as an upper bound on optimism.)
Both parametric families overshoot the known 100% point by ~+45: the data
reject a smooth volume curve, and the smooth-fit gains are therefore
upper-ish bounds. Face value: +50% more same-kind data buys ~26 SCORE,
doubling buys ~44, with 95% bounds touching 0 in the clean (v2) fits.
The local shelf slope says the marginal value of same-kind data at the
current corpus size is indistinguishable from zero.
Big caveat the curve cannot price: "same-kind" is load-bearing. Data
containing NEW failure modes (the public fleet's uncatchable tail /
train-absent ghosts, public-measured 4x) is a distribution-shift question,
not a volume question - this probe says nothing about it.
## 2. Label audit
Ledger: 82 devices with an eol; 80 with eol <= last origin (2026-07-27);
2 died 2026-07-28 (after the last origin, before data end 07-31).
* bgkf convention (origin <= max-42d): ALL 80 observable deaths appear as
dies42=1 in at least one closed row. ZERO deaths lost. The window_closed
column convention likewise loses zero (82/82 incl. the two late deaths).
* walk_forward's comment ("every death <= origin lands in some closed
window") verified on the actual table: holds at 47/48 real origins;
violated once, transiently, at 2025-10-06 (1 of 12 deaths not yet in a
closed window - the 4d synthetic grid had not yet closed a window
containing it). Immaterial: fold train sets use the max-origin cut.
* Recoverable rows the convention DOES discard: 69 open-window rows with
dies42=1 (8 devices, of which only 2 devices - the 07-28 deaths - are
entirely absent from closed-window positives).
* Recovery arm (`aug`): champion refit on train + those recovered positive
rows (train buildings only), values + all order sources. Result: SCORE
1570.1, +28.8 +/- 24.1 vs 1541.3 (1.2 sigma), 15W/33L, Wilcoxon p=0.147.
Swaps 2.28 vs ref 2.27 - count effectively flat, so the (insignificant)
harm is not an appetite artifact. No usable signal is being discarded;
adding near-boundary positive rows without their matching negatives
slightly distorts the base rate for nothing.
## 3. Per-fold death count vs cost
Raw 5-fold regression (parity run): train_deaths 58/62/66/65/69 vs mean
test cost 426/263/336/291/224 -> slope -14.3 per death (r=-0.77, p=0.13),
but this is confounded: folds with fewer train deaths are exactly the folds
whose TEST buildings hold more deaths (22/18/14/15/11), i.e. harder fleets.
The clean version is the paired fold x arm regression above (+24.0 per
dropped death, p=0.009, r=0.21), which corroborates a real but weak volume
effect swamped by composition.
## VERDICT
Label volume is NOT the binding constraint; the champion model family is
data-saturated on this corpus. The learning curve is flat from 60% to 100%
of the data (local slope ~0, p=0.68), favorable 40% subsets already
reproduce 1541.3, and even the optimistic smooth-fit extrapolation prices a
DOUBLED corpus at ~-44 SCORE [-100..0] - far less than the method-search
deltas this repo routinely adjudicates, with bounds touching zero. The
label audit closes the other escape hatch: the training convention discards
essentially nothing (0 of 80 observable deaths lost; recovering the 69
discarded open-window rows is a wash at best, +28.8 +/- 24.1).
What the response surface actually rewards is COVERAGE - specific
high-death buildings carry non-redundant failure-mode information (same
label count, 2.3x damage when they are the ones missing). Consequences for
the T5 probe: (a) further same-kind data would not move the CV much, so the
global optimum is not gated on labels nobody has; (b) capacity/structure
search on this corpus is also near its ceiling for THIS champion family
(flat shelf = the extra 40% of data adds nothing the model can use);
(c) the remaining headroom, if any, lives in mechanisms that generalize
across failure modes / fleets (distribution shift), which neither this
curve nor more of the same labels can buy.
|