Thibaut Forest Claude Fable 5 commited on
Commit
89df098
Β·
1 Parent(s): 8e0e67b

phase 71: pressure CV instrument fails calibration (limit-collision lottery ~13x signal); no instrument change, pick stays 4c93883

Browse files
Files changed (2) hide show
  1. EXPERIMENTS.md +24 -0
  2. agent_reports_bgkf/t6_pressure.md +74 -0
EXPERIMENTS.md CHANGED
@@ -3596,3 +3596,27 @@ dials on pressure-matched fleets (whole-universe-as-one-fleet + stress
3596
  fleets at 250-400 devices), not the pooled small-sub-fleet mean.
3597
  FINAL SELECTION locked: 4c93883 (measured 1575.31). 60d65da/71bfff2
3598
  only as a second slot. Never e4745d2 / 698401a.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3596
  fleets at 250-400 devices), not the pooled small-sub-fleet mean.
3597
  FINAL SELECTION locked: 4c93883 (measured 1575.31). 60d65da/71bfff2
3598
  only as a second slot. Never e4745d2 / 698401a.
3599
+
3600
+ ## Phase 71 - Pressure-matched CV instrument: CALIBRATION FAILED - no instrument change, no final round; selection stays 4c93883
3601
+
3602
+ User-approved attempt to upgrade the CV to pressure-matched evaluation
3603
+ (whole-24-building-universe as one fleet, OOF-stitched per-device
3604
+ predictions from the 5 fold fits) and re-run a packing-dial grid on it.
3605
+ Calibration gate: retrodict the public pair (B=tw1/sd7 1575.31 <
3606
+ A=tw2/sd3 1583.11, dOps +9.45). Result: PRESSURE(B)=1619.8,
3607
+ PRESSURE(A)=1615.9, A-B = -3.88+/-5.17 (22W/19L/7T), dOps -1.71 -
3608
+ WRONG SIGN on both total and ops; grid auto-stopped per protocol.
3609
+ Diagnosis (t6_pressure.md): at binding pressure the A-B delta is a
3610
+ daily/weekly-limit COLLISION LOTTERY - +/-100/200 quanta ~13x the +7.8
3611
+ target signal, landing on different origins under OOF-stitched vs
3612
+ leaky-fit predictions (per-origin deltas uncorrelated, r=-0.08) - with
3613
+ only 48 origins the one-fleet mean cannot be calibrated; any grid
3614
+ ranking would fit the lottery. Corollary: the public +7.8 itself is
3615
+ likely 1-2 collision events, i.e. the packing channel may be
3616
+ UNRESOLVABLE by any instrument at this scale; the measured public
3617
+ number is then the only defensible ranking, and it favors B.
3618
+ Redesign direction for future work (not attempted tonight):
3619
+ sign-frequency over a many-fleet stress ensemble rather than a
3620
+ one-fleet mean; stress validator ready at /tmp/c36/stress_winner.py.
3621
+ FINAL: cv.py UNCHANGED (an uncalibrated instrument would mislead);
3622
+ selection stays 4c93883 first, 60d65da/71bfff2 second slot only.
agent_reports_bgkf/t6_pressure.md ADDED
@@ -0,0 +1,74 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # t6_pressure β€” pressure-matched CV instrument: CALIBRATION FAILED, grid stopped
2
+
3
+ **Instrument (built as specified).** PRESSURE SCORE = mean over all 48 origins
4
+ of the cost of planning the ENTIRE 24-building universe as ONE fleet, with
5
+ OOF-stitched predictions: 5 bgkf-fold fits of the deployed submit.Ft2RUL
6
+ (train-building closed pairs per fold; train_uniq = 58/62/66/65/69, floor
7
+ gate inert), each device scored by the model that did not train on its
8
+ building, predict() called on the full-fleet frame so fleet-context features
9
+ see the deployment fleet. Planner state per arm exactly as run_bgkf
10
+ (alpha=1.0, ec 0.8/25, floor_scale 2.5, day_q 0.05). 5 fits + stitched preds
11
+ shared across arms; one plan+evaluate per arm per origin. Runner:
12
+ /tmp/c36/pressure.py; per-origin per-component rows in /tmp/c36/arm_A.csv,
13
+ arm_B.csv; stream log /tmp/c36/results.log. Hashes of cv.py / models_cv.py /
14
+ submit.py pinned and verified unchanged (6b3460e9… / ea340538… / de7a17f7…).
15
+
16
+ ## Step 1 β€” calibration: FAIL (wrong sign)
17
+
18
+ | arm | tw | sd | PRESSURE | Ξ” vs B | SE | W/L/T | dOps | dTiming | swaps |
19
+ |---|---|---|---|---|---|---|---|---|---|
20
+ | B (4c93883) | 1.0 | 7 | 1619.8 | β€” | β€” | β€” | β€” | β€” | 11.40 |
21
+ | A (60d65da) | 2.0 | 3 | 1615.9 | **βˆ’3.88** | 5.17 | 22/19/7 | **βˆ’1.71** | βˆ’2.71 | 11.40 (flat) |
22
+
23
+ Public truth: Aβˆ’B = **+7.8** with ops **+9.45**, timing βˆ’1.6. The instrument
24
+ gives the **wrong sign on both the total and the ops component**, so per the
25
+ protocol the 12-arm dial grid was stopped without running (results.log shows
26
+ the automatic stop).
27
+
28
+ ## Why it fails β€” the delta is limit-collision noise, not a calibratable signal
29
+
30
+ Comparing per-origin Aβˆ’B deltas from this OOF-stitched instrument against the
31
+ t5 leaky-fit monolith (same fleet, same dials, only the prediction vector
32
+ differs β€” one all-closed fit vs 5 OOF fits):
33
+
34
+ - correlation across the 48 origins: **r = βˆ’0.08** (essentially uncorrelated);
35
+ - the big daily/weekly-limit quanta (Β±100..Β±260) land on **different origins**:
36
+ OOF puts them at s34 (βˆ’208) and s38 (+100); the leaky fit put them at s16
37
+ (+214), s21 (+311), s18 (βˆ’234);
38
+ - the mean effect (|βˆ’3.9|) is far smaller than a single limit quantum and
39
+ below 1 SE (5.17); the leaky-fit variant of the same monolith gave +0.97Β±8.64.
40
+
41
+ So at monolithic-fleet pressure the A-vs-B dial delta is a daily/weekly-limit
42
+ collision lottery whose realization flips with any perturbation of the
43
+ prediction vector. A 24-building universe at 48 origins does not contain
44
+ enough independent pressure events to price a +7.8 public effect (one limit
45
+ collision = Β±100 β‰ˆ 13x the target signal). Any grid ranking computed on this
46
+ instrument would be fitting that lottery, not a transferable dial preference.
47
+
48
+ ## Steps 2–3 β€” not run (per protocol)
49
+
50
+ Grid and stress validation were conditional on calibration; both skipped.
51
+ The prepared stress validator (/tmp/c36/stress_winner.py, reuses the six t5
52
+ 18-building fleets) is ready if a differently-constructed instrument ever
53
+ calibrates.
54
+
55
+ ## What this means for the dial decision
56
+
57
+ - This instrument provides **no valid evidence to move off the incumbent B
58
+ (tw=1.0, sd=7)**, and equally no pressure-side confirmation of A.
59
+ - The better-powered pressure read remains the t5 fleet ensemble
60
+ (agent_reports_bgkf/t5_fleetcomp.md): across 39 synthetic fleets the Aβˆ’B
61
+ trade at pressure is a Β±10–25 ops lottery, sign quasi-random at high
62
+ pressure (3/7 high-pressure fleets reproduce the public loss signature).
63
+ Consistent with today's public measurement, the honest position is:
64
+ A-vs-B at deployment pressure is indistinguishable locally; the public
65
+ board (B = 1575.31 < A = 1583.11) is the only instrument that actually
66
+ ranks them, and it favors B.
67
+ - If a pressure instrument is wanted that CAN calibrate, it needs many more
68
+ independent pressure events than one fleet x 48 origins β€” e.g. the delta
69
+ distribution over an ensemble of large synthetic fleets (t5-style, more
70
+ fleets), scored by sign-frequency rather than mean.
71
+
72
+ Timeline: instrument built and calibrated within the deadline (calibration
73
+ line streamed to /tmp/c36/results.log at [~60s] into the run); grid stopped
74
+ at 2 arms as required.