Thibaut Forest Claude Fable 5 commited on
Commit Β·
89df098
1
Parent(s): 8e0e67b
phase 71: pressure CV instrument fails calibration (limit-collision lottery ~13x signal); no instrument change, pick stays 4c93883
Browse files- EXPERIMENTS.md +24 -0
- agent_reports_bgkf/t6_pressure.md +74 -0
EXPERIMENTS.md
CHANGED
|
@@ -3596,3 +3596,27 @@ dials on pressure-matched fleets (whole-universe-as-one-fleet + stress
|
|
| 3596 |
fleets at 250-400 devices), not the pooled small-sub-fleet mean.
|
| 3597 |
FINAL SELECTION locked: 4c93883 (measured 1575.31). 60d65da/71bfff2
|
| 3598 |
only as a second slot. Never e4745d2 / 698401a.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3596 |
fleets at 250-400 devices), not the pooled small-sub-fleet mean.
|
| 3597 |
FINAL SELECTION locked: 4c93883 (measured 1575.31). 60d65da/71bfff2
|
| 3598 |
only as a second slot. Never e4745d2 / 698401a.
|
| 3599 |
+
|
| 3600 |
+
## Phase 71 - Pressure-matched CV instrument: CALIBRATION FAILED - no instrument change, no final round; selection stays 4c93883
|
| 3601 |
+
|
| 3602 |
+
User-approved attempt to upgrade the CV to pressure-matched evaluation
|
| 3603 |
+
(whole-24-building-universe as one fleet, OOF-stitched per-device
|
| 3604 |
+
predictions from the 5 fold fits) and re-run a packing-dial grid on it.
|
| 3605 |
+
Calibration gate: retrodict the public pair (B=tw1/sd7 1575.31 <
|
| 3606 |
+
A=tw2/sd3 1583.11, dOps +9.45). Result: PRESSURE(B)=1619.8,
|
| 3607 |
+
PRESSURE(A)=1615.9, A-B = -3.88+/-5.17 (22W/19L/7T), dOps -1.71 -
|
| 3608 |
+
WRONG SIGN on both total and ops; grid auto-stopped per protocol.
|
| 3609 |
+
Diagnosis (t6_pressure.md): at binding pressure the A-B delta is a
|
| 3610 |
+
daily/weekly-limit COLLISION LOTTERY - +/-100/200 quanta ~13x the +7.8
|
| 3611 |
+
target signal, landing on different origins under OOF-stitched vs
|
| 3612 |
+
leaky-fit predictions (per-origin deltas uncorrelated, r=-0.08) - with
|
| 3613 |
+
only 48 origins the one-fleet mean cannot be calibrated; any grid
|
| 3614 |
+
ranking would fit the lottery. Corollary: the public +7.8 itself is
|
| 3615 |
+
likely 1-2 collision events, i.e. the packing channel may be
|
| 3616 |
+
UNRESOLVABLE by any instrument at this scale; the measured public
|
| 3617 |
+
number is then the only defensible ranking, and it favors B.
|
| 3618 |
+
Redesign direction for future work (not attempted tonight):
|
| 3619 |
+
sign-frequency over a many-fleet stress ensemble rather than a
|
| 3620 |
+
one-fleet mean; stress validator ready at /tmp/c36/stress_winner.py.
|
| 3621 |
+
FINAL: cv.py UNCHANGED (an uncalibrated instrument would mislead);
|
| 3622 |
+
selection stays 4c93883 first, 60d65da/71bfff2 second slot only.
|
agent_reports_bgkf/t6_pressure.md
ADDED
|
@@ -0,0 +1,74 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# t6_pressure β pressure-matched CV instrument: CALIBRATION FAILED, grid stopped
|
| 2 |
+
|
| 3 |
+
**Instrument (built as specified).** PRESSURE SCORE = mean over all 48 origins
|
| 4 |
+
of the cost of planning the ENTIRE 24-building universe as ONE fleet, with
|
| 5 |
+
OOF-stitched predictions: 5 bgkf-fold fits of the deployed submit.Ft2RUL
|
| 6 |
+
(train-building closed pairs per fold; train_uniq = 58/62/66/65/69, floor
|
| 7 |
+
gate inert), each device scored by the model that did not train on its
|
| 8 |
+
building, predict() called on the full-fleet frame so fleet-context features
|
| 9 |
+
see the deployment fleet. Planner state per arm exactly as run_bgkf
|
| 10 |
+
(alpha=1.0, ec 0.8/25, floor_scale 2.5, day_q 0.05). 5 fits + stitched preds
|
| 11 |
+
shared across arms; one plan+evaluate per arm per origin. Runner:
|
| 12 |
+
/tmp/c36/pressure.py; per-origin per-component rows in /tmp/c36/arm_A.csv,
|
| 13 |
+
arm_B.csv; stream log /tmp/c36/results.log. Hashes of cv.py / models_cv.py /
|
| 14 |
+
submit.py pinned and verified unchanged (6b3460e9β¦ / ea340538β¦ / de7a17f7β¦).
|
| 15 |
+
|
| 16 |
+
## Step 1 β calibration: FAIL (wrong sign)
|
| 17 |
+
|
| 18 |
+
| arm | tw | sd | PRESSURE | Ξ vs B | SE | W/L/T | dOps | dTiming | swaps |
|
| 19 |
+
|---|---|---|---|---|---|---|---|---|---|
|
| 20 |
+
| B (4c93883) | 1.0 | 7 | 1619.8 | β | β | β | β | β | 11.40 |
|
| 21 |
+
| A (60d65da) | 2.0 | 3 | 1615.9 | **β3.88** | 5.17 | 22/19/7 | **β1.71** | β2.71 | 11.40 (flat) |
|
| 22 |
+
|
| 23 |
+
Public truth: AβB = **+7.8** with ops **+9.45**, timing β1.6. The instrument
|
| 24 |
+
gives the **wrong sign on both the total and the ops component**, so per the
|
| 25 |
+
protocol the 12-arm dial grid was stopped without running (results.log shows
|
| 26 |
+
the automatic stop).
|
| 27 |
+
|
| 28 |
+
## Why it fails β the delta is limit-collision noise, not a calibratable signal
|
| 29 |
+
|
| 30 |
+
Comparing per-origin AβB deltas from this OOF-stitched instrument against the
|
| 31 |
+
t5 leaky-fit monolith (same fleet, same dials, only the prediction vector
|
| 32 |
+
differs β one all-closed fit vs 5 OOF fits):
|
| 33 |
+
|
| 34 |
+
- correlation across the 48 origins: **r = β0.08** (essentially uncorrelated);
|
| 35 |
+
- the big daily/weekly-limit quanta (Β±100..Β±260) land on **different origins**:
|
| 36 |
+
OOF puts them at s34 (β208) and s38 (+100); the leaky fit put them at s16
|
| 37 |
+
(+214), s21 (+311), s18 (β234);
|
| 38 |
+
- the mean effect (|β3.9|) is far smaller than a single limit quantum and
|
| 39 |
+
below 1 SE (5.17); the leaky-fit variant of the same monolith gave +0.97Β±8.64.
|
| 40 |
+
|
| 41 |
+
So at monolithic-fleet pressure the A-vs-B dial delta is a daily/weekly-limit
|
| 42 |
+
collision lottery whose realization flips with any perturbation of the
|
| 43 |
+
prediction vector. A 24-building universe at 48 origins does not contain
|
| 44 |
+
enough independent pressure events to price a +7.8 public effect (one limit
|
| 45 |
+
collision = Β±100 β 13x the target signal). Any grid ranking computed on this
|
| 46 |
+
instrument would be fitting that lottery, not a transferable dial preference.
|
| 47 |
+
|
| 48 |
+
## Steps 2β3 β not run (per protocol)
|
| 49 |
+
|
| 50 |
+
Grid and stress validation were conditional on calibration; both skipped.
|
| 51 |
+
The prepared stress validator (/tmp/c36/stress_winner.py, reuses the six t5
|
| 52 |
+
18-building fleets) is ready if a differently-constructed instrument ever
|
| 53 |
+
calibrates.
|
| 54 |
+
|
| 55 |
+
## What this means for the dial decision
|
| 56 |
+
|
| 57 |
+
- This instrument provides **no valid evidence to move off the incumbent B
|
| 58 |
+
(tw=1.0, sd=7)**, and equally no pressure-side confirmation of A.
|
| 59 |
+
- The better-powered pressure read remains the t5 fleet ensemble
|
| 60 |
+
(agent_reports_bgkf/t5_fleetcomp.md): across 39 synthetic fleets the AβB
|
| 61 |
+
trade at pressure is a Β±10β25 ops lottery, sign quasi-random at high
|
| 62 |
+
pressure (3/7 high-pressure fleets reproduce the public loss signature).
|
| 63 |
+
Consistent with today's public measurement, the honest position is:
|
| 64 |
+
A-vs-B at deployment pressure is indistinguishable locally; the public
|
| 65 |
+
board (B = 1575.31 < A = 1583.11) is the only instrument that actually
|
| 66 |
+
ranks them, and it favors B.
|
| 67 |
+
- If a pressure instrument is wanted that CAN calibrate, it needs many more
|
| 68 |
+
independent pressure events than one fleet x 48 origins β e.g. the delta
|
| 69 |
+
distribution over an ensemble of large synthetic fleets (t5-style, more
|
| 70 |
+
fleets), scored by sign-frequency rather than mean.
|
| 71 |
+
|
| 72 |
+
Timeline: instrument built and calibrated within the deadline (calibration
|
| 73 |
+
line streamed to /tmp/c36/results.log at [~60s] into the run); grid stopped
|
| 74 |
+
at 2 arms as required.
|