BatterySwapAI2026-Example

Model card Files Files and versions

xet

Community

BatterySwapAI2026-Example / agent_reports_bgkf /t5_fleetcomp.md

Thibaut Forest

Claude Fable 5

phase 70: fleet-composition probe - CV packing win is a low-pressure artifact; selection locks to 4c93883

8e0e67b 3 months ago

preview code

Raw History Blame Contribute Delete

7.05 kB

# t5_fleetcomp — FLEET-COMPOSITION probe of the A-vs-B planner-dial trade
**Question.** Config A (timing_weight=2.0, split_days=3; lam2sp3 head) vs Config B
(timing_weight=1.0, split_days=7; old champion 4c93883). Same model, same counts,
same ordering — only the two packing dials differ. CV says A wins −4.1±1.4;
public says A loses +7.8 (swaps identical, late/early −1.6 better, ops +9.45
worse). Is the CV win composition-robust?
**Verdict: A's win is composition-fragile — the public +7.8 is unsurprising.**
A's CV edge is earned in a low-pressure regime (five ~5-building fold fleets
where daily/weekly limits almost never bind). Under public-like fleet pressure
the A−B trade becomes an ops-limit lottery with fleet-level swings of ±10–25,
and 3 of 7 high-pressure synthetic fleets — including the monolithic
24-building universe itself — reproduce the public loss with the same
component signature (timing slightly better, ops ~+7..+11 worse).
Method: reused cv.run_bgkf machinery verbatim (same table build, complement
fit with the deployed submit.Ft2RUL, deployment-exact planner state, real
evaluator, floor gate as in run_bgkf); one fit per fleet shared across arms,
paired per fleet×origin. Runner and raw CSVs in /tmp/c35/. Module hashes
(cv.py 6b3460e9…, models_cv.py ea340538…) verified unchanged before and after.

## Stage 1 — the existing CV comparison, split by fold fleet (48 origins each)

Canonical pair: /tmp/cv_bgkfcand_ft2a10.csv (A) vs /tmp/cv_bgkfcand_ft2a10_sp7.csv
(B); reproduces the known ALL delta −4.11±1.41, 25W/4L exactly. (Note:
cv_bgkfcand_head.csv at 19:25 is a stale model state — 42 cells differ from
ft2a10.csv; do not pair against it.)
| fold | dTotal | W/L/T (48) | dLate | dEarly | dOps |
|---|---|---|---|---|---|
| 0 | −0.60 | 8/2/38 | −0.42 | −0.45 | +0.14 |
| 1 | −0.74 | 12/3/33 | −0.62 | −0.36 | +0.11 |
| 2 | −0.95 | 3/0/45 | −0.83 | +0.04 | −0.16 |
| 3 | −0.02 | 1/1/46 | 0.00 | −0.07 | +0.04 |
| 4 | −1.80 | 12/8/28 | −1.67 | −0.36 | +0.13 |
No fold-level sign flip, but only 50/240 cells are nonzero, and **A already
pays a small ops tax in 4 of 5 fold fleets** (ops-worse on 46% of origins) —
the entire CV win is late/early. The public signature (ops-dominated) is the
same trade at ~40x the ops magnitude.
## Stage 2 — 32 random building subsets as standalone fleets (sizes 3/5/8/12, all 48 origins, paired)
1536 paired cells; 432 nonzero; 157 A-loss cells; **swap counts identical in
every single cell** (the dials are count-flat everywhere). Fleet-level
(mean over 48 origins): **A wins 20 / loses 8 / ties 4**; mean −2.10,
median −0.14; all 8 losses tiny (max +0.95).
Pressure activates the trade — P(cell differs at all) and mean delta by fleet
device count:
| fleet devices | cells | P(nonzero) | P(A-loss) | mean dt |
|---|---|---|---|---|
| 0–50 | 617 | 5.5% | 1.3% | −0.04 |
| 50–100 | 180 | 19% | 6.7% | −0.35 |
| 100–200 | 376 | 24% | 9.3% | −0.62 |
| 200–300 | 310 | 74% | 28% | −7.01 |
| 300–400 | 53 | 85% | 30% | −13.7 |
Composition predictors of the fleet-level delta: corr(dt, n_dev) = −0.71,
n_bldg −0.64, max building size −0.59, n_remote −0.51. So on AVERAGE bigger
fleets favor A — but the variance explodes with size: cell deltas are late-swap
quanta of ±10..±80 (a timing lottery on which deaths each packing catches) plus
rare ±100/±200 daily/weekly-limit quanta (e.g. fleet n12_4 saw dOps −214 at two
origins: B tripped daily+weekly limits A avoided).
## Stage 3 — pressure stress: 18-building fleets and the 24-building monolith (48 origins)
Six random 18-building fleets (250–384 devices; ≥15/24 buildings shared
pairwise, yet the sign flips fleet-to-fleet):
| fleet | n_dev | dt | dOps | dLate | per-origin W/L |
|---|---|---|---|---|---|
| n18_1 | 352 | **−24.24** | −21.89 | −1.46 | 20/23 |
| n18_2 | 251 | −5.25 | −4.23 | −0.83 | 15/9 |
| n18_4 | 353 | −3.73 | −2.94 | −0.21 | 23/16 |
| n18_5 | 322 | −1.58 | −5.43 | +5.00 | 19/19 |
| n18_0 | 384 | **+7.90** | **+10.82** | −3.54 | 20/28 |
| n18_3 | 349 | **+9.31** | **+9.35** | −0.42 | 21/26 |

**3 of 6 flip to A-loses, and the two big losses are carbon copies of the
public measurement** (+7.9 with ops +10.8; +9.3 with ops +9.4; timing slightly
better in both — public: +7.8, ops +9.45, late/early −1.6).

**n24_all — the exact CV 24-building universe planned as ONE fleet instead of
five fold sub-fleets: mean +0.97 (24W/20L), dLate −4.38, dEarly −1.62,
dOps +6.47 (daily +2.08, weekly +4.17, travel +0.36).** The same buildings that
give A a 2.9σ win when planned as five small fleets give a coin-flip-to-loss
with the public's ops signature when planned as one fleet. Worst origins:
s21 +257 (ops +311), s16 +192 (ops +214) — pure daily/weekly-limit quanta.
(Caveat: complement is empty for this fleet, so the fit is self-fit — leaky,
but identical in both arms; the paired planner delta stands.)
Across all 39 synthetic fleets: A loses in 11/39 (28%); among the 15 fleets
with ≥200 devices, 3 losses, all with the ops signature.
## Which composition features predict the sign
1. **Constraint pressure (total fleet devices) is the master variable** — it
   predicts whether the dials matter at all (5% → 85% of origins) and blows up
   the variance. Below ~100 devices A≈B almost everywhere.
2. At high pressure the sign is a **daily/weekly-limit collision lottery**
   (±100/±200 quanta), quasi-random in composition; weakly suggestive (n=6,
   anecdotal): including the two death-densest mid-size buildings
   (b_9a5a45d6e498 24dev/15deaths, b_a0e6376410cb 28dev/11deaths) correlates
   with A-worse ops (membership-dOps corr +0.65/+0.74).
3. Remote buildings and the 117-device giant increase activation but do not
   fix the sign (fleets containing the giant span dt −19.9 to +8.8 at n≤12,
   and the giant is absent from both n18 losers).
## Implications
- The CV structurally cannot price this trade: its test fleets are ~5-building
  sub-fleets where limits don't bind. The public fleet is a single large fleet
  where they do. The n24 monolith is the cheapest local proxy for that regime,
  and it sides with the public (A ≈ +1, ops-worse).
- The dial choice (split3/tw2 vs split7/tw1) is a ~±10–25 ops lottery on any
  large unseen fleet; the −4.1 CV delta should not be treated as transferable
  evidence for A on the private split. On pressure-matched local evidence the
  two configs are statistically indistinguishable (n24: +0.97±8.64).
Raw data: /tmp/c35/stage1_merged.csv, stage2_all48.csv (+ per-fleet
stage2_perfleet_48orig.csv, cells stage2_cells_48orig.csv), stage3_stress.csv
(+ cells), stage3_n24.csv (+ cells), building_meta.csv, fleet defs, and the
runner fleetprobe.py. Stage-2 6-origin pilot (stage2_results.csv) superseded
by the all-48 rerun. Origins for stress/all48: all 48; fits per fleet on the
complement buildings' closed pairs (deployment-faithful), shared across arms.

	# t5_fleetcomp — FLEET-COMPOSITION probe of the A-vs-B planner-dial trade

	Question. Config A (timing_weight=2.0, split_days=3; lam2sp3 head) vs Config B
	(timing_weight=1.0, split_days=7; old champion 4c93883). Same model, same counts,
	same ordering — only the two packing dials differ. CV says A wins −4.1±1.4;
	public says A loses +7.8 (swaps identical, late/early −1.6 better, ops +9.45
	worse). Is the CV win composition-robust?

	Verdict: A's win is composition-fragile — the public +7.8 is unsurprising.
	A's CV edge is earned in a low-pressure regime (five ~5-building fold fleets
	where daily/weekly limits almost never bind). Under public-like fleet pressure
	the A−B trade becomes an ops-limit lottery with fleet-level swings of ±10–25,
	and 3 of 7 high-pressure synthetic fleets — including the monolithic
	24-building universe itself — reproduce the public loss with the same
	component signature (timing slightly better, ops ~+7..+11 worse).

	Method: reused cv.run_bgkf machinery verbatim (same table build, complement
	fit with the deployed submit.Ft2RUL, deployment-exact planner state, real
	evaluator, floor gate as in run_bgkf); one fit per fleet shared across arms,
	paired per fleet×origin. Runner and raw CSVs in /tmp/c35/. Module hashes
	(cv.py 6b3460e9…, models_cv.py ea340538…) verified unchanged before and after.

	## Stage 1 — the existing CV comparison, split by fold fleet (48 origins each)

	Canonical pair: /tmp/cv_bgkfcand_ft2a10.csv (A) vs /tmp/cv_bgkfcand_ft2a10_sp7.csv
	(B); reproduces the known ALL delta −4.11±1.41, 25W/4L exactly. (Note:
	cv_bgkfcand_head.csv at 19:25 is a stale model state — 42 cells differ from
	ft2a10.csv; do not pair against it.)

	\| fold \| dTotal \| W/L/T (48) \| dLate \| dEarly \| dOps \|
	\|---\|---\|---\|---\|---\|---\|
	\| 0 \| −0.60 \| 8/2/38 \| −0.42 \| −0.45 \| +0.14 \|
	\| 1 \| −0.74 \| 12/3/33 \| −0.62 \| −0.36 \| +0.11 \|
	\| 2 \| −0.95 \| 3/0/45 \| −0.83 \| +0.04 \| −0.16 \|
	\| 3 \| −0.02 \| 1/1/46 \| 0.00 \| −0.07 \| +0.04 \|
	\| 4 \| −1.80 \| 12/8/28 \| −1.67 \| −0.36 \| +0.13 \|

	No fold-level sign flip, but only 50/240 cells are nonzero, and **A already
	pays a small ops tax in 4 of 5 fold fleets** (ops-worse on 46% of origins) —
	the entire CV win is late/early. The public signature (ops-dominated) is the
	same trade at ~40x the ops magnitude.

	## Stage 2 — 32 random building subsets as standalone fleets (sizes 3/5/8/12, all 48 origins, paired)

	1536 paired cells; 432 nonzero; 157 A-loss cells; **swap counts identical in
	every single cell** (the dials are count-flat everywhere). Fleet-level
	(mean over 48 origins): A wins 20 / loses 8 / ties 4; mean −2.10,
	median −0.14; all 8 losses tiny (max +0.95).

	Pressure activates the trade — P(cell differs at all) and mean delta by fleet
	device count:

	\| fleet devices \| cells \| P(nonzero) \| P(A-loss) \| mean dt \|
	\|---\|---\|---\|---\|---\|
	\| 0–50 \| 617 \| 5.5% \| 1.3% \| −0.04 \|
	\| 50–100 \| 180 \| 19% \| 6.7% \| −0.35 \|
	\| 100–200 \| 376 \| 24% \| 9.3% \| −0.62 \|
	\| 200–300 \| 310 \| 74% \| 28% \| −7.01 \|
	\| 300–400 \| 53 \| 85% \| 30% \| −13.7 \|

	Composition predictors of the fleet-level delta: corr(dt, n_dev) = −0.71,
	n_bldg −0.64, max building size −0.59, n_remote −0.51. So on AVERAGE bigger
	fleets favor A — but the variance explodes with size: cell deltas are late-swap
	quanta of ±10..±80 (a timing lottery on which deaths each packing catches) plus
	rare ±100/±200 daily/weekly-limit quanta (e.g. fleet n12_4 saw dOps −214 at two
	origins: B tripped daily+weekly limits A avoided).

	## Stage 3 — pressure stress: 18-building fleets and the 24-building monolith (48 origins)

	Six random 18-building fleets (250–384 devices; ≥15/24 buildings shared
	pairwise, yet the sign flips fleet-to-fleet):

	\| fleet \| n_dev \| dt \| dOps \| dLate \| per-origin W/L \|
	\|---\|---\|---\|---\|---\|---\|
	\| n18_1 \| 352 \| −24.24 \| −21.89 \| −1.46 \| 20/23 \|
	\| n18_2 \| 251 \| −5.25 \| −4.23 \| −0.83 \| 15/9 \|
	\| n18_4 \| 353 \| −3.73 \| −2.94 \| −0.21 \| 23/16 \|
	\| n18_5 \| 322 \| −1.58 \| −5.43 \| +5.00 \| 19/19 \|
	\| n18_0 \| 384 \| +7.90 \| +10.82 \| −3.54 \| 20/28 \|
	\| n18_3 \| 349 \| +9.31 \| +9.35 \| −0.42 \| 21/26 \|

	**3 of 6 flip to A-loses, and the two big losses are carbon copies of the
	public measurement** (+7.9 with ops +10.8; +9.3 with ops +9.4; timing slightly
	better in both — public: +7.8, ops +9.45, late/early −1.6).

	**n24_all — the exact CV 24-building universe planned as ONE fleet instead of
	five fold sub-fleets: mean +0.97 (24W/20L), dLate −4.38, dEarly −1.62,
	dOps +6.47 (daily +2.08, weekly +4.17, travel +0.36).** The same buildings that
	give A a 2.9σ win when planned as five small fleets give a coin-flip-to-loss
	with the public's ops signature when planned as one fleet. Worst origins:
	s21 +257 (ops +311), s16 +192 (ops +214) — pure daily/weekly-limit quanta.
	(Caveat: complement is empty for this fleet, so the fit is self-fit — leaky,
	but identical in both arms; the paired planner delta stands.)

	Across all 39 synthetic fleets: A loses in 11/39 (28%); among the 15 fleets
	with ≥200 devices, 3 losses, all with the ops signature.

	## Which composition features predict the sign

	1. Constraint pressure (total fleet devices) is the master variable — it
	predicts whether the dials matter at all (5% → 85% of origins) and blows up
	the variance. Below ~100 devices A≈B almost everywhere.
	2. At high pressure the sign is a daily/weekly-limit collision lottery
	(±100/±200 quanta), quasi-random in composition; weakly suggestive (n=6,
	anecdotal): including the two death-densest mid-size buildings
	(b_9a5a45d6e498 24dev/15deaths, b_a0e6376410cb 28dev/11deaths) correlates
	with A-worse ops (membership-dOps corr +0.65/+0.74).
	3. Remote buildings and the 117-device giant increase activation but do not
	fix the sign (fleets containing the giant span dt −19.9 to +8.8 at n≤12,
	and the giant is absent from both n18 losers).

	## Implications

	- The CV structurally cannot price this trade: its test fleets are ~5-building
	sub-fleets where limits don't bind. The public fleet is a single large fleet
	where they do. The n24 monolith is the cheapest local proxy for that regime,
	and it sides with the public (A ≈ +1, ops-worse).
	- The dial choice (split3/tw2 vs split7/tw1) is a ~±10–25 ops lottery on any
	large unseen fleet; the −4.1 CV delta should not be treated as transferable
	evidence for A on the private split. On pressure-matched local evidence the
	two configs are statistically indistinguishable (n24: +0.97±8.64).

	Raw data: /tmp/c35/stage1_merged.csv, stage2_all48.csv (+ per-fleet
	stage2_perfleet_48orig.csv, cells stage2_cells_48orig.csv), stage3_stress.csv
	(+ cells), stage3_n24.csv (+ cells), building_meta.csv, fleet defs, and the
	runner fleetprobe.py. Stage-2 6-origin pilot (stage2_results.csv) superseded
	by the all-48 rerun. Origins for stress/all48: all 48; fits per fleet on the
	complement buildings' closed pairs (deployment-faithful), shared across arms.