Thibaut Forest
Claude Opus 5 (1M context)
Phase 56: NEW CHAMPION logbag_rank + gbm rank-blend w=0.30 (purge 1630.3, was 1972.0)
698401a |
Download agent_reports/e2_findings.md from thibautforest/BatterySwapAI2026-Example: direct link, hf CLI and curl.
- Browser
- Download file 6.71 kB
-
https://huggingface.co/thibautforest/BatterySwapAI2026-Example/resolve/main/agent_reports/e2_findings.md
- Command line
-
hf download hf://thibautforest/BatterySwapAI2026-Example/agent_reports/e2_findings.md
-
curl -L -o e2_findings.md https://huggingface.co/thibautforest/BatterySwapAI2026-Example/resolve/main/agent_reports/e2_findings.md
6.71 kB
| # e2 — end-to-end plan optimization & fleet-level forecasting (radical pipeline) | |
| STATUS: DONE — 2026-08-22 ~17:20. Top candidate: e2_hgatez (-245.8 +/- 88.3, clears 2*SE bar). | |
| Baseline: champion logbag/tsroute, SCORE 1972.0 (warmup 3560.3 / panel 1839.6 / nearend 604.8), swaps 9.21. | |
| Champion PREDICTIONS frozen & cached (48 temporal-purge logbag fits reproduce baseline EXACTLY: 1972.01, max|diff|=0.000000). | |
| Every delta below is a pure PLANNER/BUDGET change on identical predictions. True eol only ever enters evaluate_plan. | |
| Harness /tmp/e2/harness.py, cache /tmp/e2/preds.pkl, candidate CSVs /tmp/cv_oofcand_e2_*.csv. | |
| ## Results table (full-48, paired vs /tmp/cv_all48oof.csv; d/dsw = delta per extra swap) | |
| | label | ALL48 | delta | SE | W/L | swaps | d/dsw | blocks (wu/pn/ne) | verdict | | |
| |---|---|---|---|---|---|---|---|---| | |
| | e2_hgatez | 1726.3 | -245.8 | 88.3 | 17/8 | 11.96 | -89 | -853 / -146 / 0 | SURVIVOR — top pick | | |
| | e2_hgate | 1773.8 | -198.3 | 84.0 | 15/8 | 11.77 | -77 | -568 / -146 / 0 | survivor; superseded by hgatez | | |
| | e2_hgate_b | 1785.2 | -186.8 | 84.9 | 14/9 | 12.04 | -66 | -568 / -130 / 0 | budget floor on top adds nothing | | |
| | e2_nbud | 1875.5 | -96.5 | 62.2 | 11/10 | 10.67 | -66 | -211 / -87 / 0 | appetite-equivalent, not adopted | | |
| | e2_a093 | 1908.9 | -63.1 | 33.8 | 16/23 | 10.23 | -62 | -192 / -45 / +3 | control (blanket alpha 0.93) | | |
| | e2_nbud075 | 1933.2 | -38.9 | 39.2 | 8/7 | 9.75 | -72 | -42 / -45 / 0 | muted beta loses the big wins | | |
| | e2_lsadd | 1938.6 | -33.4 | 29.1 | 14/19 | 11.21 | -17 | -40 / -46 / +46 | surrogate-EV adds < simple p_be rule | | |
| | e2_opt | 1970.3 | -1.7 | 1.6 | 21/18 | 9.21 | — | +1 / -3 / +2 | day placement is CONVERGED | | |
| | triage-only: e2_ls -4.7+/-76.8 (9.71 sw), e2_cvar -4.4+/-67.6 (8.47 sw), e2_ls_pmax +97.6+/-57.9 (7.71 sw) | | | | | | | | washes / harmful | | |
| ## Top candidate: e2_hgatez — hazard-gated break-even top-up | |
| Rule (all plan-time observable): let r42 = fleet deaths RECORDED in (T-42d, T], zone42 = census of devices | |
| currently below the 2.45V corrected threshold that crossed within the last 42d (telemetry <= T). | |
| If max(r42, zone42) > sum(p42): open the break-even top-up p_be=0.2 (include every device with p42 >= 0.2). | |
| Else: plan EXACTLY as champion. Fires at ~22/48 origins (warmup s_0-2 via zone42 — the ~34d recording lag | |
| blinds r42 there; high-hazard panel eras s_4-16, s_18-23 via r42; NEVER at s_24-47 calm/near-end). | |
| - -245.8 +/- 88.3, 17W/8L, swaps 11.96 (+2.75). Clears the survivor bar (2*SE = 177) despite the swap rise. | |
| - Head-to-head vs pl_pbe20 (same p_be mechanism, always-on, -254.0 +/- 98.5, swaps 14.69): means statistically | |
| identical (+8.3 +/- 44.6), hgatez better on 14 of 20 differing origins, lower per-origin std (612 vs 683), | |
| 2.7 fewer swaps/scenario, nearend swaps unchanged (16.7 = base) vs pbe20's 18.5. | |
| - Where the gate closes, pbe20 is bimodal: it forfeits 2 big wins the gate misses (s_3 -979, s_17 -1374 — origins | |
| where the COUNT was adequate but the RANKING was broken; not plan-time detectable) but avoids 14 losses | |
| averaging +220 (calm origins where extra swaps caught nothing). | |
| - Deployment asymmetry: at near-end (most deployment-like) origins the gate never fires — adopting it cannot | |
| change the submission plan unless recorded arrivals demonstrably exceed the model's expected count at runtime. | |
| It is a safety net, not a recalibration. | |
| COORDINATION: e2_hgatez and pl_pbe20 are ONE mechanism family (gated vs always-on p_be) — adjudicate together, | |
| never stack. If appetite-driven wins are distrusted (static appetite was publicly refuted historically), the | |
| gated form is the defensible variant: swaps stay at champion level except under an observable hazard signal. | |
| MULTIPLICITY: 10 full-48 planner variants were run; the gate signal was widened once (r42 -> max(r42, zone42)) | |
| after seeing full-48 results. p_be=0.2 itself was found independently by the pl agent. Selection bias is real; | |
| the survivor margin (-245.8 vs bar -177) should be read with that in mind. | |
| ## Direction verdicts | |
| A) SURROGATE-COST PLAN SEARCH: DEAD, with strong evidence. (i) Day placement/routing: 4x search effort | |
| (restarts 8, iters 100, constructive start, NN routing) = -1.7 +/- 1.6 — the champion's optimize_days output is | |
| ~converged; no recoverable slack in WHEN/ORDER. (ii) Membership local search (drop/piggyback-add/new-group vs | |
| surrogate expected cost with per-device emergency mobility, exact mobility deltas): wash (e2_ls -4.7 +/- 76.8); | |
| add-only full-48 -33.4 +/- 29.1 with +2.0 swaps — strictly dominated by the simple p_be>=0.2 rule at the same | |
| appetite; the drop pass is actively harmful (+97.6). EV membership decisions inherit p42-scale noise, exactly | |
| the historical tspig failure, reconfirmed under the purge instrument. The champion planner is ~optimal given | |
| its beliefs; ALL remaining planner upside is in the beliefs. | |
| B) FLEET-LEVEL DEATH FORECAST: THE WINNER, but only as a GATE, not a budget. As a budget floor | |
| (n = 0.8*max(sum_p42, r42)): -96.5 +/- 62.2 — real, but per-swap efficiency (-66) matches blanket appetite | |
| (-58 to -62): targeting the count adds ~nothing because conversion of extra slots is governed by per-origin | |
| ranking luck (s_8/s_9: late 2620->660, 2310->300; s_13/14/15: late UNCHANGED, pure waste). As a gate on the | |
| p_be top-up it is the most efficient appetite form measured (-89/extra swap). Diagnostic that motivated it: | |
| sum(p42) undercounts the arrival process at warmup (s_0: 2.0 vs 9 deaths) and mid-panel (s_15: 5.3, late 3210). | |
| C) PORTFOLIO/CVaR SELECTION: DEAD. (i) Building-correlation premise falsified: marginal-zone (rank 8-40) | |
| devices die at 0.129 with NO recent building deaths vs 0.105 WITH; std logit coef of building recent-deaths | |
| 0.05 vs p42's 0.57. Deaths do not cluster by building beyond p42. (ii) Count-matched reordering by expected | |
| parked loss (p42 x early-death mass x 10/day + emergency-trip ops incl. travel/overtime): -4.4 +/- 67.6 wash — | |
| the p42 order already carries the value signal; day-quantile noise on marginal devices eats the rest. | |
| ## Conclusion | |
| The radical-pipeline premise "the planner leaks cost" is 3/4 refuted: WHEN, ORDER, and WHO-at-fixed-count are | |
| all at (or within noise of) their optimum given the champion's predictions. The one real leak is WHO-at-what- | |
| count in eras where the model's belief about fleet hazard is wrong, and the fleet's own death-arrival record | |
| (r42 + zone census) detects exactly those eras at plan time. e2_hgatez converts that detection into -245.8 | |
| +/- 88.3 (12.5% of SCORE) at +2.75 swaps. Recommend adjudicating e2_hgatez vs pl_pbe20 as the two forms of the | |
| same finding; prefer hgatez on variance, appetite, and deployment-asymmetry grounds. | |