Buckets:

AsianPlayer
/

TriDrive

Files
xet
AsianPlayer/TriDrive / code /eval /warning_head_eval.py
AsianPlayer
16 days ago
download
raw
30.6 kB
"""cwm3/eval/warning_head_eval.py — score the learned warning head on the clean human gold with the UNCHANGED cwm2/eval/warning_gold.py and
render results/tables/warning_head.md with the B4 / B5 verdicts (plan §2).
  python warning_head_eval.py --runs I1_all_s0 [I1_all_s1 I1_all_s2] [--item cwm3_v1] [--n-boot 2000] [--skip-gold]
Step 1 (unless --skip-gold): warning_gold.py --pred <pred_layout_{regime}_{test,val}.npz of results/runs/<run> (arm cwm3_I: W_direct / driver_only /
  demand_only) and results/runs/W_<run> (arm cwm3_W: W_learned_*)> --seeds <seeds of the runs> --out-dir cwm3/results/warning_gold --item <item>.
  The evaluator anchors the 512 clean-test / 107 clean-val episodes (legacy_first), pairs every model column with the DMS comparators
  (att@3s = legacy_att, cockpit@3s = legacy_cockpit_fixedD0, risk@3s, reactive, persist) on identical device-cluster resamples and fits the
  temperature on clean-val.  Rows land in cwm3/results/warning_gold/warning_gold.csv (results_schema columns, append-only).
Step 2: from warning_gold.csv (this item) — clean_test AUROC per seed + CI for W_learned_* (cwm3_W), W_direct (cwm3_I), legacy att / cockpit,
  reactive_nodms; paired Δ vs legacy_att and legacy_cockpit_fixedD0 with kill_checks.beats; the W_learned − W_direct paired Δ (not a warning_gold
  pair) from the per-episode rows CSV with warning_gold.cluster_boot on identical resamples.  Label curve / operating points are read from the
  latest results/warning_head/<run>_*/ directories.  Verdicts: B4 PASS iff W_learned beats W_direct AND beats legacy_cockpit_fixedD0 (beats =
  ≥ 2/3 seeds CI excl. 0, same sign, > seed sd) AND the label curve is monotone in n; B5 PASS iff W_learned beats legacy_att, else the honest
  wording "matches / below (Δ, CI)".  With < 3 seeds every verdict is INSUFFICIENT and the per-seed direction is reported.
No openpilot DMS field is read by this script (legacy chains enter only as labelled comparator rows inside warning_gold.py).
"""
from __future__ import annotations

import argparse
import json
import re
import subprocess
import sys
import time
from pathlib import Path

import numpy as np
import pandas as pd

CWM3 = Path(__file__).resolve().parents[1]
EXPLORE = CWM3.parent
CWM2 = EXPLORE / "cwm2"
OUT_DIR = CWM3 / "results" / "warning_gold"
TABLES = CWM3 / "results" / "tables"
for p in (CWM2 / "eval", CWM3 / "common"):
    sys.path.insert(0, str(p))
import kill_checks as KC                                            # noqa: E402
import warning_gold as WG                                           # noqa: E402
from runmeta import write_run_json                                  # noqa: E402

REGIMES = ("manual", "engaged")


def fmt(x, nd=3):
    try:
        x = float(x)
        return "—" if not np.isfinite(x) else f"{x:.{nd}f}"
    except (TypeError, ValueError):
        return "—"


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--runs", nargs="+", required=True)
    ap.add_argument("--item", default="cwm3_v1")
    ap.add_argument("--n-boot", type=int, default=2000)
    ap.add_argument("--skip-gold", action="store_true")
    ap.add_argument("--headline", default="")
    ap.add_argument("--w-suffix", default="", help="'_oof' -> use results/runs/W_<run>_oof layouts and results/warning_head/<run>_*/ dirs whose selected.json has src_tag ending in oof")
    ap.add_argument("--out-name", default="", help="table file name under results/tables (default warning_head[<w-suffix>].md; the final model writes warning_head_final.md)")
    ap.add_argument("--dir-suffixes", nargs="*", default=None, help="warning-head RESULT-dir families (train_warning_head.py --w-suffix values; '' = base) rendered as labelled rows in §3 / §4 / §4b; "
                                                                    "default = [--w-suffix]. Their columns must all be present in the scored W layout — merge the families first with eval/merge_w_layouts.py "
                                                                    "(cwm2/eval/warning_gold.py keys layouts by (arm, regime, seed, split), so several cwm3_W layouts of one run overwrite each other; 2026-09-12) "
                                                                    "and pass the merged dir as --w-suffix (e.g. --w-suffix _merged --dir-suffixes '' _k0m _k0old)")
    ap.add_argument("--extra-w-suffixes", nargs="*", default=[], help="DEPRECATED (2026-09-12): additional W layouts passed to warning_gold — they collide on (arm, regime, seed, split) and only the last "
                                                                       "survives; kept for the command line of the retracted warning_head_final_partial.md, ignored (use eval/merge_w_layouts.py + --dir-suffixes)")
    ap.add_argument("--feature-label", default="base features", help="label of the primary feature rows in the table header (2026-09-11: the warning-head agent's extended features go to warning_head_final_ext.md)")
    a = ap.parse_args()
    SUFFIX_LABEL = {"": a.feature_label, "_oof": "out-of-fold train features", "_k0m": "K0 masked (NaN) on the 64 v2 driver-box-suspect routes (train, gold, full splits)",
                    "_k0old": "K0 from the cwm2 route-median (NON-causal) kin pack — the v1 tables' K0; diagnostic",
                    # 2026-09-12 warning-head agent (roadmap §B3 1a / 1b / 3; train/warning_features_ext.py blocks appended to the current-state set)
                    "_dw": "current + dwell / duration block (11 dims from Kh8_c)", "_dwmr": "current + dwell + margin-rule block (7 dims)",
                    "_dwfc": "current + dwell + face-landmark block (16 dims, facehist_causal_routes_v2)", "_dwfcmr": "current + dwell + face + margin rule",
                    "_dwfcrm": "current + dwell + face, rows on the 64 suspect routes removed from train and evaluation", "_rm": "current, rows on the 64 suspect routes removed from train and evaluation",
                    # 2026-09-16 verified-mask sensitivity (owner's human review: 3 of the 64 suspect routes have a passenger in the causal box; results/targets_kin_causal_routes_v2/driver_box_verified_mask.json)
                    "_rmv": "current, rows on the 3 human-VERIFIED wrong-occupant routes removed from train and evaluation", "_dwfcrmv": "current + dwell + face, rows on the 3 human-VERIFIED wrong-occupant routes removed from train and evaluation",
                    "_pr": "current, one HGB per regime (pooled fallback)", "_dwpr": "current + dwell, one HGB per regime",
                    # 2026-09-12 gaze/robustness agent (coordinator request; warning_head_final_gaze.md): + the gaze_off CHAIN outputs of the U0 +chain+face run of the same seed
                    "_dwfcgz": "current + dwell + face + GAZE CHAIN (10 dims: predicted gaze_off onset / marginal / state from U0_..._gzfc; eval/predict_u0_rows.py for the train rows) — GAZE VALIDATION PENDING",
                    # 2026-09-12 gaze/robustness agent (coordinator request; warning_head_final_oof.md): OUT-OF-FOLD train features of the FINAL model (train/train_cwm3_oof.py, 5 device folds)
                    "_oofdwfc": "current + dwell + face, OUT-OF-FOLD train features (5 device folds of the final model)",
                    # 2026-09-17 (owner decision (6), F2′ into the read-out): + the F2′ future-state block (train/warning_features_ext.fs_block, results/runs/FS_dwell_s<k>)
                    "_dwfcfs": "current + dwell + face + F2′ FUTURE-STATE block (22 dims: predicted off-task occupancy @1/2/3/5 s per state hands_on / head_turned / gaze_off, expected off-task seconds, "
                               "expected remaining off dwell, P(first leave within 3 s), validity; FS_dwell_s<k> of the same seed) — PREDICTED quantities, column tag fs",
                    "_dwfcfsrmv": "current + dwell + face + F2′ future-state block, rows on the 3 human-VERIFIED wrong-occupant routes removed from train and evaluation"}
    dir_suffixes = a.dir_suffixes if a.dir_suffixes is not None else [a.w_suffix]
    base_suffix = dir_suffixes[0]
    if a.extra_w_suffixes:
        print(f"WARNING: --extra-w-suffixes {a.extra_w_suffixes} is ignored (layout key collision in warning_gold.py); use eval/merge_w_layouts.py and --dir-suffixes", flush=True)
    t0 = time.time()
    OUT_DIR.mkdir(parents=True, exist_ok=True)
    TABLES.mkdir(parents=True, exist_ok=True)
    runs = [CWM3 / "results" / "runs" / r for r in a.runs]
    seeds = sorted({int(json.loads((r / "config.json").read_text())["args"]["seed"]) for r in runs})
    preds = []
    for r in runs:
        for arm_dir in [r, r.parent / f"W_{r.name}{a.w_suffix}"]:
            for reg in REGIMES:
                for sp in ("test", "val"):
                    p = arm_dir / f"pred_layout_{reg}_{sp}.npz"
                    assert p.exists(), p
                    preds.append(str(p))

    def _match(d, suffix=None, run=None):
        """the warning-head result dir belongs to the W layout dir with this suffix: selected.json `w_suffix` (2026-09-11) or, for older dirs, the oof rule.
        2026-09-17 (ECAN read-out lane, additive): when `run` is given the dir must also belong to that joint run (selected.json `run`), because the glob
        results/warning_head/<run>_* of run X also matches the dirs of runs X_<suffix> (X_gzfc_*, X_ecan_*) and the latest of those used to win the §3 / §4 display."""
        suffix = a.w_suffix if suffix is None else suffix
        try:
            sel = json.loads((d / "selected.json").read_text())
            if run is not None and sel.get("run", run) != run:
                return False
            if "w_suffix" in sel:
                return sel["w_suffix"] == suffix
            return sel.get("src_tag", "").endswith("oof") == (suffix == "_oof")
        except Exception:                                             # noqa: BLE001
            return False
    wdirs = [d for d in sorted(CWM3.glob(f"results/warning_head/{a.runs[0]}_*")) if _match(d, base_suffix, a.runs[0])]
    headline = a.headline
    if not headline and wdirs:
        sel = json.loads((wdirs[-1] / "selected.json").read_text())
        src = sel.get("src_tag") or ("n979" if tuple(sel["sources"]) == ("consensus",) else "n" + "".join(s[:3] for s in sel["sources"]))
        headline = f"W_learned_bothsel_t15_w10_{src}"
    if not a.skip_gold:
        cmd = [sys.executable, str(CWM2 / "eval" / "warning_gold.py"), "--pred", *preds, "--seeds", ",".join(map(str, seeds)), "--n-boot", str(a.n_boot),
               "--out-dir", str(OUT_DIR), "--item", a.item, "--headline", headline or "W_learned_sel_t15_w10_n979"]
        print(" ".join(cmd[:6]), "...", flush=True)
        code = subprocess.call(cmd, cwd=str(CWM2))
        assert code == 0, f"warning_gold.py exited {code}"
    W = pd.read_csv(OUT_DIR / "warning_gold.csv", low_memory=False)
    W = W[W.item == a.item]
    stamp = W.run_stamp.max()
    W = W[W.run_stamp == stamp]
    rows_csv = sorted(OUT_DIR.glob(f"warn_gold_rows_{stamp}.csv"))
    R = pd.read_csv(rows_csv[-1], low_memory=False) if rows_csv else None
    # ---- table 1: clean_test AUROC per seed
    base = W[(W.stratum == "clean_test") & (W.metric == "auroc") & (W.calib == "raw") & (W.comparator.isna())]
    keep_arms = ["cwm3_W", "cwm3_I", "legacy_att", "legacy_cockpit_fixedD0", "legacy_risk", "reactive_nodms", "persist_joint"]
    rcfg = json.loads((runs[0] / "config.json").read_text())["cfg"]
    kin_src, kin_ver = rcfg.get("kin_source", "cwm2") or "cwm2", rcfg.get("kin_version", "v1") or "v1"
    causal = kin_src == "causal_routes"
    lines = [f"# Learned warning head W on the clean human gold — item {a.item} (warning_gold run {stamp})", "",
             f"Runs: {', '.join(a.runs)} (seeds {seeds}); evaluator: unchanged `cwm2/eval/warning_gold.py` (clean_test stratum, AUROC primary, device-cluster "
             f"bootstrap n_boot {a.n_boot}, temperature fitted on clean-val). Every number below is read from `results/warning_gold/warning_gold.csv`.", "",
             f"**Feature rows.** Columns tagged `n979` are the **{a.feature_label}** (train/train_warning_head.py `feature_names(demand='P')`: driver chain marginals, K̂, onset_p of the "
             f"driver / vehicle events, pi_resp, anchor states, P onset_p / marginals / P0, the 10 current margins + validity, R hard chain, current kinematics K0 (11) + K0_valid, "
             f"visibility bits, mode, vEgo; feature sets current = no predicted quantity / future = predicted only / both). "
             + (f"K0 comes from the joint run's driver inputs (kin_source `{kin_src}` {kin_ver}: the CAUSAL per-frame driver box; rows without a box are NaN / K0_valid 0). " if causal else "")
             + (("Sensitivity rows: " + "; ".join(f"`{'n979' + suf.replace('_', '')}` = {SUFFIX_LABEL.get(suf, suf)}" for suf in dir_suffixes[1:]) + ".") if len(dir_suffixes) > 1 else "")
             + (f" Scored W layout: `results/runs/W_<run>{a.w_suffix}` (" + ("merged families " + ", ".join(f"`W_<run>{suf}`" for suf in dir_suffixes) + " via eval/merge_w_layouts.py, values bit-identical to the sources" if a.w_suffix == "_merged" else "single family") + ").")
             + " Extended features (dwell / face / mr; warning-head agent, roadmap §B3) are reported separately in `warning_head_final_ext.md`.", "",
             "## 1. Clean-test AUROC (raw) per seed", "", "| arm | column | " + " | ".join(f"s{s}" for s in seeds) + " | mean ± sd | n / pos |", "|---|---|" + "---|" * len(seeds) + "---|---|"]
    tab1 = {}
    for (arm, ev), g in base.groupby(["arm", "event_or_family"]):
        if arm not in keep_arms:
            continue
        if arm == "cwm3_I" and not ev.startswith(("W_direct_t15", "driver_only_t15", "demand_only_t15_6fam")):
            continue
        vals = {int(s): (float(v), lo, hi) for s, v, lo, hi in zip(g.seed, g.value, g.ci_lo, g.ci_hi) if str(s).isdigit()}
        cells = [f"{fmt(vals[s][0])} [{fmt(vals[s][1], 2)},{fmt(vals[s][2], 2)}]" if s in vals else "—" for s in seeds]
        v = np.array([vals[s][0] for s in seeds if s in vals])
        tab1[(arm, ev)] = v
        lines.append(f"| {arm} | {ev} | " + " | ".join(cells) + f" | {fmt(v.mean())} ± {fmt(v.std(ddof=1) if len(v) > 1 else 0.0)} | {int(g.n.iloc[0])} / {int(g.n_pos.iloc[0])} |")
    # ---- table 2: paired deltas of W_learned columns vs the DMS comparators (warning_gold pairs) + vs W_direct (rows CSV)
    lines += ["", "## 2. Paired Δ (AUROC, clean_test, identical device-cluster resamples) of the learned head", "",
              "| column | comparator | " + " | ".join(f"s{s} Δ [CI]" for s in seeds) + " | mean Δ | verdict (kill_checks.beats) |", "|---|---|" + "---|" * len(seeds) + "---|---|"]
    verdicts = {}
    learned_cols = sorted({ev for (arm, ev) in tab1 if arm == "cwm3_W"})
    for ev in learned_cols:
        for comp in ("legacy_att", "legacy_cockpit_fixedD0", "reactive_nodms", "legacy_risk"):
            pr = KC.paired(W, "cwm3_W", comp, "gold_W", ev, metric="auroc", stratum="clean_test")
            pr = pr[pr.seed.astype(str).str.isdigit()]
            if not len(pr):
                continue
            va = KC.values(W, "cwm3_W", "gold_W", metric="auroc", stratum="clean_test", score=ev)
            vb = W[(W.arm == comp) & (W.stratum == "clean_test") & (W.metric == "auroc") & (W.calib == "raw") & (W.comparator.isna())]
            res = KC.beats(pr, va[va.seed.astype(str).str.isdigit()], vb[vb.seed.astype(str).str.isdigit()], min_seeds=3)
            verdicts[(ev, comp)] = res
            cells = []
            for s in seeds:
                x = pr[pr.seed.astype(int) == s]
                cells.append(f"{float(x.delta.iloc[0]):+.3f} [{float(x.delta_ci_lo.iloc[0]):+.3f},{float(x.delta_ci_hi.iloc[0]):+.3f}]" if len(x) else "—")
            md = res.get("mean_delta", float(np.nanmean(pr.delta.astype(float))) if len(pr) else np.nan)
            lines.append(f"| {ev} | {comp} | " + " | ".join(cells) + f" | {md:+.3f} | {res['verdict']} |")
        # vs the joint model's own W_direct (per-episode rows CSV, same resamples)
        if R is not None:
            ca, cb = f"cwm3_W::{ev}", "cwm3_I::W_direct_t15_w10_RV_6fam"
            if ca in R and cb in R:
                cells, deltas, los, his = [], [], [], []
                for s in seeds:
                    D = R[(R.seed == s) & R.clean_test & R[ca].notna() & R[cb].notna()]
                    if len(D) < 20:
                        cells.append("—")
                        continue
                    y = D.y.to_numpy().astype(int)
                    bs = WG.cluster_boot(y, {"a": D[ca].to_numpy(float), "b": D[cb].to_numpy(float)}, D.dongle.to_numpy(), a.n_boot, 0)
                    d = bs["a"]["auroc"] - bs["b"]["auroc"]
                    lo, hi = WG.pct(d)
                    dv = WG.fast_auroc(y, D[ca].to_numpy(float)) - WG.fast_auroc(y, D[cb].to_numpy(float))
                    deltas.append(dv); los.append(lo); his.append(hi)
                    cells.append(f"{dv:+.3f} [{lo:+.3f},{hi:+.3f}]")
                pr = pd.DataFrame({"delta": deltas, "delta_ci_lo": los, "delta_ci_hi": his, "status": "headline", "seed": seeds[:len(deltas)]})
                va = pd.DataFrame({"value": tab1.get(("cwm3_W", ev), np.array([]))})
                vb = pd.DataFrame({"value": tab1.get(("cwm3_I", "W_direct_t15_w10_RV_6fam"), np.array([]))})
                res = KC.beats(pr, va, vb, min_seeds=3) if len(pr) else {"verdict": "INSUFFICIENT"}
                verdicts[(ev, "W_direct(own)")] = res
                md = res.get("mean_delta", float(np.mean(deltas)) if deltas else np.nan)
                lines.append(f"| {ev} | cwm3_I W_direct_t15_w10_RV_6fam (own) | " + " | ".join(cells) + f" | {md:+.3f} | {res['verdict']} |")
    # ---- label curve + operating points from the warning-head run dirs
    lines += ["", "## 3. Feature-set ablation (same head / labels / CV; current-state only vs predicted-future only vs both; gold clean_test, device-cluster CIs)", ""]
    for suf in dir_suffixes:
      lines += [f"### Feature rows: **{SUFFIX_LABEL.get(suf, suf)}** (W layouts `results/runs/W_<run>{suf}`; columns tagged `{'n979' + suf.replace('_', '')}`)", ""]
      for r in a.runs:
        ds_ = [d for d in sorted(CWM3.glob(f"results/warning_head/{r}_*")) if _match(d, suf, r)]
        if not ds_:
            lines.append(f"**{r}**: no warning-head dir for suffix '{suf}'")
            continue
        d = ds_[-1]
        sel = json.loads((d / "selected.json").read_text())
        bc = (sel.get("base_ckpts") or {}).get("base_ckpts", {})
        lines.append(f"**{r}** — features from base checkpoints V `{bc.get('V', '?')}`, U `{bc.get('U', '?')}`, E `{bc.get('E', '?')}`; labels {sel['n_labels']} / {sel['n_pos']} pos; "
                     "selected per set: " + ", ".join(f"{fs}: {v['selected']} (cv AUROC {v['cv_auroc']:.3f}, {v['n_features']} features)" for fs, v in sel["feature_sets"].items()))
        gq = pd.read_csv(d / "gold_quick.csv")
        gq = gq[(gq.selected == 1) | (gq.feature_set == "reference")]
        lines += ["", "| feature set | model | gold AUROC [CI] | gold PR-AUC [CI] | n / pos |", "|---|---|---|---|---|"]
        for x in gq.itertuples(index=False):
            lines.append(f"| {x.feature_set} | {x.model} | {fmt(x.gold_test_auroc)} [{fmt(x.ci_lo, 2)},{fmt(x.ci_hi, 2)}] | {fmt(x.gold_test_prauc)} [{fmt(x.prauc_ci_lo, 2)},{fmt(x.prauc_ci_hi, 2)}] | {int(x.n)} / {int(x.n_pos)} |")
        fsd = d / "feature_sets.csv"
        try:                                                  # single-feature-set runs (--feature-sets current) write a header-less empty file (2026-09-12)
            fs = pd.read_csv(fsd) if fsd.exists() and fsd.stat().st_size > 0 else pd.DataFrame()
        except pd.errors.EmptyDataError:
            fs = pd.DataFrame()
        if len(fs):
            if "model_kind" not in fs:
                fs["model_kind"] = "sel"
            lines += ["", "| paired Δ | head | metric | A | B | Δ [CI] | P(Δ>0) |", "|---|---|---|---|---|---|---|"]
            for x in fs.itertuples(index=False):
                lines.append(f"| {x.a} − {x.b} | {x.model_kind} ({x.model_a} / {x.model_b}) | {x.metric} | {fmt(x.value_a)} | {fmt(x.value_b)} | {x.delta:+.3f} [{x.delta_ci_lo:+.3f},{x.delta_ci_hi:+.3f}] | {fmt(x.frac_boot_d_gt0, 2)} |")
            for kind in ("sel", "hgb", "lr"):
                bc_ = fs[(fs.a == "both") & (fs.b == "current") & (fs.metric == "auroc") & (fs.model_kind == kind)]
                if len(bc_):
                    x = bc_.iloc[0]
                    verdict_fs = "the predicted future ADDS warning value beyond the current state" if x.delta_ci_lo > 0 else \
                                 "the predicted future does NOT add measurable warning value beyond the current state ((iii) ≈ (i) within the CI)" if x.delta_ci_lo <= 0 <= x.delta_ci_hi else \
                                 "the predicted-future features HURT relative to the current state"
                    lines += ["", f"Reading (this seed, head {kind}): both − current AUROC {x.delta:+.3f} [{x.delta_ci_lo:+.3f}, {x.delta_ci_hi:+.3f}] → {verdict_fs}."]
    lines += ["", "## 4. Label curve (per feature set, selected head; gold clean-test AUROC, mean ± sd over subsamples; n = 0 is the model's own W_direct)", ""]
    mono = {}
    for suf in dir_suffixes:
      lines += [f"### Feature rows: **{SUFFIX_LABEL.get(suf, suf)}**", ""]
      for r in a.runs:
        ds_ = [d for d in sorted(CWM3.glob(f"results/warning_head/{r}_*")) if _match(d, suf, r)]
        if not ds_:
            continue
        d = ds_[-1]
        cdf = pd.read_csv(d / "label_curve.csv")
        for fs_name, cf in cdf.groupby("feature_set"):
            summ = cf.groupby("n").gold_test_auroc.agg(["mean", "std", "count"]).reset_index()
            lines.append(f"**{r} / {fs_name}**: " + "; ".join(f"n={int(x['n'])}: {x['mean']:.3f} ± {0.0 if np.isnan(x['std']) else x['std']:.3f} ({int(x['count'])})" for x in summ.to_dict("records")))
            if fs_name == "both" and suf == base_suffix:
                mono[r] = bool(np.all(np.diff(summ["mean"].to_numpy()) >= -0.005))
        op = pd.read_csv(d / "operating.csv")
        lines += ["", f"Operating points ({r}; threshold at clean-val recall {op.thr_val_recall.iloc[0]:.1f}):", "",
                  "| score | gold recall | gold precision | P@R0.3 | P@R0.5 | P@R0.7 | alerts/h manual | alerts/h engaged | lead median s | detected |", "|---|---|---|---|---|---|---|---|---|---|"]
        for x in op.itertuples(index=False):
            lines.append(f"| {x.score} | {fmt(x.gold_test_recall, 2)} | {fmt(x.gold_test_precision, 2)} | {fmt(x.p_at_r30, 2)} | {fmt(x.p_at_r50, 2)} | {fmt(x.p_at_r70, 2)} | "
                         f"{fmt(x.alerts_per_hour_manual, 1)} | {fmt(x.alerts_per_hour_engaged, 1)} | {fmt(x.lead_median_s, 1)} | {fmt(x.lead_detected_frac, 2)} |")
        lines.append("")
    # ---- DMS-alert-rate-matched operating point (train_warning_head.py operating_dms_rate.csv: every score re-thresholded per regime to the DMS att@3s chain's
    #      alerts/hour at ITS clean-val-recall-0.5 threshold; recall / precision on the gold test rows, median lead time on detected gold positives)
    lines += ["", "## 4b. Equal-alert-load operating point — every score thresholded (per regime, 60 s per-route cooldown) to the DMS att@3s chain's alerts/hour", ""]
    for suf in dir_suffixes:
      lines += [f"### Feature rows: **{SUFFIX_LABEL.get(suf, suf)}**", ""]
      for r in a.runs:
        ds_ = [d for d in sorted(CWM3.glob(f"results/warning_head/{r}_*")) if _match(d, suf, r)]
        if not ds_ or not (ds_[-1] / "operating_dms_rate.csv").exists() or (ds_[-1] / "operating_dms_rate.csv").stat().st_size == 0:
            lines.append(f"**{r}**: no operating_dms_rate.csv")
            continue
        od = pd.read_csv(ds_[-1] / "operating_dms_rate.csv")
        lines += [f"**{r}**", "", "| score | rule | gold recall | gold precision | alerts/h manual | alerts/h engaged | lead median s | detected | n gold |", "|---|---|---|---|---|---|---|---|---|"]
        for x in od.itertuples(index=False):
            lines.append(f"| {x.score} | {x.rule} | {fmt(x.gold_test_recall, 2)} | {fmt(x.gold_test_precision, 2)} | {fmt(x.alerts_per_hour_manual, 1)} | {fmt(x.alerts_per_hour_engaged, 1)} | "
                         f"{fmt(x.lead_median_s, 1)} | {fmt(x.lead_detected_frac, 2)} | {int(x.n_gold) if str(x.n_gold).strip() not in ('', 'nan') else '—'} |")
        lines.append("")
    # ---- calibration (warning_gold.py: Brier / ECE raw and after the clean-val temperature; reliability CSVs under results/warning_gold/warn_gold_reliability)
    cal = W[(W.stratum == "clean_test") & (W.comparator.isna()) & (W.metric.isin(["brier", "ece", "brier_cal", "ece_cal"]))]
    cal = cal[cal.seed.astype(str).str.isdigit()]
    if len(cal):
        lines += ["## 4c. Calibration on the clean-test gold (warning_gold.py; Brier with device-cluster CI, ECE 15 bins; `_cal` = after the temperature fitted on clean-val; reliability bins in `results/warning_gold/warn_gold_reliability/<stamp>_<column>_s<seed>.csv`)", "",
                  "| arm | column | calib | metric | " + " | ".join(f"s{s}" for s in seeds) + " | mean |", "|---|---|---|---|" + "---|" * len(seeds) + "---|"]
        keep_cols = {c for c in learned_cols if "sel" in c or "hgb" in c} | {"W_direct_t15_w10_RV_6fam", "att@3s", "cockpit@3s"}
        for (arm, ev, cb, met), g in cal.groupby(["arm", "event_or_family", "calib", "metric"]):
            if ev not in keep_cols or arm not in keep_arms:
                continue
            vals = {int(s): float(v) for s, v in zip(g.seed, g.value)}
            v = np.array([vals[s] for s in seeds if s in vals])
            lines.append(f"| {arm} | {ev} | {cb} | {met} | " + " | ".join(fmt(vals[s]) if s in vals else "—" for s in seeds) + f" | {fmt(v.mean()) if len(v) else '—'} |")
        lines.append("")
    # ---- verdicts
    hl = headline or (learned_cols[0] if learned_cols else "")
    b4_own, b4_old, b5 = verdicts.get((hl, "W_direct(own)"), {}), verdicts.get((hl, "legacy_cockpit_fixedD0"), {}), verdicts.get((hl, "legacy_att"), {})
    monotone = all(mono.values()) if mono else False
    n_seeds = len(seeds)
    def vd(res):
        return res.get("verdict", "INSUFFICIENT")
    if n_seeds < 3:
        b4 = f"INSUFFICIENT ({n_seeds} seed(s); kill rule needs 3) — direction so far: vs own W_direct {vd(b4_own)} ({b4_own.get('mean_delta', np.nan):+.3f}), vs legacy cockpit {vd(b4_old)} ({b4_old.get('mean_delta', np.nan):+.3f}), label curve monotone {monotone}"
        b5s = f"INSUFFICIENT ({n_seeds} seed(s)) — vs DMS att@3s {vd(b5)} ({b5.get('mean_delta', np.nan):+.3f})"
    else:
        b4 = "PASS" if (vd(b4_own) == "BEATS" and vd(b4_old) == "BEATS" and monotone) else f"FAIL (own W_direct {vd(b4_own)}, legacy cockpit {vd(b4_old)}, monotone {monotone})"
        b5s = "PASS — outperforms the DMS attention chain" if vd(b5) == "BEATS" else f"{vd(b5)} — matches / below the DMS attention chain (mean Δ {b5.get('mean_delta', np.nan):+.3f})"
    lines += ["## 5. Verdicts", "", f"- **B4** (W_learned > own W_direct AND > old composition legacy_cockpit_fixedD0, monotone label curve): **{b4}**",
              f"- **B5** (vs DMS att@3s, legacy_att): **{b5s}**", ""]
    # post-hoc (NOT pre-registered): the same head type fixed to HGB on every feature set — the CV selection on in-sample train features
    # picks LR / MLP for the predicted-future sets, which the gold does not confirm; reported separately and labelled post-hoc
    for col_ph in [c for c in learned_cols if c.startswith("W_learned_bothhgb") or c.startswith("W_learned_curhgb")]:
        own, old_, dms = verdicts.get((col_ph, "W_direct(own)"), {}), verdicts.get((col_ph, "legacy_cockpit_fixedD0"), {}), verdicts.get((col_ph, "legacy_att"), {})
        lines.append(f"- post-hoc (fixed HGB head, not the pre-registered CV selection) `{col_ph}`: vs own W_direct **{vd(own)}** ({own.get('mean_delta', np.nan):+.3f}); "
                     f"vs legacy cockpit **{vd(old_)}** ({old_.get('mean_delta', np.nan):+.3f}); vs DMS att@3s **{vd(dms)}** ({dms.get('mean_delta', np.nan):+.3f}, "
                     f"CI excl. 0 in {dms.get('ci_excl_zero', '?')}/{n_seeds} seeds)")
    lines += ["",
              f"Headline column: `{hl}` (pre-registered: base features, CV-selected head; the `curhgb` / `bothhgb` rows of every family are the post-hoc fixed-HGB reading). "
              f"Reference numbers from the cwm2 grid on the same gold (509 rows): W_direct 0.613, legacy cockpit 0.545, DMS att@3s 0.637 — the table above "
              f"recomputes every comparator on this run's common rows, so compare within the table. The `cwm3_I W_direct_t15_w10_RV_6fam` row is the FINAL joint model's own "
              f"label-free composition (closed-form first passage of its hazards over the six demand families + the response chain, eval/write_pred_layout_joint.py), whereas the cwm2 "
              f"0.613 belongs to the cwm2 grid's `direct_multitask` model — a different model quoted as the reference level, not the same score on another data version "
              f"(the v1 joint runs' own W_direct was 0.541–0.542, `warning_head_oof.md` §1).", "",
              "Caveats: labels are 979 train consensus anchors (no H5 records exist yet); warning-head features on train rows are in-sample for the joint model; "
              + ("driver inputs are the CAUSAL route-level kinematics (v2 packs; the 64 driver-box-suspect routes hold 17.9 % / 14.0 % of the manual / engaged test anchors — see the `k0m` sensitivity rows)."
                 if causal else "driver inputs use the route-median (non-causal) ROI (K4 pending) so every row is `secondary` in results_schema terms.")]
    md = "\n".join(lines) + "\n"
    (TABLES / (a.out_name or ("warning_head.md" if not a.w_suffix else f"warning_head{a.w_suffix}.md"))).write_text(md)
    print(md)
    write_run_json(OUT_DIR, "warning_head_eval", [Path(__file__), CWM2 / "eval" / "warning_gold.py", CWM2 / "eval" / "kill_checks.py"], [Path(p) for p in preds], seed=0, t_start=t0,
                   input_listing=False, extra={"item": a.item, "runs": a.runs, "seeds": seeds, "headline": hl, "verdicts": {f"{k[0]}|{k[1]}": v for k, v in verdicts.items()},
                                               "B4": b4, "B5": b5s, "label_curve_monotone": mono, "dms_fields_used": []})


if __name__ == "__main__":
    main()
Xet Storage Details

Size:: 30.6 kB
Xet hash:: 52edf011b167906d6c5190dde630b0df5400a800a01c29a388663635a3f0b379
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.