Buckets:
| #!/usr/bin/env python3 | |
| """cwm3/baselines_dms/eval_dms.py — LAYER 2c / 3 of the DMS-baseline comparison (plan §2 第二层 (b) 评测 / 第三层, §3 eval_dms, 2026-09-14): subprocess wrappers that run the | |
| UNCHANGED evaluators on a baseline's pred layouts, then the append-only report renderer, plus the coverage table. | |
| INPUTS results/runs/<arm>_s<k>/pred_layout_{regime}_{test,val}.npz for the arms of one baseline (abl_dms_<b>_chain, abl_dms_<b>_learned, abl_dms_<b>_ctx, ...; | |
| written by chain_style.py / fit_readout.py / make_r0_layout.py through baselines_dms/layout.py) — the same contract as the CockpitWM head layouts. | |
| OUTPUTS (all produced by the existing tools; nothing recomputed here) | |
| gold cwm2/eval/warning_gold.py (UNCHANGED; the command of docs/dms_att_chain.md §7 / results/baselines_dms/logs/gold_dms_r0.log): | |
| cd cwm2 && python eval/warning_gold.py --pred <the 4 (regime x split) layouts of every arm dir, every seed> --seeds 0,1,2 --n-boot 2000 | |
| --out-dir cwm3/results/warning_gold/dms_<b>_<stamp> --item dms_<b> --headline <primary column> | |
| -> results/warning_gold/dms_<b>_<stamp>/warning_gold.csv (AUROC / PR-AUC per stratum with device-cluster CIs, paired Δ vs att@3s / cockpit@3s / reactive on | |
| identical resamples, temperature on clean-val, ECE). One --item per baseline: warning_gold keeps, per stratum, the rows on which EVERY non-structural | |
| column is finite (:412-414), so a baseline's coverage never shrinks the gold of another item (plan §0). eval/warning_head_eval.py is the W-head wrapper of | |
| the same evaluator (it expects results/runs/<run> + W_<run> layouts of arm cwm3_I / cwm3_W); for the abl_dms_* arms warning_gold.py is called directly, as | |
| the R0 evaluation did. eval/gold_extended.py (secondary clean_test_r2 stratum) is exposed as `--gold-extended` with the same --w-runs / --seeds / --cols form. | |
| operating eval/operating_points.py (UNCHANGED) --item dms_<b> --w-runs <arm dirs> --seeds <k...> --col <column> --label "<label>" --loads <loads> | |
| -> results/operating_points/dms_<b>_<stamp>/{operating_points,lead_times,reliability,paired_lead,dms_reference}.csv + ONE section appended to | |
| results/tables/operating_points.md. Loads: dms_matched, low10, same_recall (default of the script) + the 1 / 3 / 5 alerts/h sweep as low1, low3, low5 — | |
| the script's `low<N>` token (operating_points.py:102-104, :161-163) already parses any integer N, so the sweep needs NO script change. | |
| report baselines_dms/report_dms.py --gold-item dms_<b> --gold-dir <dir> --op-item dms_<b> --arm <arm> --baseline "<label>" --status "<status>" --verify | |
| -> sections appended to results/tables/dms_warning_comparison.md and dms_operating_points.md; --verify re-reads every printed number from the CSVs. | |
| coverage results/tables/dms_coverage.md (this module, append-only sections): per baseline arm and (regime, split): rows, finite rows (w_mask & finite primary column), | |
| share, decoded routes (routes with >= 1 finite row) / routes, exposure hours (rows x 0.2 s / 3600) and finite hours — the "coverage table" of decisions.md | |
| (missing policy: non-finite = no alert, exposure unchanged). | |
| head (2026-09-14 layer-2 launch) the FROZEN CockpitWM current-state read-out W_I1_all_minus_road2drv_s<k>_final_dwfc (column W_learned_curhgb_t15_w10_n979dwfc, | |
| arm abl_W_dwfc; read-only) is passed to warning_gold.py as extra --pred layouts for the requested seeds, so its AUROC on the item's common rows is in the | |
| same CSV. warning_gold.py pairs pred columns only with the legacy comparators, so the plan's PRIMARY comparison (baseline learned arm vs the CockpitWM | |
| current-state read-out, plan §4) is computed here from warning_gold's per-episode score file warn_gold_rows_<stamp>.csv with the SAME resampler | |
| (warning_gold.cluster_boot, boot seed 0, n_boot 2000; baseline seed k vs head seed k; strata as warning_gold) -> <gold dir>/paired_vs_head.csv (+ agg rows | |
| with the seed-median CI, sd, n_seeds_ci_excl_0, same_sign as warning_gold's aggregate); report_dms renders it as the "Δ vs head" column. --no-head disables. | |
| prauc (2026-09-15, owner: "comparing AUROC alone is not enough") paired_vs_head.csv already holds the PR-AUC Δ vs head per seed (bs['b']['prauc'] − bs['h']['prauc'] | |
| of the SAME cluster_boot call as the AUROC Δ) but its `agg` row is keyed on AUROC (median seed, sd, n_seeds_ci_excl_0, same_sign all from delta_auroc). | |
| `paired_vs_head_prauc(gold_dir)` / `--paired-prauc <gold_dir>...` writes the NEW file <gold dir>/paired_vs_head_prauc.csv (never overwrites): the per-seed | |
| PR-AUC rows copied unchanged + an `agg` row by warning_gold's aggregate rule applied to the PR-AUC Δ (mean Δ, CI of the seed-median PR-AUC Δ, sd over seeds, | |
| n_seeds_ci_excl_0 from the PR-AUC CIs, same_sign). report_dms --metrics auroc,pr_auc reads it for the "Δ PR-AUC vs head" mean row. | |
| Nothing here reads a video, a model or any openpilot field. | |
| python baselines_dms/eval_dms.py --baseline l2cs_gaze --arms abl_dms_l2cs_gaze_learned --seeds 0 1 2 --col dms_l2cs_gaze_learned_sel_t15_w10_n979 --label "L2CS-Net learned read-out" | |
| python baselines_dms/eval_dms.py ... --skip-op --runs-root D --gold-root D --tables-dir D --n-boot-gold 200 # dry run on partial layouts, nothing under results/ | |
| python baselines_dms/eval_dms.py --baseline l2cs_gaze --arms abl_dms_l2cs_gaze_chain --seeds 0 --col dms_l2cs_gaze_chain_att3s --label "L2CS-Net chain-style" | |
| python baselines_dms/eval_dms.py --coverage --arms abl_dms_lmface_rule --seeds 0 --col dms_lmface_rule0 # coverage table only (CPU, seconds) | |
| python baselines_dms/eval_dms.py --dry-run # prints every argv; coverage on a synthetic layout to a temp table | |
| """ | |
| from __future__ import annotations | |
| import argparse | |
| import json | |
| import subprocess | |
| import sys | |
| import time | |
| from pathlib import Path | |
| import numpy as np | |
| PKG = Path(__file__).resolve().parent | |
| CWM3 = PKG.parent | |
| EXPLORE = CWM3.parent | |
| CWM2 = EXPLORE / "cwm2" | |
| S12R = EXPLORE / "s12_wm" / "results" | |
| if str(PKG) not in sys.path: | |
| sys.path.insert(0, str(PKG)) | |
| import layout as LY # noqa: E402 | |
| PY = sys.executable | |
| RUNS = CWM3 / "results" / "runs" | |
| GOLD_ROOT = CWM3 / "results" / "warning_gold" | |
| OP_ROOT = CWM3 / "results" / "operating_points" | |
| TABLES = CWM3 / "results" / "tables" | |
| COVERAGE_TABLE = TABLES / "dms_coverage.md" | |
| REGIMES, SPLITS = ("manual", "engaged"), ("test", "val") | |
| DEFAULT_LOADS = "dms_matched,low10,same_recall,low1,low3,low5" | |
| ROW_S = 0.2 | |
| HEAD_RUN = "W_I1_all_minus_road2drv_s{seed}_final_dwfc" # frozen CockpitWM current-state read-out (read-only comparator; docs/dms_att_chain.md §7) | |
| HEAD_COL = "W_learned_curhgb_t15_w10_n979dwfc" | |
| HEAD_ARM = "abl_W_dwfc" # the arm label inside those layouts | |
| STRATA = ("clean_test", "clean_test|manual", "clean_test|engaged", "clean_val") | |
| def layouts_of(arms, seeds, root=RUNS): | |
| out = [] | |
| for arm, seed in zip(arms, seeds): | |
| for r in REGIMES: | |
| for s in SPLITS: | |
| p = Path(root) / f"{arm}_s{seed}" / f"pred_layout_{r}_{s}.npz" | |
| assert p.exists(), f"missing layout {p}" | |
| out.append(p) | |
| return out | |
| def head_layouts(seeds, root=RUNS): | |
| """the frozen head's 4 (regime x split) layouts for every requested seed (asserted present; never written here).""" | |
| out = [] | |
| for seed in sorted(set(seeds)): | |
| for r in REGIMES: | |
| for s in SPLITS: | |
| p = Path(root) / HEAD_RUN.format(seed=seed) / f"pred_layout_{r}_{s}.npz" | |
| assert p.exists(), f"frozen head layout missing: {p}" | |
| out.append(p) | |
| return out | |
| def gold_argv(baseline, arms, seeds, col, n_boot=2000, stamp=None, root=RUNS, extra_pred=(), gold_root=GOLD_ROOT): | |
| stamp = stamp or time.strftime("%Y%m%dT%H%M%SZ", time.gmtime()) # full UTC stamp: every gold dir new (a date-only stamp would append to an earlier run's CSV) | |
| out_dir = Path(gold_root) / f"dms_{baseline}_{stamp}" | |
| preds = [str(p) for p in layouts_of(arms, seeds, root)] + [str(p) for p in extra_pred] | |
| return [PY, str(CWM2 / "eval" / "warning_gold.py"), "--pred", *preds, "--seeds", ",".join(str(s) for s in sorted(set(seeds))), "--n-boot", str(n_boot), | |
| "--out-dir", str(out_dir), "--item", f"dms_{baseline}", "--headline", col], out_dir | |
| def operating_argv(baseline, arms, seeds, col, label, loads=DEFAULT_LOADS, n_boot=1000): | |
| return [PY, str(CWM3 / "eval" / "operating_points.py"), "--item", f"dms_{baseline}", "--w-runs", *[f"{a}_s{s}" for a, s in zip(arms, seeds)], "--seeds", *[str(s) for s in seeds], | |
| "--col", col, "--label", label, "--loads", loads, "--n-boot", str(n_boot)] | |
| def gold_extended_argv(arms, seeds, cols, n_boot=2000): | |
| return [PY, str(CWM3 / "eval" / "gold_extended.py"), "--w-runs", *[f"{a}_s{s}" for a, s in zip(arms, seeds)], "--seeds", *[str(s) for s in seeds], "--cols", *cols, "--n-boot", str(n_boot)] | |
| def report_argv(baseline, gold_dir, arm, label, status, op_item=True, paired_head_csv=None, table_dir=None): | |
| argv = [PY, str(PKG / "report_dms.py"), "--gold-item", f"dms_{baseline}", "--gold-dir", str(gold_dir), "--arm", arm, "--baseline", label, "--status", status, "--verify"] | |
| if op_item: | |
| argv += ["--op-item", f"dms_{baseline}"] | |
| if paired_head_csv: | |
| argv += ["--paired-head-csv", str(paired_head_csv)] | |
| if table_dir: | |
| argv += ["--table-dir", str(table_dir)] | |
| return argv | |
| # ----------------------------------------------------------------------------- paired Δ vs the frozen CockpitWM current-state read-out (same resampler) | |
| def paired_vs_head(gold_dir, item, arms, seeds, head_col=HEAD_COL, head_arm=HEAD_ARM, n_boot=2000, boot_seed=0, log=print): | |
| """warn_gold_rows_<stamp>.csv of the gold run -> <gold_dir>/paired_vs_head.csv: for every column of every arm, stratum and seed, AUROC / PR-AUC of the baseline | |
| and of the head on the item's common-valid rows (every non-structural column finite, warning_gold's rule), Δ = baseline − head with the device-cluster CI of warning_gold.cluster_boot (identical resamples), plus | |
| per (arm, column, stratum) an `agg` row (mean Δ, sd over seeds, CI of the seed-median Δ, n_seeds_ci_excl_0, same_sign_all_seeds — warning_gold's aggregate).""" | |
| import pandas as pd | |
| for _p in (CWM2 / "eval", CWM2 / "data", CWM2, CWM2 / "baselines" / "legacy_cockpitwm"): | |
| if str(_p) not in sys.path: | |
| sys.path.insert(0, str(_p)) | |
| import warning_gold as WG # unchanged cwm2 code: cluster_boot / fast_auroc / fast_ap / pct | |
| gold_dir = Path(gold_dir) | |
| rows_csv = sorted(gold_dir.glob("warn_gold_rows_*.csv")) | |
| assert rows_csv, f"no warn_gold_rows_*.csv under {gold_dir}" | |
| rows_csv = rows_csv[-1] | |
| R = pd.read_csv(rows_csv) | |
| hc = f"{head_arm}::{head_col}" | |
| if hc not in R.columns: | |
| log(f"paired_vs_head: head column {hc} not in {rows_csv.name} (head layouts not passed?) — nothing written") | |
| return None | |
| # the item's COMMON-VALID rows exactly as warning_gold defines them (:412-414): every non-structural score column finite (legacy comparators + every pred column; | |
| # reactive_nodms is structural), so the Δ vs head stands on the same rows as the Δ vs att@3s of the same item | |
| legacy_cols = [c for c in ("att@3s", "cockpit@3s", "risk@3s", "reactive", "persist@3s") if c in R.columns] | |
| core = legacy_cols + [c for c in R.columns if "::" in c] | |
| R = R[R[core].notna().all(1)] | |
| out = [] | |
| for arm in sorted(set(arms)): | |
| cols = [c for c in R.columns if c.startswith(f"{arm}::")] | |
| for c in cols: | |
| for stratum in STRATA: | |
| if stratum == "clean_test": | |
| D0 = R[R.clean_test.astype(bool)] | |
| elif stratum == "clean_val": | |
| D0 = R[R.clean_val.astype(bool)] | |
| else: | |
| D0 = R[R.clean_test.astype(bool) & (R.regime == stratum.split("|")[1])] | |
| per_seed = [] | |
| for seed in sorted(set(seeds)): | |
| D = D0[D0.seed.astype(int) == int(seed)] | |
| both = D[c].notna().to_numpy() & D[hc].notna().to_numpy() | |
| y = D.y.to_numpy().astype(int)[both] | |
| if both.sum() < 20 or len(np.unique(y)) < 2: | |
| continue | |
| sb, sh, dong = D[c].to_numpy(float)[both], D[hc].to_numpy(float)[both], D.dongle.astype(str).to_numpy()[both] | |
| bs = WG.cluster_boot(y, {"b": sb, "h": sh}, dong, n_boot, boot_seed) | |
| rec = {"item": item, "arm": arm, "column": c.split("::", 1)[1], "stratum": stratum, "seed": str(seed), "n": int(both.sum()), "n_pos": int(y.sum()), "n_dev": int(len(np.unique(dong))), | |
| "head_arm": head_arm, "head_column": head_col, "head_seed": int(seed), "n_boot": n_boot, "boot_seed": boot_seed, "rows_csv": rows_csv.name} | |
| for met, key in (("auroc", "auroc"), ("prauc", "prauc")): | |
| f = WG.fast_auroc if met == "auroc" else WG.fast_ap | |
| vb, vh = f(y, sb), f(y, sh) | |
| d = bs["b"][key] - bs["h"][key] | |
| lo, hi = WG.pct(d) | |
| rec.update({f"{met}_baseline": vb, f"{met}_head": vh, f"delta_{met}": vb - vh, f"delta_{met}_ci_lo": lo, f"delta_{met}_ci_hi": hi, | |
| f"frac_boot_d{met}_gt0": float(np.mean(d[np.isfinite(d)] > 0)) if np.isfinite(d).any() else np.nan}) | |
| per_seed.append(rec) | |
| out += per_seed | |
| if len(per_seed) > 1: | |
| dv = np.array([r["delta_auroc"] for r in per_seed]) | |
| med = per_seed[int(np.argsort(dv)[len(dv) // 2])] | |
| n_excl = int(sum((r["delta_auroc_ci_lo"] > 0) or (r["delta_auroc_ci_hi"] < 0) for r in per_seed)) | |
| same = int(np.all(np.sign(dv) == np.sign(dv[0]))) | |
| agg = dict(med, seed="agg", head_seed=-1, n=int(np.mean([r["n"] for r in per_seed])), n_pos=int(np.mean([r["n_pos"] for r in per_seed])), | |
| auroc_baseline=float(np.mean([r["auroc_baseline"] for r in per_seed])), auroc_head=float(np.mean([r["auroc_head"] for r in per_seed])), | |
| delta_auroc=float(dv.mean()), delta_auroc_ci_lo=med["delta_auroc_ci_lo"], delta_auroc_ci_hi=med["delta_auroc_ci_hi"], | |
| prauc_baseline=float(np.mean([r["prauc_baseline"] for r in per_seed])), prauc_head=float(np.mean([r["prauc_head"] for r in per_seed])), | |
| delta_prauc=float(np.mean([r["delta_prauc"] for r in per_seed])), | |
| sd_seed_delta=float(dv.std(ddof=1)), n_seeds=len(per_seed), n_seeds_ci_excl_0=n_excl, same_sign_all_seeds=same, | |
| notes=f"mean over seeds {','.join(r['seed'] for r in per_seed)}; CI of the seed-median delta (warning_gold aggregate rule)") | |
| out.append(agg) | |
| df = pd.DataFrame(out) | |
| p = gold_dir / "paired_vs_head.csv" | |
| df.to_csv(p, index=False) | |
| log(f"paired_vs_head: {len(df)} rows -> {p}") | |
| for r in out: | |
| if r["seed"] == "agg" or len(set(seeds)) == 1: | |
| log(f" {r['arm']}::{r['column']} {r['stratum']} s{r['seed']} n={r['n']} auroc {r['auroc_baseline']:.4f} vs head {r['auroc_head']:.4f} Δ {r['delta_auroc']:+.4f} [{r['delta_auroc_ci_lo']:+.4f}, {r['delta_auroc_ci_hi']:+.4f}]") | |
| return p | |
| PRAUC_ID_COLS = ["item", "arm", "column", "stratum", "seed", "n", "n_pos", "n_dev", "head_arm", "head_column", "head_seed", "n_boot", "boot_seed", "rows_csv"] | |
| PRAUC_VAL_COLS = ["prauc_baseline", "prauc_head", "delta_prauc", "delta_prauc_ci_lo", "delta_prauc_ci_hi", "frac_boot_dprauc_gt0"] | |
| def prauc_agg_rows(PH): | |
| """per-seed rows of paired_vs_head.csv (one item) -> DataFrame of the PR-AUC paired Δ vs head: the per-seed rows (PR-AUC columns only, values unchanged — they come | |
| from the same cluster_boot resamples as the AUROC Δ) + per (arm, column, stratum) an `agg` row by warning_gold's aggregate rule applied to delta_prauc: | |
| mean Δ, CI of the seed-median PR-AUC Δ, sd_seed_delta_prauc, n_seeds_ci_excl_0 (PR-AUC CIs), same_sign_all_seeds. Items with a single seed get no agg row.""" | |
| import pandas as pd | |
| PH = PH.assign(seed=PH.seed.astype(str)) | |
| per = PH[PH.seed != "agg"] | |
| out = [] | |
| for (arm, col, st), g in per.groupby(["arm", "column", "stratum"], sort=True): | |
| g = g.sort_values("seed") | |
| for _, r in g.iterrows(): | |
| rec = {k: r[k] for k in PRAUC_ID_COLS + PRAUC_VAL_COLS} | |
| rec.update(sd_seed_delta_prauc=np.nan, n_seeds=np.nan, n_seeds_ci_excl_0=np.nan, same_sign_all_seeds=np.nan, source_csv="paired_vs_head.csv", | |
| notes="per-seed row copied from paired_vs_head.csv (same warning_gold.cluster_boot resamples as the AUROC Δ; baseline seed k vs head seed k)") | |
| out.append(rec) | |
| if len(g) > 1: | |
| dv = g.delta_prauc.to_numpy(float) | |
| med = g.iloc[int(np.argsort(dv)[len(dv) // 2])] | |
| n_excl = int(((g.delta_prauc_ci_lo.to_numpy(float) > 0) | (g.delta_prauc_ci_hi.to_numpy(float) < 0)).sum()) | |
| same = int(np.all(np.sign(dv) == np.sign(dv[0]))) | |
| rec = {k: med[k] for k in PRAUC_ID_COLS} | |
| rec.update(seed="agg", head_seed=-1, n=int(np.mean(g.n)), n_pos=int(np.mean(g.n_pos)), | |
| prauc_baseline=float(g.prauc_baseline.mean()), prauc_head=float(g.prauc_head.mean()), delta_prauc=float(dv.mean()), | |
| delta_prauc_ci_lo=float(med.delta_prauc_ci_lo), delta_prauc_ci_hi=float(med.delta_prauc_ci_hi), frac_boot_dprauc_gt0=float(med.frac_boot_dprauc_gt0), | |
| sd_seed_delta_prauc=float(dv.std(ddof=1)), n_seeds=len(g), n_seeds_ci_excl_0=n_excl, same_sign_all_seeds=same, source_csv="paired_vs_head.csv", | |
| notes=f"mean over seeds {','.join(g.seed)}; CI of the seed-median PR-AUC delta (warning_gold aggregate rule applied to delta_prauc; median seed {med.seed})") | |
| out.append(rec) | |
| return pd.DataFrame(out, columns=PRAUC_ID_COLS + PRAUC_VAL_COLS + ["sd_seed_delta_prauc", "n_seeds", "n_seeds_ci_excl_0", "same_sign_all_seeds", "source_csv", "notes"]) | |
| def paired_vs_head_prauc(gold_dir, log=print): | |
| """<gold_dir>/paired_vs_head.csv -> <gold_dir>/paired_vs_head_prauc.csv (NEW file; an existing file is never overwritten — returns its path untouched).""" | |
| import pandas as pd | |
| gold_dir = Path(gold_dir) | |
| src, dst = gold_dir / "paired_vs_head.csv", gold_dir / "paired_vs_head_prauc.csv" | |
| if not src.exists(): | |
| log(f"paired_vs_head_prauc: {src} missing (no head in that gold run) — nothing written") | |
| return None | |
| if dst.exists(): | |
| log(f"paired_vs_head_prauc: {dst} exists — left untouched") | |
| return dst | |
| df = prauc_agg_rows(pd.read_csv(src)) | |
| df.to_csv(dst, index=False) | |
| log(f"paired_vs_head_prauc: {len(df)} rows ({int((df.seed == 'agg').sum())} agg) -> {dst}") | |
| for _, r in df[df.seed == "agg"].iterrows(): | |
| log(f" {r['arm']}::{r['column']} {r['stratum']} n={r['n']} prauc {r['prauc_baseline']:.4f} vs head {r['prauc_head']:.4f} Δ {r['delta_prauc']:+.4f} " | |
| f"[{r['delta_prauc_ci_lo']:+.4f}, {r['delta_prauc_ci_hi']:+.4f}] excl0 {int(r['n_seeds_ci_excl_0'])}/{int(r['n_seeds'])} same_sign {int(r['same_sign_all_seeds'])}") | |
| return dst | |
| def run(argv, cwd, log=print, dry=False): | |
| log(f"$ (cd {cwd} && {' '.join(str(a) for a in argv)})") | |
| if dry: | |
| return 0 | |
| rc = subprocess.call([str(a) for a in argv], cwd=str(cwd)) | |
| assert rc == 0, f"exit {rc}: {argv[1]}" | |
| return rc | |
| # ----------------------------------------------------------------------------- coverage table | |
| def coverage_rows(arm, seed, col=None, root=RUNS): | |
| rows = [] | |
| for r in REGIMES: | |
| for s in SPLITS: | |
| p = Path(root) / f"{arm}_s{seed}" / f"pred_layout_{r}_{s}.npz" | |
| if not p.exists(): | |
| continue | |
| L = LY.read_layout(p) | |
| j = L["w_names"].index(col) if col in L["w_names"] else 0 | |
| fin = L["w_mask"] & np.isfinite(L["w"][:, j].astype(np.float32)) | |
| uid = L["uid"] | |
| n = len(uid) | |
| routes = np.unique(uid) | |
| dec = np.unique(uid[fin]) | |
| rows.append({"arm": arm, "seed": seed, "column": L["w_names"][j], "regime": r, "split": s, "rows": n, "finite_rows": int(fin.sum()), "finite_share": float(fin.mean()) if n else float("nan"), | |
| "routes": int(len(routes)), "decoded_routes": int(len(dec)), "hours": n * ROW_S / 3600.0, "finite_hours": float(fin.sum()) * ROW_S / 3600.0}) | |
| return rows | |
| def coverage_section(arms, seeds, col=None, root=RUNS, note=""): | |
| stamp = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) | |
| lines = [f"## coverage — {', '.join(sorted(set(arms)))} — {stamp}", "", | |
| "Missing policy (decisions.md 2026-09-14): a non-finite score is no alert with unchanged exposure (rows x 0.2 s); finite rows = w_mask & finite(column); " | |
| "decoded routes = routes with >= 1 finite row. Source: the pred_layout files listed (baselines_dms/eval_dms.py coverage_rows)." + (f" {note}" if note else ""), "", | |
| "| arm | seed | column | regime | split | rows | finite rows | share | decoded / routes | hours | finite hours |", "|---|---|---|---|---|---|---|---|---|---|---|"] | |
| all_rows = [] | |
| for arm, seed in zip(arms, seeds): | |
| for r in coverage_rows(arm, seed, col, root): | |
| all_rows.append(r) | |
| lines.append(f"| {r['arm']} | {r['seed']} | `{r['column']}` | {r['regime']} | {r['split']} | {r['rows']:,} | {r['finite_rows']:,} | {r['finite_share']:.3f} | {r['decoded_routes']} / {r['routes']} | " | |
| f"{r['hours']:.2f} | {r['finite_hours']:.2f} |") | |
| lines.append("") | |
| return "\n".join(lines) + "\n", all_rows | |
| def append_coverage(arms, seeds, col=None, table=COVERAGE_TABLE, root=RUNS, note=""): | |
| sec, rows = coverage_section(arms, seeds, col, root, note) | |
| table = Path(table) | |
| if not table.exists(): | |
| table.parent.mkdir(parents=True, exist_ok=True) | |
| table.write_text("# DMS baselines — score coverage per (regime, split): finite rows, decoded routes, exposure hours (baselines_dms/eval_dms.py; append-only sections)\n") | |
| with open(table, "a", encoding="utf-8") as fh: | |
| fh.write("\n" + sec) | |
| return sec, rows | |
| # ----------------------------------------------------------------------------- driver | |
| def evaluate(baseline, arms, seeds, col, label, status="adapted", loads=DEFAULT_LOADS, n_boot_gold=2000, n_boot_op=1000, skip_gold=False, skip_op=False, coverage=True, gold_extended=False, dry=False, log=print, | |
| runs_root=RUNS, gold_root=GOLD_ROOT, tables_dir=TABLES, head=True, head_col=HEAD_COL, coverage_note=""): | |
| """gold (warning_gold.py; + the frozen head layouts as extra --pred) -> paired Δ vs the head (same resampler) -> operating points (operating_points.py, incl. the | |
| 1/3/5 alerts/h sweep) -> report_dms.py --verify -> coverage table. -> dict of argv / dirs. runs_root / gold_root / tables_dir: dry runs point them away from results/.""" | |
| assert len(arms) == len(seeds) | |
| rec = {"baseline": baseline, "arms": list(arms), "seeds": list(seeds), "col": col, "argv": {}} | |
| extra = head_layouts(seeds, RUNS) if head else [] # the frozen head is always read from results/runs (read-only), also in dry runs | |
| if not skip_gold: | |
| argv, gold_dir = gold_argv(baseline, arms, seeds, col, n_boot_gold, root=runs_root, extra_pred=extra, gold_root=gold_root) | |
| rec["argv"]["gold"] = argv | |
| rec["gold_dir"] = str(gold_dir) | |
| run(argv, CWM2, log, dry) | |
| else: | |
| cands = sorted(Path(gold_root).glob(f"dms_{baseline}_*")) | |
| rec["gold_dir"] = str(cands[-1]) if cands else "" | |
| paired_csv = None | |
| if head and rec.get("gold_dir") and not dry and Path(rec["gold_dir"]).exists(): | |
| paired_csv = paired_vs_head(rec["gold_dir"], f"dms_{baseline}", arms, seeds, head_col, HEAD_ARM, n_boot_gold, 0, log) | |
| rec["paired_vs_head_csv"] = str(paired_csv) if paired_csv else "" | |
| if gold_extended: | |
| argv = gold_extended_argv(arms, seeds, [col]) | |
| rec["argv"]["gold_extended"] = argv | |
| run(argv, CWM3, log, dry) | |
| if not skip_op: | |
| argv = operating_argv(baseline, arms, seeds, col, label, loads, n_boot_op) | |
| rec["argv"]["operating"] = argv | |
| run(argv, CWM3, log, dry) | |
| argv = report_argv(baseline, rec.get("gold_dir") or Path(gold_root) / f"dms_{baseline}_<stamp>", arms[0], label, status, op_item=not skip_op, paired_head_csv=paired_csv, | |
| table_dir=(None if Path(tables_dir) == TABLES else tables_dir)) | |
| rec["argv"]["report"] = argv | |
| run(argv, CWM3, log, dry) | |
| if coverage and not dry: | |
| sec, rows = append_coverage(arms, seeds, col, table=Path(tables_dir) / "dms_coverage.md", root=runs_root, note=coverage_note) | |
| rec["coverage_rows"] = rows | |
| log(sec) | |
| return rec | |
| def dry_run(keep_dir=None, log=print): | |
| """prints the exact evaluator argv for a baseline (nothing executed) and writes the coverage table from a synthetic layout into a temp dir.""" | |
| import tempfile | |
| root = Path(keep_dir) if keep_dir else Path(tempfile.mkdtemp(prefix="dms_eval_dryrun_")) | |
| puid, pt = LY.pack_uid_t("manual", "val") | |
| rng = np.random.default_rng(0) | |
| N = len(puid) | |
| s = rng.random(N).astype(np.float32) | |
| routes = np.unique(puid)[:5] | |
| s[~np.isin(puid, routes)] = np.nan | |
| arm = "abl_dms_synthetic_learned" | |
| LY.write_layout(arm, "manual", 0, "val", puid, pt, {"dms_synthetic_learned_sel_t15_w10_n979": s}, meta_extra={"baseline": {"id": "synthetic"}}, out_root=root / "runs", score_source="") | |
| sec, rows = append_coverage([arm], [0], "dms_synthetic_learned_sel_t15_w10_n979", table=root / "tables" / "dms_coverage.md", root=root / "runs") | |
| rec = {"root": str(root), "coverage_rows": rows} | |
| rec["argv_example"] = {"gold": gold_argv("l2cs_gaze", ["abl_dms_l2cs_gaze_learned"] * 3, [0, 1, 2], "dms_l2cs_gaze_learned_sel_t15_w10_n979", stamp="<stamp>", root=root / "runs")[0] if False else | |
| [PY, str(CWM2 / "eval" / "warning_gold.py"), "--pred", "<results/runs/abl_dms_l2cs_gaze_learned_s{0,1,2}/pred_layout_{manual,engaged}_{test,val}.npz (12 files)>", "--seeds", "0,1,2", | |
| "--n-boot", "2000", "--out-dir", str(GOLD_ROOT / "dms_l2cs_gaze_<stamp>"), "--item", "dms_l2cs_gaze", "--headline", "dms_l2cs_gaze_learned_sel_t15_w10_n979"], | |
| "operating": operating_argv("l2cs_gaze", ["abl_dms_l2cs_gaze_learned"] * 3, [0, 1, 2], "dms_l2cs_gaze_learned_sel_t15_w10_n979", "L2CS-Net learned read-out"), | |
| "report": report_argv("l2cs_gaze", GOLD_ROOT / "dms_l2cs_gaze_<stamp>", "abl_dms_l2cs_gaze_learned", "L2CS-Net learned read-out", "adapted"), | |
| "gold_extended": gold_extended_argv(["abl_dms_l2cs_gaze_learned"] * 3, [0, 1, 2], ["dms_l2cs_gaze_learned_sel_t15_w10_n979"])} | |
| rec["loads_note"] = "low1 / low3 / low5 are parsed by the existing `low<N>` token of eval/operating_points.py (:161-163): the 1/3/5 alerts/h sweep needs no script change" | |
| print(json.dumps(rec, indent=1, default=str)) | |
| return rec | |
| def main(argv=None): | |
| ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) | |
| ap.add_argument("--baseline", default="") | |
| ap.add_argument("--arms", nargs="+", default=[]) | |
| ap.add_argument("--seeds", nargs="+", type=int, default=[0, 1, 2]) | |
| ap.add_argument("--col", default="") | |
| ap.add_argument("--label", default="") | |
| ap.add_argument("--status", default="adapted") | |
| ap.add_argument("--loads", default=DEFAULT_LOADS) | |
| ap.add_argument("--n-boot-gold", type=int, default=2000) | |
| ap.add_argument("--n-boot-op", type=int, default=1000) | |
| ap.add_argument("--skip-gold", action="store_true") | |
| ap.add_argument("--skip-op", action="store_true") | |
| ap.add_argument("--gold-extended", action="store_true") | |
| ap.add_argument("--coverage", action="store_true", help="coverage table only") | |
| ap.add_argument("--print-argv", action="store_true", help="print the evaluator commands without running them") | |
| ap.add_argument("--dry-run", action="store_true") | |
| ap.add_argument("--keep-dir", default="") | |
| ap.add_argument("--runs-root", default="", help="layout root (default results/runs); dry runs on partial layouts point elsewhere") | |
| ap.add_argument("--gold-root", default="", help="warning_gold out-dir root (default results/warning_gold)") | |
| ap.add_argument("--tables-dir", default="", help="table dir for dms_warning_comparison.md / dms_coverage.md (default results/tables)") | |
| ap.add_argument("--no-head", action="store_true", help="do not pass the frozen CockpitWM read-out layouts / skip the paired Δ vs head") | |
| ap.add_argument("--head-col", default=HEAD_COL) | |
| ap.add_argument("--coverage-note", default="", help="sentence appended to the coverage section header (e.g. which extraction tiers the caches cover)") | |
| ap.add_argument("--paired-prauc", nargs="+", default=[], metavar="GOLD_DIR", help="write <gold dir>/paired_vs_head_prauc.csv (PR-AUC-keyed aggregate; new file, never overwrites) and exit") | |
| a = ap.parse_args(argv) | |
| if a.paired_prauc: | |
| for d in a.paired_prauc: | |
| paired_vs_head_prauc(d) | |
| return 0 | |
| if a.dry_run: | |
| dry_run(a.keep_dir or None) | |
| return 0 | |
| arms = a.arms if len(a.arms) == len(a.seeds) else (a.arms * len(a.seeds) if len(a.arms) == 1 else a.arms) | |
| runs_root = Path(a.runs_root).resolve() if a.runs_root else RUNS # absolute: warning_gold.py runs with cwd = cwm2 | |
| tables_dir = Path(a.tables_dir).resolve() if a.tables_dir else TABLES | |
| if a.coverage: | |
| sec, _ = append_coverage(arms, a.seeds, a.col or None, table=tables_dir / "dms_coverage.md", root=runs_root, note=a.coverage_note) | |
| print(sec) | |
| return 0 | |
| assert a.baseline and arms and a.col, "--baseline, --arms and --col are required" | |
| evaluate(a.baseline, arms, a.seeds, a.col, a.label or a.baseline, a.status, a.loads, a.n_boot_gold, a.n_boot_op, a.skip_gold, a.skip_op, True, a.gold_extended, dry=a.print_argv, | |
| runs_root=runs_root, gold_root=(Path(a.gold_root).resolve() if a.gold_root else GOLD_ROOT), tables_dir=tables_dir, head=not a.no_head, head_col=a.head_col, coverage_note=a.coverage_note) | |
| return 0 | |
| if __name__ == "__main__": | |
| sys.exit(main()) | |
Xet Storage Details
- Size:
- 31.1 kB
- Xet hash:
- f8fe07a0bcfab5c10358bc582653990292a5e8bd92da753da2d14f64dc45bd31
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.