Buckets:
| """cwm3/eval/anticipation_setting.py — DESCRIPTIVE, POST HOC (not pre-registered) "anticipation setting" of the learned warning head vs the openpilot DMS attention chain | |
| (CPU only; existing artefacts only; no retraining). Question: do the equal-alert-load results support calling the head's warning "anticipatory" relative to the DMS chain? | |
| Setting (as in the earlier CockpitWM-2 AAAI draft: warn on the labelled positives BEFORE / WITHOUT the deployed monitor): gold-positive clean-test episodes of the cwm3 gold | |
| (512 clean-test rows, 511 with a finite DMS score, 63 positives; the cwm2 grid quoted 509 rows on the same gold), both systems at the equal-alert-load thresholds already | |
| published by eval/operating_points.py (load `dms_matched`: the head re-thresholded per regime to the DMS chain's alerts / hour at its own clean-val-recall-0.5 threshold, | |
| 60 s per-route cooldown; the DMS chain re-thresholded to the same load). Per regime and pooled, per seed (the head is seed-invariant; the DMS comparator varies by seed): | |
| (a) DMS-silent recall — among positives whose DMS att@3s score at the anchor t is BELOW the DMS equal-load threshold (DMS silent at t), the share the head flags at its | |
| own equal-load threshold at the same anchor (scores = the per-episode rows CSV of warning_gold). Comparators: the mirror image (head silent at t -> share the DMS | |
| flags), the DMS chain's overall equal-load recall, and the 2x2 (both / head-only / DMS-only / neither). The DMS chain's own recall on the DMS-silent subset is 0 by | |
| construction, so the mirror share is the like-for-like comparator. (a2) window variant: DMS never crossed in W = [t0 − lookback, t_anchor] -> share the head fired in W. | |
| (b) first-alert gap on the positives BOTH systems flag within W: Δ = lead_head − lead_dms = t_first_DMS − t_first_head (s; + = head earlier); median, IQR, device-cluster CI. | |
| (c) "warned earlier" shares — among positives where the DMS fires in W: share with a head alert (route scores) in the `--before` s BEFORE the DMS's first alert; and the | |
| reverse (DMS alert in the `--before` s before the head's first alert); plus the strict head-earlier / tie / DMS-earlier split of the both-detected pairs. | |
| Machinery reused, not re-implemented: operating_points.thr_for_rate (threshold re-derived from the route scores and checked against the published CSV), cboot (device-cluster | |
| bootstrap), rec_prec; train_warning_head.lead_times (first crossing in W, no cooldown — as in the published lead times) and alerts_per_hour; warning_gold.pct; the published | |
| per-episode lead_times.csv / paired_lead.csv of the operating-points run (tie-outs asserted). Lookback 30 s is reported as a sensitivity. | |
| Wording rule fixed in this script (post hoc): "anticipatory" is supported ONLY if, pooled, in >= 2/3 seeds, (a) Δ(head DMS-silent recall − mirror) >= +0.10 AND its CI excludes 0, | |
| AND (c) the strict head-earlier share among both-detected pairs > 0.5 AND (b) the median gap > 0 with a CI excluding 0. | |
| No openpilot DMS field is read as a model input: the DMS chain enters only as the labelled comparator (att@3s gold column; legacy marginals -> compose.legacy_compose att_only), | |
| exactly as in eval/operating_points.py. Outputs (append-only): results/anticipation_setting/<item>_<stamp>/{episodes.csv, summary.csv, log, run json} and a section appended to | |
| results/tables/anticipation_setting.md. | |
| python eval/anticipation_setting.py --item final_ext_dwfc --gold-item cwm3_final_ext --head-col "abl_W_dwfc::W_learned_curhgb_t15_w10_n979dwfc" | |
| """ | |
| from __future__ import annotations | |
| import argparse | |
| import sys | |
| import time | |
| from pathlib import Path | |
| import numpy as np | |
| import pandas as pd | |
| HERE = Path(__file__).resolve().parent | |
| sys.path.insert(0, str(HERE)) | |
| import operating_points as OP # noqa: E402 (thr_for_rate, cboot, rec_prec, TW, WG, legacy_compose, TAU, REGIMES, DMS_DIR) | |
| CWM3 = OP.CWM3 | |
| TW, WG = OP.TW, OP.WG | |
| REGIMES = OP.REGIMES | |
| CELLS = REGIMES + ("pooled",) | |
| OUT_ROOT = CWM3 / "results" / "anticipation_setting" | |
| GOLD_DIR = CWM3 / "results" / "warning_gold" | |
| TABLE = CWM3 / "results" / "tables" / "anticipation_setting.md" | |
| f3, ci = OP.f3, OP.ci | |
| def share(mask_num, mask_den): | |
| d = int(mask_den.sum()) | |
| return (float((mask_num & mask_den).sum()) / d if d else np.nan), d | |
| def boot(groups, fn, n_boot, seed): | |
| return OP.cboot(groups, fn, n_boot, seed) if len(groups) else np.array([np.nan]) | |
| def route_index(uid, t, s): | |
| """uid -> (t sorted, score sorted) for fast per-route window queries.""" | |
| order = np.lexsort((t, uid)) | |
| u, tt, ss = uid[order], t[order], s[order] | |
| cuts = np.flatnonzero(np.r_[True, u[1:] != u[:-1], True]) | |
| return {u[a]: (tt[a:b], ss[a:b]) for a, b in zip(cuts[:-1], cuts[1:])} | |
| def earliest_in(routes, uid, lo, hi, thr): | |
| """earliest alert (score >= thr) time in [lo, hi) on route uid, NaN if none / no rows.""" | |
| r = routes.get(uid) | |
| if r is None: | |
| return np.nan | |
| tt, ss = r | |
| m = (tt >= lo) & (tt < hi) & np.isfinite(ss) & (ss >= thr) | |
| return float(tt[m].min()) if m.any() else np.nan | |
| def main(): | |
| ap = argparse.ArgumentParser() | |
| ap.add_argument("--item", default="final_ext_dwfc", help="operating_points item (results/operating_points/<item>_<stamp>)") | |
| ap.add_argument("--op-run", default="", help="explicit operating_points run dir (default: latest of the item)") | |
| ap.add_argument("--gold-item", default="cwm3_final_ext", help="warning_gold item holding the head's per-episode scores") | |
| ap.add_argument("--gold-stamp", default="", help="warning_gold run_stamp (default: latest of the item)") | |
| ap.add_argument("--head-col", default="abl_W_dwfc::W_learned_curhgb_t15_w10_n979dwfc") | |
| ap.add_argument("--dms-col", default="att@3s") | |
| ap.add_argument("--label", default="HGB(current+dwell+face)") | |
| ap.add_argument("--load", default="dms_matched") | |
| ap.add_argument("--lookback", type=float, default=10.0, help="W = [t0 − lookback, t_anchor] (must equal the operating-points lookback for the tie-outs)") | |
| ap.add_argument("--lookback-sens", type=float, default=30.0) | |
| ap.add_argument("--before", type=float, default=10.0, help="(c): head alert within `before` s BEFORE the DMS first alert (and the reverse)") | |
| ap.add_argument("--n-boot", type=int, default=1000) | |
| ap.add_argument("--skip-rederive", action="store_true", help="skip the thr_for_rate re-derivation check (uses the published thresholds either way)") | |
| ap.add_argument("--out", default=str(TABLE)) | |
| ap.add_argument("--out-root", default=str(OUT_ROOT), help="run directories root (default results/anticipation_setting)") | |
| ap.add_argument("--note", default="", help="free-text note placed in the section header (e.g. what this run supersedes)") | |
| a = ap.parse_args() | |
| t_start = time.time() | |
| stamp = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime()) | |
| out = Path(a.out_root) / f"{a.item}_{stamp}" | |
| out.mkdir(parents=True, exist_ok=True) | |
| log_p = out / "anticipation_setting.log" | |
| def log(m): | |
| line = f"[{time.strftime('%H:%M:%S')}] {m}" | |
| print(line, flush=True) | |
| with open(log_p, "a") as f: | |
| f.write(line + "\n") | |
| # ---- published operating-points artefacts | |
| op_run = Path(a.op_run) if a.op_run else sorted(d for d in OP.OUT_ROOT.glob(f"{a.item}_*") if d.is_dir())[-1] | |
| OPS = pd.read_csv(op_run / "operating_points.csv") | |
| LT = pd.read_csv(op_run / "lead_times.csv") | |
| PR = pd.read_csv(op_run / "paired_lead.csv") | |
| OPS, LT, PR = OPS[OPS.load == a.load], LT[(LT.load == a.load) & (LT.regime_cell != "pooled")], PR[PR.load == a.load] | |
| assert len(OPS) and len(LT) and len(PR), (op_run, a.load) | |
| seeds = sorted(int(s) for s in OPS.seed.unique()) | |
| w_run = {int(r.seed): r.w_run for r in OPS.itertuples()} | |
| thr = {(int(r.seed), r.score, reg): float(getattr(r, f"thr_{reg}")) for r in OPS[OPS.regime == "pooled"].itertuples() for reg in REGIMES} | |
| load_target = {(int(r.seed), reg): float(getattr(r, f"load_target_{reg}")) for r in OPS[(OPS.regime == "pooled") & (OPS.score == "head")].itertuples() for reg in REGIMES} # per-seed DMS load | |
| # ---- per-episode gold scores at the anchor (warning_gold rows CSV of the gold item) | |
| W = pd.read_csv(GOLD_DIR / "warning_gold.csv", usecols=["item", "run_stamp"], low_memory=False) | |
| gstamp = a.gold_stamp or W[W.item == a.gold_item].run_stamp.max() | |
| rows_csv = GOLD_DIR / f"warn_gold_rows_{gstamp}.csv" | |
| R = pd.read_csv(rows_csv, low_memory=False) | |
| assert a.head_col in R.columns and a.dms_col in R.columns, (a.head_col, a.dms_col, rows_csv) | |
| R = R[R.clean_test.astype(bool)].copy() | |
| log(f"item {a.item} <- {op_run.name} (seeds {seeds}, load {a.load}, targets {load_target}); gold rows {rows_csv.name} ({a.gold_item}): clean-test {R[R.seed == seeds[0]].shape[0]} rows, " | |
| f"{int(R[R.seed == seeds[0]].y.sum())} positives, finite DMS {int(R[R.seed == seeds[0]][a.dms_col].notna().sum())}; head col {a.head_col}") | |
| lookbacks = [a.lookback] + ([a.lookback_sens] if a.lookback_sens and a.lookback_sens != a.lookback else []) | |
| EP, SUM, checks = [], [], [] | |
| for seed in seeds: | |
| g = R[R.seed == seed].copy() | |
| g["thr_head"] = np.array([thr[(seed, "head", r)] for r in g.regime]) | |
| g["thr_dms"] = np.array([thr[(seed, "dms", r)] for r in g.regime]) | |
| g["head_anchor"], g["dms_anchor"] = g[a.head_col].to_numpy(float), g[a.dms_col].to_numpy(float) | |
| g["head_flag"] = np.isfinite(g.head_anchor) & (g.head_anchor >= g.thr_head) | |
| g["dms_flag"] = np.isfinite(g.dms_anchor) & (g.dms_anchor >= g.thr_dms) | |
| # tie-out 1: recall at the anchor == published equal-load recall (all gold rows, per regime / pooled) | |
| for reg in CELLS: | |
| m = (g.regime == reg) if reg != "pooled" else np.ones(len(g), bool) | |
| for sc, fl, an in (("head", "head_flag", "head_anchor"), ("dms", "dms_flag", "dms_anchor")): | |
| mm = m & np.isfinite(g[an]) | |
| rc = OP.rec_prec(g.y[mm].to_numpy(), g[fl][mm].to_numpy())[0] | |
| pub = float(OPS[(OPS.seed == seed) & (OPS.regime == reg) & (OPS.score == sc)].recall.iloc[0]) | |
| checks.append({"seed": seed, "check": f"anchor recall {sc} {reg}", "ours": rc, "published": pub, "ok": abs(rc - pub) < 1e-9}) | |
| # ---- route scores (test split): head layout column, DMS = legacy marginals -> att_only (comparator only) | |
| routes, rroutes = {}, {} | |
| for reg in REGIMES: | |
| z = np.load(CWM3 / "results" / "runs" / w_run[seed] / f"pred_layout_{reg}_test.npz", allow_pickle=True) | |
| names = list(map(str, z["w_names"])) | |
| col = a.head_col.split("::", 1)[1] | |
| assert col in names, (col, names) | |
| routes[("head", reg)] = (z["uid"], z["t"].astype(float), z["w"][:, names.index(col)].astype(np.float64)) | |
| zd = np.load(OP.DMS_DIR / f"marginals_{reg}_s{seed}.npz", allow_pickle=True) | |
| routes[("dms", reg)] = (zd["uid"], zd["t"].astype(float), OP.legacy_compose(zd["pA"].astype(np.float32), None, OP.TAU)["att_only"].astype(np.float64)) | |
| for sc in ("head", "dms"): | |
| rroutes[(sc, reg)] = route_index(*routes[(sc, reg)]) | |
| if not a.skip_rederive: | |
| t1 = time.time() | |
| th = OP.thr_for_rate(*routes[(sc, reg)], load_target[(seed, reg)]) | |
| checks.append({"seed": seed, "check": f"thr_for_rate {sc} {reg}", "ours": th, "published": thr[(seed, sc, reg)], "ok": abs(th - thr[(seed, sc, reg)]) < 1e-9}) | |
| log(f" seed {seed} {sc} {reg}: re-derived thr {th:.6f} vs published {thr[(seed, sc, reg)]:.6f} ({time.time() - t1:.0f}s)") | |
| P = g[g.y == 1].copy() | |
| for L in lookbacks: | |
| for sc in ("head", "dms"): | |
| lead = np.full(len(P), np.nan) | |
| for reg in REGIMES: | |
| mr = (P.regime == reg).to_numpy() | |
| if mr.any(): | |
| lead[mr] = TW.lead_times(P[mr], {(reg, "test"): routes[(sc, reg)]}, thr[(seed, sc, reg)], L) | |
| P[f"lead_{sc}_L{L:g}"] = lead | |
| P[f"t_first_{sc}_L{L:g}"] = P.t0.to_numpy() - lead | |
| if L == a.lookback: # tie-out 2: leads == published lead_times.csv (same thresholds, same window) | |
| for sc in ("head", "dms"): | |
| pub = LT[(LT.seed == seed) & (LT.score == sc)].set_index("idx").lead_s.reindex(P.idx).to_numpy(float) | |
| ours = P[f"lead_{sc}_L{L:g}"].to_numpy(float) | |
| same = (np.isnan(pub) & np.isnan(ours)) | (np.abs(pub - ours) < 1e-6) | |
| checks.append({"seed": seed, "check": f"lead_times {sc} L{L:g} == published", "ours": float(same.mean()), "published": 1.0, "ok": bool(same.all())}) | |
| # (c) route-level: alert of the other system in the `before` s BEFORE this system's first alert in W | |
| for sc, other in (("head", "dms"), ("dms", "head")): | |
| tf = P[f"t_first_{sc}_L{L:g}"].to_numpy(float) | |
| e = np.array([earliest_in(rroutes[(other, r)], u, t - a.before, t, thr[(seed, other, r)]) if np.isfinite(t) else np.nan for u, r, t in zip(P.uid, P.regime, tf)]) | |
| P[f"{other}_before_{sc}_L{L:g}"] = np.isfinite(e) # other fired in the `before` s before sc's first alert | |
| P[f"{other}_earlier_by_L{L:g}"] = tf - e # seconds the other system was earlier (NaN if not) | |
| EP.append(P) | |
| # ---- summaries per cell | |
| for L in lookbacks: | |
| lh, ld = P[f"lead_head_L{L:g}"].to_numpy(float), P[f"lead_dms_L{L:g}"].to_numpy(float) | |
| dh, dd = np.isfinite(lh), np.isfinite(ld) | |
| gap = lh - ld # + = head earlier | |
| hb, db = P[f"head_before_dms_L{L:g}"].to_numpy(bool), P[f"dms_before_head_L{L:g}"].to_numpy(bool) | |
| heb, deb = P[f"head_earlier_by_L{L:g}"].to_numpy(float), P[f"dms_earlier_by_L{L:g}"].to_numpy(float) | |
| hf, df_, fin_d = P.head_flag.to_numpy(bool), P.dms_flag.to_numpy(bool), np.isfinite(P.dms_anchor.to_numpy(float)) | |
| dong = P.dongle.to_numpy() | |
| for reg in CELLS: | |
| m = (P.regime == reg).to_numpy() if reg != "pooled" else np.ones(len(P), bool) | |
| row = {"item": a.item, "seed": seed, "lookback": L, "regime": reg, "n_pos": int(m.sum()), "n_dev": int(len(np.unique(dong[m])))} | |
| B = lambda fn, sd: WG.pct(boot(dong[m], fn, a.n_boot, sd)) # noqa: E731 | |
| idx = np.flatnonzero(m) | |
| if L == a.lookback: # (a) anchor-instant (independent of L) | |
| sil_d, sil_h = m & fin_d & ~df_, m & ~hf # DMS silent at t (evaluable), head silent at t | |
| row["n_dms_evaluable"] = int((m & fin_d).sum()) | |
| row["a_dms_silent_n"] = int(sil_d.sum()); row["a_head_recall_dms_silent"] = share(hf, sil_d)[0] | |
| row["a_head_silent_n"] = int(sil_h.sum()); row["a_dms_recall_head_silent_mirror"] = share(df_ & fin_d, sil_h)[0] | |
| row["a_dms_overall_recall"] = share(df_, m & fin_d)[0]; row["a_head_overall_recall"] = share(hf, m)[0] | |
| row["a_delta_vs_mirror"] = row["a_head_recall_dms_silent"] - row["a_dms_recall_head_silent_mirror"] | |
| row["a_both"], row["a_head_only"], row["a_dms_only"], row["a_neither"] = (int((m & fin_d & hf & df_).sum()), int((m & fin_d & hf & ~df_).sum()), | |
| int((m & fin_d & ~hf & df_).sum()), int((m & fin_d & ~hf & ~df_).sum())) | |
| fa = lambda pk: share(hf[idx[pk]], (fin_d & ~df_)[idx[pk]])[0] # noqa: E731 | |
| fm = lambda pk: share((df_ & fin_d)[idx[pk]], (~hf)[idx[pk]])[0] # noqa: E731 | |
| row["a_head_recall_dms_silent_ci_lo"], row["a_head_recall_dms_silent_ci_hi"] = B(fa, 10) | |
| row["a_dms_recall_head_silent_mirror_ci_lo"], row["a_dms_recall_head_silent_mirror_ci_hi"] = B(fm, 10) | |
| row["a_delta_vs_mirror_ci_lo"], row["a_delta_vs_mirror_ci_hi"] = B(lambda pk: fa(pk) - fm(pk), 10) | |
| # (a2) window variant | |
| row["a2_dms_silent_W_n"] = int((m & ~dd).sum()); row["a2_head_fired_W_given_dms_silent"] = share(dh, m & ~dd)[0] | |
| row["a2_head_silent_W_n"] = int((m & ~dh).sum()); row["a2_dms_fired_W_given_head_silent_mirror"] = share(dd, m & ~dh)[0] | |
| row["a2_delta_vs_mirror"] = row["a2_head_fired_W_given_dms_silent"] - row["a2_dms_fired_W_given_head_silent_mirror"] | |
| fa2 = lambda pk: share(dh[idx[pk]], (~dd)[idx[pk]])[0] # noqa: E731 | |
| fm2 = lambda pk: share(dd[idx[pk]], (~dh)[idx[pk]])[0] # noqa: E731 | |
| row["a2_head_fired_W_given_dms_silent_ci_lo"], row["a2_head_fired_W_given_dms_silent_ci_hi"] = B(fa2, 11) | |
| row["a2_dms_fired_W_given_head_silent_mirror_ci_lo"], row["a2_dms_fired_W_given_head_silent_mirror_ci_hi"] = B(fm2, 11) | |
| row["a2_delta_vs_mirror_ci_lo"], row["a2_delta_vs_mirror_ci_hi"] = B(lambda pk: fa2(pk) - fm2(pk), 11) | |
| row["det_head_W"], row["det_dms_W"] = share(dh, m)[0], share(dd, m)[0] | |
| # (b) both detected in W | |
| both = m & dh & dd | |
| row["b_n_both"] = int(both.sum()) | |
| q = np.percentile(gap[both], [25, 50, 75]) if both.any() else [np.nan] * 3 | |
| row["b_gap_q25"], row["b_gap_median"], row["b_gap_q75"], row["b_gap_mean"] = q[0], q[1], q[2], (float(gap[both].mean()) if both.any() else np.nan) | |
| fb = lambda pk: (float(np.median(gap[idx[pk]][both[idx[pk]]])) if both[idx[pk]].any() else np.nan) # noqa: E731 | |
| row["b_gap_median_ci_lo"], row["b_gap_median_ci_hi"] = B(fb, 12) | |
| row["b_share_head_earlier"], row["b_share_tie"], row["b_share_dms_earlier"] = (share(gap > 0.1, both)[0], share(np.abs(gap) <= 0.1, both)[0], share(gap < -0.1, both)[0]) | |
| row["b_share_head_earlier_ci_lo"], row["b_share_head_earlier_ci_hi"] = B(lambda pk: share((gap > 0.1)[idx[pk]], both[idx[pk]])[0], 12) | |
| # (c) route-level `before`-window shares | |
| row["c_n_dms_fired_W"] = int((m & dd).sum()); row["c_head_before_dms"] = share(hb, m & dd)[0] | |
| row["c_head_before_dms_ci_lo"], row["c_head_before_dms_ci_hi"] = B(lambda pk: share(hb[idx[pk]], dd[idx[pk]])[0], 13) | |
| row["c_head_earlier_by_median"] = float(np.nanmedian(heb[m & dd & hb])) if (m & dd & hb).any() else np.nan | |
| row["c_n_head_fired_W"] = int((m & dh).sum()); row["c_dms_before_head"] = share(db, m & dh)[0] | |
| row["c_dms_before_head_ci_lo"], row["c_dms_before_head_ci_hi"] = B(lambda pk: share(db[idx[pk]], dh[idx[pk]])[0], 13) | |
| row["c_dms_earlier_by_median"] = float(np.nanmedian(deb[m & dh & db])) if (m & dh & db).any() else np.nan | |
| row["c_delta_shares"] = row["c_head_before_dms"] - row["c_dms_before_head"] | |
| row["c_delta_shares_ci_lo"], row["c_delta_shares_ci_hi"] = B(lambda pk: share(hb[idx[pk]], dd[idx[pk]])[0] - share(db[idx[pk]], dh[idx[pk]])[0], 13) | |
| SUM.append(row) | |
| log(f" seed {seed} L{L:g} {reg:8s}: n {row['n_pos']}; (a) DMS-silent {row.get('a_dms_silent_n', '-')} -> head {f3(row.get('a_head_recall_dms_silent', np.nan), 2)} | mirror {f3(row.get('a_dms_recall_head_silent_mirror', np.nan), 2)}; " | |
| f"(a2) {row['a2_dms_silent_W_n']} -> {f3(row['a2_head_fired_W_given_dms_silent'], 2)} | mirror {f3(row['a2_dms_fired_W_given_head_silent_mirror'], 2)}; (b) both {row['b_n_both']} median gap {f3(row['b_gap_median'], 2)} " | |
| f"[{f3(row['b_gap_q25'], 2)}, {f3(row['b_gap_q75'], 2)}] head-earlier/tie/DMS-earlier {f3(row['b_share_head_earlier'], 2)}/{f3(row['b_share_tie'], 2)}/{f3(row['b_share_dms_earlier'], 2)}; " | |
| f"(c) head-before-DMS {f3(row['c_head_before_dms'], 2)} (n {row['c_n_dms_fired_W']}) vs DMS-before-head {f3(row['c_dms_before_head'], 2)} (n {row['c_n_head_fired_W']})") | |
| # tie-out 3: (b) median gap on both-detected == paired_lead.csv delta_median_both | |
| for reg in CELLS: | |
| pub = float(PR[(PR.seed == seed) & (PR.regime == reg)].delta_median_both.iloc[0]) | |
| ours = [r for r in SUM if r["seed"] == seed and r["lookback"] == a.lookback and r["regime"] == reg][0]["b_gap_median"] | |
| checks.append({"seed": seed, "check": f"(b) median gap {reg} == paired_lead delta_median_both", "ours": ours, "published": pub, "ok": abs(ours - pub) < 1e-6}) | |
| S, E, C = pd.DataFrame(SUM), pd.concat(EP, ignore_index=True), pd.DataFrame(checks) | |
| S.to_csv(out / "summary.csv", index=False) | |
| E.to_csv(out / "episodes.csv", index=False) | |
| C.to_csv(out / "checks.csv", index=False) | |
| n_bad = int((~C.ok).sum()) | |
| log(f"consistency checks: {int(C.ok.sum())}/{len(C)} passed" + (f"; FAILED: {C[~C.ok].to_dict('records')}" if n_bad else "")) | |
| # ---- verdict (rule fixed in the docstring; pooled; >= 2/3 seeds) | |
| S0 = S[(S.lookback == a.lookback) & (S.regime == "pooled")].sort_values("seed") | |
| ok_a = [bool((r.a_delta_vs_mirror >= 0.10) and (r.a_delta_vs_mirror_ci_lo > 0)) for r in S0.itertuples()] | |
| ok_c = [bool((r.b_share_head_earlier > 0.5) and (r.b_gap_median > 0) and (r.b_gap_median_ci_lo > 0)) for r in S0.itertuples()] | |
| ok_both = [x and y for x, y in zip(ok_a, ok_c)] | |
| supported = sum(ok_both) >= 2 | |
| verdict = "SUPPORTED" if supported else "NOT SUPPORTED" | |
| # ---- table section (append-only) | |
| Lk = a.lookback | |
| H = [f"## item `{a.item}` — {a.label} (`{a.head_col}`) vs DMS att@3s — anticipation setting (descriptive, post hoc, not pre-registered) — run {stamp}", ""] + ([f"Note: {a.note}", ""] if a.note else []) + [ | |
| f"**Does the evidence support \"anticipatory\" wording relative to the openpilot DMS chain? {verdict}.** Rule (fixed in `eval/anticipation_setting.py`, pooled stratum, ≥ 2/3 seeds): " | |
| f"(a) head recall on the DMS-silent positives minus the mirror (DMS recall on the head-silent positives) ≥ +0.10 with a device-cluster CI excluding 0 — met in {sum(ok_a)}/{len(ok_a)} seeds; " | |
| f"AND (c) strict head-earlier share among the both-detected pairs > 0.5 with (b) median first-alert gap > 0 and CI excluding 0 — met in {sum(ok_c)}/{len(ok_c)} seeds. Pooled seed means: " | |
| f"(a) head {S0.a_head_recall_dms_silent.mean():.3f} vs mirror {S0.a_dms_recall_head_silent_mirror.mean():.3f} (Δ {S0.a_delta_vs_mirror.mean():+.3f}); DMS overall equal-load recall {S0.a_dms_overall_recall.mean():.3f}; " | |
| f"(b) median gap {S0.b_gap_median.mean():+.2f} s on {int(S0.b_n_both.mean())} both-detected pairs (mean over seeds); (c) head fires in the {a.before:g} s before the DMS's first alert in {S0.c_head_before_dms.mean():.2f} of the " | |
| f"DMS-fired positives vs DMS in the {a.before:g} s before the head's first alert in {S0.c_dms_before_head.mean():.2f} of the head-fired positives; strict head-earlier / tie / DMS-earlier among both-detected " | |
| f"{S0.b_share_head_earlier.mean():.2f} / {S0.b_share_tie.mean():.2f} / {S0.b_share_dms_earlier.mean():.2f}.", "", | |
| f"Definitions. Gold = the cwm3 clean human test set (unchanged `warning_gold.build_gold`, legacy_first anchors): {int(R[R.seed == seeds[0]].shape[0])} clean-test rows, " | |
| f"{int(R[R.seed == seeds[0]][a.dms_col].notna().sum())} with a finite DMS score, {int(R[R.seed == seeds[0]].y.sum())} positives (the cwm2 grid quoted 509 rows on the same gold); only the positives enter. " | |
| f"Thresholds = the published equal-alert-load (`{a.load}`) thresholds of `results/operating_points/{op_run.name}/operating_points.csv` (head per regime by bisection to the DMS chain's alerts / hour at its own " | |
| f"clean-val-recall-0.5 threshold, 60 s per-route cooldown; DMS re-thresholded to the same load), re-derived here with `operating_points.thr_for_rate` from the route scores as a check. " | |
| f"Anchor scores = the per-episode rows CSV `{rows_csv.name}` (head `{a.head_col}`, DMS `{a.dms_col}`). W = [t0 − {Lk:g} s, t_anchor]; first alert = first threshold crossing in W " | |
| f"(`train_warning_head.lead_times`, no cooldown, as in the published lead times); lead = t0 − first alert. (a) DMS silent at t := DMS anchor score < DMS threshold (finite scores only); head flags := head anchor score ≥ head threshold; " | |
| f"mirror := DMS flags among positives with head silent at t; the DMS chain's own recall on the DMS-silent subset is 0 by construction. (a2) DMS silent in W := no DMS crossing in W. (b) gap = lead_head − lead_dms " | |
| f"= t_first_DMS − t_first_head on the both-detected positives (+ = head earlier; ties = |gap| ≤ 0.1 s, one 0.2 s step). (c) head-before-DMS := any head route-score crossing in [t_first_DMS − {a.before:g} s, t_first_DMS) " | |
| f"among positives where the DMS fires in W; DMS-before-head := the reverse; \"earlier by\" = distance from that earliest crossing to the other system's first alert. CAVEAT on (c): these shares count ANY raw threshold crossing (5 Hz, no cooldown) in the pre-window, including crossings earlier than W, and are not corrected for either system's base crossing rate on the route — both systems cross often at ~50 alerts / h — so they overstate \"warned earlier\" for both sides symmetrically; the decision uses the strict first-alert ordering in (b). CIs = device-cluster bootstrap ({a.n_boot} resamples, " | |
| f"clusters = dongles of the positives). The head is seed-invariant (current-state features); the DMS comparator (legacy marginals) varies by seed, so seeds are reported separately plus their mean. " | |
| f"Consistency checks: {int(C.ok.sum())}/{len(C)} passed (anchor recall = published equal-load recall; re-derived thresholds = published; W leads = published `lead_times.csv`; (b) medians = published `paired_lead.csv`)" | |
| + (f" — FAILED: {'; '.join(f'{r.check} s{r.seed} ours {r.ours} vs {r.published}' for r in C[~C.ok].itertuples())}" if n_bad else "") + f". Raw rows: `results/anticipation_setting/{out.name}/` (episodes.csv, summary.csv, checks.csv).", ""] | |
| def mean_row(g): | |
| return g.mean(numeric_only=True) | |
| H += [f"### (a) Recall in the anticipation setting — DMS chain silent at the anchor (W-independent; head vs mirror; n = positives)", "", | |
| "| regime | seed | n pos (DMS-evaluable) | n DMS-silent | head flags given DMS silent [CI] | n head-silent | DMS flags given head silent (mirror) [CI] | Δ head − mirror [CI] | DMS overall recall | head overall recall | both / head-only / DMS-only / neither |", | |
| "|---|---|---|---|---|---|---|---|---|---|---|"] | |
| Sa = S[S.lookback == Lk] | |
| for reg in CELLS: | |
| g = Sa[Sa.regime == reg].sort_values("seed") | |
| for r in g.itertuples(): | |
| H.append(f"| {reg} | {r.seed} | {r.n_pos} ({int(r.n_dms_evaluable)}) | {int(r.a_dms_silent_n)} | {f3(r.a_head_recall_dms_silent)} [{f3(r.a_head_recall_dms_silent_ci_lo, 2)}, {f3(r.a_head_recall_dms_silent_ci_hi, 2)}] | {int(r.a_head_silent_n)} | " | |
| f"{f3(r.a_dms_recall_head_silent_mirror)} [{f3(r.a_dms_recall_head_silent_mirror_ci_lo, 2)}, {f3(r.a_dms_recall_head_silent_mirror_ci_hi, 2)}] | {r.a_delta_vs_mirror:+.3f} [{r.a_delta_vs_mirror_ci_lo:+.2f}, {r.a_delta_vs_mirror_ci_hi:+.2f}] | " | |
| f"{f3(r.a_dms_overall_recall)} | {f3(r.a_head_overall_recall)} | {int(r.a_both)} / {int(r.a_head_only)} / {int(r.a_dms_only)} / {int(r.a_neither)} |") | |
| mr = mean_row(g) | |
| H.append(f"| **{reg}** | **mean** | {int(mr.n_pos)} ({mr.n_dms_evaluable:.1f}) | {mr.a_dms_silent_n:.1f} | **{f3(mr.a_head_recall_dms_silent)}** | {mr.a_head_silent_n:.1f} | **{f3(mr.a_dms_recall_head_silent_mirror)}** | **{mr.a_delta_vs_mirror:+.3f}** | " | |
| f"{f3(mr.a_dms_overall_recall)} | {f3(mr.a_head_overall_recall)} | {mr.a_both:.1f} / {mr.a_head_only:.1f} / {mr.a_dms_only:.1f} / {mr.a_neither:.1f} |") | |
| for L in lookbacks: | |
| SL = S[S.lookback == L] | |
| tag = "" if L == Lk else f" — SENSITIVITY, lookback {L:g} s" | |
| H += ["", f"### (a2) Window variant, W = [t0 − {L:g} s, t_anchor]: head fired in W given the DMS never crossed in W (and the mirror){tag}", "", | |
| "| regime | seed | n pos | detected in W head / DMS | n DMS-silent-in-W | head fired in W given DMS silent [CI] | n head-silent-in-W | DMS fired in W given head silent (mirror) [CI] | Δ head − mirror [CI] |", "|---|---|---|---|---|---|---|---|---|"] | |
| for reg in CELLS: | |
| g = SL[SL.regime == reg].sort_values("seed") | |
| for r in g.itertuples(): | |
| H.append(f"| {reg} | {r.seed} | {r.n_pos} | {r.det_head_W:.2f} / {r.det_dms_W:.2f} | {r.a2_dms_silent_W_n} | {f3(r.a2_head_fired_W_given_dms_silent)} [{f3(r.a2_head_fired_W_given_dms_silent_ci_lo, 2)}, {f3(r.a2_head_fired_W_given_dms_silent_ci_hi, 2)}] | " | |
| f"{r.a2_head_silent_W_n} | {f3(r.a2_dms_fired_W_given_head_silent_mirror)} [{f3(r.a2_dms_fired_W_given_head_silent_mirror_ci_lo, 2)}, {f3(r.a2_dms_fired_W_given_head_silent_mirror_ci_hi, 2)}] | " | |
| f"{r.a2_delta_vs_mirror:+.3f} [{r.a2_delta_vs_mirror_ci_lo:+.2f}, {r.a2_delta_vs_mirror_ci_hi:+.2f}] |") | |
| mr = mean_row(g) | |
| H.append(f"| **{reg}** | **mean** | {int(mr.n_pos)} | {mr.det_head_W:.2f} / {mr.det_dms_W:.2f} | {mr.a2_dms_silent_W_n:.1f} | **{f3(mr.a2_head_fired_W_given_dms_silent)}** | {mr.a2_head_silent_W_n:.1f} | **{f3(mr.a2_dms_fired_W_given_head_silent_mirror)}** | **{mr.a2_delta_vs_mirror:+.3f}** |") | |
| H += ["", f"### (b) First-alert gap on the positives both systems flag in W (gap = t_first_DMS − t_first_head, + = head earlier){tag}", "", | |
| "| regime | seed | n both-detected | median gap s [CI] | IQR | mean | head earlier / tie / DMS earlier (strict, ±0.1 s) | head-earlier share CI |", "|---|---|---|---|---|---|---|---|"] | |
| for reg in CELLS: | |
| g = SL[SL.regime == reg].sort_values("seed") | |
| for r in g.itertuples(): | |
| H.append(f"| {reg} | {r.seed} | {r.b_n_both} | {r.b_gap_median:+.2f} [{r.b_gap_median_ci_lo:+.2f}, {r.b_gap_median_ci_hi:+.2f}] | [{f3(r.b_gap_q25, 2)}, {f3(r.b_gap_q75, 2)}] | {r.b_gap_mean:+.2f} | " | |
| f"{f3(r.b_share_head_earlier, 2)} / {f3(r.b_share_tie, 2)} / {f3(r.b_share_dms_earlier, 2)} | [{f3(r.b_share_head_earlier_ci_lo, 2)}, {f3(r.b_share_head_earlier_ci_hi, 2)}] |") | |
| mr = mean_row(g) | |
| H.append(f"| **{reg}** | **mean** | {mr.b_n_both:.1f} | **{mr.b_gap_median:+.2f}** | [{f3(mr.b_gap_q25, 2)}, {f3(mr.b_gap_q75, 2)}] | {mr.b_gap_mean:+.2f} | **{f3(mr.b_share_head_earlier, 2)} / {f3(mr.b_share_tie, 2)} / {f3(mr.b_share_dms_earlier, 2)}** | |") | |
| H += ["", f"### (c) \"Warned earlier\" shares — other system's route-score crossing in the {a.before:g} s BEFORE this system's first alert in W{tag}", "", | |
| f"| regime | seed | n DMS-fired in W | head alert in the {a.before:g} s before DMS's first alert [CI] | head earlier by, median s | n head-fired in W | DMS alert in the {a.before:g} s before head's first alert [CI] | DMS earlier by, median s | Δ shares (head-before-DMS − DMS-before-head) [CI] |", | |
| "|---|---|---|---|---|---|---|---|---|"] | |
| for reg in CELLS: | |
| g = SL[SL.regime == reg].sort_values("seed") | |
| for r in g.itertuples(): | |
| H.append(f"| {reg} | {r.seed} | {r.c_n_dms_fired_W} | {f3(r.c_head_before_dms)} [{f3(r.c_head_before_dms_ci_lo, 2)}, {f3(r.c_head_before_dms_ci_hi, 2)}] | {f3(r.c_head_earlier_by_median, 1)} | {r.c_n_head_fired_W} | " | |
| f"{f3(r.c_dms_before_head)} [{f3(r.c_dms_before_head_ci_lo, 2)}, {f3(r.c_dms_before_head_ci_hi, 2)}] | {f3(r.c_dms_earlier_by_median, 1)} | {r.c_delta_shares:+.3f} [{r.c_delta_shares_ci_lo:+.2f}, {r.c_delta_shares_ci_hi:+.2f}] |") | |
| mr = mean_row(g) | |
| H.append(f"| **{reg}** | **mean** | {mr.c_n_dms_fired_W:.1f} | **{f3(mr.c_head_before_dms)}** | {f3(mr.c_head_earlier_by_median, 1)} | {mr.c_n_head_fired_W:.1f} | **{f3(mr.c_dms_before_head)}** | {f3(mr.c_dms_earlier_by_median, 1)} | **{mr.c_delta_shares:+.3f}** |") | |
| H += ["", "### Decision", "", "| date | analysis | status | rule | per-seed result (pooled, W = 10 s) | verdict |", "|---|---|---|---|---|---|", | |
| f"| {time.strftime('%Y-%m-%d', time.gmtime())} | anticipation setting, item `{a.item}` ({a.label} vs DMS att@3s at equal alert load) | **descriptive, post hoc, NOT pre-registered** (rule fixed in the script before the run, after the equal-load results were known) | " | |
| f"(a) Δ(head recall on DMS-silent − mirror) ≥ +0.10 & CI excl. 0; AND (c) strict head-earlier share > 0.5 & (b) median gap > 0 with CI excl. 0; pooled; ≥ 2/3 seeds | " | |
| + "; ".join(f"s{r.seed}: (a) {r.a_delta_vs_mirror:+.3f} [{r.a_delta_vs_mirror_ci_lo:+.2f}, {r.a_delta_vs_mirror_ci_hi:+.2f}] {'PASS' if oa else 'fail'}; (c) share {r.b_share_head_earlier:.2f}, (b) median {r.b_gap_median:+.2f} [{r.b_gap_median_ci_lo:+.2f}, {r.b_gap_median_ci_hi:+.2f}] {'PASS' if oc else 'fail'}" | |
| for r, oa, oc in zip(S0.itertuples(), ok_a, ok_c)) | |
| + f" | **\"anticipatory\" wording {verdict}** ({sum(ok_both)}/{len(ok_both)} seeds meet both) |", ""] | |
| md = "\n".join(H) + "\n" | |
| outp = Path(a.out) | |
| if not outp.exists(): | |
| outp.write_text("# Anticipation setting — learned warning head vs the DMS attention chain on the gold positives (eval/anticipation_setting.py; descriptive, post hoc; sections appended per run)\n\n") | |
| with open(outp, "a") as f: | |
| f.write(md) | |
| print(md) | |
| OP.write_run_json(out, "anticipation_setting", [Path(__file__), Path(OP.__file__), Path(TW.__file__), Path(WG.__file__)], | |
| [op_run / "operating_points.csv", op_run / "lead_times.csv", op_run / "paired_lead.csv", rows_csv], seed=0, t_start=t_start, input_listing=False, | |
| extra={"item": a.item, "op_run": str(op_run), "gold_item": a.gold_item, "gold_stamp": gstamp, "head_col": a.head_col, "dms_col": a.dms_col, "load": a.load, "lookback": a.lookback, | |
| "lookback_sens": a.lookback_sens, "before": a.before, "n_boot": a.n_boot, "verdict": verdict, "ok_a": ok_a, "ok_c": ok_c, "checks": checks, | |
| "pooled_seed_rows": S0.to_dict("records"), "status": "descriptive, post hoc, not pre-registered", "dms_fields_used": []}) | |
| log(f"DONE in {time.time() - t_start:.0f}s -> {out}; section appended to {outp}; verdict {verdict}") | |
| if __name__ == "__main__": | |
| main() | |
Xet Storage Details
- Size:
- 35.1 kB
- Xet hash:
- 5f21db6cc86e9f5f58fcf9cd41f33bad1f26ad3a5173ca3fc0779503b2532929
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.