Buckets:
| #!/usr/bin/env python3 | |
| """cwm3/eval/gold_event_recall.py — pre-registered evaluation of the human GOLD EVENT SET for the engaged hand cells | |
| (docs/decisions.md 2026-09-17T00:28:00Z pre-registration; amendments 00:34:39Z (blocks W / MR), 00:46:01Z (per-route cap 8), 01:05:57Z + 01:21Z | |
| (block RE = VLM-ranked candidates, ordering only; v3 candidate precision on the random R blocks only, RE separately)). CPU only, numpy / pandas. | |
| python eval/gold_event_recall.py [--apply] [--seeds 0 1 2] [--budgets 10 20 40] [--deltas 0.5 1 2] [--gates v3 v2 none] [--n-boot 1000] | |
| [--gold-json results/review_tool/gold_engaged_events.json] [--out results/tables/gold_engaged_recall.md] [--json-out PATH] | |
| Gold events tools/review_tool/apply_gold_engaged.py (run first with --apply, or beforehand) turns the owner's labels (results/review_tool/labels_latest.json, | |
| ids G_* / GE_* / GM_* / GW_*) into results/review_tool/gold_engaged_events.json: t* = candidate time (answer 1, at_event) or the clicked frame | |
| (answer 2, ret_dt / off_dt); answers 3 / 4 (never / always) = v3 candidate false positives; unsure / unlabelled excluded. Cells = regime x event | |
| type (engaged hand_return, engaged hand_leave, manual hand_return = block MR). A cell with < MIN_GOLD = 20 gold events is INSUFFICIENT (numbers | |
| shown, no verdicts); the manual cell is evaluated only when it has >= 20 gold events. | |
| Systems frozen joint layouts results/runs/<run>/pred_layout_{regime}_test.npz: main = I1_all_minus_road2drv_s{seed}_final, indep = I1_indep_s{seed}_final, | |
| naive = I1_naive_s{seed}_final, ecan = I1_all_minus_road2drv_s{seed}_final_ecan; score per anchor = onset_p[:, tau = 15 steps = 3 s, event] | |
| = P(first onset <= 3 s | inactive at the anchor) = 1 - prod_{k<15}(1 - h_on[k]) (the six-cell PR-AUC@3s score of paired_compare_joint.py). | |
| k0 = the label-free current-state heuristic of paired_compare_joint.py --b-k0 (hand_return: -min_hand_rest_dist, hand_leave: +min_hand_rest_dist, | |
| K0_c of the v2 causal kin pack, NaN -> mean); chance = a seeded uniform score (the persistence-style constant score of alert_timing_eval.py cannot | |
| be thresholded to an alert budget — it is all-or-nothing — so the budget-matched label-free reference is the random policy under the same gate, | |
| whose expected Recall_{>= delta} is 1 - exp(-K (H / H_eligible) (3 - delta) / 3600), H = exposure, H_eligible = eligible time; also shown analytically). | |
| Gate the hazard heads are trained under the first-passage contract only at anchors where the event is possible (hands off for hand_return, hands on | |
| for hand_leave; populations.P_drv inact0); at the other anchors the main model's hand_return hazard is ~0.97 (untrained region), so a threshold | |
| on ALL anchors would spend the alert budget on impossible events. Every system's score is therefore gated by the label-free CURRENT state at t | |
| (S0_c hands_on of the causal kin pack; an invalid state = not eligible): gate v3 (PRIMARY: the label version the candidates and the gold set come | |
| from), v2 (the state version the frozen models were trained on) and none (the literal pre-registration wording) are all reported. The gate does | |
| not change the exposure: alerts/h = alerts / (row_ok anchors x 0.2 s) over ALL engaged TEST anchors. | |
| Metric for K in {10, 20, 40} alerts per hour: threshold by bisection on ALL engaged-test anchors so that alerts/h = K, an alert = score >= threshold with a | |
| 3 s per-route cool-down (alert_timing_eval.alert_rows = train_warning_head.alerts_per_hour semantics, cooldown 3 s); a score whose floor | |
| (alerts/h at threshold = max score, f16 ties) exceeds K is reported AT ITS FLOOR and flagged. Recall_{>= delta s early} = share of gold events | |
| with an alert in [t* - 3 s, t* - delta], delta in {0.5, 1, 2} s (= first alert in [t* - 3, t* + 1] has lead t* - t_alert >= delta; | |
| alert_timing_eval.match_events); also detected (lead >= 0), late (first alert in (t*, t* + 1]) and the first-alert lead distribution. | |
| Statistics paired device-cluster bootstrap over gold events (devices with replacement, 1,000 resamples, seed 0, alert_timing_eval.cluster_boot_events; | |
| thresholds fixed) of main - indep, main - k0, main - naive, ecan - main per seed; verdict over the three seeds by the project rule | |
| (cwm2/eval/kill_checks.beats: CI excl. 0 on >= 2/3 seeds, same sign, |mean| > seed noise -> BEATS / WORSE; the 3/3 count is reported too). | |
| By-products v3 candidate precision (gold / (labelled - unsure)) with Wilson 95 % CI on the RANDOM blocks R200 + R100 only, RE (VLM-ranked, heavy routes) | |
| separately, hand_leave (L100 + L50), manual MR; block W miss share and the recall estimate of the v3 candidate set recomputed with the R-only | |
| precision (apply_gold_engaged's own all-block value quoted). | |
| Outputs a dated section appended to results/tables/gold_engaged_recall.md (numbers only from this script; provenance in the header); --json-out optional. | |
| Reads only. Nothing under results/runs is written. Threads <= 4. | |
| """ | |
| from __future__ import annotations | |
| import os | |
| for _v in ("OMP_NUM_THREADS", "MKL_NUM_THREADS", "OPENBLAS_NUM_THREADS", "NUMEXPR_NUM_THREADS"): | |
| os.environ.setdefault(_v, "4") | |
| import argparse # noqa: E402 | |
| import calendar # noqa: E402 | |
| import hashlib # noqa: E402 | |
| import json # noqa: E402 | |
| import subprocess # noqa: E402 | |
| import sys # noqa: E402 | |
| import time # noqa: E402 | |
| from collections import Counter # noqa: E402 | |
| from pathlib import Path # noqa: E402 | |
| import numpy as np # noqa: E402 | |
| import pandas as pd # noqa: E402 | |
| CWM3 = Path(__file__).resolve().parents[1] | |
| EXPLORE = CWM3.parent | |
| CWM2 = EXPLORE / "cwm2" | |
| for _p in (CWM2, CWM2 / "eval", CWM3 / "common", CWM3 / "eval", CWM3 / "tools" / "review_tool"): | |
| if str(_p) not in sys.path: | |
| sys.path.insert(0, str(_p)) | |
| import alert_timing_eval as AT # noqa: E402 alert_rows (cool-down), match_events, cluster_boot_events, pct | |
| import kill_checks as KC # noqa: E402 beats = the project verdict rule | |
| from pred_layout import col # noqa: E402 layout column helper | |
| import apply_review as AR # noqa: E402 wilson, load_export | |
| from apply_gold_engaged import parse as parse_label # noqa: E402 label -> (kind, dt) | |
| RUNS = CWM3 / "results" / "runs" | |
| KIN = {"v2": CWM3 / "results" / "targets_kin_causal_routes_v2", "v3": CWM3 / "results" / "targets_kin_causal_routes_v3"} | |
| GOLD_JSON = CWM3 / "results" / "review_tool" / "gold_engaged_events.json" | |
| ITEMS = CWM3 / "results" / "review_tool" / "items_gold_engaged.json" | |
| LABELS = CWM3 / "results" / "review_tool" / "labels_latest.json" | |
| APPLY = CWM3 / "tools" / "review_tool" / "apply_gold_engaged.py" | |
| OUT_MD = CWM3 / "results" / "tables" / "gold_engaged_recall.md" | |
| SYSTEMS = {"main": "I1_all_minus_road2drv_s{seed}_final", "indep": "I1_indep_s{seed}_final", "naive": "I1_naive_s{seed}_final", | |
| "ecan": "I1_all_minus_road2drv_s{seed}_final_ecan"} | |
| REFS = ("k0", "chance") | |
| PAIRS = (("main", "indep"), ("main", "k0"), ("main", "naive"), ("ecan", "main")) | |
| EVENTS = ("hand_return", "hand_leave") | |
| CELLS = (("engaged", "hand_return"), ("engaged", "hand_leave"), ("manual", "hand_return")) | |
| GRID_DT, COOLDOWN, PRE, POST, TAU = 0.2, 3.0, 3.0, 1.0, 15 | |
| BLOCK_GROUPS = {"R (R200+R100, random)": ("R200", "R100"), "RE (VLM-ranked)": ("RE",), "L (L100+L50)": ("L100", "L50"), "MR (manual)": ("MR",)} | |
| MIN_GOLD = 20 | |
| CHANCE_SEED = 12345 | |
| # ----------------------------------------------------------------------------- grid / scores | |
| def load_grid(regime, split="test"): | |
| """row_ok anchors of the regime's split (rows sorted by (uid, t) as the layouts are): uid / dongle / t, the current hands_on state of the v2 and v3 | |
| causal kin packs (the gate), the k0 distance (v2 K0_c min_hand_rest_dist, NaN -> mean) and the exposure in hours (row_ok anchors x 0.2 s).""" | |
| with np.load(KIN["v3"] / f"kin_causal_{regime}_{split}.npz", allow_pickle=True) as z: | |
| uid, dongle, t, ok = z["uid"], z["dongle"], z["t"].astype(np.float64), z["row_ok"].astype(bool) | |
| s3 = z["S0_c"].astype(np.float32)[:, 0] | |
| with np.load(KIN["v2"] / f"kin_causal_{regime}_{split}.npz", allow_pickle=True) as z: | |
| assert np.array_equal(z["uid"], uid) and np.allclose(z["t"], t), "v2 / v3 causal kin rows differ" | |
| s2 = z["S0_c"].astype(np.float32)[:, 0] | |
| chn = [str(c) for c in z["channels"]] | |
| dist = z["K0_c"].astype(np.float64)[:, chn.index("min_hand_rest_dist")] | |
| assert np.array_equal(np.lexsort((t, uid)), np.arange(len(t))), "grid is not sorted by (uid, t)" | |
| sel = np.flatnonzero(ok) | |
| dist = np.where(np.isfinite(dist), dist, np.nanmean(dist)) # paired_compare_joint.py --b-k0 convention | |
| return dict(regime=regime, split=split, sel=sel, uid=uid[sel], dongle=dongle[sel], t=t[sel], hands_on={"v2": s2[sel], "v3": s3[sel]}, | |
| k0_dist=dist[sel].astype(np.float32), n_all=int(len(t)), n_rows=int(len(sel)), hours=len(sel) * GRID_DT / 3600.0, uid_raw=uid, t_raw=t) | |
| def layout_scores(run_dir, regime, split, grid, events=EVENTS): | |
| """onset_p[:, tau = 15, event] of the layout on the grid's row_ok anchors (f32) -> ({event: score}, meta).""" | |
| p = Path(run_dir) / f"pred_layout_{regime}_{split}.npz" | |
| with np.load(p, allow_pickle=True) as z: | |
| assert np.array_equal(z["uid"], grid["uid_raw"]) and np.allclose(z["t"], grid["t_raw"]), f"{p}: rows differ from the causal grid" | |
| names = [str(x) for x in z["event_names"]] | |
| ti = [int(x) for x in z["onset_taus"]].index(TAU) | |
| op = z["onset_p"][:, ti, :] | |
| meta = z["meta"] | |
| meta = json.loads(str(meta.item() if hasattr(meta, "item") else meta)) | |
| out = {} | |
| for ev in events: | |
| e = col(names, ev) | |
| assert e is not None, f"{p}: event {ev} not in {names}" | |
| out[ev] = op[grid["sel"], e].astype(np.float32) | |
| return out, meta | |
| def k0_scores(grid): | |
| return {"hand_return": -grid["k0_dist"], "hand_leave": grid["k0_dist"]} | |
| def chance_scores(grid, seed=CHANCE_SEED): | |
| u = np.random.default_rng(seed).random(len(grid["t"])).astype(np.float32) | |
| return {ev: u for ev in EVENTS} | |
| def eligible(grid, event, gate): | |
| """label-free gate at t: hand_return possible when hands are off, hand_leave when hands are on (S0_c hands_on of the chosen kin version; NaN = not eligible).""" | |
| if gate == "none": | |
| return np.ones(len(grid["t"]), bool) | |
| h = grid["hands_on"][gate] | |
| return np.isfinite(h) & ((h < 0.5) if event == "hand_return" else (h > 0.5)) | |
| def gated(score, elig): | |
| return np.where(elig, score, np.nan).astype(np.float32) | |
| # ----------------------------------------------------------------------------- alerts | |
| def alerts_per_hour(uid, t, s, thr, hours, cooldown=COOLDOWN): | |
| """(n alerts, alerts/h) with the per-route cool-down of alert_timing_eval.alert_rows; hours = the exposure of ALL anchors (not only the eligible ones).""" | |
| n = len(AT.alert_rows(uid, t, s, thr, cooldown)) | |
| return n, (n / hours if hours > 0 else np.nan) | |
| def thr_for_rate(uid, t, s, rate, hours, cooldown=COOLDOWN, it=24): | |
| """bisection (alert_timing_eval.thr_for_rate semantics) on the threshold so that alerts/h <= rate is met as tightly as the score's ties allow | |
| -> dict(thr, achieved, n_alerts, reachable, floor); floor = alerts/h at threshold = max score (reachable False when floor > rate: reported at the floor).""" | |
| fin = np.isfinite(s) | |
| if not fin.any(): | |
| return dict(thr=np.nan, achieved=0.0, n_alerts=0, reachable=True, floor=0.0) | |
| lo, hi = float(s[fin].min()), float(s[fin].max()) | |
| n_hi, floor = alerts_per_hour(uid, t, s, hi, hours, cooldown) | |
| if floor > rate: | |
| return dict(thr=hi, achieved=floor, n_alerts=n_hi, reachable=False, floor=floor) | |
| for _ in range(it): | |
| mid = 0.5 * (lo + hi) | |
| if alerts_per_hour(uid, t, s, mid, hours, cooldown)[1] > rate: | |
| lo = mid | |
| else: | |
| hi = mid | |
| n, r = alerts_per_hour(uid, t, s, hi, hours, cooldown) | |
| return dict(thr=hi, achieved=r, n_alerts=n, reachable=True, floor=floor) | |
| # ----------------------------------------------------------------------------- gold matching | |
| def match_gold(gold, uid, t, arows, pre=PRE, post=POST): | |
| """lead per gold event = t* - t of the FIRST alert in [t* - pre, t* + post] (NaN: none) via alert_timing_eval.match_events (t_recovery undefined).""" | |
| ts = gold["t_star"].to_numpy(np.float64) | |
| ev = pd.DataFrame({"uid": gold["uid"].to_numpy(), "onset_t": ts, "t_recovery": np.full(len(ts), np.nan), "seg_end_t": ts + post}) | |
| lead, _ = AT.match_events(ev, uid, t, arows, pre=pre + 1e-6, post=post + 1e-6) | |
| return lead | |
| def recall_indicators(lead, deltas): | |
| """[n_ev, len(deltas)] float indicators 'an alert in [t* - 3, t* - delta]' (= lead >= delta).""" | |
| ld = np.nan_to_num(np.asarray(lead, np.float64), nan=-99.0) | |
| return np.stack([(ld >= d - 1e-9).astype(float) for d in deltas], 1) if len(ld) else np.zeros((0, len(deltas))) | |
| def lead_stats(lead): | |
| ld = np.asarray(lead, np.float64) | |
| fin = np.isfinite(ld) | |
| early = fin & (ld >= -1e-9) | |
| late = fin & (ld < -1e-9) | |
| q = (lambda p: float(np.percentile(ld[early], p))) if early.any() else (lambda p: np.nan) | |
| return dict(detected=float(early.mean()) if len(ld) else np.nan, late=float(late.mean()) if len(ld) else np.nan, n_detected=int(early.sum()), | |
| lead_mean=float(ld[early].mean()) if early.any() else np.nan, lead_median=q(50), lead_q25=q(25), lead_q75=q(75)) | |
| def load_gold(gold_json, regime, event, grid): | |
| """gold events of one cell as a DataFrame with the grid's dongle; events on routes absent from the grid are dropped (counted).""" | |
| G = json.load(open(gold_json, encoding="utf-8")) | |
| rows = [g for g in G["gold_events"] if g["regime"] == regime and g["event"] == event] | |
| cols = ["id", "uid", "car", "regime", "event", "t_anchor", "t_event_candidate", "t_star", "dt_star", "label", "block"] | |
| df = pd.DataFrame(rows, columns=cols) | |
| u, i = np.unique(grid["uid"], return_index=True) | |
| u2d = dict(zip(u.tolist(), grid["dongle"][i].tolist())) | |
| keep = df["uid"].isin(u2d) | |
| n_drop = int((~keep).sum()) | |
| df = df[keep].reset_index(drop=True) | |
| df["dongle"] = df["uid"].map(u2d) | |
| df["t_star"] = np.round(df["t_star"].astype(np.float64), 3) | |
| return df, n_drop | |
| # ----------------------------------------------------------------------------- statistics | |
| def paired_boot(dongle, ind, n_boot, seed=0): | |
| """device-cluster bootstrap of the column means of ind (alert_timing_eval.cluster_boot_events) -> [n_boot, n_cols]; columns share the resamples (paired).""" | |
| return AT.cluster_boot_events(np.asarray(dongle), np.asarray(ind, np.float64), n_boot, seed) | |
| def verdict(pair_rows): | |
| """kill_checks.beats over the per-seed paired rows [{seed, diff, ci_lo, ci_hi, value_a, value_b}] -> dict(verdict, ci_excl_zero, n_seeds, ...).""" | |
| if not pair_rows: | |
| return {"verdict": "INSUFFICIENT", "n_seeds": 0, "ci_excl_zero": 0} | |
| pr = pd.DataFrame({"seed": [r["seed"] for r in pair_rows], "delta": [r["diff"] for r in pair_rows], "delta_ci_lo": [r["ci_lo"] for r in pair_rows], | |
| "delta_ci_hi": [r["ci_hi"] for r in pair_rows], "status": "secondary"}) | |
| va = pd.DataFrame({"value": [r["value_a"] for r in pair_rows]}) | |
| vb = pd.DataFrame({"value": [r["value_b"] for r in pair_rows]}) | |
| out = KC.beats(pr, va, vb, min_seeds=3) | |
| out.setdefault("ci_excl_zero", int(((pr.delta_ci_lo > 0) | (pr.delta_ci_hi < 0)).sum())) | |
| return out | |
| # ----------------------------------------------------------------------------- by-products | |
| def precision_by_block(items, export): | |
| """v3 candidate precision per block group: counts of every answer, n_eff = labelled - unsure - other, precision_event = gold / n_eff [Wilson 95 %].""" | |
| out = {} | |
| for grp, blocks in BLOCK_GROUPS.items(): | |
| c = Counter() | |
| n = 0 | |
| for it in items: | |
| m = it["meta"] | |
| if m["block"] not in blocks or m["event"] == "recall_window": | |
| continue | |
| n += 1 | |
| kind, _ = parse_label((export.get(it["id"]) or {}).get("label", "") or "") | |
| c[kind] += 1 | |
| labelled = n - c["unlabelled"] | |
| n_eff = labelled - c["unsure"] - c["other"] | |
| gold = c["at_event"] + c["shifted"] | |
| out[grp] = dict(blocks=list(blocks), n=n, labelled=labelled, at_event=c["at_event"], shifted=c["shifted"], never=c["never"], always=c["always"], unsure=c["unsure"], | |
| unlabelled=c["unlabelled"], other=c["other"], gold=gold, n_eff=n_eff, precision=(gold / n_eff if n_eff else np.nan), | |
| ci=(list(AR.wilson(gold, n_eff)) if n_eff else [np.nan, np.nan]), | |
| precision_at_event=(c["at_event"] / n_eff if n_eff else np.nan), ci_at_event=(list(AR.wilson(c["at_event"], n_eff)) if n_eff else [np.nan, np.nan])) | |
| return out | |
| def w_recall_estimate(G, p_R, ci_R): | |
| """apply_gold_engaged / apply_review_v3 recall estimate of the v3 candidate set with the R-only precision: TP = p x N_v3 returns, FN = q x negative 10 s windows.""" | |
| W, R = G.get("recall_windows", {}), G.get("recall_estimate", {}) | |
| n_ret, n_neg = R.get("n_v3_returns_engaged_test"), R.get("n_neg_windows") | |
| out = dict(n_windows=W.get("n"), yes=W.get("yes"), no=W.get("no"), unsure=W.get("unsure"), unlabelled=W.get("unlabelled"), q=W.get("q"), q_ci=W.get("q_ci"), | |
| n_v3_returns=n_ret, n_neg_windows=n_neg, apply_recall=R.get("recall"), apply_recall_lo=R.get("recall_lo"), apply_recall_hi=R.get("recall_hi"), recall_R=None) | |
| if n_ret and n_neg and W.get("q") is not None and np.isfinite(p_R): | |
| rec = lambda p, q: (p * n_ret) / (p * n_ret + q * n_neg) if (p * n_ret + q * n_neg) > 0 else np.nan # noqa: E731 | |
| out.update(recall_R=rec(p_R, W["q"]), recall_R_lo=rec(ci_R[0], W["q_ci"][1]), recall_R_hi=rec(ci_R[1], W["q_ci"][0])) | |
| return out | |
| # ----------------------------------------------------------------------------- evaluation of one cell | |
| def evaluate_cell(grid, gold, scores, budgets, deltas, gates, n_boot, seeds, log=print): | |
| """scores: {seed: {system: {event: score}}} for the model systems + {'ref': {k0, chance}} -> (rows_sys, rows_pair, verdicts) for one (regime, event) cell.""" | |
| event = gold["event"].iloc[0] if len(gold) else None | |
| rows_sys, rows_pair, verdicts = [], [], {} | |
| uid, t, hours = grid["uid"], grid["t"], grid["hours"] | |
| for gate in gates: | |
| el = eligible(grid, event, gate) | |
| n_el = int(el.sum()) | |
| for K in budgets: | |
| per = {} # (system, seed) -> (lead, thr info) | |
| for name in REFS: | |
| s = gated(scores["ref"][name][event], el) | |
| th = thr_for_rate(uid, t, s, K, hours) | |
| ar = AT.alert_rows(uid, t, s, th["thr"], COOLDOWN) | |
| per[(name, None)] = (match_gold(gold, uid, t, ar), th) | |
| for seed in seeds: | |
| for name in SYSTEMS: | |
| s = gated(scores[seed][name][event], el) | |
| th = thr_for_rate(uid, t, s, K, hours) | |
| ar = AT.alert_rows(uid, t, s, th["thr"], COOLDOWN) | |
| per[(name, seed)] = (match_gold(gold, uid, t, ar), th) | |
| log(f" gate {gate} K {K:g}/h seed {seed}: " + ", ".join(f"{n} thr {per[(n, seed)][1]['thr']:.4g} -> {per[(n, seed)][1]['achieved']:.1f}/h" for n in SYSTEMS)) | |
| names = list(SYSTEMS) + list(REFS) | |
| for seed in seeds: | |
| cols = [(n, d) for n in names for d in deltas] | |
| ind = np.concatenate([recall_indicators(per[(n, seed if n in SYSTEMS else None)][0], deltas) for n in names], 1) if len(gold) else np.zeros((0, len(cols))) | |
| bs = paired_boot(gold["dongle"].to_numpy(), ind, n_boot, 0) if len(gold) else None | |
| for n in names: | |
| lead, th = per[(n, seed if n in SYSTEMS else None)] | |
| row = dict(gate=gate, regime=grid["regime"], event=event, K=K, system=n, seed=(seed if n in SYSTEMS else "-"), n_gold=int(len(gold)), n_eligible=n_el, | |
| hours=hours, **{f"thr": th["thr"], "achieved": th["achieved"], "n_alerts": th["n_alerts"], "reachable": th["reachable"], "floor": th["floor"]}, **lead_stats(lead)) | |
| for d in deltas: | |
| j = cols.index((n, d)) | |
| row[f"recall_{d:g}"] = float(ind[:, j].mean()) if len(gold) else np.nan | |
| row[f"recall_{d:g}_lo"], row[f"recall_{d:g}_hi"] = AT.pct(bs[:, j]) if bs is not None else (np.nan, np.nan) | |
| if n in SYSTEMS or seed == seeds[0]: | |
| rows_sys.append(row) | |
| if seed == seeds[0]: # gate-adjusted analytic chance level: Poisson alerts at K / h concentrated into the eligible exposure | |
| h_el = n_el * GRID_DT / 3600.0 | |
| lam = (K * hours / h_el / 3600.0) if h_el > 0 else np.nan | |
| row = dict(gate=gate, regime=grid["regime"], event=event, K=K, system="chance (analytic)", seed="-", n_gold=int(len(gold)), n_eligible=n_el, hours=hours, | |
| thr=np.nan, achieved=float(K), n_alerts=int(round(K * hours)), reachable=True, floor=np.nan, detected=float(1.0 - np.exp(-lam * PRE)), late=np.nan, | |
| n_detected=0, lead_mean=np.nan, lead_median=np.nan, lead_q25=np.nan, lead_q75=np.nan) | |
| for d in deltas: | |
| row[f"recall_{d:g}"] = float(1.0 - np.exp(-lam * (PRE - d))) | |
| row[f"recall_{d:g}_lo"] = row[f"recall_{d:g}_hi"] = np.nan | |
| rows_sys.append(row) | |
| for a_, b_ in PAIRS: | |
| for d in deltas: | |
| ja, jb = cols.index((a_, d)), cols.index((b_, d)) | |
| if bs is None: | |
| continue | |
| dd = bs[:, ja] - bs[:, jb] | |
| lo, hi = AT.pct(dd) | |
| rows_pair.append(dict(gate=gate, regime=grid["regime"], event=event, K=K, a=a_, b=b_, delta=d, seed=seed, value_a=float(ind[:, ja].mean()), | |
| value_b=float(ind[:, jb].mean()), diff=float(ind[:, ja].mean() - ind[:, jb].mean()), ci_lo=lo, ci_hi=hi, | |
| frac_gt0=float(np.mean(dd > 0)), n_gold=int(len(gold)))) | |
| for a_, b_ in PAIRS: | |
| floor_sys = sorted({r["system"] for r in rows_sys if r["gate"] == gate and r["K"] == K and r["system"] in (a_, b_) and not r["reachable"]}) | |
| for d in deltas: | |
| pr = [r for r in rows_pair if r["gate"] == gate and r["K"] == K and r["a"] == a_ and r["b"] == b_ and r["delta"] == d] | |
| n_excl = int(sum((r["ci_lo"] > 0) or (r["ci_hi"] < 0) for r in pr)) | |
| if len(gold) < MIN_GOLD: | |
| v = {"verdict": f"INSUFFICIENT (n_gold {len(gold)} < {MIN_GOLD})", "n_seeds": len(pr), "ci_excl_zero": n_excl} | |
| elif floor_sys: # a system reported at its floor is not at the budget K: the pair is not budget-matched -> no verdict | |
| v = {"verdict": f"N/A (not budget-matched: {'/'.join(floor_sys)} at floor)", "n_seeds": len(pr), "ci_excl_zero": n_excl} | |
| else: | |
| v = verdict(pr) | |
| verdicts[(gate, K, a_, b_, d)] = v | |
| return rows_sys, rows_pair, verdicts | |
| # ----------------------------------------------------------------------------- rendering | |
| def f3(x, nd=3): | |
| try: | |
| x = float(x) | |
| except (TypeError, ValueError): | |
| return "—" | |
| return "—" if not np.isfinite(x) else f"{x:.{nd}f}" | |
| def fci(v, lo, hi, nd=3): | |
| return f"{f3(v, nd)} [{f3(lo, 2)}, {f3(hi, 2)}]" | |
| def sha256_file(p): | |
| h = hashlib.sha256() | |
| with open(p, "rb") as f: | |
| for chunk in iter(lambda: f.read(1 << 20), b""): | |
| h.update(chunk) | |
| return h.hexdigest() | |
| def render(stamp, a, G, prec, west, cells, provenance, deltas, budgets, gates): | |
| tot = G["totals"] | |
| n_gold = {c: cells[c]["n_gold"] for c in cells} | |
| L = [f"\n## Gold-event Recall_{{≥ δ s early}} @ K alerts/h — {stamp} — labels {Path(a.labels).name} ({tot['n_labelled']}/{tot['n_items']} items labelled; " | |
| f"gold engaged hand_return n = {n_gold.get(('engaged', 'hand_return'), 0)}, engaged hand_leave n = {n_gold.get(('engaged', 'hand_leave'), 0)}, " | |
| f"manual hand_return n = {n_gold.get(('manual', 'hand_return'), 0)}; MIN_GOLD = {MIN_GOLD}; re-runnable — sections append)\n", | |
| f"Provenance: `{provenance['command']}`; code sha256 gold_event_recall.py {provenance['code_sha'][:16]}, alert_timing_eval.py {provenance['at_sha'][:16]}; labels sha256 " | |
| f"{provenance['labels_sha'][:16]} (mtime {provenance['labels_mtime']}); gold json written {G.get('written_utc')} ({tot['n_gold']} gold events); items built {G.get('items_built_utc')}; " | |
| f"seeds {list(a.seeds)}; n_boot {a.n_boot}; layouts: " + "; ".join(f"{k} → {v}" for k, v in provenance["layouts"].items()) + ".\n", | |
| f"Protocol (pre-registered 2026-09-17T00:28:00Z + amendments 00:34:39Z / 00:46:01Z / 01:05:57Z / 01:21Z). Score per anchor = onset_p[τ = 15 steps = 3 s] of the event's hazard head " | |
| f"(P(first onset ≤ 3 s | inactive at t) = 1 − ∏(1 − h_on), the six-cell PR-AUC@3s score); k0 = paired_compare_joint --b-k0 heuristic (hand_return −min_hand_rest_dist, hand_leave +min_hand_rest_dist, v2 K0_c, NaN → mean); " | |
| f"chance = seeded uniform score under the same gate (its expected Recall_{{≥δ}} = 1 − exp(−K · (H / H_eligible) · (3 − δ)/3600), H = exposure, H_eligible = eligible anchors × 0.2 s — the gate " | |
| f"concentrates the budget into the eligible time, so the chance level is above the ungated K (3 − δ)/3600; shown as the `chance (analytic)` row). Exposure = ALL row_ok engaged-TEST anchors × 0.2 s; " | |
| f"alert = score ≥ threshold with a 3 s per-route cool-down (alert_timing_eval.alert_rows); threshold by bisection so that alerts/h = K ∈ {list(budgets)} (`floor` = the budget is unreachable because of " | |
| f"f16 score ties at the maximum — reported at the floor); Recall_{{≥δ}} = share of gold events with an alert in [t* − 3 s, t* − δ], δ ∈ {list(deltas)} s; detected = an alert in [t* − 3, t*]; " | |
| f"late = the first alert in [t* − 3, t* + 1] falls in (t*, t* + 1]. GATE (implementation decision 2026-09-17, made on the eligibility diagnostic, not on results): scores are eligible only where the " | |
| f"label-free current state allows the event (hands off for hand_return, hands on for hand_leave; S0_c of the causal kin pack); gate `v3` = PRIMARY (the label version of the candidates and the gold set), " | |
| f"`v2` = the frozen models' own state input, `none` = the literal pre-registration wording (all anchors). Paired device-cluster bootstrap over gold events ({a.n_boot:,} resamples, seed 0, thresholds fixed); " | |
| f"verdict = kill_checks.beats (CI excl. 0 on ≥ 2/3 seeds, same sign, |mean| > seed noise → BEATS / WORSE); the x/3 count of seeds with CI excl. 0 is shown. Cells with n_gold < {MIN_GOLD}: INSUFFICIENT (numbers shown, no verdict).\n", | |
| "### Gold set and v3 candidate precision by block (precision_event = gold / (labelled − unsure); RE is ΔP-selected on heavy routes → reported separately, NOT pooled with R)\n", | |
| "| block group | regime / event | n items | labelled | 1 at_event | 2 shifted | 3 never | 4 always | 5 unsure | unlabelled | gold | v3 precision_event [Wilson 95 %] | precision_at_event [Wilson 95 %] |", | |
| "|---|---|---|---|---|---|---|---|---|---|---|---|---|"] | |
| lab = {"R (R200+R100, random)": "engaged / hand_return", "RE (VLM-ranked)": "engaged / hand_return", "L (L100+L50)": "engaged / hand_leave", "MR (manual)": "manual / hand_return"} | |
| for grp, c in prec.items(): | |
| L.append(f"| {grp} | {lab[grp]} | {c['n']} | {c['labelled']} | {c['at_event']} | {c['shifted']} | {c['never']} | {c['always']} | {c['unsure']} | {c['unlabelled']} | {c['gold']} | " | |
| f"{fci(c['precision'], *c['ci']) if c['n_eff'] else '—'} (n_eff {c['n_eff']}) | {fci(c['precision_at_event'], *c['ci_at_event']) if c['n_eff'] else '—'} |") | |
| L.append("") | |
| for (reg, ev), c in cells.items(): | |
| L.append(f"- **{reg} {ev}**: n_gold = {c['n_gold']} on {c['n_routes']} routes / {'n/a' if c.get('skipped') else c['n_devices']} devices (blocks {c['blocks']}; at_event {c['n_at_event']}, shifted {c['n_shifted']}, " | |
| f"clicked dt median {f3(c['dt_median'], 1)} s, range [{f3(c['dt_min'], 1)}, {f3(c['dt_max'], 1)}]; dropped (route not on grid) {c['n_drop']}) → " | |
| + ("**INSUFFICIENT** (< %d)" % MIN_GOLD if c["n_gold"] < MIN_GOLD else "sufficient") + ("; not evaluated (manual cell needs ≥ %d)" % MIN_GOLD if c.get("skipped") else "")) | |
| w = west | |
| L += ["", f"- **Block W (60 engaged 10 s windows without a v3 hand event)**: labelled yes {w['yes']} / no {w['no']} / unsure {w['unsure']} / unlabelled {w['unlabelled']}; miss share q = " | |
| + (fci(w['q'], *w['q_ci']) if w.get('q') is not None else "— (no labelled windows yet)") | |
| + f"; v3 engaged-TEST hand_return candidates {w['n_v3_returns']}, negative 10 s windows {f3(w['n_neg_windows'], 0)}. Recall estimate of the v3 candidate set with the R-only precision: " | |
| + (f"**{f3(w['recall_R'])}** [{f3(w.get('recall_R_lo'))}, {f3(w.get('recall_R_hi'))}]" if w.get("recall_R") is not None else "**not available** (needs labelled W windows)") | |
| + f"; apply_gold_engaged's own all-block value: {f3(w['apply_recall'])}.", ""] | |
| for gate in gates: | |
| prim = " (PRIMARY)" if gate == gates[0] else " (sensitivity)" | |
| L.append(f"### Gate `{gate}`{prim} — per system × seed × budget\n") | |
| for (reg, ev), c in cells.items(): | |
| rs = [r for r in c["rows_sys"] if r["gate"] == gate] | |
| if not rs: | |
| continue | |
| L.append(f"#### {reg} {ev} — n_gold {c['n_gold']}{' — INSUFFICIENT' if c['n_gold'] < MIN_GOLD else ''}; eligible anchors {rs[0]['n_eligible']:,} of {c['n_rows']:,} (exposure {c['hours']:.2f} h)\n") | |
| L += ["| system | seed | K/h | threshold | alerts/h achieved (n) | " + " | ".join(f"Recall ≥ {d:g} s [CI]" for d in deltas) + " | detected | late (0, 1] | lead of detected: median / mean / IQR s |", | |
| "|---|---|---|---|---|" + "---|" * len(deltas) + "---|---|---|"] | |
| for r in rs: | |
| L.append(f"| {r['system']}{'' if r['reachable'] else ' `floor`'} | {r['seed']} | {r['K']:g} | {f3(r['thr'], 4)} | {r['achieved']:.1f} ({r['n_alerts']}) | " | |
| + " | ".join(fci(r[f"recall_{d:g}"], r[f"recall_{d:g}_lo"], r[f"recall_{d:g}_hi"]) for d in deltas) | |
| + f" | {f3(r['detected'])} | {f3(r['late'])} | {f3(r['lead_median'], 2)} / {f3(r['lead_mean'], 2)} / [{f3(r['lead_q25'], 2)}, {f3(r['lead_q75'], 2)}] |") | |
| L.append("") | |
| L.append(f"#### Gate `{gate}` — paired differences (device-cluster bootstrap, per seed) and verdicts\n") | |
| L += ["| regime / event | K/h | pair | δ s | Δ s0 / s1 / s2 [CI] | A mean | B mean | CI excl. 0 | verdict |", "|---|---|---|---|---|---|---|---|---|"] | |
| for (reg, ev), c in cells.items(): | |
| for K in budgets: | |
| for a_, b_ in PAIRS: | |
| for d in deltas: | |
| pr = [r for r in c["rows_pair"] if r["gate"] == gate and r["K"] == K and r["a"] == a_ and r["b"] == b_ and r["delta"] == d] | |
| if not pr: | |
| continue | |
| v = c["verdicts"].get((gate, K, a_, b_, d), {}) | |
| L.append(f"| {reg} {ev} | {K:g} | {a_} − {b_} | {d:g} | " + " / ".join(f"{r['diff']:+.3f} [{r['ci_lo']:+.2f}, {r['ci_hi']:+.2f}]" for r in pr) | |
| + f" | {np.mean([r['value_a'] for r in pr]):.3f} | {np.mean([r['value_b'] for r in pr]):.3f} | {v.get('ci_excl_zero', 0)}/{len(pr)} | {v.get('verdict', '—')} |") | |
| L.append("") | |
| return L | |
| def caveats(cells, prec, west, gates): | |
| C = [] | |
| for (reg, ev), c in cells.items(): | |
| if c["n_gold"] < MIN_GOLD: | |
| C.append(f"{reg} {ev}: n_gold = {c['n_gold']} < {MIN_GOLD} → INSUFFICIENT; the recall levels and Δ are shown for information only, no verdict.") | |
| fl = sorted({(r["gate"], r["regime"], r["event"], r["system"], r["K"]) for c in cells.values() for r in c["rows_sys"] if not r["reachable"]}) | |
| if fl: | |
| C.append("budget unreachable (score floor, f16 ties at the maximum; reported at the floor; pairs involving such a system carry no verdict — not budget-matched): " | |
| + "; ".join(f"{g}/{rg} {e} {s} K={K:g}" for g, rg, e, s, K in fl) + ".") | |
| gate0 = gates[0] | |
| for (reg, ev), c in cells.items(): | |
| parts = [] | |
| for K in sorted({r["K"] for r in c["rows_sys"]}): | |
| m = [r["recall_1"] for r in c["rows_sys"] if r["gate"] == gate0 and r["K"] == K and r["system"] == "main"] | |
| ch = [r["recall_1"] for r in c["rows_sys"] if r["gate"] == gate0 and r["K"] == K and r["system"] == "chance (analytic)"] | |
| sim = [r["recall_1"] for r in c["rows_sys"] if r["gate"] == gate0 and r["K"] == K and r["system"] == "chance"] | |
| if m and ch: | |
| parts.append(f"K={K:g}: main {np.mean(m):.3f} (seed mean) vs chance {ch[0]:.3f} analytic / {sim[0]:.3f} simulated → {'at/below' if np.mean(m) <= ch[0] + 1e-9 else 'above'} chance") | |
| if parts: | |
| C.append(f"Gate `{gate0}`, {reg} {ev}, Recall ≥ 1 s of main vs the gate-adjusted chance level — " + "; ".join(parts) + ".") | |
| C.append("The gate is an implementation decision (documented above), not part of the 00:28Z wording; gate `none` shows the literal reading. The gold set is conditional on the v3 candidate generator " | |
| "(first-step transitions, ≤ 8 per route, ±5 s exclusion zones) — coverage of true returns is the W-window estimate, not assumed.") | |
| pR = prec.get("R (R200+R100, random)", {}) | |
| if pR.get("n_eff"): | |
| C.append(f"v3 candidate precision (R blocks only) {f3(pR['precision'])} [{f3(pR['ci'][0])}, {f3(pR['ci'][1])}] on n_eff {pR['n_eff']}; RE (VLM-ranked) is reported separately and is not pooled.") | |
| if west.get("recall_R") is None: | |
| C.append("W-window recall estimate not available yet (no labelled W windows).") | |
| C.append("k0 / chance have no seed: their recall is identical across seed rows; the paired Δ vs k0 varies only through the model seed. Labelling is still in progress: the section is dated and states n; re-run appends.") | |
| C.append("The alert budget (K × exposure ≈ 250 / 500 / 1,000 alerts) exceeds the number of gold events many times over, so recall is limited by the ranking of the eligible anchors, not by the budget; " | |
| "the gold events are the labelled subset of the v3 candidates (not every true event of the split), so levels are estimates on a sample and the paired Δ is the quantity to read.") | |
| return C | |
| # ----------------------------------------------------------------------------- main | |
| def main(argv=None): | |
| ap = argparse.ArgumentParser() | |
| ap.add_argument("--apply", action="store_true", help="run tools/review_tool/apply_gold_engaged.py first (writes the gold json; appends its own summary section)") | |
| ap.add_argument("--labels", default=str(LABELS)) | |
| ap.add_argument("--items", default=str(ITEMS)) | |
| ap.add_argument("--gold-json", default=str(GOLD_JSON)) | |
| ap.add_argument("--seeds", nargs="+", type=int, default=[0, 1, 2]) | |
| ap.add_argument("--budgets", nargs="+", type=float, default=[10.0, 20.0, 40.0]) | |
| ap.add_argument("--deltas", nargs="+", type=float, default=[0.5, 1.0, 2.0]) | |
| ap.add_argument("--gates", nargs="+", default=["v3", "v2", "none"], help="first = PRIMARY") | |
| ap.add_argument("--n-boot", type=int, default=1000) | |
| ap.add_argument("--out", default=str(OUT_MD)) | |
| ap.add_argument("--json-out", default="") | |
| ap.add_argument("--runs", default=str(RUNS)) | |
| a = ap.parse_args(argv) | |
| t0 = time.time() | |
| stamp = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime()) | |
| log = lambda m: print(f"[{time.strftime('%H:%M:%S')}] {m}", flush=True) # noqa: E731 | |
| if a.apply: | |
| cmd = [sys.executable, str(APPLY), a.labels, "--items", a.items, "--json-out", a.gold_json] | |
| log("apply: " + " ".join(cmd)) | |
| subprocess.run(cmd, check=True, cwd=str(CWM3)) | |
| assert Path(a.gold_json).exists(), f"{a.gold_json} missing — run with --apply (or apply_gold_engaged.py) first" | |
| G = json.load(open(a.gold_json, encoding="utf-8")) | |
| # one snapshot: the gold json must not be older than the labels file it was built from (the owner labels continuously) — otherwise the by-products read | |
| # live labels while the gold events are stale; --apply rebuilds the gold json first | |
| lab_mtime = Path(a.labels).stat().st_mtime | |
| gold_written = calendar.timegm(time.strptime(G["written_utc"], "%Y-%m-%dT%H:%M:%SZ")) if G.get("written_utc") else 0.0 | |
| if lab_mtime > gold_written + 1.0: | |
| raise SystemExit(f"STALE GOLD: {a.labels} (mtime {time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime(lab_mtime))}) is newer than {a.gold_json} " | |
| f"(written {G.get('written_utc')}) — re-run with --apply so that gold events and by-products come from one label snapshot") | |
| P = json.load(open(a.items, encoding="utf-8")) | |
| export = AR.load_export(Path(a.labels)) | |
| prec = precision_by_block(P["items"], export) | |
| pR = prec["R (R200+R100, random)"] | |
| west = w_recall_estimate(G, pR["precision"], pR["ci"]) | |
| cells, provenance = {}, {"command": " ".join([Path(sys.executable).name, "eval/gold_event_recall.py"] + (argv if argv is not None else sys.argv[1:])), | |
| "code_sha": sha256_file(Path(__file__)), "at_sha": sha256_file(Path(AT.__file__)), "labels_sha": sha256_file(Path(a.labels)), | |
| "labels_mtime": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(Path(a.labels).stat().st_mtime)), "layouts": {}} | |
| grids, scores = {}, {} | |
| for reg, ev in CELLS: | |
| rows = [g for g in G["gold_events"] if g["regime"] == reg and g["event"] == ev] | |
| n_gold = len(rows) | |
| dts = [g["dt_star"] for g in rows if g["label"] != "at_event"] | |
| cell = dict(n_gold=n_gold, n_routes=len({g["uid"] for g in rows}), n_devices=0, blocks=sorted({g["block"] for g in rows}), n_at_event=sum(g["label"] == "at_event" for g in rows), | |
| n_shifted=len(dts), dt_median=(float(np.median(dts)) if dts else np.nan), dt_min=(min(dts) if dts else np.nan), dt_max=(max(dts) if dts else np.nan), | |
| n_drop=0, rows_sys=[], rows_pair=[], verdicts={}, n_rows=0, hours=np.nan, skipped=False) | |
| if reg == "manual" and n_gold < MIN_GOLD: | |
| cell["skipped"] = True | |
| cells[(reg, ev)] = cell | |
| log(f"{reg} {ev}: n_gold {n_gold} < {MIN_GOLD} — manual cell not evaluated") | |
| continue | |
| if reg not in grids: | |
| grids[reg] = load_grid(reg) | |
| log(f"grid {reg}: {grids[reg]['n_rows']:,} row_ok anchors of {grids[reg]['n_all']:,} = {grids[reg]['hours']:.2f} h, {len(np.unique(grids[reg]['dongle']))} devices") | |
| scores[reg] = {"ref": {"k0": k0_scores(grids[reg]), "chance": chance_scores(grids[reg])}} | |
| for seed in a.seeds: | |
| scores[reg][seed] = {} | |
| for name, pat in SYSTEMS.items(): | |
| run = pat.format(seed=seed) | |
| sc, meta = layout_scores(Path(a.runs) / run, reg, "test", grids[reg]) | |
| scores[reg][seed][name] = sc | |
| provenance["layouts"][f"{name} s{seed} {reg}"] = f"{run} ({meta.get('model_id', '?')})" | |
| log(f"scores {reg}: {len(a.seeds)} seeds x {len(SYSTEMS)} systems loaded ({time.time() - t0:.0f} s)") | |
| grid = grids[reg] | |
| gold, n_drop = load_gold(a.gold_json, reg, ev, grid) | |
| cell.update(n_gold=int(len(gold)), n_drop=n_drop, n_devices=int(gold["dongle"].nunique()) if len(gold) else 0, n_rows=grid["n_rows"], hours=grid["hours"]) | |
| log(f"{reg} {ev}: n_gold {len(gold)} ({cell['n_routes']} routes, {cell['n_devices']} devices; dropped {n_drop})" + (" — INSUFFICIENT" if len(gold) < MIN_GOLD else "")) | |
| if len(gold) == 0: | |
| cells[(reg, ev)] = cell | |
| continue | |
| rs, rp, vd = evaluate_cell(grid, gold, scores[reg], a.budgets, a.deltas, a.gates, a.n_boot, a.seeds, log) | |
| cell.update(rows_sys=rs, rows_pair=rp, verdicts=vd) | |
| cells[(reg, ev)] = cell | |
| L = render(stamp, a, G, prec, west, cells, provenance, a.deltas, a.budgets, a.gates) | |
| C = caveats(cells, prec, west, a.gates) | |
| L += ["### Caveats", ""] + [f"- {c}" for c in C] + ["", f"Wall {time.time() - t0:.0f} s.", ""] | |
| out = Path(a.out) | |
| out.parent.mkdir(parents=True, exist_ok=True) | |
| new = not out.exists() | |
| with open(out, "a", encoding="utf-8") as f: | |
| if new: | |
| f.write("# Human gold-event set — Recall_{≥ δ s early} @ K alerts/h of the hand hazards (append-only; eval/gold_event_recall.py)\n") | |
| f.write("\n".join(L) + "\n") | |
| if a.json_out: | |
| J = {"stamp": stamp, "args": vars(a), "provenance": provenance, "precision_by_block": prec, "w_recall": west, | |
| "cells": {f"{r}|{e}": {k: (v if k not in ("verdicts",) else {"|".join(map(str, kk)): vv for kk, vv in v.items()}) for k, v in c.items()} for (r, e), c in cells.items()}} | |
| Path(a.json_out).parent.mkdir(parents=True, exist_ok=True) | |
| Path(a.json_out).write_text(json.dumps(J, indent=1, default=lambda o: (None if isinstance(o, float) and not np.isfinite(o) else (o.item() if hasattr(o, "item") else str(o))))) | |
| print("\n".join(L[:6])) | |
| print(f"\nappended {out} ({time.time() - t0:.0f} s)") | |
| return 0 | |
| if __name__ == "__main__": | |
| sys.exit(main()) | |
Xet Storage Details
- Size:
- 42.9 kB
- Xet hash:
- 50333adafa43ff9ee289aef48edcdee35bf10c72352252543dea5e55af610b9c
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.