Buckets:
| """cwm3/eval/finish_warning_head_oof.py — banner + readout for results/tables/warning_head_final_oof.md (gaze/robustness agent, 2026-09-12; coordinator request): | |
| prepends the PRE-STATED hypotheses (in-sample optimism ≈ 0.02 as in the v1 OOF table; "the predicted future adds nothing" on the final model), the seeds available | |
| (seeds 1–2 OOF packs land later; the shell is idempotent) and copies the regime report's paired rows for the `oof_minus_insample_*` and `*_minus_cur_OOF` pairs into the | |
| banner as the readout. Idempotent (marker guard). Nothing is recomputed. python eval/finish_warning_head_oof.py --seeds "I1_..._s0_final ..." | |
| """ | |
| from __future__ import annotations | |
| import argparse | |
| import re | |
| import time | |
| from pathlib import Path | |
| CWM3 = Path(__file__).resolve().parents[1] | |
| MARK = "<!-- oof-head banner -->" | |
| def main(): | |
| ap = argparse.ArgumentParser() | |
| ap.add_argument("--table", default=str(CWM3 / "results" / "tables" / "warning_head_final_oof.md")) | |
| ap.add_argument("--seeds", default="") | |
| a = ap.parse_args() | |
| p = Path(a.table) | |
| txt = p.read_text() | |
| rows = re.findall(r"^\|\s*(?:oof_minus_insample_\w+|both_minus_cur_OOF|fut_minus_cur_OOF|both_minus_cur_insample|fut_minus_cur_insample)\s*\|[^\n]*$", txt, flags=re.M) | |
| readout = "\n".join(rows) if rows else "(pairs not found in the report)" | |
| n_seeds = len([s for s in a.seeds.split() if s]) | |
| banner = [MARK, f"# OUT-OF-FOLD WARNING HEAD of the final model — item `cwm3_final_oof` (assembled {time.strftime('%Y-%m-%d %H:%M:%S UTC', time.gmtime())}; runs with OOF packs: {a.seeds or '?'} = {n_seeds} seed(s))", "", | |
| "**Pre-stated (before the numbers were read):** (i) the in-sample train features of `warning_head_final*.md` are optimistic for the predicted-future feature sets — v1 (`warning_head_oof.md`) showed " | |
| "≈ 0.02 AUROC; the OOF − in-sample paired Δ on identical gold rows (device-cluster resamples) measures it on the FINAL model; the current-state sets (cur, cur+dwell+face) contain no model " | |
| "output and must give Δ = 0 exactly (sanity row). (ii) \"the predicted future adds nothing to the warning\": under OOF features, both − current and future − current must not be BEATS. " | |
| "Verdict rule: kill_checks.beats over the available seeds (INSUFFICIENT below 3 seeds — seeds 1–2 OOF packs are queued by the GPU-baselines agent; re-run `eval/run_warning_head_oof.sh` when they land).", "", | |
| "OOF packs: `results/runs/I1_all_minus_road2drv_s<k>_final/pred_{manual,engaged}_train_labelrows_oof.npz` (train/train_cwm3_oof.py, 5 device folds over the 91 labelled TRAIN devices, " | |
| "same rows as the in-sample packs; docs/gpu_baselines_report.md §D). Heads: `train/train_warning_head.py --oof` (cur / fut / both) and `--oof --extra-features dwell,face --w-suffix _oofdwfc`; " | |
| "the OOF (`*_n979oof*`) and in-sample (`*_n979`, `*_n979dwfc`) columns are merged into one layout (`eval/merge_w_layouts.py`, bit-identical copies) so the unchanged `cwm2/eval/warning_gold.py` scores them in one item.", "", | |
| "**Readout (copied from the regime report's paired rows below; Δ = A − B, AUROC, clean_test / manual / engaged strata; columns are `curhgb` / `futhgb` / `bothhgb` fixed-HGB heads unless `sel`):**", "", | |
| "| pair | A − B | stratum | " + " | ".join(f"Δ s{k} [CI]" for k in range(max(n_seeds, 1))) + " | mean Δ | CIs excl. 0 |", "|---|---|---|" + "---|" * max(n_seeds, 1) + "---|---|", | |
| readout, "", "---", ""] | |
| if MARK in txt: | |
| txt = re.sub(r"<!-- oof-head banner -->.*?\n---\n\n", "", txt, count=1, flags=re.S) | |
| txt = "\n".join(banner) + txt | |
| p.write_text(txt) | |
| print(f"finished {p} ({len(rows)} readout rows)") | |
| if __name__ == "__main__": | |
| main() | |
Xet Storage Details
- Size:
- 3.85 kB
- Xet hash:
- 63d35047c1f6710f378e43fb309eaf0b89874ad7334956f34a60092547731fa2
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.