benchmark/human_eval_299/newtest_prereg.md

Pre-registration — human evaluation of the CWM3 warning head on a new test set (frozen 2026-09-12, before any clip package was rendered)

This document fixes the systems, thresholds, sampling design, labels, estimators, comparisons, wording rules and minimum detectable effects of the human-evaluation programme described in docs/newtest_protocol_draft.md. Its sha256 is logged in docs/decisions.md; eval/newtest_analysis.py and eval/gaze_off_validation_analysis.py print the current sha256 next to the registered one in every result table. Anything not listed under §9 "allowed to change" is fixed. Owner's framing (binding): choices that legitimately favour the model are made here in the open (fixed low alert load as the primary axis; manual as the headline regime), and nothing may be selectively reported — every stratum, both regimes, the HT natural rates, the coverage figures and this hash are mandatory in the results.

1. Frozen systems and scores (nothing is retrained)

id	system	score	file
A	full predictive model + warning head	W_learned_bothhgb_t15_w10_n979, mean over the 3 frozen seeds	results/runs/W_I1_all_minus_road2drv_s{0,1,2}_final_merged/pred_layout_{regime}_{test,val}.npz
B	strongest current-state-only head (same inputs / supervision, no predicted block)	W_learned_curhgb_t15_w10_n979dwfc (seed-invariant)	results/runs/W_I1_all_minus_road2drv_s0_final_dwfc/pred_layout_{regime}_{test,val}.npz
C	openpilot DMS chain att@3s (deployed comparator)	compose.legacy_compose(pA, None, 15)["att_only"]	TEST cwm2/baselines/legacy_cockpitwm/results/marginals_{regime}_s0.npz; VAL marginals_{regime}_val_s0.npz

C on VAL: the draft named att_{regime}_val.npz["A"]; that array is the chain's binary future-state label pack, not a prediction. The frozen s0 chain checkpoints were therefore scored on the VAL pack by cwm2/baselines/legacy_cockpitwm/score_val_marginals.py (existing train_att.load_all reader + warn_compose.att_marginals path), verified on TEST against the published marginals (max |Δ pA| 2.44e-4 on all 510,325 / 444,734 published rows, i.e. float16 rounding). The DMS chain enters only as the labelled comparator; no DMS field is an input to A or B (tests/test_no_dms_inputs.py). sha256 of every layout / marginals file: results/newtest/thresholds_frozen_v1.json["layout_sha256"].

Test exposure (5-Hz s12 grid, rows × 0.2 s): manual 28.50 h / 158 routes / 49 devices; engaged 25.10 h / 101 routes / 39 devices. Val: 3.91 h / 6.16 h.

2. Operating points (frozen; results/newtest/thresholds_frozen_v1.json, sha256 17bece5550a68bdbf07131ed7061c08365e487da37ab8299a9ce60f8f9b2d07f)

Alert = score ≥ threshold with the 60 s per-route cooldown of train_warning_head.alerts_per_hour; thresholds by operating_points.thr_for_rate (bisection, 24 iterations) on alerts / hour.

• Primary: val-frozen thresholds at 10 / h and 20 / h fitted on the VAL layouts (deployable, no test peeking). Achieved TEST loads (manual / engaged, alerts / h): A val10 → 4.32 / 6.53, val20 → 9.40 / 14.10; B val10 → 3.83 / 2.99, val20 → 13.69 / 12.91; C val20 → 17.67 / 24.81.
• Secondary: test-achieved thresholds at 10 / h and 20 / h (equal achieved load, the operating_points.py convention).
• C cannot reach 10 / h: its score saturates at 1.0; floor = alerts / h at threshold = max score: val 17.24 / 12.85, test 15.10 / 15.66. At the two 10 / h points C is reported at its floor (threshold = max score) and, as a sensitivity, with a seeded random thinning of its alerts to 10 / h (keep probability 10 / floor, 200 replicates, mean). 20 / h is the fair three-way point; 10 / h compares A vs B.
• Thresholds: A manual 0.77799 / 0.67578 / 0.66618 / 0.53377 (val10 / val20 / test10 / test20), A engaged 0.81429 / 0.71777 / 0.75928 / 0.64111; B manual 0.62354 / 0.49316 / 0.53271 / 0.43481, B engaged 0.75586 / 0.62695 / 0.65918 / 0.54785; C = 1.0 (max score) at every point.

3. Sampling frame and design (eval/newtest_sample.py --seed 0 --version v1 --s3-hard 20; results/newtest/manifest_v1.csv sha256 2e0869c359481819d6c3eb15b92b76ac5698b00ee713e7b00d917afd3a38bafa, frame_counts_v1.json sha256 31a8844fa87aabf208980dc454619115c82a3aa21c6381df26d026a7abb3c566)

Frame = the TEST exposure, partitioned into S1 ∪ S2 (every unit carries an inclusion probability π); S3 is a third, model-independent set overlapping the frame.

• Exclusion (S1 / S2): anything within 12.5 s (same route) of a clean-test gold anchor (568), a round-1 H5 test clip (100) or a round-2 test clip (100) — 768 anchors on 143 routes.
• S1 alert union: alerts of A, B, C at the frame threshold min(thr_val20, thr_test20) per system × regime (a superset of every operating point's alerts), merged per route with a 10 s same-event guard (new event when an alert is > 10 s after the current event's first alert; anchor t = first alert): 3,316 alerts → 2,403 events → 2,188 after exclusion (178 gold, 27 round-2, 10 round-1). Pattern = systems with a frame alert in [t−10, t+10] (equals the cluster's own systems for 98.7 % of events). Cells (manual / engaged): A 212 / 199, B 216 / 175, C 315 / 361, A+B 169 / 150, A+C 56 / 46, B+C 62 / 82, A+B+C 74 / 71. Allocation 160 clips = per regime A 14, B 14, C 14, A+B 12, A+C 10, B+C 10, A+B+C 6 (discordant cells over-sampled). Device cap 10 over S1. Strata = (cell, device-group): a device whose remaining cap budget is below its count in a cell is its own stratum; among never-cappable devices those earning ≥ 1 draw are their own stratum, the rest are pooled; SRS within; π_i = n_stratum / N_stratum (exact). The 30 devices with more than 10 union events have their 10 draws pre-allocated greedily by N over their (cell, device) strata (coverage-maximising; no draw outside the plan). Events on a device-group that receives no draw are uncovered (π = 0); N_covered_cell and coverage_cell are stored: S1 overall coverage 0.809 (per cell 0.58–0.94; manual A 0.75, B 0.83, C 0.88, A+B 0.70, A+C 0.59, B+C 0.71, A+B+C 0.58; engaged A 0.94, B 0.94, C 0.81, A+B 0.88, A+C 0.70, B+C 0.80, A+B+C 0.59). 40 devices in the draw; top device 6.2 % of S1.
• S2 silent background: grid rows with speed > 2 m/s at t, ≥ 12.5 s from every union event and excluded anchor: manual 345,793 / engaged 297,106 rows (of 472,705 / 441,689 moving); 20 + 20 clips; strata (regime, device) with cap 6 (1 draw per device pre-allocated, extras proportional), 15 s same-route guard; coverage 0.956 / 0.935; π_i = n / N rows.
• S3 model-independent candidates: the 100 round-2 TEST clips (results/mining/round2_draw_v1.csv; 41 manual / 59 engaged; π = quota / N_eligible: rare_road 25 / 3,198, demand_noresp 20 / 1,875, demand_resp 20 / 2,103, hand_head 15 / 991, occlusion_night 5 / 8, background 15 / 1,472) + optional 20 hard_response_h5 clips (S3_hard; eligible 1,576 = 1,473 manual / 103 engaged; seed 1, cap 8, 10 / 10). S3 enters paired event-level analyses and AUROC only, never an HT natural rate.
• Phases: Phase 1 = S3 (100) + the first 60 S1 clips (regime-interleaved random order) + the S2 calibration clips; Phase 2 = the rest. Calibration = 8 warning clips (S1 3, S2 3, S3 2; 4 / 4 regime) + 4 gaze windows, rated by every rater + owner before Phase 1 and kept in the store (calibration = true; analysis reports with / without).
• Per clip stored server-side, never sent to a rater: fired_{A,B,C}_{val10,val20,test10,test20}, t_alert_*, score_*_anchor, stratum, cell, pattern, π, coverage. The tool's client queue is an explicit whitelist (h5_common.NEVER_SENT_KEYS / PREFIXES assertion).
• Pre-specified adaptation: if the Phase-1 S1 warranted share (Task-1 consensus) is < 0.25, the Phase-2 allocation moves from S2 to the discordant S1 cells (A+B, A+C, B+C, A+B+C); the change and its date are logged in decisions.md before it is applied; π of every already-drawn clip is unchanged.

4. Labels (tool cwm2/annotation_human/annotate_h5.py, store labels_newtest.jsonl, append-only)

• Task 1 (prefix-locked, two blind raters R(i mod 3), R((i+1) mod 3); rubric README_annotators.md unchanged): W_i = Q4 prompt_warranted_now consensus (A / S → warranted, D → not, Q → unsure; disagreement / unsure → adjudication by ADJ*; h5_to_consensus.py); onset o_i = mean start of the raters' onset_interval_s (route time) when compatibility ∈ {briefly, sustained} and concern_persists = ongoing.
• Task 2 (post clip [t, t+10] with frame t frozen 1 s; the third rater R((i+2) mod 3), served only if that rater holds no prefix record for the clip): E_i = any event_type ≠ none; onset from event_onset_s when Task 1 gives none; ongoing_at_anchor = yes without a prefix onset → o_i = t, flagged censored. Consensus rule for a second Task-2 record: same type set ∧ |Δonset| ≤ 1 s → mean onset, else adjudication (h5_to_consensus.py --task event). The outcome half is compared with the Task-1 raters' outcome stage (the 25 % second rating).
• Quality gates: Q4 binary κ ≥ 0.6, event_type κ ≥ 0.6, onset median |Δt| ≤ 1 s; interim check after 40 clips; κ < 0.4 or unsure > 25 % → pause, rubric addendum (README only; fields never change), re-calibration.

5. Estimators (eval/newtest_analysis.py; device-cluster bootstrap 2,000 resamples, dongles with replacement, identical resamples for every paired Δ)

Weights: w_i = (N_cell / N_covered_cell) / π_i on S1 ∪ S2 (post-stratified to the full cell; coverage reported beside every HT number); S3 unweighted. H_regime = 28.50 h manual, 25.10 h engaged.

At each (system X, regime, point L ∈ {val10, val20 primary; test10, test20 secondary}):

1. Event recall (primary): R_X = Σ w_i E_i 1[X alert at thr_X(L) in [o_i − 10, o_i + 3] s] / Σ w_i E_i on S1 ∪ S2; unweighted recall on all labelled events (S1 ∪ S2 ∪ S3) as secondary. Alerts are recomputed from the frozen layouts at the frozen thresholds (newtest_sample.alert_rows).
2. Effective lead time (primary): lead_i = o_i − t_first_alert_X in that window; undetected → 0; HT mean; median over both-detected events secondary; paired Δ A−B, A−C, B−C on identical events.
3. Natural false alarms: FA_X / h = Σ_{S1} w_i 1[X alert in [t−10, t+10]] (1 − W_i) / H; precision = HT ratio Σ w fired W / Σ w fired. Silent-time warranted moments / h = Σ_{S2} w_i W_i / H (the miss population). Cell-level warranted shares appear only under the header "enriched — not a natural rate".
4. Pooling (pre-fixed): manual and engaged always separate; pooled = HT event-count-weighted mixture (natural mix); equal-weight pooling shown only as a sensitivity.
5. AUROC (secondary): score at the anchor vs W_i, unweighted over all labelled clips and HT-weighted (Mann–Whitney) on S1 ∪ S2; eval/gold_extended.py still produces the pre-registered clean_test_r2 rows from the S3 clips.
6. Three questions: overall = A vs C and B vs C at 20 / h (recall, lead, FA / h); does prediction help = A vs B at 10 / h and 20 / h; vs deployed baseline = C at 20 / h and at its floor (plus the thinning sensitivity).

6. Wording rules and minimum detectable effects

• Every Δ is reported with its CI. CI containing 0 → "matches (Δ, CI)"; CI above 0 → "above (Δ, CI)"; CI below 0 → "below (Δ, CI)". The words "significantly better" and "equivalent" are never used. A per-regime comparison with fewer than 60 labelled events in that regime is labelled descriptive.
• Expected positives: S1 warranted share ≈ 0.30 → ≈ 48; S2 ≈ 2; S3 ≈ 25 → ≈ 75 events, ≈ 50 in the HT partition. Paired recall Δ with discordance 0.4: SE = √(0.4 / n) → MDE ≈ 0.20 (n 75) / 0.25 (n 50) at α 0.05 two-sided, 80 % power; lead-time Δ with SD_diff 3 s → MDE ≈ 1.0 s (n 75). The analysis prints the achieved n and the implied MDE next to every paired Δ; an observed |Δ| below the MDE with a CI containing 0 is reported as "matches", not as "no difference".
• The headline sentence of the paper uses the manual regime at the val-frozen 20 / h (three-way) and 10 / h (A vs B) points; the engaged regime, the test-achieved points, C's floor, the HT natural rates (FA / h, silent warranted / h), every S1 cell's coverage and the pre-registration hash appear in the same table.

7. gaze_off human validation (eval/gaze_off_validation_sample.py --seed 0 --version v1; results/gaze_validation/manifest_v1.csv sha256 3ffb6991d12e362e6435750451702866478333ba392ae1c7341447f271b911a4, frame_counts_v1.json sha256 d92c0466330a66687de8d24dc08a90ab9259348db813ac3683c16503edf777e1)

• VAL split only (181,278 rows, 17 devices; the head never used val, thresholds were fitted on train). State: facehist_causal_{regime}_val.npz F0_c[:,7] gaze_off, [:,8] dwell, [:,4] ear_conf_asym, F0valid; kin_causal_{regime}_val.npz S0_c[:,1] head_turned, K0valid (kin-valid = K0valid ∧ finite S0). Cells (N): on|straight 137,820; on|turned 2,594; off_brief (dwell ≤ 2 s)|straight 8,181; off_brief|turned 8,564; off_sust|straight 4,015; off_sust|turned 3,832; face_invalid|kin_invalid 15,654 (13,899 on one device); face_invalid|turned 294. Allocation 30 / 25 / 25 / 25 / 25 / 25 / 25 / 10 + 10 regime top-up (→ manual) = 200 windows (106 engaged / 94 manual, 16 devices, device cap 20); strata = cell × regime × |ear_conf_asym| > 0.3 half (50 / 50 inside every valid-face cell); π = n / N per stratum; 5 s same-route guard; face-invalid cells allocated first. Clip [t−3, t+1], player pauses once at t. Two blind raters per window (R1–R4); 4 calibration windows.
• Fields: view_region (H5 enum + down_other), head_motion, eyes_vs_head {head_and_eyes_on_road, head_on_eyes_off, head_off_eyes_on, both_off, unknown}, check_kind {none, mirror_check, instrument_or_console_check, sustained_off_road, unknown}, occlusion {none, hand_arm, glasses_hat_mask, lighting, partial_out_of_frame, camera_blocked, not_driver}, visibility.
• Metrics (eval/gaze_off_validation_analysis.py; HT-weighted with w = 1 / π to the val prevalence; device-cluster bootstrap 2,000): rater κ (view_region collapsed on / off-road, check_kind, eyes_vs_head; target ≥ 0.6); confusion automatic state {on, off_brief, off_sust, face_invalid} × human {on, off}; precision and recall of gaze_off = 1 vs human off-road and vs sustained off-road; share of gaze_off = 1 that are mirror checks; share of gaze_off = 1 and of face_invalid with occlusion ≠ none; recall of eyes-only glances (head_on_eyes_off) by gaze_off — pre-stated expectation ≈ 0, because gaze_off is a head-yaw hysteresis on |lm_yaw_dev| (on / off thresholds P90 / P75 = 0.212 / 0.103) and cannot see eye movement; the paper states "detects head-away episodes, not eyes-only glances" with the measured value, whatever it is. Wording rules of §6 apply.

8. Analysis code, provenance and reporting

• Code paths fixed here: eval/newtest_sample.py, eval/newtest_render.py, eval/newtest_analysis.py, eval/gaze_off_validation_sample.py, eval/gaze_off_validation_analysis.py, cwm2/annotation_human/{h5_common,annotate_h5,h5_to_consensus,smoke_test}.py, cwm2/baselines/legacy_cockpitwm/score_val_marginals.py. Every result table is appended (never overwritten) with a run stamp and the current sha256 of this file; CSVs and run json (common/runmeta.write_run_json) accompany every run.
• Test clips never leave the machine; no VLM or cloud service is used; raters never see any model / DMS / sampling information.
• Mandatory in the results section: both regimes, all four operating points, C's floor, all S1 cells with coverage, S2 natural rates, S3 secondary rows, AUROC, calibration-in / -out sensitivity, MDE, this hash.

9. Allowed to change after this hash (and only by a dated decisions.md row)

Rendering details that do not alter which clips are shown or their times (codec settings, layout of the rater page); the number of bootstrap resamples (≥ 1,000); the optional S3_hard 20 (in or out, decided before its labels exist); the Phase-2 reallocation of §3 under its pre-specified trigger; rubric addenda (README text only); adjudicator id; schedule dates. Not allowed: thresholds, systems, π, cell definitions, estimator formulas, windows, wording rules, MDE formula, primary / secondary labelling.