Buckets:
| # Pre-registration — human evaluation of the CWM3 warning head on a new test set (frozen 2026-09-12, before any clip package was rendered) | |
| This document fixes the systems, thresholds, sampling design, labels, estimators, comparisons, wording rules and minimum detectable effects of the human-evaluation programme described in `docs/newtest_protocol_draft.md`. Its sha256 is logged in `docs/decisions.md`; `eval/newtest_analysis.py` and `eval/gaze_off_validation_analysis.py` print the current sha256 next to the registered one in every result table. Anything not listed under §9 "allowed to change" is fixed. Owner's framing (binding): choices that legitimately favour the model are made here in the open (fixed low alert load as the primary axis; manual as the headline regime), and **nothing may be selectively reported** — every stratum, both regimes, the HT natural rates, the coverage figures and this hash are mandatory in the results. | |
| ## 1. Frozen systems and scores (nothing is retrained) | |
| | id | system | score | file | | |
| |---|---|---|---| | |
| | A | full predictive model + warning head | `W_learned_bothhgb_t15_w10_n979`, **mean over the 3 frozen seeds** | `results/runs/W_I1_all_minus_road2drv_s{0,1,2}_final_merged/pred_layout_{regime}_{test,val}.npz` | | |
| | B | strongest current-state-only head (same inputs / supervision, no predicted block) | `W_learned_curhgb_t15_w10_n979dwfc` (seed-invariant) | `results/runs/W_I1_all_minus_road2drv_s0_final_dwfc/pred_layout_{regime}_{test,val}.npz` | | |
| | C | openpilot DMS chain att@3s (deployed comparator) | `compose.legacy_compose(pA, None, 15)["att_only"]` | TEST `cwm2/baselines/legacy_cockpitwm/results/marginals_{regime}_s0.npz`; VAL `marginals_{regime}_val_s0.npz` | | |
| C on VAL: the draft named `att_{regime}_val.npz["A"]`; that array is the chain's **binary future-state label pack**, not a prediction. The frozen s0 chain checkpoints were therefore scored on the VAL pack by `cwm2/baselines/legacy_cockpitwm/score_val_marginals.py` (existing `train_att.load_all` reader + `warn_compose.att_marginals` path), verified on TEST against the published marginals (max |Δ pA| 2.44e-4 on all 510,325 / 444,734 published rows, i.e. float16 rounding). The DMS chain enters only as the labelled comparator; no DMS field is an input to A or B (`tests/test_no_dms_inputs.py`). sha256 of every layout / marginals file: `results/newtest/thresholds_frozen_v1.json["layout_sha256"]`. | |
| Test exposure (5-Hz s12 grid, rows × 0.2 s): manual 28.50 h / 158 routes / 49 devices; engaged 25.10 h / 101 routes / 39 devices. Val: 3.91 h / 6.16 h. | |
| ## 2. Operating points (frozen; `results/newtest/thresholds_frozen_v1.json`, sha256 `17bece5550a68bdbf07131ed7061c08365e487da37ab8299a9ce60f8f9b2d07f`) | |
| Alert = score ≥ threshold with the 60 s per-route cooldown of `train_warning_head.alerts_per_hour`; thresholds by `operating_points.thr_for_rate` (bisection, 24 iterations) on alerts / hour. | |
| - **Primary: val-frozen** thresholds at 10 / h and 20 / h fitted on the VAL layouts (deployable, no test peeking). Achieved TEST loads (manual / engaged, alerts / h): A val10 → 4.32 / 6.53, val20 → 9.40 / 14.10; B val10 → 3.83 / 2.99, val20 → 13.69 / 12.91; C val20 → 17.67 / 24.81. | |
| - **Secondary: test-achieved** thresholds at 10 / h and 20 / h (equal achieved load, the `operating_points.py` convention). | |
| - **C cannot reach 10 / h**: its score saturates at 1.0; floor = alerts / h at threshold = max score: val 17.24 / 12.85, test 15.10 / 15.66. At the two 10 / h points C is reported **at its floor** (threshold = max score) and, as a sensitivity, with a seeded random thinning of its alerts to 10 / h (keep probability 10 / floor, 200 replicates, mean). 20 / h is the fair three-way point; 10 / h compares A vs B. | |
| - Thresholds: A manual 0.77799 / 0.67578 / 0.66618 / 0.53377 (val10 / val20 / test10 / test20), A engaged 0.81429 / 0.71777 / 0.75928 / 0.64111; B manual 0.62354 / 0.49316 / 0.53271 / 0.43481, B engaged 0.75586 / 0.62695 / 0.65918 / 0.54785; C = 1.0 (max score) at every point. | |
| ## 3. Sampling frame and design (`eval/newtest_sample.py --seed 0 --version v1 --s3-hard 20`; `results/newtest/manifest_v1.csv` sha256 `2e0869c359481819d6c3eb15b92b76ac5698b00ee713e7b00d917afd3a38bafa`, `frame_counts_v1.json` sha256 `31a8844fa87aabf208980dc454619115c82a3aa21c6381df26d026a7abb3c566`) | |
| Frame = the TEST exposure, partitioned into S1 ∪ S2 (every unit carries an inclusion probability π); S3 is a third, model-independent set overlapping the frame. | |
| - **Exclusion (S1 / S2):** anything within 12.5 s (same route) of a clean-test gold anchor (568), a round-1 H5 test clip (100) or a round-2 test clip (100) — 768 anchors on 143 routes. | |
| - **S1 alert union:** alerts of A, B, C at the frame threshold min(thr_val20, thr_test20) per system × regime (a superset of every operating point's alerts), merged per route with a 10 s same-event guard (new event when an alert is > 10 s after the current event's first alert; anchor t = first alert): 3,316 alerts → 2,403 events → **2,188 after exclusion** (178 gold, 27 round-2, 10 round-1). Pattern = systems with a frame alert in [t−10, t+10] (equals the cluster's own systems for 98.7 % of events). Cells (manual / engaged): A 212 / 199, B 216 / 175, C 315 / 361, A+B 169 / 150, A+C 56 / 46, B+C 62 / 82, A+B+C 74 / 71. **Allocation 160 clips** = per regime A 14, B 14, C 14, A+B 12, A+C 10, B+C 10, A+B+C 6 (discordant cells over-sampled). Device cap 10 over S1. Strata = (cell, device-group): a device whose remaining cap budget is below its count in a cell is its own stratum; among never-cappable devices those earning ≥ 1 draw are their own stratum, the rest are pooled; SRS within; π_i = n_stratum / N_stratum (exact). The 30 devices with more than 10 union events have their 10 draws **pre-allocated** greedily by N over their (cell, device) strata (coverage-maximising; no draw outside the plan). Events on a device-group that receives no draw are **uncovered** (π = 0); `N_covered_cell` and `coverage_cell` are stored: S1 overall coverage 0.809 (per cell 0.58–0.94; manual A 0.75, B 0.83, C 0.88, A+B 0.70, A+C 0.59, B+C 0.71, A+B+C 0.58; engaged A 0.94, B 0.94, C 0.81, A+B 0.88, A+C 0.70, B+C 0.80, A+B+C 0.59). 40 devices in the draw; top device 6.2 % of S1. | |
| - **S2 silent background:** grid rows with speed > 2 m/s at t, ≥ 12.5 s from every union event and excluded anchor: manual 345,793 / engaged 297,106 rows (of 472,705 / 441,689 moving); 20 + 20 clips; strata (regime, device) with cap 6 (1 draw per device pre-allocated, extras proportional), 15 s same-route guard; coverage 0.956 / 0.935; π_i = n / N rows. | |
| - **S3 model-independent candidates:** the 100 round-2 TEST clips (`results/mining/round2_draw_v1.csv`; 41 manual / 59 engaged; π = quota / N_eligible: rare_road 25 / 3,198, demand_noresp 20 / 1,875, demand_resp 20 / 2,103, hand_head 15 / 991, occlusion_night 5 / 8, background 15 / 1,472) + optional 20 `hard_response_h5` clips (`S3_hard`; eligible 1,576 = 1,473 manual / 103 engaged; seed 1, cap 8, 10 / 10). S3 enters paired event-level analyses and AUROC only, **never an HT natural rate**. | |
| - Phases: Phase 1 = S3 (100) + the first 60 S1 clips (regime-interleaved random order) + the S2 calibration clips; Phase 2 = the rest. Calibration = 8 warning clips (S1 3, S2 3, S3 2; 4 / 4 regime) + 4 gaze windows, rated by every rater + owner before Phase 1 and kept in the store (`calibration = true`; analysis reports with / without). | |
| - Per clip stored server-side, never sent to a rater: `fired_{A,B,C}_{val10,val20,test10,test20}`, `t_alert_*`, `score_*_anchor`, stratum, cell, pattern, π, coverage. The tool's client queue is an explicit whitelist (`h5_common.NEVER_SENT_KEYS / PREFIXES` assertion). | |
| - **Pre-specified adaptation:** if the Phase-1 S1 warranted share (Task-1 consensus) is < 0.25, the Phase-2 allocation moves from S2 to the discordant S1 cells (A+B, A+C, B+C, A+B+C); the change and its date are logged in `decisions.md` before it is applied; π of every already-drawn clip is unchanged. | |
| ## 4. Labels (tool `cwm2/annotation_human/annotate_h5.py`, store `labels_newtest.jsonl`, append-only) | |
| - **Task 1** (prefix-locked, two blind raters R(i mod 3), R((i+1) mod 3); rubric `README_annotators.md` unchanged): W_i = Q4 `prompt_warranted_now` consensus (A / S → warranted, D → not, Q → unsure; disagreement / unsure → adjudication by `ADJ*`; `h5_to_consensus.py`); onset o_i = mean start of the raters' `onset_interval_s` (route time) when compatibility ∈ {briefly, sustained} and `concern_persists = ongoing`. | |
| - **Task 2** (post clip [t, t+10] with frame t frozen 1 s; the third rater R((i+2) mod 3), served only if that rater holds no prefix record for the clip): E_i = any `event_type` ≠ none; onset from `event_onset_s` when Task 1 gives none; `ongoing_at_anchor = yes` without a prefix onset → o_i = t, flagged censored. Consensus rule for a second Task-2 record: same type set ∧ |Δonset| ≤ 1 s → mean onset, else adjudication (`h5_to_consensus.py --task event`). The outcome half is compared with the Task-1 raters' outcome stage (the 25 % second rating). | |
| - Quality gates: Q4 binary κ ≥ 0.6, event_type κ ≥ 0.6, onset median |Δt| ≤ 1 s; interim check after 40 clips; κ < 0.4 or unsure > 25 % → pause, rubric addendum (README only; fields never change), re-calibration. | |
| ## 5. Estimators (`eval/newtest_analysis.py`; device-cluster bootstrap 2,000 resamples, dongles with replacement, identical resamples for every paired Δ) | |
| Weights: w_i = (N_cell / N_covered_cell) / π_i on S1 ∪ S2 (post-stratified to the full cell; coverage reported beside every HT number); S3 unweighted. H_regime = 28.50 h manual, 25.10 h engaged. | |
| At each (system X, regime, point L ∈ {val10, val20 primary; test10, test20 secondary}): | |
| 1. **Event recall (primary):** R_X = Σ w_i E_i 1[X alert at thr_X(L) in [o_i − 10, o_i + 3] s] / Σ w_i E_i on S1 ∪ S2; unweighted recall on all labelled events (S1 ∪ S2 ∪ S3) as secondary. Alerts are recomputed from the frozen layouts at the frozen thresholds (`newtest_sample.alert_rows`). | |
| 2. **Effective lead time (primary):** lead_i = o_i − t_first_alert_X in that window; undetected → 0; HT mean; median over both-detected events secondary; paired Δ A−B, A−C, B−C on identical events. | |
| 3. **Natural false alarms:** FA_X / h = Σ_{S1} w_i 1[X alert in [t−10, t+10]] (1 − W_i) / H; precision = HT ratio Σ w fired W / Σ w fired. **Silent-time warranted moments / h** = Σ_{S2} w_i W_i / H (the miss population). Cell-level warranted shares appear only under the header "enriched — not a natural rate". | |
| 4. **Pooling (pre-fixed):** manual and engaged always separate; pooled = HT event-count-weighted mixture (natural mix); equal-weight pooling shown only as a sensitivity. | |
| 5. **AUROC (secondary):** score at the anchor vs W_i, unweighted over all labelled clips and HT-weighted (Mann–Whitney) on S1 ∪ S2; `eval/gold_extended.py` still produces the pre-registered `clean_test_r2` rows from the S3 clips. | |
| 6. **Three questions:** overall = A vs C and B vs C at 20 / h (recall, lead, FA / h); does prediction help = A vs B at 10 / h and 20 / h; vs deployed baseline = C at 20 / h and at its floor (plus the thinning sensitivity). | |
| ## 6. Wording rules and minimum detectable effects | |
| - Every Δ is reported with its CI. CI containing 0 → "**matches** (Δ, CI)"; CI above 0 → "**above** (Δ, CI)"; CI below 0 → "**below** (Δ, CI)". The words "significantly better" and "equivalent" are never used. A per-regime comparison with fewer than 60 labelled events in that regime is labelled **descriptive**. | |
| - Expected positives: S1 warranted share ≈ 0.30 → ≈ 48; S2 ≈ 2; S3 ≈ 25 → ≈ 75 events, ≈ 50 in the HT partition. Paired recall Δ with discordance 0.4: SE = √(0.4 / n) → **MDE ≈ 0.20** (n 75) / 0.25 (n 50) at α 0.05 two-sided, 80 % power; lead-time Δ with SD_diff 3 s → **MDE ≈ 1.0 s** (n 75). The analysis prints the achieved n and the implied MDE next to every paired Δ; an observed |Δ| below the MDE with a CI containing 0 is reported as "matches", not as "no difference". | |
| - The headline sentence of the paper uses the manual regime at the val-frozen 20 / h (three-way) and 10 / h (A vs B) points; the engaged regime, the test-achieved points, C's floor, the HT natural rates (FA / h, silent warranted / h), every S1 cell's coverage and the pre-registration hash appear in the same table. | |
| ## 7. gaze_off human validation (`eval/gaze_off_validation_sample.py --seed 0 --version v1`; `results/gaze_validation/manifest_v1.csv` sha256 `3ffb6991d12e362e6435750451702866478333ba392ae1c7341447f271b911a4`, `frame_counts_v1.json` sha256 `d92c0466330a66687de8d24dc08a90ab9259348db813ac3683c16503edf777e1`) | |
| - VAL split only (181,278 rows, 17 devices; the head never used val, thresholds were fitted on train). State: `facehist_causal_{regime}_val.npz` F0_c[:,7] gaze_off, [:,8] dwell, [:,4] ear_conf_asym, F0valid; `kin_causal_{regime}_val.npz` S0_c[:,1] head_turned, K0valid (kin-valid = K0valid ∧ finite S0). Cells (N): on|straight 137,820; on|turned 2,594; off_brief (dwell ≤ 2 s)|straight 8,181; off_brief|turned 8,564; off_sust|straight 4,015; off_sust|turned 3,832; face_invalid|kin_invalid 15,654 (13,899 on one device); face_invalid|turned 294. Allocation 30 / 25 / 25 / 25 / 25 / 25 / 25 / 10 + 10 regime top-up (→ manual) = **200 windows** (106 engaged / 94 manual, 16 devices, device cap 20); strata = cell × regime × |ear_conf_asym| > 0.3 half (50 / 50 inside every valid-face cell); π = n / N per stratum; 5 s same-route guard; face-invalid cells allocated first. Clip [t−3, t+1], player pauses once at t. Two blind raters per window (R1–R4); 4 calibration windows. | |
| - Fields: `view_region` (H5 enum + down_other), `head_motion`, `eyes_vs_head` {head_and_eyes_on_road, head_on_eyes_off, head_off_eyes_on, both_off, unknown}, `check_kind` {none, mirror_check, instrument_or_console_check, sustained_off_road, unknown}, `occlusion` {none, hand_arm, glasses_hat_mask, lighting, partial_out_of_frame, camera_blocked, not_driver}, `visibility`. | |
| - Metrics (`eval/gaze_off_validation_analysis.py`; HT-weighted with w = 1 / π to the val prevalence; device-cluster bootstrap 2,000): rater κ (view_region collapsed on / off-road, check_kind, eyes_vs_head; target ≥ 0.6); confusion automatic state {on, off_brief, off_sust, face_invalid} × human {on, off}; **precision and recall of gaze_off = 1 vs human off-road and vs sustained off-road**; share of gaze_off = 1 that are mirror checks; share of gaze_off = 1 and of face_invalid with occlusion ≠ none; **recall of eyes-only glances (head_on_eyes_off) by gaze_off — pre-stated expectation ≈ 0**, because gaze_off is a head-yaw hysteresis on |lm_yaw_dev| (on / off thresholds P90 / P75 = 0.212 / 0.103) and cannot see eye movement; the paper states "detects head-away episodes, not eyes-only glances" with the measured value, whatever it is. Wording rules of §6 apply. | |
| ## 8. Analysis code, provenance and reporting | |
| - Code paths fixed here: `eval/newtest_sample.py`, `eval/newtest_render.py`, `eval/newtest_analysis.py`, `eval/gaze_off_validation_sample.py`, `eval/gaze_off_validation_analysis.py`, `cwm2/annotation_human/{h5_common,annotate_h5,h5_to_consensus,smoke_test}.py`, `cwm2/baselines/legacy_cockpitwm/score_val_marginals.py`. Every result table is appended (never overwritten) with a run stamp and the current sha256 of this file; CSVs and run json (`common/runmeta.write_run_json`) accompany every run. | |
| - Test clips never leave the machine; no VLM or cloud service is used; raters never see any model / DMS / sampling information. | |
| - Mandatory in the results section: both regimes, all four operating points, C's floor, all S1 cells with coverage, S2 natural rates, S3 secondary rows, AUROC, calibration-in / -out sensitivity, MDE, this hash. | |
| ## 9. Allowed to change after this hash (and only by a dated `decisions.md` row) | |
| Rendering details that do not alter which clips are shown or their times (codec settings, layout of the rater page); the number of bootstrap resamples (≥ 1,000); the optional S3_hard 20 (in or out, decided before its labels exist); the Phase-2 reallocation of §3 under its pre-specified trigger; rubric addenda (README text only); adjudicator id; schedule dates. Not allowed: thresholds, systems, π, cell definitions, estimator formulas, windows, wording rules, MDE formula, primary / secondary labelling. | |
Xet Storage Details
- Size:
- 16.5 kB
- Xet hash:
- 1331d177bd883e1e3012d76861ab672fc184f5e7ffb5573a2ff402a682396d22
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.