Buckets:

AsianPlayer
/

TriDrive

Files

xet

AsianPlayer/TriDrive / benchmark /human_eval_299 /README.md

AsianPlayer

16 days ago

preview code

download

raw

19.9 kB

# TriDrive — pre-registered human evaluation of the warning probe on a new test set (`benchmark/human_eval_299/`)

A model-targeted human evaluation of the TriDrive current-state warning probe (development name CockpitWM, system **A**) against a current-state-only variant (**B**) and the openpilot-derived driver-monitoring comparator (**C**) on **299 labelled cabin clips** drawn from the TEST exposure of the BATON corpus.  The design, estimators, wording rules and minimum detectable effects were frozen in `newtest_prereg.md` (byte-identical copy; sha256 in `prereg_hash.json`) before any clip was rendered; every later change is a dated row of `newtest_prereg_deviations.md`.  Paper: Appendix E.

## Files

| file | content |
|---|---|
| `manifest_v1.csv` | the frozen clip sample (320 rows: 160 S1 + 40 S2 + 100 S3 + 20 optional S3_hard) with design weights and the frozen alert decisions of A / B / C; `labelled_299` marks the 299 clips that received labels |
| `labels_newtest.jsonl` | the committee labels: 897 records = 299 clips x 3 stages (`prefix`, `outcome`, `event`); one JSON object per line |
| `thresholds_frozen_v1.json` | frozen systems, operating points (val10 / val20 / test10 / test20) and thresholds |
| `deviations_v1.json` | pre-analysis exclusions (one S1 clip without road video) and the post-hoc systems block |
| `inputs_sha256_newtest_sample.txt` | sha256 of every input of the sampling script (paths rewritten to the `/home/USER/...` placeholder) |
| `newtest_prereg.md`, `prereg_hash.json` | the pre-registration (byte-identical) and its registered hash |
| | `prereg_hash.json` also records the sha256 of the *original* `manifest_v1.csv` and `thresholds_frozen_v1.json`; the shipped copies are pseudonymised / scrubbed, so only `newtest_prereg.md` matches byte-for-byte |
| `newtest_prereg_deviations.md` | dated deviation log |
| `tables/headline_manual_valfrozen.md` | headline table of the final run (manual regime, val-frozen thresholds) plus the engaged regime, test-achieved points, calibration in / out sensitivity, AUROC and achieved MDE |
| `tables/full_results.md`, `tables/all_results.json` | index of the full result set and the machine-readable results (main / paired / AUROC / cells / calibration-out / post-hoc / coverage / consensus summary) |
| `tables/analysis_primary/` | per-clip table (`clips.csv`, whitelisted like the manifest + the analysis columns), `main.csv`, `paired.csv`, `auroc.csv`, `cells_enriched.csv`, run record |
| `counts.json` | every count measured while building this folder, the assertions and the scrub report |

Only `tables/` and `tables/analysis_primary/` of the archived run are included; `full_results.md` is the run's own index and also names intermediate files (coverage, consensus, post-hoc, calibration-out CSVs) that are summarised inside `all_results.json` but not shipped as separate files.

## Set definition (prereg Section 3)

* **Frame.**  The TEST exposure of the 5 Hz analysis grid (manual 28.50 h / engaged 25.10 h), excluding everything within 12.5 s of a clean-test gold anchor or of an earlier annotation round.
* **S1 — alert union** (160 clips, 80 per regime).  Events formed from the frame-threshold alerts of A, B and C (10 s same-event guard).  `cell` = regime | pattern of systems firing within +-10 s (A, B, C, A+B, A+C, B+C, A+B+C); discordant cells were over-sampled (per regime A 14, B 14, C 14, A+B 12, A+C 10, B+C 10, A+B+C 6); device cap 10; strata = (cell, device group); `pi` = inclusion probability; `N_covered_cell` / `coverage_cell` = events reachable under the cap.
* **S2 — silent background** (40 clips, 20 per regime).  Moving grid rows (> 2 m/s) at least 12.5 s from every union event; strata (regime, device), cap 6; `pi` = n / N rows.  S2 estimates the miss population (warranted moments per silent hour).
* **S3 — model-independent candidates** (100 clips: the round-2 TEST clips, cells rare_road / demand_noresp / demand_resp / hand_head / occlusion_night / background) and **S3_hard** (20 optional `hard_response` clips).  S3 enters paired event-level analyses and AUROC only, never a Horvitz-Thompson natural rate.
* **Phases.**  Phase 1 = S3 + the first 60 S1 clips + the S2 calibration clips; Phase 2 = the rest (`phase` 1: 160, 2: 160 clips).  `calibration = True` marks the 8 calibration clips rated by every rater before Phase 1 (the analysis reports with / without them); `optional = True` marks S3_hard.
* **Labelled set.**  299 clips: S1 159 (one S1 clip, idx 70074, excluded before analysis — no road video, `deviations_v1.json`), S2 40, S3 100; the 20 S3_hard clips were not labelled.  Regimes in the manifest: engaged 169 / manual 151.
* **Systems.**  A = full predictive model + warning head (score `W_learned_bothhgb_t15_w10_n979`, mean over 3 frozen seeds); B = strongest current-state-only head with the same inputs and supervision; C = openpilot driver-monitoring chain att@3s (deployed comparator).  Alert = score >= threshold with a 60 s per-route cooldown; thresholds fitted on the VAL layouts to 10 / h and 20 / h (primary) or on the TEST layouts (secondary).  C saturates at 1.0 and cannot reach 10 / h (reported at its floor).

## `manifest_v1.csv` columns
| column | meaning |
|---|---|
| `qpos` | position in the rating queue |
| `idx` | clip id (= the `idx` of `labels_newtest.jsonl`); S1 / S2 clips carry ids >= 70000 / 80000 minted by the sampler, S3 / S3_hard clips reuse the id of their source episode |
| `s15_idx` | id of the source episode in the project's episode store (S3 / S3_hard clips only, empty otherwise); none of these episodes is part of `../labels/` |
| `clip_id` | opaque 13-hex id of the rendered clip package |
| `stratum`, `phase`, `calibration`, `optional` | design membership (see above) |
| `regime`, `layout_regime` | regime of the anchor row and of the frozen score layout it was read from |
| `uid`, `driver`, `dev_group` | pseudonymised route-segment group `driver_N__route_M__route_a-b`, pseudonymised device `driver_N`, and the device group of the S1 / S2 stratum (`driver_N`, `_pooled`, or empty for S3) |
| `t_anchor` | anchor time, route-relative seconds |
| `anchor_s` | position of the anchor inside the rendered clip (s) |
| `cell`, `pattern`, `pattern_cluster` | S1 cell, firing pattern within +-10 s, pattern of the event's own cluster |
| `pi`, `N_cell`, `n_cell`, `N_covered_cell`, `coverage_cell`, `N_stratum`, `n_stratum` | inclusion probability and the design counts behind it |
| `s3_stratum`, `s3_draw_rank` | S3 mining stratum and draw rank |
| `speed_anchor_mps` | vehicle speed at the anchor (m/s) |
| `fired_X_<point>` | 1 if system X (A / B / C) alerts within +-10 s of the anchor at the frozen threshold of operating point `val10` / `val20` / `test10` / `test20` |
| `t_alert_X_<point>`, `t_alert_X_frame` | route time of that alert (point and frame threshold) |
| `score_X_anchor`, `thr_X_frame` | score of system X at the anchor row and the frame threshold |
| `labelled_299` | True for the 299 clips with labels |

## `labels_newtest.jsonl` records
Each clip has three records: `stage = prefix` (Task 1, Questions 1-5, judged with the player hard-limited to the anchor), `stage = outcome` (the 'what happened' half of Question 5, judged after the prefix record was locked) and `stage = event` (Task 2, Question 6 on the 10 s after the anchor, frame t frozen for 1 s).  Labels are the committee consensus (`protocol = committee_consensus`, deviation row of 2026-09-13); rater identities, timing and client metadata were removed.

Record keys: `idx`, `clip_id`, `stage`, `task`, `anchor_s` (anchor position in the clip, s), `visible_until_s` (last clip second the rater could see: the anchor for the prefix stage, anchor + 10 s otherwise), `split`, `stratum`, `cell`, `pattern`, `pi`, `regime`, `phase`, `calibration`, `mode_at_anchor` (`human` / `oem_acc` / `openpilot`), `schema_ok`, `problems` (schema checks that failed), `prefill` (which outcome / event fields were prefilled from the vehicle log and accepted), `fields` (the answers), `valid` and `unknown` (per-field flags), `uid`, `driver`, `t0` (= `t_anchor` of the manifest, route-relative s), `protocol`, `schema_version`.  Times inside `fields` are clip-relative seconds (route time = `t0` + value - `anchor_s`).

### `fields` of stage `prefix`

| field | values | meaning |
|---|---|---|
| `view_region` | road_ahead / left_mirror_or_window / right_mirror_or_window / rearview_mirror / instrument_cluster / center_console / lap_or_phone_area / passenger / unknown | where the driver is looking at the anchor (coarse region) |
| `head_motion` | steady / brief_glance / sustained_turn / unknown | head motion over the window up to the anchor |
| `left_hand` | on_wheel / off_wheel_resting / off_wheel_holding_object / not_visible / unknown | left-hand placement at the anchor |
| `right_hand` | on_wheel / off_wheel_resting / off_wheel_holding_object / not_visible / unknown | right-hand placement at the anchor |
| `object_interaction` | none / phone / food_or_drink / vehicle_control / other_object / unknown | object the driver is handling |
| `visibility` | good / partial / insufficient | how well the cabin view supports the judgement |
| `task_families` | list of: longitudinal_following / lateral_maneuver_or_check / sustained_supervision / cabin_interaction_demand / none / unknown | road-task families in force (multi-select) |
| `demand_level` | low / moderate / high / unknown | road demand at the anchor |
| `notable_event` | none / lead_vehicle_braking / cut_in / intersection_or_merge / lane_change_by_ego / pedestrian_or_cyclist / curve / other / unknown | notable road event up to the anchor |
| `compatibility` | compatible / briefly_incompatible / sustained_incompatible / unknown | behaviour-situation compatibility (Question 3) |
| `concern_persists` | no / brief / ongoing / unknown | whether the concern still holds at the anchor |
| `prompt_warranted_now` | not_warranted / warranted_attention / warranted_hands / unsure | Question 4 - is a short uniform reminder warranted at the anchor; W = 1 for warranted_attention / warranted_hands, 0 for not_warranted (unsure -> adjudication) |
| `confidence` | low / medium / high | rater confidence in Question 4 |
| `reason_codes` | list of: normal_mirror_or_instrument_check / brief_glance_low_demand / sustained_gaze_away_with_demand / hands_off_with_demand / object_interaction_while_moving / stopped_or_creeping / automation_handling_it / insufficient_visibility / insufficient_evidence / other | reason codes behind Question 4 (multi-select) |
| `evidence_marks_s` | list of numbers (s) | clip-relative times (s, <= anchor) the rater marked as evidence |
| `onset_interval_s` | null or array (s) | [start, end] clip-relative interval (s) in which the incompatible behaviour began; required when not compatible; null otherwise |
### `fields` of stage `outcome`
| field | values | meaning |
|---|---|---|
| `first_gaze_to_road_s` | null or number (s) | clip-relative time (s) of the first gaze return to the road after the anchor; null = not within the clip |
| `first_hand_to_wheel_s` | null or number (s) | clip-relative time (s) at which both hands are back on the wheel; null = not within the clip |
| `vehicle_response_kind` | none / braking / acceleration / steering_correction / lane_change / stop / unknown | vehicle response within the 10 s after the anchor (prefilled from the vehicle log, confirmed by the rater) |
| `vehicle_response_attributed_to` | driver / automation / unclear / none | who produced the vehicle response (driver / automation / none / unclear) |
| `scene_resolution` | demand_resolved / demand_persists / demand_escalated / unknown | how the road situation resolved after the anchor |
Derived labels used by the analysis: **W** = 1 if `prompt_warranted_now` in {warranted_attention, warranted_hands}, 0 if `not_warranted` (measured over the 299 prefix records: 82 warranted [65 attention / 17 hands], 215 not warranted, 2 unsure); **E** = 1 if the event-stage `event_type` is not `['none']` (measured: 78 events — S1 48, S2 5, S3 25); the event onset follows the prereg precedence Task-1 `onset_interval_s` > Task-2 `event_onset_s` > anchor (censored).
## Headline numbers (copied from `tables/headline_manual_valfrozen.md`)
Manual regime, val-frozen 20 / h (the paper's headline point, prereg Section 6): event recall on the Horvitz-Thompson partition A 0.233 [0.02, 0.54], B 0.087 [0.00, 0.42], C 0.000 [0.00, 0.00] (20 HT / 29 labelled events); false alarms per hour A 4.03 [1.23, 8.04], B 8.63 [1.41, 21.83], C 12.41 [2.96, 29.31]; precision A 0.513 [0.05, 0.85], B 0.288 [0.08, 0.78], C 0.175 [0.02, 0.55].  Paired A-C recall delta +0.233 [0.022, 0.543] reads 'above' under the registered wording rule, but the registered MDE at n = 20 is 0.40, larger than every observed |delta| in this regime: every per-regime row is **descriptive** by the prereg's own rule (fewer than 60 labelled events per regime).  Engaged regime, val10: A 0.409 [0.00, 0.93] recall HT.  AUROC of the anchor score against W over all labelled clips: manual A 0.781 [0.68, 0.87], B 0.750 [0.65, 0.85], C 0.478 [0.33, 0.70]; engaged A 0.641 [0.54, 0.74], B 0.654 [0.55, 0.77], C 0.758 [0.69, 0.83].  Device-cluster bootstrap, 2,000 resamples.
Verbatim sections of the headline table:
### 20 / h three-way (val20; C at 17.7 / h achieved)
| system | thr | achieved TEST alerts / h | n events (HT / all) | event recall HT [CI] | recall all-labelled | lead0 mean s [CI] | lead median detected s | FA / h [CI] | precision HT [CI] | silent warranted rows / h [CI] |
|---|---|---|---|---|---|---|---|---|---|---|
| A | 0.67578 | 9.40 | 20 / 29 | 0.233 [0.02, 0.54] | 0.276 | -0.14 [-0.45, -0.02] | -0.62 | 4.03 [1.23, 8.04] | 0.513 [0.05, 0.85] | 191.2 [0.0, 518.4] |
| B | 0.49316 | 13.69 | 20 / 29 | 0.087 [0.00, 0.42] | 0.207 | -0.04 [-0.21, 0.00] | -0.40 | 8.63 [1.41, 21.83] | 0.288 [0.08, 0.78] | 191.2 [0.0, 518.4] |
| C | 1.00000 | 17.67 | 20 / 29 | 0.000 [0.00, 0.00] | 0.034 | 0.00 [0.00, 0.00] | — | 12.41 [2.96, 29.31] | 0.175 [0.02, 0.55] | 191.2 [0.0, 518.4] |
S1 per-cell coverage in force (prereg §6: beside the headline; N covered = union events with pi > 0 under the device cap after the pre-analysis exclusion 70074, engaged|A+B 132 -> 128):
| cell | N | N covered | coverage | n labelled | W = 1 | E = 1 |
|---|---|---|---|---|---|---|
| engaged|A | 199 | 187 | 0.940 | 14 | 7 | 8 |
| engaged|A+B | 150 | 128 | 0.853 | 11 | 5 | 5 |
| engaged|A+B+C | 71 | 42 | 0.592 | 6 | 3 | 2 |
| engaged|A+C | 46 | 32 | 0.696 | 10 | 4 | 4 |
| engaged|B | 175 | 165 | 0.943 | 14 | 4 | 4 |
| engaged|B+C | 82 | 66 | 0.805 | 10 | 5 | 2 |
| engaged|C | 361 | 291 | 0.806 | 14 | 4 | 3 |
| manual|A | 212 | 160 | 0.755 | 14 | 4 | 5 |
| manual|A+B | 169 | 119 | 0.704 | 12 | 6 | 7 |
| manual|A+B+C | 74 | 43 | 0.581 | 6 | 2 | 1 |
| manual|A+C | 56 | 33 | 0.589 | 10 | 3 | 1 |
| manual|B | 216 | 179 | 0.829 | 14 | 5 | 4 |
| manual|B+C | 62 | 44 | 0.710 | 10 | 3 | 2 |
| manual|C | 315 | 277 | 0.879 | 14 | 1 | 0 |
Exposure H used for every /h figure: manual 28.4963 h (512,934 test rows x 0.2 s), engaged 25.1022 h (prereg §5 prints the rounded 28.50 / 25.10 h; difference 0.013 %).
| comparison (manual) | point | n events | Δ recall [CI] | MDE recall (registered, disc 0.4) | MDE recall (observed disc) | Δ lead0 s [CI] | MDE lead s | wording (rule verbatim) |
|---|---|---|---|---|---|---|---|---|
| A-C | val20 | 20 | +0.233 [0.022, 0.543] | 0.40 | 0.34 | -0.139 [-0.452, -0.019] | 1.88 | recall above (Δ +0.233, CI [+0.022, +0.543] recall); lead below (Δ -0.139 s, CI [-0.452, -0.019] lead) (descriptive) |
| B-C | val20 | 20 | +0.087 [0.000, 0.422] | 0.40 | 0.31 | -0.041 [-0.205, 0.000] | 1.88 | recall matches (Δ +0.087, CI [+0.000, +0.422] recall); lead matches (Δ -0.041 s, CI [-0.205, +0.000] lead) (descriptive) |
| A-B | val20 | 20 | +0.146 [-0.106, 0.413] | 0.40 | 0.24 | -0.097 [-0.331, 0.041] | 1.88 | recall matches (Δ +0.146, CI [-0.106, +0.413] recall); lead matches (Δ -0.097 s, CI [-0.331, +0.041] lead) (descriptive) |
| A-B | val10 | 20 | -0.056 [-0.302, 0.000] | 0.40 | 0.24 | +0.013 [-0.013, 0.087] | 1.88 | recall matches (Δ -0.056, CI [-0.302, +0.000] recall); lead matches (Δ +0.013 s, CI [-0.013, +0.087] lead) (descriptive) |
Reading notes (from `clips.csv` of this run): (i) a CI bound printed as 0.000 is a degenerate point mass at Δ = 0 in the device-cluster resamples; the wording rule uses the exact bound (lo ≤ 0 ≤ hi -> 'matches'; lo > 0 -> 'above'), so 'above' with a printed lower bound of +0.000 means lo > 0 at the 4th decimal. (ii) Lead sign: positive = alert before the event onset; lead0 counts an undetected event as 0, so a negative mean lead0 (A: alerts on average shortly AFTER onset) reads 'below' C's exact 0 under the rule. (iii) Onset precedence as implemented by `load_labels`: Task-1 prefix onset (E = 1 clips: 53) > Task-2 event onset (25) > anchor-censored (0); 21 E = 1 clips with ongoing_at_anchor = yes carry an explicit onset and are therefore not censored. (iv) Fragility of the manual HT partition (20 events on 11 devices): the single event idx 70052 (manual|A, w 76.85) holds 0.313 of the event weight (detected by none of A / B / C at val20); the A-C 'above' at val20 rests on the 2 event(s) detected only by A: idx 70115 (w share 0.023), idx 70061 (w share 0.146); the registered MDE at n = 20 is 0.40, larger than every observed |Δ| in this regime — the headline is descriptive by the prereg's own rule.

### 10 / h (val10) — A vs B; C at its floor (cannot reach 10 / h) with the thinning sensitivity

| system | thr | achieved TEST alerts / h | n events (HT / all) | event recall HT [CI] | recall all-labelled | lead0 mean s [CI] | FA / h [CI] | precision HT [CI] | note |
|---|---|---|---|---|---|---|---|---|---|
| A | 0.77799 | 4.32 | 20 / 29 | 0.008 [0.00, 0.05] | 0.103 | -0.01 [-0.09, 0.00] | 1.25 [0.15, 2.74] | 0.493 [0.00, 0.90] |  |
| B | 0.62354 | 3.83 | 20 / 29 | 0.064 [0.00, 0.34] | 0.172 | -0.03 [-0.14, 0.00] | 1.05 [0.15, 2.40] | 0.478 [0.00, 0.89] |  |
| C | 1.00000 | 15.10 | 20 / 29 | 0.000 [0.00, 0.00] | 0.034 | 0.00 [0.00, 0.00] | 11.52 [2.18, 28.67] | 0.122 [0.01, 0.53] | C at floor 15.1/h; thinned to 10/h recall HT 0.000 (keep prob 0.662, 200 replicates) |

Silent-time warranted rows / h (S2, HT): manual 191.2 [0.0, 518.4] (= 0.0106 of silent time), engaged 3083.1 [0.0, 7549.8] (= 0.1713); S2 events (E = 1) manual 0, engaged 5.

### AUROC of the anchor score vs W_i (secondary): unweighted over all labelled clips; HT-weighted on S1 u S2
| regime | system | n / pos | AUROC [CI] | AUROC HT (n) |
|---|---|---|---|---|
| manual | A | 139 / 32 | 0.781 [0.68, 0.87] | 0.996 (100) |
| manual | B | 139 / 32 | 0.750 [0.65, 0.85] | 0.994 (100) |
| manual | C | 134 / 27 | 0.478 [0.33, 0.70] | 0.984 (95) |
| engaged | A | 142 / 48 | 0.641 [0.54, 0.74] | 0.668 (99) |
| engaged | B | 142 / 48 | 0.654 [0.55, 0.77] | 0.494 (99) |
| engaged | C | 136 / 43 | 0.758 [0.69, 0.83] | 0.732 (94) |
| pooled | A | 281 / 80 | 0.712 [0.63, 0.79] | 0.869 (199) |
| pooled | B | 281 / 80 | 0.708 [0.63, 0.79] | 0.799 (199) |
| pooled | C | 270 / 70 | 0.641 [0.52, 0.75] | 0.750 (189) |
## Identifiers
`uid`, `driver` and `dev_group` are pseudonyms (`driver_N`, `driver_N__route_M__route_a-b`) shared by every file in this bucket and by the public BATON release.  Raw device ids, recording leaves, local video paths, rater identities, hosts and timing metadata were removed; `clip_id` is an opaque package id.  Where a document mentions the rating panel it uses its code name only.
## Licence
Labels, manifest and result tables: **CC BY-NC 4.0**, inherited from the BATON corpus — see `../../data/LICENSE-DATA`.  Attribution: "TriDrive authors (anonymous for double-blind review)".

Xet Storage Details

Size:: 19.9 kB
Xet hash:: 8faca7a3de742ca92e662121c9e42a6b737baa36d5d7a7d65f9c7e729b2661f9

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.

	# TriDrive — pre-registered human evaluation of the warning probe on a new test set (`benchmark/human_eval_299/`)

	A model-targeted human evaluation of the TriDrive current-state warning probe (development name CockpitWM, system A) against a current-state-only variant (B) and the openpilot-derived driver-monitoring comparator (C) on 299 labelled cabin clips drawn from the TEST exposure of the BATON corpus. The design, estimators, wording rules and minimum detectable effects were frozen in `newtest_prereg.md` (byte-identical copy; sha256 in `prereg_hash.json`) before any clip was rendered; every later change is a dated row of `newtest_prereg_deviations.md`. Paper: Appendix E.

	## Files

	\| file \| content \|
	\|---\|---\|
	\| `manifest_v1.csv` \| the frozen clip sample (320 rows: 160 S1 + 40 S2 + 100 S3 + 20 optional S3_hard) with design weights and the frozen alert decisions of A / B / C; `labelled_299` marks the 299 clips that received labels \|
	\| `labels_newtest.jsonl` \| the committee labels: 897 records = 299 clips x 3 stages (`prefix`, `outcome`, `event`); one JSON object per line \|
	\| `thresholds_frozen_v1.json` \| frozen systems, operating points (val10 / val20 / test10 / test20) and thresholds \|
	\| `deviations_v1.json` \| pre-analysis exclusions (one S1 clip without road video) and the post-hoc systems block \|
	\| `inputs_sha256_newtest_sample.txt` \| sha256 of every input of the sampling script (paths rewritten to the `/home/USER/...` placeholder) \|
	\| `newtest_prereg.md`, `prereg_hash.json` \| the pre-registration (byte-identical) and its registered hash \|
	\| \| `prereg_hash.json` also records the sha256 of the original `manifest_v1.csv` and `thresholds_frozen_v1.json`; the shipped copies are pseudonymised / scrubbed, so only `newtest_prereg.md` matches byte-for-byte \|
	\| `newtest_prereg_deviations.md` \| dated deviation log \|
	\| `tables/headline_manual_valfrozen.md` \| headline table of the final run (manual regime, val-frozen thresholds) plus the engaged regime, test-achieved points, calibration in / out sensitivity, AUROC and achieved MDE \|
	\| `tables/full_results.md`, `tables/all_results.json` \| index of the full result set and the machine-readable results (main / paired / AUROC / cells / calibration-out / post-hoc / coverage / consensus summary) \|
	\| `tables/analysis_primary/` \| per-clip table (`clips.csv`, whitelisted like the manifest + the analysis columns), `main.csv`, `paired.csv`, `auroc.csv`, `cells_enriched.csv`, run record \|
	\| `counts.json` \| every count measured while building this folder, the assertions and the scrub report \|

	Only `tables/` and `tables/analysis_primary/` of the archived run are included; `full_results.md` is the run's own index and also names intermediate files (coverage, consensus, post-hoc, calibration-out CSVs) that are summarised inside `all_results.json` but not shipped as separate files.

	## Set definition (prereg Section 3)

	* Frame. The TEST exposure of the 5 Hz analysis grid (manual 28.50 h / engaged 25.10 h), excluding everything within 12.5 s of a clean-test gold anchor or of an earlier annotation round.
	* S1 — alert union (160 clips, 80 per regime). Events formed from the frame-threshold alerts of A, B and C (10 s same-event guard). `cell` = regime \| pattern of systems firing within +-10 s (A, B, C, A+B, A+C, B+C, A+B+C); discordant cells were over-sampled (per regime A 14, B 14, C 14, A+B 12, A+C 10, B+C 10, A+B+C 6); device cap 10; strata = (cell, device group); `pi` = inclusion probability; `N_covered_cell` / `coverage_cell` = events reachable under the cap.
	* S2 — silent background (40 clips, 20 per regime). Moving grid rows (> 2 m/s) at least 12.5 s from every union event; strata (regime, device), cap 6; `pi` = n / N rows. S2 estimates the miss population (warranted moments per silent hour).
	* S3 — model-independent candidates (100 clips: the round-2 TEST clips, cells rare_road / demand_noresp / demand_resp / hand_head / occlusion_night / background) and S3_hard (20 optional `hard_response` clips). S3 enters paired event-level analyses and AUROC only, never a Horvitz-Thompson natural rate.
	* Phases. Phase 1 = S3 + the first 60 S1 clips + the S2 calibration clips; Phase 2 = the rest (`phase` 1: 160, 2: 160 clips). `calibration = True` marks the 8 calibration clips rated by every rater before Phase 1 (the analysis reports with / without them); `optional = True` marks S3_hard.
	* Labelled set. 299 clips: S1 159 (one S1 clip, idx 70074, excluded before analysis — no road video, `deviations_v1.json`), S2 40, S3 100; the 20 S3_hard clips were not labelled. Regimes in the manifest: engaged 169 / manual 151.
	* Systems. A = full predictive model + warning head (score `W_learned_bothhgb_t15_w10_n979`, mean over 3 frozen seeds); B = strongest current-state-only head with the same inputs and supervision; C = openpilot driver-monitoring chain att@3s (deployed comparator). Alert = score >= threshold with a 60 s per-route cooldown; thresholds fitted on the VAL layouts to 10 / h and 20 / h (primary) or on the TEST layouts (secondary). C saturates at 1.0 and cannot reach 10 / h (reported at its floor).

	## `manifest_v1.csv` columns

	\| column \| meaning \|
	\|---\|---\|
	\| `qpos` \| position in the rating queue \|
	\| `idx` \| clip id (= the `idx` of `labels_newtest.jsonl`); S1 / S2 clips carry ids >= 70000 / 80000 minted by the sampler, S3 / S3_hard clips reuse the id of their source episode \|
	\| `s15_idx` \| id of the source episode in the project's episode store (S3 / S3_hard clips only, empty otherwise); none of these episodes is part of `../labels/` \|
	\| `clip_id` \| opaque 13-hex id of the rendered clip package \|
	\| `stratum`, `phase`, `calibration`, `optional` \| design membership (see above) \|
	\| `regime`, `layout_regime` \| regime of the anchor row and of the frozen score layout it was read from \|
	\| `uid`, `driver`, `dev_group` \| pseudonymised route-segment group `driver_N__route_M__route_a-b`, pseudonymised device `driver_N`, and the device group of the S1 / S2 stratum (`driver_N`, `_pooled`, or empty for S3) \|
	\| `t_anchor` \| anchor time, route-relative seconds \|
	\| `anchor_s` \| position of the anchor inside the rendered clip (s) \|
	\| `cell`, `pattern`, `pattern_cluster` \| S1 cell, firing pattern within +-10 s, pattern of the event's own cluster \|
	\| `pi`, `N_cell`, `n_cell`, `N_covered_cell`, `coverage_cell`, `N_stratum`, `n_stratum` \| inclusion probability and the design counts behind it \|
	\| `s3_stratum`, `s3_draw_rank` \| S3 mining stratum and draw rank \|
	\| `speed_anchor_mps` \| vehicle speed at the anchor (m/s) \|
	\| `fired_X_<point>` \| 1 if system X (A / B / C) alerts within +-10 s of the anchor at the frozen threshold of operating point `val10` / `val20` / `test10` / `test20` \|
	\| `t_alert_X_<point>`, `t_alert_X_frame` \| route time of that alert (point and frame threshold) \|
	\| `score_X_anchor`, `thr_X_frame` \| score of system X at the anchor row and the frame threshold \|
	\| `labelled_299` \| True for the 299 clips with labels \|

	## `labels_newtest.jsonl` records

	Each clip has three records: `stage = prefix` (Task 1, Questions 1-5, judged with the player hard-limited to the anchor), `stage = outcome` (the 'what happened' half of Question 5, judged after the prefix record was locked) and `stage = event` (Task 2, Question 6 on the 10 s after the anchor, frame t frozen for 1 s). Labels are the committee consensus (`protocol = committee_consensus`, deviation row of 2026-09-13); rater identities, timing and client metadata were removed.

	Record keys: `idx`, `clip_id`, `stage`, `task`, `anchor_s` (anchor position in the clip, s), `visible_until_s` (last clip second the rater could see: the anchor for the prefix stage, anchor + 10 s otherwise), `split`, `stratum`, `cell`, `pattern`, `pi`, `regime`, `phase`, `calibration`, `mode_at_anchor` (`human` / `oem_acc` / `openpilot`), `schema_ok`, `problems` (schema checks that failed), `prefill` (which outcome / event fields were prefilled from the vehicle log and accepted), `fields` (the answers), `valid` and `unknown` (per-field flags), `uid`, `driver`, `t0` (= `t_anchor` of the manifest, route-relative s), `protocol`, `schema_version`. Times inside `fields` are clip-relative seconds (route time = `t0` + value - `anchor_s`).

	### `fields` of stage `prefix`

	\| field \| values \| meaning \|
	\|---\|---\|---\|
	\| `view_region` \| road_ahead / left_mirror_or_window / right_mirror_or_window / rearview_mirror / instrument_cluster / center_console / lap_or_phone_area / passenger / unknown \| where the driver is looking at the anchor (coarse region) \|
	\| `head_motion` \| steady / brief_glance / sustained_turn / unknown \| head motion over the window up to the anchor \|
	\| `left_hand` \| on_wheel / off_wheel_resting / off_wheel_holding_object / not_visible / unknown \| left-hand placement at the anchor \|
	\| `right_hand` \| on_wheel / off_wheel_resting / off_wheel_holding_object / not_visible / unknown \| right-hand placement at the anchor \|
	\| `object_interaction` \| none / phone / food_or_drink / vehicle_control / other_object / unknown \| object the driver is handling \|
	\| `visibility` \| good / partial / insufficient \| how well the cabin view supports the judgement \|
	\| `task_families` \| list of: longitudinal_following / lateral_maneuver_or_check / sustained_supervision / cabin_interaction_demand / none / unknown \| road-task families in force (multi-select) \|
	\| `demand_level` \| low / moderate / high / unknown \| road demand at the anchor \|
	\| `notable_event` \| none / lead_vehicle_braking / cut_in / intersection_or_merge / lane_change_by_ego / pedestrian_or_cyclist / curve / other / unknown \| notable road event up to the anchor \|
	\| `compatibility` \| compatible / briefly_incompatible / sustained_incompatible / unknown \| behaviour-situation compatibility (Question 3) \|
	\| `concern_persists` \| no / brief / ongoing / unknown \| whether the concern still holds at the anchor \|
	\| `prompt_warranted_now` \| not_warranted / warranted_attention / warranted_hands / unsure \| Question 4 - is a short uniform reminder warranted at the anchor; W = 1 for warranted_attention / warranted_hands, 0 for not_warranted (unsure -> adjudication) \|
	\| `confidence` \| low / medium / high \| rater confidence in Question 4 \|
	\| `reason_codes` \| list of: normal_mirror_or_instrument_check / brief_glance_low_demand / sustained_gaze_away_with_demand / hands_off_with_demand / object_interaction_while_moving / stopped_or_creeping / automation_handling_it / insufficient_visibility / insufficient_evidence / other \| reason codes behind Question 4 (multi-select) \|
	\| `evidence_marks_s` \| list of numbers (s) \| clip-relative times (s, <= anchor) the rater marked as evidence \|
	\| `onset_interval_s` \| null or array (s) \| [start, end] clip-relative interval (s) in which the incompatible behaviour began; required when not compatible; null otherwise \|

	### `fields` of stage `outcome`

	\| field \| values \| meaning \|
	\|---\|---\|---\|
	\| `first_gaze_to_road_s` \| null or number (s) \| clip-relative time (s) of the first gaze return to the road after the anchor; null = not within the clip \|
	\| `first_hand_to_wheel_s` \| null or number (s) \| clip-relative time (s) at which both hands are back on the wheel; null = not within the clip \|
	\| `vehicle_response_kind` \| none / braking / acceleration / steering_correction / lane_change / stop / unknown \| vehicle response within the 10 s after the anchor (prefilled from the vehicle log, confirmed by the rater) \|
	\| `vehicle_response_attributed_to` \| driver / automation / unclear / none \| who produced the vehicle response (driver / automation / none / unclear) \|
	\| `scene_resolution` \| demand_resolved / demand_persists / demand_escalated / unknown \| how the road situation resolved after the anchor \|

	Derived labels used by the analysis: W = 1 if `prompt_warranted_now` in {warranted_attention, warranted_hands}, 0 if `not_warranted` (measured over the 299 prefix records: 82 warranted [65 attention / 17 hands], 215 not warranted, 2 unsure); E = 1 if the event-stage `event_type` is not `['none']` (measured: 78 events — S1 48, S2 5, S3 25); the event onset follows the prereg precedence Task-1 `onset_interval_s` > Task-2 `event_onset_s` > anchor (censored).

	## Headline numbers (copied from `tables/headline_manual_valfrozen.md`)

	Manual regime, val-frozen 20 / h (the paper's headline point, prereg Section 6): event recall on the Horvitz-Thompson partition A 0.233 [0.02, 0.54], B 0.087 [0.00, 0.42], C 0.000 [0.00, 0.00] (20 HT / 29 labelled events); false alarms per hour A 4.03 [1.23, 8.04], B 8.63 [1.41, 21.83], C 12.41 [2.96, 29.31]; precision A 0.513 [0.05, 0.85], B 0.288 [0.08, 0.78], C 0.175 [0.02, 0.55]. Paired A-C recall delta +0.233 [0.022, 0.543] reads 'above' under the registered wording rule, but the registered MDE at n = 20 is 0.40, larger than every observed \|delta\| in this regime: every per-regime row is descriptive by the prereg's own rule (fewer than 60 labelled events per regime). Engaged regime, val10: A 0.409 [0.00, 0.93] recall HT. AUROC of the anchor score against W over all labelled clips: manual A 0.781 [0.68, 0.87], B 0.750 [0.65, 0.85], C 0.478 [0.33, 0.70]; engaged A 0.641 [0.54, 0.74], B 0.654 [0.55, 0.77], C 0.758 [0.69, 0.83]. Device-cluster bootstrap, 2,000 resamples.

	Verbatim sections of the headline table:

	### 20 / h three-way (val20; C at 17.7 / h achieved)

	\| system \| thr \| achieved TEST alerts / h \| n events (HT / all) \| event recall HT [CI] \| recall all-labelled \| lead0 mean s [CI] \| lead median detected s \| FA / h [CI] \| precision HT [CI] \| silent warranted rows / h [CI] \|
	\|---\|---\|---\|---\|---\|---\|---\|---\|---\|---\|---\|
	\| A \| 0.67578 \| 9.40 \| 20 / 29 \| 0.233 [0.02, 0.54] \| 0.276 \| -0.14 [-0.45, -0.02] \| -0.62 \| 4.03 [1.23, 8.04] \| 0.513 [0.05, 0.85] \| 191.2 [0.0, 518.4] \|
	\| B \| 0.49316 \| 13.69 \| 20 / 29 \| 0.087 [0.00, 0.42] \| 0.207 \| -0.04 [-0.21, 0.00] \| -0.40 \| 8.63 [1.41, 21.83] \| 0.288 [0.08, 0.78] \| 191.2 [0.0, 518.4] \|
	\| C \| 1.00000 \| 17.67 \| 20 / 29 \| 0.000 [0.00, 0.00] \| 0.034 \| 0.00 [0.00, 0.00] \| — \| 12.41 [2.96, 29.31] \| 0.175 [0.02, 0.55] \| 191.2 [0.0, 518.4] \|

	S1 per-cell coverage in force (prereg §6: beside the headline; N covered = union events with pi > 0 under the device cap after the pre-analysis exclusion 70074, engaged\|A+B 132 -> 128):

	\| cell \| N \| N covered \| coverage \| n labelled \| W = 1 \| E = 1 \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| engaged\|A \| 199 \| 187 \| 0.940 \| 14 \| 7 \| 8 \|
	\| engaged\|A+B \| 150 \| 128 \| 0.853 \| 11 \| 5 \| 5 \|
	\| engaged\|A+B+C \| 71 \| 42 \| 0.592 \| 6 \| 3 \| 2 \|
	\| engaged\|A+C \| 46 \| 32 \| 0.696 \| 10 \| 4 \| 4 \|
	\| engaged\|B \| 175 \| 165 \| 0.943 \| 14 \| 4 \| 4 \|
	\| engaged\|B+C \| 82 \| 66 \| 0.805 \| 10 \| 5 \| 2 \|
	\| engaged\|C \| 361 \| 291 \| 0.806 \| 14 \| 4 \| 3 \|
	\| manual\|A \| 212 \| 160 \| 0.755 \| 14 \| 4 \| 5 \|
	\| manual\|A+B \| 169 \| 119 \| 0.704 \| 12 \| 6 \| 7 \|
	\| manual\|A+B+C \| 74 \| 43 \| 0.581 \| 6 \| 2 \| 1 \|
	\| manual\|A+C \| 56 \| 33 \| 0.589 \| 10 \| 3 \| 1 \|
	\| manual\|B \| 216 \| 179 \| 0.829 \| 14 \| 5 \| 4 \|
	\| manual\|B+C \| 62 \| 44 \| 0.710 \| 10 \| 3 \| 2 \|
	\| manual\|C \| 315 \| 277 \| 0.879 \| 14 \| 1 \| 0 \|

	Exposure H used for every /h figure: manual 28.4963 h (512,934 test rows x 0.2 s), engaged 25.1022 h (prereg §5 prints the rounded 28.50 / 25.10 h; difference 0.013 %).

	\| comparison (manual) \| point \| n events \| Δ recall [CI] \| MDE recall (registered, disc 0.4) \| MDE recall (observed disc) \| Δ lead0 s [CI] \| MDE lead s \| wording (rule verbatim) \|
	\|---\|---\|---\|---\|---\|---\|---\|---\|---\|
	\| A-C \| val20 \| 20 \| +0.233 [0.022, 0.543] \| 0.40 \| 0.34 \| -0.139 [-0.452, -0.019] \| 1.88 \| recall above (Δ +0.233, CI [+0.022, +0.543] recall); lead below (Δ -0.139 s, CI [-0.452, -0.019] lead) (descriptive) \|
	\| B-C \| val20 \| 20 \| +0.087 [0.000, 0.422] \| 0.40 \| 0.31 \| -0.041 [-0.205, 0.000] \| 1.88 \| recall matches (Δ +0.087, CI [+0.000, +0.422] recall); lead matches (Δ -0.041 s, CI [-0.205, +0.000] lead) (descriptive) \|
	\| A-B \| val20 \| 20 \| +0.146 [-0.106, 0.413] \| 0.40 \| 0.24 \| -0.097 [-0.331, 0.041] \| 1.88 \| recall matches (Δ +0.146, CI [-0.106, +0.413] recall); lead matches (Δ -0.097 s, CI [-0.331, +0.041] lead) (descriptive) \|
	\| A-B \| val10 \| 20 \| -0.056 [-0.302, 0.000] \| 0.40 \| 0.24 \| +0.013 [-0.013, 0.087] \| 1.88 \| recall matches (Δ -0.056, CI [-0.302, +0.000] recall); lead matches (Δ +0.013 s, CI [-0.013, +0.087] lead) (descriptive) \|

	Reading notes (from `clips.csv` of this run): (i) a CI bound printed as 0.000 is a degenerate point mass at Δ = 0 in the device-cluster resamples; the wording rule uses the exact bound (lo ≤ 0 ≤ hi -> 'matches'; lo > 0 -> 'above'), so 'above' with a printed lower bound of +0.000 means lo > 0 at the 4th decimal. (ii) Lead sign: positive = alert before the event onset; lead0 counts an undetected event as 0, so a negative mean lead0 (A: alerts on average shortly AFTER onset) reads 'below' C's exact 0 under the rule. (iii) Onset precedence as implemented by `load_labels`: Task-1 prefix onset (E = 1 clips: 53) > Task-2 event onset (25) > anchor-censored (0); 21 E = 1 clips with ongoing_at_anchor = yes carry an explicit onset and are therefore not censored. (iv) Fragility of the manual HT partition (20 events on 11 devices): the single event idx 70052 (manual\|A, w 76.85) holds 0.313 of the event weight (detected by none of A / B / C at val20); the A-C 'above' at val20 rests on the 2 event(s) detected only by A: idx 70115 (w share 0.023), idx 70061 (w share 0.146); the registered MDE at n = 20 is 0.40, larger than every observed \|Δ\| in this regime — the headline is descriptive by the prereg's own rule.

	### 10 / h (val10) — A vs B; C at its floor (cannot reach 10 / h) with the thinning sensitivity

	\| system \| thr \| achieved TEST alerts / h \| n events (HT / all) \| event recall HT [CI] \| recall all-labelled \| lead0 mean s [CI] \| FA / h [CI] \| precision HT [CI] \| note \|
	\|---\|---\|---\|---\|---\|---\|---\|---\|---\|---\|
	\| A \| 0.77799 \| 4.32 \| 20 / 29 \| 0.008 [0.00, 0.05] \| 0.103 \| -0.01 [-0.09, 0.00] \| 1.25 [0.15, 2.74] \| 0.493 [0.00, 0.90] \| \|
	\| B \| 0.62354 \| 3.83 \| 20 / 29 \| 0.064 [0.00, 0.34] \| 0.172 \| -0.03 [-0.14, 0.00] \| 1.05 [0.15, 2.40] \| 0.478 [0.00, 0.89] \| \|
	\| C \| 1.00000 \| 15.10 \| 20 / 29 \| 0.000 [0.00, 0.00] \| 0.034 \| 0.00 [0.00, 0.00] \| 11.52 [2.18, 28.67] \| 0.122 [0.01, 0.53] \| C at floor 15.1/h; thinned to 10/h recall HT 0.000 (keep prob 0.662, 200 replicates) \|

	Silent-time warranted rows / h (S2, HT): manual 191.2 [0.0, 518.4] (= 0.0106 of silent time), engaged 3083.1 [0.0, 7549.8] (= 0.1713); S2 events (E = 1) manual 0, engaged 5.

	### AUROC of the anchor score vs W_i (secondary): unweighted over all labelled clips; HT-weighted on S1 u S2

	\| regime \| system \| n / pos \| AUROC [CI] \| AUROC HT (n) \|
	\|---\|---\|---\|---\|---\|
	\| manual \| A \| 139 / 32 \| 0.781 [0.68, 0.87] \| 0.996 (100) \|
	\| manual \| B \| 139 / 32 \| 0.750 [0.65, 0.85] \| 0.994 (100) \|
	\| manual \| C \| 134 / 27 \| 0.478 [0.33, 0.70] \| 0.984 (95) \|
	\| engaged \| A \| 142 / 48 \| 0.641 [0.54, 0.74] \| 0.668 (99) \|
	\| engaged \| B \| 142 / 48 \| 0.654 [0.55, 0.77] \| 0.494 (99) \|
	\| engaged \| C \| 136 / 43 \| 0.758 [0.69, 0.83] \| 0.732 (94) \|
	\| pooled \| A \| 281 / 80 \| 0.712 [0.63, 0.79] \| 0.869 (199) \|
	\| pooled \| B \| 281 / 80 \| 0.708 [0.63, 0.79] \| 0.799 (199) \|
	\| pooled \| C \| 270 / 70 \| 0.641 [0.52, 0.75] \| 0.750 (189) \|

	## Identifiers

	`uid`, `driver` and `dev_group` are pseudonyms (`driver_N`, `driver_N__route_M__route_a-b`) shared by every file in this bucket and by the public BATON release. Raw device ids, recording leaves, local video paths, rater identities, hosts and timing metadata were removed; `clip_id` is an opaque package id. Where a document mentions the rating panel it uses its code name only.

	## Licence

	Labels, manifest and result tables: CC BY-NC 4.0, inherited from the BATON corpus — see `../../data/LICENSE-DATA`. Attribution: "TriDrive authors (anonymous for double-blind review)".