AsianPlayer/TriDrive/benchmark/human_eval_299 - Storage Bucket

TriDrive — pre-registered human evaluation of the warning probe on a new test set (benchmark/human_eval_299/)

A model-targeted human evaluation of the TriDrive current-state warning probe (development name CockpitWM, system A) against a current-state-only variant (B) and the openpilot-derived driver-monitoring comparator (C) on 299 labelled cabin clips drawn from the TEST exposure of the BATON corpus. The design, estimators, wording rules and minimum detectable effects were frozen in newtest_prereg.md (byte-identical copy; sha256 in prereg_hash.json) before any clip was rendered; every later change is a dated row of newtest_prereg_deviations.md. Paper: Appendix E.

Files

file	content
manifest_v1.csv	the frozen clip sample (320 rows: 160 S1 + 40 S2 + 100 S3 + 20 optional S3_hard) with design weights and the frozen alert decisions of A / B / C; labelled_299 marks the 299 clips that received labels
labels_newtest.jsonl	the committee labels: 897 records = 299 clips x 3 stages (prefix, outcome, event); one JSON object per line
thresholds_frozen_v1.json	frozen systems, operating points (val10 / val20 / test10 / test20) and thresholds
deviations_v1.json	pre-analysis exclusions (one S1 clip without road video) and the post-hoc systems block
inputs_sha256_newtest_sample.txt	sha256 of every input of the sampling script (paths rewritten to the /home/USER/... placeholder)
newtest_prereg.md, prereg_hash.json	the pre-registration (byte-identical) and its registered hash
	prereg_hash.json also records the sha256 of the original manifest_v1.csv and thresholds_frozen_v1.json; the shipped copies are pseudonymised / scrubbed, so only newtest_prereg.md matches byte-for-byte
newtest_prereg_deviations.md	dated deviation log
tables/headline_manual_valfrozen.md	headline table of the final run (manual regime, val-frozen thresholds) plus the engaged regime, test-achieved points, calibration in / out sensitivity, AUROC and achieved MDE
tables/full_results.md, tables/all_results.json	index of the full result set and the machine-readable results (main / paired / AUROC / cells / calibration-out / post-hoc / coverage / consensus summary)
tables/analysis_primary/	per-clip table (clips.csv, whitelisted like the manifest + the analysis columns), main.csv, paired.csv, auroc.csv, cells_enriched.csv, run record
counts.json	every count measured while building this folder, the assertions and the scrub report

Only tables/ and tables/analysis_primary/ of the archived run are included; full_results.md is the run's own index and also names intermediate files (coverage, consensus, post-hoc, calibration-out CSVs) that are summarised inside all_results.json but not shipped as separate files.

Set definition (prereg Section 3)

• Frame. The TEST exposure of the 5 Hz analysis grid (manual 28.50 h / engaged 25.10 h), excluding everything within 12.5 s of a clean-test gold anchor or of an earlier annotation round.
• S1 — alert union (160 clips, 80 per regime). Events formed from the frame-threshold alerts of A, B and C (10 s same-event guard). cell = regime | pattern of systems firing within +-10 s (A, B, C, A+B, A+C, B+C, A+B+C); discordant cells were over-sampled (per regime A 14, B 14, C 14, A+B 12, A+C 10, B+C 10, A+B+C 6); device cap 10; strata = (cell, device group); pi = inclusion probability; N_covered_cell / coverage_cell = events reachable under the cap.
• S2 — silent background (40 clips, 20 per regime). Moving grid rows (> 2 m/s) at least 12.5 s from every union event; strata (regime, device), cap 6; pi = n / N rows. S2 estimates the miss population (warranted moments per silent hour).
• S3 — model-independent candidates (100 clips: the round-2 TEST clips, cells rare_road / demand_noresp / demand_resp / hand_head / occlusion_night / background) and S3_hard (20 optional hard_response clips). S3 enters paired event-level analyses and AUROC only, never a Horvitz-Thompson natural rate.
• Phases. Phase 1 = S3 + the first 60 S1 clips + the S2 calibration clips; Phase 2 = the rest (phase 1: 160, 2: 160 clips). calibration = True marks the 8 calibration clips rated by every rater before Phase 1 (the analysis reports with / without them); optional = True marks S3_hard.
• Labelled set. 299 clips: S1 159 (one S1 clip, idx 70074, excluded before analysis — no road video, deviations_v1.json), S2 40, S3 100; the 20 S3_hard clips were not labelled. Regimes in the manifest: engaged 169 / manual 151.
• Systems. A = full predictive model + warning head (score W_learned_bothhgb_t15_w10_n979, mean over 3 frozen seeds); B = strongest current-state-only head with the same inputs and supervision; C = openpilot driver-monitoring chain att@3s (deployed comparator). Alert = score >= threshold with a 60 s per-route cooldown; thresholds fitted on the VAL layouts to 10 / h and 20 / h (primary) or on the TEST layouts (secondary). C saturates at 1.0 and cannot reach 10 / h (reported at its floor).

manifest_v1.csv columns

column	meaning
qpos	position in the rating queue
idx	clip id (= the idx of labels_newtest.jsonl); S1 / S2 clips carry ids >= 70000 / 80000 minted by the sampler, S3 / S3_hard clips reuse the id of their source episode
s15_idx	id of the source episode in the project's episode store (S3 / S3_hard clips only, empty otherwise); none of these episodes is part of ../labels/
clip_id	opaque 13-hex id of the rendered clip package
stratum, phase, calibration, optional	design membership (see above)
regime, layout_regime	regime of the anchor row and of the frozen score layout it was read from
uid, driver, dev_group	pseudonymised route-segment group driver_N__route_M__route_a-b, pseudonymised device driver_N, and the device group of the S1 / S2 stratum (driver_N, _pooled, or empty for S3)
t_anchor	anchor time, route-relative seconds
anchor_s	position of the anchor inside the rendered clip (s)
cell, pattern, pattern_cluster	S1 cell, firing pattern within +-10 s, pattern of the event's own cluster
pi, N_cell, n_cell, N_covered_cell, coverage_cell, N_stratum, n_stratum	inclusion probability and the design counts behind it
s3_stratum, s3_draw_rank	S3 mining stratum and draw rank
speed_anchor_mps	vehicle speed at the anchor (m/s)
fired_X_<point>	1 if system X (A / B / C) alerts within +-10 s of the anchor at the frozen threshold of operating point val10 / val20 / test10 / test20
t_alert_X_<point>, t_alert_X_frame	route time of that alert (point and frame threshold)
score_X_anchor, thr_X_frame	score of system X at the anchor row and the frame threshold
labelled_299	True for the 299 clips with labels

labels_newtest.jsonl records

Each clip has three records: stage = prefix (Task 1, Questions 1-5, judged with the player hard-limited to the anchor), stage = outcome (the 'what happened' half of Question 5, judged after the prefix record was locked) and stage = event (Task 2, Question 6 on the 10 s after the anchor, frame t frozen for 1 s). Labels are the committee consensus (protocol = committee_consensus, deviation row of 2026-09-13); rater identities, timing and client metadata were removed.

Record keys: idx, clip_id, stage, task, anchor_s (anchor position in the clip, s), visible_until_s (last clip second the rater could see: the anchor for the prefix stage, anchor + 10 s otherwise), split, stratum, cell, pattern, pi, regime, phase, calibration, mode_at_anchor (human / oem_acc / openpilot), schema_ok, problems (schema checks that failed), prefill (which outcome / event fields were prefilled from the vehicle log and accepted), fields (the answers), valid and unknown (per-field flags), uid, driver, t0 (= t_anchor of the manifest, route-relative s), protocol, schema_version. Times inside fields are clip-relative seconds (route time = t0 + value - anchor_s).

fields of stage prefix

field	values	meaning
view_region	road_ahead / left_mirror_or_window / right_mirror_or_window / rearview_mirror / instrument_cluster / center_console / lap_or_phone_area / passenger / unknown	where the driver is looking at the anchor (coarse region)
head_motion	steady / brief_glance / sustained_turn / unknown	head motion over the window up to the anchor
left_hand	on_wheel / off_wheel_resting / off_wheel_holding_object / not_visible / unknown	left-hand placement at the anchor
right_hand	on_wheel / off_wheel_resting / off_wheel_holding_object / not_visible / unknown	right-hand placement at the anchor
object_interaction	none / phone / food_or_drink / vehicle_control / other_object / unknown	object the driver is handling
visibility	good / partial / insufficient	how well the cabin view supports the judgement
task_families	list of: longitudinal_following / lateral_maneuver_or_check / sustained_supervision / cabin_interaction_demand / none / unknown	road-task families in force (multi-select)
demand_level	low / moderate / high / unknown	road demand at the anchor
notable_event	none / lead_vehicle_braking / cut_in / intersection_or_merge / lane_change_by_ego / pedestrian_or_cyclist / curve / other / unknown	notable road event up to the anchor
compatibility	compatible / briefly_incompatible / sustained_incompatible / unknown	behaviour-situation compatibility (Question 3)
concern_persists	no / brief / ongoing / unknown	whether the concern still holds at the anchor
prompt_warranted_now	not_warranted / warranted_attention / warranted_hands / unsure	Question 4 - is a short uniform reminder warranted at the anchor; W = 1 for warranted_attention / warranted_hands, 0 for not_warranted (unsure -> adjudication)
confidence	low / medium / high	rater confidence in Question 4
reason_codes	list of: normal_mirror_or_instrument_check / brief_glance_low_demand / sustained_gaze_away_with_demand / hands_off_with_demand / object_interaction_while_moving / stopped_or_creeping / automation_handling_it / insufficient_visibility / insufficient_evidence / other	reason codes behind Question 4 (multi-select)
evidence_marks_s	list of numbers (s)	clip-relative times (s, <= anchor) the rater marked as evidence
onset_interval_s	null or array (s)	[start, end] clip-relative interval (s) in which the incompatible behaviour began; required when not compatible; null otherwise

fields of stage outcome

field	values	meaning
first_gaze_to_road_s	null or number (s)	clip-relative time (s) of the first gaze return to the road after the anchor; null = not within the clip
first_hand_to_wheel_s	null or number (s)	clip-relative time (s) at which both hands are back on the wheel; null = not within the clip
vehicle_response_kind	none / braking / acceleration / steering_correction / lane_change / stop / unknown	vehicle response within the 10 s after the anchor (prefilled from the vehicle log, confirmed by the rater)
vehicle_response_attributed_to	driver / automation / unclear / none	who produced the vehicle response (driver / automation / none / unclear)
scene_resolution	demand_resolved / demand_persists / demand_escalated / unknown	how the road situation resolved after the anchor

Derived labels used by the analysis: W = 1 if prompt_warranted_now in {warranted_attention, warranted_hands}, 0 if not_warranted (measured over the 299 prefix records: 82 warranted [65 attention / 17 hands], 215 not warranted, 2 unsure); E = 1 if the event-stage event_type is not ['none'] (measured: 78 events — S1 48, S2 5, S3 25); the event onset follows the prereg precedence Task-1 onset_interval_s > Task-2 event_onset_s > anchor (censored).

Headline numbers (copied from tables/headline_manual_valfrozen.md)

Manual regime, val-frozen 20 / h (the paper's headline point, prereg Section 6): event recall on the Horvitz-Thompson partition A 0.233 [0.02, 0.54], B 0.087 [0.00, 0.42], C 0.000 [0.00, 0.00] (20 HT / 29 labelled events); false alarms per hour A 4.03 [1.23, 8.04], B 8.63 [1.41, 21.83], C 12.41 [2.96, 29.31]; precision A 0.513 [0.05, 0.85], B 0.288 [0.08, 0.78], C 0.175 [0.02, 0.55]. Paired A-C recall delta +0.233 [0.022, 0.543] reads 'above' under the registered wording rule, but the registered MDE at n = 20 is 0.40, larger than every observed |delta| in this regime: every per-regime row is descriptive by the prereg's own rule (fewer than 60 labelled events per regime). Engaged regime, val10: A 0.409 [0.00, 0.93] recall HT. AUROC of the anchor score against W over all labelled clips: manual A 0.781 [0.68, 0.87], B 0.750 [0.65, 0.85], C 0.478 [0.33, 0.70]; engaged A 0.641 [0.54, 0.74], B 0.654 [0.55, 0.77], C 0.758 [0.69, 0.83]. Device-cluster bootstrap, 2,000 resamples.

Verbatim sections of the headline table:

20 / h three-way (val20; C at 17.7 / h achieved)

system	thr	achieved TEST alerts / h	n events (HT / all)	event recall HT [CI]	recall all-labelled	lead0 mean s [CI]	lead median detected s	FA / h [CI]	precision HT [CI]	silent warranted rows / h [CI]
A	0.67578	9.40	20 / 29	0.233 [0.02, 0.54]	0.276	-0.14 [-0.45, -0.02]	-0.62	4.03 [1.23, 8.04]	0.513 [0.05, 0.85]	191.2 [0.0, 518.4]
B	0.49316	13.69	20 / 29	0.087 [0.00, 0.42]	0.207	-0.04 [-0.21, 0.00]	-0.40	8.63 [1.41, 21.83]	0.288 [0.08, 0.78]	191.2 [0.0, 518.4]
C	1.00000	17.67	20 / 29	0.000 [0.00, 0.00]	0.034	0.00 [0.00, 0.00]	—	12.41 [2.96, 29.31]	0.175 [0.02, 0.55]	191.2 [0.0, 518.4]

S1 per-cell coverage in force (prereg §6: beside the headline; N covered = union events with pi > 0 under the device cap after the pre-analysis exclusion 70074, engaged|A+B 132 -> 128):

cell	N	N covered	coverage	n labelled	W = 1	E = 1
engaged	A	199	187	0.940	14	7
engaged	A+B	150	128	0.853	11	5
engaged	A+B+C	71	42	0.592	6	3
engaged	A+C	46	32	0.696	10	4
engaged	B	175	165	0.943	14	4
engaged	B+C	82	66	0.805	10	5
engaged	C	361	291	0.806	14	4
manual	A	212	160	0.755	14	4
manual	A+B	169	119	0.704	12	6
manual	A+B+C	74	43	0.581	6	2
manual	A+C	56	33	0.589	10	3
manual	B	216	179	0.829	14	5
manual	B+C	62	44	0.710	10	3
manual	C	315	277	0.879	14	1

Exposure H used for every /h figure: manual 28.4963 h (512,934 test rows x 0.2 s), engaged 25.1022 h (prereg §5 prints the rounded 28.50 / 25.10 h; difference 0.013 %).

comparison (manual)	point	n events	Δ recall [CI]	MDE recall (registered, disc 0.4)	MDE recall (observed disc)	Δ lead0 s [CI]	MDE lead s	wording (rule verbatim)
A-C	val20	20	+0.233 [0.022, 0.543]	0.40	0.34	-0.139 [-0.452, -0.019]	1.88	recall above (Δ +0.233, CI [+0.022, +0.543] recall); lead below (Δ -0.139 s, CI [-0.452, -0.019] lead) (descriptive)
B-C	val20	20	+0.087 [0.000, 0.422]	0.40	0.31	-0.041 [-0.205, 0.000]	1.88	recall matches (Δ +0.087, CI [+0.000, +0.422] recall); lead matches (Δ -0.041 s, CI [-0.205, +0.000] lead) (descriptive)
A-B	val20	20	+0.146 [-0.106, 0.413]	0.40	0.24	-0.097 [-0.331, 0.041]	1.88	recall matches (Δ +0.146, CI [-0.106, +0.413] recall); lead matches (Δ -0.097 s, CI [-0.331, +0.041] lead) (descriptive)
A-B	val10	20	-0.056 [-0.302, 0.000]	0.40	0.24	+0.013 [-0.013, 0.087]	1.88	recall matches (Δ -0.056, CI [-0.302, +0.000] recall); lead matches (Δ +0.013 s, CI [-0.013, +0.087] lead) (descriptive)

Reading notes (from clips.csv of this run): (i) a CI bound printed as 0.000 is a degenerate point mass at Δ = 0 in the device-cluster resamples; the wording rule uses the exact bound (lo ≤ 0 ≤ hi -> 'matches'; lo > 0 -> 'above'), so 'above' with a printed lower bound of +0.000 means lo > 0 at the 4th decimal. (ii) Lead sign: positive = alert before the event onset; lead0 counts an undetected event as 0, so a negative mean lead0 (A: alerts on average shortly AFTER onset) reads 'below' C's exact 0 under the rule. (iii) Onset precedence as implemented by load_labels: Task-1 prefix onset (E = 1 clips: 53) > Task-2 event onset (25) > anchor-censored (0); 21 E = 1 clips with ongoing_at_anchor = yes carry an explicit onset and are therefore not censored. (iv) Fragility of the manual HT partition (20 events on 11 devices): the single event idx 70052 (manual|A, w 76.85) holds 0.313 of the event weight (detected by none of A / B / C at val20); the A-C 'above' at val20 rests on the 2 event(s) detected only by A: idx 70115 (w share 0.023), idx 70061 (w share 0.146); the registered MDE at n = 20 is 0.40, larger than every observed |Δ| in this regime — the headline is descriptive by the prereg's own rule.

10 / h (val10) — A vs B; C at its floor (cannot reach 10 / h) with the thinning sensitivity

system	thr	achieved TEST alerts / h	n events (HT / all)	event recall HT [CI]	recall all-labelled	lead0 mean s [CI]	FA / h [CI]	precision HT [CI]	note
A	0.77799	4.32	20 / 29	0.008 [0.00, 0.05]	0.103	-0.01 [-0.09, 0.00]	1.25 [0.15, 2.74]	0.493 [0.00, 0.90]
B	0.62354	3.83	20 / 29	0.064 [0.00, 0.34]	0.172	-0.03 [-0.14, 0.00]	1.05 [0.15, 2.40]	0.478 [0.00, 0.89]
C	1.00000	15.10	20 / 29	0.000 [0.00, 0.00]	0.034	0.00 [0.00, 0.00]	11.52 [2.18, 28.67]	0.122 [0.01, 0.53]	C at floor 15.1/h; thinned to 10/h recall HT 0.000 (keep prob 0.662, 200 replicates)

Silent-time warranted rows / h (S2, HT): manual 191.2 [0.0, 518.4] (= 0.0106 of silent time), engaged 3083.1 [0.0, 7549.8] (= 0.1713); S2 events (E = 1) manual 0, engaged 5.

AUROC of the anchor score vs W_i (secondary): unweighted over all labelled clips; HT-weighted on S1 u S2

regime	system	n / pos	AUROC [CI]	AUROC HT (n)
manual	A	139 / 32	0.781 [0.68, 0.87]	0.996 (100)
manual	B	139 / 32	0.750 [0.65, 0.85]	0.994 (100)
manual	C	134 / 27	0.478 [0.33, 0.70]	0.984 (95)
engaged	A	142 / 48	0.641 [0.54, 0.74]	0.668 (99)
engaged	B	142 / 48	0.654 [0.55, 0.77]	0.494 (99)
engaged	C	136 / 43	0.758 [0.69, 0.83]	0.732 (94)
pooled	A	281 / 80	0.712 [0.63, 0.79]	0.869 (199)
pooled	B	281 / 80	0.708 [0.63, 0.79]	0.799 (199)
pooled	C	270 / 70	0.641 [0.52, 0.75]	0.750 (189)

Identifiers

uid, driver and dev_group are pseudonyms (driver_N, driver_N__route_M__route_a-b) shared by every file in this bucket and by the public BATON release. Raw device ids, recording leaves, local video paths, rater identities, hosts and timing metadata were removed; clip_id is an opaque package id. Where a document mentions the rating panel it uses its code name only.

Licence

Labels, manifest and result tables: CC BY-NC 4.0, inherited from the BATON corpus — see ../../data/LICENSE-DATA. Attribution: "TriDrive authors (anonymous for double-blind review)".