questionnaire/results/endpoints.md

Questionnaire analysis v1 (fielded short form) — 14 participants with data

Inputs: data/questionnaire.csv, data/sessions.csv; bootstrap 2000 resamples, seed 20260922; Wilcoxon two-sided (scipy). Condition rule (owner, 2026-09-23): session 1 = CockpitWM (C), session 2 = stock openpilot (S) for every respondent. Differences are C − S.

Endpoints

analysis	endpoint	n pairs	mean C	mean S	mean Δ (C−S)	median Δ	95 % CI	Wilcoxon p	verdict
primary_complete_pairs	EP0_appropriateness	14	5.27	3.49	1.79	1.25	[0.86, 2.75]	0.0076	above
primary_complete_pairs	ES1_timeliness	12	5.75	3.08	2.67	2.00	[1.33, 4.08]	0.0078	above
primary_complete_pairs	ES2_annoyance	12	2.58	1.42	1.17	0.50	[0.33, 2.08]	0.0312	above
sensitivity_NP_neutral	EP0_appropriateness	14	5.27	3.49	1.79	1.25	[0.86, 2.75]	0.0076	above
sensitivity_NP_neutral	ES1_timeliness	14	5.64	3.14	2.50	2.00	[1.36, 3.71]	0.0048	above
sensitivity_NP_neutral	ES2_annoyance	14	2.86	1.79	1.07	0.50	[0.36, 1.93]	0.0278	above

• EP0 = mean(APP1, APP2, APP3), APP3 positively worded in the fielded form → not reverse-scored (deviation from protocol §9.1).
• ES2 annoyance: higher = more annoying, so a positive Δ means C was rated MORE annoying.
• primary_complete_pairs: participants with both constructs present (cannot_judge / unanswered = missing; APP needs ≥ 2 of 3 items). sensitivity_NP_neutral: missing construct scores set to 4.

Preference (END1, decoded: 'first drive' = C)

choice	k	n	share	Wilson 95 % CI
prefer C	7	14	0.50	[0.27, 0.73]
prefer S	4	14	0.29	[0.12, 0.55]
no difference	3	14	0.21	[0.08, 0.48]

share(prefer C) − share(prefer S) = 0.21, bootstrap 95 % CI [-0.21, 0.64] → inconclusive (n 14)

Per participant (primary scoring)

pid	APP C	APP S	Δ	TIM C	TIM S	Δ	ANN C	ANN S	Δ	prefers	comment themes
P01	4.00	2.50	1.50	7	1	6	6	n/a	n/a	S	C frequent / loud / distracting; S only on hands-off / insensitive
P02	6.33	4.00	2.33	7	1	6	2	2	0	C	C context-aware (pedestrians, traffic); S only on hands-off / insensitive; wants traffic-dependent prompting
P03	4.00	1.67	2.33	4	2	2	2	1	1	C	C context-aware (pedestrians, traffic); S only on hands-off / insensitive
P04	6.33	1.67	4.67	6	n/a	n/a	3	n/a	n/a	C	C context-aware (pedestrians, traffic)
P05	6.00	5.00	1.00	7	2	5	4	4	0	C	C context-aware (pedestrians, traffic); S only on hands-off / insensitive; wants traffic-dependent prompting
P06	4.33	3.67	0.67	6	4	2	5	1	4	S	C frequent / loud / distracting; C context-aware (pedestrians, traffic); S only on hands-off / insensitive
P07	4.00	4.00	0.00	4	4	0	1	1	0	none	(no comment)
P08	3.33	3.00	0.33	4	3	1	5	1	4	S	C frequent / loud / distracting
P09	6.67	1.67	5.00	6	1	5	1	1	0	C	C frequent / loud / distracting; C context-aware (pedestrians, traffic)
P10	5.67	1.67	4.00	6	1	5	2	1	1	C	(no comment)
P11	7.00	7.00	0.00	7	7	0	1	1	0	none	(no comment)
P12	3.67	3.67	0.00	5	5	0	2	1	1	S	C frequent / loud / distracting; wants traffic-dependent prompting
P13	6.00	6.67	-0.67	6	6	0	1	1	0	none	(no comment)
P14	6.50	2.67	3.83	n/a	3	n/a	5	2	3	C	wants traffic-dependent prompting

Interpretation note: annoyance must be read together with exposure

The stock openpilot monitor (S) prompts only while engaged and reacts to a narrow cue set (hands off the wheel, sustained gaze away); in these drives it prompted rarely and, per the respondents, late or not at all. Its low annoyance score is therefore largely a consequence of issuing few prompts, not evidence of better-calibrated prompting. Supporting numbers from this sample:

• S timeliness (TIM1) rated <= 3 by 8 of 13 respondents who could judge it; S annoyance at the floor value 1 for 9 of 12 (C: 4 of 14).
• 4 of 12 respondents gave S BOTH timeliness <= 3 and annoyance = 1: the un-annoying S sessions are the ones where S did not prompt when the driver expected it.
• Free text: 5 respondents describe S as prompting only on hands-off or missing / late prompts.
• Consequence for the endpoints: EP0 (appropriateness) and ES1 (timeliness) favour C with CIs excluding 0; ES2 (annoyance) favours S but is confounded with exposure. The fair comparison of intrusiveness is annoyance conditional on prompting (per-alert ratings, ES8) and prompts per hour (ES7) from the device logs; both are pending. Until then ES2 is to be reported as 'C prompted more and was rated more intrusive; S prompted rarely', not as 'S was better calibrated'.
• Design implication for CockpitWM: the intrusiveness complaints concern prompt frequency and sound (60 s cooldown re-fires while the read-out stays above threshold; loud tone), not the situations chosen. An onset-persistence gate and a softer first-level tone address them without touching the model or thresholds.

Free-text comments (END5, verbatim, P-ids only)

• P01: While the second drive's alert system did not alert me when it should have, the constant alerts from the first system prompted me to keep looking away from the road and to the alert screen whenever it beeped (which was frequent).
• P02: Quick and accurate for the first one. Second one may not alert when there's some danger behavior.
• P03: At least the first one is accurate. Second is not accurate and came at the right time.
• P04: The first system seems very intelligent as it can provide context-aware alerts, seemingly able to distinguish the traffic context such as crossing pedestrian, stopping cars. Sometimes the alerts are unnecessary, for example, one-hand driving when the driver feels comfortable and safe.
• P05: The second always alert too late, which is bad for drivers. Also, I may get alert when I play phone for a long time, wherever I am and whatever situation I'm in. I hope it alerts me when the traffic becoma heavy but keep silent when traffic is light and the road is straight.
• P06: I like having a driver monitor to assist me, but the first one was a bit distracting. It gave alerts, but I wasn't sure what I did wrong. The second one alerted me when my hands were off the steering wheel, but didn't warn me about pedestrians or other hazards. It felt less sensitive, maybe a little too insensitive.
• P08: The first one's sound is a litttle bit loud.
• P09: The first one is accurate. It alerts frequently when I act like disattractive and most is manageable. I'm ok with the sound.
• P12: The sound of first one is a little annoying. A little sound is fine. The sound should be equal to danger. Don't mind alert in the UI.
• P14: Alert prompts is not very good. I prefer symbol. I won't have enough time to read all the words maybe I'm not a native speaker. Also, it can use different color to represent differnt level of danger.