Buckets:

AsianPlayer
/

TriDrive

Files

xet

AsianPlayer/TriDrive / questionnaire /README.md

AsianPlayer

16 days ago

preview code

download

raw

9.81 kB

# TriDrive in-car user study — questionnaire forms, data, results

This folder is the questionnaire part of the public release of **TriDrive** (development code name *CockpitWM*; the analysis
scripts and `results/SUMMARY.md` still use the development name in places). The paper is under double-blind review and this
release is anonymous: **no participant identities, name tables, audio/video recordings, cabin imagery or raw vehicle logs are
included** — only the anonymised forms (P01–P14), the tables derived from them, the scripts and the figures.

## Design
* Within-participant **paired** comparison with **two conditions**: **C = TriDrive** driver-monitoring prompts (comma four device with
  an external GPU running the TriDrive model) and **S = stock openpilot driver monitoring**. Each participant drove the same route
  twice in their own openpilot-supported car, once with each system, with two visually identical devices; the driving order was
  counterbalanced.
* **14 participants, P01–P14.** Form `forms/driverNN.pdf` is participant `PNN` (driver01 = P01, …, driver14 = P14).
* **Recording convention (experimenter):** on every form, block 1 ("After the first drive", items 1–5) records the **TriDrive**
  condition and block 2 ("After the second drive", items 6–10) records **stock openpilot**, irrespective of the actual driving order.
  Hence D1 = C and D2 = S for every respondent in the tables; "first drive" in END1/END5 means TriDrive. The actual order was not
  written on the forms, so an order effect cannot be assessed (`results/SUMMARY.md` §5.4).
* 12 forms carry AcroForm field values; two forms were completed by hand (circled answers) and were read from the rendered pages:
  `driver06` (P06) and `driver11` (P11) (`source = manual_visual` in `results/questionnaire_wide.csv`). P13's END1 answer was corrected
  from the experimenter's record (`source = acroform+override(END1)`).

## Items (per drive; 7-point Likert, 1 = strongly disagree … 7 = strongly agree, plus "Cannot judge")
| id | form item (D1 / D2) | statement |
|---|---|---|
| APP1 | 1 / 6 | The driver-monitoring prompts during this drive fit the driving situations at the time. |
| APP2 | 2 / 7 | The driver-monitoring prompts during this drive were necessary for me. |
| APP3 | 3 / 8 | Overall, the driver-monitoring prompts during this drive were appropriate. (positively worded) |
| TIM1 | 4 / 9 | The driver-monitoring prompts during this drive came at the right time. |
| ANN1 | 5 / 10 | The driver-monitoring prompts during this drive annoyed me. (higher = more annoying) |
| END1 | 11 | Which drive's driver-monitoring prompts did you prefer? (first drive / second drive / no difference) |
| END5 | 12 | Free comment on the prompts across the two drives (optional; verbatim in the tables). |

## Scoring
* **EP0 appropriateness** = mean(APP1, APP2, APP3) per drive; APP3 is positively worded in the fielded form and is **not**
  reverse-scored; the construct needs at least 2 of the 3 items.
* **ES1 timeliness** = TIM1. **ES2 annoyance** = ANN1, **higher = more annoying**, so a positive difference means TriDrive was rated
  *more* annoying.
* Paired difference per participant **Δ = C − S** (TriDrive minus openpilot).
* **END1** decoded with the convention above: first → prefer C (TriDrive), second → prefer S (openpilot), no difference → none.
* Statistics (`scripts/analysis_questionnaire_v1.py`, `scripts/paired_endpoints.py`): participant-level paired bootstrap of the mean Δ
  (2,000 resamples, seed 20260922, percentile 95 % CI) and a two-sided Wilcoxon signed-rank test (scipy, zero_method "wilcox");
  the verdict (above / below / inconclusive) follows the CI, p is reported beside it. Preference: shares with Wilson 95 % CIs and a
  participant-level bootstrap CI for share(prefer C) − share(prefer S).
## Missing data
"Cannot judge" and unanswered items are missing. **Primary analysis** (`primary_complete_pairs`): per endpoint, only participants with
both the C and the S construct score (n = 14 appropriateness, 12 timeliness, 12 annoyance).
**Sensitivity** (`sensitivity_NP_neutral`): a missing construct score is set to the neutral midpoint 4 (n = 14 for all three); no
verdict changes.
## Headline numbers (`results/endpoints.csv`, `results/preference.csv`)
| endpoint | pairs | mean C (TriDrive) | mean S (openpilot) | mean Δ (C − S) | 95 % CI | Wilcoxon p | verdict |
|---|---|---|---|---|---|---|---|
| EP0 appropriateness | 14 | 5.27 | 3.49 | +1.79 | [+0.86, +2.75] | 0.0076 | above |
| ES1 timeliness | 12 | 5.75 | 3.08 | +2.67 | [+1.33, +4.08] | 0.0078 | above |
| ES2 annoyance | 12 | 2.58 | 1.42 | +1.17 | [+0.33, +2.08] | 0.0312 | above |
Sensitivity NP-neutral (n = 14 each): Δ = +1.79 [0.86, 2.75] (p 0.0076) / +2.50 [1.36, 3.71] (p 0.0048) / +1.07 [0.36, 1.93] (p 0.0278); all verdicts unchanged.
Preference (END1, n = 14): prefer TriDrive 7 (0.50, Wilson 95 % CI [0.27, 0.73]);
prefer openpilot 4 (0.29, [0.12, 0.55]); no difference 3 (0.21, [0.08, 0.48]) —
the share difference is inconclusive at this sample size. Annoyance must be read together with exposure: the stock monitor prompted
rarely and late in these drives, so its low annoyance is not evidence of better calibration (`results/SUMMARY.md` §5.2).
## Files
* `README.md` — this file
* `LICENSE` — CC BY 4.0 notice
* `SHA256SUMS` — sha256 of every file in this folder
* `forms/README.md` — note on the forms (from the supplementary material)
* `forms/questionnaire_blank.pdf` — the form as fielded (12 items, fillable)
* `forms/driver01.pdf` … `forms/driver14.pdf` — the 14 completed forms (P01–P14); 12 with form-field values, driver06 and driver11 hand-filled (circled answers)
* `results/questionnaire.csv` — long table, one row per participant × item (data-schema layout: participant_id, session_index, condition_code, form, item_id, raw_value, scored_value, free_text, …)
* `results/questionnaire_wide.csv` — one row per participant: source, D1.* (TriDrive), D2.* (openpilot), END1, END5
* `results/per_participant.csv` — construct scores C / S / Δ per participant, preference, comment themes, verbatim comment
* `results/endpoints.csv` — the six endpoint rows (primary complete pairs + NP-neutral sensitivity): n, means, Δ, CI, Wilcoxon p, verdict
* `results/endpoints.md` — the same results as markdown, with the per-participant table and the exposure note
* `results/preference.csv` — END1 counts and shares with Wilson 95 % CIs
* `results/SUMMARY.md` — full written summary: design, raw answers, scores, endpoints, analysis, limitations, verbatim comments
* `scripts/analysis_questionnaire_v1.py` — endpoint analysis of the fielded short form (writes endpoints.csv/.md, preference.csv, per_participant.csv)
* `scripts/parse_questionnaire_pdfs.py` — PDF form fields → questionnaire.csv / questionnaire_wide.csv / sessions.csv
* `scripts/paired_endpoints.py` — pre-specified paired analysis helpers (bootstrap, Wilcoxon, verdict rules) reused by the analysis
* `scripts/analysis.py` — full protocol analysis (also covers endpoints that were not fielded in the short form)
* `scripts/make_user_study_fig_paper.py` — figures/user_study_bars.pdf/.png (paper figure, complete pairs)
* `scripts/make_user_study_fig_n14.py` — figures/user_study_bars_n14.pdf/.png (NP-neutral sensitivity, n = 14)
* `scripts/make_user_study_bars.py` — teaser chart from the result tables (figures/user_study_bars_bare.*)
* `scripts/PATHS_CHANGED.md` — the absolute paths rewritten to relative ones in the scripts
* `figures/user_study_bars.pdf` — paper figure: three paired-bar panels (appropriateness, timeliness, annoyance), complete pairs
* `figures/user_study_bars.png` — PNG of the same
* `figures/user_study_bars_n14.pdf` — sensitivity variant, missing scores imputed with 4 (n = 14)
* `figures/user_study_bars_n14.png` — PNG of the same
* `figures/user_study_bars_bare.pdf` — teaser version: appropriateness / timeliness bars and the preference split, minimal text
* `figures/user_study_bars_bare.png` — PNG of the same

## Reproducing the tables and figures
Run from the root of this folder. `results/sessions.csv` is not shipped; it is one row per participant and drive
(`participant_id,session_index,condition_code,condition` with `1,X,C` and `2,Y,S` for every participant) and is written by
`scripts/parse_questionnaire_pdfs.py`.
```
python scripts/parse_questionnaire_pdfs.py --pdf-dir forms --exclude questionnaire_blank --ids <outside>/ids.csv \
       --manual <outside>/manual.json --overrides <outside>/overrides.json --out results
python scripts/analysis_questionnaire_v1.py --questionnaire results/questionnaire.csv --sessions results/sessions.csv --out results
python scripts/make_user_study_fig_paper.py; python scripts/make_user_study_fig_n14.py
python scripts/make_user_study_bars.py --out figures/user_study_bars
```
`--ids` maps form stems to P-ids (driver01 → P01 … driver14 → P14; created automatically in sorted order); `--manual` supplies the
hand-read values of driver06 / driver11 and `--overrides` the P13 END1 correction — both are recorded in `results/questionnaire_wide.csv`
(`source` column), so the shipped tables can be checked line by line against the forms.

## Privacy and provenance
The PDFs keep every answer (form-field values and free text) but carry no participant names, no document metadata other than the
title (= file stem) and no embedded provenance manifests. The free-text comments were given with the instruction not to include
identifying details and are reproduced verbatim. Nothing in this folder links a P-id to a person.

## Licence
Everything in this folder is released under **CC BY 4.0** (see `LICENSE`, https://creativecommons.org/licenses/by/4.0/).

Xet Storage Details

Size:: 9.81 kB
Xet hash:: 6470b706f25ba33486c25ecad6f89d2ba187b98570722d1c1e1fdc7140812f56

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.

	# TriDrive in-car user study — questionnaire forms, data, results

	This folder is the questionnaire part of the public release of TriDrive (development code name CockpitWM; the analysis
	scripts and `results/SUMMARY.md` still use the development name in places). The paper is under double-blind review and this
	release is anonymous: **no participant identities, name tables, audio/video recordings, cabin imagery or raw vehicle logs are
	included** — only the anonymised forms (P01–P14), the tables derived from them, the scripts and the figures.

	## Design
	* Within-participant paired comparison with two conditions: C = TriDrive driver-monitoring prompts (comma four device with
	an external GPU running the TriDrive model) and S = stock openpilot driver monitoring. Each participant drove the same route
	twice in their own openpilot-supported car, once with each system, with two visually identical devices; the driving order was
	counterbalanced.
	* 14 participants, P01–P14. Form `forms/driverNN.pdf` is participant `PNN` (driver01 = P01, …, driver14 = P14).
	* Recording convention (experimenter): on every form, block 1 ("After the first drive", items 1–5) records the TriDrive
	condition and block 2 ("After the second drive", items 6–10) records stock openpilot, irrespective of the actual driving order.
	Hence D1 = C and D2 = S for every respondent in the tables; "first drive" in END1/END5 means TriDrive. The actual order was not
	written on the forms, so an order effect cannot be assessed (`results/SUMMARY.md` §5.4).
	* 12 forms carry AcroForm field values; two forms were completed by hand (circled answers) and were read from the rendered pages:
	`driver06` (P06) and `driver11` (P11) (`source = manual_visual` in `results/questionnaire_wide.csv`). P13's END1 answer was corrected
	from the experimenter's record (`source = acroform+override(END1)`).

	## Items (per drive; 7-point Likert, 1 = strongly disagree … 7 = strongly agree, plus "Cannot judge")
	\| id \| form item (D1 / D2) \| statement \|
	\|---\|---\|---\|
	\| APP1 \| 1 / 6 \| The driver-monitoring prompts during this drive fit the driving situations at the time. \|
	\| APP2 \| 2 / 7 \| The driver-monitoring prompts during this drive were necessary for me. \|
	\| APP3 \| 3 / 8 \| Overall, the driver-monitoring prompts during this drive were appropriate. (positively worded) \|
	\| TIM1 \| 4 / 9 \| The driver-monitoring prompts during this drive came at the right time. \|
	\| ANN1 \| 5 / 10 \| The driver-monitoring prompts during this drive annoyed me. (higher = more annoying) \|
	\| END1 \| 11 \| Which drive's driver-monitoring prompts did you prefer? (first drive / second drive / no difference) \|
	\| END5 \| 12 \| Free comment on the prompts across the two drives (optional; verbatim in the tables). \|

	## Scoring
	* EP0 appropriateness = mean(APP1, APP2, APP3) per drive; APP3 is positively worded in the fielded form and is not
	reverse-scored; the construct needs at least 2 of the 3 items.
	* ES1 timeliness = TIM1. ES2 annoyance = ANN1, higher = more annoying, so a positive difference means TriDrive was rated
	more annoying.
	* Paired difference per participant Δ = C − S (TriDrive minus openpilot).
	* END1 decoded with the convention above: first → prefer C (TriDrive), second → prefer S (openpilot), no difference → none.
	* Statistics (`scripts/analysis_questionnaire_v1.py`, `scripts/paired_endpoints.py`): participant-level paired bootstrap of the mean Δ
	(2,000 resamples, seed 20260922, percentile 95 % CI) and a two-sided Wilcoxon signed-rank test (scipy, zero_method "wilcox");
	the verdict (above / below / inconclusive) follows the CI, p is reported beside it. Preference: shares with Wilson 95 % CIs and a
	participant-level bootstrap CI for share(prefer C) − share(prefer S).

	## Missing data
	"Cannot judge" and unanswered items are missing. Primary analysis (`primary_complete_pairs`): per endpoint, only participants with
	both the C and the S construct score (n = 14 appropriateness, 12 timeliness, 12 annoyance).
	Sensitivity (`sensitivity_NP_neutral`): a missing construct score is set to the neutral midpoint 4 (n = 14 for all three); no
	verdict changes.

	## Headline numbers (`results/endpoints.csv`, `results/preference.csv`)
	\| endpoint \| pairs \| mean C (TriDrive) \| mean S (openpilot) \| mean Δ (C − S) \| 95 % CI \| Wilcoxon p \| verdict \|
	\|---\|---\|---\|---\|---\|---\|---\|---\|
	\| EP0 appropriateness \| 14 \| 5.27 \| 3.49 \| +1.79 \| [+0.86, +2.75] \| 0.0076 \| above \|
	\| ES1 timeliness \| 12 \| 5.75 \| 3.08 \| +2.67 \| [+1.33, +4.08] \| 0.0078 \| above \|
	\| ES2 annoyance \| 12 \| 2.58 \| 1.42 \| +1.17 \| [+0.33, +2.08] \| 0.0312 \| above \|

	Sensitivity NP-neutral (n = 14 each): Δ = +1.79 [0.86, 2.75] (p 0.0076) / +2.50 [1.36, 3.71] (p 0.0048) / +1.07 [0.36, 1.93] (p 0.0278); all verdicts unchanged.

	Preference (END1, n = 14): prefer TriDrive 7 (0.50, Wilson 95 % CI [0.27, 0.73]);
	prefer openpilot 4 (0.29, [0.12, 0.55]); no difference 3 (0.21, [0.08, 0.48]) —
	the share difference is inconclusive at this sample size. Annoyance must be read together with exposure: the stock monitor prompted
	rarely and late in these drives, so its low annoyance is not evidence of better calibration (`results/SUMMARY.md` §5.2).

	## Files
	* `README.md` — this file
	* `LICENSE` — CC BY 4.0 notice
	* `SHA256SUMS` — sha256 of every file in this folder
	* `forms/README.md` — note on the forms (from the supplementary material)
	* `forms/questionnaire_blank.pdf` — the form as fielded (12 items, fillable)
	* `forms/driver01.pdf` … `forms/driver14.pdf` — the 14 completed forms (P01–P14); 12 with form-field values, driver06 and driver11 hand-filled (circled answers)
	* `results/questionnaire.csv` — long table, one row per participant × item (data-schema layout: participant_id, session_index, condition_code, form, item_id, raw_value, scored_value, free_text, …)
	* `results/questionnaire_wide.csv` — one row per participant: source, D1.* (TriDrive), D2.* (openpilot), END1, END5
	* `results/per_participant.csv` — construct scores C / S / Δ per participant, preference, comment themes, verbatim comment
	* `results/endpoints.csv` — the six endpoint rows (primary complete pairs + NP-neutral sensitivity): n, means, Δ, CI, Wilcoxon p, verdict
	* `results/endpoints.md` — the same results as markdown, with the per-participant table and the exposure note
	* `results/preference.csv` — END1 counts and shares with Wilson 95 % CIs
	* `results/SUMMARY.md` — full written summary: design, raw answers, scores, endpoints, analysis, limitations, verbatim comments
	* `scripts/analysis_questionnaire_v1.py` — endpoint analysis of the fielded short form (writes endpoints.csv/.md, preference.csv, per_participant.csv)
	* `scripts/parse_questionnaire_pdfs.py` — PDF form fields → questionnaire.csv / questionnaire_wide.csv / sessions.csv
	* `scripts/paired_endpoints.py` — pre-specified paired analysis helpers (bootstrap, Wilcoxon, verdict rules) reused by the analysis
	* `scripts/analysis.py` — full protocol analysis (also covers endpoints that were not fielded in the short form)
	* `scripts/make_user_study_fig_paper.py` — figures/user_study_bars.pdf/.png (paper figure, complete pairs)
	* `scripts/make_user_study_fig_n14.py` — figures/user_study_bars_n14.pdf/.png (NP-neutral sensitivity, n = 14)
	* `scripts/make_user_study_bars.py` — teaser chart from the result tables (figures/user_study_bars_bare.*)
	* `scripts/PATHS_CHANGED.md` — the absolute paths rewritten to relative ones in the scripts
	* `figures/user_study_bars.pdf` — paper figure: three paired-bar panels (appropriateness, timeliness, annoyance), complete pairs
	* `figures/user_study_bars.png` — PNG of the same
	* `figures/user_study_bars_n14.pdf` — sensitivity variant, missing scores imputed with 4 (n = 14)
	* `figures/user_study_bars_n14.png` — PNG of the same
	* `figures/user_study_bars_bare.pdf` — teaser version: appropriateness / timeliness bars and the preference split, minimal text
	* `figures/user_study_bars_bare.png` — PNG of the same

	## Reproducing the tables and figures
	Run from the root of this folder. `results/sessions.csv` is not shipped; it is one row per participant and drive
	(`participant_id,session_index,condition_code,condition` with `1,X,C` and `2,Y,S` for every participant) and is written by
	`scripts/parse_questionnaire_pdfs.py`.
	```
	python scripts/parse_questionnaire_pdfs.py --pdf-dir forms --exclude questionnaire_blank --ids <outside>/ids.csv \
	--manual <outside>/manual.json --overrides <outside>/overrides.json --out results
	python scripts/analysis_questionnaire_v1.py --questionnaire results/questionnaire.csv --sessions results/sessions.csv --out results
	python scripts/make_user_study_fig_paper.py; python scripts/make_user_study_fig_n14.py
	python scripts/make_user_study_bars.py --out figures/user_study_bars
	```
	`--ids` maps form stems to P-ids (driver01 → P01 … driver14 → P14; created automatically in sorted order); `--manual` supplies the
	hand-read values of driver06 / driver11 and `--overrides` the P13 END1 correction — both are recorded in `results/questionnaire_wide.csv`
	(`source` column), so the shipped tables can be checked line by line against the forms.

	## Privacy and provenance
	The PDFs keep every answer (form-field values and free text) but carry no participant names, no document metadata other than the
	title (= file stem) and no embedded provenance manifests. The free-text comments were given with the instruction not to include
	identifying details and are reproduced verbatim. Nothing in this folder links a P-id to a person.

	## Licence
	Everything in this folder is released under CC BY 4.0 (see `LICENSE`, https://creativecommons.org/licenses/by/4.0/).