TriDrive

Joint Driver, Vehicle, and Road Modeling for Forecasting and Driver Monitoring

Yuhang Wang¹, Jingxin Yang², Chuheng Wei³, Yuechen Guo¹, Jinghan Xu⁴, Zhao Han¹, Hao Zhou¹

¹University of South Florida · ²NVIDIA · ³Purdue University · ⁴Hunan University

TriDrive is a unified framework that jointly models the driver, the vehicle, and the road and puts that joint model to work in two applications: multi-step forecasting of driver kinematics, vehicle dynamics, and road demands, and real-time driver monitoring with a lightweight warning probe that ran on a comma four during a paired on-road user study.

Three modality-specific predictors (an anchored kinematic representation of the driver from a frozen pose estimator, causal CAN-bus dynamics, and frozen V-JEPA 2 road latents with structured road margins) are connected by directed residual connections through which driver and road context refine vehicle forecasts. The model decodes 25 future steps (0.2 s each) of transition hazards, continuous vehicle dynamics, and future road latents directly from the encoded history, conditioned on the automation state.

Result	Where
AIDE keypoint forecasting: 48.05 All-MPJPE with BATON pose pretraining (published baselines: 71.47)	Table 1, Appendix C
BATON (197.2 h, 162 drivers): directed connections + road margins raise assistance-engaged PR-AUC by 0.084 (steering onset) and 0.286 (time-to-collision drops)	Table 3, Appendix B
Warning probe above an openpilot-based baseline on human-labeled manual-driving warnings, AUROC 0.725 vs 0.563	Section 4.4, Appendix D
On-vehicle: warning probe at 5 Hz, 177 ms p95 driver-frame-to-score latency, joint model forecasting concurrently on a comma four + 8 GB eGPU	Appendix G, TriDrive-Comma
Paired on-road study, 14 drivers: warnings rated more appropriate (+1.79) and timely (+2.67) than openpilot's driver-monitoring system	Section 4.4, Appendix H

Architecture: three pretrained branch predictors, directed residual connections into the vehicle state, direct 25-step decoding.

---

What is in this repository

Path	Contents
code/	Training and evaluation code. code/aide/ = AIDE keypoint forecasting (data protocol, our forecaster, ST-GCN / SiMLPe / MotionBERT / Driver-WM re-instantiations, evaluation harness); code/models/, code/train/, code/eval/ = BATON branch predictors, joint model, paired driver-bootstrap metrics; code/baselines_dms/ = openpilot-based warning baseline; code/paper/ = table generation
checkpoints/aide/	Every AIDE run reported in the paper and its appendix: our forecaster (5 seeds, with and without BATON pose pretraining), all ablations, the baseline re-instantiations and the released-weights MotionBERT fine-tunes, and our reproduction of the published Driver-WM pipeline. Each run directory holds ckpt_best.pt, the run manifest (full command line and hyper-parameters), training history, and the exact validation/test metrics quoted in the paper
checkpoints/baton/	BATON branch predictors (U driver GRU, V vehicle TCN, E road Transformer), the joint models of Table 3 (directed default, all-to-all, independent, single-link variants, with and without forecast features), the compact keypoint forecaster, the warning probe, and the latent-state probe experiment
checkpoints/deployed/	Exactly what ran on the vehicle: ONNX exports of the joint model and the keypoint forecaster, the distilled road student (v5), standardization statistics and PCA projections, and the fitted warning probe with its 77 feature names and validation thresholds. See checkpoints/README.md for the mapping to paper rows
tables/	Machine-readable result tables (.md / .json) that the paper's tables were generated from
paper_tables_iclr/	LaTeX tables as they appear in the manuscript, with the evidence map that ties each number to a results file
paper/	The preprint PDF
video/demo.mp4	Two-minute narrated demo: framework, on-vehicle setup, a live warning, the user study
questionnaire/	The bilingual questionnaire fielded in the paired on-road study
DATA_INVENTORY.md	Datasets, splits, label counts, on-road and user-study data, third-party components
docs/	Development notes and decision ledgers kept during the project (verbose; provenance only)

Naming. The model was developed under the working name CockpitWM; run directories, identifiers, and result tables keep that name (cwm3d_* = the AIDE forecaster, I1_* = BATON joint models). CockpitWM in the code is the same model as TriDrive in the paper.

Checkpoints referenced in the paper

Run directory	Paper row
checkpoints/aide/cwm3d_kp_s{42,123,456,789,1011}_v2	Ours (5.77 M), no pretraining: 48.60 ± .12 All-MPJPE
checkpoints/aide/cwm3d_kp_batonpre_trunkp123_s{…}_v2	Ours + BATON pose pretraining: 48.05 ± .06
checkpoints/aide/cwm3d_kp_noema_s{…}_v2, cwm3d_kp_abs_noema_s{…}_v2	EMA control for the anchoring ablation (Appendix C)
checkpoints/aide/mb_official_res_s{42,123,456}_v2	MotionBERT, released weights fine-tuned: 48.71 ± .26
checkpoints/aide/motionbert_kp_s{…}_v2, stgcn_kp_*, simlpe_kp_*	Baseline architectures adapted to our tokens
checkpoints/aide/driverwm_repro_s{42,123,456}_v2	Our reproduction of the published Driver-WM pipeline (123.7 M)
checkpoints/aide/cwm3d_vis_sema_vge_s{…}_v2	View-gate semantic variant (Table 2 / Appendix C)
checkpoints/baton/I1_all_minus_road2drv_s{0,1,2}_final_ecan	Directed joint model, ECAN inputs (Table 3 A–C default column)
checkpoints/baton/I1_all_minus_road2drv_s{0,1,2}_final_ecan_kpf	Default joint model with frozen forecast features (Table 3 D)
checkpoints/baton/I1_indep_*, I1_drv2veh_*, I1_env2veh_*, I1_naive_*	Independent branches, single links, all-to-all and other coupling variants
checkpoints/baton/U0_*, V0_*, E0_*	Pretrained driver, vehicle, and road branches
checkpoints/baton/KPF_mb_full_s{0,1,2}	Compact keypoint forecaster (0.85 M); s0 is deployed
checkpoints/baton/warning_head/	Current-state warning probe (histogram gradient boosting, 77 features)
checkpoints/deployed/	On-vehicle ONNX graphs, statistics, PCA projections, fitted probe and thresholds

Quick start

# Python 3.12; PyTorch, onnxruntime, scikit-learn, numpy, huggingface_hub
pip install huggingface_hub
hf download HenryYHW/TriDrive --include "checkpoints/deployed/*" --local-dir tridrive

Run the deployed joint model on one anchor with ONNX Runtime:

import numpy as np, onnxruntime as ort
sess = ort.InferenceSession("tridrive/checkpoints/deployed/onnx/joint_final_ecan_kpf_s0.onnx",
                            providers=["CPUExecutionProvider"])
for i in sess.get_inputs():
    print(i.name, i.shape, i.type)   # X (vehicle history), Kx (driver kinematics), kpf (forecast feats),
                                     # tok / tok_ok / tok_dt (road tokens), vf / vf_ok (road side vector), mode
outs = sess.run(None, {...})         # V: 25-step vehicle hazards and dynamics, plus driver and road heads

Reproducing the tables:

• AIDE (Table 1, Appendix C). code/aide/train/train_kp.py trains one arm per seed; code/aide/eval/ computes All-MPJPE, PCK@0.05, and the high-motion quantile sweeps. Set AIDE_ROOT to your AIDE copy (obtained from the AIDE authors).
• BATON (Table 3, Appendix B). code/train/ trains the branch predictors and the joint model; code/eval/ produces the onset PR-AUC tables with paired driver-cluster bootstrap intervals. BATON is obtained from its own release; paths are set in code/common/.
• Warning probe (Section 4.4, Appendix D). code/eval/ and code/baselines_dms/ evaluate the current-state probe and the openpilot-based baseline on the human-labeled anchor set.
• On-vehicle system (Appendix G). See TriDrive-Comma.

Paths inside the scripts point at a project root written as /home/USER/...; edit code/common/ once. Training-queue shell scripts reference a remote data host (USER@REMOTE_HOST) that is not part of this release.

Data

No naturalistic video, CAN logs, pose data, or human warning labels are redistributed here. AIDE comes from its authors; BATON from its release. The 8.4-hour dual-device recordings of the on-road study will be released after de-identification and consent review. DATA_INVENTORY.md lists every dataset, split, and label count used in the paper.

Third-party components

Downloaded from their original sources and subject to their own licences; not redistributed here except as noted.

Component	Role	Licence
V-JEPA 2 ViT-g (facebook/vjepa2-vitg-fpc64-384)	Frozen road-video encoder (teacher of the deployed student)	Meta
YOLO11x-pose	Driver detection	AGPL-3.0
RTMW-x-l (MMPose)	Whole-body pose estimation	Apache-2.0
MotionBERT release weights	Baseline fine-tunes; our fine-tunes are derivative works and keep the Apache-2.0 attribution	Apache-2.0
OpenBADAS	Teacher risk pipeline for distillation	see project
openpilot	Vehicle platform and the on-road driver-monitoring comparator	MIT

checkpoints/deployed/onnx/student_v5/ is a distilled derivative of V-JEPA 2 and OpenBADAS; check both licences before redistributing it.

Citation

@article{wang2026tridrive,
  title   = {Joint Driver, Vehicle, and Road Modeling for Forecasting and Driver Monitoring},
  author  = {Wang, Yuhang and Yang, Jingxin and Wei, Chuheng and Guo, Yuechen and Xu, Jinghan and Han, Zhao and Zhou, Hao},
  journal = {arXiv preprint},
  year    = {2026},
  note    = {Code and checkpoints: \url{https://huggingface.co/HenryYHW/TriDrive}}
}

License

Code, checkpoints, tables, and documents in this repository are released under CC BY-SA 4.0 unless a file states otherwise. Third-party components keep their own licences (table above).