benchmark/tables/aide_extra/driverwm_quantile_sweep.md

Driver-WM quantile sweep — CockpitWM, our two Driver-WM ports and the published Driver-WM on the top-q motion subsets (AIDE official test split, n = 609)

Generated 20260917T071328Z by aide/eval/report_driverwm_quantile_sweep.py from aide/results/tables/hm_subsets.json (generated 20260915T031542Z, pre-registered 2026-09-15T03:01:43Z); this table itself is item (B) of the pre-registration row docs/decisions.md 2026-09-17T07:00:22Z. No new run, no GPU, no existing file modified. Subsets: top 5 % = n 30 (motion-score threshold 97.44 px); top 10 % = n 60 (motion-score threshold 72.05 px); top 20 % = n 121 (motion-score threshold 55.35 px); top 30 % = n 182 (motion-score threshold 42.00 px); top 50 % = n 304 (motion-score threshold 28.00 px); top 100 % = n 609 (motion-score threshold 5.49 px). The top-10 % subset IS the official High-Motion (HM) subset of Driver-WM (same 60 clips, threshold 72.05 px); top 100 % IS the full test set (their 'All'). Levels are mean ± sd over seeds (ddof 1) of one test evaluation per run; Δ = paired per-clip difference on the same clips, seed-matched (42/123/456), with the 10k clip-bootstrap 95 % CI per seed; k/n = seeds whose CI excludes 0. Integrity check against hm_subsets.md: 0 mismatch(es) over all §1.2 / §1.5 / §1.4 strings reproduced here

0. Which comparisons are with our ports and which with published numbers

• Every paired Δ and every non-'published' cell below is against OUR ports, trained by us under the locked protocol (aide/data/protocol.py; 5 observed → 5 future frames, 136 joints, 1920×1080 px): (i) Driver-WM reproduction (ours, their formulation) — absolute sigmoid-normalised coordinates with an MSE loss on frozen VLM features, as described in their §3.4; it does NOT reproduce their numbers (All 130.86 ± 3.11 vs published 71.47), so margins against it say nothing about Driver-WM as published; (ii) Driver-WM architecture on CockpitWM tokens — their temporal encoder trained on our keypoint part tokens under our anchored-residual formulation (the meaningful architecture comparator).
• Published Driver-WM numbers exist for exactly three cells of this grid: All-MPJPE 71.47 (top 100 %, horizon-averaged), HM-MPJPE 138.03 (top 10 %, horizon-averaged) and HM h=5 155.82 (top 10 %) — all from ONE run (seed 42) of their Table 1 / text, without CI; their Table 8 adds 3-seed means for All and HM (71.66 ± 0.3 / 138.34 ± 1.2). No published value exists for 5 / 20 / 30 / 50 % or for h=5 outside the HM subset (Fig. 3(b) gives approximate h=1–4 HM readings only; driverwm_v2_audit.md). Published cells are single-run levels: they can be placed next to ours but cannot be paired or bootstrapped.
• Consequence for the owner's hypothesis ('Driver-WM is better only at the top-10 % subset'): on the PUBLISHED model it is untestable — one published h=5 point at 10 % and no code/weights (§4). What CAN be read is how the two ports behave across the sweep (§3).

1. Horizon-averaged MPJPE (px) per quantile — mean ± sd over seeds

model	seeds	top 5 % (n 30)	top 10 % (n 60)	top 20 % (n 121)	top 30 % (n 182)	top 50 % (n 304)	top 100 % (n 609)
zero-velocity	1	175.66	139.23	111.54	97.16	79.36	52.93
CockpitWM	5	166.13 ± 0.81	131.57 ± 0.57	103.97 ± 0.59	91.00 ± 0.39	74.01 ± 0.27	48.60 ± 0.12
CockpitWM + BATON pretraining	5	163.85 ± 0.88	128.82 ± 0.39	102.48 ± 0.38	89.81 ± 0.26	73.10 ± 0.14	48.05 ± 0.06
CockpitWM (+ semantic heads)	3	164.92 ± 1.49	130.28 ± 0.86	103.40 ± 0.49	90.68 ± 0.50	73.69 ± 0.30	48.36 ± 0.17
Driver-WM reproduction (ours, their formulation)	3	206.85 ± 3.91	176.33 ± 3.29	157.20 ± 3.29	149.06 ± 3.29	141.44 ± 3.31	130.86 ± 3.11
Driver-WM architecture on CockpitWM tokens	3	168.27 ± 0.99	132.92 ± 0.87	106.44 ± 0.65	93.39 ± 0.63	76.24 ± 0.54	50.65 ± 0.29
Driver-WM published, single run (v2 Table 1 / Table 7 seed 42; text / Tables 10–11 for h=5)	1	not published	138.03	not published	not published	not published	71.47
Driver-WM published, 3-seed mean ± sd (v2 Table 8, seeds 42/123/456)	3	not published	138.34 ± 1.2	not published	not published	not published	71.66 ± 0.3

2. h=5 MPJPE (px) per quantile — mean ± sd over seeds (seed means = §1.5 of hm_subsets.md)

model	seeds	top 5 %	top 10 %	top 20 %	top 30 %	top 50 %	top 100 %
zero-velocity	1	230.28	178.67	140.66	120.46	97.28	63.74
CockpitWM	5	212.34 ± 0.45	163.37 ± 0.63	127.46 ± 0.77	109.85 ± 0.49	88.94 ± 0.32	57.64 ± 0.18
CockpitWM + BATON pretraining	5	210.51 ± 0.90	159.92 ± 0.92	125.64 ± 0.85	108.53 ± 0.57	87.87 ± 0.38	57.02 ± 0.18
CockpitWM (+ semantic heads)	3	211.05 ± 1.93	161.93 ± 0.84	126.92 ± 0.16	109.64 ± 0.30	88.69 ± 0.19	57.47 ± 0.14
Driver-WM reproduction (ours, their formulation)	3	237.83 ± 3.11	194.34 ± 2.65	169.42 ± 3.28	158.09 ± 3.48	147.71 ± 3.68	134.50 ± 3.46
Driver-WM architecture on CockpitWM tokens	3	213.63 ± 1.55	162.83 ± 1.13	129.74 ± 0.92	112.44 ± 0.85	91.42 ± 0.69	60.04 ± 0.40
Driver-WM published, single run (v2 Table 1 / Table 7 seed 42; text / Tables 10–11 for h=5)	1	not published	155.82	not published	not published	not published	not published
Driver-WM published, 3-seed mean ± sd (v2 Table 8, seeds 42/123/456)	3	not published	not published	not published	not published	not published	not published

3. Paired Δ at h=5 — CockpitWM − Driver-WM architecture on CockpitWM tokens (seed-matched 42/123/456; negative = CockpitWM better)

top q	n clips	Δ h=5 mean ± sd over seeds	k/n seeds CI excl. 0 (h=5)	per-seed Δ h=5 [95 % CI] (s42; s123; s456)	Δ horizon-avg mean ± sd	k/n (avg)	verdict (avg, existing rule)
5 %	30	-1.41 ± 1.27	0/3	-0.10 [-8.27, +8.28]; -2.64 [-10.02, +4.43]; -1.48 [-8.31, +4.53]	-1.94 ± 2.02	0/3	INCONCLUSIVE
10 %	60	+0.54 ± 1.96	0/3	+2.47 [-3.29, +8.32]; +0.58 [-4.15, +5.05]; -1.44 [-6.63, +3.44]	-1.29 ± 1.60	0/3	INCONCLUSIVE
20 %	121	-2.16 ± 1.98	1/3	-0.35 [-3.69, +3.25]; -1.85 [-4.77, +1.10]; -4.27 [-7.38, -1.15]	-2.32 ± 1.42	2/3	BEATS
30 %	182	-2.50 ± 1.51	2/3	-1.15 [-3.63, +1.35]; -2.22 [-4.37, -0.17]; -4.13 [-6.43, -1.95]	-2.38 ± 1.13	2/3	BEATS
50 %	304	-2.44 ± 1.14	2/3	-1.45 [-3.00, +0.16]; -2.19 [-3.52, -0.80]; -3.68 [-5.21, -2.16]	-2.19 ± 0.87	3/3	BEATS
100 %	609	-2.39 ± 0.63	3/3	-1.97 [-2.90, -1.04]; -2.09 [-2.85, -1.30]; -3.12 [-4.03, -2.23]	-2.02 ± 0.45	3/3	BEATS

Secondary (same rule; h=5; mean ± sd over seeds, k/n) — the other CockpitWM variants against the same port, and all three against the reproduction port:

model − comparator	top 5 %	top 10 %	top 20 %	top 30 %	top 50 %	top 100 %
CockpitWM + BATON pretraining − Driver-WM architecture on CockpitWM tokens	-3.37 ± 1.02 (0/3)	-3.34 ± 1.42 (1/3)	-4.11 ± 1.42 (3/3)	-4.05 ± 1.22 (3/3)	-3.66 ± 0.94 (3/3)	-3.10 ± 0.50 (3/3)
CockpitWM (+ semantic heads) − Driver-WM architecture on CockpitWM tokens	-2.58 ± 2.83 (0/3)	-0.90 ± 1.96 (0/3)	-2.83 ± 0.93 (1/3)	-2.80 ± 0.90 (3/3)	-2.73 ± 0.73 (3/3)	-2.57 ± 0.43 (3/3)
CockpitWM − Driver-WM reproduction (ours)	-25.61 ± 3.13 (3/3)	-30.97 ± 2.87 (3/3)	-41.83 ± 3.63 (3/3)	-48.15 ± 3.60 (3/3)	-58.74 ± 3.81 (3/3)	-76.85 ± 3.61 (3/3)
CockpitWM + BATON pretraining − Driver-WM reproduction (ours)	-27.57 ± 4.09 (3/3)	-34.85 ± 3.20 (3/3)	-43.79 ± 4.22 (3/3)	-49.69 ± 4.11 (3/3)	-59.95 ± 4.09 (3/3)	-77.56 ± 3.60 (3/3)
CockpitWM (+ semantic heads) − Driver-WM reproduction (ours)	-26.78 ± 4.39 (3/3)	-32.41 ± 2.76 (3/3)	-42.50 ± 3.43 (3/3)	-48.45 ± 3.78 (3/3)	-59.02 ± 3.87 (3/3)	-77.03 ± 3.60 (3/3)

Reading (descriptive; the pre-registration fixes no acceptance rule for B — 'decision on entering the paper after the owner sees the table'): (i) Against the architecture port at h=5, CockpitWM's paired advantage is smallest on the high-motion tail and grows toward the full set: Δ h=5 = -1.41 (0/3) at 5 %, +0.54 (0/3) at 10 %, -2.16 (1/3) at 20 %, -2.50 (2/3) at 30 %, -2.44 (2/3) at 50 %, -2.39 (3/3) at 100 %; only the full set reaches 3/3 seeds, the 10 % point is the one quantile where the port is nominally lower (+0.54, 0/3, sd 1.96) — INCONCLUSIVE under the existing rule at every quantile except 100 %, and the level difference at 10 % (163.37 vs 162.83) is far inside the CI half-width of an n = 60 subset.
(ii) Against the reproduction port the gap is huge everywhere (3/3 at every quantile) but also smallest on the tail: Δ h=5 -25.61 at 5 % → -76.85 at 100 %; i.e. the absolute-coordinate formulation loses least where the driver actually moves, which is the pattern the owner's hypothesis predicts, yet it never becomes better than CockpitWM.
(iii) Against the published single-run numbers only the 10 % column is comparable: CockpitWM 131.57 ± 0.57 vs 138.03 horizon-averaged (ours lower), and 163.37 ± 0.63 vs 155.82 at h=5 (theirs lower by 7.55 px; CockpitWM + BATON pretraining 159.92); whether that h=5 edge persists at 5 % or vanishes at 20–100 % cannot be known without their model.

4. Release check of Driver-WM code / weights (2026-09-17T07:03Z-07:06Z; curl only (arXiv abs/HTML/API, GitHub REST search + repos API unauthenticated, HuggingFace Hub API, Papers-with-Code API); nothing downloaded, no weights fetched)

Verdict: NOT RELEASED: no official code, no official weights; official project page shows a 'Code Coming Soon' button without a link. Previous record: decisions.md row 2026-09-17T07:00:22Z (PRE-REG B) stated 'no code/weights' without a dated check; no earlier dated release check exists in decisions.md, so this is the first dated check.

Official channels:

• arXiv abstract page — https://arxiv.org/abs/2605.05092 — title 'Driver-WM: A Driver-Centric Traffic-Conditioned Latent World Model for In-Cabin Dynamics Rollout'; v1 Wed 6 May 2026 16:30:48 UTC (2,621 KB); v2 Fri 26 Jun 2026 08:53:36 UTC (2,626 KB); comment 'Accepted to the 19th European Conference on Computer Vision (ECCV 2026). This version includes the supplementary material'; no code / project link in the abstract metadata; arXiv API lists only the abs and pdf links
• arXiv HTML full text v2 — https://arxiv.org/html/2605.05092 — the only external project link is the project page below ('Project page: Driver-WM' hyperlink on the title page of the PDF as well); no github.com / huggingface.co URL; no sentence promising or announcing a code or weight release
• official project page (GitHub Pages of first-author account fisher75, repo haozhuangchi.github.io, path driver-wm/) — https://fisher75.github.io/haozhuangchi.github.io/driver-wm/ — HTTP 200; buttons: 'Paper (PDF)', 'arXiv', 'Code Coming Soon' (no href), 'Video', 'Demo' (in-page figures), 'Coverage' (3 WeChat posts), 'Invited Poster', 'Workshop Details' (https://eccv26woop.github.io/), 'Watch on YouTube' (https://www.youtube.com/watch?v=wINgebPpge0); no licence text; repo tree driver-wm/ = index.html + 12 images (no .py/.pt/.zip anywhere in the repo); driver-wm/index.html last commit 2026-09-04T13:55:14Z; page Last-Modified header Wed 09 Sep 2026 20:07:40 GMT (commits of 2026-09-09 concern another project page, 'PV-WM')
• first-author GitHub account fisher75 (8 public repos) — https://api.github.com/users/fisher75/repos — haozhuangchi.github.io (project pages, pushed 2026-09-09); Cabin-WM (created 2025-11-19, last push 2025-11-10, Apache-2.0) = a FORK of QwenLM/Qwen3-VL with 0 commits ahead / 17 behind upstream, README is the Qwen3-VL README, no releases/tags -> NOT a Driver-WM release; DRAMA-X (MIT, other paper); fisher75, Baselines, LLaVA fork, sglang_fork, deepxde fork (unrelated). No GitHub user 'haozhuangchi' (404)
• GitHub repository search (REST, unauthenticated) — https://api.github.com/search/repositories — q=Driver-WM 34 hits, none related (window managers / WMI drivers); q=DriverWM 2 unrelated; q=2605.05092 0; q='driver-wm world model' 0; q='Driver-WM in-cabin' 0; q=Cabin-WM 1 unrelated; q='in-cabin world model' 1 = likithp82/RaWM (third party, below); q='driver world model AIDE' 0
• HuggingFace Hub (models/datasets search: driver-wm, driverwm, cabin-wm, Driver-WM; author fisher75) — https://huggingface.co/api/models?search=driver-wm — no models; author fisher75 has no models; one THIRD-PARTY dataset (below)
• Papers with Code API — https://paperswithcode.com/api/v1/papers/?arxiv_id=2605.05092 — HTTP 302 -> https://huggingface.co/papers/trending (service discontinued; no information)

Third-party artefacts found (NOT official, NOT downloaded, no licence):

• HF dataset Likithp/driver-wm-aide-features (author 'Likithp', not a Driver-WM author) — https://huggingface.co/datasets/Likithp/driver-wm-aide-features — created 2026-07-04T04:27:34Z, last modified 2026-07-11T19:26:38Z, public, not gated, no dataset card / README, NO licence field, 170 downloads; 14,502 files: features/{train 7536, val 1620, test 2436} = per-clip front.pt / in_cabin.pt / left.pt / right.pt for 2,898 clips (= all AIDE clips; test 609), annotation/*.json x 2,898 (AIDE annotations redistributed), outputs/ and outputs_v2/ each holding baseline/seed42/{best,latest}.pt and seed42/{best,latest}.pt at 2,103.7 MB each (4 distinct LFS objects duplicated across the two folders) -> third-party checkpoints of undocumented architecture ('baseline' is not identified as Driver-WM); NOT downloaded
• GitHub likithp82/RaWM (Likith Prabhu, Mercedes-Benz R&D India; 'When Doing Nothing Wins: Motion Regime Imbalance in In-Cabin Driver Pose Forecasting', ECCV 2026 DriveX workshop) — https://github.com/likithp82/RaWM — created 2026-08-14, 3 commits same day, README.md + CITATION.cff only, no code, no licence, README says 'Full dataset setup and pretrained weights coming soon'; quotes Driver-WM 71.47 / 138.03 from the paper (not a reimplementation with weights) and reports Zero-Velocity 53.42 / 140.63 and RaWM v6 51.05 +- 0.43 / 130.38 +- 0.93 (All / HM MPJPE) -> independent third-party statement that zero-velocity beats the published Driver-WM on All-MPJPE (our recomputed ZV: 52.93 / 139.23)

The pre-registered branch 'if released, evaluate the released model on all quantiles under our protocol' is NOT triggered; the third-party checkpoints are not Driver-WM and have no licence, so no evaluation of them is proposed; re-check recommended before camera-ready.

5. Provenance

• Runs (all pre-existing, aide/results/runs/): CockpitWM cwm3d_kp_s{42,123,456,789,1011}_v2; + BATON pretraining cwm3d_kp_batonpre_trunkp123_s*_v2 (5 seeds); (+ semantic heads) cwm3d_vis_sem_s{42,123,456}_v2; Driver-WM architecture on CockpitWM tokens driverwm_kp_s{42,123,456}_v2; Driver-WM reproduction driverwm_repro_s{42,123,456}_v2; zero-velocity recomputed from the observed frames. All numbers are read from hm_subsets.json (levels: summary[q][model].mpjpe / .h[4]; Δ: summary[q][model].delta[comp].per_seed[*] and .h[4]), which report_hm_subsets.py verified against every run's metrics_test.json and master_aide.json (594 cells, max abs diff 1.99e-13).
• Published cells: driverwm_v2_audit.md:23 (71.47 / 138.03), :56 and :58 (155.82); driverwm_v2_audit.md:235; local PDF /home/USER/project/related_works/DriverWM-V2.pdf (arXiv 2605.05092v2, 26 Jun 2026).
• Release-check evidence: aide/results/tables/driverwm_release_check_20260917T07Z.json (URLs, dates, observations as fetched). Wall time 0.1 s, CPU only.