benchmark/aide_protocol/driverwm_kp_port.md

Driver-WM-kp — port of the Driver-WM reproduction to keypoint part tokens (arch attribution experiment (2))

Owner-approved rule (docs/decisions.md 2026-09-14, "AIDE 架构归因公平实验" (2)): take the Driver-WM reproduction architecture (dual stream + gated causal injection + ST-GCN decoder), replace the INTERNAL-stream input (frozen Qwen3-VL in-cabin features) by the keypoint part tokens of K (cwm3d_kp, same tokenizer), keep the external-context stream, and train it with the shared keypoint recipe (residual over the last observed pose, zero-initialised output = zero-velocity at init, identical augmentation / selection), 3 seeds. Pre-specified reading: per-seed paired Δ(Driver-WM-kp − K) on All-MPJPE with CI containing 0 and |Δ| < 1 px on every seed → the lead belongs to the keypoint formulation; K significantly better (CI excl. 0 on ≥ 2/3 seeds) → supports an architecture advantage under the keypoint formulation; Driver-WM-kp better → reported as is.

Code: aide/models/driverwm_kp.py (model), aide/train/train_kp.py (--model driverwm_kp: additive dispatch branch; the exterior features are loaded and passed exactly through the existing --visual path), tests aide/tests/test_arch_attribution.py. Runs aide/results/runs/driverwm_kp_s{42,123,456}_v2; table aide/results/tables/arch_attribution.md §2 (aide/eval/report_arch_attribution.py).

1. What was kept from the reproduction (aide/models/driverwm_repro.py, protocol.md C10–C16)

component	reproduction	Driver-WM-kp
external-context stream	mean over the 3 exterior views (front / left / right) of the pooled Qwen3-VL-2B features (2048-d, frozen, results/feats/feats_qwen.npz[pooled]) + learned view embedding + step embedding → one causal pre-norm TransformerEncoderLayer(2048, 4 heads, d_ff 4096) (pre_ext)	identical (same classes / shapes; view_emb, pos_ext, pre_ext)
external transition	residual MLP f_ext: 2048 → 4096 → 2048, e_{t+1} = e_t + f_ext(e_t)	identical
context module	Perceiver-style: mean-pooled internal history → LN + Linear → 4 latent queries in a 256-d bottleneck; keys / values = LN + Linear(256) of the external prefix ≤ t; MultiheadAttention(256, 4 heads); output Linear → added to the last internal state (m_t)	identical structure; the query / output projections are D_int-wide (1024) instead of 2048 because the internal latent is 1024-d (PerceiverCtxKP)
gated causal injection	g_t = sigmoid(MLP(e_t)) (LN → Linear(2048, 2048) → GELU → Linear(2048, D)), z_{t+1} = (1 − g_t) ⊙ (z_t + f_int(z_t)) + g_t ⊙ m_t; override clamps g (Table-3 interventions)	identical (last Linear outputs D_int = 1024)
rollout	5 closed-loop steps, both histories grow by the predicted latent, prefix-only context (causal)	identical
skeleton decoder	Linear(D → 136 × 32) → 2 × STGCNBlock (spatial graph conv over the normalised HALPE-136 adjacency protocol.EDGES_ALL + self loops, temporal conv k = 3 over the 5 rolled-out steps, residual, LayerNorm) → Linear(32 → 2)	identical blocks (STGCNBlock, normalized_adjacency imported unchanged); read-out changed, see §2
leak guard	forward accepts exactly T_OBS frames	identical (obs and vis both asserted to T_OBS frames; test_arch_attribution.py checks that the future keypoints / features cannot reach the output)

2. What was replaced

component	reproduction	Driver-WM-kp	why
internal-stream input	frozen 2048-d Qwen3-VL feature of the in-cabin view per observed frame (+ view / step embedding) → causal TransformerEncoderLayer(2048, 4, 4096)	heads.PartTokenizer(d = 256) — K's tokenizer, unchanged: 4 part tokens per observed frame (body 26 joints, face 68, left hand 21, right hand 21; per joint (Δx, Δy)/200 to the LAST observed pose, x/W, y/H, confidence; shared hand Linear + side embedding; part + step embeddings), concatenated to ONE 1024-d latent per frame → causal pre-norm TransformerEncoderLayer(1024, 4 heads, d_ff 4096) (pre_int)	the rule: same tokenizer as K. Driver-WM's latent world model is "one latent per step" (z_t), so the 4 part tokens are concatenated (D_int = 4 × 256) rather than kept as separate tokens; the causal pre-encoder, f_int, ctx and gate then run in 1024-d. The in-cabin Qwen feature is not read at all (test: perturbing it leaves the output bit-identical).
internal transition	f_int: 2048 → 4096 → 2048	f_int: 1024 → 4096 → 1024 (same hidden width)	follows D_int
decoder read-out	sigmoid → absolute normalised coordinates, pixels = (x·W, y·H) (Eq. 8; no residual anchor, C10)	pred = last observed xy + 200 · Linear(32 → 2)(·), Linear ZERO-initialised → the untrained model is exactly zero-velocity	the shared recipe (residual over the last observed pose, zero-init output) — the same parametrisation as cwm3d_kp / SiMLPe-kp / MotionBERT-kp / ST-GCN-kp
internal positional / view embedding	learned view embedding + step embedding added to the 2048-d feature	none on top of the tokenizer: the PartTokenizer already carries part + step embeddings; a view embedding is meaningless for a single-view keypoint stream	avoid a duplicated step embedding

3. What was dropped (loss and heads)

item	reproduction (C13)	Driver-WM-kp	why
L_lat^vel (Eq. 14)	latent-velocity regression; target = the same in-cabin pre-encoder run causally over the 10-frame in-cabin FEATURE sequence [observed; GT future], detached	dropped	its target requires the frozen in-cabin features of the future frames, which the keypoint internal stream does not have; re-running the keypoint tokenizer over the 10-frame block would change the token semantics ((Δx, Δy) relative to the last frame of the sequence) and is not part of the shared recipe
L_aux (4 CE heads DBR / DER / TCR / VCR)	on the last rolled-out internal / external states	dropped (no semantic heads)	the comparison target K = cwm3d_kp is geometry-only; the shared keypoint recipe has no semantic loss
L_skel (Eq. 23, conf-masked squared error in normalised coordinates) + L_phys = L_bone + L_smooth (Eq. 25–26), λ_lat = λ_phys = 0.1		replaced by the shared recipe's loss: the model returns a plain tensor and train_kp --loss euc applies the mean Euclidean pixel error (== the metric), exactly as for MotionBERT-kp / ST-GCN-kp (hardening D; cwm3d_kp additionally has its Laplace-NLL / bone / jerk terms, which need its dict output)	same loss as the other shared-recipe baselines
training (C14)	AdamW 5e-5, wd 1e-5, cosine without warm-up, bs 32, 100 epochs, no early stop, no augmentation, no EMA	shared recipe verbatim (run_stage_bc.RECIPE): AdamW 3e-4, wd 0.05, 10 warm-up epochs + cosine, bs 32, ≤ 200 epochs, EMA 0.999, early stop patience 40 on val All-MPJPE (EMA weights), sub-window augmentation (--aug-window 1), keypoint noise 2(1 − conf) px, joint dropout 0.1, translate ± 20 px, scale ± 5 %, no flip; exterior features of a training sub-window frame = feature of the nearest protocol frame (as for cwm3d --visual, S2); no modality dropout (the reproduction has none)	the rule: identical augmentation / selection

Selection is val-only; the official test split is evaluated once per run by the existing harness (paired Δ vs zero-velocity / ZV-shrink / seed-matched SiMLPe-kp, cwm3d_kp and MotionBERT-kp as refs; the table's paired bootstraps are recomputed from the pred_test.npz files by the report script).

4. Parameter count (measured, python aide/models/driverwm_kp.py)

module	Driver-WM reproduction (123,655,254)	Driver-WM-kp (85,721,410)
tokenizer (tok, = cwm3d_kp's PartTokenizer)	—	150,784
pre_int (internal causal pre-encoder)	33,576,960 (2048-d)	12,596,224 (1024-d)
pre_ext (external causal pre-encoder)	33,576,960	33,576,960
f_int	16,787,456	8,395,776
f_ext	16,787,456	16,787,456
ctx (Perceiver context)	5,517,824	3,417,600
gate	8,396,800	6,298,624
dec (ST-GCN skeleton decoder)	8,925,762	4,469,314
semantic heads	57,364	—
view_emb / pos	8,192 / 20,480	8,192 / 20,480

For reference K = cwm3d_kp has 5,771,449 parameters (same 150,784-parameter tokenizer), MotionBERT-kp 813,088, ST-GCN-kp 530,080. Driver-WM-kp is ≈ 15 × K; the comparison is deliberately not parameter-matched (the rule fixes the architecture, not the size).

5. Guards (aide/tests/test_arch_attribution.py, all passing)

• exact zero-velocity at initialisation; 10-frame inputs rejected;
• the output is bit-identical when the FUTURE keypoints / features are changed (only the observed block is handed in), changes when the observed exterior features change (the external stream is live), and is bit-identical when the in-cabin Qwen feature changes (it is not read);
• the internal stream uses heads.PartTokenizer (K's tokenizer) with D_int = 4 × 256.

6. Deviations from the pre-specified rule

None in the architecture or recipe. The two judgement calls that the rule left open are (i) concatenating the 4 part tokens into one per-step latent (Driver-WM's single-latent transition) and (ii) dropping L_lat / L_aux / L_skel-L_phys in favour of the shared recipe's plain Euclidean loss (the rule says "shared recipe"); both are recorded above and in the model docstring.

7. Result (from aide/results/tables/arch_attribution.md §2, generated 20260914T182913Z)

Test All-MPJPE 50.40 / 50.57 / 50.97 (seeds 42 / 123 / 456; best epoch 61 / 62 / 74, early stop patience 40; 6.1 / 5.9 / 5.2 GPU-min). Paired Δ (Driver-WM-kp − K) All = +1.63 [+0.99, +2.31] / +1.91 [+1.43, +2.40] / +2.51 [+1.89, +3.14] (K better on 3/3 seeds, CI excl. 0); PCK@0.05 −0.79 ± 0.20 (WORSE); HM / HM h=5 INCONCLUSIVE. Pre-specified reading obtained: (b) "支持架构在关键点表述下的优势". Against MotionBERT-kp: Δ All +2.27 ± 0.20 (WORSE).