Buckets:

AsianPlayer
/

TriDrive

Files

xet

AsianPlayer/TriDrive / benchmark /aide_protocol /driverwm_kp_port.md

AsianPlayer

16 days ago

preview code

download

raw

10.1 kB

# Driver-WM-kp — port of the Driver-WM reproduction to keypoint part tokens (arch attribution experiment (2))

Owner-approved rule (docs/decisions.md 2026-09-14, "AIDE 架构归因公平实验" (2)): take the Driver-WM reproduction architecture (dual stream + gated causal injection + ST-GCN decoder), replace the INTERNAL-stream input (frozen Qwen3-VL in-cabin features) by the keypoint part tokens of K (`cwm3d_kp`, same tokenizer), keep the external-context stream, and train it with the shared keypoint recipe (residual over the last observed pose, zero-initialised output = zero-velocity at init, identical augmentation / selection), 3 seeds. Pre-specified reading: per-seed paired Δ(Driver-WM-kp − K) on All-MPJPE with CI containing 0 and |Δ| < 1 px on every seed → the lead belongs to the keypoint formulation; K significantly better (CI excl. 0 on ≥ 2/3 seeds) → supports an architecture advantage under the keypoint formulation; Driver-WM-kp better → reported as is.

Code: `aide/models/driverwm_kp.py` (model), `aide/train/train_kp.py` (`--model driverwm_kp`: additive dispatch branch; the exterior features are loaded and passed exactly through the existing `--visual` path), tests `aide/tests/test_arch_attribution.py`. Runs `aide/results/runs/driverwm_kp_s{42,123,456}_v2`; table `aide/results/tables/arch_attribution.md` §2 (`aide/eval/report_arch_attribution.py`).

## 1. What was kept from the reproduction (`aide/models/driverwm_repro.py`, protocol.md C10–C16)
| component | reproduction | Driver-WM-kp |
|---|---|---|
| external-context stream | mean over the 3 exterior views (front / left / right) of the pooled Qwen3-VL-2B features (2048-d, frozen, `results/feats/feats_qwen.npz[pooled]`) + learned view embedding + step embedding → one causal pre-norm `TransformerEncoderLayer(2048, 4 heads, d_ff 4096)` (`pre_ext`) | identical (same classes / shapes; `view_emb`, `pos_ext`, `pre_ext`) |
| external transition | residual MLP `f_ext`: 2048 → 4096 → 2048, e_{t+1} = e_t + f_ext(e_t) | identical |
| context module | Perceiver-style: mean-pooled internal history → LN + Linear → 4 latent queries in a 256-d bottleneck; keys / values = LN + Linear(256) of the external prefix ≤ t; MultiheadAttention(256, 4 heads); output Linear → added to the last internal state (m_t) | identical structure; the query / output projections are D_int-wide (1024) instead of 2048 because the internal latent is 1024-d (`PerceiverCtxKP`) |
| gated causal injection | g_t = sigmoid(MLP(e_t)) (LN → Linear(2048, 2048) → GELU → Linear(2048, D)), z_{t+1} = (1 − g_t) ⊙ (z_t + f_int(z_t)) + g_t ⊙ m_t; `override` clamps g (Table-3 interventions) | identical (last Linear outputs D_int = 1024) |
| rollout | 5 closed-loop steps, both histories grow by the predicted latent, prefix-only context (causal) | identical |
| skeleton decoder | Linear(D → 136 × 32) → 2 × `STGCNBlock` (spatial graph conv over the normalised HALPE-136 adjacency `protocol.EDGES_ALL` + self loops, temporal conv k = 3 over the 5 rolled-out steps, residual, LayerNorm) → Linear(32 → 2) | identical blocks (`STGCNBlock`, `normalized_adjacency` imported unchanged); read-out changed, see §2 |
| leak guard | forward accepts exactly T_OBS frames | identical (obs and vis both asserted to T_OBS frames; `test_arch_attribution.py` checks that the future keypoints / features cannot reach the output) |

## 2. What was replaced

| component | reproduction | Driver-WM-kp | why |
|---|---|---|---|
| internal-stream input | frozen 2048-d Qwen3-VL feature of the in-cabin view per observed frame (+ view / step embedding) → causal `TransformerEncoderLayer(2048, 4, 4096)` | `heads.PartTokenizer(d = 256)` — K's tokenizer, unchanged: 4 part tokens per observed frame (body 26 joints, face 68, left hand 21, right hand 21; per joint (Δx, Δy)/200 to the LAST observed pose, x/W, y/H, confidence; shared hand Linear + side embedding; part + step embeddings), concatenated to ONE 1024-d latent per frame → causal pre-norm `TransformerEncoderLayer(1024, 4 heads, d_ff 4096)` (`pre_int`) | the rule: same tokenizer as K. Driver-WM's latent world model is "one latent per step" (z_t), so the 4 part tokens are concatenated (D_int = 4 × 256) rather than kept as separate tokens; the causal pre-encoder, f_int, ctx and gate then run in 1024-d. The in-cabin Qwen feature is not read at all (test: perturbing it leaves the output bit-identical). |
| internal transition | `f_int`: 2048 → 4096 → 2048 | `f_int`: 1024 → 4096 → 1024 (same hidden width) | follows D_int |
| decoder read-out | sigmoid → absolute normalised coordinates, pixels = (x·W, y·H) (Eq. 8; no residual anchor, C10) | `pred = last observed xy + 200 · Linear(32 → 2)(·)`, Linear ZERO-initialised → the untrained model is exactly zero-velocity | the shared recipe (residual over the last observed pose, zero-init output) — the same parametrisation as cwm3d_kp / SiMLPe-kp / MotionBERT-kp / ST-GCN-kp |
| internal positional / view embedding | learned view embedding + step embedding added to the 2048-d feature | none on top of the tokenizer: the PartTokenizer already carries part + step embeddings; a view embedding is meaningless for a single-view keypoint stream | avoid a duplicated step embedding |
## 3. What was dropped (loss and heads)
| item | reproduction (C13) | Driver-WM-kp | why |
|---|---|---|---|
| L_lat^vel (Eq. 14) | latent-velocity regression; target = the same in-cabin pre-encoder run causally over the 10-frame in-cabin FEATURE sequence [observed; GT future], detached | dropped | its target requires the frozen in-cabin features of the future frames, which the keypoint internal stream does not have; re-running the keypoint tokenizer over the 10-frame block would change the token semantics ((Δx, Δy) relative to the last frame of the sequence) and is not part of the shared recipe |
| L_aux (4 CE heads DBR / DER / TCR / VCR) | on the last rolled-out internal / external states | dropped (no semantic heads) | the comparison target K = `cwm3d_kp` is geometry-only; the shared keypoint recipe has no semantic loss |
| L_skel (Eq. 23, conf-masked squared error in normalised coordinates) + L_phys = L_bone + L_smooth (Eq. 25–26), λ_lat = λ_phys = 0.1 | | replaced by the shared recipe's loss: the model returns a plain tensor and `train_kp --loss euc` applies the mean Euclidean pixel error (== the metric), exactly as for MotionBERT-kp / ST-GCN-kp (hardening D; cwm3d_kp additionally has its Laplace-NLL / bone / jerk terms, which need its dict output) | same loss as the other shared-recipe baselines |
| training (C14) | AdamW 5e-5, wd 1e-5, cosine without warm-up, bs 32, 100 epochs, no early stop, no augmentation, no EMA | shared recipe verbatim (`run_stage_bc.RECIPE`): AdamW 3e-4, wd 0.05, 10 warm-up epochs + cosine, bs 32, ≤ 200 epochs, EMA 0.999, early stop patience 40 on val All-MPJPE (EMA weights), sub-window augmentation (`--aug-window 1`), keypoint noise 2(1 − conf) px, joint dropout 0.1, translate ± 20 px, scale ± 5 %, no flip; exterior features of a training sub-window frame = feature of the nearest protocol frame (as for cwm3d --visual, S2); no modality dropout (the reproduction has none) | the rule: identical augmentation / selection |
Selection is val-only; the official test split is evaluated once per run by the existing harness (paired Δ vs zero-velocity / ZV-shrink / seed-matched SiMLPe-kp, cwm3d_kp and MotionBERT-kp as refs; the table's paired bootstraps are recomputed from the pred_test.npz files by the report script).
## 4. Parameter count (measured, `python aide/models/driverwm_kp.py`)

| module | Driver-WM reproduction (123,655,254) | Driver-WM-kp (85,721,410) |
|---|---|---|
| tokenizer (`tok`, = cwm3d_kp's PartTokenizer) | — | 150,784 |
| `pre_int` (internal causal pre-encoder) | 33,576,960 (2048-d) | 12,596,224 (1024-d) |
| `pre_ext` (external causal pre-encoder) | 33,576,960 | 33,576,960 |
| `f_int` | 16,787,456 | 8,395,776 |
| `f_ext` | 16,787,456 | 16,787,456 |
| `ctx` (Perceiver context) | 5,517,824 | 3,417,600 |
| `gate` | 8,396,800 | 6,298,624 |
| `dec` (ST-GCN skeleton decoder) | 8,925,762 | 4,469,314 |
| semantic heads | 57,364 | — |
| `view_emb` / `pos` | 8,192 / 20,480 | 8,192 / 20,480 |

For reference K = `cwm3d_kp` has 5,771,449 parameters (same 150,784-parameter tokenizer), MotionBERT-kp 813,088, ST-GCN-kp 530,080. Driver-WM-kp is ≈ 15 × K; the comparison is deliberately not parameter-matched (the rule fixes the architecture, not the size).

## 5. Guards (aide/tests/test_arch_attribution.py, all passing)

- exact zero-velocity at initialisation; 10-frame inputs rejected;
- the output is bit-identical when the FUTURE keypoints / features are changed (only the observed block is handed in), changes when the observed exterior features change (the external stream is live), and is bit-identical when the in-cabin Qwen feature changes (it is not read);
- the internal stream uses `heads.PartTokenizer` (K's tokenizer) with D_int = 4 × 256.
## 6. Deviations from the pre-specified rule
None in the architecture or recipe. The two judgement calls that the rule left open are (i) concatenating the 4 part tokens into one per-step latent (Driver-WM's single-latent transition) and (ii) dropping L_lat / L_aux / L_skel-L_phys in favour of the shared recipe's plain Euclidean loss (the rule says "shared recipe"); both are recorded above and in the model docstring.
## 7. Result (from `aide/results/tables/arch_attribution.md` §2, generated 20260914T182913Z)

Test All-MPJPE 50.40 / 50.57 / 50.97 (seeds 42 / 123 / 456; best epoch 61 / 62 / 74, early stop patience 40; 6.1 / 5.9 / 5.2 GPU-min). Paired Δ (Driver-WM-kp − K) All = +1.63 [+0.99, +2.31] / +1.91 [+1.43, +2.40] / +2.51 [+1.89, +3.14] (K better on 3/3 seeds, CI excl. 0); PCK@0.05 −0.79 ± 0.20 (WORSE); HM / HM h=5 INCONCLUSIVE. Pre-specified reading obtained: (b) "支持架构在关键点表述下的优势". Against MotionBERT-kp: Δ All +2.27 ± 0.20 (WORSE).

Xet Storage Details

Size:: 10.1 kB
Xet hash:: 9cf120ab4046cfc20a1fe5e2022fffc58907cd252762f1c636cd9d2e2d8b28a4

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.

	# Driver-WM-kp — port of the Driver-WM reproduction to keypoint part tokens (arch attribution experiment (2))

	Owner-approved rule (docs/decisions.md 2026-09-14, "AIDE 架构归因公平实验" (2)): take the Driver-WM reproduction architecture (dual stream + gated causal injection + ST-GCN decoder), replace the INTERNAL-stream input (frozen Qwen3-VL in-cabin features) by the keypoint part tokens of K (`cwm3d_kp`, same tokenizer), keep the external-context stream, and train it with the shared keypoint recipe (residual over the last observed pose, zero-initialised output = zero-velocity at init, identical augmentation / selection), 3 seeds. Pre-specified reading: per-seed paired Δ(Driver-WM-kp − K) on All-MPJPE with CI containing 0 and \|Δ\| < 1 px on every seed → the lead belongs to the keypoint formulation; K significantly better (CI excl. 0 on ≥ 2/3 seeds) → supports an architecture advantage under the keypoint formulation; Driver-WM-kp better → reported as is.

	Code: `aide/models/driverwm_kp.py` (model), `aide/train/train_kp.py` (`--model driverwm_kp`: additive dispatch branch; the exterior features are loaded and passed exactly through the existing `--visual` path), tests `aide/tests/test_arch_attribution.py`. Runs `aide/results/runs/driverwm_kp_s{42,123,456}_v2`; table `aide/results/tables/arch_attribution.md` §2 (`aide/eval/report_arch_attribution.py`).

	## 1. What was kept from the reproduction (`aide/models/driverwm_repro.py`, protocol.md C10–C16)

	\| component \| reproduction \| Driver-WM-kp \|
	\|---\|---\|---\|
	\| external-context stream \| mean over the 3 exterior views (front / left / right) of the pooled Qwen3-VL-2B features (2048-d, frozen, `results/feats/feats_qwen.npz[pooled]`) + learned view embedding + step embedding → one causal pre-norm `TransformerEncoderLayer(2048, 4 heads, d_ff 4096)` (`pre_ext`) \| identical (same classes / shapes; `view_emb`, `pos_ext`, `pre_ext`) \|
	\| external transition \| residual MLP `f_ext`: 2048 → 4096 → 2048, e_{t+1} = e_t + f_ext(e_t) \| identical \|
	\| context module \| Perceiver-style: mean-pooled internal history → LN + Linear → 4 latent queries in a 256-d bottleneck; keys / values = LN + Linear(256) of the external prefix ≤ t; MultiheadAttention(256, 4 heads); output Linear → added to the last internal state (m_t) \| identical structure; the query / output projections are D_int-wide (1024) instead of 2048 because the internal latent is 1024-d (`PerceiverCtxKP`) \|
	\| gated causal injection \| g_t = sigmoid(MLP(e_t)) (LN → Linear(2048, 2048) → GELU → Linear(2048, D)), z_{t+1} = (1 − g_t) ⊙ (z_t + f_int(z_t)) + g_t ⊙ m_t; `override` clamps g (Table-3 interventions) \| identical (last Linear outputs D_int = 1024) \|
	\| rollout \| 5 closed-loop steps, both histories grow by the predicted latent, prefix-only context (causal) \| identical \|
	\| skeleton decoder \| Linear(D → 136 × 32) → 2 × `STGCNBlock` (spatial graph conv over the normalised HALPE-136 adjacency `protocol.EDGES_ALL` + self loops, temporal conv k = 3 over the 5 rolled-out steps, residual, LayerNorm) → Linear(32 → 2) \| identical blocks (`STGCNBlock`, `normalized_adjacency` imported unchanged); read-out changed, see §2 \|
	\| leak guard \| forward accepts exactly T_OBS frames \| identical (obs and vis both asserted to T_OBS frames; `test_arch_attribution.py` checks that the future keypoints / features cannot reach the output) \|

	## 2. What was replaced

	\| component \| reproduction \| Driver-WM-kp \| why \|
	\|---\|---\|---\|---\|
	\| internal-stream input \| frozen 2048-d Qwen3-VL feature of the in-cabin view per observed frame (+ view / step embedding) → causal `TransformerEncoderLayer(2048, 4, 4096)` \| `heads.PartTokenizer(d = 256)` — K's tokenizer, unchanged: 4 part tokens per observed frame (body 26 joints, face 68, left hand 21, right hand 21; per joint (Δx, Δy)/200 to the LAST observed pose, x/W, y/H, confidence; shared hand Linear + side embedding; part + step embeddings), concatenated to ONE 1024-d latent per frame → causal pre-norm `TransformerEncoderLayer(1024, 4 heads, d_ff 4096)` (`pre_int`) \| the rule: same tokenizer as K. Driver-WM's latent world model is "one latent per step" (z_t), so the 4 part tokens are concatenated (D_int = 4 × 256) rather than kept as separate tokens; the causal pre-encoder, f_int, ctx and gate then run in 1024-d. The in-cabin Qwen feature is not read at all (test: perturbing it leaves the output bit-identical). \|
	\| internal transition \| `f_int`: 2048 → 4096 → 2048 \| `f_int`: 1024 → 4096 → 1024 (same hidden width) \| follows D_int \|
	\| decoder read-out \| sigmoid → absolute normalised coordinates, pixels = (x·W, y·H) (Eq. 8; no residual anchor, C10) \| `pred = last observed xy + 200 · Linear(32 → 2)(·)`, Linear ZERO-initialised → the untrained model is exactly zero-velocity \| the shared recipe (residual over the last observed pose, zero-init output) — the same parametrisation as cwm3d_kp / SiMLPe-kp / MotionBERT-kp / ST-GCN-kp \|
	\| internal positional / view embedding \| learned view embedding + step embedding added to the 2048-d feature \| none on top of the tokenizer: the PartTokenizer already carries part + step embeddings; a view embedding is meaningless for a single-view keypoint stream \| avoid a duplicated step embedding \|

	## 3. What was dropped (loss and heads)

	\| item \| reproduction (C13) \| Driver-WM-kp \| why \|
	\|---\|---\|---\|---\|
	\| L_lat^vel (Eq. 14) \| latent-velocity regression; target = the same in-cabin pre-encoder run causally over the 10-frame in-cabin FEATURE sequence [observed; GT future], detached \| dropped \| its target requires the frozen in-cabin features of the future frames, which the keypoint internal stream does not have; re-running the keypoint tokenizer over the 10-frame block would change the token semantics ((Δx, Δy) relative to the last frame of the sequence) and is not part of the shared recipe \|
	\| L_aux (4 CE heads DBR / DER / TCR / VCR) \| on the last rolled-out internal / external states \| dropped (no semantic heads) \| the comparison target K = `cwm3d_kp` is geometry-only; the shared keypoint recipe has no semantic loss \|
	\| L_skel (Eq. 23, conf-masked squared error in normalised coordinates) + L_phys = L_bone + L_smooth (Eq. 25–26), λ_lat = λ_phys = 0.1 \| \| replaced by the shared recipe's loss: the model returns a plain tensor and `train_kp --loss euc` applies the mean Euclidean pixel error (== the metric), exactly as for MotionBERT-kp / ST-GCN-kp (hardening D; cwm3d_kp additionally has its Laplace-NLL / bone / jerk terms, which need its dict output) \| same loss as the other shared-recipe baselines \|
	\| training (C14) \| AdamW 5e-5, wd 1e-5, cosine without warm-up, bs 32, 100 epochs, no early stop, no augmentation, no EMA \| shared recipe verbatim (`run_stage_bc.RECIPE`): AdamW 3e-4, wd 0.05, 10 warm-up epochs + cosine, bs 32, ≤ 200 epochs, EMA 0.999, early stop patience 40 on val All-MPJPE (EMA weights), sub-window augmentation (`--aug-window 1`), keypoint noise 2(1 − conf) px, joint dropout 0.1, translate ± 20 px, scale ± 5 %, no flip; exterior features of a training sub-window frame = feature of the nearest protocol frame (as for cwm3d --visual, S2); no modality dropout (the reproduction has none) \| the rule: identical augmentation / selection \|

	Selection is val-only; the official test split is evaluated once per run by the existing harness (paired Δ vs zero-velocity / ZV-shrink / seed-matched SiMLPe-kp, cwm3d_kp and MotionBERT-kp as refs; the table's paired bootstraps are recomputed from the pred_test.npz files by the report script).

	## 4. Parameter count (measured, `python aide/models/driverwm_kp.py`)

	\| module \| Driver-WM reproduction (123,655,254) \| Driver-WM-kp (85,721,410) \|
	\|---\|---\|---\|
	\| tokenizer (`tok`, = cwm3d_kp's PartTokenizer) \| — \| 150,784 \|
	\| `pre_int` (internal causal pre-encoder) \| 33,576,960 (2048-d) \| 12,596,224 (1024-d) \|
	\| `pre_ext` (external causal pre-encoder) \| 33,576,960 \| 33,576,960 \|
	\| `f_int` \| 16,787,456 \| 8,395,776 \|
	\| `f_ext` \| 16,787,456 \| 16,787,456 \|
	\| `ctx` (Perceiver context) \| 5,517,824 \| 3,417,600 \|
	\| `gate` \| 8,396,800 \| 6,298,624 \|
	\| `dec` (ST-GCN skeleton decoder) \| 8,925,762 \| 4,469,314 \|
	\| semantic heads \| 57,364 \| — \|
	\| `view_emb` / `pos` \| 8,192 / 20,480 \| 8,192 / 20,480 \|

	For reference K = `cwm3d_kp` has 5,771,449 parameters (same 150,784-parameter tokenizer), MotionBERT-kp 813,088, ST-GCN-kp 530,080. Driver-WM-kp is ≈ 15 × K; the comparison is deliberately not parameter-matched (the rule fixes the architecture, not the size).

	## 5. Guards (aide/tests/test_arch_attribution.py, all passing)

	- exact zero-velocity at initialisation; 10-frame inputs rejected;
	- the output is bit-identical when the FUTURE keypoints / features are changed (only the observed block is handed in), changes when the observed exterior features change (the external stream is live), and is bit-identical when the in-cabin Qwen feature changes (it is not read);
	- the internal stream uses `heads.PartTokenizer` (K's tokenizer) with D_int = 4 × 256.

	## 6. Deviations from the pre-specified rule

	None in the architecture or recipe. The two judgement calls that the rule left open are (i) concatenating the 4 part tokens into one per-step latent (Driver-WM's single-latent transition) and (ii) dropping L_lat / L_aux / L_skel-L_phys in favour of the shared recipe's plain Euclidean loss (the rule says "shared recipe"); both are recorded above and in the model docstring.

	## 7. Result (from `aide/results/tables/arch_attribution.md` §2, generated 20260914T182913Z)

	Test All-MPJPE 50.40 / 50.57 / 50.97 (seeds 42 / 123 / 456; best epoch 61 / 62 / 74, early stop patience 40; 6.1 / 5.9 / 5.2 GPU-min). Paired Δ (Driver-WM-kp − K) All = +1.63 [+0.99, +2.31] / +1.91 [+1.43, +2.40] / +2.51 [+1.89, +3.14] (K better on 3/3 seeds, CI excl. 0); PCK@0.05 −0.79 ± 0.20 (WORSE); HM / HM h=5 INCONCLUSIVE. Pre-specified reading obtained: (b) "支持架构在关键点表述下的优势". Against MotionBERT-kp: Δ All +2.27 ± 0.20 (WORSE).