maliang SPEC — gc25d on GameCrafter-Data-v10 (proxy → video, numerical proxy)

Authoritative design for the rebuild. Background: docs/CONTEXT.md; investigator reports: docs/understand/*.md (train_recipe, distill_port, infer_contract, data_v10 — read the one for your component, they have file:line refs). Paths are relative to /opt/dlami/nvme/zhiyangdeng/maliang (= $M).

§0 Ground rules

Code tree: code/gc25d/ (our package; flat modules imported with PYTHONPATH=$M/code/gc25d:$M/code/arope), code/arope/ = verbatim vendored ActionRoPE (actionrope/{arope,geometry}.py, distill/{arope_dit,flow_pf,runlib,ema,data}.py, distill/tests/preflight.py) — never edit files under code/arope/ (fork into code/gc25d instead). code/arope/models -> ../../models, code/arope/outputs -> ../../outputs (symlinks so vendored defaults resolve).
Python: $M/.venv/bin/python (torch 2.11 cu128, DiffSynth 2.1.6 at commit 4dbf980 editable, deepspeed 0.19.6, accelerate 1.14). Env for every run: DIFFSYNTH_SKIP_DOWNLOAD=True PYTHONPATH=$M/code/gc25d:$M/code/arope HF_HOME=/opt/dlami/nvme/zhiyangdeng/.cache/huggingface TMPDIR=$M/.tmp TORCHINDUCTOR_CACHE_DIR=$M/.tmp/inductor TRITON_CACHE_DIR=$M/.tmp/triton. Never install flash-attn / sageattention.
Outputs only under $M/outputs/ or $M/data/ (NVMe). Nothing large on /. Never touch AWS credentials.
The condition is a numerical proxy: per-pixel uint8 semantic ids encoded by gc_sem_cond (pooled one-hot → convs → added after patch embedding). It never goes through the VAE. Rope is plain (no ActionRoPE world RoPE, no mask channel, offset_px=None).
Already done (by the lead, tested): code/gc25d/model.py (vocab-generalized: pooled_onehot(..., vocab), SemCondEncoder(vocab=), install_sem_cond(dit, vocab), load_gc_weights detects vocab from ckpt, collapse_parts, valid_raw_ids, load_dit local), code/gc25d/gcdistill/gc_dit.py (GcDiT(vocab=), ckpt_vocab, build_gcdit auto-detects), code/gc25d/capsules.py (ray-cast part labelling label_parts, code-world render_capsules, ground_points, capsules_from_json/to_array/from_array). The "ids" vocab path is bit-identical to the deployed gc25d_v8 model (checked).

§1 Geometry (unchanged from gc25d)

Source 1280×704 → crop x∈[30,1250) → 832×480. RGB: geometry.resample_rgb (INTER_AREA). ids: geometry.resample_ids (centre-aligned nearest, _IDX_X/_IDX_Y). Player at dst (415.5, 239.5) in v10_real and P9C runs; P9 (framed) runs drift (keep them).

§2 Time

81 frames = 21 latent cells; cell 0 = frame 0 (clean TI2V anchor, t=0); cell j = frames 4j−3…4j. Clips = index/clips_81.json (end exclusive, start ∈ 96ℤ). Actions: blocks20 = actions_block_0.25s.npy[s//4 : s//4+20]; text.cell_blocks(blocks20) [21,5]; text.cell_prompts(blocks20) 21 strings (cell 0 = cell 1). Rollout extension (stage 3 only): cells 21..27 = frames s+81…s+108 and blocks s//4+20 … s//4+26; valid only if frames s…s+108 share episode and camera_variant (check per-frame jsons or frames.jsonl).

§3 Condition = numeric ids, vocab "parts" (default) / "ids" (legacy)

raw values: static 0 sky,1 terrain,2 paving,3 architecture,4 rock,5 vegetation,6 prop; player parts 10 core,12 l_arm,13 r_arm,14 l_leg, 15 r_leg,16 weapon; boss parts 11 core,17 l_arm,18 r_arm,19 l_leg,20 r_leg,21 weapon; 255 UNKNOWN. (model.PLAYER_PART_IDS, BOSS_PART_IDS.)
Construction per frame: ids = resample_ids(ids.png) (UE labels incl. scene occlusion of characters) → capsules.label_parts(ids, json['world_to_clip'], json['world_to_view'], capsules.capsules_from_json(json)). Assert values ⊆ model.valid_raw_ids("parts").
"ids" vocab = model.collapse_parts(parts) (12..16→10, 17..21→11) — available for A/B, default training uses "parts" (K=20).
At inference the code world renders the same numbers with capsules.render_capsules (+ static semantic layer); training never sees ellipse markers.

§4 Text (unchanged)

Template/motions/NULL from code/gc25d/text.py byte-identical. Text table = ref/gc25d-live-demo/weights/text_table_gc25d.pt (19 keys, bf16 [512,4096], zero past seq len) copied to data/gc25d_v10/text_table_gc25d.pt. key_to_id = {k:i for i,k in enumerate(sorted(table))} (must match infer.py _KEY_TO_ID).

§5 Processed data layout: `data/gc25d_v10/`

index/clips.jsonl   one row per clip (12,800): clip_id, subset, scene, run, start, end, split ("train"|"val"), drop (bool),
                    drop_reason, episode, camera_variant, look_name, centered, n_events, dead_frac, has_ext (bool), ortho (bool)
index/runs.json     {run: split, "_rule": "..."}
latents/<clip_id>.pt   {latent: bf16 [48,21,30,52], prompts: list[21] str, actions_block: uint8 [21,5], clip_id, subset, scene, run, start}
cond/<clip_id>.npz     ids: uint8 [81,480,832] (parts vocab raw ids)  — np.savez_compressed
cond_ext/<clip_id>.npz (only if has_ext) ids: uint8 [28,480,832] frames s+81..s+108; blocks: uint8 [7,5] = actions_block[s//4+20 : s//4+27];
                       prompts: 7 strings (text.block_to_prompt of each block)
meta/<clip_id>.npz     world_to_clip f64[F,16], world_to_view f64[F,16], projection f64[F,16], orthographic bool, fov_x_deg f32[F],
                       ortho_width_cm f32[F], frame_offset_cm f32[F,3], capsules f32[F,2,12,7] (capsules.capsules_to_array),
                       actor_pos f32[F,2,3], ground_z f32[F] (min over player capsule endpoint z minus radius), player_state/boss_state
                       (unicode arrays [F]), player_px/boss_px int32[F] (at 832×480) — F = 81 (+28 if has_ext, frames continue)
text_table_gc25d.pt

clip_id = f"{scene}__{run}__{start:06d}". Latent = Wan2.2 VAE (Wan2.2_VAE.pth, DiffSynth WanVideoVAE38) fp32 weights with PyTorch default TF32 convs, non-tiled, input [−1,1] from resample_rgb uint8, stored bf16 (≈1.06 s/clip/H200 → ≈30 min on 8 GPUs). Split (run-level, 10 val runs = 500 clips): val runs = v10_real R10_s1512, R10_s3512, R10_s5512, R10_s7511, R10_s9511; v9_stylized P9_golden_s2909 (02), P9_autumn_s4908 (04), P9C_golden_s6946 (06), P9C_default_s8946 (08), P9C_snow_s10945 (10). Everything else train. Drop (flag, still processed): the 348 buried-player clips docs/understand/data_v10_tools/v10_real_player_buried_clips.txt.

§6 Model

Wan2.2-TI2V-5B DiT (models/Wan2.2-TI2V-5B, 3 fp32 shards → bf16) + gc_sem_cond (vocab parts: in 1280 → 512 → 512 → zero-init 3072), forward = model.gc_forward (bidirectional) / gcdistill.gc_dit.GcDiT (distill + deploy). Export = 831 bf16 tensors (825 Wan + 6 gc_sem_cond), keys without dit., string metadata incl. format=gc25d, arm=gc25d, sem_cond=gc_sem_cond, vocab=<parts|ids>, data=v10.

§7 SFT (bidirectional teacher) — `code/gc25d/train.py` (fork of ActionRoPE `actionrope/train.py`, see train_recipe.md §8)

accelerate + DeepSpeed ZeRO-2, bf16, 8 GPU × micro 1 × GA 2 = 16. Flow matching "Wan" shift 5, t uniform over 1000 shifted steps (one t per micro-batch), loss = MSE(frames 1..20) × training_weight(t); TI2V: cell 0 = clean x0, t=0, excluded from loss. AdamW lr 1e-5 (DiT) / cond_lr 1e-4 (dit.gc_sem_cond.* group), wd 0.01, betas (0.9,0.999), warmup 500 then constant, clip 1.0, grad ckpt on, max_steps 20000, save_every 1000 (step-N.safetensors), rotating save_state (2 slots), val_every 500 (16 fixed val clips × t∈{200,500,800}: cond / uncond(ids all 255) / null-text), log sem_rms/x_rms. Per-sample dropouts (dataset worker, python random): text p=0.1 → NULL_PROMPT ×21; cond p=0.05 → ids all 255 (independent); UNKNOWN masking p=0.5 (only if cond not dropped): world-anchored ground-plane half-planes — pick 1 (p .7, q~~U(0.05,0.40)) or 2 (p .3, each q~~U(0.05,0.25)) random directions θ in world XY; for each, threshold τ = max over calibration frames (every 4th + last) of that frame's (1−q)-quantile of static grid ground-point projections ⇒ no frame has more than ≈q of its static pixels beyond (revised 2026-10-22 after review: frame-0-only calibration let later frames reach ~100 % masked; now max-frame p90 0.37, mean 0.06); for every frame mark static pixels (raw 0..6) whose ground point (capsules.ground_points at meta.ground_z[0], computed on the 4×4 pooled grid centres and nearest-upsampled ×4) lies beyond any plane → 255. Character pixels are never masked. NaN ground points → masked. This mimics the code world's unseen canvas (255) sliding with the camera; it is per-frame so cell-local (causal-safe).

§8 CF++ distillation (`code/gc25d/gcdistill/`; forks of ActionRoPE runners, see distill_port.md §8–9)

stage1 TF (ar_tf_gc.py): init = SFT ckpt; causal GcDiT, teacher forcing, lr 1e-5, betas 0.9/0.999, wd 0.01, warmup 200, clip 10 (all three places), eff batch 8 (8 GPU, GA1, configs/accelerate_zero2_8gpu_ga1.yaml), 8000 steps, save_every 500 (+state), same dropouts/masking as SFT, windowed 0/0. Metadata must include teacher_forcing / windowed_train / windowed_sink / init_ckpt.
stage2 CD (cd_gc.py): teacher = student = stage1 final; deploy axis 1000/750/500/250, solver 1, cfg 1.0, lr 2e-6, betas 0/0.999, wd 0, warmup 100, clip 10, EMA 0.99 start 200 (fp32 shadow), eff 8, 8000 steps (v8 used 5000; stage-2 length is the main quality lever per WorldMind/ActionRoPE, 8000 = upstream default), save_every 500 (ema/ = deploy + stage3 init), dropouts 0.
stage3 DMD (dmd_gc.py): student = stage2 ema/ema_step8000; teacher = critic = SFT (bidirectional); FSDP1 FULL_SHARD 8 GPU (configs/accelerate_fsdp_dmd_8gpu.yaml); sink 3 / kv_window 17 / rope_cap 20; denoise 1000/750/500/250; frankenstein tail (cell 0 + last K generated); K = grad_horizon = 20; rollout 21..27 cells; real z0 + real per-cell ids/prompts for 1+N cells from cond + extension (clips with has_ext only). Extension source (decided 2026-10-22): v10_real swaps look (re-scattered set dressing) exactly at frame s+81, so its raw cond_ext pops at cell 21 → for rows with ext_look_change use cond_ext_canvas/<clip>.npz (same keys: ids [28,480,832], blocks, prompts) produced by code/gc25d/ext_canvas.py: ground-plane canvas from frames 0..80 (latest wins) re-rendered with each extension frame's camera + code-drawn capsules from meta, unseen = 255 (= code-world semantics; static agreement 97–98 % on in-look frames, ~17 % unknown in extension frames). v9 rows (no look change) use the raw cond_ext. lr 2e-6 / critic 1e-6, betas 0/0.999, wd 0.01, warmup 100, clip 10, critic_steps 5, EMA 0.99 start 100, 500 steps, save_every 50; cfg 1.0.
Deploy contract (unchanged): ref/gc25d-live-demo/code/gc25d/codeworld/{live,fast_live}.py must load our stage3 EMA (831 keys, vocab auto-detected) and run 4 steps/cell with sink 3 + window 17, commit at t=0.

§9 Smoke-test policy

Every component ships with (a) CPU/tiny tests (pytest under code/gc25d/tests/), (b) a real-weight smoke on assigned GPUs with tiny step counts, (c) a written log in outputs/_smoke/<component>/. GPU assignment during parallel development: prep 0–1, SFT 2–5, distill 6–7 — use CUDA_VISIBLE_DEVICES and never kill processes you did not start.

§10 Orchestration (2026-10-22)

scripts/train_sft.sh (running → outputs/sft_v10), then scripts/chain_after_sft.sh (waits for SFT done.json → scripts/pick_sft.py picks the teacher = lowest mean cond val mse among step ≥ 6000, latest within 0.5 % → stage1 8000 → stage2 8000 → stage3 500 → milestone eval sweep outputs/distill/gc25d_v10/eval/stage3_ema*/{chain241,clip28}). Log: outputs/distill/gc25d_v10/chain.log. Deploy package for v10 (parts vocab + capsule code world) is not built yet: the old ref/gc25d-live-demo live.py needs the vocab patch in scripts/distill_smoke/deploy_vocab/ and its code world still draws ellipse markers.