# maliang SPEC — gc25d on GameCrafter-Data-v10 (proxy → video, numerical proxy)

Authoritative design for the rebuild. Background: `docs/CONTEXT.md`; investigator reports: `docs/understand/*.md`
(train_recipe, distill_port, infer_contract, data_v10 — read the one for your component, they have file:line refs).
Paths are relative to `/opt/dlami/nvme/zhiyangdeng/maliang` (= `$M`).

## §0 Ground rules
- Code tree: `code/gc25d/` (our package; flat modules imported with `PYTHONPATH=$M/code/gc25d:$M/code/arope`),
  `code/arope/` = **verbatim vendored** ActionRoPE (`actionrope/{arope,geometry}.py`, `distill/{arope_dit,flow_pf,runlib,ema,data}.py`,
  `distill/tests/preflight.py`) — **never edit files under code/arope/** (fork into code/gc25d instead).
  `code/arope/models -> ../../models`, `code/arope/outputs -> ../../outputs` (symlinks so vendored defaults resolve).
- Python: `$M/.venv/bin/python` (torch 2.11 cu128, DiffSynth 2.1.6 at commit 4dbf980 editable, deepspeed 0.19.6, accelerate 1.14).
  Env for every run: `DIFFSYNTH_SKIP_DOWNLOAD=True PYTHONPATH=$M/code/gc25d:$M/code/arope HF_HOME=/opt/dlami/nvme/zhiyangdeng/.cache/huggingface
  TMPDIR=$M/.tmp TORCHINDUCTOR_CACHE_DIR=$M/.tmp/inductor TRITON_CACHE_DIR=$M/.tmp/triton`. Never install flash-attn / sageattention.
- Outputs only under `$M/outputs/` or `$M/data/` (NVMe). Nothing large on `/`. Never touch AWS credentials.
- The condition is a **numerical proxy**: per-pixel uint8 semantic ids encoded by `gc_sem_cond` (pooled one-hot → convs → added after
  patch embedding). It never goes through the VAE. Rope is plain (no ActionRoPE world RoPE, no mask channel, `offset_px=None`).
- Already done (by the lead, tested): `code/gc25d/model.py` (vocab-generalized: `pooled_onehot(..., vocab)`, `SemCondEncoder(vocab=)`,
  `install_sem_cond(dit, vocab)`, `load_gc_weights` detects vocab from ckpt, `collapse_parts`, `valid_raw_ids`, `load_dit` local),
  `code/gc25d/gcdistill/gc_dit.py` (`GcDiT(vocab=)`, `ckpt_vocab`, `build_gcdit` auto-detects), `code/gc25d/capsules.py`
  (ray-cast part labelling `label_parts`, code-world `render_capsules`, `ground_points`, `capsules_from_json/to_array/from_array`).
  The "ids" vocab path is bit-identical to the deployed gc25d_v8 model (checked).

## §1 Geometry (unchanged from gc25d)
Source 1280×704 → crop x∈[30,1250) → 832×480. RGB: `geometry.resample_rgb` (INTER_AREA). ids: `geometry.resample_ids`
(centre-aligned nearest, `_IDX_X/_IDX_Y`). Player at dst (415.5, 239.5) in v10_real and P9C runs; P9 (framed) runs drift (keep them).

## §2 Time
81 frames = 21 latent cells; cell 0 = frame 0 (clean TI2V anchor, t=0); cell j = frames 4j−3…4j. Clips = `index/clips_81.json`
(end exclusive, start ∈ 96ℤ). Actions: `blocks20 = actions_block_0.25s.npy[s//4 : s//4+20]`; `text.cell_blocks(blocks20)` [21,5];
`text.cell_prompts(blocks20)` 21 strings (cell 0 = cell 1). **Rollout extension** (stage 3 only): cells 21..27 = frames s+81…s+108 and
blocks `s//4+20 … s//4+26`; valid only if frames s…s+108 share `episode` and `camera_variant` (check per-frame jsons or frames.jsonl).

## §3 Condition = numeric ids, vocab "parts" (default) / "ids" (legacy)
- raw values: static 0 sky,1 terrain,2 paving,3 architecture,4 rock,5 vegetation,6 prop; player parts 10 core,12 l_arm,13 r_arm,14 l_leg,
  15 r_leg,16 weapon; boss parts 11 core,17 l_arm,18 r_arm,19 l_leg,20 r_leg,21 weapon; 255 UNKNOWN. (`model.PLAYER_PART_IDS`, `BOSS_PART_IDS`.)
- Construction per frame: `ids = resample_ids(ids.png)` (UE labels incl. scene occlusion of characters) → `capsules.label_parts(ids,
  json['world_to_clip'], json['world_to_view'], capsules.capsules_from_json(json))`. Assert values ⊆ `model.valid_raw_ids("parts")`.
- "ids" vocab = `model.collapse_parts(parts)` (12..16→10, 17..21→11) — available for A/B, default training uses "parts" (K=20).
- At inference the code world renders the same numbers with `capsules.render_capsules` (+ static semantic layer); training never sees
  ellipse markers.

## §4 Text (unchanged)
Template/motions/NULL from `code/gc25d/text.py` byte-identical. Text table = `ref/gc25d-live-demo/weights/text_table_gc25d.pt`
(19 keys, bf16 [512,4096], zero past seq len) copied to `data/gc25d_v10/text_table_gc25d.pt`. `key_to_id = {k:i for i,k in
enumerate(sorted(table))}` (must match infer.py `_KEY_TO_ID`).

## §5 Processed data layout: `data/gc25d_v10/`
```
index/clips.jsonl   one row per clip (12,800): clip_id, subset, scene, run, start, end, split ("train"|"val"), drop (bool),
                    drop_reason, episode, camera_variant, look_name, centered, n_events, dead_frac, has_ext (bool), ortho (bool)
index/runs.json     {run: split, "_rule": "..."}
latents/<clip_id>.pt   {latent: bf16 [48,21,30,52], prompts: list[21] str, actions_block: uint8 [21,5], clip_id, subset, scene, run, start}
cond/<clip_id>.npz     ids: uint8 [81,480,832] (parts vocab raw ids)  — np.savez_compressed
cond_ext/<clip_id>.npz (only if has_ext) ids: uint8 [28,480,832] frames s+81..s+108; blocks: uint8 [7,5] = actions_block[s//4+20 : s//4+27];
                       prompts: 7 strings (text.block_to_prompt of each block)
meta/<clip_id>.npz     world_to_clip f64[F,16], world_to_view f64[F,16], projection f64[F,16], orthographic bool, fov_x_deg f32[F],
                       ortho_width_cm f32[F], frame_offset_cm f32[F,3], capsules f32[F,2,12,7] (capsules.capsules_to_array),
                       actor_pos f32[F,2,3], ground_z f32[F] (min over player capsule endpoint z minus radius), player_state/boss_state
                       (unicode arrays [F]), player_px/boss_px int32[F] (at 832×480) — F = 81 (+28 if has_ext, frames continue)
text_table_gc25d.pt
```
clip_id = `f"{scene}__{run}__{start:06d}"`. Latent = Wan2.2 VAE (`Wan2.2_VAE.pth`, DiffSynth `WanVideoVAE38`) fp32 weights with
PyTorch default TF32 convs, non-tiled, input [−1,1] from `resample_rgb` uint8, stored bf16 (≈1.06 s/clip/H200 → ≈30 min on 8 GPUs).
Split (run-level, 10 val runs = 500 clips): val runs = v10_real R10_s1512, R10_s3512, R10_s5512, R10_s7511, R10_s9511; v9_stylized
P9_golden_s2909 (02), P9_autumn_s4908 (04), P9C_golden_s6946 (06), P9C_default_s8946 (08), P9C_snow_s10945 (10). Everything else train.
Drop (flag, still processed): the 348 buried-player clips `docs/understand/data_v10_tools/v10_real_player_buried_clips.txt`.

## §6 Model
Wan2.2-TI2V-5B DiT (`models/Wan2.2-TI2V-5B`, 3 fp32 shards → bf16) + `gc_sem_cond` (vocab parts: in 1280 → 512 → 512 → zero-init 3072),
forward = `model.gc_forward` (bidirectional) / `gcdistill.gc_dit.GcDiT` (distill + deploy). Export = 831 bf16 tensors (825 Wan + 6
gc_sem_cond), keys without `dit.`, string metadata incl. `format=gc25d, arm=gc25d, sem_cond=gc_sem_cond, vocab=<parts|ids>, data=v10`.

## §7 SFT (bidirectional teacher) — `code/gc25d/train.py` (fork of ActionRoPE `actionrope/train.py`, see train_recipe.md §8)
accelerate + DeepSpeed ZeRO-2, bf16, 8 GPU × micro 1 × GA 2 = 16. Flow matching "Wan" shift 5, t uniform over 1000 shifted steps
(one t per micro-batch), loss = MSE(frames 1..20) × training_weight(t); TI2V: cell 0 = clean x0, t=0, excluded from loss.
AdamW lr 1e-5 (DiT) / **cond_lr 1e-4** (`dit.gc_sem_cond.*` group), wd 0.01, betas (0.9,0.999), warmup 500 then constant, clip 1.0,
grad ckpt on, max_steps 20000, save_every 1000 (`step-N.safetensors`), rotating `save_state` (2 slots), val_every 500 (16 fixed val clips
× t∈{200,500,800}: cond / uncond(ids all 255) / null-text), log sem_rms/x_rms.
Per-sample dropouts (dataset worker, python random): text p=0.1 → NULL_PROMPT ×21; cond p=0.05 → ids all 255 (independent);
**UNKNOWN masking p=0.5** (only if cond not dropped): world-anchored ground-plane half-planes — pick 1 (p .7, q~U(0.05,0.40)) or
2 (p .3, each q~U(0.05,0.25)) random directions θ in world XY; for each, threshold τ = max over calibration frames (every 4th + last)
of that frame's (1−q)-quantile of static grid ground-point projections ⇒ no frame has more than ≈q of its static pixels beyond
(revised 2026-10-22 after review: frame-0-only calibration let later frames reach ~100 % masked; now max-frame p90 0.37, mean 0.06);
for every frame mark static pixels (raw 0..6) whose ground point (`capsules.ground_points` at `meta.ground_z[0]`, computed on the 4×4
pooled grid centres and nearest-upsampled ×4) lies beyond any plane → 255. Character pixels are never masked. NaN ground points → masked.
This mimics the code world's unseen canvas (255) sliding with the camera; it is per-frame so cell-local (causal-safe).

## §8 CF++ distillation (`code/gc25d/gcdistill/`; forks of ActionRoPE runners, see distill_port.md §8–9)
- stage1 TF (`ar_tf_gc.py`): init = SFT ckpt; causal GcDiT, teacher forcing, lr 1e-5, betas 0.9/0.999, wd 0.01, warmup 200, clip 10
  (all three places), eff batch 8 (8 GPU, GA1, `configs/accelerate_zero2_8gpu_ga1.yaml`), 8000 steps, save_every 500 (+state),
  same dropouts/masking as SFT, windowed 0/0. Metadata must include teacher_forcing / windowed_train / windowed_sink / init_ckpt.
- stage2 CD (`cd_gc.py`): teacher = student = stage1 final; deploy axis 1000/750/500/250, solver 1, cfg 1.0, lr 2e-6, betas 0/0.999,
  wd 0, warmup 100, clip 10, EMA 0.99 start 200 (fp32 shadow), eff 8, **8000 steps** (v8 used 5000; stage-2 length is the
  main quality lever per WorldMind/ActionRoPE, 8000 = upstream default), save_every 500 (ema/ = deploy + stage3 init), dropouts 0.
- stage3 DMD (`dmd_gc.py`): student = stage2 `ema/ema_step8000`; teacher = critic = SFT (bidirectional); FSDP1 FULL_SHARD 8 GPU
  (`configs/accelerate_fsdp_dmd_8gpu.yaml`); sink 3 / kv_window 17 / rope_cap 20; denoise 1000/750/500/250; **frankenstein tail**
  (cell 0 + last K generated); K = grad_horizon = 20; rollout 21..27 cells; real z0 + real per-cell ids/prompts for 1+N cells from
  `cond` + extension (clips with has_ext only). **Extension source (decided 2026-10-22):** v10_real swaps look (re-scattered set
  dressing) exactly at frame s+81, so its raw `cond_ext` pops at cell 21 → for rows with `ext_look_change` use
  `cond_ext_canvas/<clip>.npz` (same keys: ids [28,480,832], blocks, prompts) produced by `code/gc25d/ext_canvas.py`: ground-plane
  canvas from frames 0..80 (latest wins) re-rendered with each extension frame's camera + code-drawn capsules from meta, unseen = 255
  (= code-world semantics; static agreement 97–98 % on in-look frames, ~17 % unknown in extension frames). v9 rows (no look change)
  use the raw `cond_ext`. lr 2e-6 / critic 1e-6, betas 0/0.999, wd 0.01, warmup 100, clip 10, critic_steps 5,
  EMA 0.99 start 100, 500 steps, save_every 50; cfg 1.0.
- Deploy contract (unchanged): `ref/gc25d-live-demo/code/gc25d/codeworld/{live,fast_live}.py` must load our stage3 EMA (831 keys, vocab
  auto-detected) and run 4 steps/cell with sink 3 + window 17, commit at t=0.

## §9 Smoke-test policy
Every component ships with (a) CPU/tiny tests (pytest under `code/gc25d/tests/`), (b) a real-weight smoke on assigned GPUs with tiny
step counts, (c) a written log in `outputs/_smoke/<component>/`. GPU assignment during parallel development: prep 0–1, SFT 2–5,
distill 6–7 — use `CUDA_VISIBLE_DEVICES` and never kill processes you did not start.

## §10 Orchestration (2026-10-22)
`scripts/train_sft.sh` (running → `outputs/sft_v10`), then `scripts/chain_after_sft.sh` (waits for SFT `done.json` → `scripts/pick_sft.py`
picks the teacher = lowest mean cond val mse among step ≥ 6000, latest within 0.5 % → stage1 8000 → stage2 8000 → stage3 500 →
milestone eval sweep `outputs/distill/gc25d_v10/eval/stage3_ema*/{chain241,clip28}`). Log: `outputs/distill/gc25d_v10/chain.log`.
Deploy package for v10 (parts vocab + capsule code world) is not built yet: the old `ref/gc25d-live-demo` live.py needs the vocab patch
in `scripts/distill_smoke/deploy_vocab/` and its code world still draws ellipse markers.