|
Download docs/SPEC.md from teawhite/maliang: direct link, hf CLI and curl.
- Browser
- Download file 12 kB
-
https://huggingface.co/teawhite/maliang/resolve/main/docs/SPEC.md
- Command line
-
hf download hf://teawhite/maliang/docs/SPEC.md
-
curl -L -o SPEC.md https://huggingface.co/teawhite/maliang/resolve/main/docs/SPEC.md
12 kB
| # maliang SPEC β gc25d on GameCrafter-Data-v10 (proxy β video, numerical proxy) | |
| Authoritative design for the rebuild. Background: `docs/CONTEXT.md`; investigator reports: `docs/understand/*.md` | |
| (train_recipe, distill_port, infer_contract, data_v10 β read the one for your component, they have file:line refs). | |
| Paths are relative to `/opt/dlami/nvme/zhiyangdeng/maliang` (= `$M`). | |
| ## Β§0 Ground rules | |
| - Code tree: `code/gc25d/` (our package; flat modules imported with `PYTHONPATH=$M/code/gc25d:$M/code/arope`), | |
| `code/arope/` = **verbatim vendored** ActionRoPE (`actionrope/{arope,geometry}.py`, `distill/{arope_dit,flow_pf,runlib,ema,data}.py`, | |
| `distill/tests/preflight.py`) β **never edit files under code/arope/** (fork into code/gc25d instead). | |
| `code/arope/models -> ../../models`, `code/arope/outputs -> ../../outputs` (symlinks so vendored defaults resolve). | |
| - Python: `$M/.venv/bin/python` (torch 2.11 cu128, DiffSynth 2.1.6 at commit 4dbf980 editable, deepspeed 0.19.6, accelerate 1.14). | |
| Env for every run: `DIFFSYNTH_SKIP_DOWNLOAD=True PYTHONPATH=$M/code/gc25d:$M/code/arope HF_HOME=/opt/dlami/nvme/zhiyangdeng/.cache/huggingface | |
| TMPDIR=$M/.tmp TORCHINDUCTOR_CACHE_DIR=$M/.tmp/inductor TRITON_CACHE_DIR=$M/.tmp/triton`. Never install flash-attn / sageattention. | |
| - Outputs only under `$M/outputs/` or `$M/data/` (NVMe). Nothing large on `/`. Never touch AWS credentials. | |
| - The condition is a **numerical proxy**: per-pixel uint8 semantic ids encoded by `gc_sem_cond` (pooled one-hot β convs β added after | |
| patch embedding). It never goes through the VAE. Rope is plain (no ActionRoPE world RoPE, no mask channel, `offset_px=None`). | |
| - Already done (by the lead, tested): `code/gc25d/model.py` (vocab-generalized: `pooled_onehot(..., vocab)`, `SemCondEncoder(vocab=)`, | |
| `install_sem_cond(dit, vocab)`, `load_gc_weights` detects vocab from ckpt, `collapse_parts`, `valid_raw_ids`, `load_dit` local), | |
| `code/gc25d/gcdistill/gc_dit.py` (`GcDiT(vocab=)`, `ckpt_vocab`, `build_gcdit` auto-detects), `code/gc25d/capsules.py` | |
| (ray-cast part labelling `label_parts`, code-world `render_capsules`, `ground_points`, `capsules_from_json/to_array/from_array`). | |
| The "ids" vocab path is bit-identical to the deployed gc25d_v8 model (checked). | |
| ## Β§1 Geometry (unchanged from gc25d) | |
| Source 1280Γ704 β crop xβ[30,1250) β 832Γ480. RGB: `geometry.resample_rgb` (INTER_AREA). ids: `geometry.resample_ids` | |
| (centre-aligned nearest, `_IDX_X/_IDX_Y`). Player at dst (415.5, 239.5) in v10_real and P9C runs; P9 (framed) runs drift (keep them). | |
| ## Β§2 Time | |
| 81 frames = 21 latent cells; cell 0 = frame 0 (clean TI2V anchor, t=0); cell j = frames 4jβ3β¦4j. Clips = `index/clips_81.json` | |
| (end exclusive, start β 96β€). Actions: `blocks20 = actions_block_0.25s.npy[s//4 : s//4+20]`; `text.cell_blocks(blocks20)` [21,5]; | |
| `text.cell_prompts(blocks20)` 21 strings (cell 0 = cell 1). **Rollout extension** (stage 3 only): cells 21..27 = frames s+81β¦s+108 and | |
| blocks `s//4+20 β¦ s//4+26`; valid only if frames sβ¦s+108 share `episode` and `camera_variant` (check per-frame jsons or frames.jsonl). | |
| ## Β§3 Condition = numeric ids, vocab "parts" (default) / "ids" (legacy) | |
| - raw values: static 0 sky,1 terrain,2 paving,3 architecture,4 rock,5 vegetation,6 prop; player parts 10 core,12 l_arm,13 r_arm,14 l_leg, | |
| 15 r_leg,16 weapon; boss parts 11 core,17 l_arm,18 r_arm,19 l_leg,20 r_leg,21 weapon; 255 UNKNOWN. (`model.PLAYER_PART_IDS`, `BOSS_PART_IDS`.) | |
| - Construction per frame: `ids = resample_ids(ids.png)` (UE labels incl. scene occlusion of characters) β `capsules.label_parts(ids, | |
| json['world_to_clip'], json['world_to_view'], capsules.capsules_from_json(json))`. Assert values β `model.valid_raw_ids("parts")`. | |
| - "ids" vocab = `model.collapse_parts(parts)` (12..16β10, 17..21β11) β available for A/B, default training uses "parts" (K=20). | |
| - At inference the code world renders the same numbers with `capsules.render_capsules` (+ static semantic layer); training never sees | |
| ellipse markers. | |
| ## Β§4 Text (unchanged) | |
| Template/motions/NULL from `code/gc25d/text.py` byte-identical. Text table = `ref/gc25d-live-demo/weights/text_table_gc25d.pt` | |
| (19 keys, bf16 [512,4096], zero past seq len) copied to `data/gc25d_v10/text_table_gc25d.pt`. `key_to_id = {k:i for i,k in | |
| enumerate(sorted(table))}` (must match infer.py `_KEY_TO_ID`). | |
| ## Β§5 Processed data layout: `data/gc25d_v10/` | |
| ``` | |
| index/clips.jsonl one row per clip (12,800): clip_id, subset, scene, run, start, end, split ("train"|"val"), drop (bool), | |
| drop_reason, episode, camera_variant, look_name, centered, n_events, dead_frac, has_ext (bool), ortho (bool) | |
| index/runs.json {run: split, "_rule": "..."} | |
| latents/<clip_id>.pt {latent: bf16 [48,21,30,52], prompts: list[21] str, actions_block: uint8 [21,5], clip_id, subset, scene, run, start} | |
| cond/<clip_id>.npz ids: uint8 [81,480,832] (parts vocab raw ids) β np.savez_compressed | |
| cond_ext/<clip_id>.npz (only if has_ext) ids: uint8 [28,480,832] frames s+81..s+108; blocks: uint8 [7,5] = actions_block[s//4+20 : s//4+27]; | |
| prompts: 7 strings (text.block_to_prompt of each block) | |
| meta/<clip_id>.npz world_to_clip f64[F,16], world_to_view f64[F,16], projection f64[F,16], orthographic bool, fov_x_deg f32[F], | |
| ortho_width_cm f32[F], frame_offset_cm f32[F,3], capsules f32[F,2,12,7] (capsules.capsules_to_array), | |
| actor_pos f32[F,2,3], ground_z f32[F] (min over player capsule endpoint z minus radius), player_state/boss_state | |
| (unicode arrays [F]), player_px/boss_px int32[F] (at 832Γ480) β F = 81 (+28 if has_ext, frames continue) | |
| text_table_gc25d.pt | |
| ``` | |
| clip_id = `f"{scene}__{run}__{start:06d}"`. Latent = Wan2.2 VAE (`Wan2.2_VAE.pth`, DiffSynth `WanVideoVAE38`) fp32 weights with | |
| PyTorch default TF32 convs, non-tiled, input [β1,1] from `resample_rgb` uint8, stored bf16 (β1.06 s/clip/H200 β β30 min on 8 GPUs). | |
| Split (run-level, 10 val runs = 500 clips): val runs = v10_real R10_s1512, R10_s3512, R10_s5512, R10_s7511, R10_s9511; v9_stylized | |
| P9_golden_s2909 (02), P9_autumn_s4908 (04), P9C_golden_s6946 (06), P9C_default_s8946 (08), P9C_snow_s10945 (10). Everything else train. | |
| Drop (flag, still processed): the 348 buried-player clips `docs/understand/data_v10_tools/v10_real_player_buried_clips.txt`. | |
| ## Β§6 Model | |
| Wan2.2-TI2V-5B DiT (`models/Wan2.2-TI2V-5B`, 3 fp32 shards β bf16) + `gc_sem_cond` (vocab parts: in 1280 β 512 β 512 β zero-init 3072), | |
| forward = `model.gc_forward` (bidirectional) / `gcdistill.gc_dit.GcDiT` (distill + deploy). Export = 831 bf16 tensors (825 Wan + 6 | |
| gc_sem_cond), keys without `dit.`, string metadata incl. `format=gc25d, arm=gc25d, sem_cond=gc_sem_cond, vocab=<parts|ids>, data=v10`. | |
| ## Β§7 SFT (bidirectional teacher) β `code/gc25d/train.py` (fork of ActionRoPE `actionrope/train.py`, see train_recipe.md Β§8) | |
| accelerate + DeepSpeed ZeRO-2, bf16, 8 GPU Γ micro 1 Γ GA 2 = 16. Flow matching "Wan" shift 5, t uniform over 1000 shifted steps | |
| (one t per micro-batch), loss = MSE(frames 1..20) Γ training_weight(t); TI2V: cell 0 = clean x0, t=0, excluded from loss. | |
| AdamW lr 1e-5 (DiT) / **cond_lr 1e-4** (`dit.gc_sem_cond.*` group), wd 0.01, betas (0.9,0.999), warmup 500 then constant, clip 1.0, | |
| grad ckpt on, max_steps 20000, save_every 1000 (`step-N.safetensors`), rotating `save_state` (2 slots), val_every 500 (16 fixed val clips | |
| Γ tβ{200,500,800}: cond / uncond(ids all 255) / null-text), log sem_rms/x_rms. | |
| Per-sample dropouts (dataset worker, python random): text p=0.1 β NULL_PROMPT Γ21; cond p=0.05 β ids all 255 (independent); | |
| **UNKNOWN masking p=0.5** (only if cond not dropped): world-anchored ground-plane half-planes β pick 1 (p .7, q~U(0.05,0.40)) or | |
| 2 (p .3, each q~U(0.05,0.25)) random directions ΞΈ in world XY; for each, threshold Ο = max over calibration frames (every 4th + last) | |
| of that frame's (1βq)-quantile of static grid ground-point projections β no frame has more than βq of its static pixels beyond | |
| (revised 2026-10-22 after review: frame-0-only calibration let later frames reach ~100 % masked; now max-frame p90 0.37, mean 0.06); | |
| for every frame mark static pixels (raw 0..6) whose ground point (`capsules.ground_points` at `meta.ground_z[0]`, computed on the 4Γ4 | |
| pooled grid centres and nearest-upsampled Γ4) lies beyond any plane β 255. Character pixels are never masked. NaN ground points β masked. | |
| This mimics the code world's unseen canvas (255) sliding with the camera; it is per-frame so cell-local (causal-safe). | |
| ## Β§8 CF++ distillation (`code/gc25d/gcdistill/`; forks of ActionRoPE runners, see distill_port.md Β§8β9) | |
| - stage1 TF (`ar_tf_gc.py`): init = SFT ckpt; causal GcDiT, teacher forcing, lr 1e-5, betas 0.9/0.999, wd 0.01, warmup 200, clip 10 | |
| (all three places), eff batch 8 (8 GPU, GA1, `configs/accelerate_zero2_8gpu_ga1.yaml`), 8000 steps, save_every 500 (+state), | |
| same dropouts/masking as SFT, windowed 0/0. Metadata must include teacher_forcing / windowed_train / windowed_sink / init_ckpt. | |
| - stage2 CD (`cd_gc.py`): teacher = student = stage1 final; deploy axis 1000/750/500/250, solver 1, cfg 1.0, lr 2e-6, betas 0/0.999, | |
| wd 0, warmup 100, clip 10, EMA 0.99 start 200 (fp32 shadow), eff 8, **8000 steps** (v8 used 5000; stage-2 length is the | |
| main quality lever per WorldMind/ActionRoPE, 8000 = upstream default), save_every 500 (ema/ = deploy + stage3 init), dropouts 0. | |
| - stage3 DMD (`dmd_gc.py`): student = stage2 `ema/ema_step8000`; teacher = critic = SFT (bidirectional); FSDP1 FULL_SHARD 8 GPU | |
| (`configs/accelerate_fsdp_dmd_8gpu.yaml`); sink 3 / kv_window 17 / rope_cap 20; denoise 1000/750/500/250; **frankenstein tail** | |
| (cell 0 + last K generated); K = grad_horizon = 20; rollout 21..27 cells; real z0 + real per-cell ids/prompts for 1+N cells from | |
| `cond` + extension (clips with has_ext only). **Extension source (decided 2026-10-22):** v10_real swaps look (re-scattered set | |
| dressing) exactly at frame s+81, so its raw `cond_ext` pops at cell 21 β for rows with `ext_look_change` use | |
| `cond_ext_canvas/<clip>.npz` (same keys: ids [28,480,832], blocks, prompts) produced by `code/gc25d/ext_canvas.py`: ground-plane | |
| canvas from frames 0..80 (latest wins) re-rendered with each extension frame's camera + code-drawn capsules from meta, unseen = 255 | |
| (= code-world semantics; static agreement 97β98 % on in-look frames, ~17 % unknown in extension frames). v9 rows (no look change) | |
| use the raw `cond_ext`. lr 2e-6 / critic 1e-6, betas 0/0.999, wd 0.01, warmup 100, clip 10, critic_steps 5, | |
| EMA 0.99 start 100, 500 steps, save_every 50; cfg 1.0. | |
| - Deploy contract (unchanged): `ref/gc25d-live-demo/code/gc25d/codeworld/{live,fast_live}.py` must load our stage3 EMA (831 keys, vocab | |
| auto-detected) and run 4 steps/cell with sink 3 + window 17, commit at t=0. | |
| ## Β§9 Smoke-test policy | |
| Every component ships with (a) CPU/tiny tests (pytest under `code/gc25d/tests/`), (b) a real-weight smoke on assigned GPUs with tiny | |
| step counts, (c) a written log in `outputs/_smoke/<component>/`. GPU assignment during parallel development: prep 0β1, SFT 2β5, | |
| distill 6β7 β use `CUDA_VISIBLE_DEVICES` and never kill processes you did not start. | |
| ## Β§10 Orchestration (2026-10-22) | |
| `scripts/train_sft.sh` (running β `outputs/sft_v10`), then `scripts/chain_after_sft.sh` (waits for SFT `done.json` β `scripts/pick_sft.py` | |
| picks the teacher = lowest mean cond val mse among step β₯ 6000, latest within 0.5 % β stage1 8000 β stage2 8000 β stage3 500 β | |
| milestone eval sweep `outputs/distill/gc25d_v10/eval/stage3_ema*/{chain241,clip28}`). Log: `outputs/distill/gc25d_v10/chain.log`. | |
| Deploy package for v10 (parts vocab + capsule code world) is not built yet: the old `ref/gc25d-live-demo` live.py needs the vocab patch | |
| in `scripts/distill_smoke/deploy_vocab/` and its code world still draws ellipse markers. | |