docs/impl/prep.md

prep — GameCrafter-Data-v10 → data/gc25d_v10/ (SPEC §1–§5)

Status 2026-10-22: code done and tested. MINI dataset built, verified and marked READY (data/gc25d_v10_mini/READY). The full prep has not been run yet; its command is below.

Open decision before stage 3: ext_look_change. 42 % of the DMD pool has a semantic-map pop at cell 21. See Notes for consumers.

Review-fix pass (same day; MINI outputs unchanged, no rebuild needed):

• Source ids.png values are now asserted ⊆ {0..6,10,11}, in prep and in the verify spot check.
• Killing the main process no longer leaves orphan ranks: there is a SIGTERM handler, and ranks and workers set PDEATHSIG.
• A torn last row in rows_rank*.jsonl no longer breaks the merge on resume.
• The ext_look_change note has been corrected (it is a real pop), and verify prints the counts.

Smoke logs are in outputs/_smoke/prep/fixer/.

What

file	role
code/gc25d/prep_v10.py	the prep CLI: one spawned process per GPU; DataLoader workers do the CPU work; the main thread runs the VAE; atomic writes; resume; merge
code/gc25d/verify_prep.py	checks index, files and shapes on every clip, then GPU spot checks (VAE PSNR, bit-exact re-encode, cell 0, label_parts, capsule re-render IoU)
code/gc25d/tests/test_prep.py	26 CPU tests (~10 s): block/prompt alignment, ext validity, split/drop, atomic write / resume / merge, torn-row resume, raw ids check, PDEATHSIG, VAE input LUT, one real clip (no VAE)

The output layout follows SPEC §5 exactly. Additions beyond SPEC:

• index/clips.jsonl also has look, max_frame_offset_cm, ext_look_change, mirrored_world, player_px_mean, boss_visible_frac, player_hidden_frac (non-dead frames with 0 player px). drop_reason is "buried_player" or null.
• meta/*.npz also has bot_mode (str [F]) and frames (int32 [F], absolute frame index).
• index/ also holds rows_rank<r>.jsonl (per-rank rows used for resume), prep_args.json, prep_stats_<time>.json (throughput) and verify_report.json.
• latents/*.pt stores actions_block as a torch uint8 tensor, so torch.load(..., weights_only=True) works.

How (conventions that matter)

• Per clip, inside a worker (OMP_NUM_THREADS=1, cv2.setNumThreads(1), one LRU-cached RunTars per run with shards indexed lazily):
  1. Read the per-frame tar jsons s…s+108.
  2. has_ext ⇔ all 109 frames exist, episode and camera_variant are constant, and block s//4+26 exists.
  3. Decode jpg (cv2, BGR→RGB) → geometry.resample_rgb. Decode ids.png (UNCHANGED), assert the full-resolution values ⊆ {0..6,10,11} (check_raw_ids, data_v10.md §8.2 step 3), then resample_ids → capsules.label_parts, with the output asserted ⊆ model.valid_raw_ids("parts") minus 255. Without the source check, a stray raw value in 12..21 would pass straight through label_parts as a body-part id.
  4. Build the meta from the jsons (capsules_to_array, ground_z = min over player capsules of endpoint z − r, player_px/boss_px from the parts map).
  5. Write cond_ext → meta → cond, each with np.savez_compressed to *.tmp<pid> and then os.replace.
• VAE in the rank's main thread: WanVideoPipeline.from_pretrained(... ModelConfig(path=Wan2.2_VAE.pth)), then vae.model.float(). The PyTorch default TF32 flags are left alone (logged as cudnn.allow_tf32=True matmul.allow_tf32=False).
  • Input: uint8 → [−1,1] through a 256-entry LUT computed on the CPU exactly as infer.encode_video does. On CUDA, div_(127.5) becomes a multiply-by-reciprocal and can be 1 ulp off, which the LUT avoids.
  • Encode: vae.single_encode(x[None]). This is the body of vae.encode([x], tiled=False) without its 388 MB GPU→CPU→GPU round trip.
  • The latent is cast to bf16 and written to latents/<clip>.pt atomically. The index row is appended after the latent.
  • Checked: re-encoding with the official vae.encode([x_cpu], tiled=False) is bit-identical on all 8 spot clips.
• Resume: a clip is skipped if it has a row and all of its outputs exist. If all outputs exist but the row is missing, the worker rebuilds the row from the jsons and meta without decoding. Partial or tmp files are simply redone. --merge_only rebuilds clips.jsonl and runs.json. A rows file left with a torn last line by a kill is closed with a '\n' before appending (open_rows). Otherwise the first resumed row would be glued to the fragment, both would be dropped, and the final merge would fail on a "missing" clip.
• Stopping: kill <main pid> (SIGTERM) or Ctrl-C stops the run cleanly. Main turns SIGTERM into SystemExit, and a finally terminates and joins every rank. Ranks and their DataLoader workers also set PR_SET_PDEATHSIG (die_with_parent), so they exit even if main is SIGKILLed. Before this fix, killing main left the ranks running as orphans with PPID 1. They kept writing into the output directory and raced a resume. Re-run the same command to resume. A kill can leave *.tmp<pid> files behind. They never count as outputs, so the clip is redone, and verify warns about leftovers. Delete them with find <out> -name '*.tmp[0-9]*' -delete once no prep is running.
• Ranks get contiguous chunks of the sorted clip list, which keeps tar reads local and balances the load.
• --gpus lists indices within CUDA_VISIBLE_DEVICES.

Numbers (H200, measured)

MINI (4 runs, 200 clips, GPUs 0,1, 12 workers/GPU, idle machine; outputs/_smoke/prep/mini_build.log):

• 113 s wall, including about 10 s of startup.
• 0.997 clips/s/GPU steady, VAE-bound: VAE 0.98 s/clip, the main thread waited on data 0.002 s/clip.
• Worker CPU 3.95 s/clip: json 0.21, decode+resample 1.09, label_parts 1.94, npz 0.63. Total including the VAE process: 5.1 core-s/clip.
• Machine CPU 5.7 % of 192 cores. GPU peak 6.5 GiB/rank. VAE load 4.7 s.

Contention test (stress_*.log). A CPU-only --no_vae job ran 6 ranks × 12 workers (72 busy workers, 17.5 clips/s, no GPU use) next to the 2 VAE ranks, while other agents were also training on GPUs 2–7:

• The VAE ranks still ran 0.96–1.00 clips/s/GPU. The main thread waited 0.025 s/clip.
• Worker time rose to 4.6–5.4 s/clip, at 40 % machine CPU.
• Twelve workers per GPU leave more than 2× CPU headroom.

Extrapolation for the full run (12,800 clips, 8 GPUs):

• Wall time: 12,800 / (8 × 0.96–1.0) = 1,600–1,670 s, plus about 1 min of startup and merge, so ≈ 28–30 min.
• CPU: about 8 clips/s × 5–6 core-s ≈ 40–50 busy cores (~25 % of the machine).
• Disk read: ≈ 23 MB/clip, so about 185 MB/s.
• Output: v10 4.11 MB/clip and v9 4.97 MB/clip (latent 3.15, cond 0.69/1.37, cond_ext 0.21/0.40, meta 0.06), so ≈ 59 GB.
• Expected has_ext: v10 87 % / v9 84 % in MINI.

verify_prep on MINI (mini_verify.log, 141 s): 0 fails, 0 warnings.

• All 200 clips pass the per-clip checks.
• Player/boss px per frame at 832×480: v10 3,410/3,370, v9 3,479/4,853.
• 8 spot clips:
  • Re-encode is exact (max diff 0). Latent cell 0 equals encode(frame 0) at bf16 (exact fraction 1.0; rel L2 1.7e-3 is the bf16 storage error).
  • VAE recon PSNR 21.3–36.4 dB, mean 27.2. Photoreal R10_s1512 clips score 21–22 dB; a flat-shaded v9 look scores 36. The SPEC's 23–26 dB is only typical, so the check fails below 18 dB and warns outside 21–40 dB.
  • collapse_parts(cond) == resample_ids(ids.png) on every frame including extension frames, with 0 mismatching px. Re-running label_parts from the tar json gives an exact match, and from meta 1.0 agreement on character pixels.
• Capsule re-render from meta only (render_capsules):
  • v10 player: unoccluded median IoU 0.9978, pooled 0.9977, GT ⊆ render 1.0000. v10 boss: 0.9967.
  • v9 player: pooled IoU 0.65 because scene occlusion hides 35 % of the rendered px; on unoccluded frames it is still 0.9977.

Review-fix smoke (outputs/_smoke/prep/fixer/, GPUs 0,1):

• pytest.log: 26 passed in 10.4 s. A mutation check (stubbing die_with_parent / open_rows out) makes both new tests fail.
• kill_resume_smoke.log: prep on R10_s1501 + P9_golden_s2909, 20 clips each.
  • kill -TERM main after the first progress line: main exits 143 and prints the resume hint. All 15 descendants (2 ranks + 12 workers + tracker) are gone within 0.5 s.
  • kill -KILL main: main exits 137, and all 15 descendants exit through PDEATHSIG within 0.5 s.
  • Then a half row is appended to rows_rank0.jsonl without a newline, and the same command is re-run. The resume says 11 done / 29 to do and exits 0, and merge finds all 40 clips.
  • verify_prep --spot 2: 0 fails. The 1 warning is the 7 leftover .tmp<pid> files from the two kills. Re-encode is exact, PSNR is 29.8 and 23.1 dB, and the v10 render IoU is 0.998 for the player and 0.997 for the boss.
• rawscan_no_vae.log: --no_vae --clips_per_run 3 over all 256 runs, 768 clips, about 73k ids.png frames including extension frames. The new source-id check never fires, and the run took 83 s (10.7 clips/s on 48 CPU workers). The check costs 1.1 ms/frame, about 0.12 s/clip (~3 % of worker CPU).
• mini_verify.log: MINI re-verified with the updated verifier. Result: OK, 200 clips, 0 fails, 0 warns, 121 s. The 8 spot clips pass the source-id check, and the numbers are unchanged: PSNR 21.25–36.40 dB, v10 unoccluded player IoU 0.9978. New ext: line: v10 has_ext 87, of which 85 change look, and the DMD train pool keeps 1/43 v10 clips when same-look is required; v9 has_ext 84, of which 0 change look (41/41 kept).

Notes for consumers

• ext_look_change (OPEN DECISION for the lead / distill owner, SPEC §2 gap). v10_real switches the look at frame s+81, which is every clip boundary and also the first extension frame (cell 21). A look is a biome plus lighting. The switch changes the RGB, and it also re-lays out the static semantic map. An earlier version of this note said "the ids change only modestly, so DMD is unaffected". That was wrong. It came from data_v10.md §9, which said "the ids continue".

  • Measured on MINI cond+cond_ext (all 171 has_ext clips), 80→81 vs. other adjacent frames:

    subset	look change at 80→81	static-class histogram L1 at 80→81 (median / p90 / max)	other frame steps (median)	biggest jump in the clip is at 80→81	static px changing class (80→81 vs. within clip, median)
    v10	85/87	0.088 / 0.22 / 0.62	0.0026	78/87	0.107 vs 0.042
    v9	0/84	0.016	0.015	2/84	0.112 vs 0.109

    Example: 01_…__R10_s1512__003456 goes from 0.5 % to 25 % vegetation in one frame. The player and camera are continuous across the boundary.

  • Reviewer scan of all 12,800 clips: v10 has_ext 4,888, of which 4,772 change look, all at offset 81. v9 has_ext 5,897, of which 0 change look.

  • The stage-3 DMD train pool (has_ext, not dropped, train split) is ≈ 10,069 clips, ≈ 4,274 (42 %) of them with this pop at cell 21. N is in 21..27, so cell 21 is in every rollout. The bidirectional teacher never saw a mid-clip biome change, so these samples are out of its distribution.

  • What prep does: prep keeps SPEC §2 literally. has_ext checks episode and camera_variant only, and cond_ext is written for these clips. The decision therefore stays reversible without re-running the 30 min prep. Every row carries ext_look_change, and verify_prep prints the counts (ext: … dmd_pool_train_same_look=…).

  • Recommended: stage 3 drops these rows by adding and not r.get("ext_look_change", False) to the need_ext branch of gcdistill/data_gc.read_index. That leaves ≈ 5.8k train clips (≈ 5.7k v9, ≈ 100 v10), which is still more than 500 steps × 8 = 4,000 samples. The trade-off is that DMD then sees almost no v10_real looks. The alternative is to accept the pop and amend SPEC §2. Either way, the call belongs to the lead and is not made here. No v10 clip has a usable extension without the pop, because the switch lands exactly on frame s+81.

• ground_z is per frame and follows the walk cycle and the terrain. In MINI, ground_z[0] sits a median of 12–15 cm above the clip's minimum, and the range within a clip has p90 ≈ 130 cm (slopes and steps). SPEC §7 masking uses ground_z[0], which is fine, but don't expect it to be constant.
• MINI has 0 dropped clips (scene 01/02). The 348 buried clips are all in v10 scenes 10/07/09; test_plan_split_drop covers the drop flag path.

Commands

M=/opt/dlami/nvme/zhiyangdeng/maliang
ENV="DIFFSYNTH_SKIP_DOWNLOAD=True PYTHONPATH=$M/code/gc25d:$M/code/arope HF_HOME=/opt/dlami/nvme/zhiyangdeng/.cache/huggingface TMPDIR=$M/.tmp OMP_NUM_THREADS=8"
# tests
env $ENV $M/.venv/bin/python -m pytest -q $M/code/gc25d/tests/test_prep.py
# MINI (done)
env $ENV CUDA_VISIBLE_DEVICES=0,1 $M/.venv/bin/python $M/code/gc25d/prep_v10.py --src $M/data/GameCrafter-Data-v10/data \
    --out $M/data/gc25d_v10_mini --gpus 0,1 --runs R10_s1501,R10_s1512,P9C_autumn_s1941,P9_golden_s2909 --workers_per_gpu 12
env $ENV CUDA_VISIBLE_DEVICES=0,1 $M/.venv/bin/python $M/code/gc25d/verify_prep.py --root $M/data/gc25d_v10_mini --spot 8
# FULL (not run yet; ≈ 30 min, all 8 GPUs; resumable: just re-run the same command)
cd $M && env $ENV CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 nohup $M/.venv/bin/python $M/code/gc25d/prep_v10.py \
    --src $M/data/GameCrafter-Data-v10/data --out $M/data/gc25d_v10 --gpus 0,1,2,3,4,5,6,7 --workers_per_gpu 12 \
    > $M/outputs/_smoke/prep/full_build.log 2>&1 &
# to stop it: kill <that PID> (SIGTERM) — ranks and workers exit too; re-run the same command to resume
# then (≈ 15 min: phase B ≈ 4 min with 64 procs, 32 spot clips ≈ 10 min)
env $ENV CUDA_VISIBLE_DEVICES=0 $M/.venv/bin/python $M/code/gc25d/verify_prep.py --root $M/data/gc25d_v10 --spot 32 --workers 64