| # prep — GameCrafter-Data-v10 → `data/gc25d_v10/` (SPEC §1–§5) |
| |
| Status 2026-10-22: code done and tested. MINI dataset built, verified and marked READY (`data/gc25d_v10_mini/READY`). |
| **The full prep has not been run yet**; its command is below. |
| |
| **Open decision before stage 3: `ext_look_change`.** 42 % of the DMD pool has a semantic-map pop at cell 21. See *Notes for consumers*. |
| |
| Review-fix pass (same day; MINI outputs unchanged, no rebuild needed): |
| * Source ids.png values are now asserted ⊆ {0..6,10,11}, in prep and in the verify spot check. |
| * Killing the main process no longer leaves orphan ranks: there is a SIGTERM handler, and ranks and workers set PDEATHSIG. |
| * A torn last row in `rows_rank*.jsonl` no longer breaks the merge on resume. |
| * The `ext_look_change` note has been corrected (it is a real pop), and verify prints the counts. |
|
|
| Smoke logs are in `outputs/_smoke/prep/fixer/`. |
|
|
| ## What |
|
|
| | file | role | |
| |---|---| |
| | `code/gc25d/prep_v10.py` | the prep CLI: one spawned process per GPU; DataLoader workers do the CPU work; the main thread runs the VAE; atomic writes; resume; merge | |
| | `code/gc25d/verify_prep.py` | checks index, files and shapes on every clip, then GPU spot checks (VAE PSNR, bit-exact re-encode, cell 0, label_parts, capsule re-render IoU) | |
| | `code/gc25d/tests/test_prep.py` | 26 CPU tests (~10 s): block/prompt alignment, ext validity, split/drop, atomic write / resume / merge, torn-row resume, raw ids check, PDEATHSIG, VAE input LUT, one real clip (no VAE) | |
|
|
| The output layout follows SPEC §5 exactly. Additions beyond SPEC: |
| * `index/clips.jsonl` also has `look`, `max_frame_offset_cm`, `ext_look_change`, `mirrored_world`, `player_px_mean`, `boss_visible_frac`, |
| `player_hidden_frac` (non-dead frames with 0 player px). `drop_reason` is `"buried_player"` or `null`. |
| * `meta/*.npz` also has `bot_mode` (str [F]) and `frames` (int32 [F], absolute frame index). |
| * `index/` also holds `rows_rank<r>.jsonl` (per-rank rows used for resume), `prep_args.json`, `prep_stats_<time>.json` (throughput) and |
| `verify_report.json`. |
| * `latents/*.pt` stores `actions_block` as a torch uint8 tensor, so `torch.load(..., weights_only=True)` works. |
|
|
| ## How (conventions that matter) |
| * Per clip, inside a worker (`OMP_NUM_THREADS=1`, `cv2.setNumThreads(1)`, one LRU-cached `RunTars` per run with shards indexed lazily): |
| 1. Read the per-frame tar jsons s…s+108. |
| 2. `has_ext` ⇔ all 109 frames exist, `episode` and `camera_variant` are constant, and block s//4+26 exists. |
| 3. Decode jpg (cv2, BGR→RGB) → `geometry.resample_rgb`. Decode ids.png (UNCHANGED), assert the full-resolution values ⊆ {0..6,10,11} |
| (`check_raw_ids`, data_v10.md §8.2 step 3), then `resample_ids` → `capsules.label_parts`, with the output asserted ⊆ |
| `model.valid_raw_ids("parts")` minus 255. Without the source check, a stray raw value in 12..21 would pass straight through |
| `label_parts` as a body-part id. |
| 4. Build the meta from the jsons (`capsules_to_array`, `ground_z` = min over player capsules of endpoint z − r, `player_px`/`boss_px` from |
| the parts map). |
| 5. Write `cond_ext` → `meta` → `cond`, each with `np.savez_compressed` to `*.tmp<pid>` and then `os.replace`. |
| * VAE in the rank's main thread: `WanVideoPipeline.from_pretrained(... ModelConfig(path=Wan2.2_VAE.pth))`, then `vae.model.float()`. The |
| PyTorch default TF32 flags are left alone (logged as `cudnn.allow_tf32=True matmul.allow_tf32=False`). |
| * Input: uint8 → [−1,1] through a 256-entry LUT computed on the CPU exactly as `infer.encode_video` does. On CUDA, `div_(127.5)` becomes a |
| multiply-by-reciprocal and can be 1 ulp off, which the LUT avoids. |
| * Encode: `vae.single_encode(x[None])`. This is the body of `vae.encode([x], tiled=False)` without its 388 MB GPU→CPU→GPU round trip. |
| * The latent is cast to bf16 and written to `latents/<clip>.pt` atomically. The index row is appended **after** the latent. |
| * **Checked:** re-encoding with the official `vae.encode([x_cpu], tiled=False)` is bit-identical on all 8 spot clips. |
| * Resume: a clip is skipped if it has a row and all of its outputs exist. If all outputs exist but the row is missing, the worker rebuilds the |
| row from the jsons and meta without decoding. Partial or tmp files are simply redone. `--merge_only` rebuilds `clips.jsonl` and `runs.json`. |
| A rows file left with a torn last line by a kill is closed with a `'\n'` before appending (`open_rows`). Otherwise the first resumed row |
| would be glued to the fragment, both would be dropped, and the final merge would fail on a "missing" clip. |
| * Stopping: `kill <main pid>` (SIGTERM) or Ctrl-C stops the run cleanly. Main turns SIGTERM into SystemExit, and a `finally` terminates and |
| joins every rank. Ranks and their DataLoader workers also set `PR_SET_PDEATHSIG` (`die_with_parent`), so they exit even if main is |
| SIGKILLed. Before this fix, killing main left the ranks running as orphans with PPID 1. They kept writing into the output directory and |
| raced a resume. Re-run the same command to resume. A kill can leave `*.tmp<pid>` files behind. They never count as outputs, so the clip |
| is redone, and verify warns about leftovers. Delete them with `find <out> -name '*.tmp[0-9]*' -delete` once no prep is running. |
| * Ranks get contiguous chunks of the sorted clip list, which keeps tar reads local and balances the load. |
| * `--gpus` lists indices within `CUDA_VISIBLE_DEVICES`. |
| |
| ## Numbers (H200, measured) |
| **MINI** (4 runs, 200 clips, GPUs 0,1, 12 workers/GPU, idle machine; `outputs/_smoke/prep/mini_build.log`): |
| * 113 s wall, including about 10 s of startup. |
| * **0.997 clips/s/GPU steady**, VAE-bound: VAE 0.98 s/clip, the main thread waited on data 0.002 s/clip. |
| * Worker CPU 3.95 s/clip: json 0.21, decode+resample 1.09, label_parts 1.94, npz 0.63. Total including the VAE process: **5.1 core-s/clip**. |
| * Machine CPU 5.7 % of 192 cores. GPU peak 6.5 GiB/rank. VAE load 4.7 s. |
| |
| **Contention test** (`stress_*.log`). A CPU-only `--no_vae` job ran 6 ranks × 12 workers (72 busy workers, 17.5 clips/s, no GPU use) next to |
| the 2 VAE ranks, while other agents were also training on GPUs 2–7: |
| * The VAE ranks still ran **0.96–1.00 clips/s/GPU**. The main thread waited 0.025 s/clip. |
| * Worker time rose to 4.6–5.4 s/clip, at 40 % machine CPU. |
| * Twelve workers per GPU leave more than 2× CPU headroom. |
|
|
| **Extrapolation for the full run** (12,800 clips, 8 GPUs): |
| * Wall time: 12,800 / (8 × 0.96–1.0) = 1,600–1,670 s, plus about 1 min of startup and merge, so **≈ 28–30 min**. |
| * CPU: about 8 clips/s × 5–6 core-s ≈ **40–50 busy cores** (~25 % of the machine). |
| * Disk read: ≈ 23 MB/clip, so about 185 MB/s. |
| * Output: v10 4.11 MB/clip and v9 4.97 MB/clip (latent 3.15, cond 0.69/1.37, cond_ext 0.21/0.40, meta 0.06), so **≈ 59 GB**. |
| * Expected `has_ext`: v10 87 % / v9 84 % in MINI. |
|
|
| **verify_prep on MINI** (`mini_verify.log`, 141 s): 0 fails, 0 warnings. |
| * All 200 clips pass the per-clip checks. |
| * Player/boss px per frame at 832×480: v10 3,410/3,370, v9 3,479/4,853. |
| * 8 spot clips: |
| * Re-encode is exact (max diff 0). Latent cell 0 equals encode(frame 0) at bf16 (exact fraction 1.0; rel L2 1.7e-3 is the bf16 storage |
| error). |
| * VAE recon PSNR 21.3–36.4 dB, mean 27.2. Photoreal R10_s1512 clips score 21–22 dB; a flat-shaded v9 look scores 36. The SPEC's |
| 23–26 dB is only typical, so the check fails below 18 dB and warns outside 21–40 dB. |
| * `collapse_parts(cond) == resample_ids(ids.png)` on every frame including extension frames, with 0 mismatching px. Re-running |
| `label_parts` from the tar json gives an exact match, and from meta 1.0 agreement on character pixels. |
| * Capsule re-render from meta only (`render_capsules`): |
| * v10 player: unoccluded median IoU **0.9978**, pooled 0.9977, GT ⊆ render 1.0000. v10 boss: 0.9967. |
| * v9 player: pooled IoU 0.65 because scene occlusion hides 35 % of the rendered px; on unoccluded frames it is still 0.9977. |
| |
| **Review-fix smoke** (`outputs/_smoke/prep/fixer/`, GPUs 0,1): |
| * `pytest.log`: 26 passed in 10.4 s. A mutation check (stubbing `die_with_parent` / `open_rows` out) makes both new tests fail. |
| * `kill_resume_smoke.log`: prep on R10_s1501 + P9_golden_s2909, 20 clips each. |
| * `kill -TERM main` after the first progress line: main exits 143 and prints the resume hint. All 15 descendants (2 ranks + 12 workers + |
| tracker) are gone within 0.5 s. |
| * `kill -KILL main`: main exits 137, and all 15 descendants exit through PDEATHSIG within 0.5 s. |
| * Then a half row is appended to `rows_rank0.jsonl` without a newline, and the same command is re-run. The resume says 11 done / 29 to do |
| and exits 0, and `merge` finds all 40 clips. |
| * `verify_prep --spot 2`: 0 fails. The 1 warning is the 7 leftover `.tmp<pid>` files from the two kills. Re-encode is exact, PSNR is 29.8 |
| and 23.1 dB, and the v10 render IoU is 0.998 for the player and 0.997 for the boss. |
| * `rawscan_no_vae.log`: `--no_vae --clips_per_run 3` over **all 256 runs**, 768 clips, about 73k ids.png frames including extension frames. |
| The new source-id check never fires, and the run took 83 s (10.7 clips/s on 48 CPU workers). The check costs 1.1 ms/frame, about |
| 0.12 s/clip (~3 % of worker CPU). |
| * `mini_verify.log`: MINI re-verified with the updated verifier. Result: OK, 200 clips, 0 fails, 0 warns, 121 s. The 8 spot clips pass the |
| source-id check, and the numbers are unchanged: PSNR 21.25–36.40 dB, v10 unoccluded player IoU 0.9978. New `ext:` line: v10 has_ext 87, |
| of which 85 change look, and the DMD train pool keeps **1/43** v10 clips when same-look is required; v9 has_ext 84, of which 0 change look |
| (41/41 kept). |
| |
| ## Notes for consumers |
| * **`ext_look_change` (OPEN DECISION for the lead / distill owner, SPEC §2 gap).** v10_real switches the look at frame s+81, which is every |
| clip boundary and also the first extension frame (cell 21). A look is a biome plus lighting. The switch changes the RGB, **and it also |
| re-lays out the static semantic map.** An earlier version of this note said "the ids change only modestly, so DMD is unaffected". That was |
| wrong. It came from data_v10.md §9, which said "the ids continue". |
| * Measured on MINI cond+cond_ext (all 171 has_ext clips), 80→81 vs. other adjacent frames: |
|
|
| | subset | look change at 80→81 | static-class histogram L1 at 80→81 (median / p90 / max) | other frame steps (median) | biggest jump in the clip is at 80→81 | static px changing class (80→81 vs. within clip, median) | |
| |---|---|---|---|---|---| |
| | v10 | 85/87 | 0.088 / 0.22 / 0.62 | 0.0026 | 78/87 | 0.107 vs 0.042 | |
| | v9 | 0/84 | 0.016 | 0.015 | 2/84 | 0.112 vs 0.109 | |
| |
| Example: `01_…__R10_s1512__003456` goes from 0.5 % to 25 % vegetation in one frame. The player and camera are continuous across the boundary. |
| * Reviewer scan of all 12,800 clips: v10 has_ext 4,888, of which 4,772 change look, all at offset 81. v9 has_ext 5,897, of which 0 change look. |
| * The stage-3 DMD train pool (has_ext, not dropped, train split) is ≈ 10,069 clips, ≈ 4,274 (42 %) of them with this pop at cell 21. N is in |
| 21..27, so cell 21 is in every rollout. The bidirectional teacher never saw a mid-clip biome change, so these samples are out of its |
| distribution. |
| * **What prep does:** prep keeps SPEC §2 literally. `has_ext` checks episode and camera_variant only, and cond_ext is written for these clips. |
| The decision therefore stays reversible without re-running the 30 min prep. Every row carries `ext_look_change`, and `verify_prep` prints |
| the counts (`ext: … dmd_pool_train_same_look=…`). |
| * **Recommended:** stage 3 drops these rows by adding `and not r.get("ext_look_change", False)` to the `need_ext` branch of |
| `gcdistill/data_gc.read_index`. That leaves ≈ 5.8k train clips (≈ 5.7k v9, ≈ 100 v10), which is still more than 500 steps × 8 = 4,000 |
| samples. The trade-off is that DMD then sees almost no v10_real looks. The alternative is to accept the pop and amend SPEC §2. Either way, |
| the call belongs to the lead and is not made here. No v10 clip has a usable extension without the pop, because the switch lands exactly |
| on frame s+81. |
| * `ground_z` is per frame and follows the walk cycle and the terrain. In MINI, `ground_z[0]` sits a median of 12–15 cm above the clip's |
| minimum, and the range within a clip has p90 ≈ 130 cm (slopes and steps). SPEC §7 masking uses `ground_z[0]`, which is fine, but don't |
| expect it to be constant. |
| * MINI has 0 dropped clips (scene 01/02). The 348 buried clips are all in v10 scenes 10/07/09; `test_plan_split_drop` covers the drop flag |
| path. |
| |
| ## Commands |
| ```bash |
| M=/opt/dlami/nvme/zhiyangdeng/maliang |
| ENV="DIFFSYNTH_SKIP_DOWNLOAD=True PYTHONPATH=$M/code/gc25d:$M/code/arope HF_HOME=/opt/dlami/nvme/zhiyangdeng/.cache/huggingface TMPDIR=$M/.tmp OMP_NUM_THREADS=8" |
| # tests |
| env $ENV $M/.venv/bin/python -m pytest -q $M/code/gc25d/tests/test_prep.py |
| # MINI (done) |
| env $ENV CUDA_VISIBLE_DEVICES=0,1 $M/.venv/bin/python $M/code/gc25d/prep_v10.py --src $M/data/GameCrafter-Data-v10/data \ |
| --out $M/data/gc25d_v10_mini --gpus 0,1 --runs R10_s1501,R10_s1512,P9C_autumn_s1941,P9_golden_s2909 --workers_per_gpu 12 |
| env $ENV CUDA_VISIBLE_DEVICES=0,1 $M/.venv/bin/python $M/code/gc25d/verify_prep.py --root $M/data/gc25d_v10_mini --spot 8 |
| # FULL (not run yet; ≈ 30 min, all 8 GPUs; resumable: just re-run the same command) |
| cd $M && env $ENV CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 nohup $M/.venv/bin/python $M/code/gc25d/prep_v10.py \ |
| --src $M/data/GameCrafter-Data-v10/data --out $M/data/gc25d_v10 --gpus 0,1,2,3,4,5,6,7 --workers_per_gpu 12 \ |
| > $M/outputs/_smoke/prep/full_build.log 2>&1 & |
| # to stop it: kill <that PID> (SIGTERM) — ranks and workers exit too; re-run the same command to resume |
| # then (≈ 15 min: phase B ≈ 4 min with 64 procs, 32 spot clips ≈ 10 min) |
| env $ENV CUDA_VISIBLE_DEVICES=0 $M/.venv/bin/python $M/code/gc25d/verify_prep.py --root $M/data/gc25d_v10 --spot 32 --workers 64 |
| ``` |
|
|
