Blame for docs/impl/prep.md

# prep — GameCrafter-Data-v10 → `data/gc25d_v10/` (SPEC §1–§5)

Status 2026-10-22: code done and tested. MINI dataset built, verified and marked READY (`data/gc25d_v10_mini/READY`).
**The full prep has not been run yet**; its command is below.

**Open decision before stage 3: `ext_look_change`.** 42 % of the DMD pool has a semantic-map pop at cell 21. See *Notes for consumers*.

Review-fix pass (same day; MINI outputs unchanged, no rebuild needed):
* Source ids.png values are now asserted ⊆ {0..6,10,11}, in prep and in the verify spot check.
* Killing the main process no longer leaves orphan ranks: there is a SIGTERM handler, and ranks and workers set PDEATHSIG.
* A torn last row in `rows_rank*.jsonl` no longer breaks the merge on resume.
* The `ext_look_change` note has been corrected (it is a real pop), and verify prints the counts.

Smoke logs are in `outputs/_smoke/prep/fixer/`.

## What

| file | role |
|---|---|
| `code/gc25d/prep_v10.py` | the prep CLI: one spawned process per GPU; DataLoader workers do the CPU work; the main thread runs the VAE; atomic writes; resume; merge |
| `code/gc25d/verify_prep.py` | checks index, files and shapes on every clip, then GPU spot checks (VAE PSNR, bit-exact re-encode, cell 0, label_parts, capsule re-render IoU) |
| `code/gc25d/tests/test_prep.py` | 26 CPU tests (~10 s): block/prompt alignment, ext validity, split/drop, atomic write / resume / merge, torn-row resume, raw ids check, PDEATHSIG, VAE input LUT, one real clip (no VAE) |

The output layout follows SPEC §5 exactly. Additions beyond SPEC:
* `index/clips.jsonl` also has `look`, `max_frame_offset_cm`, `ext_look_change`, `mirrored_world`, `player_px_mean`, `boss_visible_frac`,
  `player_hidden_frac` (non-dead frames with 0 player px). `drop_reason` is `"buried_player"` or `null`.
* `meta/*.npz` also has `bot_mode` (str [F]) and `frames` (int32 [F], absolute frame index).
* `index/` also holds `rows_rank<r>.jsonl` (per-rank rows used for resume), `prep_args.json`, `prep_stats_<time>.json` (throughput) and
  `verify_report.json`.
* `latents/*.pt` stores `actions_block` as a torch uint8 tensor, so `torch.load(..., weights_only=True)` works.

## How (conventions that matter)
* Per clip, inside a worker (`OMP_NUM_THREADS=1`, `cv2.setNumThreads(1)`, one LRU-cached `RunTars` per run with shards indexed lazily):
  1. Read the per-frame tar jsons s…s+108.
  2. `has_ext` ⇔ all 109 frames exist, `episode` and `camera_variant` are constant, and block s//4+26 exists.
  3. Decode jpg (cv2, BGR→RGB) → `geometry.resample_rgb`. Decode ids.png (UNCHANGED), assert the full-resolution values ⊆ {0..6,10,11}
     (`check_raw_ids`, data_v10.md §8.2 step 3), then `resample_ids` → `capsules.label_parts`, with the output asserted ⊆
     `model.valid_raw_ids("parts")` minus 255. Without the source check, a stray raw value in 12..21 would pass straight through
     `label_parts` as a body-part id.
  4. Build the meta from the jsons (`capsules_to_array`, `ground_z` = min over player capsules of endpoint z − r, `player_px`/`boss_px` from
     the parts map).
  5. Write `cond_ext` → `meta` → `cond`, each with `np.savez_compressed` to `*.tmp<pid>` and then `os.replace`.
* VAE in the rank's main thread: `WanVideoPipeline.from_pretrained(... ModelConfig(path=Wan2.2_VAE.pth))`, then `vae.model.float()`. The
  PyTorch default TF32 flags are left alone (logged as `cudnn.allow_tf32=True matmul.allow_tf32=False`).
  * Input: uint8 → [−1,1] through a 256-entry LUT computed on the CPU exactly as `infer.encode_video` does. On CUDA, `div_(127.5)` becomes a
    multiply-by-reciprocal and can be 1 ulp off, which the LUT avoids.
  * Encode: `vae.single_encode(x[None])`. This is the body of `vae.encode([x], tiled=False)` without its 388 MB GPU→CPU→GPU round trip.
  * The latent is cast to bf16 and written to `latents/<clip>.pt` atomically. The index row is appended **after** the latent.
  * **Checked:** re-encoding with the official `vae.encode([x_cpu], tiled=False)` is bit-identical on all 8 spot clips.
* Resume: a clip is skipped if it has a row and all of its outputs exist. If all outputs exist but the row is missing, the worker rebuilds the
  row from the jsons and meta without decoding. Partial or tmp files are simply redone. `--merge_only` rebuilds `clips.jsonl` and `runs.json`.
  A rows file left with a torn last line by a kill is closed with a `'\n'` before appending (`open_rows`). Otherwise the first resumed row
  would be glued to the fragment, both would be dropped, and the final merge would fail on a "missing" clip.
* Stopping: `kill <main pid>` (SIGTERM) or Ctrl-C stops the run cleanly. Main turns SIGTERM into SystemExit, and a `finally` terminates and
  joins every rank. Ranks and their DataLoader workers also set `PR_SET_PDEATHSIG` (`die_with_parent`), so they exit even if main is
  SIGKILLed. Before this fix, killing main left the ranks running as orphans with PPID 1. They kept writing into the output directory and
  raced a resume. Re-run the same command to resume. A kill can leave `*.tmp<pid>` files behind. They never count as outputs, so the clip
  is redone, and verify warns about leftovers. Delete them with `find <out> -name '*.tmp[0-9]*' -delete` once no prep is running.
* Ranks get contiguous chunks of the sorted clip list, which keeps tar reads local and balances the load.
* `--gpus` lists indices within `CUDA_VISIBLE_DEVICES`.

## Numbers (H200, measured)
**MINI** (4 runs, 200 clips, GPUs 0,1, 12 workers/GPU, idle machine; `outputs/_smoke/prep/mini_build.log`):
* 113 s wall, including about 10 s of startup.
* **0.997 clips/s/GPU steady**, VAE-bound: VAE 0.98 s/clip, the main thread waited on data 0.002 s/clip.
* Worker CPU 3.95 s/clip: json 0.21, decode+resample 1.09, label_parts 1.94, npz 0.63. Total including the VAE process: **5.1 core-s/clip**.
* Machine CPU 5.7 % of 192 cores. GPU peak 6.5 GiB/rank. VAE load 4.7 s.

**Contention test** (`stress_*.log`). A CPU-only `--no_vae` job ran 6 ranks × 12 workers (72 busy workers, 17.5 clips/s, no GPU use) next to
the 2 VAE ranks, while other agents were also training on GPUs 2–7:
* The VAE ranks still ran **0.96–1.00 clips/s/GPU**. The main thread waited 0.025 s/clip.
* Worker time rose to 4.6–5.4 s/clip, at 40 % machine CPU.
* Twelve workers per GPU leave more than 2× CPU headroom.

**Extrapolation for the full run** (12,800 clips, 8 GPUs):
* Wall time: 12,800 / (8 × 0.96–1.0) = 1,600–1,670 s, plus about 1 min of startup and merge, so **≈ 28–30 min**.
* CPU: about 8 clips/s × 5–6 core-s ≈ **40–50 busy cores** (~25 % of the machine).
* Disk read: ≈ 23 MB/clip, so about 185 MB/s.
* Output: v10 4.11 MB/clip and v9 4.97 MB/clip (latent 3.15, cond 0.69/1.37, cond_ext 0.21/0.40, meta 0.06), so **≈ 59 GB**.
* Expected `has_ext`: v10 87 % / v9 84 % in MINI.

**verify_prep on MINI** (`mini_verify.log`, 141 s): 0 fails, 0 warnings.
* All 200 clips pass the per-clip checks.
* Player/boss px per frame at 832×480: v10 3,410/3,370, v9 3,479/4,853.
* 8 spot clips:
  * Re-encode is exact (max diff 0). Latent cell 0 equals encode(frame 0) at bf16 (exact fraction 1.0; rel L2 1.7e-3 is the bf16 storage
    error).
  * VAE recon PSNR 21.3–36.4 dB, mean 27.2. Photoreal R10_s1512 clips score 21–22 dB; a flat-shaded v9 look scores 36. The SPEC's
    23–26 dB is only typical, so the check fails below 18 dB and warns outside 21–40 dB.
  * `collapse_parts(cond) == resample_ids(ids.png)` on every frame including extension frames, with 0 mismatching px. Re-running
    `label_parts` from the tar json gives an exact match, and from meta 1.0 agreement on character pixels.
* Capsule re-render from meta only (`render_capsules`):
  * v10 player: unoccluded median IoU **0.9978**, pooled 0.9977, GT ⊆ render 1.0000. v10 boss: 0.9967.
  * v9 player: pooled IoU 0.65 because scene occlusion hides 35 % of the rendered px; on unoccluded frames it is still 0.9977.

**Review-fix smoke** (`outputs/_smoke/prep/fixer/`, GPUs 0,1):
* `pytest.log`: 26 passed in 10.4 s. A mutation check (stubbing `die_with_parent` / `open_rows` out) makes both new tests fail.
* `kill_resume_smoke.log`: prep on R10_s1501 + P9_golden_s2909, 20 clips each.
  * `kill -TERM main` after the first progress line: main exits 143 and prints the resume hint. All 15 descendants (2 ranks + 12 workers +
    tracker) are gone within 0.5 s.
  * `kill -KILL main`: main exits 137, and all 15 descendants exit through PDEATHSIG within 0.5 s.
  * Then a half row is appended to `rows_rank0.jsonl` without a newline, and the same command is re-run. The resume says 11 done / 29 to do
    and exits 0, and `merge` finds all 40 clips.
  * `verify_prep --spot 2`: 0 fails. The 1 warning is the 7 leftover `.tmp<pid>` files from the two kills. Re-encode is exact, PSNR is 29.8
    and 23.1 dB, and the v10 render IoU is 0.998 for the player and 0.997 for the boss.
* `rawscan_no_vae.log`: `--no_vae --clips_per_run 3` over **all 256 runs**, 768 clips, about 73k ids.png frames including extension frames.
  The new source-id check never fires, and the run took 83 s (10.7 clips/s on 48 CPU workers). The check costs 1.1 ms/frame, about
  0.12 s/clip (~3 % of worker CPU).
* `mini_verify.log`: MINI re-verified with the updated verifier. Result: OK, 200 clips, 0 fails, 0 warns, 121 s. The 8 spot clips pass the
  source-id check, and the numbers are unchanged: PSNR 21.25–36.40 dB, v10 unoccluded player IoU 0.9978. New `ext:` line: v10 has_ext 87,
  of which 85 change look, and the DMD train pool keeps **1/43** v10 clips when same-look is required; v9 has_ext 84, of which 0 change look
  (41/41 kept).

## Notes for consumers
* **`ext_look_change` (OPEN DECISION for the lead / distill owner, SPEC §2 gap).** v10_real switches the look at frame s+81, which is every
  clip boundary and also the first extension frame (cell 21). A look is a biome plus lighting. The switch changes the RGB, **and it also
  re-lays out the static semantic map.** An earlier version of this note said "the ids change only modestly, so DMD is unaffected". That was
  wrong. It came from data_v10.md §9, which said "the ids continue".
  * Measured on MINI cond+cond_ext (all 171 has_ext clips), 80→81 vs. other adjacent frames:

    | subset | look change at 80→81 | static-class histogram L1 at 80→81 (median / p90 / max) | other frame steps (median) | biggest jump in the clip is at 80→81 | static px changing class (80→81 vs. within clip, median) |
    |---|---|---|---|---|---|
    | v10 | 85/87 | 0.088 / 0.22 / 0.62 | 0.0026 | 78/87 | 0.107 vs 0.042 |
    | v9 | 0/84 | 0.016 | 0.015 | 2/84 | 0.112 vs 0.109 |

    Example: `01_…__R10_s1512__003456` goes from 0.5 % to 25 % vegetation in one frame. The player and camera are continuous across the boundary.
  * Reviewer scan of all 12,800 clips: v10 has_ext 4,888, of which 4,772 change look, all at offset 81. v9 has_ext 5,897, of which 0 change look.
  * The stage-3 DMD train pool (has_ext, not dropped, train split) is ≈ 10,069 clips, ≈ 4,274 (42 %) of them with this pop at cell 21. N is in
    21..27, so cell 21 is in every rollout. The bidirectional teacher never saw a mid-clip biome change, so these samples are out of its
    distribution.
  * **What prep does:** prep keeps SPEC §2 literally. `has_ext` checks episode and camera_variant only, and cond_ext is written for these clips.
    The decision therefore stays reversible without re-running the 30 min prep. Every row carries `ext_look_change`, and `verify_prep` prints
    the counts (`ext: … dmd_pool_train_same_look=…`).
  * **Recommended:** stage 3 drops these rows by adding `and not r.get("ext_look_change", False)` to the `need_ext` branch of
    `gcdistill/data_gc.read_index`. That leaves ≈ 5.8k train clips (≈ 5.7k v9, ≈ 100 v10), which is still more than 500 steps × 8 = 4,000
    samples. The trade-off is that DMD then sees almost no v10_real looks. The alternative is to accept the pop and amend SPEC §2. Either way,
    the call belongs to the lead and is not made here. No v10 clip has a usable extension without the pop, because the switch lands exactly
    on frame s+81.
* `ground_z` is per frame and follows the walk cycle and the terrain. In MINI, `ground_z[0]` sits a median of 12–15 cm above the clip's
  minimum, and the range within a clip has p90 ≈ 130 cm (slopes and steps). SPEC §7 masking uses `ground_z[0]`, which is fine, but don't
  expect it to be constant.
* MINI has 0 dropped clips (scene 01/02). The 348 buried clips are all in v10 scenes 10/07/09; `test_plan_split_drop` covers the drop flag
  path.

## Commands
```bash
M=/opt/dlami/nvme/zhiyangdeng/maliang
ENV="DIFFSYNTH_SKIP_DOWNLOAD=True PYTHONPATH=$M/code/gc25d:$M/code/arope HF_HOME=/opt/dlami/nvme/zhiyangdeng/.cache/huggingface TMPDIR=$M/.tmp OMP_NUM_THREADS=8"
# tests
env $ENV $M/.venv/bin/python -m pytest -q $M/code/gc25d/tests/test_prep.py
# MINI (done)
env $ENV CUDA_VISIBLE_DEVICES=0,1 $M/.venv/bin/python $M/code/gc25d/prep_v10.py --src $M/data/GameCrafter-Data-v10/data \
    --out $M/data/gc25d_v10_mini --gpus 0,1 --runs R10_s1501,R10_s1512,P9C_autumn_s1941,P9_golden_s2909 --workers_per_gpu 12
env $ENV CUDA_VISIBLE_DEVICES=0,1 $M/.venv/bin/python $M/code/gc25d/verify_prep.py --root $M/data/gc25d_v10_mini --spot 8
# FULL (not run yet; ≈ 30 min, all 8 GPUs; resumable: just re-run the same command)
cd $M && env $ENV CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 nohup $M/.venv/bin/python $M/code/gc25d/prep_v10.py \
    --src $M/data/GameCrafter-Data-v10/data --out $M/data/gc25d_v10 --gpus 0,1,2,3,4,5,6,7 --workers_per_gpu 12 \
    > $M/outputs/_smoke/prep/full_build.log 2>&1 &
# to stop it: kill <that PID> (SIGTERM) — ranks and workers exit too; re-run the same command to resume
# then (≈ 15 min: phase B ≈ 4 min with 64 procs, 32 spot clips ≈ 10 min)
env $ENV CUDA_VISIBLE_DEVICES=0 $M/.venv/bin/python $M/code/gc25d/verify_prep.py --root $M/data/gc25d_v10 --spot 32 --workers 64
```