# sft — bidirectional gc25d teacher on v10 (SPEC §6–§7)

Status 2026-10-22: code done, CPU-tested (30 tests) and smoke-tested with real weights on `data/gc25d_v10_mini` (4×H200, GPUs 2–5).
A review-fix pass then made resume exact (data position and lineage) and added data-wait timing. See "Review fixes" at the end.
**The full run has not been started.** It needs `data/gc25d_v10` (full prep), and its command is at the end.

## What

| file | role |
|---|---|
| `code/gc25d/dataset.py` | `GcLatentDataset` (latents + parts ids + text table + the three dropouts + world-anchored UNKNOWN masking) and `gc_collate` |
| `code/gc25d/train.py` | fork of ActionRoPE `actionrope/train.py` (same section order): `GcTrainModule`, `sem_param_groups`, `warmup_lambda`, `run_validation` (cond / uncond / null-text), export through `gcdistill.gc_dit.save_gc` + `check_gc_ckpt`, rotating `save_state`, resume / warm start |
| `scripts/train_sft.sh` | launcher: env vars → accelerate yaml (8×GA2 and 8×GA1 use the existing files, other combinations are generated as `configs/accelerate_zero2_<N>gpu_ga<G>.yaml`), `tee -a $OUT/train.log`, refuses a second job on the same OUT |
| `code/gc25d/sample.py` | offline eval: TI2V sampling (`infer.sample` + `model.gc_forward`) on val clips → mp4 `GT \| generated \| cond (parts coloured)` + per-frame PSNR/SSIM json + `metrics.json` |
| `code/gc25d/tests/test_dataset.py` | 15 tests: contract, split/drop/limit, val never drops, `ids` vocab, dropout rates/independence/determinism, masking (world-anchored, characters untouched, coverage, NaN ⇒ masked, speed), collate, real mini clip |
| `code/gc25d/tests/test_train.py` | 15 tests: tiny WanModel forward/backward through `GcTrainModule`, manual loss + frame-0 exclusion, `sem_rms` grows from 0, param groups + warmup, CLI defaults = SPEC §7, val variants, export keys/metadata, lineage metadata, `epoch_steps` = ceil, `resume_position` (exact / epoch end / geometry changed / legacy), `init_ckpt_of_state`, CSV header rotation, and **real accelerate on CPU**: the seedable-sampler permutation depends only on (seed, epoch), `skip_first_batches` continues the uninterrupted order, and the sync pattern is `[F,T,F,T,T]` |

Nothing under `code/arope/` was touched. `model.py`, `capsules.py` and `gcdistill/gc_dit.py` are unchanged; they are only imported.

## How (conventions that matter)

### Dataset (`dataset.py`)
* Sample = `input_latents` bf16 [48,21,30,52], `context` bf16 [21,512,4096], `context_ids` int64 [21], `sem_ids` uint8 [81,480,832],
  `clip_id`, `dropped = {"text": bool, "cond": bool, "mask": float}`. Here `mask` is the fraction of all 81×480×832 pixels that the mask set to 255.
* `key_to_id = {k: i for i, k in enumerate(sorted(text_table))}`. This equals `infer._KEY_TO_ID` (tested).
* Rows come from `index/clips.jsonl` filtered by `split`. Rows with `drop=true` are excluded unless `include_dropped=True`.
  `limit` shuffles with `Random(seed)`, keeps the first `limit` rows and sorts them, as `AropeLatentDataset` does.
  **The val split forces all three dropout probabilities to 0.**
* `vocab="ids"` ⇒ `model.collapse_parts` after masking.
* RNG: each sample draws from `random.Random(f"gc25d:{seed}:{epoch}:{index}")`. It always draws three uniforms (text, cond, mask) in
  that order, then the mask parameters. Results are identical for a given (seed, epoch, index), whatever the worker count or rank.
  The trainer calls `train_ds.set_epoch(epoch)` itself before every epoch. With the seedable sampler, accelerate's `DataLoaderShard.set_epoch`
  only reaches the sampler and never the dataset. **`persistent_workers=False`** because persistent workers would keep a stale epoch.
* Dropouts, as in SPEC §7: text p=0.1 replaces all 21 cells with `NULL_PROMPT`. cond p=0.05 sets the ids to all 255 (the npz is not even
  read), independently of text. mask p=0.5 applies only when cond was not dropped.
* World-anchored UNKNOWN mask (SPEC §7, `apply_unknown_mask`):
  * 1 half-plane with p .7, 2 with p .3. θ ~ U[0,2π) in world XY and q ~ U(0.05, 0.40).
  * The ground plane is Z = `meta.ground_z[0]`. If that value is NaN the code falls back to the nanmedian, and if there is none it skips the mask.
  * Ground points come from `capsules.ground_points` on the 4×4 pooled grid, then nearest-upsampled ×4. `ground_points` only accepts integer
    dst pixels and the geometric cell centre (4j+1.5, 4i+1.5) falls between pixels, so cell (i, j) uses dst pixel **(4j+2, 4i+2)**. That is half a dst pixel, under 1 source pixel.
  * τ = the (1−q) quantile of the frame-0 static pixels' projections onto θ, weighted per pixel. Every frame then sets static pixels (raw ≤ 6)
    to 255 when their ground point is beyond any plane or is NaN. Characters (10..21) are never touched.
  * Cost is ≈0.27–0.37 s per masked sample (81 frames × 25k rays). An unmasked sample takes 0.09 s. Training uses 4 workers per rank.

### Trainer (`train.py`)
The section order follows upstream (args / model / export+log / validation / main), so `diff -u ref/arope_git/ActionRoPE/actionrope/train.py
code/gc25d/train.py` is readable. These parts are kept verbatim:
* `Accelerator(gradient_accumulation_steps)` and `set_seed(device_specific=True)`.
* The DS overrides: `--grad_clip` is written into `deepspeed_config.gradient_clipping`, and a GA mismatch with the yaml ⇒ SystemExit.
* `LambdaLR` is stepped manually, and only on `sync_gradients`.
* The per-rank `cpu_gen` timestep `tid ~ U{0..999}` (seed·1000 + rank + start_step).
* The `accumulate` block, `dit.freqs` moved to the GPU, `config.json`/`done.json`, and the CSV + tensorboard logs.

Changes from upstream:
* `GcTrainModule.forward`: `x_t = add_noise`, then frame 0 is replaced by clean x0, then `model.gc_forward(..., first_frame_cond, context_ids,
  grad ckpt when training, aux)`. Frame 0 is dropped from the loss, and `loss = MSE × training_weight(t)`. The returned stats are `{mse, timestep, sem_rms, x_rms}`.
* Optimizer: `AdamW(sem_param_groups(module, lr, cond_lr, wd))` = [all DiT params at `--lr`, `dit.gc_sem_cond.*` at `--cond_lr`]. The betas
  stay at the torch defaults (0.9, 0.999). The warmup factor multiplies both groups. `sem_param_groups` takes a `prefix` (default
  `dit.gc_sem_cond.`) so the distill runners can reuse it.
* Validation: fixed `val_n` val clips (`limit` + seed) × `--val_timesteps` × fixed noise. Each (clip, t) runs three forwards with **the same noise**:
  * cond: true ids and true text;
  * uncond: ids all 255;
  * nulltext: `NULL_PROMPT` × 21.

  `val_log.csv` holds `step, timestep, mse, mse_uncond, mse_nulltext, gap_uncond, gap_nulltext, n`, and tensorboard gets `val/*_t{t}` and `val/*_mean`.
* Export: `gcdistill.gc_dit.save_gc` writes 831 BF16 keys with the `dit.` prefix stripped, through `.tmp` + rename and chmod 644. It is
  followed by `check_gc_ckpt` and a vocab check. Metadata = `GC_META` (`arm=gc25d, format=gc25d, sem_cond=gc_sem_cond, mask_channel=False`) plus
  `vocab, data=v10, data_root, step, stage=sft_bidirectional, cond_type=ids, cond_key=ids, text_dropout, cond_dropout, mask_p, lr, cond_lr,
  weight_decay, warmup_steps, first_frame_cond, seed, init_ckpt, model_dir, trainer`, all as strings.
* `save_state` = `distill.runlib.save_state_rotating`: two slots plus the `state` symlink. `trainer_state.json` = `{step, arm, vocab, init_ckpt,
  epoch, epoch_micro, num_processes, grad_accum, batch_size, n_micro, n_train, seed}`. Here `epoch_micro` is the number of micro-batches this rank
  has consumed in `epoch`, and `n_micro` is `len(dataloader)` per rank.
* **Data order**: `Accelerator(dataloader_config=DataLoaderConfiguration(use_seedable_sampler=True, data_seed=--seed))`. accelerate replaces the
  RandomSampler with `SeedableRandomSampler`, so each epoch's permutation is a pure function of (seed, epoch) and is the same on every rank before
  sharding. An explicit `data_seed` is needed because `set_seed(device_specific=True)` gives each rank a different torch seed. Upstream instead
  uses the sampler generator's running state, which `load_state` does not save (accelerate only saves sampler state for IterableDatasets), so
  upstream cannot resume mid-epoch.
* **Steps per epoch** = `epoch_steps(len(dataloader), GA)` = **ceil**. accelerate forces a sync at the end of the dataloader, so a tail shorter
  than GA is still one step (mini 4 GPUs: 25 micro ⇒ 13 steps; tested against real accelerate: `[F,T,F,T,T]` for 5 micro at GA 2).
* Resume:
  * `--resume <dir>`: `load_state`, fast-forward the LR schedule and reseed `cpu_gen` (seed·1000 + rank + start_step, as upstream). A vocab
    mismatch ⇒ SystemExit. **The data position continues exactly** (`resume_position`):
    * If the stored geometry (`num_processes, grad_accum, batch_size, n_micro, n_train, seed`) matches, the trainer resumes in the stored
      `epoch` and skips its first `epoch_micro` micro-batches with `accelerator.skip_first_batches`. The skip happens in the batch sampler, so
      skipped clips are never loaded. Every later (clip, text/cond/mask draw) is then the same as in an uninterrupted run. `epoch_micro` is a
      multiple of GA (or the whole epoch), so the accumulation boundaries do not move. If the save fell on the epoch's last micro-batch, the
      run starts the next epoch.
    * If the geometry changed, the rest of the current epoch is abandoned and the run starts the next epoch from its beginning. That gives a new
      permutation and new draws, so no sample is replayed with identical draws.
    * A legacy `trainer_state.json` with only `{step, arm, vocab}` derives `epoch = step // ceil-steps-per-epoch` and skips the rest of the
      way into it.
    * What is still not bit-exact: `t` (`cpu_gen` is reseeded, as upstream) and DataLoader worker seeds. The noise ε comes from the CUDA RNG,
      which `load_state` restores.
  * `--resume X.safetensors`: warm start through `model.load_gc_weights`. A gc25d export must have the same vocab. A plain Wan export with no
    `gc_sem_cond.*` keys gets a zero-initialised cond.
  * **Lineage metadata**: `init_ckpt` is the warm-start `.safetensors`, or `""` for base Wan with a zero-initialised cond. A dir resume never
    changes it: the value is carried in `trainer_state.json`, and a legacy state falls back to the run's `config.json`. The resume is recorded
    separately as `resumed_from` (realpath of the slot) and `resumed_step`.
* `train_log.csv` holds `step, loss, mse, timestep, lr, cond_lr, grad_norm, sem_rms, x_rms, text_drop, cond_drop, mask_frac, step_time,
  data_time, mem_gb, epoch, clips`:
  * **`step_time` is the wall time of the whole optimizer step, including waiting for data.** `data_time` is the part spent waiting for the
    DataLoader, which includes the worker re-fork at each epoch start. Validation and save time are excluded.
  * `clips` lists rank 0's clip_ids for the step's micro-batches, joined with `;`.
  * `done.json` has `s_per_step_mean` (wall time), `data_s_per_step_mean` and `data_pos`.
  * If an existing CSV in OUT has a different header (code upgrade plus resume), it is renamed to `*.old<N>.csv` first, so rows never land
    under the wrong header.
  * A non-finite reduced loss ⇒ SystemExit.
* Default `--num_workers` is 4 (upstream 2) because of the mask cost. `--save_state` and `--val_uncond` default on (SPEC §7).
* A train/val leak check (shared clip or shared run) ⇒ SystemExit.

### Launcher (`scripts/train_sft.sh`)
* Environment variables: `OUT` (required, must resolve under `$M/outputs/`), `NGPU` (8), `GPUS` (default 0..NGPU−1; NGPU must match), `GA` (2),
  `MAX_STEPS` (20000), `RESUME`, `VOCAB` (parts), `DATA` (`$M/data/gc25d_v10`), `PORT` (default: a free port), `EXTRA`, and
  `ALLOW_EXISTING=1` to start from 0 in an OUT that already has checkpoints.
* Exports `DIFFSYNTH_SKIP_DOWNLOAD`, `PYTHONPATH`, `HF_HOME`, and puts `TMPDIR`, `TORCHINDUCTOR_CACHE_DIR`, `TRITON_CACHE_DIR`,
  `TORCH_EXTENSIONS_DIR`, `TORCH_HOME` and `XDG_CACHE_HOME` all under `$M/.tmp`.
* Refuses to start (exit 3) in three cases:
  * the `flock` on `$OUT/.train_sft.lock` is taken;
  * a `train.py … --output $OUT` process exists;
  * OUT already holds `step-*.safetensors` or `state` but no `RESUME` was given.

  It also refuses when `$DATA/index/clips.jsonl` is missing, and only warns when `READY` is missing.
* Generated configs: `configs/accelerate_zero2_4gpu_ga2.yaml`, made during the smoke from `accelerate_zero2_ga2.yaml` by changing
  `num_processes` and `gradient_accumulation_steps` only.

### sample.py
* Supports `--ckpt X` | `--base [--vocab]`, `--data_root`, `--raw_root`, `--clip_ids a,b | @file` | `--val_n N` (val, drop=false, round-robin over
  runs), `--steps 50 --shift 5 --cfg 1 --sem_cfg 1 --sem_mode cond|unknown --seed 0 --out_dir`, `--no_ssim` and `--no_recon_ceiling`.
* It does not use `infer.load_models`, which installs the default `ids` vocab first, so a parts checkpoint would raise. Instead it calls
  `GM.load_dit` + `GM.load_gc_weights` (the vocab comes from the checkpoint).
* GT = raw tar jpgs `<raw_root>/<subset>/<scene>/<run>` read with `infer.read_raw_frames`, then `geometry.resample_rgb`.
* The ids panel shows what the model received. Static classes use `geometry.PALETTE`; player parts are a magenta/violet family with a cyan
  weapon, boss parts a red/orange family with a blue weapon; 255 is hatched.

## Tests (CPU)
```
cd /opt/dlami/nvme/zhiyangdeng/maliang
DIFFSYNTH_SKIP_DOWNLOAD=True PYTHONPATH=$PWD/code/gc25d:$PWD/code/arope TMPDIR=$PWD/.tmp OMP_NUM_THREADS=8 \
  .venv/bin/python -m pytest -q code/gc25d/tests/test_dataset.py code/gc25d/tests/test_train.py     # 30 passed, ~31 s
```
What the masking tests check:
* **World anchoring.** Synthetic UE-convention cameras (orthographic and reversed-Z perspective) translate over 81 frames. World ground points
  more than 60 cm from a plane boundary are projected into every 5th frame. At static pixels, the mask equals the world half-plane test exactly.
  More than 500 points are seen in at least 3 frames.
* **Characters.** Pixels 10..21 never change, and only static pixels become 255.
* **Coverage.** With one plane, the frame-0 static fraction is within 0.01 of q. With two planes it lies in [max q, Σq]. Over 40 seeds, 2-plane draws stay within the expected count.
* **NaN.** Sky rays from a shallow perspective camera are masked.

## Smoke (real weights, `outputs/_smoke/sft/`)
Setup: 4 procs on GPUs 2–5, **GA 2 ⇒ global batch 8**. I chose GA 2 over GA 4 (batch 16) so that one optimizer step does the same per-GPU
work (2 micro-batches) as the full 8-GPU GA2 run, which makes the measured s/step the full-run s/step. Data: `data/gc25d_v10_mini`
(100 train / 100 val clips).
```
OUT=outputs/_smoke/sft NGPU=4 GPUS=2,3,4,5 GA=2 MAX_STEPS=10 DATA=data/gc25d_v10_mini \
  EXTRA="--save_every 5 --val_every 5 --val_n 4 --warmup_steps 2" bash scripts/train_sft.sh
OUT=outputs/_smoke/sft NGPU=4 GPUS=2,3,4,5 GA=2 MAX_STEPS=13 RESUME=outputs/_smoke/sft/state DATA=data/gc25d_v10_mini \
  EXTRA="--save_every 5 --val_every 5 --val_n 4 --warmup_steps 2" bash scripts/train_sft.sh
```
| check | result |
|---|---|
| loss finite | steps 1–13: 0.05–0.42 (× training_weight). grad_norm 0.16–0.78, with one 6.07 spike in a repeat run that did not reappear across 3 more epoch boundaries |
| `sem_rms` grows from 0 | step 1: 0.0, step 2: 3.8e-4, step 3: 2.3e-3, step 10: 3.9e-3 (x_rms ≈ 0.35, so ≈1 %). cond lr warmup 5e-5 → 1e-4 |
| val rows | `val_log.csv`: steps 5 and 10 × t ∈ {201.3, 500.6, 799.7}, n=4. At step 10: mse 0.377/0.176/0.149, gap_uncond +0.0013/+0.0005/−0.0045, gap_nulltext +0.0022/+0.0016/+0.0010. As expected after 10 steps, the cond gap is ~0 |
| export | `step-5`, `step-10` and `step-13.safetensors`: each 831 tensors, all BF16, 10.02 GB, `check_gc_ckpt` OK, `ckpt_vocab` = parts, metadata as above |
| save_state | `state_A` (step 5) and `state_B` (step 10), 66 GB each on 4 GPUs. The `state` symlink rotated correctly |
| resume (pre-fix code) | `--resume …/state` (→ state_B, step 10) continued at 11–13 with the LR at its plateau and `sem_rms` continuous (3.3e-3). It exported step-13 and rotated `state` → state_A. **Review found that steps 11–13 replayed the exact samples of steps 1–3** (same clips and the same mask_frac 0.04015…/text drop/0.08591…). This is fixed; see "Review fixes" |
| second launch on the same OUT | refused: `已有训练在跑（锁 … 被占用）`, exit 3 |
| single process, no DS (`python train.py --limit 8 --max_steps 2`) | OK: 831-key export, peak 46.9 GB (plain bf16 AdamW, numbers not representative) |
| **s/step** | **2.40 s** of compute (40-step run with no val/saves, GPUs uncontended) ≈ 1.2 s per micro-batch. This matches ActionRoPE's 1.20 s/step at GA 1. The first step takes 7.1 s. Each epoch ends with one 1-micro step (~1.35 s): per-rank epoch length 25 is odd, and accelerate syncs at the end of the dataloader. Since the fix, `step_time` is wall time including data wait: **2.41–2.45 s steady with 0.018 s data wait**, and **2.51 s mean** over steps 3–16 including the epoch-boundary steps (see the fix smoke) |
| **peak memory** | 35.2 GB (main) / 35.3 GB (max) per GPU on 4 GPUs. On 8 GPUs ZeRO-2 shards the optimizer 8-way: expect ≈ 25–28 GB (ActionRoPE measured 32.8 GB on 4 and 22.7 GB on 8) |

The first smoke ran while another project's (PCE) job held ~39 GB at 100 % utilisation on all 8 GPUs. The step time was the same once it had left.

UNKNOWN mask statistics on mini train with default probabilities (53/100 samples masked):

| measure | mean | median | p90 |
|---|---|---|---|
| frame-0 static fraction masked | 0.31 | | |
| whole-clip pixel fraction | 0.39 | | 0.80 |
| frame-80 static fraction | 0.47 | | 1.00 |
| share of frames > 90 % masked | 0.126 | 0 | 0.54 |

sample.py on 1 val clip (`02_Autumn_Woodland_Shrine__P9_golden_s2909__002400`), 8 steps:
| model | PSNR frames 1–80 | SSIM | frame-0 PSNR | VAE ceiling | sampling | total |
|---|---|---|---|---|---|---|
| `--ckpt outputs/_smoke/sft/step-10.safetensors` | 15.46 dB | 0.571 | 36.86 | 31.54 | 3.6 s | 28 s |
| `--base` | 15.01 dB | 0.540 | 36.86 | 31.54 | 3.4 s | 24 s |

Outputs are in `outputs/_smoke/sft/sample/{step-10_s8,base_s8}/`. I removed the timing run's and the 1-proc run's checkpoints, and the stale `state_B`.
`outputs/_smoke/sft/` keeps `step-{5,10,13}.safetensors` (real-weight 831-key parts checkpoints; the distill smokes can use them) and `state_A`.

## Full run
```
cd /opt/dlami/nvme/zhiyangdeng/maliang && OUT=/opt/dlami/nvme/zhiyangdeng/maliang/outputs/sft_v10 NGPU=8 GA=2 MAX_STEPS=20000 VOCAB=parts \
  DATA=/opt/dlami/nvme/zhiyangdeng/maliang/data/gc25d_v10 nohup bash scripts/train_sft.sh \
  > /opt/dlami/nvme/zhiyangdeng/maliang/outputs/sft_v10.launcher.out 2>&1 &
# crash → same command plus RESUME=/opt/dlami/nvme/zhiyangdeng/maliang/outputs/sft_v10/state
```
This uses the SPEC §7 defaults throughout:
* lr 1e-5, cond_lr 1e-4, wd 0.01, warmup 500, clip 1.0;
* 8 × micro 1 × GA 2 = 16;
* save_every 1000 plus rotating state, val_every 500 with val_n 16 at t 200/500/800 in all three variants;
* dropouts 0.1 / 0.05 / 0.5, grad ckpt on.

Estimate: 20,000 × ~2.43 s wall ≈ 13.5 h of training. Data wait is ~0.02 s/step, plus a ~1.1 s worker re-fork once per 747-step epoch, which is negligible. Add 20 exports (~10 GB, <30 s each), 20 state saves (~66 GB per slot, <1 min each) and 40
validations (16 clips × 3 t × 3 variants over 8 ranks, ~10 s each), for ≈ 14 h in total. Disk: 20 × 10 GB step files + 2 × ~66 GB state ≈ 330 GB on NVMe.
Under the SPEC §5 split (10 val runs = 500 clips), computed from the raw per-run `clips_81.json` files and the buried list:
* 12,800 clips = 12,300 train + 500 val. 345 of the 348 buried clips are in train runs and 3 in val runs.
* That leaves **11,955 train clips**: 11,955 // 8 = **1,494 micro-batches per rank** (even), **747 steps per epoch**, ≈ **26.8 epochs** in 20k steps.
* An earlier version of this doc used the investigator's 20-run split, which gives 11,452 train, 715 steps per epoch and 28 epochs. That
  split is not the SPEC split. Re-check against `data/gc25d_v10/index/clips.jsonl` once prep finishes: `config.json` →
  `dataset.geometry` / `steps_per_epoch` prints the real values.

Watch:
* `train/sem_rms` and `train/sem_over_x` should keep rising.
* `val/gap_uncond_mean` should become clearly positive, which shows the condition is used. `gap_nulltext` measures the text.

## Open issues / notes
* **LEAD DECISION NEEDED BEFORE THE FULL RUN: mask coverage grows over the clip.** Both reviewers flagged this, and it is not a code
  defect: the code follows SPEC §7 to the letter, with τ calibrated on frame 0 only. The half-planes are fixed in the world and the camera
  travels several metres in 81 frames, so the mask grows. Evidence on mini:
  * My run (53/100 masked): frame 80 averages 47 % masked and 12.6 % of frames are > 90 % masked.
  * Reviewer A (100 train clips, mask forced on), last-cell static coverage: ortho mean 0.35, p90 0.96; perspective mean 0.47, p90 1.00.
    18–20 % of masked clips end more than 90 % masked.
  * Reviewer B (50 perspective + 50 ortho clips × 3 seeds), whole-clip static coverage: mean 0.40 (perspective) / 0.37 (ortho), p90 0.86 / 0.82.
    Frame 80 has p90 = 1.0. 14.0 % / 11.7 % of masked frames are > 90 % masked, and 12–16 % are < 1 % masked because the mask slid off screen.
  * Overall about 6 % of all training frames have no static condition, on top of the 5 % cond dropout.

  At deploy, canvases are 8–40 % unseen, and the auto-demo keeps black ≤ 10 % in almost every frame (`ref/gc25d-live-demo/README.md:36,
  :102–103`). Training is therefore much heavier than deploy in the tail.

  Options, all keeping the half-planes world-anchored and so cell-local / causal-safe:
  * (a) calibrate q on the mean frame or the last frame instead of frame 0;
  * (b) cap per-frame coverage by clamping τ per frame, e.g. τ_f = max(τ, the (1−0.4) quantile of frame f's static projections);
  * (c) keep the current behaviour and record the decision in SPEC §7.

  This fix pass did not change the behaviour because SPEC §7 is authoritative. Options (a) and (b) are each a few lines in
  `dataset.sample_planes` / `planes_mask` and need a matching update to the coverage test.
* A half-weight tail step (one micro-batch) happens only when the per-rank epoch length is odd. That is true on mini with 4 GPUs (25),
  but **not for the full run under the SPEC split (1,494 per rank, even)**. When it does happen, accelerate passes the boundary to DeepSpeed
  (`set_gradient_accumulation_boundary`) and the step count uses the ceiling (see Trainer), so the bookkeeping is consistent.
* The val subset is upstream's `limit` (seeded shuffle), so it is not run-balanced. sample.py's `--val_n` is run-balanced.
* Warm start from an `ids`-vocab checkpoint into a `parts` run is not supported (`load_gc_weights` refuses the vocab mismatch). It would need a
  proj_in channel expansion.
* `sample.py` loads the DiT straight onto the GPU. Load takes 6–8 s and an 8-step clip with VAE decode, recon ceiling and SSIM takes ~25 s.

## Review fixes (2026-10-22, fixer pass)
Two independent reviewers produced 9 findings, which reduce to 5 distinct issues. Each was verified against the code, the logs and SPEC before any change.

**Fixed**
1. **Resume replayed identical samples, and the epoch counter used floor** (`train.py`). This was the only finding with code consequences.
   * Verified: in the smoke, steps 11/12/13 reproduced steps 1/2/3 exactly, and the timing run had 13 steps per epoch while the log printed 12.
   * Root cause: accelerate does not save the sampler generator for map-style datasets, the epoch was derived as `start_step // floor`, and
     the dropout RNG is keyed on (seed, epoch, idx).
   * Fix: `SeedableRandomSampler` with `data_seed`, the data position stored in `trainer_state.json`, `skip_first_batches` on resume,
     `epoch_steps` = ceil, and a geometry check that falls back to the next epoch. Details are under Trainer → Resume.
2. **`init_ckpt` lineage after a dir resume.** Verified: step-13 recorded `init_ckpt=…/sft/state`, a rotating symlink. Now `init_ckpt` is only
   ever a warm-start `.safetensors` or `""`. It is carried through `trainer_state.json` (legacy states fall back to `config.json`), and the
   resume is recorded separately as `resumed_from` (slot realpath) and `resumed_step`.
3. **`step_time` excluded data wait.** Verified: the timing run had 106.6 s of wall time against Σstep_time = 100.9 s.
   * `step_time` is now the wall time from the end of the previous step, with validation and save time excluded.
   * The new `data_time` column records the data-wait part, and `done.json` has `data_s_per_step_mean`.
   * `clips` (rank-0 clip_ids) was added so data continuity can be checked from the log.
   * Because the columns changed, an old-header CSV in a resumed OUT is renamed first.
4. **Full-run numbers used the 20-run split** (this doc). Verified: SPEC §5 gives 11,955 train clips, 1,494 per rank, 747 steps per epoch and ≈26.8 epochs.
   The half-weight-step note is now scoped to odd per-rank epochs, which the full run does not have.

**Not changed (lead decision)**
5. **Mask coverage grows over the clip** (both reviewers, "design risk, not a code defect"). The code implements SPEC §7 literally, and SPEC is
   authoritative, so I kept the behaviour. The evidence and options (a)/(b)/(c) are under Open issues and need a decision before the full run.

**Fix smoke** (4 procs on GPUs 2–5, GA 2, mini, `outputs/_smoke/sft/fix_A`, `fix_B`; comparison in `fix_tools/compare_resume.{py,out}`):
```
# A: warm start from step-13.safetensors, 16 steps uninterrupted (crosses the 13-step epoch boundary), save/val at 8 and 16
OUT=outputs/_smoke/sft/fix_A RESUME=outputs/_smoke/sft/step-13.safetensors NGPU=4 GPUS=2,3,4,5 GA=2 MAX_STEPS=16 DATA=data/gc25d_v10_mini \
  EXTRA="--save_every 8 --val_every 8 --val_n 4 --warmup_steps 2" bash scripts/train_sft.sh
# B: resume from A's step-8 slot into a separate OUT, run to 16
OUT=outputs/_smoke/sft/fix_B RESUME=outputs/_smoke/sft/fix_A/state_A … (same) bash scripts/train_sft.sh
```
| check | result |
|---|---|
| warm start | `step-13.safetensors` loaded (831 tensors, `has_sem_cond`). Step 1 has sem_rms 2.95e-3, not 0 |
| ceil steps/epoch | log: `25 个 micro batch = 13 步`. Step 13 is the 1-micro tail (one clip in `clips`), and epoch 1 starts at step 14 |
| state position | A's `state_A` holds `{step 8, epoch 0, epoch_micro 16, …}` and `state_B` holds `{step 16, epoch 1, epoch_micro 6}` |
| **exact resume** | B: `数据从 epoch 0 的第 16 个 micro batch 接上（精确）`. **Steps 9–16 of B equal A row by row** in `clips`, `epoch`, `text_drop`, `cond_drop` and `mask_frac`. No (clip, epoch) pair from steps 1–8 reappears. Loss differs because `t` is reseeded, as upstream does |
| lineage | A's step-8 and step-16 have `init_ckpt=…/sft/step-13.safetensors`, `resumed_from=""`. B's step-16 has the **same `init_ckpt`**, `resumed_from=…/fix_A/state_A`, `resumed_step=8` |
| exports | 831 BF16, `check_gc_ckpt` OK, vocab parts (A: step 8 and 16; B: step 16) |
| timing (A) | steady 2.41–2.45 s/step wall, data 0.018 s; tail step 13: 1.95 s (data 0.30); epoch start 14: 3.93 s (data 1.11, worker re-fork); mean over steps 3–16 **2.51 s (data 0.12)**. Peak memory 35.2 / 35.3 GB |
| timing (B) | 6.4 s/step: another job's 8 processes (~9.5 GB, up to 100 % utilisation on all 8 GPUs) started during the smoke. I did not touch them. The numbers above are run A's |

After checking, I deleted the fix smoke's state slots and exports (≈225 GB) and kept the logs, configs and comparison output. The older
`outputs/_smoke/sft/step-{5,10,13}.safetensors` and `state_A` (legacy `trainer_state` = `{step, arm, vocab}`) are untouched. Resuming that legacy
state uses the "derived from step" path (step 13 ⇒ epoch 1, skip 0).