| # sft — bidirectional gc25d teacher on v10 (SPEC §6–§7) |
|
|
| Status 2026-10-22: code done, CPU-tested (30 tests) and smoke-tested with real weights on `data/gc25d_v10_mini` (4×H200, GPUs 2–5). |
| A review-fix pass then made resume exact (data position and lineage) and added data-wait timing. See "Review fixes" at the end. |
| **The full run has not been started.** It needs `data/gc25d_v10` (full prep), and its command is at the end. |
|
|
| ## What |
|
|
| | file | role | |
| |---|---| |
| | `code/gc25d/dataset.py` | `GcLatentDataset` (latents + parts ids + text table + the three dropouts + world-anchored UNKNOWN masking) and `gc_collate` | |
| | `code/gc25d/train.py` | fork of ActionRoPE `actionrope/train.py` (same section order): `GcTrainModule`, `sem_param_groups`, `warmup_lambda`, `run_validation` (cond / uncond / null-text), export through `gcdistill.gc_dit.save_gc` + `check_gc_ckpt`, rotating `save_state`, resume / warm start | |
| | `scripts/train_sft.sh` | launcher: env vars → accelerate yaml (8×GA2 and 8×GA1 use the existing files, other combinations are generated as `configs/accelerate_zero2_<N>gpu_ga<G>.yaml`), `tee -a $OUT/train.log`, refuses a second job on the same OUT | |
| | `code/gc25d/sample.py` | offline eval: TI2V sampling (`infer.sample` + `model.gc_forward`) on val clips → mp4 `GT \| generated \| cond (parts coloured)` + per-frame PSNR/SSIM json + `metrics.json` | |
| | `code/gc25d/tests/test_dataset.py` | 15 tests: contract, split/drop/limit, val never drops, `ids` vocab, dropout rates/independence/determinism, masking (world-anchored, characters untouched, coverage, NaN ⇒ masked, speed), collate, real mini clip | |
| | `code/gc25d/tests/test_train.py` | 15 tests: tiny WanModel forward/backward through `GcTrainModule`, manual loss + frame-0 exclusion, `sem_rms` grows from 0, param groups + warmup, CLI defaults = SPEC §7, val variants, export keys/metadata, lineage metadata, `epoch_steps` = ceil, `resume_position` (exact / epoch end / geometry changed / legacy), `init_ckpt_of_state`, CSV header rotation, and **real accelerate on CPU**: the seedable-sampler permutation depends only on (seed, epoch), `skip_first_batches` continues the uninterrupted order, and the sync pattern is `[F,T,F,T,T]` | |
|
|
| Nothing under `code/arope/` was touched. `model.py`, `capsules.py` and `gcdistill/gc_dit.py` are unchanged; they are only imported. |
|
|
| ## How (conventions that matter) |
|
|
| ### Dataset (`dataset.py`) |
| * Sample = `input_latents` bf16 [48,21,30,52], `context` bf16 [21,512,4096], `context_ids` int64 [21], `sem_ids` uint8 [81,480,832], |
| `clip_id`, `dropped = {"text": bool, "cond": bool, "mask": float}`. Here `mask` is the fraction of all 81×480×832 pixels that the mask set to 255. |
| * `key_to_id = {k: i for i, k in enumerate(sorted(text_table))}`. This equals `infer._KEY_TO_ID` (tested). |
| * Rows come from `index/clips.jsonl` filtered by `split`. Rows with `drop=true` are excluded unless `include_dropped=True`. |
| `limit` shuffles with `Random(seed)`, keeps the first `limit` rows and sorts them, as `AropeLatentDataset` does. |
| **The val split forces all three dropout probabilities to 0.** |
| * `vocab="ids"` ⇒ `model.collapse_parts` after masking. |
| * RNG: each sample draws from `random.Random(f"gc25d:{seed}:{epoch}:{index}")`. It always draws three uniforms (text, cond, mask) in |
| that order, then the mask parameters. Results are identical for a given (seed, epoch, index), whatever the worker count or rank. |
| The trainer calls `train_ds.set_epoch(epoch)` itself before every epoch. With the seedable sampler, accelerate's `DataLoaderShard.set_epoch` |
| only reaches the sampler and never the dataset. **`persistent_workers=False`** because persistent workers would keep a stale epoch. |
| * Dropouts, as in SPEC §7: text p=0.1 replaces all 21 cells with `NULL_PROMPT`. cond p=0.05 sets the ids to all 255 (the npz is not even |
| read), independently of text. mask p=0.5 applies only when cond was not dropped. |
| * World-anchored UNKNOWN mask (SPEC §7, `apply_unknown_mask`): |
| * 1 half-plane with p .7, 2 with p .3. θ ~ U[0,2π) in world XY and q ~ U(0.05, 0.40). |
| * The ground plane is Z = `meta.ground_z[0]`. If that value is NaN the code falls back to the nanmedian, and if there is none it skips the mask. |
| * Ground points come from `capsules.ground_points` on the 4×4 pooled grid, then nearest-upsampled ×4. `ground_points` only accepts integer |
| dst pixels and the geometric cell centre (4j+1.5, 4i+1.5) falls between pixels, so cell (i, j) uses dst pixel **(4j+2, 4i+2)**. That is half a dst pixel, under 1 source pixel. |
| * τ = the (1−q) quantile of the frame-0 static pixels' projections onto θ, weighted per pixel. Every frame then sets static pixels (raw ≤ 6) |
| to 255 when their ground point is beyond any plane or is NaN. Characters (10..21) are never touched. |
| * Cost is ≈0.27–0.37 s per masked sample (81 frames × 25k rays). An unmasked sample takes 0.09 s. Training uses 4 workers per rank. |
| |
| ### Trainer (`train.py`) |
| The section order follows upstream (args / model / export+log / validation / main), so `diff -u ref/arope_git/ActionRoPE/actionrope/train.py |
| code/gc25d/train.py` is readable. These parts are kept verbatim: |
| * `Accelerator(gradient_accumulation_steps)` and `set_seed(device_specific=True)`. |
| * The DS overrides: `--grad_clip` is written into `deepspeed_config.gradient_clipping`, and a GA mismatch with the yaml ⇒ SystemExit. |
| * `LambdaLR` is stepped manually, and only on `sync_gradients`. |
| * The per-rank `cpu_gen` timestep `tid ~ U{0..999}` (seed·1000 + rank + start_step). |
| * The `accumulate` block, `dit.freqs` moved to the GPU, `config.json`/`done.json`, and the CSV + tensorboard logs. |
| |
| Changes from upstream: |
| * `GcTrainModule.forward`: `x_t = add_noise`, then frame 0 is replaced by clean x0, then `model.gc_forward(..., first_frame_cond, context_ids, |
| grad ckpt when training, aux)`. Frame 0 is dropped from the loss, and `loss = MSE × training_weight(t)`. The returned stats are `{mse, timestep, sem_rms, x_rms}`. |
| * Optimizer: `AdamW(sem_param_groups(module, lr, cond_lr, wd))` = [all DiT params at `--lr`, `dit.gc_sem_cond.*` at `--cond_lr`]. The betas |
| stay at the torch defaults (0.9, 0.999). The warmup factor multiplies both groups. `sem_param_groups` takes a `prefix` (default |
| `dit.gc_sem_cond.`) so the distill runners can reuse it. |
| * Validation: fixed `val_n` val clips (`limit` + seed) × `--val_timesteps` × fixed noise. Each (clip, t) runs three forwards with **the same noise**: |
| * cond: true ids and true text; |
| * uncond: ids all 255; |
| * nulltext: `NULL_PROMPT` × 21. |
| |
| `val_log.csv` holds `step, timestep, mse, mse_uncond, mse_nulltext, gap_uncond, gap_nulltext, n`, and tensorboard gets `val/*_t{t}` and `val/*_mean`. |
| * Export: `gcdistill.gc_dit.save_gc` writes 831 BF16 keys with the `dit.` prefix stripped, through `.tmp` + rename and chmod 644. It is |
| followed by `check_gc_ckpt` and a vocab check. Metadata = `GC_META` (`arm=gc25d, format=gc25d, sem_cond=gc_sem_cond, mask_channel=False`) plus |
| `vocab, data=v10, data_root, step, stage=sft_bidirectional, cond_type=ids, cond_key=ids, text_dropout, cond_dropout, mask_p, lr, cond_lr, |
| weight_decay, warmup_steps, first_frame_cond, seed, init_ckpt, model_dir, trainer`, all as strings. |
| * `save_state` = `distill.runlib.save_state_rotating`: two slots plus the `state` symlink. `trainer_state.json` = `{step, arm, vocab, init_ckpt, |
| epoch, epoch_micro, num_processes, grad_accum, batch_size, n_micro, n_train, seed}`. Here `epoch_micro` is the number of micro-batches this rank |
| has consumed in `epoch`, and `n_micro` is `len(dataloader)` per rank. |
| * **Data order**: `Accelerator(dataloader_config=DataLoaderConfiguration(use_seedable_sampler=True, data_seed=--seed))`. accelerate replaces the |
| RandomSampler with `SeedableRandomSampler`, so each epoch's permutation is a pure function of (seed, epoch) and is the same on every rank before |
| sharding. An explicit `data_seed` is needed because `set_seed(device_specific=True)` gives each rank a different torch seed. Upstream instead |
| uses the sampler generator's running state, which `load_state` does not save (accelerate only saves sampler state for IterableDatasets), so |
| upstream cannot resume mid-epoch. |
| * **Steps per epoch** = `epoch_steps(len(dataloader), GA)` = **ceil**. accelerate forces a sync at the end of the dataloader, so a tail shorter |
| than GA is still one step (mini 4 GPUs: 25 micro ⇒ 13 steps; tested against real accelerate: `[F,T,F,T,T]` for 5 micro at GA 2). |
| * Resume: |
| * `--resume <dir>`: `load_state`, fast-forward the LR schedule and reseed `cpu_gen` (seed·1000 + rank + start_step, as upstream). A vocab |
| mismatch ⇒ SystemExit. **The data position continues exactly** (`resume_position`): |
| * If the stored geometry (`num_processes, grad_accum, batch_size, n_micro, n_train, seed`) matches, the trainer resumes in the stored |
| `epoch` and skips its first `epoch_micro` micro-batches with `accelerator.skip_first_batches`. The skip happens in the batch sampler, so |
| skipped clips are never loaded. Every later (clip, text/cond/mask draw) is then the same as in an uninterrupted run. `epoch_micro` is a |
| multiple of GA (or the whole epoch), so the accumulation boundaries do not move. If the save fell on the epoch's last micro-batch, the |
| run starts the next epoch. |
| * If the geometry changed, the rest of the current epoch is abandoned and the run starts the next epoch from its beginning. That gives a new |
| permutation and new draws, so no sample is replayed with identical draws. |
| * A legacy `trainer_state.json` with only `{step, arm, vocab}` derives `epoch = step // ceil-steps-per-epoch` and skips the rest of the |
| way into it. |
| * What is still not bit-exact: `t` (`cpu_gen` is reseeded, as upstream) and DataLoader worker seeds. The noise ε comes from the CUDA RNG, |
| which `load_state` restores. |
| * `--resume X.safetensors`: warm start through `model.load_gc_weights`. A gc25d export must have the same vocab. A plain Wan export with no |
| `gc_sem_cond.*` keys gets a zero-initialised cond. |
| * **Lineage metadata**: `init_ckpt` is the warm-start `.safetensors`, or `""` for base Wan with a zero-initialised cond. A dir resume never |
| changes it: the value is carried in `trainer_state.json`, and a legacy state falls back to the run's `config.json`. The resume is recorded |
| separately as `resumed_from` (realpath of the slot) and `resumed_step`. |
| * `train_log.csv` holds `step, loss, mse, timestep, lr, cond_lr, grad_norm, sem_rms, x_rms, text_drop, cond_drop, mask_frac, step_time, |
| data_time, mem_gb, epoch, clips`: |
| * **`step_time` is the wall time of the whole optimizer step, including waiting for data.** `data_time` is the part spent waiting for the |
| DataLoader, which includes the worker re-fork at each epoch start. Validation and save time are excluded. |
| * `clips` lists rank 0's clip_ids for the step's micro-batches, joined with `;`. |
| * `done.json` has `s_per_step_mean` (wall time), `data_s_per_step_mean` and `data_pos`. |
| * If an existing CSV in OUT has a different header (code upgrade plus resume), it is renamed to `*.old<N>.csv` first, so rows never land |
| under the wrong header. |
| * A non-finite reduced loss ⇒ SystemExit. |
| * Default `--num_workers` is 4 (upstream 2) because of the mask cost. `--save_state` and `--val_uncond` default on (SPEC §7). |
| * A train/val leak check (shared clip or shared run) ⇒ SystemExit. |
| |
| ### Launcher (`scripts/train_sft.sh`) |
| * Environment variables: `OUT` (required, must resolve under `$M/outputs/`), `NGPU` (8), `GPUS` (default 0..NGPU−1; NGPU must match), `GA` (2), |
| `MAX_STEPS` (20000), `RESUME`, `VOCAB` (parts), `DATA` (`$M/data/gc25d_v10`), `PORT` (default: a free port), `EXTRA`, and |
| `ALLOW_EXISTING=1` to start from 0 in an OUT that already has checkpoints. |
| * Exports `DIFFSYNTH_SKIP_DOWNLOAD`, `PYTHONPATH`, `HF_HOME`, and puts `TMPDIR`, `TORCHINDUCTOR_CACHE_DIR`, `TRITON_CACHE_DIR`, |
| `TORCH_EXTENSIONS_DIR`, `TORCH_HOME` and `XDG_CACHE_HOME` all under `$M/.tmp`. |
| * Refuses to start (exit 3) in three cases: |
| * the `flock` on `$OUT/.train_sft.lock` is taken; |
| * a `train.py … --output $OUT` process exists; |
| * OUT already holds `step-*.safetensors` or `state` but no `RESUME` was given. |
|
|
| It also refuses when `$DATA/index/clips.jsonl` is missing, and only warns when `READY` is missing. |
| * Generated configs: `configs/accelerate_zero2_4gpu_ga2.yaml`, made during the smoke from `accelerate_zero2_ga2.yaml` by changing |
| `num_processes` and `gradient_accumulation_steps` only. |
|
|
| ### sample.py |
| * Supports `--ckpt X` | `--base [--vocab]`, `--data_root`, `--raw_root`, `--clip_ids a,b | @file` | `--val_n N` (val, drop=false, round-robin over |
| runs), `--steps 50 --shift 5 --cfg 1 --sem_cfg 1 --sem_mode cond|unknown --seed 0 --out_dir`, `--no_ssim` and `--no_recon_ceiling`. |
| * It does not use `infer.load_models`, which installs the default `ids` vocab first, so a parts checkpoint would raise. Instead it calls |
| `GM.load_dit` + `GM.load_gc_weights` (the vocab comes from the checkpoint). |
| * GT = raw tar jpgs `<raw_root>/<subset>/<scene>/<run>` read with `infer.read_raw_frames`, then `geometry.resample_rgb`. |
| * The ids panel shows what the model received. Static classes use `geometry.PALETTE`; player parts are a magenta/violet family with a cyan |
| weapon, boss parts a red/orange family with a blue weapon; 255 is hatched. |
|
|
| ## Tests (CPU) |
| ``` |
| cd /opt/dlami/nvme/zhiyangdeng/maliang |
| DIFFSYNTH_SKIP_DOWNLOAD=True PYTHONPATH=$PWD/code/gc25d:$PWD/code/arope TMPDIR=$PWD/.tmp OMP_NUM_THREADS=8 \ |
| .venv/bin/python -m pytest -q code/gc25d/tests/test_dataset.py code/gc25d/tests/test_train.py # 30 passed, ~31 s |
| ``` |
| What the masking tests check: |
| * **World anchoring.** Synthetic UE-convention cameras (orthographic and reversed-Z perspective) translate over 81 frames. World ground points |
| more than 60 cm from a plane boundary are projected into every 5th frame. At static pixels, the mask equals the world half-plane test exactly. |
| More than 500 points are seen in at least 3 frames. |
| * **Characters.** Pixels 10..21 never change, and only static pixels become 255. |
| * **Coverage.** With one plane, the frame-0 static fraction is within 0.01 of q. With two planes it lies in [max q, Σq]. Over 40 seeds, 2-plane draws stay within the expected count. |
| * **NaN.** Sky rays from a shallow perspective camera are masked. |
|
|
| ## Smoke (real weights, `outputs/_smoke/sft/`) |
| Setup: 4 procs on GPUs 2–5, **GA 2 ⇒ global batch 8**. I chose GA 2 over GA 4 (batch 16) so that one optimizer step does the same per-GPU |
| work (2 micro-batches) as the full 8-GPU GA2 run, which makes the measured s/step the full-run s/step. Data: `data/gc25d_v10_mini` |
| (100 train / 100 val clips). |
| ``` |
| OUT=outputs/_smoke/sft NGPU=4 GPUS=2,3,4,5 GA=2 MAX_STEPS=10 DATA=data/gc25d_v10_mini \ |
| EXTRA="--save_every 5 --val_every 5 --val_n 4 --warmup_steps 2" bash scripts/train_sft.sh |
| OUT=outputs/_smoke/sft NGPU=4 GPUS=2,3,4,5 GA=2 MAX_STEPS=13 RESUME=outputs/_smoke/sft/state DATA=data/gc25d_v10_mini \ |
| EXTRA="--save_every 5 --val_every 5 --val_n 4 --warmup_steps 2" bash scripts/train_sft.sh |
| ``` |
| | check | result | |
| |---|---| |
| | loss finite | steps 1–13: 0.05–0.42 (× training_weight). grad_norm 0.16–0.78, with one 6.07 spike in a repeat run that did not reappear across 3 more epoch boundaries | |
| | `sem_rms` grows from 0 | step 1: 0.0, step 2: 3.8e-4, step 3: 2.3e-3, step 10: 3.9e-3 (x_rms ≈ 0.35, so ≈1 %). cond lr warmup 5e-5 → 1e-4 | |
| | val rows | `val_log.csv`: steps 5 and 10 × t ∈ {201.3, 500.6, 799.7}, n=4. At step 10: mse 0.377/0.176/0.149, gap_uncond +0.0013/+0.0005/−0.0045, gap_nulltext +0.0022/+0.0016/+0.0010. As expected after 10 steps, the cond gap is ~0 | |
| | export | `step-5`, `step-10` and `step-13.safetensors`: each 831 tensors, all BF16, 10.02 GB, `check_gc_ckpt` OK, `ckpt_vocab` = parts, metadata as above | |
| | save_state | `state_A` (step 5) and `state_B` (step 10), 66 GB each on 4 GPUs. The `state` symlink rotated correctly | |
| | resume (pre-fix code) | `--resume …/state` (→ state_B, step 10) continued at 11–13 with the LR at its plateau and `sem_rms` continuous (3.3e-3). It exported step-13 and rotated `state` → state_A. **Review found that steps 11–13 replayed the exact samples of steps 1–3** (same clips and the same mask_frac 0.04015…/text drop/0.08591…). This is fixed; see "Review fixes" | |
| | second launch on the same OUT | refused: `已有训练在跑（锁 … 被占用）`, exit 3 | |
| | single process, no DS (`python train.py --limit 8 --max_steps 2`) | OK: 831-key export, peak 46.9 GB (plain bf16 AdamW, numbers not representative) | |
| | **s/step** | **2.40 s** of compute (40-step run with no val/saves, GPUs uncontended) ≈ 1.2 s per micro-batch. This matches ActionRoPE's 1.20 s/step at GA 1. The first step takes 7.1 s. Each epoch ends with one 1-micro step (~1.35 s): per-rank epoch length 25 is odd, and accelerate syncs at the end of the dataloader. Since the fix, `step_time` is wall time including data wait: **2.41–2.45 s steady with 0.018 s data wait**, and **2.51 s mean** over steps 3–16 including the epoch-boundary steps (see the fix smoke) | |
| | **peak memory** | 35.2 GB (main) / 35.3 GB (max) per GPU on 4 GPUs. On 8 GPUs ZeRO-2 shards the optimizer 8-way: expect ≈ 25–28 GB (ActionRoPE measured 32.8 GB on 4 and 22.7 GB on 8) | |
| |
| The first smoke ran while another project's (PCE) job held ~39 GB at 100 % utilisation on all 8 GPUs. The step time was the same once it had left. |
| |
| UNKNOWN mask statistics on mini train with default probabilities (53/100 samples masked): |
| |
| | measure | mean | median | p90 | |
| |---|---|---|---| |
| | frame-0 static fraction masked | 0.31 | | | |
| | whole-clip pixel fraction | 0.39 | | 0.80 | |
| | frame-80 static fraction | 0.47 | | 1.00 | |
| | share of frames > 90 % masked | 0.126 | 0 | 0.54 | |
| |
| sample.py on 1 val clip (`02_Autumn_Woodland_Shrine__P9_golden_s2909__002400`), 8 steps: |
| | model | PSNR frames 1–80 | SSIM | frame-0 PSNR | VAE ceiling | sampling | total | |
| |---|---|---|---|---|---|---| |
| | `--ckpt outputs/_smoke/sft/step-10.safetensors` | 15.46 dB | 0.571 | 36.86 | 31.54 | 3.6 s | 28 s | |
| | `--base` | 15.01 dB | 0.540 | 36.86 | 31.54 | 3.4 s | 24 s | |
| |
| Outputs are in `outputs/_smoke/sft/sample/{step-10_s8,base_s8}/`. I removed the timing run's and the 1-proc run's checkpoints, and the stale `state_B`. |
| `outputs/_smoke/sft/` keeps `step-{5,10,13}.safetensors` (real-weight 831-key parts checkpoints; the distill smokes can use them) and `state_A`. |
| |
| ## Full run |
| ``` |
| cd /opt/dlami/nvme/zhiyangdeng/maliang && OUT=/opt/dlami/nvme/zhiyangdeng/maliang/outputs/sft_v10 NGPU=8 GA=2 MAX_STEPS=20000 VOCAB=parts \ |
| DATA=/opt/dlami/nvme/zhiyangdeng/maliang/data/gc25d_v10 nohup bash scripts/train_sft.sh \ |
| > /opt/dlami/nvme/zhiyangdeng/maliang/outputs/sft_v10.launcher.out 2>&1 & |
| # crash → same command plus RESUME=/opt/dlami/nvme/zhiyangdeng/maliang/outputs/sft_v10/state |
| ``` |
| This uses the SPEC §7 defaults throughout: |
| * lr 1e-5, cond_lr 1e-4, wd 0.01, warmup 500, clip 1.0; |
| * 8 × micro 1 × GA 2 = 16; |
| * save_every 1000 plus rotating state, val_every 500 with val_n 16 at t 200/500/800 in all three variants; |
| * dropouts 0.1 / 0.05 / 0.5, grad ckpt on. |
| |
| Estimate: 20,000 × ~2.43 s wall ≈ 13.5 h of training. Data wait is ~0.02 s/step, plus a ~1.1 s worker re-fork once per 747-step epoch, which is negligible. Add 20 exports (~10 GB, <30 s each), 20 state saves (~66 GB per slot, <1 min each) and 40 |
| validations (16 clips × 3 t × 3 variants over 8 ranks, ~10 s each), for ≈ 14 h in total. Disk: 20 × 10 GB step files + 2 × ~66 GB state ≈ 330 GB on NVMe. |
| Under the SPEC §5 split (10 val runs = 500 clips), computed from the raw per-run `clips_81.json` files and the buried list: |
| * 12,800 clips = 12,300 train + 500 val. 345 of the 348 buried clips are in train runs and 3 in val runs. |
| * That leaves **11,955 train clips**: 11,955 // 8 = **1,494 micro-batches per rank** (even), **747 steps per epoch**, ≈ **26.8 epochs** in 20k steps. |
| * An earlier version of this doc used the investigator's 20-run split, which gives 11,452 train, 715 steps per epoch and 28 epochs. That |
| split is not the SPEC split. Re-check against `data/gc25d_v10/index/clips.jsonl` once prep finishes: `config.json` → |
| `dataset.geometry` / `steps_per_epoch` prints the real values. |
| |
| Watch: |
| * `train/sem_rms` and `train/sem_over_x` should keep rising. |
| * `val/gap_uncond_mean` should become clearly positive, which shows the condition is used. `gap_nulltext` measures the text. |
| |
| ## Open issues / notes |
| * **LEAD DECISION NEEDED BEFORE THE FULL RUN: mask coverage grows over the clip.** Both reviewers flagged this, and it is not a code |
| defect: the code follows SPEC §7 to the letter, with τ calibrated on frame 0 only. The half-planes are fixed in the world and the camera |
| travels several metres in 81 frames, so the mask grows. Evidence on mini: |
| * My run (53/100 masked): frame 80 averages 47 % masked and 12.6 % of frames are > 90 % masked. |
| * Reviewer A (100 train clips, mask forced on), last-cell static coverage: ortho mean 0.35, p90 0.96; perspective mean 0.47, p90 1.00. |
| 18–20 % of masked clips end more than 90 % masked. |
| * Reviewer B (50 perspective + 50 ortho clips × 3 seeds), whole-clip static coverage: mean 0.40 (perspective) / 0.37 (ortho), p90 0.86 / 0.82. |
| Frame 80 has p90 = 1.0. 14.0 % / 11.7 % of masked frames are > 90 % masked, and 12–16 % are < 1 % masked because the mask slid off screen. |
| * Overall about 6 % of all training frames have no static condition, on top of the 5 % cond dropout. |
| |
| At deploy, canvases are 8–40 % unseen, and the auto-demo keeps black ≤ 10 % in almost every frame (`ref/gc25d-live-demo/README.md:36, |
| :102–103`). Training is therefore much heavier than deploy in the tail. |
| |
| Options, all keeping the half-planes world-anchored and so cell-local / causal-safe: |
| * (a) calibrate q on the mean frame or the last frame instead of frame 0; |
| * (b) cap per-frame coverage by clamping τ per frame, e.g. τ_f = max(τ, the (1−0.4) quantile of frame f's static projections); |
| * (c) keep the current behaviour and record the decision in SPEC §7. |
| |
| This fix pass did not change the behaviour because SPEC §7 is authoritative. Options (a) and (b) are each a few lines in |
| `dataset.sample_planes` / `planes_mask` and need a matching update to the coverage test. |
| * A half-weight tail step (one micro-batch) happens only when the per-rank epoch length is odd. That is true on mini with 4 GPUs (25), |
| but **not for the full run under the SPEC split (1,494 per rank, even)**. When it does happen, accelerate passes the boundary to DeepSpeed |
| (`set_gradient_accumulation_boundary`) and the step count uses the ceiling (see Trainer), so the bookkeeping is consistent. |
| * The val subset is upstream's `limit` (seeded shuffle), so it is not run-balanced. sample.py's `--val_n` is run-balanced. |
| * Warm start from an `ids`-vocab checkpoint into a `parts` run is not supported (`load_gc_weights` refuses the vocab mismatch). It would need a |
| proj_in channel expansion. |
| * `sample.py` loads the DiT straight onto the GPU. Load takes 6–8 s and an 8-step clip with VAE decode, recon ceiling and SSIM takes ~25 s. |
| |
| ## Review fixes (2026-10-22, fixer pass) |
| Two independent reviewers produced 9 findings, which reduce to 5 distinct issues. Each was verified against the code, the logs and SPEC before any change. |
| |
| **Fixed** |
| 1. **Resume replayed identical samples, and the epoch counter used floor** (`train.py`). This was the only finding with code consequences. |
| * Verified: in the smoke, steps 11/12/13 reproduced steps 1/2/3 exactly, and the timing run had 13 steps per epoch while the log printed 12. |
| * Root cause: accelerate does not save the sampler generator for map-style datasets, the epoch was derived as `start_step // floor`, and |
| the dropout RNG is keyed on (seed, epoch, idx). |
| * Fix: `SeedableRandomSampler` with `data_seed`, the data position stored in `trainer_state.json`, `skip_first_batches` on resume, |
| `epoch_steps` = ceil, and a geometry check that falls back to the next epoch. Details are under Trainer → Resume. |
| 2. **`init_ckpt` lineage after a dir resume.** Verified: step-13 recorded `init_ckpt=…/sft/state`, a rotating symlink. Now `init_ckpt` is only |
| ever a warm-start `.safetensors` or `""`. It is carried through `trainer_state.json` (legacy states fall back to `config.json`), and the |
| resume is recorded separately as `resumed_from` (slot realpath) and `resumed_step`. |
| 3. **`step_time` excluded data wait.** Verified: the timing run had 106.6 s of wall time against Σstep_time = 100.9 s. |
| * `step_time` is now the wall time from the end of the previous step, with validation and save time excluded. |
| * The new `data_time` column records the data-wait part, and `done.json` has `data_s_per_step_mean`. |
| * `clips` (rank-0 clip_ids) was added so data continuity can be checked from the log. |
| * Because the columns changed, an old-header CSV in a resumed OUT is renamed first. |
| 4. **Full-run numbers used the 20-run split** (this doc). Verified: SPEC §5 gives 11,955 train clips, 1,494 per rank, 747 steps per epoch and ≈26.8 epochs. |
| The half-weight-step note is now scoped to odd per-rank epochs, which the full run does not have. |
| |
| **Not changed (lead decision)** |
| 5. **Mask coverage grows over the clip** (both reviewers, "design risk, not a code defect"). The code implements SPEC §7 literally, and SPEC is |
| authoritative, so I kept the behaviour. The evidence and options (a)/(b)/(c) are under Open issues and need a decision before the full run. |
| |
| **Fix smoke** (4 procs on GPUs 2–5, GA 2, mini, `outputs/_smoke/sft/fix_A`, `fix_B`; comparison in `fix_tools/compare_resume.{py,out}`): |
| ``` |
| # A: warm start from step-13.safetensors, 16 steps uninterrupted (crosses the 13-step epoch boundary), save/val at 8 and 16 |
| OUT=outputs/_smoke/sft/fix_A RESUME=outputs/_smoke/sft/step-13.safetensors NGPU=4 GPUS=2,3,4,5 GA=2 MAX_STEPS=16 DATA=data/gc25d_v10_mini \ |
| EXTRA="--save_every 8 --val_every 8 --val_n 4 --warmup_steps 2" bash scripts/train_sft.sh |
| # B: resume from A's step-8 slot into a separate OUT, run to 16 |
| OUT=outputs/_smoke/sft/fix_B RESUME=outputs/_smoke/sft/fix_A/state_A … (same) bash scripts/train_sft.sh |
| ``` |
| | check | result | |
| |---|---| |
| | warm start | `step-13.safetensors` loaded (831 tensors, `has_sem_cond`). Step 1 has sem_rms 2.95e-3, not 0 | |
| | ceil steps/epoch | log: `25 个 micro batch = 13 步`. Step 13 is the 1-micro tail (one clip in `clips`), and epoch 1 starts at step 14 | |
| | state position | A's `state_A` holds `{step 8, epoch 0, epoch_micro 16, …}` and `state_B` holds `{step 16, epoch 1, epoch_micro 6}` | |
| | **exact resume** | B: `数据从 epoch 0 的第 16 个 micro batch 接上（精确）`. **Steps 9–16 of B equal A row by row** in `clips`, `epoch`, `text_drop`, `cond_drop` and `mask_frac`. No (clip, epoch) pair from steps 1–8 reappears. Loss differs because `t` is reseeded, as upstream does | |
| | lineage | A's step-8 and step-16 have `init_ckpt=…/sft/step-13.safetensors`, `resumed_from=""`. B's step-16 has the **same `init_ckpt`**, `resumed_from=…/fix_A/state_A`, `resumed_step=8` | |
| | exports | 831 BF16, `check_gc_ckpt` OK, vocab parts (A: step 8 and 16; B: step 16) | |
| | timing (A) | steady 2.41–2.45 s/step wall, data 0.018 s; tail step 13: 1.95 s (data 0.30); epoch start 14: 3.93 s (data 1.11, worker re-fork); mean over steps 3–16 **2.51 s (data 0.12)**. Peak memory 35.2 / 35.3 GB | |
| | timing (B) | 6.4 s/step: another job's 8 processes (~9.5 GB, up to 100 % utilisation on all 8 GPUs) started during the smoke. I did not touch them. The numbers above are run A's | |
| |
| After checking, I deleted the fix smoke's state slots and exports (≈225 GB) and kept the logs, configs and comparison output. The older |
| `outputs/_smoke/sft/step-{5,10,13}.safetensors` and `state_A` (legacy `trainer_state` = `{step, arm, vocab}`) are untouched. Resuming that legacy |
| state uses the "derived from step" path (step 13 ⇒ epoch 1, skip 0). |
| |
