docs/impl/sft.md

sft — bidirectional gc25d teacher on v10 (SPEC §6–§7)

Status 2026-10-22: code done, CPU-tested (30 tests) and smoke-tested with real weights on data/gc25d_v10_mini (4×H200, GPUs 2–5). A review-fix pass then made resume exact (data position and lineage) and added data-wait timing. See "Review fixes" at the end. The full run has not been started. It needs data/gc25d_v10 (full prep), and its command is at the end.

What

file	role
code/gc25d/dataset.py	GcLatentDataset (latents + parts ids + text table + the three dropouts + world-anchored UNKNOWN masking) and gc_collate
code/gc25d/train.py	fork of ActionRoPE actionrope/train.py (same section order): GcTrainModule, sem_param_groups, warmup_lambda, run_validation (cond / uncond / null-text), export through gcdistill.gc_dit.save_gc + check_gc_ckpt, rotating save_state, resume / warm start
scripts/train_sft.sh	launcher: env vars → accelerate yaml (8×GA2 and 8×GA1 use the existing files, other combinations are generated as configs/accelerate_zero2_<N>gpu_ga<G>.yaml), tee -a $OUT/train.log, refuses a second job on the same OUT
code/gc25d/sample.py	offline eval: TI2V sampling (infer.sample + model.gc_forward) on val clips → mp4 GT | generated | cond (parts coloured) + per-frame PSNR/SSIM json + metrics.json
code/gc25d/tests/test_dataset.py	15 tests: contract, split/drop/limit, val never drops, ids vocab, dropout rates/independence/determinism, masking (world-anchored, characters untouched, coverage, NaN ⇒ masked, speed), collate, real mini clip
code/gc25d/tests/test_train.py	15 tests: tiny WanModel forward/backward through GcTrainModule, manual loss + frame-0 exclusion, sem_rms grows from 0, param groups + warmup, CLI defaults = SPEC §7, val variants, export keys/metadata, lineage metadata, epoch_steps = ceil, resume_position (exact / epoch end / geometry changed / legacy), init_ckpt_of_state, CSV header rotation, and real accelerate on CPU: the seedable-sampler permutation depends only on (seed, epoch), skip_first_batches continues the uninterrupted order, and the sync pattern is [F,T,F,T,T]

Nothing under code/arope/ was touched. model.py, capsules.py and gcdistill/gc_dit.py are unchanged; they are only imported.

How (conventions that matter)

Dataset (dataset.py)

• Sample = input_latents bf16 [48,21,30,52], context bf16 [21,512,4096], context_ids int64 [21], sem_ids uint8 [81,480,832], clip_id, dropped = {"text": bool, "cond": bool, "mask": float}. Here mask is the fraction of all 81×480×832 pixels that the mask set to 255.
• key_to_id = {k: i for i, k in enumerate(sorted(text_table))}. This equals infer._KEY_TO_ID (tested).
• Rows come from index/clips.jsonl filtered by split. Rows with drop=true are excluded unless include_dropped=True. limit shuffles with Random(seed), keeps the first limit rows and sorts them, as AropeLatentDataset does. The val split forces all three dropout probabilities to 0.
• vocab="ids" ⇒ model.collapse_parts after masking.
• RNG: each sample draws from random.Random(f"gc25d:{seed}:{epoch}:{index}"). It always draws three uniforms (text, cond, mask) in that order, then the mask parameters. Results are identical for a given (seed, epoch, index), whatever the worker count or rank. The trainer calls train_ds.set_epoch(epoch) itself before every epoch. With the seedable sampler, accelerate's DataLoaderShard.set_epoch only reaches the sampler and never the dataset. persistent_workers=False because persistent workers would keep a stale epoch.
• Dropouts, as in SPEC §7: text p=0.1 replaces all 21 cells with NULL_PROMPT. cond p=0.05 sets the ids to all 255 (the npz is not even read), independently of text. mask p=0.5 applies only when cond was not dropped.
• World-anchored UNKNOWN mask (SPEC §7, apply_unknown_mask):
  • 1 half-plane with p .7, 2 with p .3. θ ~ U[0,2π) in world XY and q ~ U(0.05, 0.40).
  • The ground plane is Z = meta.ground_z[0]. If that value is NaN the code falls back to the nanmedian, and if there is none it skips the mask.
  • Ground points come from capsules.ground_points on the 4×4 pooled grid, then nearest-upsampled ×4. ground_points only accepts integer dst pixels and the geometric cell centre (4j+1.5, 4i+1.5) falls between pixels, so cell (i, j) uses dst pixel (4j+2, 4i+2). That is half a dst pixel, under 1 source pixel.
  • τ = the (1−q) quantile of the frame-0 static pixels' projections onto θ, weighted per pixel. Every frame then sets static pixels (raw ≤ 6) to 255 when their ground point is beyond any plane or is NaN. Characters (10..21) are never touched.
  • Cost is ≈0.27–0.37 s per masked sample (81 frames × 25k rays). An unmasked sample takes 0.09 s. Training uses 4 workers per rank.

Trainer (train.py)

The section order follows upstream (args / model / export+log / validation / main), so diff -u ref/arope_git/ActionRoPE/actionrope/train.py code/gc25d/train.py is readable. These parts are kept verbatim:

• Accelerator(gradient_accumulation_steps) and set_seed(device_specific=True).
• The DS overrides: --grad_clip is written into deepspeed_config.gradient_clipping, and a GA mismatch with the yaml ⇒ SystemExit.
• LambdaLR is stepped manually, and only on sync_gradients.
• The per-rank cpu_gen timestep tid ~ U{0..999} (seed·1000 + rank + start_step).
• The accumulate block, dit.freqs moved to the GPU, config.json/done.json, and the CSV + tensorboard logs.

Changes from upstream:

• GcTrainModule.forward: x_t = add_noise, then frame 0 is replaced by clean x0, then model.gc_forward(..., first_frame_cond, context_ids, grad ckpt when training, aux). Frame 0 is dropped from the loss, and loss = MSE × training_weight(t). The returned stats are {mse, timestep, sem_rms, x_rms}.

• Optimizer: AdamW(sem_param_groups(module, lr, cond_lr, wd)) = [all DiT params at --lr, dit.gc_sem_cond.* at --cond_lr]. The betas stay at the torch defaults (0.9, 0.999). The warmup factor multiplies both groups. sem_param_groups takes a prefix (default dit.gc_sem_cond.) so the distill runners can reuse it.

• Validation: fixed val_n val clips (limit + seed) × --val_timesteps × fixed noise. Each (clip, t) runs three forwards with the same noise:

  • cond: true ids and true text;
  • uncond: ids all 255;
  • nulltext: NULL_PROMPT × 21.

  val_log.csv holds step, timestep, mse, mse_uncond, mse_nulltext, gap_uncond, gap_nulltext, n, and tensorboard gets val/*_t{t} and val/*_mean.

• Export: gcdistill.gc_dit.save_gc writes 831 BF16 keys with the dit. prefix stripped, through .tmp + rename and chmod 644. It is followed by check_gc_ckpt and a vocab check. Metadata = GC_META (arm=gc25d, format=gc25d, sem_cond=gc_sem_cond, mask_channel=False) plus vocab, data=v10, data_root, step, stage=sft_bidirectional, cond_type=ids, cond_key=ids, text_dropout, cond_dropout, mask_p, lr, cond_lr, weight_decay, warmup_steps, first_frame_cond, seed, init_ckpt, model_dir, trainer, all as strings.

• save_state = distill.runlib.save_state_rotating: two slots plus the state symlink. trainer_state.json = {step, arm, vocab, init_ckpt, epoch, epoch_micro, num_processes, grad_accum, batch_size, n_micro, n_train, seed}. Here epoch_micro is the number of micro-batches this rank has consumed in epoch, and n_micro is len(dataloader) per rank.

• Data order: Accelerator(dataloader_config=DataLoaderConfiguration(use_seedable_sampler=True, data_seed=--seed)). accelerate replaces the RandomSampler with SeedableRandomSampler, so each epoch's permutation is a pure function of (seed, epoch) and is the same on every rank before sharding. An explicit data_seed is needed because set_seed(device_specific=True) gives each rank a different torch seed. Upstream instead uses the sampler generator's running state, which load_state does not save (accelerate only saves sampler state for IterableDatasets), so upstream cannot resume mid-epoch.

• Steps per epoch = epoch_steps(len(dataloader), GA) = ceil. accelerate forces a sync at the end of the dataloader, so a tail shorter than GA is still one step (mini 4 GPUs: 25 micro ⇒ 13 steps; tested against real accelerate: [F,T,F,T,T] for 5 micro at GA 2).

• Resume:

  • --resume <dir>: load_state, fast-forward the LR schedule and reseed cpu_gen (seed·1000 + rank + start_step, as upstream). A vocab mismatch ⇒ SystemExit. The data position continues exactly (resume_position):
    • If the stored geometry (num_processes, grad_accum, batch_size, n_micro, n_train, seed) matches, the trainer resumes in the stored epoch and skips its first epoch_micro micro-batches with accelerator.skip_first_batches. The skip happens in the batch sampler, so skipped clips are never loaded. Every later (clip, text/cond/mask draw) is then the same as in an uninterrupted run. epoch_micro is a multiple of GA (or the whole epoch), so the accumulation boundaries do not move. If the save fell on the epoch's last micro-batch, the run starts the next epoch.
    • If the geometry changed, the rest of the current epoch is abandoned and the run starts the next epoch from its beginning. That gives a new permutation and new draws, so no sample is replayed with identical draws.
    • A legacy trainer_state.json with only {step, arm, vocab} derives epoch = step // ceil-steps-per-epoch and skips the rest of the way into it.
    • What is still not bit-exact: t (cpu_gen is reseeded, as upstream) and DataLoader worker seeds. The noise ε comes from the CUDA RNG, which load_state restores.
  • --resume X.safetensors: warm start through model.load_gc_weights. A gc25d export must have the same vocab. A plain Wan export with no gc_sem_cond.* keys gets a zero-initialised cond.
  • Lineage metadata: init_ckpt is the warm-start .safetensors, or "" for base Wan with a zero-initialised cond. A dir resume never changes it: the value is carried in trainer_state.json, and a legacy state falls back to the run's config.json. The resume is recorded separately as resumed_from (realpath of the slot) and resumed_step.

• train_log.csv holds step, loss, mse, timestep, lr, cond_lr, grad_norm, sem_rms, x_rms, text_drop, cond_drop, mask_frac, step_time, data_time, mem_gb, epoch, clips:

  • step_time is the wall time of the whole optimizer step, including waiting for data. data_time is the part spent waiting for the DataLoader, which includes the worker re-fork at each epoch start. Validation and save time are excluded.
  • clips lists rank 0's clip_ids for the step's micro-batches, joined with ;.
  • done.json has s_per_step_mean (wall time), data_s_per_step_mean and data_pos.
  • If an existing CSV in OUT has a different header (code upgrade plus resume), it is renamed to *.old<N>.csv first, so rows never land under the wrong header.
  • A non-finite reduced loss ⇒ SystemExit.

• Default --num_workers is 4 (upstream 2) because of the mask cost. --save_state and --val_uncond default on (SPEC §7).

• A train/val leak check (shared clip or shared run) ⇒ SystemExit.

Launcher (scripts/train_sft.sh)

• Environment variables: OUT (required, must resolve under $M/outputs/), NGPU (8), GPUS (default 0..NGPU−1; NGPU must match), GA (2), MAX_STEPS (20000), RESUME, VOCAB (parts), DATA ($M/data/gc25d_v10), PORT (default: a free port), EXTRA, and ALLOW_EXISTING=1 to start from 0 in an OUT that already has checkpoints.

• Exports DIFFSYNTH_SKIP_DOWNLOAD, PYTHONPATH, HF_HOME, and puts TMPDIR, TORCHINDUCTOR_CACHE_DIR, TRITON_CACHE_DIR, TORCH_EXTENSIONS_DIR, TORCH_HOME and XDG_CACHE_HOME all under $M/.tmp.

• Refuses to start (exit 3) in three cases:

  • the flock on $OUT/.train_sft.lock is taken;
  • a train.py … --output $OUT process exists;
  • OUT already holds step-*.safetensors or state but no RESUME was given.

  It also refuses when $DATA/index/clips.jsonl is missing, and only warns when READY is missing.

• Generated configs: configs/accelerate_zero2_4gpu_ga2.yaml, made during the smoke from accelerate_zero2_ga2.yaml by changing num_processes and gradient_accumulation_steps only.

sample.py

• Supports --ckpt X | --base [--vocab], --data_root, --raw_root, --clip_ids a,b | @file | --val_n N (val, drop=false, round-robin over runs), --steps 50 --shift 5 --cfg 1 --sem_cfg 1 --sem_mode cond|unknown --seed 0 --out_dir, --no_ssim and --no_recon_ceiling.
• It does not use infer.load_models, which installs the default ids vocab first, so a parts checkpoint would raise. Instead it calls GM.load_dit + GM.load_gc_weights (the vocab comes from the checkpoint).
• GT = raw tar jpgs <raw_root>/<subset>/<scene>/<run> read with infer.read_raw_frames, then geometry.resample_rgb.
• The ids panel shows what the model received. Static classes use geometry.PALETTE; player parts are a magenta/violet family with a cyan weapon, boss parts a red/orange family with a blue weapon; 255 is hatched.

Tests (CPU)

cd /opt/dlami/nvme/zhiyangdeng/maliang
DIFFSYNTH_SKIP_DOWNLOAD=True PYTHONPATH=$PWD/code/gc25d:$PWD/code/arope TMPDIR=$PWD/.tmp OMP_NUM_THREADS=8 \
  .venv/bin/python -m pytest -q code/gc25d/tests/test_dataset.py code/gc25d/tests/test_train.py     # 30 passed, ~31 s

What the masking tests check:

• World anchoring. Synthetic UE-convention cameras (orthographic and reversed-Z perspective) translate over 81 frames. World ground points more than 60 cm from a plane boundary are projected into every 5th frame. At static pixels, the mask equals the world half-plane test exactly. More than 500 points are seen in at least 3 frames.
• Characters. Pixels 10..21 never change, and only static pixels become 255.
• Coverage. With one plane, the frame-0 static fraction is within 0.01 of q. With two planes it lies in [max q, Σq]. Over 40 seeds, 2-plane draws stay within the expected count.
• NaN. Sky rays from a shallow perspective camera are masked.

Smoke (real weights, outputs/_smoke/sft/)

Setup: 4 procs on GPUs 2–5, GA 2 ⇒ global batch 8. I chose GA 2 over GA 4 (batch 16) so that one optimizer step does the same per-GPU work (2 micro-batches) as the full 8-GPU GA2 run, which makes the measured s/step the full-run s/step. Data: data/gc25d_v10_mini (100 train / 100 val clips).

OUT=outputs/_smoke/sft NGPU=4 GPUS=2,3,4,5 GA=2 MAX_STEPS=10 DATA=data/gc25d_v10_mini \
  EXTRA="--save_every 5 --val_every 5 --val_n 4 --warmup_steps 2" bash scripts/train_sft.sh
OUT=outputs/_smoke/sft NGPU=4 GPUS=2,3,4,5 GA=2 MAX_STEPS=13 RESUME=outputs/_smoke/sft/state DATA=data/gc25d_v10_mini \
  EXTRA="--save_every 5 --val_every 5 --val_n 4 --warmup_steps 2" bash scripts/train_sft.sh

check	result
loss finite	steps 1–13: 0.05–0.42 (× training_weight). grad_norm 0.16–0.78, with one 6.07 spike in a repeat run that did not reappear across 3 more epoch boundaries
sem_rms grows from 0	step 1: 0.0, step 2: 3.8e-4, step 3: 2.3e-3, step 10: 3.9e-3 (x_rms ≈ 0.35, so ≈1 %). cond lr warmup 5e-5 → 1e-4
val rows	val_log.csv: steps 5 and 10 × t ∈ {201.3, 500.6, 799.7}, n=4. At step 10: mse 0.377/0.176/0.149, gap_uncond +0.0013/+0.0005/−0.0045, gap_nulltext +0.0022/+0.0016/+0.0010. As expected after 10 steps, the cond gap is ~0
export	step-5, step-10 and step-13.safetensors: each 831 tensors, all BF16, 10.02 GB, check_gc_ckpt OK, ckpt_vocab = parts, metadata as above
save_state	state_A (step 5) and state_B (step 10), 66 GB each on 4 GPUs. The state symlink rotated correctly
resume (pre-fix code)	--resume …/state (→ state_B, step 10) continued at 11–13 with the LR at its plateau and sem_rms continuous (3.3e-3). It exported step-13 and rotated state → state_A. Review found that steps 11–13 replayed the exact samples of steps 1–3 (same clips and the same mask_frac 0.04015…/text drop/0.08591…). This is fixed; see "Review fixes"
second launch on the same OUT	refused: 已有训练在跑（锁 … 被占用）, exit 3
single process, no DS (python train.py --limit 8 --max_steps 2)	OK: 831-key export, peak 46.9 GB (plain bf16 AdamW, numbers not representative)
s/step	2.40 s of compute (40-step run with no val/saves, GPUs uncontended) ≈ 1.2 s per micro-batch. This matches ActionRoPE's 1.20 s/step at GA 1. The first step takes 7.1 s. Each epoch ends with one 1-micro step (~1.35 s): per-rank epoch length 25 is odd, and accelerate syncs at the end of the dataloader. Since the fix, step_time is wall time including data wait: 2.41–2.45 s steady with 0.018 s data wait, and 2.51 s mean over steps 3–16 including the epoch-boundary steps (see the fix smoke)
peak memory	35.2 GB (main) / 35.3 GB (max) per GPU on 4 GPUs. On 8 GPUs ZeRO-2 shards the optimizer 8-way: expect ≈ 25–28 GB (ActionRoPE measured 32.8 GB on 4 and 22.7 GB on 8)

The first smoke ran while another project's (PCE) job held ~39 GB at 100 % utilisation on all 8 GPUs. The step time was the same once it had left.

UNKNOWN mask statistics on mini train with default probabilities (53/100 samples masked):

measure	mean	median	p90
frame-0 static fraction masked	0.31
whole-clip pixel fraction	0.39		0.80
frame-80 static fraction	0.47		1.00
share of frames > 90 % masked	0.126	0	0.54

sample.py on 1 val clip (02_Autumn_Woodland_Shrine__P9_golden_s2909__002400), 8 steps:

model	PSNR frames 1–80	SSIM	frame-0 PSNR	VAE ceiling	sampling	total
--ckpt outputs/_smoke/sft/step-10.safetensors	15.46 dB	0.571	36.86	31.54	3.6 s	28 s
--base	15.01 dB	0.540	36.86	31.54	3.4 s	24 s

Outputs are in outputs/_smoke/sft/sample/{step-10_s8,base_s8}/. I removed the timing run's and the 1-proc run's checkpoints, and the stale state_B. outputs/_smoke/sft/ keeps step-{5,10,13}.safetensors (real-weight 831-key parts checkpoints; the distill smokes can use them) and state_A.

Full run

cd /opt/dlami/nvme/zhiyangdeng/maliang && OUT=/opt/dlami/nvme/zhiyangdeng/maliang/outputs/sft_v10 NGPU=8 GA=2 MAX_STEPS=20000 VOCAB=parts \
  DATA=/opt/dlami/nvme/zhiyangdeng/maliang/data/gc25d_v10 nohup bash scripts/train_sft.sh \
  > /opt/dlami/nvme/zhiyangdeng/maliang/outputs/sft_v10.launcher.out 2>&1 &
# crash → same command plus RESUME=/opt/dlami/nvme/zhiyangdeng/maliang/outputs/sft_v10/state

This uses the SPEC §7 defaults throughout:

• lr 1e-5, cond_lr 1e-4, wd 0.01, warmup 500, clip 1.0;
• 8 × micro 1 × GA 2 = 16;
• save_every 1000 plus rotating state, val_every 500 with val_n 16 at t 200/500/800 in all three variants;
• dropouts 0.1 / 0.05 / 0.5, grad ckpt on.

Estimate: 20,000 × 2.43 s wall ≈ 13.5 h of training. Data wait is ~0.02 s/step, plus a ~1.1 s worker re-fork once per 747-step epoch, which is negligible. Add 20 exports (10 GB, <30 s each), 20 state saves (~66 GB per slot, <1 min each) and 40 validations (16 clips × 3 t × 3 variants over 8 ranks, ~10 s each), for ≈ 14 h in total. Disk: 20 × 10 GB step files + 2 × ~66 GB state ≈ 330 GB on NVMe. Under the SPEC §5 split (10 val runs = 500 clips), computed from the raw per-run clips_81.json files and the buried list:

• 12,800 clips = 12,300 train + 500 val. 345 of the 348 buried clips are in train runs and 3 in val runs.
• That leaves 11,955 train clips: 11,955 // 8 = 1,494 micro-batches per rank (even), 747 steps per epoch, ≈ 26.8 epochs in 20k steps.
• An earlier version of this doc used the investigator's 20-run split, which gives 11,452 train, 715 steps per epoch and 28 epochs. That split is not the SPEC split. Re-check against data/gc25d_v10/index/clips.jsonl once prep finishes: config.json → dataset.geometry / steps_per_epoch prints the real values.

Watch:

• train/sem_rms and train/sem_over_x should keep rising.
• val/gap_uncond_mean should become clearly positive, which shows the condition is used. gap_nulltext measures the text.

Open issues / notes

• LEAD DECISION NEEDED BEFORE THE FULL RUN: mask coverage grows over the clip. Both reviewers flagged this, and it is not a code defect: the code follows SPEC §7 to the letter, with τ calibrated on frame 0 only. The half-planes are fixed in the world and the camera travels several metres in 81 frames, so the mask grows. Evidence on mini:

  • My run (53/100 masked): frame 80 averages 47 % masked and 12.6 % of frames are > 90 % masked.
  • Reviewer A (100 train clips, mask forced on), last-cell static coverage: ortho mean 0.35, p90 0.96; perspective mean 0.47, p90 1.00. 18–20 % of masked clips end more than 90 % masked.
  • Reviewer B (50 perspective + 50 ortho clips × 3 seeds), whole-clip static coverage: mean 0.40 (perspective) / 0.37 (ortho), p90 0.86 / 0.82. Frame 80 has p90 = 1.0. 14.0 % / 11.7 % of masked frames are > 90 % masked, and 12–16 % are < 1 % masked because the mask slid off screen.
  • Overall about 6 % of all training frames have no static condition, on top of the 5 % cond dropout.

  At deploy, canvases are 8–40 % unseen, and the auto-demo keeps black ≤ 10 % in almost every frame (ref/gc25d-live-demo/README.md:36, :102–103). Training is therefore much heavier than deploy in the tail.

  Options, all keeping the half-planes world-anchored and so cell-local / causal-safe:

  • (a) calibrate q on the mean frame or the last frame instead of frame 0;
  • (b) cap per-frame coverage by clamping τ per frame, e.g. τ_f = max(τ, the (1−0.4) quantile of frame f's static projections);
  • (c) keep the current behaviour and record the decision in SPEC §7.

  This fix pass did not change the behaviour because SPEC §7 is authoritative. Options (a) and (b) are each a few lines in dataset.sample_planes / planes_mask and need a matching update to the coverage test.

• A half-weight tail step (one micro-batch) happens only when the per-rank epoch length is odd. That is true on mini with 4 GPUs (25), but not for the full run under the SPEC split (1,494 per rank, even). When it does happen, accelerate passes the boundary to DeepSpeed (set_gradient_accumulation_boundary) and the step count uses the ceiling (see Trainer), so the bookkeeping is consistent.

• The val subset is upstream's limit (seeded shuffle), so it is not run-balanced. sample.py's --val_n is run-balanced.

• Warm start from an ids-vocab checkpoint into a parts run is not supported (load_gc_weights refuses the vocab mismatch). It would need a proj_in channel expansion.

• sample.py loads the DiT straight onto the GPU. Load takes 6–8 s and an 8-step clip with VAE decode, recon ceiling and SSIM takes ~25 s.

Review fixes (2026-10-22, fixer pass)

Two independent reviewers produced 9 findings, which reduce to 5 distinct issues. Each was verified against the code, the logs and SPEC before any change.

Fixed

1. Resume replayed identical samples, and the epoch counter used floor (train.py). This was the only finding with code consequences.
   • Verified: in the smoke, steps 11/12/13 reproduced steps 1/2/3 exactly, and the timing run had 13 steps per epoch while the log printed 12.
   • Root cause: accelerate does not save the sampler generator for map-style datasets, the epoch was derived as start_step // floor, and the dropout RNG is keyed on (seed, epoch, idx).
   • Fix: SeedableRandomSampler with data_seed, the data position stored in trainer_state.json, skip_first_batches on resume, epoch_steps = ceil, and a geometry check that falls back to the next epoch. Details are under Trainer → Resume.
2. init_ckpt lineage after a dir resume. Verified: step-13 recorded init_ckpt=…/sft/state, a rotating symlink. Now init_ckpt is only ever a warm-start .safetensors or "". It is carried through trainer_state.json (legacy states fall back to config.json), and the resume is recorded separately as resumed_from (slot realpath) and resumed_step.
3. step_time excluded data wait. Verified: the timing run had 106.6 s of wall time against Σstep_time = 100.9 s.
   • step_time is now the wall time from the end of the previous step, with validation and save time excluded.
   • The new data_time column records the data-wait part, and done.json has data_s_per_step_mean.
   • clips (rank-0 clip_ids) was added so data continuity can be checked from the log.
   • Because the columns changed, an old-header CSV in a resumed OUT is renamed first.
4. Full-run numbers used the 20-run split (this doc). Verified: SPEC §5 gives 11,955 train clips, 1,494 per rank, 747 steps per epoch and ≈26.8 epochs. The half-weight-step note is now scoped to odd per-rank epochs, which the full run does not have.

Not changed (lead decision) 5. Mask coverage grows over the clip (both reviewers, "design risk, not a code defect"). The code implements SPEC §7 literally, and SPEC is authoritative, so I kept the behaviour. The evidence and options (a)/(b)/(c) are under Open issues and need a decision before the full run.

Fix smoke (4 procs on GPUs 2–5, GA 2, mini, outputs/_smoke/sft/fix_A, fix_B; comparison in fix_tools/compare_resume.{py,out}):

# A: warm start from step-13.safetensors, 16 steps uninterrupted (crosses the 13-step epoch boundary), save/val at 8 and 16
OUT=outputs/_smoke/sft/fix_A RESUME=outputs/_smoke/sft/step-13.safetensors NGPU=4 GPUS=2,3,4,5 GA=2 MAX_STEPS=16 DATA=data/gc25d_v10_mini \
  EXTRA="--save_every 8 --val_every 8 --val_n 4 --warmup_steps 2" bash scripts/train_sft.sh
# B: resume from A's step-8 slot into a separate OUT, run to 16
OUT=outputs/_smoke/sft/fix_B RESUME=outputs/_smoke/sft/fix_A/state_A … (same) bash scripts/train_sft.sh

check	result
warm start	step-13.safetensors loaded (831 tensors, has_sem_cond). Step 1 has sem_rms 2.95e-3, not 0
ceil steps/epoch	log: 25 个 micro batch = 13 步. Step 13 is the 1-micro tail (one clip in clips), and epoch 1 starts at step 14
state position	A's state_A holds {step 8, epoch 0, epoch_micro 16, …} and state_B holds {step 16, epoch 1, epoch_micro 6}
exact resume	B: 数据从 epoch 0 的第 16 个 micro batch 接上（精确）. Steps 9–16 of B equal A row by row in clips, epoch, text_drop, cond_drop and mask_frac. No (clip, epoch) pair from steps 1–8 reappears. Loss differs because t is reseeded, as upstream does
lineage	A's step-8 and step-16 have init_ckpt=…/sft/step-13.safetensors, resumed_from="". B's step-16 has the same init_ckpt, resumed_from=…/fix_A/state_A, resumed_step=8
exports	831 BF16, check_gc_ckpt OK, vocab parts (A: step 8 and 16; B: step 16)
timing (A)	steady 2.41–2.45 s/step wall, data 0.018 s; tail step 13: 1.95 s (data 0.30); epoch start 14: 3.93 s (data 1.11, worker re-fork); mean over steps 3–16 2.51 s (data 0.12). Peak memory 35.2 / 35.3 GB
timing (B)	6.4 s/step: another job's 8 processes (~9.5 GB, up to 100 % utilisation on all 8 GPUs) started during the smoke. I did not touch them. The numbers above are run A's

After checking, I deleted the fix smoke's state slots and exports (≈225 GB) and kept the logs, configs and comparison output. The older outputs/_smoke/sft/step-{5,10,13}.safetensors and state_A (legacy trainer_state = {step, arm, vocab}) are untouched. Resuming that legacy state uses the "derived from step" path (step 13 ⇒ epoch 1, skip 0).