maliang / docs /impl /res704.md
teawhite's picture
Add files using upload-large-folder tool
51ac7cc verified
|
Raw History Blame Contribute Delete
38.7 kB

res704: resolution-configurable gc25d stack (GC_RES), target 1280×704

Status 2026-10-23. Code done. Tests pass at both resolutions. Real-weight smokes ran at 1280×704 on GPUs 0–3: prep, verify, ext_canvas, 4-GPU SFT, sample, three full-shape dry runs, stage1/2/3 memory probes and the eval twin. The 480p pipeline is unchanged: the default GC_RES=832x480 reproduces every old constant and index table bit for bit (tested). data/gc25d_v10 and outputs/sft_v10 were not touched. Nothing has been launched at full scale. The commands for the full 704p prep → SFT → distill chain are at the end. A review fix pass on the same day is summarised right below: deploy res guard, parse_res lower bound, launcher normalisation, chain path guards, the capsule bbox, ext_canvas threads and verify thresholds. The final test and smoke numbers are there too.

Review fix pass (2026-10-23)

Ten reviewer findings were each verified first, then fixed, partly fixed or rejected. 832×480 output stays bit-identical (the old-constant / _IDX sha256 tests still pass; the capsule bbox default is unchanged at 832×480). Logs are in outputs/_smoke/res704_fix/.

# finding verdict what changed
1 deploy engines never check ckpt res confirmed, fixed Res guard at the top of LiveGame.__init__ in the pending deploy patch (deploy_vocab/live_vocab.patch), before any weights are read; FastLive / gen_server inherit it. The same check is in live_path_check.py. ref/ untouched. Also fixed a regression the res704 change had introduced: live_path_check --no_inject (the documented deploy-contract check) crashed, because maliang's data_gc now reads GEO.LAT_H, which the deploy package's hard-coded geometry lacks. The twin side now imports maliang's geometry
2 parse_res accepts values that crash at import; odd CROP_W off-centre confirmed, fixed parse_res requires H ≥ 384 (downscale < 2) and raises a GC_RES ValueError; 115 of the 450 previously accepted values were crashers. CROP_W is rounded to the nearest even number, so the crop is exactly centred and PLAYER_DST = frame centre at every legal res. 480p / 704p unchanged; 1152×640 goes 1267 → 1268. firstframe_flux copy updated. Tests cover all 335 accepted values (import-time asserts simulated) plus a subprocess import per listed res
3 launchers compare raw vs normalised GC_RES confirmed, fixed train_sft.sh, common_gc.sh and chain_after_sft.sh normalise to geometry.RES_TAG after validating. GC_RES=1280X704 now passes against the 704 mini (probe S2: reaches the existing-ckpt gate; S4b: require_data_res OK)
4 chain path defaults ignore GC_RES; unanchored pgrep confirmed, fixed At GC_RES ≠ 832x480 the chain requires SFT_OUT, D and DATA_ROOT. Before waiting it checks the res of DATA_ROOT, each existing $D/stage*/config.json and SFT=; it checks the picked SFT again. pgrep is anchored. All three runners now refuse another resolution's run dir before opening train.log (DG.check_res(run_dir=…))
5 infer.load_ids raw-silhouette guard bypassed at 704p rejected (documented instead) The 10/11 refusal is a gc25d_v8 (ellipse-marker) rule. For v10, silhouettes are the condition: collapse_parts(cond) = raw ids.png for an ids-vocab model, and docs/understand/infer_contract.md:285 already notes the rationale no longer holds. Enforcing it at 704p would block valid v10 ids-vocab inputs. The proposed "parts vocab without 12..21" check cannot apply because infer.py only accepts {0..6, 10, 11, 255}; v10 parts conditions go through sample_user.py. Documented in the module doc and load_ids
6 / 8 render_capsules pad is fixed in dst px, so it clips more at 704p confirmed, fixed differently Scaling the pad only shrinks the error. At 704p, missed character px vs a pad-64 reference over 29 has_ext clips (every 3rd frame) are: pad 2 0.149 %, 2.93 0.103 %, 4 0.067 %, 6 0.030 %. The cause is the world-axis bbox, so cam_axes adds ± r along the camera right/up/forward axes. That gives 0.000 % at both resolutions; on at every res except 832×480, which keeps the old bbox bit for bit (open issue 2). (Finding 8's formula 2·max(1, H/480)·1.4 would give 2.8, not 2.0, at 480p.) The verify_prep 50 px gate is scaled by AREA_VS_480
7 704p answer rests on detail ratio alone confirmed (evidence caveat) scripts/res_test_edge.py adds SSIM / Laplacian correlation / edge F1 (table at the top). 704 vs 640 ties on SSIM and Laplacian correlation, wins on edge recall/F1, and loses 0.24 dB PSNR. Stated in "Why 1280×704"
9 ext_canvas thread env set after numpy import confirmed, fixed The four thread vars are forced to 1 at module top, before numpy (prep_v10 pattern). The plain $ENV (OMP=8) command is now safe
10 verify_prep 480p thresholds; fixed-shape docstrings confirmed (thresholds), rejected (docstrings) Visible-frame gate 50·AREA_VS_480; PSNR WARN lower bound 20 dB off 832×480 (21 at 832×480). The docstrings flagged in verify_prep.py / data_gc.py already carry an "these are the 832×480 example (1280×704: …)" qualifier, so they were left alone

Smoke (1280×704, GPUs 0–3)

  • ext_canvas, 704 mini, 85 clips, 64 workers, plain $ENV (OMP_NUM_THREADS=8 inherited, overridden inside): ≈ 200 s, rc 0. Peak load average 78 while sharing the CPUs with a 28-process job, vs 1,850 before. unknown 0.169, self-agree f80 0.974, f76 0.969 (same as before). New vs old canvases (ext_canvas_old_vs_new.json): blocks / prompts identical in 85/85. 57,081 px changed in 467 of 2,380 frames, all of them character pixels the old bbox had clipped (+0.155 % character px, 0 removed). The old canvases are in .tmp/cond_ext_canvas_704mini_old/.
  • verify_prep, 704 mini, 8 spots: OK, 200 clips, 0 fails, 0 warns (was 2 warns: the two R10_s1512 clips at 20.26 / 20.42 dB are inside the [20, 40] band used off 832×480). The render stats use the camera-axis bbox: v10 player unoccluded IoU median 0.9977 (p10 0.9968), v9_stylized player coverage 0.9997 → 1.0000; boss unchanged (verify_mini.log).
  • 832×480 bit-identity of the bbox change: render_capsules default vs the pre-fix module on 550 real 480p frames (10 mini clips, every 2nd frame): 550/550 bitwise equal.
  • deploy path, live_path_check (3 streamed cells vs eval_rollout.deploy_rollout):
    • 480p EMA through the freshly patched package (--no_inject, .tmp/deploy_patched_fix704): bitwise equal 4/4 cells (regression fixed);
    • 480p EMA through ref/ live.py (injected): bitwise equal 4/4;
    • 704p probe EMA through ref/ live.py at GC_RES=1280x704: frames 13×704×1280, bitwise equal 4/4, DiT ≈ 715 ms/cell (the same overhead-bound eager loop as at 480p, ≈ 700 ms);
    • refusals: the 704p EMA vs the patched package, or vs ref/ at default GC_RES, stops in live_path_check before loading. Patched LiveGame(...) and FastLive(...) called directly raise in 0.00 s (deploy_guard_probes.log).
  • launcher probes (launcher_probes.log): 12 cases (both mismatch directions, illegal res, upper-case res, chain without D, chain aimed at the 480p distill dir / 480p data / a 480p SFT ckpt, 480p default chain); all behave as intended. Nothing was launched.
  • res_test_edge.py (GPU 0, ≈ 3 min): PSNR / detail reproduce res_test.json exactly for all 5 resolutions.

Tests (tests_default.log, tests_704.log)

  • default GC_RES, CUDA_VISIBLE_DEVICES=3: 160 passed, 4 skipped, 322 s (was 158 / 4; the +2 are the new tests below). This includes test_704_suite_subprocess.
  • GC_RES=1280x704, CPU: 156 passed, 8 skipped, 188 s (was 154 / 8). The skips are the same kinds as before.
  • New or extended in test_res.py:
    • test_parse_res: rejects 640x352 / 608x320 / 32x96;
    • test_parse_res_accepted_are_import_safe: all 335 accepted W×H; _IDX asserts hold, the crop is even and centred, aspect error ≤ 1 px;
    • test_listed_res_import_subprocess: real import of geometry + firstframe_flux for 7 resolutions; PLAYER_DST = frame centre; an illegal res is a GC_RES ValueError, never an AssertionError;
    • test_guards: check_run_dir_res, plus stage 1/2/3 main() refusing another resolution's run dir with only config.json left in it;
    • test_capsules_rays_consistent_across_res: with the camera-axis bbox and the default pad 2, both resolutions equal the pad-64 render on the synthetic oblique cameras; the 704p default uses it and the 480p default is the old bbox.

Why 1280×704 (and not 540p / 600-something p)

The user asked "540p or 600+p, whichever fits better". The lead's resolution test is scripts/res_test.py → outputs/char_diag/res_test.json. It measures 40 player crops from 8 clips (5 frames each): VAE round trip at each resolution, upsampled back, compared with the native 1280×704 GT. It also times one 21-cell bidirectional DiT forward. scripts/res_test_edge.py → outputs/char_diag/res_test_edge.json (added in review) reruns the same crops through the same code path and adds three metrics that check where the detail is, not just how much of it there is: grey SSIM, the correlation between the Laplacians of reconstruction and GT, and Canny edge F1 with 1 px tolerance. Its PSNR / detail values match res_test.json exactly.

GC_RES detail ratio crop PSNR SSIM Laplacian corr edge P / R / F1 DiT forward (21 cells) tokens / cell
832x480 (current) 0.375 24.74 0.634 0.207 0.933 / 0.624 / 0.733 0.262 s (1.00×) 390
960x544 ("540p") 0.419 25.24 0.655 0.242 0.941 / 0.652 / 0.759 0.360 s (1.37×) 510
1024x576 0.437 25.47 0.665 0.255 0.943 / 0.664 / 0.768 0.409 s (1.56×) 576
1152x640 ("600+p") 0.468 25.93 0.686 0.288 0.946 / 0.686 / 0.784 0.546 s (2.08×) 720
1280x704 (native) 0.779 25.69 0.679 0.291 0.921 / 0.803 / 0.855 0.691 s (2.64×) 880

Every non-native resolution goes through crop + INTER_AREA downscaling. That resampling throws away fine character detail before the VAE sees the frame. 480p → 640p improves every metric a little. Going from there to native 704p:

  • Edge recall jumps from 0.69 to 0.80 (F1 0.78 → 0.86): many more of the GT's real edges come back.
  • Placement metrics tie. SSIM is 0.686 vs 0.679 and Laplacian correlation 0.288 vs 0.291.
  • Edge precision drops slightly (0.946 → 0.921), and crop PSNR is 0.24 dB lower than at 640p.

Caveat: the detail ratio (0.468 → 0.779) overstates the 640p → 704p gain. It measures Laplacian energy, so it also rewards the absent resampling low-pass, and part of the extra high frequency at 704p is VAE artifact rather than faithful detail. Lower edge precision and PSNR, and a flat SSIM, all show this. The honest summary:

  • 704p beats 480p and 540p on every metric.
  • Against 640p, 704p recovers clearly more of the real edges but is no better on SSIM or PSNR, and costs about 27 % more DiT compute (2.64× vs 2.08× of 480p).
  • 704p is also Wan2.2-TI2V-5B's own resolution, and the data's. Frames and ids pass through unchanged (no crop or rescale), so 704p is still the pick for "600+p". 540p is a cheap middle ground that gains little.

All of these are one env var. If the real-time budget later rules out 704p, GC_RES=960x544 or 1152x640 runs through the same code. The general rule is W, H multiples of 32, 384 ≤ H ≤ 704, W ≤ 1280 and aspect ≤ 1280:704. Each resolution needs its own prep / SFT, because a data root and a ckpt are tied to one resolution.

The contract

  • code/gc25d/geometry.py is the only source of resolution. It parses GC_RES="<W>x<H>" once at import (default "832x480"). Rules: W and H are multiples of 32, 384 ≤ H ≤ 704, W ≤ 1280, and the aspect ratio is at most the source's 1280:704 (wider would need a vertical crop, which is rejected with a message). H ≥ 384 means the downscale factor is < 2. Below that, the centre-aligned _IDX tables cannot reach the first/last source row and column, so the import-time asserts would fail; parse_res now rejects those values first with a GC_RES ValueError. Derived values:
    • DST_W, DST_H, RES_TAG = "<W>x<H>", MIN_H = 384;
    • CROP_W = crop_w_for(W, H) = min(1280, 2·round(352·W/H)), i.e. 704·W/H rounded to the nearest even number, so the crop is exactly centred. CROP_X0 = (1280 − CROP_W)//2. 832×480 gives 1220 / 30, exactly as before; 1280×704 gives 1280 / 0. Only odd cases change, e.g. 1152×640 is now 1268 / 6 instead of 1267 / 6, which was 0.5 source px off-centre;
    • SX, SY, SCALE, _IDX_X/_IDX_Y (centre-aligned nearest), and PLAYER_DST. PLAYER_DST is the exact frame centre at every legal resolution (tested). It is computed multiply-then-divide, so 832×480 gives exactly (415.5, 239.5) as the old literal did, while plain src_to_dst would give 239.49999999999997;
    • new: LAT_H, LAT_W = H/16, W/16 (30×52 → 44×80), POOL_H, POOL_W = H/4, W/4 (120×208 → 176×320), TOK_PER_CELL (390 → 880), AREA_VS_480 (for pixel-count thresholds), IDENTITY (1280×704: no crop, no scaling).
    • resample_rgb / resample_ids take an identity fast path when IDENTITY is set. It returns a C-contiguous copy and is tested to equal the general path, both same-size INTER_AREA and fancy indexing.
    • Child processes (accelerate ranks, DataLoader workers, prep's spawned ranks, ProcessPool workers) inherit the env var, so every process in a run agrees.
  • Resolution guards. A missing res always means "832x480", so all existing 480p roots and ckpts keep working unchanged.
    • Data root: index/prep_args.json["res"]. prep writes it, plus res on every index row. It is checked in GcLatentDataset and GcRolloutDataset, sample.py, sample_user.py --export_clip, infer.py (clip modes), eval_rollout.py (also chain mode, which bypasses the dataset), ext_canvas.py, verify_prep.py, all three distill runners (in parse_args, before loading 5B weights) and scripts/train_sft.sh / run_stage*.sh (require_data_res). prep also refuses to write into an --out that already holds another resolution (prep_args or rows), and merge refuses mixed rows.
    • Ckpt: safetensors metadata res. gc_dit.save_gc always writes it, so SFT exports and every distill product (dits/ ema/ critic/) carry it, and train.export_metadata adds it explicitly. sample.py, sample_user.py, infer.py, eval_rollout.py and the stage1/2/3 runners refuse a ckpt whose res ≠ GC_RES. The runners check the init, teacher and student ckpts, warm starts, and the dits/ema/critic picked up on resume. train.py dir-resume refuses a trainer_state.json of another res (res is now written there). A safetensors warm start with a different res only prints a note, because cross-resolution warm starting is legitimate: the DiT and gc_sem_cond are shape-generic.
    • Distill run dir: <OUT>/config.json["res"] (missing = 832x480; no config.json = new dir). All three runners call DG.check_res(run_dir=…) → geometry.check_run_dir_res before opening train.log. A run aimed at another resolution's run dir therefore stops without appending to its logs and without loading 5B (tested: the refused dir still holds only config.json).
    • Deploy: the pending deploy patch scripts/distill_smoke/deploy_vocab/live_vocab.patch, applied by make_patched_deploy.sh, now adds a res check at the top of LiveGame.__init__, before any weights are read. It refuses a ckpt whose metadata res (missing = 832x480) ≠ the engine's G.DST_W×G.DST_H. fast_live.FastLive and gen_server inherit LiveGame, so they are covered too. scripts/distill_smoke/live_path_check.py does the same check against the geometry its live.py actually uses (maliang's when injecting, which follows GC_RES; the package's own 832×480 with --no_inject). ref/ itself is still untouched, as with the vocab patch.
    • Every error message names GC_RES, the value found, and the fix. Example: --ckpt …/step-8.safetensors 是在 1280x704 上训练 / 导出的(metadata res,缺失 = 832x480),本进程 GC_RES=832x480:设 GC_RES=1280x704,或换成 832x480 的 ckpt.
  • Launchers. All three validate GC_RES and normalise it to geometry.RES_TAG, so " 1280X704" becomes 1280x704. Every shell string comparison and log line therefore uses the canonical value.
    • scripts/train_sft.sh exports GC_RES (default 832x480), refuses to start if DATA's res ≠ GC_RES, and logs it in the banner.
    • gcdistill/common_gc.sh logs [res] GC_RES=… and provides require_data_res, which run_stage{1,2,3}.sh call on real data; their banners print res=.
    • scripts/chain_after_sft.sh takes GC_RES, D (distill root) and DATA_ROOT, and passes each stage its OUT=$D/stage*.
      • With the default GC_RES its defaults equal the old 480p chain exactly; eval_rollout gets --data_root.
      • With any other GC_RES it requires SFT_OUT, D and DATA_ROOT explicitly, because the defaults are 480p directories.
      • Before waiting for the SFT, it checks the res of DATA_ROOT, of every existing $D/stage*/config.json and of an explicit SFT=. It checks the picked SFT ckpt again before stage 1.
      • The SFT wait loop's pgrep is anchored like train_sft.sh's (--output $SFT_OUT( |$)), so a 480p chain no longer waits on a running sft_v10_704 trainer.

What changed per file (shapes that were hard-coded → geometry)

file change
geometry.py GC_RES parsing (H ≥ 384, even crop_w_for), derived constants, identity fast paths, root_res / check_root_res / ckpt_res / check_ckpt_res / check_run_dir_res, docstrings generalised
dataset.py LATENT_SHAPE, IDS_SHAPE, mask grid from geometry; check_root_res in the constructor (ds.res)
infer.py H, W, LAT_H, LAT_W, SRC_* from geometry; messages print the actual shapes; ckpt/data-root res checks; a source-resolution ids array is not resampled when source = target. The --ids 10/11 "raw silhouette" refusal (a gc25d_v8 ellipse-marker rule) can only tell raw from processed ids by shape, so at 1280×704 it does not apply. This is documented in the module doc and load_ids; v10 parts conditions go through sample_user.py
sample.py, sample_user.py shape asserts / warmup latents from geometry; ckpt + data-root res checks; res in json; at 1280×704 a 1280×704 first frame / ids are taken as-is
viz.py PANEL_H, PANEL_W = DST_H, DST_W; the title bar stays 48 px, total height H+48 (752 at 704p) is still a multiple of 16 (asserted)
prep_v10.py LAT_SHAPE from geometry; res in prep_args.json (+ crop_x0/crop_w) and every row; refuses a foreign-res --out. At 1280×704: RGB = raw frame, ids = raw ids.png, label_parts at full res (identity _IDX)
verify_prep.py res check (prep_args + every row), res in the report, log says px per frame (<res>). Thresholds set at 480p are scaled: the visible-frame gate is GT ≥ 50·AREA_VS_480 px; the PSNR WARN band is [21, 40] at 832×480 and [20, 40] elsewhere (busy clips sit ≈ 1 dB lower against native GT)
ext_canvas.py data-root res check, res in the report; canvas cell stays 6 cm (world units). Forces OMP/OPENBLAS/MKL/NUMEXPR_NUM_THREADS=1 at the top, before numpy, like prep_v10. The old setdefault inside main() ran after numpy was imported and never overrode $ENV's 8
capsules.py render_capsules(..., cam_axes=None). The bbox also projects the endpoints ± r along the camera right/up/forward axes (columns of world_to_view). The default is on at every resolution except 832×480, which keeps the old world-axis bbox bit for bit (the existing data/gc25d_v10/cond_ext_canvas was drawn that way). Measured on 29 has_ext clips, every 3rd frame, vs a pad-64 reference: 704p clipped character px 0.149 % → 0; 480p 0.080 % (unchanged default) → 0 with cam_axes=True
model.py docstrings only (already shape-generic)
procmap.py player_px0 ≥ 500·AREA_VS_480; the default out dir gets a _<res> suffix at non-default res (the 480p benchmark cases are not overwritten)
firstframe_flux.py runs in the PCE env (no cv2 ⇒ cannot import geometry) ⇒ copy of parse_res / crop_w_for (tested equal to geometry); output = DST; generation grid gen_grid(W,H) = 16·(a,b)·t with a:b = W:H reduced, t chosen so the area is closest to 0.9 MP and the grid is at least the output size (832×480 → 1248×720 as before, 1280×704 → 1280×704); MIN_PX scaled by area; clip_frame uses the crop rule; clip_ids refuses a cond of another res
gcdistill/gc_dit.py save_gc writes res
gcdistill/data_gc.py LAT_HW, check_res(ckpts=…, root=…, run_dir=…) (SystemExit), GcRolloutDataset root check + latent shape from geometry
gcdistill/ar_tf_gc.py, cd_gc.py, dmd_gc.py synthetic shapes (30,52) → DG.LAT_HW; res checks (parse_args + every real-weight load + run dir before the logs open); res in config / logs / stage1 trainer_state; new --dry_run_full_hw (tiny model, full-res latent + sem grids); dmd budget gate: the KV / scoring / per-grad-frame terms × TOK_SCALE = TOK_PER_CELL/390 (1.0 at 480p ⇒ identical constants), and dry runs skip the gate (it fired on the 5B constants with tiny models whenever a GPU was visible — the two pre-existing test_stage3 failures under CUDA_VISIBLE_DEVICES=0)
gcdistill/eval_rollout.py ckpt + data-root res check (before loading), res in summary
train.py res in export metadata, config.json, trainer_state; dir-resume refuses another res; warm-start note
scripts/train_sft.sh, gcdistill/common_gc.sh, run_stage{1,2,3}.sh, scripts/chain_after_sft.sh see above
configs/accelerate_fsdp_dmd_4gpu.yaml new: 2gpu yaml with num_processes 4. Needed for the 704p stage3 probe, because 2 GPUs fail the budget gate at 704p (126.1 > 126.0 GiB)
scripts/distill_smoke/deploy_vocab/{live_vocab.patch,make_patched_deploy.sh}, scripts/distill_smoke/live_path_check.py deploy res guard (see the contract); make_patched_deploy.sh self-checks that the guard is in the patched live.py; live_path_check asserts the frame shape from live.G instead of a hard-coded 480×832
scripts/res_test_edge.py new: alignment metrics (SSIM, Laplacian correlation, edge F1) for the resolution decision, same crops and code path as res_test.py

Not changed: code/arope/ (vendored; already shape-generic), the other analysis scripts in scripts/ (genbench_compose.py, char_*, distill_smoke/* are 480p tools tied to existing 480p outputs), and ref/gc25d-live-demo (deploy package, see open issues; its guard ships as part of the pending patch).

Tests

  • tests/test_res.py (new):
    • default res: every old geometry name equals the hard-coded old constants, plus _IDX_X/_IDX_Y by sha256; the shapes derived in dataset / infer / viz / prep / data_gc / firstframe_flux / dmd budget equal the old literals; resample_* equal the old implementation;
    • parse_res good and bad values;
    • guards: root / ckpt / prep out / DG.check_res / save_gc writes res;
    • a tiny DiT GcTrainModule step on the real latent + sem grids of the current res;
    • infer.sample / viz.SideBySide shapes;
    • the firstframe_flux copy matches geometry for 8 resolutions;
    • cross-resolution capsules: a 704p subprocess renders capsules, labels parts and casts ground points; sampled with the 480p _IDX, they equal the 480p results (render / label bitwise, ground points ≤ 1e-6);
    • test_704_suite_subprocess: reruns test_res / dataset (synthetic) / train / gcdit (+ data_gc rollout dataset) / stage1 / stage2 / stage3 dry runs / prep / sample_user / firstframe_flux / procmap with GC_RES=1280x704, and the 704p-only checks (identity paths, derived constants, budget scaling) run inside it.
  • Existing tests made resolution-aware, but only where they encoded 480p legitimately: synthetic roots write prep_args.json, rows carry res, shapes come from geometry, and "real mini" tests use data/gc25d_v10_mini at 480p and data/gc25d_v10_704_mini at 704p (skipped if the res differs). Files: test_dataset, test_gcdit, test_stage1, test_stage3, test_prep, test_sample_user, test_firstframe_flux.
  • Results of the implementation pass (outputs/_smoke/res704/tests_default.log, tests_704.log; the review fix pass numbers are in the section at the top):
    • default GC_RES, CUDA_VISIBLE_DEVICES=3: 158 passed, 4 skipped (326 s, includes the 704p subprocess suite). The skips are the two GPU/real-weight gates, which were skipped before too, and two 704p-only tests. Before this change the same command had 2 failures (the dmd budget gate in dry runs).
    • GC_RES=1280x704, CPU, with the 704 mini READY: 154 passed, 8 skipped (189 s). The real-mini tests (test_dataset::test_real_mini, test_stage1::test_7, test_stage3::test_18) run against data/gc25d_v10_704_mini. The skips are the 4 default-only tests in test_res, the 2 GPU/real-weight gates, and 2 tests that need the 480p data/gc25d_v10.

Real smoke at 1280×704 (GPUs 0–3, all logs in outputs/_smoke/res704/)

Mini prep (prep_mini.log): runs R10_s1501, R10_s1512, P9C_autumn_s1941, P9_golden_s2909 → data/gc25d_v10_704_mini (200 clips, 2 GPUs × 12 workers).

  • VAE: 2.15 s/clip (480p: 1.07). Throughput 0.45 clips/s per GPU (GPU-bound; 480p was 0.72, worker-bound). GPU peak 11.4 GiB (480p: 6.5).
  • Worker: 6.2 s/clip (label 3.9 s, write 1.3 s) × 12 workers per GPU ⇒ not the bottleneck.
  • Extrapolated full set: 12,800 clips on 8 GPUs ≈ 60 min (the 480p full prep took 37 min).
  • Disk: latent 7.1 MB/clip (480p 3.15), cond 2.04 MB, cond_ext 0.71 MB ⇒ full set ≈ 125–130 GB (480p: 53 GB).

verify_prep (verify_mini.log): OK, 200 clips, 0 fails, 2 warns, 250 s.

  • A: res 1280x704 everywhere. B: 200/200. C: 8 spot clips: official vae.encode re-encode exact 1.00000; cell 0 bf16-exact; collapse mismatch 0; relabel from tar json 28/28.
  • Capsule re-render from meta: v10 player unoccluded IoU median 0.9977, coverage 1.0000; boss 0.9967. The 480p mini gives 0.9978 / 0.9967.
  • VAE reconstruction PSNR vs native GT: mean 26.62 dB (min 20.26, max 38.14). The 2 warns are two R10_s1512 clips at 20.3 / 20.4 dB, just below the [21, 40] band, which was calibrated at 480p (the same clips at 480p: 21.25 / 22.39). Native-resolution GT has more high frequencies, so expect about 1 dB lower PSNR on busy clips. These are warnings, not failures.

ext_canvas (ext_canvas_mini.log): 85 ext_look_change clips, 64 workers, 180 s. unknown 0.169, self-agreement f80 0.974, f76 0.969. 480p full-set values: 0.157 / 0.977 / 0.969. About 80 s per clip with --report (480p ≈ 52 s). The implementer's first attempt ran without OMP_NUM_THREADS=1, and each of the 64 fork-workers started a full BLAS thread pool (load average 1,850). ext_canvas.py now forces single-threaded BLAS itself (review fix pass), so the plain $ENV command is safe. These canvases were then re-rendered with the camera-axis bbox (see the review fix pass).

SFT smoke (outputs/_smoke/sft704, outputs/_smoke/sft704.launcher.out): 4 GPUs (0–3) ZeRO-2, micro 1 × GA 2, 8 steps, val at step 8 (4 clips × 3 t × 3 variants), export + save_state.

480p (_smoke/sft_timing, same 4-GPU GA2 setup) 704p
s/step (steady) 2.40 5.92 (2.47×), data wait 0.03 s
peak allocated / GPU 35.2 GiB 37.9 GiB (max over ranks 40.0)
first step — 13.0 s
  • Grad checkpointing keeps activations small, so memory barely moves. From 4 to 8 GPUs ZeRO-2 halves the optimizer/grad shard (480p: 35.2 → 24.4 GiB), so 8 GPUs at 704p ≈ 27–29 GiB/GPU.
  • sem_rms rises from 0 (zero init), and val runs (gaps ~0 after 8 steps, as expected).
  • step-8.safetensors: 831 BF16 keys, check_gc_ckpt OK, metadata res=1280x704. trainer_state.json has res. The state slot is 66 GB (4 GPUs).
  • Full run estimate (8 GPUs, GA2, 20,000 steps): per-GPU work is the same as at 4 GPUs, so ≈ 5.9–6.1 s/step ⇒ ≈ 33–34 h of training. Add validation (every 500 steps, ~30 s) and saves (every 1000 steps: 10 GB export + state slot of roughly 70 GB at 8 GPUs) ⇒ ≈ 35 h. At 480p the same run is ≈ 14 h.

sample.py (sample_step8.log): 1 val clip (02_Autumn_Woodland_Shrine__P9_golden_s2909__002400), 8 steps.

  • Sampling 6.6 s (0.83 s/step, one forward per step). Peak 22.6 GB including the non-tiled VAE decode of 81×704×1280. Wall 52 s.
  • Side-by-side mp4 is 752×3840, 81 frames. PSNR is meaningless after 8 SFT steps (15.9 dB; VAE ceiling 33.6).
  • Refusals checked: the 704p ckpt at default GC_RES, and the 480p mini at GC_RES=1280x704. Both stop before loading, with the messages above. train.py --resume outputs/_smoke/sft704/state at default GC_RES stops with … 是 res=1280x704 的存档(缺失 = 832x480),本次 GC_RES=832x480. train_sft.sh refuses both mismatch directions and an illegal GC_RES (1280x640, aspect wider than the source).

Distill dry runs at 704p shapes (dry_s{1,2,3}.log): --dry_run --dry_run_full_hw. Tiny models, but the latent is 44×80 and sem ids are 704×1280 (verified: input_latents (48,5,44,80), sem_ids (17,704,1280)). stage1 13 s, stage2 15 s, stage3 21 s, all exit 0. Exports carry res=1280x704.

Real-weight probes, 704p (init = the SFT smoke ckpt; quality irrelevant):

stage config result
3 DMD FSDP1, 2 GPUs refused by the budget gate: predicted 126.1 GiB > 0.9 × 140 = 126.0 (stage3_probe_2gpu_gate_abort.log)
3 DMD FSDP1, 4 GPUs (0–3), 2 steps, PROFILE_MEM, N = 22 / 27 (stage3_probe/) gate predicted 84.1 GiB. Measured peak 68.5 GiB/GPU (step 0: after rollout 44.7, after critic 61.2; step 1 with N=27: rollout peak 68.2). Probe gnorm 12.1; step 0 dmd = gnorm_g = 0 exactly; replay_dev = 0; step 2: 33 s (rollout 5.9 s, exit 0); step 1: 77 s (first step, includes warm-up). dits/critic/ema exported with res
1 TF ZeRO-2, 2 GPUs (0,1), GA1, 3 steps (stage1_probe/) 11.7 s/step (480p 2-GPU smoke: 3.5–3.7), peak 67.7 GiB (480p: 56.5). gnorm 2.9–4.2; export + state OK
2 CD ZeRO-2, 2 GPUs (2,3), GA1, 3 steps (stage2_probe/) 17.2 s/step (480p: 5.1), peak 98.3 GiB (480p: 85.0). dits/ + ema/ exported

The teacher-forcing double stream (2 × 21 × 880 = 36,960 tokens) makes attention the dominant cost at 704p, so stage1/2 slow down by 3.2–3.4×, more than the SFT's 2.5×. Going from 2 to 8 GPUs removes ≈ 30 GiB of ZeRO-2 shard per GPU (optimizer m/v/master + grads, (1/2 − 1/8) of ≈ 80 GB). Expected 8-GPU peaks: stage1 ≈ 38 GiB, stage2 ≈ 68 GiB. Both fit.

8-GPU stage3 at 704p: of the 68.5 GiB, ≈ 41.9 is FULL_SHARD state (167.7 GiB / 4). At 8 GPUs that state is 21.0, so expect ≈ 48 GiB/GPU peak (gate prediction 63.2 GiB). At 480p the expectation was ≈ 40 GB. It fits comfortably.

Wall-clock plan at 704p, 8 GPUs (per-GPU work at 8 GPUs equals the measured smokes, because the batch per GPU is the same):

step 704p 480p (for reference)
prep / verify / ext_canvas ≈ 1 h / ≈ 0.75 h / ≈ 0.6 h 37 min / 27 min / —
SFT 20k steps (GA2) ≈ 35 h (5.9 s/step + val/save) ≈ 14 h (2.4–2.5 s/step)
stage1 TF 8000 steps ≈ 26 h (11.7 s/step) ≈ 8 h
stage2 CD 8000 steps ≈ 38 h (17.2 s/step) ≈ 11 h
stage3 DMD 500 steps ≈ 5–8 h (33–77 s/step in the 4-GPU probe; rollout length varies) ≈ 3 h
total ≈ 4.5 days ≈ 1.5 days

Stage-2 length (8000) is the documented quality lever. If the calendar is tight, shorten stage 1 first, not stage 2.

Deploy twin (eval_rollout.py) at 704p (eval_clip28_wan/, eval_chain121_tae/, speed_{704,480}/): runs end to end with the probe EMA. clip28 + Wan VAE: 55 s wall. chain121 + TAEHV: 95 s. Same GPU, idle, chain121 + TAEHV, DiT only: 704p 420 ms/cell vs 480p 361 ms/cell. This eval loop is launch-overhead-bound (5 single-cell forwards per cell, eager PyTorch). The compute ratio is ≈ 2.6× (table above), so an optimised deploy path (fast_live.py-style) will pay close to the compute ratio. Real time needs ≤ 250 ms per cell (4 frames at 16 fps). 704p real-time is not established here. It needs the deploy package (see open issues).

Commands (M=/opt/dlami/nvme/zhiyangdeng/maliang)

M=/opt/dlami/nvme/zhiyangdeng/maliang
ENV="DIFFSYNTH_SKIP_DOWNLOAD=True PYTHONPATH=$M/code/gc25d:$M/code/arope HF_HOME=/opt/dlami/nvme/zhiyangdeng/.cache/huggingface TMPDIR=$M/.tmp OMP_NUM_THREADS=8 GC_RES=1280x704"

# 1) full 704p prep, 8 GPUs (≈ 60 min, ≈ 130 GB; resumable: rerun the same command)
env $ENV CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 $M/.venv/bin/python $M/code/gc25d/prep_v10.py \
    --out $M/data/gc25d_v10_704 --gpus 0,1,2,3,4,5,6,7 --workers_per_gpu 12 > $M/outputs/prep_full/prep_704.log 2>&1

# 2) verify (480p full verify took 27 min; expect ~45 min)
env $ENV CUDA_VISIBLE_DEVICES=0 $M/.venv/bin/python $M/code/gc25d/verify_prep.py \
    --root $M/data/gc25d_v10_704 --spot 16 --device cuda:0 --workers 64 > $M/outputs/prep_full/verify_704.log 2>&1

# 3) stage-3 extension canvases (4,772 ext_look_change clips; ≈ 70 s/clip/worker ⇒ ~35 min at 160 workers).
#    ext_canvas.py forces OMP/OPENBLAS/MKL/NUMEXPR threads = 1 itself (the 8 in $ENV is overridden).
env $ENV $M/.venv/bin/python $M/code/gc25d/ext_canvas.py \
    --root $M/data/gc25d_v10_704 --workers 160 > $M/outputs/prep_full/ext_canvas_704.log 2>&1
touch $M/data/gc25d_v10_704/READY            # after verify says OK (exit code 0, "OK" in verify_704.log)

# 4) full SFT from scratch, same recipe as 480p (8 GPUs × micro 1 × GA 2 = 16, 20k steps, lr 1e-5 / cond_lr 1e-4, warmup 500)
GC_RES=1280x704 DATA=data/gc25d_v10_704 OUT=outputs/sft_v10_704 NGPU=8 GA=2 \
    setsid nohup bash $M/scripts/train_sft.sh > $M/outputs/sft_v10_704.launcher.out 2>&1 &
#    crash resume: same + RESUME=outputs/sft_v10_704/state

# 5) distill chain at 704p (waits for sft_v10_704/done.json, picks the teacher, stage1 8000 → stage2 8000 → stage3 500 → eval).
#    At GC_RES ≠ 832x480 the chain refuses to start unless SFT_OUT, D and DATA_ROOT are all given; it checks their res up front.
mkdir -p $M/outputs/distill/gc25d_v10_704
GC_RES=1280x704 SFT_OUT=$M/outputs/sft_v10_704 D=$M/outputs/distill/gc25d_v10_704 DATA_ROOT=$M/data/gc25d_v10_704 \
    setsid nohup bash $M/scripts/chain_after_sft.sh > $M/outputs/distill/gc25d_v10_704/chain.log 2>&1 &
#    to skip the wait/pick: add SFT=<abs path of step-N.safetensors>
#    per stage by hand: GC_RES=1280x704 DATA_ROOT=$M/data/gc25d_v10_704 OUT=$M/outputs/distill/gc25d_v10_704/stage1_tf \
#                       SFT=<sft ckpt> bash $M/code/gc25d/gcdistill/run_stage1.sh   (likewise stage2 AR=…, stage3 STUDENT=… SFT=…)

# sampling / eval at 704p
env $ENV CUDA_VISIBLE_DEVICES=0 $M/.venv/bin/python $M/code/gc25d/sample.py --ckpt $M/outputs/sft_v10_704/step-N.safetensors \
    --data_root $M/data/gc25d_v10_704 --val_n 8 --out_dir $M/outputs/sft_v10_704/eval_step-N

# deploy-path parity at 704p: the live.py model path (ref/ live.py, maliang modules injected ⇒ follows GC_RES) vs
# eval_rollout.deploy_rollout, 3 streamed cells, must be bitwise equal
env $ENV AROPE_ROOT=$M/code/arope CUDA_VISIBLE_DEVICES=0 $M/.venv/bin/python $M/scripts/distill_smoke/live_path_check.py \
    --ckpt $M/outputs/distill/gc25d_v10_704/stage3_dmd/ema/ema_step500.safetensors --data_root $M/data/gc25d_v10_704 \
    --out $M/outputs/distill/gc25d_v10_704/eval/live_path

Open issues / caveats

  1. Deploy package is still 480p. ref/gc25d-live-demo/code/gc25d/{geometry.py,codeworld/*} is a separate hard-coded 832×480 copy (e.g. gen_server.py has H, W = 480, 832) and was not touched. A 704p deploy needs it ported (geometry + code world + server shapes). Until then the pending deploy patch refuses a 704p ckpt in LiveGame.__init__, before any weights are read; it no longer loads silently at 480p. The maliang geometry.py was deliberately not copied into the patched package: the package would then follow an inherited GC_RES=1280x704, and the guard would pass a 704p ckpt into a server that still hard-codes 480×832. The model path itself is resolution-generic: live_path_check at GC_RES=1280x704 (injected maliang modules) streams 3 cells at 704×1280, bitwise equal to eval_rollout (see the review fix pass).
  2. capsules.render_capsules bounding box at 832×480. The world-axis bbox (endpoints ± r along the world axes) underestimates a sphere's screen radius under an oblique camera. The default is now cam_axes on (adds ± r along the camera axes) at every resolution except 832×480, which keeps the old bbox bit for bit, so data/gc25d_v10/cond_ext_canvas and the 480p verify reports reproduce. At 480p the old bbox misses 0.080 % of character px (29 has_ext clips, every 3rd frame); cam_axes=True misses 0. Turning it on at 480p is a one-line default change in render_capsules (the 480p code world and canvases would then gain those pixels); that is the lead's call.
  3. Verify PSNR WARN band: [21, 40] at 832×480 (as before), [20, 40] at other resolutions (native GT is ≈ 1 dB lower on busy clips).
  4. Full-res Wan-VAE decode for long chains (eval_rollout --vae wan --chain_cells 241) at 704p was not tried. The chain eval in chain_after_sft.sh already uses TAEHV for chain241 and Wan only for clip28 (which runs fine: 55 s).
  5. Smoke artifacts take ≈ 200 GB under outputs/_smoke/{sft704,res704}. Most of it is the state_A slots of sft704 and res704/stage1_probe (≈ 66 GB each). They can be deleted, but sft704/step-8.safetensors is the 704p init the probes used.
  6. SPEC §1/§5 text and the other impl docs describe the 832×480 instance. Their shapes are what geometry gives at the default GC_RES.