# res704: resolution-configurable gc25d stack (`GC_RES`), target 1280×704 Status 2026-10-23. Code done. Tests pass at both resolutions. Real-weight smokes ran at 1280×704 on GPUs 0–3: prep, verify, ext_canvas, 4-GPU SFT, sample, three full-shape dry runs, stage1/2/3 memory probes and the eval twin. The 480p pipeline is unchanged: the default `GC_RES=832x480` reproduces every old constant and index table bit for bit (tested). `data/gc25d_v10` and `outputs/sft_v10` were not touched. **Nothing has been launched at full scale.** The commands for the full 704p prep → SFT → distill chain are at the end. A review fix pass on the same day is summarised right below: deploy res guard, `parse_res` lower bound, launcher normalisation, chain path guards, the capsule bbox, ext_canvas threads and verify thresholds. The final test and smoke numbers are there too. ## Review fix pass (2026-10-23) Ten reviewer findings were each verified first, then fixed, partly fixed or rejected. 832×480 output stays bit-identical (the old-constant / `_IDX` sha256 tests still pass; the capsule bbox default is unchanged at 832×480). Logs are in `outputs/_smoke/res704_fix/`. | # | finding | verdict | what changed | |---|---|---|---| | 1 | deploy engines never check ckpt `res` | **confirmed, fixed** | Res guard at the top of `LiveGame.__init__` in the pending deploy patch (`deploy_vocab/live_vocab.patch`), before any weights are read; `FastLive` / `gen_server` inherit it. The same check is in `live_path_check.py`. `ref/` untouched. Also fixed a **regression the res704 change had introduced**: `live_path_check --no_inject` (the documented deploy-contract check) crashed, because maliang's `data_gc` now reads `GEO.LAT_H`, which the deploy package's hard-coded geometry lacks. The twin side now imports maliang's geometry | | 2 | `parse_res` accepts values that crash at import; odd `CROP_W` off-centre | **confirmed, fixed** | `parse_res` requires H ≥ 384 (downscale < 2) and raises a `GC_RES` ValueError; 115 of the 450 previously accepted values were crashers. `CROP_W` is rounded to the nearest even number, so the crop is exactly centred and `PLAYER_DST` = frame centre at every legal res. 480p / 704p unchanged; 1152×640 goes 1267 → 1268. `firstframe_flux` copy updated. Tests cover all 335 accepted values (import-time asserts simulated) plus a subprocess import per listed res | | 3 | launchers compare raw vs normalised `GC_RES` | **confirmed, fixed** | `train_sft.sh`, `common_gc.sh` and `chain_after_sft.sh` normalise to `geometry.RES_TAG` after validating. `GC_RES=1280X704` now passes against the 704 mini (probe S2: reaches the existing-ckpt gate; S4b: `require_data_res` OK) | | 4 | chain path defaults ignore `GC_RES`; unanchored pgrep | **confirmed, fixed** | At GC_RES ≠ 832x480 the chain requires `SFT_OUT`, `D` and `DATA_ROOT`. Before waiting it checks the res of DATA_ROOT, each existing `$D/stage*/config.json` and `SFT=`; it checks the picked SFT again. pgrep is anchored. All three runners now refuse another resolution's run dir **before opening train.log** (`DG.check_res(run_dir=…)`) | | 5 | `infer.load_ids` raw-silhouette guard bypassed at 704p | **rejected (documented instead)** | The 10/11 refusal is a gc25d_v8 (ellipse-marker) rule. For v10, silhouettes *are* the condition: `collapse_parts(cond)` = raw ids.png for an ids-vocab model, and `docs/understand/infer_contract.md:285` already notes the rationale no longer holds. Enforcing it at 704p would block valid v10 ids-vocab inputs. The proposed "parts vocab without 12..21" check cannot apply because `infer.py` only accepts {0..6, 10, 11, 255}; v10 parts conditions go through `sample_user.py`. Documented in the module doc and `load_ids` | | 6 / 8 | `render_capsules` pad is fixed in dst px, so it clips more at 704p | **confirmed, fixed differently** | Scaling the pad only shrinks the error. At 704p, missed character px vs a pad-64 reference over 29 has_ext clips (every 3rd frame) are: pad 2 0.149 %, 2.93 0.103 %, 4 0.067 %, 6 0.030 %. The cause is the world-axis bbox, so `cam_axes` adds ± r along the camera right/up/forward axes. That gives **0.000 %** at both resolutions; on at every res except 832×480, which keeps the old bbox bit for bit (open issue 2). (Finding 8's formula `2·max(1, H/480)·1.4` would give 2.8, not 2.0, at 480p.) The verify_prep 50 px gate is scaled by `AREA_VS_480` | | 7 | 704p answer rests on detail ratio alone | **confirmed (evidence caveat)** | `scripts/res_test_edge.py` adds SSIM / Laplacian correlation / edge F1 (table at the top). 704 vs 640 ties on SSIM and Laplacian correlation, wins on edge recall/F1, and loses 0.24 dB PSNR. Stated in "Why 1280×704" | | 9 | ext_canvas thread env set after numpy import | **confirmed, fixed** | The four thread vars are forced to 1 at module top, before numpy (prep_v10 pattern). The plain `$ENV` (OMP=8) command is now safe | | 10 | verify_prep 480p thresholds; fixed-shape docstrings | **confirmed (thresholds), rejected (docstrings)** | Visible-frame gate `50·AREA_VS_480`; PSNR WARN lower bound 20 dB off 832×480 (21 at 832×480). The docstrings flagged in `verify_prep.py` / `data_gc.py` already carry an "these are the 832×480 example (1280×704: …)" qualifier, so they were left alone | **Smoke (1280×704, GPUs 0–3)** * ext_canvas, 704 mini, 85 clips, 64 workers, plain `$ENV` (OMP_NUM_THREADS=8 inherited, overridden inside): ≈ 200 s, rc 0. Peak load average 78 while sharing the CPUs with a 28-process job, vs 1,850 before. unknown 0.169, self-agree f80 0.974, f76 0.969 (same as before). New vs old canvases (`ext_canvas_old_vs_new.json`): blocks / prompts identical in 85/85. 57,081 px changed in 467 of 2,380 frames, **all of them character pixels the old bbox had clipped** (+0.155 % character px, 0 removed). The old canvases are in `.tmp/cond_ext_canvas_704mini_old/`. * verify_prep, 704 mini, 8 spots: **OK, 200 clips, 0 fails, 0 warns** (was 2 warns: the two R10_s1512 clips at 20.26 / 20.42 dB are inside the [20, 40] band used off 832×480). The render stats use the camera-axis bbox: v10 player unoccluded IoU median 0.9977 (p10 0.9968), v9_stylized player coverage 0.9997 → **1.0000**; boss unchanged (`verify_mini.log`). * 832×480 bit-identity of the bbox change: `render_capsules` default vs the pre-fix module on 550 real 480p frames (10 mini clips, every 2nd frame): **550/550 bitwise equal**. * deploy path, `live_path_check` (3 streamed cells vs `eval_rollout.deploy_rollout`): * 480p EMA through the freshly patched package (`--no_inject`, `.tmp/deploy_patched_fix704`): bitwise equal 4/4 cells (regression fixed); * 480p EMA through `ref/` live.py (injected): bitwise equal 4/4; * **704p** probe EMA through `ref/` live.py at `GC_RES=1280x704`: frames 13×704×1280, **bitwise equal 4/4**, DiT ≈ 715 ms/cell (the same overhead-bound eager loop as at 480p, ≈ 700 ms); * refusals: the 704p EMA vs the patched package, or vs `ref/` at default GC_RES, stops in `live_path_check` before loading. Patched `LiveGame(...)` and `FastLive(...)` called directly raise in 0.00 s (`deploy_guard_probes.log`). * launcher probes (`launcher_probes.log`): 12 cases (both mismatch directions, illegal res, upper-case res, chain without D, chain aimed at the 480p distill dir / 480p data / a 480p SFT ckpt, 480p default chain); all behave as intended. Nothing was launched. * `res_test_edge.py` (GPU 0, ≈ 3 min): PSNR / detail reproduce `res_test.json` exactly for all 5 resolutions. **Tests** (`tests_default.log`, `tests_704.log`) * default `GC_RES`, `CUDA_VISIBLE_DEVICES=3`: **160 passed, 4 skipped**, 322 s (was 158 / 4; the +2 are the new tests below). This includes `test_704_suite_subprocess`. * `GC_RES=1280x704`, CPU: **156 passed, 8 skipped**, 188 s (was 154 / 8). The skips are the same kinds as before. * New or extended in `test_res.py`: * `test_parse_res`: rejects 640x352 / 608x320 / 32x96; * `test_parse_res_accepted_are_import_safe`: all 335 accepted W×H; `_IDX` asserts hold, the crop is even and centred, aspect error ≤ 1 px; * `test_listed_res_import_subprocess`: real import of geometry + firstframe_flux for 7 resolutions; `PLAYER_DST` = frame centre; an illegal res is a `GC_RES` ValueError, never an AssertionError; * `test_guards`: `check_run_dir_res`, plus stage 1/2/3 `main()` refusing another resolution's run dir with only `config.json` left in it; * `test_capsules_rays_consistent_across_res`: with the camera-axis bbox and the default pad 2, both resolutions equal the pad-64 render on the synthetic oblique cameras; the 704p default uses it and the 480p default is the old bbox. ## Why 1280×704 (and not 540p / 600-something p) The user asked "540p or 600+p, whichever fits better". The lead's resolution test is `scripts/res_test.py` → `outputs/char_diag/res_test.json`. It measures 40 player crops from 8 clips (5 frames each): VAE round trip at each resolution, upsampled back, compared with the native 1280×704 GT. It also times one 21-cell bidirectional DiT forward. `scripts/res_test_edge.py` → `outputs/char_diag/res_test_edge.json` (added in review) reruns the same crops through the same code path and adds three metrics that check *where* the detail is, not just how much of it there is: grey SSIM, the correlation between the Laplacians of reconstruction and GT, and Canny edge F1 with 1 px tolerance. Its PSNR / detail values match `res_test.json` exactly. | GC_RES | detail ratio | crop PSNR | SSIM | Laplacian corr | edge P / R / **F1** | DiT forward (21 cells) | tokens / cell | |---|---|---|---|---|---|---|---| | 832x480 (current) | 0.375 | 24.74 | 0.634 | 0.207 | 0.933 / 0.624 / 0.733 | 0.262 s (1.00×) | 390 | | 960x544 ("540p") | 0.419 | 25.24 | 0.655 | 0.242 | 0.941 / 0.652 / 0.759 | 0.360 s (1.37×) | 510 | | 1024x576 | 0.437 | 25.47 | 0.665 | 0.255 | 0.943 / 0.664 / 0.768 | 0.409 s (1.56×) | 576 | | 1152x640 ("600+p") | 0.468 | **25.93** | **0.686** | 0.288 | 0.946 / 0.686 / 0.784 | 0.546 s (2.08×) | 720 | | **1280x704 (native)** | **0.779** | 25.69 | 0.679 | **0.291** | 0.921 / **0.803** / **0.855** | 0.691 s (2.64×) | 880 | Every non-native resolution goes through crop + INTER_AREA downscaling. That resampling throws away fine character detail before the VAE sees the frame. 480p → 640p improves every metric a little. Going from there to native 704p: * **Edge recall** jumps from 0.69 to 0.80 (F1 0.78 → 0.86): many more of the GT's real edges come back. * **Placement metrics tie.** SSIM is 0.686 vs 0.679 and Laplacian correlation 0.288 vs 0.291. * **Edge precision drops slightly** (0.946 → 0.921), and crop PSNR is 0.24 dB *lower* than at 640p. **Caveat:** the detail ratio (0.468 → 0.779) overstates the 640p → 704p gain. It measures Laplacian *energy*, so it also rewards the absent resampling low-pass, and part of the extra high frequency at 704p is VAE artifact rather than faithful detail. Lower edge precision and PSNR, and a flat SSIM, all show this. The honest summary: * 704p beats 480p and 540p on every metric. * Against 640p, 704p recovers clearly more of the real edges but is no better on SSIM or PSNR, and costs about 27 % more DiT compute (2.64× vs 2.08× of 480p). * 704p is also Wan2.2-TI2V-5B's own resolution, and the data's. Frames and ids pass through unchanged (no crop or rescale), so 704p is still the pick for "600+p". 540p is a cheap middle ground that gains little. All of these are one env var. If the real-time budget later rules out 704p, `GC_RES=960x544` or `1152x640` runs through the same code. The general rule is W, H multiples of 32, 384 ≤ H ≤ 704, W ≤ 1280 and aspect ≤ 1280:704. Each resolution needs its own prep / SFT, because a data root and a ckpt are tied to one resolution. ## The contract * **`code/gc25d/geometry.py` is the only source of resolution.** It parses `GC_RES="x"` once at import (default `"832x480"`). Rules: W and H are multiples of 32, **384 ≤ H** ≤ 704, W ≤ 1280, and the aspect ratio is at most the source's 1280:704 (wider would need a vertical crop, which is rejected with a message). H ≥ 384 means the downscale factor is < 2. Below that, the centre-aligned `_IDX` tables cannot reach the first/last source row and column, so the import-time asserts would fail; `parse_res` now rejects those values first with a `GC_RES` ValueError. Derived values: * `DST_W, DST_H`, `RES_TAG = "x"`, `MIN_H = 384`; * `CROP_W = crop_w_for(W, H) = min(1280, 2·round(352·W/H))`, i.e. 704·W/H rounded to the nearest **even** number, so the crop is exactly centred. `CROP_X0 = (1280 − CROP_W)//2`. 832×480 gives 1220 / 30, exactly as before; 1280×704 gives 1280 / 0. Only odd cases change, e.g. 1152×640 is now 1268 / 6 instead of 1267 / 6, which was 0.5 source px off-centre; * `SX, SY, SCALE`, `_IDX_X/_IDX_Y` (centre-aligned nearest), and `PLAYER_DST`. `PLAYER_DST` is the exact frame centre at every legal resolution (tested). It is computed multiply-then-divide, so 832×480 gives exactly (415.5, 239.5) as the old literal did, while plain `src_to_dst` would give 239.49999999999997; * new: `LAT_H, LAT_W = H/16, W/16` (30×52 → 44×80), `POOL_H, POOL_W = H/4, W/4` (120×208 → 176×320), `TOK_PER_CELL` (390 → 880), `AREA_VS_480` (for pixel-count thresholds), `IDENTITY` (1280×704: no crop, no scaling). * `resample_rgb` / `resample_ids` take an identity fast path when `IDENTITY` is set. It returns a C-contiguous **copy** and is tested to equal the general path, both same-size INTER_AREA and fancy indexing. * Child processes (accelerate ranks, DataLoader workers, prep's spawned ranks, ProcessPool workers) inherit the env var, so every process in a run agrees. * **Resolution guards.** A missing `res` always means `"832x480"`, so all existing 480p roots and ckpts keep working unchanged. * Data root: `index/prep_args.json["res"]`. prep writes it, plus `res` on **every index row**. It is checked in `GcLatentDataset` and `GcRolloutDataset`, `sample.py`, `sample_user.py --export_clip`, `infer.py` (clip modes), `eval_rollout.py` (also chain mode, which bypasses the dataset), `ext_canvas.py`, `verify_prep.py`, all three distill runners (in `parse_args`, before loading 5B weights) and `scripts/train_sft.sh` / `run_stage*.sh` (`require_data_res`). prep also refuses to write into an `--out` that already holds another resolution (prep_args or rows), and `merge` refuses mixed rows. * Ckpt: safetensors metadata `res`. `gc_dit.save_gc` always writes it, so SFT exports and every distill product (dits/ ema/ critic/) carry it, and `train.export_metadata` adds it explicitly. `sample.py`, `sample_user.py`, `infer.py`, `eval_rollout.py` and the stage1/2/3 runners refuse a ckpt whose res ≠ GC_RES. The runners check the init, teacher and student ckpts, warm starts, and the dits/ema/critic picked up on resume. `train.py` dir-resume refuses a `trainer_state.json` of another res (`res` is now written there). A **safetensors warm start** with a different res only prints a note, because cross-resolution warm starting is legitimate: the DiT and `gc_sem_cond` are shape-generic. * Distill run dir: `/config.json["res"]` (missing = 832x480; no config.json = new dir). All three runners call `DG.check_res(run_dir=…)` → `geometry.check_run_dir_res` **before opening `train.log`**. A run aimed at another resolution's run dir therefore stops without appending to its logs and without loading 5B (tested: the refused dir still holds only `config.json`). * Deploy: the pending deploy patch `scripts/distill_smoke/deploy_vocab/live_vocab.patch`, applied by `make_patched_deploy.sh`, now adds a res check at the top of `LiveGame.__init__`, before any weights are read. It refuses a ckpt whose metadata res (missing = 832x480) ≠ the engine's `G.DST_W×G.DST_H`. `fast_live.FastLive` and `gen_server` inherit `LiveGame`, so they are covered too. `scripts/distill_smoke/live_path_check.py` does the same check against the geometry its `live.py` actually uses (maliang's when injecting, which follows GC_RES; the package's own 832×480 with `--no_inject`). `ref/` itself is still untouched, as with the vocab patch. * Every error message names `GC_RES`, the value found, and the fix. Example: `--ckpt …/step-8.safetensors 是在 1280x704 上训练 / 导出的(metadata res,缺失 = 832x480),本进程 GC_RES=832x480:设 GC_RES=1280x704,或换成 832x480 的 ckpt`. * **Launchers.** All three validate `GC_RES` and **normalise it to `geometry.RES_TAG`**, so `" 1280X704"` becomes `1280x704`. Every shell string comparison and log line therefore uses the canonical value. * `scripts/train_sft.sh` exports `GC_RES` (default 832x480), **refuses to start if DATA's res ≠ GC_RES**, and logs it in the banner. * `gcdistill/common_gc.sh` logs `[res] GC_RES=…` and provides `require_data_res`, which `run_stage{1,2,3}.sh` call on real data; their banners print `res=`. * `scripts/chain_after_sft.sh` takes `GC_RES`, `D` (distill root) and `DATA_ROOT`, and passes each stage its `OUT=$D/stage*`. * With the default GC_RES its defaults equal the old 480p chain exactly; eval_rollout gets `--data_root`. * **With any other GC_RES it requires `SFT_OUT`, `D` and `DATA_ROOT` explicitly**, because the defaults are 480p directories. * Before waiting for the SFT, it checks the res of DATA_ROOT, of every existing `$D/stage*/config.json` and of an explicit `SFT=`. It checks the picked SFT ckpt again before stage 1. * The SFT wait loop's `pgrep` is anchored like train_sft.sh's (`--output $SFT_OUT( |$)`), so a 480p chain no longer waits on a running `sft_v10_704` trainer. ### What changed per file (shapes that were hard-coded → geometry) | file | change | |---|---| | `geometry.py` | GC_RES parsing (H ≥ 384, even `crop_w_for`), derived constants, identity fast paths, `root_res / check_root_res / ckpt_res / check_ckpt_res / check_run_dir_res`, docstrings generalised | | `dataset.py` | `LATENT_SHAPE`, `IDS_SHAPE`, mask grid from geometry; `check_root_res` in the constructor (`ds.res`) | | `infer.py` | `H, W, LAT_H, LAT_W, SRC_*` from geometry; messages print the actual shapes; ckpt/data-root res checks; a source-resolution ids array is not resampled when source = target. The `--ids` 10/11 "raw silhouette" refusal (a gc25d_v8 ellipse-marker rule) can only tell raw from processed ids by shape, so at 1280×704 it does not apply. This is documented in the module doc and `load_ids`; v10 parts conditions go through `sample_user.py` | | `sample.py`, `sample_user.py` | shape asserts / warmup latents from geometry; ckpt + data-root res checks; `res` in json; at 1280×704 a 1280×704 first frame / ids are taken as-is | | `viz.py` | `PANEL_H, PANEL_W = DST_H, DST_W`; the title bar stays 48 px, total height H+48 (752 at 704p) is still a multiple of 16 (asserted) | | `prep_v10.py` | `LAT_SHAPE` from geometry; `res` in prep_args.json (+ crop_x0/crop_w) and every row; refuses a foreign-res `--out`. At 1280×704: RGB = raw frame, ids = raw ids.png, `label_parts` at full res (identity `_IDX`) | | `verify_prep.py` | res check (prep_args + every row), `res` in the report, log says `px per frame ()`. Thresholds set at 480p are scaled: the visible-frame gate is `GT ≥ 50·AREA_VS_480` px; the PSNR WARN band is [21, 40] at 832×480 and [20, 40] elsewhere (busy clips sit ≈ 1 dB lower against native GT) | | `ext_canvas.py` | data-root res check, `res` in the report; canvas cell stays 6 cm (world units). Forces `OMP/OPENBLAS/MKL/NUMEXPR_NUM_THREADS=1` at the top, before numpy, like prep_v10. The old `setdefault` inside `main()` ran after numpy was imported and never overrode `$ENV`'s 8 | | `capsules.py` | `render_capsules(..., cam_axes=None)`. The bbox also projects the endpoints ± r along the camera right/up/forward axes (columns of `world_to_view`). The default is on at every resolution except 832×480, which keeps the old world-axis bbox bit for bit (the existing `data/gc25d_v10/cond_ext_canvas` was drawn that way). Measured on 29 has_ext clips, every 3rd frame, vs a pad-64 reference: 704p clipped character px 0.149 % → 0; 480p 0.080 % (unchanged default) → 0 with `cam_axes=True` | | `model.py` | docstrings only (already shape-generic) | | `procmap.py` | `player_px0 ≥ 500·AREA_VS_480`; the default out dir gets a `_` suffix at non-default res (the 480p benchmark cases are not overwritten) | | `firstframe_flux.py` | runs in the PCE env (no cv2 ⇒ cannot import geometry) ⇒ **copy** of `parse_res` / `crop_w_for` (tested equal to geometry); output = DST; generation grid `gen_grid(W,H)` = 16·(a,b)·t with a:b = W:H reduced, t chosen so the area is closest to 0.9 MP and the grid is at least the output size (832×480 → 1248×720 as before, 1280×704 → 1280×704); `MIN_PX` scaled by area; `clip_frame` uses the crop rule; `clip_ids` refuses a cond of another res | | `gcdistill/gc_dit.py` | `save_gc` writes `res` | | `gcdistill/data_gc.py` | `LAT_HW`, `check_res(ckpts=…, root=…, run_dir=…)` (SystemExit), `GcRolloutDataset` root check + latent shape from geometry | | `gcdistill/ar_tf_gc.py`, `cd_gc.py`, `dmd_gc.py` | synthetic shapes `(30,52)` → `DG.LAT_HW`; res checks (parse_args + every real-weight load + run dir before the logs open); `res` in config / logs / stage1 trainer_state; new `--dry_run_full_hw` (tiny model, full-res latent + sem grids); **dmd budget gate**: the KV / scoring / per-grad-frame terms × `TOK_SCALE = TOK_PER_CELL/390` (1.0 at 480p ⇒ identical constants), and dry runs skip the gate (it fired on the 5B constants with tiny models whenever a GPU was visible — the two pre-existing `test_stage3` failures under `CUDA_VISIBLE_DEVICES=0`) | | `gcdistill/eval_rollout.py` | ckpt + data-root res check (before loading), `res` in summary | | `train.py` | `res` in export metadata, config.json, trainer_state; dir-resume refuses another res; warm-start note | | `scripts/train_sft.sh`, `gcdistill/common_gc.sh`, `run_stage{1,2,3}.sh`, `scripts/chain_after_sft.sh` | see above | | `configs/accelerate_fsdp_dmd_4gpu.yaml` | new: 2gpu yaml with num_processes 4. Needed for the 704p stage3 probe, because 2 GPUs fail the budget gate at 704p (126.1 > 126.0 GiB) | | `scripts/distill_smoke/deploy_vocab/{live_vocab.patch,make_patched_deploy.sh}`, `scripts/distill_smoke/live_path_check.py` | deploy res guard (see the contract); `make_patched_deploy.sh` self-checks that the guard is in the patched live.py; `live_path_check` asserts the frame shape from `live.G` instead of a hard-coded 480×832 | | `scripts/res_test_edge.py` | new: alignment metrics (SSIM, Laplacian correlation, edge F1) for the resolution decision, same crops and code path as `res_test.py` | Not changed: `code/arope/` (vendored; already shape-generic), the other analysis scripts in `scripts/` (`genbench_compose.py`, `char_*`, `distill_smoke/*` are 480p tools tied to existing 480p outputs), and `ref/gc25d-live-demo` (deploy package, see open issues; its guard ships as part of the pending patch). ## Tests * `tests/test_res.py` (new): * default res: every old geometry name equals the hard-coded old constants, plus `_IDX_X/_IDX_Y` by sha256; the shapes derived in dataset / infer / viz / prep / data_gc / firstframe_flux / dmd budget equal the old literals; `resample_*` equal the old implementation; * `parse_res` good and bad values; * guards: root / ckpt / prep out / `DG.check_res` / `save_gc` writes res; * a tiny DiT `GcTrainModule` step on the **real latent + sem grids** of the current res; * `infer.sample` / `viz.SideBySide` shapes; * the firstframe_flux copy matches geometry for 8 resolutions; * cross-resolution capsules: a 704p subprocess renders capsules, labels parts and casts ground points; sampled with the 480p `_IDX`, they **equal** the 480p results (render / label bitwise, ground points ≤ 1e-6); * `test_704_suite_subprocess`: reruns test_res / dataset (synthetic) / train / gcdit (+ data_gc rollout dataset) / stage1 / stage2 / stage3 dry runs / prep / sample_user / firstframe_flux / procmap with `GC_RES=1280x704`, and the 704p-only checks (identity paths, derived constants, budget scaling) run inside it. * Existing tests made resolution-aware, but only where they encoded 480p legitimately: synthetic roots write `prep_args.json`, rows carry `res`, shapes come from geometry, and "real mini" tests use `data/gc25d_v10_mini` at 480p and `data/gc25d_v10_704_mini` at 704p (skipped if the res differs). Files: `test_dataset`, `test_gcdit`, `test_stage1`, `test_stage3`, `test_prep`, `test_sample_user`, `test_firstframe_flux`. * Results of the implementation pass (`outputs/_smoke/res704/tests_default.log`, `tests_704.log`; the review fix pass numbers are in the section at the top): * default `GC_RES`, `CUDA_VISIBLE_DEVICES=3`: **158 passed, 4 skipped** (326 s, includes the 704p subprocess suite). The skips are the two GPU/real-weight gates, which were skipped before too, and two 704p-only tests. Before this change the same command had 2 failures (the dmd budget gate in dry runs). * `GC_RES=1280x704`, CPU, with the 704 mini READY: **154 passed, 8 skipped** (189 s). The real-mini tests (`test_dataset::test_real_mini`, `test_stage1::test_7`, `test_stage3::test_18`) run against `data/gc25d_v10_704_mini`. The skips are the 4 default-only tests in test_res, the 2 GPU/real-weight gates, and 2 tests that need the 480p `data/gc25d_v10`. ## Real smoke at 1280×704 (GPUs 0–3, all logs in `outputs/_smoke/res704/`) **Mini prep** (`prep_mini.log`): runs R10_s1501, R10_s1512, P9C_autumn_s1941, P9_golden_s2909 → `data/gc25d_v10_704_mini` (200 clips, 2 GPUs × 12 workers). * VAE: **2.15 s/clip** (480p: 1.07). Throughput 0.45 clips/s per GPU (GPU-bound; 480p was 0.72, worker-bound). GPU peak **11.4 GiB** (480p: 6.5). * Worker: 6.2 s/clip (label 3.9 s, write 1.3 s) × 12 workers per GPU ⇒ not the bottleneck. * Extrapolated full set: 12,800 clips on 8 GPUs ≈ **60 min** (the 480p full prep took 37 min). * Disk: latent 7.1 MB/clip (480p 3.15), cond 2.04 MB, cond_ext 0.71 MB ⇒ full set ≈ **125–130 GB** (480p: 53 GB). **verify_prep** (`verify_mini.log`): **OK, 200 clips, 0 fails, 2 warns**, 250 s. * A: res 1280x704 everywhere. B: 200/200. C: 8 spot clips: official `vae.encode` re-encode exact 1.00000; cell 0 bf16-exact; collapse mismatch 0; relabel from tar json 28/28. * Capsule re-render from meta: v10 player unoccluded IoU median **0.9977**, coverage 1.0000; boss 0.9967. The 480p mini gives 0.9978 / 0.9967. * VAE reconstruction PSNR vs native GT: mean 26.62 dB (min 20.26, max 38.14). The 2 warns are two R10_s1512 clips at 20.3 / 20.4 dB, just below the [21, 40] band, which was calibrated at 480p (the same clips at 480p: 21.25 / 22.39). Native-resolution GT has more high frequencies, so expect about 1 dB lower PSNR on busy clips. These are warnings, not failures. **ext_canvas** (`ext_canvas_mini.log`): 85 ext_look_change clips, 64 workers, **180 s**. unknown 0.169, self-agreement f80 0.974, f76 0.969. 480p full-set values: 0.157 / 0.977 / 0.969. About 80 s per clip with `--report` (480p ≈ 52 s). The implementer's first attempt ran without `OMP_NUM_THREADS=1`, and each of the 64 fork-workers started a full BLAS thread pool (load average 1,850). `ext_canvas.py` now forces single-threaded BLAS itself (review fix pass), so the plain `$ENV` command is safe. These canvases were then re-rendered with the camera-axis bbox (see the review fix pass). **SFT smoke** (`outputs/_smoke/sft704`, `outputs/_smoke/sft704.launcher.out`): 4 GPUs (0–3) ZeRO-2, micro 1 × GA 2, 8 steps, val at step 8 (4 clips × 3 t × 3 variants), export + `save_state`. | | 480p (`_smoke/sft_timing`, same 4-GPU GA2 setup) | **704p** | |---|---|---| | s/step (steady) | 2.40 | **5.92** (2.47×), data wait 0.03 s | | peak allocated / GPU | 35.2 GiB | **37.9 GiB (max over ranks 40.0)** | | first step | — | 13.0 s | * Grad checkpointing keeps activations small, so memory barely moves. From 4 to 8 GPUs ZeRO-2 halves the optimizer/grad shard (480p: 35.2 → 24.4 GiB), so **8 GPUs at 704p ≈ 27–29 GiB/GPU**. * `sem_rms` rises from 0 (zero init), and val runs (gaps ~0 after 8 steps, as expected). * `step-8.safetensors`: 831 BF16 keys, `check_gc_ckpt` OK, metadata `res=1280x704`. `trainer_state.json` has `res`. The state slot is 66 GB (4 GPUs). * Full run estimate (8 GPUs, GA2, 20,000 steps): per-GPU work is the same as at 4 GPUs, so ≈ 5.9–6.1 s/step ⇒ **≈ 33–34 h** of training. Add validation (every 500 steps, ~30 s) and saves (every 1000 steps: 10 GB export + state slot of roughly 70 GB at 8 GPUs) ⇒ **≈ 35 h**. At 480p the same run is ≈ 14 h. **sample.py** (`sample_step8.log`): 1 val clip (`02_Autumn_Woodland_Shrine__P9_golden_s2909__002400`), 8 steps. * Sampling 6.6 s (0.83 s/step, one forward per step). Peak 22.6 GB including the non-tiled VAE decode of 81×704×1280. Wall 52 s. * Side-by-side mp4 is 752×3840, 81 frames. PSNR is meaningless after 8 SFT steps (15.9 dB; VAE ceiling 33.6). * Refusals checked: the 704p ckpt at default GC_RES, and the 480p mini at GC_RES=1280x704. Both stop before loading, with the messages above. `train.py --resume outputs/_smoke/sft704/state` at default GC_RES stops with `… 是 res=1280x704 的存档(缺失 = 832x480),本次 GC_RES=832x480`. `train_sft.sh` refuses both mismatch directions and an illegal `GC_RES` (`1280x640`, aspect wider than the source). **Distill dry runs at 704p shapes** (`dry_s{1,2,3}.log`): `--dry_run --dry_run_full_hw`. Tiny models, but the latent is 44×80 and sem ids are 704×1280 (verified: `input_latents (48,5,44,80)`, `sem_ids (17,704,1280)`). stage1 13 s, stage2 15 s, stage3 21 s, all exit 0. Exports carry `res=1280x704`. **Real-weight probes, 704p** (init = the SFT smoke ckpt; quality irrelevant): | stage | config | result | |---|---|---| | 3 DMD | FSDP1, **2 GPUs** | **refused by the budget gate**: predicted 126.1 GiB > 0.9 × 140 = 126.0 (`stage3_probe_2gpu_gate_abort.log`) | | 3 DMD | FSDP1, **4 GPUs** (0–3), 2 steps, PROFILE_MEM, N = 22 / 27 (`stage3_probe/`) | gate predicted 84.1 GiB. **Measured peak 68.5 GiB/GPU** (step 0: after rollout 44.7, after critic 61.2; step 1 with N=27: rollout peak 68.2). Probe gnorm 12.1; step 0 dmd = gnorm_g = 0 exactly; replay_dev = 0; step 2: 33 s (rollout 5.9 s, exit 0); step 1: 77 s (first step, includes warm-up). dits/critic/ema exported with res | | 1 TF | ZeRO-2, 2 GPUs (0,1), GA1, 3 steps (`stage1_probe/`) | **11.7 s/step** (480p 2-GPU smoke: 3.5–3.7), peak **67.7 GiB** (480p: 56.5). gnorm 2.9–4.2; export + state OK | | 2 CD | ZeRO-2, 2 GPUs (2,3), GA1, 3 steps (`stage2_probe/`) | **17.2 s/step** (480p: 5.1), peak **98.3 GiB** (480p: 85.0). dits/ + ema/ exported | The teacher-forcing double stream (2 × 21 × 880 = 36,960 tokens) makes attention the dominant cost at 704p, so stage1/2 slow down by 3.2–3.4×, more than the SFT's 2.5×. Going from 2 to 8 GPUs removes ≈ 30 GiB of ZeRO-2 shard per GPU (optimizer m/v/master + grads, (1/2 − 1/8) of ≈ 80 GB). Expected 8-GPU peaks: **stage1 ≈ 38 GiB, stage2 ≈ 68 GiB**. Both fit. **8-GPU stage3 at 704p:** of the 68.5 GiB, ≈ 41.9 is FULL_SHARD state (167.7 GiB / 4). At 8 GPUs that state is 21.0, so expect **≈ 48 GiB/GPU peak** (gate prediction 63.2 GiB). At 480p the expectation was ≈ 40 GB. It fits comfortably. **Wall-clock plan at 704p, 8 GPUs** (per-GPU work at 8 GPUs equals the measured smokes, because the batch per GPU is the same): | step | 704p | 480p (for reference) | |---|---|---| | prep / verify / ext_canvas | ≈ 1 h / ≈ 0.75 h / ≈ 0.6 h | 37 min / 27 min / — | | SFT 20k steps (GA2) | **≈ 35 h** (5.9 s/step + val/save) | ≈ 14 h (2.4–2.5 s/step) | | stage1 TF 8000 steps | **≈ 26 h** (11.7 s/step) | ≈ 8 h | | stage2 CD 8000 steps | **≈ 38 h** (17.2 s/step) | ≈ 11 h | | stage3 DMD 500 steps | ≈ 5–8 h (33–77 s/step in the 4-GPU probe; rollout length varies) | ≈ 3 h | | **total** | **≈ 4.5 days** | ≈ 1.5 days | Stage-2 length (8000) is the documented quality lever. If the calendar is tight, shorten stage 1 first, not stage 2. **Deploy twin (`eval_rollout.py`) at 704p** (`eval_clip28_wan/`, `eval_chain121_tae/`, `speed_{704,480}/`): runs end to end with the probe EMA. clip28 + Wan VAE: 55 s wall. chain121 + TAEHV: 95 s. Same GPU, idle, chain121 + TAEHV, DiT only: **704p 420 ms/cell vs 480p 361 ms/cell**. This eval loop is launch-overhead-bound (5 single-cell forwards per cell, eager PyTorch). The compute ratio is ≈ 2.6× (table above), so an optimised deploy path (`fast_live.py`-style) will pay close to the compute ratio. Real time needs ≤ 250 ms per cell (4 frames at 16 fps). **704p real-time is not established here.** It needs the deploy package (see open issues). ## Commands (M=/opt/dlami/nvme/zhiyangdeng/maliang) ```bash M=/opt/dlami/nvme/zhiyangdeng/maliang ENV="DIFFSYNTH_SKIP_DOWNLOAD=True PYTHONPATH=$M/code/gc25d:$M/code/arope HF_HOME=/opt/dlami/nvme/zhiyangdeng/.cache/huggingface TMPDIR=$M/.tmp OMP_NUM_THREADS=8 GC_RES=1280x704" # 1) full 704p prep, 8 GPUs (≈ 60 min, ≈ 130 GB; resumable: rerun the same command) env $ENV CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 $M/.venv/bin/python $M/code/gc25d/prep_v10.py \ --out $M/data/gc25d_v10_704 --gpus 0,1,2,3,4,5,6,7 --workers_per_gpu 12 > $M/outputs/prep_full/prep_704.log 2>&1 # 2) verify (480p full verify took 27 min; expect ~45 min) env $ENV CUDA_VISIBLE_DEVICES=0 $M/.venv/bin/python $M/code/gc25d/verify_prep.py \ --root $M/data/gc25d_v10_704 --spot 16 --device cuda:0 --workers 64 > $M/outputs/prep_full/verify_704.log 2>&1 # 3) stage-3 extension canvases (4,772 ext_look_change clips; ≈ 70 s/clip/worker ⇒ ~35 min at 160 workers). # ext_canvas.py forces OMP/OPENBLAS/MKL/NUMEXPR threads = 1 itself (the 8 in $ENV is overridden). env $ENV $M/.venv/bin/python $M/code/gc25d/ext_canvas.py \ --root $M/data/gc25d_v10_704 --workers 160 > $M/outputs/prep_full/ext_canvas_704.log 2>&1 touch $M/data/gc25d_v10_704/READY # after verify says OK (exit code 0, "OK" in verify_704.log) # 4) full SFT from scratch, same recipe as 480p (8 GPUs × micro 1 × GA 2 = 16, 20k steps, lr 1e-5 / cond_lr 1e-4, warmup 500) GC_RES=1280x704 DATA=data/gc25d_v10_704 OUT=outputs/sft_v10_704 NGPU=8 GA=2 \ setsid nohup bash $M/scripts/train_sft.sh > $M/outputs/sft_v10_704.launcher.out 2>&1 & # crash resume: same + RESUME=outputs/sft_v10_704/state # 5) distill chain at 704p (waits for sft_v10_704/done.json, picks the teacher, stage1 8000 → stage2 8000 → stage3 500 → eval). # At GC_RES ≠ 832x480 the chain refuses to start unless SFT_OUT, D and DATA_ROOT are all given; it checks their res up front. mkdir -p $M/outputs/distill/gc25d_v10_704 GC_RES=1280x704 SFT_OUT=$M/outputs/sft_v10_704 D=$M/outputs/distill/gc25d_v10_704 DATA_ROOT=$M/data/gc25d_v10_704 \ setsid nohup bash $M/scripts/chain_after_sft.sh > $M/outputs/distill/gc25d_v10_704/chain.log 2>&1 & # to skip the wait/pick: add SFT= # per stage by hand: GC_RES=1280x704 DATA_ROOT=$M/data/gc25d_v10_704 OUT=$M/outputs/distill/gc25d_v10_704/stage1_tf \ # SFT= bash $M/code/gc25d/gcdistill/run_stage1.sh (likewise stage2 AR=…, stage3 STUDENT=… SFT=…) # sampling / eval at 704p env $ENV CUDA_VISIBLE_DEVICES=0 $M/.venv/bin/python $M/code/gc25d/sample.py --ckpt $M/outputs/sft_v10_704/step-N.safetensors \ --data_root $M/data/gc25d_v10_704 --val_n 8 --out_dir $M/outputs/sft_v10_704/eval_step-N # deploy-path parity at 704p: the live.py model path (ref/ live.py, maliang modules injected ⇒ follows GC_RES) vs # eval_rollout.deploy_rollout, 3 streamed cells, must be bitwise equal env $ENV AROPE_ROOT=$M/code/arope CUDA_VISIBLE_DEVICES=0 $M/.venv/bin/python $M/scripts/distill_smoke/live_path_check.py \ --ckpt $M/outputs/distill/gc25d_v10_704/stage3_dmd/ema/ema_step500.safetensors --data_root $M/data/gc25d_v10_704 \ --out $M/outputs/distill/gc25d_v10_704/eval/live_path ``` ## Open issues / caveats 1. **Deploy package is still 480p.** `ref/gc25d-live-demo/code/gc25d/{geometry.py,codeworld/*}` is a separate hard-coded 832×480 copy (e.g. `gen_server.py` has `H, W = 480, 832`) and was not touched. A 704p deploy needs it ported (geometry + code world + server shapes). Until then the pending deploy patch **refuses** a 704p ckpt in `LiveGame.__init__`, before any weights are read; it no longer loads silently at 480p. The maliang `geometry.py` was deliberately **not** copied into the patched package: the package would then follow an inherited `GC_RES=1280x704`, and the guard would pass a 704p ckpt into a server that still hard-codes 480×832. The model path itself is resolution-generic: `live_path_check` at `GC_RES=1280x704` (injected maliang modules) streams 3 cells at 704×1280, bitwise equal to `eval_rollout` (see the review fix pass). 2. **`capsules.render_capsules` bounding box at 832×480.** The world-axis bbox (endpoints ± r along the world axes) underestimates a sphere's screen radius under an oblique camera. The default is now `cam_axes` on (adds ± r along the camera axes) at every resolution **except** 832×480, which keeps the old bbox bit for bit, so `data/gc25d_v10/cond_ext_canvas` and the 480p verify reports reproduce. At 480p the old bbox misses 0.080 % of character px (29 has_ext clips, every 3rd frame); `cam_axes=True` misses 0. Turning it on at 480p is a one-line default change in `render_capsules` (the 480p code world and canvases would then gain those pixels); that is the lead's call. 3. Verify PSNR WARN band: [21, 40] at 832×480 (as before), [20, 40] at other resolutions (native GT is ≈ 1 dB lower on busy clips). 4. Full-res Wan-VAE decode for long chains (`eval_rollout --vae wan --chain_cells 241`) at 704p was not tried. The chain eval in `chain_after_sft.sh` already uses TAEHV for chain241 and Wan only for clip28 (which runs fine: 55 s). 5. Smoke artifacts take ≈ 200 GB under `outputs/_smoke/{sft704,res704}`. Most of it is the `state_A` slots of `sft704` and `res704/stage1_probe` (≈ 66 GB each). They can be deleted, but `sft704/step-8.safetensors` is the 704p init the probes used. 6. SPEC §1/§5 text and the other impl docs describe the 832×480 instance. Their shapes are what geometry gives at the default `GC_RES`.