docs/impl/genbench3.md

genbench3: generalization of the distilled DMD student (2026-10-24)

The question: after CF++ distillation, does the 480p student still generalise to first frames it never saw? That means very different art styles and different player or boss appearances. The first frame must stay consistent with the semantic map. Otherwise the test is meaningless, because the model is told to draw the map.

• Model: outputs/distill/gc25d_v10/stage3_dmd/ema/ema_step500.safetensors.
• Teacher used for comparison: outputs/sft_v10/step-8000.safetensors.

Pipeline (scripts/genbench3/, outputs outputs/genbench3/)

step	script	what it does
sources	sources.py	8 chains that fill 241 cells (60 s), one scene each. 3 are val runs and 5 are train runs (the val split has only 10 runs and few heads qualify). Frame 0 must have the player ≥ 400 px and the boss ≥ 1200 px (600 px in the relaxed pass). Both must be fully inside the frame (≤ 0.5 % of their pixels within 24 px of the border), their boxes must overlap ≤ 0.3 IoU, and UNKNOWN pixels ≤ 5 %.
first frames	make_f0.py (PCE env pce_f3, FLUX.2 klein-9B)	16 styles × 6 seeds. The prompt is RESTYLE_V4, a material/rendering-only rewrite: nothing is added or removed. Themed styles are phrased as "existing X becomes Y". 4 player looks and 4 boss looks × 2 seeds, made by crop-editing.
gate	vlm_gate.py (Qwen3-VL-32B, deterministic P(yes))	Automatic check plus VLM questions; details below. The best seed per (source, variant) slot is kept.
combos	plan.py combo / both	Style + one character: a second FLUX edit on the gated style frame, then gated again. Both characters edited: the two gated char edits are composited, since each one changes pixels only near its own character.
rollouts	rollout.py	dmd: eval_rollout.deploy_rollout, the same path as live.py: 4 steps, sink 3 / window 17, seed 0, 241 cells, TAEHV decoding. bidir: the teacher, 50 steps, the first 81 frames of the chain head. z0 is the Wan VAE encoding of the png; orig is encoded the same way. Each job is claimed with mkdir <variant>.claim.
metrics	metrics.py, vlm_eval.py	Paired metrics against the source's orig rollout (same semantics, actions and noise), plus Qwen3-VL questions on the generated frames.
report	report.py, grid_video.py	report/sheet_<tag>.jpg: variants × t. report/chars_<tag>.jpg: crops of the edited character over time. Grid videos are 3×3, 60 s.

Making first frames that agree with the semantics

• Character edits.

  • The edit crop is the character's box with the W:H aspect ratio and 1.7× padding. FLUX edits it at gen resolution.
  • The crop is composited back into the capsule mask (+2 px) ∪ the region FLUX changed (ΔE > 14) connected to the mask within 0.18 × the character's size. The other character is protected.
  • The capsule proxies are smaller than the real silhouettes: the lava golem's fists are outside the mask. Compositing only inside the mask left chimeras, as in the smoke test with the ice troll.
  • Outside that region the pixels are bit-identical to the original frame.

• Styles. FLUX's restyle with the v1 prompt adds walls, lollipops and brick buildings to empty ground in candy and lego. That contradicts the map. Two changes helped:

  • the "material-only" prompt v4 with styles rephrased as "existing ground / rocks / plants become …";
  • 6 seeds per style.

  SDEdit (strength 0.8 / 0.9) did not work: with klein's 4 steps the output barely restyles.

• Gate.

  • style / combo, automatic: coarse-scale edge precision against the original ≥ 0.6. This is the struct metric at ½ resolution with σ 2 blur, which removes texture and keeps object outlines. Also, semantic-boundary edge recall must be ≥ 0.8 × the original's.
  • char, automatic: the composite region is ≤ 3.5 × the mask, and the character still fills its silhouette (bg_fill rises by ≤ 0.15).
  • VLM questions: layout ≥ 0.15; all other questions ≥ 0.5. The questions are:
    • style: chars, style;
    • char: changed, pose, rest, clean;
    • combo: layout, changed, pose, clean.
  • Calibration. The thresholds were set by eye on every style of train_v10_03 and val_v9_04. VLM layout scores are strict and noisy for texture-heavy styles: voxel, ukiyoe, papyrus and watercolor frames with the correct layout scored only 0.2–0.4. So VLM layout is a loose second check and the coarse-edge precision is the main criterion.
  • Every accepted style frame was then reviewed by eye. Nothing was vetoed; a few borderline cases remain, such as scattered candies in train_v9_08 candy.

Slots that pass (per source, out of 8):

result	variants
8/8	blueprint, chalkboard; player ninja, ranger, silverknight; boss demonlord, skeletonking
7/8	papyrus; boss icetroll
6/8	pixelart, watercolor; player firemage
5/8	voxel; boss stonegolem
4/8	knitted, porcelain, winter
2/8	ukiyoe
1/8	candy, mosaic, neon
0/8	lava, lego, stainedglass
combos	15/16

Themed styles that need content changes (lava, lego, stained glass, candy, neon, mosaic) cannot be made layout-consistent with FLUX klein editing. They are absent or n = 1 below.

146 rollout jobs: 8 orig, 57 style, 58 char, 8 both, 15 combo; each one has both dmd and bidir.

Metrics

• style_clip@t = d(V_t, O_t) / d(V_0, O_0), with d = 1 − cos(CLIP ViT-L/14) on the whole frame. 1 means the new look is fully kept and 0 means it reverted to orig.
• look is a colour index that is robust to how far the two rollouts diverge. Take the role's mask pixels at frame t. Compute the Lab-distribution W1 distance to the frame-0 new look (V0) and to the frame-0 original look (O0). Then look = d(·, O0) / (d(·, O0) + d(·, V0)).
  • Report look − look_orig, the same index computed on the orig rollout. It is 1 at t = 0 and ≈ 0 once the look reverts.
  • Whole-frame ratios are meaningless for char-only variants: d(0) ≈ 0, so the later divergence gets amplified.
• edge_rel = layout_metrics edge_recall of V_t against the semantic map of frame t, divided by the same for O_t.
• VLM, Qwen3-VL-32B P(yes):
  • "is this frame clearly