maliang / docs /impl /genbench3.md
teawhite's picture
Add files using upload-large-folder tool
51ac7cc verified
|
Raw History Blame Contribute Delete
12.6 kB
# genbench3: generalization of the distilled DMD student (2026-10-24)
The question: after CF++ distillation, does the 480p student still generalise to first frames it never saw?
That means very different art styles and different player or boss appearances. The first frame must stay
consistent with the semantic map. Otherwise the test is meaningless, because the model is told to draw the map.
* Model: `outputs/distill/gc25d_v10/stage3_dmd/ema/ema_step500.safetensors`.
* Teacher used for comparison: `outputs/sft_v10/step-8000.safetensors`.
## Pipeline (`scripts/genbench3/`, outputs `outputs/genbench3/`)
| step | script | what it does |
|---|---|---|
| sources | `sources.py` | 8 chains that fill 241 cells (60 s), one scene each. 3 are val runs and 5 are train runs (the val split has only 10 runs and few heads qualify). Frame 0 must have the player β‰₯ 400 px and the boss β‰₯ 1200 px (600 px in the relaxed pass). Both must be fully inside the frame (≀ 0.5 % of their pixels within 24 px of the border), their boxes must overlap ≀ 0.3 IoU, and UNKNOWN pixels ≀ 5 %. |
| first frames | `make_f0.py` (PCE env `pce_f3`, FLUX.2 klein-9B) | 16 styles Γ— 6 seeds. The prompt is `RESTYLE_V4`, a material/rendering-only rewrite: nothing is added or removed. Themed styles are phrased as "existing X becomes Y". 4 player looks and 4 boss looks Γ— 2 seeds, made by crop-editing. |
| gate | `vlm_gate.py` (Qwen3-VL-32B, deterministic P(yes)) | Automatic check plus VLM questions; details below. The best seed per (source, variant) slot is kept. |
| combos | `plan.py combo / both` | Style + one character: a second FLUX edit on the gated style frame, then gated again. Both characters edited: the two gated char edits are composited, since each one changes pixels only near its own character. |
| rollouts | `rollout.py` | **dmd**: `eval_rollout.deploy_rollout`, the same path as live.py: 4 steps, sink 3 / window 17, seed 0, 241 cells, TAEHV decoding. **bidir**: the teacher, 50 steps, the first 81 frames of the chain head. z0 is the Wan VAE encoding of the png; `orig` is encoded the same way. Each job is claimed with `mkdir <variant>.claim`. |
| metrics | `metrics.py`, `vlm_eval.py` | Paired metrics against the source's `orig` rollout (same semantics, actions and noise), plus Qwen3-VL questions on the generated frames. |
| report | `report.py`, `grid_video.py` | `report/sheet_<tag>.jpg`: variants Γ— t. `report/chars_<tag>.jpg`: crops of the edited character over time. Grid videos are 3Γ—3, 60 s. |
### Making first frames that agree with the semantics
* **Character edits.**
* The edit crop is the character's box with the W:H aspect ratio and 1.7Γ— padding. FLUX edits it at gen resolution.
* The crop is composited back into the **capsule mask (+2 px) βˆͺ the region FLUX changed (Ξ”E > 14) connected to the mask
within 0.18 Γ— the character's size**. The other character is protected.
* The capsule proxies are smaller than the real silhouettes: the lava golem's fists are outside the mask. Compositing only
inside the mask left chimeras, as in the smoke test with the ice troll.
* Outside that region the pixels are bit-identical to the original frame.
* **Styles.** FLUX's restyle with the v1 prompt adds walls, lollipops and brick buildings to empty ground in candy and lego.
That contradicts the map. Two changes helped:
* the "material-only" prompt v4 with styles rephrased as "existing ground / rocks / plants become …";
* 6 seeds per style.
SDEdit (strength 0.8 / 0.9) did not work: with klein's 4 steps the output barely restyles.
* **Gate.**
* **style / combo, automatic:** coarse-scale edge precision against the original β‰₯ 0.6. This is the struct metric at Β½
resolution with Οƒ 2 blur, which removes texture and keeps object outlines. Also, semantic-boundary edge recall must be
β‰₯ 0.8 Γ— the original's.
* **char, automatic:** the composite region is ≀ 3.5 Γ— the mask, and the character still fills its silhouette (bg_fill
rises by ≀ 0.15).
* **VLM questions:** layout β‰₯ 0.15; all other questions β‰₯ 0.5. The questions are:
* style: chars, style;
* char: changed, pose, rest, clean;
* combo: layout, changed, pose, clean.
* **Calibration.** The thresholds were set by eye on every style of train_v10_03 and val_v9_04. VLM layout scores are
strict and noisy for texture-heavy styles: voxel, ukiyoe, papyrus and watercolor frames with the correct layout scored
only 0.2–0.4. So VLM layout is a loose second check and the coarse-edge precision is the main criterion.
* Every accepted style frame was then reviewed by eye. Nothing was vetoed; a few borderline cases remain, such as scattered
candies in train_v9_08 candy.
**Slots that pass** (per source, out of 8):
| result | variants |
|---|---|
| 8/8 | blueprint, chalkboard; player ninja, ranger, silverknight; boss demonlord, skeletonking |
| 7/8 | papyrus; boss icetroll |
| 6/8 | pixelart, watercolor; player firemage |
| 5/8 | voxel; boss stonegolem |
| 4/8 | knitted, porcelain, winter |
| 2/8 | ukiyoe |
| 1/8 | candy, mosaic, neon |
| 0/8 | lava, lego, stainedglass |
| combos | 15/16 |
Themed styles that need content changes (lava, lego, stained glass, candy, neon, mosaic) cannot be made layout-consistent
with FLUX klein editing. They are absent or n = 1 below.
146 rollout jobs: 8 orig, 57 style, 58 char, 8 both, 15 combo; each one has both dmd and bidir.
## Metrics
* `style_clip@t` = d(V_t, O_t) / d(V_0, O_0), with d = 1 βˆ’ cos(CLIP ViT-L/14) on the whole frame. 1 means the new look is
fully kept and 0 means it reverted to orig.
* `look` is a colour index that is robust to how far the two rollouts diverge. Take the role's mask pixels at frame t.
Compute the Lab-distribution W1 distance to the frame-0 new look (V0) and to the frame-0 original look (O0). Then
look = d(Β·, O0) / (d(Β·, O0) + d(Β·, V0)).
* Report `look βˆ’ look_orig`, the same index computed on the orig rollout. It is 1 at t = 0 and β‰ˆ 0 once the look reverts.
* Whole-frame ratios are meaningless for char-only variants: d(0) β‰ˆ 0, so the later divergence gets amplified.
* `edge_rel` = layout_metrics edge_recall of V_t against the semantic map of frame t, divided by the same for O_t.
* VLM, Qwen3-VL-32B P(yes):
* "is this frame clearly <style>" on the whole frame;
* "does <who> look like <look>" on the mask-box crop;
* a "coherent frame" question.
* The orig rollout gets the same questions; that is the false-positive baseline, β‰ˆ 0.01–0.03.
## Results (DMD, 60 s, all 8 sources)
| style | n | VLM "is <style>" at 0 / 5 / 15 / 30 / 60 s | CLIP kept at 5 / 15 / 30 / 60 s | edge_rel at 60 s |
|---|---|---|---|---|
| winter | 4 | 1.00 / 1.00 / 0.99 / 1.00 / 0.99 | 0.84 / 0.75 / 0.68 / 0.67 | 1.01 |
| watercolor | 6 | 1.00 / 1.00 / 0.83 / 0.50 / 0.81 | 0.68 / 0.63 / 0.63 / 0.62 | 1.00 |
| papyrus | 7 | 0.95 / 0.87 / 0.67 / 0.79 / 0.64 | 0.74 / 0.71 / 0.74 / 0.68 | 1.06 |
| pixelart | 6 | 0.95 / 0.19 / 0.19 / 0.18 / 0.36 | 1.06 / 1.16 / 1.08 / 0.81 | 1.04 |
| chalkboard | 8 | 1.00 / 0.86 / 0.38 / 0.14 / 0.26 | 0.89 / 0.73 / 0.78 / 0.67 | 1.01 |
| porcelain | 4 | 0.81 / 0.61 / 0.19 / 0.30 / 0.25 | 0.89 / 0.99 / 0.86 / 0.81 | 0.99 |
| blueprint | 8 | 1.00 / 0.73 / 0.36 / 0.05 / 0.13 | 0.95 / 0.89 / 0.82 / 0.72 | 1.06 |
| knitted | 4 | 0.93 / 0.06 / 0.00 / 0.00 / 0.06 | 0.75 / 0.79 / 0.85 / 0.65 | 1.00 |
| ukiyoe | 2 | 0.95 / 0.16 / 0.00 / 0.00 / 0.00 | 0.61 / 0.90 / 0.60 / 0.67 | 1.08 |
| voxel | 5 | 0.69 / 0.00 / 0.00 / 0.00 / 0.00 | 1.05 / 0.96 / 0.82 / 0.85 | 1.06 |
| all styles | 57 | 0.94 / 0.63 / 0.41 / 0.33 / 0.38 | 0.86 / 0.84 / 0.79 / 0.72 | 1.03 |
(candy, mosaic and neon have n = 1 each and are omitted.)
* **Character looks.**
* The VLM says "looks like <look>" with probability 0.93 at 0 s, 0.22 at 5 s, 0.06 at 15 s and 0.08 at 60 s; the orig
baseline is ≀ 0.03.
* The colour index (look βˆ’ orig baseline) for the player is 0.38 at 2 s, 0.35 at 5 s, 0.32 at 15 s, 0.30 at 30 s and
0.32 at 60 s (n = 30).
* The colour index for the boss is 0.55 at 2 s (n = 28), 0.48 at 5 s (n = 19) and 0.30–0.44 at 15–60 s (n = 8). The boss
is often out of view later.
* The ice troll is the most persistent: VLM 0.59 at 5 s and 0.84 at 60 s; colour index 0.6–0.7.
* **DMD against the bidirectional teacher at 5 s (frame 80).** Style VLM: 0.73 for the teacher, 0.63 for DMD.
* Equal for palette styles: watercolor, winter, candy, neon and mosaic are all 1.00 against 1.00.
* Worse for texture styles:
* knitted 0.29 β†’ 0.06;
* pixelart 0.47 β†’ 0.19;
* ukiyoe 0.44 β†’ 0.16;
* porcelain 0.91 β†’ 0.61;
* chalkboard 0.99 β†’ 0.86.
* CLIP-kept is identical, 0.86 against 0.86.
* Character looks are equally lost (VLM 0.18 against 0.19): **appearance loss is inherited from the teacher**, not caused
by distillation.
* edge_rel is equal.
* The "coherent frame" VLM score is no worse than orig at any time.
## Verified conclusions (adversarial check, 3 independent agents over all 8 sources; transcript wf_934ced7e-65a)
My first reading of three findings came from 2–3 sources. Each was re-checked by a skeptic who looked at every sheet and crop,
plus zoomed frames. The corrected versions:
* **Styles (partially supported):**
* Restyles whose identity is a large colour or lighting shift persist for 60 s. Winter: 7/7 variants have VLM β‰₯ 0.95 at
every time.
* Watercolor and papyrus usually persist (VLM β‰₯ 0.68 at 60 s in 6/7 and 6/8 variants) but can revert within 15–30 s:
watercolor train_v9_07, papyrus train_v10_03 and train_v10_05.
* Geometry, material and pixel-grid styles (voxel, knitted, pixel-art, ukiyo-e) lose that structure within about 5 s:
VLM ≀ 0.53 at 5 s in all 17 variants. They converge to the **source's own rendering**: realistic for the v10 sources,
flat or cel-shaded for the stylized v9 sources. Residues remain, such as felt patches or posterized colour.
* Line-art on a coloured ground (blueprint, chalkboard) sits in between. The ground colour usually persists to 60 s.
The line art fades mostly by 15–30 s, but residual grids and chalk strokes stay visible to 60 s in several sources.
* The paired CLIP ratio does not separate these groups, because it is confounded by trajectory divergence. Rely on the
VLM scores and on visual inspection.
* **Newly revealed objects (partially supported):**
* Only for **vegetation in the realistic v10 outdoor scenes** are trees, bushes and ferns entering the view often drawn in
the realistic training look while the ground keeps the style. This is clearest for chalkboard, watercolor, papyrus,
knitted and mosaic.
* Porcelain and winter usually recolour new vegetation; blueprint draws new rocks and trees as outlines.
* In the v9 sources, newly revealed buildings, props and trees are drawn in the new style.
* In several rollouts the ground reverts too.
* **Characters (my first claim, "bosses revert fast, players keep their colours", was refuted):**
* In 6/8 sources the boss leaves the screen early, after 2–15 s. Up to its last visible frame the boss edit is usually
intact.
* The only clear early reversion is train_v10_03: every boss edit holds through 10 s and reverts at 15 s.
* Where the boss stays on screen (train_v10_06, val_v9_02), the main colours of most boss edits last to 60 s, and the
finer identity (skull, crown, horns) fades after about 10–20 s.
* Players keep their main colours to 60 s in most visible cases. The exceptions are val_v10_01 ranger, train_v9_08
silverknight and train_v10_05 at about 45 s.
* VLM identity at 5 s is **higher for bosses (β‰ˆ 0.46) than for players (β‰ˆ 0.12)**, probably because bosses have more
pixels. **Bosses do not revert faster than players.** The idea that the capsule proxy encodes boss identity has no
support.
* Confounds: several orig players already resemble an edit. In train_v10_05 the orig is a white/blue knight; in
val_v9_02 the orig wears a grey-green cloak.
## Caveats
* n is small per style: 1–8 sources. The VLM style question asks whether the frame is "clearly" in that style. It is strict,
and 480p TAEHV frames lose fine strokes.
* The colour look-index cannot tell looks apart when the old and new looks share a palette: demonlord against the black and
red lava golem.
* Boss statistics after about 5 s rest on few rollouts, because the boss leaves the view.
* All 10 scenes appear in training; "val" means held-out runs only. There are no unseen maps here.