|
Download docs/impl/genbench3.md from teawhite/maliang: direct link, hf CLI and curl.
- Browser
- Download file 12.6 kB
-
https://huggingface.co/teawhite/maliang/resolve/main/docs/impl/genbench3.md
- Command line
-
hf download hf://teawhite/maliang/docs/impl/genbench3.md
-
curl -L -o genbench3.md https://huggingface.co/teawhite/maliang/resolve/main/docs/impl/genbench3.md
12.6 kB
| # genbench3: generalization of the distilled DMD student (2026-10-24) | |
| The question: after CF++ distillation, does the 480p student still generalise to first frames it never saw? | |
| That means very different art styles and different player or boss appearances. The first frame must stay | |
| consistent with the semantic map. Otherwise the test is meaningless, because the model is told to draw the map. | |
| * Model: `outputs/distill/gc25d_v10/stage3_dmd/ema/ema_step500.safetensors`. | |
| * Teacher used for comparison: `outputs/sft_v10/step-8000.safetensors`. | |
| ## Pipeline (`scripts/genbench3/`, outputs `outputs/genbench3/`) | |
| | step | script | what it does | | |
| |---|---|---| | |
| | sources | `sources.py` | 8 chains that fill 241 cells (60 s), one scene each. 3 are val runs and 5 are train runs (the val split has only 10 runs and few heads qualify). Frame 0 must have the player β₯ 400 px and the boss β₯ 1200 px (600 px in the relaxed pass). Both must be fully inside the frame (β€ 0.5 % of their pixels within 24 px of the border), their boxes must overlap β€ 0.3 IoU, and UNKNOWN pixels β€ 5 %. | | |
| | first frames | `make_f0.py` (PCE env `pce_f3`, FLUX.2 klein-9B) | 16 styles Γ 6 seeds. The prompt is `RESTYLE_V4`, a material/rendering-only rewrite: nothing is added or removed. Themed styles are phrased as "existing X becomes Y". 4 player looks and 4 boss looks Γ 2 seeds, made by crop-editing. | | |
| | gate | `vlm_gate.py` (Qwen3-VL-32B, deterministic P(yes)) | Automatic check plus VLM questions; details below. The best seed per (source, variant) slot is kept. | | |
| | combos | `plan.py combo / both` | Style + one character: a second FLUX edit on the gated style frame, then gated again. Both characters edited: the two gated char edits are composited, since each one changes pixels only near its own character. | | |
| | rollouts | `rollout.py` | **dmd**: `eval_rollout.deploy_rollout`, the same path as live.py: 4 steps, sink 3 / window 17, seed 0, 241 cells, TAEHV decoding. **bidir**: the teacher, 50 steps, the first 81 frames of the chain head. z0 is the Wan VAE encoding of the png; `orig` is encoded the same way. Each job is claimed with `mkdir <variant>.claim`. | | |
| | metrics | `metrics.py`, `vlm_eval.py` | Paired metrics against the source's `orig` rollout (same semantics, actions and noise), plus Qwen3-VL questions on the generated frames. | | |
| | report | `report.py`, `grid_video.py` | `report/sheet_<tag>.jpg`: variants Γ t. `report/chars_<tag>.jpg`: crops of the edited character over time. Grid videos are 3Γ3, 60 s. | | |
| ### Making first frames that agree with the semantics | |
| * **Character edits.** | |
| * The edit crop is the character's box with the W:H aspect ratio and 1.7Γ padding. FLUX edits it at gen resolution. | |
| * The crop is composited back into the **capsule mask (+2 px) βͺ the region FLUX changed (ΞE > 14) connected to the mask | |
| within 0.18 Γ the character's size**. The other character is protected. | |
| * The capsule proxies are smaller than the real silhouettes: the lava golem's fists are outside the mask. Compositing only | |
| inside the mask left chimeras, as in the smoke test with the ice troll. | |
| * Outside that region the pixels are bit-identical to the original frame. | |
| * **Styles.** FLUX's restyle with the v1 prompt adds walls, lollipops and brick buildings to empty ground in candy and lego. | |
| That contradicts the map. Two changes helped: | |
| * the "material-only" prompt v4 with styles rephrased as "existing ground / rocks / plants become β¦"; | |
| * 6 seeds per style. | |
| SDEdit (strength 0.8 / 0.9) did not work: with klein's 4 steps the output barely restyles. | |
| * **Gate.** | |
| * **style / combo, automatic:** coarse-scale edge precision against the original β₯ 0.6. This is the struct metric at Β½ | |
| resolution with Ο 2 blur, which removes texture and keeps object outlines. Also, semantic-boundary edge recall must be | |
| β₯ 0.8 Γ the original's. | |
| * **char, automatic:** the composite region is β€ 3.5 Γ the mask, and the character still fills its silhouette (bg_fill | |
| rises by β€ 0.15). | |
| * **VLM questions:** layout β₯ 0.15; all other questions β₯ 0.5. The questions are: | |
| * style: chars, style; | |
| * char: changed, pose, rest, clean; | |
| * combo: layout, changed, pose, clean. | |
| * **Calibration.** The thresholds were set by eye on every style of train_v10_03 and val_v9_04. VLM layout scores are | |
| strict and noisy for texture-heavy styles: voxel, ukiyoe, papyrus and watercolor frames with the correct layout scored | |
| only 0.2β0.4. So VLM layout is a loose second check and the coarse-edge precision is the main criterion. | |
| * Every accepted style frame was then reviewed by eye. Nothing was vetoed; a few borderline cases remain, such as scattered | |
| candies in train_v9_08 candy. | |
| **Slots that pass** (per source, out of 8): | |
| | result | variants | | |
| |---|---| | |
| | 8/8 | blueprint, chalkboard; player ninja, ranger, silverknight; boss demonlord, skeletonking | | |
| | 7/8 | papyrus; boss icetroll | | |
| | 6/8 | pixelart, watercolor; player firemage | | |
| | 5/8 | voxel; boss stonegolem | | |
| | 4/8 | knitted, porcelain, winter | | |
| | 2/8 | ukiyoe | | |
| | 1/8 | candy, mosaic, neon | | |
| | 0/8 | lava, lego, stainedglass | | |
| | combos | 15/16 | | |
| Themed styles that need content changes (lava, lego, stained glass, candy, neon, mosaic) cannot be made layout-consistent | |
| with FLUX klein editing. They are absent or n = 1 below. | |
| 146 rollout jobs: 8 orig, 57 style, 58 char, 8 both, 15 combo; each one has both dmd and bidir. | |
| ## Metrics | |
| * `style_clip@t` = d(V_t, O_t) / d(V_0, O_0), with d = 1 β cos(CLIP ViT-L/14) on the whole frame. 1 means the new look is | |
| fully kept and 0 means it reverted to orig. | |
| * `look` is a colour index that is robust to how far the two rollouts diverge. Take the role's mask pixels at frame t. | |
| Compute the Lab-distribution W1 distance to the frame-0 new look (V0) and to the frame-0 original look (O0). Then | |
| look = d(Β·, O0) / (d(Β·, O0) + d(Β·, V0)). | |
| * Report `look β look_orig`, the same index computed on the orig rollout. It is 1 at t = 0 and β 0 once the look reverts. | |
| * Whole-frame ratios are meaningless for char-only variants: d(0) β 0, so the later divergence gets amplified. | |
| * `edge_rel` = layout_metrics edge_recall of V_t against the semantic map of frame t, divided by the same for O_t. | |
| * VLM, Qwen3-VL-32B P(yes): | |
| * "is this frame clearly <style>" on the whole frame; | |
| * "does <who> look like <look>" on the mask-box crop; | |
| * a "coherent frame" question. | |
| * The orig rollout gets the same questions; that is the false-positive baseline, β 0.01β0.03. | |
| ## Results (DMD, 60 s, all 8 sources) | |
| | style | n | VLM "is <style>" at 0 / 5 / 15 / 30 / 60 s | CLIP kept at 5 / 15 / 30 / 60 s | edge_rel at 60 s | | |
| |---|---|---|---|---| | |
| | winter | 4 | 1.00 / 1.00 / 0.99 / 1.00 / 0.99 | 0.84 / 0.75 / 0.68 / 0.67 | 1.01 | | |
| | watercolor | 6 | 1.00 / 1.00 / 0.83 / 0.50 / 0.81 | 0.68 / 0.63 / 0.63 / 0.62 | 1.00 | | |
| | papyrus | 7 | 0.95 / 0.87 / 0.67 / 0.79 / 0.64 | 0.74 / 0.71 / 0.74 / 0.68 | 1.06 | | |
| | pixelart | 6 | 0.95 / 0.19 / 0.19 / 0.18 / 0.36 | 1.06 / 1.16 / 1.08 / 0.81 | 1.04 | | |
| | chalkboard | 8 | 1.00 / 0.86 / 0.38 / 0.14 / 0.26 | 0.89 / 0.73 / 0.78 / 0.67 | 1.01 | | |
| | porcelain | 4 | 0.81 / 0.61 / 0.19 / 0.30 / 0.25 | 0.89 / 0.99 / 0.86 / 0.81 | 0.99 | | |
| | blueprint | 8 | 1.00 / 0.73 / 0.36 / 0.05 / 0.13 | 0.95 / 0.89 / 0.82 / 0.72 | 1.06 | | |
| | knitted | 4 | 0.93 / 0.06 / 0.00 / 0.00 / 0.06 | 0.75 / 0.79 / 0.85 / 0.65 | 1.00 | | |
| | ukiyoe | 2 | 0.95 / 0.16 / 0.00 / 0.00 / 0.00 | 0.61 / 0.90 / 0.60 / 0.67 | 1.08 | | |
| | voxel | 5 | 0.69 / 0.00 / 0.00 / 0.00 / 0.00 | 1.05 / 0.96 / 0.82 / 0.85 | 1.06 | | |
| | all styles | 57 | 0.94 / 0.63 / 0.41 / 0.33 / 0.38 | 0.86 / 0.84 / 0.79 / 0.72 | 1.03 | | |
| (candy, mosaic and neon have n = 1 each and are omitted.) | |
| * **Character looks.** | |
| * The VLM says "looks like <look>" with probability 0.93 at 0 s, 0.22 at 5 s, 0.06 at 15 s and 0.08 at 60 s; the orig | |
| baseline is β€ 0.03. | |
| * The colour index (look β orig baseline) for the player is 0.38 at 2 s, 0.35 at 5 s, 0.32 at 15 s, 0.30 at 30 s and | |
| 0.32 at 60 s (n = 30). | |
| * The colour index for the boss is 0.55 at 2 s (n = 28), 0.48 at 5 s (n = 19) and 0.30β0.44 at 15β60 s (n = 8). The boss | |
| is often out of view later. | |
| * The ice troll is the most persistent: VLM 0.59 at 5 s and 0.84 at 60 s; colour index 0.6β0.7. | |
| * **DMD against the bidirectional teacher at 5 s (frame 80).** Style VLM: 0.73 for the teacher, 0.63 for DMD. | |
| * Equal for palette styles: watercolor, winter, candy, neon and mosaic are all 1.00 against 1.00. | |
| * Worse for texture styles: | |
| * knitted 0.29 β 0.06; | |
| * pixelart 0.47 β 0.19; | |
| * ukiyoe 0.44 β 0.16; | |
| * porcelain 0.91 β 0.61; | |
| * chalkboard 0.99 β 0.86. | |
| * CLIP-kept is identical, 0.86 against 0.86. | |
| * Character looks are equally lost (VLM 0.18 against 0.19): **appearance loss is inherited from the teacher**, not caused | |
| by distillation. | |
| * edge_rel is equal. | |
| * The "coherent frame" VLM score is no worse than orig at any time. | |
| ## Verified conclusions (adversarial check, 3 independent agents over all 8 sources; transcript wf_934ced7e-65a) | |
| My first reading of three findings came from 2β3 sources. Each was re-checked by a skeptic who looked at every sheet and crop, | |
| plus zoomed frames. The corrected versions: | |
| * **Styles (partially supported):** | |
| * Restyles whose identity is a large colour or lighting shift persist for 60 s. Winter: 7/7 variants have VLM β₯ 0.95 at | |
| every time. | |
| * Watercolor and papyrus usually persist (VLM β₯ 0.68 at 60 s in 6/7 and 6/8 variants) but can revert within 15β30 s: | |
| watercolor train_v9_07, papyrus train_v10_03 and train_v10_05. | |
| * Geometry, material and pixel-grid styles (voxel, knitted, pixel-art, ukiyo-e) lose that structure within about 5 s: | |
| VLM β€ 0.53 at 5 s in all 17 variants. They converge to the **source's own rendering**: realistic for the v10 sources, | |
| flat or cel-shaded for the stylized v9 sources. Residues remain, such as felt patches or posterized colour. | |
| * Line-art on a coloured ground (blueprint, chalkboard) sits in between. The ground colour usually persists to 60 s. | |
| The line art fades mostly by 15β30 s, but residual grids and chalk strokes stay visible to 60 s in several sources. | |
| * The paired CLIP ratio does not separate these groups, because it is confounded by trajectory divergence. Rely on the | |
| VLM scores and on visual inspection. | |
| * **Newly revealed objects (partially supported):** | |
| * Only for **vegetation in the realistic v10 outdoor scenes** are trees, bushes and ferns entering the view often drawn in | |
| the realistic training look while the ground keeps the style. This is clearest for chalkboard, watercolor, papyrus, | |
| knitted and mosaic. | |
| * Porcelain and winter usually recolour new vegetation; blueprint draws new rocks and trees as outlines. | |
| * In the v9 sources, newly revealed buildings, props and trees are drawn in the new style. | |
| * In several rollouts the ground reverts too. | |
| * **Characters (my first claim, "bosses revert fast, players keep their colours", was refuted):** | |
| * In 6/8 sources the boss leaves the screen early, after 2β15 s. Up to its last visible frame the boss edit is usually | |
| intact. | |
| * The only clear early reversion is train_v10_03: every boss edit holds through 10 s and reverts at 15 s. | |
| * Where the boss stays on screen (train_v10_06, val_v9_02), the main colours of most boss edits last to 60 s, and the | |
| finer identity (skull, crown, horns) fades after about 10β20 s. | |
| * Players keep their main colours to 60 s in most visible cases. The exceptions are val_v10_01 ranger, train_v9_08 | |
| silverknight and train_v10_05 at about 45 s. | |
| * VLM identity at 5 s is **higher for bosses (β 0.46) than for players (β 0.12)**, probably because bosses have more | |
| pixels. **Bosses do not revert faster than players.** The idea that the capsule proxy encodes boss identity has no | |
| support. | |
| * Confounds: several orig players already resemble an edit. In train_v10_05 the orig is a white/blue knight; in | |
| val_v9_02 the orig wears a grey-green cloak. | |
| ## Caveats | |
| * n is small per style: 1β8 sources. The VLM style question asks whether the frame is "clearly" in that style. It is strict, | |
| and 480p TAEHV frames lose fine strokes. | |
| * The colour look-index cannot tell looks apart when the old and new looks share a palette: demonlord against the black and | |
| red lava golem. | |
| * Boss statistics after about 5 s rest on few rollouts, because the boss leaves the view. | |
| * All 10 scenes appear in training; "val" means held-out runs only. There are no unseen maps here. | |