Add evaluator models (9B step100, 27B step296): checkpoints, benchmark, usage

README.md

@@ -11,6 +11,8 @@ tags:
 - javascript
 - code-generation
 - full-sft
+- evaluator
+- reward-model
 ---
 
 # DesignCoder
@@ -18,14 +20,25 @@ tags:
 Checkpoint collection for **DesignCoder**, a family of full-parameter SFT models for UI design
 research and end-to-end HTML/CSS/JavaScript implementation.
 
+The repository holds **two kinds of model**:
+
+| Kind | What it does | Folder prefix |
+|---|---|---|
+| **Generation** | writes HTML/CSS/JS from a design brief | `designcoder_{basemodel}_...` |
+| **Evaluator** | scores a rendered UI screenshot against a rubric | `designcoder_evaluator_{basemodel}_...` |
+
+They are trained on **different datasets** and answer **different inputs** — an evaluator will
+not generate pages, and a generation model will not produce valid rubric verdicts.
+
 Each subfolder in this repository is a self-contained, directly loadable checkpoint.
 
 ## Naming convention
 
 ```
-designcoder_{basemodel}_{size}_{optimizer}_bs{global_batch}[_{extra_axes}]_step{global_step}
+designcoder_[evaluator_]{basemodel}_{size}_{optimizer}_bs{global_batch}[_{extra_axes}]_step{global_step}
 ```
 
+- `evaluator`: present only for rubric-scoring models; absent means generation
 - `basemodel` / `size`: base model family and parameter scale
 - `optimizer`: `muon` or `adamw`
 - `bs`: global batch size (`per_device × grad_accum × world_size`)
@@ -35,16 +48,22 @@ designcoder_{basemodel}_{size}_{optimizer}_bs{global_batch}[_{extra_axes}]_step{
 
 ## Dataset revisions
 
-Checkpoints in this repository come from two different dataset revisions. **Scores and loss
+Checkpoints in this repository come from three different dataset revisions. **Scores and loss
 values are only comparable within the same revision.**
 
-| Tag | Samples | Used by |
-|---|---:|---|
-| *(untagged)* `data37865` | 37,865 | `*_step1900`, `*_step3800` |
-| `data41287` | 41,287 | `*_data41287_step200`, `*_data41287_step400` |
+| Tag | Samples | Dataset | Used by |
+|---|---:|---|---|
+| *(untagged)* `data37865` | 37,865 | `designcoder_sft_v2` (generation) | `*_step1900`, `*_step3800` |
+| `data41287` | 41,287 | `designcoder_sft_v2` (generation) | `*_data41287_step200`, `*_data41287_step400` |
+| `data37847` | 37,847 | **`DesignCoder-evaluate`** (screenshot scoring) | `designcoder_evaluator_*` |
+
+> ⚠️ `data37847` and `data37865` are **entirely different datasets** — the near-identical
+> sample counts are a coincidence. Never compare loss across them.
 
 ## Checkpoints
 
+### Generation models
+
 | Subfolder | Base model | Optimizer | LR | Global batch | Dataset | Step | bench-200 (full, n=200) | Notes |
 |---|---|---|---|---|---|---|---|---|
 | `designcoder_qwen3.5_4b_muon_bs32_step1900` | Qwen3.5-4B | Muon | 1e-5 | 32 | 37,865 | 1900 | – | smallest of the first release |
@@ -59,6 +78,29 @@ values are only comparable within the same revision.**
 All four `data41287` scores are **final full-benchmark runs: 200/200 rollouts, 200/200
 screenshot captures, 200/200 judge evaluations** per model (no subsetting).
 
+### Evaluator models
+
+Rubric scorers for UI screenshots — intended as reward / judge models during rollout, **not**
+for page generation. Trained on `DesignCoder-evaluate` (37,847 samples), evaluated on
+held-out benchmark screenshots scored by a stronger external vision judge (κ = agreement
+with that judge; see [Evaluator benchmark](#evaluator-benchmark)).
+
+| Subfolder | Base model | Optimizer | LR | Global batch | Dataset | Step | OOD κ | Notes |
+|---|---|---|---|---|---|---|---:|---|
+| `designcoder_evaluator_qwen3.5_9b_adamw_bs256_data37847_step100` | Qwen3.5-9B | AdamW | 1e-5 | 256 | 37,847 | 100 | **0.694** | 1/3 memory, 2× speed; **requires exactly 10-item rubrics** |
+| `designcoder_evaluator_qwen3.8_27b_adamw_bs256_data37847_step296` | Qwen3.8-27B | AdamW | 5e-6 | 256 | 37,847 | 296 | 0.664 | **robust to any rubric length** — default choice |
+
+**Which one to use.** Counter-intuitively the 9B has slightly *higher* agreement; the 27B's
+advantage is **format robustness**. Verdict-count exact match across rubric lengths:
+
+| Rubric items | 5 | 10 | 15 | 20 | 25 |
+|---|---:|---:|---:|---:|---:|
+| 9B step100 | 0.00 | **1.00** | 0.92 | 0.94 | 0.85 |
+| 27B step296 | **1.00** | **1.00** | **0.99** | **1.00** | **1.00** |
+
+Training rubrics are always exactly 10 items, so anything else is out-of-distribution.
+Use the 9B when your rubric is fixed at 10 items; use the 27B otherwise.
+
 ## Benchmark
 
 `bench-200` is the frozen 200-case DesignCoder benchmark (100 Track A landing, 40 Track A
@@ -133,8 +175,80 @@ subset score collapsed from 84.22 to 68.35. **Do not pick checkpoints from this
 loss.** Note also that small subsets systematically overestimate: the subset ranks checkpoints
 correctly but runs several points above the full 200-case benchmark.
 
+## Evaluator benchmark
+
+`DesignCoder-evaluate` ships **no test split**, and both evaluators trained on all 37,847
+samples for 2 epochs — so there is no clean in-training validation set. Evaluation instead
+uses **held-out inputs**: 239 generated-page screenshots from the DesignCoder benchmark, each
+already scored check-by-check by a stronger external vision judge. A byte-level md5 check
+confirmed **0 overlap** between those benchmark screenshots and the 37,851 training images.
+
+Alignment is possible because the training target's `frozen_dynamic_scores.verdicts` and the
+external judge's `verdict` are both binary and share the same `(dimension, point)` structure.
+
+**Headline metrics** (239 held-out cases, original 24–25-item benchmark rubrics):
+
+| Metric | 9B step100 | 27B step296 |
+|---|---:|---:|
+| JSON parse rate | 1.0000 | **1.0000** |
+| Per-check agreement | 0.9226 | **0.9313** |
+| Cohen's κ vs external judge | 0.6353 | **0.6802** |
+| Defect recall (verdict = 0) | 0.6581 | **0.6996** |
+| Case-level Pearson | 0.8417 | **0.8572** |
+
+Raw agreement is a weak signal here — 87% of checks are `1`, so always-pass already scores
+~88%. **κ and defect recall are the meaningful numbers.**
+
+**Reproducing the known quality ordering.** The external judge ranks three generation
+checkpoints 4B `0.833` < 9B `0.884` < 27B `0.905`. Both published evaluators recover that
+ordering to within ~2pp:
+
+| Scorer | 4B | 9B | 27B | Ordering |
+|---|---:|---:|---:|---|
+| External judge (reference) | 0.833 | 0.884 | 0.905 | – |
+| Evaluator 9B step100 | 0.855 | 0.886 | 0.910 | ✅ |
+| Evaluator 27B step296 | 0.840 | 0.893 | 0.909 | ✅ |
+
+### Evaluator checkpoint selection
+
+Checkpoints were chosen on a purpose-built **out-of-distribution suite** (576 prompts:
+rubric lengths 5/10/15/20/25 × three dimension-grouping shapes), not on training loss.
+The two families behave differently:
+
+| Run | Step | Train loss | OOD κ | Overfit gap (in-dist − OOD) |
+|---|---:|---:|---:|---:|
+| 9B AdamW | 50 | – | 0.6568 | +0.004 |
+| 9B AdamW | **100** | 0.4956 | **0.6943** | +0.007 |
+| 9B AdamW | 150 | – | 0.6730 | +0.018 |
+| 9B AdamW | 200 | – | 0.6649 | +0.011 |
+| 9B AdamW | 250 | – | 0.6765 | −0.019 |
+| 9B AdamW | 296 (final) | 0.4358 | 0.6501 | +0.027 |
+| 27B AdamW | 200 | – | 0.6625 | +0.002 |
+| 27B AdamW | **296 (final)** | 0.4246 | **0.6640** | −0.029 |
+
+The 9B reproduces the "loss keeps falling, held-out quality degrades" pattern seen in the
+generation runs — but peaks **much earlier** (~34% of training vs ~75%). The 27B shows no
+late-training degradation, so its final checkpoint is published.
+
+### Evaluator optimizer ablation
+
+A 9B run with **Muon + `pure_bf16`** (identical data, batch, and steps) failed badly:
+
+| | Muon + pure_bf16 | AdamW + ZeRO-3 |
+|---|---:|---:|
+| Final train loss | 0.7505 | **0.4358** |
+| Truncated generations | 14 / 389 | **0 / 389** |
+| Cohen's κ | 0.4831 | **0.6353** |
+| Quality ordering | ❌ inverted | ✅ correct |
+
+The Muon arm stalled early (loss 1.376 → 1.138 over 50 steps) and degenerated into repetition
+loops hitting the 8192-token cap. It is not published. **Do not use Muon + `pure_bf16` for
+this task** — note this differs from the generation family, where Muon is viable.
+
 ## Shared training setup
 
+**Generation models**
+
 - Objective: full-parameter supervised fine-tuning (no LoRA / adapters)
 - Dataset: `designcoder_sft_v2_train` in ShareGPT format (see revision table above)
 - Chat template: `qwen3_5` with thinking enabled
@@ -142,8 +256,19 @@ correctly but runs several points above the full 200-case benchmark.
 - Sequence packing: enabled, with neat packing (no cross-sample attention)
 - LR schedule: cosine, warmup ratio 0.1
 
+**Evaluator models**
+
+- Objective: full-parameter SFT, **vision tower frozen** (LM + multimodal projector tuned)
+- Dataset: `DesignCoder-evaluate`, 37,847 samples, one screenshot per sample
+- Chat template: `qwen3_5` with thinking enabled; context length 32,768
+- `image_max_pixels`: 1,048,576 — inputs must be downscaled the same way at inference
+- Precision / parallel: bf16 + DeepSpeed ZeRO-3; 2 epochs, global batch 256
+- LR schedule: cosine, warmup ratio 0.1
+
 ## Usage
 
+### Generation models
+
 ```python
 from transformers import AutoModelForCausalLM, AutoProcessor
 
@@ -160,7 +285,7 @@ To download a single checkpoint only:
 hf download xingxm/DesignCoder --include "designcoder_qwen3.8_27b_adamw_bs128_data41287_step400/*" --local-dir ./DesignCoder
 ```
 
-### Inference contract
+#### Inference contract
 
 These models are trained as tool-using agents, not single-turn generators. A case runs
 `design_search` → (`websearch`, landing only) → a final answer containing exactly three code
@@ -169,6 +294,61 @@ format from `examples/designcoder/runtime/infer_designcoder.py`; prompting with
 instruction and no tool turns does not match the training distribution and will score far
 below the numbers above.
 
+### Evaluator models
+
+Note the different loader class (`AutoModelForImageTextToText`) — `AutoModelForCausalLM`
+resolves to a text-only shell and `generate()` will reject the image tensors.
+
+```python
+from transformers import AutoModelForImageTextToText, AutoProcessor, AutoTokenizer
+from PIL import Image
+import torch, math
+
+repo = "xingxm/DesignCoder"
+subfolder = "designcoder_evaluator_qwen3.8_27b_adamw_bs256_data37847_step296"
+
+tok = AutoTokenizer.from_pretrained(repo, subfolder=subfolder, trust_remote_code=True)
+proc = AutoProcessor.from_pretrained(repo, subfolder=subfolder, trust_remote_code=True)
+model = AutoModelForImageTextToText.from_pretrained(
+    repo, subfolder=subfolder, dtype=torch.bfloat16, device_map="cuda", trust_remote_code=True
+)
+
+img = Image.open("screenshot.png").convert("RGB")
+w, h = img.size                                   # match training preprocessing
+if w * h > 1048576:
+    s = math.sqrt(1048576 / (w * h))
+    img = img.resize((int(w * s), int(h * s)))
+
+user = (
+    "<image>\n"
+    "Evaluate the attached UI screenshot using the selected visual criteria.\n\n"
+    "<surface>\nlanding\n</surface>\n\n"
+    "<generation_brief>\n...brief...\n</generation_brief>\n\n"
+    '<frozen_rubric>\n{"rubric":{"Alignment":["..."],"Layout":["..."]}}\n</frozen_rubric>'
+)
+text = tok.apply_chat_template(
+    [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": user}],
+    tokenize=False, add_generation_prompt=True,
+).replace("<image>", "<|vision_start|><|image_pad|><|vision_end|>")
+
+inputs = proc(text=[text], images=[img], return_tensors="pt").to("cuda")
+out = model.generate(**inputs, max_new_tokens=4096, do_sample=False)
+print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
+```
+
+#### Evaluator inference contract
+
+- `SYSTEM_PROMPT` must be the one shipped with the `DesignCoder-evaluate` dataset — it
+  enumerates the rubric namespace the model was trained against.
+- The chat template already ends with `<think>\n`, so generated text continues *inside* the
+  reasoning block and carries **no opening `<think>` tag**. Split on `</think>` to reach the
+  JSON payload.
+- Downscale images to ≤ 1,048,576 px as shown; larger inputs drift from training.
+- Parse `frozen_dynamic_scores.verdicts`. **Ignore the sibling `summary` string** — it
+  miscounts (e.g. reads `"24/24"` next to 25 emitted verdicts) in both models.
+- `static_scores` / `dynamic_scores` (40 items, 0/1/2) have **no held-out validation** — they
+  were only checked against contaminated training samples. Treat them as unverified.
+
 ## Provenance
 
 Each subfolder additionally ships `trainer_state.json` / `trainer_log.jsonl` (and