Add designcoder_evaluator_qwen3.8_27b_adamw_bs256_data37847_step296/README.md

designcoder_evaluator_qwen3.8_27b_adamw_bs256_data37847_step296/README.md

@@ -0,0 +1,171 @@
+---
+license: other
+base_model: Qwen/Qwen3.8-27B
+tags:
+- designcoder
+- evaluator
+- reward-model
+- ui-evaluation
+- multimodal
+- image-text-to-text
+language:
+- en
+pipeline_tag: image-text-to-text
+---
+
+# DesignCoder Evaluator · Qwen3.8-27B · AdamW · bs256 · data37847 · step296
+
+> **This is an evaluator, not a generation model.**
+> Every other checkpoint in this repository *writes* HTML/CSS/JS. This one *scores*
+> the rendered result. It takes a UI screenshot plus a rubric and returns structured
+> per-criterion scores — it is intended as a reward / judge model for rollout scoring,
+> not for page generation.
+
+## Task
+
+Input: one full-page UI screenshot + `<surface>` (landing | dashboard) +
+`<generation_brief>` + `<frozen_rubric>`.
+Output: `<think>` reasoning followed by a strict JSON payload with
+
+| Field | Content |
+|---|---|
+| `dynamic_scores` | 5 prompt-fit criteria, scored 0/1/2 |
+| `static_scores` | 27~35 integrity + detail rubrics, 0/2 (integrity) or 0/1/2 (detail), `"N/A"` where genuinely inapplicable |
+| `interaction_rubrics` | free-text interaction test cases |
+| `frozen_dynamic_scores.verdicts` | binary 0/1 verdict per supplied rubric item |
+
+## Training
+
+| | |
+|---|---|
+| Base model | Qwen3.8-27B (`Qwen3_5ForConditionalGeneration`) |
+| Dataset | **DesignCoder-evaluate**, 37,847 filtered samples |
+| Optimizer | AdamW, lr 5e-6, cosine, warmup 0.1, wd 0.0 |
+| Precision / parallel | bf16 + DeepSpeed ZeRO-3, 8 nodes × 8 H20 = 64 GPUs |
+| Global batch | 256 (per_device 1 × grad_accum 4 × 64) |
+| Epochs / steps | 2.0 / 296 |
+| `cutoff_len` / `image_max_pixels` | 32768 / 1048576 |
+| Frozen | vision tower (LM + projector fully tuned) |
+| **train_loss** | **0.4939** |
+
+> ⚠️ `data37847` refers to the **DesignCoder-evaluate** dataset (UI screenshot scoring).
+> It is **not** the generation dataset used by the `data37865` / `data41287` checkpoints
+> in this repo — the similar sample count is a coincidence.
+
+### Checkpoint selection
+
+Both saved checkpoints were evaluated on a purpose-built **out-of-distribution suite**
+(576 prompts: rubric lengths 5/10/15/20/25 plus three dimension-grouping shapes).
+Unlike the generation runs in this repo — and unlike the 9B evaluator, which peaks at
+34% of training — the 27B shows **no late-training degradation**:
+
+| Step | OOD κ | In-dist κ | Overfit gap | OOD count-match |
+|---:|---:|---:|---:|---:|
+| 200 | 0.6625 | 0.6643 | +0.002 | 0.9977 |
+| **296 (this one)** | **0.6640** | 0.6353 | **−0.029** | **1.0000** |
+
+The two are statistically indistinguishable on κ (Δ 0.0015, well inside variant-to-variant
+noise), but step 296 has the better overfit gap and perfect verdict counting, so the final
+checkpoint is published.
+
+## Evaluation
+
+The dataset has no test split, so evaluation uses **held-out inputs**: 239 generated-page
+screenshots from the DesignCoder benchmark, each already scored check-by-check by a
+stronger external vision judge. Byte-level md5 check confirmed **0 overlap** between these
+710 benchmark screenshots and the 37,851 training images.
+
+Because the training rubric has a **fixed shape** (10 items:
+`Alignment:2, Layout:2, Typography:2, Components:2, Assets:1, Aesthetics:1`, 100% of
+3,000 sampled records) while benchmark rubrics carry 24~25 items, two variants were run:
+**Set B** = original benchmark rubric (out-of-distribution length),
+**Set C** = same screenshots, rubric down-sampled to the training shape.
+
+| Metric | Set B | Set C |
+|---|---|---|
+| JSON parse rate | **1.0000** | **1.0000** |
+| Verdict-count exact match | **1.0000** | **1.0000** |
+| Per-check agreement | 0.9313 | 0.9347 |
+| Cohen's κ | **0.6802** | 0.6645 |
+| Defect recall (GT=0) | 0.6996 | 0.6958 |
+| Defect precision | 0.7401 | 0.7066 |
+| Predicted vs true defect rate | 0.119 / 0.126 | 0.108 / 0.110 |
+| Case-level Pearson | **0.8572** | 0.7890 |
+
+Agreement alone is misleading (87% of checks are `1`, so always-pass scores ~88%);
+κ and defect recall are the meaningful signals.
+
+### Reproducing the known quality ordering
+
+Three generation checkpoints, ranked by the external judge — this model reproduces the
+ordering with **<1pp absolute error**, which is what matters for reward use:
+
+| Scorer | 4B-Muon | 9B-AdamW | 27B-v1 | Ordering |
+|---|---|---|---|---|
+| External judge | 0.833 | 0.884 | 0.905 | reference |
+| **This model** | 0.840 | 0.893 | 0.909 | ✅ correct |
+| 9B evaluator (Muon) | 0.896 | 0.940 | 0.933 | ❌ inverted |
+
+## Usage
+
+```python
+from transformers import AutoModelForImageTextToText, AutoProcessor, AutoTokenizer
+from PIL import Image
+import torch, math
+
+MODEL = "xingxm/DesignCoder"
+SUB = "designcoder_evaluator_qwen3.8_27b_adamw_bs256_data37847_step296"
+
+tok = AutoTokenizer.from_pretrained(MODEL, subfolder=SUB, trust_remote_code=True)
+proc = AutoProcessor.from_pretrained(MODEL, subfolder=SUB, trust_remote_code=True)
+model = AutoModelForImageTextToText.from_pretrained(
+    MODEL, subfolder=SUB, dtype=torch.bfloat16, device_map="cuda", trust_remote_code=True
+)
+
+img = Image.open("screenshot.png").convert("RGB")
+w, h = img.size                                    # match training preprocessing
+if w * h > 1048576:
+    s = math.sqrt(1048576 / (w * h))
+    img = img.resize((int(w * s), int(h * s)))
+
+user = (
+    "<image>\n"
+    "Evaluate the attached UI screenshot using the selected visual criteria.\n\n"
+    "<surface>\nlanding\n</surface>\n\n"
+    "<generation_brief>\n...brief...\n</generation_brief>\n\n"
+    '<frozen_rubric>\n{"rubric":{"Alignment":["..."],"Layout":["..."]}}\n</frozen_rubric>'
+)
+text = tok.apply_chat_template(
+    [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": user}],
+    tokenize=False, add_generation_prompt=True,
+).replace("<image>", "<|vision_start|><|image_pad|><|vision_end|>")
+
+inputs = proc(text=[text], images=[img], return_tensors="pt").to("cuda")
+out = model.generate(**inputs, max_new_tokens=4096, do_sample=False)
+print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
+```
+
+`SYSTEM_PROMPT` must be the one shipped with the DesignCoder-evaluate dataset — the model
+relies on it to enumerate the rubric namespace. The chat template already ends with
+`<think>\n`, so generated text continues *inside* the reasoning block and contains no
+opening `<think>` tag.
+
+## Limitations
+
+- **`summary` field miscounts.** `frozen_dynamic_scores.summary` is self-consistent in only
+  2.1% of cases (e.g. writes `"24/24"` when 25 verdicts were emitted). Parse the
+  `verdicts` array; ignore `summary`.
+- **`static_scores` / `dynamic_scores` are unvalidated on unseen data.** No ground truth
+  exists for these 40 items outside the training distribution; they were only checked on a
+  contaminated training sample (static exact-match 0.9196, dynamic 0.8376).
+- **Trained on a single rubric shape.** All training rubrics have exactly 10 items. The
+  model handles 25-item rubrics correctly, but longer or structurally different rubrics are
+  untested.
+- **Judgement quality is not where this model beats the 9B.** On the OOD suite the
+  [9B evaluator](../designcoder_evaluator_qwen3.5_9b_adamw_bs256_data37847_step100)
+  actually scores slightly higher on κ (0.694 vs 0.664). The 27B's real advantage is
+  **format robustness**: verdict-count match stays at 1.00 across rubric lengths 5→25,
+  where the 9B drops to 0.00 at length 5 and ~0.85 at length 25. Use this model when
+  rubric length varies; the 9B is the cheaper choice for fixed 10-item rubrics.
+- Evidence is screenshot-only; the model cannot verify interaction behaviour, yet
+  `interaction_rubrics` asks it to describe interactions — treat that field as speculative.