Add designcoder_evaluator_qwen3.5_9b_adamw_bs256_data37847_step100/README.md

designcoder_evaluator_qwen3.5_9b_adamw_bs256_data37847_step100/README.md

@@ -0,0 +1,182 @@
+---
+license: other
+base_model: Qwen/Qwen3.5-9B
+tags:
+- designcoder
+- evaluator
+- reward-model
+- ui-evaluation
+- multimodal
+- image-text-to-text
+language:
+- en
+pipeline_tag: image-text-to-text
+---
+
+# DesignCoder Evaluator · Qwen3.5-9B · AdamW · bs256 · data37847 · step100
+
+> **This is an evaluator, not a generation model.**
+> Most checkpoints in this repository *write* HTML/CSS/JS. This one *scores* the
+> rendered result: given a UI screenshot and a rubric it returns structured
+> per-criterion verdicts, for use as a reward / judge model during rollout.
+
+The lightweight companion to
+[`designcoder_evaluator_qwen3.8_27b_adamw_bs256_data37847_step296`](../designcoder_evaluator_qwen3.8_27b_adamw_bs256_data37847_step296).
+Roughly **1/3 the memory and 2× the speed** of the 27B, with equal or better
+agreement — **provided the rubric has exactly 10 items** (see Limitations).
+
+## Task
+
+Input: one full-page UI screenshot + `<surface>` (landing | dashboard) +
+`<generation_brief>` + `<frozen_rubric>`.
+Output: `<think>` reasoning followed by strict JSON with `dynamic_scores` (5 × 0/1/2),
+`static_scores` (34 dashboard / 35 landing), `interaction_rubrics` (free text) and
+`frozen_dynamic_scores.verdicts` (binary per rubric item).
+
+## Training
+
+| | |
+|---|---|
+| Base model | Qwen3.5-9B (`Qwen3_5ForConditionalGeneration`) |
+| Dataset | **DesignCoder-evaluate**, 37,847 filtered samples |
+| Optimizer | AdamW, lr 1e-5, cosine, warmup 0.1, wd 0.0 |
+| Precision / parallel | bf16 + DeepSpeed ZeRO-3, 8 nodes × 8 H20 = 64 GPUs |
+| Global batch | 256 (per_device 2 × grad_accum 2 × 64) |
+| **Exported step** | **100 / 296** (0.68 epoch), loss 0.4956 |
+| Full-run final loss | 0.4358 @ step 296 |
+| Frozen | vision tower (LM + projector fully tuned) |
+
+> ⚠️ `data37847` is the **DesignCoder-evaluate** dataset (UI screenshot scoring).
+> It is **not** the generation dataset behind the `data37865` / `data41287`
+> checkpoints — the similar sample count is a coincidence.
+
+### Why step 100 and not the final checkpoint
+
+All six checkpoints (50/100/150/200/250/296) were evaluated on a purpose-built
+**out-of-distribution suite** (576 prompts: rubric lengths 5/10/15/20/25 and three
+dimension-grouping shapes). Step 100 wins on OOD agreement, and the final checkpoint
+is measurably worse:
+
+| Step | OOD κ | In-dist κ | Overfit gap | Train loss |
+|---:|---:|---:|---:|---:|
+| 50 | 0.6568 | 0.6610 | +0.004 | – |
+| **100** | **0.6943** | 0.7010 | +0.007 | 0.4956 |
+| 150 | 0.6730 | 0.6908 | +0.018 | – |
+| 200 | 0.6649 | 0.6758 | +0.011 | – |
+| 250 | 0.6765 | 0.6571 | −0.019 | – |
+| 296 (final) | 0.6501 | 0.6772 | +0.027 | 0.4358 |
+
+This mirrors the pattern documented for the generation runs in this repo — loss keeps
+falling while held-out quality degrades — except the optimum arrives **earlier**
+(~34% of training vs ~75% for generation).
+
+### Optimizer ablation
+
+A sibling 9B run using **Muon + `pure_bf16`** (otherwise identical data, batch, and
+steps) failed badly, which is why this run uses AdamW:
+
+| | Muon + pure_bf16 | **AdamW + ZeRO-3** |
+|---|---:|---:|
+| Final train loss | 0.7505 | **0.4358** |
+| Truncated generations | 14 / 389 | **0 / 389** |
+| Cohen's κ (25-item rubric) | 0.4831 | **0.6353** |
+| Reproduces known quality ordering | ❌ inverted | ✅ correct |
+
+The Muon arm stalled early (loss 1.376 → 1.138 over the first 50 steps) and degenerated
+into repetition loops that ran into the 8192-token cap. Do not use Muon + `pure_bf16`
+for this task.
+
+## Evaluation
+
+The dataset has no test split, so evaluation uses **held-out inputs**: 239 generated-page
+screenshots already scored check-by-check by a stronger external vision judge
+(md5 check: **0 overlap** with the 37,851 training images).
+
+| Metric | 25-item rubric (OOD) | 10-item rubric (in-dist) |
+|---|---:|---:|
+| JSON parse rate | 1.0000 | 1.0000 |
+| Verdict-count exact match | 0.6276 | **1.0000** |
+| Per-check agreement | 0.9226 | 0.9343 |
+| Cohen's κ | 0.6353 | 0.6560 |
+| Defect recall | 0.6581 | 0.6730 |
+| Defect precision | 0.7018 | 0.7137 |
+| Case-level Pearson | 0.8417 | 0.7809 |
+
+> Numbers above are for **step 296** (measured before checkpoint selection).
+> Step 100 — the weights published here — scores **higher** on the OOD suite
+> (κ 0.6943 vs 0.6501); the table is retained as a conservative reference.
+
+### Reproducing the known quality ordering
+
+External judge on three generation checkpoints: 4B `0.833` < 9B `0.884` < 27B `0.905`
+
+| Scorer | 4B | 9B | 27B | Ordering |
+|---|---:|---:|---:|---|
+| This model (9B AdamW) | 0.855 | **0.886** | 0.910 | ✅ correct |
+| 27B evaluator | 0.840 | 0.893 | 0.909 | ✅ correct |
+| 9B Muon evaluator | 0.896 | 0.940 | 0.933 | ❌ inverted |
+
+## Usage
+
+```python
+from transformers import AutoModelForImageTextToText, AutoProcessor, AutoTokenizer
+from PIL import Image
+import torch, math
+
+MODEL = "xingxm/DesignCoder"
+SUB = "designcoder_evaluator_qwen3.5_9b_adamw_bs256_data37847_step100"
+
+tok = AutoTokenizer.from_pretrained(MODEL, subfolder=SUB, trust_remote_code=True)
+proc = AutoProcessor.from_pretrained(MODEL, subfolder=SUB, trust_remote_code=True)
+model = AutoModelForImageTextToText.from_pretrained(
+    MODEL, subfolder=SUB, dtype=torch.bfloat16, device_map="cuda", trust_remote_code=True
+)
+
+img = Image.open("screenshot.png").convert("RGB")
+w, h = img.size                                    # match training preprocessing
+if w * h > 1048576:
+    s = math.sqrt(1048576 / (w * h))
+    img = img.resize((int(w * s), int(h * s)))
+
+user = (
+    "<image>\n"
+    "Evaluate the attached UI screenshot using the selected visual criteria.\n\n"
+    "<surface>\nlanding\n</surface>\n\n"
+    "<generation_brief>\n...brief...\n</generation_brief>\n\n"
+    '<frozen_rubric>\n{"rubric":{"Alignment":["..."],"Layout":["..."]}}\n</frozen_rubric>'
+)
+text = tok.apply_chat_template(
+    [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": user}],
+    tokenize=False, add_generation_prompt=True,
+).replace("<image>", "<|vision_start|><|image_pad|><|vision_end|>")
+
+inputs = proc(text=[text], images=[img], return_tensors="pt").to("cuda")
+out = model.generate(**inputs, max_new_tokens=4096, do_sample=False)
+print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
+```
+
+`SYSTEM_PROMPT` must be the one shipped with the DesignCoder-evaluate dataset. The chat
+template already ends with `<think>\n`, so generated text continues *inside* the reasoning
+block and carries no opening `<think>` tag.
+
+## Limitations
+
+- **Feed it exactly 10 rubric items.** Every training rubric has exactly 10 (fixed shape
+  `Alignment:2, Layout:2, Typography:2, Components:2, Assets:1, Aesthetics:1`, 100% of
+  37,847 records). Off-length rubrics degrade verdict counting badly — the dimension names
+  stay correct but the per-dimension item count drifts, typically by ±1:
+
+  | Rubric length | 5 | 10 | 15 | 20 | 25 |
+  |---|---:|---:|---:|---:|---:|
+  | Count-match (this model) | 0.00 | **1.00** | 0.92 | 0.94 | 0.85 |
+  | Count-match (27B) | 1.00 | 1.00 | 0.99 | 1.00 | 1.00 |
+
+  **If your rubric length varies, use the 27B instead** — that is its main advantage,
+  not raw judgement quality (OOD κ is actually 0.694 here vs 0.664 for the 27B).
+- **`summary` miscounts.** `frozen_dynamic_scores.summary` is rarely self-consistent
+  (e.g. `"24/24"` alongside 25 verdicts). Parse the `verdicts` array; ignore `summary`.
+- **`static_scores` / `dynamic_scores` unvalidated on unseen data.** No ground truth exists
+  for those 40 items outside the training distribution; they were only checked on a
+  contaminated training sample (static exact-match 0.9135, dynamic 0.8400).
+- Evidence is screenshot-only; the model cannot verify interaction behaviour, yet
+  `interaction_rubrics` asks it to describe interactions — treat that field as speculative.