Update model card: add 4 data41287 checkpoints, fix dataset revision, add bench-200 scores

README.md

@@ -30,22 +30,69 @@ designcoder_{basemodel}_{size}_{optimizer}_bs{global_batch}[_{extra_axes}]_step{
 - `optimizer`: `muon` or `adamw`
 - `bs`: global batch size (`per_device × grad_accum × world_size`)
 - `extra_axes`: any hyper-parameter that deviates from the default recipe, e.g. `wd0.05`
-  (weight decay, default 0.0) or `ep20` (epochs, default 2)
+  (weight decay, default 0.0), `ep20` (epochs, default 2), or `data41287` (dataset revision)
 - `step`: trainer `global_step` of the exported weights
 
+## Dataset revisions
+
+Checkpoints in this repository come from two different dataset revisions. **Scores and loss
+values are only comparable within the same revision.**
+
+| Tag | Samples | Used by |
+|---|---:|---|
+| *(untagged)* `data37865` | 37,865 | `*_step1900`, `*_step3800` |
+| `data41287` | 41,287 | `*_data41287_step200`, `*_data41287_step400` |
+
 ## Checkpoints
 
-| Subfolder | Base model | Optimizer | LR | Global batch | Epochs | Weight decay | Step | Notes |
+| Subfolder | Base model | Optimizer | LR | Global batch | Dataset | Step | bench-200 | Notes |
 |---|---|---|---|---|---|---|---|---|
-| `designcoder_qwen3.5_4b_muon_bs32_step1900` | Qwen3.5-4B | Muon | 1e-5 | 32 | 2 | 0.0 | 1900 | smallest release |
-| `designcoder_qwen3.5_9b_muon_bs16_step3800` | Qwen3.5-9B | Muon | 1e-5 | 16 | 2 | 0.0 | 3800 | optimizer ablation (Muon arm) |
-| `designcoder_qwen3.5_9b_adamw_bs16_step3800` | Qwen3.5-9B | AdamW | 2e-5 | 16 | 2 | 0.0 | 3800 | optimizer ablation (AdamW arm) |
-| `designcoder_qwen3.6_27b_adamw_bs32_step1900` | Qwen3.6-27B | AdamW | 1e-5 | 32 | 2 | 0.0 | 1900 | largest release |
+| `designcoder_qwen3.5_4b_muon_bs32_step1900` | Qwen3.5-4B | Muon | 1e-5 | 32 | 37,865 | 1900 | – | smallest of the first release |
+| `designcoder_qwen3.5_9b_muon_bs16_step3800` | Qwen3.5-9B | Muon | 1e-5 | 16 | 37,865 | 3800 | – | optimizer ablation (Muon arm) |
+| `designcoder_qwen3.5_9b_adamw_bs16_step3800` | Qwen3.5-9B | AdamW | 2e-5 | 16 | 37,865 | 3800 | – | optimizer ablation (AdamW arm) |
+| `designcoder_qwen3.6_27b_adamw_bs32_step1900` | Qwen3.6-27B | AdamW | 1e-5 | 32 | 37,865 | 1900 | – | largest of the first release |
+| `designcoder_qwen3.5_4b_adamw_bs256_data41287_step200` | Qwen3.5-4B | AdamW | 2e-5 | 256 | 41,287 | 200 | 84.22 | best 4B / AdamW |
+| `designcoder_qwen3.5_4b_muon_bs256_data41287_step200` | Qwen3.5-4B | Muon | 2e-5 | 256 | 41,287 | 200 | 83.36 | best 4B / Muon; degrades less late in training |
+| `designcoder_qwen3.5_9b_adamw_bs256_data41287_step200` | Qwen3.5-9B | AdamW | 2e-5 | 256 | 41,287 | 200 | **84.40** | only checkpoint scored on all 200 cases |
+| `designcoder_qwen3.8_27b_adamw_bs128_data41287_step400` | Qwen3.8-27B | AdamW | 1e-5 | 128 | 41,287 | 400 | **91.19** | strongest checkpoint in the collection |
+
+## Benchmark
+
+`bench-200` is the frozen 200-case DesignCoder benchmark (100 Track A landing, 40 Track A
+dashboard, 30 Track B landing, 30 Track B dashboard). Every rubric item is a binary
+screenshot check scored by a vision judge over full-page renders; the reported number is the
+unweighted mean of Prompt Fit and the six rubric dimensions (Alignment, Layout, Typography,
+Components, Assets, Aesthetics).
+
+The 9B score is a full 200-case run. The 4B and 27B scores come from an 8-case subset
+reweighted to the benchmark's real landing/dashboard split, so they are indicative rather
+than final — and the subset was sampled around the 9B mid-range, which understates 9B
+relative to 4B. Use the 9B full-run number (84.40) when comparing against 27B (91.19).
+
+### Checkpoint selection
+
+The `data41287` checkpoints were selected by **running the benchmark**, not by taking the
+lowest training loss. In all four runs the best checkpoint sits at roughly 75% of training,
+and loss kept improving while benchmark scores fell:
+
+| Run | Step | Train loss | bench-200 |
+|---|---:|---:|---:|
+| 4B AdamW | 200 | 0.2696 | **84.22** |
+| 4B AdamW | 266 | 0.2682 | 68.35 |
+| 4B Muon | 200 | 0.3339 | **83.36** |
+| 4B Muon | 266 | 0.3349 | 81.27 |
+| 9B AdamW | 200 | 0.2518 | **84.40** |
+| 9B AdamW | 266 | 0.2504 | lowest of the three |
+| 27B AdamW | 400 | 0.2067 | **91.19** |
+| 27B AdamW | 530 | 0.2059 | 86.37 |
+
+The 4B AdamW pair is the clearest example: loss improved from 0.2696 to 0.2682 while the
+score collapsed from 84.22 to 68.35. **Do not pick checkpoints from this family by loss.**
 
 ## Shared training setup
 
 - Objective: full-parameter supervised fine-tuning (no LoRA / adapters)
-- Dataset: `designcoder_sft_v2_train`, 41,287 ShareGPT-format records
+- Dataset: `designcoder_sft_v2_train` in ShareGPT format (see revision table above)
 - Chat template: `qwen3_5` with thinking enabled
 - Context length: 32,768
 - Sequence packing: enabled, with neat packing (no cross-sample attention)
@@ -57,7 +104,7 @@ designcoder_{basemodel}_{size}_{optimizer}_bs{global_batch}[_{extra_axes}]_step{
 from transformers import AutoModelForCausalLM, AutoProcessor
 
 repo = "xingxm/DesignCoder"
-subfolder = "designcoder_qwen3.5_4b_muon_bs32_step1900"
+subfolder = "designcoder_qwen3.8_27b_adamw_bs128_data41287_step400"
 
 model = AutoModelForCausalLM.from_pretrained(repo, subfolder=subfolder, dtype="auto", device_map="auto")
 processor = AutoProcessor.from_pretrained(repo, subfolder=subfolder)
@@ -66,9 +113,18 @@ processor = AutoProcessor.from_pretrained(repo, subfolder=subfolder)
 To download a single checkpoint only:
 
 ```bash
-hf download xingxm/DesignCoder --include "designcoder_qwen3.5_4b_muon_bs32_step1900/*" --local-dir ./DesignCoder
+hf download xingxm/DesignCoder --include "designcoder_qwen3.8_27b_adamw_bs128_data41287_step400/*" --local-dir ./DesignCoder
 ```
 
+### Inference contract
+
+These models are trained as tool-using agents, not single-turn generators. A case runs
+`design_search` → (`websearch`, landing only) → a final answer containing exactly three code
+blocks in the order `html`, `css`, `js`. Reproduce the system prompts and tool observation
+format from `examples/designcoder/runtime/infer_designcoder.py`; prompting with a bare
+instruction and no tool turns does not match the training distribution and will score far
+below the numbers above.
+
 ## Provenance
 
 Each subfolder additionally ships `trainer_state.json` / `trainer_log.jsonl` (and