Update to full V5.9.2 scores (all three rubric families, n=200)

README.md

@@ -11,8 +11,6 @@ tags:
 - javascript
 - code-generation
 - full-sft
-- evaluator
-- reward-model
 ---
 
 # DesignCoder
@@ -20,25 +18,14 @@ tags:
 Checkpoint collection for **DesignCoder**, a family of full-parameter SFT models for UI design
 research and end-to-end HTML/CSS/JavaScript implementation.
 
-The repository holds **two kinds of model**:
-
-| Kind | What it does | Folder prefix |
-|---|---|---|
-| **Generation** | writes HTML/CSS/JS from a design brief | `designcoder_{basemodel}_...` |
-| **Evaluator** | scores a rendered UI screenshot against a rubric | `designcoder_evaluator_{basemodel}_...` |
-
-They are trained on **different datasets** and answer **different inputs** — an evaluator will
-not generate pages, and a generation model will not produce valid rubric verdicts.
-
 Each subfolder in this repository is a self-contained, directly loadable checkpoint.
 
 ## Naming convention
 
 ```
-designcoder_[evaluator_]{basemodel}_{size}_{optimizer}_bs{global_batch}[_{extra_axes}]_step{global_step}
+designcoder_{basemodel}_{size}_{optimizer}_bs{global_batch}[_{extra_axes}]_step{global_step}
 ```
 
-- `evaluator`: present only for rubric-scoring models; absent means generation
 - `basemodel` / `size`: base model family and parameter scale
 - `optimizer`: `muon` or `adamw`
 - `bs`: global batch size (`per_device × grad_accum × world_size`)
@@ -48,58 +35,30 @@ designcoder_[evaluator_]{basemodel}_{size}_{optimizer}_bs{global_batch}[_{extra_
 
 ## Dataset revisions
 
-Checkpoints in this repository come from three different dataset revisions. **Scores and loss
+Checkpoints in this repository come from two different dataset revisions. **Scores and loss
 values are only comparable within the same revision.**
 
-| Tag | Samples | Dataset | Used by |
-|---|---:|---|---|
-| *(untagged)* `data37865` | 37,865 | `designcoder_sft_v2` (generation) | `*_step1900`, `*_step3800` |
-| `data41287` | 41,287 | `designcoder_sft_v2` (generation) | `*_data41287_step200`, `*_data41287_step400` |
-| `data37847` | 37,847 | **`DesignCoder-evaluate`** (screenshot scoring) | `designcoder_evaluator_*` |
-
-> ⚠️ `data37847` and `data37865` are **entirely different datasets** — the near-identical
-> sample counts are a coincidence. Never compare loss across them.
+| Tag | Samples | Used by |
+|---|---:|---|
+| *(untagged)* `data37865` | 37,865 | `*_step1900`, `*_step3800` |
+| `data41287` | 41,287 | `*_data41287_step200`, `*_data41287_step400` |
 
 ## Checkpoints
 
-### Generation models
-
-| Subfolder | Base model | Optimizer | LR | Global batch | Dataset | Step | bench-200 (full, n=200) | Notes |
+| Subfolder | Base model | Optimizer | LR | Global batch | Dataset | Step | bench-200 V5.9.2 (n=200) | Notes |
 |---|---|---|---|---|---|---|---|---|
 | `designcoder_qwen3.5_4b_muon_bs32_step1900` | Qwen3.5-4B | Muon | 1e-5 | 32 | 37,865 | 1900 | – | smallest of the first release |
 | `designcoder_qwen3.5_9b_muon_bs16_step3800` | Qwen3.5-9B | Muon | 1e-5 | 16 | 37,865 | 3800 | – | optimizer ablation (Muon arm) |
 | `designcoder_qwen3.5_9b_adamw_bs16_step3800` | Qwen3.5-9B | AdamW | 2e-5 | 16 | 37,865 | 3800 | – | optimizer ablation (AdamW arm) |
 | `designcoder_qwen3.6_27b_adamw_bs32_step1900` | Qwen3.6-27B | AdamW | 1e-5 | 32 | 37,865 | 1900 | – | largest of the first release |
-| `designcoder_qwen3.5_4b_adamw_bs256_data41287_step200` | Qwen3.5-4B | AdamW | 2e-5 | 256 | 41,287 | 200 | 82.12 | best 4B / AdamW |
-| `designcoder_qwen3.5_4b_muon_bs256_data41287_step200` | Qwen3.5-4B | Muon | 2e-5 | 256 | 41,287 | 200 | 77.62 | best 4B / Muon |
-| `designcoder_qwen3.5_9b_adamw_bs256_data41287_step200` | Qwen3.5-9B | AdamW | 2e-5 | 256 | 41,287 | 200 | **84.40** | best 9B |
-| `designcoder_qwen3.8_27b_adamw_bs128_data41287_step400` | Qwen3.8-27B | AdamW | 1e-5 | 128 | 41,287 | 400 | **87.89** | strongest checkpoint in the collection |
+| `designcoder_qwen3.5_4b_adamw_bs256_data41287_step200` | Qwen3.5-4B | AdamW | 2e-5 | 256 | 41,287 | 200 | 80.06 | best 4B / AdamW |
+| `designcoder_qwen3.5_4b_muon_bs256_data41287_step200` | Qwen3.5-4B | Muon | 2e-5 | 256 | 41,287 | 200 | 71.34 | best 4B / Muon |
+| `designcoder_qwen3.5_9b_adamw_bs256_data41287_step200` | Qwen3.5-9B | AdamW | 2e-5 | 256 | 41,287 | 200 | **82.14** | best 9B |
+| `designcoder_qwen3.8_27b_adamw_bs128_data41287_step400` | Qwen3.8-27B | AdamW | 1e-5 | 128 | 41,287 | 400 | **87.04** | strongest checkpoint in the collection |
 
 All four `data41287` scores are **final full-benchmark runs: 200/200 rollouts, 200/200
-screenshot captures, 200/200 judge evaluations** per model (no subsetting).
-
-### Evaluator models
-
-Rubric scorers for UI screenshots — intended as reward / judge models during rollout, **not**
-for page generation. Trained on `DesignCoder-evaluate` (37,847 samples), evaluated on
-held-out benchmark screenshots scored by a stronger external vision judge (κ = agreement
-with that judge; see [Evaluator benchmark](#evaluator-benchmark)).
-
-| Subfolder | Base model | Optimizer | LR | Global batch | Dataset | Step | OOD κ | Notes |
-|---|---|---|---|---|---|---|---:|---|
-| `designcoder_evaluator_qwen3.5_9b_adamw_bs256_data37847_step100` | Qwen3.5-9B | AdamW | 1e-5 | 256 | 37,847 | 100 | **0.694** | 1/3 memory, 2× speed; **requires exactly 10-item rubrics** |
-| `designcoder_evaluator_qwen3.8_27b_adamw_bs256_data37847_step296` | Qwen3.8-27B | AdamW | 5e-6 | 256 | 37,847 | 296 | 0.664 | **robust to any rubric length** — default choice |
-
-**Which one to use.** Counter-intuitively the 9B has slightly *higher* agreement; the 27B's
-advantage is **format robustness**. Verdict-count exact match across rubric lengths:
-
-| Rubric items | 5 | 10 | 15 | 20 | 25 |
-|---|---:|---:|---:|---:|---:|
-| 9B step100 | 0.00 | **1.00** | 0.92 | 0.94 | 0.85 |
-| 27B step296 | **1.00** | **1.00** | **0.99** | **1.00** | **1.00** |
-
-Training rubrics are always exactly 10 items, so anything else is out-of-distribution.
-Use the 9B when your rubric is fixed at 10 items; use the 27B otherwise.
+screenshot captures, 200/200 judge evaluations** per model, scored with the **complete
+V5.9.2 rubric set (all three families)** — no subsetting, no omitted rubric family.
 
 ## Benchmark
 
@@ -107,48 +66,56 @@ Use the 9B when your rubric is fixed at 10 items; use the 27B otherwise.
 dashboard, 30 Track B landing, 30 Track B dashboard; Track A cases specify a style, Track B
 cases are style-free).
 
-**Rubric composition.** Every prompt ships with its own reference rubric of **23–25 binary
-screenshot checks** (184 prompts carry 25 checks, 15 carry 24, 1 carries 23 — 4,983 frozen
-checks in total), all evaluated with `check_with=screenshot` by a vision judge over the
-full-page render. Check distribution across dimensions:
+**Rubric composition.** Scoring uses three independent rubric families. Only the first varies
+per case; the other two are fixed for every case of a given surface.
+
+| Family | Scope | Size | Scale |
+|---|---|---|---|
+| **Frozen** | per case | 23–25 checks/case, 4,983 total (184 cases carry 25, 15 carry 24, 1 carries 23) | binary 0/1 |
+| **Prompt Fit & Product** | fixed per surface | 5 rubrics | 0/1/2 |
+| **Static** | fixed per surface | landing 27 `d_*` + 8 `q_*`; dashboard 25 `d_*` + 9 `q_*` | `d_*` 2/0/N-A, `q_*` 0/1/2/N-A |
 
-| Dimension | Checks | Share |
-|---|---:|---:|
-| Components | 1,517 | 30.4% |
-| Layout | 842 | 16.9% |
-| Aesthetics | 782 | 15.7% |
-| Typography | 642 | 12.9% |
-| Alignment | 616 | 12.4% |
-| Assets | 584 | 11.7% |
+Frozen checks are distributed across six dimensions: Components 1,517 (30.4%), Layout 842
+(16.9%), Aesthetics 782 (15.7%), Typography 642 (12.9%), Alignment 616 (12.4%), Assets 584
+(11.7%). Every check is `check_with=screenshot`.
 
-On top of the frozen checks, the judge scores 5 surface-specific **Prompt-Fit** items (0–2
-each) per case. The reported `overall_score` (0–100) is the unweighted mean of Prompt Fit and
-the six rubric dimensions. Judge: `gpt-5.6-sol` (vision) with structured JSON output.
+`overall_score` (0–100) is the unweighted mean of top-level slots: Prompt Fit (1 slot), each
+active Static dimension (1 slot each), and the whole Frozen family (1 slot). Aggregation is
+performed by the reference collector `DesignEvaluator-Skill/scripts/collect_unified.py`, not
+by a reimplementation. Judge: `gpt-5.6-sol` (vision) with structured JSON output.
 
 ### Full-run results (n=200 per model)
 
-| Model | Overall | Landing | Dashboard | Track A | Track B | Prompt Fit | Frozen pass rate | Render fails |
-|---|---:|---:|---:|---:|---:|---:|---:|---:|
-| 27B AdamW step400 | **87.89** | 88.55 | 86.67 | 86.90 | 90.20 | 84.10 | 88.6% | 0/200 |
-| 9B AdamW step200 | 84.40 | 86.31 | 80.85 | 84.10 | 85.09 | 78.35 | 85.4% | 1/200 |
-| 4B AdamW step200 | 82.12 | 85.07 | 76.65 | 82.39 | 81.50 | 75.25 | 83.2% | 1/200 |
-| 4B Muon step200 | 77.62 | 81.65 | 70.14 | 77.64 | 77.58 | 64.60 | 79.4% | 3/200 |
+| Model | **Overall (V5.9.2)** | Prompt Fit | Static | Frozen | Landing | Dashboard | Track A | Track B | Render fails |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| 27B AdamW step400 | **87.04** | 84.10 | 87.31 | 88.52 | 89.65 | 82.19 | 86.56 | 88.16 | 0/200 |
+| 9B AdamW step200 | **82.14** | 78.35 | 82.20 | 85.41 | 86.96 | 73.19 | 82.71 | 80.82 | 1/200 |
+| 4B AdamW step200 | **80.06** | 75.25 | 80.40 | 83.27 | 85.70 | 69.59 | 80.61 | 78.78 | 1/200 |
+| 4B Muon step200 | **71.34** | 64.60 | 70.96 | 79.79 | 78.48 | 58.06 | 72.05 | 69.68 | 3/200 |
+
+Scores are monotone in scale. Five of the six pairwise differences are significant (paired
+bootstrap 10k-resample 95% CI excludes 0 and Wilcoxon p < 4e-7); **9B vs 4B AdamW is not
+significant** (mean diff +2.08, CI [-0.03, 4.18], p = 0.065). See
+[`eval/significance_tests_v592.json`](./blob/main/eval/significance_tests_v592.json).
 
-Scores increase strictly monotonically with scale (all 6 pairwise differences significant,
-paired bootstrap 10k-resample 95% CI excludes 0 and Wilcoxon p < 0.013 — see
-[`eval/significance_tests.json`](./blob/main/eval/significance_tests.json)). The gap is far
-larger on dashboards (+16.5 from 4B Muon to 27B) than on landings (+6.9), and **Assets** is
-the weakest dimension for every scale (55–67% pass rate), indicating a data-level bottleneck
-rather than a capability ceiling.
+Two observations that only the full rubric set exposes:
+
+- **Dashboards are the bottleneck, and they degrade faster than landings.** The 27B loses 7.5
+  points moving from landing to dashboard; the 4B Muon loses 20.4. The Static family's
+  chart (`d_data_*`) and workflow (`d_work_*`) checks catch empty or non-functional charts
+  that the Frozen checks largely miss.
+- **The Frozen family alone compresses the ranking.** Across the four models Frozen spans only
+  8.7 points (88.52 → 79.79) while Prompt Fit spans 19.5 and Static spans 16.4. Reporting
+  Frozen-heavy scores therefore understates the gap between scales.
 
 ### Evaluation artifacts (`eval/`)
 
 | File | Content |
 |---|---|
-| [`eval/benchmark_summary.csv`](./blob/main/eval/benchmark_summary.csv) | per-model aggregates: overall, Track/Surface splits, six dimensions, Prompt Fit, frozen pass rate |
-| [`eval/benchmark_per_case.csv`](./blob/main/eval/benchmark_per_case.csv) | long-form per-case scores for all 4 models × 200 cases |
-| [`eval/significance_tests.json`](./blob/main/eval/significance_tests.json) | paired bootstrap (10k resamples) + Wilcoxon signed-rank for all 6 model pairs |
-| [`eval/rubric_stats.json`](./blob/main/eval/rubric_stats.json) | rubric composition statistics (checks per prompt, per dimension, per track) |
+| [`eval/benchmark_summary_v592.csv`](./blob/main/eval/benchmark_summary_v592.csv) | per-model aggregates: overall, three family scores, Track/Surface splits, Static and Frozen dimensions |
+| [`eval/benchmark_per_case_v592.csv`](./blob/main/eval/benchmark_per_case_v592.csv) | long-form per-case scores (overall + three families) for all 4 models × 200 cases |
+| [`eval/significance_tests_v592.json`](./blob/main/eval/significance_tests_v592.json) | paired bootstrap (10k resamples) + Wilcoxon signed-rank for all 6 model pairs |
+| [`eval/rubric_stats.json`](./blob/main/eval/rubric_stats.json) | composition of all three rubric families and the aggregation rule |
 | [`eval/reports.html`](./blob/main/eval/reports.html) | self-contained interactive HTML report: model comparison, dimension heatmap, score distributions, per-case tables |
 
 ### Checkpoint selection
@@ -159,96 +126,28 @@ and loss kept improving while benchmark scores fell. The table below shows the *
 selection subset** (used only to rank checkpoints, not comparable to the final full-run
 numbers in the tables above):
 
-| Run | Step | Train loss | subset bench (n=8, selection only) | final full bench (n=200) |
+| Run | Step | Train loss | subset bench (n=8, Frozen+Prompt-Fit only) | final V5.9.2 (n=200) |
 |---|---:|---:|---:|---:|
-| 4B AdamW | 200 | 0.2696 | 84.22 | **82.12** |
+| 4B AdamW | 200 | 0.2696 | 84.22 | **80.06** |
 | 4B AdamW | 266 | 0.2682 | 68.35 | – |
-| 4B Muon | 200 | 0.3339 | 83.36 | **77.62** |
+| 4B Muon | 200 | 0.3339 | 83.36 | **71.34** |
 | 4B Muon | 266 | 0.3349 | 81.27 | – |
-| 9B AdamW | 200 | 0.2518 | 84.40 | **84.40** |
+| 9B AdamW | 200 | 0.2518 | 84.40 | **82.14** |
 | 9B AdamW | 266 | 0.2504 | lowest of the three | – |
-| 27B AdamW | 400 | 0.2067 | 91.19 | **87.89** |
+| 27B AdamW | 400 | 0.2067 | 91.19 | **87.04** |
 | 27B AdamW | 530 | 0.2059 | 86.37 | – |
 
 The 4B AdamW pair is the clearest example: loss improved from 0.2696 to 0.2682 while the
 subset score collapsed from 84.22 to 68.35. **Do not pick checkpoints from this family by
-loss.** Note also that small subsets systematically overestimate: the subset ranks checkpoints
-correctly but runs several points above the full 200-case benchmark.
-
-## Evaluator benchmark
-
-`DesignCoder-evaluate` ships **no test split**, and both evaluators trained on all 37,847
-samples for 2 epochs — so there is no clean in-training validation set. Evaluation instead
-uses **held-out inputs**: 239 generated-page screenshots from the DesignCoder benchmark, each
-already scored check-by-check by a stronger external vision judge. A byte-level md5 check
-confirmed **0 overlap** between those benchmark screenshots and the 37,851 training images.
-
-Alignment is possible because the training target's `frozen_dynamic_scores.verdicts` and the
-external judge's `verdict` are both binary and share the same `(dimension, point)` structure.
-
-**Headline metrics** (239 held-out cases, original 24–25-item benchmark rubrics):
-
-| Metric | 9B step100 | 27B step296 |
-|---|---:|---:|
-| JSON parse rate | 1.0000 | **1.0000** |
-| Per-check agreement | 0.9226 | **0.9313** |
-| Cohen's κ vs external judge | 0.6353 | **0.6802** |
-| Defect recall (verdict = 0) | 0.6581 | **0.6996** |
-| Case-level Pearson | 0.8417 | **0.8572** |
+loss.**
 
-Raw agreement is a weak signal here — 87% of checks are `1`, so always-pass already scores
-~88%. **κ and defect recall are the meaningful numbers.**
-
-**Reproducing the known quality ordering.** The external judge ranks three generation
-checkpoints 4B `0.833` < 9B `0.884` < 27B `0.905`. Both published evaluators recover that
-ordering to within ~2pp:
-
-| Scorer | 4B | 9B | 27B | Ordering |
-|---|---:|---:|---:|---|
-| External judge (reference) | 0.833 | 0.884 | 0.905 | – |
-| Evaluator 9B step100 | 0.855 | 0.886 | 0.910 | ✅ |
-| Evaluator 27B step296 | 0.840 | 0.893 | 0.909 | ✅ |
-
-### Evaluator checkpoint selection
-
-Checkpoints were chosen on a purpose-built **out-of-distribution suite** (576 prompts:
-rubric lengths 5/10/15/20/25 × three dimension-grouping shapes), not on training loss.
-The two families behave differently:
-
-| Run | Step | Train loss | OOD κ | Overfit gap (in-dist − OOD) |
-|---|---:|---:|---:|---:|
-| 9B AdamW | 50 | – | 0.6568 | +0.004 |
-| 9B AdamW | **100** | 0.4956 | **0.6943** | +0.007 |
-| 9B AdamW | 150 | – | 0.6730 | +0.018 |
-| 9B AdamW | 200 | – | 0.6649 | +0.011 |
-| 9B AdamW | 250 | – | 0.6765 | −0.019 |
-| 9B AdamW | 296 (final) | 0.4358 | 0.6501 | +0.027 |
-| 27B AdamW | 200 | – | 0.6625 | +0.002 |
-| 27B AdamW | **296 (final)** | 0.4246 | **0.6640** | −0.029 |
-
-The 9B reproduces the "loss keeps falling, held-out quality degrades" pattern seen in the
-generation runs — but peaks **much earlier** (~34% of training vs ~75%). The 27B shows no
-late-training degradation, so its final checkpoint is published.
-
-### Evaluator optimizer ablation
-
-A 9B run with **Muon + `pure_bf16`** (identical data, batch, and steps) failed badly:
-
-| | Muon + pure_bf16 | AdamW + ZeRO-3 |
-|---|---:|---:|
-| Final train loss | 0.7505 | **0.4358** |
-| Truncated generations | 14 / 389 | **0 / 389** |
-| Cohen's κ | 0.4831 | **0.6353** |
-| Quality ordering | ❌ inverted | ✅ correct |
-
-The Muon arm stalled early (loss 1.376 → 1.138 over 50 steps) and degenerated into repetition
-loops hitting the 8192-token cap. It is not published. **Do not use Muon + `pure_bf16` for
-this task** — note this differs from the generation family, where Muon is viable.
+The two score columns are not comparable: the selection subset used 8 cases and only two of
+the three rubric families, and it overestimates by 4–12 points, with the largest error on the
+weakest model. It is reliable enough to rank checkpoints within a run, which is all it was
+used for — every number reported elsewhere in this card is the full 200-case V5.9.2 score.
 
 ## Shared training setup
 
-**Generation models**
-
 - Objective: full-parameter supervised fine-tuning (no LoRA / adapters)
 - Dataset: `designcoder_sft_v2_train` in ShareGPT format (see revision table above)
 - Chat template: `qwen3_5` with thinking enabled
@@ -256,19 +155,8 @@ this task** — note this differs from the generation family, where Muon is viab
 - Sequence packing: enabled, with neat packing (no cross-sample attention)
 - LR schedule: cosine, warmup ratio 0.1
 
-**Evaluator models**
-
-- Objective: full-parameter SFT, **vision tower frozen** (LM + multimodal projector tuned)
-- Dataset: `DesignCoder-evaluate`, 37,847 samples, one screenshot per sample
-- Chat template: `qwen3_5` with thinking enabled; context length 32,768
-- `image_max_pixels`: 1,048,576 — inputs must be downscaled the same way at inference
-- Precision / parallel: bf16 + DeepSpeed ZeRO-3; 2 epochs, global batch 256
-- LR schedule: cosine, warmup ratio 0.1
-
 ## Usage
 
-### Generation models
-
 ```python
 from transformers import AutoModelForCausalLM, AutoProcessor
 
@@ -285,7 +173,7 @@ To download a single checkpoint only:
 hf download xingxm/DesignCoder --include "designcoder_qwen3.8_27b_adamw_bs128_data41287_step400/*" --local-dir ./DesignCoder
 ```
 
-#### Inference contract
+### Inference contract
 
 These models are trained as tool-using agents, not single-turn generators. A case runs
 `design_search` → (`websearch`, landing only) → a final answer containing exactly three code
@@ -294,61 +182,6 @@ format from `examples/designcoder/runtime/infer_designcoder.py`; prompting with
 instruction and no tool turns does not match the training distribution and will score far
 below the numbers above.
 
-### Evaluator models
-
-Note the different loader class (`AutoModelForImageTextToText`) — `AutoModelForCausalLM`
-resolves to a text-only shell and `generate()` will reject the image tensors.
-
-```python
-from transformers import AutoModelForImageTextToText, AutoProcessor, AutoTokenizer
-from PIL import Image
-import torch, math
-
-repo = "xingxm/DesignCoder"
-subfolder = "designcoder_evaluator_qwen3.8_27b_adamw_bs256_data37847_step296"
-
-tok = AutoTokenizer.from_pretrained(repo, subfolder=subfolder, trust_remote_code=True)
-proc = AutoProcessor.from_pretrained(repo, subfolder=subfolder, trust_remote_code=True)
-model = AutoModelForImageTextToText.from_pretrained(
-    repo, subfolder=subfolder, dtype=torch.bfloat16, device_map="cuda", trust_remote_code=True
-)
-
-img = Image.open("screenshot.png").convert("RGB")
-w, h = img.size                                   # match training preprocessing
-if w * h > 1048576:
-    s = math.sqrt(1048576 / (w * h))
-    img = img.resize((int(w * s), int(h * s)))
-
-user = (
-    "<image>\n"
-    "Evaluate the attached UI screenshot using the selected visual criteria.\n\n"
-    "<surface>\nlanding\n</surface>\n\n"
-    "<generation_brief>\n...brief...\n</generation_brief>\n\n"
-    '<frozen_rubric>\n{"rubric":{"Alignment":["..."],"Layout":["..."]}}\n</frozen_rubric>'
-)
-text = tok.apply_chat_template(
-    [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": user}],
-    tokenize=False, add_generation_prompt=True,
-).replace("<image>", "<|vision_start|><|image_pad|><|vision_end|>")
-
-inputs = proc(text=[text], images=[img], return_tensors="pt").to("cuda")
-out = model.generate(**inputs, max_new_tokens=4096, do_sample=False)
-print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
-```
-
-#### Evaluator inference contract
-
-- `SYSTEM_PROMPT` must be the one shipped with the `DesignCoder-evaluate` dataset — it
-  enumerates the rubric namespace the model was trained against.
-- The chat template already ends with `<think>\n`, so generated text continues *inside* the
-  reasoning block and carries **no opening `<think>` tag**. Split on `</think>` to reach the
-  JSON payload.
-- Downscale images to ≤ 1,048,576 px as shown; larger inputs drift from training.
-- Parse `frozen_dynamic_scores.verdicts`. **Ignore the sibling `summary` string** — it
-  miscounts (e.g. reads `"24/24"` next to 25 emitted verdicts) in both models.
-- `static_scores` / `dynamic_scores` (40 items, 0/1/2) have **no held-out validation** — they
-  were only checked against contaminated training samples. Treat them as unverified.
-
 ## Provenance
 
 Each subfolder additionally ships `trainer_state.json` / `trainer_log.jsonl` (and