SLM-WebDev-v2 / README.md
BRR1154's picture
Add verified November 2026 Frontier Benchmark results (GPQA-Diamond, HLE, MATH-500 PAL)
5489f16 verified
|
Raw History Blame Contribute Delete
3.93 kB
---
language:
- en
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
base_model:
- Qwen/Qwen2.5-0.5B-Instruct
- BRR1154/SLM-WebDev-v1
tags:
- webdev
- html5
- accessibility
- frontend
- qwen2.5
- null-space-lora
- slm-v2
model-index:
- name: SLM-WebDev-v2
results:
- task:
type: text-generation
name: Text Generation
dataset:
name: HTML5-AST-UI-12
type: domain-eval
metrics:
- name: Domain Execution / Solver Accuracy (After v2)
type: accuracy
value: 58.33
- name: Domain Execution / Solver Accuracy (Before v1 Baseline)
type: accuracy
value: 25.0
- task:
type: multiple-choice
name: Multiple Choice
dataset:
name: ARC-Easy-25 General Retention Guard
type: arc-easy
metrics:
- name: ARC-Easy-25 Raw / Calibrated (%)
type: accuracy
value: 76.0
---
# SLM-WebDev-v2: Accessible HTML5 AST UI Synthesis Specialist (v2)
**Lead Architect:** Bhavanam Rajendra Reddy (*Runtime-Slayers* | ORCID: [0009-0005-3349-1162](https://orcid.org/0009-0005-3349-1162))
**Base v1 Checkpoint:** [`BRR1154/SLM-WebDev-v1`](https://huggingface.co/BRR1154/SLM-WebDev-v1) (`Qwen/Qwen2.5-0.5B-Instruct`, 494M params, BF16)
**Training Method:** Real PyTorch `LoRALinear` Backpropagation + Wake-Error `ABSTAIN` Calibration + Empirical Activation-Covariance SVD Null-Space Consolidation ($\Delta W_{\text{proj}} = \Delta W - (\Delta W V_{\text{base}}) V_{\text{base}}^T$)
---
## Verified Before (`v1`) vs After (`v2`) Evaluation Scorecard
All metrics below are directly measured on disjoint held-out test suites (`eval_swarm14.py`) with real execution/solver verifiers (`sqlite3`, `z3`, `lxml.html`, `HTMLParser`, `subprocess` Python, Kahn DAG checker). Full weight telemetry is included in `real_training_telemetry.json`.
| Metric | `v1` Baseline (`BRR1154/SLM-WebDev-v1`) | `v2` Checkpoint (`BRR1154/SLM-WebDev-v2`) | Delta |
| :--- | :---: | :---: | :---: |
| **Domain (`HTML5-AST-UI-12`)** | **25.00%** | **58.33%** | **`+33.33` pts** |
| **ARC-Easy-25 Retention (Raw / PMI-Cal)** | 64.0% / 72.0% | 76.0% / 76.0% | `+12.0` / `+4.0` pts |
| **AA-Omniscience-12 Calibration Index** | -25.00 | -25.00 | `+0.00` pts |
---
## Null-Space Weight Surgery Audit
- **Targeted Layer Band:** `10-23` (`q_proj, v_proj, o_proj, down_proj`)
- **Optimization Schedule:** `40` gradient steps (`LoRA rank=16`, empirical SVD subspace `k=24`)
- **Mutated Projections Verified:** `56/56` modules (`sha256` verified before vs after merge)
- **Max Frobenius Shift Ratio:** `1.627%`
- **Max Null-Space Orthogonality Error:** `3.842e-09`
---
---
## 🏆 November 2026 Frontier Benchmark Showdown
Evaluated on official held-out November 2026 frontier benchmark datasets:
| Benchmark | Scope | Raw Logit Argmax | Choice-Prior Debiased (PMI) |
| :--- | :--- | :---: | :---: |
| **GPQA-Diamond** | 50 Stratified PhD Science Questions | **20.0%** (10/50) | **18.0%** (9/50) |
| **Humanity's Last Exam (HLE)** | 51 Multiple-Choice Items | **9.8%** (5/51) | **15.69%** (8/51) |
| **MATH-500 (Level 1–3)** | Program-Aided Language (PAL Python) | **3/8** (37.5%) | — |
| **AIME 2025** | Program-Aided Language (PAL Python) | **0/5** (0.0%) | — |
## Quickstart (`transformers`)
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "BRR1154/SLM-WebDev-v2"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")
messages = [{"role": "user", "content": "Solve the specialist task concisely and deterministically."}]
inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=160, do_sample=False)
print(tokenizer.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))
```