Nadir 1.0: 15.9M on-device dark-mode model, int8/int4 quants, runtime, extension

README.md

@@ -0,0 +1,218 @@
+---
+license: apache-2.0
+library_name: onnx
+pipeline_tag: other
+tags:
+  - dark-mode
+  - browser-extension
+  - onnx
+  - onnxruntime-web
+  - accessibility
+  - design
+  - distillation
+---
+
+# Nadir
+
+**Nadir** is a small on-device model that gives light websites a tasteful, readable dark mode, with the
+care of a real dark theme rather than a filter. It runs entirely in your browser (onnxruntime-web, WASM),
+in about 35 ms of model time per page, and ships with a ready-to-load Chromium extension.
+
+Nadir follows three rules:
+
+1. **Already dark? Leave it alone.** Pages that are already dark are detected and never touched.
+2. **Has its own dark mode? Use it.** Nadir switches on the site's own dark theme (`prefers-color-scheme`
+   media rules, or `dark`/`theme-dark` class and attribute toggles it detects in the page's CSS).
+3. **No dark mode? The model designs one.** Nadir reads the page's colors and images and predicts a dark
+   palette for that specific site: brand colors stay recognizable, surfaces keep their hierarchy, text stays
+   readable, and logos stay visible: a logo on its own white rectangle is adapted to sit cleanly on the dark
+   surface. Photos, videos and ads are never color-shifted.
+
+In a blind pairwise test on 146 sites it never trained on, judged by Claude Haiku 5.5, the preference
+went to Nadir's dark mode over Dark Reader's on **93 sites vs 50** (3 ties): 65% of the decided pages.
+
+## How it works
+
+Nadir doesn't paint pixels. It redesigns the page's **color system**:
+
+1. **Extraction (`runtime/extract.js`).** One synchronous pass over the DOM collects every distinct color
+   *in its role*: background, text, border, shadow, gradient, SVG fill, SVG stroke. Each token gets 49
+   features: lightness, chroma, hue, area, text size and weight, where it appears (header, nav, link, form,
+   code, footer), what it sits on and the original contrast. It also records four relations between tokens
+   (text-on-surface, surface-on-surface, border-on-surface, same-element) and every visible image or
+   inline-SVG logo with 39 features plus a 32×32 thumbnail. Ad slots are excluded.
+2. **Model (`NadirNet`).** A relation-biased transformer over `[page] + tokens + images`. Each attention head
+   gets a learned bias from the 11 relation channels, so the model knows which text sits on which surface.
+   It predicts for every token an OKLab color as a parameterized edit of the input (lightness, chroma scale,
+   hue rotation, tint, alpha). This keeps brand hues stable and makes impossible colors rare. For every
+   image it predicts a treatment: `keep`, `dim`, `invert`, `light` (monochrome dark glyph → near-white) or
+   `plate` (dark multicolor logo that needs help; shown with a hue-preserving inversion at runtime). A small
+   CNN reads the thumbnails.
+3. **Design rules (`runtime/nadir-infer.js`, `runtime/nadir-map.js`).** Deterministic guards on top of the
+   model:
+   - The page canvas is always dark.
+   - Every text and icon color keeps WCAG contrast on *every* surface it actually sits on. It never needs
+     more contrast than the original design had.
+   - Brand colors keep their hue (±45°).
+   - Things that were invisible in the light design stay invisible: text-fade gradients and borders that
+     matched their surface follow the surface.
+   - Colors drawn over video and photos are left alone when they already read as dark mode (player
+     controls, scrims, captions).
+   - Photos and video are never recolored.
+   - Light, flat background images behind content (paper textures, pale washes, SVG gradients) are cleared so the
+     designed dark surface shows through. Images with real detail are left alone.
+   - Inline SVG logos stay readable: their shapes are drawn as ink, and SVG mask and clip-path shapes (knockout
+     text, cut-outs) are never recolored.
+   - **Logos on their own white rectangle are adapted.** The white background is removed so Nadir's surface
+     shows through, black or gray ink becomes light, and brand colors keep their hue (lifted only as far as
+     needed to read on dark). This applies only to images the page itself marks as its logo.
+   - Dark logos on transparency get a hue-preserving inversion, and already-light ones are kept.
+4. **Engine.** The predicted palette is applied through a patched
+   [Dark Reader](https://github.com/darkreader/darkreader) 4.9.133 dynamic engine (MIT), which handles the
+   hard plumbing: stylesheets, inline styles, shadow DOM and dynamically inserted content. It asks Nadir for
+   every color it rewrites. Colors the model didn't see (new content) are mapped by nearest-neighbor
+   lookup in the predicted palette.
+
+## Model
+
+| | |
+|---|---|
+| Parameters | 15.9M (15,895,995) |
+| Width / depth / heads | 384 / 8 layers / 6 heads |
+| Inputs | ≤256 color tokens, ≤32 images, 11 relation channels |
+| Outputs | OKLab color + alpha per token, 5-way image treatment |
+| Runtime | onnxruntime-web 1.29 (WASM SIMD, multi-threaded) |
+
+### Files
+
+| File | Precision | Size | Δ color vs fp32 (mean / max, OKLab) | Image agreement |
+|---|---|---|---|---|
+| `onnx/nadir-fp32.onnx` | fp32 | 63.7 MB | 0 / 0 | 100% |
+| `onnx/nadir-int8.onnx` | 8-bit weights | 20.6 MB | 0.00015 / 0.025 | 99.99% |
+| `onnx/nadir-int4.onnx` | 4-bit weights (8-bit input/output layers) | 13.3 MB | 0.00036 / 0.071 | 99.76% |
+
+Fidelity is measured on 400 held-out pages. A difference of 0.02 in OKLab is about the smallest one you can see.
+
+Speed in onnxruntime-web 1.29 (Chrome, Apple M4 Max, 4 WASM threads), per page:
+
+| File | Load | Inference |
+|---|---|---|
+| fp32 | ~7 s | 34 ms |
+| int8 | ~1.5 s | 33–35 ms |
+| int4 | ~1.0 s | 38 ms |
+
+The extension's own end-to-end time for a first visit is 0.7–3 s, mostly reading the page and its images. Revisits apply
+from cache.
+
+- `onnx/nadir-int8.onnx` is the extension's default.
+- The 8-bit and 4-bit models are **calibrated weight-only quants**:
+  - Layer-sequential GPTQ, with Hessians collected from 256 real crawled pages flowing through the
+    already-quantized earlier layers.
+  - Asymmetric codes in groups of 32, with each group's clipping range chosen by an MSE search.
+  - The 4-bit model keeps its input and output layers at 8 bits.
+- Storage format:
+  - 4-bit layers use ONNX `MatMulNBits` (block 32).
+  - 8-bit layers store uint8 codes with a per-block scale and zero point, dequantized in the graph by standard
+    ops (`Cast`/`Sub`/`Mul`/`Reshape`). onnxruntime folds these into the weights when it loads the model, so
+    the 8-bit layers run as exact fp32 math with no special kernels. Both files are tested on onnxruntime 1.22
+    (CPU) and onnxruntime-web 1.29.
+- Each quant is verified three ways: packing parity against a PyTorch fake-quant model (≤2e-6), fidelity
+  against fp32, and quality against the training targets.
+- They run in onnxruntime-web with outputs matching onnxruntime CPU to ~1e-5.
+- Activation-quantized int8 (ORT dynamic/static) was rejected: worst-case color error up to 0.99.
+
+`model.safetensors` + `model.py` hold the PyTorch weights and architecture.
+
+## Training data
+
+- **Pages.** A crawl of 47,569 sites from the [Tranco](https://tranco-list.eu/) top list at 1280×800.
+  Each page was captured in light mode, in emulated dark mode, and with dark-class toggles applied.
+- **Teacher.** 19,563 light pages were redesigned by **Claude Haiku 5.5** (Anthropic Batch API,
+  high effort) acting as the site's own product designer. It was given the screenshot, a contact sheet of
+  the images, and the full color inventory with roles and locations, and returned a dark palette plus image
+  treatments in a strict JSON schema. The design brief is in `training/haiku_gen.py`.
+- **Native pairs.** 4,142 pages with a real built-in dark theme give human-designed light→dark
+  pairs, aligned token by token.
+- **Quality control.**
+  - Designs that missed ids, kept a light canvas, or left >15% of text below 3:1 contrast were dropped.
+  - The rest were passed through the same contrast rule the runtime uses (14,338 of 19,464 designs adjusted).
+  - The train/validation split is by site (hash of the hostname).
+- **Augmentation.** Random token dropout (it also simulates ad blockers and partial loads) and whole-design
+  hue rotation.
+
+## Evaluation
+
+The judge was a blind pairwise one: Claude Haiku 5.5 saw the original light page plus two dark versions in
+random order, A and B. It judged readability first, then breakage (glaring light patches, lost logos,
+distorted photos), then design quality. All pages are held-out sites the model never trained on.
+
+| Nadir vs … (146 held-out sites) | Nadir preferred | Other preferred | Ties |
+|---|---|---|---|
+| Dark Reader 4.9 (dynamic mode) | **93** | 50 | 3 |
+| Claude Haiku 5.5's own designs (the teacher)¹ | 76 | 61 | 9 |
+| Nadir before its final design rules² | 45 | 29 | 72 |
+
+¹ The teacher's designs were rendered with the earlier runtime, so part of this margin comes from the runtime's newer
+design rules rather than from the model itself. Before those rules, the model and its teacher were even (63 vs 67).
+² The rules for light background textures, SVG logos and SVG masks.
+
+**Model size.** A 34.0M-parameter version (512 / 10 layers / 8 heads) trained on the same data was no better: 49 vs 53
+with 45 ties against this one. It reached the same color error and runs about 2× slower in the browser, so the 15.9M
+model is the release.
+
+Held-out color error vs the teacher designs (OKLab ΔE): 0.040 (Haiku pages), 0.103
+(native dark themes). Image-treatment accuracy: 79.4% over 5 classes.
+
+## Use it
+
+### Browser extension (Chrome, Edge, Brave, Arc, Aside and other Chromium browsers)
+
+1. Download `nadir-extension.zip` and unzip it.
+2. Open `chrome://extensions`, enable **Developer mode**, click **Load unpacked**, and choose the folder.
+3. Browse. The toolbar badge shows what Nadir did on each site:
+   - `AI`: model-designed dark mode.
+   - `ON`: the site's own dark mode.
+   - No badge: already dark.
+
+The popup can turn Nadir off for a site.
+
+The extension works alongside content blockers such as uBlock Origin Lite: removed ads simply aren't part
+of the palette, and ads that remain are treated as images (never recolored).
+
+### In your own code
+
+```js
+import * as ort from 'onnxruntime-web';
+// runtime/extract.js, runtime/nadir-infer.js and runtime/nadir-map.js define the globals used below
+const page = await __nadirExtract({describe: false, thumbs: false});   // in the page: colors, roles, relations, images
+// pixel stats + thumbnails for the images (any context that can fetch them, e.g. an extension page)
+const st = await __nadirImageStats(page.images, {dpr: devicePixelRatio});
+page.images.forEach((im, i) => { Object.assign(im, st[i]); im.f = __nadirImageFeatures(im, page.vw, page.vh, page.docH); });
+const built = NadirInfer.buildInputs(page);
+const session = await ort.InferenceSession.create('nadir-int8.onnx', {executionProviders: ['wasm']});
+const feeds = Object.fromEntries(Object.entries(built.feeds).map(([k, [data, dims]]) =>
+  [k, new ort.Tensor(k === 'tok_role' || k === 'src' ? 'int64' : k === 'mask' ? 'bool' : 'float32', data, dims)]));
+const out = await session.run(feeds);
+const {colors, keep, treatments} = NadirInfer.decode(page, built,
+  {lab: out.lab.data, alpha_logit: out.alpha_logit.data, img_logits: out.img_logits.data});
+// colors[i] is the dark replacement for page.tokens[i] (null = keep); treatments[j] applies to page.images[j]
+```
+
+## Limitations
+
+- Nadir recolors CSS. Text and UI baked into images, canvas drawings and cross-origin iframes keep their
+  original colors, though the iframes get their own Nadir pass in the extension.
+- Colors are mapped per (color, role). If one exact color is used both as a light card and as a light
+  button, both get the same dark color.
+- Background images that mix light areas with strong color (for example a colored hero shape with a white base) are
+  kept as they are, so their light parts stay light.
+- The model reads at most 256 colors per page. On very colorful pages the rarest colors are mapped by nearest
+  neighbor in the predicted palette.
+- The teacher is an LLM. Its taste is good but not perfect, and the model inherits its preferences.
+- The judge is also an LLM, so the evaluation is automated, not a human study.
+
+## License
+
+Apache-2.0 for the model, weights and Nadir code. The bundled page engine is a modified
+[Dark Reader](https://github.com/darkreader/darkreader) (MIT License, © Dark Reader Ltd.); its license is
+included in the extension (`lib/DARKREADER-LICENSE`).

config.json

@@ -0,0 +1,36 @@
+{
+ "d": 384,
+ "layers": 8,
+ "heads": 6,
+ "ff": 4,
+ "drop": 0.1,
+ "architecture": "NadirNet",
+ "token_features": 49,
+ "image_features": 39,
+ "relation_channels": 11,
+ "max_tokens": 256,
+ "max_images": 32,
+ "image_treatments": [
+  "keep",
+  "dim",
+  "invert",
+  "light",
+  "plate"
+ ],
+ "inputs": [
+  "tok_f",
+  "tok_role",
+  "tok_lab",
+  "img_f",
+  "img_thumb",
+  "page_f",
+  "src",
+  "rel",
+  "mask"
+ ],
+ "outputs": [
+  "lab",
+  "alpha_logit",
+  "img_logits"
+ ]
+}

model.py

@@ -0,0 +1,141 @@
+"""NadirNet: a relation-biased set transformer over a page's colour palette and images.
+
+Tokens: [PAGE] + one token per (colour, role) the page uses + one token per image.
+Attention logits get a learned per-head bias from the page's structure graph (text-on-surface,
+surface-on-surface, border-on-surface, same-element, image-on-surface, same-colour), so the model
+reasons about hierarchy and contrast the way a designer would. Outputs a dark colour per token
+(OKLab, parameterised relative to the input so hue is kept unless the model chooses otherwise),
+an alpha, and a treatment class per image.
+"""
+import math
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+FT, FI, FP = 49, 39, 8
+N_ROLES, N_TREAT = 7, 5
+K_REL = 11
+
+
+def fourier_lab(lab, n=6):
+    x = torch.stack([lab[..., 0], lab[..., 1] * 4, lab[..., 2] * 4], -1)  # (..., 3)
+    freqs = (2.0 ** torch.arange(n, device=lab.device, dtype=lab.dtype)) * math.pi
+    ang = x[..., None] * freqs  # (...,3,n)
+    return torch.cat([torch.sin(ang), torch.cos(ang)], -1).flatten(-2)  # (..., 6n)
+
+
+class MLP(nn.Module):
+    def __init__(self, i, h, o, drop=0.0):
+        super().__init__()
+        self.net = nn.Sequential(nn.Linear(i, h), nn.GELU(), nn.Dropout(drop), nn.Linear(h, o))
+
+    def forward(self, x):
+        return self.net(x)
+
+
+class ThumbCNN(nn.Module):
+    """32x32 RGBA thumbnail -> embedding. Alpha matters: transparent logos behave differently from photos."""
+    def __init__(self, o=128):
+        super().__init__()
+        c = [5, 32, 64, 96, 128]
+        layers = []
+        for a, b in zip(c[:-1], c[1:]):
+            layers += [nn.Conv2d(a, b, 3, padding=1), nn.GELU(), nn.Conv2d(b, b, 3, padding=1, stride=2), nn.GELU()]
+        self.net = nn.Sequential(*layers)
+        self.proj = nn.Linear(128 * 2 * 2, o)
+
+    def forward(self, x):  # (B, M, 4, 32, 32) in [0,1]
+        B, M = x.shape[:2]
+        x = x.reshape(B * M, 4, 32, 32)
+        a = x[:, 3:4]
+        lum = (0.2126 * x[:, 0:1] + 0.7152 * x[:, 1:2] + 0.0722 * x[:, 2:3]) * a  # premultiplied luminance
+        x = torch.cat([x[:, :3] * a, a, lum], 1)
+        x = self.net(x)  # (BM,128,2,2)
+        return self.proj(x.flatten(1)).reshape(B, M, -1)
+
+
+class Block(nn.Module):
+    def __init__(self, d, h, ff, drop):
+        super().__init__()
+        self.h = h
+        self.ln1 = nn.LayerNorm(d)
+        self.qkv = nn.Linear(d, 3 * d)
+        self.proj = nn.Linear(d, d)
+        self.rel = nn.Linear(K_REL, h, bias=False)
+        nn.init.normal_(self.rel.weight, std=0.5)
+        self.ln2 = nn.LayerNorm(d)
+        self.ff = nn.Sequential(nn.Linear(d, ff * d), nn.GELU(), nn.Dropout(drop), nn.Linear(ff * d, d))
+        self.drop = nn.Dropout(drop)
+
+    def forward(self, x, rel, pad_bias):
+        B, T, D = x.shape
+        q, k, v = self.qkv(self.ln1(x)).reshape(B, T, 3, self.h, D // self.h).permute(2, 0, 3, 1, 4)
+        bias = self.rel(rel).permute(0, 3, 1, 2) + pad_bias  # (B,H,T,T)
+        a = F.scaled_dot_product_attention(q, k, v, attn_mask=bias.to(q.dtype))
+        x = x + self.drop(self.proj(a.transpose(1, 2).reshape(B, T, D)))
+        return x + self.drop(self.ff(self.ln2(x)))
+
+
+class NadirNet(nn.Module):
+    def __init__(self, d=384, layers=8, heads=6, ff=4, drop=0.1):
+        super().__init__()
+        self.cfg = dict(d=d, layers=layers, heads=heads, ff=ff, drop=drop)
+        self.tok_in = MLP(FT + N_ROLES + 36, 2 * d, d)
+        self.cnn = ThumbCNN(128)
+        self.img_in = MLP(FI + 128, 2 * d, d)
+        self.page_in = MLP(FP, d, d)
+        self.src = nn.Embedding(2, d)
+        self.typ = nn.Embedding(3, d)
+        self.blocks = nn.ModuleList([Block(d, heads, ff, drop) for _ in range(layers)])
+        self.ln = nn.LayerNorm(d)
+        self.color_head = MLP(d + 3, d, 6)
+        self.img_head = MLP(d, d // 2, N_TREAT)
+
+    def forward(self, tok_f, tok_role, tok_lab, img_f, img_thumb, page_f, src, rel, mask):
+        """tok_f (B,N,FT) tok_role (B,N) long, tok_lab (B,N,3), img_f (B,M,FI), img_thumb (B,M,4,32,32) float 0-1,
+        page_f (B,FP), src (B,) long, rel (B,T,T,K) with T=1+N+M, mask (B,T) bool (True=real)."""
+        B, N = tok_role.shape
+        M = img_f.shape[1]
+        tok = self.tok_in(torch.cat([tok_f, F.one_hot(tok_role, N_ROLES).to(tok_f.dtype), fourier_lab(tok_lab)], -1)) + self.typ.weight[1]
+        if M > 0:
+            img = self.img_in(torch.cat([img_f, self.cnn(img_thumb)], -1)) + self.typ.weight[2]
+        else:
+            img = tok.new_zeros(B, 0, tok.shape[-1])
+        cls = self.page_in(page_f)[:, None] + self.typ.weight[0]
+        x = torch.cat([cls, tok, img], 1) + self.src(src)[:, None]
+        pad_bias = torch.zeros(mask.shape, dtype=x.dtype, device=x.device).masked_fill(~mask, -1e4)[:, None, None, :]
+        for blk in self.blocks:
+            x = blk(x, rel, pad_bias)
+        x = self.ln(x)
+        ht = x[:, 1:1 + N]
+        z = self.color_head(torch.cat([ht, tok_lab], -1))
+        L = torch.sigmoid(z[..., 0])
+        s = 1.6 * torch.sigmoid(z[..., 1])
+        r = 0.6 * torch.tanh(z[..., 2])
+        a_in, b_in = tok_lab[..., 1], tok_lab[..., 2]
+        a = s * a_in - r * b_in + 0.12 * torch.tanh(z[..., 3])
+        b = r * a_in + s * b_in + 0.12 * torch.tanh(z[..., 4])
+        lab = torch.stack([L, a, b], -1)
+        alpha_logit = z[..., 5]
+        img_logits = self.img_head(x[:, 1 + N:]) if M > 0 else x.new_zeros(B, 0, N_TREAT)
+        return lab, alpha_logit, img_logits
+
+
+# ---------- colour maths (differentiable) ----------
+OKLAB_TO_LMS = torch.tensor([[1.0, 0.3963377774, 0.2158037573], [1.0, -0.1055613458, -0.0638541728], [1.0, -0.0894841775, -1.2914855480]])
+LMS_TO_RGB = torch.tensor([[4.0767416621, -3.3077115913, 0.2309699292], [-1.2684380046, 2.6097574011, -0.3413193965], [-0.0041960863, -0.7034186147, 1.7076147010]])
+
+
+def oklab_to_linrgb(lab):
+    lms = lab @ OKLAB_TO_LMS.to(lab).T
+    return (lms ** 3) @ LMS_TO_RGB.to(lab).T
+
+
+def luminance(lab):
+    rgb = oklab_to_linrgb(lab).clamp(0, 1)
+    return rgb @ torch.tensor([0.2126, 0.7152, 0.0722]).to(lab)
+
+
+def contrast_ratio(lab1, lab2):
+    l1, l2 = luminance(lab1), luminance(lab2)
+    return (torch.maximum(l1, l2) + 0.05) / (torch.minimum(l1, l2) + 0.05)

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e7acc89ba8a039a4889d89ec9f01afce49b2f765d1ac57a489eb4c604aef2ceb
+size 63596788

nadir-extension.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4a5da0a2671306af61a162f81c8cfa276ab2dd70fb05f7e927a401d9e1f08757
+size 23421769

onnx/nadir-fp32.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:83c54558313415596d4d3b3e753ff30e3a72cdc65e27a5c8eefcb132b9d5c934
+size 63699911

onnx/nadir-int4.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5e11864ad3d7f1edb7bc8b7006e5d7a1c12401bbc17578f590182c0ca764802a
+size 13313059

onnx/nadir-int8.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bbdb0a2b13bfca0c05f8d19d54d3b721fb4fcd459d40e8519a7a5f77af15d634
+size 20626640

reports/eval_judge.json

@@ -0,0 +1,88 @@
+{
+ "judge": "Claude Haiku 5.5 (Batch API, effort medium), blind pairwise: original light page + two dark versions in random order; criteria readability > breakage > design",
+ "pages": "held-out sites (hash split by hostname), rendered live at 1280x800",
+ "final_release_vs_others": {
+  "final_m3:dr": {
+   "n": 146,
+   "final_m3_wins": 93,
+   "dr_wins": 50,
+   "ties": 3,
+   "final_m3_winrate_excl_ties": 0.65,
+   "mean_score_final_m3": 6.85,
+   "mean_score_dr": 6.1
+  },
+  "final_m3:final_m": {
+   "n": 146,
+   "final_m3_wins": 45,
+   "final_m_wins": 29,
+   "ties": 72,
+   "final_m3_winrate_excl_ties": 0.608,
+   "mean_score_final_m3": 6.62,
+   "mean_score_final_m": 6.16
+  },
+  "final_m3:haiku": {
+   "n": 146,
+   "final_m3_wins": 76,
+   "haiku_wins": 61,
+   "ties": 9,
+   "final_m3_winrate_excl_ties": 0.555,
+   "mean_score_final_m3": 6.72,
+   "mean_score_haiku": 6.31
+  }
+ },
+ "before_final_design_rules": {
+  "final_m:dr": {
+   "n": 147,
+   "final_m_wins": 83,
+   "dr_wins": 57,
+   "ties": 7,
+   "final_m_winrate_excl_ties": 0.593,
+   "mean_score_final_m": 6.5,
+   "mean_score_dr": 6.28
+  },
+  "final_l:dr": {
+   "n": 147,
+   "final_l_wins": 81,
+   "dr_wins": 60,
+   "ties": 6,
+   "final_l_winrate_excl_ties": 0.574,
+   "mean_score_final_l": 6.61,
+   "mean_score_dr": 6.31
+  },
+  "final_m:final_l": {
+   "n": 147,
+   "final_m_wins": 49,
+   "final_l_wins": 53,
+   "ties": 45,
+   "final_m_winrate_excl_ties": 0.48,
+   "mean_score_final_m": 6.43,
+   "mean_score_final_l": 6.54
+  },
+  "final_m:haiku": {
+   "n": 146,
+   "final_m_wins": 63,
+   "haiku_wins": 67,
+   "ties": 16,
+   "final_m_winrate_excl_ties": 0.485,
+   "mean_score_final_m": 6.32,
+   "mean_score_haiku": 6.38
+  },
+  "final_l:haiku": {
+   "n": 146,
+   "final_l_wins": 64,
+   "haiku_wins": 70,
+   "ties": 12,
+   "final_l_winrate_excl_ties": 0.478,
+   "mean_score_final_l": 6.47,
+   "mean_score_haiku": 6.44
+  }
+ },
+ "notes": {
+  "final_m3": "released model (15.9M) with final runtime rules",
+  "final_m": "same model, runtime before the backdrop/SVG rules",
+  "final_l": "34.0M model, earlier runtime",
+  "dr": "Dark Reader 4.9.133 dynamic mode",
+  "haiku": "teacher designs from Claude Haiku 5.5, earlier runtime"
+ },
+ "cost_usd": 0.368
+}

reports/export_report.json

@@ -0,0 +1,20 @@
+{
+ "params": 15895995,
+ "parity_max_abs": 1.2516975402832031e-06,
+ "variants": {
+  "fp32": {
+   "dE_vs_fp32": 0.0,
+   "dE_max": 0.0,
+   "img_agree": 1.0,
+   "size_mb": 63.7,
+   "cpu_ms_median": 7.48
+  },
+  "fp16": {
+   "dE_vs_fp32": 0.0002421401860071152,
+   "dE_max": 0.23341728746891022,
+   "img_agree": 1.0,
+   "size_mb": 31.91,
+   "cpu_ms_median": 10.17
+  }
+ }
+}

reports/ortweb_timing_m4max.json

@@ -0,0 +1,67 @@
+{
+ "runtime": "onnxruntime-web 1.29 wasm, Chrome for Testing 141 headless, Apple M4 Max; t1/t4 = 1/4 threads; 12 held-out pages, median of 3 runs each",
+ "fp32": {
+  "t1": {
+   "isolated": false,
+   "load_ms": 7246.9,
+   "ms_median": 117.08,
+   "max_abs_lab_vs_ort_cpu": 6.16908073425293e-06,
+   "max_abs_img_vs_ort_cpu": 4.172325134277344e-06
+  },
+  "t4": {
+   "isolated": false,
+   "load_ms": 7363.2,
+   "ms_median": 33.93,
+   "max_abs_lab_vs_ort_cpu": 6.16908073425293e-06,
+   "max_abs_img_vs_ort_cpu": 4.172325134277344e-06
+  }
+ },
+ "int8": {
+  "t1": {
+   "isolated": false,
+   "load_ms": 1356.6,
+   "ms_median": 116.78,
+   "max_abs_lab_vs_ort_cpu": 1.9669532775878906e-06,
+   "max_abs_img_vs_ort_cpu": 2.9802322387695312e-06
+  },
+  "t4": {
+   "isolated": false,
+   "load_ms": 1628.3,
+   "ms_median": 34.78,
+   "max_abs_lab_vs_ort_cpu": 1.9669532775878906e-06,
+   "max_abs_img_vs_ort_cpu": 2.9802322387695312e-06
+  }
+ },
+ "int8_repeat": {
+  "t1": {
+   "isolated": false,
+   "load_ms": 1523.5,
+   "ms_median": 115.82,
+   "max_abs_lab_vs_ort_cpu": 1.9669532775878906e-06,
+   "max_abs_img_vs_ort_cpu": 2.9802322387695312e-06
+  },
+  "t4": {
+   "isolated": false,
+   "load_ms": 1519,
+   "ms_median": 32.68,
+   "max_abs_lab_vs_ort_cpu": 1.9669532775878906e-06,
+   "max_abs_img_vs_ort_cpu": 2.9802322387695312e-06
+  }
+ },
+ "int4": {
+  "t1": {
+   "isolated": false,
+   "load_ms": 1074.8,
+   "ms_median": 137.33,
+   "max_abs_lab_vs_ort_cpu": 1.3709068298339844e-06,
+   "max_abs_img_vs_ort_cpu": 7.659196853637695e-06
+  },
+  "t4": {
+   "isolated": false,
+   "load_ms": 1047,
+   "ms_median": 38.47,
+   "max_abs_lab_vs_ort_cpu": 1.3709068298339844e-06,
+   "max_abs_img_vs_ort_cpu": 7.659196853637695e-06
+  }
+ }
+}

reports/quant_report.json

@@ -0,0 +1,63 @@
+{
+ "format": {
+  "8-bit layers": "uint8 blocks of 32 with fp32 scale + uint8 zero point, dequantized in-graph (Cast/Sub/Mul, folded at load)",
+  "4-bit layers": "uint4 blocks of 32, MatMulNBits"
+ },
+ "group_size": 32,
+ "calib_pages": 256,
+ "eval_pages": 400,
+ "fp32_layers": [
+  "cnn.proj",
+  "page_in.net.0",
+  "page_in.net.3",
+  "blocks.0.rel",
+  "blocks.1.rel",
+  "blocks.2.rel",
+  "blocks.3.rel",
+  "blocks.4.rel",
+  "blocks.5.rel",
+  "blocks.6.rel",
+  "blocks.7.rel"
+ ],
+ "int4_8bit_edge_layers": [
+  "tok_in.net.0",
+  "tok_in.net.3",
+  "img_in.net.0",
+  "img_in.net.3",
+  "color_head.net.0",
+  "color_head.net.3",
+  "img_head.net.0",
+  "img_head.net.3"
+ ],
+ "variants": {
+  "fp32": {
+   "haiku_dE": 0.039349029815445344,
+   "native_dE": 0.08639045081450604,
+   "img_acc": 0.7966292134831461,
+   "size_mb": 63.7,
+   "cpu_ms_median": 11.42
+  },
+  "int8": {
+   "dE_vs_fp32": 0.00015194275738394936,
+   "dE_max": 0.02521813102066517,
+   "img_agree": 0.9998743718592965,
+   "haiku_dE": 0.03935358844697476,
+   "native_dE": 0.0863548206735868,
+   "img_acc": 0.7968164794007491,
+   "pack_parity_max_abs": 2.5033950805664062e-06,
+   "size_mb": 20.63,
+   "cpu_ms_median": 9.19
+  },
+  "int4": {
+   "dE_vs_fp32": 0.00036296586506068707,
+   "dE_max": 0.0709674283862114,
+   "img_agree": 0.9976130653266332,
+   "haiku_dE": 0.039340872559696434,
+   "native_dE": 0.08632776886341162,
+   "img_acc": 0.797003745318352,
+   "pack_parity_max_abs": 1.5497207641601562e-06,
+   "size_mb": 13.31,
+   "cpu_ms_median": 9.76
+  }
+ }
+}

reports/train_log.json

@@ -0,0 +1,501 @@
+{
+ "args": {
+  "data": "/home/user/.local/share/rtx-temporary-jobs/tj-nadir-0fe76575c0f3/payload/work/data/ds_final.pkl",
+  "out": "/job/export/run_m",
+  "d": 384,
+  "layers": 8,
+  "heads": 6,
+  "epochs": 30,
+  "bs": 48,
+  "lr": 0.0004,
+  "wd": 0.05,
+  "drop": 0.1,
+  "native_weight": 0.5,
+  "seed": 0,
+  "workers": 4
+ },
+ "params": 15895995,
+ "log": [
+  {
+   "epoch": 0,
+   "step": 468,
+   "secs": 81,
+   "loss": 0.3405,
+   "dE": 0.12132,
+   "con": 0.50822,
+   "img": 1.29224,
+   "s0_dE": 0.06306,
+   "s0_dL": 0.06024,
+   "s0_con": 0.04883,
+   "img_acc": 0.73416,
+   "s1_dE": 0.16366,
+   "s1_dL": 0.15909,
+   "s1_con": 0.11938
+  },
+  {
+   "epoch": 1,
+   "step": 936,
+   "secs": 148,
+   "loss": 0.18873,
+   "dE": 0.06161,
+   "con": 0.03771,
+   "img": 0.99138,
+   "s0_dE": 0.05432,
+   "s0_dL": 0.05155,
+   "s0_con": 0.03668,
+   "img_acc": 0.78603,
+   "s1_dE": 0.1523,
+   "s1_dL": 0.14838,
+   "s1_con": 0.08049
+  },
+  {
+   "epoch": 2,
+   "step": 1404,
+   "secs": 215,
+   "loss": 0.17375,
+   "dE": 0.05668,
+   "con": 0.03498,
+   "img": 0.90371,
+   "s0_dE": 0.04977,
+   "s0_dL": 0.04725,
+   "s0_con": 0.02243,
+   "img_acc": 0.78005,
+   "s1_dE": 0.14476,
+   "s1_dL": 0.14153,
+   "s1_con": 0.26806
+  },
+  {
+   "epoch": 3,
+   "step": 1872,
+   "secs": 281,
+   "loss": 0.16779,
+   "dE": 0.05488,
+   "con": 0.04139,
+   "img": 0.86775,
+   "s0_dE": 0.0506,
+   "s0_dL": 0.04808,
+   "s0_con": 0.0331,
+   "img_acc": 0.79611,
+   "s1_dE": 0.13697,
+   "s1_dL": 0.13372,
+   "s1_con": 0.1237
+  },
+  {
+   "epoch": 4,
+   "step": 2340,
+   "secs": 349,
+   "loss": 0.16278,
+   "dE": 0.05274,
+   "con": 0.0414,
+   "img": 0.85412,
+   "s0_dE": 0.04849,
+   "s0_dL": 0.04604,
+   "s0_con": 0.02439,
+   "img_acc": 0.78982,
+   "s1_dE": 0.13195,
+   "s1_dL": 0.12841,
+   "s1_con": 0.16648
+  },
+  {
+   "epoch": 5,
+   "step": 2808,
+   "secs": 416,
+   "loss": 0.15903,
+   "dE": 0.05166,
+   "con": 0.03833,
+   "img": 0.84086,
+   "s0_dE": 0.0472,
+   "s0_dL": 0.04477,
+   "s0_con": 0.03465,
+   "img_acc": 0.80467,
+   "s1_dE": 0.12882,
+   "s1_dL": 0.12538,
+   "s1_con": 0.12963
+  },
+  {
+   "epoch": 6,
+   "step": 3276,
+   "secs": 484,
+   "loss": 0.15729,
+   "dE": 0.05076,
+   "con": 0.04035,
+   "img": 0.83185,
+   "s0_dE": 0.0457,
+   "s0_dL": 0.04338,
+   "s0_con": 0.02999,
+   "img_acc": 0.80345,
+   "s1_dE": 0.15936,
+   "s1_dL": 0.15619,
+   "s1_con": 0.51491
+  },
+  {
+   "epoch": 7,
+   "step": 3744,
+   "secs": 552,
+   "loss": 0.15392,
+   "dE": 0.04978,
+   "con": 0.0403,
+   "img": 0.81841,
+   "s0_dE": 0.04612,
+   "s0_dL": 0.04395,
+   "s0_con": 0.01962,
+   "img_acc": 0.79899,
+   "s1_dE": 0.14854,
+   "s1_dL": 0.14578,
+   "s1_con": 0.42056
+  },
+  {
+   "epoch": 8,
+   "step": 4212,
+   "secs": 619,
+   "loss": 0.15244,
+   "dE": 0.04894,
+   "con": 0.04062,
+   "img": 0.81195,
+   "s0_dE": 0.04632,
+   "s0_dL": 0.04425,
+   "s0_con": 0.02413,
+   "img_acc": 0.79917,
+   "s1_dE": 0.12446,
+   "s1_dL": 0.12131,
+   "s1_con": 0.23072
+  },
+  {
+   "epoch": 9,
+   "step": 4680,
+   "secs": 687,
+   "loss": 0.14931,
+   "dE": 0.0481,
+   "con": 0.03785,
+   "img": 0.79837,
+   "s0_dE": 0.04452,
+   "s0_dL": 0.04238,
+   "s0_con": 0.02434,
+   "img_acc": 0.80094,
+   "s1_dE": 0.12096,
+   "s1_dL": 0.11779,
+   "s1_con": 0.21844
+  },
+  {
+   "epoch": 10,
+   "step": 5148,
+   "secs": 755,
+   "loss": 0.1484,
+   "dE": 0.04786,
+   "con": 0.03854,
+   "img": 0.79393,
+   "s0_dE": 0.04549,
+   "s0_dL": 0.04344,
+   "s0_con": 0.02666,
+   "img_acc": 0.79123,
+   "s1_dE": 0.11822,
+   "s1_dL": 0.11502,
+   "s1_con": 0.25328
+  },
+  {
+   "epoch": 11,
+   "step": 5616,
+   "secs": 822,
+   "loss": 0.14645,
+   "dE": 0.04703,
+   "con": 0.03983,
+   "img": 0.78304,
+   "s0_dE": 0.04345,
+   "s0_dL": 0.04139,
+   "s0_con": 0.01948,
+   "img_acc": 0.81053,
+   "s1_dE": 0.11486,
+   "s1_dL": 0.11153,
+   "s1_con": 0.20122
+  },
+  {
+   "epoch": 12,
+   "step": 6084,
+   "secs": 891,
+   "loss": 0.13803,
+   "dE": 0.04635,
+   "con": 0.03723,
+   "img": 0.76961,
+   "s0_dE": 0.04328,
+   "s0_dL": 0.04125,
+   "s0_con": 0.01831,
+   "img_acc": 0.8051,
+   "s1_dE": 0.11435,
+   "s1_dL": 0.11091,
+   "s1_con": 0.16656
+  },
+  {
+   "epoch": 13,
+   "step": 6552,
+   "secs": 958,
+   "loss": 0.13627,
+   "dE": 0.04589,
+   "con": 0.03889,
+   "img": 0.76454,
+   "s0_dE": 0.04247,
+   "s0_dL": 0.04047,
+   "s0_con": 0.01875,
+   "img_acc": 0.80161,
+   "s1_dE": 0.11794,
+   "s1_dL": 0.11469,
+   "s1_con": 0.23754
+  },
+  {
+   "epoch": 14,
+   "step": 7020,
+   "secs": 1025,
+   "loss": 0.13324,
+   "dE": 0.045,
+   "con": 0.0383,
+   "img": 0.75126,
+   "s0_dE": 0.04194,
+   "s0_dL": 0.03998,
+   "s0_con": 0.02424,
+   "img_acc": 0.80332,
+   "s1_dE": 0.11805,
+   "s1_dL": 0.11474,
+   "s1_con": 0.22873
+  },
+  {
+   "epoch": 15,
+   "step": 7488,
+   "secs": 1093,
+   "loss": 0.13267,
+   "dE": 0.04502,
+   "con": 0.0394,
+   "img": 0.74724,
+   "s0_dE": 0.04183,
+   "s0_dL": 0.03983,
+   "s0_con": 0.01954,
+   "img_acc": 0.81353,
+   "s1_dE": 0.11306,
+   "s1_dL": 0.10954,
+   "s1_con": 0.20409
+  },
+  {
+   "epoch": 16,
+   "step": 7956,
+   "secs": 1161,
+   "loss": 0.1305,
+   "dE": 0.04465,
+   "con": 0.03933,
+   "img": 0.72731,
+   "s0_dE": 0.04195,
+   "s0_dL": 0.04006,
+   "s0_con": 0.02284,
+   "img_acc": 0.80485,
+   "s1_dE": 0.11437,
+   "s1_dL": 0.11122,
+   "s1_con": 0.27215
+  },
+  {
+   "epoch": 17,
+   "step": 8424,
+   "secs": 1229,
+   "loss": 0.12864,
+   "dE": 0.04415,
+   "con": 0.03796,
+   "img": 0.71419,
+   "s0_dE": 0.04177,
+   "s0_dL": 0.0399,
+   "s0_con": 0.0207,
+   "img_acc": 0.79611,
+   "s1_dE": 0.10924,
+   "s1_dL": 0.10592,
+   "s1_con": 0.22063
+  },
+  {
+   "epoch": 18,
+   "step": 8892,
+   "secs": 1296,
+   "loss": 0.12691,
+   "dE": 0.04379,
+   "con": 0.03646,
+   "img": 0.69736,
+   "s0_dE": 0.04175,
+   "s0_dL": 0.03993,
+   "s0_con": 0.02416,
+   "img_acc": 0.801,
+   "s1_dE": 0.10774,
+   "s1_dL": 0.10449,
+   "s1_con": 0.24435
+  },
+  {
+   "epoch": 19,
+   "step": 9360,
+   "secs": 1364,
+   "loss": 0.12499,
+   "dE": 0.04319,
+   "con": 0.036,
+   "img": 0.68816,
+   "s0_dE": 0.0413,
+   "s0_dL": 0.03945,
+   "s0_con": 0.02047,
+   "img_acc": 0.79813,
+   "s1_dE": 0.11372,
+   "s1_dL": 0.11037,
+   "s1_con": 0.20421
+  },
+  {
+   "epoch": 20,
+   "step": 9828,
+   "secs": 1432,
+   "loss": 0.12434,
+   "dE": 0.04334,
+   "con": 0.03581,
+   "img": 0.67546,
+   "s0_dE": 0.0409,
+   "s0_dL": 0.03905,
+   "s0_con": 0.01907,
+   "img_acc": 0.79483,
+   "s1_dE": 0.10692,
+   "s1_dL": 0.1036,
+   "s1_con": 0.22495
+  },
+  {
+   "epoch": 21,
+   "step": 10296,
+   "secs": 1499,
+   "loss": 0.12248,
+   "dE": 0.04287,
+   "con": 0.03581,
+   "img": 0.65995,
+   "s0_dE": 0.04105,
+   "s0_dL": 0.0392,
+   "s0_con": 0.02163,
+   "img_acc": 0.7999,
+   "s1_dE": 0.10461,
+   "s1_dL": 0.10127,
+   "s1_con": 0.18494
+  },
+  {
+   "epoch": 22,
+   "step": 10764,
+   "secs": 1567,
+   "loss": 0.12033,
+   "dE": 0.04222,
+   "con": 0.03395,
+   "img": 0.64253,
+   "s0_dE": 0.04075,
+   "s0_dL": 0.03893,
+   "s0_con": 0.02266,
+   "img_acc": 0.7999,
+   "s1_dE": 0.10178,
+   "s1_dL": 0.09855,
+   "s1_con": 0.23005
+  },
+  {
+   "epoch": 23,
+   "step": 11232,
+   "secs": 1635,
+   "loss": 0.11905,
+   "dE": 0.04192,
+   "con": 0.03691,
+   "img": 0.63201,
+   "s0_dE": 0.04024,
+   "s0_dL": 0.03843,
+   "s0_con": 0.01664,
+   "img_acc": 0.79049,
+   "s1_dE": 0.10458,
+   "s1_dL": 0.10125,
+   "s1_con": 0.1973
+  },
+  {
+   "epoch": 24,
+   "step": 11700,
+   "secs": 1703,
+   "loss": 0.11767,
+   "dE": 0.04169,
+   "con": 0.03462,
+   "img": 0.61761,
+   "s0_dE": 0.04043,
+   "s0_dL": 0.03861,
+   "s0_con": 0.02038,
+   "img_acc": 0.79379,
+   "s1_dE": 0.1025,
+   "s1_dL": 0.09926,
+   "s1_con": 0.2135
+  },
+  {
+   "epoch": 25,
+   "step": 12168,
+   "secs": 1770,
+   "loss": 0.11655,
+   "dE": 0.04131,
+   "con": 0.03564,
+   "img": 0.60858,
+   "s0_dE": 0.04068,
+   "s0_dL": 0.0389,
+   "s0_con": 0.01981,
+   "img_acc": 0.79715,
+   "s1_dE": 0.10372,
+   "s1_dL": 0.10038,
+   "s1_con": 0.20402
+  },
+  {
+   "epoch": 26,
+   "step": 12636,
+   "secs": 1838,
+   "loss": 0.11502,
+   "dE": 0.04106,
+   "con": 0.03415,
+   "img": 0.59825,
+   "s0_dE": 0.04032,
+   "s0_dL": 0.03852,
+   "s0_con": 0.01906,
+   "img_acc": 0.79691,
+   "s1_dE": 0.10293,
+   "s1_dL": 0.09969,
+   "s1_con": 0.22009
+  },
+  {
+   "epoch": 27,
+   "step": 13104,
+   "secs": 1906,
+   "loss": 0.11521,
+   "dE": 0.04109,
+   "con": 0.03603,
+   "img": 0.59315,
+   "s0_dE": 0.0404,
+   "s0_dL": 0.03861,
+   "s0_con": 0.02053,
+   "img_acc": 0.79721,
+   "s1_dE": 0.10302,
+   "s1_dL": 0.09978,
+   "s1_con": 0.2078
+  },
+  {
+   "epoch": 28,
+   "step": 13572,
+   "secs": 1973,
+   "loss": 0.11402,
+   "dE": 0.04082,
+   "con": 0.03683,
+   "img": 0.58722,
+   "s0_dE": 0.04034,
+   "s0_dL": 0.03856,
+   "s0_con": 0.01831,
+   "img_acc": 0.79098,
+   "s1_dE": 0.10383,
+   "s1_dL": 0.10053,
+   "s1_con": 0.181
+  },
+  {
+   "epoch": 29,
+   "step": 14040,
+   "secs": 2041,
+   "loss": 0.11321,
+   "dE": 0.04059,
+   "con": 0.03495,
+   "img": 0.5803,
+   "s0_dE": 0.04018,
+   "s0_dL": 0.03841,
+   "s0_con": 0.01978,
+   "img_acc": 0.79416,
+   "s1_dE": 0.10315,
+   "s1_dL": 0.09986,
+   "s1_con": 0.19971
+  }
+ ],
+ "best_score": 0.07564117624840601
+}

runtime/DARKREADER-LICENSE

@@ -0,0 +1,23 @@
+MIT License
+
+Copyright (c) 2026 Dark Reader Ltd.
+
+All rights reserved.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

runtime/extract.js

@@ -0,0 +1,825 @@
+// Nadir page analyzer. Shared by the training crawler (Playwright) and the browser extension.
+// Defines globalThis.__nadirExtract(opts) -> Promise<PageDescription>.
+// Everything the model sees is computed here, so training and inference features match exactly.
+(function () {
+  'use strict';
+  if (globalThis.__nadirExtract) return;
+
+  const ROLES = ['bg', 'text', 'border', 'shadow', 'gradient', 'fill', 'stroke'];
+  const R = Object.fromEntries(ROLES.map((r, i) => [r, i]));
+  const SKIP_TAGS = new Set(['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEMPLATE', 'HEAD', 'META', 'LINK', 'TITLE', 'BR', 'WBR', 'SOURCE', 'TRACK', 'PARAM']);
+  const INTERACTIVE = new Set(['A', 'BUTTON', 'INPUT', 'SELECT', 'TEXTAREA', 'SUMMARY', 'LABEL']);
+  const FORM = new Set(['INPUT', 'SELECT', 'TEXTAREA']);
+  const HEADINGS = new Set(['H1', 'H2', 'H3', 'H4', 'H5', 'H6']);
+  const CODE = new Set(['CODE', 'PRE', 'KBD', 'SAMP']);
+  const SVG_SHAPES = new Set(['path', 'circle', 'rect', 'ellipse', 'line', 'polyline', 'polygon', 'text', 'tspan', 'use', 'g', 'svg']);
+
+  // ---------- color parsing ----------
+  const colorCache = new Map();
+  let cctx = null;
+  const BAD_COLOR = /var\(|env\(|attr\(|calc\(|^(inherit|initial|unset|revert|revert-layer|currentcolor|auto|none|transparent|-webkit-[\w-]+|-moz-[\w-]+)$/i;
+  function splitTop(s) { // split on top-level commas
+    const out = []; let depth = 0, cur = '';
+    for (const ch of s) { if (ch === '(') depth++; else if (ch === ')') depth--; if (ch === ',' && depth === 0) { out.push(cur); cur = ''; } else cur += ch; }
+    out.push(cur); return out.map(x => x.trim());
+  }
+  function canvasParse(str) {
+    const ld = str.match(/^light-dark\((.*)\)$/i);
+    if (ld) { const parts = splitTop(ld[1]); return parts.length === 2 ? parseColor(parts[0]) : null; }
+    if (BAD_COLOR.test(str)) return null;
+    if (typeof CSS !== 'undefined' && CSS.supports && !CSS.supports('color', str)) return null;
+    if (!cctx) {
+      const c = (typeof OffscreenCanvas !== 'undefined') ? new OffscreenCanvas(1, 1) : Object.assign(document.createElement('canvas'), {width: 1, height: 1});
+      cctx = c.getContext('2d', {willReadFrequently: true});
+    }
+    cctx.fillStyle = '#010203';
+    const before = cctx.fillStyle;
+    cctx.fillStyle = str;
+    if (cctx.fillStyle === before && !/^#?010203/i.test(str)) return null; // canvas rejected it
+    cctx.clearRect(0, 0, 1, 1);
+    cctx.fillRect(0, 0, 1, 1);
+    const d = cctx.getImageData(0, 0, 1, 1).data;
+    return [d[0], d[1], d[2], d[3] / 255];
+  }
+  function parseColor(str) {
+    if (!str) return null;
+    let c = colorCache.get(str);
+    if (c !== undefined) return c;
+    c = null;
+    const s = str.trim();
+    if (s === 'transparent' || s === 'none' || s === '' || s.startsWith('url(') || s === 'currentcolor') c = null;
+    else {
+      const m = s.match(/^rgba?\(\s*([\d.]+)[\s,]+([\d.]+)[\s,]+([\d.]+)(?:\s*[,/]\s*([\d.]+%?))?\s*\)$/);
+      if (m) {
+        let a = m[4] === undefined ? 1 : (m[4].endsWith('%') ? parseFloat(m[4]) / 100 : parseFloat(m[4]));
+        c = [Math.round(+m[1]), Math.round(+m[2]), Math.round(+m[3]), a];
+      } else {
+        try { c = canvasParse(s); } catch (e) { c = null; }
+      }
+      if (c && c[3] <= 0.004) c = null;
+    }
+    colorCache.set(str, c);
+    return c;
+  }
+  // all colors inside a composite value (box-shadow, gradients)
+  const COLOR_RE = /(rgba?\([^)]*\)|hsla?\([^)]*\)|oklch\([^)]*\)|oklab\([^)]*\)|lab\([^)]*\)|lch\([^)]*\)|color\([^)]*\)|#[0-9a-fA-F]{3,8}\b)/g;
+  function colorsIn(str) {
+    if (!str || str === 'none') return [];
+    const out = [];
+    for (const m of str.matchAll(COLOR_RE)) { const c = parseColor(m[1]); if (c) out.push(c); }
+    return out;
+  }
+  const hex2 = (n) => n.toString(16).padStart(2, '0');
+  const keyOf = (c) => hex2(c[0]) + hex2(c[1]) + hex2(c[2]) + hex2(Math.round(c[3] * 255));
+
+  // sRGB -> OKLab
+  function lin(u) { u /= 255; return u <= 0.04045 ? u / 12.92 : Math.pow((u + 0.055) / 1.055, 2.4); }
+  function oklab(c) {
+    const r = lin(c[0]), g = lin(c[1]), b = lin(c[2]);
+    const l = Math.cbrt(0.4122214708 * r + 0.5363325363 * g + 0.0514459929 * b);
+    const m = Math.cbrt(0.2119034982 * r + 0.6806995451 * g + 0.1073969566 * b);
+    const s = Math.cbrt(0.0883024619 * r + 0.2817188376 * g + 0.6299787005 * b);
+    return [0.2104542553 * l + 0.7936177850 * m - 0.0040720468 * s,
+      1.9779984951 * l - 2.4285922050 * m + 0.4505937099 * s,
+      0.0259040371 * l + 0.7827717662 * m - 0.8086757660 * s];
+  }
+  function relLum(c) { return 0.2126 * lin(c[0]) + 0.7152 * lin(c[1]) + 0.0722 * lin(c[2]); }
+  function over(top, bottom) { // composite rgba over opaque rgb
+    const a = top[3];
+    return [top[0] * a + bottom[0] * (1 - a), top[1] * a + bottom[1] * (1 - a), top[2] * a + bottom[2] * (1 - a), 1];
+  }
+
+  // ---------- dark-mode support scan ----------
+  const DARK_SEL_RE = /(?:^|[\s,>+~(])((?:html|body|:root)?(?:\.(?:[\w-]*-)?dark(?:-[\w-]+)?|\.(?:[\w-]*-)?night[\w-]*|\[data-[\w-]*(?:theme|mode|scheme|color-mode|appearance)[\w-]*(?:[~|^*$]?=\s*["']?[\w-]*dark[\w-]*["']?)\s*(?:i\s*)?\]|\[(?:theme|mode)\s*=\s*["']?dark["']?\]|\[dark(?:-theme|-mode)?\]))/i;
+  function scanSheets(limitRules) {
+    const res = {sheets: 0, blocked: 0, rules: 0, darkMediaRules: 0, darkMediaColorDecls: 0, lightMediaRules: 0,
+      darkSelectors: {}, ruleColors: [], hoverColors: 0};
+    let budget = limitRules;
+    const seenRuleColors = new Map();
+    const pushRuleColor = (role, c, state) => {
+      const k = role + ':' + keyOf(c);
+      const e = seenRuleColors.get(k);
+      if (e) { e.n++; if (state) e.state++; return; }
+      seenRuleColors.set(k, {role, c, n: 1, state: state ? 1 : 0});
+    };
+    function walk(rules, inDarkMedia, inLightMedia) {
+      for (const rule of rules) {
+        if (budget-- <= 0) return;
+        if (rule.cssRules && rule.media) {
+          const mt = rule.media.mediaText || '';
+          const dark = /prefers-color-scheme\s*:\s*dark/i.test(mt);
+          const light = /prefers-color-scheme\s*:\s*light/i.test(mt);
+          if (dark) res.darkMediaRules++;
+          if (light) res.lightMediaRules++;
+          walk(rule.cssRules, inDarkMedia || dark, inLightMedia || light);
+          continue;
+        }
+        if (rule.cssRules && !rule.style) { walk(rule.cssRules, inDarkMedia, inLightMedia); continue; }
+        if (!rule.style) continue;
+        res.rules++;
+        const st = rule.style;
+        let colorDecls = 0;
+        const sel = rule.selectorText || '';
+        const state = /:(hover|focus|active|focus-visible|focus-within|checked)/.test(sel);
+        for (let i = 0; i < st.length; i++) {
+          const p = st[i];
+          const v = st.getPropertyValue(p);
+          if (!v) continue;
+          let role = -1;
+          if (p === 'background-color' || p === 'background') role = R.bg;
+          else if (p === 'color' || p === '-webkit-text-fill-color') role = R.text;
+          else if (p.startsWith('border') && p.endsWith('color') || p === 'outline-color') role = R.border;
+          else if (p === 'fill') role = R.fill;
+          else if (p === 'stroke') role = R.stroke;
+          else if (p === 'box-shadow' || p === 'text-shadow') role = R.shadow;
+          else if (p === 'background-image') role = R.gradient;
+          else if (p.startsWith('--')) { if (COLOR_RE.test(v)) colorDecls++; COLOR_RE.lastIndex = 0; continue; }
+          if (role < 0) continue;
+          colorDecls++;
+          if (inDarkMedia || inLightMedia) continue; // light page colors only
+          const cs = (role === R.shadow || role === R.gradient || p === 'background') ? colorsIn(v) : [parseColor(v)].filter(Boolean);
+          for (const c of cs) { pushRuleColor(role === R.gradient && p === 'background' ? R.bg : role, c, state); if (state) res.hoverColors++; }
+        }
+        if (inDarkMedia) res.darkMediaColorDecls += colorDecls;
+        if (colorDecls && sel) {
+          const m = sel.match(DARK_SEL_RE);
+          if (m) {
+            const k = m[1].trim();
+            res.darkSelectors[k] = (res.darkSelectors[k] || 0) + colorDecls;
+          }
+        }
+      }
+    }
+    const sheets = [...document.styleSheets, ...(document.adoptedStyleSheets || []), ...(scanSheets.extra || [])];
+    for (const sh of sheets) {
+      if (scanSheets.filter && !scanSheets.filter(sh)) continue;
+      res.sheets++;
+      let rules;
+      try { rules = sh.cssRules; } catch (e) { res.blocked++; continue; }
+      if (!rules) { res.blocked++; continue; }
+      walk(rules, sh.media && /prefers-color-scheme\s*:\s*dark/i.test(sh.media.mediaText || ''), false);
+    }
+    res.ruleColors = [...seenRuleColors.values()];
+    return res;
+  }
+
+  function schemeInfo(sheetScan) {
+    const meta = document.querySelector('meta[name="color-scheme"]');
+    const rootCS = getComputedStyle(document.documentElement).colorScheme || '';
+    const darkLinks = [...document.querySelectorAll('link[rel~="stylesheet"][media]')].filter(l => /prefers-color-scheme\s*:\s*dark/i.test(l.media)).length;
+    const darkSources = [...document.querySelectorAll('picture source[media]')].filter(s => /prefers-color-scheme\s*:\s*dark/i.test(s.media)).length;
+    const html = document.documentElement, body = document.body;
+    const attrs = {};
+    for (const el of [html, body]) {
+      if (!el) continue;
+      for (const a of el.attributes) if (/theme|mode|scheme|appearance|dark|light/i.test(a.name + '=' + a.value) && a.name !== 'style') attrs[el.tagName.toLowerCase() + '@' + a.name] = a.value.slice(0, 80);
+    }
+    const sel = Object.entries(sheetScan.darkSelectors).sort((a, b) => b[1] - a[1]).slice(0, 12);
+    return {
+      metaColorScheme: meta ? meta.content : null,
+      rootColorScheme: rootCS,
+      darkMediaRules: sheetScan.darkMediaRules,
+      darkMediaColorDecls: sheetScan.darkMediaColorDecls,
+      lightMediaRules: sheetScan.lightMediaRules,
+      darkLinks, darkSources,
+      darkSelectors: sel,
+      rootAttrs: attrs,
+      rootClass: (html.className && typeof html.className === 'string' ? html.className : '').slice(0, 200),
+      bodyClass: (body && typeof body.className === 'string' ? body.className : '').slice(0, 200),
+      sheets: sheetScan.sheets, blockedSheets: sheetScan.blocked, rules: sheetScan.rules,
+    };
+  }
+
+  // ---------- ads ----------
+  // Ad slots are ignored entirely, so a page looks the same to the model with or without an ad blocker.
+  const AD_TOKEN = /^(ad|ads|adv|advert|adverts|advertisement|advertising|adsbygoogle|adslot|adunit|adbox|adblock|adbanner|ad[-_](slot|unit|container|wrapper|banner|box|block|placeholder|label|space|zone|frame|holder|area|leaderboard|rail|sidebar|top|bottom|inline|native|mpu|billboard|skyscraper|sticky)|(top|side|bottom|inline|sticky|native|header|footer|right|left)[-_]?ads?|banner[-_]?ads?|gpt[-_]ad.*|dfp[-_]?ad.*|google[-_]ads?.*|taboola.*|trc_.*|outbrain.*|ob[-_]widget.*|sponsored([-_].*)?|promoted[-_]?(content|post|ad)s?|mgid.*|revcontent.*|teads.*|criteo.*|amzn[-_]?ad.*|adthrive.*|ezoic.*|mediavine.*)$/i;
+  const AD_ID = /^(google_ads|div-gpt-ad|gpt-ad|ad[-_]|ads[-_]|taboola|outbrain|adthrive|ezoic-pub-ad|mgid|revcontent)/i;
+  const AD_SRC = /(doubleclick\.net|googlesyndication\.com|googleadservices\.com|adservice\.google|amazon-adsystem\.com|adnxs\.com|taboola\.com|outbrain\.com|criteo\.|pubmatic\.com|rubiconproject\.com|openx\.net|adsrvr\.org|moatads\.com|mgid\.com|revcontent\.com|teads\.tv|media\.net\/|ezoic|adthrive|mediavine)/i;
+  function isAd(el) {
+    const tag = el.tagName;
+    if (tag === 'INS' && /adsbygoogle/.test(el.className || '')) return true;
+    if ((tag === 'IFRAME' || tag === 'IMG' || tag === 'A') && AD_SRC.test(el.getAttribute('src') || el.getAttribute('href') || '')) return true;
+    if (el.hasAttribute('data-ad-slot') || el.hasAttribute('data-google-query-id') || el.hasAttribute('data-ad-unit') || el.hasAttribute('data-adunit') || el.hasAttribute('data-ad-client')) return true;
+    const id = el.id;
+    if (id && typeof id === 'string' && AD_ID.test(id)) return true;
+    const cls = typeof el.className === 'string' ? el.className : '';
+    if (cls) { for (const c of cls.split(/\s+/)) if (c && AD_TOKEN.test(c)) return true; }
+    const al = el.getAttribute('aria-label');
+    if (al && /^(advertisement|ad|sponsored|anzeige|publicidad|publicité|pubblicità|реклама|広告)$/i.test(al.trim())) return true;
+    return false;
+  }
+
+  // ---------- element walk ----------
+  function describe(el) {
+    let s = el.tagName.toLowerCase();
+    if (el.id && typeof el.id === 'string') s += '#' + el.id.slice(0, 24);
+    const cls = typeof el.className === 'string' ? el.className : (el.className && el.className.baseVal) || '';
+    if (cls) s += '.' + cls.trim().split(/\s+/).slice(0, 3).join('.').slice(0, 48);
+    return s;
+  }
+  function ownText(el) {
+    let t = '';
+    for (const n of el.childNodes) if (n.nodeType === 3) t += n.nodeValue;
+    t = t.replace(/\s+/g, ' ').trim();
+    if (!t && FORM.has(el.tagName)) {
+      const v = el.value || el.placeholder || '';
+      if (el.type !== 'hidden' && el.type !== 'checkbox' && el.type !== 'radio') t = String(v).slice(0, 60) || ' ';
+    }
+    return t;
+  }
+
+  async function extract(opts) {
+    opts = Object.assign({maxElements: 5000, maxRules: 40000, maxTokens: 384, maxImages: 48, describe: false,
+      thumbs: false, thumbSize: 32, bigThumbSize: 0, fetchImage: null, pseudo: true, time: true}, opts || {});
+    const t0 = performance.now();
+    const vw = innerWidth, vh = innerHeight;
+    const sx = scrollX, sy = scrollY;
+    const docEl = document.documentElement;
+    const docH = Math.max(docEl.scrollHeight, document.body ? document.body.scrollHeight : 0, vh);
+    const docW = Math.max(vw, Math.min(docEl.scrollWidth, vw * 1.5));
+    const pageArea = docW * Math.min(docH, vh * 8);
+    scanSheets.filter = opts.sheetFilter || null;
+    scanSheets.extra = opts.extraSheets || [];
+    const sheetScan = scanSheets(opts.maxRules);
+
+    // canvas default background (white unless the page opts into dark color-scheme)
+    const rootCS = getComputedStyle(docEl);
+    const canvasDark = /dark/.test(rootCS.colorScheme) && !/light/.test(rootCS.colorScheme);
+    const canvasColor = canvasDark ? [18, 18, 18, 1] : [255, 255, 255, 1];
+
+    const tokens = new Map(); // key role:hex -> token
+    function tok(role, c) {
+      const k = role + ':' + keyOf(c);
+      let t = tokens.get(k);
+      if (!t) {
+        t = {k, role, c, i: -1, area: 0, area1: 0, n: 0, nVis: 0, chars: 0, fsSum: 0, fsMax: 0, bold: 0, root: 0,
+          inter: 0, link: 0, head: 0, code: 0, form: 0, header: 0, footer: 0, nav: 0, under: [0, 0, 0, 0], over: [0, 0, 0, 0],
+          contrastSum: 0, contrastW: 0, rules: 0, state: 0, depthSum: 0, ex: [], xs: 0, ys: 0, ws: 0, hs: 0, imgUnder: 0, fixed: 0, media: 0};
+        tokens.set(k, t);
+      }
+      return t;
+    }
+    const rel = new Map(); // "type|a|b" -> weight
+    function addRel(type, a, b, w) {
+      if (!a || !b || a === b) return;
+      const k = type + '|' + a.k + '|' + b.k;
+      rel.set(k, (rel.get(k) || 0) + w);
+    }
+
+    // walk DOM (including open shadow roots) in document order
+    let adsSkipped = 0;
+    const els = [];
+    const stack = document.body ? [document.body] : [];
+    const deferred = [];
+    const parentOf = new Map();
+    while ((stack.length || deferred.length) && els.length < opts.maxElements) {
+      const el = stack.length ? stack.pop() : deferred.shift();
+      els.push(el);
+      if (el !== document.body && el.getClientRects().length === 0 && !(el instanceof SVGElement)) {
+        // hidden subtree (display:none etc.): visit later, only if budget remains
+        const kids = [...el.children].filter(k => !SKIP_TAGS.has(k.tagName) && !isAd(k));
+        for (const k of kids) { parentOf.set(k, el); deferred.push(k); }
+        continue;
+      }
+      const kids = [];
+      const root = el.shadowRoot || null;
+      if (root) for (const k of root.children) kids.push(k);
+      for (const k of el.children) kids.push(k);
+      for (let i = kids.length - 1; i >= 0; i--) {
+        const k = kids[i];
+        if (SKIP_TAGS.has(k.tagName)) continue;
+        if (isAd(k)) { adsSkipped++; continue; }
+        parentOf.set(k, el);
+        stack.push(k);
+      }
+    }
+
+    // media (video players, embeds, canvases, large images): overlays on these must keep their colours
+    const media = [];
+    for (const m of document.querySelectorAll('video, iframe, canvas, embed, object, img')) {
+      if (m.tagName === 'IFRAME' && AD_SRC.test(m.src || '')) continue;
+      const r = m.getBoundingClientRect();
+      if (r.width * r.height < (m.tagName === 'IMG' ? 40000 : 6000)) continue;
+      media.push({m, x: r.left + sx, y: r.top + sy, w: r.width, h: r.height});
+      if (media.length > 64) break;
+    }
+    const onMedia = (el, x, y, w, h) => {
+      if (!media.length || w * h <= 0) return 0;
+      for (const md of media) {
+        if (md.m === el) continue;
+        if (el.contains(md.m) && w * h < md.w * md.h * 2.5) return 1;   // player/media container
+        const ix = Math.max(0, Math.min(x + w, md.x + md.w) - Math.max(x, md.x));
+        const iy = Math.max(0, Math.min(y + h, md.y + md.h) - Math.max(y, md.y));
+        if (ix * iy >= 0.6 * w * h) return 1;                            // drawn on top of media
+      }
+      return 0;
+    };
+    // per-element info
+    const info = new Map();
+    const htmlBg = parseColor(rootCS.backgroundColor);
+    const bodyCS = document.body ? getComputedStyle(document.body) : null;
+    const bodyBg = bodyCS ? parseColor(bodyCS.backgroundColor) : null;
+    // effective root surface: body bg propagates to canvas if html has none
+    let rootSurface = htmlBg ? over(htmlBg, canvasColor) : (bodyBg ? over(bodyBg, canvasColor) : canvasColor);
+    const rootTok = tok(R.bg, htmlBg || bodyBg || canvasColor);
+    rootTok.root = 1;
+    rootTok.n++; rootTok.nVis++;
+    const rootRec = {surf: rootSurface, surfTok: rootTok, imgBg: false, region: 0, depth: 0, fixed: false};
+    const images = [];
+    const svgRoots = [];
+    let visibleCount = 0;
+
+    const ec = opts.elemColors ? [] : null;
+    for (const el of els) {
+      const cs = getComputedStyle(el);
+      const ecRow = ec ? [el.tagName, 0, null, null, null, null, null, null, null] : null;
+      if (ec) ec.push(ecRow);
+      const parent = parentOf.get(el);
+      const p = parent ? info.get(parent) : rootRec;
+      const pr = p || rootRec;
+      if (cs.display === 'none') { info.set(el, Object.assign({}, pr, {hidden: true})); }
+      const hidden = cs.display === 'none' || (pr && pr.hidden) || cs.visibility === 'hidden' || cs.opacity === '0';
+      const rect = el.getBoundingClientRect();
+      const x = rect.left + sx, y = rect.top + sy;
+      const w = Math.max(0, Math.min(rect.width, docW - Math.max(0, x)));
+      const h = Math.max(0, rect.height);
+      const vis = !hidden && w > 1 && h > 1 && x + w > 0 && y + h > 0;
+      const area = vis ? Math.min(w * h, docW * docH) : 0;
+      const area1 = vis ? Math.max(0, Math.min(x + w, vw) - Math.max(x, 0)) * Math.max(0, Math.min(y + h, vh) - Math.max(y, 0)) : 0;
+      if (vis) visibleCount++;
+      if (ecRow) ecRow[1] = Math.round(area);
+      const tag = el.tagName;
+      const isSvg = el instanceof SVGElement;
+      // regions
+      let region = pr.region;
+      if (tag === 'HEADER' || el.getAttribute && el.getAttribute('role') === 'banner') region |= 1;
+      if (tag === 'NAV' || el.getAttribute && el.getAttribute('role') === 'navigation') region |= 2;
+      if (tag === 'FOOTER' || el.getAttribute && el.getAttribute('role') === 'contentinfo') region |= 4;
+      const fixed = pr.fixed || cs.position === 'fixed' || cs.position === 'sticky';
+      const om = vis ? onMedia(el, x, y, w, h) : 0;
+      const rec = {surf: pr.surf, surfTok: pr.surfTok, imgBg: pr.imgBg, region, depth: pr.depth + 1, hidden: hidden, fixed, om,
+        inLink: pr.inLink || tag === 'A', link: tag === 'A' ? el : pr.link};
+      info.set(el, rec);
+      const flags = (t) => {
+        t.n++;
+        if (vis) { t.nVis++; t.depthSum += rec.depth; }
+        if (INTERACTIVE.has(tag) || el.getAttribute && el.getAttribute('role') === 'button') t.inter++;
+        if (rec.inLink) t.link++;
+        if (HEADINGS.has(tag)) t.head++;
+        if (CODE.has(tag)) t.code++;
+        if (FORM.has(tag)) t.form++;
+        if (region & 1) t.header++;
+        if (region & 2) t.nav++;
+        if (region & 4) t.footer++;
+        if (fixed) t.fixed++;
+        if (om) t.media++;
+        if (opts.describe && t.ex.length < 3 && vis) {
+          const txt = ownText(el);
+          t.ex.push(describe(el) + (txt ? ' "' + txt.slice(0, 40) + '"' : ''));
+        }
+      };
+
+      // background
+      const bg = parseColor(cs.backgroundColor);
+      const bgImg = cs.backgroundImage;
+      if (bg) {
+        const t = tok(R.bg, bg);
+        if (ecRow) ecRow[2] = keyOf(bg);
+        flags(t);
+        t.area += area; t.area1 += area1;
+        if (vis) { t.xs += x * area; t.ys += y * area; t.ws += w * area; t.hs += h * area; }
+        const under = pr.surf;
+        if (vis) { addRel('bb', t, pr.surfTok, area); }
+        const ul = oklab(under);
+        t.under[0] += ul[0] * (area + 1); t.under[1] += ul[1] * (area + 1); t.under[2] += ul[2] * (area + 1); t.under[3] += area + 1;
+        if (pr.imgBg) t.imgUnder += 1;
+        rec.surf = over(bg, under);
+        rec.surfTok = t;
+        rec.surfEl = el;
+        rec.ownArea = area;
+      }
+      if (bgImg && bgImg !== 'none') {
+        if (bgImg.includes('gradient')) {
+          if (ecRow) ecRow[7] = colorsIn(bgImg).map(keyOf).join(',');
+          for (const c of colorsIn(bgImg)) { const t = tok(R.gradient, c); flags(t); t.area += area / 2; t.area1 += area1 / 2; }
+          const gc = colorsIn(bgImg);
+          if (gc.length) {
+            let avg = [0, 0, 0, 1];
+            for (const c of gc) { const o = over(c, pr.surf); avg[0] += o[0] / gc.length; avg[1] += o[1] / gc.length; avg[2] += o[2] / gc.length; }
+            rec.surf = avg; rec.imgBg = false;
+          }
+        }
+        const urls = [...bgImg.matchAll(/url\(\s*["']?([^"')]+)["']?\s*\)/g)].map(m => m[1]);
+        if (urls.length && vis && area > 64) {
+          images.push({el, kind: 'bg', src: urls[0], x, y, w, h, area, area1, rec});
+          if (area > 0.02 * vw * vh) rec.imgBg = true;
+        }
+      }
+      if (ecRow) ecRow[8] = keyOf(rec.surf.map((v, i) => i < 3 ? Math.round(v) : 1));
+      // text
+      const txt = ownText(el);
+      if (txt) {
+        const col = parseColor(cs.color);
+        if (col) {
+          const t = tok(R.text, col);
+          if (ecRow) ecRow[3] = keyOf(col);
+          flags(t);
+          const n = txt.length;
+          t.chars += vis ? n : 0;
+          const fs = parseFloat(cs.fontSize) || 16;
+          t.fsSum += fs * n; t.fsMax = Math.max(t.fsMax, vis ? fs : 0);
+          if ((parseInt(cs.fontWeight) || 400) >= 600) t.bold += n;
+          t.area += vis ? Math.min(area, n * fs * fs * 0.5) : 0;
+          t.area1 += area1 ? Math.min(area1, n * fs * fs * 0.5) : 0;
+          const under = rec.surf;
+          const ul = oklab(under);
+          const wgt = vis ? n : 0.1;
+          t.under[0] += ul[0] * wgt; t.under[1] += ul[1] * wgt; t.under[2] += ul[2] * wgt; t.under[3] += wgt;
+          const fg = over(col, under);
+          const l1 = relLum(fg), l2 = relLum(under);
+          t.contrastSum += ((Math.max(l1, l2) + 0.05) / (Math.min(l1, l2) + 0.05)) * wgt; t.contrastW += wgt;
+          if (rec.imgBg) t.imgUnder += 1;
+          if (vis) addRel('tb', t, rec.surfTok, n);
+          // reverse: text sitting on this surface
+          const st = rec.surfTok;
+          const tl = oklab(col);
+          st.over[0] += tl[0] * wgt; st.over[1] += tl[1] * wgt; st.over[2] += tl[2] * wgt; st.over[3] += wgt;
+          if (bg) addRel('se', t, tok(R.bg, bg), n);
+        }
+      }
+      // borders
+      if (!isSvg) {
+        const seen = new Set();
+        for (const side of ['Top', 'Right', 'Bottom', 'Left']) {
+          const bw = parseFloat(cs['border' + side + 'Width']);
+          if (!bw || cs['border' + side + 'Style'] === 'none' || cs['border' + side + 'Style'] === 'hidden') continue;
+          const bc = parseColor(cs['border' + side + 'Color']);
+          if (!bc) continue;
+          const k = keyOf(bc);
+          const len = (side === 'Top' || side === 'Bottom') ? w : h;
+          if (seen.has(k)) { const t = tok(R.border, bc); t.area += vis ? bw * len : 0; continue; }
+          seen.add(k);
+          if (ecRow && !ecRow[4]) ecRow[4] = k;
+          const t = tok(R.border, bc);
+          flags(t);
+          t.area += vis ? bw * len : 0;
+          const ul = oklab(pr.surf);
+          t.under[0] += ul[0]; t.under[1] += ul[1]; t.under[2] += ul[2]; t.under[3] += 1;
+          if (vis) addRel('rb', t, pr.surfTok, bw * len);
+          if (bg) addRel('se', t, tok(R.bg, bg), 1);
+        }
+        const ow = parseFloat(cs.outlineWidth);
+        if (ow && cs.outlineStyle !== 'none') {
+          const oc = parseColor(cs.outlineColor);
+          if (oc) { const t = tok(R.border, oc); flags(t); t.area += vis ? ow * 2 * (w + h) : 0; }
+        }
+        // shadows
+        for (const sh of [cs.boxShadow, cs.textShadow]) {
+          if (!sh || sh === 'none') continue;
+          for (const c of colorsIn(sh)) { const t = tok(R.shadow, c); flags(t); t.area += vis ? Math.min(area, 2 * (w + h) * 4) : 0;
+            const ul = oklab(pr.surf); t.under[0] += ul[0]; t.under[1] += ul[1]; t.under[2] += ul[2]; t.under[3] += 1; }
+        }
+      }
+      // svg paint
+      if (isSvg) {
+        const ltag = tag.toLowerCase();
+        if (ltag === 'svg' && !(parent instanceof SVGElement) && vis && area >= 400) {
+          const meta = ((el.getAttribute('class') || '') + ' ' + (el.id || '') + ' ' + (el.getAttribute('aria-label') || '') + ' ' + ((el.querySelector('title') || {}).textContent || '')).toLowerCase();
+          const useExt = [...el.querySelectorAll('use')].some(u => { const h = u.getAttribute('href') || u.getAttribute('xlink:href') || ''; return h && !h.startsWith('#'); });
+          if (rec.inLink || (region & 1) || /logo|brand|wordmark/.test(meta) || useExt || area >= 4000)
+            images.push({el, kind: 'svg', src: '', x, y, w, h, area, area1, rec, svgMeta: meta, useExt});
+        }
+        if (SVG_SHAPES.has(ltag) && ltag !== 'svg' && ltag !== 'g') {
+          const fc = cs.fill && cs.fill.startsWith('url') ? null : parseColor(cs.fill);
+          const sc = cs.stroke && cs.stroke.startsWith('url') ? null : parseColor(cs.stroke);
+          const sw = parseFloat(cs.strokeWidth) || 0;
+          const fo = parseFloat(cs.fillOpacity); const so = parseFloat(cs.strokeOpacity);
+          if (ecRow && fc) ecRow[5] = keyOf(fc);
+          if (ecRow && sc) ecRow[6] = keyOf(sc);
+          if (fc && fo !== 0) { const t = tok(R.fill, fc); flags(t); t.area += area * 0.5; t.area1 += area1 * 0.5;
+            const ul = oklab(rec.surf); t.under[0] += ul[0]; t.under[1] += ul[1]; t.under[2] += ul[2]; t.under[3] += 1; if (vis) addRel('tb', t, rec.surfTok, Math.sqrt(area + 1)); }
+          if (sc && sw > 0 && so !== 0) { const t = tok(R.stroke, sc); flags(t); t.area += vis ? sw * (w + h) : 0;
+            const ul = oklab(rec.surf); t.under[0] += ul[0]; t.under[1] += ul[1]; t.under[2] += ul[2]; t.under[3] += 1; if (vis) addRel('tb', t, rec.surfTok, Math.sqrt(area + 1)); }
+        }
+      }
+      // images
+      if (tag === 'IMG' && vis && area > 16) {
+        images.push({el, kind: 'img', src: el.currentSrc || el.src, x, y, w, h, area, area1, rec});
+      } else if ((tag === 'IMAGE' || tag === 'image') && vis) {
+        const href = el.getAttribute('href') || el.getAttribute('xlink:href');
+        if (href) images.push({el, kind: 'svgimage', src: new URL(href, location.href).href, x, y, w, h, area, area1, rec});
+      }
+      // pseudo elements
+      if (opts.pseudo && vis && !isSvg && els.length < 20000) {
+        for (const pseudo of ['::before', '::after']) {
+          const ps = getComputedStyle(el, pseudo);
+          const ct = ps.content;
+          if (!ct || ct === 'none' || ct === 'normal') continue;
+          const pb = parseColor(ps.backgroundColor);
+          const pw = parseFloat(ps.width) || 0, ph = parseFloat(ps.height) || 0;
+          const parea = Math.min(area, pw * ph);
+          if (pb) { const t = tok(R.bg, pb); flags(t); t.area += parea; addRel('bb', t, rec.surfTok, parea); }
+          if (ct !== '""' && ct !== "''") {
+            const pc = parseColor(ps.color);
+            if (pc) { const t = tok(R.text, pc); flags(t); t.chars += 1; const ul = oklab(rec.surf); t.under[0] += ul[0]; t.under[1] += ul[1]; t.under[2] += ul[2]; t.under[3] += 1; addRel('tb', t, rec.surfTok, 1); }
+          }
+          const bw = parseFloat(ps.borderTopWidth) || parseFloat(ps.borderBottomWidth) || parseFloat(ps.borderLeftWidth);
+          if (bw) { const bc = parseColor(ps.borderTopColor !== 'rgba(0, 0, 0, 0)' ? ps.borderTopColor : ps.borderBottomColor); if (bc) { const t = tok(R.border, bc); flags(t); t.area += bw * Math.max(pw, ph); } }
+          const psh = ps.boxShadow;
+          if (psh && psh !== 'none') for (const c of colorsIn(psh)) { const t = tok(R.shadow, c); flags(t); }
+          if (ps.backgroundImage && ps.backgroundImage.includes('gradient')) for (const c of colorsIn(ps.backgroundImage)) { const t = tok(R.gradient, c); flags(t); t.area += parea / 2; }
+        }
+      }
+    }
+
+    // own area for surfaces: subtract direct child surfaces
+    // (approximation: walk surfaces; area already includes descendants)
+    const childSurfArea = new Map();
+    for (const el of els) {
+      const r = info.get(el);
+      if (!r || !r.surfEl || r.surfEl !== el || !r.ownArea) continue;
+      // find nearest ancestor surface element
+      let p = parentOf.get(el);
+      while (p && !(info.get(p) && info.get(p).surfEl === p)) p = parentOf.get(p);
+      const key = p || 'root';
+      childSurfArea.set(key, (childSurfArea.get(key) || 0) + r.ownArea);
+    }
+    const ownAreaTok = new Map();
+    let rootOwn = Math.max(0, docW * docH - (childSurfArea.get('root') || 0));
+    ownAreaTok.set(rootTok, rootOwn);
+    for (const el of els) {
+      const r = info.get(el);
+      if (!r || r.surfEl !== el || !r.ownArea) continue;
+      const own = Math.max(0, r.ownArea - (childSurfArea.get(el) || 0));
+      ownAreaTok.set(r.surfTok, (ownAreaTok.get(r.surfTok) || 0) + own);
+    }
+
+    // rule-only colors
+    for (const rc of sheetScan.ruleColors) {
+      const t = tok(rc.role, rc.c);
+      t.rules += rc.n; t.state += rc.state;
+    }
+    // root token area = canvas own area
+    rootTok.area += rootOwn;
+
+    // ---------- darkness measurement ----------
+    let bgLsum = 0, bgW = 0, darkW = 0;
+    for (const [t, own] of ownAreaTok) {
+      if (t.role !== R.bg) continue;
+      const eff = over(t.c, t.root ? canvasColor : (t.under[3] ? labToApproxRgb(t.under) : canvasColor));
+      const L = oklab(eff)[0];
+      bgLsum += L * own; bgW += own; if (L < 0.45) darkW += own;
+    }
+    let txLsum = 0, txW = 0;
+    for (const t of tokens.values()) if (t.role === R.text && t.chars) { txLsum += oklab(t.c)[0] * t.chars; txW += t.chars; }
+    const darkness = {bgL: bgW ? bgLsum / bgW : 1, darkFrac: bgW ? darkW / bgW : 0, textL: txW ? txLsum / txW : 0, rootL: oklab(rootSurface)[0]};
+
+    // ---------- token selection & features ----------
+    let list = [...tokens.values()];
+    const score = (t) => (t.root ? 1e12 : 0) + Math.log1p(t.area) * 3 + Math.log1p(t.chars) * 2 + Math.log1p(t.nVis) * 2 + Math.log1p(t.n) + Math.log1p(t.rules) * 0.7 + Math.log1p(t.state);
+    list.sort((a, b) => score(b) - score(a));
+    const dropped = Math.max(0, list.length - opts.maxTokens);
+    list = list.slice(0, opts.maxTokens);
+    list.forEach((t, i) => t.i = i);
+
+    // bg lightness rank among visible bg tokens
+    const bgVis = list.filter(t => t.role === R.bg && t.area > 0).map(t => oklab(t.c)[0]).sort((a, b) => a - b);
+    const rankL = (L) => { if (!bgVis.length) return 0.5; let i = 0; while (i < bgVis.length && bgVis[i] < L) i++; return i / bgVis.length; };
+    const totalChars = list.reduce((s, t) => s + (t.role === R.text ? t.chars : 0), 0) || 1;
+
+    const out = [];
+    for (const t of list) {
+      const lab = oklab(t.c);
+      const C = Math.hypot(lab[1], lab[2]);
+      const under = t.under[3] ? [t.under[0] / t.under[3], t.under[1] / t.under[3], t.under[2] / t.under[3]] : oklab(canvasColor);
+      const overL = t.over[3] ? [t.over[0] / t.over[3], t.over[1] / t.over[3], t.over[2] / t.over[3]] : [0, 0, 0];
+      const n = Math.max(1, t.n);
+      const own = ownAreaTok.get(t) || 0;
+      const f = [
+        lab[0], lab[1] * 4, lab[2] * 4, C * 4, t.c[3], C > 0.02 ? lab[1] / C : 0, C > 0.02 ? lab[2] / C : 0,
+        t.c[0] / 255, t.c[1] / 255, t.c[2] / 255,
+        Math.log1p(t.area / 1000) / 10, Math.sqrt(Math.min(1, t.area / pageArea)), Math.sqrt(Math.min(1, t.area1 / (vw * vh))), Math.sqrt(Math.min(1, own / pageArea)),
+        Math.log1p(t.n) / 8, Math.log1p(t.nVis) / 8, Math.log1p(t.rules) / 6, t.rules ? t.state / (t.rules) : 0, (t.n === 0 && t.rules > 0) ? 1 : 0,
+        Math.log1p(t.chars) / 10, t.chars / totalChars, t.chars ? (t.fsSum / t.chars) / 32 : 0, t.fsMax / 48, t.chars ? t.bold / t.chars : 0,
+        t.root, t.inter / n, t.link / n, t.head / n, t.code / n, t.form / n, t.header / n, t.nav / n, t.footer / n, t.fixed / n,
+        under[0], under[1] * 4, under[2] * 4, t.contrastW ? Math.log(t.contrastSum / t.contrastW) / 3 : 0, Math.min(1, t.imgUnder / n),
+        overL[0], overL[1] * 4, overL[2] * 4, t.over[3] ? 1 : 0,
+        t.role === R.bg ? rankL(lab[0]) : 0, t.nVis ? Math.min(1, t.depthSum / t.nVis / 30) : 0,
+        t.ws && t.area ? Math.min(1, t.ws / t.area / docW) : 0, t.hs && t.area ? Math.min(1, t.hs / t.area / vh) : 0,
+        t.ys && t.area ? Math.min(1, t.ys / t.area / docH) : 0,
+        t.media / n,
+      ];
+      for (let q = 0; q < f.length; q++) if (!Number.isFinite(f[q])) f[q] = 0; // JSON turns NaN into null; never feed it to the model
+      const o = {k: t.k, role: t.role, rgba: [t.c[0], t.c[1], t.c[2], +t.c[3].toFixed(3)], f};
+      if (opts.describe) {
+        o.d = {area: Math.round(t.area), area1: Math.round(t.area1), own: Math.round(own), n: t.n, nVis: t.nVis, rules: t.rules, state: t.state, chars: t.chars,
+          fs: t.chars ? +(t.fsSum / t.chars).toFixed(1) : 0, root: t.root, inter: t.inter, link: t.link, head: t.head, header: t.header, nav: t.nav, footer: t.footer,
+          contrast: t.contrastW ? +(t.contrastSum / t.contrastW).toFixed(2) : null, media: t.media, ex: t.ex};
+      }
+      out.push(o);
+    }
+    const relOut = [];
+    for (const [k, w] of rel) {
+      const [type, a, b] = k.split('|');
+      const ta = tokens.get(a), tb = tokens.get(b);
+      if (!ta || !tb || ta.i < 0 || tb.i < 0) continue;
+      relOut.push([['tb', 'bb', 'rb', 'se'].indexOf(type), ta.i, tb.i, +Math.log1p(w).toFixed(3)]);
+    }
+
+    // ---------- images ----------
+    const imgSeen = new Map();
+    let svgCount = 0;
+    // logo-like images first, then by area
+    const logoish = (im) => (im.rec.link && im.rec.inLink ? 1 : 0) + ((im.rec.region & 1) ? 1 : 0) + (im.kind === 'svg' && /logo|brand/.test(im.svgMeta || '') ? 2 : 0);
+    images.sort((a, b) => (logoish(b) - logoish(a)) || (b.area - a.area));
+    const imgOut = [];
+    for (const im of images) {
+      if (imgOut.length >= opts.maxImages) break;
+      if (im.kind !== 'svg' && (!im.src || im.src.startsWith('data:image/gif;base64,R0lGOD'))) continue;
+      if (im.kind === 'svg') { im.src = 'svg:' + svgCount++; if (opts.markSvg) im.el.setAttribute('data-nadir-svg', im.src); }
+      const key = im.src;
+      if (imgSeen.has(key)) { const o = imgOut[imgSeen.get(key)]; o.count++; o.areaSum += im.area; continue; }
+      const el = im.el;
+      const a = im.rec.link;
+      const href = a ? (a.getAttribute('href') || '') : '';
+      let homeLink = 0;
+      try { if (a) { const u = new URL(href, location.href); homeLink = (u.origin === location.origin && (u.pathname === '/' || u.pathname === '')) ? 1 : 0; } } catch (e) {}
+      const meta = im.kind === 'svg' ? (im.svgMeta || '') : ((el.getAttribute && (el.getAttribute('alt') || '')) + ' ' + (typeof el.className === 'string' ? el.className : '') + ' ' + (el.id || '') + ' ' + (im.src.split('?')[0].split('/').pop() || '')).toLowerCase();
+      const ext = im.kind === 'svg' ? 'svg' : (im.src.split('?')[0].match(/\.(svg|png|jpe?g|webp|gif|avif|ico)$/i) || [, ''])[1].toLowerCase();
+      const under = oklab(im.rec.surf);
+      const o = {src: im.src.slice(0, 600), kind: im.kind, x: Math.round(im.x), y: Math.round(im.y), w: Math.round(im.w), h: Math.round(im.h), count: 1, areaSum: im.area,
+        nw: el.naturalWidth || 0, nh: el.naturalHeight || 0, homeLink, inLink: im.rec.inLink ? 1 : 0, header: (im.rec.region & 3) ? 1 : 0, footer: (im.rec.region & 4) ? 1 : 0,
+        logo: /logo|brand|wordmark/.test(meta) ? 1 : 0, icon: /icon|ico\b|glyph|arrow|chevron|caret|svg/.test(meta) ? 1 : 0, avatar: /avatar|profile|user/.test(meta) ? 1 : 0,
+        ext, underTok: im.rec.surfTok && im.rec.surfTok.i >= 0 ? im.rec.surfTok.i : -1, under,
+        surf: im.rec.surf.slice(0, 3).map(Math.round), vy: Math.round(im.y - sy), vx: Math.round(im.x - sx), useExt: im.useExt ? 1 : 0,
+        alt: opts.describe ? ((el.getAttribute && el.getAttribute('alt')) || '').slice(0, 60) : undefined,
+        stats: null, thumb: null};
+      if (im.kind === 'img' && el.complete === false) o.loading = 1;
+      imgSeen.set(key, imgOut.length);
+      imgOut.push(o);
+    }
+    if (opts.thumbs) await imageStats(imgOut, opts);
+    for (const o of imgOut) o.f = imageFeatures(o, vw, vh, docH);
+    const res = {
+      v: 5, url: location.href, title: document.title.slice(0, 200), vw, vh, docH, docW,
+      scheme: schemeInfo(sheetScan), darkness, canvasDark,
+      tokens: out, rel: relOut, images: imgOut,
+      ec: ec || undefined,
+      stats: {ads: adsSkipped, elements: els.length, visible: visibleCount, tokensTotal: tokens.size, tokensDropped: dropped, ms: Math.round(performance.now() - t0)},
+    };
+    return res;
+  }
+
+  function labToApproxRgb(u) {
+    // inverse OKLab for averaged under-colour
+    const L = u[0] / u[3], a = u[1] / u[3], b = u[2] / u[3];
+    const l_ = L + 0.3963377774 * a + 0.2158037573 * b;
+    const m_ = L - 0.1055613458 * a - 0.0638541728 * b;
+    const s_ = L - 0.0894841775 * a - 1.2914855480 * b;
+    const l = l_ ** 3, m = m_ ** 3, s = s_ ** 3;
+    const r = 4.0767416621 * l - 3.3077115913 * m + 0.2309699292 * s;
+    const g = -1.2684380046 * l + 2.6097574011 * m - 0.3413193965 * s;
+    const bb = -0.0041960863 * l - 0.7034186147 * m + 1.7076147010 * s;
+    const enc = (x) => { x = Math.max(0, Math.min(1, x)); return 255 * (x <= 0.0031308 ? 12.92 * x : 1.055 * Math.pow(x, 1 / 2.4) - 0.055); };
+    return [enc(r), enc(g), enc(bb), 1];
+  }
+
+  // ---------- image pixels ----------
+  function blobToDataURL(blob) {
+    return new Promise((res) => { const fr = new FileReader(); fr.onload = () => res(fr.result); fr.onerror = () => res(null); fr.readAsDataURL(blob); });
+  }
+  function loadViaImg(url) {
+    return new Promise((res) => {
+      const img = new Image(); img.decoding = 'async';
+      const done = (ok) => res(ok && img.naturalWidth ? img : null);
+      img.onload = () => done(true); img.onerror = () => done(false); setTimeout(() => done(false), 4000);
+      img.src = url;
+    });
+  }
+  // Decode without loading a URL into the page where possible (page CSP often blocks blob: images).
+  async function loadImage(src, fetchImage) {
+    if (fetchImage && !src.startsWith('data:')) {
+      const blob = await fetchImage(src);
+      if (!blob || !blob.size) return null;
+      const isSvg = /svg/.test(blob.type) || /\.svg(\?|#|$)/i.test(src);
+      if (!isSvg) { try { return await createImageBitmap(blob); } catch (e) {} }
+      const du = await blobToDataURL(isSvg ? new Blob([blob], {type: 'image/svg+xml'}) : blob);
+      return du ? await loadViaImg(du) : null;
+    }
+    return await loadViaImg(src);
+  }
+
+  async function imageStats(list, opts) {
+    const jobs = list.map(async (o) => {
+      try {
+        const img = await loadImage(o.src, opts.fetchImage);
+        if (!img) return;
+        const IW = img.naturalWidth || img.width, IH = img.naturalHeight || img.height;
+        o.nw = o.nw || IW; o.nh = o.nh || IH;
+        statsFromSource(o, img, IW, IH, opts);
+        if (img.close) img.close();
+      } catch (e) { o.err = String(e).slice(0, 80); }
+    });
+    await Promise.all(jobs);
+  }
+  function drawThumbSrc(src, iw, ih, size) {
+    const c = document.createElement('canvas'); c.width = size; c.height = size;
+    const ctx = c.getContext('2d', {willReadFrequently: true});
+    const s = Math.min(size / iw, size / ih);
+    const dw = Math.max(1, Math.round(iw * s)), dh = Math.max(1, Math.round(ih * s));
+    ctx.drawImage(src, Math.floor((size - dw) / 2), Math.floor((size - dh) / 2), dw, dh);
+    return {c, ctx, dw, dh};
+  }
+  function statsFromSource(o, img, IW, IH, opts) {
+    const S = opts.thumbSize;
+    const {ctx, dw, dh} = drawThumbSrc(img, IW, IH, S);
+    const d = ctx.getImageData(0, 0, S, S).data;
+    o.thumb = Array.from(d); // RGBA bytes, letterboxed (transparent pad)
+    let n = 0, tr = 0, semi = 0, Ls = 0, L2 = 0, Cs = 0, white = 0, black = 0, ringA = 0, ringL = 0, ringN = 0, total = 0;
+    const x0 = Math.floor((S - dw) / 2), y0 = Math.floor((S - dh) / 2);
+    const qset = new Set();
+    for (let y = y0; y < y0 + dh; y++) for (let x = x0; x < x0 + dw; x++) {
+      const i = (y * S + x) * 4; const a = d[i + 3] / 255; total++;
+      const ring = (x === x0 || y === y0 || x === x0 + dw - 1 || y === y0 + dh - 1);
+      if (ring) { ringA += a; ringN++; }
+      if (a < 0.1) { tr++; continue; }
+      if (a < 0.9) semi++;
+      const lab = oklab([d[i], d[i + 1], d[i + 2]]);
+      const C = Math.hypot(lab[1], lab[2]);
+      n++; Ls += lab[0]; L2 += lab[0] * lab[0]; Cs += C;
+      if (lab[0] > 0.93 && C < 0.03) white++;
+      if (lab[0] < 0.3 && C < 0.04) black++;
+      if (ring) ringL += lab[0];
+      qset.add((d[i] >> 5) << 6 | (d[i + 1] >> 5) << 3 | (d[i + 2] >> 5));
+    }
+    const mL = n ? Ls / n : 0;
+    o.stats = {transp: total ? tr / total : 0, semi: total ? semi / total : 0, meanL: mL, stdL: n ? Math.sqrt(Math.max(0, L2 / n - mL * mL)) : 0,
+      meanC: n ? Cs / n : 0, white: n ? white / n : 0, black: n ? black / n : 0, ringA: ringN ? ringA / ringN : 0,
+      ringL: ringN ? ringL / Math.max(1, ringN * (ringA / ringN || 1)) : 0, colors: qset.size / 512};
+    if (opts.bigThumbSize) o.big = drawThumbSrc(img, IW, IH, opts.bigThumbSize).c.toDataURL('image/png');
+  }
+  function imageFeatures(o, vw, vh, docH) {
+    const s = o.stats || {};
+    const ext = o.ext;
+    return sanitize([
+      Math.log1p(o.w) / 8, Math.log1p(o.h) / 8, Math.log1p(o.areaSum / 100) / 12, Math.min(1, (o.w * o.h) / (vw * vh)), Math.log(Math.max(1, o.w) / Math.max(1, o.h)) / 3,
+      Math.log1p(o.nw) / 9, Math.log1p(o.nh) / 9, Math.min(1, o.y / Math.max(1, docH)), o.y < vh ? 1 : 0, Math.log1p(o.count) / 3,
+      o.homeLink, o.inLink, o.header, o.footer, o.logo, o.icon, o.avatar,
+      ext === 'svg' ? 1 : 0, ext === 'png' ? 1 : 0, (ext === 'jpg' || ext === 'jpeg') ? 1 : 0, ext === 'webp' ? 1 : 0, ext === 'gif' ? 1 : 0, o.kind === 'bg' ? 1 : 0,
+      o.under ? o.under[0] : 1, o.under ? o.under[1] * 4 : 0, o.under ? o.under[2] * 4 : 0,
+      o.stats ? 1 : 0, s.transp || 0, s.semi || 0, s.meanL || 0, s.stdL || 0, (s.meanC || 0) * 4, s.white || 0, s.black || 0, s.ringA || 0, s.ringL || 0, s.colors || 0,
+      o.kind === 'svg' ? 1 : 0, o.useExt || 0,
+    ]);
+  }
+  const sanitize = (f) => f.map((v) => (Number.isFinite(+v) ? +v : 0));
+
+  // Compute pixel stats for a list of image entries outside the page (helper page / extension offscreen document).
+  // opts.fetchImage(src) -> Blob; opts.shot: dataURL of the viewport screenshot (for inline-SVG crops).
+  async function defaultFetch(src) {
+    const ac = new AbortController(); const t = setTimeout(() => ac.abort(), 6000);
+    try { const r = await fetch(src, {signal: ac.signal, credentials: 'omit'}); if (!r.ok) return null; const b = await r.blob(); return b.size > 8e6 ? null : b; }
+    catch (e) { return null; } finally { clearTimeout(t); }
+  }
+  async function imageStatsStandalone(list, opts) {
+    opts = Object.assign({thumbSize: 32, bigThumbSize: 0, concurrency: 8, fetchImage: defaultFetch}, opts || {});
+    let shotImg = null;
+    if (opts.shot && list.some(o => o.kind === 'svg')) shotImg = await loadViaImg(opts.shot);
+    // inline-SVG crops first (local, fast), then fetched images with limited concurrency
+    for (const o of list) {
+      if (o.kind !== 'svg' || !shotImg) continue;
+      const dpr = opts.dpr || 1;
+      if (o.vy < 0 || (o.vy + o.h) * dpr > shotImg.naturalHeight + 1 || o.w < 2 || o.h < 2) continue;
+      cropStats(o, shotImg, opts);
+    }
+    const raster = list.filter(o => o.kind !== 'svg');
+    let k = 0;
+    const run = async () => { while (k < raster.length) { const o = raster[k++]; await imageStats([o], opts); } };
+    await Promise.all(Array.from({length: opts.concurrency}, run));
+    return list.map(o => ({stats: o.stats || null, thumb: o.thumb || null, big: o.big || null, nw: o.nw, nh: o.nh, err: o.err}));
+  }
+  // inline SVGs: crop the rendered element from a screenshot; alpha is keyed off the surface colour beneath it
+  function cropStats(o, shot, opts) {
+    const S = opts.thumbSize;
+    const c = document.createElement('canvas'); c.width = Math.max(1, Math.round(o.w)); c.height = Math.max(1, Math.round(o.h));
+    const cx = c.getContext('2d', {willReadFrequently: true});
+    const dpr = opts.dpr || 1;
+    cx.drawImage(shot, o.vx * dpr, o.vy * dpr, o.w * dpr, o.h * dpr, 0, 0, c.width, c.height);
+    const src = cx.getImageData(0, 0, c.width, c.height);
+    const sl = oklab(o.surf || [255, 255, 255]);
+    const d = src.data;
+    for (let i = 0; i < d.length; i += 4) {
+      const lab = oklab([d[i], d[i + 1], d[i + 2]]);
+      const dist = Math.hypot(lab[0] - sl[0], lab[1] - sl[1], lab[2] - sl[2]);
+      d[i + 3] = Math.round(255 * Math.max(0, Math.min(1, (dist - 0.02) / 0.1)));
+    }
+    cx.putImageData(src, 0, 0);
+    o.nw = c.width; o.nh = c.height;
+    statsFromSource(o, c, c.width, c.height, opts);
+  }
+
+  globalThis.__nadirImageStats = imageStatsStandalone;
+  globalThis.__nadirImageFeatures = imageFeatures;
+  globalThis.__nadirExtract = extract;
+  globalThis.__nadirOklab = oklab;
+  globalThis.__nadirParseColor = parseColor;
+})();

runtime/nadir-infer.js

@@ -0,0 +1,193 @@
+// Nadir inference glue: page description (from extract.js + image stats) -> model inputs -> dark mapping.
+// Mirrors train.py collate() for a single page. Works in browsers (offscreen document) and Node (parity tests).
+(function (root) {
+  'use strict';
+  const MAX_TOK = 256, MAX_IMG = 32, FT = 49, FI = 39, K = 11;
+  const TREAT = ['keep', 'dim', 'invert', 'light', 'plate'];
+  const FG_ROLES = new Set([1, 5, 6]); // text, svg fill, svg stroke
+  const lin = (u) => { u /= 255; return u <= 0.04045 ? u / 12.92 : Math.pow((u + 0.055) / 1.055, 2.4); };
+  function oklab(r, g, b) {
+    r = lin(r); g = lin(g); b = lin(b);
+    const l = Math.cbrt(0.4122214708 * r + 0.5363325363 * g + 0.0514459929 * b);
+    const m = Math.cbrt(0.2119034982 * r + 0.6806995451 * g + 0.1073969566 * b);
+    const s = Math.cbrt(0.0883024619 * r + 0.2817188376 * g + 0.6299787005 * b);
+    return [0.2104542553 * l + 0.7936177850 * m - 0.0040720468 * s, 1.9779984951 * l - 2.4285922050 * m + 0.4505937099 * s, 0.0259040371 * l + 0.7827717662 * m - 0.8086757660 * s];
+  }
+  function labToRgb(L, a, b) {
+    const l_ = L + 0.3963377774 * a + 0.2158037573 * b, m_ = L - 0.1055613458 * a - 0.0638541728 * b, s_ = L - 0.0894841775 * a - 1.2914855480 * b;
+    const l = l_ ** 3, m = m_ ** 3, s = s_ ** 3;
+    const enc = (x) => { x = Math.max(0, Math.min(1, x)); return Math.round(255 * (x <= 0.0031308 ? 12.92 * x : 1.055 * Math.pow(x, 1 / 2.4) - 0.055)); };
+    return [enc(4.0767416621 * l - 3.3077115913 * m + 0.2309699292 * s), enc(-1.2684380046 * l + 2.6097574011 * m - 0.3413193965 * s), enc(-0.0041960863 * l - 0.7034186147 * m + 1.7076147010 * s)];
+  }
+  const relLum = (rgb) => 0.2126 * lin(rgb[0]) + 0.7152 * lin(rgb[1]) + 0.0722 * lin(rgb[2]);
+  const contrast = (a, b) => { const x = relLum(a), y = relLum(b); return (Math.max(x, y) + 0.05) / (Math.min(x, y) + 0.05); };
+  const hex = (rgb, a) => '#' + rgb.map(v => v.toString(16).padStart(2, '0')).join('') + (a < 0.999 ? Math.round(a * 255).toString(16).padStart(2, '0') : '');
+
+  function buildInputs(page) {
+    const toks = page.tokens.slice(0, MAX_TOK);
+    const N = Math.max(1, toks.length);
+    const imgs = page.images.slice(0, MAX_IMG);
+    const Mreal = imgs.length, M = Math.max(1, Mreal);
+    const T = 1 + N + M;
+    const tok_f = new Float32Array(N * FT), role = new BigInt64Array(N), lab = new Float32Array(N * 3);
+    toks.forEach((t, i) => {
+      tok_f.set(t.f.slice(0, FT).map((v) => (Number.isFinite(v) ? v : 0)), i * FT); role[i] = BigInt(t.role);
+      const o = oklab(t.rgba[0], t.rgba[1], t.rgba[2]); lab.set(o, i * 3);
+    });
+    const img_f = new Float32Array(M * FI), thumbs = new Float32Array(M * 4 * 32 * 32);
+    imgs.forEach((im, j) => {
+      img_f.set(im.f.slice(0, FI).map((v) => (Number.isFinite(v) ? v : 0)), j * FI);
+      if (im.thumb && im.thumb.length === 4096) {
+        for (let y = 0; y < 32; y++) for (let x = 0; x < 32; x++) for (let c = 0; c < 4; c++)
+          thumbs[j * 4096 + c * 1024 + y * 32 + x] = im.thumb[(y * 32 + x) * 4 + c] / 255;
+      }
+    });
+    const st = page.stats || {};
+    const page_f = new Float32Array([page.vw / 1280, Math.min(8, page.docH / page.vh) / 8, toks.length / MAX_TOK, Mreal / MAX_IMG,
+      page.darkness.bgL, page.darkness.textL, page.canvasDark ? 1 : 0, Math.min(1, (st.tokensDropped || 0) / 200)]);
+    // relations
+    const rel = new Float32Array(T * T * K);
+    const n = toks.length;
+    const R = new Float32Array(4 * (n + 1) * (n + 1));
+    const P = n + 1;
+    const tbBest = new Map();
+    for (const r of page.rel) {
+      const [typ, a, b, w] = r;
+      if (a >= n || b >= n) continue;
+      const idx = typ * P * P + (a + 1) * P + (b + 1);
+      if (w > R[idx]) R[idx] = w;
+      if (typ === 0) { const cur = tbBest.get(a); if (!cur || w > cur[0]) tbBest.set(a, [w, b]); }
+    }
+    for (let k = 0; k < 4; k++) for (let a = 0; a < P; a++) {
+      let mx = 0; for (let b = 0; b < P; b++) mx = Math.max(mx, R[k * P * P + a * P + b]);
+      const den = mx + 1e-6;
+      for (let b = 0; b < P; b++) {
+        const v = R[k * P * P + a * P + b] / den;
+        if (!v) continue;
+        rel[(a * T + b) * K + k] = v;          // a -> b
+        rel[(b * T + a) * K + 4 + k] = v;      // transpose channel
+      }
+    }
+    imgs.forEach((im, j) => {
+      const u = im.underTok;
+      if (u != null && u >= 0 && u < n) { rel[((1 + N + j) * T + (1 + u)) * K + 8] = 1; rel[((1 + u) * T + (1 + N + j)) * K + 9] = 1; }
+    });
+    const keyOf = (t) => t.rgba[0] * 65536 + t.rgba[1] * 256 + t.rgba[2];
+    const byKey = new Map();
+    toks.forEach((t, i) => { const k = keyOf(t); if (!byKey.has(k)) byKey.set(k, []); byKey.get(k).push(i); });
+    for (const list of byKey.values()) if (list.length > 1) for (const i of list) for (const j of list) if (i !== j) rel[((1 + i) * T + (1 + j)) * K + 10] = 1;
+    const mask = new Uint8Array(T);
+    for (let i = 0; i <= n; i++) mask[i] = 1;
+    for (let j = 0; j < Mreal; j++) mask[1 + N + j] = 1;
+    return {N, M, T, Mreal, tbBest, feeds: {
+      tok_f: [tok_f, [1, N, FT]], tok_role: [role, [1, N]], tok_lab: [lab, [1, N, 3]], img_f: [img_f, [1, M, FI]], img_thumb: [thumbs, [1, M, 4, 32, 32]],
+      page_f: [page_f, [1, 8]], src: [new BigInt64Array([0n]), [1]], rel: [rel, [1, T, T, K]], mask: [mask, [1, T]],
+    }};
+  }
+
+  // outputs: lab Float32Array(N*3), alpha_logit Float32Array(N), img_logits Float32Array(M*5)
+  // brand colours keep their hue: a coloured input whose output drifts more than 45 degrees is turned back to its
+  // own hue (keeping the predicted lightness and chroma); neutral inputs get at most a faint tint
+  function hueGuard(t, o) {
+    const i = oklab(t.rgba[0], t.rgba[1], t.rgba[2]), q = oklab(...o.rgb);
+    const cin = Math.hypot(i[1], i[2]), cout = Math.hypot(q[1], q[2]);
+    if (cin > 0.03 && cout > 0.03) {
+      const d = Math.abs(((Math.atan2(q[2], q[1]) - Math.atan2(i[2], i[1]) + Math.PI) % (2 * Math.PI) + 2 * Math.PI) % (2 * Math.PI) - Math.PI);
+      if (d <= Math.PI / 4) return;
+      const h = Math.atan2(i[2], i[1]);
+      o.rgb = labToRgb(q[0], cout * Math.cos(h), cout * Math.sin(h));
+    } else if (cin <= 0.03 && cout > 0.06) {
+      o.rgb = labToRgb(q[0], q[1] * 0.06 / cout, q[2] * 0.06 / cout);
+    }
+  }
+
+  function decode(page, built, out, opts) {
+    opts = opts || {};
+    const toks = page.tokens.slice(0, MAX_TOK);
+    const n = toks.length;
+    const outputs = new Array(n);
+    const keep = new Set();
+    toks.forEach((t, i) => {
+      const L = out.lab[i * 3], a = out.lab[i * 3 + 1], b = out.lab[i * 3 + 2];
+      const ain = Math.min(1 - 1e-3, Math.max(1e-3, t.rgba[3]));
+      const alpha = 1 / (1 + Math.exp(-(out.alpha_logit[i] + Math.log(ain / (1 - ain)))));
+      outputs[i] = {rgb: labToRgb(L, a, b), a: t.rgba[3] >= 0.999 && alpha > 0.97 ? 1 : alpha, L};
+      // colours over video/photos that already read as dark mode (dark scrims, saturated player accents, light
+      // captions and icons) are shown as-is; anything else over media is redesigned like any other colour
+      const f = t.f;
+      const bgLike = t.role === 0 || t.role === 3 || t.role === 4;
+      if (f[48] >= 0.5 && (bgLike ? (f[24] <= 0.5 && f[11] < 0.25 && (f[0] < 0.5 || f[3] > 0.4)) : f[0] > 0.6)) {
+        keep.add(i);
+        const rgb = t.rgba.slice(0, 3);
+        outputs[i] = {rgb, a: t.rgba[3], L: oklab(...rgb)[0]};
+      } else hueGuard(t, outputs[i]);
+    });
+    // safety: page canvas must be dark
+    toks.forEach((t, i) => {
+      if (t.f[24] > 0.5 && t.role === 0 && outputs[i].L > 0.35) { const o = oklab(...outputs[i].rgb); outputs[i].rgb = labToRgb(0.2, o[1] * 0.3, o[2] * 0.3); outputs[i].L = 0.2; }
+    });
+    // things that were invisible against their surface stay invisible: gradient stops with exactly a surface's
+    // colour (text fades) take that surface's dark colour, and so do borders that matched their surface
+    const keyOf = (t) => t.rgba[0] * 65536 + t.rgba[1] * 256 + t.rgba[2];
+    const surfOf = new Map();
+    toks.forEach((t, i) => { if (t.role === 0 && t.rgba[3] >= 0.999 && !keep.has(i) && outputs[i].a >= 0.999 && !surfOf.has(keyOf(t))) surfOf.set(keyOf(t), i); });
+    toks.forEach((t, i) => {
+      if (t.role === 4 && !keep.has(i) && surfOf.has(keyOf(t))) { const j = surfOf.get(keyOf(t)); outputs[i].rgb = outputs[j].rgb.slice(); outputs[i].L = outputs[j].L; }
+    });
+    const borderOn = new Map();
+    for (const [typ, a, b, w] of page.rel) {
+      if (typ !== 2 || a >= n || b >= n || toks[a].role !== 2 || toks[b].role !== 0) continue;
+      const cur = borderOn.get(a); if (!cur || w > cur[0]) borderOn.set(a, [w, b]);
+    }
+    for (const [i, [, j]] of borderOn) {
+      if (keep.has(i) || keep.has(j)) continue;
+      if (contrast(toks[i].rgba, toks[j].rgba) < 1.05 && toks[i].rgba[3] >= 0.5) { outputs[i].rgb = outputs[j].rgb.slice(); outputs[i].L = outputs[j].L; }
+    }
+    // safety: every foreground colour (text, svg fill/stroke) stays readable on all the surfaces it sits on,
+    // never asking for more contrast than the light design had; one lightness per shared token, chosen on a grid
+    const surf = new Map();
+    for (const [typ, a, b, w] of page.rel) {
+      if (typ !== 0 || a >= n || b >= n || toks[b].role !== 0 || outputs[b].a < 0.999) continue;
+      if (!surf.has(a)) surf.set(a, []);
+      surf.get(a).push([b, w + 0.1]);
+    }
+    for (const [i, lst] of surf) {
+      const t = toks[i];
+      if (keep.has(i) || !FG_ROLES.has(t.role) || outputs[i].a < 0.999) continue;
+      let need0 = 3;
+      if (t.role === 1) { const fs = t.f[21] * 32, bold = t.f[23] > 0.5; need0 = (fs >= 24 || (fs >= 18.5 && bold)) ? 3 : 4.5; }
+      if (opts.minContrast) need0 = Math.min(need0, opts.minContrast);
+      const reqs = [];
+      for (const [j, w] of lst) {
+        const need = Math.min(need0, contrast(t.rgba, toks[j].rgba));
+        if (need >= 1.5) reqs.push([outputs[j].rgb, need, w]);
+      }
+      if (!reqs.length) continue;
+      const deficit = (rgb) => { let d = 0; for (const [bg, need, w] of reqs) d += w * Math.max(0, Math.log(need) - Math.log(contrast(rgb, bg))); return d; };
+      if (deficit(outputs[i].rgb) <= 0) continue;
+      const o = oklab(...outputs[i].rgb);
+      let W = 0; for (const r of reqs) W += r[2];
+      let best = null;
+      for (let k = 0; k <= 50; k++) {
+        const Lx = k / 50, rgb = labToRgb(Lx, o[1], o[2]);
+        const c = deficit(rgb) / W + 0.05 * Math.abs(Lx - o[0]);
+        if (!best || c < best[0]) best = [c, rgb, Lx];
+      }
+      outputs[i].rgb = best[1]; outputs[i].L = best[2];
+    }
+    const colors = toks.map((t, i) => keep.has(i) ? null : hex(outputs[i].rgb, outputs[i].a));
+    const imgs = page.images.slice(0, MAX_IMG);
+    const treatments = imgs.map((im, j) => {
+      let best = 0;
+      for (let c = 1; c < 5; c++) if (out.img_logits[j * 5 + c] > out.img_logits[j * 5 + best]) best = c;
+      const t = TREAT[best];
+      const u = im.underTok;
+      return root.NadirSafeTreatment ? root.NadirSafeTreatment(t, im, u != null && u >= 0 && u < n ? outputs[u].L : null) : t;
+    });
+    return {colors, keep: [...keep], treatments};
+  }
+
+  const api = {buildInputs, decode, oklab, labToRgb, TREAT, MAX_TOK, MAX_IMG};
+  root.NadirInfer = api;
+  if (typeof module !== 'undefined') module.exports = api;
+})(typeof globalThis !== 'undefined' ? globalThis : this);

runtime/nadir-map.js

@@ -0,0 +1,244 @@
+// Nadir colour mapping runtime: turns per-token dark colours (from the model or a teacher) into a lookup
+// for every colour the page might use. Shared by the extension and the evaluation renderer.
+(function () {
+  'use strict';
+  const ROLES = ['bg', 'text', 'border', 'shadow', 'gradient', 'fill', 'stroke'];
+  // which token roles answer a query role, in order of preference
+  const CHAIN = {
+    bg: ['bg', 'gradient', 'shadow'],
+    gradient: ['gradient', 'bg', 'shadow'],
+    shadow: ['shadow', 'bg', 'gradient'],
+    text: ['text', 'fill', 'stroke'],
+    fill: ['fill', 'text', 'stroke'],
+    stroke: ['stroke', 'fill', 'text'],
+    border: ['border', 'bg', 'stroke'],
+  };
+  const lin = (u) => { u /= 255; return u <= 0.04045 ? u / 12.92 : Math.pow((u + 0.055) / 1.055, 2.4); };
+  function oklab(r, g, b) {
+    r = lin(r); g = lin(g); b = lin(b);
+    const l = Math.cbrt(0.4122214708 * r + 0.5363325363 * g + 0.0514459929 * b);
+    const m = Math.cbrt(0.2119034982 * r + 0.6806995451 * g + 0.1073969566 * b);
+    const s = Math.cbrt(0.0883024619 * r + 0.2817188376 * g + 0.6299787005 * b);
+    return [0.2104542553 * l + 0.7936177850 * m - 0.0040720468 * s,
+      1.9779984951 * l - 2.4285922050 * m + 0.4505937099 * s,
+      0.0259040371 * l + 0.7827717662 * m - 0.8086757660 * s];
+  }
+  function toRgb(L, a, b) {
+    const l_ = L + 0.3963377774 * a + 0.2158037573 * b;
+    const m_ = L - 0.1055613458 * a - 0.0638541728 * b;
+    const s_ = L - 0.0894841775 * a - 1.2914855480 * b;
+    const l = l_ ** 3, m = m_ ** 3, s = s_ ** 3;
+    const enc = (x) => { x = Math.max(0, Math.min(1, x)); return Math.round(255 * (x <= 0.0031308 ? 12.92 * x : 1.055 * Math.pow(x, 1 / 2.4) - 0.055)); };
+    return [enc(4.0767416621 * l - 3.3077115913 * m + 0.2309699292 * s), enc(-1.2684380046 * l + 2.6097574011 * m - 0.3413193965 * s), enc(-0.0041960863 * l - 0.7034186147 * m + 1.7076147010 * s)];
+  }
+  const hex2 = (n) => n.toString(16).padStart(2, '0');
+  const keyOf = (r, g, b, a) => hex2(r) + hex2(g) + hex2(b) + hex2(Math.round(a * 255));
+  function css(rgb, a) {
+    return a >= 0.999 ? `#${hex2(rgb[0])}${hex2(rgb[1])}${hex2(rgb[2])}` : `rgba(${rgb[0]}, ${rgb[1]}, ${rgb[2]}, ${+a.toFixed(3)})`;
+  }
+  function parseHex(h) {
+    h = h.replace('#', '');
+    return [parseInt(h.slice(0, 2), 16), parseInt(h.slice(2, 4), 16), parseInt(h.slice(4, 6), 16), h.length >= 8 ? parseInt(h.slice(6, 8), 16) / 255 : 1];
+  }
+
+  class NadirMap {
+    // tokens: [{role, rgba:[r,g,b,a]}]; outputs: same length, each '#rrggbb[aa]' | [r,g,b,a] | null
+    // opts.keep: set of token indices that must stay unchanged (media overlays)
+    constructor(tokens, outputs, opts) {
+      opts = opts || {};
+      this.exact = {};           // role -> key -> css
+      this.pts = {};             // role -> [{lab, alpha, dl, da, db, dA}]
+      for (const r of ROLES) { this.exact[r] = new Map(); this.pts[r] = []; }
+      this.fillL = new Map();    // SVG fill colour -> output lightness
+      const keep = opts.keep || new Set();
+      tokens.forEach((t, i) => {
+        let o = outputs[i];
+        if (keep.has(i)) o = t.rgba;
+        if (o == null) return;
+        if (typeof o === 'string') o = parseHex(o);
+        const role = typeof t.role === 'number' ? ROLES[t.role] : t.role;
+        const [r, g, b, a] = t.rgba;
+        this.exact[role].set(keyOf(r, g, b, a), css([o[0], o[1], o[2]], o[3] == null ? a : o[3]));
+        const li = oklab(r, g, b), lo = oklab(o[0], o[1], o[2]);
+        if (role === 'fill' && !keep.has(i)) this.fillL.set(keyOf(r, g, b, a), lo[0]); // media keeps say nothing about ink
+        this.pts[role].push({lab: li, alpha: a, d: [lo[0] - li[0], lo[1] - li[1], lo[2] - li[2]], out: lo, oA: o[3] == null ? a : o[3]});
+      });
+      this.cache = new Map();
+      this.maxDist = opts.maxDist || 0.16;
+    }
+    // An SVG fill the model saw on this page: drawn as light ink (logo, icon) or as a dark surface (wave, panel)?
+    // null when the colour was never seen as a fill: the engine then falls back to its own size heuristic.
+    fillSide(r, g, b, a) {
+      const L = this.fillL.get(keyOf(r, g, b, a));
+      return L == null ? null : (L >= 0.5 ? 'fg' : 'bg');
+    }
+    lookup(role, r, g, b, a) {
+      const ck = role + keyOf(r, g, b, a);
+      let v = this.cache.get(ck);
+      if (v !== undefined) return v;
+      v = this._lookup(role, r, g, b, a);
+      this.cache.set(ck, v);
+      return v;
+    }
+    _lookup(role, r, g, b, a) {
+      const key = keyOf(r, g, b, a);
+      const chain = CHAIN[role] || [role];
+      for (const rr of chain) { const e = this.exact[rr].get(key); if (e) return e; }
+      // nearest-neighbour transfer within the role chain (first role with points)
+      const q = oklab(r, g, b);
+      let best = [];
+      for (let ci = 0; ci < chain.length; ci++) {
+        const pts = this.pts[chain[ci]];
+        for (const p of pts) {
+          const dl = p.lab[0] - q[0], da = p.lab[1] - q[1], db = p.lab[2] - q[2];
+          const d = Math.sqrt(dl * dl * 1.5 + da * da + db * db + (p.alpha - a) * (p.alpha - a) * 0.25) + ci * 0.03;
+          best.push([d, p]);
+        }
+        if (best.length >= 6) break;
+      }
+      if (!best.length) return null;
+      best.sort((x, y) => x[0] - y[0]);
+      best = best.slice(0, 4);
+      if (best[0][0] > this.maxDist) return null; // too far from anything the model saw: built-in fallback
+      let w = 0, d0 = 0, d1 = 0, d2 = 0, dA = 0;
+      for (const [d, p] of best) {
+        const wi = 1 / (d * d + 1e-4);
+        w += wi; d0 += p.d[0] * wi; d1 += p.d[1] * wi; d2 += p.d[2] * wi; dA += (p.oA - p.alpha) * wi;
+      }
+      const L = Math.max(0, Math.min(1, q[0] + d0 / w));
+      // keep chroma direction of the query, scale like the neighbours did
+      const rgb = toRgb(L, q[1] + d1 / w, q[2] + d2 / w);
+      const alpha = Math.max(0, Math.min(1, a + dA / w));
+      return css(rgb, alpha);
+    }
+  }
+
+  // image treatments as CSS
+  const TREAT_CSS = {
+    invert: 'filter: invert(1) hue-rotate(180deg) !important;',
+    light: 'filter: brightness(0) invert(0.92) !important;',
+    dim: 'filter: brightness(0.88) !important;',
+    plate: 'background-color: rgba(255,255,255,0.92) !important; box-shadow: 0 0 0 3px rgba(255,255,255,0.92) !important; border-radius: 4px !important;',
+  };
+  // Content images must never be colour-altered: only flat graphics (logos, icons, diagrams) may get a treatment.
+  // a logo on its own white rectangle: adapt it instead of leaving a white box. Only images the page marks as its
+  // logo (logo/brand/wordmark naming, or the link home); product shots, photos and other images are never touched
+  // Returns 'adapt:<fallback>': the page tries the adaptation (its pixel check confirms the white background)
+  // and uses the fallback treatment when there is nothing to adapt. Without pixel stats yet, logos still try.
+  function adaptable(im, s, surfL) {
+    if (!(im.logo || (im.homeLink && im.header)) || im.h > 200 || (surfL != null && surfL >= 0.4)) return false;
+    if (!s) return im.w * im.h <= 250000;
+    return s.transp < 0.05 && (s.ringA || 0) >= 0.95 && (s.ringL || 0) >= 0.85 && (s.white || 0) >= 0.3 && s.colors < 0.2;
+  }
+  function baseTreatment(t, im, surfL) {
+    const s = im && im.stats;
+    // a light, flat CSS background image (texture, pale wash, pattern) sits behind content whose text is now light:
+    // clear it so the dark surface designed for that element shows through. Photos and detailed images stay.
+    // (almost no lightness variation, so never a photo; SVG backgrounds can't be photos, so a smooth gradient qualifies).
+    // Logos drawn as backgrounds are never cleared.
+    if (im && im.kind === 'bg' && !im.logo && !im.homeLink && s && s.transp < 0.05 && s.meanL > 0.8 && im.w * im.h >= 40000 &&
+        s.stdL < (/\.svg$/i.test((im.src || '').split('?')[0]) ? 0.15 : 0.07)) return 'clear';
+    // a mostly dark graphic on transparency would vanish on a dark surface: hue-preserving inversion
+    if ((!t || t === 'keep') && s && s.transp > 0.15 && s.meanL < 0.35 && im.w * im.h < 200000 && (surfL == null || surfL < 0.4)) return 'invert';
+    if (!t || t === 'keep') return 'keep';
+    const photoLike = s ? (s.transp < 0.02 && s.colors > 0.18 && s.stdL > 0.12) : (im && im.w * im.h > 60000);
+    if (photoLike && t !== 'dim') return 'keep';
+    if (t === 'dim' && im && im.w * im.h < 20000) return 'keep';
+    // a transparent graphic that is already light reads fine on a dark surface: leave it alone
+    if ((t === 'plate' || t === 'light' || t === 'invert') && s && s.transp > 0.15 && s.meanL >= 0.55 && (s.black || 0) < 0.15 &&
+        (surfL == null || surfL < 0.4)) return 'keep';
+    // no light boxes on a dark page: a dark logo on transparency gets a hue-preserving inversion instead of a plate,
+    // an opaque one already carries its own background
+    if (t === 'plate') return (!s || s.transp > 0.15) ? 'invert' : 'keep';
+    return t;
+  }
+  function safeTreatment(t, im, surfL) {
+    const base = baseTreatment(t, im, surfL);
+    return base !== 'clear' && im && adaptable(im, im.stats, surfL) ? 'adapt:' + base : base;
+  }
+
+  // ---------- flat graphics on a white background (logos, icons, diagrams): adapt the image itself ----------
+  // The white background becomes transparent so the page's own dark surface shows through, neutral ink flips to
+  // light, and brand colours keep their hue and are lifted only as far as needed to read on dark.
+  function adaptPixels(d, w, h, info) {
+    info = info || {};
+    let n = 0, light = 0, sr = 0, sg = 0, sb = 0;
+    const ringPx = (p) => {
+      const i = p * 4;
+      if (d[i + 3] < 250) return;
+      n++;
+      const o = oklab(d[i], d[i + 1], d[i + 2]);
+      if (o[0] > 0.9 && Math.hypot(o[1], o[2]) < 0.04) { light++; sr += d[i]; sg += d[i + 1]; sb += d[i + 2]; }
+    };
+    for (let x = 0; x < w; x++) { ringPx(x); ringPx((h - 1) * w + x); }
+    for (let y = 1; y < h - 1; y++) { ringPx(y * w); ringPx(y * w + w - 1); }
+    info.ring = n ? light / n : 0;
+    if (!n || light / n < 0.5) { info.why = 'ring'; return false; } // no light background to remove (tight crops touch the edge)
+    const bg = oklab(sr / light, sg / light, sb / light);
+    const cache = new Map();
+    for (let p = 0, P = w * h; p < P; p++) {
+      const i = p * 4;
+      if (!d[i + 3]) continue;
+      const key = (d[i] << 16) | (d[i + 1] << 8) | d[i + 2];
+      let r = cache.get(key);
+      if (!r) {
+        const o = oklab(d[i], d[i + 1], d[i + 2]);
+        const dist = Math.hypot(o[0] - bg[0], o[1] - bg[1], o[2] - bg[2]);
+        const cov = Math.max(0, Math.min(1, (dist - 0.03) / 0.22)); // 0 = background, 1 = ink, between = anti-aliased edge
+        if (cov <= 0) r = [0, 0, 0, 0];
+        else {
+          // unmix the edge pixel from the background, then restyle the ink for a dark surface
+          let L = (o[0] - (1 - cov) * bg[0]) / cov, a = (o[1] - (1 - cov) * bg[1]) / cov, b = (o[2] - (1 - cov) * bg[2]) / cov;
+          L = Math.max(0, Math.min(1, L));
+          const C = Math.hypot(a, b);
+          if (C < 0.05) L = 0.94 - 0.75 * L;                       // black/gray ink -> light
+          else if (L < 0.62) { L = 0.62 + 0.15 * L / 0.62; a *= 0.92; b *= 0.92; } // brand colour: same hue, lifted to read on dark
+          const rgb = toRgb(L, a, b);
+          r = [rgb[0], rgb[1], rgb[2], cov];
+        }
+        cache.set(key, r);
+      }
+      d[i] = r[0]; d[i + 1] = r[1]; d[i + 2] = r[2]; d[i + 3] = Math.round(d[i + 3] * r[3]);
+    }
+    return true;
+  }
+  // blob -> data URL of the adapted image at its natural size (so layout never shifts), or null
+  async function sniffType(blob) {
+    const h = new Uint8Array(await blob.slice(0, 64).arrayBuffer());
+    const txt = String.fromCharCode(...h);
+    if (h[0] === 0x89 && txt.slice(1, 4) === 'PNG') return 'image/png';
+    if (h[0] === 0xff && h[1] === 0xd8) return 'image/jpeg';
+    if (txt.startsWith('GIF8')) return 'image/gif';
+    if (txt.startsWith('RIFF') && txt.slice(8, 12) === 'WEBP') return 'image/webp';
+    if (/<svg|<\?xml/i.test(txt)) return 'image/svg+xml';
+    return blob.type;
+  }
+  async function adaptImage(blob, info) {
+    info = info || {};
+    const type = await sniffType(blob); // servers mislabel images (e.g. video/webp)
+    if (type !== blob.type) blob = new Blob([blob], {type});
+    let src = null;
+    if (!/svg/.test(blob.type)) { try { src = await createImageBitmap(blob); } catch (e) { src = null; } }
+    if (!src) {
+      const u = URL.createObjectURL(blob.type ? blob : new Blob([blob], {type: 'image/svg+xml'}));
+      src = await new Promise((res) => { const im = new Image(); im.onload = () => res(im); im.onerror = () => res(null); im.src = u; });
+      URL.revokeObjectURL(u);
+    }
+    if (!src) { info.why = 'decode'; return null; }
+    const w = src.naturalWidth || src.width, h = src.naturalHeight || src.height;
+    if (!w || !h || w * h > 2400 * 2400) { info.why = 'size'; return null; }
+    const c = document.createElement('canvas'); c.width = w; c.height = h;
+    const ctx = c.getContext('2d', {willReadFrequently: true});
+    ctx.drawImage(src, 0, 0, w, h);
+    if (src.close) src.close();
+    const im = ctx.getImageData(0, 0, w, h);
+    if (!adaptPixels(im.data, w, h, info)) return null;
+    ctx.putImageData(im, 0, 0);
+    return c.toDataURL('image/png');
+  }
+
+  globalThis.NadirMap = NadirMap;
+  globalThis.NadirAdaptImage = adaptImage;
+  globalThis.NadirImageCSS = TREAT_CSS;
+  globalThis.NadirSafeTreatment = safeTreatment;
+})();

training/build_ds.py

@@ -0,0 +1,281 @@
+#!/usr/bin/env python3
+"""Build Nadir training data: page features + per-token dark targets (Haiku designs and native dark-mode pairs)."""
+import argparse, collections, difflib, gzip, hashlib, json, math, os, random, sys
+from pathlib import Path
+import numpy as np
+
+import safety
+
+ROLES = ["bg", "text", "border", "shadow", "gradient", "fill", "stroke"]
+TREAT = ["keep", "dim", "invert", "light", "plate"]
+MAX_TOK, MAX_IMG = 256, 32
+
+
+def lin(u):
+    u = np.asarray(u, dtype=np.float64) / 255.0
+    return np.where(u <= 0.04045, u / 12.92, ((u + 0.055) / 1.055) ** 2.4)
+
+
+def oklab(rgb):
+    r, g, b = lin(rgb[..., 0]), lin(rgb[..., 1]), lin(rgb[..., 2])
+    l = np.cbrt(0.4122214708 * r + 0.5363325363 * g + 0.0514459929 * b)
+    m = np.cbrt(0.2119034982 * r + 0.6806995451 * g + 0.1073969566 * b)
+    s = np.cbrt(0.0883024619 * r + 0.2817188376 * g + 0.6299787005 * b)
+    return np.stack([0.2104542553 * l + 0.7936177850 * m - 0.0040720468 * s,
+                     1.9779984951 * l - 2.4285922050 * m + 0.4505937099 * s,
+                     0.0259040371 * l + 0.7827717662 * m - 0.8086757660 * s], -1)
+
+
+def rel_lum(rgb):
+    c = lin(rgb)
+    return 0.2126 * c[..., 0] + 0.7152 * c[..., 1] + 0.0722 * c[..., 2]
+
+
+def hex2rgba(h):
+    h = h.lstrip("#")
+    return [int(h[0:2], 16), int(h[2:4], 16), int(h[4:6], 16), int(h[6:8], 16) / 255 if len(h) >= 8 else 1.0]
+
+
+def contrast(fg, bg):
+    l1, l2 = rel_lum(np.array(fg[:3])), rel_lum(np.array(bg[:3]))
+    return (max(l1, l2) + 0.05) / (min(l1, l2) + 0.05)
+
+
+def over(top, bottom):
+    a = top[3]
+    return [top[i] * a + bottom[i] * (1 - a) for i in range(3)] + [1.0]
+
+
+def page_arrays(L):
+    toks = L["tokens"][:MAX_TOK]
+    n = len(toks)
+    F = np.array([t["f"] for t in toks], dtype=np.float32)
+    role = np.array([t["role"] for t in toks], dtype=np.int64)
+    rgba = np.array([t["rgba"] for t in toks], dtype=np.float32)
+    rel = np.array([r for r in L["rel"] if r[1] < n and r[2] < n], dtype=np.float32).reshape(-1, 4)
+    imgs = L["images"][:MAX_IMG]
+    IF = np.array([im["f"] for im in imgs], dtype=np.float32).reshape(len(imgs), 39)
+    thumbs = np.zeros((len(imgs), 32, 32, 4), dtype=np.uint8)
+    for j, im in enumerate(imgs):
+        if im.get("thumb") and len(im["thumb"]) == 4096:
+            thumbs[j] = np.array(im["thumb"], dtype=np.uint8).reshape(32, 32, 4)
+    img_under = np.array([im.get("underTok", -1) if (im.get("underTok", -1) < n) else -1 for im in imgs], dtype=np.int64)
+    w = []
+    for t in toks:
+        d = t.get("d") or {}
+        w.append(1.0 + math.log1p(d.get("area", 0) / 2000) + math.log1p(d.get("chars", 0) / 20) + 0.5 * math.log1p(d.get("nVis", 0)) + 0.3 * math.log1p(d.get("rules", 0)))
+    keep = [i for i, t in enumerate(toks) if safety.is_media_keep(t)]
+    page = np.array([L["vw"] / 1280, min(8, L["docH"] / L["vh"]) / 8, n / MAX_TOK, len(imgs) / MAX_IMG, L["darkness"]["bgL"], L["darkness"]["textL"],
+                     1.0 if L.get("canvasDark") else 0.0, min(1, L["stats"].get("tokensDropped", 0) / 200)], dtype=np.float32)
+    F, IF, page = (np.nan_to_num(x, nan=0.0, posinf=0.0, neginf=0.0) for x in (F, IF, page))
+    return dict(F=F, role=role, rgba=rgba, rel=rel, IF=IF, thumbs=thumbs, img_under=img_under, w=np.array(w, dtype=np.float32), page=page, keep=np.array(keep, dtype=np.int64))
+
+
+def haiku_targets(L, h, arrs, qc):
+    n = len(arrs["role"])
+    tgt = np.full((n, 4), np.nan, dtype=np.float32)
+    for k, v in h["colors"].items():
+        i = int(k)
+        if i < n:
+            tgt[i] = hex2rgba(v)
+    imgt = np.full((len(arrs["IF"]),), -1, dtype=np.int64)
+    for k, v in (h.get("images") or {}).items():
+        j = int(k)
+        if j < len(imgt) and v in TREAT:
+            imgt[j] = TREAT.index(v)
+    # ---- QC ----
+    asked = min(n, h.get("n_tok", n))
+    got = np.isfinite(tgt[:asked, 0]).sum()
+    if asked == 0 or got / asked < 0.9:
+        qc["coverage"] += 1
+        return None
+    root = [i for i, t in enumerate(L["tokens"][:n]) if (t.get("d") or {}).get("root")]
+    if root and np.isfinite(tgt[root[0], 0]):
+        rl = oklab(np.array(tgt[root[0], :3]))[0]
+        if rl > 0.4:
+            qc["light_root"] += 1
+            return None
+    # text contrast vs mapped dominant surface
+    tb = collections.defaultdict(list)
+    for typ, a, b, w in arrs["rel"]:
+        if int(typ) == 0:
+            tb[int(a)].append((w, int(b)))
+    bad = tot = 0.0
+    for i, t in enumerate(L["tokens"][:n]):
+        if t["role"] != 1 or not np.isfinite(tgt[i, 0]):
+            continue
+        chars = (t.get("d") or {}).get("chars", 0)
+        if not chars or not tb.get(i):
+            continue
+        j = max(tb[i])[1]
+        if not np.isfinite(tgt[j, 0]):
+            continue
+        bgc = tgt[j]
+        base = over(list(bgc), [18, 18, 18, 1]) if bgc[3] < 1 else list(bgc)
+        fg = over(list(tgt[i]), base)
+        c = contrast(fg, base)
+        tot += chars
+        if c < 3.0:
+            bad += chars
+    if tot and bad / tot > 0.15:
+        qc["contrast"] += 1
+        return None
+    qc["ok"] += 1
+    return tgt, imgt
+
+
+def repair_contrast(arrs, tgt):
+    """Apply the runtime rules (hue guard, invisible-stays-invisible, contrast) to a Haiku design, so the model
+    learns what the runtime would show."""
+    n = len(arrs["role"])
+    toks = [{"role": int(arrs["role"][i]), "rgba": arrs["rgba"][i].tolist(), "f": arrs["F"][i]} for i in range(n)]
+    out, keep = [], set(int(k) for k in arrs["keep"])
+    for i in range(n):
+        if not np.isfinite(tgt[i, 0]):
+            keep.add(i)
+            out.append({"rgb": [0, 0, 0], "a": 0.0, "L": 0.0})
+            continue
+        rgb = [int(round(float(x))) for x in tgt[i, :3]]
+        out.append({"rgb": rgb, "a": float(tgt[i, 3]), "L": safety.oklab(*rgb)[0]})
+    before = [list(o["rgb"]) for o in out]
+    for i in range(n):
+        if i not in keep:
+            safety.hue_guard(toks[i], out[i])
+    safety.keep_invisible(toks, out, arrs["rel"].tolist(), keep)
+    safety.enforce_contrast(toks, out, arrs["rel"].tolist(), keep)
+    changed = 0
+    for i in range(n):
+        if out[i]["rgb"] != before[i]:
+            tgt[i, :3] = out[i]["rgb"]
+            changed += 1
+    return changed
+
+
+def native_targets(L, D, arrs, qc):
+    le, de = L.get("ec") or [], D.get("ec") or []
+    if not le or not de:
+        qc["no_ec"] += 1
+        return None
+    if len(le) == len(de) and all(a[0] == b[0] for a, b in zip(le, de)):
+        pairs = list(zip(range(len(le)), range(len(de))))
+    else:
+        sm = difflib.SequenceMatcher(None, [r[0] for r in le], [r[0] for r in de], autojunk=False)
+        pairs = []
+        for blk in sm.get_matching_blocks():
+            pairs += [(blk.a + k, blk.b + k) for k in range(blk.size)]
+        if len(pairs) < 0.8 * len(le):
+            qc["align"] += 1
+            return None
+    votes = collections.defaultdict(collections.Counter)
+    for ia, ib in pairs:
+        a, b = le[ia], de[ib]
+        w = (a[1] or 0) + 1
+        if a[2]:
+            # bg: effective dark surface at this element (opaque light bg) or raw dark bg (translucent)
+            alpha = int(a[2][6:8], 16)
+            tgt = (b[8] if alpha >= 250 else b[2]) or b[8]
+            votes[(0, a[2])][tgt] += w
+        for j, role in ((3, 1), (4, 2), (5, 5), (6, 6)):
+            if a[j]:
+                votes[(role, a[j])][b[j] or "00000000"] += w
+        if a[7] and b[7]:
+            la, lb = a[7].split(","), b[7].split(",")
+            if len(la) == len(lb):
+                for x, y in zip(la, lb):
+                    votes[(4, x)][y] += w
+    n = len(arrs["role"])
+    tgt = np.full((n, 4), np.nan, dtype=np.float32)
+    hit = 0
+    for i, t in enumerate(L["tokens"][:n]):
+        key = t["k"].split(":")[1]
+        c = votes.get((t["role"], key))
+        if not c:
+            continue
+        best, bw = c.most_common(1)[0]
+        if bw / sum(c.values()) < 0.5:
+            continue
+        tgt[i] = hex2rgba(best)
+        hit += 1
+    if hit < 4:
+        qc["few"] += 1
+        return None
+    qc["ok"] += 1
+    return tgt, np.full((len(arrs["IF"]),), -1, dtype=np.int64)
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--crawl", required=True)
+    ap.add_argument("--haiku", nargs="*", default=[])
+    ap.add_argument("--out", required=True)
+    ap.add_argument("--val-frac", type=float, default=0.05)
+    args = ap.parse_args()
+    crawl = Path(args.crawl)
+    index = {}
+    for line in (l for ip in sorted(crawl.glob("index*.jsonl")) for l in open(ip)):
+        try:
+            r = json.loads(line)
+            index[r["id"]] = r
+        except Exception:
+            pass
+    samples = []
+    qc_h = collections.Counter(); qc_n = collections.Counter()
+    # Haiku-designed pages (source 0)
+    for hd in args.haiku:
+        for fp in Path(hd).glob("*.json"):
+            if fp.name in ("ledger.jsonl",):
+                continue
+            try:
+                h = json.loads(fp.read_text())
+            except Exception:
+                continue
+            if h.get("refusal") or "colors" not in h:
+                continue
+            pid = h["id"]
+            try:
+                L = json.load(gzip.open(crawl / "pages" / pid / "light.json.gz"))
+            except Exception:
+                continue
+            arrs = page_arrays(L)
+            r = haiku_targets(L, h, arrs, qc_h)
+            if r is None:
+                continue
+            arrs["tgt"], arrs["imgt"] = r
+            if repair_contrast(arrs, arrs["tgt"]):
+                qc_h["repaired"] += 1
+            arrs["src"] = 0
+            arrs["id"] = pid
+            arrs["host"] = index.get(pid, {}).get("url", L["url"]).split("/")[2]
+            samples.append(arrs)
+    # native dark-mode pairs (source 1)
+    for pid, r in index.items():
+        if r.get("kind") not in ("native_media", "native_class"):
+            continue
+        try:
+            L = json.load(gzip.open(crawl / "pages" / pid / "light.json.gz"))
+            D = json.load(gzip.open(crawl / "pages" / pid / "dark.json.gz"))
+        except Exception:
+            continue
+        arrs = page_arrays(L)
+        res = native_targets(L, D, arrs, qc_n)
+        if res is None:
+            continue
+        arrs["tgt"], arrs["imgt"] = res
+        arrs["src"] = 1
+        arrs["id"] = pid
+        arrs["host"] = r["url"].split("/")[2]
+        samples.append(arrs)
+    print("haiku QC", dict(qc_h), "native QC", dict(qc_n), "samples", len(samples), flush=True)
+    hosts = sorted(set(s["host"].replace("www.", "") for s in samples))
+    # stable split: a host's side never changes as the crawl grows
+    val_hosts = {h for h in hosts if int(hashlib.md5(h.encode()).hexdigest()[:8], 16) % 1000 < args.val_frac * 1000}
+    train = [s for s in samples if s["host"].replace("www.", "") not in val_hosts]
+    val = [s for s in samples if s["host"].replace("www.", "") in val_hosts]
+    import pickle
+    with open(args.out, "wb") as f:
+        pickle.dump({"train": train, "val": val, "roles": ROLES, "treat": TREAT}, f, protocol=4)
+    print("train", len(train), "val", len(val), "haiku", sum(s["src"] == 0 for s in samples), "native", sum(s["src"] == 1 for s in samples))
+
+
+if __name__ == "__main__":
+    main()

training/export.py

@@ -0,0 +1,211 @@
+#!/usr/bin/env python3
+"""Export NadirNet to ONNX (fp32/fp16) and build calibrated 8-bit and 4-bit variants; measure each against fp32."""
+import argparse, json, os, pickle, time
+import numpy as np
+import torch
+import torch.nn as nn
+
+from model import NadirNet
+from train import collate, prep, evaluate
+
+NAMES_IN = ["tok_f", "tok_role", "tok_lab", "img_f", "img_thumb", "page_f", "src", "rel", "mask"]
+NAMES_OUT = ["lab", "alpha_logit", "img_logits"]
+
+
+class Wrapped(nn.Module):
+    def __init__(self, m):
+        super().__init__()
+        self.m = m
+
+    def forward(self, tok_f, tok_role, tok_lab, img_f, img_thumb, page_f, src, rel, mask):
+        return self.m(tok_f, tok_role, tok_lab, img_f, img_thumb, page_f, src, rel, mask)
+
+
+def feeds_for(s, pad=None):
+    b = collate([s], "cpu")
+    if pad:  # fixed shapes for calibration (quantizers stack activations across samples)
+        N0, M0 = b["tok_f"].shape[1], b["img_f"].shape[1]
+        N, M = pad
+        if N0 <= N and M0 <= M:
+            T0, T = 1 + N0 + M0, 1 + N + M
+            def padto(x, dims, val=0):
+                shape = list(x.shape)
+                for d, sz in dims.items():
+                    shape[d] = sz
+                y = torch.full(shape, val, dtype=x.dtype)
+                y[tuple(slice(0, x.shape[i]) for i in range(x.ndim))] = x
+                return y
+            idx = list(range(1 + N0)) + list(range(1 + N, 1 + N + M0))
+            rel = torch.zeros(1, T, T, b["rel"].shape[-1]); mask = torch.zeros(1, T, dtype=torch.bool)
+            ii = torch.tensor(idx)
+            rel[0][ii[:, None], ii[None, :]] = b["rel"][0]
+            mask[0, ii] = b["mask"][0]
+            b = dict(b, tok_f=padto(b["tok_f"], {1: N}), role=padto(b["role"], {1: N}), lab=padto(b["lab"], {1: N}),
+                     img_f=padto(b["img_f"], {1: M}), thumbs=padto(b["thumbs"], {1: M}), rel=rel, mask=mask)
+    M = b["img_f"].shape[1]
+    if M == 0:  # the runtime always sends at least one (masked) image slot
+        T = b["mask"].shape[1]
+        b["img_f"] = torch.zeros(1, 1, 39); b["thumbs"] = torch.zeros(1, 1, 4, 32, 32)
+        rel = torch.zeros(1, T + 1, T + 1, b["rel"].shape[-1]); rel[:, :T, :T] = b["rel"]; b["rel"] = rel
+        b["mask"] = torch.cat([b["mask"], torch.zeros(1, 1, dtype=torch.bool)], 1)
+    return {"tok_f": b["tok_f"].numpy(), "tok_role": b["role"].numpy(), "tok_lab": b["lab"].numpy(), "img_f": b["img_f"].numpy(),
+            "img_thumb": b["thumbs"].numpy(), "page_f": b["page"].numpy(), "src": b["src"].numpy(), "rel": b["rel"].numpy(), "mask": b["mask"].numpy()}
+
+
+def fidelity(sess, ref_sess, samples):
+    """Mean OKLab distance between this model's colours and fp32's, plus image-decision agreement."""
+    de, agree, n_img, worst = [], 0, 0, 0.0
+    for s in samples:
+        f = feeds_for(s)
+        a = sess.run(None, f); r = ref_sess.run(None, f)
+        n = s["F"].shape[0]
+        d = np.linalg.norm(a[0][0, :n] - r[0][0, :n], axis=-1)
+        w = s["w"][:n]
+        de.append(float((d * w).sum() / w.sum())); worst = max(worst, float(d.max()))
+        m = len(s["IF"])
+        if m:
+            agree += int((a[2][0, :m].argmax(-1) == r[2][0, :m].argmax(-1)).sum()); n_img += m
+    return {"dE_vs_fp32": float(np.mean(de)), "dE_max": worst, "img_agree": agree / max(1, n_img)}
+
+
+def bench(sess, samples, reps=3):
+    ts = []
+    for s in samples[:20]:
+        f = feeds_for(s)
+        sess.run(None, f)
+        t = time.time()
+        for _ in range(reps):
+            sess.run(None, f)
+        ts.append((time.time() - t) / reps * 1000)
+    return float(np.median(ts))
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--ckpt", required=True)
+    ap.add_argument("--data", required=True)
+    ap.add_argument("--out", required=True)
+    ap.add_argument("--calib", type=int, default=256)
+    ap.add_argument("--only-float", action="store_true", help="fp32/fp16 only (calibrated 8/4-bit come from quantize.py)")
+    args = ap.parse_args()
+    os.makedirs(args.out, exist_ok=True)
+    import onnx, onnxruntime as ort
+    ck = torch.load(args.ckpt, map_location="cpu")
+    model = NadirNet(**{k: v for k, v in ck["cfg"].items()}); model.load_state_dict(ck["model"]); model.eval()
+    data = pickle.load(open(args.data, "rb"))
+    val = [prep(s) for s in data["val"]]
+    rng = np.random.default_rng(0)
+    calib = [prep(s) for s in rng.choice(np.array(data["train"], dtype=object), size=min(args.calib, len(data["train"])), replace=False)]
+    s0 = max(val, key=lambda s: len(s["role"]))
+    ex = feeds_for(s0)
+    par_samples = val[:8]
+    with torch.no_grad():
+        ref_out = [model(*[torch.from_numpy(feeds_for(s)[k]) for k in NAMES_IN])[0].numpy() for s in par_samples]
+    fp32 = os.path.join(args.out, "nadir-fp32.onnx")
+    torch.onnx.export(Wrapped(model), tuple(torch.from_numpy(ex[k]) for k in NAMES_IN), fp32, input_names=NAMES_IN, output_names=NAMES_OUT,
+                      dynamic_axes={"tok_f": {1: "N"}, "tok_role": {1: "N"}, "tok_lab": {1: "N"}, "img_f": {1: "M"}, "img_thumb": {1: "M"},
+                                    "rel": {1: "T", 2: "T"}, "mask": {1: "T"}, "lab": {1: "N"}, "alpha_logit": {1: "N"}, "img_logits": {1: "M"}},
+                      opset_version=17, do_constant_folding=True, dynamo=False)
+    onnx.checker.check_model(fp32)
+    so = ort.SessionOptions(); so.intra_op_num_threads = 4
+    ref = ort.InferenceSession(fp32, so, providers=["CPUExecutionProvider"])
+    # parity torch vs onnx (reference computed before export)
+    parity = max(float(np.abs(r - ref.run(None, feeds_for(s))[0]).max()) for r, s in zip(ref_out, par_samples))
+    report = {"params": sum(p.numel() for p in model.parameters()), "parity_max_abs": parity, "variants": {}}
+    print("parity", parity, flush=True)
+
+    # fp16 (weights + compute in fp16, IO kept fp32)
+    # fp16: export a half-precision copy with fp32 inputs/outputs (casts inside the graph)
+    have16 = True
+    try:
+        class Half(nn.Module):
+            def __init__(self, m):
+                super().__init__(); self.m = m.half()
+            def forward(self, tok_f, tok_role, tok_lab, img_f, img_thumb, page_f, src, rel, mask):
+                o = self.m(tok_f.half(), tok_role, tok_lab.half(), img_f.half(), img_thumb.half(), page_f.half(), src, rel.half(), mask)
+                return tuple(x.float() for x in o)
+        import copy
+        torch.onnx.export(Half(copy.deepcopy(model)), tuple(torch.from_numpy(ex[k]) for k in NAMES_IN), os.path.join(args.out, "nadir-fp16.onnx"),
+                          input_names=NAMES_IN, output_names=NAMES_OUT,
+                          dynamic_axes={"tok_f": {1: "N"}, "tok_role": {1: "N"}, "tok_lab": {1: "N"}, "img_f": {1: "M"}, "img_thumb": {1: "M"},
+                                        "rel": {1: "T", 2: "T"}, "mask": {1: "T"}, "lab": {1: "N"}, "alpha_logit": {1: "N"}, "img_logits": {1: "M"}},
+                          opset_version=17, do_constant_folding=True, dynamo=False)
+        ort.InferenceSession(os.path.join(args.out, "nadir-fp16.onnx"), providers=["CPUExecutionProvider"])
+    except Exception as e:
+        print("fp16 failed:", str(e)[:300], flush=True); have16 = False
+
+    if args.only_float:
+        variants = {"fp32": fp32}
+        if have16: variants["fp16"] = os.path.join(args.out, "nadir-fp16.onnx")
+        for name, path in variants.items():
+            sess = ort.InferenceSession(path, so, providers=["CPUExecutionProvider"])
+            r = fidelity(sess, ref, val[:300]); r["size_mb"] = round(os.path.getsize(path) / 1e6, 2); r["cpu_ms_median"] = round(bench(sess, val[:300]), 2)
+            report["variants"][name] = r
+            print(name, r, flush=True)
+        json.dump(report, open(os.path.join(args.out, "export_report.json"), "w"), indent=1)
+        return
+    from onnxruntime.quantization import quantize_dynamic, quantize_static, QuantType, QuantFormat, CalibrationDataReader, CalibrationMethod
+    from onnxruntime.quantization.shape_inference import quant_pre_process
+    pre = os.path.join(args.out, "_pre.onnx")
+    quant_pre_process(fp32, pre, skip_symbolic_shape=True)
+
+    class Reader(CalibrationDataReader):
+        def __init__(self, ss):
+            self.it = iter([feeds_for(s, pad=(256, 32)) for s in ss])
+        def get_next(self):
+            return next(self.it, None)
+
+    # 8-bit A: per-channel int8 weights, dynamic activations
+    q8d = os.path.join(args.out, "nadir-int8-dynamic.onnx")
+    quantize_dynamic(pre, q8d, weight_type=QuantType.QInt8, per_channel=True, op_types_to_quantize=["MatMul", "Gemm"])
+    # 8-bit B: static QDQ, activations calibrated on real pages (percentile)
+    q8s = os.path.join(args.out, "nadir-int8-static.onnx")
+    try:
+        quantize_static(pre, q8s, Reader(calib[:128]), quant_format=QuantFormat.QDQ, per_channel=True, activation_type=QuantType.QInt8,
+                        weight_type=QuantType.QInt8, op_types_to_quantize=["MatMul", "Gemm"], calibrate_method=CalibrationMethod.Percentile,
+                        extra_options={"CalibPercentile": 99.99, "ActivationSymmetric": True, "WeightSymmetric": True})
+    except Exception as e:
+        print("static int8 failed:", e, flush=True); q8s = None
+    # 4-bit: block-wise weights (MatMulNBits), GPTQ calibrated on real pages, RTN fallback
+    q4 = os.path.join(args.out, "nadir-int4.onnx")
+    from onnxruntime.quantization import matmul_nbits_quantizer as mnq
+    done4 = False
+    # GPTQ needs reduction dims that are multiples of the block size; leave tiny matrices (relation bias, 11 wide) in fp32
+    pm = onnx.load(pre)
+    inits = {i.name: i for i in pm.graph.initializer}
+    skip = [n.name for n in pm.graph.node if n.op_type == "MatMul" and len(n.input) > 1 and n.input[1] in inits and inits[n.input[1]].dims[0] % 32 != 0]
+    report["int4_fp32_nodes"] = len(skip)
+    for algo in ("gptq", "hqq", "rtn"):
+        try:
+            if algo == "gptq":
+                cfg = mnq.GPTQWeightOnlyQuantConfig(calibration_data_reader=Reader(calib), percdamp=0.01, block_size=32, actorder=True, mse=True)
+            elif algo == "hqq":
+                cfg = mnq.HQQWeightOnlyQuantConfig(block_size=32, bits=4)
+            else:
+                cfg = mnq.RTNWeightOnlyQuantConfig()
+            qz = mnq.MatMulNBitsQuantizer(onnx.load(pre), block_size=32, is_symmetric=False, accuracy_level=4, algo_config=cfg, nodes_to_exclude=skip)
+            qz.process()
+            qz.model.save_model_to_file(q4, use_external_data_format=False)
+            report["int4_algo"] = algo
+            done4 = True
+            break
+        except Exception as e:
+            print(f"int4 {algo} failed: {str(e)[:300]}", flush=True)
+    variants = {"fp32": fp32, "int8-dynamic": q8d}
+    if have16: variants["fp16"] = os.path.join(args.out, "nadir-fp16.onnx")
+    if q8s: variants["int8-static"] = q8s
+    if done4: variants["int4"] = q4
+    evs = val[:300]
+    for name, path in variants.items():
+        sess = ort.InferenceSession(path, so, providers=["CPUExecutionProvider"])
+        r = fidelity(sess, ref, evs)
+        r["size_mb"] = round(os.path.getsize(path) / 1e6, 2)
+        r["cpu_ms_median"] = round(bench(sess, evs), 2)
+        report["variants"][name] = r
+        print(name, r, flush=True)
+    os.remove(pre)
+    json.dump(report, open(os.path.join(args.out, "export_report.json"), "w"), indent=1)
+
+
+if __name__ == "__main__":
+    main()

training/haiku_gen.py

@@ -0,0 +1,621 @@
+#!/usr/bin/env python3
+"""Nadir teacher data: Claude Haiku 5.5 designs a dark theme for each crawled light page.
+
+Runs on Vega's host (the API key vault proxy is only reachable there). Writes one JSON per page to
+OUT/<id>.json and keeps a running cost ledger; stops at --budget dollars.
+"""
+import argparse, asyncio, base64, gzip, io, json, math, os, random, re, sys, time
+from pathlib import Path
+
+import anthropic
+from PIL import Image, ImageDraw, ImageFont
+
+MODEL = "claude-haiku-5-5"
+PRICE_IN, PRICE_OUT = 0.10 / 1e6, 0.50 / 1e6          # prompts <= 100K tokens
+PRICE_CACHE_READ, PRICE_CACHE_WRITE = 0.01 / 1e6, 0.125 / 1e6
+ROLES = ["bg", "text", "border", "shadow", "gradient", "fill", "stroke"]
+TREATMENTS = ["keep", "dim", "invert", "light", "plate"]
+
+SYSTEM = """You are a senior product designer. A company has a light-themed website and has asked you to design its dark theme: the version their own design team would ship, not a mechanical inversion. You are given a screenshot of the light page, a contact sheet of its images, and the complete inventory of colors the page uses, each with its role and where it appears. You return the dark-theme replacement for every color and a treatment for every image.
+
+How to design it:
+- Surfaces. Choose one base dark surface for the page canvas. Near-black neutrals in the #0d0f12-#1c1e22 range work for most sites; a faint tint of the brand hue is welcome when it suits the brand. Avoid pure #000000 unless the brand is starkly monochrome. Keep the surface hierarchy of the light design but express it in dark-UI terms: anything that sits ON another surface (cards, panels, popovers, inputs, sticky headers, table stripes) is slightly LIGHTER than what is beneath it, in small steps (about 3-7 points of lightness). Alternating light-gray section bands become subtly different dark tones. Never let a nested surface end up darker than its parent unless it is a deliberate inset (code wells, input fields may be a touch darker).
+- Colors that are already dark in the light design (dark heroes, footers, banners, code blocks, dark buttons) stay dark. Harmonize them with the new palette; never flip them to light.
+- Text. Primary text is light but not pure white for body copy (around #e4e6ea-#f0f1f3). Secondary and muted text keep their hierarchy as progressively dimmer grays, and every text color must keep at least 4.5:1 contrast on the surface it actually sits on (3:1 for large or bold headings). Text that sits on a colored or dark-in-light-mode surface keeps working on that surface's new color.
+- Brand and accent colors keep their hue so the brand stays recognizable. When an accent is used as text, a link or an icon on a dark surface, raise its lightness and slightly reduce chroma until it reads comfortably. When an accent is a filled button or badge background, keep it close to the original (deepen slightly if very light) and keep its label readable.
+- Status colors (error, warning, success, info) keep their meaning and become readable on dark.
+- Borders and dividers become subtle lines slightly lighter than the surface they sit on. Focus rings stay clearly visible.
+- Translucent colors (8-digit hex): keep a similar alpha unless it no longer works. Black-tinted overlays used to darken or separate things in light mode usually become white-tinted at a similar or slightly lower alpha (rgba(0,0,0,.05) hover -> #ffffff0f). Shadows and scrims stay dark; shadows may get a little stronger.
+- Hover, focus and active colors (marked as rule-only / state) keep their relationship to the resting state: if hover was darker in light mode, make it visibly lighter (or more emphasized) in dark mode.
+- Overlays drawn on top of videos or images (video player controls, captions on photos, scrims, gradient fades over media) and the black backgrounds of media players are part of the media: return them unchanged.
+- Syntax highlighting and data-visualization colors stay distinguishable from each other and readable on dark.
+- The same input color in the same role always gets the same output. Every listed id must appear in your answer exactly once.
+
+Images (choose one treatment each; the contact sheet shows transparency as a checkerboard):
+- keep: photos, video thumbnails, illustrations, screenshots of colorful UI, colorful logos or icons that remain clear on a dark background, avatars, product shots on their own backgrounds.
+- dim: large, very bright images (mostly white backgrounds, bright illustrations) that would glare on a dark page; they are shown slightly dimmed.
+- invert: dark-on-light or dark-on-transparent line art, diagrams, charts, black text images, and mostly-monochrome logos whose few colored parts still look right after a hue-preserving inversion.
+- light: monochrome dark glyphs, icons and wordmarks on transparent backgrounds that should simply become near-white.
+- plate: logos or images with important dark parts on transparency whose colors must not be altered (multicolor brand marks, partner logos); they get a subtle light rounded backing.
+Think about the site logo especially carefully: it must stay recognizable and clearly visible.
+
+Answer with JSON only, following the schema. Use "#rrggbb", or "#rrggbbaa" when alpha is below 1."""
+
+SCHEMA = {
+    "type": "object",
+    "properties": {
+        "design_notes": {"type": "string"},
+        "colors": {"type": "array", "items": {"type": "object", "properties": {"id": {"type": "string"}, "dark": {"type": "string"}},
+                                              "required": ["id", "dark"], "additionalProperties": False}},
+        "images": {"type": "array", "items": {"type": "object", "properties": {"id": {"type": "string"}, "treatment": {"type": "string", "enum": TREATMENTS}},
+                                              "required": ["id", "treatment"], "additionalProperties": False}},
+    },
+    "required": ["design_notes", "colors", "images"],
+    "additionalProperties": False,
+}
+
+
+def hexa(rgba):
+    r, g, b, a = rgba
+    h = "#%02x%02x%02x" % (r, g, b)
+    if a < 0.999:
+        h += "%02x" % round(a * 255)
+    return h
+
+
+def pct(x):
+    if x >= 0.095:
+        return f"{x * 100:.0f}%"
+    if x >= 0.0005:
+        return f"{x * 100:.1f}%"
+    return "<0.1%"
+
+
+def describe_tokens(page, max_tokens):
+    toks = page["tokens"][:max_tokens]
+    n = len(toks)
+    pa = max(1.0, page["docW"] * min(page["docH"], page["vh"] * 8))
+    va = page["vw"] * page["vh"]
+    # relations: (type, a, b, w)
+    best = {}
+    for typ, a, b, w in page["rel"]:
+        if a >= n or b >= n:
+            continue
+        best.setdefault((typ, a), []).append((w, b))
+        if typ in (0, 1):
+            best.setdefault((10 + typ, b), []).append((w, a))  # reverse: what sits on b
+    def top(key, k=1):
+        lst = sorted(best.get(key, []), reverse=True)[:k]
+        return [f"t{b}" for _, b in lst]
+    lines = []
+    for i, t in enumerate(toks):
+        d = t["d"]
+        role = ROLES[t["role"]]
+        parts = [f"t{i}", role, hexa(t["rgba"])]
+        info = []
+        if d["root"]:
+            info.append("PAGE CANVAS (html/body background)")
+        visible = d["nVis"] > 0
+        if role == "bg":
+            if d["area"]:
+                info.append(f"covers {pct(min(1, d['area'] / pa))} of page (unobstructed {pct(min(1, d['own'] / pa))}), first screen {pct(min(1, d['area1'] / va))}")
+            on = top((1, i))
+            if on and not d["root"]:
+                info.append(f"sits on {on[0]}")
+            txt = top((10, i), 3)
+            if txt:
+                info.append("text on it: " + ",".join(txt))
+        elif role in ("text", "fill", "stroke"):
+            if role == "text" and d["chars"]:
+                info.append(f"{d['chars']} chars, ~{d['fs']:.0f}px")
+            elif role != "text":
+                info.append("SVG " + ("fill (icons/logos/shapes)" if role == "fill" else "stroke (icons/lines)"))
+            on = top((0, i))
+            if on:
+                info.append(f"on {on[0]}" + (f" (contrast {d['contrast']:.1f}:1)" if d.get("contrast") else ""))
+        elif role == "border":
+            on = top((2, i))
+            if on:
+                info.append(f"line on {on[0]}")
+        elif role == "shadow":
+            info.append("box/text-shadow color")
+        elif role == "gradient":
+            info.append("gradient stop in a background-image")
+        if d["n"]:
+            info.append(f"{d['n']} elements" + ("" if visible else ", none visible now (hidden menus etc.)"))
+        flags = []
+        n_el = max(1, d["n"])
+        if d.get("media", 0) / n_el >= 0.5:
+            info.append("ON TOP OF / CONTAINS video or images (player UI, caption, scrim)")
+        for key, label in (("inter", "interactive"), ("link", "in links"), ("head", "headings"), ("header", "header"), ("nav", "nav"), ("footer", "footer")):
+            if d.get(key, 0) / n_el >= 0.3:
+                flags.append(label)
+        if flags:
+            info.append("/".join(flags))
+        if d["rules"] and not d["n"]:
+            info.append(f"CSS-rule only ({d['rules']} rules" + (f", {d['state']} hover/focus/active" if d["state"] else "") + ")")
+        elif d["state"]:
+            info.append(f"also in {d['state']} hover/focus/active rules")
+        if d.get("ex"):
+            info.append("e.g. " + "; ".join(x[:70] for x in d["ex"][:2]))
+        lines.append(" ".join(parts) + " | " + " | ".join(info))
+    return lines
+
+
+def describe_images(page, max_images):
+    out = []
+    for j, im in enumerate(page["images"][:max_images]):
+        s = im.get("stats") or {}
+        bits = [f"i{j}", f"{im['w']}x{im['h']}px", im.get("ext") or im["kind"]]
+        if im["kind"] == "bg":
+            bits.append("CSS background image")
+        where = []
+        if im.get("header"): where.append("header/nav")
+        if im.get("footer"): where.append("footer")
+        if im.get("homeLink"): where.append("links to homepage")
+        elif im.get("inLink"): where.append("inside a link")
+        if im["y"] < page["vh"]: where.append("first screen")
+        if where: bits.append(", ".join(where))
+        if im.get("logo"): bits.append("named like a logo")
+        if im.get("icon"): bits.append("named like an icon")
+        if im.get("alt"): bits.append(f'alt "{im["alt"][:50]}"')
+        if s:
+            bits.append(f"transparent {pct(s.get('transp', 0))}, opaque mean lightness {s.get('meanL', 0):.2f}, near-white {pct(s.get('white', 0))}, near-black {pct(s.get('black', 0))}, chroma {s.get('meanC', 0):.2f}")
+        else:
+            bits.append("pixels unavailable")
+        if im.get("underTok", -1) >= 0:
+            bits.append(f"on surface t{im['underTok']}")
+        if im.get("count", 1) > 1:
+            bits.append(f"used {im['count']}x")
+        out.append(" | ".join(bits))
+    return out
+
+
+def contact_sheet(page, max_images):
+    ims = [im for im in page["images"][:max_images]]
+    if not ims:
+        return None
+    cols = 6
+    cell, pad, lab = 112, 8, 16
+    rows = math.ceil(len(ims) / cols)
+    W, H = cols * (cell + pad) + pad, rows * (cell + pad + lab) + pad
+    sheet = Image.new("RGB", (W, H), (255, 255, 255))
+    dr = ImageDraw.Draw(sheet)
+    try:
+        font = ImageFont.load_default(size=13)
+    except Exception:
+        font = ImageFont.load_default()
+    for j, im in enumerate(ims):
+        x = pad + (j % cols) * (cell + pad)
+        y = pad + (j // cols) * (cell + pad + lab)
+        # checkerboard
+        for cy in range(0, cell, 8):
+            for cx in range(0, cell, 8):
+                c = (204, 204, 204) if ((cx + cy) // 8) % 2 else (245, 245, 245)
+                dr.rectangle([x + cx, y + lab + cy, x + cx + 7, y + lab + cy + 7], fill=c)
+        dr.text((x, y), f"i{j}", fill=(200, 0, 0), font=font)
+        if im.get("big"):
+            try:
+                b = base64.b64decode(im["big"].split(",", 1)[1])
+                th = Image.open(io.BytesIO(b)).convert("RGBA").resize((cell, cell), Image.LANCZOS)
+                sheet.paste(th, (x, y + lab), th)
+            except Exception:
+                dr.text((x + 4, y + lab + 40), "n/a", fill=(0, 0, 0), font=font)
+        else:
+            dr.text((x + 4, y + lab + 40), "n/a", fill=(0, 0, 0), font=font)
+    buf = io.BytesIO()
+    sheet.save(buf, "PNG", optimize=True)
+    return buf.getvalue()
+
+
+def screenshot(pdir, max_w=900, max_h=1400):
+    im = Image.open(pdir / "shot.jpg").convert("RGB")
+    s = max_w / im.width
+    im = im.resize((max_w, int(im.height * s)), Image.LANCZOS)
+    if im.height > max_h:
+        im = im.crop((0, 0, max_w, max_h))
+    buf = io.BytesIO()
+    im.save(buf, "JPEG", quality=82)
+    return buf.getvalue()
+
+
+def build_request(page, pdir, args):
+    lines = describe_tokens(page, args.max_tokens)
+    imgs = describe_images(page, args.max_images)
+    sheet = contact_sheet(page, args.max_images) if imgs else None
+    host = re.sub(r"^https?://", "", page["url"]).split("/")[0]
+    text = [f"Website: {host} — \"{page['title'][:100]}\"", f"Viewport 1280x{page['vh']}, page height {page['docH']}px.",
+            "", f"COLOR INVENTORY ({len(lines)} colors; id role color | usage):"] + lines
+    if imgs:
+        text += ["", f"IMAGES ({len(imgs)}; thumbnails in the contact sheet):"] + imgs
+    text += ["", f"Return the dark-theme color for every id t0-t{len(lines) - 1}" + (f" and a treatment for every image i0-i{len(imgs) - 1}." if imgs else " (images: return an empty list).")]
+    content = [{"type": "text", "text": "Screenshot of the light page (top section):"},
+               {"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": base64.b64encode(screenshot(pdir)).decode()}}]
+    if sheet:
+        content += [{"type": "text", "text": "Contact sheet of the page's images:"},
+                    {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": base64.b64encode(sheet).decode()}}]
+    content.append({"type": "text", "text": "\n".join(text)})
+    return content, len(lines), len(imgs)
+
+
+HEX_RE = re.compile(r"^#?([0-9a-fA-F]{6})([0-9a-fA-F]{2})?$")
+
+
+def parse_answer(msg, n_tok, n_img):
+    txt = "".join(b.text for b in msg.content if b.type == "text")
+    data = json.loads(txt)
+    colors, images = {}, {}
+    for c in data.get("colors", []):
+        m = HEX_RE.match(c.get("dark", "").strip())
+        mid = re.match(r"^t(\d+)$", c.get("id", ""))
+        if m and mid and int(mid.group(1)) < n_tok:
+            colors[int(mid.group(1))] = "#" + m.group(1).lower() + (m.group(2) or "").lower()
+    for c in data.get("images", []):
+        mid = re.match(r"^i(\d+)$", c.get("id", ""))
+        if mid and int(mid.group(1)) < n_img and c.get("treatment") in TREATMENTS:
+            images[int(mid.group(1))] = c["treatment"]
+    return data.get("design_notes", ""), colors, images
+
+
+class Gate:
+    """Adaptive concurrency: grow slowly while healthy, back off on the org concurrency cap and stay below it."""
+    def __init__(self, start, cap):
+        self.limit, self.cap, self.active = float(start), cap, 0
+        self.cond = asyncio.Condition()
+        self.last429 = 0.0
+
+    async def acquire(self):
+        async with self.cond:
+            while self.active >= int(self.limit):
+                await self.cond.wait()
+            self.active += 1
+
+    async def release(self, ok=True, limited=False):
+        async with self.cond:
+            self.active -= 1
+            if limited:
+                seen = self.active + 1
+                self.cap = max(2, min(self.cap, int(seen * 0.75)))
+                self.limit = max(2.0, min(self.limit * 0.7, self.cap))
+                self.last429 = time.time()
+            elif ok and time.time() - self.last429 > 120:
+                self.limit = min(self.cap, self.limit + 0.2)
+            self.cond.notify_all()
+
+
+class Ledger:
+    def __init__(self, path, budget):
+        self.path, self.budget = Path(path), budget
+        self.cost = 0.0
+        self.n = 0
+        if self.path.exists():
+            for line in self.path.open():
+                try:
+                    r = json.loads(line)
+                    self.cost += r["cost"]; self.n += 1
+                except Exception:
+                    pass
+        self.f = self.path.open("a")
+
+    def add(self, rec):
+        self.cost += rec["cost"]; self.n += 1
+        self.f.write(json.dumps(rec) + "\n"); self.f.flush()
+
+
+def usage_cost(u):
+    cr = getattr(u, "cache_read_input_tokens", 0) or 0
+    cw = getattr(u, "cache_creation_input_tokens", 0) or 0
+    return u.input_tokens * PRICE_IN + u.output_tokens * PRICE_OUT + cr * PRICE_CACHE_READ + cw * PRICE_CACHE_WRITE
+
+
+ERRLOG = {}
+
+
+def logerr(kind, e):
+    ERRLOG[kind] = ERRLOG.get(kind, 0) + 1
+    if ERRLOG[kind] <= 3:
+        print(f"[{kind}] {type(e).__name__}: {str(e)[:300]}", flush=True)
+
+
+async def one(client, sem, pid, crawl_dir, out_dir, ledger, args, stats):
+    out = out_dir / f"{pid}.json"
+    if out.exists():
+        return
+    pdir = crawl_dir / "pages" / pid
+    try:
+        page = json.load(gzip.open(pdir / "light.json.gz"))
+        content, n_tok, n_img = build_request(page, pdir, args)
+    except Exception as e:
+        stats["prep_err"] = stats.get("prep_err", 0) + 1
+        return
+    for attempt in range(8):
+        if ledger.cost >= args.budget:
+            return
+        await sem.acquire()
+        ok, limited = False, False
+        t0 = time.time()
+        try:
+            async with client.messages.stream(
+                model=MODEL, max_tokens=args.max_out, system=SYSTEM,
+                thinking={"type": "adaptive"}, output_config={"effort": args.effort, "format": {"type": "json_schema", "schema": SCHEMA}},
+                messages=[{"role": "user", "content": content}],
+            ) as stream:
+                msg = await stream.get_final_message()
+            ok = True
+        except anthropic.RateLimitError as e:
+            stats["429"] = stats.get("429", 0) + 1
+            limited = "concurrent" in str(e).lower()
+            logerr("429", e)
+        except anthropic.APIStatusError as e:
+            stats[f"http{e.status_code}"] = stats.get(f"http{e.status_code}", 0) + 1
+            logerr(f"http{e.status_code}", e)
+            if e.status_code in (400, 401, 403, 404, 413):
+                (out_dir / "errors").mkdir(exist_ok=True)
+                (out_dir / "errors" / f"{pid}.txt").write_text(str(e)[:2000])
+                await sem.release(False)
+                return
+        except (anthropic.APIConnectionError, anthropic.APITimeoutError) as e:
+            stats["conn"] = stats.get("conn", 0) + 1
+            logerr("conn", e)
+        except Exception as e:
+            stats["other"] = stats.get("other", 0) + 1
+            logerr("other", e)
+        await sem.release(ok, limited)
+        if not ok:
+            await asyncio.sleep(min(60, 4 * (attempt + 1)) + random.random() * 4)
+            continue
+        cost = usage_cost(msg.usage)
+        rec = {"id": pid, "cost": cost, "in": msg.usage.input_tokens, "out": msg.usage.output_tokens, "stop": msg.stop_reason,
+               "secs": round(time.time() - t0, 1), "t": time.time()}
+        ledger.add(rec)
+        if msg.stop_reason == "refusal":
+            stats["refusal"] = stats.get("refusal", 0) + 1
+            out.write_text(json.dumps({"id": pid, "refusal": True, "usage": rec}))
+            return
+        try:
+            notes, colors, images = parse_answer(msg, n_tok, n_img)
+        except Exception as e:
+            stats["parse"] = stats.get("parse", 0) + 1
+            (out_dir / "errors").mkdir(exist_ok=True)
+            (out_dir / "errors" / f"{pid}.txt").write_text("parse: " + str(e)[:500])
+            return
+        out.write_text(json.dumps({"id": pid, "url": page["url"], "n_tok": n_tok, "n_img": n_img, "notes": notes,
+                                   "colors": colors, "images": images, "usage": rec, "effort": args.effort, "model": MODEL}))
+        stats["ok"] = stats.get("ok", 0) + 1
+        return
+
+
+# ---------------- batch mode ----------------
+import urllib.request
+
+PROXY = os.environ.get("ANTHROPIC_BASE_URL", "https://api.anthropic.com")
+HDRS = {"x-api-key": os.environ.get("ANTHROPIC_API_KEY", ""), "anthropic-version": "2023-06-01", "content-type": "application/json"}
+
+
+def http(method, path, body=None, timeout=300):
+    req = urllib.request.Request(PROXY + path, data=json.dumps(body).encode() if body is not None else None, headers=HDRS, method=method)
+    with urllib.request.urlopen(req, timeout=timeout) as r:
+        return r.read()
+
+
+def params_for(content, args):
+    return {"model": MODEL, "max_tokens": args.max_out, "system": SYSTEM, "thinking": {"type": "adaptive"},
+            "output_config": {"effort": args.effort, "format": {"type": "json_schema", "schema": SCHEMA}},
+            "messages": [{"role": "user", "content": content}]}
+
+
+class Msg:  # minimal adapter so parse_answer works on batch result dicts
+    def __init__(self, d):
+        self.content = [type("B", (), {"type": b.get("type"), "text": b.get("text", "")}) for b in d.get("content", [])]
+
+
+def collect_batch(bid, meta, crawl_dir, out_dir, ledger, stats):
+    raw = http("GET", f"/v1/messages/batches/{bid}/results", timeout=600).decode()
+    seen = set()
+    for line in raw.splitlines():
+        if not line.strip():
+            continue
+        r = json.loads(line)
+        pid = r["custom_id"]
+        seen.add(pid)
+        res = r["result"]
+        if res["type"] != "succeeded":
+            stats["b_" + res["type"]] = stats.get("b_" + res["type"], 0) + 1
+            continue
+        m = res["message"]
+        u = m.get("usage", {})
+        cost = 0.5 * (u.get("input_tokens", 0) * PRICE_IN + u.get("output_tokens", 0) * PRICE_OUT
+                      + (u.get("cache_read_input_tokens") or 0) * PRICE_CACHE_READ + (u.get("cache_creation_input_tokens") or 0) * PRICE_CACHE_WRITE)
+        rec = {"id": pid, "cost": cost, "in": u.get("input_tokens", 0), "out": u.get("output_tokens", 0), "stop": m.get("stop_reason"), "batch": bid, "t": time.time()}
+        ledger.add(rec)
+        out = out_dir / f"{pid}.json"
+        if m.get("stop_reason") == "refusal":
+            stats["refusal"] = stats.get("refusal", 0) + 1
+            out.write_text(json.dumps({"id": pid, "refusal": True, "usage": rec}))
+            continue
+        n_tok, n_img = meta[pid]
+        try:
+            notes, colors, images = parse_answer(Msg(m), n_tok, n_img)
+        except Exception as e:
+            stats["parse"] = stats.get("parse", 0) + 1
+            continue
+        url = ""
+        out.write_text(json.dumps({"id": pid, "n_tok": n_tok, "n_img": n_img, "notes": notes, "colors": colors, "images": images,
+                                   "usage": rec, "effort": args_global.effort, "model": MODEL}))
+        stats["ok"] = stats.get("ok", 0) + 1
+    return seen
+
+
+args_global = None
+
+
+def batch_main(args):
+    global args_global
+    args_global = args
+    crawl_dir, out_dir = Path(args.crawl), Path(args.out)
+    out_dir.mkdir(parents=True, exist_ok=True)
+    ledger = Ledger(out_dir / "ledger.jsonl", args.budget)
+    state_p = out_dir / "batches.jsonl"
+    batches = {}
+    if state_p.exists():
+        for line in state_p.open():
+            b = json.loads(line)
+            batches.setdefault(b["id"], {}).update(b)
+    submitted = set(pid for b in batches.values() for pid in b.get("ids", []))
+    stats = {}
+    est_per = 0.0035
+    last_submit = 0
+    while True:
+        # collect finished batches
+        pending_est = 0.0
+        for bid, b in list(batches.items()):
+            if b.get("collected"):
+                continue
+            try:
+                info = json.loads(http("GET", f"/v1/messages/batches/{bid}"))
+            except Exception as e:
+                print("poll error", bid, str(e)[:200], flush=True)
+                pending_est += len(b["ids"]) * est_per
+                continue
+            if info["processing_status"] == "ended":
+                try:
+                    collect_batch(bid, b["meta"], crawl_dir, out_dir, ledger, stats)
+                    b["collected"] = True
+                    with state_p.open("a") as f:
+                        f.write(json.dumps({"id": bid, "collected": True}) + "\n")
+                    print(f"[{time.strftime('%H:%M:%S')}] collected {bid} {info['request_counts']}", flush=True)
+                except Exception as e:
+                    print("collect error", bid, str(e)[:300], flush=True)
+                    pending_est += len(b["ids"]) * est_per
+            else:
+                pending_est += len(b["ids"]) * est_per
+        if ledger.n > 50:
+            est_per = max(0.002, ledger.cost / ledger.n)
+        # candidates
+        cands = []
+        if args.ids:
+            cands = [l.strip() for l in open(args.ids) if l.strip()]
+        else:
+            for line in (l for ip in sorted(crawl_dir.glob("index*.jsonl")) for l in open(ip)):
+                try:
+                    r = json.loads(line)
+                except Exception:
+                    continue
+                if r.get("kind") in args.kinds.split(",") and (r.get("n_tokens") or 0) >= 8 and (r.get("status") or 200) < 400:
+                    cands.append(r["id"])
+        new = [c for c in cands if c not in submitted and not (out_dir / f"{c}.json").exists()]
+        room = args.budget - ledger.cost - pending_est
+        can = int(max(0, room) / est_per)
+        ready = len(new) >= args.batch_size or (new and time.time() - last_submit > args.max_wait)
+        if ready and can > 0:
+            chunk = new[: min(args.batch_size, can)]
+            reqs, meta = [], {}
+            for pid in chunk:
+                pdir = crawl_dir / "pages" / pid
+                try:
+                    page = json.load(gzip.open(pdir / "light.json.gz"))
+                    content, n_tok, n_img = build_request(page, pdir, args)
+                except Exception as e:
+                    submitted.add(pid)
+                    continue
+                reqs.append({"custom_id": pid, "params": params_for(content, args)})
+                meta[pid] = (n_tok, n_img)
+            if reqs:
+                try:
+                    info = json.loads(http("POST", "/v1/messages/batches", {"requests": reqs}, timeout=900))
+                    b = {"id": info["id"], "ids": list(meta), "meta": meta, "t": time.time()}
+                    batches[info["id"]] = b
+                    with state_p.open("a") as f:
+                        f.write(json.dumps(b) + "\n")
+                    submitted.update(meta)
+                    last_submit = time.time()
+                    print(f"[{time.strftime('%H:%M:%S')}] submitted {info['id']} with {len(reqs)} (spent ${ledger.cost:.2f}, pending ~${pending_est:.2f}, est/page ${est_per:.4f})", flush=True)
+                except Exception as e:
+                    print("submit error", str(e)[:300], flush=True)
+                    time.sleep(60)
+        print(f"[{time.strftime('%H:%M:%S')}] spent ${ledger.cost:.3f} on {ledger.n} | open batches {sum(1 for b in batches.values() if not b.get('collected'))} | waiting pages {len(new)} | {stats}", flush=True)
+        open_batches = [b for b in batches.values() if not b.get("collected")]
+        if not args.follow and not open_batches and (not new or can <= 0):
+            break
+        if room <= 0 and not open_batches:
+            print("budget reached", flush=True)
+            break
+        time.sleep(args.poll)
+    print(f"DONE ${ledger.cost:.3f} {ledger.n} {stats}", flush=True)
+
+
+async def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--crawl", required=True)
+    ap.add_argument("--out", required=True)
+    ap.add_argument("--ids", help="file with page ids (default: all light pages in index)")
+    ap.add_argument("--budget", type=float, default=95.0)
+    ap.add_argument("--concurrency", type=int, default=64)
+    ap.add_argument("--effort", default="high")
+    ap.add_argument("--max-tokens", type=int, default=140)
+    ap.add_argument("--max-images", type=int, default=24)
+    ap.add_argument("--max-out", type=int, default=48000)
+    ap.add_argument("--limit", type=int, default=0)
+    ap.add_argument("--follow", action="store_true", help="keep polling the crawl index for new pages")
+    ap.add_argument("--mode", default="batch", choices=["batch", "realtime"])
+    ap.add_argument("--batch-size", type=int, default=250)
+    ap.add_argument("--max-wait", type=int, default=900)
+    ap.add_argument("--poll", type=int, default=60)
+    ap.add_argument("--kinds", default="light")
+    args = ap.parse_args()
+    if args.mode == "batch":
+        return batch_main(args)
+    crawl_dir, out_dir = Path(args.crawl), Path(args.out)
+    out_dir.mkdir(parents=True, exist_ok=True)
+    ledger = Ledger(out_dir / "ledger.jsonl", args.budget)
+    client = anthropic.AsyncAnthropic(base_url=PROXY, api_key=os.environ.get("ANTHROPIC_API_KEY", ""),
+                                      max_retries=0, timeout=600)
+    sem = Gate(min(8, args.concurrency), args.concurrency)
+    stats = {}
+    started = set()
+    tasks = []
+    t_start = time.time()
+
+    def candidates():
+        if args.ids:
+            return [l.strip() for l in open(args.ids) if l.strip()]
+        ids = []
+        for line in (l for ip in sorted(crawl_dir.glob("index*.jsonl")) for l in open(ip)):
+            try:
+                r = json.loads(line)
+            except Exception:
+                continue
+            if r.get("kind") == "light" and (r.get("n_tokens") or 0) >= 8 and (r.get("status") or 200) < 400:
+                ids.append(r["id"])
+        return ids
+
+    async def reporter():
+        while True:
+            await asyncio.sleep(60)
+            el = time.time() - t_start
+            print(f"[{time.strftime('%H:%M:%S')}] ${ledger.cost:.3f} spent, {ledger.n} calls, {stats}, {ledger.n / max(1, el) * 3600:.0f}/h, conc {sem.active}/{sem.limit:.1f} cap {sem.cap}", flush=True)
+    rep = asyncio.create_task(reporter())
+    while True:
+        ids = candidates()
+        new = [i for i in ids if i not in started]
+        if args.limit:
+            new = new[: max(0, args.limit - len(started))]
+        for pid in new:
+            started.add(pid)
+            tasks.append(asyncio.create_task(one(client, sem, pid, crawl_dir, out_dir, ledger, args, stats)))
+        tasks = [t for t in tasks if not t.done()]
+        if ledger.cost >= args.budget:
+            print("budget reached", flush=True)
+            break
+        if not args.follow and not tasks:
+            break
+        if args.limit and len(started) >= args.limit and not tasks:
+            break
+        await asyncio.sleep(30 if args.follow else 2)
+    await asyncio.gather(*tasks, return_exceptions=True)
+    rep.cancel()
+    print(f"DONE ${ledger.cost:.3f} {ledger.n} calls {stats}", flush=True)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

training/judge.py

@@ -0,0 +1,110 @@
+#!/usr/bin/env python3
+"""Blind pairwise judging of dark-mode renders with Claude Haiku 5.5 (Batch API). Runs on Vega's host."""
+import argparse, base64, io, json, random, time
+from pathlib import Path
+from PIL import Image
+
+import haiku_gen as hg
+
+JUDGE_SYSTEM = """You are a meticulous product-design reviewer. You compare two dark-mode versions (A and B) of the same web page, given the original light page for reference. Judge which dark mode a demanding user would rather browse with, considering in this order: (1) readability: every piece of text clearly legible against its background, nothing lost; (2) nothing broken: no large light patches left glaring, logos and icons still visible and recognizable, photos and videos not color-distorted; (3) design quality: surfaces, hierarchy and elevation preserved, brand colors recognizable, pleasant and coherent palette, no muddy or harsh colors. Small differences in exact shades matter less than legibility and breakage. Answer with JSON only."""
+
+SCHEMA = {"type": "object", "properties": {
+    "winner": {"type": "string", "enum": ["A", "B", "tie"]},
+    "score_A": {"type": "integer"}, "score_B": {"type": "integer"},
+    "reason": {"type": "string"}},
+    "required": ["winner", "score_A", "score_B", "reason"], "additionalProperties": False}
+
+
+def img_block(path, w=800, h=1000):
+    im = Image.open(path).convert("RGB")
+    s = w / im.width
+    im = im.resize((w, int(im.height * s)), Image.LANCZOS).crop((0, 0, w, min(h, int(im.height * s))))
+    buf = io.BytesIO(); im.save(buf, "JPEG", quality=80)
+    return {"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": base64.b64encode(buf.getvalue()).decode()}}
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--renders", required=True, help="render output dir with one subdir per tag")
+    ap.add_argument("--crawl", required=True)
+    ap.add_argument("--pairs", default="nadir:dr,nadir:haiku")
+    ap.add_argument("--out", required=True)
+    ap.add_argument("--effort", default="medium")
+    ap.add_argument("--budget", type=float, default=3.0)
+    ap.add_argument("--chunk", type=int, default=400, help="requests per batch (3 images each, ~150 KB)")
+    args = ap.parse_args()
+    R, crawl, out = Path(args.renders), Path(args.crawl), Path(args.out)
+    out.mkdir(parents=True, exist_ok=True)
+    rng = random.Random(0)
+    reqs, meta = [], {}
+    for pair in args.pairs.split(","):
+        x, y = pair.split(":")
+        for rx in sorted((R / x).glob("*.json")):
+            pid = rx.stem
+            ry = R / y / f"{pid}.json"
+            if not ry.exists():
+                continue
+            jx, jy = json.loads(rx.read_text()), json.loads(ry.read_text())
+            if not (jx.get("ok") and jy.get("ok")) or not (R / x / f"{pid}.jpg").exists() or not (R / y / f"{pid}.jpg").exists():
+                continue
+            flip = rng.random() < 0.5
+            a, b = (y, x) if flip else (x, y)
+            cid = f"{pid}-{x}-{y}"
+            content = [{"type": "text", "text": "Original light page:"}, img_block(crawl / "pages" / pid / "shot.jpg"),
+                       {"type": "text", "text": "Dark mode A:"}, img_block(R / a / f"{pid}.jpg"),
+                       {"type": "text", "text": "Dark mode B:"}, img_block(R / b / f"{pid}.jpg"),
+                       {"type": "text", "text": "Which dark mode is better? Scores are 1-10."}]
+            reqs.append({"custom_id": cid, "params": {"model": hg.MODEL, "max_tokens": 16000, "system": JUDGE_SYSTEM, "thinking": {"type": "adaptive"},
+                                                      "output_config": {"effort": args.effort, "format": {"type": "json_schema", "schema": SCHEMA}},
+                                                      "messages": [{"role": "user", "content": content}]}})
+            meta[cid] = {"pid": pid, "x": x, "y": y, "A": a, "B": b}
+    print(len(reqs), "comparisons", flush=True)
+    est = len(reqs) * 0.0015
+    if est > args.budget:
+        reqs = reqs[: int(args.budget / 0.0015)]
+    results, cost = [], 0.0
+    bids = []
+    for k in range(0, len(reqs), args.chunk):  # submit every chunk first, then wait for all of them together
+        info = json.loads(hg.http("POST", "/v1/messages/batches", {"requests": reqs[k:k + args.chunk]}, timeout=900))
+        bids.append(info["id"])
+        print("submitted", info["id"], len(reqs[k:k + args.chunk]), flush=True)
+    pending = set(bids)
+    while pending:
+        time.sleep(30)
+        for bid in list(pending):
+            if json.loads(hg.http("GET", f"/v1/messages/batches/{bid}"))["processing_status"] == "ended":
+                pending.discard(bid)
+    for bid in bids:
+        raw = hg.http("GET", f"/v1/messages/batches/{bid}/results", timeout=600).decode()
+        for line in raw.splitlines():
+            if not line.strip():
+                continue
+            r = json.loads(line)
+            if r["result"]["type"] != "succeeded":
+                continue
+            m = r["result"]["message"]; u = m["usage"]
+            cost += 0.5 * (u["input_tokens"] * hg.PRICE_IN + u["output_tokens"] * hg.PRICE_OUT)
+            txt = "".join(b.get("text", "") for b in m["content"] if b.get("type") == "text")
+            try:
+                v = json.loads(txt)
+            except Exception:
+                continue
+            mt = meta[r["custom_id"]]
+            win = {"A": mt["A"], "B": mt["B"], "tie": "tie"}[v["winner"]]
+            sc = {mt["A"]: v["score_A"], mt["B"]: v["score_B"]}
+            results.append({**mt, "winner": win, "score_x": sc[mt["x"]], "score_y": sc[mt["y"]], "reason": v["reason"]})
+    summary = {}
+    for pair in args.pairs.split(","):
+        x, y = pair.split(":")
+        rs = [r for r in results if r["x"] == x and r["y"] == y]
+        if not rs:
+            continue
+        wx = sum(r["winner"] == x for r in rs); wy = sum(r["winner"] == y for r in rs); t = sum(r["winner"] == "tie" for r in rs)
+        summary[pair] = {"n": len(rs), f"{x}_wins": wx, f"{y}_wins": wy, "ties": t, f"{x}_winrate_excl_ties": round(wx / max(1, wx + wy), 3),
+                         f"mean_score_{x}": round(sum(r["score_x"] for r in rs) / len(rs), 2), f"mean_score_{y}": round(sum(r["score_y"] for r in rs) / len(rs), 2)}
+    json.dump({"summary": summary, "cost": cost, "results": results}, open(out / "judge.json", "w"), indent=1)
+    print(json.dumps(summary, indent=1), f"cost ${cost:.3f}", flush=True)
+
+
+if __name__ == "__main__":
+    main()

training/model.py

@@ -0,0 +1,141 @@
+"""NadirNet: a relation-biased set transformer over a page's colour palette and images.
+
+Tokens: [PAGE] + one token per (colour, role) the page uses + one token per image.
+Attention logits get a learned per-head bias from the page's structure graph (text-on-surface,
+surface-on-surface, border-on-surface, same-element, image-on-surface, same-colour), so the model
+reasons about hierarchy and contrast the way a designer would. Outputs a dark colour per token
+(OKLab, parameterised relative to the input so hue is kept unless the model chooses otherwise),
+an alpha, and a treatment class per image.
+"""
+import math
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+FT, FI, FP = 49, 39, 8
+N_ROLES, N_TREAT = 7, 5
+K_REL = 11
+
+
+def fourier_lab(lab, n=6):
+    x = torch.stack([lab[..., 0], lab[..., 1] * 4, lab[..., 2] * 4], -1)  # (..., 3)
+    freqs = (2.0 ** torch.arange(n, device=lab.device, dtype=lab.dtype)) * math.pi
+    ang = x[..., None] * freqs  # (...,3,n)
+    return torch.cat([torch.sin(ang), torch.cos(ang)], -1).flatten(-2)  # (..., 6n)
+
+
+class MLP(nn.Module):
+    def __init__(self, i, h, o, drop=0.0):
+        super().__init__()
+        self.net = nn.Sequential(nn.Linear(i, h), nn.GELU(), nn.Dropout(drop), nn.Linear(h, o))
+
+    def forward(self, x):
+        return self.net(x)
+
+
+class ThumbCNN(nn.Module):
+    """32x32 RGBA thumbnail -> embedding. Alpha matters: transparent logos behave differently from photos."""
+    def __init__(self, o=128):
+        super().__init__()
+        c = [5, 32, 64, 96, 128]
+        layers = []
+        for a, b in zip(c[:-1], c[1:]):
+            layers += [nn.Conv2d(a, b, 3, padding=1), nn.GELU(), nn.Conv2d(b, b, 3, padding=1, stride=2), nn.GELU()]
+        self.net = nn.Sequential(*layers)
+        self.proj = nn.Linear(128 * 2 * 2, o)
+
+    def forward(self, x):  # (B, M, 4, 32, 32) in [0,1]
+        B, M = x.shape[:2]
+        x = x.reshape(B * M, 4, 32, 32)
+        a = x[:, 3:4]
+        lum = (0.2126 * x[:, 0:1] + 0.7152 * x[:, 1:2] + 0.0722 * x[:, 2:3]) * a  # premultiplied luminance
+        x = torch.cat([x[:, :3] * a, a, lum], 1)
+        x = self.net(x)  # (BM,128,2,2)
+        return self.proj(x.flatten(1)).reshape(B, M, -1)
+
+
+class Block(nn.Module):
+    def __init__(self, d, h, ff, drop):
+        super().__init__()
+        self.h = h
+        self.ln1 = nn.LayerNorm(d)
+        self.qkv = nn.Linear(d, 3 * d)
+        self.proj = nn.Linear(d, d)
+        self.rel = nn.Linear(K_REL, h, bias=False)
+        nn.init.normal_(self.rel.weight, std=0.5)
+        self.ln2 = nn.LayerNorm(d)
+        self.ff = nn.Sequential(nn.Linear(d, ff * d), nn.GELU(), nn.Dropout(drop), nn.Linear(ff * d, d))
+        self.drop = nn.Dropout(drop)
+
+    def forward(self, x, rel, pad_bias):
+        B, T, D = x.shape
+        q, k, v = self.qkv(self.ln1(x)).reshape(B, T, 3, self.h, D // self.h).permute(2, 0, 3, 1, 4)
+        bias = self.rel(rel).permute(0, 3, 1, 2) + pad_bias  # (B,H,T,T)
+        a = F.scaled_dot_product_attention(q, k, v, attn_mask=bias.to(q.dtype))
+        x = x + self.drop(self.proj(a.transpose(1, 2).reshape(B, T, D)))
+        return x + self.drop(self.ff(self.ln2(x)))
+
+
+class NadirNet(nn.Module):
+    def __init__(self, d=384, layers=8, heads=6, ff=4, drop=0.1):
+        super().__init__()
+        self.cfg = dict(d=d, layers=layers, heads=heads, ff=ff, drop=drop)
+        self.tok_in = MLP(FT + N_ROLES + 36, 2 * d, d)
+        self.cnn = ThumbCNN(128)
+        self.img_in = MLP(FI + 128, 2 * d, d)
+        self.page_in = MLP(FP, d, d)
+        self.src = nn.Embedding(2, d)
+        self.typ = nn.Embedding(3, d)
+        self.blocks = nn.ModuleList([Block(d, heads, ff, drop) for _ in range(layers)])
+        self.ln = nn.LayerNorm(d)
+        self.color_head = MLP(d + 3, d, 6)
+        self.img_head = MLP(d, d // 2, N_TREAT)
+
+    def forward(self, tok_f, tok_role, tok_lab, img_f, img_thumb, page_f, src, rel, mask):
+        """tok_f (B,N,FT) tok_role (B,N) long, tok_lab (B,N,3), img_f (B,M,FI), img_thumb (B,M,4,32,32) float 0-1,
+        page_f (B,FP), src (B,) long, rel (B,T,T,K) with T=1+N+M, mask (B,T) bool (True=real)."""
+        B, N = tok_role.shape
+        M = img_f.shape[1]
+        tok = self.tok_in(torch.cat([tok_f, F.one_hot(tok_role, N_ROLES).to(tok_f.dtype), fourier_lab(tok_lab)], -1)) + self.typ.weight[1]
+        if M > 0:
+            img = self.img_in(torch.cat([img_f, self.cnn(img_thumb)], -1)) + self.typ.weight[2]
+        else:
+            img = tok.new_zeros(B, 0, tok.shape[-1])
+        cls = self.page_in(page_f)[:, None] + self.typ.weight[0]
+        x = torch.cat([cls, tok, img], 1) + self.src(src)[:, None]
+        pad_bias = torch.zeros(mask.shape, dtype=x.dtype, device=x.device).masked_fill(~mask, -1e4)[:, None, None, :]
+        for blk in self.blocks:
+            x = blk(x, rel, pad_bias)
+        x = self.ln(x)
+        ht = x[:, 1:1 + N]
+        z = self.color_head(torch.cat([ht, tok_lab], -1))
+        L = torch.sigmoid(z[..., 0])
+        s = 1.6 * torch.sigmoid(z[..., 1])
+        r = 0.6 * torch.tanh(z[..., 2])
+        a_in, b_in = tok_lab[..., 1], tok_lab[..., 2]
+        a = s * a_in - r * b_in + 0.12 * torch.tanh(z[..., 3])
+        b = r * a_in + s * b_in + 0.12 * torch.tanh(z[..., 4])
+        lab = torch.stack([L, a, b], -1)
+        alpha_logit = z[..., 5]
+        img_logits = self.img_head(x[:, 1 + N:]) if M > 0 else x.new_zeros(B, 0, N_TREAT)
+        return lab, alpha_logit, img_logits
+
+
+# ---------- colour maths (differentiable) ----------
+OKLAB_TO_LMS = torch.tensor([[1.0, 0.3963377774, 0.2158037573], [1.0, -0.1055613458, -0.0638541728], [1.0, -0.0894841775, -1.2914855480]])
+LMS_TO_RGB = torch.tensor([[4.0767416621, -3.3077115913, 0.2309699292], [-1.2684380046, 2.6097574011, -0.3413193965], [-0.0041960863, -0.7034186147, 1.7076147010]])
+
+
+def oklab_to_linrgb(lab):
+    lms = lab @ OKLAB_TO_LMS.to(lab).T
+    return (lms ** 3) @ LMS_TO_RGB.to(lab).T
+
+
+def luminance(lab):
+    rgb = oklab_to_linrgb(lab).clamp(0, 1)
+    return rgb @ torch.tensor([0.2126, 0.7152, 0.0722]).to(lab)
+
+
+def contrast_ratio(lab1, lab2):
+    l1, l2 = luminance(lab1), luminance(lab2)
+    return (torch.maximum(l1, l2) + 0.05) / (torch.minimum(l1, l2) + 0.05)

training/predict.py

@@ -0,0 +1,93 @@
+#!/usr/bin/env python3
+"""Run a trained NadirNet over held-out pages and write render jobs (Python port of nadir-infer.js decode)."""
+import argparse, gzip, json, math, pickle
+from pathlib import Path
+import numpy as np
+import torch
+
+from model import NadirNet
+from train import collate, prep
+from mkjobs import job_from
+
+TREAT = ["keep", "dim", "invert", "light", "plate"]
+
+
+from safety import oklab, lab_to_rgb, contrast, safe_treatment, enforce_contrast, is_media_keep, hue_guard, keep_invisible
+
+
+def decode(L, lab, alpha_logit, img_logits, rels):
+    toks = L["tokens"][:256]
+    out, keep = [], set()
+    for i, t in enumerate(toks):
+        ain = min(1 - 1e-3, max(1e-3, t["rgba"][3]))
+        alpha = 1 / (1 + math.exp(-(alpha_logit[i] + math.log(ain / (1 - ain)))))
+        a = 1.0 if (t["rgba"][3] >= 0.999 and alpha > 0.97) else alpha
+        out.append({"rgb": lab_to_rgb(*lab[i]), "a": a, "L": float(lab[i][0])})
+        if is_media_keep(t):  # shown as-is: later safety passes see its real colour
+            keep.add(i)
+            rgb = [int(x) for x in t["rgba"][:3]]
+            out[i] = {"rgb": rgb, "a": float(t["rgba"][3]), "L": oklab(*rgb)[0]}
+        else:
+            hue_guard(t, out[i])
+    for i, t in enumerate(toks):
+        if t["f"][24] > 0.5 and t["role"] == 0 and out[i]["L"] > 0.35:
+            o = oklab(*out[i]["rgb"]); out[i]["rgb"] = lab_to_rgb(0.2, o[1] * 0.3, o[2] * 0.3); out[i]["L"] = 0.2
+    keep_invisible(toks, out, rels, keep)
+    enforce_contrast(toks, out, rels, keep)
+    def hx(o):
+        h = "#%02x%02x%02x" % tuple(o["rgb"])
+        return h + ("%02x" % round(o["a"] * 255) if o["a"] < 0.999 else "")
+    colors = [None if i in keep else hx(o) for i, o in enumerate(out)]
+    treat = {}
+    for j, im in enumerate(L["images"][:32]):
+        t = TREAT[int(np.argmax(img_logits[j]))] if j < len(img_logits) else "keep"
+        u = im.get("underTok", -1)
+        treat[str(j)] = safe_treatment(t, im, out[u]["L"] if u is not None and 0 <= u < len(out) else None)
+    return colors, sorted(keep), treat
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--ckpt", required=True)
+    ap.add_argument("--data", required=True)
+    ap.add_argument("--crawl", required=True)
+    ap.add_argument("--out", required=True)
+    ap.add_argument("--n", type=int, default=150)
+    ap.add_argument("--src", type=int, default=0)
+    ap.add_argument("--tag", default="nadir")
+    ap.add_argument("--also", default="haiku,dr", help="extra render tags to emit for the same pages")
+    ap.add_argument("--haiku", default="")
+    args = ap.parse_args()
+    ck = torch.load(args.ckpt, map_location="cpu")
+    model = NadirNet(**ck["cfg"]); model.load_state_dict(ck["model"]); model.eval()
+    dev = "cuda" if torch.cuda.is_available() else "cpu"
+    model.to(dev)
+    data = pickle.load(open(args.data, "rb"))
+    val = [s for s in data["val"] if s["src"] == args.src][: args.n]
+    with open(args.out, "w") as f:
+        for s in val:
+            s = prep(s)
+            L = json.load(gzip.open(Path(args.crawl) / "pages" / s["id"] / "light.json.gz"))
+            b = collate([s], dev)
+            with torch.no_grad():
+                lab, al, il = model(b["tok_f"], b["role"], b["lab"], b["img_f"], b["thumbs"], b["page"], torch.zeros_like(b["src"]), b["rel"], b["mask"])
+            n = len(s["role"])
+            colors, keep, treat = decode(L, lab[0, :n].float().cpu().numpy().tolist(), al[0, :n].float().cpu().numpy().tolist(),
+                                         il[0].float().cpu().numpy().tolist(), s["rel"])
+            colors = colors + [None] * (len(L["tokens"]) - len(colors))
+            j = job_from(L, s["id"], args.tag, colors, treat)
+            j["keep"] = sorted(set(j["keep"]) | set(keep))
+            f.write(json.dumps(j) + "\n")
+            if "dr" in args.also:
+                f.write(json.dumps(job_from(L, s["id"], "dr", [None] * len(L["tokens"]), {}, baseline=True)) + "\n")
+            if "haiku" in args.also and args.haiku:
+                hp = Path(args.haiku) / f"{s['id']}.json"
+                if hp.exists():
+                    h = json.loads(hp.read_text())
+                    outs = [h["colors"].get(str(i)) for i in range(len(L["tokens"]))]
+                    f.write(json.dumps(job_from(L, s["id"], "haiku", outs, h["images"])) + "\n")
+    print("wrote", len(val), "pages")
+
+
+if __name__ == "__main__":
+    main()

training/quantize.py

@@ -0,0 +1,330 @@
+#!/usr/bin/env python3
+"""Calibrated weight-only quantization of NadirNet into ONNX MatMulNBits (8-bit and 4-bit).
+
+Layer-sequential GPTQ: each layer's Hessian comes from real crawled pages flowing through the already
+quantized earlier layers. Each group of 32 weights gets asymmetric uint codes whose clipping range is
+chosen by an MSE search. Codes are packed in ORT's MatMulNBits layout, and every variant is checked
+three ways: against a PyTorch fake-quant copy (packing parity), against fp32 (fidelity) and against
+the Haiku targets (quality)."""
+import argparse, copy, json, os, pickle, time
+import numpy as np
+import torch
+import torch.nn as nn
+
+from model import NadirNet
+from train import prep
+from export import feeds_for, fidelity, bench, NAMES_IN
+
+GS = 32
+
+
+class Stop(Exception):
+    pass
+
+
+def group_params(w, bits, maxshrink, grid=100):
+    """Per-row scale/zero-point for one group (rows, GS), minimizing |q(w)-w|^2.4 over clipping ranges."""
+    maxq = 2 ** bits - 1
+    xmin = w.min(1).values.clamp(max=0)
+    xmax = w.max(1).values.clamp(min=0)
+    best = torch.full_like(xmin, float("inf"))
+    bs, bz = torch.ones_like(xmin), torch.zeros_like(xmin)
+    for i in range(int(maxshrink * grid) + 1):
+        p = 1 - i / grid
+        lo, hi = p * xmin, p * xmax
+        s = ((hi - lo) / maxq).clamp(min=1e-10)
+        z = torch.round(-lo / s).clamp(0, maxq)
+        q = (torch.clamp(torch.round(w / s[:, None]) + z[:, None], 0, maxq) - z[:, None]) * s[:, None]
+        err = (q - w).abs().pow(2.4).sum(1)
+        m = err < best
+        best[m], bs[m], bz[m] = err[m], s[m], z[m]
+    return bs, bz
+
+
+def gptq(W, H, bits, percdamp=0.01, blocksize=128, gs=GS):
+    """W (out, Kpad) with Kpad % gs == 0, H (Kpad, Kpad); gs=None quantizes per output channel (one group per row).
+    Returns dequantized W, codes, scales, zero points."""
+    maxq = 2 ** bits - 1
+    maxshrink = 0.5 if bits == 4 else 0.2
+    W = W.clone().double()
+    H = H.clone().double()
+    rows, cols = W.shape
+    dead = torch.diag(H) == 0
+    H[dead, dead] = 1
+    W[:, dead] = 0
+    H += percdamp * torch.mean(torch.diag(H)) * torch.eye(cols, dtype=H.dtype)
+    Hinv = torch.linalg.cholesky(torch.cholesky_inverse(torch.linalg.cholesky(H)), upper=True)
+    Q = torch.zeros_like(W)
+    codes = torch.zeros(rows, cols, dtype=torch.uint8)
+    if gs is None:  # per channel: one range per output row, chosen on the whole row up front
+        s, z = group_params(W, bits, maxshrink)
+        S, Z = s[:, None].clone(), z[:, None].clone()
+    else:
+        S = torch.zeros(rows, cols // gs, dtype=torch.float64)
+        Z = torch.zeros(rows, cols // gs, dtype=torch.float64)
+    for i1 in range(0, cols, blocksize):
+        i2 = min(i1 + blocksize, cols)
+        W1 = W[:, i1:i2].clone()
+        E1 = torch.zeros_like(W1)
+        H1 = Hinv[i1:i2, i1:i2]
+        for i in range(i2 - i1):
+            c = i1 + i
+            if gs is not None and c % gs == 0:  # groups never straddle blocks (blocksize % gs == 0)
+                s, z = group_params(W1[:, i:i + gs], bits, maxshrink)
+                S[:, c // gs], Z[:, c // gs] = s, z
+            w = W1[:, i]
+            qc = torch.clamp(torch.round(w / s) + z, 0, maxq)
+            q = (qc - z) * s
+            codes[:, c] = qc.to(torch.uint8)
+            Q[:, c] = q
+            err = (w - q) / H1[i, i]
+            W1[:, i:] -= err[:, None] * H1[i, i:][None, :]
+            E1[:, i] = err
+        W[:, i2:] -= E1 @ Hinv[i1:i2, i2:]
+    return Q.float(), codes, S.float(), Z.to(torch.uint8)
+
+
+def pack(codes, S, Z, bits):
+    """MatMulNBits layout: B (N, k_blocks, blob), scales (N, k_blocks), zero points (N, ceil(k_blocks*bits/8))."""
+    c = codes.numpy()
+    N, Kp = c.shape
+    kb = Kp // GS
+    z = Z.numpy().astype(np.uint8)
+    if bits == 8:
+        B = c.reshape(N, kb, GS).astype(np.uint8)
+        zp = z
+    else:
+        c = c.reshape(N, kb, GS)
+        B = (c[..., 0::2] | (c[..., 1::2] << 4)).astype(np.uint8)
+        if kb % 2:
+            z = np.concatenate([z, np.full((N, 1), 8, np.uint8)], 1)
+        zp = (z[:, 0::2] | (z[:, 1::2] << 4)).astype(np.uint8)
+    return B, S.numpy().astype(np.float32), zp
+
+
+def linear_order(model, sample):
+    """Linear modules in the order the forward pass calls them."""
+    order, hooks = [], []
+    for name, mod in model.named_modules():
+        if isinstance(mod, nn.Linear):
+            hooks.append(mod.register_forward_pre_hook(lambda m, a, n=name: order.append(n) if n not in order else None))
+    with torch.no_grad():
+        model(*[torch.from_numpy(sample[k]) for k in NAMES_IN])
+    for h in hooks:
+        h.remove()
+    return order
+
+
+def onnx_matmuls(model, onnx_model):
+    """Map each Linear to the MatMul node whose initializer is its transposed weight (Gemm exports stay fp32)."""
+    from onnx import numpy_helper
+    inits = {i.name: i for i in onnx_model.graph.initializer}
+    by_shape = {}
+    for node in onnx_model.graph.node:
+        if node.op_type == "MatMul" and node.input[1] in inits:
+            arr = numpy_helper.to_array(inits[node.input[1]])
+            by_shape.setdefault(arr.shape, []).append((node, arr))
+    out = {}
+    for name, mod in model.named_modules():
+        if isinstance(mod, nn.Linear):
+            wt = mod.weight.detach().numpy().T
+            for node, arr in by_shape.get(wt.shape, []):
+                if np.array_equal(arr, wt):
+                    out[name] = node
+                    break
+    return out
+
+
+def hessian(model, name, calib):
+    """2/n * X^T X of the inputs to one Linear, over all real token rows of the calibration pages."""
+    mod = dict(model.named_modules())[name]
+    K = mod.in_features
+    H = torch.zeros(K, K, dtype=torch.float64)
+    n = [0]
+
+    def hook(m, a):
+        x = a[0].detach().reshape(-1, K).double()
+        H.add_(x.T @ x)
+        n[0] += x.shape[0]
+        raise Stop
+
+    h = mod.register_forward_pre_hook(hook)
+    with torch.no_grad():
+        for f in calib:
+            try:
+                model(*[torch.from_numpy(f[k]) for k in NAMES_IN])
+            except Stop:
+                pass
+    h.remove()
+    return H * (2.0 / max(1, n[0]))
+
+
+def quantize_model(model, calib, bits_for, order):
+    """Sequential GPTQ over `order`; returns the fake-quant model and packed tensors per layer."""
+    qm = copy.deepcopy(model).eval()
+    mods = dict(qm.named_modules())
+    packed = {}
+    for name in order:
+        bits = bits_for.get(name)
+        if not bits:
+            continue
+        mod = mods[name]
+        K = mod.in_features
+        H = hessian(qm, name, calib)
+        t = time.time()
+        # blocks of 32 along K (zero-padded to a multiple of 32)
+        Kp = -(-K // GS) * GS
+        Hp = torch.zeros(Kp, Kp, dtype=torch.float64); Hp[:K, :K] = H
+        Wp = torch.zeros(mod.out_features, Kp); Wp[:, :K] = mod.weight.detach()
+        Q, codes, S, Z = gptq(Wp, Hp, bits)
+        if bits == 8:
+            # 8-bit: uint8 codes as a (K/32, 32, N) tensor with per-block scale and zero point, dequantized in-graph
+            N, kb = mod.out_features, Kp // GS
+            packed[name] = ("dq8", K, codes.numpy().T.reshape(kb, GS, N).copy(), S.numpy().T[:, None, :].astype(np.float32).copy(),
+                            Z.numpy().T[:, None, :].astype(np.uint8).copy())
+        else:
+            # 4-bit: packed as MatMulNBits
+            packed[name] = ("nbits", K) + pack(codes, S, Z, bits)
+        with torch.no_grad():
+            mod.weight.copy_(Q[:, :K])
+        print(f"  {name} {tuple(mod.weight.shape)} {bits}-bit  {time.time() - t:.1f}s", flush=True)
+    return qm, packed
+
+
+def build_onnx(fp32_path, mapping, packed, out_path, accuracy_level=0):
+    import onnx
+    from onnx import helper, numpy_helper
+    m = onnx.load(fp32_path)
+    g = m.graph
+    drop_inits = set()
+    for name, (kind, K, B, S, zp) in packed.items():
+        node = mapping[name]
+        wname = node.input[1]
+        idx = list(g.node).index(node)
+        if kind == "dq8":
+            # uint8 codes dequantized with plain Cast/Sub/Mul: every onnxruntime constant-folds this into exact fp32
+            # weights at load. (DequantizeLinear->MatMul gets rewritten by onnxruntime into an activation-quantized
+            # kernel, which costs accuracy.)
+            kb, gs, N = B.shape
+            inits = [numpy_helper.from_array(B, wname + "_q8"), numpy_helper.from_array(zp, wname + "_zero_point"),
+                     numpy_helper.from_array(S, wname + "_scale"), numpy_helper.from_array(np.array([kb * gs, N], np.int64), wname + "_kn")]
+            g.initializer.extend(inits)
+            base = (node.name or wname) + "_deq8"
+            F = onnx.TensorProto.FLOAT
+            deq = [helper.make_node("Cast", [inits[0].name], [wname + "_f"], name=base + "_cast", to=F),
+                   helper.make_node("Cast", [inits[1].name], [wname + "_zf"], name=base + "_castz", to=F),
+                   helper.make_node("Sub", [wname + "_f", wname + "_zf"], [wname + "_c"], name=base + "_sub"),
+                   helper.make_node("Mul", [wname + "_c", inits[2].name], [wname + "_d3"], name=base + "_mul"),
+                   helper.make_node("Reshape", [wname + "_d3", inits[3].name], [wname + ("_dq" if kb * gs == K else "_dp")], name=base + "_reshape")]
+            if kb * gs != K:  # drop the zero padding rows
+                lims = [numpy_helper.from_array(np.array(v, np.int64), wname + n) for v, n in (([0], "_s0"), ([K], "_sK"), ([0], "_ax"))]
+                g.initializer.extend(lims)
+                deq.append(helper.make_node("Slice", [wname + "_dp"] + [i.name for i in lims], [wname + "_dq"], name=base + "_slice"))
+            g.node[idx].input[1] = wname + "_dq"  # `node` belongs to another copy of the graph: edit this one
+            for k, dn in enumerate(deq):
+                g.node.insert(idx + k, dn)
+        else:
+            N = B.shape[0]
+            inits = [numpy_helper.from_array(B, wname + "_Q4"), numpy_helper.from_array(S, wname + "_scales"),
+                     numpy_helper.from_array(zp, wname + "_zero_points")]
+            g.initializer.extend(inits)
+            kw = dict(K=K, N=N, bits=4, block_size=GS)
+            if accuracy_level:
+                kw["accuracy_level"] = accuracy_level
+            new = helper.make_node("MatMulNBits", [node.input[0]] + [i.name for i in inits], list(node.output),
+                                   name=(node.name or wname) + "_Q4", domain="com.microsoft", **kw)
+            g.node.remove(node)
+            g.node.insert(idx, new)
+        drop_inits.add(wname)
+    keep = [i for i in g.initializer if i.name not in drop_inits]
+    del g.initializer[:]
+    g.initializer.extend(keep)
+    if any(n.domain == "com.microsoft" for n in g.node) and not any(o.domain == "com.microsoft" for o in m.opset_import):
+        m.opset_import.append(helper.make_opsetid("com.microsoft", 1))
+    onnx.checker.check_model(m)
+    onnx.save(m, out_path)
+
+
+def quality(sess, samples):
+    """dE vs the training targets (Haiku for src 0, native dark for src 1) and image-treatment accuracy."""
+    agg = {0: [0.0, 0.0], 1: [0.0, 0.0]}
+    ok = n = 0
+    for s in samples:
+        f = feeds_for(s)
+        lab, _, imgl = sess.run(None, f)
+        k = len(s["role"])
+        m = s["tmask"][:k]
+        if m.any():
+            d = np.linalg.norm(lab[0, :k] - s["lab_t"][:k], axis=-1)
+            w = s["w"][:k] * m
+            agg[int(s["src"])][0] += float((d * w).sum() / max(w.sum(), 1e-9)); agg[int(s["src"])][1] += 1
+        mi = len(s["IF"])
+        if mi:
+            t = np.asarray(s["imgt"][:mi])
+            v = t >= 0
+            ok += int((imgl[0, :mi].argmax(-1)[v] == t[v]).sum()); n += int(v.sum())
+    return {"haiku_dE": agg[0][0] / max(1, agg[0][1]), "native_dE": agg[1][0] / max(1, agg[1][1]), "img_acc": ok / max(1, n)}
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--ckpt", required=True)
+    ap.add_argument("--data", required=True)
+    ap.add_argument("--fp32", required=True, help="fp32 ONNX from export.py")
+    ap.add_argument("--out", required=True)
+    ap.add_argument("--calib", type=int, default=256)
+    ap.add_argument("--eval", type=int, default=400)
+    ap.add_argument("--threads", type=int, default=8)
+    args = ap.parse_args()
+    torch.set_num_threads(args.threads)
+    os.makedirs(args.out, exist_ok=True)
+    import onnx, onnxruntime as ort
+    ck = torch.load(args.ckpt, map_location="cpu")
+    model = NadirNet(**ck["cfg"]); model.load_state_dict(ck["model"]); model.eval()
+    data = pickle.load(open(args.data, "rb"))
+    hv = [s for s in data["val"] if s["src"] == 0]; nv = [s for s in data["val"] if s["src"] == 1]
+    val = [prep(s) for s in hv[: args.eval * 3 // 4] + nv[: args.eval - min(len(hv), args.eval * 3 // 4)]]  # both sources
+    rng = np.random.default_rng(0)
+    pick = rng.choice(len(data["train"]), size=min(args.calib, len(data["train"])), replace=False)
+    calib = [feeds_for(prep(data["train"][i])) for i in pick]
+    order = linear_order(model, calib[0])
+    mapping = onnx_matmuls(model, onnx.load(args.fp32))
+    quantizable = [n for n in order if n in mapping and not n.endswith(".rel")]
+    print(f"{len(quantizable)}/{len(order)} linears map to MatMul; fp32 kept: {[n for n in order if n not in quantizable]}", flush=True)
+    edges = [n for n in quantizable if n.startswith(("tok_in.", "img_in.", "color_head.", "img_head."))]
+    plans = {"int8": {n: 8 for n in quantizable},
+             "int4": {n: (8 if n in edges else 4) for n in quantizable}}
+    so = ort.SessionOptions(); so.intra_op_num_threads = 4
+    ref = ort.InferenceSession(args.fp32, so, providers=["CPUExecutionProvider"])
+    report = {"format": {"8-bit layers": f"uint8 blocks of {GS} with fp32 scale + uint8 zero point, dequantized in-graph (Cast/Sub/Mul, folded at load)", "4-bit layers": f"uint4 blocks of {GS}, MatMulNBits"},
+              "group_size": GS, "calib_pages": len(calib), "eval_pages": len(val), "fp32_layers": [n for n in order if n not in quantizable],
+              "int4_8bit_edge_layers": edges, "variants": {}}
+    r = quality(ref, val); r["size_mb"] = round(os.path.getsize(args.fp32) / 1e6, 2); r["cpu_ms_median"] = round(bench(ref, val), 2)
+    report["variants"]["fp32"] = r
+    print("fp32", r, flush=True)
+    for vname, bits_for in plans.items():
+        print(f"== {vname}", flush=True)
+        qm, packed = quantize_model(model, calib, bits_for, quantizable)
+        path = os.path.join(args.out, f"nadir-{vname}.onnx")
+        build_onnx(args.fp32, mapping, packed, path)
+        sess = ort.InferenceSession(path, so, providers=["CPUExecutionProvider"])
+        # packing parity: ORT MatMulNBits graph vs the PyTorch fake-quant model
+        par = 0.0
+        with torch.no_grad():
+            for s in val[:8]:
+                f = feeds_for(s)
+                tq = qm(*[torch.from_numpy(f[k]) for k in NAMES_IN])[0].numpy()
+                par = max(par, float(np.abs(tq - sess.run(None, f)[0]).max()))
+        r = fidelity(sess, ref, val)
+        r.update(quality(sess, val))
+        r["pack_parity_max_abs"] = par
+        r["size_mb"] = round(os.path.getsize(path) / 1e6, 2)
+        r["cpu_ms_median"] = round(bench(sess, val), 2)
+        report["variants"][vname] = r
+        print(vname, r, flush=True)
+        torch.save({"cfg": ck["cfg"], "model": qm.state_dict()}, os.path.join(args.out, f"fakequant-{vname}.pt"))
+    json.dump(report, open(os.path.join(args.out, "quant_report.json"), "w"), indent=1)
+
+
+if __name__ == "__main__":
+    main()

training/safety.py

@@ -0,0 +1,188 @@
+"""Deterministic safety rules shared by the dataset builder (target repair) and the decoder (runtime).
+Mirrors nadir-infer.js / nadir-map.js."""
+import math
+
+
+def lin(u):
+    u = u / 255.0
+    return u / 12.92 if u <= 0.04045 else ((u + 0.055) / 1.055) ** 2.4
+
+
+def oklab(r, g, b):
+    r, g, b = lin(r), lin(g), lin(b)
+    l = (0.4122214708 * r + 0.5363325363 * g + 0.0514459929 * b) ** (1 / 3)
+    m = (0.2119034982 * r + 0.6806995451 * g + 0.1073969566 * b) ** (1 / 3)
+    s = (0.0883024619 * r + 0.2817188376 * g + 0.6299787005 * b) ** (1 / 3)
+    return [0.2104542553 * l + 0.7936177850 * m - 0.0040720468 * s, 1.9779984951 * l - 2.4285922050 * m + 0.4505937099 * s,
+            0.0259040371 * l + 0.7827717662 * m - 0.8086757660 * s]
+
+
+def lab_to_rgb(L, a, b):
+    l_, m_, s_ = L + 0.3963377774 * a + 0.2158037573 * b, L - 0.1055613458 * a - 0.0638541728 * b, L - 0.0894841775 * a - 1.2914855480 * b
+    l, m, s = l_ ** 3, m_ ** 3, s_ ** 3
+    def enc(x):
+        x = max(0.0, min(1.0, x))
+        return round(255 * (12.92 * x if x <= 0.0031308 else 1.055 * x ** (1 / 2.4) - 0.055))
+    return [enc(4.0767416621 * l - 3.3077115913 * m + 0.2309699292 * s), enc(-1.2684380046 * l + 2.6097574011 * m - 0.3413193965 * s),
+            enc(-0.0041960863 * l - 0.7034186147 * m + 1.7076147010 * s)]
+
+
+def rel_lum(c):
+    return 0.2126 * lin(c[0]) + 0.7152 * lin(c[1]) + 0.0722 * lin(c[2])
+
+
+def contrast(a, b):
+    x, y = rel_lum(a), rel_lum(b)
+    return (max(x, y) + 0.05) / (min(x, y) + 0.05)
+
+
+def adaptable(im, s, surf_L=None):
+    """A logo on its own white rectangle: adapt it instead of leaving a white box. Only images the page marks as its
+    logo (logo/brand/wordmark naming, or the link home); product shots, photos and other images are never touched.
+    Without pixel stats a logo still tries (the adaptation itself checks for the white background)."""
+    if not (im.get("logo") or (im.get("homeLink") and im.get("header"))) or im["h"] > 200 or (surf_L is not None and surf_L >= 0.4):
+        return False
+    if not s:
+        return im["w"] * im["h"] <= 250000
+    return s["transp"] < 0.05 and s.get("ringA", 0) >= 0.95 and s.get("ringL", 0) >= 0.85 and s.get("white", 0) >= 0.3 and s["colors"] < 0.2
+
+
+def is_light_backdrop(im, s):
+    """Opaque, light, flat (almost no lightness variation, so never a photo; SVG backgrounds can't be photos, so a
+    smooth gradient qualifies too) and big enough to sit behind content. Logos drawn as backgrounds are never cleared."""
+    if im.get("kind") != "bg" or im.get("logo") or im.get("homeLink") or not s or s["transp"] >= 0.05 or s["meanL"] <= 0.8 or im["w"] * im["h"] < 40000:
+        return False
+    svg = im.get("src", "").split("?")[0].lower().endswith(".svg")
+    return s["stdL"] < (0.15 if svg else 0.07)
+
+
+def safe_treatment(t, im, surf_L=None):
+    """'adapt:<fallback>' for logos on a white rectangle (fallback used when there is nothing to adapt)."""
+    base = _base_treatment(t, im, surf_L)
+    return "adapt:" + base if base != "clear" and adaptable(im, im.get("stats"), surf_L) else base
+
+
+def _base_treatment(t, im, surf_L=None):
+    s = im.get("stats")
+    # a light, flat CSS background image (texture, pale wash, pattern) sits behind content whose text is now light:
+    # clear it so the dark surface designed for that element shows through. Photos and detailed images stay.
+    if is_light_backdrop(im, s):
+        return "clear"
+    # a mostly dark graphic on transparency would vanish on a dark surface: hue-preserving inversion
+    if (not t or t == "keep") and s and s["transp"] > 0.15 and s["meanL"] < 0.35 and im["w"] * im["h"] < 200000 and (surf_L is None or surf_L < 0.4):
+        return "invert"
+    if not t or t == "keep":
+        return "keep"
+    photo = (s["transp"] < 0.02 and s["colors"] > 0.18 and s["stdL"] > 0.12) if s else (im["w"] * im["h"] > 60000)
+    if photo and t != "dim":
+        return "keep"
+    if t == "dim" and im["w"] * im["h"] < 20000:
+        return "keep"
+    # a transparent graphic that is already light reads fine on a dark surface: leave it alone
+    if t in ("plate", "light", "invert") and s and s["transp"] > 0.15 and s["meanL"] >= 0.55 and s.get("black", 0) < 0.15 and (surf_L is None or surf_L < 0.4):
+        return "keep"
+    # no light boxes on a dark page: a dark logo on transparency gets a hue-preserving inversion instead of a plate,
+    # an opaque one already carries its own background
+    if t == "plate":
+        return "invert" if (s is None or s["transp"] > 0.15) else "keep"
+    return t
+
+
+FG_ROLES = (1, 5, 6)  # text, svg fill, svg stroke
+
+
+def hue_guard(t, o):
+    """Brand colours keep their hue: a coloured input whose output drifts more than 45 degrees is turned back to
+    its own hue (keeping the predicted lightness and chroma); neutral inputs get at most a faint tint."""
+    rgb = [int(x) for x in t["rgba"][:3]]
+    i = oklab(*rgb)
+    q = oklab(*o["rgb"])
+    cin, cout = math.hypot(i[1], i[2]), math.hypot(q[1], q[2])
+    if cin > 0.03 and cout > 0.03:
+        d = abs((math.atan2(q[2], q[1]) - math.atan2(i[2], i[1]) + math.pi) % (2 * math.pi) - math.pi)
+        if d <= math.radians(45):
+            return
+        h = math.atan2(i[2], i[1])
+        o["rgb"] = lab_to_rgb(q[0], cout * math.cos(h), cout * math.sin(h))
+    elif cin <= 0.03 and cout > 0.06:
+        o["rgb"] = lab_to_rgb(q[0], q[1] * 0.06 / cout, q[2] * 0.06 / cout)
+
+
+def keep_invisible(toks, out, rels, keep):
+    """Things that were invisible against their surface in the light design stay invisible: a gradient stop
+    with exactly a surface's colour (text fades, edge fades) takes that surface's dark colour, and a border that
+    matched the surface it sits on takes the surface's dark colour too (otherwise: dark slabs, white edge bars)."""
+    n = len(toks)
+    key = lambda t: tuple(int(x) for x in t["rgba"][:3])
+    surf = {}
+    for i, t in enumerate(toks):  # first (most prominent) opaque surface of each colour
+        if t["role"] == 0 and t["rgba"][3] >= 0.999 and i not in keep and out[i]["a"] >= 0.999:
+            surf.setdefault(key(t), i)
+    for i, t in enumerate(toks):
+        if t["role"] == 4 and i not in keep and key(t) in surf:
+            j = surf[key(t)]
+            out[i]["rgb"], out[i]["L"] = list(out[j]["rgb"]), out[j]["L"]
+    best = {}
+    for typ, a, b, w in rels:
+        a, b = int(a), int(b)
+        if int(typ) == 2 and a < n and b < n and toks[a]["role"] == 2 and toks[b]["role"] == 0 and (a not in best or w > best[a][0]):
+            best[a] = (w, b)
+    for i, (_, j) in best.items():
+        if i in keep or j in keep:
+            continue
+        if contrast(toks[i]["rgba"][:3], toks[j]["rgba"][:3]) < 1.05 and toks[i]["rgba"][3] >= 0.5:
+            out[i]["rgb"], out[i]["L"] = list(out[j]["rgb"]), out[j]["L"]
+
+
+def is_media_keep(t):
+    """Colours drawn over video/photos that already read as dark mode stay exactly as they are: dark scrims and
+    player bars, light captions and icons. Everything else over media (a white page or a frosted white button
+    laid over a decorative canvas) is redesigned like any other colour."""
+    f = t["f"]
+    if f[48] < 0.5:
+        return False
+    if t["role"] in (0, 3, 4):  # bg, shadow, gradient: dark scrims, or saturated accents such as a progress bar
+        return f[24] <= 0.5 and f[11] < 0.25 and (f[0] < 0.5 or f[3] > 0.4)
+    return f[0] > 0.6
+
+
+def enforce_contrast(toks, out, rels, keep):
+    """Keep every foreground colour readable on all the surfaces it sits on, never asking for more contrast
+    than the light design had. One lightness per token (tokens are shared), chosen on a grid to minimise the
+    weighted log-contrast deficit, with a small pull towards the model's own lightness."""
+    n = len(toks)
+    surf = {}
+    for typ, a, b, w in rels:
+        a, b = int(a), int(b)
+        if int(typ) == 0 and a < n and b < n and toks[b]["role"] == 0 and out[b]["a"] >= 0.999:
+            surf.setdefault(a, []).append((b, float(w) + 0.1))  # log-area weights: small buttons still count
+    for i, lst in surf.items():
+        t = toks[i]
+        if i in keep or t["role"] not in FG_ROLES or out[i]["a"] < 0.999:
+            continue
+        if t["role"] == 1:
+            fs, bold = t["f"][21] * 32, t["f"][23] > 0.5
+            need0 = 3.0 if (fs >= 24 or (fs >= 18.5 and bold)) else 4.5
+        else:
+            need0 = 3.0
+        reqs = []
+        for j, w in lst:
+            need = min(need0, contrast(t["rgba"][:3], toks[j]["rgba"][:3]))
+            if need >= 1.5:
+                reqs.append((out[j]["rgb"], need, w))
+        if not reqs:
+            continue
+        def deficit(rgb):
+            return sum(w * max(0.0, math.log(need) - math.log(contrast(rgb, bg))) for bg, need, w in reqs)
+        if deficit(out[i]["rgb"]) <= 0:
+            continue
+        o = oklab(*out[i]["rgb"])
+        W = sum(w for _, _, w in reqs)
+        best = None
+        for k in range(51):
+            Lx = k / 50
+            rgb = lab_to_rgb(Lx, o[1], o[2])
+            c = deficit(rgb) / W + 0.05 * abs(Lx - o[0])
+            if best is None or c < best[0]:
+                best = (c, rgb, Lx)
+        out[i]["rgb"], out[i]["L"] = best[1], best[2]

training/train.py

@@ -0,0 +1,292 @@
+#!/usr/bin/env python3
+"""Train NadirNet on Haiku-designed and native dark-mode targets."""
+import argparse, json, math, os, pickle, random, time
+from collections import defaultdict
+import numpy as np
+import torch
+import torch.nn.functional as F
+
+from model import NadirNet, K_REL, contrast_ratio
+from build_ds import oklab
+
+COLOR_IDX = dict(lab=(0, 1, 2), C=3, alpha=4, cs=(5, 6), rgb=(7, 8, 9), under=(34, 35, 36), over=(39, 40, 41))
+
+
+def lab_to_rgb_np(lab):
+    M1 = np.array([[1.0, 0.3963377774, 0.2158037573], [1.0, -0.1055613458, -0.0638541728], [1.0, -0.0894841775, -1.2914855480]])
+    M2 = np.array([[4.0767416621, -3.3077115913, 0.2309699292], [-1.2684380046, 2.6097574011, -0.3413193965], [-0.0041960863, -0.7034186147, 1.7076147010]])
+    lin = np.clip((lab @ M1.T) ** 3 @ M2.T, 0, 1)
+    return np.where(lin <= 0.0031308, 12.92 * lin, 1.055 * np.power(lin, 1 / 2.4) - 0.055)
+
+
+def prep(s):
+    """Precompute OKLab of inputs/targets once per sample."""
+    if "lab_in" in s:
+        return s
+    s["lab_in"] = oklab(s["rgba"][:, :3]).astype(np.float32)
+    t = s["tgt"]
+    ok = np.isfinite(t[:, 0])
+    lab_t = np.zeros((len(t), 3), np.float32)
+    if ok.any():
+        lab_t[ok] = oklab(t[ok, :3]).astype(np.float32)
+    s["lab_t"], s["tmask"] = lab_t, ok
+    s["alpha_t"] = np.where(ok, t[:, 3], 1).astype(np.float32)
+    km = np.zeros(len(t), bool)
+    if len(s["keep"]):
+        km[s["keep"]] = True
+    s["tmask"] = s["tmask"] & ~km
+    return s
+
+
+def augment(s, rng):
+    s = dict(s)
+    n = len(s["role"])
+    # token dropout (simulates ad blockers, A/B variants, partial loads)
+    if rng.random() < 0.6 and n > 8:
+        p = rng.uniform(0, 0.25)
+        root = s["F"][:, 24] > 0.5
+        keepm = (rng.random(n) > p) | root
+        idx = np.nonzero(keepm)[0]
+        remap = -np.ones(n, np.int64); remap[idx] = np.arange(len(idx))
+        for k in ("F", "role", "rgba", "w", "lab_in", "lab_t", "tmask", "alpha_t"):
+            s[k] = s[k][idx]
+        rel = s["rel"]
+        if len(rel):
+            a, b = rel[:, 1].astype(int), rel[:, 2].astype(int)
+            good = (remap[a] >= 0) & (remap[b] >= 0)
+            rel = rel[good].copy(); rel[:, 1] = remap[a[good]]; rel[:, 2] = remap[b[good]]
+        s["rel"] = rel
+        iu = s["img_under"].copy(); m = iu >= 0; iu[m] = remap[iu[m]]; s["img_under"] = iu
+        n = len(idx)
+    # hue rotation of the whole design (inputs and targets together)
+    if rng.random() < 0.3:
+        th = rng.uniform(0, 2 * math.pi)
+        c, si = math.cos(th), math.sin(th)
+        rot = lambda a, b: (c * a - si * b, si * a + c * b)
+        F_ = s["F"].copy()
+        for (ia, ib) in ((1, 2), (35, 36), (40, 41), (5, 6)):
+            F_[:, ia], F_[:, ib] = rot(F_[:, ia], F_[:, ib])
+        lab = s["lab_in"].copy(); lab[:, 1], lab[:, 2] = rot(lab[:, 1], lab[:, 2])
+        F_[:, 7:10] = lab_to_rgb_np(lab.astype(np.float64)).astype(np.float32)
+        s["F"], s["lab_in"] = F_, lab.astype(np.float32)
+        lt = s["lab_t"].copy(); lt[:, 1], lt[:, 2] = rot(lt[:, 1], lt[:, 2]); s["lab_t"] = lt.astype(np.float32)
+    return s
+
+
+def collate(batch, device):
+    B = len(batch)
+    N = max(1, max(len(s["role"]) for s in batch))
+    M = max(len(s["IF"]) for s in batch)
+    T = 1 + N + M
+    tok_f = np.zeros((B, N, 49), np.float32); role = np.zeros((B, N), np.int64); lab = np.zeros((B, N, 3), np.float32)
+    lab_t = np.zeros((B, N, 3), np.float32); tmask = np.zeros((B, N), bool); w = np.zeros((B, N), np.float32)
+    alpha_in = np.ones((B, N), np.float32); alpha_t = np.ones((B, N), np.float32)
+    img_f = np.zeros((B, M, 39), np.float32); thumbs = np.zeros((B, M, 4, 32, 32), np.float32); imgt = -np.ones((B, M), np.int64)
+    page = np.zeros((B, 8), np.float32); src = np.zeros(B, np.int64)
+    rel = np.zeros((B, T, T, K_REL), np.float32); mask = np.zeros((B, T), bool)
+    tb_pair = -np.ones((B, N), np.int64)
+    for i, s in enumerate(batch):
+        n, m = len(s["role"]), len(s["IF"])
+        tok_f[i, :n] = s["F"]; role[i, :n] = s["role"]; lab[i, :n] = s["lab_in"]; lab_t[i, :n] = s["lab_t"]; tmask[i, :n] = s["tmask"]
+        w[i, :n] = s["w"]; alpha_in[i, :n] = s["rgba"][:, 3]; alpha_t[i, :n] = s["alpha_t"]
+        if m:
+            img_f[i, :m] = s["IF"]; thumbs[i, :m] = s["thumbs"].transpose(0, 3, 1, 2) / 255.0; imgt[i, :m] = s["imgt"]
+        page[i] = s["page"]; src[i] = s["src"]
+        mask[i, :1 + n] = True; mask[i, 1 + N:1 + N + m] = True
+        # relations
+        r = s["rel"]
+        if len(r):
+            typ, a, b, wt = r[:, 0].astype(int), r[:, 1].astype(int) + 1, r[:, 2].astype(int) + 1, r[:, 3]
+            R = np.zeros((4, n + 1, n + 1), np.float32)
+            np.maximum.at(R, (typ, a, b), wt)
+            R /= R.max(axis=2, keepdims=True) + 1e-6
+            rel[i, :n + 1, :n + 1, 0:4] = R.transpose(1, 2, 0)
+            rel[i, :n + 1, :n + 1, 4:8] = R.transpose(2, 1, 0)
+            # dominant surface for each text token (contrast loss)
+            tbm = typ == 0
+            best = {}
+            for aa, bb, ww in zip(a[tbm] - 1, b[tbm] - 1, wt[tbm]):
+                if ww > best.get(aa, (-1, -1))[0]:
+                    best[aa] = (ww, bb)
+            for aa, (ww, bb) in best.items():
+                tb_pair[i, aa] = bb
+        for j in range(m):
+            u = s["img_under"][j]
+            if u >= 0:
+                rel[i, 1 + N + j, 1 + u, 8] = 1.0
+                rel[i, 1 + u, 1 + N + j, 9] = 1.0
+        # same colour, different role
+        key = s["rgba"][:, :3] @ np.array([65536, 256, 1], np.float32)
+        same = (key[:, None] == key[None, :]) & ~np.eye(n, dtype=bool)
+        rel[i, 1:1 + n, 1:1 + n, 10] = same
+    if device is None:
+        return dict(tok_f=tok_f, role=role, lab=lab, lab_t=lab_t, tmask=tmask, w=w, alpha_in=alpha_in, alpha_t=alpha_t,
+                    img_f=img_f, thumbs=thumbs, imgt=imgt, page=page, src=src, rel=rel, mask=mask, tb_pair=tb_pair)
+    t = lambda x: torch.from_numpy(x).to(device, non_blocking=True)
+    return dict(tok_f=t(tok_f), role=t(role), lab=t(lab), lab_t=t(lab_t), tmask=t(tmask), w=t(w), alpha_in=t(alpha_in), alpha_t=t(alpha_t),
+                img_f=t(img_f), thumbs=t(thumbs), imgt=t(imgt), page=t(page), src=t(src), rel=t(rel), mask=t(mask), tb_pair=t(tb_pair))
+
+
+class Batches(torch.utils.data.Dataset):
+    """Each item is a whole (augmented) batch, assembled in a worker process."""
+    def __init__(self, data, order, bs, aug, seed):
+        self.data, self.order, self.bs, self.aug, self.seed = data, order, bs, aug, seed
+
+    def __len__(self):
+        return math.ceil(len(self.order) / self.bs)
+
+    def __getitem__(self, k):
+        rng = np.random.default_rng((self.seed, k))
+        items = [self.data[i] for i in self.order[k * self.bs:(k + 1) * self.bs]]
+        if self.aug:
+            items = [augment(s, rng) for s in items]
+        return collate(items, None)
+
+
+def to_dev(b, device):
+    return {k: (v if torch.is_tensor(v) else torch.from_numpy(v)).to(device, non_blocking=True) for k, v in b.items()}
+
+
+def huber(x, d):
+    ax = x.abs()
+    return torch.where(ax < d, 0.5 * x * x / d, ax - 0.5 * d)
+
+
+def losses(model, b, img_w=None):
+    lab, alpha_logit, img_logits = model(b["tok_f"], b["role"], b["lab"], b["img_f"], b["thumbs"], b["page"], b["src"], b["rel"], b["mask"])
+    lab = lab.float(); alpha_logit = alpha_logit.float(); img_logits = img_logits.float()
+    m = b["tmask"].float() * b["w"]
+    d = lab - b["lab_t"]
+    per = 2.0 * huber(d[..., 0], 0.02) + huber(d[..., 1], 0.01) + huber(d[..., 2], 0.01)
+    lc = (per * m).sum() / m.sum().clamp_min(1)
+    # alpha (only where translucency is involved)
+    am = m * ((b["alpha_in"] < 0.999) | (b["alpha_t"] < 0.999)).float()
+    a_in = b["alpha_in"].clamp(1e-3, 1 - 1e-3)
+    alpha = torch.sigmoid(alpha_logit + torch.log(a_in / (1 - a_in)))
+    la = (huber(alpha - b["alpha_t"], 0.05) * am).sum() / am.sum().clamp_min(1)
+    # readability: text must not lose contrast against its own surface
+    pair = b["tb_pair"]
+    pm = (pair >= 0) & b["tmask"] & (b["role"] == 1) & (b["alpha_in"] > 0.99)
+    if pm.any():
+        idx = pair.clamp_min(0)
+        bg_pred = torch.gather(lab, 1, idx[..., None].expand(-1, -1, 3))
+        bg_t = torch.gather(b["lab_t"], 1, idx[..., None].expand(-1, -1, 3))
+        bgm = torch.gather(b["tmask"], 1, idx)
+        pm = pm & bgm
+        c_pred = contrast_ratio(lab, bg_pred)
+        c_t = contrast_ratio(b["lab_t"], bg_t).clamp(max=4.5)
+        lcon = (F.relu(c_t - c_pred) * pm.float()).sum() / pm.float().sum().clamp_min(1)
+    else:
+        lcon = lab.new_zeros(())
+    # images
+    im = b["imgt"] >= 0
+    if im.any():
+        li = F.cross_entropy(img_logits[im], b["imgt"][im], weight=img_w, label_smoothing=0.05)
+        acc = (img_logits[im].argmax(-1) == b["imgt"][im]).float().mean()
+    else:
+        li = lab.new_zeros(()); acc = lab.new_zeros(())
+    with torch.no_grad():
+        de = (d.norm(dim=-1) * m).sum() / m.sum().clamp_min(1)
+        dl = (d[..., 0].abs() * m).sum() / m.sum().clamp_min(1)
+    return lc, la, lcon, li, dict(dE=de.item(), dL=dl.item(), img_acc=acc.item(), n_img=int(im.sum()))
+
+
+def evaluate(model, val, device, bs, img_w):
+    model.eval()
+    agg = defaultdict(float); cnt = defaultdict(float)
+    with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
+        for src in (0, 1):
+            vs = [s for s in val if s["src"] == src]
+            for k in range(0, len(vs), bs):
+                b = collate(vs[k:k + bs], device)
+                lc, la, lcon, li, st = losses(model, b, img_w)
+                n = len(vs[k:k + bs])
+                agg[f"s{src}_dE"] += st["dE"] * n; agg[f"s{src}_dL"] += st["dL"] * n; agg[f"s{src}_con"] += lcon.item() * n
+                cnt[f"s{src}"] += n
+                if st["n_img"]:
+                    agg["img_acc"] += st["img_acc"] * st["n_img"]; cnt["img"] += st["n_img"]
+    out = {}
+    for k, v in agg.items():
+        if k == "img_acc":
+            out[k] = v / max(1, cnt["img"])
+        else:
+            out[k] = v / max(1, cnt[k.split("_")[0]])
+    model.train()
+    return out
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--data", required=True)
+    ap.add_argument("--out", required=True)
+    ap.add_argument("--d", type=int, default=384)
+    ap.add_argument("--layers", type=int, default=8)
+    ap.add_argument("--heads", type=int, default=6)
+    ap.add_argument("--epochs", type=int, default=40)
+    ap.add_argument("--bs", type=int, default=48)
+    ap.add_argument("--lr", type=float, default=4e-4)
+    ap.add_argument("--wd", type=float, default=0.05)
+    ap.add_argument("--drop", type=float, default=0.1)
+    ap.add_argument("--native-weight", type=float, default=0.5)
+    ap.add_argument("--seed", type=int, default=0)
+    ap.add_argument("--workers", type=int, default=8)
+    args = ap.parse_args()
+    torch.manual_seed(args.seed); rng = np.random.default_rng(args.seed)
+    os.makedirs(args.out, exist_ok=True)
+    data = pickle.load(open(args.data, "rb"))
+    train, val = [prep(s) for s in data["train"]], [prep(s) for s in data["val"]]
+    print(f"train {len(train)} val {len(val)} (haiku {sum(s['src'] == 0 for s in train)}, native {sum(s['src'] == 1 for s in train)})", flush=True)
+    device = "cuda"
+    model = NadirNet(d=args.d, layers=args.layers, heads=args.heads, drop=args.drop).to(device)
+    nparam = sum(p.numel() for p in model.parameters())
+    print(f"params {nparam / 1e6:.2f}M", flush=True)
+    # image class weights (inverse sqrt frequency)
+    cnt = np.ones(5)
+    for s in train:
+        for v in s["imgt"]:
+            if v >= 0:
+                cnt[v] += 1
+    img_w = torch.tensor((cnt.sum() / cnt) ** 0.5, dtype=torch.float32, device=device)
+    img_w = img_w / img_w.mean()
+    print("image label counts", cnt.astype(int).tolist(), flush=True)
+    opt = torch.optim.AdamW(model.parameters(), lr=args.lr, weight_decay=args.wd, betas=(0.9, 0.98))
+    steps_per = math.ceil(len(train) / args.bs)
+    total = steps_per * args.epochs
+    warm = min(800, total // 20)
+    sched = torch.optim.lr_scheduler.LambdaLR(opt, lambda st: min(1, (st + 1) / warm) * (0.1 + 0.9 * 0.5 * (1 + math.cos(math.pi * min(1, st / total)))))
+    # sample weights: native pages count less (different designers, noisier pairing)
+    sw = np.array([1.0 if s["src"] == 0 else args.native_weight for s in train]); sw /= sw.sum()
+    best = 1e9; step = 0; log = []
+    t0 = time.time()
+    for ep in range(args.epochs):
+        order = rng.choice(len(train), size=len(train), p=sw)
+        run = defaultdict(float); nb = 0
+        loader = torch.utils.data.DataLoader(Batches(train, order, args.bs, True, args.seed * 1000 + ep), batch_size=None, shuffle=False,
+                                             num_workers=args.workers, persistent_workers=False, prefetch_factor=4 if args.workers else None)
+        for bnp in loader:
+            b = to_dev(bnp, device)
+            with torch.autocast("cuda", dtype=torch.bfloat16):
+                lc, la, lcon, li, st = losses(model, b, img_w)
+            loss = lc + 0.5 * la + 0.02 * lcon + 0.05 * li
+            opt.zero_grad(set_to_none=True)
+            if not torch.isfinite(loss):  # one bad page must never poison the weights
+                run["skipped"] += 1
+                continue
+            loss.backward()
+            torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
+            opt.step(); sched.step(); step += 1
+            run["loss"] += loss.item(); run["dE"] += st["dE"]; run["con"] += lcon.item(); run["img"] += li.item(); nb += 1
+        ev = evaluate(model, val, device, 64, img_w)
+        rec = {"epoch": ep, "step": step, "secs": round(time.time() - t0), **{k: round(v / nb, 5) for k, v in run.items()}, **{k: round(v, 5) for k, v in ev.items()}}
+        log.append(rec)
+        print(json.dumps(rec), flush=True)
+        score = ev.get("s0_dE", 9) + 0.3 * ev.get("s1_dE", 0) + 0.02 * ev.get("s0_con", 0) + 0.02 * (1 - ev.get("img_acc", 0))
+        if score < best:
+            best = score
+            torch.save({"model": model.state_dict(), "cfg": model.cfg, "epoch": ep, "eval": ev}, os.path.join(args.out, "best.pt"))
+    torch.save({"model": model.state_dict(), "cfg": model.cfg, "epoch": args.epochs - 1}, os.path.join(args.out, "last.pt"))
+    json.dump({"args": vars(args), "params": nparam, "log": log, "best_score": best}, open(os.path.join(args.out, "train_log.json"), "w"), indent=1)
+
+
+if __name__ == "__main__":
+    main()