NeonClary commited on
Commit
0faa1e7
·
verified ·
1 Parent(s): 38c304b

Add Neon leaderboard dashboard (single-viewport, Docker static)

Browse files
Files changed (8) hide show
  1. .gitignore +4 -0
  2. Dockerfile +14 -0
  3. README.md +100 -11
  4. app.js +569 -0
  5. data.js +0 -0
  6. data.json +0 -0
  7. index.html +313 -0
  8. scripts/build_data.py +493 -0
.gitignore ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ .shots/
2
+ __pycache__/
3
+ *.pyc
4
+ .DS_Store
Dockerfile ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Static dashboard served on the port Hugging Face Spaces expects (7860).
2
+ # The app is a single HTML page + precomputed data; no backend logic is needed,
3
+ # so a tiny static file server is all that's required.
4
+ FROM python:3.12-slim
5
+
6
+ WORKDIR /app
7
+
8
+ # Only the files the browser actually needs.
9
+ COPY index.html app.js data.js data.json ./
10
+
11
+ EXPOSE 7860
12
+
13
+ # Serve from /app on 0.0.0.0:7860. index.html is returned for "/".
14
+ CMD ["python", "-m", "http.server", "7860", "--bind", "0.0.0.0"]
README.md CHANGED
@@ -1,11 +1,100 @@
1
- ---
2
- title: Leaderboard
3
- emoji: 🔥
4
- colorFrom: gray
5
- colorTo: indigo
6
- sdk: docker
7
- pinned: false
8
- short_description: Model performance comparison leaderboard
9
- ---
10
-
11
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ title: Neon Leaderboard
3
+ emoji: 📊
4
+ colorFrom: green
5
+ colorTo: blue
6
+ sdk: docker
7
+ app_port: 7860
8
+ pinned: false
9
+ short_description: Single-viewport dashboard for the Neon model leaderboard
10
+ ---
11
+
12
+ # Neon Leaderboard
13
+
14
+ A single-page, single-viewport dashboard for exploring the
15
+ [`neon-router`](https://github.com/NeonGeckoCom/neon-router) weekly leaderboard —
16
+ Neon.ai models and retrieval pipelines evaluated across seven QA benchmarks.
17
+
18
+ Everything is **static**: no backend, no build step at serve time. The page reads
19
+ one precomputed file (`data.js`) and renders entirely in the browser, so it deploys
20
+ identically to a Hugging Face *Static* Space or GitHub Pages.
21
+
22
+ ## What's inside
23
+
24
+ | Tab | What it shows |
25
+ |---|---|
26
+ | **Compare** | The full results grid. Pick a benchmark and see every pipeline ranked, or switch to **By model** to follow one pipeline across all benchmarks. Searchable benchmark / model dropdowns, sort by any metric (MRR, speed, cost…), filter to a single model, toggle Neon-only / disqualified rows, and relative color heat per column. |
27
+ | **Picks** | The router's single recommended pipeline per client **profile** × benchmark, with a statistical verdict (winner clear / inconclusive / all disqualified). Hover a profile header for its hard gates. |
28
+ | **Models** | A summary page for any model: pipeline, repo, mean MRR / gen / latency, Pareto and DQ counts, and its metrics across every benchmark. |
29
+ | **Metrics** | All models × all metrics — either **mean scores** (color-coded) or **wins / losses** per metric. Sortable by any metric. |
30
+ | **Guide** | Full metrics glossary, every client profile's hard gates, and run provenance. |
31
+
32
+ Hover any column header anywhere for a plain-language definition (e.g. *P50 = 50th
33
+ percentile latency*, *MRR = Mean Reciprocal Rank*). A light/dark toggle (◐) sits in
34
+ the top-right; append `?theme=light` to the URL to share a light-mode link, and
35
+ `#picks` / `#models` / `#metrics` / `#guide` to deep-link a tab.
36
+
37
+ ## Faithfulness to the source
38
+
39
+ `data.js` is generated from the leaderboard's own source of truth — `run.json` plus
40
+ the profile YAMLs — and the picks, Pareto front, disqualifications, verdicts, and
41
+ win/loss tallies are recomputed with logic ported directly from
42
+ `router/pareto.py` and `router/reports/leaderboard.py`. Spot-checked to match
43
+ `reports/dashboard/leaderboard.md` exactly.
44
+
45
+ ## Regenerating the data
46
+
47
+ When the leaderboard is re-run, rebuild `data.js` from a `neon-router` checkout:
48
+
49
+ ```bash
50
+ python scripts/build_data.py \
51
+ --run-json /path/to/neon-router/reports/dashboard/raw/run.json \
52
+ --profiles-dir /path/to/neon-router/configs/profiles \
53
+ --out data.js
54
+ ```
55
+
56
+ With a `neon-router` clone sitting next to this folder, the defaults already point
57
+ at it, so `python scripts/build_data.py` is enough. The script needs only
58
+ `pyyaml` (`pip install pyyaml`).
59
+
60
+ To change which models get the **NEON** badge, edit the `NEON_MODELS` set near the
61
+ top of `scripts/build_data.py` and re-run it.
62
+
63
+ ## Deploying
64
+
65
+ ### Hugging Face Static Space (BrainForge)
66
+
67
+ 1. Create a new Space → SDK **Static** under the `BrainForge` org.
68
+ 2. Push `index.html`, `app.js`, `data.js`, and this `README.md` (its front-matter
69
+ above configures the Space). That's the whole site.
70
+
71
+ ```bash
72
+ # from this folder, with the Space repo as the remote
73
+ git lfs install --skip-repo
74
+ git add index.html app.js data.js README.md scripts/build_data.py
75
+ git commit -m "Neon leaderboard dashboard"
76
+ git push
77
+ ```
78
+
79
+ ### GitHub Pages (NeonGeckoCom)
80
+
81
+ 1. Put these files at the repo root (or in `/docs`).
82
+ 2. Settings → Pages → deploy from branch, root (or `/docs`).
83
+ No build step is required — it's plain static files.
84
+
85
+ ## Local preview
86
+
87
+ ```bash
88
+ python -m http.server 8000
89
+ # open http://localhost:8000/
90
+ ```
91
+
92
+ Files:
93
+
94
+ ```
95
+ index.html # markup + styles (single viewport shell)
96
+ app.js # all rendering / interactions (vanilla JS, no deps)
97
+ data.js # precomputed dataset (window.LEADERBOARD_DATA)
98
+ data.json # same data as plain JSON (for non-browser consumers)
99
+ scripts/build_data.py # regenerates data.js / data.json from run.json + profiles
100
+ ```
app.js ADDED
@@ -0,0 +1,569 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ /* Neon Leaderboard - static dashboard.
2
+ All data is precomputed in data.js (window.LEADERBOARD_DATA) by
3
+ scripts/build_data.py, faithful to the neon-router report logic. */
4
+ (function () {
5
+ "use strict";
6
+ const D = window.LEADERBOARD_DATA;
7
+ if (!D) { document.body.innerHTML = "<p style='padding:30px'>data.js failed to load.</p>"; return; }
8
+
9
+ const $ = (s, r) => (r || document).querySelector(s);
10
+ const $$ = (s, r) => Array.from((r || document).querySelectorAll(s));
11
+ const el = (tag, attrs, kids) => {
12
+ const n = document.createElement(tag);
13
+ if (attrs) for (const k in attrs) {
14
+ if (k === "class") n.className = attrs[k];
15
+ else if (k === "html") n.innerHTML = attrs[k];
16
+ else if (k === "text") n.textContent = attrs[k];
17
+ else if (k.startsWith("on") && typeof attrs[k] === "function") n.addEventListener(k.slice(2), attrs[k]);
18
+ else if (attrs[k] != null) n.setAttribute(k, attrs[k]);
19
+ }
20
+ if (kids) (Array.isArray(kids) ? kids : [kids]).forEach(c => c != null && n.appendChild(typeof c === "string" ? document.createTextNode(c) : c));
21
+ return n;
22
+ };
23
+
24
+ const GLOSS = D.glossary || {};
25
+ const METRICS = D.metrics_def; // [{key,label,field,higher,fmt,tip}]
26
+ const BENCH = D.benchmarks;
27
+ const AGG = D.aggregates;
28
+ const CANDS = D.candidates; // map id -> meta
29
+
30
+ const aggBy = {}; // benchmark -> [agg]
31
+ const aggByCand = {}; // candidate_id -> [agg] (across benchmarks)
32
+ AGG.forEach(a => {
33
+ (aggBy[a.benchmark] = aggBy[a.benchmark] || []).push(a);
34
+ (aggByCand[a.candidate_id] = aggByCand[a.candidate_id] || []).push(a);
35
+ });
36
+ const CAND_IDS = Object.keys(CANDS);
37
+ const MODEL_NAMES = Array.from(new Set(CAND_IDS.map(id => CANDS[id].model)));
38
+ // Prefer a Neon model with generation data as the default detail selection.
39
+ const DEFAULT_CAND = CAND_IDS.find(id => CANDS[id].is_neon) || CAND_IDS[0];
40
+
41
+ // ---- formatting ----
42
+ function num(v, code) {
43
+ if (v == null || v === "") return "";
44
+ const d = { f0: 0, f1: 1, f2: 2, f3: 3, f5: 5 }[code];
45
+ return Number(v).toFixed(d == null ? 3 : d);
46
+ }
47
+ function pipeLabel(c) {
48
+ const r = c.reranker && c.reranker !== "none" ? " · " + c.reranker : "";
49
+ return c.model + " · " + c.retriever + r;
50
+ }
51
+ function benchShort(b) { return b.replace(/^dataset-/, "").replace(/-qa$/, ""); }
52
+
53
+ // ---- tooltip ----
54
+ const tip = $("#tip");
55
+ function showTip(html, x, y) {
56
+ tip.innerHTML = html; tip.style.opacity = "1";
57
+ const r = tip.getBoundingClientRect();
58
+ let left = x + 14, top = y + 16;
59
+ if (left + r.width > innerWidth - 8) left = x - r.width - 12;
60
+ if (top + r.height > innerHeight - 8) top = y - r.height - 12;
61
+ tip.style.left = Math.max(6, left) + "px"; tip.style.top = Math.max(6, top) + "px";
62
+ }
63
+ function hideTip() { tip.style.opacity = "0"; }
64
+ function bindTip(node, html) {
65
+ if (!html) return node;
66
+ node.addEventListener("mousemove", e => showTip(html, e.clientX, e.clientY));
67
+ node.addEventListener("mouseleave", hideTip);
68
+ return node;
69
+ }
70
+ function glossTip(term, extra) {
71
+ const def = GLOSS[term];
72
+ if (!def && !extra) return null;
73
+ return "<b>" + term + "</b><br>" + (def || "") + (extra ? "<br><br>" + extra : "");
74
+ }
75
+
76
+ // ---- color heat (diverging, low-glare) ----
77
+ function heatStyle(v, min, max, higher) {
78
+ if (v == null || v === "" || max === min) return "";
79
+ let t = (v - min) / (max - min); // 0..1, 1 = numerically max
80
+ if (!higher) t = 1 - t; // 1 = good
81
+ const intensity = Math.abs(t - 0.5) * 2; // 0 mid .. 1 extreme
82
+ const hue = t >= 0.5 ? 168 : 28;
83
+ const dark = document.documentElement.getAttribute("data-theme") !== "light";
84
+ const light = dark ? 46 : 60;
85
+ const sat = dark ? 52 : 62;
86
+ const alpha = (0.05 + intensity * 0.30).toFixed(3);
87
+ return "background:hsla(" + hue + "," + sat + "%," + light + "%," + alpha + ")";
88
+ }
89
+
90
+ // ---- searchable combobox (single select) ----
91
+ function Combo(host, opts) {
92
+ // opts: {options:[{value,label,sub,neon}], value, placeholder, onChange, width}
93
+ let value = opts.value;
94
+ let open = false, kbd = -1, filtered = opts.options;
95
+ host.className = "combo";
96
+ const btn = el("button", { class: "field", style: "text-align:left;min-width:" + (opts.width || 180) + "px;display:flex;align-items:center;gap:6px" });
97
+ const menu = el("div", { class: "menu" });
98
+ const search = el("input", { class: "field", placeholder: "Type to search…", style: "width:100%;margin-bottom:5px" });
99
+ const list = el("div");
100
+ menu.appendChild(search); menu.appendChild(list);
101
+ host.appendChild(btn); host.appendChild(menu);
102
+
103
+ function curLabel() { const o = opts.options.find(o => o.value === value); return o ? o.label : (opts.placeholder || "Select…"); }
104
+ function paintBtn() {
105
+ btn.innerHTML = "";
106
+ const o = opts.options.find(o => o.value === value);
107
+ if (o && o.neon) btn.appendChild(el("span", { class: "badge neon", text: "NEON" }));
108
+ btn.appendChild(el("span", { text: curLabel(), style: "flex:1;overflow:hidden;text-overflow:ellipsis;white-space:nowrap" }));
109
+ btn.appendChild(el("span", { text: "▾", style: "color:var(--text-faint);font-size:10px" }));
110
+ }
111
+ function paintList() {
112
+ list.innerHTML = "";
113
+ filtered.forEach((o, i) => {
114
+ const row = el("div", { class: "opt" + (o.value === value ? " sel" : "") + (i === kbd ? " kbd" : "") });
115
+ if (o.neon) row.appendChild(el("span", { class: "badge neon", text: "NEON" }));
116
+ row.appendChild(el("span", { text: o.label }));
117
+ if (o.sub) row.appendChild(el("small", { text: o.sub }));
118
+ row.addEventListener("click", () => { value = o.value; close(); opts.onChange && opts.onChange(value); });
119
+ list.appendChild(row);
120
+ });
121
+ if (!filtered.length) list.appendChild(el("div", { class: "opt", style: "color:var(--text-faint)", text: "No matches" }));
122
+ }
123
+ function doFilter() {
124
+ const q = search.value.trim().toLowerCase();
125
+ filtered = !q ? opts.options : opts.options.filter(o =>
126
+ (o.label + " " + (o.sub || "")).toLowerCase().includes(q));
127
+ kbd = filtered.length ? 0 : -1; paintList();
128
+ }
129
+ function openMenu() { open = true; host.classList.add("open"); search.value = ""; doFilter(); setTimeout(() => search.focus(), 0); }
130
+ function close() { open = false; host.classList.remove("open"); paintBtn(); }
131
+ btn.addEventListener("click", e => { e.stopPropagation(); open ? close() : openMenu(); });
132
+ search.addEventListener("input", doFilter);
133
+ search.addEventListener("keydown", e => {
134
+ if (e.key === "ArrowDown") { kbd = Math.min(filtered.length - 1, kbd + 1); paintList(); e.preventDefault(); }
135
+ else if (e.key === "ArrowUp") { kbd = Math.max(0, kbd - 1); paintList(); e.preventDefault(); }
136
+ else if (e.key === "Enter" && filtered[kbd]) { value = filtered[kbd].value; close(); opts.onChange && opts.onChange(value); }
137
+ else if (e.key === "Escape") close();
138
+ });
139
+ document.addEventListener("click", e => { if (open && !host.contains(e.target)) close(); });
140
+ paintBtn();
141
+ return {
142
+ get value() { return value; },
143
+ set value(v) { value = v; paintBtn(); },
144
+ setOptions(o, v) { opts.options = o; if (v !== undefined) value = v; paintBtn(); }
145
+ };
146
+ }
147
+
148
+ // ===================================================================
149
+ // State
150
+ // ===================================================================
151
+ const S = {
152
+ tab: "compare",
153
+ cmpMode: "byBench",
154
+ bench: BENCH[0],
155
+ cand: null, // selected pipeline id (byModel)
156
+ filterModel: "__all__",
157
+ sortKey: "mrr",
158
+ sortDir: -1, // -1 desc, 1 asc
159
+ neonOnly: false,
160
+ showDQ: true,
161
+ heat: true,
162
+ metricMode: "scores",
163
+ metricSort: "mrr",
164
+ metricNeonOnly: false,
165
+ metricHeat: true,
166
+ pageCand: null,
167
+ };
168
+
169
+ // ===================================================================
170
+ // Header / footer
171
+ // ===================================================================
172
+ $("#headerMeta").textContent = (D.date || "") + " · " + D.n_candidates + " pipelines · " +
173
+ BENCH.length + " benchmarks · " + (D.n_queries || "?") + " queries";
174
+ const prov = D.provenance || {};
175
+ $("#provFoot").textContent = (prov.weekly ? "weekly roster sweep" : "sweep") + (prov.merged_at ? " · merged " + prov.merged_at.slice(0, 19).replace("T", " ") + "Z" : "");
176
+ function paintLegendSwatches() {
177
+ $("#lgGood").style.cssText = "width:11px;height:11px;border-radius:3px;display:inline-block;" + heatStyle(1, 0, 1, true);
178
+ $("#lgBad").style.cssText = "width:11px;height:11px;border-radius:3px;display:inline-block;" + heatStyle(0, 0, 1, true);
179
+ }
180
+
181
+ // ===================================================================
182
+ // Tabs
183
+ // ===================================================================
184
+ const CTLS = { compare: "#ctl-compare", picks: "#ctl-picks", models: "#ctl-models", metrics: "#ctl-metrics", guide: "#ctl-guide" };
185
+ $$("#tabs button").forEach(b => b.addEventListener("click", () => setTab(b.dataset.tab)));
186
+ function setTab(t) {
187
+ S.tab = t;
188
+ try { history.replaceState(null, "", "#" + t); } catch (e) {}
189
+ $$("#tabs button").forEach(b => b.classList.toggle("active", b.dataset.tab === t));
190
+ $$(".view").forEach(v => v.classList.toggle("active", v.id === "view-" + t));
191
+ for (const k in CTLS) $(CTLS[k]).style.display = (k === t ? "flex" : "none");
192
+ render();
193
+ }
194
+
195
+ // ===================================================================
196
+ // Compare controls
197
+ // ===================================================================
198
+ const benchCombo = Combo($("#benchCombo"), {
199
+ options: BENCH.map(b => ({ value: b, label: benchShort(b), sub: b })),
200
+ value: S.bench, width: 170, onChange: v => { S.bench = v; render(); }
201
+ });
202
+ const candOptions = () => CAND_IDS.map(id => ({ value: id, label: pipeLabel(CANDS[id]), neon: CANDS[id].is_neon, sub: CANDS[id].is_neon ? "Neon" : (CANDS[id].is_retrieval_only ? "retrieval" : "") }));
203
+ const modelCombo = Combo($("#modelCombo"), { options: candOptions(), value: DEFAULT_CAND, width: 230, onChange: v => { S.cand = v; render(); } });
204
+ S.cand = DEFAULT_CAND;
205
+ const filterCombo = Combo($("#filterCombo"), {
206
+ options: [{ value: "__all__", label: "All models" }].concat(MODEL_NAMES.map(m => ({ value: m, label: m, neon: (CANDS[CAND_IDS.find(id => CANDS[id].model === m)] || {}).is_neon }))),
207
+ value: "__all__", width: 160, onChange: v => { S.filterModel = v; render(); }
208
+ });
209
+ const sortSel = $("#sortSel");
210
+ function fillSortSel() {
211
+ sortSel.innerHTML = "";
212
+ sortSel.appendChild(el("option", { value: "pipe", text: "Pipeline (A–Z)" }));
213
+ METRICS.forEach(m => sortSel.appendChild(el("option", { value: m.field, text: m.label })));
214
+ sortSel.value = S.sortKey;
215
+ }
216
+ fillSortSel();
217
+ sortSel.addEventListener("change", () => { S.sortKey = sortSel.value; render(); });
218
+ $("#sortDir").addEventListener("click", () => { S.sortDir *= -1; $("#sortDir").textContent = S.sortDir < 0 ? "▾" : "▴"; render(); });
219
+ $$("#cmpModeSeg button").forEach(b => b.addEventListener("click", () => {
220
+ S.cmpMode = b.dataset.mode;
221
+ $$("#cmpModeSeg button").forEach(x => x.classList.toggle("active", x === b));
222
+ $("#benchPickWrap").style.display = S.cmpMode === "byBench" ? "flex" : "none";
223
+ $("#modelPickWrap").style.display = S.cmpMode === "byModel" ? "flex" : "none";
224
+ render();
225
+ }));
226
+ $("#neonOnly").addEventListener("change", e => { S.neonOnly = e.target.checked; render(); });
227
+ $("#showDQ").addEventListener("change", e => { S.showDQ = e.target.checked; render(); });
228
+ $("#heat").addEventListener("change", e => { S.heat = e.target.checked; render(); });
229
+
230
+ // ===================================================================
231
+ // Compare render
232
+ // ===================================================================
233
+ function activeMetricCols(rows) {
234
+ return METRICS.filter(m => rows.some(r => r[m.field] != null && r[m.field] !== ""));
235
+ }
236
+ function renderCompare() {
237
+ const table = $("#cmpTable");
238
+ table.innerHTML = "";
239
+ let rows, flagFor, leftHead, leftCell;
240
+
241
+ if (S.cmpMode === "byBench") {
242
+ const bt = D.bench_tables[S.bench] || { order: [], pareto: [], dq: {} };
243
+ const paretoSet = new Set(bt.pareto);
244
+ rows = (aggBy[S.bench] || []).slice();
245
+ if (S.filterModel !== "__all__") rows = rows.filter(r => r.model === S.filterModel);
246
+ if (S.neonOnly) rows = rows.filter(r => r.is_neon);
247
+ if (!S.showDQ) rows = rows.filter(r => !bt.dq[r.candidate_id]);
248
+ flagFor = r => ({ pareto: paretoSet.has(r.candidate_id), dq: bt.dq[r.candidate_id] });
249
+ leftHead = "pipeline"; leftCell = r => pipeCell(r, flagFor(r));
250
+ } else {
251
+ rows = (aggByCand[S.cand] || []).slice();
252
+ flagFor = () => ({});
253
+ leftHead = "benchmark";
254
+ leftCell = r => bindTip(el("td", { class: "txt" }, el("span", { class: "pipe", text: benchShort(r.benchmark) })), "<b>" + r.benchmark + "</b>");
255
+ }
256
+
257
+ const cols = activeMetricCols(rows);
258
+ // sort
259
+ rows.sort((a, b) => {
260
+ if (S.sortKey === "pipe") {
261
+ const av = S.cmpMode === "byBench" ? pipeLabel(CANDS[a.candidate_id]) : a.benchmark;
262
+ const bv = S.cmpMode === "byBench" ? pipeLabel(CANDS[b.candidate_id]) : b.benchmark;
263
+ return av < bv ? S.sortDir : av > bv ? -S.sortDir : 0;
264
+ }
265
+ const av = a[S.sortKey], bv = b[S.sortKey];
266
+ if (av == null && bv == null) return 0;
267
+ if (av == null) return 1; if (bv == null) return -1;
268
+ return (av - bv) * S.sortDir;
269
+ });
270
+
271
+ // column ranges for heat
272
+ const ranges = {};
273
+ cols.forEach(m => {
274
+ const vs = rows.map(r => r[m.field]).filter(v => v != null && v !== "");
275
+ ranges[m.field] = vs.length ? { min: Math.min(...vs), max: Math.max(...vs) } : null;
276
+ });
277
+
278
+ // head
279
+ const thead = el("thead");
280
+ const htr = el("tr");
281
+ htr.appendChild(bindTip(el("th", { class: "txt lbl sortable", onclick: () => sortByCol("pipe"), html: leftHead + (S.sortKey === "pipe" ? " <span class='arrow'>" + (S.sortDir < 0 ? "▼" : "▲") + "</span>" : "") }),
282
+ S.cmpMode === "byBench" ? glossTip("pipeline", "model · retriever · reranker") : "<b>benchmark</b><br>The evaluation dataset for this row."));
283
+ cols.forEach(m => {
284
+ const arrow = S.sortKey === m.field ? " <span class='arrow'>" + (S.sortDir < 0 ? "▼" : "▲") + "</span>" : "";
285
+ htr.appendChild(bindTip(el("th", { class: "sortable", onclick: () => sortByCol(m.field), html: m.label + arrow }),
286
+ glossTip(m.tip, (m.higher ? "Higher is better." : "Lower is better."))));
287
+ });
288
+ thead.appendChild(htr); table.appendChild(thead);
289
+
290
+ // body
291
+ const tb = el("tbody");
292
+ if (!rows.length) { tb.appendChild(el("tr", null, el("td", { colspan: cols.length + 1, class: "empty", text: "No rows match the current filters." }))); }
293
+ rows.forEach(r => {
294
+ const tr = el("tr", { class: r.is_neon ? "neon" : "" });
295
+ tr.appendChild(leftCell(r));
296
+ cols.forEach(m => {
297
+ const v = r[m.field];
298
+ const td = el("td", { text: num(v, m.fmt) });
299
+ if (S.heat && ranges[m.field]) td.style.cssText = heatStyle(v, ranges[m.field].min, ranges[m.field].max, m.higher);
300
+ tr.appendChild(td);
301
+ });
302
+ tb.appendChild(tr);
303
+ });
304
+ table.appendChild(tb);
305
+ }
306
+ function sortByCol(k) { if (S.sortKey === k) S.sortDir *= -1; else { S.sortKey = k; S.sortDir = (k === "pipe") ? 1 : -1; } sortSel.value = k; $("#sortDir").textContent = S.sortDir < 0 ? "▾" : "▴"; render(); }
307
+
308
+ function pipeCell(r, flag) {
309
+ const c = CANDS[r.candidate_id];
310
+ const inner = el("span", { class: "pipe" });
311
+ if (flag.pareto) inner.appendChild(el("span", { class: "star", text: "★ " }));
312
+ inner.appendChild(el("b", { text: c.model }));
313
+ inner.appendChild(el("small", { text: " " + c.retriever + (c.reranker !== "none" ? " · " + c.reranker : "") }));
314
+ const td = el("td", { class: "txt" });
315
+ if (c.is_neon) td.appendChild(el("span", { class: "badge neon", text: "NEON", style: "margin-right:6px" }));
316
+ td.appendChild(inner);
317
+ if (flag.dq) td.appendChild(el("span", { class: "badge dq", text: "DQ:" + flag.dq, style: "margin-left:7px" }));
318
+ return bindTip(td, "<b>" + pipeLabel(c) + "</b>" +
319
+ (c.model_repo ? "<br>repo: " + c.model_repo : "") +
320
+ (flag.dq ? "<br><br>Disqualified by a profile's <b>" + flag.dq + "</b> gate." : "") +
321
+ (flag.pareto ? "<br><br>★ On the Pareto front (not dominated on accuracy / latency / build-cost)." : ""));
322
+ }
323
+
324
+ // ===================================================================
325
+ // Picks render
326
+ // ===================================================================
327
+ const VCLASS = { "winner clear": "v-clear", "inconclusive": "v-inconclusive", "all disqualified": "v-dq", "no winner": "v-none", "no signal": "v-none" };
328
+ function renderPicks() {
329
+ const table = $("#picksTable"); table.innerHTML = "";
330
+ const profs = D.profiles;
331
+ const thead = el("thead"); const htr = el("tr");
332
+ htr.appendChild(el("th", { class: "txt", text: "benchmark", style: "text-align:left" }));
333
+ profs.forEach(p => {
334
+ const extra = "Hard gates: " + (p.gates.join(" · ") || "—") + "<br>Ranks on: " + p.metric +
335
+ (p.corpus_profile ? "<br>Corpus: " + p.corpus_profile : "") +
336
+ (p.deployment_constraint ? "<br>Deploy: " + p.deployment_constraint : "");
337
+ htr.appendChild(bindTip(el("th", { html: p.name.replace(/_/g, " ") }), glossTip("profile", extra)));
338
+ });
339
+ thead.appendChild(htr); table.appendChild(thead);
340
+
341
+ const tb = el("tbody");
342
+ BENCH.forEach(b => {
343
+ const tr = el("tr");
344
+ tr.appendChild(bindTip(el("th", { text: benchShort(b) }), "<b>" + b + "</b>"));
345
+ const row = D.picks[b] || {};
346
+ profs.forEach(p => {
347
+ const cell = row[p.name];
348
+ const td = el("td");
349
+ if (cell && cell.pick) {
350
+ const c = CANDS[cell.pick];
351
+ const wrap = el("div");
352
+ const line = el("div", { style: "font-family:var(--mono);font-size:11.5px;margin-bottom:4px" });
353
+ if (c && c.is_neon) line.appendChild(el("span", { class: "badge neon", text: "NEON", style: "margin-right:5px" }));
354
+ line.appendChild(el("span", { text: cell.model + " · " + cell.retriever }));
355
+ wrap.appendChild(line);
356
+ const mv = cell.metric === "generation_acc" ? "gen " + num(cell.metric_val, "f2") : "H@1 " + num(cell.metric_val, "f2");
357
+ wrap.appendChild(bindTip(el("span", { class: "verdict " + (VCLASS[cell.verdict] || "v-none"), text: cell.verdict }),
358
+ glossTip("verdict") ));
359
+ wrap.appendChild(el("span", { class: "cell-sub", text: "MRR " + num(cell.mrr, "f3") + " · " + mv + " · p50 " + num(cell.p50, "f0") + "ms" }));
360
+ td.appendChild(wrap);
361
+ } else {
362
+ td.appendChild(el("span", { class: "verdict " + (VCLASS[(cell && cell.verdict)] || "v-dq"), text: (cell && cell.verdict) || "—" }));
363
+ }
364
+ tr.appendChild(td);
365
+ });
366
+ tb.appendChild(tr);
367
+ });
368
+ table.appendChild(tb);
369
+ }
370
+
371
+ // ===================================================================
372
+ // Models render
373
+ // ===================================================================
374
+ const modelPageCombo = Combo($("#modelPageCombo"), { options: candOptions(), value: DEFAULT_CAND, width: 240, onChange: v => { S.pageCand = v; render(); } });
375
+ S.pageCand = DEFAULT_CAND;
376
+ function renderModels() {
377
+ const host = $("#modelPage"); host.innerHTML = "";
378
+ const id = S.pageCand; const c = CANDS[id];
379
+ const rows = (aggByCand[id] || []).slice().sort((a, b) => benchShort(a.benchmark) < benchShort(b.benchmark) ? -1 : 1);
380
+ $("#modelPageBadge").innerHTML = c.is_neon ? "<span class='badge neon'>NEON.ai model</span>" : (c.is_retrieval_only ? "<span class='badge pareto'>retrieval baseline</span>" : "");
381
+
382
+ // aggregate stats
383
+ const mean = f => { const vs = rows.map(r => r[f]).filter(v => v != null); return vs.length ? vs.reduce((a, b) => a + b, 0) / vs.length : null; };
384
+ let paretoCount = 0; BENCH.forEach(b => { const bt = D.bench_tables[b]; if (bt && bt.pareto.includes(id)) paretoCount++; });
385
+ let dqCount = 0; BENCH.forEach(b => { const bt = D.bench_tables[b]; if (bt && bt.dq[id]) dqCount++; });
386
+
387
+ // left card
388
+ const left = el("div", { class: "card" });
389
+ left.appendChild(el("h3", { style: "margin:0 0 10px;font-size:14px", text: c.model }));
390
+ const kvs = [
391
+ ["pipeline", c.retriever + (c.reranker !== "none" ? " · " + c.reranker : "")],
392
+ ["model repo", c.model_repo || "—"],
393
+ ["serving", c.model_adapter || "—"],
394
+ ["benchmarks", String(rows.length)],
395
+ ["type", c.is_neon ? "Neon.ai model" : (c.is_retrieval_only ? "retrieval-only" : "external")],
396
+ ];
397
+ kvs.forEach(([k, v]) => left.appendChild(el("div", { class: "kv" }, [el("span", { class: "k", text: k }), el("span", { class: "v", text: v })])));
398
+ const stats = el("div", { class: "stat-row", style: "margin-top:14px" });
399
+ const stat = (n, l, tipTerm) => { const s = el("div", { class: "stat" }, [el("div", { class: "n", text: n }), bindTip(el("div", { class: "l", text: l }), glossTip(tipTerm))]); stats.appendChild(s); };
400
+ stat(num(mean("mrr"), "f3"), "mean MRR", "MRR");
401
+ const gm = mean("generation_acc"); stat(gm == null ? "—" : num(gm, "f2"), "mean gen", "gen");
402
+ stat(num(mean("p50_retrieval_latency_ms"), "f0"), "p50 ms", "P50");
403
+ stat(String(paretoCount), "Pareto ★", "Pareto");
404
+ stat(String(dqCount), "DQ count", "DQ");
405
+ left.appendChild(stats);
406
+ host.appendChild(left);
407
+
408
+ // right card: metrics across benchmarks
409
+ const right = el("div", { class: "card scroll" });
410
+ const cols = activeMetricCols(rows);
411
+ const table = el("table");
412
+ const thead = el("thead"); const htr = el("tr");
413
+ htr.appendChild(el("th", { class: "txt", text: "benchmark", style: "text-align:left;position:sticky;top:0;background:var(--bg-elev)" }));
414
+ cols.forEach(m => htr.appendChild(bindTip(el("th", { html: m.label, style: "position:sticky;top:0;background:var(--bg-elev)" }), glossTip(m.tip, m.higher ? "Higher is better." : "Lower is better."))));
415
+ thead.appendChild(htr); table.appendChild(thead);
416
+ const ranges = {}; cols.forEach(m => { const vs = rows.map(r => r[m.field]).filter(v => v != null); ranges[m.field] = vs.length ? { min: Math.min(...vs), max: Math.max(...vs) } : null; });
417
+ const tb = el("tbody");
418
+ rows.forEach(r => {
419
+ const bt = D.bench_tables[r.benchmark];
420
+ const tr = el("tr");
421
+ const star = bt && bt.pareto.includes(id) ? el("span", { class: "star", text: "★ " }) : null;
422
+ const dq = bt && bt.dq[id] ? el("span", { class: "badge dq", text: "DQ:" + bt.dq[id], style: "margin-left:6px" }) : null;
423
+ const tdL = el("td", { class: "txt" }); if (star) tdL.appendChild(star); tdL.appendChild(el("span", { text: benchShort(r.benchmark) })); if (dq) tdL.appendChild(dq);
424
+ tr.appendChild(tdL);
425
+ cols.forEach(m => { const v = r[m.field]; const td = el("td", { text: num(v, m.fmt) }); if (ranges[m.field]) td.style.cssText = heatStyle(v, ranges[m.field].min, ranges[m.field].max, m.higher); tr.appendChild(td); });
426
+ tb.appendChild(tr);
427
+ });
428
+ table.appendChild(tb); right.appendChild(table); host.appendChild(right);
429
+ }
430
+
431
+ // ===================================================================
432
+ // Metrics render
433
+ // ===================================================================
434
+ const metricSortSel = $("#metricSortSel");
435
+ function fillMetricSort() {
436
+ metricSortSel.innerHTML = "";
437
+ metricSortSel.appendChild(el("option", { value: "name", text: "Pipeline (A–Z)" }));
438
+ METRICS.forEach(m => metricSortSel.appendChild(el("option", { value: m.key, text: m.label })));
439
+ metricSortSel.value = S.metricSort;
440
+ }
441
+ fillMetricSort();
442
+ metricSortSel.addEventListener("change", () => { S.metricSort = metricSortSel.value; render(); });
443
+ $$("#metricModeSeg button").forEach(b => b.addEventListener("click", () => { S.metricMode = b.dataset.mode; $$("#metricModeSeg button").forEach(x => x.classList.toggle("active", x === b)); render(); }));
444
+ $("#metricNeonOnly").addEventListener("change", e => { S.metricNeonOnly = e.target.checked; render(); });
445
+ $("#metricHeat").addEventListener("change", e => { S.metricHeat = e.target.checked; render(); });
446
+
447
+ function renderMetrics() {
448
+ const table = $("#metricsTable"); table.innerHTML = "";
449
+ const stats = D.metric_stats; // key -> cid -> {wins,losses,mean,count}
450
+ let ids = CAND_IDS.slice();
451
+ if (S.metricNeonOnly) ids = ids.filter(id => CANDS[id].is_neon);
452
+
453
+ // sort rows
454
+ if (S.metricSort === "name") ids.sort((a, b) => pipeLabel(CANDS[a]) < pipeLabel(CANDS[b]) ? -1 : 1);
455
+ else {
456
+ const mdef = METRICS.find(m => m.key === S.metricSort);
457
+ const st = stats[S.metricSort] || {};
458
+ ids.sort((a, b) => {
459
+ const sa = st[a], sb = st[b];
460
+ const va = S.metricMode === "winloss" ? (sa ? sa.wins - sa.losses : -1e9) : (sa && sa.mean != null ? sa.mean : (mdef.higher ? -1e9 : 1e9));
461
+ const vb = S.metricMode === "winloss" ? (sb ? sb.wins - sb.losses : -1e9) : (sb && sb.mean != null ? sb.mean : (mdef.higher ? -1e9 : 1e9));
462
+ return S.metricMode === "winloss" ? (vb - va) : ((vb - va) * (mdef.higher ? 1 : -1));
463
+ });
464
+ }
465
+
466
+ const thead = el("thead"); const htr = el("tr");
467
+ htr.appendChild(el("th", { class: "txt", text: "pipeline", style: "text-align:left;position:sticky;left:0;z-index:6" }));
468
+ METRICS.forEach(m => htr.appendChild(bindTip(el("th", { html: m.label }), glossTip(m.tip, m.higher ? "Higher is better." : "Lower is better."))));
469
+ thead.appendChild(htr); table.appendChild(thead);
470
+
471
+ // ranges for mean heat per metric
472
+ const ranges = {};
473
+ METRICS.forEach(m => { const vs = ids.map(id => (stats[m.key][id] || {}).mean).filter(v => v != null); ranges[m.key] = vs.length ? { min: Math.min(...vs), max: Math.max(...vs) } : null; });
474
+
475
+ const tb = el("tbody");
476
+ ids.forEach(id => {
477
+ const c = CANDS[id];
478
+ const tr = el("tr", { class: c.is_neon ? "neon" : "" });
479
+ const th = el("th", { style: "font-family:var(--mono);font-size:11.5px" });
480
+ if (c.is_neon) th.appendChild(el("span", { class: "badge neon", text: "NEON", style: "margin-right:5px" }));
481
+ th.appendChild(el("span", { text: pipeLabel(c) }));
482
+ bindTip(th, "<b>" + pipeLabel(c) + "</b>" + (c.model_repo ? "<br>repo: " + c.model_repo : ""));
483
+ tr.appendChild(th);
484
+ METRICS.forEach(m => {
485
+ const s = (stats[m.key] || {})[id];
486
+ const td = el("td");
487
+ if (!s) { td.textContent = ""; }
488
+ else if (S.metricMode === "winloss") {
489
+ td.innerHTML = "<span style='color:var(--neon)'>" + s.wins + "</span> – <span style='color:#ff9b7d'>" + s.losses + "</span>";
490
+ const net = s.wins - s.losses;
491
+ if (S.metricHeat) td.style.cssText = heatStyle(net, -3, 3, true);
492
+ } else {
493
+ td.textContent = num(s.mean, m.fmt);
494
+ if (S.metricHeat && ranges[m.key]) td.style.cssText = heatStyle(s.mean, ranges[m.key].min, ranges[m.key].max, m.higher);
495
+ }
496
+ tr.appendChild(td);
497
+ });
498
+ tb.appendChild(tr);
499
+ });
500
+ table.appendChild(tb);
501
+ }
502
+
503
+ // ===================================================================
504
+ // Guide render
505
+ // ===================================================================
506
+ function renderGuide() {
507
+ const host = $("#guideGrid"); host.innerHTML = "";
508
+ // left: glossary
509
+ const left = el("div", { class: "guide-col" });
510
+ left.appendChild(el("h3", { text: "Metrics glossary" }));
511
+ const gt = el("table", { class: "gloss" });
512
+ const order = ["MRR", "H@1", "H@5", "nDCG", "P50", "P95", "build", "gen", "gen ms", "tok/s", "gen $", "$/q", "par", "shift", "n", "Pareto", "DQ", "verdict"];
513
+ order.forEach(k => { if (GLOSS[k]) gt.appendChild(el("tr", null, [el("td", { class: "term", text: k }), el("td", { class: "def", text: GLOSS[k] })])); });
514
+ left.appendChild(gt); host.appendChild(left);
515
+ // right: profiles + provenance
516
+ const right = el("div", { class: "guide-col" });
517
+ right.appendChild(el("h3", { text: "Client profiles (hard gates)" }));
518
+ D.profiles.forEach(p => {
519
+ const box = el("div", { class: "prof" });
520
+ box.appendChild(el("h4", { text: p.name.replace(/_/g, " ") + " — ranks on " + p.metric }));
521
+ const gates = el("div", { class: "gates" });
522
+ (p.gates.length ? p.gates : ["no hard gates"]).forEach(g => gates.appendChild(el("span", { class: "gate", text: g })));
523
+ box.appendChild(gates);
524
+ const sub = [];
525
+ if (p.corpus_profile) sub.push("corpus: " + p.corpus_profile);
526
+ if (p.deployment_constraint) sub.push("deploy: " + p.deployment_constraint);
527
+ if (sub.length) box.appendChild(el("div", { style: "color:var(--text-faint);font-size:11px;margin-top:6px", text: sub.join(" · ") }));
528
+ right.appendChild(box);
529
+ });
530
+ right.appendChild(el("h3", { text: "Provenance", style: "margin-top:10px" }));
531
+ const pv = el("div", { class: "prof" });
532
+ const sources = (prov.merged_from || []);
533
+ pv.appendChild(el("div", { style: "font-size:11.5px;color:var(--text-dim)", text: "Type: " + (prov.weekly ? "weekly roster sweep" : "sweep") }));
534
+ if (prov.merged_at) pv.appendChild(el("div", { style: "font-size:11.5px;color:var(--text-dim)", text: "Merged at: " + prov.merged_at }));
535
+ sources.forEach(s => pv.appendChild(el("div", { style: "font-family:var(--mono);font-size:10.5px;color:var(--text-faint);margin-top:3px", text: s })));
536
+ right.appendChild(pv);
537
+ host.appendChild(right);
538
+ }
539
+
540
+ // ===================================================================
541
+ // Theme
542
+ // ===================================================================
543
+ function applyTheme(t) {
544
+ document.documentElement.setAttribute("data-theme", t);
545
+ try { localStorage.setItem("nlb-theme", t); } catch (e) {}
546
+ paintLegendSwatches(); render();
547
+ }
548
+ $("#themeBtn").addEventListener("click", () => applyTheme(document.documentElement.getAttribute("data-theme") === "light" ? "dark" : "light"));
549
+ let savedTheme = "dark"; try { savedTheme = localStorage.getItem("nlb-theme") || "dark"; } catch (e) {}
550
+ const qTheme = new URLSearchParams(location.search).get("theme");
551
+ if (qTheme === "light" || qTheme === "dark") savedTheme = qTheme;
552
+ document.documentElement.setAttribute("data-theme", savedTheme);
553
+
554
+ // ===================================================================
555
+ // Master render
556
+ // ===================================================================
557
+ function render() {
558
+ if (S.tab === "compare") renderCompare();
559
+ else if (S.tab === "picks") renderPicks();
560
+ else if (S.tab === "models") renderModels();
561
+ else if (S.tab === "metrics") renderMetrics();
562
+ else if (S.tab === "guide") renderGuide();
563
+ }
564
+
565
+ paintLegendSwatches();
566
+ const initTab = (location.hash || "").replace("#", "");
567
+ if (["compare", "picks", "models", "metrics", "guide"].includes(initTab)) setTab(initTab);
568
+ else render();
569
+ })();
data.js ADDED
The diff for this file is too large to render. See raw diff
 
data.json ADDED
The diff for this file is too large to render. See raw diff
 
index.html ADDED
@@ -0,0 +1,313 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!doctype html>
2
+ <html lang="en">
3
+ <head>
4
+ <meta charset="utf-8" />
5
+ <meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover" />
6
+ <title>Neon Leaderboard</title>
7
+ <style>
8
+ :root {
9
+ --bg: #0f1419;
10
+ --bg-elev: #161c24;
11
+ --bg-elev2: #1c242e;
12
+ --panel: #131a22;
13
+ --border: #283441;
14
+ --border-soft: #1f2a35;
15
+ --text: #d7dee7;
16
+ --text-dim: #93a1b0;
17
+ --text-faint: #64727f;
18
+ --accent: #3fb6a8; /* neon teal */
19
+ --accent-dim: #2c7a72;
20
+ --neon: #5ad1c2;
21
+ --good-h: 168; /* teal-green */
22
+ --bad-h: 28; /* warm amber */
23
+ --chip-bg: #1d2731;
24
+ --shadow: 0 4px 18px rgba(0,0,0,.35);
25
+ --radius: 9px;
26
+ --row-h: 26px;
27
+ --font: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
28
+ --mono: ui-monospace, "SF Mono", "Cascadia Code", "Roboto Mono", Menlo, Consolas, monospace;
29
+ }
30
+ html[data-theme="light"] {
31
+ --bg: #f3f5f8;
32
+ --bg-elev: #ffffff;
33
+ --bg-elev2: #eef1f5;
34
+ --panel: #ffffff;
35
+ --border: #d4dbe3;
36
+ --border-soft: #e4e9ef;
37
+ --text: #1f2933;
38
+ --text-dim: #56636f;
39
+ --text-faint: #8a97a3;
40
+ --accent: #1f8f82;
41
+ --accent-dim: #5fb5ab;
42
+ --neon: #178a7c;
43
+ --chip-bg: #eef2f6;
44
+ --shadow: 0 4px 18px rgba(40,55,75,.12);
45
+ }
46
+
47
+ * { box-sizing: border-box; }
48
+ html, body { height: 100%; margin: 0; }
49
+ body {
50
+ background: var(--bg);
51
+ color: var(--text);
52
+ font-family: var(--font);
53
+ font-size: 13px;
54
+ line-height: 1.35;
55
+ -webkit-font-smoothing: antialiased;
56
+ overflow: hidden; /* single viewport, no page scroll */
57
+ }
58
+ button { font-family: inherit; color: inherit; cursor: pointer; }
59
+ select, input { font-family: inherit; font-size: 12.5px; }
60
+
61
+ .app { display: flex; flex-direction: column; height: 100vh; height: 100dvh; }
62
+
63
+ /* ---------- Header ---------- */
64
+ header.top {
65
+ display: flex; align-items: center; gap: 18px;
66
+ padding: 0 16px; height: 50px; flex: 0 0 auto;
67
+ background: var(--bg-elev); border-bottom: 1px solid var(--border);
68
+ }
69
+ .brand { display: flex; align-items: baseline; gap: 9px; white-space: nowrap; }
70
+ .brand .dot { width: 9px; height: 9px; border-radius: 50%; background: var(--accent);
71
+ box-shadow: 0 0 9px var(--accent); align-self: center; }
72
+ .brand h1 { font-size: 15px; font-weight: 650; margin: 0; letter-spacing: .2px; }
73
+ .brand .meta { color: var(--text-faint); font-size: 11px; }
74
+
75
+ nav.tabs { display: flex; gap: 2px; margin-left: 4px; }
76
+ nav.tabs button {
77
+ background: transparent; border: 0; padding: 8px 14px; border-radius: 7px;
78
+ color: var(--text-dim); font-size: 13px; font-weight: 550; letter-spacing: .2px;
79
+ transition: background .12s, color .12s;
80
+ }
81
+ nav.tabs button:hover { background: var(--bg-elev2); color: var(--text); }
82
+ nav.tabs button.active { background: var(--accent); color: #06231f; }
83
+ html[data-theme="light"] nav.tabs button.active { color: #fff; }
84
+
85
+ .spacer { flex: 1 1 auto; }
86
+ .icon-btn {
87
+ background: var(--bg-elev2); border: 1px solid var(--border); border-radius: 7px;
88
+ width: 32px; height: 32px; display: grid; place-items: center; color: var(--text-dim);
89
+ font-size: 15px;
90
+ }
91
+ .icon-btn:hover { color: var(--text); border-color: var(--accent-dim); }
92
+
93
+ /* ---------- Controls bar ---------- */
94
+ .controls {
95
+ display: flex; align-items: center; gap: 14px; flex-wrap: wrap;
96
+ padding: 9px 16px; flex: 0 0 auto;
97
+ background: var(--panel); border-bottom: 1px solid var(--border-soft);
98
+ min-height: 46px;
99
+ }
100
+ .ctl { display: flex; align-items: center; gap: 7px; }
101
+ .ctl label { color: var(--text-faint); font-size: 11px; text-transform: uppercase; letter-spacing: .6px; }
102
+ .field {
103
+ background: var(--bg-elev2); border: 1px solid var(--border); color: var(--text);
104
+ border-radius: 7px; padding: 6px 9px; min-width: 90px; outline: none;
105
+ }
106
+ .field:focus { border-color: var(--accent-dim); }
107
+ select.field { padding-right: 26px; appearance: none;
108
+ background-image: linear-gradient(45deg, transparent 50%, var(--text-faint) 50%),
109
+ linear-gradient(135deg, var(--text-faint) 50%, transparent 50%);
110
+ background-position: calc(100% - 14px) 50%, calc(100% - 9px) 50%;
111
+ background-size: 5px 5px, 5px 5px; background-repeat: no-repeat; }
112
+ .seg { display: inline-flex; background: var(--bg-elev2); border: 1px solid var(--border);
113
+ border-radius: 7px; overflow: hidden; }
114
+ .seg button { background: transparent; border: 0; padding: 6px 11px; color: var(--text-dim); font-size: 12px; }
115
+ .seg button.active { background: var(--accent); color: #06231f; }
116
+ html[data-theme="light"] .seg button.active { color: #fff; }
117
+ .toggle { display: inline-flex; align-items: center; gap: 6px; color: var(--text-dim); font-size: 12px; user-select: none; }
118
+ .toggle input { accent-color: var(--accent); }
119
+
120
+ /* combobox (searchable dropdown) */
121
+ .combo { position: relative; }
122
+ .combo .menu {
123
+ position: absolute; z-index: 40; top: calc(100% + 4px); left: 0; min-width: 240px; max-width: 360px;
124
+ max-height: 320px; overflow: auto; background: var(--bg-elev); border: 1px solid var(--border);
125
+ border-radius: 8px; box-shadow: var(--shadow); padding: 5px; display: none;
126
+ }
127
+ .combo.open .menu { display: block; }
128
+ .combo .opt { padding: 6px 9px; border-radius: 6px; cursor: pointer; display: flex; align-items: center; gap: 8px; white-space: nowrap; }
129
+ .combo .opt:hover, .combo .opt.kbd { background: var(--bg-elev2); }
130
+ .combo .opt.sel { color: var(--accent); }
131
+ .combo .opt small { color: var(--text-faint); margin-left: auto; }
132
+
133
+ /* ---------- Main ---------- */
134
+ main { flex: 1 1 auto; min-height: 0; display: flex; position: relative; }
135
+ .view { display: none; flex: 1; min-height: 0; flex-direction: column; padding: 12px 16px 8px; }
136
+ .view.active { display: flex; }
137
+
138
+ .table-wrap { flex: 1 1 auto; min-height: 0; overflow: auto; border: 1px solid var(--border-soft);
139
+ border-radius: var(--radius); background: var(--panel); }
140
+ table { border-collapse: separate; border-spacing: 0; width: 100%; }
141
+ thead th {
142
+ position: sticky; top: 0; z-index: 5; background: var(--bg-elev);
143
+ text-align: right; padding: 7px 9px; font-size: 11px; font-weight: 600; color: var(--text-dim);
144
+ border-bottom: 1px solid var(--border); white-space: nowrap; cursor: help; user-select: none;
145
+ }
146
+ thead th.lbl, thead th.sortable { cursor: pointer; }
147
+ thead th.txt { text-align: left; }
148
+ thead th .arrow { color: var(--accent); font-size: 9px; margin-left: 3px; }
149
+ tbody td { padding: 4px 9px; text-align: right; font-variant-numeric: tabular-nums;
150
+ border-bottom: 1px solid var(--border-soft); white-space: nowrap; height: var(--row-h); }
151
+ tbody td.txt { text-align: left; font-family: var(--mono); font-size: 12px; }
152
+ tbody tr:hover td { background: rgba(127,150,170,.07); }
153
+ tbody tr.neon td.txt .pipe { font-weight: 600; }
154
+
155
+ .badge { display: inline-block; padding: 1px 6px; border-radius: 999px; font-size: 10px; font-weight: 650;
156
+ letter-spacing: .3px; vertical-align: middle; }
157
+ .badge.neon { background: rgba(90,209,194,.16); color: var(--neon); border: 1px solid rgba(90,209,194,.3); }
158
+ .badge.pareto { background: rgba(120,170,255,.14); color: #8fb6ff; }
159
+ .badge.dq { background: rgba(255,120,90,.13); color: #ff9b7d; border: 1px solid rgba(255,120,90,.25); }
160
+ html[data-theme="light"] .badge.pareto { color: #2f5fc0; }
161
+ .star { color: #8fb6ff; font-size: 11px; }
162
+ .pipe small { color: var(--text-faint); }
163
+
164
+ .verdict { display: inline-block; padding: 2px 8px; border-radius: 6px; font-size: 11px; font-weight: 600; }
165
+ .v-clear { background: rgba(63,182,168,.18); color: var(--neon); }
166
+ .v-inconclusive { background: rgba(190,170,90,.16); color: #d9c178; }
167
+ .v-dq { background: rgba(255,120,90,.13); color: #ff9b7d; }
168
+ .v-none { background: var(--chip-bg); color: var(--text-faint); }
169
+
170
+ /* Picks grid + Metrics grid share matrix styles */
171
+ .matrix-wrap { flex: 1 1 auto; min-height: 0; overflow: auto; border: 1px solid var(--border-soft);
172
+ border-radius: var(--radius); background: var(--panel); }
173
+ .matrix { width: 100%; border-collapse: separate; border-spacing: 0; }
174
+ .matrix th, .matrix td { padding: 8px 10px; border-bottom: 1px solid var(--border-soft); border-right: 1px solid var(--border-soft); }
175
+ .matrix thead th { position: sticky; top: 0; background: var(--bg-elev); z-index: 5; cursor: help; font-size: 11.5px; }
176
+ .matrix tbody th { position: sticky; left: 0; background: var(--bg-elev); z-index: 4; text-align: left; cursor: help; }
177
+ .matrix td { text-align: center; font-variant-numeric: tabular-nums; }
178
+ .matrix .cell-sub { display: block; font-size: 10px; color: var(--text-faint); margin-top: 2px; }
179
+
180
+ /* Models tab */
181
+ .model-page { flex: 1 1 auto; min-height: 0; display: grid; grid-template-columns: 280px 1fr; gap: 14px; }
182
+ .card { background: var(--panel); border: 1px solid var(--border-soft); border-radius: var(--radius); padding: 16px; }
183
+ .card.scroll { overflow: auto; }
184
+ .kv { display: flex; justify-content: space-between; gap: 12px; padding: 6px 0; border-bottom: 1px solid var(--border-soft); font-size: 12.5px; }
185
+ .kv:last-child { border-bottom: 0; }
186
+ .kv .k { color: var(--text-faint); }
187
+ .kv .v { color: var(--text); text-align: right; font-family: var(--mono); }
188
+ .stat-row { display: flex; gap: 10px; flex-wrap: wrap; margin-bottom: 12px; }
189
+ .stat { background: var(--bg-elev2); border: 1px solid var(--border-soft); border-radius: 8px; padding: 9px 13px; min-width: 92px; }
190
+ .stat .n { font-size: 19px; font-weight: 650; font-variant-numeric: tabular-nums; }
191
+ .stat .l { font-size: 10.5px; color: var(--text-faint); text-transform: uppercase; letter-spacing: .5px; cursor: help; }
192
+
193
+ /* Guide tab */
194
+ .guide-grid { flex: 1 1 auto; min-height: 0; display: grid; grid-template-columns: 1fr 1fr; gap: 14px; }
195
+ .guide-col { overflow: auto; }
196
+ .guide-col h3 { margin: 0 0 8px; font-size: 13px; color: var(--accent); }
197
+ .gloss { width: 100%; border-collapse: collapse; }
198
+ .gloss td { padding: 5px 8px; border-bottom: 1px solid var(--border-soft); vertical-align: top; font-size: 12px; }
199
+ .gloss td.term { color: var(--text); font-weight: 600; white-space: nowrap; width: 78px; font-family: var(--mono); }
200
+ .gloss td.def { color: var(--text-dim); }
201
+ .prof { background: var(--bg-elev2); border: 1px solid var(--border-soft); border-radius: 8px; padding: 10px 12px; margin-bottom: 8px; }
202
+ .prof h4 { margin: 0 0 4px; font-size: 12.5px; }
203
+ .prof .gates { display: flex; gap: 6px; flex-wrap: wrap; }
204
+ .gate { background: var(--chip-bg); border-radius: 6px; padding: 2px 8px; font-size: 11px; color: var(--text-dim); font-family: var(--mono); }
205
+
206
+ footer.bottom { flex: 0 0 auto; display: flex; align-items: center; gap: 16px; padding: 4px 16px;
207
+ height: 26px; background: var(--bg-elev); border-top: 1px solid var(--border); font-size: 10.5px; color: var(--text-faint); }
208
+ footer .lg { display: inline-flex; align-items: center; gap: 5px; }
209
+ footer .sw { width: 11px; height: 11px; border-radius: 3px; display: inline-block; }
210
+
211
+ /* tooltip */
212
+ #tip { position: fixed; z-index: 200; max-width: 320px; background: #0a0e12; color: #e7edf3;
213
+ border: 1px solid var(--border); border-radius: 8px; padding: 9px 11px; font-size: 12px; line-height: 1.45;
214
+ box-shadow: var(--shadow); pointer-events: none; opacity: 0; transition: opacity .1s; }
215
+ html[data-theme="light"] #tip { background: #1f2933; color: #f3f5f8; border-color: #1f2933; }
216
+ #tip b { color: var(--neon); }
217
+
218
+ .empty { color: var(--text-faint); padding: 30px; text-align: center; font-size: 13px; }
219
+ ::-webkit-scrollbar { width: 10px; height: 10px; }
220
+ ::-webkit-scrollbar-thumb { background: var(--border); border-radius: 6px; }
221
+ ::-webkit-scrollbar-track { background: transparent; }
222
+ </style>
223
+ </head>
224
+ <body>
225
+ <div class="app">
226
+ <header class="top">
227
+ <div class="brand"><span class="dot"></span><h1>Neon Leaderboard</h1><span class="meta" id="headerMeta"></span></div>
228
+ <nav class="tabs" id="tabs">
229
+ <button data-tab="compare" class="active">Compare</button>
230
+ <button data-tab="picks">Picks</button>
231
+ <button data-tab="models">Models</button>
232
+ <button data-tab="metrics">Metrics</button>
233
+ <button data-tab="guide">Guide</button>
234
+ </nav>
235
+ <span class="spacer"></span>
236
+ <button class="icon-btn" id="themeBtn" title="Toggle light / dark">◐</button>
237
+ </header>
238
+
239
+ <!-- COMPARE -->
240
+ <div class="controls" id="ctl-compare">
241
+ <div class="ctl"><label>View</label>
242
+ <div class="seg" id="cmpModeSeg">
243
+ <button data-mode="byBench" class="active">By benchmark</button>
244
+ <button data-mode="byModel">By model</button>
245
+ </div>
246
+ </div>
247
+ <div class="ctl" id="benchPickWrap"><label>Benchmark</label><div class="combo" id="benchCombo"></div></div>
248
+ <div class="ctl" id="modelPickWrap" style="display:none"><label>Model</label><div class="combo" id="modelCombo"></div></div>
249
+ <div class="ctl"><label>Filter model</label><div class="combo" id="filterCombo"></div></div>
250
+ <div class="ctl"><label>Sort</label><select class="field" id="sortSel"></select>
251
+ <button class="icon-btn" id="sortDir" title="Sort direction" style="width:30px;height:30px">▾</button>
252
+ </div>
253
+ <span class="spacer"></span>
254
+ <label class="toggle"><input type="checkbox" id="neonOnly" /> Neon only</label>
255
+ <label class="toggle"><input type="checkbox" id="showDQ" checked /> Show DQ</label>
256
+ <label class="toggle"><input type="checkbox" id="heat" checked /> Color</label>
257
+ </div>
258
+
259
+ <!-- PICKS -->
260
+ <div class="controls" id="ctl-picks" style="display:none">
261
+ <div class="ctl"><label>Router recommendations</label></div>
262
+ <span class="meta" style="color:var(--text-faint);font-size:11.5px">Pick = model · retriever · reranker per client profile. Hover a profile column for its hard gates.</span>
263
+ </div>
264
+
265
+ <!-- MODELS -->
266
+ <div class="controls" id="ctl-models" style="display:none">
267
+ <div class="ctl"><label>Model</label><div class="combo" id="modelPageCombo"></div></div>
268
+ <span class="meta" id="modelPageBadge"></span>
269
+ </div>
270
+
271
+ <!-- METRICS -->
272
+ <div class="controls" id="ctl-metrics" style="display:none">
273
+ <div class="ctl"><label>Show</label>
274
+ <div class="seg" id="metricModeSeg">
275
+ <button data-mode="scores" class="active">Mean scores</button>
276
+ <button data-mode="winloss">Wins / losses</button>
277
+ </div>
278
+ </div>
279
+ <div class="ctl"><label>Sort by</label><select class="field" id="metricSortSel"></select></div>
280
+ <span class="spacer"></span>
281
+ <label class="toggle"><input type="checkbox" id="metricNeonOnly" /> Neon only</label>
282
+ <label class="toggle"><input type="checkbox" id="metricHeat" checked /> Color</label>
283
+ </div>
284
+
285
+ <!-- GUIDE -->
286
+ <div class="controls" id="ctl-guide" style="display:none">
287
+ <div class="ctl"><label>Reference</label></div>
288
+ <span class="meta" style="color:var(--text-faint);font-size:11.5px">Every column, profile gate, and provenance source behind the numbers.</span>
289
+ </div>
290
+
291
+ <main>
292
+ <section class="view active" id="view-compare"><div class="table-wrap"><table id="cmpTable"></table></div></section>
293
+ <section class="view" id="view-picks"><div class="matrix-wrap"><table class="matrix" id="picksTable"></table></div></section>
294
+ <section class="view" id="view-models"><div class="model-page" id="modelPage"></div></section>
295
+ <section class="view" id="view-metrics"><div class="matrix-wrap"><table class="matrix" id="metricsTable"></table></div></section>
296
+ <section class="view" id="view-guide"><div class="guide-grid" id="guideGrid"></div></section>
297
+ </main>
298
+
299
+ <footer class="bottom">
300
+ <span class="lg"><span class="sw" id="lgGood"></span> higher rank</span>
301
+ <span class="lg"><span class="sw" id="lgBad"></span> lower rank</span>
302
+ <span class="lg"><span class="badge neon" style="padding:0 6px">NEON</span> Neon.ai model</span>
303
+ <span class="lg"><span class="star">★</span> Pareto-optimal</span>
304
+ <span class="lg"><span class="badge dq" style="padding:0 6px">DQ</span> disqualified by a profile gate</span>
305
+ <span class="spacer"></span>
306
+ <span id="provFoot"></span>
307
+ </footer>
308
+ </div>
309
+ <div id="tip"></div>
310
+ <script src="data.js"></script>
311
+ <script src="app.js"></script>
312
+ </body>
313
+ </html>
scripts/build_data.py ADDED
@@ -0,0 +1,493 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Build the static dashboard data bundle (``data.js``) for the Neon leaderboard.
3
+
4
+ This reads the leaderboard's own source of truth -- ``run.json`` plus the
5
+ profile YAMLs -- and recomputes picks, the Pareto front, disqualifications,
6
+ verdicts, and per-metric win/loss tallies using logic faithful to
7
+ ``router/pareto.py`` and ``router/reports/leaderboard.py`` in the neon-router
8
+ repo. The output is a single ``data.js`` file that assigns
9
+ ``window.LEADERBOARD_DATA`` so the dashboard loads with no server / no CORS.
10
+
11
+ Usage:
12
+ python scripts/build_data.py \
13
+ --run-json /path/to/reports/dashboard/raw/run.json \
14
+ --profiles-dir /path/to/configs/profiles \
15
+ --out ../data.js
16
+
17
+ Defaults point at a sibling ``neon-router`` checkout.
18
+ """
19
+ from __future__ import annotations
20
+
21
+ import argparse
22
+ import json
23
+ import math
24
+ import os
25
+ from dataclasses import dataclass, field
26
+ from datetime import datetime, timezone
27
+ from pathlib import Path
28
+ from typing import Dict, List, Optional, Sequence, Tuple
29
+
30
+ try:
31
+ import yaml
32
+ except ImportError: # pragma: no cover
33
+ raise SystemExit("PyYAML is required: pip install pyyaml")
34
+
35
+
36
+ # ---------------------------------------------------------------------------
37
+ # Profile model (mirror of router.pareto.ProfilePreference, subset we need)
38
+ # ---------------------------------------------------------------------------
39
+ @dataclass
40
+ class Profile:
41
+ name: str
42
+ latency_budget_ms: Optional[float] = None
43
+ generation_latency_budget_ms: Optional[float] = None
44
+ cost_ceiling_per_query_usd: Optional[float] = None
45
+ accuracy_floor_mrr: Optional[float] = None
46
+ accuracy_floor_generation_acc: Optional[float] = None
47
+ corpus_profile: Optional[str] = None
48
+ query_mix: Dict[str, float] = field(default_factory=dict)
49
+ deployment_constraint: Optional[str] = None
50
+ quality_weight: float = 0.7
51
+ latency_penalty_per_ms: float = 0.0001
52
+ generation_latency_penalty_per_ms: float = 0.0
53
+ cost_penalty_per_token: float = 0.0
54
+ metric: str = "mrr"
55
+
56
+
57
+ # Fields the dashboard reads off each aggregate row.
58
+ NUMERIC_FIELDS = [
59
+ "mrr", "hit_at_1", "hit_at_5", "ndcg_at_10",
60
+ "p50_retrieval_latency_ms", "p95_retrieval_latency_ms",
61
+ "avg_retrieval_latency_ms", "index_build_seconds",
62
+ "est_cost_per_query_usd", "generation_acc",
63
+ "p50_generation_latency_ms", "avg_generation_latency_ms",
64
+ "generation_tokens_per_sec", "est_generation_cost_per_query_usd",
65
+ "paraphrase_drop", "shift_drop", "n_queries",
66
+ ]
67
+
68
+ DQ_LATENCY, DQ_COST, DQ_ACCURACY = "latency", "cost", "accuracy"
69
+
70
+ # Models served/fine-tuned by Neon.ai (vs. retrieval-only baselines and any
71
+ # external references). Edit this set to control the "Neon" badge in the UI.
72
+ # Match on the candidate's `model` name (the first ``::`` segment).
73
+ NEON_MODELS = {
74
+ "devstral-123b",
75
+ "ministral-14b-reasoning",
76
+ "magistral-small",
77
+ "devstral-small-24b",
78
+ "mistral-small-3.2-24b",
79
+ "granite-4-30b",
80
+ }
81
+
82
+
83
+ def _get(agg: dict, key: str):
84
+ return agg.get(key)
85
+
86
+
87
+ def _acc_axis(agg: dict, metric: str) -> float:
88
+ return float(agg.get(metric) or 0.0)
89
+
90
+
91
+ def _cost_per_query(agg: dict) -> float:
92
+ return float(agg.get("est_cost_per_query_usd") or 0.0)
93
+
94
+
95
+ # ---------------------------------------------------------------------------
96
+ # Faithful port of router.pareto
97
+ # ---------------------------------------------------------------------------
98
+ def apply_hard_constraints(aggs: List[dict], p: Profile) -> Tuple[List[dict], List[Tuple[dict, str]]]:
99
+ survivors: List[dict] = []
100
+ dq: List[Tuple[dict, str]] = []
101
+ for agg in aggs:
102
+ reason: Optional[str] = None
103
+ avg_lat = agg.get("avg_retrieval_latency_ms")
104
+ avg_gen_lat = agg.get("avg_generation_latency_ms")
105
+ if p.latency_budget_ms is not None and avg_lat is not None and avg_lat > p.latency_budget_ms:
106
+ reason = DQ_LATENCY
107
+ elif (
108
+ p.generation_latency_budget_ms is not None
109
+ and avg_gen_lat is not None
110
+ and avg_gen_lat > p.generation_latency_budget_ms
111
+ ):
112
+ reason = DQ_LATENCY
113
+ elif (
114
+ p.cost_ceiling_per_query_usd is not None
115
+ and p.cost_ceiling_per_query_usd > 0.0
116
+ and _cost_per_query(agg) > p.cost_ceiling_per_query_usd
117
+ ):
118
+ reason = DQ_COST
119
+ elif (
120
+ p.accuracy_floor_generation_acc is not None
121
+ and agg.get("generation_acc") is not None
122
+ and agg.get("generation_acc") < p.accuracy_floor_generation_acc
123
+ ):
124
+ reason = DQ_ACCURACY
125
+ elif (
126
+ p.accuracy_floor_mrr is not None
127
+ and p.metric == "mrr"
128
+ and _acc_axis(agg, p.metric) < p.accuracy_floor_mrr
129
+ ):
130
+ reason = DQ_ACCURACY
131
+ if reason is None:
132
+ survivors.append(agg)
133
+ else:
134
+ dq.append((agg, reason))
135
+ return survivors, dq
136
+
137
+
138
+ def pareto_front(aggs: List[dict], metric: str = "mrr") -> List[dict]:
139
+ front: List[dict] = []
140
+ for a in aggs:
141
+ dominated = False
142
+ for b in aggs:
143
+ if a is b:
144
+ continue
145
+ a_lat = a.get("avg_retrieval_latency_ms") or 0.0
146
+ b_lat = b.get("avg_retrieval_latency_ms") or 0.0
147
+ a_build = a.get("index_build_seconds") or 0.0
148
+ b_build = b.get("index_build_seconds") or 0.0
149
+ ge = (
150
+ _acc_axis(b, metric) >= _acc_axis(a, metric)
151
+ and b_lat <= a_lat
152
+ and b_build <= a_build
153
+ )
154
+ gt = (
155
+ _acc_axis(b, metric) > _acc_axis(a, metric)
156
+ or b_lat < a_lat
157
+ or b_build < a_build
158
+ )
159
+ if ge and gt:
160
+ dominated = True
161
+ break
162
+ if not dominated:
163
+ front.append(a)
164
+ return front
165
+
166
+
167
+ def select_winner(aggs: List[dict], p: Profile) -> Optional[dict]:
168
+ survivors, _ = apply_hard_constraints(aggs, p)
169
+ front = pareto_front(survivors, metric=p.metric)
170
+ if not front:
171
+ return None
172
+
173
+ def utility(agg: dict) -> float:
174
+ accuracy = _acc_axis(agg, p.metric)
175
+ lat = agg.get("avg_retrieval_latency_ms") or 0.0
176
+ gen_lat = agg.get("avg_generation_latency_ms") or 0.0
177
+ return (
178
+ p.quality_weight * accuracy
179
+ - p.latency_penalty_per_ms * lat
180
+ - p.generation_latency_penalty_per_ms * gen_lat
181
+ )
182
+
183
+ return sorted(front, key=utility, reverse=True)[0]
184
+
185
+
186
+ def _wilson_ci(n_succ: float, n_total: int, z: float = 1.96) -> Tuple[float, float]:
187
+ if n_total == 0:
188
+ return (0.0, 0.0)
189
+ pp = n_succ / n_total
190
+ denom = 1.0 + z * z / n_total
191
+ centre = (pp + z * z / (2.0 * n_total)) / denom
192
+ margin = z * math.sqrt(pp * (1.0 - pp) / n_total + z * z / (4.0 * n_total * n_total)) / denom
193
+ return (max(0.0, centre - margin), min(1.0, centre + margin))
194
+
195
+
196
+ def value_add_verdict(aggs: List[dict], p: Profile) -> str:
197
+ survivors, _ = apply_hard_constraints(aggs, p)
198
+ if len(survivors) < 2:
199
+ return "no signal"
200
+
201
+ def key(a: dict) -> float:
202
+ if p.metric == "generation_acc" and a.get("generation_acc") is not None:
203
+ return a["generation_acc"]
204
+ return a.get("mrr") or 0.0
205
+
206
+ ranked = sorted(survivors, key=key, reverse=True)
207
+ winner, runner = ranked[0], ranked[1]
208
+ n = int(winner.get("n_queries") or 0)
209
+ w_lo, _ = _wilson_ci(key(winner) * n, n)
210
+ _, r_hi = _wilson_ci(key(runner) * n, n)
211
+ return "winner clear" if w_lo > r_hi else "inconclusive"
212
+
213
+
214
+ # ---------------------------------------------------------------------------
215
+ # Loaders
216
+ # ---------------------------------------------------------------------------
217
+ def load_profiles(profiles_dir: Path) -> List[Profile]:
218
+ valid = set(Profile.__dataclass_fields__)
219
+ profiles: List[Profile] = []
220
+ for path in sorted(profiles_dir.glob("*.yaml")):
221
+ raw = yaml.safe_load(path.read_text(encoding="utf-8")) or {}
222
+ clean = {k: v for k, v in raw.items() if k in valid}
223
+ profiles.append(Profile(**clean))
224
+ return profiles
225
+
226
+
227
+ def split_candidate(cid: str) -> Tuple[str, str, str]:
228
+ parts = cid.split("::")
229
+ return (parts[0], parts[1], parts[2]) if len(parts) == 3 else (cid, "", "")
230
+
231
+
232
+ # Mirror of leaderboard.py _GLOSSARY (kept in sync with the source report).
233
+ GLOSSARY = {
234
+ "model": "The generation model in the pipeline. 'retrieval-only' means no LLM generation step was run for this row.",
235
+ "retriever": "Retrieval strategy used to fetch candidate documents (e.g. hybrid = BM25 + dense embeddings; bm25-only = lexical; splade-only = learned sparse).",
236
+ "rerank": "Cross-encoder reranker applied after retrieval ('none' = no reranking; minilm / mxbai are reranker models).",
237
+ "MRR": "Mean Reciprocal Rank \u2014 average of 1/rank for the first relevant doc retrieved per query. Range 0\u20131; higher is better.",
238
+ "H@1": "Hit@1 \u2014 fraction of queries where the top-1 retrieved doc is relevant. Higher is better.",
239
+ "H@5": "Hit@5 \u2014 fraction of queries where any of the top-5 retrieved docs is relevant. Higher is better.",
240
+ "nDCG": "Normalized Discounted Cumulative Gain @10 \u2014 position-weighted relevance of the top-10 results. Higher is better.",
241
+ "P50": "Median (50th percentile) per-query retrieval latency in milliseconds. Lower is better.",
242
+ "P95": "95th percentile retrieval latency \u2014 the tail (worst-case) cost per query, in milliseconds. Lower is better.",
243
+ "build": "Index build time in seconds for this benchmark corpus. Lower is better.",
244
+ "gen": "Generation score (task-dependent): for multiple-choice benchmarks, the fraction of questions answered correctly given retrieved context; for the language-following benchmark, the fraction of responses written in the user's language. Higher is better.",
245
+ "gen ms": "Median per-query generation latency (time-to-first-token + generation) in milliseconds. Lower is better.",
246
+ "tok/s": "LLM output tokens generated per second on this benchmark. Higher is better.",
247
+ "gen $": "Estimated generation cost per query in USD (token usage \u00d7 model price). Lower is better.",
248
+ "$/q": "Total estimated cost per query in USD (retrieval + generation). Lower is better.",
249
+ "par": "Paraphrase robustness \u2014 MRR on paraphrased queries minus MRR on originals. Near zero is expected; large negative means fragile exact-match retrieval.",
250
+ "shift": "Held-out (domain-shifted) slice MRR \u2014 measures out-of-distribution generalization on an unseen query subset.",
251
+ "n": "Number of queries evaluated for this row.",
252
+ "verdict": "Whether the winning candidate is statistically distinct from the runner-up. 'winner clear' = winner's Wilson CI lower bound exceeds runner-up's upper bound; 'inconclusive' = confidence intervals overlap (within noise); 'no signal' = too few valid candidates; 'all disqualified' = every candidate failed a hard gate.",
253
+ "Pareto": "Pareto front \u2014 a pipeline not dominated by any other on (accuracy, latency, build-cost) simultaneously. Marked with \u2605.",
254
+ "DQ": "Disqualified \u2014 violated one of the profile's hard gates: DQ:latency (p50 over budget), DQ:cost (cost per query over ceiling), or DQ:accuracy (MRR or generation score below the floor).",
255
+ "pick": "The router's single recommended pipeline (model / retriever / reranker) for this client profile on this benchmark.",
256
+ "profile": "A client requirements file (not a model) declaring latency / cost / accuracy hard gates plus utility weights used to rank surviving candidates.",
257
+ }
258
+
259
+ # Per-metric definitions used by the Compare and Metrics tabs.
260
+ # key -> (label, field, higher_is_better, fmt, tooltip-key)
261
+ METRICS = [
262
+ ("mrr", "MRR", "mrr", True, "f3", "MRR"),
263
+ ("h1", "H@1", "hit_at_1", True, "f2", "H@1"),
264
+ ("h5", "H@5", "hit_at_5", True, "f2", "H@5"),
265
+ ("ndcg", "nDCG", "ndcg_at_10", True, "f3", "nDCG"),
266
+ ("p50", "P50 ms", "p50_retrieval_latency_ms", False, "f0", "P50"),
267
+ ("p95", "P95 ms", "p95_retrieval_latency_ms", False, "f0", "P95"),
268
+ ("build", "Build s", "index_build_seconds", False, "f1", "build"),
269
+ ("gen", "gen", "generation_acc", True, "f3", "gen"),
270
+ ("gen_ms", "gen ms", "p50_generation_latency_ms", False, "f0", "gen ms"),
271
+ ("tok_s", "tok/s", "generation_tokens_per_sec", True, "f0", "tok/s"),
272
+ ("gen_cost", "gen $", "est_generation_cost_per_query_usd", False, "f5", "gen $"),
273
+ ("cost", "$/q", "est_cost_per_query_usd", False, "f5", "$/q"),
274
+ ("par", "par", "paraphrase_drop", True, "f3", "par"),
275
+ ("shift", "shift", "shift_drop", True, "f3", "shift"),
276
+ ]
277
+
278
+
279
+ def build(run_json: Path, profiles_dir: Path) -> dict:
280
+ raw = json.loads(run_json.read_text(encoding="utf-8"))
281
+ profiles = load_profiles(profiles_dir)
282
+ benchmarks: List[str] = list(raw.get("benchmarks", []))
283
+ aggregates: List[dict] = raw.get("aggregates", [])
284
+ candidates_meta: List[dict] = raw.get("candidates", [])
285
+
286
+ # Candidate metadata map keyed by candidate_id.
287
+ cand_map: Dict[str, dict] = {}
288
+ for c in candidates_meta:
289
+ llm = c.get("llm", {}) or {}
290
+ retr = c.get("retriever", {}) or {}
291
+ rer = c.get("reranker", {}) or {}
292
+ cid = f"{llm.get('name')}::{retr.get('name')}::{rer.get('name')}"
293
+ cand_map[cid] = {
294
+ "id": cid,
295
+ "model": llm.get("name"),
296
+ "model_adapter": llm.get("adapter"),
297
+ "model_repo": llm.get("model"),
298
+ "retriever": retr.get("name"),
299
+ "retriever_adapter": retr.get("adapter"),
300
+ "reranker": rer.get("name"),
301
+ "reranker_model": rer.get("model"),
302
+ "is_neon": llm.get("name") in NEON_MODELS,
303
+ "is_retrieval_only": llm.get("name") == "retrieval-only",
304
+ }
305
+
306
+ # Trim aggregates to the fields the dashboard needs + identity.
307
+ def trim(agg: dict) -> dict:
308
+ model, retriever, reranker = split_candidate(agg["candidate_id"])
309
+ out = {
310
+ "candidate_id": agg["candidate_id"],
311
+ "benchmark": agg["benchmark"],
312
+ "model": model,
313
+ "retriever": retriever,
314
+ "reranker": reranker,
315
+ "is_neon": model in NEON_MODELS,
316
+ "error": agg.get("error"),
317
+ }
318
+ for f in NUMERIC_FIELDS:
319
+ out[f] = agg.get(f)
320
+ return out
321
+
322
+ trimmed = [trim(a) for a in aggregates]
323
+
324
+ by_bench: Dict[str, List[dict]] = {}
325
+ for a in trimmed:
326
+ by_bench.setdefault(a["benchmark"], []).append(a)
327
+
328
+ # ----- Picks (per benchmark x profile) -----
329
+ picks: Dict[str, Dict[str, dict]] = {}
330
+ for bench in benchmarks:
331
+ aggs = by_bench.get(bench, [])
332
+ picks[bench] = {}
333
+ for p in profiles:
334
+ winner = select_winner(aggs, p)
335
+ if winner is None:
336
+ survivors, _ = apply_hard_constraints(aggs, p)
337
+ status = "all disqualified" if not survivors else "no winner"
338
+ picks[bench][p.name] = {"pick": None, "verdict": status}
339
+ continue
340
+ verdict = value_add_verdict(aggs, p)
341
+ metric_val = (
342
+ winner.get("generation_acc")
343
+ if p.metric == "generation_acc"
344
+ else winner.get("hit_at_1")
345
+ )
346
+ picks[bench][p.name] = {
347
+ "pick": winner["candidate_id"],
348
+ "model": winner["model"],
349
+ "retriever": winner["retriever"],
350
+ "reranker": winner["reranker"],
351
+ "mrr": winner.get("mrr"),
352
+ "metric": p.metric,
353
+ "metric_val": metric_val,
354
+ "p50": winner.get("p50_retrieval_latency_ms"),
355
+ "verdict": verdict,
356
+ }
357
+
358
+ # ----- Per-benchmark ordering, Pareto, DQ -----
359
+ bench_tables: Dict[str, dict] = {}
360
+ for bench in benchmarks:
361
+ aggs = sorted(by_bench.get(bench, []), key=lambda a: a.get("mrr") or 0.0, reverse=True)
362
+ front_ids = {a["candidate_id"] for a in pareto_front(aggs)}
363
+ dq_by_id: Dict[str, str] = {}
364
+ for p in profiles:
365
+ _, dqs = apply_hard_constraints(aggs, p)
366
+ for agg, reason in dqs:
367
+ dq_by_id.setdefault(agg["candidate_id"], reason)
368
+ bench_tables[bench] = {
369
+ "order": [a["candidate_id"] for a in aggs],
370
+ "pareto": sorted(front_ids),
371
+ "dq": dq_by_id,
372
+ }
373
+
374
+ # ----- Per-metric win / loss tally across benchmarks -----
375
+ EPS = 1e-9
376
+ metric_stats: Dict[str, dict] = {}
377
+ for key, label, fieldname, higher, _fmt, _tip in METRICS:
378
+ per_cand: Dict[str, dict] = {}
379
+ for bench in benchmarks:
380
+ vals = [
381
+ (a["candidate_id"], a.get(fieldname))
382
+ for a in by_bench.get(bench, [])
383
+ if a.get(fieldname) is not None
384
+ ]
385
+ if not vals:
386
+ continue
387
+ best = (max if higher else min)(v for _, v in vals)
388
+ worst = (min if higher else max)(v for _, v in vals)
389
+ for cid, v in vals:
390
+ s = per_cand.setdefault(cid, {"wins": 0, "losses": 0, "sum": 0.0, "count": 0})
391
+ s["sum"] += v
392
+ s["count"] += 1
393
+ if abs(v - best) <= EPS:
394
+ s["wins"] += 1
395
+ if abs(v - worst) <= EPS and abs(best - worst) > EPS:
396
+ s["losses"] += 1
397
+ for cid, s in per_cand.items():
398
+ s["mean"] = s["sum"] / s["count"] if s["count"] else None
399
+ del s["sum"]
400
+ metric_stats[key] = per_cand
401
+
402
+ profiles_out = []
403
+ for p in profiles:
404
+ gates = []
405
+ if p.latency_budget_ms:
406
+ gates.append(f"p50 \u2264 {p.latency_budget_ms:g} ms")
407
+ if p.generation_latency_budget_ms:
408
+ gates.append(f"gen \u2264 {p.generation_latency_budget_ms:g} ms")
409
+ if p.cost_ceiling_per_query_usd:
410
+ gates.append(f"$/q \u2264 {p.cost_ceiling_per_query_usd:g}")
411
+ if p.accuracy_floor_mrr:
412
+ gates.append(f"MRR \u2265 {p.accuracy_floor_mrr:g}")
413
+ if p.accuracy_floor_generation_acc:
414
+ gates.append(f"gen \u2265 {p.accuracy_floor_generation_acc:g}")
415
+ profiles_out.append({
416
+ "name": p.name,
417
+ "metric": p.metric,
418
+ "gates": gates,
419
+ "corpus_profile": p.corpus_profile,
420
+ "query_mix": p.query_mix,
421
+ "deployment_constraint": p.deployment_constraint,
422
+ "quality_weight": p.quality_weight,
423
+ })
424
+
425
+ header = raw.get("provenance", {}) or {}
426
+ date_str = ""
427
+ merged_at = header.get("merged_at")
428
+ if merged_at:
429
+ date_str = merged_at[:10]
430
+
431
+ return {
432
+ "generated_at": datetime.now(timezone.utc).isoformat(),
433
+ "date": date_str,
434
+ "benchmarks": benchmarks,
435
+ "candidates": cand_map,
436
+ "aggregates": trimmed,
437
+ "profiles": profiles_out,
438
+ "picks": picks,
439
+ "bench_tables": bench_tables,
440
+ "metric_stats": metric_stats,
441
+ "metrics_def": [
442
+ {"key": k, "label": l, "field": f, "higher": h, "fmt": fmt, "tip": t}
443
+ for (k, l, f, h, fmt, t) in METRICS
444
+ ],
445
+ "glossary": GLOSSARY,
446
+ "provenance": header,
447
+ "n_candidates": len({a["candidate_id"] for a in trimmed}),
448
+ "n_queries": (trimmed[0].get("n_queries") if trimmed else None),
449
+ }
450
+
451
+
452
+ def main() -> None:
453
+ here = Path(__file__).resolve().parent
454
+ default_repo = here.parent.parent / "neon-router"
455
+ ap = argparse.ArgumentParser()
456
+ ap.add_argument(
457
+ "--run-json",
458
+ type=Path,
459
+ default=default_repo / "reports" / "dashboard" / "raw" / "run.json",
460
+ )
461
+ ap.add_argument(
462
+ "--profiles-dir",
463
+ type=Path,
464
+ default=default_repo / "configs" / "profiles",
465
+ )
466
+ ap.add_argument("--out", type=Path, default=here.parent / "data.js")
467
+ args = ap.parse_args()
468
+
469
+ if not args.run_json.exists():
470
+ raise SystemExit(f"run.json not found: {args.run_json}")
471
+ if not args.profiles_dir.exists():
472
+ raise SystemExit(f"profiles dir not found: {args.profiles_dir}")
473
+
474
+ data = build(args.run_json, args.profiles_dir)
475
+ payload = json.dumps(data, ensure_ascii=False, separators=(",", ":"))
476
+ args.out.write_text(
477
+ "// Auto-generated by scripts/build_data.py - do not edit by hand.\n"
478
+ "window.LEADERBOARD_DATA = " + payload + ";\n",
479
+ encoding="utf-8",
480
+ )
481
+ # Also emit a plain JSON sidecar for non-browser consumers.
482
+ (args.out.parent / "data.json").write_text(
483
+ json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8"
484
+ )
485
+ print(f"Wrote {args.out} ({args.out.stat().st_size/1024:.1f} KB)")
486
+ print(f" benchmarks={len(data['benchmarks'])} "
487
+ f"candidates={data['n_candidates']} "
488
+ f"aggregates={len(data['aggregates'])} "
489
+ f"profiles={len(data['profiles'])}")
490
+
491
+
492
+ if __name__ == "__main__":
493
+ main()