Spaces:
Runtime error
Runtime error
Add Neon leaderboard dashboard (single-viewport, Docker static)
Browse files- .gitignore +4 -0
- Dockerfile +14 -0
- README.md +100 -11
- app.js +569 -0
- data.js +0 -0
- data.json +0 -0
- index.html +313 -0
- scripts/build_data.py +493 -0
.gitignore
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
.shots/
|
| 2 |
+
__pycache__/
|
| 3 |
+
*.pyc
|
| 4 |
+
.DS_Store
|
Dockerfile
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Static dashboard served on the port Hugging Face Spaces expects (7860).
|
| 2 |
+
# The app is a single HTML page + precomputed data; no backend logic is needed,
|
| 3 |
+
# so a tiny static file server is all that's required.
|
| 4 |
+
FROM python:3.12-slim
|
| 5 |
+
|
| 6 |
+
WORKDIR /app
|
| 7 |
+
|
| 8 |
+
# Only the files the browser actually needs.
|
| 9 |
+
COPY index.html app.js data.js data.json ./
|
| 10 |
+
|
| 11 |
+
EXPOSE 7860
|
| 12 |
+
|
| 13 |
+
# Serve from /app on 0.0.0.0:7860. index.html is returned for "/".
|
| 14 |
+
CMD ["python", "-m", "http.server", "7860", "--bind", "0.0.0.0"]
|
README.md
CHANGED
|
@@ -1,11 +1,100 @@
|
|
| 1 |
-
---
|
| 2 |
-
title: Leaderboard
|
| 3 |
-
emoji:
|
| 4 |
-
colorFrom:
|
| 5 |
-
colorTo:
|
| 6 |
-
sdk: docker
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
-
|
| 10 |
-
|
| 11 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
title: Neon Leaderboard
|
| 3 |
+
emoji: 📊
|
| 4 |
+
colorFrom: green
|
| 5 |
+
colorTo: blue
|
| 6 |
+
sdk: docker
|
| 7 |
+
app_port: 7860
|
| 8 |
+
pinned: false
|
| 9 |
+
short_description: Single-viewport dashboard for the Neon model leaderboard
|
| 10 |
+
---
|
| 11 |
+
|
| 12 |
+
# Neon Leaderboard
|
| 13 |
+
|
| 14 |
+
A single-page, single-viewport dashboard for exploring the
|
| 15 |
+
[`neon-router`](https://github.com/NeonGeckoCom/neon-router) weekly leaderboard —
|
| 16 |
+
Neon.ai models and retrieval pipelines evaluated across seven QA benchmarks.
|
| 17 |
+
|
| 18 |
+
Everything is **static**: no backend, no build step at serve time. The page reads
|
| 19 |
+
one precomputed file (`data.js`) and renders entirely in the browser, so it deploys
|
| 20 |
+
identically to a Hugging Face *Static* Space or GitHub Pages.
|
| 21 |
+
|
| 22 |
+
## What's inside
|
| 23 |
+
|
| 24 |
+
| Tab | What it shows |
|
| 25 |
+
|---|---|
|
| 26 |
+
| **Compare** | The full results grid. Pick a benchmark and see every pipeline ranked, or switch to **By model** to follow one pipeline across all benchmarks. Searchable benchmark / model dropdowns, sort by any metric (MRR, speed, cost…), filter to a single model, toggle Neon-only / disqualified rows, and relative color heat per column. |
|
| 27 |
+
| **Picks** | The router's single recommended pipeline per client **profile** × benchmark, with a statistical verdict (winner clear / inconclusive / all disqualified). Hover a profile header for its hard gates. |
|
| 28 |
+
| **Models** | A summary page for any model: pipeline, repo, mean MRR / gen / latency, Pareto and DQ counts, and its metrics across every benchmark. |
|
| 29 |
+
| **Metrics** | All models × all metrics — either **mean scores** (color-coded) or **wins / losses** per metric. Sortable by any metric. |
|
| 30 |
+
| **Guide** | Full metrics glossary, every client profile's hard gates, and run provenance. |
|
| 31 |
+
|
| 32 |
+
Hover any column header anywhere for a plain-language definition (e.g. *P50 = 50th
|
| 33 |
+
percentile latency*, *MRR = Mean Reciprocal Rank*). A light/dark toggle (◐) sits in
|
| 34 |
+
the top-right; append `?theme=light` to the URL to share a light-mode link, and
|
| 35 |
+
`#picks` / `#models` / `#metrics` / `#guide` to deep-link a tab.
|
| 36 |
+
|
| 37 |
+
## Faithfulness to the source
|
| 38 |
+
|
| 39 |
+
`data.js` is generated from the leaderboard's own source of truth — `run.json` plus
|
| 40 |
+
the profile YAMLs — and the picks, Pareto front, disqualifications, verdicts, and
|
| 41 |
+
win/loss tallies are recomputed with logic ported directly from
|
| 42 |
+
`router/pareto.py` and `router/reports/leaderboard.py`. Spot-checked to match
|
| 43 |
+
`reports/dashboard/leaderboard.md` exactly.
|
| 44 |
+
|
| 45 |
+
## Regenerating the data
|
| 46 |
+
|
| 47 |
+
When the leaderboard is re-run, rebuild `data.js` from a `neon-router` checkout:
|
| 48 |
+
|
| 49 |
+
```bash
|
| 50 |
+
python scripts/build_data.py \
|
| 51 |
+
--run-json /path/to/neon-router/reports/dashboard/raw/run.json \
|
| 52 |
+
--profiles-dir /path/to/neon-router/configs/profiles \
|
| 53 |
+
--out data.js
|
| 54 |
+
```
|
| 55 |
+
|
| 56 |
+
With a `neon-router` clone sitting next to this folder, the defaults already point
|
| 57 |
+
at it, so `python scripts/build_data.py` is enough. The script needs only
|
| 58 |
+
`pyyaml` (`pip install pyyaml`).
|
| 59 |
+
|
| 60 |
+
To change which models get the **NEON** badge, edit the `NEON_MODELS` set near the
|
| 61 |
+
top of `scripts/build_data.py` and re-run it.
|
| 62 |
+
|
| 63 |
+
## Deploying
|
| 64 |
+
|
| 65 |
+
### Hugging Face Static Space (BrainForge)
|
| 66 |
+
|
| 67 |
+
1. Create a new Space → SDK **Static** under the `BrainForge` org.
|
| 68 |
+
2. Push `index.html`, `app.js`, `data.js`, and this `README.md` (its front-matter
|
| 69 |
+
above configures the Space). That's the whole site.
|
| 70 |
+
|
| 71 |
+
```bash
|
| 72 |
+
# from this folder, with the Space repo as the remote
|
| 73 |
+
git lfs install --skip-repo
|
| 74 |
+
git add index.html app.js data.js README.md scripts/build_data.py
|
| 75 |
+
git commit -m "Neon leaderboard dashboard"
|
| 76 |
+
git push
|
| 77 |
+
```
|
| 78 |
+
|
| 79 |
+
### GitHub Pages (NeonGeckoCom)
|
| 80 |
+
|
| 81 |
+
1. Put these files at the repo root (or in `/docs`).
|
| 82 |
+
2. Settings → Pages → deploy from branch, root (or `/docs`).
|
| 83 |
+
No build step is required — it's plain static files.
|
| 84 |
+
|
| 85 |
+
## Local preview
|
| 86 |
+
|
| 87 |
+
```bash
|
| 88 |
+
python -m http.server 8000
|
| 89 |
+
# open http://localhost:8000/
|
| 90 |
+
```
|
| 91 |
+
|
| 92 |
+
Files:
|
| 93 |
+
|
| 94 |
+
```
|
| 95 |
+
index.html # markup + styles (single viewport shell)
|
| 96 |
+
app.js # all rendering / interactions (vanilla JS, no deps)
|
| 97 |
+
data.js # precomputed dataset (window.LEADERBOARD_DATA)
|
| 98 |
+
data.json # same data as plain JSON (for non-browser consumers)
|
| 99 |
+
scripts/build_data.py # regenerates data.js / data.json from run.json + profiles
|
| 100 |
+
```
|
app.js
ADDED
|
@@ -0,0 +1,569 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/* Neon Leaderboard - static dashboard.
|
| 2 |
+
All data is precomputed in data.js (window.LEADERBOARD_DATA) by
|
| 3 |
+
scripts/build_data.py, faithful to the neon-router report logic. */
|
| 4 |
+
(function () {
|
| 5 |
+
"use strict";
|
| 6 |
+
const D = window.LEADERBOARD_DATA;
|
| 7 |
+
if (!D) { document.body.innerHTML = "<p style='padding:30px'>data.js failed to load.</p>"; return; }
|
| 8 |
+
|
| 9 |
+
const $ = (s, r) => (r || document).querySelector(s);
|
| 10 |
+
const $$ = (s, r) => Array.from((r || document).querySelectorAll(s));
|
| 11 |
+
const el = (tag, attrs, kids) => {
|
| 12 |
+
const n = document.createElement(tag);
|
| 13 |
+
if (attrs) for (const k in attrs) {
|
| 14 |
+
if (k === "class") n.className = attrs[k];
|
| 15 |
+
else if (k === "html") n.innerHTML = attrs[k];
|
| 16 |
+
else if (k === "text") n.textContent = attrs[k];
|
| 17 |
+
else if (k.startsWith("on") && typeof attrs[k] === "function") n.addEventListener(k.slice(2), attrs[k]);
|
| 18 |
+
else if (attrs[k] != null) n.setAttribute(k, attrs[k]);
|
| 19 |
+
}
|
| 20 |
+
if (kids) (Array.isArray(kids) ? kids : [kids]).forEach(c => c != null && n.appendChild(typeof c === "string" ? document.createTextNode(c) : c));
|
| 21 |
+
return n;
|
| 22 |
+
};
|
| 23 |
+
|
| 24 |
+
const GLOSS = D.glossary || {};
|
| 25 |
+
const METRICS = D.metrics_def; // [{key,label,field,higher,fmt,tip}]
|
| 26 |
+
const BENCH = D.benchmarks;
|
| 27 |
+
const AGG = D.aggregates;
|
| 28 |
+
const CANDS = D.candidates; // map id -> meta
|
| 29 |
+
|
| 30 |
+
const aggBy = {}; // benchmark -> [agg]
|
| 31 |
+
const aggByCand = {}; // candidate_id -> [agg] (across benchmarks)
|
| 32 |
+
AGG.forEach(a => {
|
| 33 |
+
(aggBy[a.benchmark] = aggBy[a.benchmark] || []).push(a);
|
| 34 |
+
(aggByCand[a.candidate_id] = aggByCand[a.candidate_id] || []).push(a);
|
| 35 |
+
});
|
| 36 |
+
const CAND_IDS = Object.keys(CANDS);
|
| 37 |
+
const MODEL_NAMES = Array.from(new Set(CAND_IDS.map(id => CANDS[id].model)));
|
| 38 |
+
// Prefer a Neon model with generation data as the default detail selection.
|
| 39 |
+
const DEFAULT_CAND = CAND_IDS.find(id => CANDS[id].is_neon) || CAND_IDS[0];
|
| 40 |
+
|
| 41 |
+
// ---- formatting ----
|
| 42 |
+
function num(v, code) {
|
| 43 |
+
if (v == null || v === "") return "";
|
| 44 |
+
const d = { f0: 0, f1: 1, f2: 2, f3: 3, f5: 5 }[code];
|
| 45 |
+
return Number(v).toFixed(d == null ? 3 : d);
|
| 46 |
+
}
|
| 47 |
+
function pipeLabel(c) {
|
| 48 |
+
const r = c.reranker && c.reranker !== "none" ? " · " + c.reranker : "";
|
| 49 |
+
return c.model + " · " + c.retriever + r;
|
| 50 |
+
}
|
| 51 |
+
function benchShort(b) { return b.replace(/^dataset-/, "").replace(/-qa$/, ""); }
|
| 52 |
+
|
| 53 |
+
// ---- tooltip ----
|
| 54 |
+
const tip = $("#tip");
|
| 55 |
+
function showTip(html, x, y) {
|
| 56 |
+
tip.innerHTML = html; tip.style.opacity = "1";
|
| 57 |
+
const r = tip.getBoundingClientRect();
|
| 58 |
+
let left = x + 14, top = y + 16;
|
| 59 |
+
if (left + r.width > innerWidth - 8) left = x - r.width - 12;
|
| 60 |
+
if (top + r.height > innerHeight - 8) top = y - r.height - 12;
|
| 61 |
+
tip.style.left = Math.max(6, left) + "px"; tip.style.top = Math.max(6, top) + "px";
|
| 62 |
+
}
|
| 63 |
+
function hideTip() { tip.style.opacity = "0"; }
|
| 64 |
+
function bindTip(node, html) {
|
| 65 |
+
if (!html) return node;
|
| 66 |
+
node.addEventListener("mousemove", e => showTip(html, e.clientX, e.clientY));
|
| 67 |
+
node.addEventListener("mouseleave", hideTip);
|
| 68 |
+
return node;
|
| 69 |
+
}
|
| 70 |
+
function glossTip(term, extra) {
|
| 71 |
+
const def = GLOSS[term];
|
| 72 |
+
if (!def && !extra) return null;
|
| 73 |
+
return "<b>" + term + "</b><br>" + (def || "") + (extra ? "<br><br>" + extra : "");
|
| 74 |
+
}
|
| 75 |
+
|
| 76 |
+
// ---- color heat (diverging, low-glare) ----
|
| 77 |
+
function heatStyle(v, min, max, higher) {
|
| 78 |
+
if (v == null || v === "" || max === min) return "";
|
| 79 |
+
let t = (v - min) / (max - min); // 0..1, 1 = numerically max
|
| 80 |
+
if (!higher) t = 1 - t; // 1 = good
|
| 81 |
+
const intensity = Math.abs(t - 0.5) * 2; // 0 mid .. 1 extreme
|
| 82 |
+
const hue = t >= 0.5 ? 168 : 28;
|
| 83 |
+
const dark = document.documentElement.getAttribute("data-theme") !== "light";
|
| 84 |
+
const light = dark ? 46 : 60;
|
| 85 |
+
const sat = dark ? 52 : 62;
|
| 86 |
+
const alpha = (0.05 + intensity * 0.30).toFixed(3);
|
| 87 |
+
return "background:hsla(" + hue + "," + sat + "%," + light + "%," + alpha + ")";
|
| 88 |
+
}
|
| 89 |
+
|
| 90 |
+
// ---- searchable combobox (single select) ----
|
| 91 |
+
function Combo(host, opts) {
|
| 92 |
+
// opts: {options:[{value,label,sub,neon}], value, placeholder, onChange, width}
|
| 93 |
+
let value = opts.value;
|
| 94 |
+
let open = false, kbd = -1, filtered = opts.options;
|
| 95 |
+
host.className = "combo";
|
| 96 |
+
const btn = el("button", { class: "field", style: "text-align:left;min-width:" + (opts.width || 180) + "px;display:flex;align-items:center;gap:6px" });
|
| 97 |
+
const menu = el("div", { class: "menu" });
|
| 98 |
+
const search = el("input", { class: "field", placeholder: "Type to search…", style: "width:100%;margin-bottom:5px" });
|
| 99 |
+
const list = el("div");
|
| 100 |
+
menu.appendChild(search); menu.appendChild(list);
|
| 101 |
+
host.appendChild(btn); host.appendChild(menu);
|
| 102 |
+
|
| 103 |
+
function curLabel() { const o = opts.options.find(o => o.value === value); return o ? o.label : (opts.placeholder || "Select…"); }
|
| 104 |
+
function paintBtn() {
|
| 105 |
+
btn.innerHTML = "";
|
| 106 |
+
const o = opts.options.find(o => o.value === value);
|
| 107 |
+
if (o && o.neon) btn.appendChild(el("span", { class: "badge neon", text: "NEON" }));
|
| 108 |
+
btn.appendChild(el("span", { text: curLabel(), style: "flex:1;overflow:hidden;text-overflow:ellipsis;white-space:nowrap" }));
|
| 109 |
+
btn.appendChild(el("span", { text: "▾", style: "color:var(--text-faint);font-size:10px" }));
|
| 110 |
+
}
|
| 111 |
+
function paintList() {
|
| 112 |
+
list.innerHTML = "";
|
| 113 |
+
filtered.forEach((o, i) => {
|
| 114 |
+
const row = el("div", { class: "opt" + (o.value === value ? " sel" : "") + (i === kbd ? " kbd" : "") });
|
| 115 |
+
if (o.neon) row.appendChild(el("span", { class: "badge neon", text: "NEON" }));
|
| 116 |
+
row.appendChild(el("span", { text: o.label }));
|
| 117 |
+
if (o.sub) row.appendChild(el("small", { text: o.sub }));
|
| 118 |
+
row.addEventListener("click", () => { value = o.value; close(); opts.onChange && opts.onChange(value); });
|
| 119 |
+
list.appendChild(row);
|
| 120 |
+
});
|
| 121 |
+
if (!filtered.length) list.appendChild(el("div", { class: "opt", style: "color:var(--text-faint)", text: "No matches" }));
|
| 122 |
+
}
|
| 123 |
+
function doFilter() {
|
| 124 |
+
const q = search.value.trim().toLowerCase();
|
| 125 |
+
filtered = !q ? opts.options : opts.options.filter(o =>
|
| 126 |
+
(o.label + " " + (o.sub || "")).toLowerCase().includes(q));
|
| 127 |
+
kbd = filtered.length ? 0 : -1; paintList();
|
| 128 |
+
}
|
| 129 |
+
function openMenu() { open = true; host.classList.add("open"); search.value = ""; doFilter(); setTimeout(() => search.focus(), 0); }
|
| 130 |
+
function close() { open = false; host.classList.remove("open"); paintBtn(); }
|
| 131 |
+
btn.addEventListener("click", e => { e.stopPropagation(); open ? close() : openMenu(); });
|
| 132 |
+
search.addEventListener("input", doFilter);
|
| 133 |
+
search.addEventListener("keydown", e => {
|
| 134 |
+
if (e.key === "ArrowDown") { kbd = Math.min(filtered.length - 1, kbd + 1); paintList(); e.preventDefault(); }
|
| 135 |
+
else if (e.key === "ArrowUp") { kbd = Math.max(0, kbd - 1); paintList(); e.preventDefault(); }
|
| 136 |
+
else if (e.key === "Enter" && filtered[kbd]) { value = filtered[kbd].value; close(); opts.onChange && opts.onChange(value); }
|
| 137 |
+
else if (e.key === "Escape") close();
|
| 138 |
+
});
|
| 139 |
+
document.addEventListener("click", e => { if (open && !host.contains(e.target)) close(); });
|
| 140 |
+
paintBtn();
|
| 141 |
+
return {
|
| 142 |
+
get value() { return value; },
|
| 143 |
+
set value(v) { value = v; paintBtn(); },
|
| 144 |
+
setOptions(o, v) { opts.options = o; if (v !== undefined) value = v; paintBtn(); }
|
| 145 |
+
};
|
| 146 |
+
}
|
| 147 |
+
|
| 148 |
+
// ===================================================================
|
| 149 |
+
// State
|
| 150 |
+
// ===================================================================
|
| 151 |
+
const S = {
|
| 152 |
+
tab: "compare",
|
| 153 |
+
cmpMode: "byBench",
|
| 154 |
+
bench: BENCH[0],
|
| 155 |
+
cand: null, // selected pipeline id (byModel)
|
| 156 |
+
filterModel: "__all__",
|
| 157 |
+
sortKey: "mrr",
|
| 158 |
+
sortDir: -1, // -1 desc, 1 asc
|
| 159 |
+
neonOnly: false,
|
| 160 |
+
showDQ: true,
|
| 161 |
+
heat: true,
|
| 162 |
+
metricMode: "scores",
|
| 163 |
+
metricSort: "mrr",
|
| 164 |
+
metricNeonOnly: false,
|
| 165 |
+
metricHeat: true,
|
| 166 |
+
pageCand: null,
|
| 167 |
+
};
|
| 168 |
+
|
| 169 |
+
// ===================================================================
|
| 170 |
+
// Header / footer
|
| 171 |
+
// ===================================================================
|
| 172 |
+
$("#headerMeta").textContent = (D.date || "") + " · " + D.n_candidates + " pipelines · " +
|
| 173 |
+
BENCH.length + " benchmarks · " + (D.n_queries || "?") + " queries";
|
| 174 |
+
const prov = D.provenance || {};
|
| 175 |
+
$("#provFoot").textContent = (prov.weekly ? "weekly roster sweep" : "sweep") + (prov.merged_at ? " · merged " + prov.merged_at.slice(0, 19).replace("T", " ") + "Z" : "");
|
| 176 |
+
function paintLegendSwatches() {
|
| 177 |
+
$("#lgGood").style.cssText = "width:11px;height:11px;border-radius:3px;display:inline-block;" + heatStyle(1, 0, 1, true);
|
| 178 |
+
$("#lgBad").style.cssText = "width:11px;height:11px;border-radius:3px;display:inline-block;" + heatStyle(0, 0, 1, true);
|
| 179 |
+
}
|
| 180 |
+
|
| 181 |
+
// ===================================================================
|
| 182 |
+
// Tabs
|
| 183 |
+
// ===================================================================
|
| 184 |
+
const CTLS = { compare: "#ctl-compare", picks: "#ctl-picks", models: "#ctl-models", metrics: "#ctl-metrics", guide: "#ctl-guide" };
|
| 185 |
+
$$("#tabs button").forEach(b => b.addEventListener("click", () => setTab(b.dataset.tab)));
|
| 186 |
+
function setTab(t) {
|
| 187 |
+
S.tab = t;
|
| 188 |
+
try { history.replaceState(null, "", "#" + t); } catch (e) {}
|
| 189 |
+
$$("#tabs button").forEach(b => b.classList.toggle("active", b.dataset.tab === t));
|
| 190 |
+
$$(".view").forEach(v => v.classList.toggle("active", v.id === "view-" + t));
|
| 191 |
+
for (const k in CTLS) $(CTLS[k]).style.display = (k === t ? "flex" : "none");
|
| 192 |
+
render();
|
| 193 |
+
}
|
| 194 |
+
|
| 195 |
+
// ===================================================================
|
| 196 |
+
// Compare controls
|
| 197 |
+
// ===================================================================
|
| 198 |
+
const benchCombo = Combo($("#benchCombo"), {
|
| 199 |
+
options: BENCH.map(b => ({ value: b, label: benchShort(b), sub: b })),
|
| 200 |
+
value: S.bench, width: 170, onChange: v => { S.bench = v; render(); }
|
| 201 |
+
});
|
| 202 |
+
const candOptions = () => CAND_IDS.map(id => ({ value: id, label: pipeLabel(CANDS[id]), neon: CANDS[id].is_neon, sub: CANDS[id].is_neon ? "Neon" : (CANDS[id].is_retrieval_only ? "retrieval" : "") }));
|
| 203 |
+
const modelCombo = Combo($("#modelCombo"), { options: candOptions(), value: DEFAULT_CAND, width: 230, onChange: v => { S.cand = v; render(); } });
|
| 204 |
+
S.cand = DEFAULT_CAND;
|
| 205 |
+
const filterCombo = Combo($("#filterCombo"), {
|
| 206 |
+
options: [{ value: "__all__", label: "All models" }].concat(MODEL_NAMES.map(m => ({ value: m, label: m, neon: (CANDS[CAND_IDS.find(id => CANDS[id].model === m)] || {}).is_neon }))),
|
| 207 |
+
value: "__all__", width: 160, onChange: v => { S.filterModel = v; render(); }
|
| 208 |
+
});
|
| 209 |
+
const sortSel = $("#sortSel");
|
| 210 |
+
function fillSortSel() {
|
| 211 |
+
sortSel.innerHTML = "";
|
| 212 |
+
sortSel.appendChild(el("option", { value: "pipe", text: "Pipeline (A–Z)" }));
|
| 213 |
+
METRICS.forEach(m => sortSel.appendChild(el("option", { value: m.field, text: m.label })));
|
| 214 |
+
sortSel.value = S.sortKey;
|
| 215 |
+
}
|
| 216 |
+
fillSortSel();
|
| 217 |
+
sortSel.addEventListener("change", () => { S.sortKey = sortSel.value; render(); });
|
| 218 |
+
$("#sortDir").addEventListener("click", () => { S.sortDir *= -1; $("#sortDir").textContent = S.sortDir < 0 ? "▾" : "▴"; render(); });
|
| 219 |
+
$$("#cmpModeSeg button").forEach(b => b.addEventListener("click", () => {
|
| 220 |
+
S.cmpMode = b.dataset.mode;
|
| 221 |
+
$$("#cmpModeSeg button").forEach(x => x.classList.toggle("active", x === b));
|
| 222 |
+
$("#benchPickWrap").style.display = S.cmpMode === "byBench" ? "flex" : "none";
|
| 223 |
+
$("#modelPickWrap").style.display = S.cmpMode === "byModel" ? "flex" : "none";
|
| 224 |
+
render();
|
| 225 |
+
}));
|
| 226 |
+
$("#neonOnly").addEventListener("change", e => { S.neonOnly = e.target.checked; render(); });
|
| 227 |
+
$("#showDQ").addEventListener("change", e => { S.showDQ = e.target.checked; render(); });
|
| 228 |
+
$("#heat").addEventListener("change", e => { S.heat = e.target.checked; render(); });
|
| 229 |
+
|
| 230 |
+
// ===================================================================
|
| 231 |
+
// Compare render
|
| 232 |
+
// ===================================================================
|
| 233 |
+
function activeMetricCols(rows) {
|
| 234 |
+
return METRICS.filter(m => rows.some(r => r[m.field] != null && r[m.field] !== ""));
|
| 235 |
+
}
|
| 236 |
+
function renderCompare() {
|
| 237 |
+
const table = $("#cmpTable");
|
| 238 |
+
table.innerHTML = "";
|
| 239 |
+
let rows, flagFor, leftHead, leftCell;
|
| 240 |
+
|
| 241 |
+
if (S.cmpMode === "byBench") {
|
| 242 |
+
const bt = D.bench_tables[S.bench] || { order: [], pareto: [], dq: {} };
|
| 243 |
+
const paretoSet = new Set(bt.pareto);
|
| 244 |
+
rows = (aggBy[S.bench] || []).slice();
|
| 245 |
+
if (S.filterModel !== "__all__") rows = rows.filter(r => r.model === S.filterModel);
|
| 246 |
+
if (S.neonOnly) rows = rows.filter(r => r.is_neon);
|
| 247 |
+
if (!S.showDQ) rows = rows.filter(r => !bt.dq[r.candidate_id]);
|
| 248 |
+
flagFor = r => ({ pareto: paretoSet.has(r.candidate_id), dq: bt.dq[r.candidate_id] });
|
| 249 |
+
leftHead = "pipeline"; leftCell = r => pipeCell(r, flagFor(r));
|
| 250 |
+
} else {
|
| 251 |
+
rows = (aggByCand[S.cand] || []).slice();
|
| 252 |
+
flagFor = () => ({});
|
| 253 |
+
leftHead = "benchmark";
|
| 254 |
+
leftCell = r => bindTip(el("td", { class: "txt" }, el("span", { class: "pipe", text: benchShort(r.benchmark) })), "<b>" + r.benchmark + "</b>");
|
| 255 |
+
}
|
| 256 |
+
|
| 257 |
+
const cols = activeMetricCols(rows);
|
| 258 |
+
// sort
|
| 259 |
+
rows.sort((a, b) => {
|
| 260 |
+
if (S.sortKey === "pipe") {
|
| 261 |
+
const av = S.cmpMode === "byBench" ? pipeLabel(CANDS[a.candidate_id]) : a.benchmark;
|
| 262 |
+
const bv = S.cmpMode === "byBench" ? pipeLabel(CANDS[b.candidate_id]) : b.benchmark;
|
| 263 |
+
return av < bv ? S.sortDir : av > bv ? -S.sortDir : 0;
|
| 264 |
+
}
|
| 265 |
+
const av = a[S.sortKey], bv = b[S.sortKey];
|
| 266 |
+
if (av == null && bv == null) return 0;
|
| 267 |
+
if (av == null) return 1; if (bv == null) return -1;
|
| 268 |
+
return (av - bv) * S.sortDir;
|
| 269 |
+
});
|
| 270 |
+
|
| 271 |
+
// column ranges for heat
|
| 272 |
+
const ranges = {};
|
| 273 |
+
cols.forEach(m => {
|
| 274 |
+
const vs = rows.map(r => r[m.field]).filter(v => v != null && v !== "");
|
| 275 |
+
ranges[m.field] = vs.length ? { min: Math.min(...vs), max: Math.max(...vs) } : null;
|
| 276 |
+
});
|
| 277 |
+
|
| 278 |
+
// head
|
| 279 |
+
const thead = el("thead");
|
| 280 |
+
const htr = el("tr");
|
| 281 |
+
htr.appendChild(bindTip(el("th", { class: "txt lbl sortable", onclick: () => sortByCol("pipe"), html: leftHead + (S.sortKey === "pipe" ? " <span class='arrow'>" + (S.sortDir < 0 ? "▼" : "▲") + "</span>" : "") }),
|
| 282 |
+
S.cmpMode === "byBench" ? glossTip("pipeline", "model · retriever · reranker") : "<b>benchmark</b><br>The evaluation dataset for this row."));
|
| 283 |
+
cols.forEach(m => {
|
| 284 |
+
const arrow = S.sortKey === m.field ? " <span class='arrow'>" + (S.sortDir < 0 ? "▼" : "▲") + "</span>" : "";
|
| 285 |
+
htr.appendChild(bindTip(el("th", { class: "sortable", onclick: () => sortByCol(m.field), html: m.label + arrow }),
|
| 286 |
+
glossTip(m.tip, (m.higher ? "Higher is better." : "Lower is better."))));
|
| 287 |
+
});
|
| 288 |
+
thead.appendChild(htr); table.appendChild(thead);
|
| 289 |
+
|
| 290 |
+
// body
|
| 291 |
+
const tb = el("tbody");
|
| 292 |
+
if (!rows.length) { tb.appendChild(el("tr", null, el("td", { colspan: cols.length + 1, class: "empty", text: "No rows match the current filters." }))); }
|
| 293 |
+
rows.forEach(r => {
|
| 294 |
+
const tr = el("tr", { class: r.is_neon ? "neon" : "" });
|
| 295 |
+
tr.appendChild(leftCell(r));
|
| 296 |
+
cols.forEach(m => {
|
| 297 |
+
const v = r[m.field];
|
| 298 |
+
const td = el("td", { text: num(v, m.fmt) });
|
| 299 |
+
if (S.heat && ranges[m.field]) td.style.cssText = heatStyle(v, ranges[m.field].min, ranges[m.field].max, m.higher);
|
| 300 |
+
tr.appendChild(td);
|
| 301 |
+
});
|
| 302 |
+
tb.appendChild(tr);
|
| 303 |
+
});
|
| 304 |
+
table.appendChild(tb);
|
| 305 |
+
}
|
| 306 |
+
function sortByCol(k) { if (S.sortKey === k) S.sortDir *= -1; else { S.sortKey = k; S.sortDir = (k === "pipe") ? 1 : -1; } sortSel.value = k; $("#sortDir").textContent = S.sortDir < 0 ? "▾" : "▴"; render(); }
|
| 307 |
+
|
| 308 |
+
function pipeCell(r, flag) {
|
| 309 |
+
const c = CANDS[r.candidate_id];
|
| 310 |
+
const inner = el("span", { class: "pipe" });
|
| 311 |
+
if (flag.pareto) inner.appendChild(el("span", { class: "star", text: "★ " }));
|
| 312 |
+
inner.appendChild(el("b", { text: c.model }));
|
| 313 |
+
inner.appendChild(el("small", { text: " " + c.retriever + (c.reranker !== "none" ? " · " + c.reranker : "") }));
|
| 314 |
+
const td = el("td", { class: "txt" });
|
| 315 |
+
if (c.is_neon) td.appendChild(el("span", { class: "badge neon", text: "NEON", style: "margin-right:6px" }));
|
| 316 |
+
td.appendChild(inner);
|
| 317 |
+
if (flag.dq) td.appendChild(el("span", { class: "badge dq", text: "DQ:" + flag.dq, style: "margin-left:7px" }));
|
| 318 |
+
return bindTip(td, "<b>" + pipeLabel(c) + "</b>" +
|
| 319 |
+
(c.model_repo ? "<br>repo: " + c.model_repo : "") +
|
| 320 |
+
(flag.dq ? "<br><br>Disqualified by a profile's <b>" + flag.dq + "</b> gate." : "") +
|
| 321 |
+
(flag.pareto ? "<br><br>★ On the Pareto front (not dominated on accuracy / latency / build-cost)." : ""));
|
| 322 |
+
}
|
| 323 |
+
|
| 324 |
+
// ===================================================================
|
| 325 |
+
// Picks render
|
| 326 |
+
// ===================================================================
|
| 327 |
+
const VCLASS = { "winner clear": "v-clear", "inconclusive": "v-inconclusive", "all disqualified": "v-dq", "no winner": "v-none", "no signal": "v-none" };
|
| 328 |
+
function renderPicks() {
|
| 329 |
+
const table = $("#picksTable"); table.innerHTML = "";
|
| 330 |
+
const profs = D.profiles;
|
| 331 |
+
const thead = el("thead"); const htr = el("tr");
|
| 332 |
+
htr.appendChild(el("th", { class: "txt", text: "benchmark", style: "text-align:left" }));
|
| 333 |
+
profs.forEach(p => {
|
| 334 |
+
const extra = "Hard gates: " + (p.gates.join(" · ") || "—") + "<br>Ranks on: " + p.metric +
|
| 335 |
+
(p.corpus_profile ? "<br>Corpus: " + p.corpus_profile : "") +
|
| 336 |
+
(p.deployment_constraint ? "<br>Deploy: " + p.deployment_constraint : "");
|
| 337 |
+
htr.appendChild(bindTip(el("th", { html: p.name.replace(/_/g, " ") }), glossTip("profile", extra)));
|
| 338 |
+
});
|
| 339 |
+
thead.appendChild(htr); table.appendChild(thead);
|
| 340 |
+
|
| 341 |
+
const tb = el("tbody");
|
| 342 |
+
BENCH.forEach(b => {
|
| 343 |
+
const tr = el("tr");
|
| 344 |
+
tr.appendChild(bindTip(el("th", { text: benchShort(b) }), "<b>" + b + "</b>"));
|
| 345 |
+
const row = D.picks[b] || {};
|
| 346 |
+
profs.forEach(p => {
|
| 347 |
+
const cell = row[p.name];
|
| 348 |
+
const td = el("td");
|
| 349 |
+
if (cell && cell.pick) {
|
| 350 |
+
const c = CANDS[cell.pick];
|
| 351 |
+
const wrap = el("div");
|
| 352 |
+
const line = el("div", { style: "font-family:var(--mono);font-size:11.5px;margin-bottom:4px" });
|
| 353 |
+
if (c && c.is_neon) line.appendChild(el("span", { class: "badge neon", text: "NEON", style: "margin-right:5px" }));
|
| 354 |
+
line.appendChild(el("span", { text: cell.model + " · " + cell.retriever }));
|
| 355 |
+
wrap.appendChild(line);
|
| 356 |
+
const mv = cell.metric === "generation_acc" ? "gen " + num(cell.metric_val, "f2") : "H@1 " + num(cell.metric_val, "f2");
|
| 357 |
+
wrap.appendChild(bindTip(el("span", { class: "verdict " + (VCLASS[cell.verdict] || "v-none"), text: cell.verdict }),
|
| 358 |
+
glossTip("verdict") ));
|
| 359 |
+
wrap.appendChild(el("span", { class: "cell-sub", text: "MRR " + num(cell.mrr, "f3") + " · " + mv + " · p50 " + num(cell.p50, "f0") + "ms" }));
|
| 360 |
+
td.appendChild(wrap);
|
| 361 |
+
} else {
|
| 362 |
+
td.appendChild(el("span", { class: "verdict " + (VCLASS[(cell && cell.verdict)] || "v-dq"), text: (cell && cell.verdict) || "—" }));
|
| 363 |
+
}
|
| 364 |
+
tr.appendChild(td);
|
| 365 |
+
});
|
| 366 |
+
tb.appendChild(tr);
|
| 367 |
+
});
|
| 368 |
+
table.appendChild(tb);
|
| 369 |
+
}
|
| 370 |
+
|
| 371 |
+
// ===================================================================
|
| 372 |
+
// Models render
|
| 373 |
+
// ===================================================================
|
| 374 |
+
const modelPageCombo = Combo($("#modelPageCombo"), { options: candOptions(), value: DEFAULT_CAND, width: 240, onChange: v => { S.pageCand = v; render(); } });
|
| 375 |
+
S.pageCand = DEFAULT_CAND;
|
| 376 |
+
function renderModels() {
|
| 377 |
+
const host = $("#modelPage"); host.innerHTML = "";
|
| 378 |
+
const id = S.pageCand; const c = CANDS[id];
|
| 379 |
+
const rows = (aggByCand[id] || []).slice().sort((a, b) => benchShort(a.benchmark) < benchShort(b.benchmark) ? -1 : 1);
|
| 380 |
+
$("#modelPageBadge").innerHTML = c.is_neon ? "<span class='badge neon'>NEON.ai model</span>" : (c.is_retrieval_only ? "<span class='badge pareto'>retrieval baseline</span>" : "");
|
| 381 |
+
|
| 382 |
+
// aggregate stats
|
| 383 |
+
const mean = f => { const vs = rows.map(r => r[f]).filter(v => v != null); return vs.length ? vs.reduce((a, b) => a + b, 0) / vs.length : null; };
|
| 384 |
+
let paretoCount = 0; BENCH.forEach(b => { const bt = D.bench_tables[b]; if (bt && bt.pareto.includes(id)) paretoCount++; });
|
| 385 |
+
let dqCount = 0; BENCH.forEach(b => { const bt = D.bench_tables[b]; if (bt && bt.dq[id]) dqCount++; });
|
| 386 |
+
|
| 387 |
+
// left card
|
| 388 |
+
const left = el("div", { class: "card" });
|
| 389 |
+
left.appendChild(el("h3", { style: "margin:0 0 10px;font-size:14px", text: c.model }));
|
| 390 |
+
const kvs = [
|
| 391 |
+
["pipeline", c.retriever + (c.reranker !== "none" ? " · " + c.reranker : "")],
|
| 392 |
+
["model repo", c.model_repo || "—"],
|
| 393 |
+
["serving", c.model_adapter || "—"],
|
| 394 |
+
["benchmarks", String(rows.length)],
|
| 395 |
+
["type", c.is_neon ? "Neon.ai model" : (c.is_retrieval_only ? "retrieval-only" : "external")],
|
| 396 |
+
];
|
| 397 |
+
kvs.forEach(([k, v]) => left.appendChild(el("div", { class: "kv" }, [el("span", { class: "k", text: k }), el("span", { class: "v", text: v })])));
|
| 398 |
+
const stats = el("div", { class: "stat-row", style: "margin-top:14px" });
|
| 399 |
+
const stat = (n, l, tipTerm) => { const s = el("div", { class: "stat" }, [el("div", { class: "n", text: n }), bindTip(el("div", { class: "l", text: l }), glossTip(tipTerm))]); stats.appendChild(s); };
|
| 400 |
+
stat(num(mean("mrr"), "f3"), "mean MRR", "MRR");
|
| 401 |
+
const gm = mean("generation_acc"); stat(gm == null ? "—" : num(gm, "f2"), "mean gen", "gen");
|
| 402 |
+
stat(num(mean("p50_retrieval_latency_ms"), "f0"), "p50 ms", "P50");
|
| 403 |
+
stat(String(paretoCount), "Pareto ★", "Pareto");
|
| 404 |
+
stat(String(dqCount), "DQ count", "DQ");
|
| 405 |
+
left.appendChild(stats);
|
| 406 |
+
host.appendChild(left);
|
| 407 |
+
|
| 408 |
+
// right card: metrics across benchmarks
|
| 409 |
+
const right = el("div", { class: "card scroll" });
|
| 410 |
+
const cols = activeMetricCols(rows);
|
| 411 |
+
const table = el("table");
|
| 412 |
+
const thead = el("thead"); const htr = el("tr");
|
| 413 |
+
htr.appendChild(el("th", { class: "txt", text: "benchmark", style: "text-align:left;position:sticky;top:0;background:var(--bg-elev)" }));
|
| 414 |
+
cols.forEach(m => htr.appendChild(bindTip(el("th", { html: m.label, style: "position:sticky;top:0;background:var(--bg-elev)" }), glossTip(m.tip, m.higher ? "Higher is better." : "Lower is better."))));
|
| 415 |
+
thead.appendChild(htr); table.appendChild(thead);
|
| 416 |
+
const ranges = {}; cols.forEach(m => { const vs = rows.map(r => r[m.field]).filter(v => v != null); ranges[m.field] = vs.length ? { min: Math.min(...vs), max: Math.max(...vs) } : null; });
|
| 417 |
+
const tb = el("tbody");
|
| 418 |
+
rows.forEach(r => {
|
| 419 |
+
const bt = D.bench_tables[r.benchmark];
|
| 420 |
+
const tr = el("tr");
|
| 421 |
+
const star = bt && bt.pareto.includes(id) ? el("span", { class: "star", text: "★ " }) : null;
|
| 422 |
+
const dq = bt && bt.dq[id] ? el("span", { class: "badge dq", text: "DQ:" + bt.dq[id], style: "margin-left:6px" }) : null;
|
| 423 |
+
const tdL = el("td", { class: "txt" }); if (star) tdL.appendChild(star); tdL.appendChild(el("span", { text: benchShort(r.benchmark) })); if (dq) tdL.appendChild(dq);
|
| 424 |
+
tr.appendChild(tdL);
|
| 425 |
+
cols.forEach(m => { const v = r[m.field]; const td = el("td", { text: num(v, m.fmt) }); if (ranges[m.field]) td.style.cssText = heatStyle(v, ranges[m.field].min, ranges[m.field].max, m.higher); tr.appendChild(td); });
|
| 426 |
+
tb.appendChild(tr);
|
| 427 |
+
});
|
| 428 |
+
table.appendChild(tb); right.appendChild(table); host.appendChild(right);
|
| 429 |
+
}
|
| 430 |
+
|
| 431 |
+
// ===================================================================
|
| 432 |
+
// Metrics render
|
| 433 |
+
// ===================================================================
|
| 434 |
+
const metricSortSel = $("#metricSortSel");
|
| 435 |
+
function fillMetricSort() {
|
| 436 |
+
metricSortSel.innerHTML = "";
|
| 437 |
+
metricSortSel.appendChild(el("option", { value: "name", text: "Pipeline (A–Z)" }));
|
| 438 |
+
METRICS.forEach(m => metricSortSel.appendChild(el("option", { value: m.key, text: m.label })));
|
| 439 |
+
metricSortSel.value = S.metricSort;
|
| 440 |
+
}
|
| 441 |
+
fillMetricSort();
|
| 442 |
+
metricSortSel.addEventListener("change", () => { S.metricSort = metricSortSel.value; render(); });
|
| 443 |
+
$$("#metricModeSeg button").forEach(b => b.addEventListener("click", () => { S.metricMode = b.dataset.mode; $$("#metricModeSeg button").forEach(x => x.classList.toggle("active", x === b)); render(); }));
|
| 444 |
+
$("#metricNeonOnly").addEventListener("change", e => { S.metricNeonOnly = e.target.checked; render(); });
|
| 445 |
+
$("#metricHeat").addEventListener("change", e => { S.metricHeat = e.target.checked; render(); });
|
| 446 |
+
|
| 447 |
+
function renderMetrics() {
|
| 448 |
+
const table = $("#metricsTable"); table.innerHTML = "";
|
| 449 |
+
const stats = D.metric_stats; // key -> cid -> {wins,losses,mean,count}
|
| 450 |
+
let ids = CAND_IDS.slice();
|
| 451 |
+
if (S.metricNeonOnly) ids = ids.filter(id => CANDS[id].is_neon);
|
| 452 |
+
|
| 453 |
+
// sort rows
|
| 454 |
+
if (S.metricSort === "name") ids.sort((a, b) => pipeLabel(CANDS[a]) < pipeLabel(CANDS[b]) ? -1 : 1);
|
| 455 |
+
else {
|
| 456 |
+
const mdef = METRICS.find(m => m.key === S.metricSort);
|
| 457 |
+
const st = stats[S.metricSort] || {};
|
| 458 |
+
ids.sort((a, b) => {
|
| 459 |
+
const sa = st[a], sb = st[b];
|
| 460 |
+
const va = S.metricMode === "winloss" ? (sa ? sa.wins - sa.losses : -1e9) : (sa && sa.mean != null ? sa.mean : (mdef.higher ? -1e9 : 1e9));
|
| 461 |
+
const vb = S.metricMode === "winloss" ? (sb ? sb.wins - sb.losses : -1e9) : (sb && sb.mean != null ? sb.mean : (mdef.higher ? -1e9 : 1e9));
|
| 462 |
+
return S.metricMode === "winloss" ? (vb - va) : ((vb - va) * (mdef.higher ? 1 : -1));
|
| 463 |
+
});
|
| 464 |
+
}
|
| 465 |
+
|
| 466 |
+
const thead = el("thead"); const htr = el("tr");
|
| 467 |
+
htr.appendChild(el("th", { class: "txt", text: "pipeline", style: "text-align:left;position:sticky;left:0;z-index:6" }));
|
| 468 |
+
METRICS.forEach(m => htr.appendChild(bindTip(el("th", { html: m.label }), glossTip(m.tip, m.higher ? "Higher is better." : "Lower is better."))));
|
| 469 |
+
thead.appendChild(htr); table.appendChild(thead);
|
| 470 |
+
|
| 471 |
+
// ranges for mean heat per metric
|
| 472 |
+
const ranges = {};
|
| 473 |
+
METRICS.forEach(m => { const vs = ids.map(id => (stats[m.key][id] || {}).mean).filter(v => v != null); ranges[m.key] = vs.length ? { min: Math.min(...vs), max: Math.max(...vs) } : null; });
|
| 474 |
+
|
| 475 |
+
const tb = el("tbody");
|
| 476 |
+
ids.forEach(id => {
|
| 477 |
+
const c = CANDS[id];
|
| 478 |
+
const tr = el("tr", { class: c.is_neon ? "neon" : "" });
|
| 479 |
+
const th = el("th", { style: "font-family:var(--mono);font-size:11.5px" });
|
| 480 |
+
if (c.is_neon) th.appendChild(el("span", { class: "badge neon", text: "NEON", style: "margin-right:5px" }));
|
| 481 |
+
th.appendChild(el("span", { text: pipeLabel(c) }));
|
| 482 |
+
bindTip(th, "<b>" + pipeLabel(c) + "</b>" + (c.model_repo ? "<br>repo: " + c.model_repo : ""));
|
| 483 |
+
tr.appendChild(th);
|
| 484 |
+
METRICS.forEach(m => {
|
| 485 |
+
const s = (stats[m.key] || {})[id];
|
| 486 |
+
const td = el("td");
|
| 487 |
+
if (!s) { td.textContent = ""; }
|
| 488 |
+
else if (S.metricMode === "winloss") {
|
| 489 |
+
td.innerHTML = "<span style='color:var(--neon)'>" + s.wins + "</span> – <span style='color:#ff9b7d'>" + s.losses + "</span>";
|
| 490 |
+
const net = s.wins - s.losses;
|
| 491 |
+
if (S.metricHeat) td.style.cssText = heatStyle(net, -3, 3, true);
|
| 492 |
+
} else {
|
| 493 |
+
td.textContent = num(s.mean, m.fmt);
|
| 494 |
+
if (S.metricHeat && ranges[m.key]) td.style.cssText = heatStyle(s.mean, ranges[m.key].min, ranges[m.key].max, m.higher);
|
| 495 |
+
}
|
| 496 |
+
tr.appendChild(td);
|
| 497 |
+
});
|
| 498 |
+
tb.appendChild(tr);
|
| 499 |
+
});
|
| 500 |
+
table.appendChild(tb);
|
| 501 |
+
}
|
| 502 |
+
|
| 503 |
+
// ===================================================================
|
| 504 |
+
// Guide render
|
| 505 |
+
// ===================================================================
|
| 506 |
+
function renderGuide() {
|
| 507 |
+
const host = $("#guideGrid"); host.innerHTML = "";
|
| 508 |
+
// left: glossary
|
| 509 |
+
const left = el("div", { class: "guide-col" });
|
| 510 |
+
left.appendChild(el("h3", { text: "Metrics glossary" }));
|
| 511 |
+
const gt = el("table", { class: "gloss" });
|
| 512 |
+
const order = ["MRR", "H@1", "H@5", "nDCG", "P50", "P95", "build", "gen", "gen ms", "tok/s", "gen $", "$/q", "par", "shift", "n", "Pareto", "DQ", "verdict"];
|
| 513 |
+
order.forEach(k => { if (GLOSS[k]) gt.appendChild(el("tr", null, [el("td", { class: "term", text: k }), el("td", { class: "def", text: GLOSS[k] })])); });
|
| 514 |
+
left.appendChild(gt); host.appendChild(left);
|
| 515 |
+
// right: profiles + provenance
|
| 516 |
+
const right = el("div", { class: "guide-col" });
|
| 517 |
+
right.appendChild(el("h3", { text: "Client profiles (hard gates)" }));
|
| 518 |
+
D.profiles.forEach(p => {
|
| 519 |
+
const box = el("div", { class: "prof" });
|
| 520 |
+
box.appendChild(el("h4", { text: p.name.replace(/_/g, " ") + " — ranks on " + p.metric }));
|
| 521 |
+
const gates = el("div", { class: "gates" });
|
| 522 |
+
(p.gates.length ? p.gates : ["no hard gates"]).forEach(g => gates.appendChild(el("span", { class: "gate", text: g })));
|
| 523 |
+
box.appendChild(gates);
|
| 524 |
+
const sub = [];
|
| 525 |
+
if (p.corpus_profile) sub.push("corpus: " + p.corpus_profile);
|
| 526 |
+
if (p.deployment_constraint) sub.push("deploy: " + p.deployment_constraint);
|
| 527 |
+
if (sub.length) box.appendChild(el("div", { style: "color:var(--text-faint);font-size:11px;margin-top:6px", text: sub.join(" · ") }));
|
| 528 |
+
right.appendChild(box);
|
| 529 |
+
});
|
| 530 |
+
right.appendChild(el("h3", { text: "Provenance", style: "margin-top:10px" }));
|
| 531 |
+
const pv = el("div", { class: "prof" });
|
| 532 |
+
const sources = (prov.merged_from || []);
|
| 533 |
+
pv.appendChild(el("div", { style: "font-size:11.5px;color:var(--text-dim)", text: "Type: " + (prov.weekly ? "weekly roster sweep" : "sweep") }));
|
| 534 |
+
if (prov.merged_at) pv.appendChild(el("div", { style: "font-size:11.5px;color:var(--text-dim)", text: "Merged at: " + prov.merged_at }));
|
| 535 |
+
sources.forEach(s => pv.appendChild(el("div", { style: "font-family:var(--mono);font-size:10.5px;color:var(--text-faint);margin-top:3px", text: s })));
|
| 536 |
+
right.appendChild(pv);
|
| 537 |
+
host.appendChild(right);
|
| 538 |
+
}
|
| 539 |
+
|
| 540 |
+
// ===================================================================
|
| 541 |
+
// Theme
|
| 542 |
+
// ===================================================================
|
| 543 |
+
function applyTheme(t) {
|
| 544 |
+
document.documentElement.setAttribute("data-theme", t);
|
| 545 |
+
try { localStorage.setItem("nlb-theme", t); } catch (e) {}
|
| 546 |
+
paintLegendSwatches(); render();
|
| 547 |
+
}
|
| 548 |
+
$("#themeBtn").addEventListener("click", () => applyTheme(document.documentElement.getAttribute("data-theme") === "light" ? "dark" : "light"));
|
| 549 |
+
let savedTheme = "dark"; try { savedTheme = localStorage.getItem("nlb-theme") || "dark"; } catch (e) {}
|
| 550 |
+
const qTheme = new URLSearchParams(location.search).get("theme");
|
| 551 |
+
if (qTheme === "light" || qTheme === "dark") savedTheme = qTheme;
|
| 552 |
+
document.documentElement.setAttribute("data-theme", savedTheme);
|
| 553 |
+
|
| 554 |
+
// ===================================================================
|
| 555 |
+
// Master render
|
| 556 |
+
// ===================================================================
|
| 557 |
+
function render() {
|
| 558 |
+
if (S.tab === "compare") renderCompare();
|
| 559 |
+
else if (S.tab === "picks") renderPicks();
|
| 560 |
+
else if (S.tab === "models") renderModels();
|
| 561 |
+
else if (S.tab === "metrics") renderMetrics();
|
| 562 |
+
else if (S.tab === "guide") renderGuide();
|
| 563 |
+
}
|
| 564 |
+
|
| 565 |
+
paintLegendSwatches();
|
| 566 |
+
const initTab = (location.hash || "").replace("#", "");
|
| 567 |
+
if (["compare", "picks", "models", "metrics", "guide"].includes(initTab)) setTab(initTab);
|
| 568 |
+
else render();
|
| 569 |
+
})();
|
data.js
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
data.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
index.html
ADDED
|
@@ -0,0 +1,313 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<!doctype html>
|
| 2 |
+
<html lang="en">
|
| 3 |
+
<head>
|
| 4 |
+
<meta charset="utf-8" />
|
| 5 |
+
<meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover" />
|
| 6 |
+
<title>Neon Leaderboard</title>
|
| 7 |
+
<style>
|
| 8 |
+
:root {
|
| 9 |
+
--bg: #0f1419;
|
| 10 |
+
--bg-elev: #161c24;
|
| 11 |
+
--bg-elev2: #1c242e;
|
| 12 |
+
--panel: #131a22;
|
| 13 |
+
--border: #283441;
|
| 14 |
+
--border-soft: #1f2a35;
|
| 15 |
+
--text: #d7dee7;
|
| 16 |
+
--text-dim: #93a1b0;
|
| 17 |
+
--text-faint: #64727f;
|
| 18 |
+
--accent: #3fb6a8; /* neon teal */
|
| 19 |
+
--accent-dim: #2c7a72;
|
| 20 |
+
--neon: #5ad1c2;
|
| 21 |
+
--good-h: 168; /* teal-green */
|
| 22 |
+
--bad-h: 28; /* warm amber */
|
| 23 |
+
--chip-bg: #1d2731;
|
| 24 |
+
--shadow: 0 4px 18px rgba(0,0,0,.35);
|
| 25 |
+
--radius: 9px;
|
| 26 |
+
--row-h: 26px;
|
| 27 |
+
--font: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
|
| 28 |
+
--mono: ui-monospace, "SF Mono", "Cascadia Code", "Roboto Mono", Menlo, Consolas, monospace;
|
| 29 |
+
}
|
| 30 |
+
html[data-theme="light"] {
|
| 31 |
+
--bg: #f3f5f8;
|
| 32 |
+
--bg-elev: #ffffff;
|
| 33 |
+
--bg-elev2: #eef1f5;
|
| 34 |
+
--panel: #ffffff;
|
| 35 |
+
--border: #d4dbe3;
|
| 36 |
+
--border-soft: #e4e9ef;
|
| 37 |
+
--text: #1f2933;
|
| 38 |
+
--text-dim: #56636f;
|
| 39 |
+
--text-faint: #8a97a3;
|
| 40 |
+
--accent: #1f8f82;
|
| 41 |
+
--accent-dim: #5fb5ab;
|
| 42 |
+
--neon: #178a7c;
|
| 43 |
+
--chip-bg: #eef2f6;
|
| 44 |
+
--shadow: 0 4px 18px rgba(40,55,75,.12);
|
| 45 |
+
}
|
| 46 |
+
|
| 47 |
+
* { box-sizing: border-box; }
|
| 48 |
+
html, body { height: 100%; margin: 0; }
|
| 49 |
+
body {
|
| 50 |
+
background: var(--bg);
|
| 51 |
+
color: var(--text);
|
| 52 |
+
font-family: var(--font);
|
| 53 |
+
font-size: 13px;
|
| 54 |
+
line-height: 1.35;
|
| 55 |
+
-webkit-font-smoothing: antialiased;
|
| 56 |
+
overflow: hidden; /* single viewport, no page scroll */
|
| 57 |
+
}
|
| 58 |
+
button { font-family: inherit; color: inherit; cursor: pointer; }
|
| 59 |
+
select, input { font-family: inherit; font-size: 12.5px; }
|
| 60 |
+
|
| 61 |
+
.app { display: flex; flex-direction: column; height: 100vh; height: 100dvh; }
|
| 62 |
+
|
| 63 |
+
/* ---------- Header ---------- */
|
| 64 |
+
header.top {
|
| 65 |
+
display: flex; align-items: center; gap: 18px;
|
| 66 |
+
padding: 0 16px; height: 50px; flex: 0 0 auto;
|
| 67 |
+
background: var(--bg-elev); border-bottom: 1px solid var(--border);
|
| 68 |
+
}
|
| 69 |
+
.brand { display: flex; align-items: baseline; gap: 9px; white-space: nowrap; }
|
| 70 |
+
.brand .dot { width: 9px; height: 9px; border-radius: 50%; background: var(--accent);
|
| 71 |
+
box-shadow: 0 0 9px var(--accent); align-self: center; }
|
| 72 |
+
.brand h1 { font-size: 15px; font-weight: 650; margin: 0; letter-spacing: .2px; }
|
| 73 |
+
.brand .meta { color: var(--text-faint); font-size: 11px; }
|
| 74 |
+
|
| 75 |
+
nav.tabs { display: flex; gap: 2px; margin-left: 4px; }
|
| 76 |
+
nav.tabs button {
|
| 77 |
+
background: transparent; border: 0; padding: 8px 14px; border-radius: 7px;
|
| 78 |
+
color: var(--text-dim); font-size: 13px; font-weight: 550; letter-spacing: .2px;
|
| 79 |
+
transition: background .12s, color .12s;
|
| 80 |
+
}
|
| 81 |
+
nav.tabs button:hover { background: var(--bg-elev2); color: var(--text); }
|
| 82 |
+
nav.tabs button.active { background: var(--accent); color: #06231f; }
|
| 83 |
+
html[data-theme="light"] nav.tabs button.active { color: #fff; }
|
| 84 |
+
|
| 85 |
+
.spacer { flex: 1 1 auto; }
|
| 86 |
+
.icon-btn {
|
| 87 |
+
background: var(--bg-elev2); border: 1px solid var(--border); border-radius: 7px;
|
| 88 |
+
width: 32px; height: 32px; display: grid; place-items: center; color: var(--text-dim);
|
| 89 |
+
font-size: 15px;
|
| 90 |
+
}
|
| 91 |
+
.icon-btn:hover { color: var(--text); border-color: var(--accent-dim); }
|
| 92 |
+
|
| 93 |
+
/* ---------- Controls bar ---------- */
|
| 94 |
+
.controls {
|
| 95 |
+
display: flex; align-items: center; gap: 14px; flex-wrap: wrap;
|
| 96 |
+
padding: 9px 16px; flex: 0 0 auto;
|
| 97 |
+
background: var(--panel); border-bottom: 1px solid var(--border-soft);
|
| 98 |
+
min-height: 46px;
|
| 99 |
+
}
|
| 100 |
+
.ctl { display: flex; align-items: center; gap: 7px; }
|
| 101 |
+
.ctl label { color: var(--text-faint); font-size: 11px; text-transform: uppercase; letter-spacing: .6px; }
|
| 102 |
+
.field {
|
| 103 |
+
background: var(--bg-elev2); border: 1px solid var(--border); color: var(--text);
|
| 104 |
+
border-radius: 7px; padding: 6px 9px; min-width: 90px; outline: none;
|
| 105 |
+
}
|
| 106 |
+
.field:focus { border-color: var(--accent-dim); }
|
| 107 |
+
select.field { padding-right: 26px; appearance: none;
|
| 108 |
+
background-image: linear-gradient(45deg, transparent 50%, var(--text-faint) 50%),
|
| 109 |
+
linear-gradient(135deg, var(--text-faint) 50%, transparent 50%);
|
| 110 |
+
background-position: calc(100% - 14px) 50%, calc(100% - 9px) 50%;
|
| 111 |
+
background-size: 5px 5px, 5px 5px; background-repeat: no-repeat; }
|
| 112 |
+
.seg { display: inline-flex; background: var(--bg-elev2); border: 1px solid var(--border);
|
| 113 |
+
border-radius: 7px; overflow: hidden; }
|
| 114 |
+
.seg button { background: transparent; border: 0; padding: 6px 11px; color: var(--text-dim); font-size: 12px; }
|
| 115 |
+
.seg button.active { background: var(--accent); color: #06231f; }
|
| 116 |
+
html[data-theme="light"] .seg button.active { color: #fff; }
|
| 117 |
+
.toggle { display: inline-flex; align-items: center; gap: 6px; color: var(--text-dim); font-size: 12px; user-select: none; }
|
| 118 |
+
.toggle input { accent-color: var(--accent); }
|
| 119 |
+
|
| 120 |
+
/* combobox (searchable dropdown) */
|
| 121 |
+
.combo { position: relative; }
|
| 122 |
+
.combo .menu {
|
| 123 |
+
position: absolute; z-index: 40; top: calc(100% + 4px); left: 0; min-width: 240px; max-width: 360px;
|
| 124 |
+
max-height: 320px; overflow: auto; background: var(--bg-elev); border: 1px solid var(--border);
|
| 125 |
+
border-radius: 8px; box-shadow: var(--shadow); padding: 5px; display: none;
|
| 126 |
+
}
|
| 127 |
+
.combo.open .menu { display: block; }
|
| 128 |
+
.combo .opt { padding: 6px 9px; border-radius: 6px; cursor: pointer; display: flex; align-items: center; gap: 8px; white-space: nowrap; }
|
| 129 |
+
.combo .opt:hover, .combo .opt.kbd { background: var(--bg-elev2); }
|
| 130 |
+
.combo .opt.sel { color: var(--accent); }
|
| 131 |
+
.combo .opt small { color: var(--text-faint); margin-left: auto; }
|
| 132 |
+
|
| 133 |
+
/* ---------- Main ---------- */
|
| 134 |
+
main { flex: 1 1 auto; min-height: 0; display: flex; position: relative; }
|
| 135 |
+
.view { display: none; flex: 1; min-height: 0; flex-direction: column; padding: 12px 16px 8px; }
|
| 136 |
+
.view.active { display: flex; }
|
| 137 |
+
|
| 138 |
+
.table-wrap { flex: 1 1 auto; min-height: 0; overflow: auto; border: 1px solid var(--border-soft);
|
| 139 |
+
border-radius: var(--radius); background: var(--panel); }
|
| 140 |
+
table { border-collapse: separate; border-spacing: 0; width: 100%; }
|
| 141 |
+
thead th {
|
| 142 |
+
position: sticky; top: 0; z-index: 5; background: var(--bg-elev);
|
| 143 |
+
text-align: right; padding: 7px 9px; font-size: 11px; font-weight: 600; color: var(--text-dim);
|
| 144 |
+
border-bottom: 1px solid var(--border); white-space: nowrap; cursor: help; user-select: none;
|
| 145 |
+
}
|
| 146 |
+
thead th.lbl, thead th.sortable { cursor: pointer; }
|
| 147 |
+
thead th.txt { text-align: left; }
|
| 148 |
+
thead th .arrow { color: var(--accent); font-size: 9px; margin-left: 3px; }
|
| 149 |
+
tbody td { padding: 4px 9px; text-align: right; font-variant-numeric: tabular-nums;
|
| 150 |
+
border-bottom: 1px solid var(--border-soft); white-space: nowrap; height: var(--row-h); }
|
| 151 |
+
tbody td.txt { text-align: left; font-family: var(--mono); font-size: 12px; }
|
| 152 |
+
tbody tr:hover td { background: rgba(127,150,170,.07); }
|
| 153 |
+
tbody tr.neon td.txt .pipe { font-weight: 600; }
|
| 154 |
+
|
| 155 |
+
.badge { display: inline-block; padding: 1px 6px; border-radius: 999px; font-size: 10px; font-weight: 650;
|
| 156 |
+
letter-spacing: .3px; vertical-align: middle; }
|
| 157 |
+
.badge.neon { background: rgba(90,209,194,.16); color: var(--neon); border: 1px solid rgba(90,209,194,.3); }
|
| 158 |
+
.badge.pareto { background: rgba(120,170,255,.14); color: #8fb6ff; }
|
| 159 |
+
.badge.dq { background: rgba(255,120,90,.13); color: #ff9b7d; border: 1px solid rgba(255,120,90,.25); }
|
| 160 |
+
html[data-theme="light"] .badge.pareto { color: #2f5fc0; }
|
| 161 |
+
.star { color: #8fb6ff; font-size: 11px; }
|
| 162 |
+
.pipe small { color: var(--text-faint); }
|
| 163 |
+
|
| 164 |
+
.verdict { display: inline-block; padding: 2px 8px; border-radius: 6px; font-size: 11px; font-weight: 600; }
|
| 165 |
+
.v-clear { background: rgba(63,182,168,.18); color: var(--neon); }
|
| 166 |
+
.v-inconclusive { background: rgba(190,170,90,.16); color: #d9c178; }
|
| 167 |
+
.v-dq { background: rgba(255,120,90,.13); color: #ff9b7d; }
|
| 168 |
+
.v-none { background: var(--chip-bg); color: var(--text-faint); }
|
| 169 |
+
|
| 170 |
+
/* Picks grid + Metrics grid share matrix styles */
|
| 171 |
+
.matrix-wrap { flex: 1 1 auto; min-height: 0; overflow: auto; border: 1px solid var(--border-soft);
|
| 172 |
+
border-radius: var(--radius); background: var(--panel); }
|
| 173 |
+
.matrix { width: 100%; border-collapse: separate; border-spacing: 0; }
|
| 174 |
+
.matrix th, .matrix td { padding: 8px 10px; border-bottom: 1px solid var(--border-soft); border-right: 1px solid var(--border-soft); }
|
| 175 |
+
.matrix thead th { position: sticky; top: 0; background: var(--bg-elev); z-index: 5; cursor: help; font-size: 11.5px; }
|
| 176 |
+
.matrix tbody th { position: sticky; left: 0; background: var(--bg-elev); z-index: 4; text-align: left; cursor: help; }
|
| 177 |
+
.matrix td { text-align: center; font-variant-numeric: tabular-nums; }
|
| 178 |
+
.matrix .cell-sub { display: block; font-size: 10px; color: var(--text-faint); margin-top: 2px; }
|
| 179 |
+
|
| 180 |
+
/* Models tab */
|
| 181 |
+
.model-page { flex: 1 1 auto; min-height: 0; display: grid; grid-template-columns: 280px 1fr; gap: 14px; }
|
| 182 |
+
.card { background: var(--panel); border: 1px solid var(--border-soft); border-radius: var(--radius); padding: 16px; }
|
| 183 |
+
.card.scroll { overflow: auto; }
|
| 184 |
+
.kv { display: flex; justify-content: space-between; gap: 12px; padding: 6px 0; border-bottom: 1px solid var(--border-soft); font-size: 12.5px; }
|
| 185 |
+
.kv:last-child { border-bottom: 0; }
|
| 186 |
+
.kv .k { color: var(--text-faint); }
|
| 187 |
+
.kv .v { color: var(--text); text-align: right; font-family: var(--mono); }
|
| 188 |
+
.stat-row { display: flex; gap: 10px; flex-wrap: wrap; margin-bottom: 12px; }
|
| 189 |
+
.stat { background: var(--bg-elev2); border: 1px solid var(--border-soft); border-radius: 8px; padding: 9px 13px; min-width: 92px; }
|
| 190 |
+
.stat .n { font-size: 19px; font-weight: 650; font-variant-numeric: tabular-nums; }
|
| 191 |
+
.stat .l { font-size: 10.5px; color: var(--text-faint); text-transform: uppercase; letter-spacing: .5px; cursor: help; }
|
| 192 |
+
|
| 193 |
+
/* Guide tab */
|
| 194 |
+
.guide-grid { flex: 1 1 auto; min-height: 0; display: grid; grid-template-columns: 1fr 1fr; gap: 14px; }
|
| 195 |
+
.guide-col { overflow: auto; }
|
| 196 |
+
.guide-col h3 { margin: 0 0 8px; font-size: 13px; color: var(--accent); }
|
| 197 |
+
.gloss { width: 100%; border-collapse: collapse; }
|
| 198 |
+
.gloss td { padding: 5px 8px; border-bottom: 1px solid var(--border-soft); vertical-align: top; font-size: 12px; }
|
| 199 |
+
.gloss td.term { color: var(--text); font-weight: 600; white-space: nowrap; width: 78px; font-family: var(--mono); }
|
| 200 |
+
.gloss td.def { color: var(--text-dim); }
|
| 201 |
+
.prof { background: var(--bg-elev2); border: 1px solid var(--border-soft); border-radius: 8px; padding: 10px 12px; margin-bottom: 8px; }
|
| 202 |
+
.prof h4 { margin: 0 0 4px; font-size: 12.5px; }
|
| 203 |
+
.prof .gates { display: flex; gap: 6px; flex-wrap: wrap; }
|
| 204 |
+
.gate { background: var(--chip-bg); border-radius: 6px; padding: 2px 8px; font-size: 11px; color: var(--text-dim); font-family: var(--mono); }
|
| 205 |
+
|
| 206 |
+
footer.bottom { flex: 0 0 auto; display: flex; align-items: center; gap: 16px; padding: 4px 16px;
|
| 207 |
+
height: 26px; background: var(--bg-elev); border-top: 1px solid var(--border); font-size: 10.5px; color: var(--text-faint); }
|
| 208 |
+
footer .lg { display: inline-flex; align-items: center; gap: 5px; }
|
| 209 |
+
footer .sw { width: 11px; height: 11px; border-radius: 3px; display: inline-block; }
|
| 210 |
+
|
| 211 |
+
/* tooltip */
|
| 212 |
+
#tip { position: fixed; z-index: 200; max-width: 320px; background: #0a0e12; color: #e7edf3;
|
| 213 |
+
border: 1px solid var(--border); border-radius: 8px; padding: 9px 11px; font-size: 12px; line-height: 1.45;
|
| 214 |
+
box-shadow: var(--shadow); pointer-events: none; opacity: 0; transition: opacity .1s; }
|
| 215 |
+
html[data-theme="light"] #tip { background: #1f2933; color: #f3f5f8; border-color: #1f2933; }
|
| 216 |
+
#tip b { color: var(--neon); }
|
| 217 |
+
|
| 218 |
+
.empty { color: var(--text-faint); padding: 30px; text-align: center; font-size: 13px; }
|
| 219 |
+
::-webkit-scrollbar { width: 10px; height: 10px; }
|
| 220 |
+
::-webkit-scrollbar-thumb { background: var(--border); border-radius: 6px; }
|
| 221 |
+
::-webkit-scrollbar-track { background: transparent; }
|
| 222 |
+
</style>
|
| 223 |
+
</head>
|
| 224 |
+
<body>
|
| 225 |
+
<div class="app">
|
| 226 |
+
<header class="top">
|
| 227 |
+
<div class="brand"><span class="dot"></span><h1>Neon Leaderboard</h1><span class="meta" id="headerMeta"></span></div>
|
| 228 |
+
<nav class="tabs" id="tabs">
|
| 229 |
+
<button data-tab="compare" class="active">Compare</button>
|
| 230 |
+
<button data-tab="picks">Picks</button>
|
| 231 |
+
<button data-tab="models">Models</button>
|
| 232 |
+
<button data-tab="metrics">Metrics</button>
|
| 233 |
+
<button data-tab="guide">Guide</button>
|
| 234 |
+
</nav>
|
| 235 |
+
<span class="spacer"></span>
|
| 236 |
+
<button class="icon-btn" id="themeBtn" title="Toggle light / dark">◐</button>
|
| 237 |
+
</header>
|
| 238 |
+
|
| 239 |
+
<!-- COMPARE -->
|
| 240 |
+
<div class="controls" id="ctl-compare">
|
| 241 |
+
<div class="ctl"><label>View</label>
|
| 242 |
+
<div class="seg" id="cmpModeSeg">
|
| 243 |
+
<button data-mode="byBench" class="active">By benchmark</button>
|
| 244 |
+
<button data-mode="byModel">By model</button>
|
| 245 |
+
</div>
|
| 246 |
+
</div>
|
| 247 |
+
<div class="ctl" id="benchPickWrap"><label>Benchmark</label><div class="combo" id="benchCombo"></div></div>
|
| 248 |
+
<div class="ctl" id="modelPickWrap" style="display:none"><label>Model</label><div class="combo" id="modelCombo"></div></div>
|
| 249 |
+
<div class="ctl"><label>Filter model</label><div class="combo" id="filterCombo"></div></div>
|
| 250 |
+
<div class="ctl"><label>Sort</label><select class="field" id="sortSel"></select>
|
| 251 |
+
<button class="icon-btn" id="sortDir" title="Sort direction" style="width:30px;height:30px">▾</button>
|
| 252 |
+
</div>
|
| 253 |
+
<span class="spacer"></span>
|
| 254 |
+
<label class="toggle"><input type="checkbox" id="neonOnly" /> Neon only</label>
|
| 255 |
+
<label class="toggle"><input type="checkbox" id="showDQ" checked /> Show DQ</label>
|
| 256 |
+
<label class="toggle"><input type="checkbox" id="heat" checked /> Color</label>
|
| 257 |
+
</div>
|
| 258 |
+
|
| 259 |
+
<!-- PICKS -->
|
| 260 |
+
<div class="controls" id="ctl-picks" style="display:none">
|
| 261 |
+
<div class="ctl"><label>Router recommendations</label></div>
|
| 262 |
+
<span class="meta" style="color:var(--text-faint);font-size:11.5px">Pick = model · retriever · reranker per client profile. Hover a profile column for its hard gates.</span>
|
| 263 |
+
</div>
|
| 264 |
+
|
| 265 |
+
<!-- MODELS -->
|
| 266 |
+
<div class="controls" id="ctl-models" style="display:none">
|
| 267 |
+
<div class="ctl"><label>Model</label><div class="combo" id="modelPageCombo"></div></div>
|
| 268 |
+
<span class="meta" id="modelPageBadge"></span>
|
| 269 |
+
</div>
|
| 270 |
+
|
| 271 |
+
<!-- METRICS -->
|
| 272 |
+
<div class="controls" id="ctl-metrics" style="display:none">
|
| 273 |
+
<div class="ctl"><label>Show</label>
|
| 274 |
+
<div class="seg" id="metricModeSeg">
|
| 275 |
+
<button data-mode="scores" class="active">Mean scores</button>
|
| 276 |
+
<button data-mode="winloss">Wins / losses</button>
|
| 277 |
+
</div>
|
| 278 |
+
</div>
|
| 279 |
+
<div class="ctl"><label>Sort by</label><select class="field" id="metricSortSel"></select></div>
|
| 280 |
+
<span class="spacer"></span>
|
| 281 |
+
<label class="toggle"><input type="checkbox" id="metricNeonOnly" /> Neon only</label>
|
| 282 |
+
<label class="toggle"><input type="checkbox" id="metricHeat" checked /> Color</label>
|
| 283 |
+
</div>
|
| 284 |
+
|
| 285 |
+
<!-- GUIDE -->
|
| 286 |
+
<div class="controls" id="ctl-guide" style="display:none">
|
| 287 |
+
<div class="ctl"><label>Reference</label></div>
|
| 288 |
+
<span class="meta" style="color:var(--text-faint);font-size:11.5px">Every column, profile gate, and provenance source behind the numbers.</span>
|
| 289 |
+
</div>
|
| 290 |
+
|
| 291 |
+
<main>
|
| 292 |
+
<section class="view active" id="view-compare"><div class="table-wrap"><table id="cmpTable"></table></div></section>
|
| 293 |
+
<section class="view" id="view-picks"><div class="matrix-wrap"><table class="matrix" id="picksTable"></table></div></section>
|
| 294 |
+
<section class="view" id="view-models"><div class="model-page" id="modelPage"></div></section>
|
| 295 |
+
<section class="view" id="view-metrics"><div class="matrix-wrap"><table class="matrix" id="metricsTable"></table></div></section>
|
| 296 |
+
<section class="view" id="view-guide"><div class="guide-grid" id="guideGrid"></div></section>
|
| 297 |
+
</main>
|
| 298 |
+
|
| 299 |
+
<footer class="bottom">
|
| 300 |
+
<span class="lg"><span class="sw" id="lgGood"></span> higher rank</span>
|
| 301 |
+
<span class="lg"><span class="sw" id="lgBad"></span> lower rank</span>
|
| 302 |
+
<span class="lg"><span class="badge neon" style="padding:0 6px">NEON</span> Neon.ai model</span>
|
| 303 |
+
<span class="lg"><span class="star">★</span> Pareto-optimal</span>
|
| 304 |
+
<span class="lg"><span class="badge dq" style="padding:0 6px">DQ</span> disqualified by a profile gate</span>
|
| 305 |
+
<span class="spacer"></span>
|
| 306 |
+
<span id="provFoot"></span>
|
| 307 |
+
</footer>
|
| 308 |
+
</div>
|
| 309 |
+
<div id="tip"></div>
|
| 310 |
+
<script src="data.js"></script>
|
| 311 |
+
<script src="app.js"></script>
|
| 312 |
+
</body>
|
| 313 |
+
</html>
|
scripts/build_data.py
ADDED
|
@@ -0,0 +1,493 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Build the static dashboard data bundle (``data.js``) for the Neon leaderboard.
|
| 3 |
+
|
| 4 |
+
This reads the leaderboard's own source of truth -- ``run.json`` plus the
|
| 5 |
+
profile YAMLs -- and recomputes picks, the Pareto front, disqualifications,
|
| 6 |
+
verdicts, and per-metric win/loss tallies using logic faithful to
|
| 7 |
+
``router/pareto.py`` and ``router/reports/leaderboard.py`` in the neon-router
|
| 8 |
+
repo. The output is a single ``data.js`` file that assigns
|
| 9 |
+
``window.LEADERBOARD_DATA`` so the dashboard loads with no server / no CORS.
|
| 10 |
+
|
| 11 |
+
Usage:
|
| 12 |
+
python scripts/build_data.py \
|
| 13 |
+
--run-json /path/to/reports/dashboard/raw/run.json \
|
| 14 |
+
--profiles-dir /path/to/configs/profiles \
|
| 15 |
+
--out ../data.js
|
| 16 |
+
|
| 17 |
+
Defaults point at a sibling ``neon-router`` checkout.
|
| 18 |
+
"""
|
| 19 |
+
from __future__ import annotations
|
| 20 |
+
|
| 21 |
+
import argparse
|
| 22 |
+
import json
|
| 23 |
+
import math
|
| 24 |
+
import os
|
| 25 |
+
from dataclasses import dataclass, field
|
| 26 |
+
from datetime import datetime, timezone
|
| 27 |
+
from pathlib import Path
|
| 28 |
+
from typing import Dict, List, Optional, Sequence, Tuple
|
| 29 |
+
|
| 30 |
+
try:
|
| 31 |
+
import yaml
|
| 32 |
+
except ImportError: # pragma: no cover
|
| 33 |
+
raise SystemExit("PyYAML is required: pip install pyyaml")
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
# ---------------------------------------------------------------------------
|
| 37 |
+
# Profile model (mirror of router.pareto.ProfilePreference, subset we need)
|
| 38 |
+
# ---------------------------------------------------------------------------
|
| 39 |
+
@dataclass
|
| 40 |
+
class Profile:
|
| 41 |
+
name: str
|
| 42 |
+
latency_budget_ms: Optional[float] = None
|
| 43 |
+
generation_latency_budget_ms: Optional[float] = None
|
| 44 |
+
cost_ceiling_per_query_usd: Optional[float] = None
|
| 45 |
+
accuracy_floor_mrr: Optional[float] = None
|
| 46 |
+
accuracy_floor_generation_acc: Optional[float] = None
|
| 47 |
+
corpus_profile: Optional[str] = None
|
| 48 |
+
query_mix: Dict[str, float] = field(default_factory=dict)
|
| 49 |
+
deployment_constraint: Optional[str] = None
|
| 50 |
+
quality_weight: float = 0.7
|
| 51 |
+
latency_penalty_per_ms: float = 0.0001
|
| 52 |
+
generation_latency_penalty_per_ms: float = 0.0
|
| 53 |
+
cost_penalty_per_token: float = 0.0
|
| 54 |
+
metric: str = "mrr"
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
# Fields the dashboard reads off each aggregate row.
|
| 58 |
+
NUMERIC_FIELDS = [
|
| 59 |
+
"mrr", "hit_at_1", "hit_at_5", "ndcg_at_10",
|
| 60 |
+
"p50_retrieval_latency_ms", "p95_retrieval_latency_ms",
|
| 61 |
+
"avg_retrieval_latency_ms", "index_build_seconds",
|
| 62 |
+
"est_cost_per_query_usd", "generation_acc",
|
| 63 |
+
"p50_generation_latency_ms", "avg_generation_latency_ms",
|
| 64 |
+
"generation_tokens_per_sec", "est_generation_cost_per_query_usd",
|
| 65 |
+
"paraphrase_drop", "shift_drop", "n_queries",
|
| 66 |
+
]
|
| 67 |
+
|
| 68 |
+
DQ_LATENCY, DQ_COST, DQ_ACCURACY = "latency", "cost", "accuracy"
|
| 69 |
+
|
| 70 |
+
# Models served/fine-tuned by Neon.ai (vs. retrieval-only baselines and any
|
| 71 |
+
# external references). Edit this set to control the "Neon" badge in the UI.
|
| 72 |
+
# Match on the candidate's `model` name (the first ``::`` segment).
|
| 73 |
+
NEON_MODELS = {
|
| 74 |
+
"devstral-123b",
|
| 75 |
+
"ministral-14b-reasoning",
|
| 76 |
+
"magistral-small",
|
| 77 |
+
"devstral-small-24b",
|
| 78 |
+
"mistral-small-3.2-24b",
|
| 79 |
+
"granite-4-30b",
|
| 80 |
+
}
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
def _get(agg: dict, key: str):
|
| 84 |
+
return agg.get(key)
|
| 85 |
+
|
| 86 |
+
|
| 87 |
+
def _acc_axis(agg: dict, metric: str) -> float:
|
| 88 |
+
return float(agg.get(metric) or 0.0)
|
| 89 |
+
|
| 90 |
+
|
| 91 |
+
def _cost_per_query(agg: dict) -> float:
|
| 92 |
+
return float(agg.get("est_cost_per_query_usd") or 0.0)
|
| 93 |
+
|
| 94 |
+
|
| 95 |
+
# ---------------------------------------------------------------------------
|
| 96 |
+
# Faithful port of router.pareto
|
| 97 |
+
# ---------------------------------------------------------------------------
|
| 98 |
+
def apply_hard_constraints(aggs: List[dict], p: Profile) -> Tuple[List[dict], List[Tuple[dict, str]]]:
|
| 99 |
+
survivors: List[dict] = []
|
| 100 |
+
dq: List[Tuple[dict, str]] = []
|
| 101 |
+
for agg in aggs:
|
| 102 |
+
reason: Optional[str] = None
|
| 103 |
+
avg_lat = agg.get("avg_retrieval_latency_ms")
|
| 104 |
+
avg_gen_lat = agg.get("avg_generation_latency_ms")
|
| 105 |
+
if p.latency_budget_ms is not None and avg_lat is not None and avg_lat > p.latency_budget_ms:
|
| 106 |
+
reason = DQ_LATENCY
|
| 107 |
+
elif (
|
| 108 |
+
p.generation_latency_budget_ms is not None
|
| 109 |
+
and avg_gen_lat is not None
|
| 110 |
+
and avg_gen_lat > p.generation_latency_budget_ms
|
| 111 |
+
):
|
| 112 |
+
reason = DQ_LATENCY
|
| 113 |
+
elif (
|
| 114 |
+
p.cost_ceiling_per_query_usd is not None
|
| 115 |
+
and p.cost_ceiling_per_query_usd > 0.0
|
| 116 |
+
and _cost_per_query(agg) > p.cost_ceiling_per_query_usd
|
| 117 |
+
):
|
| 118 |
+
reason = DQ_COST
|
| 119 |
+
elif (
|
| 120 |
+
p.accuracy_floor_generation_acc is not None
|
| 121 |
+
and agg.get("generation_acc") is not None
|
| 122 |
+
and agg.get("generation_acc") < p.accuracy_floor_generation_acc
|
| 123 |
+
):
|
| 124 |
+
reason = DQ_ACCURACY
|
| 125 |
+
elif (
|
| 126 |
+
p.accuracy_floor_mrr is not None
|
| 127 |
+
and p.metric == "mrr"
|
| 128 |
+
and _acc_axis(agg, p.metric) < p.accuracy_floor_mrr
|
| 129 |
+
):
|
| 130 |
+
reason = DQ_ACCURACY
|
| 131 |
+
if reason is None:
|
| 132 |
+
survivors.append(agg)
|
| 133 |
+
else:
|
| 134 |
+
dq.append((agg, reason))
|
| 135 |
+
return survivors, dq
|
| 136 |
+
|
| 137 |
+
|
| 138 |
+
def pareto_front(aggs: List[dict], metric: str = "mrr") -> List[dict]:
|
| 139 |
+
front: List[dict] = []
|
| 140 |
+
for a in aggs:
|
| 141 |
+
dominated = False
|
| 142 |
+
for b in aggs:
|
| 143 |
+
if a is b:
|
| 144 |
+
continue
|
| 145 |
+
a_lat = a.get("avg_retrieval_latency_ms") or 0.0
|
| 146 |
+
b_lat = b.get("avg_retrieval_latency_ms") or 0.0
|
| 147 |
+
a_build = a.get("index_build_seconds") or 0.0
|
| 148 |
+
b_build = b.get("index_build_seconds") or 0.0
|
| 149 |
+
ge = (
|
| 150 |
+
_acc_axis(b, metric) >= _acc_axis(a, metric)
|
| 151 |
+
and b_lat <= a_lat
|
| 152 |
+
and b_build <= a_build
|
| 153 |
+
)
|
| 154 |
+
gt = (
|
| 155 |
+
_acc_axis(b, metric) > _acc_axis(a, metric)
|
| 156 |
+
or b_lat < a_lat
|
| 157 |
+
or b_build < a_build
|
| 158 |
+
)
|
| 159 |
+
if ge and gt:
|
| 160 |
+
dominated = True
|
| 161 |
+
break
|
| 162 |
+
if not dominated:
|
| 163 |
+
front.append(a)
|
| 164 |
+
return front
|
| 165 |
+
|
| 166 |
+
|
| 167 |
+
def select_winner(aggs: List[dict], p: Profile) -> Optional[dict]:
|
| 168 |
+
survivors, _ = apply_hard_constraints(aggs, p)
|
| 169 |
+
front = pareto_front(survivors, metric=p.metric)
|
| 170 |
+
if not front:
|
| 171 |
+
return None
|
| 172 |
+
|
| 173 |
+
def utility(agg: dict) -> float:
|
| 174 |
+
accuracy = _acc_axis(agg, p.metric)
|
| 175 |
+
lat = agg.get("avg_retrieval_latency_ms") or 0.0
|
| 176 |
+
gen_lat = agg.get("avg_generation_latency_ms") or 0.0
|
| 177 |
+
return (
|
| 178 |
+
p.quality_weight * accuracy
|
| 179 |
+
- p.latency_penalty_per_ms * lat
|
| 180 |
+
- p.generation_latency_penalty_per_ms * gen_lat
|
| 181 |
+
)
|
| 182 |
+
|
| 183 |
+
return sorted(front, key=utility, reverse=True)[0]
|
| 184 |
+
|
| 185 |
+
|
| 186 |
+
def _wilson_ci(n_succ: float, n_total: int, z: float = 1.96) -> Tuple[float, float]:
|
| 187 |
+
if n_total == 0:
|
| 188 |
+
return (0.0, 0.0)
|
| 189 |
+
pp = n_succ / n_total
|
| 190 |
+
denom = 1.0 + z * z / n_total
|
| 191 |
+
centre = (pp + z * z / (2.0 * n_total)) / denom
|
| 192 |
+
margin = z * math.sqrt(pp * (1.0 - pp) / n_total + z * z / (4.0 * n_total * n_total)) / denom
|
| 193 |
+
return (max(0.0, centre - margin), min(1.0, centre + margin))
|
| 194 |
+
|
| 195 |
+
|
| 196 |
+
def value_add_verdict(aggs: List[dict], p: Profile) -> str:
|
| 197 |
+
survivors, _ = apply_hard_constraints(aggs, p)
|
| 198 |
+
if len(survivors) < 2:
|
| 199 |
+
return "no signal"
|
| 200 |
+
|
| 201 |
+
def key(a: dict) -> float:
|
| 202 |
+
if p.metric == "generation_acc" and a.get("generation_acc") is not None:
|
| 203 |
+
return a["generation_acc"]
|
| 204 |
+
return a.get("mrr") or 0.0
|
| 205 |
+
|
| 206 |
+
ranked = sorted(survivors, key=key, reverse=True)
|
| 207 |
+
winner, runner = ranked[0], ranked[1]
|
| 208 |
+
n = int(winner.get("n_queries") or 0)
|
| 209 |
+
w_lo, _ = _wilson_ci(key(winner) * n, n)
|
| 210 |
+
_, r_hi = _wilson_ci(key(runner) * n, n)
|
| 211 |
+
return "winner clear" if w_lo > r_hi else "inconclusive"
|
| 212 |
+
|
| 213 |
+
|
| 214 |
+
# ---------------------------------------------------------------------------
|
| 215 |
+
# Loaders
|
| 216 |
+
# ---------------------------------------------------------------------------
|
| 217 |
+
def load_profiles(profiles_dir: Path) -> List[Profile]:
|
| 218 |
+
valid = set(Profile.__dataclass_fields__)
|
| 219 |
+
profiles: List[Profile] = []
|
| 220 |
+
for path in sorted(profiles_dir.glob("*.yaml")):
|
| 221 |
+
raw = yaml.safe_load(path.read_text(encoding="utf-8")) or {}
|
| 222 |
+
clean = {k: v for k, v in raw.items() if k in valid}
|
| 223 |
+
profiles.append(Profile(**clean))
|
| 224 |
+
return profiles
|
| 225 |
+
|
| 226 |
+
|
| 227 |
+
def split_candidate(cid: str) -> Tuple[str, str, str]:
|
| 228 |
+
parts = cid.split("::")
|
| 229 |
+
return (parts[0], parts[1], parts[2]) if len(parts) == 3 else (cid, "", "")
|
| 230 |
+
|
| 231 |
+
|
| 232 |
+
# Mirror of leaderboard.py _GLOSSARY (kept in sync with the source report).
|
| 233 |
+
GLOSSARY = {
|
| 234 |
+
"model": "The generation model in the pipeline. 'retrieval-only' means no LLM generation step was run for this row.",
|
| 235 |
+
"retriever": "Retrieval strategy used to fetch candidate documents (e.g. hybrid = BM25 + dense embeddings; bm25-only = lexical; splade-only = learned sparse).",
|
| 236 |
+
"rerank": "Cross-encoder reranker applied after retrieval ('none' = no reranking; minilm / mxbai are reranker models).",
|
| 237 |
+
"MRR": "Mean Reciprocal Rank \u2014 average of 1/rank for the first relevant doc retrieved per query. Range 0\u20131; higher is better.",
|
| 238 |
+
"H@1": "Hit@1 \u2014 fraction of queries where the top-1 retrieved doc is relevant. Higher is better.",
|
| 239 |
+
"H@5": "Hit@5 \u2014 fraction of queries where any of the top-5 retrieved docs is relevant. Higher is better.",
|
| 240 |
+
"nDCG": "Normalized Discounted Cumulative Gain @10 \u2014 position-weighted relevance of the top-10 results. Higher is better.",
|
| 241 |
+
"P50": "Median (50th percentile) per-query retrieval latency in milliseconds. Lower is better.",
|
| 242 |
+
"P95": "95th percentile retrieval latency \u2014 the tail (worst-case) cost per query, in milliseconds. Lower is better.",
|
| 243 |
+
"build": "Index build time in seconds for this benchmark corpus. Lower is better.",
|
| 244 |
+
"gen": "Generation score (task-dependent): for multiple-choice benchmarks, the fraction of questions answered correctly given retrieved context; for the language-following benchmark, the fraction of responses written in the user's language. Higher is better.",
|
| 245 |
+
"gen ms": "Median per-query generation latency (time-to-first-token + generation) in milliseconds. Lower is better.",
|
| 246 |
+
"tok/s": "LLM output tokens generated per second on this benchmark. Higher is better.",
|
| 247 |
+
"gen $": "Estimated generation cost per query in USD (token usage \u00d7 model price). Lower is better.",
|
| 248 |
+
"$/q": "Total estimated cost per query in USD (retrieval + generation). Lower is better.",
|
| 249 |
+
"par": "Paraphrase robustness \u2014 MRR on paraphrased queries minus MRR on originals. Near zero is expected; large negative means fragile exact-match retrieval.",
|
| 250 |
+
"shift": "Held-out (domain-shifted) slice MRR \u2014 measures out-of-distribution generalization on an unseen query subset.",
|
| 251 |
+
"n": "Number of queries evaluated for this row.",
|
| 252 |
+
"verdict": "Whether the winning candidate is statistically distinct from the runner-up. 'winner clear' = winner's Wilson CI lower bound exceeds runner-up's upper bound; 'inconclusive' = confidence intervals overlap (within noise); 'no signal' = too few valid candidates; 'all disqualified' = every candidate failed a hard gate.",
|
| 253 |
+
"Pareto": "Pareto front \u2014 a pipeline not dominated by any other on (accuracy, latency, build-cost) simultaneously. Marked with \u2605.",
|
| 254 |
+
"DQ": "Disqualified \u2014 violated one of the profile's hard gates: DQ:latency (p50 over budget), DQ:cost (cost per query over ceiling), or DQ:accuracy (MRR or generation score below the floor).",
|
| 255 |
+
"pick": "The router's single recommended pipeline (model / retriever / reranker) for this client profile on this benchmark.",
|
| 256 |
+
"profile": "A client requirements file (not a model) declaring latency / cost / accuracy hard gates plus utility weights used to rank surviving candidates.",
|
| 257 |
+
}
|
| 258 |
+
|
| 259 |
+
# Per-metric definitions used by the Compare and Metrics tabs.
|
| 260 |
+
# key -> (label, field, higher_is_better, fmt, tooltip-key)
|
| 261 |
+
METRICS = [
|
| 262 |
+
("mrr", "MRR", "mrr", True, "f3", "MRR"),
|
| 263 |
+
("h1", "H@1", "hit_at_1", True, "f2", "H@1"),
|
| 264 |
+
("h5", "H@5", "hit_at_5", True, "f2", "H@5"),
|
| 265 |
+
("ndcg", "nDCG", "ndcg_at_10", True, "f3", "nDCG"),
|
| 266 |
+
("p50", "P50 ms", "p50_retrieval_latency_ms", False, "f0", "P50"),
|
| 267 |
+
("p95", "P95 ms", "p95_retrieval_latency_ms", False, "f0", "P95"),
|
| 268 |
+
("build", "Build s", "index_build_seconds", False, "f1", "build"),
|
| 269 |
+
("gen", "gen", "generation_acc", True, "f3", "gen"),
|
| 270 |
+
("gen_ms", "gen ms", "p50_generation_latency_ms", False, "f0", "gen ms"),
|
| 271 |
+
("tok_s", "tok/s", "generation_tokens_per_sec", True, "f0", "tok/s"),
|
| 272 |
+
("gen_cost", "gen $", "est_generation_cost_per_query_usd", False, "f5", "gen $"),
|
| 273 |
+
("cost", "$/q", "est_cost_per_query_usd", False, "f5", "$/q"),
|
| 274 |
+
("par", "par", "paraphrase_drop", True, "f3", "par"),
|
| 275 |
+
("shift", "shift", "shift_drop", True, "f3", "shift"),
|
| 276 |
+
]
|
| 277 |
+
|
| 278 |
+
|
| 279 |
+
def build(run_json: Path, profiles_dir: Path) -> dict:
|
| 280 |
+
raw = json.loads(run_json.read_text(encoding="utf-8"))
|
| 281 |
+
profiles = load_profiles(profiles_dir)
|
| 282 |
+
benchmarks: List[str] = list(raw.get("benchmarks", []))
|
| 283 |
+
aggregates: List[dict] = raw.get("aggregates", [])
|
| 284 |
+
candidates_meta: List[dict] = raw.get("candidates", [])
|
| 285 |
+
|
| 286 |
+
# Candidate metadata map keyed by candidate_id.
|
| 287 |
+
cand_map: Dict[str, dict] = {}
|
| 288 |
+
for c in candidates_meta:
|
| 289 |
+
llm = c.get("llm", {}) or {}
|
| 290 |
+
retr = c.get("retriever", {}) or {}
|
| 291 |
+
rer = c.get("reranker", {}) or {}
|
| 292 |
+
cid = f"{llm.get('name')}::{retr.get('name')}::{rer.get('name')}"
|
| 293 |
+
cand_map[cid] = {
|
| 294 |
+
"id": cid,
|
| 295 |
+
"model": llm.get("name"),
|
| 296 |
+
"model_adapter": llm.get("adapter"),
|
| 297 |
+
"model_repo": llm.get("model"),
|
| 298 |
+
"retriever": retr.get("name"),
|
| 299 |
+
"retriever_adapter": retr.get("adapter"),
|
| 300 |
+
"reranker": rer.get("name"),
|
| 301 |
+
"reranker_model": rer.get("model"),
|
| 302 |
+
"is_neon": llm.get("name") in NEON_MODELS,
|
| 303 |
+
"is_retrieval_only": llm.get("name") == "retrieval-only",
|
| 304 |
+
}
|
| 305 |
+
|
| 306 |
+
# Trim aggregates to the fields the dashboard needs + identity.
|
| 307 |
+
def trim(agg: dict) -> dict:
|
| 308 |
+
model, retriever, reranker = split_candidate(agg["candidate_id"])
|
| 309 |
+
out = {
|
| 310 |
+
"candidate_id": agg["candidate_id"],
|
| 311 |
+
"benchmark": agg["benchmark"],
|
| 312 |
+
"model": model,
|
| 313 |
+
"retriever": retriever,
|
| 314 |
+
"reranker": reranker,
|
| 315 |
+
"is_neon": model in NEON_MODELS,
|
| 316 |
+
"error": agg.get("error"),
|
| 317 |
+
}
|
| 318 |
+
for f in NUMERIC_FIELDS:
|
| 319 |
+
out[f] = agg.get(f)
|
| 320 |
+
return out
|
| 321 |
+
|
| 322 |
+
trimmed = [trim(a) for a in aggregates]
|
| 323 |
+
|
| 324 |
+
by_bench: Dict[str, List[dict]] = {}
|
| 325 |
+
for a in trimmed:
|
| 326 |
+
by_bench.setdefault(a["benchmark"], []).append(a)
|
| 327 |
+
|
| 328 |
+
# ----- Picks (per benchmark x profile) -----
|
| 329 |
+
picks: Dict[str, Dict[str, dict]] = {}
|
| 330 |
+
for bench in benchmarks:
|
| 331 |
+
aggs = by_bench.get(bench, [])
|
| 332 |
+
picks[bench] = {}
|
| 333 |
+
for p in profiles:
|
| 334 |
+
winner = select_winner(aggs, p)
|
| 335 |
+
if winner is None:
|
| 336 |
+
survivors, _ = apply_hard_constraints(aggs, p)
|
| 337 |
+
status = "all disqualified" if not survivors else "no winner"
|
| 338 |
+
picks[bench][p.name] = {"pick": None, "verdict": status}
|
| 339 |
+
continue
|
| 340 |
+
verdict = value_add_verdict(aggs, p)
|
| 341 |
+
metric_val = (
|
| 342 |
+
winner.get("generation_acc")
|
| 343 |
+
if p.metric == "generation_acc"
|
| 344 |
+
else winner.get("hit_at_1")
|
| 345 |
+
)
|
| 346 |
+
picks[bench][p.name] = {
|
| 347 |
+
"pick": winner["candidate_id"],
|
| 348 |
+
"model": winner["model"],
|
| 349 |
+
"retriever": winner["retriever"],
|
| 350 |
+
"reranker": winner["reranker"],
|
| 351 |
+
"mrr": winner.get("mrr"),
|
| 352 |
+
"metric": p.metric,
|
| 353 |
+
"metric_val": metric_val,
|
| 354 |
+
"p50": winner.get("p50_retrieval_latency_ms"),
|
| 355 |
+
"verdict": verdict,
|
| 356 |
+
}
|
| 357 |
+
|
| 358 |
+
# ----- Per-benchmark ordering, Pareto, DQ -----
|
| 359 |
+
bench_tables: Dict[str, dict] = {}
|
| 360 |
+
for bench in benchmarks:
|
| 361 |
+
aggs = sorted(by_bench.get(bench, []), key=lambda a: a.get("mrr") or 0.0, reverse=True)
|
| 362 |
+
front_ids = {a["candidate_id"] for a in pareto_front(aggs)}
|
| 363 |
+
dq_by_id: Dict[str, str] = {}
|
| 364 |
+
for p in profiles:
|
| 365 |
+
_, dqs = apply_hard_constraints(aggs, p)
|
| 366 |
+
for agg, reason in dqs:
|
| 367 |
+
dq_by_id.setdefault(agg["candidate_id"], reason)
|
| 368 |
+
bench_tables[bench] = {
|
| 369 |
+
"order": [a["candidate_id"] for a in aggs],
|
| 370 |
+
"pareto": sorted(front_ids),
|
| 371 |
+
"dq": dq_by_id,
|
| 372 |
+
}
|
| 373 |
+
|
| 374 |
+
# ----- Per-metric win / loss tally across benchmarks -----
|
| 375 |
+
EPS = 1e-9
|
| 376 |
+
metric_stats: Dict[str, dict] = {}
|
| 377 |
+
for key, label, fieldname, higher, _fmt, _tip in METRICS:
|
| 378 |
+
per_cand: Dict[str, dict] = {}
|
| 379 |
+
for bench in benchmarks:
|
| 380 |
+
vals = [
|
| 381 |
+
(a["candidate_id"], a.get(fieldname))
|
| 382 |
+
for a in by_bench.get(bench, [])
|
| 383 |
+
if a.get(fieldname) is not None
|
| 384 |
+
]
|
| 385 |
+
if not vals:
|
| 386 |
+
continue
|
| 387 |
+
best = (max if higher else min)(v for _, v in vals)
|
| 388 |
+
worst = (min if higher else max)(v for _, v in vals)
|
| 389 |
+
for cid, v in vals:
|
| 390 |
+
s = per_cand.setdefault(cid, {"wins": 0, "losses": 0, "sum": 0.0, "count": 0})
|
| 391 |
+
s["sum"] += v
|
| 392 |
+
s["count"] += 1
|
| 393 |
+
if abs(v - best) <= EPS:
|
| 394 |
+
s["wins"] += 1
|
| 395 |
+
if abs(v - worst) <= EPS and abs(best - worst) > EPS:
|
| 396 |
+
s["losses"] += 1
|
| 397 |
+
for cid, s in per_cand.items():
|
| 398 |
+
s["mean"] = s["sum"] / s["count"] if s["count"] else None
|
| 399 |
+
del s["sum"]
|
| 400 |
+
metric_stats[key] = per_cand
|
| 401 |
+
|
| 402 |
+
profiles_out = []
|
| 403 |
+
for p in profiles:
|
| 404 |
+
gates = []
|
| 405 |
+
if p.latency_budget_ms:
|
| 406 |
+
gates.append(f"p50 \u2264 {p.latency_budget_ms:g} ms")
|
| 407 |
+
if p.generation_latency_budget_ms:
|
| 408 |
+
gates.append(f"gen \u2264 {p.generation_latency_budget_ms:g} ms")
|
| 409 |
+
if p.cost_ceiling_per_query_usd:
|
| 410 |
+
gates.append(f"$/q \u2264 {p.cost_ceiling_per_query_usd:g}")
|
| 411 |
+
if p.accuracy_floor_mrr:
|
| 412 |
+
gates.append(f"MRR \u2265 {p.accuracy_floor_mrr:g}")
|
| 413 |
+
if p.accuracy_floor_generation_acc:
|
| 414 |
+
gates.append(f"gen \u2265 {p.accuracy_floor_generation_acc:g}")
|
| 415 |
+
profiles_out.append({
|
| 416 |
+
"name": p.name,
|
| 417 |
+
"metric": p.metric,
|
| 418 |
+
"gates": gates,
|
| 419 |
+
"corpus_profile": p.corpus_profile,
|
| 420 |
+
"query_mix": p.query_mix,
|
| 421 |
+
"deployment_constraint": p.deployment_constraint,
|
| 422 |
+
"quality_weight": p.quality_weight,
|
| 423 |
+
})
|
| 424 |
+
|
| 425 |
+
header = raw.get("provenance", {}) or {}
|
| 426 |
+
date_str = ""
|
| 427 |
+
merged_at = header.get("merged_at")
|
| 428 |
+
if merged_at:
|
| 429 |
+
date_str = merged_at[:10]
|
| 430 |
+
|
| 431 |
+
return {
|
| 432 |
+
"generated_at": datetime.now(timezone.utc).isoformat(),
|
| 433 |
+
"date": date_str,
|
| 434 |
+
"benchmarks": benchmarks,
|
| 435 |
+
"candidates": cand_map,
|
| 436 |
+
"aggregates": trimmed,
|
| 437 |
+
"profiles": profiles_out,
|
| 438 |
+
"picks": picks,
|
| 439 |
+
"bench_tables": bench_tables,
|
| 440 |
+
"metric_stats": metric_stats,
|
| 441 |
+
"metrics_def": [
|
| 442 |
+
{"key": k, "label": l, "field": f, "higher": h, "fmt": fmt, "tip": t}
|
| 443 |
+
for (k, l, f, h, fmt, t) in METRICS
|
| 444 |
+
],
|
| 445 |
+
"glossary": GLOSSARY,
|
| 446 |
+
"provenance": header,
|
| 447 |
+
"n_candidates": len({a["candidate_id"] for a in trimmed}),
|
| 448 |
+
"n_queries": (trimmed[0].get("n_queries") if trimmed else None),
|
| 449 |
+
}
|
| 450 |
+
|
| 451 |
+
|
| 452 |
+
def main() -> None:
|
| 453 |
+
here = Path(__file__).resolve().parent
|
| 454 |
+
default_repo = here.parent.parent / "neon-router"
|
| 455 |
+
ap = argparse.ArgumentParser()
|
| 456 |
+
ap.add_argument(
|
| 457 |
+
"--run-json",
|
| 458 |
+
type=Path,
|
| 459 |
+
default=default_repo / "reports" / "dashboard" / "raw" / "run.json",
|
| 460 |
+
)
|
| 461 |
+
ap.add_argument(
|
| 462 |
+
"--profiles-dir",
|
| 463 |
+
type=Path,
|
| 464 |
+
default=default_repo / "configs" / "profiles",
|
| 465 |
+
)
|
| 466 |
+
ap.add_argument("--out", type=Path, default=here.parent / "data.js")
|
| 467 |
+
args = ap.parse_args()
|
| 468 |
+
|
| 469 |
+
if not args.run_json.exists():
|
| 470 |
+
raise SystemExit(f"run.json not found: {args.run_json}")
|
| 471 |
+
if not args.profiles_dir.exists():
|
| 472 |
+
raise SystemExit(f"profiles dir not found: {args.profiles_dir}")
|
| 473 |
+
|
| 474 |
+
data = build(args.run_json, args.profiles_dir)
|
| 475 |
+
payload = json.dumps(data, ensure_ascii=False, separators=(",", ":"))
|
| 476 |
+
args.out.write_text(
|
| 477 |
+
"// Auto-generated by scripts/build_data.py - do not edit by hand.\n"
|
| 478 |
+
"window.LEADERBOARD_DATA = " + payload + ";\n",
|
| 479 |
+
encoding="utf-8",
|
| 480 |
+
)
|
| 481 |
+
# Also emit a plain JSON sidecar for non-browser consumers.
|
| 482 |
+
(args.out.parent / "data.json").write_text(
|
| 483 |
+
json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8"
|
| 484 |
+
)
|
| 485 |
+
print(f"Wrote {args.out} ({args.out.stat().st_size/1024:.1f} KB)")
|
| 486 |
+
print(f" benchmarks={len(data['benchmarks'])} "
|
| 487 |
+
f"candidates={data['n_candidates']} "
|
| 488 |
+
f"aggregates={len(data['aggregates'])} "
|
| 489 |
+
f"profiles={len(data['profiles'])}")
|
| 490 |
+
|
| 491 |
+
|
| 492 |
+
if __name__ == "__main__":
|
| 493 |
+
main()
|