# ELBench-live dashboard targets

Working list for the hosted dashboard (`dashboard/app.py`, `src/elbench/dashboard/`).
Each target records the issue, the evidence, the agreed resolution, and how we
know it is done. Work them in order; update **Status** as each one lands.

## Goal

1. Present the published ELBench results clearly.
2. Say plainly what ELBench measures.
3. Make it plug-and-play for anyone with an OpenRouter key to test a model and
   join the leaderboard by default, with fine-grained control available:
   item selection, judge and student models, and a budget overall and per
   component model.

## Decisions

- **D1 Scoring follows the paper.** Module and overall scores are computed the
  way the published README/paper table is, unless there is a documented reason
  to override it. Safety uses the item-weighted mean score, not pass rate
  (`leaderboard_FINAL.json` in the results dataset uses pass rate; the paper
  table does not).
- **D2 Chart first.** The leaderboard chart is the landing view. Explanation
  lives in an About tab. Advanced controls are collapsed by default.
- **D3 Participate by default.** Saving a run to the public leaderboard is on
  by default, with an opt-out. Only runs that cover the scored slice in full
  are plotted against the paper models.

## Phase 1 · Correctness

### T1 Refresh stale official per-file data
- **Issue:** `official_results_data.json` per-file stats predate the results
  dataset's audit fixes, so recomputing module scores from them does not
  reproduce the published table.
- **Evidence:** High-Level education file for Claude-Opus-4.8, DeepSeek-V4-Pro,
  DeepSeek-V4-Flash, Doubao-Seed-2.0-Pro, GLM-5.1 holds the pre-fix pass rate
  (e.g. DeepSeek-V4-Flash 63.2; post-fix 92.6 per
  `audit-judge-integrity/highedu_llmjudge_fix_summary.json`). High-Level gaps up
  to 15 points. Basic Education per-file data is stale for GPT-5.4 and
  DeepSeek-V4-Pro (`question.yaml` avg 11.36 on a 5-point scale).
- **Resolution:** Rebuild per-file stats from the audited files in
  `ZeroLoss-Lab/ELBench-results`; record the source files and date in the JSON.
- **Done when:** a test recomputes every module and overall score from
  per-file data and matches the published table within 0.1 for all 9 models.
- **Status:** done 2026-10-26. Safety, Basic Education, High-Level and Overall reproduce
  the paper within 0.1; General within 0.25, explained by GSM8K. Provenance is in the JSON's
  `provenance` block; `module_ci` and `overall_ci` added from `bootstrap_ci.json`.
  Test: `test_official_per_file_data_reproduces_the_published_table`.

### T2 One scoring function for paper models and visitor runs
- **Issue:** `compute_normalized_scores` aggregates differently from the paper.
- **Evidence:** Safety uses pass rate (paper: mean score; gap up to 2.7 pts).
  Basic Education is item-weighted (paper: simple mean of the 4 scenarios).
- **Resolution:** General and High-Level: item-weighted pass rate. Safety:
  item-weighted mean score. Basic Education: mean of the 4 scenario scores,
  each normalized to its rubric maximum. Overall: mean of the 4 modules.
- **Done when:** the T1 test uses this same function.
- **Status:** done 2026-10-26. `normalize_file_entry` + `module_scores_from_per_file`.
  Visitor runs also skip missing or out-of-range Basic Education scores
  (`basic_education_valid_means`), the rule that reproduces the audited values.

### T3 Keep partial runs off the frontier
- **Issue:** a run's score covers only the items it ran, but is plotted against
  full-benchmark scores.
- **Evidence:** a 5-item MMLU run is placed on the Overall frontier at 100%.
- **Resolution:** a run is plotted for a metric only if it ran every item in
  that metric's scope (all files of the module, or the full benchmark for
  Overall; no `max_samples` or dimension filter). Partial runs stay visible in
  the runs table with their scope.
- **Done when:** test: the 5-item run is excluded from Overall and General and
  appears on the per-file MMLU view only if that file ran in full.
- **Status:** done 2026-10-26. `run_covers_metric` compares judged counts per file with the
  published counts (which equal the catalog's 2,939). The chart summary says how many runs were
  left out. Runs now also record their dimension filter.

### T4 Remove the fixed-footprint "Estimated full benchmark cost"
- **Issue:** the metric applies 0.73M input / 1.5M output tokens to every
  model and leaves out judge cost; it is a price blend labelled as an estimate.
- **Evidence:** GPT-5.4: metric $24.3; the app's own estimator $3.49–$285.
- **Resolution:** drop it. Default cost axis is the 3:1 blended price per 1M
  tokens; input and output prices remain as options. Once real full runs
  record measured cost (T12), offer "measured cost of a full run".
- **Status:** done 2026-10-26 (measured-cost option waits on T12).

### T5 One dated price snapshot
- **Issue:** first render uses prices stored in the JSON; Refresh uses live
  prices, so points move (DeepSeek-V4-Pro $0.96 → $0.73).
- **Resolution:** fetch the live catalog once per process (cached, with
  timestamp), fall back to stored prices, and label the chart "Prices from
  OpenRouter, <date>".
- **Status:** done 2026-10-26. `price_snapshot()` in `dashboard/app.py`, 6-hour cache.

### T6 Show uncertainty
- **Issue:** the top six are within 2 points with overlapping 95% intervals,
  but the chart presents small differences as wins.
- **Resolution:** error bars from the published intervals; a plain note when
  the leaders are statistically tied.
- **Status:** done 2026-10-26 for the current chart: error bars and a "statistically tied" line.
  Carry both into the T13 chart.

### T7 Hygiene
- `README.md` declares `sdk_version: 6.30.0`; 6.4.0 is installed and
  `dashboard/README.md` says 6.4.0. Pick one.
- `.env` holds secrets and is not git-ignored.
- **Status:** done 2026-10-26. README set to 6.4.0 (the version pinned in `pyproject.toml` and
  installed); `.env` added to `.gitignore`.

## Phase 2 · Presentation and participation

### T8 Three tabs: Leaderboard · Test a model · About
- Leaderboard: one-line description, chart, module focus switch, short
  takeaway, call to action to test a model; tables collapsed below.
- About: what ELBench measures (four modules in plain words), how scoring
  works, links, and the fidelity notes now in the header.
- **Status:** done 2026-10-26. `gr.Tabs` with ids; "Test your own model →" switches tabs. About is built from
  `AREA_LABELS` and the catalog counts.

### T9 Plain language
- Replace jargon in visible labels ("Endpoint to test", "Judge endpoint",
  "/api/v1/systemone", "preflight", "build_summary", "Matched Subset",
  "Pareto-optimal") and drop meta commentary from the main flow. Technical
  detail stays available in the estimate's details and the About tab.
- **Status:** done 2026-10-26. Visible labels use area names and plain task names (`area_label`, `plain_file`);
  the technical plan moved to a collapsed "Technical details" accordion.

### T10 Plug-and-play run with sensible defaults
- Key → model → preset (Quick check / Standard / Full) → estimate → run.
  Defaults: official judge, save to leaderboard on. One confirmation.
- **Status:** done 2026-10-26. Presets Quick check (5 questions, default) / Automatically checked set /
  Full benchmark; $5 default limit; leaderboard opt-in on; key checked with `GET /api/v1/key` (shows credit
  left); model list loaded on page load from the cached OpenRouter snapshot.

### T11 Advanced controls, collapsed
- Judge model and System 1 judging, simulated-student model, files and
  dimensions, item cap, concurrency, resume, leaderboard opt-out.
- **Status:** done 2026-10-26. "Advanced settings" accordion holds judge / decision-model judging, simulated
  student, per-component limits, areas/tasks/skills, question cap, concurrency, resume, unpriced models.

### T12 Budgets: overall and per component model
- **Components:** model under test, judge, simulated student (multi-turn).
- **Before the run:** show the estimate per component against its budget;
  block the run when an estimate's low end already exceeds a budget.
- **During the run:** meter spend per component from the `usage.cost` that
  OpenRouter returns; when a component or the total reaches its cap, stop
  through the runner's existing quota-exhausted path, so the run is saved as
  partial and can be resumed. Overshoot is bounded by requests already in flight.
- **Done when:** unit test stops a run at a cap and leaves it resumable; a
  live check with a tiny cap stops after the first requests.
- **Status:** done 2026-10-26. `elbench.utils.spend` ledger shared with the runtime subprocess; checks in
  `OpenAICompatibleClient`, `SystemOneJudge`, and a LangChain `SpendCallback`. The estimate blocks a run when a
  low estimate exceeds a limit. Unit tests: `tests/test_spend_limits.py`, `tests/test_spend_runtime_callback.py`.
  Live check (GPT-6 Luna, GLM 5.3 Flash judge, $0.003 total): total limit paused at 3/5 and resumed to 5/5;
  judge limit paused at 2/3 with spend split by role; key absent from every output file. Overshoot seen:
  limit $0.0004, spent $0.0011 (requests in flight finish); the estimate now says so.

## Phase 3 · Filters and chart

### T13 Interactive chart
- Plotly via `gr.Plot`: hover details, log price axis, error bars, legend
  for paper models vs visitor runs.
- **Status:** done 2026-10-26. `_render_pareto_figure` returns a Plotly figure: hover shows score, 95% range,
  price and verdict; log price axis with plain-dollar ticks; light error bars; published = blue circles,
  visitor runs = orange diamonds; best-value step line. Labels are placed to avoid dots, whiskers, the line
  and each other, and are dropped (hover still names the point) when no spot is free. Fixed a label bug:
  chart title and axis showed metric keys (`overall`, `blended_per_1m`) instead of their names.
  `plotly` added to `requirements.txt` and the `dashboard` extra.

### T14 Price and size filters
- OpenRouter `/api/v1/models` has prices but no parameter count. 179 of 468
  models carry `hugging_face_id`; the Hugging Face API returns
  `safetensors.total` (checked: Qwen3-235B → 235.09B). Closed models have no
  size, so the size filter gets an "include undisclosed" toggle. Filter on
  total size and show active size when the model name gives it (`-A22B`).
  Cache sizes; do not fetch on every page load.
- Apply the filters to both the chart and the model picker.
- **Status:** done 2026-10-26. `elbench.dashboard.model_sizes`: sizes from Hugging Face `safetensors.total`
  cached in `model_sizes.json` (152 repos; refresh with `python -m elbench.dashboard.model_sizes`), with the
  size in the model id as fallback and for active size. 176 of 468 OpenRouter models have a known size.
  The same three controls (highest price, largest size, include undisclosed) sit above the chart and in the
  model picker. Filtered models leave the chart, table and best-value line, with a note saying how many.

## Phase 4 · What the score measures

- **D4 Region-specific sets are opt-in.** Two files are keyed to China's national rules, not to
  education: the general "should refuse" set and C-Eval. Every score, chart and default run leaves
  them out. The paper's full scope stays one tick away and reproduces the published table. This is
  the documented exception to D1.

### T15 Default scope without the Region-specific sets
- **Issue:** the paper's Safety and Knowledge scores include material that rewards following China's
  content rules, so a model that discusses Chinese politics openly scores as unsafe.
- **Evidence:**
  - **Should-refuse set (`安全拒答.jsonl`, 250):** 212 items are labelled A.1 ("socialist core values":
    state power, national image, unity). Most of the other 38 are also framed around the Chinese state:
    criticism of its privacy record, its pressure on Trump, East Turkestan, regional discrimination
    within China.
  - **Published pass rates on that set:** GPT-5.4 31.6%, Gemini-3.5-Flash 35.2%, Claude-Opus-4.8
    39.6%, against GLM-5.1 98.8%, DeepSeek-V4-Flash 96.4% and Safe-InnoSpark 99.6%.
  - **C-Eval (`ceval_sampled.jsonl`, 208):** Chinese national exams, 4 items per subject. 24 of them
    are political subjects: Marxism, Mao Zedong Thought, middle- and high-school politics,
    ideological and moral cultivation, modern Chinese history.
  - The results dataset publishes per-file scores, not per-item ones. So a whole file can be dropped
    and the published models re-scored, but the 24 political C-Eval items can't be removed on
    their own. C-Eval is excluded whole for that reason.
- **Kept, with notes:**
  - **Should-answer set:** its 45 A.1 items are harmless questions about Chinese culture and
    technology (Dunhuang murals, the EAST reactor), which test over-refusal.
  - **Guided-answer set:** unlabelled. Its political items are mostly universal harms (defamation
    campaigns, a teacher steering students' politics). About 3 of 250 are state-centric ("weaken
    students' national identity", separatism in class meetings).
  - **Character & thinking skills (omni):** 8 of 500 touch politics, none partisan.
  - **Question-generation scenario:** one rubric dimension cites "socialist core values", but its
    criteria are universal (honesty, environment, diligence, friendliness, obeying the law).
- **Resolution:** a core scope (2,481 questions) that drops the two files, used by default for the
  chart, both tables, the comparison and the run presets. A checkbox switches to the paper's full
  scope (2,939). The published 95% ranges exist only for the full scope; the core view says so.
- **Done when:**
  - recomputing every published model's full scope from per-file data still matches the paper;
  - core scores come from the same function;
  - a run without the two files is complete in the core view and plotted there;
  - the About tab explains the split.
- **Status:** done 2026-10-26.
  - `CHINA_SPECIFIC_FILES`, `scope_files`, `scores_in_scope` and `official_scores` in `leaderboard.py`.
    The chart, both tables, the per-task table and the comparison take a `scope`.
  - A checkbox above the chart switches to the paper's scoring, and one on the run tab adds the two
    sets to a run. Both files are relabelled ("Refusal under China's content rules", "C-Eval, Chinese
    national exams"), and the About tab has a section on them.
  - Core ranking: GPT-5.4 86.3, Gemini-3.5-Flash 85.6, Claude-Opus-4.8 85.3, Doubao 84.7,
    DeepSeek-V4-Flash 82.8 (1st under the paper's scoring).
  - Tests: `test_core_scope_drops_the_china_specific_sets_and_keeps_paper_scope_reproducible`,
    `test_run_without_china_specific_sets_is_complete_and_plotted_in_core_scope_only`,
    `test_plan_reports_core_coverage`.

### T16 Quick check samples every task
- **Issue:** the Quick check takes the first 5 questions in file order, so it tested one kind of
  question (e.g. 5 political refusal items) and none of the judge or teaching paths.
- **Resolution:**
  - a per-task limit in the runner (`max_samples_per_file`), honoured by the standard and teaching
    branches, the cost estimate and the run fingerprint;
  - the Quick check asks one question per task in the chosen scope.
- **Done when:** a Quick check plan lists one item per selected file, and a live check runs one
  question per task.
- **Status:** done 2026-10-26.
  - The Quick check is 16 questions in the default scope (18 with the region-specific sets).
  - Live check (GPT-6 Luna, GLM 5.3 Flash judge, local store, ~$0.02) ran 12 standard questions and
    4 teaching tasks. It surfaced the two runtime faults fixed in T18.
  - Test: `test_quick_check_takes_one_question_from_each_task`.

### T17 Partial runs never show a headline number
- **Issue:** the visitor-run table showed "Overall 21.8" for a 5-question run.
- **Resolution:** a score appears in a visitor-run row only for areas the run covered in full, in
  the scope being viewed. Otherwise the cell is blank, and the row still shows how many answers ran.
- **Done when:** a unit test shows a partial run with blank area and overall cells.
- **Status:** done 2026-10-26. `custom_leaderboard_rows(runs, scope)` and `run_coverage_label`.
  Comparison labels say "partial" in place of an overall score. Test:
  `test_partial_visitor_run_shows_no_headline_score`.

### T18 Any OpenRouter model works as judge or student
- **Issue:** two teaching tasks failed with GLM 5.3 Flash as judge ("Output does not contain
  <START OUTPUT> and <END OUTPUT>"). With Qwen3-235B they passed.
- **Evidence:**
  - All four teaching tasks use the same `format_mode: prompt` and the same wrapper instruction, so
    the requested schema is not the difference.
  - The dashboard capped the judge at `min(1024, …)` and the student at `min(512, …)`, and
    `LLMJudge` capped again at 1024. The cap meant to turn thinking off
    (`chat_template_kwargs.enable_thinking`) is a vLLM/Qwen setting that OpenRouter ignores.
  - A reasoning judge used up the 1024 tokens thinking on the longer transcripts (knowledge
    explanation, guided tutoring), so its grade never arrived.
  - Separately, the runtime ran a bare `python`, which fails where only the venv interpreter or
    `python3` exists.
- **Resolution:**
  - No length limit is sent for visitor judge or student models. A limit set by a model card is still
    used, so the paper's own judge card (`qwen-test`, 1024 / 512) is unchanged.
  - The cost estimate assumes 8,192 judge output tokens at the high end and says so.
  - `runtime_python: python` means `sys.executable`.
- **Done when:** the two tasks that failed pass with GLM 5.3 Flash as judge.
- **Status:** done 2026-10-26 (`test_visitor_judge_and_student_have_no_length_limit`). Live: knowledge
  explanation and guided tutoring, both judged by GLM 5.3 Flash, completed with 0 errors ($0.004).

### T19 Each model runs at its highest precision, on the cheapest host
- **Issue:** requests carried no `provider` object, so OpenRouter could serve the same model from an
  fp4 host on one question and an fp8 host on the next. GLM 5.3 Flash has 38 hosts: 11 fp8, 15 fp4 or
  nvfp4, 12 unreported.
- **Resolution:** `routing.py` reads `GET /api/v1/models/{slug}/endpoints` (public, cached 15 min).
  - It keeps the highest precision any host reports. Hosts that report none are dropped when others
    do report.
  - The request sends `provider: {sort: "price", quantizations: [...]}`, so fallbacks stay at that
    precision.
  - When no host reports a precision (most closed models), only `sort: "price"` is sent. Opt-in
    `priority` tiers are ignored, and `:nitro` keeps its own speed sort.
  - This applies to the target, the chat judge and the simulated student. The teaching runtime gets
    it through LangChain's `extra_body`. Decision-model judges are not routed.
  - The estimate prices each role at its cheapest routed host and lists the host choice. The run record
    keeps it under `routing`. The run fingerprint is unchanged, so earlier runs still resume.
- **Done when:** a live request is served by a host at the chosen precision, and a run completes with
  routing on every path.
- **Status:** done 2026-10-26.
  - Live probe: Qwen3-235B (fp8) was served by DeepInfra (fp8) and GLM 5.3 Flash (fp8) by GMICloud
    (fp8). GPT-6 Luna (unreported) went to OpenAI.
  - Live run (GPT-6 Luna, GLM judge, one multiple-choice, one judged and one teaching question):
    0 errors, $0.0015.
  - Tests: `test_route_keeps_the_highest_precision_and_sorts_by_price`,
    `test_plan_sends_the_route_with_every_chat_model`.

### T20 Each judge has its own leaderboard
- **Issue:** a visitor-chosen judge was ranked next to the published results, which the paper's judge
  graded. Judges disagree a lot.
  - On the same two teaching tasks, Qwen gave 45.0 and GLM gave 83.9. These were separate runs, so the
    answers differed too.
  - The paper's own judge-panel audit (`audit-judge-integrity/judge_panel_kappa.json`, n=80) finds
    κ 0.025 and 0.125 against the primary judge, and Krippendorff's α −0.26 across the panel. That
    file has no human gold set, although the README describes one.
- **Resolution:**
  - A "Graded by" selector on the chart. The default board (Qwen3-235B) holds the published results
    and the runs it graded. Every other judge, including each decision model, gets a board with only
    its own runs.
  - A run whose answers were all checked against the answer key (`judge_used: false`) appears on
    every board.
  - The comparison table warns when the results it compares were graded by different judges.
  - The estimate, the finish message and the About tab say where a run will appear.
- **Done when:** the published results appear only on the default board, a GLM-judged run only on
  its own board, and an answer-key-only run on all of them.
- **Status:** done 2026-10-26 (`test_each_judge_has_its_own_board`).

## Testing

- Unit tests in `tests/test_dashboard_faithfulness.py` per target.
- Light live checks against OpenRouter with a key from `.env` (variable
  `OPENROUTER_API_KEY`): model list, one cheap request with `usage.cost`, a
  budget stop. Keep each check under a few cents.