Spaces:
Running
Running
Commit History
16 days ago
-
Sync from evaleval/eval-cards via hub-sync
Verified signaturecfd41ff -
Sync from evaleval/eval-cards via hub-sync
Verified signature966ebcd -
Sync from evaleval/eval-cards via hub-sync
Verified signaturec685464 -
Sync from evaleval/eval-cards via hub-sync
Verified signaturea3005bd
29 days ago
-
Sync from evaleval/eval-cards via hub-sync
Verified signatureea6ddc0 -
Sync from evaleval/eval-cards via hub-sync
Verified signature6eddff5 -
Sync from evaleval/eval-cards via hub-sync
Verified signaturec4756f2 -
Sync from evaleval/eval-cards via hub-sync
Verified signature4755eec -
Sync from evaleval/eval-cards via hub-sync
Verified signature97caaa8 -
Sync from evaleval/eval-cards via hub-sync
Verified signaturefdd48cd
30 days ago
-
Sync from evaleval/eval-cards via hub-sync
Verified signatureedadf53
-
Sync from evaleval/eval-cards via hub-sync
Verified signature29cd253
-
Sync from evaleval/eval-cards via hub-sync
Verified signaturee59af24
-
Sync from evaleval/eval-cards via hub-sync
Verified signatureeea1441
-
Sync from evaleval/eval-cards via hub-sync
Verified signature1b448a6
-
Keep curated family names that merely slugify to their key
The family table humanized any display_name whose slug equalled the family key, on the theory that such a name is the raw slug with different separators. A curated name like "HELM", "MMMU", "LiveBench" or "CapArena" also slugifies to its key, and humanizeFamilyKey lowercases mid-word capitals, so the rows rendered "Helm", "Caparena", "Vals-AI". Measured over the cleaned hierarchy the table actually receives (80 families after cleanHierarchy): 49 were humanized, 38 of them wrongly; this change restores all 38 to their curated spelling and leaves the 11 genuinely raw slugs (gsm8k, hellaswag, piqa, ...) humanized. The rule now fires only when the display is still all-lowercase, i.e. actually a raw slug. The registry's incoming "aiXamine" family is the same shape and is covered by the unit test as a synthetic case. The rule is extracted as familyRowLabel (named apart from lib/hierarchy-lookup's familyDisplayName field, which carries the raw value) and unit-tested, including the misleading…c8cec8c
-
Show sources in context plot tooltips
6588001 -
Exclude assisted results from merged views
5b1f5c8 -
Clarify study context score semantics
cd7beed
-
Show both study conditions among all reported evaluations; retire the compute view
The context view read as "this study versus the official leaderboard", with each circle standing for a named agent scaffold. A scaffold name turns out not to pin a setup: within one scaffold the configuration still varies, and nothing short of a config hash would fix that. So circles are now every reported evaluation of the same model from any source, scaffold demoted to hover metadata, and the copy says measurements rather than leaderboard entries. Both halves of the study's design are now visible. The plot showed only the no-feedback condition, which left the answer-feedback half of a fully crossed experiment with nowhere to appear. Each strip gains a second diamond for the model's fullest oracle-feedback cell, in a second hue validated for colour-vision separation against the accent rather than distinguished by fill alone. Uncertainty is now the source's own published standard error, on the study's marks, replacing an estimated re-run band we computed ourselves. External standard …1d3058f -
Add scaffold-context view; fix folded-model URL 404s and AIR-Bench phantom splits
Context view (curated collection benchmark pages): - New Context tab in the score plotbox and ?view=context embed: per-model dot strips placing the study's score and re-run band among the official leaderboard's per-scaffold entries, fed by the producer's collection_context.json sidecar (probe-gated; older snapshots render exactly as before). Harvest date shown in the embed study strip. Production fixes (pre-existing, live on the current snapshot): - Folded-model pages 404ed because the fallback matched the percent-encoded route id against plain raw_model_ids; the id is now decoded before matching (repairs 239 folded model URLs). - consolidateAirBench synthesised split ids as a prefix x slice cross product, inventing 59 nonexistent ids once a second source prefix appeared; synthesised ids are now validated against the hierarchy's real constituent evaluation ids (120 -> 61 splits, other nodes byte-identical). Preflight hardening: error-marker matching made case-inse…6b4f9e2
-
Collection study embeds: compute view, trajectories route, context strip
The two study surfaces gain the embed affordance every other plot has. The distribution embed accepts ?view=compute and mounts ComputePlot directly from a shared pure builder (the same marks feed as the page's Compute chip; the distribution machinery never runs in that branch, so an empty result is a declared absence line, never a substituted histogram). A new /embed/eval/trajectories route serves the panel cards with ?panel= and ?condition= params; the page owns the fetch, keyed by the evaluation id it was issued for, so switching sources can never paint stale panels. Study embeds carry a context strip inside the iframe: attribution with the paper link (http(s) only), the budget sentence, and the oracle-feedback run count. The plotbox embed popover gains a Compute variant and the trajectories section an embed button, both only where the data supports them. Card titles and the embed route's labels read from one shared map, and the compute availability gate is the single buildComputePro…0d81559 -
Collection study UI: compute view, trajectory panels, source switcher
Per-source pages of curated study collections gain a Compute chip on the plotbox (per-run scores over the nominal budget axis with condition-labeled marks, an always-visible legend, and no trend lines), a trajectories section (release-order colored step curves with on-curve tooltips, a task-difficulty heatmap, and a termination summary with hover breakdowns), a source and merged-page switcher above EvalDetail, and study attribution in the header. The merged page's protocol banner links to the study source and never mounts collection panels. Copy uses the study's vocabulary (feedback condition, oracle score feedback) and the plain-language standard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>3be50ca -
Use American spelling in user-visible strings (organisation → organization)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>f05c576 -
Matrix leaderboard: canonical-scale cells from the pipeline's ranked row
The matrix precompute stamped each (eval, model, metric) cell from whichever eval_results_view row happened to arrive last, on the raw source scale — so one column could mix 0-1 fractions with 0-100 percents (Scale SEAL hle vs the AISI collection rows), the verified checkmark rode an arbitrary row, and the frontier plot disagreed with the leaderboard ordering. Pick exactly one row per cell — the pipeline's ranked-best (position, then canonical score) — and emit score_canonical so every cell of a column sits on the metric's registry scale; the verified flag rides the same chosen row. Percent-unit columns whose values are all fractions render ×100 with a % suffix (column-uniform, so old snapshots without canonical scores are never half-converted), and the distribution/ frontier series apply the same presentation so plot and table agree. Also make the model_id→route mapping deterministic (min() instead of first-seen DISTINCT order). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>6a311d7
-
Mixed-page banner wording + clean-first synthesized leaderboard fallback
Two correctness patches from the spec-v2 adversarial review: - The hle per-source page carries both Scale SEAL rows and AISI protocol rows (both sources submit under the hle config), so the study-protocol banner's "these results" claim over-reached there. Mixed pages now say "N of the results below…"; homogeneous study pages keep the strong wording. - The synthesized leaderboard-rows fallback (snapshots without the precomputed matrix) deduped by model identity with scan-order pick semantics — an assisted run could become a model's cell. Rows now sort clean-first, best-score-first before the dedupe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>9f416c2 -
Source column: prefer the de-aliased evaluator identity
Upstream raw source strings can carry stale spellings the registry has already resolved — at the current EEE pin every AISI record still says "UK AI Security Initiative" (the upstream rename landed after the pin). Every other evaluator surface already shows the de-aliased name; the Source column was the one holdout reading source_metadata verbatim. Thread the view's per-row evaluator_display_name (registry canonical display when the org resolves, raw otherwise) through the per-source and merged payloads and prefer it for the Source label. Raw source_metadata strings remain untouched in row details for provenance. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>b7439ff -
Assisted runs: shown, labeled, and unranked by default
Protocol-varied collections (first: the UK AISI inference-scaling study) deliver several results per model, including oracle answer-feedback runs. The backend never serves those a rank, but the single-metric table (and the merged page, which renders through it) re-rank client-side by score — which would have crowned an assisted 1.00 as #1. - Plumb collection_id / protocol_condition through the eval-summary and merged queries, probe-gated so pre-collections snapshots keep NULL aliases (no binder errors, feature self-disabling). - Assisted rows keep their place in the table but take no rank (em dash + "assisted" chip); an explicit banner control re-includes them in ranking (default off). - Study-protocol caveat banner on pages carrying protocol rows: expanded budgets are not comparable to standard published results. - Distribution/Frontier plotbox stats exclude assisted rows — a "best" that needed the answer oracle is not a frontier. Pages without protocol data render ident…b597ca9
-
Unit-aware score display: units outrank the magnitude heuristic
Overlap rows gain plainUnit: when appearances agree on a plain-quantity unit, scores render as raw quantities ("1753 points", "0.80 seconds", "0.0012 usd") instead of being forced through percent formatting — the "score > 1.5 => %" guess produced "1525.0%" for GDPval-AA's Elo. Precedence: producer-resolved scale conversion (spec F5, e.g. WildBench's curated points->[0,1]) > declared unit > magnitude heuristic (last resort). Units come in two trust classes: physical units (seconds/usd/tokens/attempts...) are trusted at any magnitude; ambiguous score-like units (points/elo) need |score| > 1.5, because sources stamp "points" on genuine 0-1 fractions (omni-math accuracy). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>0ab5b2a -
Source select always displays 'All sources (merged)' on merged pages
Binding the ?source= came-from param as the select's VALUE made a merged page read as a single-source view (maintainer report: mmlu-pro showing 'Artificial Analysis' in the dropdown while the hero says 'merged across 7 sources'). The select is a navigator, not state; the came-from source keeps only its row highlight. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>04bb7fe
-
Merged-page review fixes: exclude flagged rows, wire comparability, thread card
Fixes from the adversarial review of 731462e: - BLOCKER: flagged observations (scale_conversion='flagged', score_canonical NULL) leaked RAW scores into the merged pool — on /evals/livebench a raw 1.42 outranked the true 0.85 best AND dragged the bounds inference to a bogus 0-100 scale (lying header, clamped score bars). lib/merged-adapter.ts now excludes null-canonical rows from model_results, the leaderboard matrix, the distribution pool, avg_score, best_model and the bounds inference; the merged page discloses the excluded count ("N results are not shown: their scores could not be converted to this metric's common scale"). - BLOCKER: the comparability tile read "Only one source reports this benchmark" on merged pages because EvalDetail was mounted without evalHierarchy/comparisonIndex. The route page now loads the hierarchy for merged routes too and threads both through MergedBenchmarkView (gpqa now shows 54% / 6 sources / 148 models compared). - Leaderboard …c5c4016 -
Merged pages: restore full eval-page plots by mounting EvalDetail at merged grain
The merged /evals/<benchmark-id> pages shipped as a bare observation table, dropping every plot the per-source pages have. Rebuild them as the SAME full EvalDetail experience — hero, benchmark signals, metric spec, score distribution/frontier, leaderboard, embed links — fed with the pooled all-sources observation rows on the metric's canonical scale (score_canonical, raw score for flagged rows): - components/merged-benchmark-view.tsx is now a thin wrapper: merged scope bar (results/sources/models counts), Source narrower (navigate-on-select, ?source= pre-highlight), slice selector for slice-grain pages, and disclosure notes around an EvalDetail mount. The metric switcher rides EvalDetail's split-picker slot and re-queries ?metric= without navigation. - lib/merged-adapter.ts: infer canonical-scale bounds (0-1 / 0-100 / data range) so score bars normalise, and stamp each row with its source composite slug for the ?source= highlight; rows keep their own per-source source_…731462e -
Resolve score scales from producer-stamped registry bounds
New snapshots stamp every comparison-index metric entry with canonical_min_score / canonical_max_score — the effective metric's registry bounds, i.e. the scale score_canonical sits on. Scale resolution now consults them, with precedence: 1. mul100 seen => percent registry 2. div100 seen => fraction registry 3. registry bounds when usable (min 0 / max 100 => percent; max <= 1.5 => fraction; anything else unusable) 4. the existing 'none'-cell unit vote, keeping both guards (unit conflict => null, anchor required => null) Bounds let two previously-guessed group shapes resolve exactly: - all-'curated' groups (anchor-neutral tags, e.g. WildBench's 1-10 points folded onto [0,1] wb-score): a model whose only appearance is the curated source now shows ~46% instead of the legacy "4.6%"; - unit-conflicted all-'none' groups (terminalbench-hard's lying "percent" unit): now exact 11.0% instead of the legacy fallback. The old-snapshot and no_bounds guards are unchanged. …fc25dc8 -
Guard against lying percent units in unanchored scale groups
Adversarial regression (terminalbench-hard / DeepSeek-R1): in an all-'none' group a single cell whose source-reported unit mislabels itself "percent" flipped the whole group's axis (0.11 rendered "0.1%" instead of "11.0%"). When no div100/mul100 conversion anchors the scale and the 'none' cells' units disagree, at least one unit is lying — stay on the legacy vote. Regression test mirrors the real composition. Also drop the now-unused isPreferredMetric local. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>3e89c0f -
Remove the Best headline from merged pages (maintainer decision)
Per-source pages have no best headline; the sorted table's top row already shows who leads. The merged hero now shows just results/sources/ models counts. best_result stays in the data (link-preview images and the embeds adapter still use it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>4b681cb -
Prefer producer canonical scores over >1.5 scale guessing (spec F5)
New snapshots stamp every per-source comparison-index score cell with score_canonical (the score on the metric's registry scale) and scale_conversion. The three client-side ">1.5 => percent" heuristics now apply a prefer-canonical, fallback-heuristic rule: - When every cell in a scale group carries usable canonical fields, the producer values settle the common axis exactly and the magnitude guess is skipped. The display scale mirrors each site's legacy convention (majority vote / any-percent) counted from the cells' actual source scales, so output is unchanged wherever the old guess was right. - Any old-snapshot cell in a group (missing keys), a bounds-less metric (no_bounds: canonical === raw, guarantees nothing), or an all-flagged group keeps the legacy heuristic byte-for-byte -- groups are never half-converted. - Flagged cells (score_canonical null) among canonical siblings keep the legacy per-row guess mapped onto the group's scale and stay included, matching the…e4bfb13 -
Apply adversarial-review fixes to the merged page and API route
- Source narrower: 'All sources (merged)' now clears the ?source pre-highlight instead of being a dead option. - eval-summary route: merged lookup failures degrade to the per-source lookup/404 instead of a 500 (connection-reset race on the optional merged table). - hasMergedEvalsView: only cache the positive probe, so a transient init failure of the optional parquet can't pin merged pages to 'absent' for the process lifetime. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>10276e4 -
Route single-segment eval ids to the merged benchmark page (F2-F4)
/evals/[...id]: one URL segment renders the new MergedBenchmarkView; two segments keep the per-source EvalDetail path untouched. ?source=/?metric=/?slice= live in the URL query so selections survive reload and back. MergedBenchmarkView: observation-grain hero ("N results from K sources - M models", best single result on the canonical scale), source narrower that NAVIGATES to the per-source page (Q5; slice-only sources disabled), metric switcher that re-queries via ?metric=, slice selector + note on slice-grain pages, flat interleaved table dense- ranked by score_canonical (flagged rows show the raw score with an unconverted warning and rank last), 50-per-click pagination, and per-source disclosure notes (protocol-variant-only and slice-only sources). No hierarchyLocation lookups — merged ids aren't keyed there. Browse-tree leaves route on the hierarchy node's new benchmark_id field (additive HierarchyBenchmark type): when present the leaf opens the merged page with the clicked source…d20fb23
-
Honor ?mode= query param (summary/policy|research) for embeds and deep-links
Verified signature5e2801e
-
Update to uniform US spelling
2fd6a71
-
Tolerate missing localStorage in sandboxed embed iframes
Hub blog sanitizer forces sandbox="allow-scripts" (no allow-same-origin) on every iframe, so /embed/* cards run as an opaque origin where even reading the window.localStorage getter throws a DOMException. That uncaught throw in AudienceModeProvider aborted React startup, leaving a blank frame. Add lib/safe-storage.ts (safeStorage() probes once, falls back to an in-memory shim) and route AudienceModeProvider + survey page through it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>01f29b6
-
Fix model comparison + add evaluator logo
4945098
-
Stop propagating verified into detail route
66f4cb7 -
Update verified counts for grey validated orgs
2b5a024
-
Fix embeds and duplicates
7260042 -
Fix embeds to filter by canonical benchmark only (no splits in embed until we decide how to display splits) + suppress popup on embed routes
e74e7be
-
Fix embed - missing model name in benchmark details
b12c58d -
Update mobile table rendering to stack
9d7e3c7
-
Make sure embeds dont have intro popup
1c48b8f -
Update embeds title and move toggle order
4e96330