Spaces:
Running
Running
Commit History
16 days ago
-
Sync from evaleval/eval-cards via hub-sync
Verified signaturecfd41ff -
Sync from evaleval/eval-cards via hub-sync
Verified signature966ebcd -
Sync from evaleval/eval-cards via hub-sync
Verified signaturec685464 -
Sync from evaleval/eval-cards via hub-sync
Verified signaturea3005bd
29 days ago
-
Sync from evaleval/eval-cards via hub-sync
Verified signature6eddff5 -
Sync from evaleval/eval-cards via hub-sync
Verified signaturec4756f2 -
Sync from evaleval/eval-cards via hub-sync
Verified signature97caaa8
30 days ago
-
Sync from evaleval/eval-cards via hub-sync
Verified signatureedadf53
-
Sync from evaleval/eval-cards via hub-sync
Verified signature29cd253
-
Sync from evaleval/eval-cards via hub-sync
Verified signaturee59af24
-
Keep the word stems prefix-matching and take the first matching fallback rule
Mirror of the same change in eval_cards_backend_pipeline: faithful / factual / corrupted anchor at the word start only, so FactualBench and corrupted_visual_genome keep their tags. inferTagsFromBenchmark now returns the first matching rule, as the producer's resolve_benchmark_tags does; unioning all matches made the two repos tag the same fallback name differently even with identical regexes. The rule sets are now textually identical in both repos. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1uBAjR5FZuzD2B8ZfwwjP5a30bf2 -
Mirror the agent rule, use the tag vocabulary, and gate both
Same change as in eval_cards_backend_pipeline (the tag table stays byte-identical): the frontend's `agent` rule had kept its trailing boundary, so eleven agent benchmarks tagged differently in the two repos; the three stem rules are now textually mirrored and the whole-word alternatives stay anchored. The one "security" entry, outside the 17-tag vocabulary, becomes safety; a vitest asserts every list in the table draws from EVALUATION_TAGS and pins the stem behaviour. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1uBAjR5FZuzD2B8ZfwwjP34b018c -
Tag the aiXamine leaderboards and fix the fallback stems that never matched
Mirror of the same change in eval_cards_backend_pipeline (the two tag files are kept byte-identical). Three fallback rules in benchmark-schema.ts ended in a word boundary right after a stem (hallucin, robust, agent), so "Hallucination", "Robustness" and "Agentic" never matched and those benchmarks tagged general. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1uBAjR5FZuzD2B8ZfwwjP1efee54 -
Read infinite metric bounds from their JSON wire form
The registry now marks a metric bound that is unbounded by definition with an infinite float, and the producer's sidecars carry it as the string "Infinity" / "-Infinity" because JSON has no literal for it. lib/json-bounds.ts is the one place the frontend crosses that boundary, in both directions, importable from server and client code alike. Reading applies a policy per key: - canonical_min_score / canonical_max_score (the registry's stamp on a comparison-index metric) come back as the infinite number. Their only consumers are the scale resolvers, which read an open side as neither fraction nor percent: registryBoundsIsPercent already returns null for it, and mergeRegistryBounds now folds a non-finite side to null before comparing sibling metrics, so a group on an open-ended metric keeps the same fallback it had when the registry said nothing. - every other bound key (metric_config.min_score and friends) is a per-fact range used for normalisation, "0 to N scale" copy and…ca658d4
-
Exclude assisted results from merged views
5b1f5c8
-
Show both study conditions among all reported evaluations; retire the compute view
The context view read as "this study versus the official leaderboard", with each circle standing for a named agent scaffold. A scaffold name turns out not to pin a setup: within one scaffold the configuration still varies, and nothing short of a config hash would fix that. So circles are now every reported evaluation of the same model from any source, scaffold demoted to hover metadata, and the copy says measurements rather than leaderboard entries. Both halves of the study's design are now visible. The plot showed only the no-feedback condition, which left the answer-feedback half of a fully crossed experiment with nowhere to appear. Each strip gains a second diamond for the model's fullest oracle-feedback cell, in a second hue validated for colour-vision separation against the accent rather than distinguished by fill alone. Uncertainty is now the source's own published standard error, on the study's marks, replacing an estimated re-run band we computed ourselves. External standard …1d3058f -
Add scaffold-context view; fix folded-model URL 404s and AIR-Bench phantom splits
Context view (curated collection benchmark pages): - New Context tab in the score plotbox and ?view=context embed: per-model dot strips placing the study's score and re-run band among the official leaderboard's per-scaffold entries, fed by the producer's collection_context.json sidecar (probe-gated; older snapshots render exactly as before). Harvest date shown in the embed study strip. Production fixes (pre-existing, live on the current snapshot): - Folded-model pages 404ed because the fallback matched the percent-encoded route id against plain raw_model_ids; the id is now decoded before matching (repairs 239 folded model URLs). - consolidateAirBench synthesised split ids as a prefix x slice cross product, inventing 59 nonexistent ids once a second source prefix appeared; synthesised ids are now validated against the hierarchy's real constituent evaluation ids (120 -> 61 splits, other nodes byte-identical). Preflight hardening: error-marker matching made case-inse…6b4f9e2
-
Collection study embeds: compute view, trajectories route, context strip
The two study surfaces gain the embed affordance every other plot has. The distribution embed accepts ?view=compute and mounts ComputePlot directly from a shared pure builder (the same marks feed as the page's Compute chip; the distribution machinery never runs in that branch, so an empty result is a declared absence line, never a substituted histogram). A new /embed/eval/trajectories route serves the panel cards with ?panel= and ?condition= params; the page owns the fetch, keyed by the evaluation id it was issued for, so switching sources can never paint stale panels. Study embeds carry a context strip inside the iframe: attribution with the paper link (http(s) only), the budget sentence, and the oracle-feedback run count. The plotbox embed popover gains a Compute variant and the trajectories section an embed button, both only where the data supports them. Card titles and the embed route's labels read from one shared map, and the compute availability gate is the single buildComputePro…0d81559 -
Collection study data layer: payload attachments, trajectory route, panel builders
Per-source eval summaries gain two optional fields: source_options (sibling sources plus the merged page target when one exists) and a curated collection attachment built from the new collections.json sidecar fetch, keyed by the rows' collection_id and carrying the server-chosen compute axis (the first numeric protocol axis with two or more values within a feedback condition). New /api/eval-trajectories route serves pre-shaped panels from the collection_trajectories table: tokens-to-success step series (solve events are completed_on_successful_submit runs only; unsolved tasks censor at their observed token count), reliability bins (1 - the task's median cross-model solve rate pooled over both feedback conditions, shared across condition panels), and termination partitions in the study's own table grain with a separate outcome line. Trajectory model ids canonicalise through models_view raw id lists; unmatched ids stay flagged. Every accessor returns absence on snapshots that predate th…59c7b6c -
Only replace display names showing the copy-paste signature
sanitizeName required the whole key to appear inside the display name, clobbering legitimate names that share nothing with their slug — the AISI study's paper title became "AISI-Inference-Scaling" and "Humanity's Last Exam" became "HLE". The bug the guard exists for is one display string shipping under 2+ distinct keys (math-mc/gsm-mc both carrying "wasp (…)"), so require that duplication signature before falling back to the humanized key. Verified against the published hierarchy: unique names survive, the duplicate class still gets replaced. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>928757f -
Benchmark cards: also key "Name (ACRONYM)" titles by acronym and bare name
A tile displaying "HLE" couldn't reach the card titled "Humanity's Last Exam (HLE)" — the name-keyed map only carried the full title. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>285f739 -
Use American spelling in user-visible strings (organisation → organization)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>f05c576
-
Mixed-page banner wording + clean-first synthesized leaderboard fallback
Two correctness patches from the spec-v2 adversarial review: - The hle per-source page carries both Scale SEAL rows and AISI protocol rows (both sources submit under the hle config), so the study-protocol banner's "these results" claim over-reached there. Mixed pages now say "N of the results below…"; homogeneous study pages keep the strong wording. - The synthesized leaderboard-rows fallback (snapshots without the precomputed matrix) deduped by model identity with scan-order pick semantics — an assisted run could become a model's cell. Rows now sort clean-first, best-score-first before the dedupe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>9f416c2 -
Probe-gate evaluator_display_name like every additive view column
The previous commit selected r.evaluator_display_name unconditionally; the pinned-fixture tests correctly binder-errored — older snapshots predate the column. Fold it into the shared additive-columns helper with the same DESCRIBE-probe NULL-alias degradation as the collection columns (probe cache now holds the full column set). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>28baa4e -
Source column: prefer the de-aliased evaluator identity
Upstream raw source strings can carry stale spellings the registry has already resolved — at the current EEE pin every AISI record still says "UK AI Security Initiative" (the upstream rename landed after the pin). Every other evaluator surface already shows the de-aliased name; the Source column was the one holdout reading source_metadata verbatim. Thread the view's per-row evaluator_display_name (registry canonical display when the org resolves, raw otherwise) through the per-source and merged payloads and prefer it for the Source label. Raw source_metadata strings remain untouched in row details for provenance. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>b7439ff -
Assisted runs: shown, labeled, and unranked by default
Protocol-varied collections (first: the UK AISI inference-scaling study) deliver several results per model, including oracle answer-feedback runs. The backend never serves those a rank, but the single-metric table (and the merged page, which renders through it) re-rank client-side by score — which would have crowned an assisted 1.00 as #1. - Plumb collection_id / protocol_condition through the eval-summary and merged queries, probe-gated so pre-collections snapshots keep NULL aliases (no binder errors, feature self-disabling). - Assisted rows keep their place in the table but take no rank (em dash + "assisted" chip); an explicit banner control re-includes them in ranking (default off). - Study-protocol caveat banner on pages carrying protocol rows: expanded budgets are not comparable to standard published results. - Distribution/Frontier plotbox stats exclude assisted rows — a "best" that needed the answer oracle is not a frontier. Pages without protocol data render ident…b597ca9 -
Collections groundwork: uk-aisi logo asset + additive snapshot plumbing
- public/org-logos/uk-aisi.svg — served at the path the registry's canonical_orgs.logo_url points to (evaluator letterhead). - Optional collection fields on ModelResultForBenchmark (not yet populated by any adapter — display integration is spec'd separately). - collection_trajectories registered as an optional snapshot table (best-effort load; absent on older snapshots). - Integrity suite: collections.json presence/coverage checks, self-skipping on pre-collections snapshots. No behavior change on any page. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>68d591a
-
Unit-aware score display: units outrank the magnitude heuristic
Overlap rows gain plainUnit: when appearances agree on a plain-quantity unit, scores render as raw quantities ("1753 points", "0.80 seconds", "0.0012 usd") instead of being forced through percent formatting — the "score > 1.5 => %" guess produced "1525.0%" for GDPval-AA's Elo. Precedence: producer-resolved scale conversion (spec F5, e.g. WildBench's curated points->[0,1]) > declared unit > magnitude heuristic (last resort). Units come in two trust classes: physical units (seconds/usd/tokens/attempts...) are trusted at any magnitude; ambiguous score-like units (points/elo) need |score| > 1.5, because sources stamp "points" on genuine 0-1 fractions (omni-math accuracy). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>0ab5b2a -
Close re-review residuals: slice-grain card fallback, shown-pool model count
Cards for slice-grain merged pages live under the slice ids — include them in the lookup. models_count now counts distinct models among the rows actually shown, so hiding an unconvertible row can't produce '80 results · 81 models'. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>d20ae12
-
Merged-page review fixes: exclude flagged rows, wire comparability, thread card
Fixes from the adversarial review of 731462e: - BLOCKER: flagged observations (scale_conversion='flagged', score_canonical NULL) leaked RAW scores into the merged pool — on /evals/livebench a raw 1.42 outranked the true 0.85 best AND dragged the bounds inference to a bogus 0-100 scale (lying header, clamped score bars). lib/merged-adapter.ts now excludes null-canonical rows from model_results, the leaderboard matrix, the distribution pool, avg_score, best_model and the bounds inference; the merged page discloses the excluded count ("N results are not shown: their scores could not be converted to this metric's common scale"). - BLOCKER: the comparability tile read "Only one source reports this benchmark" on merged pages because EvalDetail was mounted without evalHierarchy/comparisonIndex. The route page now loads the hierarchy for merged routes too and threads both through MergedBenchmarkView (gpqa now shows 54% / 6 sources / 148 models compared). - Leaderboard …c5c4016 -
Merged pages: restore full eval-page plots by mounting EvalDetail at merged grain
The merged /evals/<benchmark-id> pages shipped as a bare observation table, dropping every plot the per-source pages have. Rebuild them as the SAME full EvalDetail experience — hero, benchmark signals, metric spec, score distribution/frontier, leaderboard, embed links — fed with the pooled all-sources observation rows on the metric's canonical scale (score_canonical, raw score for flagged rows): - components/merged-benchmark-view.tsx is now a thin wrapper: merged scope bar (results/sources/models counts), Source narrower (navigate-on-select, ?source= pre-highlight), slice selector for slice-grain pages, and disclosure notes around an EvalDetail mount. The metric switcher rides EvalDetail's split-picker slot and re-queries ?metric= without navigation. - lib/merged-adapter.ts: infer canonical-scale bounds (0-1 / 0-100 / data range) so score bars normalise, and stamp each row with its source composite slug for the ?source= highlight; rows keep their own per-source source_…731462e -
Resolve score scales from producer-stamped registry bounds
New snapshots stamp every comparison-index metric entry with canonical_min_score / canonical_max_score — the effective metric's registry bounds, i.e. the scale score_canonical sits on. Scale resolution now consults them, with precedence: 1. mul100 seen => percent registry 2. div100 seen => fraction registry 3. registry bounds when usable (min 0 / max 100 => percent; max <= 1.5 => fraction; anything else unusable) 4. the existing 'none'-cell unit vote, keeping both guards (unit conflict => null, anchor required => null) Bounds let two previously-guessed group shapes resolve exactly: - all-'curated' groups (anchor-neutral tags, e.g. WildBench's 1-10 points folded onto [0,1] wb-score): a model whose only appearance is the curated source now shows ~46% instead of the legacy "4.6%"; - unit-conflicted all-'none' groups (terminalbench-hard's lying "percent" unit): now exact 11.0% instead of the legacy fallback. The old-snapshot and no_bounds guards are unchanged. …fc25dc8 -
Guard against lying percent units in unanchored scale groups
Adversarial regression (terminalbench-hard / DeepSeek-R1): in an all-'none' group a single cell whose source-reported unit mislabels itself "percent" flipped the whole group's axis (0.11 rendered "0.1%" instead of "11.0%"). When no div100/mul100 conversion anchors the scale and the 'none' cells' units disagree, at least one unit is lying — stay on the legacy vote. Regression test mirrors the real composition. Also drop the now-unused isPreferredMetric local. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>3e89c0f -
Handle the 'curated' scale-conversion tag (registry-declared factors)
Curated canonicals (e.g. wildbench's 1-10 rows folded onto the 0-1 wb-score scale) display exactly via score_canonical but stay neutral for group scale-anchoring — the tag alone can't say fraction-vs-percent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>4010530 -
Prefer producer canonical scores over >1.5 scale guessing (spec F5)
New snapshots stamp every per-source comparison-index score cell with score_canonical (the score on the metric's registry scale) and scale_conversion. The three client-side ">1.5 => percent" heuristics now apply a prefer-canonical, fallback-heuristic rule: - When every cell in a scale group carries usable canonical fields, the producer values settle the common axis exactly and the magnitude guess is skipped. The display scale mirrors each site's legacy convention (majority vote / any-percent) counted from the cells' actual source scales, so output is unchanged wherever the old guess was right. - Any old-snapshot cell in a group (missing keys), a bounds-less metric (no_bounds: canonical === raw, guarantees nothing), or an all-flagged group keeps the legacy heuristic byte-for-byte -- groups are never half-converted. - Flagged cells (score_canonical null) among canonical siblings keep the legacy per-row guess mapped onto the group's scale and stay included, matching the…e4bfb13 -
Apply adversarial-review fixes to the merged page and API route
- Source narrower: 'All sources (merged)' now clears the ?source pre-highlight instead of being a dead option. - eval-summary route: merged lookup failures degrade to the per-source lookup/404 instead of a 500 (connection-reset race on the optional merged table). - hasMergedEvalsView: only cache the positive probe, so a transient init failure of the optional parquet can't pin merged pages to 'absent' for the process lifetime. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>10276e4 -
Route single-segment eval ids to the merged benchmark page (F2-F4)
/evals/[...id]: one URL segment renders the new MergedBenchmarkView; two segments keep the per-source EvalDetail path untouched. ?source=/?metric=/?slice= live in the URL query so selections survive reload and back. MergedBenchmarkView: observation-grain hero ("N results from K sources - M models", best single result on the canonical scale), source narrower that NAVIGATES to the per-source page (Q5; slice-only sources disabled), metric switcher that re-queries via ?metric=, slice selector + note on slice-grain pages, flat interleaved table dense- ranked by score_canonical (flagged rows show the raw score with an unconverted warning and rank last), 50-per-click pagination, and per-source disclosure notes (protocol-variant-only and slice-only sources). No hierarchyLocation lookups — merged ids aren't keyed there. Browse-tree leaves route on the hierarchy node's new benchmark_id field (additive HierarchyBenchmark type): when present the leaf opens the merged page with the clicked source…d20fb23 -
Add merged-benchmark accessor + API routing (spec F1)
getMergedBenchmarkSummary reads the new merged_evals_view row and the observation-grain leaderboard live from eval_results_view on metric_id_effective, sorted by score_canonical in the metric's direction. Echo rows are kept — the accessor never calls dedupeLeaderboardRowsByModelIdentity (design pt 3/Q3) — and eval-matrices.json is not consulted. Slice-grain pages (design pt 7b) query the slice's own benchmark_id with is_slice. merged_evals_view loads best-effort at connection init so pre-merged snapshots keep working; the accessor probes the catalog and returns null when the artifact is absent. /api/eval-summary resolves single-segment ids (no %2F after routeIdFromSegments normalization — the new isMergedEvalId helper) through the merged accessor with ?metric=/?slice= params, returning a { merged: true } discriminated payload; two-segment ids keep the existing path byte-identical. fetchEvalSummary adapts merged payloads to the BenchmarkEvalSummary shape so the eval embeds render merg…0fd9809 -
Fix percent-encoded id round-trip in routeIdFromSegments (F7)
Eval detail pages 404ed for ids containing percent-encoded characters other than %2F (spaces, @, parens): Next decodes catch-all params, but routeIdFromSegments only re-encoded "/" -> %2F, so rebuilt ids never matched the stored fully-encoded evaluation_id. routeIdFromSegments now re-encodes every segment with a strict encoder matching Python's urllib.parse.quote(value, safe="") — i.e. encodeURIComponent plus explicit %-encoding of ! ' ( ) * — then joins with %2F. Segments are loose-decoded first (try/catch decodeURIComponent) so the transform is idempotent: call sites span client useParams() and server awaited params, where Next has historically been inconsistent about delivering decoded vs raw-encoded values, and both now converge to the canonical stored form. This accepts the same rare tradeoff as middleware safeDecode and the developer-lookup fallback: a decoded value that literally contains %XX gets double-decoded. Adds unit tests for routeIdFromSegments (expected outputs verifi…c8d2308
-
Regenerate model URL redirects for the 2026-08-13 snapshot
The registry's OpenRouter id adoption repointed canonical model routes; the old map 301'd six now-canonical pages to dead targets. 1107 -> 1613 entries, validated by test:integrity against the published snapshot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>6c6caea
-
Update to uniform US spelling
2fd6a71
-
Tolerate missing localStorage in sandboxed embed iframes
Hub blog sanitizer forces sandbox="allow-scripts" (no allow-same-origin) on every iframe, so /embed/* cards run as an opaque origin where even reading the window.localStorage getter throws a DOMException. That uncaught throw in AudienceModeProvider aborted React startup, leaving a blank frame. Add lib/safe-storage.ts (safeStorage() probes once, falls back to an in-memory shim) and route AudienceModeProvider + survey page through it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>01f29b6
-
Add dynamic thumbnails for orgs
e068beb
-
Fix model comparison + add evaluator logo
4945098
-
Update verified counts for grey validated orgs
2b5a024 -
Update verified list
8a710d4 -
Help docs: verified-checkmark gif + refreshed cross-post tutorial
- get-verified: use the verification-tooltip gif as the page screenshot - tutorials.ts: screenshot blocks now accept .gif as well as .png - cross-post-to-hugging-face: rewrite to the updated blog content (Feb 2026 launches, two-sided Community Evals model, converter audit flow + OPEN PRS sign-off, four supported benchmarks, verified badge); iframes -> links since the renderer has no rehype-raw - add real screenshots for the two cross-post figures - gitattributes: track screenshot .gif via LFS (matching the .png rule) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>f7f3c2c
-
Fix embeds to filter by canonical benchmark only (no splits in embed until we decide how to display splits) + suppress popup on embed routes
e74e7be
-
Update embeds title and move toggle order
4e96330 -
Update wording
6d05aa6