Spaces:
Running
Running
Commit History
3 days ago
-
Sync from evaleval/eval-cards via hub-sync (part 2) Running
Verified signature2c19b0f -
Sync from evaleval/eval-cards via hub-sync
Verified signature0a3a7e0
8 days ago
-
Sync from evaleval/eval-cards via hub-sync
Verified signature46d6cd4
27 days ago
-
Sync from evaleval/eval-cards via hub-sync (part 2)
Verified signatureac827bb -
Sync from evaleval/eval-cards via hub-sync
Verified signaturecfd41ff -
Sync from evaleval/eval-cards via hub-sync (part 2)
Verified signature8ade5c1 -
Sync from evaleval/eval-cards via hub-sync
Verified signature966ebcd -
Sync from evaleval/eval-cards via hub-sync (part 2)
Verified signaturee3aa6bf -
Sync from evaleval/eval-cards via hub-sync
Verified signaturec685464 -
Sync from evaleval/eval-cards via hub-sync (part 2)
Verified signature109cc95 -
Sync from evaleval/eval-cards via hub-sync
Verified signaturea3005bd
-
Sync from evaleval/eval-cards via hub-sync
Verified signatureea6ddc0 -
Sync from evaleval/eval-cards via hub-sync (part 2)
Verified signaturea736240 -
Sync from evaleval/eval-cards via hub-sync
Verified signature6eddff5 -
Sync from evaleval/eval-cards via hub-sync (part 2)
Verified signaturef8acc78 -
Sync from evaleval/eval-cards via hub-sync
Verified signaturec4756f2 -
Sync from evaleval/eval-cards via hub-sync (part 2)
Verified signaturec42c305 -
Sync from evaleval/eval-cards via hub-sync
Verified signature4755eec -
Sync from evaleval/eval-cards via hub-sync (part 2)
Verified signaturef896ca1 -
Sync from evaleval/eval-cards via hub-sync
Verified signature97caaa8 -
Sync from evaleval/eval-cards via hub-sync
Verified signaturefdd48cd
-
Sync from evaleval/eval-cards via hub-sync (part 2)
Verified signatureae25365 -
Sync from evaleval/eval-cards via hub-sync
Verified signatureedadf53 -
Sync from evaleval/eval-cards via hub-sync
Verified signatured696af3
-
Sync from evaleval/eval-cards via hub-sync (part 2)
Verified signature341ceb7 -
Sync from evaleval/eval-cards via hub-sync
Verified signature29cd253
-
Sync from evaleval/eval-cards via hub-sync (part 2)
Verified signature2a41883 -
Sync from evaleval/eval-cards via hub-sync
Verified signaturee59af24
-
Sync from evaleval/eval-cards via hub-sync (part 2)
Verified signature2fd4390 -
Sync from evaleval/eval-cards via hub-sync
Verified signatureeea1441
-
Sync from evaleval/eval-cards via hub-sync
Verified signature1b448a6 -
Sync from evaleval/eval-cards via hub-sync (part 2)
Verified signature2ce4374 -
Sync from evaleval/eval-cards via hub-sync
Verified signature013c1cd
-
Move the Docker base image from bullseye to bookworm (node 24)
Both stages used node:18-bullseye-slim. Debian 11 (bullseye) left LTS support on 2026-08-31 and its security pool is being cleared: the bullseye-security index still lists libnghttp2-14 1.43.0-1+deb11u3 but the pool directory no longer holds any deb11 build of it, so the runner stage's apt-get install of ca-certificates and git fails with a 404 on every build, including a factory rebuild. The failure is deterministic, not a mirror hiccup. node:24-bookworm-slim matches the Node version the repo is developed and tested on (v24) and is a maintained image; node 18 left upstream support in 2025 and its image has not been rebuilt since May 2025. Next 15.2 and @duckdb/node-api (N-API) run unchanged on it: the full production build against the 2026-09-07T07-40-20Z snapshot passes locally on node 24. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1uBAjR5FZuzD2B8ZfwwjP993b525
-
Keep curated family names that merely slugify to their key
The family table humanized any display_name whose slug equalled the family key, on the theory that such a name is the raw slug with different separators. A curated name like "HELM", "MMMU", "LiveBench" or "CapArena" also slugifies to its key, and humanizeFamilyKey lowercases mid-word capitals, so the rows rendered "Helm", "Caparena", "Vals-AI". Measured over the cleaned hierarchy the table actually receives (80 families after cleanHierarchy): 49 were humanized, 38 of them wrongly; this change restores all 38 to their curated spelling and leaves the 11 genuinely raw slugs (gsm8k, hellaswag, piqa, ...) humanized. The rule now fires only when the display is still all-lowercase, i.e. actually a raw slug. The registry's incoming "aiXamine" family is the same shape and is covered by the unit test as a synthetic case. The rule is extracted as familyRowLabel (named apart from lib/hierarchy-lookup's familyDisplayName field, which carries the raw value) and unit-tested, including the misleading…c8cec8c -
Keep the word stems prefix-matching and take the first matching fallback rule
Mirror of the same change in eval_cards_backend_pipeline: faithful / factual / corrupted anchor at the word start only, so FactualBench and corrupted_visual_genome keep their tags. inferTagsFromBenchmark now returns the first matching rule, as the producer's resolve_benchmark_tags does; unioning all matches made the two repos tag the same fallback name differently even with identical regexes. The rule sets are now textually identical in both repos. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1uBAjR5FZuzD2B8ZfwwjP5a30bf2 -
Mirror the agent rule, use the tag vocabulary, and gate both
Same change as in eval_cards_backend_pipeline (the tag table stays byte-identical): the frontend's `agent` rule had kept its trailing boundary, so eleven agent benchmarks tagged differently in the two repos; the three stem rules are now textually mirrored and the whole-word alternatives stay anchored. The one "security" entry, outside the 17-tag vocabulary, becomes safety; a vitest asserts every list in the table draws from EVALUATION_TAGS and pins the stem behaviour. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1uBAjR5FZuzD2B8ZfwwjP34b018c -
Tag the aiXamine leaderboards and fix the fallback stems that never matched
Mirror of the same change in eval_cards_backend_pipeline (the two tag files are kept byte-identical). Three fallback rules in benchmark-schema.ts ended in a word boundary right after a stem (hallucin, robust, agent), so "Hallucination", "Robustness" and "Agentic" never matched and those benchmarks tagged general. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1uBAjR5FZuzD2B8ZfwwjP1efee54 -
Tag AlpacaEval 1.0 and the LEXam benchmarks
Registered by eval-card-registry #55 and #54 (integration batch); keyed by display name here, they would otherwise fall through to `general`. Same change ships as evalcard_tags.json in eval_cards_backend_pipeline. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1uBAjR5FZuzD2B8ZfwwjPcc77dc0 -
Tag the GLUE, SuperGLUE, ANLI, HEAD-QA, LAMBADA and classic LM benchmarks
The registry (eval-card-registry#64 and its follow-up) adds the lm-evaluation-harness task families as standalone benchmarks grouped by family. categories.json is keyed by display name, so without entries every one of them fell through getBenchmarkTags to the regex fallback and tagged `general`, polluting the /evals category pills. Entries follow the existing neighbours: GLUE mirrors SuperGLUE (linguistic_core, general); NLI / paraphrase / sentiment tasks are linguistic_core; COPA and WSC are commonsense_reasoning like Winogrande; HEAD-QA and MedMCQA mirror MedQA (knowledge, natural_sciences). Variant display names such as "MNLI (mismatched)" and "HEAD-QA (English)" reach the base entry through the paren-suffix strip; "ANLI R1/2/3" get their own keys because the tight normaliser does not strip the round. The same table ships as evalcard_tags.json in eval_cards_backend_pipeline and is updated identically there. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: …cf3c3ab -
Read infinite metric bounds from their JSON wire form
The registry now marks a metric bound that is unbounded by definition with an infinite float, and the producer's sidecars carry it as the string "Infinity" / "-Infinity" because JSON has no literal for it. lib/json-bounds.ts is the one place the frontend crosses that boundary, in both directions, importable from server and client code alike. Reading applies a policy per key: - canonical_min_score / canonical_max_score (the registry's stamp on a comparison-index metric) come back as the infinite number. Their only consumers are the scale resolvers, which read an open side as neither fraction nor percent: registryBoundsIsPercent already returns null for it, and mergeRegistryBounds now folds a non-finite side to null before comparing sibling metrics, so a group on an open-ended metric keeps the same fallback it had when the registry said nothing. - every other bound key (metric_config.min_score and friends) is a per-fact range used for normalisation, "0 to N scale" copy and…ca658d4
-
Show sources in context plot tooltips
6588001 -
Exclude assisted results from merged views
5b1f5c8 -
Clarify study context score semantics
cd7beed
-
Show both study conditions among all reported evaluations; retire the compute view
The context view read as "this study versus the official leaderboard", with each circle standing for a named agent scaffold. A scaffold name turns out not to pin a setup: within one scaffold the configuration still varies, and nothing short of a config hash would fix that. So circles are now every reported evaluation of the same model from any source, scaffold demoted to hover metadata, and the copy says measurements rather than leaderboard entries. Both halves of the study's design are now visible. The plot showed only the no-feedback condition, which left the answer-feedback half of a fully crossed experiment with nowhere to appear. Each strip gains a second diamond for the model's fullest oracle-feedback cell, in a second hue validated for colour-vision separation against the accent rather than distinguished by fill alone. Uncertainty is now the source's own published standard error, on the study's marks, replacing an estimated re-run band we computed ourselves. External standard …1d3058f -
Add scaffold-context view; fix folded-model URL 404s and AIR-Bench phantom splits
Context view (curated collection benchmark pages): - New Context tab in the score plotbox and ?view=context embed: per-model dot strips placing the study's score and re-run band among the official leaderboard's per-scaffold entries, fed by the producer's collection_context.json sidecar (probe-gated; older snapshots render exactly as before). Harvest date shown in the embed study strip. Production fixes (pre-existing, live on the current snapshot): - Folded-model pages 404ed because the fallback matched the percent-encoded route id against plain raw_model_ids; the id is now decoded before matching (repairs 239 folded model URLs). - consolidateAirBench synthesised split ids as a prefix x slice cross product, inventing 59 nonexistent ids once a second source prefix appeared; synthesised ids are now validated against the hierarchy's real constituent evaluation ids (120 -> 61 splits, other nodes byte-identical). Preflight hardening: error-marker matching made case-inse…6b4f9e2
-
Assert the study strip by its own sentence, not the study name
The source picker's option label carries the study title on snapshots that use it as the composite display name, so name-containment both false-failed the merged no-strip check and vacuously passed the pinned strip check. The strip's budget sentence appears nowhere else. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>bf6e621 -
Collection study embeds: compute view, trajectories route, context strip
The two study surfaces gain the embed affordance every other plot has. The distribution embed accepts ?view=compute and mounts ComputePlot directly from a shared pure builder (the same marks feed as the page's Compute chip; the distribution machinery never runs in that branch, so an empty result is a declared absence line, never a substituted histogram). A new /embed/eval/trajectories route serves the panel cards with ?panel= and ?condition= params; the page owns the fetch, keyed by the evaluation id it was issued for, so switching sources can never paint stale panels. Study embeds carry a context strip inside the iframe: attribution with the paper link (http(s) only), the budget sentence, and the oracle-feedback run count. The plotbox embed popover gains a Compute variant and the trajectories section an embed button, both only where the data supports them. Card titles and the embed route's labels read from one shared map, and the compute availability gate is the single buildComputePro…0d81559 -
Collection study UI: compute view, trajectory panels, source switcher
Per-source pages of curated study collections gain a Compute chip on the plotbox (per-run scores over the nominal budget axis with condition-labeled marks, an always-visible legend, and no trend lines), a trajectories section (release-order colored step curves with on-curve tooltips, a task-difficulty heatmap, and a termination summary with hover breakdowns), a source and merged-page switcher above EvalDetail, and study attribution in the header. The merged page's protocol banner links to the study source and never mounts collection panels. Copy uses the study's vocabulary (feedback condition, oracle score feedback) and the plain-language standard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>3be50ca -
Collection study data layer: payload attachments, trajectory route, panel builders
Per-source eval summaries gain two optional fields: source_options (sibling sources plus the merged page target when one exists) and a curated collection attachment built from the new collections.json sidecar fetch, keyed by the rows' collection_id and carrying the server-chosen compute axis (the first numeric protocol axis with two or more values within a feedback condition). New /api/eval-trajectories route serves pre-shaped panels from the collection_trajectories table: tokens-to-success step series (solve events are completed_on_successful_submit runs only; unsolved tasks censor at their observed token count), reliability bins (1 - the task's median cross-model solve rate pooled over both feedback conditions, shared across condition panels), and termination partitions in the study's own table grain with a separate outcome line. Trajectory model ids canonicalise through models_view raw id lists; unmatched ids stay flagged. Every accessor returns absence on snapshots that predate th…59c7b6c