Spaces:
Running
Running
Commit History
26 days ago
-
Sync from evaleval/eval-cards via hub-sync
Verified signaturecfd41ff -
Sync from evaleval/eval-cards via hub-sync
Verified signaturea3005bd
-
Sync from evaleval/eval-cards via hub-sync
Verified signaturec4756f2
-
Show both study conditions among all reported evaluations; retire the compute view
The context view read as "this study versus the official leaderboard", with each circle standing for a named agent scaffold. A scaffold name turns out not to pin a setup: within one scaffold the configuration still varies, and nothing short of a config hash would fix that. So circles are now every reported evaluation of the same model from any source, scaffold demoted to hover metadata, and the copy says measurements rather than leaderboard entries. Both halves of the study's design are now visible. The plot showed only the no-feedback condition, which left the answer-feedback half of a fully crossed experiment with nowhere to appear. Each strip gains a second diamond for the model's fullest oracle-feedback cell, in a second hue validated for colour-vision separation against the accent rather than distinguished by fill alone. Uncertainty is now the source's own published standard error, on the study's marks, replacing an estimated re-run band we computed ourselves. External standard …1d3058f -
Add scaffold-context view; fix folded-model URL 404s and AIR-Bench phantom splits
Context view (curated collection benchmark pages): - New Context tab in the score plotbox and ?view=context embed: per-model dot strips placing the study's score and re-run band among the official leaderboard's per-scaffold entries, fed by the producer's collection_context.json sidecar (probe-gated; older snapshots render exactly as before). Harvest date shown in the embed study strip. Production fixes (pre-existing, live on the current snapshot): - Folded-model pages 404ed because the fallback matched the percent-encoded route id against plain raw_model_ids; the id is now decoded before matching (repairs 239 folded model URLs). - consolidateAirBench synthesised split ids as a prefix x slice cross product, inventing 59 nonexistent ids once a second source prefix appeared; synthesised ids are now validated against the hierarchy's real constituent evaluation ids (120 -> 61 splits, other nodes byte-identical). Preflight hardening: error-marker matching made case-inse…6b4f9e2
-
Collection study embeds: compute view, trajectories route, context strip
The two study surfaces gain the embed affordance every other plot has. The distribution embed accepts ?view=compute and mounts ComputePlot directly from a shared pure builder (the same marks feed as the page's Compute chip; the distribution machinery never runs in that branch, so an empty result is a declared absence line, never a substituted histogram). A new /embed/eval/trajectories route serves the panel cards with ?panel= and ?condition= params; the page owns the fetch, keyed by the evaluation id it was issued for, so switching sources can never paint stale panels. Study embeds carry a context strip inside the iframe: attribution with the paper link (http(s) only), the budget sentence, and the oracle-feedback run count. The plotbox embed popover gains a Compute variant and the trajectories section an embed button, both only where the data supports them. Card titles and the embed route's labels read from one shared map, and the compute availability gate is the single buildComputePro…0d81559
-
Merged-page review fixes: exclude flagged rows, wire comparability, thread card
Fixes from the adversarial review of 731462e: - BLOCKER: flagged observations (scale_conversion='flagged', score_canonical NULL) leaked RAW scores into the merged pool — on /evals/livebench a raw 1.42 outranked the true 0.85 best AND dragged the bounds inference to a bogus 0-100 scale (lying header, clamped score bars). lib/merged-adapter.ts now excludes null-canonical rows from model_results, the leaderboard matrix, the distribution pool, avg_score, best_model and the bounds inference; the merged page discloses the excluded count ("N results are not shown: their scores could not be converted to this metric's common scale"). - BLOCKER: the comparability tile read "Only one source reports this benchmark" on merged pages because EvalDetail was mounted without evalHierarchy/comparisonIndex. The route page now loads the hierarchy for merged routes too and threads both through MergedBenchmarkView (gpqa now shows 54% / 6 sources / 148 models compared). - Leaderboard …c5c4016 -
Merged pages: restore full eval-page plots by mounting EvalDetail at merged grain
The merged /evals/<benchmark-id> pages shipped as a bare observation table, dropping every plot the per-source pages have. Rebuild them as the SAME full EvalDetail experience — hero, benchmark signals, metric spec, score distribution/frontier, leaderboard, embed links — fed with the pooled all-sources observation rows on the metric's canonical scale (score_canonical, raw score for flagged rows): - components/merged-benchmark-view.tsx is now a thin wrapper: merged scope bar (results/sources/models counts), Source narrower (navigate-on-select, ?source= pre-highlight), slice selector for slice-grain pages, and disclosure notes around an EvalDetail mount. The metric switcher rides EvalDetail's split-picker slot and re-queries ?metric= without navigation. - lib/merged-adapter.ts: infer canonical-scale bounds (0-1 / 0-100 / data range) so score bars normalise, and stamp each row with its source composite slug for the ?source= highlight; rows keep their own per-source source_…731462e
-
Fix embeds and duplicates
7260042 -
Fix embeds to filter by canonical benchmark only (no splits in embed until we decide how to display splits) + suppress popup on embed routes
e74e7be
-
Update embeds title and move toggle order
4e96330
-
view-data: coerce BIGINT-encoded numeric strings; restore distribution embed route
Two follow-ups from the getRowObjectsJson() switch: - `getRowObjectsJson()` serialises BIGINT inconsistently — values inside int32 range come back as JS numbers, larger ones as ASCII numeric strings. Same column (evidence_count, evaluations_count, evaluator_count) ends up as a mix of `number` and `string` across rows, which silently turned the /models page total via `sum + row.evidence_count` into giant string concatenations (visible on the page as a long digit blob next to "5,498 MODELS"). Detect the `^-?(?:0|[1-9]\d*)$` pattern in normalizeDuckDBValue and coerce when the value round-trips as a safe integer. - Restore the /embed/eval/distribution/<eval> route that was accidentally `git rm --cached`'d in the cleanup commit (445ce35). Externally-embedded iframes still point at this path; removing it yielded 404s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>d49f850 -
Untrack accidentally-committed noise files; gitignore the patterns
The previous commit (b6dab21) pulled in several files that should not be tracked: - "(1).tsx" — Finder duplicates of route files - output/hierarchy_explorer.html — local probe output - public/peer-ranks.json — local-generated cache (~370k lines) - app/embed/eval/distribution/[...id]/page.tsx — orphan route Remove from the index and extend .gitignore so they can't sneak back through a `git add -A` or accidental wildcard. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>445ce35 -
view-data: paranoid CAST every output column to a primitive type
The previous patch left simple scalars (VARCHAR / BOOLEAN / DOUBLE / INTEGER) untouched, on the assumption the binding handles those natively. The deployed log still surfaces "Invalid Error: don't know what type:" on the new explicit-column SELECT, which means whatever the binding is choking on is reachable even without r.* expansion and even with every nested type wrapped. The only thing left to vary is whether the binding ever sees the parquet's native logical types at all. Force every output column through `CAST(... AS <primitive>)`. The binding now only ever sees VARCHAR / BOOLEAN / INTEGER / DOUBLE in the result schema regardless of what type the underlying parquet column declared. Local prod build keeps returning 200 on /api/model-summary with model_info correctly nested. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>b6dab21 -
Embeds: histogram route, leaderboard slices + sort, brand mark; cross-source row dedup
Embed routes - New /embed/eval/histogram/<evalId>?model=<routeId> renders a single model/benchmark plotbox inside <BenchmarkDetail> via a new embedSurface="histogram" + embedTargetEvalId; the on-page split picker, view selector, peer rank, cross-family whisker and setup notes all flow through unchanged. The plotbox EmbedButton points at this route with label "Histogram". - Leaderboard embed: SPLIT dropdown for subtask-sliced evals (Global MMLU style), click-to-sort column headers with direction toggle, and getMetricChipLabel fallback so cells stop collapsing to "Score" when display_name is generic. - Frontier embed: same slice picker, locked to frontier view (showViewToggle=false), shared metric-label fallback. - Embed layout: small EvalEval logo + text in the top-left of every embed, linking to evalcards.evalevalai.com. Cross-source row dedup - New dedupeLeaderboardRowsByModelIdentity in lib/eval-processing.ts collapses two-source duplicates (e.g. one row develo…7a54021
