Spaces:
Running
Running
Commit History
17 days ago
-
Sync from evaleval/eval-cards via hub-sync
Verified signaturea3005bd
30 days ago
-
Sync from evaleval/eval-cards via hub-sync
Verified signaturec4756f2
-
Collection study data layer: payload attachments, trajectory route, panel builders
Per-source eval summaries gain two optional fields: source_options (sibling sources plus the merged page target when one exists) and a curated collection attachment built from the new collections.json sidecar fetch, keyed by the rows' collection_id and carrying the server-chosen compute axis (the first numeric protocol axis with two or more values within a feedback condition). New /api/eval-trajectories route serves pre-shaped panels from the collection_trajectories table: tokens-to-success step series (solve events are completed_on_successful_submit runs only; unsolved tasks censor at their observed token count), reliability bins (1 - the task's median cross-model solve rate pooled over both feedback conditions, shared across condition panels), and termination partitions in the study's own table grain with a separate outcome line. Trajectory model ids canonicalise through models_view raw id lists; unmatched ids stay flagged. Every accessor returns absence on snapshots that predate th…59c7b6c
-
Resolve merged benchmarks in the eval OG-image route
Merged all-sources pages (single-segment ids, spec F2) live in merged_evals_view, so getEvalSummaryById returned nothing and the card fell back to the generic "Benchmark" frame while the page's text metadata resolved fine. Branch on isMergedEvalId exactly like app/evals/[...id]/layout.tsx and surface display_name, models_count, and the producer's best_result on the card. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>fe2b03f
-
Apply adversarial-review fixes to the merged page and API route
- Source narrower: 'All sources (merged)' now clears the ?source pre-highlight instead of being a dead option. - eval-summary route: merged lookup failures degrade to the per-source lookup/404 instead of a 500 (connection-reset race on the optional merged table). - hasMergedEvalsView: only cache the positive probe, so a transient init failure of the optional parquet can't pin merged pages to 'absent' for the process lifetime. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>10276e4 -
Add merged-benchmark accessor + API routing (spec F1)
getMergedBenchmarkSummary reads the new merged_evals_view row and the observation-grain leaderboard live from eval_results_view on metric_id_effective, sorted by score_canonical in the metric's direction. Echo rows are kept — the accessor never calls dedupeLeaderboardRowsByModelIdentity (design pt 3/Q3) — and eval-matrices.json is not consulted. Slice-grain pages (design pt 7b) query the slice's own benchmark_id with is_slice. merged_evals_view loads best-effort at connection init so pre-merged snapshots keep working; the accessor probes the catalog and returns null when the artifact is absent. /api/eval-summary resolves single-segment ids (no %2F after routeIdFromSegments normalization — the new isMergedEvalId helper) through the merged accessor with ?metric=/?slice= params, returning a { merged: true } discriminated payload; two-segment ids keep the existing path byte-identical. fetchEvalSummary adapts merged payloads to the BenchmarkEvalSummary shape so the eval embeds render merg…0fd9809
-
Add dynamic thumbnails for orgs
e068beb
-
Fix model comparison + add evaluator logo
4945098
-
Gzip the detail-page sidecars via cachedGzipJson
comparison-index (~20MB), peer-ranks, benchmark-metadata, eval-hierarchy were shipped RAW on detail pages — the comparison-index dominated the ~5s stall. Route them through the existing cachedGzipJson (gzip ~10-13x + per-process memoize + ETag/304); same data, no shape change, same pattern already proven on the /models + /evals index routes. Interim transfer fix; the scoped-fetch migration supersedes it later. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>f323554 -
Cache + gzip the /models and /evals index payloads; remove temp diag route
/models and /evals fetch /api/model-cards-lite (~15.5 MB) and /api/eval-list-lite (~5.4 MB) client-side. Server compute is only ~1s (warm in-memory tables), but the responses were recomputed every request and shipped uncompressed, so first visits showed a long loading state dominated by the transfer. Add lib/cached-json-response.ts: memoise the serialized + gzipped bytes per process (10-min TTL) and serve with Content-Encoding: gzip + an ETag (conditional 304s). warm-startup-cache already hits both routes at boot, so the cache is primed before the first visitor — who now downloads the gzipped (~10x smaller) payload instead of 15.5 MB of raw JSON. Also removes the temporary /api/diag endpoint (and the duckdb diagnostic exports it used) now that the parquet read-path bug is fixed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>7888b0e -
diag: add file-integrity hash + /tmp-vs-/data read probes
scan_primitives fails on the local /data cache but succeeds over httpfs with identical data, and threads=1 doesn't help. Add probes to sha256 the cached file vs remote (rule out download corruption) and to read the same bytes from os.tmpdir() vs /data, to confirm whether the HF /data mount + DuckDB local reader (mmap) is the cause. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>368cea5 -
Rename _diag -> diag (underscore folders are private in App Router)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>2b79eaf -
Add /api/_diag probe; restore @duckdb/node-api 1.5.3-r.2
The 1.5.2 rollback did not help — the rebuilt Space still fails with "don't know what type:" on a freshly purged + re-downloaded cache, on the old binding. So neither the disk cache nor the binding version is the cause. The Space's pinned snapshot (warehouse/2026-05-29T00-24-44Z) also reads cleanly under linux/amd64 Rosetta emulation locally, which points at a real-hardware (Sapphire Rapids / AVX-512) Parquet-decode path that emulation doesn't exercise. Adds a temporary read-only /api/_diag endpoint to isolate, on the Space hardware itself, which nested column's to_json decode trips the Thrift error, and whether thread count or the local-file vs httpfs read path matters. Restores the binding to the preferred 1.5.3-r.2 (version proven irrelevant to the failure). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>5fe6792
-
Warm sidecar cache and show loading progress
83ef54a
-
Adapt upstream category-based code to v2 tag taxonomy
Upstream's new OG routes, eval layout, and benchmark-detail additions referenced the retired v1 category model. Map to v2 equivalents: - evaluations_by_category -> evaluations_by_tag (OG model card) - summary.category -> derived_tags[0] (OG/eval-detail social subtitle) - category: CategoryType -> EvalTag (benchmark-detail local row types) - summary_eval_ids -> constituent_evaluation_ids (clean-hierarchy Vals.ai merge) Also cast a pre-existing parseMaybeJson(evalcards_annotations) to RowAnnotations in view-data (latent, masked by ignoreBuildErrors).bfb71af -
Add per-model + per-eval OpenGraph thumbnails; replace "EE/EC" mark with logo image
Social/chat unfurls now identify the specific model or benchmark being shared instead of showing the generic homepage card. - /api/og/models/[...id] and /api/og/evals/[...id] render dynamic 1200x630 PNGs via next/og's ImageResponse, pulling display name / developer / category / models-evaluated / top-model from the existing DuckDB-backed summary getters. Cards include the EvalEval brand logo (logo-square.png) in place of the previous text mark. The handlers live under /api/og/ rather than as `opengraph-image.tsx` siblings because Next.js 15 rejects metadata files inside catch-all (`[...id]`) folders. - app/models/[...id]/layout.tsx and app/evals/[...id]/layout.tsx own the `generateMetadata` for those routes (the page components are client-side, so they can't export metadata directly). The layouts also seed the per-entity `title`/`description`/`og:image`/`twitter:image` so non-image unfurls also identify the entity. - app/opengraph-image.tsx swaps the inline "EE" te…c315a26
-
Dedup logic to counts
aac276a -
Route peer-ranks fetch through SNAPSHOT_URL sidecar
The model-detail benchmark grid was hardcoded to fetch peer-ranks.json from the dataset *root* on `main`. The rest of the app reads from the pinned `SNAPSHOT_URL=warehouse/<id>/` snapshot, so peer ranks could drift from the views they were paired with — different eval IDs, totals, or models than the comparison-index showed. The producer now emits peer-ranks.json as a Stage J sidecar (eval_cards_backend_pipeline ffbfe71). Add `fetchPeerRanks()` to `lib/sidecars.ts` (unwraps the new `{generated_at, ranks}` envelope; gracefully returns `{}` for older snapshots that 404), expose it through `lib/hf-data.ts` + `/api/peer-ranks` + `dashboard-data-client.ts`, and switch `loadPeerRanks()` in benchmark-detail.tsx off the hardcoded URL. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>6cc7b0b
-
Add DuckDB shadow-read backend with source-metadata fix
Jenny Chim +1: Claude Opus 4.7 (1M context)Two-part change behind a feature flag. Backend dispatcher (lib/data-backend.ts) routes 11 API routes + app/page.tsx to either lib/model-data.ts (default JSON path) or the new lib/duckdb-data.ts based on DATA_BACKEND=duckdb. The DuckDB path reads payload_json from local Parquet via @duckdb/node-api; pipeline-side export is gated by EXPORT_EXPERIMENTAL_PARQUET=1 (lives in eval_cards_backend_pipeline). For parity testing, lib/hf-data.ts gains HF_DATA_LOCAL_DIR (override the default .cache/hf-data path) and HF_DATA_OFFLINE=1 (refuse remote fetches so background refreshes can't poison parity tests). scripts/compare-data-backends.mjs diffs the two backends endpoint-by-endpoint. Source-metadata fallback (item #4 in notes/ts-to-pipeline-migration.md) deleted: pipeline now carries source_metadata on every model_result row (verified 86,183/86,183 production hierarchy + 43,859/43,859 eval-detail). Removed getCanonicalSourceMetadata, buildSourceMetadataIndex, and three inline duplicates in lib/…2fcae3f
-
Separate policy and researcher views
- Foundations: glossary + Term + ModeSwitch helpers - Policy: plain-language summary, scope chips, provenance row, collapsible subtasks, technical details pushed into expander - Researcher: per-row Reproducibility card with adaptive compact mode for limited-disclosure rows; agent setup group for agentic evals; lazy-fetched generation_config + score_details from model files - Replace boilerplate IBM risk field with curated known-issues registry - Apples-to-apples banner above leaderboards, anchored to Comparability panel - Methodology split: surface task setup, scoring, validation - Compact row-signals indicator replacing repetitive coloured badges - Cleaner divergence chips in Comparability panel; full-width when only one divergence type fires - Flag-this-score button that links to the dataset record on HF and pre-fills a discussion title plus copyable context - Defensive guard so an embedded ancestor card no longer leaks its text onto a leaf-benchmark page9b4cdbb
-
Add interpretive signals, corpus dashboard, and slice browser
Surfaces reproducibility, reporting completeness, provenance, and comparability signals from the backend across eval detail, model compare, eval list cards, and a new /corpus dashboard. Adapts to the upstream 2-level hierarchy (family → leaf), caps the leaderboard at 24 default columns and replaces the slice tabs with a search dialog when a benchmark has more than 5 slices. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>bca888a
-
improve ux
8058fce
-
Improve eval/model UX, lite data paths, and leaderboard clarity
436ada0
-
Add per-benchmark comparison histograms on model detail
Rewires the model detail page to render one histogram per (benchmark, metric) backed by the new comparison-index.json artifact. Each card groups peers from the same family (Fibble1…Fibble5 under Fibble Arena, RewardBench 2's components under one card), shows metric tabs with backend-sorted order, exposes submission-axis metadata (harness / variant / rerun) on multi-submission peers, and surfaces secondary metrics or per-split subtasks in an accordion below the chart. Ranks are read directly from comparison-index by_model, retiring the peer-ranks round-trip on this surface. Also adds `/api/eval-detail` for the deep-dive modal and wires `comparison-index.json` into the build-time cache script. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>415ac43
-
Refresh eval cards UI and backend data flow
c1f2130
-
Fix survey submit: use correct HF commit API JSON format with files array
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>c5372a8 -
Fix survey submit: use multipart form data for HF commit API
The HF commit API requires multipart with a JSON header part and file content as a separate part, not inline JSON. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>9481599 -
Fix HF commit API field: summary not commit_message
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>023694a -
Fix survey submission: use HF commit API instead of deprecated upload
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>ddf16f4 -
Add survey submission and update survey text for public use
- Add /api/survey-submit route that POSTs to HF dataset - Make survey fields optional with floating action bar - Update text from guided interview to public online survey - Auto-set date instead of manual interview date field Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>516ec04
-
fix bugs
04b4cff
-
feat: refine model and benchmark exploration
03e2430
-
redesigned
3a12290