Commit History
-
leaderboard: admin rescore (selected + all) reusing the eval pipeline
Michael Rabinovich +1: CursorAdd a maintainer-only rescore that re-evaluates submissions against the current ground truth/data, reusing the submit worker end to end: each target row is reset to the pending regime (scores cleared, submitted_at preserved) and _spawn_worker is re-dispatched, which re-renders the gallery, regenerates reports/<id>.{html,json}, and recomputes the score. - rescore_rows(ids): selected rows; LookupError on unknown id before any write. rescore_all(): every row with a stored zip that isn't pending (skips legacy zip-less seed rows + in-flight evals). - Bulk dispatch is staggered on a background thread so a board-wide rescore doesn't fire N run_job calls at once; fully re-runnable, so an interrupted rescore converges on a retry. - UI in the Admin tab: "Rescore selected" behind a confirm checkbox and "Rescore ALL" behind a type-to-confirm phrase, both admin-gated with a server-side re-check and disarmed after firing / on load. Co-authored-by: Cursor <cursoragent@cursor.com>2893b22 -
leaderboard: rebuild Tasks tab as grouped thumbnail grid
Michael Rabinovich +1: CursorReplace the flat 84-row Tasks list with a responsive card grid grouped by type (Generation / Editing) with live count badges, a number search, and an All/Generation/Editing segmented filter. Each card shows the task's input geometry thumbnail (generation -> input.png; editing -> renders/iso.png, the iso render of input.step) plus the sample number and type tag. Thumbnails use loading="lazy" + decoding="async" with reserved 4:3 dimensions and CSS-constrained width so the 84 images don't all fetch at once. Card click opens the task's existing detail card (prompt + input), keeping j/k/Esc navigation. System fonts only. Co-authored-by: Cursor <cursoragent@cursor.com>95f3ee8 -
leaderboard: rename tabs, relabel fixtures as samples, inline gallery row stats
Michael Rabinovich +1: Cursor- Rename the visual "Gallery" tab to "Leaderboard" and the table tab to "Detailed View" (order unchanged); update the smoke test accordingly. - Replace user-facing "fixture" wording with "sample" across the gallery, task browser, submit instructions, and submitter status/error messages. Internal data keys, routes, CSS, and identifiers are unchanged. - Drop the gallery's "more numbers" expander: surface Generation/Editing inline under the aggregate score and move Download ZIP under the name. - Link each gallery submission title to its hosted report (same /reports/<id>.html the Detailed View table uses); drop the date line. Co-authored-by: Cursor <cursoragent@cursor.com>461547b -
leaderboard: serve renders from the public bucket, not the dataset proxy
Michael Rabinovich +1: CursorCandidate renders (turntables + edit-diff WebP) now live in the public HF Storage Bucket under renders/<id>/, uploaded once by the eval job. The gallery grid and the hosted report reference them by anonymous bucket URL, so the Space is out of the render read path and the binary renders stop bloating the submissions dataset / its commit queue. - leaderboard: HF_RENDER_BUCKET + render URL/path/prefix helpers. - app: render resolvers return public bucket URLs; the two render proxy routes + fetchers are retired. Only the private GT render keeps its token-held proxy. - submit: merge passes render_base_url to generate_html and commits report-only; dispatched job gets CADGENBENCH_RENDER_BUCKET/HF_ENDPOINT. - admin: delete also purges the bucket renders/<id>/ prefix (previously orphaned). - Dockerfile: bump cadgenbench pin to 3d49822 (render_base_url support). Co-authored-by: Cursor <cursoragent@cursor.com>d2161b1
-
leaderboard: add Tasks tab to browse benchmark fixtures
Michael Rabinovich +1: CursorA read-only task browser that mirrors the per-submission report's summary-table -> detail-card navigation (j/k, Esc, Prev/Next, deep-linkable) but shows only the prompt + input (drawing for generation, starting-shape renders for editing) with no scores or ground truth. Fixtures come from each <fixture>/description.yaml in the inputs dataset; input images are proxied through the Space's read token via /task-input and lazy-loaded. Co-authored-by: Cursor <cursoragent@cursor.com>f4924d6 -
Serve rotating WebP turntables + GT generator
Michael Rabinovich +1: CursorPoint the gallery and proxy routes at rotating.webp (submissions and GT), publish the full per-fixture renders folder, and add a reusable one-time GT turntable generator that renders from the cached GT mesh. Co-authored-by: Cursor <cursoragent@cursor.com>c1cb5e4 -
stage shards via bucket API instead of volume mount
Michael RabinovichAvoids Space volume-mount issues on an existing Space: shard jobs sync artifacts to the HF bucket via the bucket API, and the Space syncs them back down to merge. Gated on CADGENBENCH_SHARD_BUCKET.920b1b4 -
fix HF Jobs polling namespace
Michael Rabinovichdd284f3 -
leaderboard: open reports in new tab
Michael RabinovichNow that the Space is public, link completed submissions directly to their proxied HTML reports instead of inlining large report payloads through the row-click detail panel.5fb3ebc -
stage shard artifacts through HF bucket
Michael Rabinovich6ffd043 -
test: fix leaderboard smoke test for gallery-default tab + sr-only label
Michael Rabinovich +1: CursorGallery is now the default tab, so the Leaderboard content is hidden on load; click the Leaderboard tab before asserting. Scope the label locator to the visible <p> since the Leaderboard widget also renders a screen-reader-only <caption> with the same text (strict-mode 2-element violation otherwise). Co-authored-by: Cursor <cursoragent@cursor.com>1004adf -
gallery: searchable fixture picker, URL-persisted picks, cached render proxies
Michael Rabinovich +1: CursorReplace the always-open fixture chip grid with the compact reference UI: current picks as removable pills + an "Add fixture" button opening a searchable, grouped (Generation/Editing) dropdown, capped at 3 with a "remove one to add another" note. Persist the 3 picks in ?fixtures= so shared links reproduce the view; URL read/replaceState are try/catch guarded for sandboxed iframes. Drop the external Google Fonts @import for system font stacks so the page never errors in a sandboxed frame. Serve thumbnails as lazy-loaded, hard-cached /render and /gt-render proxy URLs (loading="lazy" + decoding="async" + CSS-constrained width) instead of base64-inlining every tile, so only the ~33 visible thumbnails load and fixture swaps / repeat visits hit cache. Requires the Space to be public. Fix a stale test expecting the old 6-tuple _admin_delete signature. Co-authored-by: Cursor <cursoragent@cursor.com>4a9408a -
admin: stop pending jobs before delete
Michael RabinovichAdd an admin action that cancels matching in-flight HF Jobs before deleting submissions, and keep the admin table refreshed so pending rows stay visible.c4b5d70 -
submit: stream live eval progress to the submitter's status panel
Michael Rabinovich +1: CursorAdds a small in-process progress registry (progress.py) that the eval worker publishes stage notes to (queued -> waiting for GPU -> evaluating -> collecting results -> done/failed, plus an "X of N chunks done" count for sharded runs). handle_submit keeps its generator alive after queuing and streams those notes into the personal status panel until a terminal stage or a backstop deadline. The shared leaderboard table stays coarse; this is the personal-view feedback only, so it adds no write load to the results file. Also fixes the _with_hub_retries tests for the newer huggingface_hub HfHubHTTPError(response=...) signature. Co-authored-by: Cursor <cursoragent@cursor.com>b0f4559 -
submit: retry Hub commits on 429/5xx + persistent status panel
Michael Rabinovich +1: CursorWrap every Space-side commit (zip upload, results.jsonl RMW, report + gallery uploads, shard cleanup) in a backoff-with-jitter retry that honors Retry-After, so a rate-limited submissions repo delays rather than fails a submit. Make handle_submit a generator streaming a durable status panel (validating -> uploading -> queued, or the rejection reason) instead of relying on transient toasts. Co-authored-by: Cursor <cursoragent@cursor.com>b224eee
-
submit: allow missing fixture outputs
Michael Rabinovichdc8ff2a -
gallery: refresh after admin changes
Michael Rabinovichdaae24c -
admin: refresh table on auth load
Michael Rabinovich02751ff
-
leaderboard: drop silent fallback; boot resilient on Hub read failure
Michael RabinovichThe leaderboard previously fell back to a bundled/local results.jsonl on any Hub error, which let an under-scoped Space HF_TOKEN silently serve stale baked-in rows that looked up-to-date. Remove the fallback: the live results.jsonl is the single source of truth and any read failure raises LeaderboardDataError. To stay robust rather than crash the Space at construction time, app.py wraps the readers (_safe_load_split / _safe_load_admin): a failed read yields empty, correctly-shaped tables plus a loud, persistent banner and a gr.Warning toast on refresh / Timer tick, and is logged via logger.exception. No stale or cached data is ever shown in place of a failed read. Also git rm the bundled results.jsonl and add regression tests for the no-fallback contract and the boot-resilience wrappers.a662bfa
-
admin: bulk promote/demote/delete + admin table loader
Michael Rabinovichadds load_admin_table to the leaderboard reader: a single flat frame of every row (both tiers) with a leading editable select column, for the admin tab to act on. reworks admin.py around bulk helpers (promote_rows, demote_rows, delete_rows) that each do one results.jsonl write for the whole selection; delete also best-effort removes the companion zip and report artifacts. the singular promote_row/demote_row stay as thin wrappers. tests cover the bulk paths, a missing-id abort, empty-selection guards, and artifact deletion, all with the hub mocked.0957a56 -
tests: cover the admin gate and promote/demote helpers
Michael Rabinovichfour unit tests for admin.py: the CADGENBENCH_ADMINS gate (in-set, out-of-set, logged-out, empty set), promote and demote happy paths, and idempotent re-promotion. the results.jsonl read and write that _hub_rmw_results drives are mocked, so the suite makes no hub calls.007a50e
-
leaderboard: format submitted_at as `YYYY-MM-DD HH:MM UTC`; lock tables read-only
Michael RabinovichTable cells were editable (gradio_leaderboard inherits gr.Dataframe's interactive default); add interactive=False to both Validated and Unvalidated widgets so cell contents aren't accidentally typed over. submitted_at column rendered as raw ISO-8601 ("2026-05-28T07:13:16Z"); apply the same _fmt_timestamp helper the detail panel already uses so the table reads as "2026-05-28 07:13 UTC". Helper moved into leaderboard.py (formatter module) and re-imported by app.py.c4e21b3 -
submit+app: gate Submit on HF OAuth; populate hf_username from profile
Michael RabinovichBundle 1+2 C10. Login-required submit shaped per the bundle spec. submit.py: - `handle_submit` grows a `profile: gr.OAuthProfile | None` arg. Gradio dependency-injects the visitor's profile based on the signature. None -> immediate `gr.Error("Please log in via the HF button before submitting.")` so a UI mishap can't write an anonymous row. - `_build_pending_row` grows an `hf_username=None` kwarg. `handle_submit` passes `profile.username` through so the row carries the canonical HF identity (not a free-text claim in meta.json's submitter_name). app.py: - New `gr.LoginButton()` in the Submit tab above the zip dropzone. - Submit button starts `interactive=False`; new `_enable_submit_when_logged_in(profile)` reads the visitor's OAuth profile via `blocks.load` and flips the button on / off per page load. Server-side gate in `handle_submit` mirrors this so a logged-out user who somehow bypasses the disabled-button UI still gets rejected. requirements.txt: - `gradio…c87b253
-
leaderboard+app: combined CSV download with validation_status discriminator
Michael RabinovichBundle 1+2 C8. One file, both tiers, downloaded via `gr.DownloadButton` next to the existing Refresh button. leaderboard.py: - New `CSV_COLUMNS` constant (wider than the on-screen table: raw aggregate_score / validity_rate, plus provenance fields like submission_sha256 + cadgenbench_data_revision + notes + failure_reason). `validation_status` is the discriminator that lets a reader grep the file for one tier or the other. - New `build_combined_csv()` reads the same rows the leaderboard reader does, applies the same status / validation_status defaults, projects to CSV_COLUMNS, sort: validated rows on top (highest score first) then unvalidated, then writes to a unique /tmp file and returns the path. Fresh on every click so the export reflects the latest data, not a snapshot captured at boot. app.py: - Imports `build_combined_csv`. The Refresh button and the new `gr.DownloadButton("Download CSV", size="sm")` live in a single `gr.Row` so they sit side-by-side unde…f585077 -
app: render reports inline via iframe srcdoc (private-Space safe)
Michael RabinovichPolish follow-up on C6. Reports now render inside the Space, inside the same iframe HF uses to host the leaderboard, with no second HTTP request leaving the browser. Why: the Space is private, and HF's edge gates same-origin pathname navigations on a short-lived __sign JWT that browsers don't carry forward when a user clicks a relative link. The previous design (submission_name -> /reports/<id>.html) worked server-side under Bearer auth (verified) but 404'd in the browser. Confirmed via external review and reproduced with a requests.Session cookie test. This commit: - Drops the submission_name -> /reports/<id>.html link wrapping from the rendered table; the cell stays plain text. The underlying `report_url` column (relative URL to the proxy) is still computed for the modern-pipeline gate but no longer surfaced as a clickable cell. - Adds a `gr.HTML` viewer below the detail panel. Row-click triggers `_format_detail_and_report`, which returns (detail_markdown, report_ifram…0e3b21f -
app+leaderboard: link submission_name to a Space-side report proxy
Michael RabinovichPolish follow-up #2 on C6. Three user-visible changes plus the plumbing required to make them work. 1. Drop the `report` and `zip` columns from the rendered table. `submission_name` becomes the report's deep link (typical HF leaderboard pattern: the system / model name doubles as the link target; cf. Open LLM Leaderboard, DABstep, BigCodeBench). Modern completed rows get `[<name>](/reports/<sid>.html)`; pending, failed, and legacy seed rows fall back to plain text so there's no broken click. The hidden `submission_blob_url` and the computed `report_url` ride along in the DataFrame and surface in the row-click detail panel only. 2. Rename the agent-URL column from `agent` to `model details (optional)`. Missing cells render as italic `_None_` so the optionality is explicit instead of a blank. "Agent" was confusing; the schema lets `agent_url` be code OR paper OR a blog post and "agent" reads narrow. 3. Add a Space-side proxy for the per-submission H…77edebf -
app+leaderboard: detail panel polish (rename link columns, fix broken report links)
Michael RabinovichPolish follow-up on C6, three fixes after the first detail-panel commit landed: 1. Rename display columns from snake_case schema names to clean short headers. `agent_url` -> `agent`, `submission_blob_url` -> `zip`, `report_url` -> `report`. The rename happens at projection time via `df.rename`, so the schema in results.jsonl is unchanged. Headers now read as plain English ("agent", "zip", "report") rather than a python dict. 2. The `agent` column shows the URL itself (scheme stripped, truncated past 40 chars) as link text instead of a fixed "code" label. The schema says agent_url can be code OR paper; the shortened URL is the only honest hint about what's behind the click. Missing values render as italic `_None_` rather than a blank cell, so the optionality is explicit. 3. The `report` link is now gated on the row's `submission_sha256` being non-null in addition to status == "completed". Legacy seed rows pre-date the modern submit pipeline (which …1a8f331 -
leaderboard: markdown link columns for agent_url, submission, report
Michael RabinovichBundle 1+2 C5. Three new columns on both tables (validated + unvalidated) carry clickable links rendered via gradio_leaderboard's `datatype="markdown"`: - agent_url: link to the agent's source / paper if the row's meta.json provided one (label "code"; empty cell when null). - submission_blob_url: link to the uploaded zip on the submissions dataset (label "zip"; populated on every non-failed row, the same URL the submit handler computed at upload time). - report_url: built at read time from `submission_id` plus HF_SUBMISSIONS_REPO; points at `reports/<id>.html` under /resolve/main/ so the browser renders the report directly. Only emitted for `status == "completed"` rows. leaderboard.py: - LEADERBOARD_COLS + VALIDATED_LEADERBOARD_COLS grow by three. - New LEADERBOARD_DATATYPES + VALIDATED_LEADERBOARD_DATATYPES constants mark the link columns as "markdown", everything else as "str". Tied to the column lists by a single helper so the two stay in lockstep. - New _agent_u…53de73a -
submit: default validation + hf_username fields on the pending row
Michael RabinovichBundle 1+2 C4. `_build_pending_row` adds three new keys matching the schema bump from C1, in the schema.md column order: - validation_status = "unvalidated" (every new row starts in the unvalidated tier; maintainers promote via the dataset repo after methodology review). - validation_method = None (set by the maintainer at promotion time to one of code | traces | api | manual). - hf_username = None (placeholder; C10 plumbs through the value from gradio.OAuthProfile.username once the LoginButton lands). Pre-existing fields, submitted_at stamping, the cadgenbench version + data revision tags, the score-shaped null fields, the blob URL, and the sha256 are untouched. Dedup gate still pivots on submission_sha256 exactly as before. tests/test_submit.py covers the C4 acceptance: - new-row defaults: validation_status is "unvalidated", validation_method + hf_username are None; - sha256 path passthrough: the hash argument lands on the row's submission_sha256 key intact (existing …a533dd2 -
app: split Leaderboard into Validated + Unvalidated tables
Michael RabinovichBundle 1+2 C3. Swaps the single Leaderboard widget for two stacked ones (Validated on top, Unvalidated below) and wires the existing auto-refresh Timer + Refresh button to push fresh dataframes into both. Initial render reads via the C2 `load_leaderboard_split()`; existing rows all land in Unvalidated until a maintainer flips `validation_status` to `"validated"` on a row in the submissions dataset. - app.py: replaces `df_view` with `validated_view` + `unvalidated_view`. Single timer tick now fans out to both widgets (load split runs once per tick, both halves get pushed). Refresh button same fan-out shape. Inline comment points at the validation policy decision doc. - leaderboard.py: drops the now-unused single-DataFrame `load_leaderboard()` per the C3 cleanup half. Stale docstring references that pointed at the dead function got smoothed out (the split reader's doc is self-contained now). - tests/test_smoke.py: picks up the C3 acceptance with a new `test_two_leaderboa…046548a -
leaderboard: add load_leaderboard_split() for two-tier viewer
Michael RabinovichBundle 1+2 C2. Additive new function for the C3 two-stacked-table swap; existing single-DataFrame load_leaderboard() stays put until C3 flips the caller in app.py + drops it. Keeps the live Space green between commits. - VALIDATED_LEADERBOARD_COLS: same shape as LEADERBOARD_COLS plus `validation_method` (always null on unvalidated rows, so omitted from that tier rather than rendered). - load_leaderboard_split() -> (validated_df, unvalidated_df): reads results.jsonl the same way as load_leaderboard(), splits rows on `validation_status` (defensive: anything not literally "validated" lands in unvalidated, which covers legacy + null + any future-unknown value), runs the same status-aware cell formatting per tier via the new _project_and_format helper. - _project_and_format: shared sort + projection + pending/failed cell tagging so the two tiers produce identically-shaped cells. Legacy default: rows without `validation_status` get "unvalidated" via row-level fill. No resul…0dd7215 -
tests: pytest + playwright headless smoke harness
Michael RabinovichFirst Space-side test harness; sets up the floor for the Phase D bundle's per-commit unit + UI tests (split leaderboard, pending-row defaults, OAuth-required submit) without slipping a regression past a build-only check. requirements-dev.txt pulls -r requirements.txt then adds pytest + playwright. One `pip install -r requirements-dev.txt && playwright install chromium` brings a fresh venv up to speed. tests/conftest.py exposes a session-scoped `app_url` fixture: boots app.py in a subprocess on a free port, polls until the HTTP server answers, yields the URL, tears the process down on teardown. Logs land in a tmp file so a startup failure surfaces the real Gradio traceback rather than a bare RuntimeError. Boot-time stuck-pending sweep in submit.py is disabled via CADGENBENCH_DISABLE_BOOT_SWEEP=1 so the fixture is fast and offline-tolerant. tests/test_smoke.py launches headless Chromium via playwright's sync API, navigates to the fixture's URL, and asserts the Leaderboard / Submit / A…21b071a