Commit History
-
Serve rotating WebP turntables + GT generator
Michael Rabinovich +1: CursorPoint the gallery and proxy routes at rotating.webp (submissions and GT), publish the full per-fixture renders folder, and add a reusable one-time GT turntable generator that renders from the cached GT mesh. Co-authored-by: Cursor <cursoragent@cursor.com>c1cb5e4 -
stage shards via bucket API instead of volume mount
Michael RabinovichAvoids Space volume-mount issues on an existing Space: shard jobs sync artifacts to the HF bucket via the bucket API, and the Space syncs them back down to merge. Gated on CADGENBENCH_SHARD_BUCKET.920b1b4 -
fix HF Jobs polling namespace
Michael Rabinovichdd284f3 -
leaderboard: open reports in new tab
Michael RabinovichNow that the Space is public, link completed submissions directly to their proxied HTML reports instead of inlining large report payloads through the row-click detail panel.5fb3ebc -
stage shard artifacts through HF bucket
Michael Rabinovich6ffd043 -
test: fix leaderboard smoke test for gallery-default tab + sr-only label
Michael Rabinovich +1: CursorGallery is now the default tab, so the Leaderboard content is hidden on load; click the Leaderboard tab before asserting. Scope the label locator to the visible <p> since the Leaderboard widget also renders a screen-reader-only <caption> with the same text (strict-mode 2-element violation otherwise). Co-authored-by: Cursor <cursoragent@cursor.com>1004adf -
gallery: searchable fixture picker, URL-persisted picks, cached render proxies
Michael Rabinovich +1: CursorReplace the always-open fixture chip grid with the compact reference UI: current picks as removable pills + an "Add fixture" button opening a searchable, grouped (Generation/Editing) dropdown, capped at 3 with a "remove one to add another" note. Persist the 3 picks in ?fixtures= so shared links reproduce the view; URL read/replaceState are try/catch guarded for sandboxed iframes. Drop the external Google Fonts @import for system font stacks so the page never errors in a sandboxed frame. Serve thumbnails as lazy-loaded, hard-cached /render and /gt-render proxy URLs (loading="lazy" + decoding="async" + CSS-constrained width) instead of base64-inlining every tile, so only the ~33 visible thumbnails load and fixture swaps / repeat visits hit cache. Requires the Space to be public. Fix a stale test expecting the old 6-tuple _admin_delete signature. Co-authored-by: Cursor <cursoragent@cursor.com>4a9408a -
admin: stop pending jobs before delete
Michael RabinovichAdd an admin action that cancels matching in-flight HF Jobs before deleting submissions, and keep the admin table refreshed so pending rows stay visible.c4b5d70 -
submit: stream live eval progress to the submitter's status panel
Michael Rabinovich +1: CursorAdds a small in-process progress registry (progress.py) that the eval worker publishes stage notes to (queued -> waiting for GPU -> evaluating -> collecting results -> done/failed, plus an "X of N chunks done" count for sharded runs). handle_submit keeps its generator alive after queuing and streams those notes into the personal status panel until a terminal stage or a backstop deadline. The shared leaderboard table stays coarse; this is the personal-view feedback only, so it adds no write load to the results file. Also fixes the _with_hub_retries tests for the newer huggingface_hub HfHubHTTPError(response=...) signature. Co-authored-by: Cursor <cursoragent@cursor.com>b0f4559 -
submit: retry Hub commits on 429/5xx + persistent status panel
Michael Rabinovich +1: CursorWrap every Space-side commit (zip upload, results.jsonl RMW, report + gallery uploads, shard cleanup) in a backoff-with-jitter retry that honors Retry-After, so a rate-limited submissions repo delays rather than fails a submit. Make handle_submit a generator streaming a durable status panel (validating -> uploading -> queued, or the rejection reason) instead of relying on transient toasts. Co-authored-by: Cursor <cursoragent@cursor.com>b224eee
-
submit: allow missing fixture outputs
Michael Rabinovichdc8ff2a -
gallery: refresh after admin changes
Michael Rabinovichdaae24c -
admin: refresh table on auth load
Michael Rabinovich02751ff
-
leaderboard: drop silent fallback; boot resilient on Hub read failure
Michael RabinovichThe leaderboard previously fell back to a bundled/local results.jsonl on any Hub error, which let an under-scoped Space HF_TOKEN silently serve stale baked-in rows that looked up-to-date. Remove the fallback: the live results.jsonl is the single source of truth and any read failure raises LeaderboardDataError. To stay robust rather than crash the Space at construction time, app.py wraps the readers (_safe_load_split / _safe_load_admin): a failed read yields empty, correctly-shaped tables plus a loud, persistent banner and a gr.Warning toast on refresh / Timer tick, and is logged via logger.exception. No stale or cached data is ever shown in place of a failed read. Also git rm the bundled results.jsonl and add regression tests for the no-fallback contract and the boot-resilience wrappers.a662bfa
-
admin: bulk promote/demote/delete + admin table loader
Michael Rabinovichadds load_admin_table to the leaderboard reader: a single flat frame of every row (both tiers) with a leading editable select column, for the admin tab to act on. reworks admin.py around bulk helpers (promote_rows, demote_rows, delete_rows) that each do one results.jsonl write for the whole selection; delete also best-effort removes the companion zip and report artifacts. the singular promote_row/demote_row stay as thin wrappers. tests cover the bulk paths, a missing-id abort, empty-selection guards, and artifact deletion, all with the hub mocked.0957a56 -
tests: cover the admin gate and promote/demote helpers
Michael Rabinovichfour unit tests for admin.py: the CADGENBENCH_ADMINS gate (in-set, out-of-set, logged-out, empty set), promote and demote happy paths, and idempotent re-promotion. the results.jsonl read and write that _hub_rmw_results drives are mocked, so the suite makes no hub calls.007a50e
-
leaderboard: format submitted_at as `YYYY-MM-DD HH:MM UTC`; lock tables read-only
Michael RabinovichTable cells were editable (gradio_leaderboard inherits gr.Dataframe's interactive default); add interactive=False to both Validated and Unvalidated widgets so cell contents aren't accidentally typed over. submitted_at column rendered as raw ISO-8601 ("2026-05-28T07:13:16Z"); apply the same _fmt_timestamp helper the detail panel already uses so the table reads as "2026-05-28 07:13 UTC". Helper moved into leaderboard.py (formatter module) and re-imported by app.py.c4e21b3 -
submit+app: gate Submit on HF OAuth; populate hf_username from profile
Michael RabinovichBundle 1+2 C10. Login-required submit shaped per the bundle spec. submit.py: - `handle_submit` grows a `profile: gr.OAuthProfile | None` arg. Gradio dependency-injects the visitor's profile based on the signature. None -> immediate `gr.Error("Please log in via the HF button before submitting.")` so a UI mishap can't write an anonymous row. - `_build_pending_row` grows an `hf_username=None` kwarg. `handle_submit` passes `profile.username` through so the row carries the canonical HF identity (not a free-text claim in meta.json's submitter_name). app.py: - New `gr.LoginButton()` in the Submit tab above the zip dropzone. - Submit button starts `interactive=False`; new `_enable_submit_when_logged_in(profile)` reads the visitor's OAuth profile via `blocks.load` and flips the button on / off per page load. Server-side gate in `handle_submit` mirrors this so a logged-out user who somehow bypasses the disabled-button UI still gets rejected. requirements.txt: - `gradio…c87b253
-
leaderboard+app: combined CSV download with validation_status discriminator
Michael RabinovichBundle 1+2 C8. One file, both tiers, downloaded via `gr.DownloadButton` next to the existing Refresh button. leaderboard.py: - New `CSV_COLUMNS` constant (wider than the on-screen table: raw aggregate_score / validity_rate, plus provenance fields like submission_sha256 + cadgenbench_data_revision + notes + failure_reason). `validation_status` is the discriminator that lets a reader grep the file for one tier or the other. - New `build_combined_csv()` reads the same rows the leaderboard reader does, applies the same status / validation_status defaults, projects to CSV_COLUMNS, sort: validated rows on top (highest score first) then unvalidated, then writes to a unique /tmp file and returns the path. Fresh on every click so the export reflects the latest data, not a snapshot captured at boot. app.py: - Imports `build_combined_csv`. The Refresh button and the new `gr.DownloadButton("Download CSV", size="sm")` live in a single `gr.Row` so they sit side-by-side unde…f585077 -
app: render reports inline via iframe srcdoc (private-Space safe)
Michael RabinovichPolish follow-up on C6. Reports now render inside the Space, inside the same iframe HF uses to host the leaderboard, with no second HTTP request leaving the browser. Why: the Space is private, and HF's edge gates same-origin pathname navigations on a short-lived __sign JWT that browsers don't carry forward when a user clicks a relative link. The previous design (submission_name -> /reports/<id>.html) worked server-side under Bearer auth (verified) but 404'd in the browser. Confirmed via external review and reproduced with a requests.Session cookie test. This commit: - Drops the submission_name -> /reports/<id>.html link wrapping from the rendered table; the cell stays plain text. The underlying `report_url` column (relative URL to the proxy) is still computed for the modern-pipeline gate but no longer surfaced as a clickable cell. - Adds a `gr.HTML` viewer below the detail panel. Row-click triggers `_format_detail_and_report`, which returns (detail_markdown, report_ifram…0e3b21f -
app+leaderboard: link submission_name to a Space-side report proxy
Michael RabinovichPolish follow-up #2 on C6. Three user-visible changes plus the plumbing required to make them work. 1. Drop the `report` and `zip` columns from the rendered table. `submission_name` becomes the report's deep link (typical HF leaderboard pattern: the system / model name doubles as the link target; cf. Open LLM Leaderboard, DABstep, BigCodeBench). Modern completed rows get `[<name>](/reports/<sid>.html)`; pending, failed, and legacy seed rows fall back to plain text so there's no broken click. The hidden `submission_blob_url` and the computed `report_url` ride along in the DataFrame and surface in the row-click detail panel only. 2. Rename the agent-URL column from `agent` to `model details (optional)`. Missing cells render as italic `_None_` so the optionality is explicit instead of a blank. "Agent" was confusing; the schema lets `agent_url` be code OR paper OR a blog post and "agent" reads narrow. 3. Add a Space-side proxy for the per-submission H…77edebf -
app+leaderboard: detail panel polish (rename link columns, fix broken report links)
Michael RabinovichPolish follow-up on C6, three fixes after the first detail-panel commit landed: 1. Rename display columns from snake_case schema names to clean short headers. `agent_url` -> `agent`, `submission_blob_url` -> `zip`, `report_url` -> `report`. The rename happens at projection time via `df.rename`, so the schema in results.jsonl is unchanged. Headers now read as plain English ("agent", "zip", "report") rather than a python dict. 2. The `agent` column shows the URL itself (scheme stripped, truncated past 40 chars) as link text instead of a fixed "code" label. The schema says agent_url can be code OR paper; the shortened URL is the only honest hint about what's behind the click. Missing values render as italic `_None_` rather than a blank cell, so the optionality is explicit. 3. The `report` link is now gated on the row's `submission_sha256` being non-null in addition to status == "completed". Legacy seed rows pre-date the modern submit pipeline (which …1a8f331 -
leaderboard: markdown link columns for agent_url, submission, report
Michael RabinovichBundle 1+2 C5. Three new columns on both tables (validated + unvalidated) carry clickable links rendered via gradio_leaderboard's `datatype="markdown"`: - agent_url: link to the agent's source / paper if the row's meta.json provided one (label "code"; empty cell when null). - submission_blob_url: link to the uploaded zip on the submissions dataset (label "zip"; populated on every non-failed row, the same URL the submit handler computed at upload time). - report_url: built at read time from `submission_id` plus HF_SUBMISSIONS_REPO; points at `reports/<id>.html` under /resolve/main/ so the browser renders the report directly. Only emitted for `status == "completed"` rows. leaderboard.py: - LEADERBOARD_COLS + VALIDATED_LEADERBOARD_COLS grow by three. - New LEADERBOARD_DATATYPES + VALIDATED_LEADERBOARD_DATATYPES constants mark the link columns as "markdown", everything else as "str". Tied to the column lists by a single helper so the two stay in lockstep. - New _agent_u…53de73a -
submit: default validation + hf_username fields on the pending row
Michael RabinovichBundle 1+2 C4. `_build_pending_row` adds three new keys matching the schema bump from C1, in the schema.md column order: - validation_status = "unvalidated" (every new row starts in the unvalidated tier; maintainers promote via the dataset repo after methodology review). - validation_method = None (set by the maintainer at promotion time to one of code | traces | api | manual). - hf_username = None (placeholder; C10 plumbs through the value from gradio.OAuthProfile.username once the LoginButton lands). Pre-existing fields, submitted_at stamping, the cadgenbench version + data revision tags, the score-shaped null fields, the blob URL, and the sha256 are untouched. Dedup gate still pivots on submission_sha256 exactly as before. tests/test_submit.py covers the C4 acceptance: - new-row defaults: validation_status is "unvalidated", validation_method + hf_username are None; - sha256 path passthrough: the hash argument lands on the row's submission_sha256 key intact (existing …a533dd2 -
app: split Leaderboard into Validated + Unvalidated tables
Michael RabinovichBundle 1+2 C3. Swaps the single Leaderboard widget for two stacked ones (Validated on top, Unvalidated below) and wires the existing auto-refresh Timer + Refresh button to push fresh dataframes into both. Initial render reads via the C2 `load_leaderboard_split()`; existing rows all land in Unvalidated until a maintainer flips `validation_status` to `"validated"` on a row in the submissions dataset. - app.py: replaces `df_view` with `validated_view` + `unvalidated_view`. Single timer tick now fans out to both widgets (load split runs once per tick, both halves get pushed). Refresh button same fan-out shape. Inline comment points at the validation policy decision doc. - leaderboard.py: drops the now-unused single-DataFrame `load_leaderboard()` per the C3 cleanup half. Stale docstring references that pointed at the dead function got smoothed out (the split reader's doc is self-contained now). - tests/test_smoke.py: picks up the C3 acceptance with a new `test_two_leaderboa…046548a -
leaderboard: add load_leaderboard_split() for two-tier viewer
Michael RabinovichBundle 1+2 C2. Additive new function for the C3 two-stacked-table swap; existing single-DataFrame load_leaderboard() stays put until C3 flips the caller in app.py + drops it. Keeps the live Space green between commits. - VALIDATED_LEADERBOARD_COLS: same shape as LEADERBOARD_COLS plus `validation_method` (always null on unvalidated rows, so omitted from that tier rather than rendered). - load_leaderboard_split() -> (validated_df, unvalidated_df): reads results.jsonl the same way as load_leaderboard(), splits rows on `validation_status` (defensive: anything not literally "validated" lands in unvalidated, which covers legacy + null + any future-unknown value), runs the same status-aware cell formatting per tier via the new _project_and_format helper. - _project_and_format: shared sort + projection + pending/failed cell tagging so the two tiers produce identically-shaped cells. Legacy default: rows without `validation_status` get "unvalidated" via row-level fill. No resul…0dd7215 -
tests: pytest + playwright headless smoke harness
Michael RabinovichFirst Space-side test harness; sets up the floor for the Phase D bundle's per-commit unit + UI tests (split leaderboard, pending-row defaults, OAuth-required submit) without slipping a regression past a build-only check. requirements-dev.txt pulls -r requirements.txt then adds pytest + playwright. One `pip install -r requirements-dev.txt && playwright install chromium` brings a fresh venv up to speed. tests/conftest.py exposes a session-scoped `app_url` fixture: boots app.py in a subprocess on a free port, polls until the HTTP server answers, yields the URL, tears the process down on teardown. Logs land in a tmp file so a startup failure surfaces the real Gradio traceback rather than a bare RuntimeError. Boot-time stuck-pending sweep in submit.py is disabled via CADGENBENCH_DISABLE_BOOT_SWEEP=1 so the fixture is fast and offline-tolerant. tests/test_smoke.py launches headless Chromium via playwright's sync API, navigates to the fixture's URL, and asserts the Leaderboard / Submit / A…21b071a