Commit History
-
fix(oauth): use real HF OAuth on this Docker Space instead of mock login
Michael Rabinovich +1: CursorGradio's get_space() only returns truthy when SYSTEM=="spaces", which HF sets on Gradio-SDK Spaces but not on sdk: docker Spaces like this one. Without it, mount_gradio_app wired up Gradio's MOCK OAuth routes: they never contact hf.co and log every visitor in as the container token's owner (our HF_TOKEN account). Because that account is in CADGENBENCH_ADMINS, this leaked the "Logout (michaelr27)" identity to all visitors and granted admin to anyone who completed the (mock) login. Set SYSTEM=spaces when running on a Space (SPACE_ID present) so the real hf_oauth: true flow runs; left unset locally so the local mock login still works for dev. Co-authored-by: Cursor <cursoragent@cursor.com>f17ac64 -
leaderboard: make the gallery the single scroller
Michael Rabinovich +1: CursorSizing the iframe to a fixed 80vh left two scrollbars (the iframe body and the outer Gradio page), which was confusing. Now fitIframe measures the available height from the iframe's own top down to the bottom of the viewport (via the same-origin parent) and sizes the iframe to min(content, available), reserving room for the Refresh button. Short content shrinks the iframe (nothing scrolls); tall content fills the viewport so only the iframe body scrolls and the sticky header + GT row stay locked. The outer page no longer needs its own scrollbar. Co-authored-by: Cursor <cursoragent@cursor.com>5eaf6b8 -
leaderboard: lock GT row via in-iframe scroll capped by max-height
Michael Rabinovich +1: CursorThe previous attempt capped the gallery scroll box from JS using window.parent.innerHeight, but inside HF's nested iframes that is not the browser viewport, so the cap came out huge: the box never scrolled, the iframe grew to full height, and the page scrolled the GT row away. Instead, scroll inside the iframe itself (the column header + GT row are sticky against the iframe viewport, which reliably works) and cap the iframe with `max-height: 80vh` in app.py, where vh is the Gradio doc viewport. fitIframe still shrinks the iframe to content for a compact page when there are few rows; once content exceeds the cap the iframe body scrolls and the GT row stays pinned. Drops the unreliable JS cap and the parent-aware modal positioning (back to plain fixed centering). Co-authored-by: Cursor <cursoragent@cursor.com>a0462d9
-
leaderboard: lock GT row + header in a capped gallery scroll box
Michael Rabinovich +1: CursorMake the gallery a scroll container capped at ~78% of the visible viewport so the column header and ground-truth row stay pinned (sticky) at the top while submissions scroll inside it. The cap is set from the parent viewport in JS and the iframe is sized to the (bounded) content, so the page stays compact and shows no scrollbar when the rows are shorter than the cap. Co-authored-by: Cursor <cursoragent@cursor.com>5dd0c03 -
leaderboard: fixed 4-sample gallery, auto-size iframe, center modal
Michael Rabinovich +1: CursorReplace the gallery's interactive fixture picker with a fixed comparison sheet: two generation + two editing samples, one Medium and one Hard per task. Difficulty is chosen once from the Claude Opus 4.8 baseline's per-sample CAD scores (median = Medium, 20th percentile = Hard) and frozen into FIXED_FIXTURES; the GUI no longer recomputes it as the leaderboard evolves, trading a moving ground truth for a simple, stable interface. Adds tools/pick_gallery_fixtures.py to regenerate the constant. Size the gallery iframe to its content (no more 90vh fixed box / nested scrollbar) and pin the compare modal to the visible part of the iframe so it opens centered in view instead of far below the fold. Co-authored-by: Cursor <cursoragent@cursor.com>08eae45 -
deploy: bump cadgenbench to 72f6e82 (scale-relative edit-diff tol, viz polish)
Michael Rabinovich +1: CursorCo-authored-by: Cursor <cursoragent@cursor.com>96dcaf6 -
deploy: bump cadgenbench to 850281d (unified viz colours)
Michael Rabinovich +1: CursorCo-authored-by: Cursor <cursoragent@cursor.com>4bfd457
-
leaderboard: bump cadgenbench to d514d6a (report thumbnail grid)
Michael Rabinovich +1: CursorPins the cadgenbench install to the report-grid commit so newly generated hosted reports render the grouped thumbnail-grid summary view. Excludes the later f9b142b scoring change deliberately. Co-authored-by: Cursor <cursoragent@cursor.com>fd0c525 -
leaderboard: backfill tool to grid-ify already-published reports
Michael Rabinovich +1: CursorRewrites the summary view of published reports/<id>.html in the submissions dataset from the old flat table to the thumbnail grid, in place and without re-evaluating. Parses each report's existing rows (sample/status/CAD) + the render-bucket base already embedded in the file, classifies editing by the starting-shape renders (so invalid edits group correctly), and rebuilds the grid via single_run's shared builders (byte-identical to a fresh report), pointing editing cards at edit_diff.png and generation cards at output iso. Injects the shared grid CSS/JS; detail cards, header and download button are left untouched. Supports --files and --dataset (--dry-run); idempotent. Co-authored-by: Cursor <cursoragent@cursor.com>ba3eefb -
leaderboard: admin rescore (selected + all) reusing the eval pipeline
Michael Rabinovich +1: CursorAdd a maintainer-only rescore that re-evaluates submissions against the current ground truth/data, reusing the submit worker end to end: each target row is reset to the pending regime (scores cleared, submitted_at preserved) and _spawn_worker is re-dispatched, which re-renders the gallery, regenerates reports/<id>.{html,json}, and recomputes the score. - rescore_rows(ids): selected rows; LookupError on unknown id before any write. rescore_all(): every row with a stored zip that isn't pending (skips legacy zip-less seed rows + in-flight evals). - Bulk dispatch is staggered on a background thread so a board-wide rescore doesn't fire N run_job calls at once; fully re-runnable, so an interrupted rescore converges on a retry. - UI in the Admin tab: "Rescore selected" behind a confirm checkbox and "Rescore ALL" behind a type-to-confirm phrase, both admin-gated with a server-side re-check and disarmed after firing / on load. Co-authored-by: Cursor <cursoragent@cursor.com>2893b22 -
leaderboard: backfill tool for edit_diff.png stills in render bucket
Michael Rabinovich +1: CursorOne-time tool to close the gap for submissions evaluated before the eval pipeline started writing edit_diff.png: lists the render bucket, finds every <fixture>/edit_diff.webp lacking a sibling edit_diff.png, extracts frame 0 via cadgenbench.common.imaging.first_frame_png (the exact frame the pipeline now saves), and uploads the still to the same bucket prefix. Idempotent (skips existing stills); --dry-run lists without a token. Co-authored-by: Cursor <cursoragent@cursor.com>c7f83a5 -
leaderboard: bump cadgenbench to 1010043 + legend backfill tool
Michael Rabinovich +1: CursorPin to the report generator with color-chip interface/edit-diff legends, and add tools/backfill_report_legends.py to retro-patch the corrected legends into already-published reports/<id>.html in place (no re-eval). Co-authored-by: Cursor <cursoragent@cursor.com>1a24d4a -
leaderboard: rebuild Tasks tab as grouped thumbnail grid
Michael Rabinovich +1: CursorReplace the flat 84-row Tasks list with a responsive card grid grouped by type (Generation / Editing) with live count badges, a number search, and an All/Generation/Editing segmented filter. Each card shows the task's input geometry thumbnail (generation -> input.png; editing -> renders/iso.png, the iso render of input.step) plus the sample number and type tag. Thumbnails use loading="lazy" + decoding="async" with reserved 4:3 dimensions and CSS-constrained width so the 84 images don't all fetch at once. Card click opens the task's existing detail card (prompt + input), keeping j/k/Esc navigation. System fonts only. Co-authored-by: Cursor <cursoragent@cursor.com>95f3ee8 -
leaderboard: download-zip button in merge-path report + backfill tool
Michael Rabinovich +1: CursorPass the submission-zip URL to generate_html in the merge path so reassembled reports show the "Download submission ZIP" button, and add a one-off tools/backfill_report_download_button.py that injects the button into already-published reports/<id>.html in place (no re-eval). Bump CADGENBENCH_SHA to 0c7690e. Co-authored-by: Cursor <cursoragent@cursor.com>c48c18f -
leaderboard: rename tabs, relabel fixtures as samples, inline gallery row stats
Michael Rabinovich +1: Cursor- Rename the visual "Gallery" tab to "Leaderboard" and the table tab to "Detailed View" (order unchanged); update the smoke test accordingly. - Replace user-facing "fixture" wording with "sample" across the gallery, task browser, submit instructions, and submitter status/error messages. Internal data keys, routes, CSS, and identifiers are unchanged. - Drop the gallery's "more numbers" expander: surface Generation/Editing inline under the aggregate score and move Download ZIP under the name. - Link each gallery submission title to its hosted report (same /reports/<id>.html the Detailed View table uses); drop the date line. Co-authored-by: Cursor <cursoragent@cursor.com>461547b -
leaderboard: serve GT report assets via proxy; link them in hosted report
Michael Rabinovich +1: CursorAdd a /gt/{fixture}/{relpath} proxy route that streams the private GT view PNGs + PDF (token-holding), and pass gt_base_url/input_base_url to generate_html in the merge path so the hosted report links GT/input/overlay (lazy) instead of base64-inlining them. Bump CADGENBENCH_SHA to 25943a0. Co-authored-by: Cursor <cursoragent@cursor.com>5140b0a -
leaderboard: serve renders from the public bucket, not the dataset proxy
Michael Rabinovich +1: CursorCandidate renders (turntables + edit-diff WebP) now live in the public HF Storage Bucket under renders/<id>/, uploaded once by the eval job. The gallery grid and the hosted report reference them by anonymous bucket URL, so the Space is out of the render read path and the binary renders stop bloating the submissions dataset / its commit queue. - leaderboard: HF_RENDER_BUCKET + render URL/path/prefix helpers. - app: render resolvers return public bucket URLs; the two render proxy routes + fetchers are retired. Only the private GT render keeps its token-held proxy. - submit: merge passes render_base_url to generate_html and commits report-only; dispatched job gets CADGENBENCH_RENDER_BUCKET/HF_ENDPOINT. - admin: delete also purges the bucket renders/<id>/ prefix (previously orphaned). - Dockerfile: bump cadgenbench pin to 3d49822 (render_base_url support). Co-authored-by: Cursor <cursoragent@cursor.com>d2161b1
-
leaderboard: show edit-diff turntable in gallery grid for editing fixtures
Michael Rabinovich +1: CursorThe gallery grid tile for an editing fixture now serves the ghost edit-diff WebP (renders/<id>/<fixture>/edit_diff.webp) instead of the plain candidate turntable, so a small edit is legible at a glance. Generation tiles, the GT column, and the cell-click modal are unchanged (the modal keeps the plain aligned output via img; the grid uses a new gridImg). - gallery: per-cell gridImg + gridRenderFor hook; build/render take a diff_resolver. - app: _fetch_render_diff + /render-diff proxy route + resolver. A miss 404s to the dashed cell (no fallback to rotating.webp). Co-authored-by: Cursor <cursoragent@cursor.com>e611f15 -
leaderboard: move Tasks tab after Leaderboard
Michael Rabinovich +1: CursorCo-authored-by: Cursor <cursoragent@cursor.com>31854f7 -
leaderboard: add Tasks tab to browse benchmark fixtures
Michael Rabinovich +1: CursorA read-only task browser that mirrors the per-submission report's summary-table -> detail-card navigation (j/k, Esc, Prev/Next, deep-linkable) but shows only the prompt + input (drawing for generation, starting-shape renders for editing) with no scores or ground truth. Fixtures come from each <fixture>/description.yaml in the inputs dataset; input images are proxied through the Space's read token via /task-input and lazy-loaded. Co-authored-by: Cursor <cursoragent@cursor.com>f4924d6 -
Serve rotating WebP turntables + GT generator
Michael Rabinovich +1: CursorPoint the gallery and proxy routes at rotating.webp (submissions and GT), publish the full per-fixture renders folder, and add a reusable one-time GT turntable generator that renders from the cached GT mesh. Co-authored-by: Cursor <cursoragent@cursor.com>c1cb5e4 -
stage shards via bucket API instead of volume mount
Michael RabinovichAvoids Space volume-mount issues on an existing Space: shard jobs sync artifacts to the HF bucket via the bucket API, and the Space syncs them back down to merge. Gated on CADGENBENCH_SHARD_BUCKET.920b1b4 -
fix HF Jobs polling namespace
Michael Rabinovichdd284f3 -
leaderboard: open reports in new tab
Michael RabinovichNow that the Space is public, link completed submissions directly to their proxied HTML reports instead of inlining large report payloads through the row-click detail panel.5fb3ebc -
stage shard artifacts through HF bucket
Michael Rabinovich6ffd043 -
test: fix leaderboard smoke test for gallery-default tab + sr-only label
Michael Rabinovich +1: CursorGallery is now the default tab, so the Leaderboard content is hidden on load; click the Leaderboard tab before asserting. Scope the label locator to the visible <p> since the Leaderboard widget also renders a screen-reader-only <caption> with the same text (strict-mode 2-element violation otherwise). Co-authored-by: Cursor <cursoragent@cursor.com>1004adf -
gallery: searchable fixture picker, URL-persisted picks, cached render proxies
Michael Rabinovich +1: CursorReplace the always-open fixture chip grid with the compact reference UI: current picks as removable pills + an "Add fixture" button opening a searchable, grouped (Generation/Editing) dropdown, capped at 3 with a "remove one to add another" note. Persist the 3 picks in ?fixtures= so shared links reproduce the view; URL read/replaceState are try/catch guarded for sandboxed iframes. Drop the external Google Fonts @import for system font stacks so the page never errors in a sandboxed frame. Serve thumbnails as lazy-loaded, hard-cached /render and /gt-render proxy URLs (loading="lazy" + decoding="async" + CSS-constrained width) instead of base64-inlining every tile, so only the ~33 visible thumbnails load and fixture swaps / repeat visits hit cache. Requires the Space to be public. Fix a stale test expecting the old 6-tuple _admin_delete signature. Co-authored-by: Cursor <cursoragent@cursor.com>4a9408a -
admin: stop pending jobs before delete
Michael RabinovichAdd an admin action that cancels matching in-flight HF Jobs before deleting submissions, and keep the admin table refreshed so pending rows stay visible.c4b5d70 -
submit: stream live eval progress to the submitter's status panel
Michael Rabinovich +1: CursorAdds a small in-process progress registry (progress.py) that the eval worker publishes stage notes to (queued -> waiting for GPU -> evaluating -> collecting results -> done/failed, plus an "X of N chunks done" count for sharded runs). handle_submit keeps its generator alive after queuing and streams those notes into the personal status panel until a terminal stage or a backstop deadline. The shared leaderboard table stays coarse; this is the personal-view feedback only, so it adds no write load to the results file. Also fixes the _with_hub_retries tests for the newer huggingface_hub HfHubHTTPError(response=...) signature. Co-authored-by: Cursor <cursoragent@cursor.com>b0f4559 -
submit: publish merged report + gallery in one commit
Michael Rabinovich +1: CursorReplace the per-file gallery-thumbnail uploads (one upload_file commit per fixture, ~80+ for a full fan-out) with a single create_commit that stages reports/<id>.{html,json} plus every renders/<id>/<fixture>.png at once. Collapses finalize from ~86 commits to one publish commit, cutting post-shard finalize from minutes to well under a minute and removing the 429 "concurrency queue" failures that stranded earlier submissions. Co-authored-by: Cursor <cursoragent@cursor.com>07430f9 -
submit: retry Hub commits on 429/5xx + persistent status panel
Michael Rabinovich +1: CursorWrap every Space-side commit (zip upload, results.jsonl RMW, report + gallery uploads, shard cleanup) in a backoff-with-jitter retry that honors Retry-After, so a rate-limited submissions repo delays rather than fails a submit. Make handle_submit a generator streaming a durable status panel (validating -> uploading -> queued, or the rejection reason) instead of relying on transient toasts. Co-authored-by: Cursor <cursoragent@cursor.com>b224eee
-
submit: allow missing fixture outputs
Michael Rabinovichdc8ff2a -
gallery: refresh after admin changes
Michael Rabinovichdaae24c -
pin cadgenbench to e446cca (benchmark-v1 aligner + sidecar meshes)
Michael Rabinovich +1: CursorCo-authored-by: Cursor <cursoragent@cursor.com>30909aa -
admin: refresh table on auth load
Michael Rabinovich02751ff
-
submit: shard large submissions across HF Jobs + merge (UC3)
Michael Rabinovich +1: CursorSubmissions over SHARD_THRESHOLD (12) fixtures now fan out into 12-fixture shards dispatched at once, polled to terminal with ERROR-only retry, then merged into one run via write_run_summary (regenerating the report + gallery from the merged whole) before the row flips. At/under the threshold a submission stays a single job, so the original path is unchanged. Eval is CPU-bound, so more shards is the throughput lever. Co-authored-by: Cursor <cursoragent@cursor.com>293bc3b -
gallery: stop negative-caching render fetches
Michael Rabinovich +1: CursorRenders land after boot (a submission completes, or an existing row is backfilled), so lru_cache was pinning boot-time 404s and keeping thumbnails dashed until the next restart. Drop the memoization on the render byte-fetchers; hf_hub_download keeps its own per-revision disk cache, so re-fetching an unchanged thumbnail stays cheap. Also picks up the 3 baseline rows' just-backfilled renders on this redeploy. Co-authored-by: Cursor <cursoragent@cursor.com>8eb8954 -
add visual Gallery tab (top-10 verified, sticky GT, fixture picker)
Michael Rabinovich +1: CursorVisual-first leaderboard as a new default tab. Self-contained HTML (gallery.py) inlined into an iframe srcdoc with base64 thumbnails, so it works on the private Space where HF's edge 404s in-browser custom-route fetches (same constraint the report viewer's srcdoc path already handles). Top-10 verified only, teal sticky ground-truth row, 3-fixture picker (drop-oldest, global re-render), dashed cell for invalid/missing fixtures, per-row generation/editing/validity expander, and a GT-vs-output compare modal per thumbnail. Existing Leaderboard / Submit / About / Admin tabs unchanged. Co-authored-by: Cursor <cursoragent@cursor.com>01d67e9 -
docker: cache-bust the cadgenbench @main install
Michael Rabinovich +1: CursorADD the GitHub commits endpoint for the tracked ref before the pip install so BuildKit invalidates the cached layer whenever main moves; otherwise @main would stay pinned to the first build's HEAD. No-op when CADGENBENCH_SHA is a fixed commit SHA. Co-authored-by: Cursor <cursoragent@cursor.com>3909559 -
docker: track cadgenbench @main instead of a pinned SHA
Michael Rabinovich +1: CursorPre-v1 the Space should always rebuild against the latest cadgenbench. Default CADGENBENCH_SHA to `main`; lock to a specific commit SHA at the v1 release for reproducible scores (post-gt-swap.md Stage F). Co-authored-by: Cursor <cursoragent@cursor.com>40dd166 -
docker: stop COPYing the removed results.jsonl
Michael RabinovichThe bundled results.jsonl was git rm'd along with the leaderboard's local fallback, but the Dockerfile still listed it in the final COPY, so the build failed with `"/results.jsonl": not found`. Drop it from the COPY; only the *.py sources are needed now.3e18d9a -
leaderboard: drop silent fallback; boot resilient on Hub read failure
Michael RabinovichThe leaderboard previously fell back to a bundled/local results.jsonl on any Hub error, which let an under-scoped Space HF_TOKEN silently serve stale baked-in rows that looked up-to-date. Remove the fallback: the live results.jsonl is the single source of truth and any read failure raises LeaderboardDataError. To stay robust rather than crash the Space at construction time, app.py wraps the readers (_safe_load_split / _safe_load_admin): a failed read yields empty, correctly-shaped tables plus a loud, persistent banner and a gr.Warning toast on refresh / Timer tick, and is logged via logger.exception. No stale or cached data is ever shown in place of a failed read. Also git rm the bundled results.jsonl and add regression tests for the no-fallback contract and the boot-resilience wrappers.a662bfa
-
submit: derive eval-gpu image repo from HF_ORG
Michael Rabinovichthe image lives in the org, so the repo id tracks HF_ORG (matching leaderboard.py) and an org rename stays one space-variable change. on the space HF_ORG is set, so this resolves to the same value as before. the job namespace stays a fixed michaelr27 constant since dispatch is billed to that account. verified resolution for the space, override, and no-HF_ORG cases.c78e980 -
submit: keep eval-gpu image repo as a fixed literal
Michael Rabinovichrevert the env-var indirection; the audit sweep stays prose-only and must not alter dispatch resolution. the repo id and job namespace are real deployed identities, kept as documented constants.bd9b6ba -
submit: derive eval-gpu image repo from HF_ORG
Michael Rabinovich0a04e61 -
leaderboard: log the local-fallback path instead of printing
Michael Rabinovich3dba2e2
-
license: add apache-2.0 header to space modules
Michael Rabinovichb048bc2 -
app: use a colon in the submit-tab system-agnostic note
Michael Rabinovich8a21dae -
readme: describe the space card in client-facing terms
Michael Rabinovich6d31809 -
app: default the admin validation_method to manual
Michael Rabinovichmanual (we re-ran the agent ourselves) is the common case for the maintainer team, so preselect it on the promote radio.46e3a72