Commit History
-
Bump CADGENBENCH_SHA to 037eade; add GT edit-diff report backfill tool
Michael Rabinovich +1: CursorPins the leaderboard image to the cadgenbench commit that renders the GT answer-key edit-diff report column. Adds tools/backfill_report_gt_edit_diff.py, the one-off (idempotent) patcher that swaps existing reports' editing GT columns to the answer-key turntable without re-evaluating. Co-authored-by: Cursor <cursoragent@cursor.com>b885669 -
Gallery: show GT "answer-key" edit-diff for editing fixtures
Michael Rabinovich +1: CursorThe ground-truth row now renders the blue edit-diff turntable (the correct change vs the input) for editing fixtures instead of a plain GT turntable, so the reference reads the same way as the candidate red/amber diff in the same column. Adds a gt_diff_resolver pointing at the one-time GT-dataset webp (<fixture>/renders/edit_diff_gt.webp, via the existing /gt proxy) and updates the compare modal caption/legend. tools/generate_gt_edit_diff.py renders/uploads those webps for the 32 editing fixtures (one-time, per data revision). Co-authored-by: Cursor <cursoragent@cursor.com>49e27be -
Rename fixture-named result keys to sample-named keys
Michael Rabinovich +1: CursorRead/write per_sample_scores, per_sample_breakdown, n_samples, per_sample_results across submit/gallery/admin and tools. Also link the canonical submission docs from the Submit tab. Co-authored-by: Cursor <cursoragent@cursor.com>be6fa3d
-
leaderboard: fixed 4-sample gallery, auto-size iframe, center modal
Michael Rabinovich +1: CursorReplace the gallery's interactive fixture picker with a fixed comparison sheet: two generation + two editing samples, one Medium and one Hard per task. Difficulty is chosen once from the Claude Opus 4.8 baseline's per-sample CAD scores (median = Medium, 20th percentile = Hard) and frozen into FIXED_FIXTURES; the GUI no longer recomputes it as the leaderboard evolves, trading a moving ground truth for a simple, stable interface. Adds tools/pick_gallery_fixtures.py to regenerate the constant. Size the gallery iframe to its content (no more 90vh fixed box / nested scrollbar) and pin the compare modal to the visible part of the iframe so it opens centered in view instead of far below the fold. Co-authored-by: Cursor <cursoragent@cursor.com>08eae45
-
leaderboard: backfill tool to grid-ify already-published reports
Michael Rabinovich +1: CursorRewrites the summary view of published reports/<id>.html in the submissions dataset from the old flat table to the thumbnail grid, in place and without re-evaluating. Parses each report's existing rows (sample/status/CAD) + the render-bucket base already embedded in the file, classifies editing by the starting-shape renders (so invalid edits group correctly), and rebuilds the grid via single_run's shared builders (byte-identical to a fresh report), pointing editing cards at edit_diff.png and generation cards at output iso. Injects the shared grid CSS/JS; detail cards, header and download button are left untouched. Supports --files and --dataset (--dry-run); idempotent. Co-authored-by: Cursor <cursoragent@cursor.com>ba3eefb -
leaderboard: backfill tool for edit_diff.png stills in render bucket
Michael Rabinovich +1: CursorOne-time tool to close the gap for submissions evaluated before the eval pipeline started writing edit_diff.png: lists the render bucket, finds every <fixture>/edit_diff.webp lacking a sibling edit_diff.png, extracts frame 0 via cadgenbench.common.imaging.first_frame_png (the exact frame the pipeline now saves), and uploads the still to the same bucket prefix. Idempotent (skips existing stills); --dry-run lists without a token. Co-authored-by: Cursor <cursoragent@cursor.com>c7f83a5 -
leaderboard: bump cadgenbench to 1010043 + legend backfill tool
Michael Rabinovich +1: CursorPin to the report generator with color-chip interface/edit-diff legends, and add tools/backfill_report_legends.py to retro-patch the corrected legends into already-published reports/<id>.html in place (no re-eval). Co-authored-by: Cursor <cursoragent@cursor.com>1a24d4a -
leaderboard: download-zip button in merge-path report + backfill tool
Michael Rabinovich +1: CursorPass the submission-zip URL to generate_html in the merge path so reassembled reports show the "Download submission ZIP" button, and add a one-off tools/backfill_report_download_button.py that injects the button into already-published reports/<id>.html in place (no re-eval). Bump CADGENBENCH_SHA to 0c7690e. Co-authored-by: Cursor <cursoragent@cursor.com>c48c18f
-
leaderboard: add Tasks tab to browse benchmark fixtures
Michael Rabinovich +1: CursorA read-only task browser that mirrors the per-submission report's summary-table -> detail-card navigation (j/k, Esc, Prev/Next, deep-linkable) but shows only the prompt + input (drawing for generation, starting-shape renders for editing) with no scores or ground truth. Fixtures come from each <fixture>/description.yaml in the inputs dataset; input images are proxied through the Space's read token via /task-input and lazy-loaded. Co-authored-by: Cursor <cursoragent@cursor.com>f4924d6 -
Serve rotating WebP turntables + GT generator
Michael Rabinovich +1: CursorPoint the gallery and proxy routes at rotating.webp (submissions and GT), publish the full per-fixture renders folder, and add a reusable one-time GT turntable generator that renders from the cached GT mesh. Co-authored-by: Cursor <cursoragent@cursor.com>c1cb5e4