Commit History
-
leaderboard: add Metrics tab/page + report deep-links
Michael Rabinovich +1: CursorNew self-contained Metrics explainer (metrics_page.py) served at /metrics and embedded in a Metrics tab: validity gate, the three axes, editing renormalization, with formulas and the interface mating-group illustration (vendored under assets/metrics/ via LFS, served by the /metrics-assets route). Submit's merge path passes the submission name and the /metrics base URL to the report generator so hosted reports title themselves and their metric pills deep-link to the explainer. Co-authored-by: Cursor <cursoragent@cursor.com>28e0081
-
leaderboard: download-zip button in merge-path report + backfill tool
Michael Rabinovich +1: CursorPass the submission-zip URL to generate_html in the merge path so reassembled reports show the "Download submission ZIP" button, and add a one-off tools/backfill_report_download_button.py that injects the button into already-published reports/<id>.html in place (no re-eval). Bump CADGENBENCH_SHA to 0c7690e. Co-authored-by: Cursor <cursoragent@cursor.com>c48c18f -
leaderboard: rename tabs, relabel fixtures as samples, inline gallery row stats
Michael Rabinovich +1: Cursor- Rename the visual "Gallery" tab to "Leaderboard" and the table tab to "Detailed View" (order unchanged); update the smoke test accordingly. - Replace user-facing "fixture" wording with "sample" across the gallery, task browser, submit instructions, and submitter status/error messages. Internal data keys, routes, CSS, and identifiers are unchanged. - Drop the gallery's "more numbers" expander: surface Generation/Editing inline under the aggregate score and move Download ZIP under the name. - Link each gallery submission title to its hosted report (same /reports/<id>.html the Detailed View table uses); drop the date line. Co-authored-by: Cursor <cursoragent@cursor.com>461547b -
leaderboard: serve GT report assets via proxy; link them in hosted report
Michael Rabinovich +1: CursorAdd a /gt/{fixture}/{relpath} proxy route that streams the private GT view PNGs + PDF (token-holding), and pass gt_base_url/input_base_url to generate_html in the merge path so the hosted report links GT/input/overlay (lazy) instead of base64-inlining them. Bump CADGENBENCH_SHA to 25943a0. Co-authored-by: Cursor <cursoragent@cursor.com>5140b0a -
leaderboard: serve renders from the public bucket, not the dataset proxy
Michael Rabinovich +1: CursorCandidate renders (turntables + edit-diff WebP) now live in the public HF Storage Bucket under renders/<id>/, uploaded once by the eval job. The gallery grid and the hosted report reference them by anonymous bucket URL, so the Space is out of the render read path and the binary renders stop bloating the submissions dataset / its commit queue. - leaderboard: HF_RENDER_BUCKET + render URL/path/prefix helpers. - app: render resolvers return public bucket URLs; the two render proxy routes + fetchers are retired. Only the private GT render keeps its token-held proxy. - submit: merge passes render_base_url to generate_html and commits report-only; dispatched job gets CADGENBENCH_RENDER_BUCKET/HF_ENDPOINT. - admin: delete also purges the bucket renders/<id>/ prefix (previously orphaned). - Dockerfile: bump cadgenbench pin to 3d49822 (render_base_url support). Co-authored-by: Cursor <cursoragent@cursor.com>d2161b1
-
Serve rotating WebP turntables + GT generator
Michael Rabinovich +1: CursorPoint the gallery and proxy routes at rotating.webp (submissions and GT), publish the full per-fixture renders folder, and add a reusable one-time GT turntable generator that renders from the cached GT mesh. Co-authored-by: Cursor <cursoragent@cursor.com>c1cb5e4 -
stage shards via bucket API instead of volume mount
Michael RabinovichAvoids Space volume-mount issues on an existing Space: shard jobs sync artifacts to the HF bucket via the bucket API, and the Space syncs them back down to merge. Gated on CADGENBENCH_SHARD_BUCKET.920b1b4 -
fix HF Jobs polling namespace
Michael Rabinovichdd284f3 -
stage shard artifacts through HF bucket
Michael Rabinovich6ffd043 -
submit: stream live eval progress to the submitter's status panel
Michael Rabinovich +1: CursorAdds a small in-process progress registry (progress.py) that the eval worker publishes stage notes to (queued -> waiting for GPU -> evaluating -> collecting results -> done/failed, plus an "X of N chunks done" count for sharded runs). handle_submit keeps its generator alive after queuing and streams those notes into the personal status panel until a terminal stage or a backstop deadline. The shared leaderboard table stays coarse; this is the personal-view feedback only, so it adds no write load to the results file. Also fixes the _with_hub_retries tests for the newer huggingface_hub HfHubHTTPError(response=...) signature. Co-authored-by: Cursor <cursoragent@cursor.com>b0f4559 -
submit: publish merged report + gallery in one commit
Michael Rabinovich +1: CursorReplace the per-file gallery-thumbnail uploads (one upload_file commit per fixture, ~80+ for a full fan-out) with a single create_commit that stages reports/<id>.{html,json} plus every renders/<id>/<fixture>.png at once. Collapses finalize from ~86 commits to one publish commit, cutting post-shard finalize from minutes to well under a minute and removing the 429 "concurrency queue" failures that stranded earlier submissions. Co-authored-by: Cursor <cursoragent@cursor.com>07430f9 -
submit: retry Hub commits on 429/5xx + persistent status panel
Michael Rabinovich +1: CursorWrap every Space-side commit (zip upload, results.jsonl RMW, report + gallery uploads, shard cleanup) in a backoff-with-jitter retry that honors Retry-After, so a rate-limited submissions repo delays rather than fails a submit. Make handle_submit a generator streaming a durable status panel (validating -> uploading -> queued, or the rejection reason) instead of relying on transient toasts. Co-authored-by: Cursor <cursoragent@cursor.com>b224eee
-
submit: allow missing fixture outputs
Michael Rabinovichdc8ff2a
-
submit: shard large submissions across HF Jobs + merge (UC3)
Michael Rabinovich +1: CursorSubmissions over SHARD_THRESHOLD (12) fixtures now fan out into 12-fixture shards dispatched at once, polled to terminal with ERROR-only retry, then merged into one run via write_run_summary (regenerating the report + gallery from the merged whole) before the row flips. At/under the threshold a submission stays a single job, so the original path is unchanged. Eval is CPU-bound, so more shards is the throughput lever. Co-authored-by: Cursor <cursoragent@cursor.com>293bc3b
-
submit: derive eval-gpu image repo from HF_ORG
Michael Rabinovichthe image lives in the org, so the repo id tracks HF_ORG (matching leaderboard.py) and an org rename stays one space-variable change. on the space HF_ORG is set, so this resolves to the same value as before. the job namespace stays a fixed michaelr27 constant since dispatch is billed to that account. verified resolution for the space, override, and no-HF_ORG cases.c78e980 -
submit: keep eval-gpu image repo as a fixed literal
Michael Rabinovichrevert the env-var indirection; the audit sweep stays prose-only and must not alter dispatch resolution. the repo id and job namespace are real deployed identities, kept as documented constants.bd9b6ba -
submit: derive eval-gpu image repo from HF_ORG
Michael Rabinovich0a04e61
-
license: add apache-2.0 header to space modules
Michael Rabinovichb048bc2 -
submit: drop invisible button-progress bar
Verified signature7655daf -
submit: restore Validating... toast on click
Verified signatureef37489 -
step 10 / jobs-migration: submit.py -> dispatch eval to HF Jobs
Verified signaturefd61157
-
submit: drop eval workers 4 -> 1 (OOM under parallel osmesa)
Michael Rabinovich +1: CursorCo-authored-by: Cursor <cursoragent@cursor.com>ed2a486 -
submit: dump full eval/report subprocess output to container log on failure
Michael Rabinovich +1: CursorCo-authored-by: Cursor <cursoragent@cursor.com>0648533 -
Dockerfile + submit: swap Chromium for VTK; bump eval workers 1 -> 4
Michael Rabinovich +1: CursorPairs with cadgenbench 3fd213d (in-process VTK / PyVista renderer). Dockerfile drops the playwright install + Chromium browser layer (~200 MB), drops PLAYWRIGHT_BROWSERS_PATH, and adds the Mesa GL runtime libs VTK needs for off-screen rendering on the Space's cpu-upgrade tier (libgl1 was already there; libegl1, libegl-mesa0, libglx-mesa0 are new). CADGENBENCH_SHA bumps to 3fd213d. EVAL_WORKER_COUNT goes from "1" back to "4": the VTK renderer is cheap enough that one ProcessPoolExecutor worker per fixture no longer oversubscribes the host the way the old Chromium subprocess did. Co-authored-by: Cursor <cursoragent@cursor.com>0626a6b -
submit: serialize cadgenbench evaluate to dodge cpu-upgrade contention
Michael RabinovichEVAL_WORKER_COUNT goes from "8" to "1". Removes the debug /debug/render-bench route added in the previous commit. Why: on the Space's cpu-upgrade tier today, a single headless- Chromium render takes ~7s; five in parallel collapse the host and each individual render blows past its 120s subprocess timeout. Same renderer, same fixtures, same Playwright version run on a laptop in ~5s per render with no slowdown under 5-way parallelism. The slow path is HF's shared host, not our renderer. Sequential eval (one fixture at a time, one render at a time) gives each Chromium the whole box. Five fixtures finish in ~30-60s wall time, far under the 15-minute outer EVAL_TIMEOUT_SECONDS budget. Tradeoff: total eval wall time grows linearly with fixture count. For today's five fixtures that's a few seconds slower than the healthy-host parallel case, and a huge cliff better than a failed eval. For the 100-fixture launch this becomes ~10 minutes per submission, which is the limit of this approach: a r…6e3ab50 -
submit+app: gate Submit on HF OAuth; populate hf_username from profile
Michael RabinovichBundle 1+2 C10. Login-required submit shaped per the bundle spec. submit.py: - `handle_submit` grows a `profile: gr.OAuthProfile | None` arg. Gradio dependency-injects the visitor's profile based on the signature. None -> immediate `gr.Error("Please log in via the HF button before submitting.")` so a UI mishap can't write an anonymous row. - `_build_pending_row` grows an `hf_username=None` kwarg. `handle_submit` passes `profile.username` through so the row carries the canonical HF identity (not a free-text claim in meta.json's submitter_name). app.py: - New `gr.LoginButton()` in the Submit tab above the zip dropzone. - Submit button starts `interactive=False`; new `_enable_submit_when_logged_in(profile)` reads the visitor's OAuth profile via `blocks.load` and flips the button on / off per page load. Server-side gate in `handle_submit` mirrors this so a logged-out user who somehow bypasses the disabled-button UI still gets rejected. requirements.txt: - `gradio…c87b253 -
submit+app: toast-based feedback (validating / queued / refreshed)
Michael RabinovichBundle 1+2 C9. Replaces the static markdown reply on the Submit tab with gr.Info / gr.Error toasts, and adds a refresh toast on the manual Refresh button (the Timer auto-refresh stays silent - a toast every 10s would be noise). submit.py: - handle_submit no longer returns a markdown string. It side- effects: gr.Info("Validating submission...") at the top of the validation stage, gr.Info("Submission <id> queued ...") after the row + zip are on the Hub and the worker has been spawned. Every rejection path raises gr.Error so the user gets a red toast and the handler aborts; the outer try/finally still runs cleanup. Internal _ValidationError + _HubWriteError sentinels preserved. - _validate_form returns a plain message (no `**Error:**` markdown wrap; the caller wraps into gr.Error). - All four rejection paths covered: 1. Form (no zip attached). 2. Validation gate (extract / meta / fixtures / steps). 3. Dedup gate (sha256 match on existing row). 4. Hub write…6facf47
-
submit: default validation + hf_username fields on the pending row
Michael RabinovichBundle 1+2 C4. `_build_pending_row` adds three new keys matching the schema bump from C1, in the schema.md column order: - validation_status = "unvalidated" (every new row starts in the unvalidated tier; maintainers promote via the dataset repo after methodology review). - validation_method = None (set by the maintainer at promotion time to one of code | traces | api | manual). - hf_username = None (placeholder; C10 plumbs through the value from gradio.OAuthProfile.username once the LoginButton lands). Pre-existing fields, submitted_at stamping, the cadgenbench version + data revision tags, the score-shaped null fields, the blob URL, and the sha256 are untouched. Dedup gate still pivots on submission_sha256 exactly as before. tests/test_submit.py covers the C4 acceptance: - new-row defaults: validation_status is "unvalidated", validation_method + hf_username are None; - sha256 path passthrough: the hash argument lands on the row's submission_sha256 key intact (existing …a533dd2 -
submit: stuck-pending sweep on boot (chunk 6)
Michael RabinovichStep 6 (E) chunk 6. A pending row whose worker died (Space restart for a deploy, OOM, crash) has no one to flip it; without this sweep it stays "pending" in the leaderboard forever (we hit this twice during chunk-3..5 churn already). On submit.py module import, spawn a daemon thread that: 1. Downloads results.jsonl (no lock; read-only). 2. Iterates rows; for each pending row whose `submitted_at` is older than 30 min, calls _flip_row_to_failed(sid, "evaluation interrupted by Space restart"). 3. Logs the list of stuck ids so a deployer can see what was swept. Threshold is 30 min: well above the real eval ceiling on cpu-upgrade (~5 min), so a genuinely-still-running submission is safe. Hub-fetch failure at boot is non-fatal (logged warning, sweep skipped; next boot retries). Per-row flip failures are caught + logged + skipped (the rest of the sweep continues). Opt-out via CADGENBENCH_DISABLE_BOOT_SWEEP=1 for test imports that don't want the Hub round-trip. Two tes…0a2a2c6 -
submit: reject byte-identical re-submits via SHA-256 dedup
Michael RabinovichBefore: clicking Submit twice (or re-uploading the same zip after a transient confusion) created two rows pointing at byte-identical content, ran the eval twice for the same input, and bloated the submissions/reports blobs. After: the cheap-sync handler computes sha256 of the uploaded zip right after validation passes, queries results.jsonl for any row with the same submission_sha256, and rejects with a pointer to the existing submission_id if a match is found. Hash also lands on the new row's submission_sha256 field (schema bump in the companion commit on the submissions dataset). Dedup-fetch failures fall back to letting the submit through rather than rejecting on a Hub blip (logged). Race window: two near-simultaneous identical submits whose dedup checks both return None before either appends will land two rows; recoverable by manual cleanup if it ever happens, not worth tightening before auth + login throttling lands at pre-launch. Drive-by: _build_pending_row's unused fixture_n…3237736
-
submit: background-thread eval, flip row pending -> completed / failed
Michael RabinovichStep 6 (E) chunk 4. On a successful submit handle_submit now spawns a daemon thread that runs `cadgenbench evaluate` then `cadgenbench report single` over the unpacked submission, uploads reports/<id>.{html,json} to the submissions dataset, reads run_summary.json, and flips the row pending -> completed under the existing _HUB_LOCK. Any worker-side exception flips the row to failed with a short failure_reason (<=200 chars; full traceback goes to Space logs). Tempdir cleanup always runs in finally. Pipeline per submission, in the worker: 1. cadgenbench evaluate <run_dir> --workers 8 (subprocess via `python -m cadgenbench.cli`; the eval CLI already fans out across fixtures with ProcessPoolExecutor, self-throttles to n_fixtures when smaller). 2. cadgenbench report single <run_dir> -o <tmp>/<id>.html 3. Build reports JSON: run_summary.json + each fixture's result.json bundled together (mirrors README's description of reports/<id>.json as the "machine-readable…855bb8a -
submit: drop the agree checkbox, meta.json is the only consent gate
Michael RabinovichThe Submit tab had both an "I agree" checkbox and required meta.json to carry "agree_to_publish": true. Two gates expressing the same intent, neither covering a case the other didn't, and the UI half doesn't extend to programmatic submitters (gradio_client, future REST API). Drop the UI checkbox. meta.json["agree_to_publish"] stays the single consent gate, uniform across browser + programmatic submission. handle_submit signature shrinks to (zip_file). Submit form is now: zip upload + Submit button.9e84592 -
submit: drop dead form fields, meta.json is the only source
Michael RabinovichBefore: the Submit tab carried four text inputs (submission name, submitter name, agent URL, notes) alongside the zip upload, and handle_submit took them as args but ignored everything except the agree checkbox. They survived chunk 2's move from "form-driven stub" to "meta.json contract" only to keep the gr.click signature stable across one commit; chunk 3 didn't touch them either. After: Submit tab is just zip upload + agree checkbox + submit button. handle_submit signature is (zip_file, agree). All five submission fields come from meta.json inside the zip, full stop. Cuts 30 lines of UI + 4 unused handler params; removes a real user-confusion source (filling the form did nothing).f571140 -
submit: write pending row + upload zip on successful validation
Michael RabinovichStep 6 (E) chunk 3. Cheap-sync validation (chunk 2) now also commits to the Hub: on success the handler uploads submissions/<id>.zip and appends a status=pending row to results.jsonl under a process-wide threading.Lock. Eval still stubbed, the row stays pending forever until chunk 4 lands the background worker. Hub-write ordering (after validation passes): 1. Upload submissions/<id>.zip. Unique path per submission, no lock needed (HfApi handles its own commit serialization). 2. Build the pending row: metadata fields + status="pending" + failure_reason=null + submission_blob_url + null score fields, matching the "Pending" regime in cadgenbench-submissions/schema.md. 3. Acquire _HUB_LOCK, download current results.jsonl (treat EntryNotFoundError as empty), append one JSON line, re-upload. Lock held only for the RMW cycle (~1-2s), not for eval time; concurrent submitters serialise here, not in the worker. If step 1 fails the submitter sees a clean "Hub…1748c76 -
submit: move handle_submit into its own module + add validation pipeline
Michael RabinovichStep 6 (E) chunk 2 of the async submit work. Lifts the UI-only stub out of app.py into submit.py and grows it into the real cheap-sync validation pipeline. Still no Hub writes and no eval kick-off; the handler returns "would queue here" placeholder text. Async write + worker thread land in later chunks. Validation gates, in order: 1. Form: zip attached, agree-checkbox ticked. 2. Zip safety: valid zip, no absolute / parent-traversing entry names, no symlinks (Python 3.12's extractall normalises unsafe paths silently; we reject so submitters see a clear error). 3. meta.json schema: required keys, types, agree_to_publish is literal true, notes within the 500-char cap. meta.json is the source of truth for submitter / submission identity; the other form fields are advisory for now and kept only so the gr.click signature stays stable. 4. Fixture set: top-level dirs in the zip equal the fixture set reported by cadgenbench.common.paths.data_inputs_di…0501689