Commit History
-
submit: drop invisible button-progress bar
Verified signature7655daf -
submit: restore Validating... toast on click
Verified signatureef37489 -
step 10 / jobs-migration: submit.py -> dispatch eval to HF Jobs
Verified signaturefd61157
-
submit: drop eval workers 4 -> 1 (OOM under parallel osmesa)
Michael Rabinovich +1: CursorCo-authored-by: Cursor <cursoragent@cursor.com>ed2a486 -
submit: dump full eval/report subprocess output to container log on failure
Michael Rabinovich +1: CursorCo-authored-by: Cursor <cursoragent@cursor.com>0648533 -
Dockerfile + submit: swap Chromium for VTK; bump eval workers 1 -> 4
Michael Rabinovich +1: CursorPairs with cadgenbench 3fd213d (in-process VTK / PyVista renderer). Dockerfile drops the playwright install + Chromium browser layer (~200 MB), drops PLAYWRIGHT_BROWSERS_PATH, and adds the Mesa GL runtime libs VTK needs for off-screen rendering on the Space's cpu-upgrade tier (libgl1 was already there; libegl1, libegl-mesa0, libglx-mesa0 are new). CADGENBENCH_SHA bumps to 3fd213d. EVAL_WORKER_COUNT goes from "1" back to "4": the VTK renderer is cheap enough that one ProcessPoolExecutor worker per fixture no longer oversubscribes the host the way the old Chromium subprocess did. Co-authored-by: Cursor <cursoragent@cursor.com>0626a6b -
submit: serialize cadgenbench evaluate to dodge cpu-upgrade contention
Michael RabinovichEVAL_WORKER_COUNT goes from "8" to "1". Removes the debug /debug/render-bench route added in the previous commit. Why: on the Space's cpu-upgrade tier today, a single headless- Chromium render takes ~7s; five in parallel collapse the host and each individual render blows past its 120s subprocess timeout. Same renderer, same fixtures, same Playwright version run on a laptop in ~5s per render with no slowdown under 5-way parallelism. The slow path is HF's shared host, not our renderer. Sequential eval (one fixture at a time, one render at a time) gives each Chromium the whole box. Five fixtures finish in ~30-60s wall time, far under the 15-minute outer EVAL_TIMEOUT_SECONDS budget. Tradeoff: total eval wall time grows linearly with fixture count. For today's five fixtures that's a few seconds slower than the healthy-host parallel case, and a huge cliff better than a failed eval. For the 100-fixture launch this becomes ~10 minutes per submission, which is the limit of this approach: a r…6e3ab50 -
submit+app: gate Submit on HF OAuth; populate hf_username from profile
Michael RabinovichBundle 1+2 C10. Login-required submit shaped per the bundle spec. submit.py: - `handle_submit` grows a `profile: gr.OAuthProfile | None` arg. Gradio dependency-injects the visitor's profile based on the signature. None -> immediate `gr.Error("Please log in via the HF button before submitting.")` so a UI mishap can't write an anonymous row. - `_build_pending_row` grows an `hf_username=None` kwarg. `handle_submit` passes `profile.username` through so the row carries the canonical HF identity (not a free-text claim in meta.json's submitter_name). app.py: - New `gr.LoginButton()` in the Submit tab above the zip dropzone. - Submit button starts `interactive=False`; new `_enable_submit_when_logged_in(profile)` reads the visitor's OAuth profile via `blocks.load` and flips the button on / off per page load. Server-side gate in `handle_submit` mirrors this so a logged-out user who somehow bypasses the disabled-button UI still gets rejected. requirements.txt: - `gradio…c87b253 -
submit+app: toast-based feedback (validating / queued / refreshed)
Michael RabinovichBundle 1+2 C9. Replaces the static markdown reply on the Submit tab with gr.Info / gr.Error toasts, and adds a refresh toast on the manual Refresh button (the Timer auto-refresh stays silent - a toast every 10s would be noise). submit.py: - handle_submit no longer returns a markdown string. It side- effects: gr.Info("Validating submission...") at the top of the validation stage, gr.Info("Submission <id> queued ...") after the row + zip are on the Hub and the worker has been spawned. Every rejection path raises gr.Error so the user gets a red toast and the handler aborts; the outer try/finally still runs cleanup. Internal _ValidationError + _HubWriteError sentinels preserved. - _validate_form returns a plain message (no `**Error:**` markdown wrap; the caller wraps into gr.Error). - All four rejection paths covered: 1. Form (no zip attached). 2. Validation gate (extract / meta / fixtures / steps). 3. Dedup gate (sha256 match on existing row). 4. Hub write…6facf47
-
submit: default validation + hf_username fields on the pending row
Michael RabinovichBundle 1+2 C4. `_build_pending_row` adds three new keys matching the schema bump from C1, in the schema.md column order: - validation_status = "unvalidated" (every new row starts in the unvalidated tier; maintainers promote via the dataset repo after methodology review). - validation_method = None (set by the maintainer at promotion time to one of code | traces | api | manual). - hf_username = None (placeholder; C10 plumbs through the value from gradio.OAuthProfile.username once the LoginButton lands). Pre-existing fields, submitted_at stamping, the cadgenbench version + data revision tags, the score-shaped null fields, the blob URL, and the sha256 are untouched. Dedup gate still pivots on submission_sha256 exactly as before. tests/test_submit.py covers the C4 acceptance: - new-row defaults: validation_status is "unvalidated", validation_method + hf_username are None; - sha256 path passthrough: the hash argument lands on the row's submission_sha256 key intact (existing …a533dd2 -
submit: stuck-pending sweep on boot (chunk 6)
Michael RabinovichStep 6 (E) chunk 6. A pending row whose worker died (Space restart for a deploy, OOM, crash) has no one to flip it; without this sweep it stays "pending" in the leaderboard forever (we hit this twice during chunk-3..5 churn already). On submit.py module import, spawn a daemon thread that: 1. Downloads results.jsonl (no lock; read-only). 2. Iterates rows; for each pending row whose `submitted_at` is older than 30 min, calls _flip_row_to_failed(sid, "evaluation interrupted by Space restart"). 3. Logs the list of stuck ids so a deployer can see what was swept. Threshold is 30 min: well above the real eval ceiling on cpu-upgrade (~5 min), so a genuinely-still-running submission is safe. Hub-fetch failure at boot is non-fatal (logged warning, sweep skipped; next boot retries). Per-row flip failures are caught + logged + skipped (the rest of the sweep continues). Opt-out via CADGENBENCH_DISABLE_BOOT_SWEEP=1 for test imports that don't want the Hub round-trip. Two tes…0a2a2c6 -
submit: reject byte-identical re-submits via SHA-256 dedup
Michael RabinovichBefore: clicking Submit twice (or re-uploading the same zip after a transient confusion) created two rows pointing at byte-identical content, ran the eval twice for the same input, and bloated the submissions/reports blobs. After: the cheap-sync handler computes sha256 of the uploaded zip right after validation passes, queries results.jsonl for any row with the same submission_sha256, and rejects with a pointer to the existing submission_id if a match is found. Hash also lands on the new row's submission_sha256 field (schema bump in the companion commit on the submissions dataset). Dedup-fetch failures fall back to letting the submit through rather than rejecting on a Hub blip (logged). Race window: two near-simultaneous identical submits whose dedup checks both return None before either appends will land two rows; recoverable by manual cleanup if it ever happens, not worth tightening before auth + login throttling lands at pre-launch. Drive-by: _build_pending_row's unused fixture_n…3237736
-
submit: background-thread eval, flip row pending -> completed / failed
Michael RabinovichStep 6 (E) chunk 4. On a successful submit handle_submit now spawns a daemon thread that runs `cadgenbench evaluate` then `cadgenbench report single` over the unpacked submission, uploads reports/<id>.{html,json} to the submissions dataset, reads run_summary.json, and flips the row pending -> completed under the existing _HUB_LOCK. Any worker-side exception flips the row to failed with a short failure_reason (<=200 chars; full traceback goes to Space logs). Tempdir cleanup always runs in finally. Pipeline per submission, in the worker: 1. cadgenbench evaluate <run_dir> --workers 8 (subprocess via `python -m cadgenbench.cli`; the eval CLI already fans out across fixtures with ProcessPoolExecutor, self-throttles to n_fixtures when smaller). 2. cadgenbench report single <run_dir> -o <tmp>/<id>.html 3. Build reports JSON: run_summary.json + each fixture's result.json bundled together (mirrors README's description of reports/<id>.json as the "machine-readable…855bb8a -
submit: drop the agree checkbox, meta.json is the only consent gate
Michael RabinovichThe Submit tab had both an "I agree" checkbox and required meta.json to carry "agree_to_publish": true. Two gates expressing the same intent, neither covering a case the other didn't, and the UI half doesn't extend to programmatic submitters (gradio_client, future REST API). Drop the UI checkbox. meta.json["agree_to_publish"] stays the single consent gate, uniform across browser + programmatic submission. handle_submit signature shrinks to (zip_file). Submit form is now: zip upload + Submit button.9e84592 -
submit: drop dead form fields, meta.json is the only source
Michael RabinovichBefore: the Submit tab carried four text inputs (submission name, submitter name, agent URL, notes) alongside the zip upload, and handle_submit took them as args but ignored everything except the agree checkbox. They survived chunk 2's move from "form-driven stub" to "meta.json contract" only to keep the gr.click signature stable across one commit; chunk 3 didn't touch them either. After: Submit tab is just zip upload + agree checkbox + submit button. handle_submit signature is (zip_file, agree). All five submission fields come from meta.json inside the zip, full stop. Cuts 30 lines of UI + 4 unused handler params; removes a real user-confusion source (filling the form did nothing).f571140 -
submit: write pending row + upload zip on successful validation
Michael RabinovichStep 6 (E) chunk 3. Cheap-sync validation (chunk 2) now also commits to the Hub: on success the handler uploads submissions/<id>.zip and appends a status=pending row to results.jsonl under a process-wide threading.Lock. Eval still stubbed, the row stays pending forever until chunk 4 lands the background worker. Hub-write ordering (after validation passes): 1. Upload submissions/<id>.zip. Unique path per submission, no lock needed (HfApi handles its own commit serialization). 2. Build the pending row: metadata fields + status="pending" + failure_reason=null + submission_blob_url + null score fields, matching the "Pending" regime in cadgenbench-submissions/schema.md. 3. Acquire _HUB_LOCK, download current results.jsonl (treat EntryNotFoundError as empty), append one JSON line, re-upload. Lock held only for the RMW cycle (~1-2s), not for eval time; concurrent submitters serialise here, not in the worker. If step 1 fails the submitter sees a clean "Hub…1748c76 -
submit: move handle_submit into its own module + add validation pipeline
Michael RabinovichStep 6 (E) chunk 2 of the async submit work. Lifts the UI-only stub out of app.py into submit.py and grows it into the real cheap-sync validation pipeline. Still no Hub writes and no eval kick-off; the handler returns "would queue here" placeholder text. Async write + worker thread land in later chunks. Validation gates, in order: 1. Form: zip attached, agree-checkbox ticked. 2. Zip safety: valid zip, no absolute / parent-traversing entry names, no symlinks (Python 3.12's extractall normalises unsafe paths silently; we reject so submitters see a clear error). 3. meta.json schema: required keys, types, agree_to_publish is literal true, notes within the 500-char cap. meta.json is the source of truth for submitter / submission identity; the other form fields are advisory for now and kept only so the gr.click signature stays stable. 4. Fixture set: top-level dirs in the zip equal the fixture set reported by cadgenbench.common.paths.data_inputs_di…0501689