Commit History
-
step 10 / jobs-migration: submit.py -> dispatch eval to HF Jobs
Verified signaturefd61157
-
submit: drop eval workers 4 -> 1 (OOM under parallel osmesa)
Michael Rabinovich +1: CursorCo-authored-by: Cursor <cursoragent@cursor.com>ed2a486 -
Dockerfile: swap vtk for vtk-osmesa so off-screen rendering works headless
Michael Rabinovich +1: CursorThe stock pyvista wheel pulls the stock vtk wheel from PyPI, which is built with vtkXOpenGLRenderWindow. That render window class needs an X server even in off_screen mode, so it segfaults the eval worker on the Space's truly-headless container ("DISPLAY=" with no X11 socket). The parent ProcessPoolExecutor sees the segfault as BrokenProcessPool six minutes into the eval, which is exactly the failure mode the first VTK submission hit. vtk-osmesa is the same VTK build but compiled against Mesa OSMesa (pure CPU software rasteriser, statically linked, no display server required). It shares the `import vtk` namespace so pyvista picks it up transparently. This is the canonical PyVista headless-Docker fix (per the PyVista docs + the maintainers' own Docker images). The wheel lives on Kitware's index, not PyPI proper. Co-authored-by: Cursor <cursoragent@cursor.com>545c818 -
submit: dump full eval/report subprocess output to container log on failure
Michael Rabinovich +1: CursorCo-authored-by: Cursor <cursoragent@cursor.com>0648533 -
Dockerfile: bump CADGENBENCH_SHA to b22a53c (ProcessPool spawn fix)
Michael Rabinovich +1: CursorCo-authored-by: Cursor <cursoragent@cursor.com>fa32cdd -
Dockerfile + submit: swap Chromium for VTK; bump eval workers 1 -> 4
Michael Rabinovich +1: CursorPairs with cadgenbench 3fd213d (in-process VTK / PyVista renderer). Dockerfile drops the playwright install + Chromium browser layer (~200 MB), drops PLAYWRIGHT_BROWSERS_PATH, and adds the Mesa GL runtime libs VTK needs for off-screen rendering on the Space's cpu-upgrade tier (libgl1 was already there; libegl1, libegl-mesa0, libglx-mesa0 are new). CADGENBENCH_SHA bumps to 3fd213d. EVAL_WORKER_COUNT goes from "1" back to "4": the VTK renderer is cheap enough that one ProcessPoolExecutor worker per fixture no longer oversubscribes the host the way the old Chromium subprocess did. Co-authored-by: Cursor <cursoragent@cursor.com>0626a6b -
leaderboard: format submitted_at as `YYYY-MM-DD HH:MM UTC`; lock tables read-only
Michael RabinovichTable cells were editable (gradio_leaderboard inherits gr.Dataframe's interactive default); add interactive=False to both Validated and Unvalidated widgets so cell contents aren't accidentally typed over. submitted_at column rendered as raw ISO-8601 ("2026-05-28T07:13:16Z"); apply the same _fmt_timestamp helper the detail panel already uses so the table reads as "2026-05-28 07:13 UTC". Helper moved into leaderboard.py (formatter module) and re-imported by app.py.c4e21b3 -
submit: serialize cadgenbench evaluate to dodge cpu-upgrade contention
Michael RabinovichEVAL_WORKER_COUNT goes from "8" to "1". Removes the debug /debug/render-bench route added in the previous commit. Why: on the Space's cpu-upgrade tier today, a single headless- Chromium render takes ~7s; five in parallel collapse the host and each individual render blows past its 120s subprocess timeout. Same renderer, same fixtures, same Playwright version run on a laptop in ~5s per render with no slowdown under 5-way parallelism. The slow path is HF's shared host, not our renderer. Sequential eval (one fixture at a time, one render at a time) gives each Chromium the whole box. Five fixtures finish in ~30-60s wall time, far under the 15-minute outer EVAL_TIMEOUT_SECONDS budget. Tradeoff: total eval wall time grows linearly with fixture count. For today's five fixtures that's a few seconds slower than the healthy-host parallel case, and a huge cliff better than a failed eval. For the 100-fixture launch this becomes ~10 minutes per submission, which is the limit of this approach: a r…6e3ab50 -
debug: add /debug/render-bench route for one-shot Chromium timing
Michael RabinovichRead-only FastAPI endpoint mounted on the Space's parent app. Sequentially times `render_step` against each fixture's input.step in cadgenbench-data and returns per-fixture seconds + view count. Used to compare per-render cost on the Space's container vs a local reference when eval has been timing out at the 120s render_step ceiling but nothing on the render path changed in our code. Curl-fetchable under Bearer auth (the Space is private): curl -H "Authorization: Bearer $HF_TOKEN" \ https://<space>.hf.space/debug/render-bench Not user-visible (no UI), no side effects, no row writes. Safe to remove once the diagnosis lands.37e45d8 -
submit+app: gate Submit on HF OAuth; populate hf_username from profile
Michael RabinovichBundle 1+2 C10. Login-required submit shaped per the bundle spec. submit.py: - `handle_submit` grows a `profile: gr.OAuthProfile | None` arg. Gradio dependency-injects the visitor's profile based on the signature. None -> immediate `gr.Error("Please log in via the HF button before submitting.")` so a UI mishap can't write an anonymous row. - `_build_pending_row` grows an `hf_username=None` kwarg. `handle_submit` passes `profile.username` through so the row carries the canonical HF identity (not a free-text claim in meta.json's submitter_name). app.py: - New `gr.LoginButton()` in the Submit tab above the zip dropzone. - Submit button starts `interactive=False`; new `_enable_submit_when_logged_in(profile)` reads the visitor's OAuth profile via `blocks.load` and flips the button on / off per page load. Server-side gate in `handle_submit` mirrors this so a logged-out user who somehow bypasses the disabled-button UI still gets rejected. requirements.txt: - `gradio…c87b253 -
submit+app: toast-based feedback (validating / queued / refreshed)
Michael RabinovichBundle 1+2 C9. Replaces the static markdown reply on the Submit tab with gr.Info / gr.Error toasts, and adds a refresh toast on the manual Refresh button (the Timer auto-refresh stays silent - a toast every 10s would be noise). submit.py: - handle_submit no longer returns a markdown string. It side- effects: gr.Info("Validating submission...") at the top of the validation stage, gr.Info("Submission <id> queued ...") after the row + zip are on the Hub and the worker has been spawned. Every rejection path raises gr.Error so the user gets a red toast and the handler aborts; the outer try/finally still runs cleanup. Internal _ValidationError + _HubWriteError sentinels preserved. - _validate_form returns a plain message (no `**Error:**` markdown wrap; the caller wraps into gr.Error). - All four rejection paths covered: 1. Form (no zip attached). 2. Validation gate (extract / meta / fixtures / steps). 3. Dedup gate (sha256 match on existing row). 4. Hub write…6facf47
-
leaderboard+app: combined CSV download with validation_status discriminator
Michael RabinovichBundle 1+2 C8. One file, both tiers, downloaded via `gr.DownloadButton` next to the existing Refresh button. leaderboard.py: - New `CSV_COLUMNS` constant (wider than the on-screen table: raw aggregate_score / validity_rate, plus provenance fields like submission_sha256 + cadgenbench_data_revision + notes + failure_reason). `validation_status` is the discriminator that lets a reader grep the file for one tier or the other. - New `build_combined_csv()` reads the same rows the leaderboard reader does, applies the same status / validation_status defaults, projects to CSV_COLUMNS, sort: validated rows on top (highest score first) then unvalidated, then writes to a unique /tmp file and returns the path. Fresh on every click so the export reflects the latest data, not a snapshot captured at boot. app.py: - Imports `build_combined_csv`. The Refresh button and the new `gr.DownloadButton("Download CSV", size="sm")` live in a single `gr.Row` so they sit side-by-side unde…f585077 -
app: Citation + Validation Guidelines accordions, validation link in About
Michael RabinovichBundle 1+2 C7. Two collapsed accordions above the leaderboard tables plus a validation-policy bullet on the About tab. - Validation Guidelines accordion (collapsed by default, sits above the tables). Carries the two-tier story in two lines (unvalidated -> validated, four evidence types) plus a link to the canonical policy doc in the main repo at `cadgenbench/docs/benchmark/validation.md`. Bundle 4 creates the file; the link will 404 until that lands, then resolves. - Citation accordion (collapsed by default, below Validation Guidelines). Carries the verbatim BibTeX entry locked in space-setup/bundles/1-2-space-ux.md (cite key cadgenbench2026, Space URL as howpublished). Rendered in a `gr.Code` with language=None (no BibTeX highlighter ships with gradio) and show_line_numbers off so the entry stays copy-paste tidy. - About tab picks up a new bullet linking the same validation doc. VALIDATION_DOC_URL is centralised at module level so the Validation Guidelines markdo…97b9a4a -
app: render reports inline via iframe srcdoc (private-Space safe)
Michael RabinovichPolish follow-up on C6. Reports now render inside the Space, inside the same iframe HF uses to host the leaderboard, with no second HTTP request leaving the browser. Why: the Space is private, and HF's edge gates same-origin pathname navigations on a short-lived __sign JWT that browsers don't carry forward when a user clicks a relative link. The previous design (submission_name -> /reports/<id>.html) worked server-side under Bearer auth (verified) but 404'd in the browser. Confirmed via external review and reproduced with a requests.Session cookie test. This commit: - Drops the submission_name -> /reports/<id>.html link wrapping from the rendered table; the cell stays plain text. The underlying `report_url` column (relative URL to the proxy) is still computed for the modern-pipeline gate but no longer surfaced as a clickable cell. - Adds a `gr.HTML` viewer below the detail panel. Row-click triggers `_format_detail_and_report`, which returns (detail_markdown, report_ifram…0e3b21f -
app+leaderboard: link submission_name to a Space-side report proxy
Michael RabinovichPolish follow-up #2 on C6. Three user-visible changes plus the plumbing required to make them work. 1. Drop the `report` and `zip` columns from the rendered table. `submission_name` becomes the report's deep link (typical HF leaderboard pattern: the system / model name doubles as the link target; cf. Open LLM Leaderboard, DABstep, BigCodeBench). Modern completed rows get `[<name>](/reports/<sid>.html)`; pending, failed, and legacy seed rows fall back to plain text so there's no broken click. The hidden `submission_blob_url` and the computed `report_url` ride along in the DataFrame and surface in the row-click detail panel only. 2. Rename the agent-URL column from `agent` to `model details (optional)`. Missing cells render as italic `_None_` so the optionality is explicit instead of a blank. "Agent" was confusing; the schema lets `agent_url` be code OR paper OR a blog post and "agent" reads narrow. 3. Add a Space-side proxy for the per-submission H…77edebf -
app+leaderboard: detail panel polish (rename link columns, fix broken report links)
Michael RabinovichPolish follow-up on C6, three fixes after the first detail-panel commit landed: 1. Rename display columns from snake_case schema names to clean short headers. `agent_url` -> `agent`, `submission_blob_url` -> `zip`, `report_url` -> `report`. The rename happens at projection time via `df.rename`, so the schema in results.jsonl is unchanged. Headers now read as plain English ("agent", "zip", "report") rather than a python dict. 2. The `agent` column shows the URL itself (scheme stripped, truncated past 40 chars) as link text instead of a fixed "code" label. The schema says agent_url can be code OR paper; the shortened URL is the only honest hint about what's behind the click. Missing values render as italic `_None_` rather than a blank cell, so the optionality is explicit. 3. The `report` link is now gated on the row's `submission_sha256` being non-null in addition to status == "completed". Legacy seed rows pre-date the modern submit pipeline (which …1a8f331 -
app: row-click detail panel below the leaderboard tables
Michael RabinovichBundle 1+2 C6. Single shared `gr.Markdown` panel below both Leaderboard widgets; either table's `.select(...)` event populates it with the clicked row's notes, agent_url, submitted_at, submission zip + report links (always), and failure_reason (failed rows only). Spec'd shape: one panel, two select sources. leaderboard.py: - LEADERBOARD_COLS + VALIDATED_LEADERBOARD_COLS each grow by three hidden detail-only fields (submission_id, notes, failure_reason). They ride in the underlying DataFrame so the select handler can read them out directly without a parallel state cache. - New LEADERBOARD_HIDE_COLUMNS constant lists the columns hidden from view; both Leaderboard widgets in app.py consume it. - `_project_and_format` now fills missing columns with None (legacy rows can lack optional fields like notes; the panel needs the column-set to be stable regardless of source-row shape). app.py: - pandas imported (for the helper's pd.isna check). - New `_format_detail(df, evt)` helper …3112173 -
leaderboard: markdown link columns for agent_url, submission, report
Michael RabinovichBundle 1+2 C5. Three new columns on both tables (validated + unvalidated) carry clickable links rendered via gradio_leaderboard's `datatype="markdown"`: - agent_url: link to the agent's source / paper if the row's meta.json provided one (label "code"; empty cell when null). - submission_blob_url: link to the uploaded zip on the submissions dataset (label "zip"; populated on every non-failed row, the same URL the submit handler computed at upload time). - report_url: built at read time from `submission_id` plus HF_SUBMISSIONS_REPO; points at `reports/<id>.html` under /resolve/main/ so the browser renders the report directly. Only emitted for `status == "completed"` rows. leaderboard.py: - LEADERBOARD_COLS + VALIDATED_LEADERBOARD_COLS grow by three. - New LEADERBOARD_DATATYPES + VALIDATED_LEADERBOARD_DATATYPES constants mark the link columns as "markdown", everything else as "str". Tied to the column lists by a single helper so the two stay in lockstep. - New _agent_u…53de73a -
submit: default validation + hf_username fields on the pending row
Michael RabinovichBundle 1+2 C4. `_build_pending_row` adds three new keys matching the schema bump from C1, in the schema.md column order: - validation_status = "unvalidated" (every new row starts in the unvalidated tier; maintainers promote via the dataset repo after methodology review). - validation_method = None (set by the maintainer at promotion time to one of code | traces | api | manual). - hf_username = None (placeholder; C10 plumbs through the value from gradio.OAuthProfile.username once the LoginButton lands). Pre-existing fields, submitted_at stamping, the cadgenbench version + data revision tags, the score-shaped null fields, the blob URL, and the sha256 are untouched. Dedup gate still pivots on submission_sha256 exactly as before. tests/test_submit.py covers the C4 acceptance: - new-row defaults: validation_status is "unvalidated", validation_method + hf_username are None; - sha256 path passthrough: the hash argument lands on the row's submission_sha256 key intact (existing …a533dd2 -
docs: add leaderboard discovery tags to README front-matter
Michael Rabinovichtags: surfaces this Space in the Hub's leaderboard discovery filter (huggingface.co/spaces?other=leaderboard) and faceted searches by modality, submission type, and eval type. The set blends the empirical vocabulary across the top 50 leaderboard-tagged Spaces with three truthful novel descriptors: modality:3d (closest existing values are modality:text / tools, neither accurate for STEP geometry), eval:cad (paired with eval:generation so the standard "generation" chip also surfaces us), and judge:auto (eval is metric-based, no LLM/human judge).36546fc -
app: split Leaderboard into Validated + Unvalidated tables
Michael RabinovichBundle 1+2 C3. Swaps the single Leaderboard widget for two stacked ones (Validated on top, Unvalidated below) and wires the existing auto-refresh Timer + Refresh button to push fresh dataframes into both. Initial render reads via the C2 `load_leaderboard_split()`; existing rows all land in Unvalidated until a maintainer flips `validation_status` to `"validated"` on a row in the submissions dataset. - app.py: replaces `df_view` with `validated_view` + `unvalidated_view`. Single timer tick now fans out to both widgets (load split runs once per tick, both halves get pushed). Refresh button same fan-out shape. Inline comment points at the validation policy decision doc. - leaderboard.py: drops the now-unused single-DataFrame `load_leaderboard()` per the C3 cleanup half. Stale docstring references that pointed at the dead function got smoothed out (the split reader's doc is self-contained now). - tests/test_smoke.py: picks up the C3 acceptance with a new `test_two_leaderboa…046548a -
license: add Apache-2.0
Michael Rabinovich8676e14 -
leaderboard: add load_leaderboard_split() for two-tier viewer
Michael RabinovichBundle 1+2 C2. Additive new function for the C3 two-stacked-table swap; existing single-DataFrame load_leaderboard() stays put until C3 flips the caller in app.py + drops it. Keeps the live Space green between commits. - VALIDATED_LEADERBOARD_COLS: same shape as LEADERBOARD_COLS plus `validation_method` (always null on unvalidated rows, so omitted from that tier rather than rendered). - load_leaderboard_split() -> (validated_df, unvalidated_df): reads results.jsonl the same way as load_leaderboard(), splits rows on `validation_status` (defensive: anything not literally "validated" lands in unvalidated, which covers legacy + null + any future-unknown value), runs the same status-aware cell formatting per tier via the new _project_and_format helper. - _project_and_format: shared sort + projection + pending/failed cell tagging so the two tiers produce identically-shaped cells. Legacy default: rows without `validation_status` get "unvalidated" via row-level fill. No resul…0dd7215 -
docs: link companion datasets in README front-matter
Michael Rabinovichdatasets: lists the two companion dataset repos so the Hub renders them as cards on the Space page (and so the link is explicit rather than relying on auto-parse from hardcoded repo IDs in app.py / leaderboard.py). - HuggingAI4Engineering/cadgenbench-submissions: the results + per-submission report dataset the leaderboard reads from. - HuggingAI4Engineering/cadgenbench-data: the public fixture inputs. cadgenbench-data-gt is omitted on purpose (permanently private).1402833 -
docs: add hf_oauth + apache-2.0 license to README front-matter
Michael Rabinovichhf_oauth: true is what wires the LoginButton + OAuth callback the Submit tab relies on; without it the auth-required submit flow is silently broken (login click no-ops, no token reaches the handler). license: apache-2.0 matches the HF Leaderboards template (gradio-templates/leaderboard) and the rest of the cadgenbench code surface, and renders a license badge on the Space page.0bb6044 -
tests: pytest + playwright headless smoke harness
Michael RabinovichFirst Space-side test harness; sets up the floor for the Phase D bundle's per-commit unit + UI tests (split leaderboard, pending-row defaults, OAuth-required submit) without slipping a regression past a build-only check. requirements-dev.txt pulls -r requirements.txt then adds pytest + playwright. One `pip install -r requirements-dev.txt && playwright install chromium` brings a fresh venv up to speed. tests/conftest.py exposes a session-scoped `app_url` fixture: boots app.py in a subprocess on a free port, polls until the HTTP server answers, yields the URL, tears the process down on teardown. Logs land in a tmp file so a startup failure surfaces the real Gradio traceback rather than a bare RuntimeError. Boot-time stuck-pending sweep in submit.py is disabled via CADGENBENCH_DISABLE_BOOT_SWEEP=1 so the fixture is fast and offline-tolerant. tests/test_smoke.py launches headless Chromium via playwright's sync API, navigates to the fixture's URL, and asserts the Leaderboard / Submit / A…21b071a -
Dockerfile: bump CADGENBENCH_SHA to 056cfbb (invalid-candidate fast path)
Michael Rabinovichff674cc -
submit: stuck-pending sweep on boot (chunk 6)
Michael RabinovichStep 6 (E) chunk 6. A pending row whose worker died (Space restart for a deploy, OOM, crash) has no one to flip it; without this sweep it stays "pending" in the leaderboard forever (we hit this twice during chunk-3..5 churn already). On submit.py module import, spawn a daemon thread that: 1. Downloads results.jsonl (no lock; read-only). 2. Iterates rows; for each pending row whose `submitted_at` is older than 30 min, calls _flip_row_to_failed(sid, "evaluation interrupted by Space restart"). 3. Logs the list of stuck ids so a deployer can see what was swept. Threshold is 30 min: well above the real eval ceiling on cpu-upgrade (~5 min), so a genuinely-still-running submission is safe. Hub-fetch failure at boot is non-fatal (logged warning, sweep skipped; next boot retries). Per-row flip failures are caught + logged + skipped (the rest of the sweep continues). Opt-out via CADGENBENCH_DISABLE_BOOT_SWEEP=1 for test imports that don't want the Hub round-trip. Two tes…0a2a2c6 -
submit: reject byte-identical re-submits via SHA-256 dedup
Michael RabinovichBefore: clicking Submit twice (or re-uploading the same zip after a transient confusion) created two rows pointing at byte-identical content, ran the eval twice for the same input, and bloated the submissions/reports blobs. After: the cheap-sync handler computes sha256 of the uploaded zip right after validation passes, queries results.jsonl for any row with the same submission_sha256, and rejects with a pointer to the existing submission_id if a match is found. Hash also lands on the new row's submission_sha256 field (schema bump in the companion commit on the submissions dataset). Dedup-fetch failures fall back to letting the submit through rather than rejecting on a Hub blip (logged). Race window: two near-simultaneous identical submits whose dedup checks both return None before either appends will land two rows; recoverable by manual cleanup if it ever happens, not worth tightening before auth + login throttling lands at pre-launch. Drive-by: _build_pending_row's unused fixture_n…3237736
-
app: move theme= from Blocks.launch() to Blocks() ctor (Gradio 5 API)
Michael Rabinovich76f0611 -
app+requirements: pin Gradio 5 + gradio_leaderboard, auto-refresh now works
Michael RabinovichStep 6 (E) chunk 5, take 2. The Gradio 6.14 gr.Dataframe component silently swallows row updates after the first render: server-side load_leaderboard fires every N seconds (logged: "Loaded 11 rows from Hub" on each tick) but the rendered DOM never picks the new rows up. Confirmed with a local row-growing fixture (load_leaderboard monkey-patched to return N+1 rows on each call); the server fired 10 times, the DOM stuck on the first call's row count. Tried in order, all failed against the local fixture: - gr.Dataframe(every=10, value=callable) - gr.Timer(10).tick(outputs=gr.Dataframe) - manual Refresh button click - @gr.render(inputs=[gr.State, gr.State], key=f"...-{tick}") - @gr.render(inputs=[hidden gr.Textbox], key=f"...-{tick}") - streaming generator on app.load (first yield delivered; every subsequent yield ignored client-side) What actually works: pin to Gradio 5 (latest 5.50.0) and use the gradio_leaderboard.Leaderboard custom component, which has its own update …4e86f82 -
app: fix leaderboard auto-refresh via @gr .render(key=...)
Michael RabinovichIn Gradio 6.14 a `gr.Dataframe` identity-checks its component id and short-circuits the diff when the new value's shape + column structure look unchanged, even when the row data differs. That swallowed updates from `every=` on the Dataframe AND from `gr.Timer().tick(outputs=df_view)` AND from a manual Refresh button click: the server-side fetch ran every tick (`Loaded 10 rows from Hub` in logs) but the browser never saw the new rows. Only a full page reload picked them up. Documented fix (post Gradio's March 2026 Dataframe overhaul): wrap the component in `@gr.render` and give the inner Dataframe a `key=` that changes on every tick. Gradio then tears down and rebuilds the component subtree in place, picking up the fresh value instead of trying to diff in place. Shape: - tick_state: gr.State holding a ms-timestamp that bumps on every refresh; included in the Dataframe `key=`. - table_state: gr.State holding the pandas DataFrame. - `_refresh_table()` runs once per tick, retur…f2f35be -
app: drive auto-refresh via gr.Timer().tick instead of every= on Dataframe
Michael Rabinovichgr.Dataframe(every=N) was firing load_leaderboard on the server every 10s (verified via runtime logs), but the component's client-side re-render after the initial value silently dropped rows added between ticks. Same shape as gradio-app/gradio#8160 ("when the data already exists ... it cannot be updated and the page needs to be refreshed"). Switch to gr.Timer(10).tick(fn=load_leaderboard, outputs=df_view). Identical code path to the manual Refresh button: each tick is an event fired through the standard component-update channel, which re-renders reliably. The Refresh button stays as the "I want it now" override.4ee70ef -
app: auto-refresh leaderboard + bootstrap logging
Michael RabinovichStep 6 (E) chunk 5 plus a diagnostic. Auto-refresh: - gr.Dataframe now takes `value=load_leaderboard` (callable, not its call result) plus `every=10`. Gradio re-invokes the function every 10 seconds and re-renders the dataframe, so pending rows visibly flip to completed / failed and freshly-submitted rows appear within 10s without any user action. Manual Refresh button stays as an "I want it now" override. Logging bootstrap: - logging.basicConfig(level=INFO, ...) at app startup so the `logger.info` / `logger.warning` / `logger.exception` calls already sprinkled through leaderboard.py and submit.py reach the Space's runtime logs. Without this they go nowhere (Python's default no-handler behaviour), which is why every refresh / worker pathology so far has been silent. Combined push: the auto-refresh might not surface new rows if the underlying load_leaderboard returns stale data on the Space for any reason (CDN, auth, frontend re-render bug)…b2f3ce6 -
Dockerfile: bump CADGENBENCH_SHA to 9bd02fb (lazy baseline import)
Michael RabinovichPicks up the cadgenbench fix that makes `cadgenbench --help` and `cadgenbench evaluate` work in the eval-only install (no `[baseline]` extra). Before this bump, the chunk-4 worker's `python -m cadgenbench.cli evaluate <run_dir>` subprocess died during argparse setup with `ModuleNotFoundError: No module named 'dotenv'`, flipping every submission to status=failed before any work happened. Bumping CADGENBENCH_SHA invalidates the cadgenbench layer in the Dockerfile cache (Chromium layer + earlier layers stay cached), so the rebuild is ~2-3 min not the ~5 min cold cycle.67fd44c -
leaderboard: fetch results.jsonl via direct cache-busted HTTPS
Michael Rabinovichhf_hub_download (even with force_download=True) hands back stale bytes for several minutes after a commit: the local disk cache, revision pinning, and the Hub's CDN-fronted resolve endpoint all add their own staleness windows. End effect: pending and failed rows landed by the submit worker were on the Hub but invisible on the leaderboard tab until you waited it out, defeating the whole point of Refresh. Switch _load_rows_from_hub to a plain requests.get against the dataset's resolve/main URL with Cache-Control: no-cache and a millisecond cache-bust query param. Auth via huggingface_hub get_token() so the same code reads the still-private submissions dataset from the Space (HF_TOKEN secret) and from local dev (~/.cache/huggingface/token) without forking the code path. Adds requests to requirements.txt (was a transitive dep via huggingface_hub; making it explicit so a future upstream change doesn't silently drop it).9fcb684 -
submit: background-thread eval, flip row pending -> completed / failed
Michael RabinovichStep 6 (E) chunk 4. On a successful submit handle_submit now spawns a daemon thread that runs `cadgenbench evaluate` then `cadgenbench report single` over the unpacked submission, uploads reports/<id>.{html,json} to the submissions dataset, reads run_summary.json, and flips the row pending -> completed under the existing _HUB_LOCK. Any worker-side exception flips the row to failed with a short failure_reason (<=200 chars; full traceback goes to Space logs). Tempdir cleanup always runs in finally. Pipeline per submission, in the worker: 1. cadgenbench evaluate <run_dir> --workers 8 (subprocess via `python -m cadgenbench.cli`; the eval CLI already fans out across fixtures with ProcessPoolExecutor, self-throttles to n_fixtures when smaller). 2. cadgenbench report single <run_dir> -o <tmp>/<id>.html 3. Build reports JSON: run_summary.json + each fixture's result.json bundled together (mirrors README's description of reports/<id>.json as the "machine-readable…855bb8a -
leaderboard: render pending / failed score cells with a status tag
Michael RabinovichBlank score cells on a pending row left the only "this is in progress" cue in the status column, which read as broken to a user expecting at least *some* signal in the score columns. _fmt_pct and _fmt_score are now status-aware: pending rows render the tag "evaluating..." (hourglass prefix) in both score columns, failed rows render "failed" (cross prefix). Completed rows render the score number as before. Matches schema.md's "table renders the three states distinctly (spinner / score / error tag)" description and lines up with the visual pattern other HF leaderboards use while a row is queued.5aee3e5 -
leaderboard: handle null score fields on pending / failed rows
Michael RabinovichThe pending row landed by chunk 3 carries null validity_rate + aggregate_score (correct per the schema). pandas coerces JSON null to NaN on column construction, and _fmt_pct's `if x is None` gate didn't catch NaN; int(NaN) then raised ValueError inside .map(), bringing down load_leaderboard and rendering the leaderboard tab as a generic "Error". Fix: - _fmt_pct: gate on pd.isna(x) (catches both None and NaN), return "" for null. - _fmt_score: matching formatter for aggregate_score (was previously rendered as the raw float; with nulls in the mix it'd show as "NaN" cells). Returns "" for null, "%.4f" otherwise. Pending and failed rows now render with blank cells in both score columns; the row-level status column carries the meaning.00d091d -
app: drop history-narrative prose from Submit tab
Michael RabinovichThe Submit tab carried a sentence explaining why there's no form to fill in ("All five fields come from meta.json inside the zip, no separate form to fill in."). To a launch visitor that explains an absence that was never there; the right thing is to describe the contract as it stands, not to narrate a removal. Cut the sentence; keep the one useful half as a "Consent." line documenting that agree_to_publish: true in meta.json is the consent gate.a58058c -
submit: drop the agree checkbox, meta.json is the only consent gate
Michael RabinovichThe Submit tab had both an "I agree" checkbox and required meta.json to carry "agree_to_publish": true. Two gates expressing the same intent, neither covering a case the other didn't, and the UI half doesn't extend to programmatic submitters (gradio_client, future REST API). Drop the UI checkbox. meta.json["agree_to_publish"] stays the single consent gate, uniform across browser + programmatic submission. handle_submit signature shrinks to (zip_file). Submit form is now: zip upload + Submit button.9e84592 -
submit: drop dead form fields, meta.json is the only source
Michael RabinovichBefore: the Submit tab carried four text inputs (submission name, submitter name, agent URL, notes) alongside the zip upload, and handle_submit took them as args but ignored everything except the agree checkbox. They survived chunk 2's move from "form-driven stub" to "meta.json contract" only to keep the gr.click signature stable across one commit; chunk 3 didn't touch them either. After: Submit tab is just zip upload + agree checkbox + submit button. handle_submit signature is (zip_file, agree). All five submission fields come from meta.json inside the zip, full stop. Cuts 30 lines of UI + 4 unused handler params; removes a real user-confusion source (filling the form did nothing).f571140 -
submit: write pending row + upload zip on successful validation
Michael RabinovichStep 6 (E) chunk 3. Cheap-sync validation (chunk 2) now also commits to the Hub: on success the handler uploads submissions/<id>.zip and appends a status=pending row to results.jsonl under a process-wide threading.Lock. Eval still stubbed, the row stays pending forever until chunk 4 lands the background worker. Hub-write ordering (after validation passes): 1. Upload submissions/<id>.zip. Unique path per submission, no lock needed (HfApi handles its own commit serialization). 2. Build the pending row: metadata fields + status="pending" + failure_reason=null + submission_blob_url + null score fields, matching the "Pending" regime in cadgenbench-submissions/schema.md. 3. Acquire _HUB_LOCK, download current results.jsonl (treat EntryNotFoundError as empty), append one JSON line, re-upload. Lock held only for the RMW cycle (~1-2s), not for eval time; concurrent submitters serialise here, not in the worker. If step 1 fails the submitter sees a clean "Hub…1748c76 -
submit: move handle_submit into its own module + add validation pipeline
Michael RabinovichStep 6 (E) chunk 2 of the async submit work. Lifts the UI-only stub out of app.py into submit.py and grows it into the real cheap-sync validation pipeline. Still no Hub writes and no eval kick-off; the handler returns "would queue here" placeholder text. Async write + worker thread land in later chunks. Validation gates, in order: 1. Form: zip attached, agree-checkbox ticked. 2. Zip safety: valid zip, no absolute / parent-traversing entry names, no symlinks (Python 3.12's extractall normalises unsafe paths silently; we reject so submitters see a clear error). 3. meta.json schema: required keys, types, agree_to_publish is literal true, notes within the 500-char cap. meta.json is the source of truth for submitter / submission identity; the other form fields are advisory for now and kept only so the gr.click signature stays stable. 4. Fixture set: top-level dirs in the zip equal the fixture set reported by cadgenbench.common.paths.data_inputs_di…0501689 -
Dockerfile: bump CADGENBENCH_SHA to ed31274 (parallel evaluate)
Michael RabinovichPicks up the eval orchestrator's switch to ProcessPoolExecutor across fixtures (cadgenbench commit ed31274). On the Space's cpu-upgrade tier (8 vCPU) this should give a near-linear speedup on real-GT-scale runs; locally on the 5-stub set we saw ~2.4x going from --workers 1 to --workers 4 (1m13s -> 31s). The default worker count is min(8, os.cpu_count()) so the Space gets the speedup without setting any env var. Verified locally: per-fixture cad_score outputs are byte-identical between sequential and parallel runs (eval is per-fixture independent + deterministic).91244c6 -
Dockerfile: copy all Python modules, not just app.py
Michael RabinovichPrevious COPY hardcoded app.py only, which broke the moment app.py imported sibling leaderboard.py: the new module never made it into the image and the runtime hit ModuleNotFoundError. Switching to *.py picks up every Python file in AI4Engineering/ (currently app.py + leaderboard.py; submit.py lands next), so future module additions don't need a Dockerfile bump.f26021f -
refactor: split leaderboard read path into its own module
Michael RabinovichMirrors the shape adyen/DABstep uses on their reference Space (UI assembly in app.py, business logic in a sibling module). app.py drops from 224 lines to 158 by extracting the env-driven repo constants, the LOCAL_RESULTS_PATH + LEADERBOARD_COLS schema, the Hub/local row loaders, and load_leaderboard() into leaderboard.py. The submit handler stays in app.py for now; it'll move to a third module (submit.py) when Step 6 (E) lands the real validation + async eval. No behaviour change: load_leaderboard is imported back into app.py and wired into the same gr.Dataframe / refresh button pair as before. Smoke-tested locally (cadgenbench-space env): app module imports cleanly, load_leaderboard() returns a dataframe with the expected columns.b5ad973 -
docs: rewrite Space README + About tab as declarative current state
Michael Rabinovich- README.md: drop the (private; was michaelr27/...) history aside, drop the broken pointer to legacy/ (the directory was emptied during the NIST scrub), tighten the status sentence. - app.py ABOUT_MD: rewrite as a clean four-bullet "what this is" rather than a "status: active development" disclaimer block. Drops the broken pointer to space-setup/ (those notes are local-only and not in any public repo). Datasets bullet now describes both repos (inputs at HF_DATA_REPO, results at HF_SUBMISSIONS_REPO) with proper Hub links rather than just the repo names.628bc9e -
Dockerfile: bump CADGENBENCH_SHA to 1747493 (Hub-only fixtures)
Michael Rabinovichcadgenbench main now resolves fixtures exclusively from the Hub dataset repos: the wheel-bundled _data/ and the in-repo data/ folder are both gone (commit 1747493). The Space's Settings variables CADGENBENCH_DATA_REPO and CADGENBENCH_DATA_GT_REPO are already set (chunk 5), and HF_TOKEN is the existing Space secret, so this rebuild verifies the end-to-end Hub-only path inside the production runtime.d8207be -
Dockerfile: bump CADGENBENCH_SHA to 36ac6b4 (Hub-aware paths.py)
Michael Rabinovichcadgenbench commit 36ac6b4 adds the Hub-dataset resolution branch to common/paths.py (snapshot_download from $CADGENBENCH_DATA_REPO / $CADGENBENCH_DATA_GT_REPO when set, falling back to the existing local/bundled chain otherwise). The Space's Settings variables CADGENBENCH_DATA_REPO and CADGENBENCH_DATA_GT_REPO are set in lockstep with this bump (HuggingAI4Engineering/cadgenbench-data and HuggingAI4Engineering/cadgenbench-data-gt respectively), so the runtime now resolves fixtures from the Hub dataset repos instead of the wheel-bundled fallback. GT dataset is private; the existing HF_TOKEN Space secret provides read access (already set during Step 5 for the submissions dataset). No new secret required.aa1e886