Commit History
-
app: use a colon in the submit-tab system-agnostic note
Michael Rabinovich8a21dae -
readme: describe the space card in client-facing terms
Michael Rabinovich6d31809 -
app: default the admin validation_method to manual
Michael Rabinovichmanual (we re-ran the agent ourselves) is the common case for the maintainer team, so preselect it on the promote radio.46e3a72 -
app: credit Mecado as CAD data source in About tab
Michael Rabinovich +1: CursorCo-authored-by: Cursor <cursoragent@cursor.com>0c44305 -
app: admin tab as a multi-select table with delete
Michael Rabinovichreplaces the single-row dropdown with an editable table: one row per submission, a leading select checkbox the only editable column, the rest read-only context. promote, demote and delete now act on every ticked row at once. promote applies the chosen validation_method to the whole selection. delete sits in a danger-zone accordion behind a confirm checkbox that arms the button, and clears companion artifacts plus the row. a live count line reports how many rows are selected.58ce093 -
admin: bulk promote/demote/delete + admin table loader
Michael Rabinovichadds load_admin_table to the leaderboard reader: a single flat frame of every row (both tiers) with a leading editable select column, for the admin tab to act on. reworks admin.py around bulk helpers (promote_rows, demote_rows, delete_rows) that each do one results.jsonl write for the whole selection; delete also best-effort removes the companion zip and report artifacts. the singular promote_row/demote_row stay as thin wrappers. tests cover the bulk paths, a missing-id abort, empty-selection guards, and artifact deletion, all with the hub mocked.0957a56 -
tests: cover the admin gate and promote/demote helpers
Michael Rabinovichfour unit tests for admin.py: the CADGENBENCH_ADMINS gate (in-set, out-of-set, logged-out, empty set), promote and demote happy paths, and idempotent re-promotion. the results.jsonl read and write that _hub_rmw_results drives are mocked, so the suite makes no hub calls.007a50e -
app: add admin tab for promoting and demoting rows
Michael Rabinovichnew admin tab with a login button, a submission dropdown, a validation_method radio, and mark validated / mark unvalidated buttons. the controls are gated on membership in the CADGENBENCH_ADMINS set via a blocks.load handler, mirroring the submit button's login gate; the promote and demote handlers re-check admin rights server-side and refresh the dropdown plus both leaderboard tables after a write.47c86cf -
admin: promote/demote helpers gated by CADGENBENCH_ADMINS
Michael Rabinovichadds admin.py with is_admin, promote_row and demote_row. promotion sets validation_status to validated and records the evidence type; demotion reverses it and nulls validation_method. row writes reuse submit._hub_rmw_results so the lock and read-modify-write semantics match the submit path. the admin set is read from the CADGENBENCH_ADMINS space variable.0bd3c0a -
submit: drop invisible button-progress bar
Verified signature7655daf -
submit: restore Validating... toast on click
Verified signatureef37489 -
step 10 / jobs-migration: requirements.txt -> dispatch eval to HF Jobs
Verified signature8ff1be8 -
step 10 / jobs-migration: submit.py -> dispatch eval to HF Jobs
Verified signaturefd61157
-
submit: drop eval workers 4 -> 1 (OOM under parallel osmesa)
Michael Rabinovich +1: CursorCo-authored-by: Cursor <cursoragent@cursor.com>ed2a486 -
Dockerfile: swap vtk for vtk-osmesa so off-screen rendering works headless
Michael Rabinovich +1: CursorThe stock pyvista wheel pulls the stock vtk wheel from PyPI, which is built with vtkXOpenGLRenderWindow. That render window class needs an X server even in off_screen mode, so it segfaults the eval worker on the Space's truly-headless container ("DISPLAY=" with no X11 socket). The parent ProcessPoolExecutor sees the segfault as BrokenProcessPool six minutes into the eval, which is exactly the failure mode the first VTK submission hit. vtk-osmesa is the same VTK build but compiled against Mesa OSMesa (pure CPU software rasteriser, statically linked, no display server required). It shares the `import vtk` namespace so pyvista picks it up transparently. This is the canonical PyVista headless-Docker fix (per the PyVista docs + the maintainers' own Docker images). The wheel lives on Kitware's index, not PyPI proper. Co-authored-by: Cursor <cursoragent@cursor.com>545c818 -
submit: dump full eval/report subprocess output to container log on failure
Michael Rabinovich +1: CursorCo-authored-by: Cursor <cursoragent@cursor.com>0648533 -
Dockerfile: bump CADGENBENCH_SHA to b22a53c (ProcessPool spawn fix)
Michael Rabinovich +1: CursorCo-authored-by: Cursor <cursoragent@cursor.com>fa32cdd -
Dockerfile + submit: swap Chromium for VTK; bump eval workers 1 -> 4
Michael Rabinovich +1: CursorPairs with cadgenbench 3fd213d (in-process VTK / PyVista renderer). Dockerfile drops the playwright install + Chromium browser layer (~200 MB), drops PLAYWRIGHT_BROWSERS_PATH, and adds the Mesa GL runtime libs VTK needs for off-screen rendering on the Space's cpu-upgrade tier (libgl1 was already there; libegl1, libegl-mesa0, libglx-mesa0 are new). CADGENBENCH_SHA bumps to 3fd213d. EVAL_WORKER_COUNT goes from "1" back to "4": the VTK renderer is cheap enough that one ProcessPoolExecutor worker per fixture no longer oversubscribes the host the way the old Chromium subprocess did. Co-authored-by: Cursor <cursoragent@cursor.com>0626a6b -
leaderboard: format submitted_at as `YYYY-MM-DD HH:MM UTC`; lock tables read-only
Michael RabinovichTable cells were editable (gradio_leaderboard inherits gr.Dataframe's interactive default); add interactive=False to both Validated and Unvalidated widgets so cell contents aren't accidentally typed over. submitted_at column rendered as raw ISO-8601 ("2026-05-28T07:13:16Z"); apply the same _fmt_timestamp helper the detail panel already uses so the table reads as "2026-05-28 07:13 UTC". Helper moved into leaderboard.py (formatter module) and re-imported by app.py.c4e21b3 -
submit: serialize cadgenbench evaluate to dodge cpu-upgrade contention
Michael RabinovichEVAL_WORKER_COUNT goes from "8" to "1". Removes the debug /debug/render-bench route added in the previous commit. Why: on the Space's cpu-upgrade tier today, a single headless- Chromium render takes ~7s; five in parallel collapse the host and each individual render blows past its 120s subprocess timeout. Same renderer, same fixtures, same Playwright version run on a laptop in ~5s per render with no slowdown under 5-way parallelism. The slow path is HF's shared host, not our renderer. Sequential eval (one fixture at a time, one render at a time) gives each Chromium the whole box. Five fixtures finish in ~30-60s wall time, far under the 15-minute outer EVAL_TIMEOUT_SECONDS budget. Tradeoff: total eval wall time grows linearly with fixture count. For today's five fixtures that's a few seconds slower than the healthy-host parallel case, and a huge cliff better than a failed eval. For the 100-fixture launch this becomes ~10 minutes per submission, which is the limit of this approach: a r…6e3ab50 -
debug: add /debug/render-bench route for one-shot Chromium timing
Michael RabinovichRead-only FastAPI endpoint mounted on the Space's parent app. Sequentially times `render_step` against each fixture's input.step in cadgenbench-data and returns per-fixture seconds + view count. Used to compare per-render cost on the Space's container vs a local reference when eval has been timing out at the 120s render_step ceiling but nothing on the render path changed in our code. Curl-fetchable under Bearer auth (the Space is private): curl -H "Authorization: Bearer $HF_TOKEN" \ https://<space>.hf.space/debug/render-bench Not user-visible (no UI), no side effects, no row writes. Safe to remove once the diagnosis lands.37e45d8 -
submit+app: gate Submit on HF OAuth; populate hf_username from profile
Michael RabinovichBundle 1+2 C10. Login-required submit shaped per the bundle spec. submit.py: - `handle_submit` grows a `profile: gr.OAuthProfile | None` arg. Gradio dependency-injects the visitor's profile based on the signature. None -> immediate `gr.Error("Please log in via the HF button before submitting.")` so a UI mishap can't write an anonymous row. - `_build_pending_row` grows an `hf_username=None` kwarg. `handle_submit` passes `profile.username` through so the row carries the canonical HF identity (not a free-text claim in meta.json's submitter_name). app.py: - New `gr.LoginButton()` in the Submit tab above the zip dropzone. - Submit button starts `interactive=False`; new `_enable_submit_when_logged_in(profile)` reads the visitor's OAuth profile via `blocks.load` and flips the button on / off per page load. Server-side gate in `handle_submit` mirrors this so a logged-out user who somehow bypasses the disabled-button UI still gets rejected. requirements.txt: - `gradio…c87b253 -
submit+app: toast-based feedback (validating / queued / refreshed)
Michael RabinovichBundle 1+2 C9. Replaces the static markdown reply on the Submit tab with gr.Info / gr.Error toasts, and adds a refresh toast on the manual Refresh button (the Timer auto-refresh stays silent - a toast every 10s would be noise). submit.py: - handle_submit no longer returns a markdown string. It side- effects: gr.Info("Validating submission...") at the top of the validation stage, gr.Info("Submission <id> queued ...") after the row + zip are on the Hub and the worker has been spawned. Every rejection path raises gr.Error so the user gets a red toast and the handler aborts; the outer try/finally still runs cleanup. Internal _ValidationError + _HubWriteError sentinels preserved. - _validate_form returns a plain message (no `**Error:**` markdown wrap; the caller wraps into gr.Error). - All four rejection paths covered: 1. Form (no zip attached). 2. Validation gate (extract / meta / fixtures / steps). 3. Dedup gate (sha256 match on existing row). 4. Hub write…6facf47
-
leaderboard+app: combined CSV download with validation_status discriminator
Michael RabinovichBundle 1+2 C8. One file, both tiers, downloaded via `gr.DownloadButton` next to the existing Refresh button. leaderboard.py: - New `CSV_COLUMNS` constant (wider than the on-screen table: raw aggregate_score / validity_rate, plus provenance fields like submission_sha256 + cadgenbench_data_revision + notes + failure_reason). `validation_status` is the discriminator that lets a reader grep the file for one tier or the other. - New `build_combined_csv()` reads the same rows the leaderboard reader does, applies the same status / validation_status defaults, projects to CSV_COLUMNS, sort: validated rows on top (highest score first) then unvalidated, then writes to a unique /tmp file and returns the path. Fresh on every click so the export reflects the latest data, not a snapshot captured at boot. app.py: - Imports `build_combined_csv`. The Refresh button and the new `gr.DownloadButton("Download CSV", size="sm")` live in a single `gr.Row` so they sit side-by-side unde…f585077 -
app: Citation + Validation Guidelines accordions, validation link in About
Michael RabinovichBundle 1+2 C7. Two collapsed accordions above the leaderboard tables plus a validation-policy bullet on the About tab. - Validation Guidelines accordion (collapsed by default, sits above the tables). Carries the two-tier story in two lines (unvalidated -> validated, four evidence types) plus a link to the canonical policy doc in the main repo at `cadgenbench/docs/benchmark/validation.md`. Bundle 4 creates the file; the link will 404 until that lands, then resolves. - Citation accordion (collapsed by default, below Validation Guidelines). Carries the verbatim BibTeX entry locked in space-setup/bundles/1-2-space-ux.md (cite key cadgenbench2026, Space URL as howpublished). Rendered in a `gr.Code` with language=None (no BibTeX highlighter ships with gradio) and show_line_numbers off so the entry stays copy-paste tidy. - About tab picks up a new bullet linking the same validation doc. VALIDATION_DOC_URL is centralised at module level so the Validation Guidelines markdo…97b9a4a -
app: render reports inline via iframe srcdoc (private-Space safe)
Michael RabinovichPolish follow-up on C6. Reports now render inside the Space, inside the same iframe HF uses to host the leaderboard, with no second HTTP request leaving the browser. Why: the Space is private, and HF's edge gates same-origin pathname navigations on a short-lived __sign JWT that browsers don't carry forward when a user clicks a relative link. The previous design (submission_name -> /reports/<id>.html) worked server-side under Bearer auth (verified) but 404'd in the browser. Confirmed via external review and reproduced with a requests.Session cookie test. This commit: - Drops the submission_name -> /reports/<id>.html link wrapping from the rendered table; the cell stays plain text. The underlying `report_url` column (relative URL to the proxy) is still computed for the modern-pipeline gate but no longer surfaced as a clickable cell. - Adds a `gr.HTML` viewer below the detail panel. Row-click triggers `_format_detail_and_report`, which returns (detail_markdown, report_ifram…0e3b21f -
app+leaderboard: link submission_name to a Space-side report proxy
Michael RabinovichPolish follow-up #2 on C6. Three user-visible changes plus the plumbing required to make them work. 1. Drop the `report` and `zip` columns from the rendered table. `submission_name` becomes the report's deep link (typical HF leaderboard pattern: the system / model name doubles as the link target; cf. Open LLM Leaderboard, DABstep, BigCodeBench). Modern completed rows get `[<name>](/reports/<sid>.html)`; pending, failed, and legacy seed rows fall back to plain text so there's no broken click. The hidden `submission_blob_url` and the computed `report_url` ride along in the DataFrame and surface in the row-click detail panel only. 2. Rename the agent-URL column from `agent` to `model details (optional)`. Missing cells render as italic `_None_` so the optionality is explicit instead of a blank. "Agent" was confusing; the schema lets `agent_url` be code OR paper OR a blog post and "agent" reads narrow. 3. Add a Space-side proxy for the per-submission H…77edebf -
app+leaderboard: detail panel polish (rename link columns, fix broken report links)
Michael RabinovichPolish follow-up on C6, three fixes after the first detail-panel commit landed: 1. Rename display columns from snake_case schema names to clean short headers. `agent_url` -> `agent`, `submission_blob_url` -> `zip`, `report_url` -> `report`. The rename happens at projection time via `df.rename`, so the schema in results.jsonl is unchanged. Headers now read as plain English ("agent", "zip", "report") rather than a python dict. 2. The `agent` column shows the URL itself (scheme stripped, truncated past 40 chars) as link text instead of a fixed "code" label. The schema says agent_url can be code OR paper; the shortened URL is the only honest hint about what's behind the click. Missing values render as italic `_None_` rather than a blank cell, so the optionality is explicit. 3. The `report` link is now gated on the row's `submission_sha256` being non-null in addition to status == "completed". Legacy seed rows pre-date the modern submit pipeline (which …1a8f331 -
app: row-click detail panel below the leaderboard tables
Michael RabinovichBundle 1+2 C6. Single shared `gr.Markdown` panel below both Leaderboard widgets; either table's `.select(...)` event populates it with the clicked row's notes, agent_url, submitted_at, submission zip + report links (always), and failure_reason (failed rows only). Spec'd shape: one panel, two select sources. leaderboard.py: - LEADERBOARD_COLS + VALIDATED_LEADERBOARD_COLS each grow by three hidden detail-only fields (submission_id, notes, failure_reason). They ride in the underlying DataFrame so the select handler can read them out directly without a parallel state cache. - New LEADERBOARD_HIDE_COLUMNS constant lists the columns hidden from view; both Leaderboard widgets in app.py consume it. - `_project_and_format` now fills missing columns with None (legacy rows can lack optional fields like notes; the panel needs the column-set to be stable regardless of source-row shape). app.py: - pandas imported (for the helper's pd.isna check). - New `_format_detail(df, evt)` helper …3112173 -
leaderboard: markdown link columns for agent_url, submission, report
Michael RabinovichBundle 1+2 C5. Three new columns on both tables (validated + unvalidated) carry clickable links rendered via gradio_leaderboard's `datatype="markdown"`: - agent_url: link to the agent's source / paper if the row's meta.json provided one (label "code"; empty cell when null). - submission_blob_url: link to the uploaded zip on the submissions dataset (label "zip"; populated on every non-failed row, the same URL the submit handler computed at upload time). - report_url: built at read time from `submission_id` plus HF_SUBMISSIONS_REPO; points at `reports/<id>.html` under /resolve/main/ so the browser renders the report directly. Only emitted for `status == "completed"` rows. leaderboard.py: - LEADERBOARD_COLS + VALIDATED_LEADERBOARD_COLS grow by three. - New LEADERBOARD_DATATYPES + VALIDATED_LEADERBOARD_DATATYPES constants mark the link columns as "markdown", everything else as "str". Tied to the column lists by a single helper so the two stay in lockstep. - New _agent_u…53de73a -
submit: default validation + hf_username fields on the pending row
Michael RabinovichBundle 1+2 C4. `_build_pending_row` adds three new keys matching the schema bump from C1, in the schema.md column order: - validation_status = "unvalidated" (every new row starts in the unvalidated tier; maintainers promote via the dataset repo after methodology review). - validation_method = None (set by the maintainer at promotion time to one of code | traces | api | manual). - hf_username = None (placeholder; C10 plumbs through the value from gradio.OAuthProfile.username once the LoginButton lands). Pre-existing fields, submitted_at stamping, the cadgenbench version + data revision tags, the score-shaped null fields, the blob URL, and the sha256 are untouched. Dedup gate still pivots on submission_sha256 exactly as before. tests/test_submit.py covers the C4 acceptance: - new-row defaults: validation_status is "unvalidated", validation_method + hf_username are None; - sha256 path passthrough: the hash argument lands on the row's submission_sha256 key intact (existing …a533dd2 -
docs: add leaderboard discovery tags to README front-matter
Michael Rabinovichtags: surfaces this Space in the Hub's leaderboard discovery filter (huggingface.co/spaces?other=leaderboard) and faceted searches by modality, submission type, and eval type. The set blends the empirical vocabulary across the top 50 leaderboard-tagged Spaces with three truthful novel descriptors: modality:3d (closest existing values are modality:text / tools, neither accurate for STEP geometry), eval:cad (paired with eval:generation so the standard "generation" chip also surfaces us), and judge:auto (eval is metric-based, no LLM/human judge).36546fc -
app: split Leaderboard into Validated + Unvalidated tables
Michael RabinovichBundle 1+2 C3. Swaps the single Leaderboard widget for two stacked ones (Validated on top, Unvalidated below) and wires the existing auto-refresh Timer + Refresh button to push fresh dataframes into both. Initial render reads via the C2 `load_leaderboard_split()`; existing rows all land in Unvalidated until a maintainer flips `validation_status` to `"validated"` on a row in the submissions dataset. - app.py: replaces `df_view` with `validated_view` + `unvalidated_view`. Single timer tick now fans out to both widgets (load split runs once per tick, both halves get pushed). Refresh button same fan-out shape. Inline comment points at the validation policy decision doc. - leaderboard.py: drops the now-unused single-DataFrame `load_leaderboard()` per the C3 cleanup half. Stale docstring references that pointed at the dead function got smoothed out (the split reader's doc is self-contained now). - tests/test_smoke.py: picks up the C3 acceptance with a new `test_two_leaderboa…046548a -
license: add Apache-2.0
Michael Rabinovich8676e14 -
leaderboard: add load_leaderboard_split() for two-tier viewer
Michael RabinovichBundle 1+2 C2. Additive new function for the C3 two-stacked-table swap; existing single-DataFrame load_leaderboard() stays put until C3 flips the caller in app.py + drops it. Keeps the live Space green between commits. - VALIDATED_LEADERBOARD_COLS: same shape as LEADERBOARD_COLS plus `validation_method` (always null on unvalidated rows, so omitted from that tier rather than rendered). - load_leaderboard_split() -> (validated_df, unvalidated_df): reads results.jsonl the same way as load_leaderboard(), splits rows on `validation_status` (defensive: anything not literally "validated" lands in unvalidated, which covers legacy + null + any future-unknown value), runs the same status-aware cell formatting per tier via the new _project_and_format helper. - _project_and_format: shared sort + projection + pending/failed cell tagging so the two tiers produce identically-shaped cells. Legacy default: rows without `validation_status` get "unvalidated" via row-level fill. No resul…0dd7215 -
docs: link companion datasets in README front-matter
Michael Rabinovichdatasets: lists the two companion dataset repos so the Hub renders them as cards on the Space page (and so the link is explicit rather than relying on auto-parse from hardcoded repo IDs in app.py / leaderboard.py). - HuggingAI4Engineering/cadgenbench-submissions: the results + per-submission report dataset the leaderboard reads from. - HuggingAI4Engineering/cadgenbench-data: the public fixture inputs. cadgenbench-data-gt is omitted on purpose (permanently private).1402833 -
docs: add hf_oauth + apache-2.0 license to README front-matter
Michael Rabinovichhf_oauth: true is what wires the LoginButton + OAuth callback the Submit tab relies on; without it the auth-required submit flow is silently broken (login click no-ops, no token reaches the handler). license: apache-2.0 matches the HF Leaderboards template (gradio-templates/leaderboard) and the rest of the cadgenbench code surface, and renders a license badge on the Space page.0bb6044 -
tests: pytest + playwright headless smoke harness
Michael RabinovichFirst Space-side test harness; sets up the floor for the Phase D bundle's per-commit unit + UI tests (split leaderboard, pending-row defaults, OAuth-required submit) without slipping a regression past a build-only check. requirements-dev.txt pulls -r requirements.txt then adds pytest + playwright. One `pip install -r requirements-dev.txt && playwright install chromium` brings a fresh venv up to speed. tests/conftest.py exposes a session-scoped `app_url` fixture: boots app.py in a subprocess on a free port, polls until the HTTP server answers, yields the URL, tears the process down on teardown. Logs land in a tmp file so a startup failure surfaces the real Gradio traceback rather than a bare RuntimeError. Boot-time stuck-pending sweep in submit.py is disabled via CADGENBENCH_DISABLE_BOOT_SWEEP=1 so the fixture is fast and offline-tolerant. tests/test_smoke.py launches headless Chromium via playwright's sync API, navigates to the fixture's URL, and asserts the Leaderboard / Submit / A…21b071a -
Dockerfile: bump CADGENBENCH_SHA to 056cfbb (invalid-candidate fast path)
Michael Rabinovichff674cc -
submit: stuck-pending sweep on boot (chunk 6)
Michael RabinovichStep 6 (E) chunk 6. A pending row whose worker died (Space restart for a deploy, OOM, crash) has no one to flip it; without this sweep it stays "pending" in the leaderboard forever (we hit this twice during chunk-3..5 churn already). On submit.py module import, spawn a daemon thread that: 1. Downloads results.jsonl (no lock; read-only). 2. Iterates rows; for each pending row whose `submitted_at` is older than 30 min, calls _flip_row_to_failed(sid, "evaluation interrupted by Space restart"). 3. Logs the list of stuck ids so a deployer can see what was swept. Threshold is 30 min: well above the real eval ceiling on cpu-upgrade (~5 min), so a genuinely-still-running submission is safe. Hub-fetch failure at boot is non-fatal (logged warning, sweep skipped; next boot retries). Per-row flip failures are caught + logged + skipped (the rest of the sweep continues). Opt-out via CADGENBENCH_DISABLE_BOOT_SWEEP=1 for test imports that don't want the Hub round-trip. Two tes…0a2a2c6 -
submit: reject byte-identical re-submits via SHA-256 dedup
Michael RabinovichBefore: clicking Submit twice (or re-uploading the same zip after a transient confusion) created two rows pointing at byte-identical content, ran the eval twice for the same input, and bloated the submissions/reports blobs. After: the cheap-sync handler computes sha256 of the uploaded zip right after validation passes, queries results.jsonl for any row with the same submission_sha256, and rejects with a pointer to the existing submission_id if a match is found. Hash also lands on the new row's submission_sha256 field (schema bump in the companion commit on the submissions dataset). Dedup-fetch failures fall back to letting the submit through rather than rejecting on a Hub blip (logged). Race window: two near-simultaneous identical submits whose dedup checks both return None before either appends will land two rows; recoverable by manual cleanup if it ever happens, not worth tightening before auth + login throttling lands at pre-launch. Drive-by: _build_pending_row's unused fixture_n…3237736
-
app: move theme= from Blocks.launch() to Blocks() ctor (Gradio 5 API)
Michael Rabinovich76f0611 -
app+requirements: pin Gradio 5 + gradio_leaderboard, auto-refresh now works
Michael RabinovichStep 6 (E) chunk 5, take 2. The Gradio 6.14 gr.Dataframe component silently swallows row updates after the first render: server-side load_leaderboard fires every N seconds (logged: "Loaded 11 rows from Hub" on each tick) but the rendered DOM never picks the new rows up. Confirmed with a local row-growing fixture (load_leaderboard monkey-patched to return N+1 rows on each call); the server fired 10 times, the DOM stuck on the first call's row count. Tried in order, all failed against the local fixture: - gr.Dataframe(every=10, value=callable) - gr.Timer(10).tick(outputs=gr.Dataframe) - manual Refresh button click - @gr.render(inputs=[gr.State, gr.State], key=f"...-{tick}") - @gr.render(inputs=[hidden gr.Textbox], key=f"...-{tick}") - streaming generator on app.load (first yield delivered; every subsequent yield ignored client-side) What actually works: pin to Gradio 5 (latest 5.50.0) and use the gradio_leaderboard.Leaderboard custom component, which has its own update …4e86f82 -
app: fix leaderboard auto-refresh via @gr .render(key=...)
Michael RabinovichIn Gradio 6.14 a `gr.Dataframe` identity-checks its component id and short-circuits the diff when the new value's shape + column structure look unchanged, even when the row data differs. That swallowed updates from `every=` on the Dataframe AND from `gr.Timer().tick(outputs=df_view)` AND from a manual Refresh button click: the server-side fetch ran every tick (`Loaded 10 rows from Hub` in logs) but the browser never saw the new rows. Only a full page reload picked them up. Documented fix (post Gradio's March 2026 Dataframe overhaul): wrap the component in `@gr.render` and give the inner Dataframe a `key=` that changes on every tick. Gradio then tears down and rebuilds the component subtree in place, picking up the fresh value instead of trying to diff in place. Shape: - tick_state: gr.State holding a ms-timestamp that bumps on every refresh; included in the Dataframe `key=`. - table_state: gr.State holding the pandas DataFrame. - `_refresh_table()` runs once per tick, retur…f2f35be -
app: drive auto-refresh via gr.Timer().tick instead of every= on Dataframe
Michael Rabinovichgr.Dataframe(every=N) was firing load_leaderboard on the server every 10s (verified via runtime logs), but the component's client-side re-render after the initial value silently dropped rows added between ticks. Same shape as gradio-app/gradio#8160 ("when the data already exists ... it cannot be updated and the page needs to be refreshed"). Switch to gr.Timer(10).tick(fn=load_leaderboard, outputs=df_view). Identical code path to the manual Refresh button: each tick is an event fired through the standard component-update channel, which re-renders reliably. The Refresh button stays as the "I want it now" override.4ee70ef -
app: auto-refresh leaderboard + bootstrap logging
Michael RabinovichStep 6 (E) chunk 5 plus a diagnostic. Auto-refresh: - gr.Dataframe now takes `value=load_leaderboard` (callable, not its call result) plus `every=10`. Gradio re-invokes the function every 10 seconds and re-renders the dataframe, so pending rows visibly flip to completed / failed and freshly-submitted rows appear within 10s without any user action. Manual Refresh button stays as an "I want it now" override. Logging bootstrap: - logging.basicConfig(level=INFO, ...) at app startup so the `logger.info` / `logger.warning` / `logger.exception` calls already sprinkled through leaderboard.py and submit.py reach the Space's runtime logs. Without this they go nowhere (Python's default no-handler behaviour), which is why every refresh / worker pathology so far has been silent. Combined push: the auto-refresh might not surface new rows if the underlying load_leaderboard returns stale data on the Space for any reason (CDN, auth, frontend re-render bug)…b2f3ce6 -
Dockerfile: bump CADGENBENCH_SHA to 9bd02fb (lazy baseline import)
Michael RabinovichPicks up the cadgenbench fix that makes `cadgenbench --help` and `cadgenbench evaluate` work in the eval-only install (no `[baseline]` extra). Before this bump, the chunk-4 worker's `python -m cadgenbench.cli evaluate <run_dir>` subprocess died during argparse setup with `ModuleNotFoundError: No module named 'dotenv'`, flipping every submission to status=failed before any work happened. Bumping CADGENBENCH_SHA invalidates the cadgenbench layer in the Dockerfile cache (Chromium layer + earlier layers stay cached), so the rebuild is ~2-3 min not the ~5 min cold cycle.67fd44c -
leaderboard: fetch results.jsonl via direct cache-busted HTTPS
Michael Rabinovichhf_hub_download (even with force_download=True) hands back stale bytes for several minutes after a commit: the local disk cache, revision pinning, and the Hub's CDN-fronted resolve endpoint all add their own staleness windows. End effect: pending and failed rows landed by the submit worker were on the Hub but invisible on the leaderboard tab until you waited it out, defeating the whole point of Refresh. Switch _load_rows_from_hub to a plain requests.get against the dataset's resolve/main URL with Cache-Control: no-cache and a millisecond cache-bust query param. Auth via huggingface_hub get_token() so the same code reads the still-private submissions dataset from the Space (HF_TOKEN secret) and from local dev (~/.cache/huggingface/token) without forking the code path. Adds requests to requirements.txt (was a transitive dep via huggingface_hub; making it explicit so a future upstream change doesn't silently drop it).9fcb684 -
submit: background-thread eval, flip row pending -> completed / failed
Michael RabinovichStep 6 (E) chunk 4. On a successful submit handle_submit now spawns a daemon thread that runs `cadgenbench evaluate` then `cadgenbench report single` over the unpacked submission, uploads reports/<id>.{html,json} to the submissions dataset, reads run_summary.json, and flips the row pending -> completed under the existing _HUB_LOCK. Any worker-side exception flips the row to failed with a short failure_reason (<=200 chars; full traceback goes to Space logs). Tempdir cleanup always runs in finally. Pipeline per submission, in the worker: 1. cadgenbench evaluate <run_dir> --workers 8 (subprocess via `python -m cadgenbench.cli`; the eval CLI already fans out across fixtures with ProcessPoolExecutor, self-throttles to n_fixtures when smaller). 2. cadgenbench report single <run_dir> -o <tmp>/<id>.html 3. Build reports JSON: run_summary.json + each fixture's result.json bundled together (mirrors README's description of reports/<id>.json as the "machine-readable…855bb8a -
leaderboard: render pending / failed score cells with a status tag
Michael RabinovichBlank score cells on a pending row left the only "this is in progress" cue in the status column, which read as broken to a user expecting at least *some* signal in the score columns. _fmt_pct and _fmt_score are now status-aware: pending rows render the tag "evaluating..." (hourglass prefix) in both score columns, failed rows render "failed" (cross prefix). Completed rows render the score number as before. Matches schema.md's "table renders the three states distinctly (spinner / score / error tag)" description and lines up with the visual pattern other HF leaderboards use while a row is queued.5aee3e5