Commit History
-
app: use a colon in the submit-tab system-agnostic note
Michael Rabinovich8a21dae -
app: default the admin validation_method to manual
Michael Rabinovichmanual (we re-ran the agent ourselves) is the common case for the maintainer team, so preselect it on the promote radio.46e3a72 -
app: credit Mecado as CAD data source in About tab
Michael Rabinovich +1: CursorCo-authored-by: Cursor <cursoragent@cursor.com>0c44305 -
app: admin tab as a multi-select table with delete
Michael Rabinovichreplaces the single-row dropdown with an editable table: one row per submission, a leading select checkbox the only editable column, the rest read-only context. promote, demote and delete now act on every ticked row at once. promote applies the chosen validation_method to the whole selection. delete sits in a danger-zone accordion behind a confirm checkbox that arms the button, and clears companion artifacts plus the row. a live count line reports how many rows are selected.58ce093 -
app: add admin tab for promoting and demoting rows
Michael Rabinovichnew admin tab with a login button, a submission dropdown, a validation_method radio, and mark validated / mark unvalidated buttons. the controls are gated on membership in the CADGENBENCH_ADMINS set via a blocks.load handler, mirroring the submit button's login gate; the promote and demote handlers re-check admin rights server-side and refresh the dropdown plus both leaderboard tables after a write.47c86cf
-
leaderboard: format submitted_at as `YYYY-MM-DD HH:MM UTC`; lock tables read-only
Michael RabinovichTable cells were editable (gradio_leaderboard inherits gr.Dataframe's interactive default); add interactive=False to both Validated and Unvalidated widgets so cell contents aren't accidentally typed over. submitted_at column rendered as raw ISO-8601 ("2026-05-28T07:13:16Z"); apply the same _fmt_timestamp helper the detail panel already uses so the table reads as "2026-05-28 07:13 UTC". Helper moved into leaderboard.py (formatter module) and re-imported by app.py.c4e21b3 -
submit: serialize cadgenbench evaluate to dodge cpu-upgrade contention
Michael RabinovichEVAL_WORKER_COUNT goes from "8" to "1". Removes the debug /debug/render-bench route added in the previous commit. Why: on the Space's cpu-upgrade tier today, a single headless- Chromium render takes ~7s; five in parallel collapse the host and each individual render blows past its 120s subprocess timeout. Same renderer, same fixtures, same Playwright version run on a laptop in ~5s per render with no slowdown under 5-way parallelism. The slow path is HF's shared host, not our renderer. Sequential eval (one fixture at a time, one render at a time) gives each Chromium the whole box. Five fixtures finish in ~30-60s wall time, far under the 15-minute outer EVAL_TIMEOUT_SECONDS budget. Tradeoff: total eval wall time grows linearly with fixture count. For today's five fixtures that's a few seconds slower than the healthy-host parallel case, and a huge cliff better than a failed eval. For the 100-fixture launch this becomes ~10 minutes per submission, which is the limit of this approach: a r…6e3ab50 -
debug: add /debug/render-bench route for one-shot Chromium timing
Michael RabinovichRead-only FastAPI endpoint mounted on the Space's parent app. Sequentially times `render_step` against each fixture's input.step in cadgenbench-data and returns per-fixture seconds + view count. Used to compare per-render cost on the Space's container vs a local reference when eval has been timing out at the 120s render_step ceiling but nothing on the render path changed in our code. Curl-fetchable under Bearer auth (the Space is private): curl -H "Authorization: Bearer $HF_TOKEN" \ https://<space>.hf.space/debug/render-bench Not user-visible (no UI), no side effects, no row writes. Safe to remove once the diagnosis lands.37e45d8 -
submit+app: gate Submit on HF OAuth; populate hf_username from profile
Michael RabinovichBundle 1+2 C10. Login-required submit shaped per the bundle spec. submit.py: - `handle_submit` grows a `profile: gr.OAuthProfile | None` arg. Gradio dependency-injects the visitor's profile based on the signature. None -> immediate `gr.Error("Please log in via the HF button before submitting.")` so a UI mishap can't write an anonymous row. - `_build_pending_row` grows an `hf_username=None` kwarg. `handle_submit` passes `profile.username` through so the row carries the canonical HF identity (not a free-text claim in meta.json's submitter_name). app.py: - New `gr.LoginButton()` in the Submit tab above the zip dropzone. - Submit button starts `interactive=False`; new `_enable_submit_when_logged_in(profile)` reads the visitor's OAuth profile via `blocks.load` and flips the button on / off per page load. Server-side gate in `handle_submit` mirrors this so a logged-out user who somehow bypasses the disabled-button UI still gets rejected. requirements.txt: - `gradio…c87b253 -
submit+app: toast-based feedback (validating / queued / refreshed)
Michael RabinovichBundle 1+2 C9. Replaces the static markdown reply on the Submit tab with gr.Info / gr.Error toasts, and adds a refresh toast on the manual Refresh button (the Timer auto-refresh stays silent - a toast every 10s would be noise). submit.py: - handle_submit no longer returns a markdown string. It side- effects: gr.Info("Validating submission...") at the top of the validation stage, gr.Info("Submission <id> queued ...") after the row + zip are on the Hub and the worker has been spawned. Every rejection path raises gr.Error so the user gets a red toast and the handler aborts; the outer try/finally still runs cleanup. Internal _ValidationError + _HubWriteError sentinels preserved. - _validate_form returns a plain message (no `**Error:**` markdown wrap; the caller wraps into gr.Error). - All four rejection paths covered: 1. Form (no zip attached). 2. Validation gate (extract / meta / fixtures / steps). 3. Dedup gate (sha256 match on existing row). 4. Hub write…6facf47
-
leaderboard+app: combined CSV download with validation_status discriminator
Michael RabinovichBundle 1+2 C8. One file, both tiers, downloaded via `gr.DownloadButton` next to the existing Refresh button. leaderboard.py: - New `CSV_COLUMNS` constant (wider than the on-screen table: raw aggregate_score / validity_rate, plus provenance fields like submission_sha256 + cadgenbench_data_revision + notes + failure_reason). `validation_status` is the discriminator that lets a reader grep the file for one tier or the other. - New `build_combined_csv()` reads the same rows the leaderboard reader does, applies the same status / validation_status defaults, projects to CSV_COLUMNS, sort: validated rows on top (highest score first) then unvalidated, then writes to a unique /tmp file and returns the path. Fresh on every click so the export reflects the latest data, not a snapshot captured at boot. app.py: - Imports `build_combined_csv`. The Refresh button and the new `gr.DownloadButton("Download CSV", size="sm")` live in a single `gr.Row` so they sit side-by-side unde…f585077 -
app: Citation + Validation Guidelines accordions, validation link in About
Michael RabinovichBundle 1+2 C7. Two collapsed accordions above the leaderboard tables plus a validation-policy bullet on the About tab. - Validation Guidelines accordion (collapsed by default, sits above the tables). Carries the two-tier story in two lines (unvalidated -> validated, four evidence types) plus a link to the canonical policy doc in the main repo at `cadgenbench/docs/benchmark/validation.md`. Bundle 4 creates the file; the link will 404 until that lands, then resolves. - Citation accordion (collapsed by default, below Validation Guidelines). Carries the verbatim BibTeX entry locked in space-setup/bundles/1-2-space-ux.md (cite key cadgenbench2026, Space URL as howpublished). Rendered in a `gr.Code` with language=None (no BibTeX highlighter ships with gradio) and show_line_numbers off so the entry stays copy-paste tidy. - About tab picks up a new bullet linking the same validation doc. VALIDATION_DOC_URL is centralised at module level so the Validation Guidelines markdo…97b9a4a -
app: render reports inline via iframe srcdoc (private-Space safe)
Michael RabinovichPolish follow-up on C6. Reports now render inside the Space, inside the same iframe HF uses to host the leaderboard, with no second HTTP request leaving the browser. Why: the Space is private, and HF's edge gates same-origin pathname navigations on a short-lived __sign JWT that browsers don't carry forward when a user clicks a relative link. The previous design (submission_name -> /reports/<id>.html) worked server-side under Bearer auth (verified) but 404'd in the browser. Confirmed via external review and reproduced with a requests.Session cookie test. This commit: - Drops the submission_name -> /reports/<id>.html link wrapping from the rendered table; the cell stays plain text. The underlying `report_url` column (relative URL to the proxy) is still computed for the modern-pipeline gate but no longer surfaced as a clickable cell. - Adds a `gr.HTML` viewer below the detail panel. Row-click triggers `_format_detail_and_report`, which returns (detail_markdown, report_ifram…0e3b21f -
app+leaderboard: link submission_name to a Space-side report proxy
Michael RabinovichPolish follow-up #2 on C6. Three user-visible changes plus the plumbing required to make them work. 1. Drop the `report` and `zip` columns from the rendered table. `submission_name` becomes the report's deep link (typical HF leaderboard pattern: the system / model name doubles as the link target; cf. Open LLM Leaderboard, DABstep, BigCodeBench). Modern completed rows get `[<name>](/reports/<sid>.html)`; pending, failed, and legacy seed rows fall back to plain text so there's no broken click. The hidden `submission_blob_url` and the computed `report_url` ride along in the DataFrame and surface in the row-click detail panel only. 2. Rename the agent-URL column from `agent` to `model details (optional)`. Missing cells render as italic `_None_` so the optionality is explicit instead of a blank. "Agent" was confusing; the schema lets `agent_url` be code OR paper OR a blog post and "agent" reads narrow. 3. Add a Space-side proxy for the per-submission H…77edebf -
app+leaderboard: detail panel polish (rename link columns, fix broken report links)
Michael RabinovichPolish follow-up on C6, three fixes after the first detail-panel commit landed: 1. Rename display columns from snake_case schema names to clean short headers. `agent_url` -> `agent`, `submission_blob_url` -> `zip`, `report_url` -> `report`. The rename happens at projection time via `df.rename`, so the schema in results.jsonl is unchanged. Headers now read as plain English ("agent", "zip", "report") rather than a python dict. 2. The `agent` column shows the URL itself (scheme stripped, truncated past 40 chars) as link text instead of a fixed "code" label. The schema says agent_url can be code OR paper; the shortened URL is the only honest hint about what's behind the click. Missing values render as italic `_None_` rather than a blank cell, so the optionality is explicit. 3. The `report` link is now gated on the row's `submission_sha256` being non-null in addition to status == "completed". Legacy seed rows pre-date the modern submit pipeline (which …1a8f331 -
app: row-click detail panel below the leaderboard tables
Michael RabinovichBundle 1+2 C6. Single shared `gr.Markdown` panel below both Leaderboard widgets; either table's `.select(...)` event populates it with the clicked row's notes, agent_url, submitted_at, submission zip + report links (always), and failure_reason (failed rows only). Spec'd shape: one panel, two select sources. leaderboard.py: - LEADERBOARD_COLS + VALIDATED_LEADERBOARD_COLS each grow by three hidden detail-only fields (submission_id, notes, failure_reason). They ride in the underlying DataFrame so the select handler can read them out directly without a parallel state cache. - New LEADERBOARD_HIDE_COLUMNS constant lists the columns hidden from view; both Leaderboard widgets in app.py consume it. - `_project_and_format` now fills missing columns with None (legacy rows can lack optional fields like notes; the panel needs the column-set to be stable regardless of source-row shape). app.py: - pandas imported (for the helper's pd.isna check). - New `_format_detail(df, evt)` helper …3112173 -
leaderboard: markdown link columns for agent_url, submission, report
Michael RabinovichBundle 1+2 C5. Three new columns on both tables (validated + unvalidated) carry clickable links rendered via gradio_leaderboard's `datatype="markdown"`: - agent_url: link to the agent's source / paper if the row's meta.json provided one (label "code"; empty cell when null). - submission_blob_url: link to the uploaded zip on the submissions dataset (label "zip"; populated on every non-failed row, the same URL the submit handler computed at upload time). - report_url: built at read time from `submission_id` plus HF_SUBMISSIONS_REPO; points at `reports/<id>.html` under /resolve/main/ so the browser renders the report directly. Only emitted for `status == "completed"` rows. leaderboard.py: - LEADERBOARD_COLS + VALIDATED_LEADERBOARD_COLS grow by three. - New LEADERBOARD_DATATYPES + VALIDATED_LEADERBOARD_DATATYPES constants mark the link columns as "markdown", everything else as "str". Tied to the column lists by a single helper so the two stay in lockstep. - New _agent_u…53de73a -
app: split Leaderboard into Validated + Unvalidated tables
Michael RabinovichBundle 1+2 C3. Swaps the single Leaderboard widget for two stacked ones (Validated on top, Unvalidated below) and wires the existing auto-refresh Timer + Refresh button to push fresh dataframes into both. Initial render reads via the C2 `load_leaderboard_split()`; existing rows all land in Unvalidated until a maintainer flips `validation_status` to `"validated"` on a row in the submissions dataset. - app.py: replaces `df_view` with `validated_view` + `unvalidated_view`. Single timer tick now fans out to both widgets (load split runs once per tick, both halves get pushed). Refresh button same fan-out shape. Inline comment points at the validation policy decision doc. - leaderboard.py: drops the now-unused single-DataFrame `load_leaderboard()` per the C3 cleanup half. Stale docstring references that pointed at the dead function got smoothed out (the split reader's doc is self-contained now). - tests/test_smoke.py: picks up the C3 acceptance with a new `test_two_leaderboa…046548a
-
app: move theme= from Blocks.launch() to Blocks() ctor (Gradio 5 API)
Michael Rabinovich76f0611 -
app+requirements: pin Gradio 5 + gradio_leaderboard, auto-refresh now works
Michael RabinovichStep 6 (E) chunk 5, take 2. The Gradio 6.14 gr.Dataframe component silently swallows row updates after the first render: server-side load_leaderboard fires every N seconds (logged: "Loaded 11 rows from Hub" on each tick) but the rendered DOM never picks the new rows up. Confirmed with a local row-growing fixture (load_leaderboard monkey-patched to return N+1 rows on each call); the server fired 10 times, the DOM stuck on the first call's row count. Tried in order, all failed against the local fixture: - gr.Dataframe(every=10, value=callable) - gr.Timer(10).tick(outputs=gr.Dataframe) - manual Refresh button click - @gr.render(inputs=[gr.State, gr.State], key=f"...-{tick}") - @gr.render(inputs=[hidden gr.Textbox], key=f"...-{tick}") - streaming generator on app.load (first yield delivered; every subsequent yield ignored client-side) What actually works: pin to Gradio 5 (latest 5.50.0) and use the gradio_leaderboard.Leaderboard custom component, which has its own update …4e86f82 -
app: fix leaderboard auto-refresh via @gr .render(key=...)
Michael RabinovichIn Gradio 6.14 a `gr.Dataframe` identity-checks its component id and short-circuits the diff when the new value's shape + column structure look unchanged, even when the row data differs. That swallowed updates from `every=` on the Dataframe AND from `gr.Timer().tick(outputs=df_view)` AND from a manual Refresh button click: the server-side fetch ran every tick (`Loaded 10 rows from Hub` in logs) but the browser never saw the new rows. Only a full page reload picked them up. Documented fix (post Gradio's March 2026 Dataframe overhaul): wrap the component in `@gr.render` and give the inner Dataframe a `key=` that changes on every tick. Gradio then tears down and rebuilds the component subtree in place, picking up the fresh value instead of trying to diff in place. Shape: - tick_state: gr.State holding a ms-timestamp that bumps on every refresh; included in the Dataframe `key=`. - table_state: gr.State holding the pandas DataFrame. - `_refresh_table()` runs once per tick, retur…f2f35be -
app: drive auto-refresh via gr.Timer().tick instead of every= on Dataframe
Michael Rabinovichgr.Dataframe(every=N) was firing load_leaderboard on the server every 10s (verified via runtime logs), but the component's client-side re-render after the initial value silently dropped rows added between ticks. Same shape as gradio-app/gradio#8160 ("when the data already exists ... it cannot be updated and the page needs to be refreshed"). Switch to gr.Timer(10).tick(fn=load_leaderboard, outputs=df_view). Identical code path to the manual Refresh button: each tick is an event fired through the standard component-update channel, which re-renders reliably. The Refresh button stays as the "I want it now" override.4ee70ef -
app: auto-refresh leaderboard + bootstrap logging
Michael RabinovichStep 6 (E) chunk 5 plus a diagnostic. Auto-refresh: - gr.Dataframe now takes `value=load_leaderboard` (callable, not its call result) plus `every=10`. Gradio re-invokes the function every 10 seconds and re-renders the dataframe, so pending rows visibly flip to completed / failed and freshly-submitted rows appear within 10s without any user action. Manual Refresh button stays as an "I want it now" override. Logging bootstrap: - logging.basicConfig(level=INFO, ...) at app startup so the `logger.info` / `logger.warning` / `logger.exception` calls already sprinkled through leaderboard.py and submit.py reach the Space's runtime logs. Without this they go nowhere (Python's default no-handler behaviour), which is why every refresh / worker pathology so far has been silent. Combined push: the auto-refresh might not surface new rows if the underlying load_leaderboard returns stale data on the Space for any reason (CDN, auth, frontend re-render bug)…b2f3ce6 -
app: drop history-narrative prose from Submit tab
Michael RabinovichThe Submit tab carried a sentence explaining why there's no form to fill in ("All five fields come from meta.json inside the zip, no separate form to fill in."). To a launch visitor that explains an absence that was never there; the right thing is to describe the contract as it stands, not to narrate a removal. Cut the sentence; keep the one useful half as a "Consent." line documenting that agree_to_publish: true in meta.json is the consent gate.a58058c -
submit: drop the agree checkbox, meta.json is the only consent gate
Michael RabinovichThe Submit tab had both an "I agree" checkbox and required meta.json to carry "agree_to_publish": true. Two gates expressing the same intent, neither covering a case the other didn't, and the UI half doesn't extend to programmatic submitters (gradio_client, future REST API). Drop the UI checkbox. meta.json["agree_to_publish"] stays the single consent gate, uniform across browser + programmatic submission. handle_submit signature shrinks to (zip_file). Submit form is now: zip upload + Submit button.9e84592 -
submit: drop dead form fields, meta.json is the only source
Michael RabinovichBefore: the Submit tab carried four text inputs (submission name, submitter name, agent URL, notes) alongside the zip upload, and handle_submit took them as args but ignored everything except the agree checkbox. They survived chunk 2's move from "form-driven stub" to "meta.json contract" only to keep the gr.click signature stable across one commit; chunk 3 didn't touch them either. After: Submit tab is just zip upload + agree checkbox + submit button. handle_submit signature is (zip_file, agree). All five submission fields come from meta.json inside the zip, full stop. Cuts 30 lines of UI + 4 unused handler params; removes a real user-confusion source (filling the form did nothing).f571140 -
submit: move handle_submit into its own module + add validation pipeline
Michael RabinovichStep 6 (E) chunk 2 of the async submit work. Lifts the UI-only stub out of app.py into submit.py and grows it into the real cheap-sync validation pipeline. Still no Hub writes and no eval kick-off; the handler returns "would queue here" placeholder text. Async write + worker thread land in later chunks. Validation gates, in order: 1. Form: zip attached, agree-checkbox ticked. 2. Zip safety: valid zip, no absolute / parent-traversing entry names, no symlinks (Python 3.12's extractall normalises unsafe paths silently; we reject so submitters see a clear error). 3. meta.json schema: required keys, types, agree_to_publish is literal true, notes within the 500-char cap. meta.json is the source of truth for submitter / submission identity; the other form fields are advisory for now and kept only so the gr.click signature stays stable. 4. Fixture set: top-level dirs in the zip equal the fixture set reported by cadgenbench.common.paths.data_inputs_di…0501689 -
refactor: split leaderboard read path into its own module
Michael RabinovichMirrors the shape adyen/DABstep uses on their reference Space (UI assembly in app.py, business logic in a sibling module). app.py drops from 224 lines to 158 by extracting the env-driven repo constants, the LOCAL_RESULTS_PATH + LEADERBOARD_COLS schema, the Hub/local row loaders, and load_leaderboard() into leaderboard.py. The submit handler stays in app.py for now; it'll move to a third module (submit.py) when Step 6 (E) lands the real validation + async eval. No behaviour change: load_leaderboard is imported back into app.py and wired into the same gr.Dataframe / refresh button pair as before. Smoke-tested locally (cadgenbench-space env): app module imports cleanly, load_leaderboard() returns a dataframe with the expected columns.b5ad973 -
docs: rewrite Space README + About tab as declarative current state
Michael Rabinovich- README.md: drop the (private; was michaelr27/...) history aside, drop the broken pointer to legacy/ (the directory was emptied during the NIST scrub), tighten the status sentence. - app.py ABOUT_MD: rewrite as a clean four-bullet "what this is" rather than a "status: active development" disclaimer block. Drops the broken pointer to space-setup/ (those notes are local-only and not in any public repo). Datasets bullet now describes both repos (inputs at HF_DATA_REPO, results at HF_SUBMISSIONS_REPO) with proper Hub links rather than just the repo names.628bc9e -
docs: refresh stale status strings after org transfer
Michael Rabinovich +1: CursorSpace + submissions dataset have moved from michaelr27/... to the HuggingAI4Engineering org (Phase C of space-setup/migration.md). README body and app.py ABOUT_MD now reference the current Space slug (HuggingAI4Engineering/cadgenbench-leaderboard) instead of the pre-transfer michaelr27/AI4Engineering, and drop the stale 'science/...' migration target that was never used. Co-authored-by: Cursor <cursoragent@cursor.com>7dc3bc3
-
ui: render validity_rate as percentage in the leaderboard table
Michael Rabinovich +1: CursorConvert the 0-1 float into 'NN%' (or 'NN.N%' for non-whole values) just before passing the table to gr.Dataframe, so the column reads as '100%' instead of '1' / '1.0'. Underlying row schema is unchanged - validity_rate is still stored as a float in [0, 1] in results.jsonl. Side-effect: this column now sorts alphabetically rather than numerically because gr.Dataframe sees strings. Acceptable for v1 - the primary sort is aggregate_score and validity_rate hovers around 100% for serious submissions anyway. If finer sorting matters later, switch to a numeric column with a Gradio formatter. Co-authored-by: Cursor <cursoragent@cursor.com>00d6e0c -
schema: rename model -> submission_name; seed with 3 baseline rows
Michael Rabinovich +1: CursorMirrors the cadgenbench-submissions schema change. LEADERBOARD_COLS exposes submission_name as the lead identity column instead of model; the handle_submit signature, the Submit-tab form Textbox, and the meta.json example in the markdown all match. Submit-tab markdown gets a short "Submission name" explainer making explicit that the benchmark is system-agnostic - a submission may use no LLM, one, or several. If the submitter wants to disclose their stack they put it in submission_name or notes. Local results.jsonl mirrors the 3 baseline seed rows from cadgenbench-submissions so the offline fallback shows real data, not the old fake demos. Co-authored-by: Cursor <cursoragent@cursor.com>952dbca -
step 5: leaderboard reads from submissions dataset on hub
Michael Rabinovich +1: CursorReplace the local-only JSONL read with hf_hub_download(force_download=True) against michaelr27/cadgenbench-submissions. force_download avoids the stale-snapshot trap that datasets.load_dataset would create per process. Falls back to AI4Engineering/results.jsonl on any Hub error (broad except) so the table stays up during a transient outage or a mid-migration rename. HF_TOKEN secret already set on the Space via HfApi.add_space_secret to read the private dataset. Secret becomes write-only at launch when the dataset goes public; rotation TODO documented in space-setup/migration.md. Co-authored-by: Cursor <cursoragent@cursor.com>afed315 -
step 3: gradio leaderboard scaffold
Michael Rabinovich +1: CursorReplace the static placeholder index with a Gradio app: Leaderboard tab backed by a hand-crafted results.jsonl, Submit tab (UI-only stub, real eval wiring lands in step 6), About tab. Org slugs parameterised via HF_ORG / HF_SUBMISSIONS_REPO / HF_DATA_REPO env vars so the future move to the science org is a secret change, not a code change. Files: - app.py: gradio 6.14 app, three tabs - requirements.txt: gradio, pandas, huggingface_hub, datasets - results.jsonl: two dev seed rows - README.md: sdk: static -> sdk: gradio - legacy/: archive of the prototype nist comparison HTMLs - .gitignore: __pycache__, .venv, .gradio Schema: validity_rate (float [0,1]) instead of valid (bool); notes kept in the row but hidden from the visible table. All copy uses "AI-driven" rather than "LLM-driven". Co-authored-by: Cursor <cursoragent@cursor.com>c040324