Commit History
-
app: fix leaderboard auto-refresh via @gr .render(key=...)
Michael RabinovichIn Gradio 6.14 a `gr.Dataframe` identity-checks its component id and short-circuits the diff when the new value's shape + column structure look unchanged, even when the row data differs. That swallowed updates from `every=` on the Dataframe AND from `gr.Timer().tick(outputs=df_view)` AND from a manual Refresh button click: the server-side fetch ran every tick (`Loaded 10 rows from Hub` in logs) but the browser never saw the new rows. Only a full page reload picked them up. Documented fix (post Gradio's March 2026 Dataframe overhaul): wrap the component in `@gr.render` and give the inner Dataframe a `key=` that changes on every tick. Gradio then tears down and rebuilds the component subtree in place, picking up the fresh value instead of trying to diff in place. Shape: - tick_state: gr.State holding a ms-timestamp that bumps on every refresh; included in the Dataframe `key=`. - table_state: gr.State holding the pandas DataFrame. - `_refresh_table()` runs once per tick, retur…f2f35be -
app: drive auto-refresh via gr.Timer().tick instead of every= on Dataframe
Michael Rabinovichgr.Dataframe(every=N) was firing load_leaderboard on the server every 10s (verified via runtime logs), but the component's client-side re-render after the initial value silently dropped rows added between ticks. Same shape as gradio-app/gradio#8160 ("when the data already exists ... it cannot be updated and the page needs to be refreshed"). Switch to gr.Timer(10).tick(fn=load_leaderboard, outputs=df_view). Identical code path to the manual Refresh button: each tick is an event fired through the standard component-update channel, which re-renders reliably. The Refresh button stays as the "I want it now" override.4ee70ef -
app: auto-refresh leaderboard + bootstrap logging
Michael RabinovichStep 6 (E) chunk 5 plus a diagnostic. Auto-refresh: - gr.Dataframe now takes `value=load_leaderboard` (callable, not its call result) plus `every=10`. Gradio re-invokes the function every 10 seconds and re-renders the dataframe, so pending rows visibly flip to completed / failed and freshly-submitted rows appear within 10s without any user action. Manual Refresh button stays as an "I want it now" override. Logging bootstrap: - logging.basicConfig(level=INFO, ...) at app startup so the `logger.info` / `logger.warning` / `logger.exception` calls already sprinkled through leaderboard.py and submit.py reach the Space's runtime logs. Without this they go nowhere (Python's default no-handler behaviour), which is why every refresh / worker pathology so far has been silent. Combined push: the auto-refresh might not surface new rows if the underlying load_leaderboard returns stale data on the Space for any reason (CDN, auth, frontend re-render bug)…b2f3ce6 -
Dockerfile: bump CADGENBENCH_SHA to 9bd02fb (lazy baseline import)
Michael RabinovichPicks up the cadgenbench fix that makes `cadgenbench --help` and `cadgenbench evaluate` work in the eval-only install (no `[baseline]` extra). Before this bump, the chunk-4 worker's `python -m cadgenbench.cli evaluate <run_dir>` subprocess died during argparse setup with `ModuleNotFoundError: No module named 'dotenv'`, flipping every submission to status=failed before any work happened. Bumping CADGENBENCH_SHA invalidates the cadgenbench layer in the Dockerfile cache (Chromium layer + earlier layers stay cached), so the rebuild is ~2-3 min not the ~5 min cold cycle.67fd44c -
leaderboard: fetch results.jsonl via direct cache-busted HTTPS
Michael Rabinovichhf_hub_download (even with force_download=True) hands back stale bytes for several minutes after a commit: the local disk cache, revision pinning, and the Hub's CDN-fronted resolve endpoint all add their own staleness windows. End effect: pending and failed rows landed by the submit worker were on the Hub but invisible on the leaderboard tab until you waited it out, defeating the whole point of Refresh. Switch _load_rows_from_hub to a plain requests.get against the dataset's resolve/main URL with Cache-Control: no-cache and a millisecond cache-bust query param. Auth via huggingface_hub get_token() so the same code reads the still-private submissions dataset from the Space (HF_TOKEN secret) and from local dev (~/.cache/huggingface/token) without forking the code path. Adds requests to requirements.txt (was a transitive dep via huggingface_hub; making it explicit so a future upstream change doesn't silently drop it).9fcb684 -
submit: background-thread eval, flip row pending -> completed / failed
Michael RabinovichStep 6 (E) chunk 4. On a successful submit handle_submit now spawns a daemon thread that runs `cadgenbench evaluate` then `cadgenbench report single` over the unpacked submission, uploads reports/<id>.{html,json} to the submissions dataset, reads run_summary.json, and flips the row pending -> completed under the existing _HUB_LOCK. Any worker-side exception flips the row to failed with a short failure_reason (<=200 chars; full traceback goes to Space logs). Tempdir cleanup always runs in finally. Pipeline per submission, in the worker: 1. cadgenbench evaluate <run_dir> --workers 8 (subprocess via `python -m cadgenbench.cli`; the eval CLI already fans out across fixtures with ProcessPoolExecutor, self-throttles to n_fixtures when smaller). 2. cadgenbench report single <run_dir> -o <tmp>/<id>.html 3. Build reports JSON: run_summary.json + each fixture's result.json bundled together (mirrors README's description of reports/<id>.json as the "machine-readable…855bb8a -
leaderboard: render pending / failed score cells with a status tag
Michael RabinovichBlank score cells on a pending row left the only "this is in progress" cue in the status column, which read as broken to a user expecting at least *some* signal in the score columns. _fmt_pct and _fmt_score are now status-aware: pending rows render the tag "evaluating..." (hourglass prefix) in both score columns, failed rows render "failed" (cross prefix). Completed rows render the score number as before. Matches schema.md's "table renders the three states distinctly (spinner / score / error tag)" description and lines up with the visual pattern other HF leaderboards use while a row is queued.5aee3e5 -
leaderboard: handle null score fields on pending / failed rows
Michael RabinovichThe pending row landed by chunk 3 carries null validity_rate + aggregate_score (correct per the schema). pandas coerces JSON null to NaN on column construction, and _fmt_pct's `if x is None` gate didn't catch NaN; int(NaN) then raised ValueError inside .map(), bringing down load_leaderboard and rendering the leaderboard tab as a generic "Error". Fix: - _fmt_pct: gate on pd.isna(x) (catches both None and NaN), return "" for null. - _fmt_score: matching formatter for aggregate_score (was previously rendered as the raw float; with nulls in the mix it'd show as "NaN" cells). Returns "" for null, "%.4f" otherwise. Pending and failed rows now render with blank cells in both score columns; the row-level status column carries the meaning.00d091d -
app: drop history-narrative prose from Submit tab
Michael RabinovichThe Submit tab carried a sentence explaining why there's no form to fill in ("All five fields come from meta.json inside the zip, no separate form to fill in."). To a launch visitor that explains an absence that was never there; the right thing is to describe the contract as it stands, not to narrate a removal. Cut the sentence; keep the one useful half as a "Consent." line documenting that agree_to_publish: true in meta.json is the consent gate.a58058c -
submit: drop the agree checkbox, meta.json is the only consent gate
Michael RabinovichThe Submit tab had both an "I agree" checkbox and required meta.json to carry "agree_to_publish": true. Two gates expressing the same intent, neither covering a case the other didn't, and the UI half doesn't extend to programmatic submitters (gradio_client, future REST API). Drop the UI checkbox. meta.json["agree_to_publish"] stays the single consent gate, uniform across browser + programmatic submission. handle_submit signature shrinks to (zip_file). Submit form is now: zip upload + Submit button.9e84592 -
submit: drop dead form fields, meta.json is the only source
Michael RabinovichBefore: the Submit tab carried four text inputs (submission name, submitter name, agent URL, notes) alongside the zip upload, and handle_submit took them as args but ignored everything except the agree checkbox. They survived chunk 2's move from "form-driven stub" to "meta.json contract" only to keep the gr.click signature stable across one commit; chunk 3 didn't touch them either. After: Submit tab is just zip upload + agree checkbox + submit button. handle_submit signature is (zip_file, agree). All five submission fields come from meta.json inside the zip, full stop. Cuts 30 lines of UI + 4 unused handler params; removes a real user-confusion source (filling the form did nothing).f571140 -
submit: write pending row + upload zip on successful validation
Michael RabinovichStep 6 (E) chunk 3. Cheap-sync validation (chunk 2) now also commits to the Hub: on success the handler uploads submissions/<id>.zip and appends a status=pending row to results.jsonl under a process-wide threading.Lock. Eval still stubbed, the row stays pending forever until chunk 4 lands the background worker. Hub-write ordering (after validation passes): 1. Upload submissions/<id>.zip. Unique path per submission, no lock needed (HfApi handles its own commit serialization). 2. Build the pending row: metadata fields + status="pending" + failure_reason=null + submission_blob_url + null score fields, matching the "Pending" regime in cadgenbench-submissions/schema.md. 3. Acquire _HUB_LOCK, download current results.jsonl (treat EntryNotFoundError as empty), append one JSON line, re-upload. Lock held only for the RMW cycle (~1-2s), not for eval time; concurrent submitters serialise here, not in the worker. If step 1 fails the submitter sees a clean "Hub…1748c76 -
submit: move handle_submit into its own module + add validation pipeline
Michael RabinovichStep 6 (E) chunk 2 of the async submit work. Lifts the UI-only stub out of app.py into submit.py and grows it into the real cheap-sync validation pipeline. Still no Hub writes and no eval kick-off; the handler returns "would queue here" placeholder text. Async write + worker thread land in later chunks. Validation gates, in order: 1. Form: zip attached, agree-checkbox ticked. 2. Zip safety: valid zip, no absolute / parent-traversing entry names, no symlinks (Python 3.12's extractall normalises unsafe paths silently; we reject so submitters see a clear error). 3. meta.json schema: required keys, types, agree_to_publish is literal true, notes within the 500-char cap. meta.json is the source of truth for submitter / submission identity; the other form fields are advisory for now and kept only so the gr.click signature stays stable. 4. Fixture set: top-level dirs in the zip equal the fixture set reported by cadgenbench.common.paths.data_inputs_di…0501689 -
Dockerfile: bump CADGENBENCH_SHA to ed31274 (parallel evaluate)
Michael RabinovichPicks up the eval orchestrator's switch to ProcessPoolExecutor across fixtures (cadgenbench commit ed31274). On the Space's cpu-upgrade tier (8 vCPU) this should give a near-linear speedup on real-GT-scale runs; locally on the 5-stub set we saw ~2.4x going from --workers 1 to --workers 4 (1m13s -> 31s). The default worker count is min(8, os.cpu_count()) so the Space gets the speedup without setting any env var. Verified locally: per-fixture cad_score outputs are byte-identical between sequential and parallel runs (eval is per-fixture independent + deterministic).91244c6 -
Dockerfile: copy all Python modules, not just app.py
Michael RabinovichPrevious COPY hardcoded app.py only, which broke the moment app.py imported sibling leaderboard.py: the new module never made it into the image and the runtime hit ModuleNotFoundError. Switching to *.py picks up every Python file in AI4Engineering/ (currently app.py + leaderboard.py; submit.py lands next), so future module additions don't need a Dockerfile bump.f26021f -
refactor: split leaderboard read path into its own module
Michael RabinovichMirrors the shape adyen/DABstep uses on their reference Space (UI assembly in app.py, business logic in a sibling module). app.py drops from 224 lines to 158 by extracting the env-driven repo constants, the LOCAL_RESULTS_PATH + LEADERBOARD_COLS schema, the Hub/local row loaders, and load_leaderboard() into leaderboard.py. The submit handler stays in app.py for now; it'll move to a third module (submit.py) when Step 6 (E) lands the real validation + async eval. No behaviour change: load_leaderboard is imported back into app.py and wired into the same gr.Dataframe / refresh button pair as before. Smoke-tested locally (cadgenbench-space env): app module imports cleanly, load_leaderboard() returns a dataframe with the expected columns.b5ad973 -
docs: rewrite Space README + About tab as declarative current state
Michael Rabinovich- README.md: drop the (private; was michaelr27/...) history aside, drop the broken pointer to legacy/ (the directory was emptied during the NIST scrub), tighten the status sentence. - app.py ABOUT_MD: rewrite as a clean four-bullet "what this is" rather than a "status: active development" disclaimer block. Drops the broken pointer to space-setup/ (those notes are local-only and not in any public repo). Datasets bullet now describes both repos (inputs at HF_DATA_REPO, results at HF_SUBMISSIONS_REPO) with proper Hub links rather than just the repo names.628bc9e -
Dockerfile: bump CADGENBENCH_SHA to 1747493 (Hub-only fixtures)
Michael Rabinovichcadgenbench main now resolves fixtures exclusively from the Hub dataset repos: the wheel-bundled _data/ and the in-repo data/ folder are both gone (commit 1747493). The Space's Settings variables CADGENBENCH_DATA_REPO and CADGENBENCH_DATA_GT_REPO are already set (chunk 5), and HF_TOKEN is the existing Space secret, so this rebuild verifies the end-to-end Hub-only path inside the production runtime.d8207be -
Dockerfile: bump CADGENBENCH_SHA to 36ac6b4 (Hub-aware paths.py)
Michael Rabinovichcadgenbench commit 36ac6b4 adds the Hub-dataset resolution branch to common/paths.py (snapshot_download from $CADGENBENCH_DATA_REPO / $CADGENBENCH_DATA_GT_REPO when set, falling back to the existing local/bundled chain otherwise). The Space's Settings variables CADGENBENCH_DATA_REPO and CADGENBENCH_DATA_GT_REPO are set in lockstep with this bump (HuggingAI4Engineering/cadgenbench-data and HuggingAI4Engineering/cadgenbench-data-gt respectively), so the runtime now resolves fixtures from the Hub dataset repos instead of the wheel-bundled fallback. GT dataset is private; the existing HF_TOKEN Space secret provides read access (already set during Step 5 for the submissions dataset). No new secret required.aa1e886 -
scrub for the cadgenbench Public flip: drop NIST + remove build secret
Michael Rabinovich +1: CursorThree changes bundled together so the next Space rebuild lands in the post-NIST, post-secret end state in one go: - legacy/nist_*_comparison_3d.html (9 MB each) deleted; these were old NIST-stub comparison artifacts already labelled legacy, kept around for visual reference only. Regenerate from the real benchmark dataset post-launch if useful. - results.jsonl emptied. The three baseline seed rows it held were scheduled for pre-launch deletion anyway (per space-setup/migration.md Phase D), and the same rows were just rewritten on the Hub side of cadgenbench-submissions to drop the NIST per-fixture entries. The Space's leaderboard read path uses results.jsonl only as a Hub-fetch fallback; an empty file just produces an empty table on fallback, which is the right behaviour for a development Space. - Dockerfile: dropped the `RUN --mount=type=secret,id=GH_PAT` line and switched the cadgenbench install URL from the authenticated `git+https://michaelr27:$(cat /run/secre…a80542f -
Dockerfile: use actual GH username in classic-PAT URL, not x-access-token
Michael Rabinovich +1: CursorPrevious push tried 'https://x-access-token:TOKEN@github.com/...' (GitHub-Actions placeholder convention). GitHub rejected it: 'Invalid username or token. Password authentication is not supported for Git operations.' Classic PATs over HTTPS Basic Auth need a real GitHub username as the user half of the URL, even though the username is ignored once GitHub validates the PAT as the credential. Switching to 'https://michaelr27:TOKEN@github.com/...' which is the form GitHub's docs themselves recommend. Hardcoding 'michaelr27' is acceptable temporary debt: this whole URL goes away when we either swap GH_PAT for the read-only deploy key (URL becomes git+ssh://) or when huggingface/cadgenbench flips Public (URL drops auth entirely). Co-authored-by: Cursor <cursoragent@cursor.com>12c3a54 -
Dockerfile: use x-access-token:TOKEN URL form for classic PAT
Michael Rabinovich +1: CursorPrevious URL form 'https://$TOKEN@github.com/...' relied on GitHub's special-case where fine-grained PATs are accepted as a single-value username. Classic PATs don't get that special-casing: git parses 'TOKEN@github.com' as username=TOKEN with no password and then prompts for one, which fails in the non-interactive Docker build with 'No such device or address'. Switching to 'https://x-access-token:$TOKEN@github.com/...' (GitHub Actions convention) is the standard explicit-Basic-Auth form and works uniformly for classic + fine-grained PATs, so the URL stays the same when we eventually swap GH_PAT for a deploy key (the URL changes to ssh+git anyway in that case). Co-authored-by: Cursor <cursoragent@cursor.com>818fd28 -
Dockerfile: point cadgenbench install at huggingface/cadgenbench
Michael Rabinovich +1: CursorSwitch source from MichaelRabinovich/LeForge to huggingface/cadgenbench now that the latter has the full history (181 commits, see space-setup/migration.md Phase C). The pinned commit (d7e0468) exists on both remotes so this is URL-only; cadgenbench version bumps are a separate concern handled by ARG CADGENBENCH_SHA. The GH_PAT secret on the Space is being rotated in lockstep from a fine-grained PAT scoped to LeForge to a classic PAT with 'repo' scope. Classic is broader than ideal (gives the PAT read on any private repo the issuing user can see, not just huggingface/cadgenbench) but it's the only self-serve option: the huggingface GH org has fine-grained PATs disabled, and the cadgenbench space-setup user isn't an admin on the org repo so can't add a per-repo deploy key either. Tracked as follow-up debt to swap for a deploy key once admin permissions land (asked Leandro). Co-authored-by: Cursor <cursoragent@cursor.com>aa263ac -
docs: refresh stale status strings after org transfer
Michael Rabinovich +1: CursorSpace + submissions dataset have moved from michaelr27/... to the HuggingAI4Engineering org (Phase C of space-setup/migration.md). README body and app.py ABOUT_MD now reference the current Space slug (HuggingAI4Engineering/cadgenbench-leaderboard) instead of the pre-transfer michaelr27/AI4Engineering, and drop the stale 'science/...' migration target that was never used. Co-authored-by: Cursor <cursoragent@cursor.com>7dc3bc3
-
docs: refresh stale science/ migration target in space README
Michael Rabinovich +1: CursorLeandro decided against the existing science/ org; the migration target is now a new dedicated CAD/Engineering org (name TBD, picked by user, candidates include HuggingFaceCAD / HuggingEngineering). Mirrors the <hf-org> placeholder convention used throughout space-setup/. Co-authored-by: Cursor <cursoragent@cursor.com>93ddafd -
step 6 (A2): switch space to sdk:docker, install cadgenbench from private LeForge via GH_PAT build secret
Michael Rabinovich +1: CursorThe HF Spaces Gradio SDK can't substitute Space secrets into requirements.txt at build time (only at runtime), and embedding the literal token in requirements.txt is a no-go (would land in git history of the Space). Switching to sdk: docker collapses two open Step 6 sub-decisions into one small change: (A2) private GitHub dep install -- handled by `RUN --mount=type=secret` in the Dockerfile; the PAT (Space secret GH_PAT) is mounted at /run/secrets/GH_PAT for that one RUN and never lands in any layer. (D) Playwright/Chromium for STEP -> PNG rendering -- handled by `playwright install --with-deps chromium` in the same Dockerfile. Layer ordering optimises rebuild speed: OS deps + requirements.txt deps + Chromium come early (stable), cadgenbench from git comes mid (bumps on each commit), app.py + results.jsonl come last (changes most). Front-matter: sdk: gradio -> sdk: docker, drop sdk_version + app_file (Docker SDK uses Dockerfile CMD), add app_port: 7860. .…900c7c1 -
ui: render validity_rate as percentage in the leaderboard table
Michael Rabinovich +1: CursorConvert the 0-1 float into 'NN%' (or 'NN.N%' for non-whole values) just before passing the table to gr.Dataframe, so the column reads as '100%' instead of '1' / '1.0'. Underlying row schema is unchanged - validity_rate is still stored as a float in [0, 1] in results.jsonl. Side-effect: this column now sorts alphabetically rather than numerically because gr.Dataframe sees strings. Acceptable for v1 - the primary sort is aggregate_score and validity_rate hovers around 100% for serious submissions anyway. If finer sorting matters later, switch to a numeric column with a Gradio formatter. Co-authored-by: Cursor <cursoragent@cursor.com>00d6e0c -
schema: rename model -> submission_name; seed with 3 baseline rows
Michael Rabinovich +1: CursorMirrors the cadgenbench-submissions schema change. LEADERBOARD_COLS exposes submission_name as the lead identity column instead of model; the handle_submit signature, the Submit-tab form Textbox, and the meta.json example in the markdown all match. Submit-tab markdown gets a short "Submission name" explainer making explicit that the benchmark is system-agnostic - a submission may use no LLM, one, or several. If the submitter wants to disclose their stack they put it in submission_name or notes. Local results.jsonl mirrors the 3 baseline seed rows from cadgenbench-submissions so the offline fallback shows real data, not the old fake demos. Co-authored-by: Cursor <cursoragent@cursor.com>952dbca -
step 5: leaderboard reads from submissions dataset on hub
Michael Rabinovich +1: CursorReplace the local-only JSONL read with hf_hub_download(force_download=True) against michaelr27/cadgenbench-submissions. force_download avoids the stale-snapshot trap that datasets.load_dataset would create per process. Falls back to AI4Engineering/results.jsonl on any Hub error (broad except) so the table stays up during a transient outage or a mid-migration rename. HF_TOKEN secret already set on the Space via HfApi.add_space_secret to read the private dataset. Secret becomes write-only at launch when the dataset goes public; rotation TODO documented in space-setup/migration.md. Co-authored-by: Cursor <cursoragent@cursor.com>afed315 -
step 3: gradio leaderboard scaffold
Michael Rabinovich +1: CursorReplace the static placeholder index with a Gradio app: Leaderboard tab backed by a hand-crafted results.jsonl, Submit tab (UI-only stub, real eval wiring lands in step 6), About tab. Org slugs parameterised via HF_ORG / HF_SUBMISSIONS_REPO / HF_DATA_REPO env vars so the future move to the science org is a secret change, not a code change. Files: - app.py: gradio 6.14 app, three tabs - requirements.txt: gradio, pandas, huggingface_hub, datasets - results.jsonl: two dev seed rows - README.md: sdk: static -> sdk: gradio - legacy/: archive of the prototype nist comparison HTMLs - .gitignore: __pycache__, .venv, .gradio Schema: validity_rate (float [0,1]) instead of valid (bool); notes kept in the row but hidden from the visible table. All copy uses "AI-driven" rather than "LLM-driven". Co-authored-by: Cursor <cursoragent@cursor.com>c040324
-
Update index.html
Verified signature64a522a
-
changed index
Michael Rabinovich79711e2 -
Added comparison files
Michael Rabinovich69c9226
-
initial commit
Verified signature1222e5c