Commit History
-
Dockerfile: bump CADGENBENCH_SHA to 1747493 (Hub-only fixtures)
Michael Rabinovichcadgenbench main now resolves fixtures exclusively from the Hub dataset repos: the wheel-bundled _data/ and the in-repo data/ folder are both gone (commit 1747493). The Space's Settings variables CADGENBENCH_DATA_REPO and CADGENBENCH_DATA_GT_REPO are already set (chunk 5), and HF_TOKEN is the existing Space secret, so this rebuild verifies the end-to-end Hub-only path inside the production runtime.d8207be -
Dockerfile: bump CADGENBENCH_SHA to 36ac6b4 (Hub-aware paths.py)
Michael Rabinovichcadgenbench commit 36ac6b4 adds the Hub-dataset resolution branch to common/paths.py (snapshot_download from $CADGENBENCH_DATA_REPO / $CADGENBENCH_DATA_GT_REPO when set, falling back to the existing local/bundled chain otherwise). The Space's Settings variables CADGENBENCH_DATA_REPO and CADGENBENCH_DATA_GT_REPO are set in lockstep with this bump (HuggingAI4Engineering/cadgenbench-data and HuggingAI4Engineering/cadgenbench-data-gt respectively), so the runtime now resolves fixtures from the Hub dataset repos instead of the wheel-bundled fallback. GT dataset is private; the existing HF_TOKEN Space secret provides read access (already set during Step 5 for the submissions dataset). No new secret required.aa1e886 -
scrub for the cadgenbench Public flip: drop NIST + remove build secret
Michael Rabinovich +1: CursorThree changes bundled together so the next Space rebuild lands in the post-NIST, post-secret end state in one go: - legacy/nist_*_comparison_3d.html (9 MB each) deleted; these were old NIST-stub comparison artifacts already labelled legacy, kept around for visual reference only. Regenerate from the real benchmark dataset post-launch if useful. - results.jsonl emptied. The three baseline seed rows it held were scheduled for pre-launch deletion anyway (per space-setup/migration.md Phase D), and the same rows were just rewritten on the Hub side of cadgenbench-submissions to drop the NIST per-fixture entries. The Space's leaderboard read path uses results.jsonl only as a Hub-fetch fallback; an empty file just produces an empty table on fallback, which is the right behaviour for a development Space. - Dockerfile: dropped the `RUN --mount=type=secret,id=GH_PAT` line and switched the cadgenbench install URL from the authenticated `git+https://michaelr27:$(cat /run/secre…a80542f -
Dockerfile: use actual GH username in classic-PAT URL, not x-access-token
Michael Rabinovich +1: CursorPrevious push tried 'https://x-access-token:TOKEN@github.com/...' (GitHub-Actions placeholder convention). GitHub rejected it: 'Invalid username or token. Password authentication is not supported for Git operations.' Classic PATs over HTTPS Basic Auth need a real GitHub username as the user half of the URL, even though the username is ignored once GitHub validates the PAT as the credential. Switching to 'https://michaelr27:TOKEN@github.com/...' which is the form GitHub's docs themselves recommend. Hardcoding 'michaelr27' is acceptable temporary debt: this whole URL goes away when we either swap GH_PAT for the read-only deploy key (URL becomes git+ssh://) or when huggingface/cadgenbench flips Public (URL drops auth entirely). Co-authored-by: Cursor <cursoragent@cursor.com>12c3a54 -
Dockerfile: use x-access-token:TOKEN URL form for classic PAT
Michael Rabinovich +1: CursorPrevious URL form 'https://$TOKEN@github.com/...' relied on GitHub's special-case where fine-grained PATs are accepted as a single-value username. Classic PATs don't get that special-casing: git parses 'TOKEN@github.com' as username=TOKEN with no password and then prompts for one, which fails in the non-interactive Docker build with 'No such device or address'. Switching to 'https://x-access-token:$TOKEN@github.com/...' (GitHub Actions convention) is the standard explicit-Basic-Auth form and works uniformly for classic + fine-grained PATs, so the URL stays the same when we eventually swap GH_PAT for a deploy key (the URL changes to ssh+git anyway in that case). Co-authored-by: Cursor <cursoragent@cursor.com>818fd28 -
Dockerfile: point cadgenbench install at huggingface/cadgenbench
Michael Rabinovich +1: CursorSwitch source from MichaelRabinovich/LeForge to huggingface/cadgenbench now that the latter has the full history (181 commits, see space-setup/migration.md Phase C). The pinned commit (d7e0468) exists on both remotes so this is URL-only; cadgenbench version bumps are a separate concern handled by ARG CADGENBENCH_SHA. The GH_PAT secret on the Space is being rotated in lockstep from a fine-grained PAT scoped to LeForge to a classic PAT with 'repo' scope. Classic is broader than ideal (gives the PAT read on any private repo the issuing user can see, not just huggingface/cadgenbench) but it's the only self-serve option: the huggingface GH org has fine-grained PATs disabled, and the cadgenbench space-setup user isn't an admin on the org repo so can't add a per-repo deploy key either. Tracked as follow-up debt to swap for a deploy key once admin permissions land (asked Leandro). Co-authored-by: Cursor <cursoragent@cursor.com>aa263ac -
docs: refresh stale status strings after org transfer
Michael Rabinovich +1: CursorSpace + submissions dataset have moved from michaelr27/... to the HuggingAI4Engineering org (Phase C of space-setup/migration.md). README body and app.py ABOUT_MD now reference the current Space slug (HuggingAI4Engineering/cadgenbench-leaderboard) instead of the pre-transfer michaelr27/AI4Engineering, and drop the stale 'science/...' migration target that was never used. Co-authored-by: Cursor <cursoragent@cursor.com>7dc3bc3
-
docs: refresh stale science/ migration target in space README
Michael Rabinovich +1: CursorLeandro decided against the existing science/ org; the migration target is now a new dedicated CAD/Engineering org (name TBD, picked by user, candidates include HuggingFaceCAD / HuggingEngineering). Mirrors the <hf-org> placeholder convention used throughout space-setup/. Co-authored-by: Cursor <cursoragent@cursor.com>93ddafd -
step 6 (A2): switch space to sdk:docker, install cadgenbench from private LeForge via GH_PAT build secret
Michael Rabinovich +1: CursorThe HF Spaces Gradio SDK can't substitute Space secrets into requirements.txt at build time (only at runtime), and embedding the literal token in requirements.txt is a no-go (would land in git history of the Space). Switching to sdk: docker collapses two open Step 6 sub-decisions into one small change: (A2) private GitHub dep install -- handled by `RUN --mount=type=secret` in the Dockerfile; the PAT (Space secret GH_PAT) is mounted at /run/secrets/GH_PAT for that one RUN and never lands in any layer. (D) Playwright/Chromium for STEP -> PNG rendering -- handled by `playwright install --with-deps chromium` in the same Dockerfile. Layer ordering optimises rebuild speed: OS deps + requirements.txt deps + Chromium come early (stable), cadgenbench from git comes mid (bumps on each commit), app.py + results.jsonl come last (changes most). Front-matter: sdk: gradio -> sdk: docker, drop sdk_version + app_file (Docker SDK uses Dockerfile CMD), add app_port: 7860. .…900c7c1 -
ui: render validity_rate as percentage in the leaderboard table
Michael Rabinovich +1: CursorConvert the 0-1 float into 'NN%' (or 'NN.N%' for non-whole values) just before passing the table to gr.Dataframe, so the column reads as '100%' instead of '1' / '1.0'. Underlying row schema is unchanged - validity_rate is still stored as a float in [0, 1] in results.jsonl. Side-effect: this column now sorts alphabetically rather than numerically because gr.Dataframe sees strings. Acceptable for v1 - the primary sort is aggregate_score and validity_rate hovers around 100% for serious submissions anyway. If finer sorting matters later, switch to a numeric column with a Gradio formatter. Co-authored-by: Cursor <cursoragent@cursor.com>00d6e0c -
schema: rename model -> submission_name; seed with 3 baseline rows
Michael Rabinovich +1: CursorMirrors the cadgenbench-submissions schema change. LEADERBOARD_COLS exposes submission_name as the lead identity column instead of model; the handle_submit signature, the Submit-tab form Textbox, and the meta.json example in the markdown all match. Submit-tab markdown gets a short "Submission name" explainer making explicit that the benchmark is system-agnostic - a submission may use no LLM, one, or several. If the submitter wants to disclose their stack they put it in submission_name or notes. Local results.jsonl mirrors the 3 baseline seed rows from cadgenbench-submissions so the offline fallback shows real data, not the old fake demos. Co-authored-by: Cursor <cursoragent@cursor.com>952dbca -
step 5: leaderboard reads from submissions dataset on hub
Michael Rabinovich +1: CursorReplace the local-only JSONL read with hf_hub_download(force_download=True) against michaelr27/cadgenbench-submissions. force_download avoids the stale-snapshot trap that datasets.load_dataset would create per process. Falls back to AI4Engineering/results.jsonl on any Hub error (broad except) so the table stays up during a transient outage or a mid-migration rename. HF_TOKEN secret already set on the Space via HfApi.add_space_secret to read the private dataset. Secret becomes write-only at launch when the dataset goes public; rotation TODO documented in space-setup/migration.md. Co-authored-by: Cursor <cursoragent@cursor.com>afed315 -
step 3: gradio leaderboard scaffold
Michael Rabinovich +1: CursorReplace the static placeholder index with a Gradio app: Leaderboard tab backed by a hand-crafted results.jsonl, Submit tab (UI-only stub, real eval wiring lands in step 6), About tab. Org slugs parameterised via HF_ORG / HF_SUBMISSIONS_REPO / HF_DATA_REPO env vars so the future move to the science org is a secret change, not a code change. Files: - app.py: gradio 6.14 app, three tabs - requirements.txt: gradio, pandas, huggingface_hub, datasets - results.jsonl: two dev seed rows - README.md: sdk: static -> sdk: gradio - legacy/: archive of the prototype nist comparison HTMLs - .gitignore: __pycache__, .venv, .gradio Schema: validity_rate (float [0,1]) instead of valid (bool); notes kept in the row but hidden from the visible table. All copy uses "AI-driven" rather than "LLM-driven". Co-authored-by: Cursor <cursoragent@cursor.com>c040324
-
Update index.html
Verified signature64a522a
-
changed index
Michael Rabinovich79711e2 -
Added comparison files
Michael Rabinovich69c9226
-
initial commit
Verified signature1222e5c