Commit History
-
app+leaderboard: link submission_name to a Space-side report proxy
Michael RabinovichPolish follow-up #2 on C6. Three user-visible changes plus the plumbing required to make them work. 1. Drop the `report` and `zip` columns from the rendered table. `submission_name` becomes the report's deep link (typical HF leaderboard pattern: the system / model name doubles as the link target; cf. Open LLM Leaderboard, DABstep, BigCodeBench). Modern completed rows get `[<name>](/reports/<sid>.html)`; pending, failed, and legacy seed rows fall back to plain text so there's no broken click. The hidden `submission_blob_url` and the computed `report_url` ride along in the DataFrame and surface in the row-click detail panel only. 2. Rename the agent-URL column from `agent` to `model details (optional)`. Missing cells render as italic `_None_` so the optionality is explicit instead of a blank. "Agent" was confusing; the schema lets `agent_url` be code OR paper OR a blog post and "agent" reads narrow. 3. Add a Space-side proxy for the per-submission H…77edebf -
app+leaderboard: detail panel polish (rename link columns, fix broken report links)
Michael RabinovichPolish follow-up on C6, three fixes after the first detail-panel commit landed: 1. Rename display columns from snake_case schema names to clean short headers. `agent_url` -> `agent`, `submission_blob_url` -> `zip`, `report_url` -> `report`. The rename happens at projection time via `df.rename`, so the schema in results.jsonl is unchanged. Headers now read as plain English ("agent", "zip", "report") rather than a python dict. 2. The `agent` column shows the URL itself (scheme stripped, truncated past 40 chars) as link text instead of a fixed "code" label. The schema says agent_url can be code OR paper; the shortened URL is the only honest hint about what's behind the click. Missing values render as italic `_None_` rather than a blank cell, so the optionality is explicit. 3. The `report` link is now gated on the row's `submission_sha256` being non-null in addition to status == "completed". Legacy seed rows pre-date the modern submit pipeline (which …1a8f331 -
app: row-click detail panel below the leaderboard tables
Michael RabinovichBundle 1+2 C6. Single shared `gr.Markdown` panel below both Leaderboard widgets; either table's `.select(...)` event populates it with the clicked row's notes, agent_url, submitted_at, submission zip + report links (always), and failure_reason (failed rows only). Spec'd shape: one panel, two select sources. leaderboard.py: - LEADERBOARD_COLS + VALIDATED_LEADERBOARD_COLS each grow by three hidden detail-only fields (submission_id, notes, failure_reason). They ride in the underlying DataFrame so the select handler can read them out directly without a parallel state cache. - New LEADERBOARD_HIDE_COLUMNS constant lists the columns hidden from view; both Leaderboard widgets in app.py consume it. - `_project_and_format` now fills missing columns with None (legacy rows can lack optional fields like notes; the panel needs the column-set to be stable regardless of source-row shape). app.py: - pandas imported (for the helper's pd.isna check). - New `_format_detail(df, evt)` helper …3112173 -
leaderboard: markdown link columns for agent_url, submission, report
Michael RabinovichBundle 1+2 C5. Three new columns on both tables (validated + unvalidated) carry clickable links rendered via gradio_leaderboard's `datatype="markdown"`: - agent_url: link to the agent's source / paper if the row's meta.json provided one (label "code"; empty cell when null). - submission_blob_url: link to the uploaded zip on the submissions dataset (label "zip"; populated on every non-failed row, the same URL the submit handler computed at upload time). - report_url: built at read time from `submission_id` plus HF_SUBMISSIONS_REPO; points at `reports/<id>.html` under /resolve/main/ so the browser renders the report directly. Only emitted for `status == "completed"` rows. leaderboard.py: - LEADERBOARD_COLS + VALIDATED_LEADERBOARD_COLS grow by three. - New LEADERBOARD_DATATYPES + VALIDATED_LEADERBOARD_DATATYPES constants mark the link columns as "markdown", everything else as "str". Tied to the column lists by a single helper so the two stay in lockstep. - New _agent_u…53de73a -
submit: default validation + hf_username fields on the pending row
Michael RabinovichBundle 1+2 C4. `_build_pending_row` adds three new keys matching the schema bump from C1, in the schema.md column order: - validation_status = "unvalidated" (every new row starts in the unvalidated tier; maintainers promote via the dataset repo after methodology review). - validation_method = None (set by the maintainer at promotion time to one of code | traces | api | manual). - hf_username = None (placeholder; C10 plumbs through the value from gradio.OAuthProfile.username once the LoginButton lands). Pre-existing fields, submitted_at stamping, the cadgenbench version + data revision tags, the score-shaped null fields, the blob URL, and the sha256 are untouched. Dedup gate still pivots on submission_sha256 exactly as before. tests/test_submit.py covers the C4 acceptance: - new-row defaults: validation_status is "unvalidated", validation_method + hf_username are None; - sha256 path passthrough: the hash argument lands on the row's submission_sha256 key intact (existing …a533dd2 -
docs: add leaderboard discovery tags to README front-matter
Michael Rabinovichtags: surfaces this Space in the Hub's leaderboard discovery filter (huggingface.co/spaces?other=leaderboard) and faceted searches by modality, submission type, and eval type. The set blends the empirical vocabulary across the top 50 leaderboard-tagged Spaces with three truthful novel descriptors: modality:3d (closest existing values are modality:text / tools, neither accurate for STEP geometry), eval:cad (paired with eval:generation so the standard "generation" chip also surfaces us), and judge:auto (eval is metric-based, no LLM/human judge).36546fc -
app: split Leaderboard into Validated + Unvalidated tables
Michael RabinovichBundle 1+2 C3. Swaps the single Leaderboard widget for two stacked ones (Validated on top, Unvalidated below) and wires the existing auto-refresh Timer + Refresh button to push fresh dataframes into both. Initial render reads via the C2 `load_leaderboard_split()`; existing rows all land in Unvalidated until a maintainer flips `validation_status` to `"validated"` on a row in the submissions dataset. - app.py: replaces `df_view` with `validated_view` + `unvalidated_view`. Single timer tick now fans out to both widgets (load split runs once per tick, both halves get pushed). Refresh button same fan-out shape. Inline comment points at the validation policy decision doc. - leaderboard.py: drops the now-unused single-DataFrame `load_leaderboard()` per the C3 cleanup half. Stale docstring references that pointed at the dead function got smoothed out (the split reader's doc is self-contained now). - tests/test_smoke.py: picks up the C3 acceptance with a new `test_two_leaderboa…046548a -
license: add Apache-2.0
Michael Rabinovich8676e14 -
leaderboard: add load_leaderboard_split() for two-tier viewer
Michael RabinovichBundle 1+2 C2. Additive new function for the C3 two-stacked-table swap; existing single-DataFrame load_leaderboard() stays put until C3 flips the caller in app.py + drops it. Keeps the live Space green between commits. - VALIDATED_LEADERBOARD_COLS: same shape as LEADERBOARD_COLS plus `validation_method` (always null on unvalidated rows, so omitted from that tier rather than rendered). - load_leaderboard_split() -> (validated_df, unvalidated_df): reads results.jsonl the same way as load_leaderboard(), splits rows on `validation_status` (defensive: anything not literally "validated" lands in unvalidated, which covers legacy + null + any future-unknown value), runs the same status-aware cell formatting per tier via the new _project_and_format helper. - _project_and_format: shared sort + projection + pending/failed cell tagging so the two tiers produce identically-shaped cells. Legacy default: rows without `validation_status` get "unvalidated" via row-level fill. No resul…0dd7215 -
docs: link companion datasets in README front-matter
Michael Rabinovichdatasets: lists the two companion dataset repos so the Hub renders them as cards on the Space page (and so the link is explicit rather than relying on auto-parse from hardcoded repo IDs in app.py / leaderboard.py). - HuggingAI4Engineering/cadgenbench-submissions: the results + per-submission report dataset the leaderboard reads from. - HuggingAI4Engineering/cadgenbench-data: the public fixture inputs. cadgenbench-data-gt is omitted on purpose (permanently private).1402833 -
docs: add hf_oauth + apache-2.0 license to README front-matter
Michael Rabinovichhf_oauth: true is what wires the LoginButton + OAuth callback the Submit tab relies on; without it the auth-required submit flow is silently broken (login click no-ops, no token reaches the handler). license: apache-2.0 matches the HF Leaderboards template (gradio-templates/leaderboard) and the rest of the cadgenbench code surface, and renders a license badge on the Space page.0bb6044 -
tests: pytest + playwright headless smoke harness
Michael RabinovichFirst Space-side test harness; sets up the floor for the Phase D bundle's per-commit unit + UI tests (split leaderboard, pending-row defaults, OAuth-required submit) without slipping a regression past a build-only check. requirements-dev.txt pulls -r requirements.txt then adds pytest + playwright. One `pip install -r requirements-dev.txt && playwright install chromium` brings a fresh venv up to speed. tests/conftest.py exposes a session-scoped `app_url` fixture: boots app.py in a subprocess on a free port, polls until the HTTP server answers, yields the URL, tears the process down on teardown. Logs land in a tmp file so a startup failure surfaces the real Gradio traceback rather than a bare RuntimeError. Boot-time stuck-pending sweep in submit.py is disabled via CADGENBENCH_DISABLE_BOOT_SWEEP=1 so the fixture is fast and offline-tolerant. tests/test_smoke.py launches headless Chromium via playwright's sync API, navigates to the fixture's URL, and asserts the Leaderboard / Submit / A…21b071a -
Dockerfile: bump CADGENBENCH_SHA to 056cfbb (invalid-candidate fast path)
Michael Rabinovichff674cc -
submit: stuck-pending sweep on boot (chunk 6)
Michael RabinovichStep 6 (E) chunk 6. A pending row whose worker died (Space restart for a deploy, OOM, crash) has no one to flip it; without this sweep it stays "pending" in the leaderboard forever (we hit this twice during chunk-3..5 churn already). On submit.py module import, spawn a daemon thread that: 1. Downloads results.jsonl (no lock; read-only). 2. Iterates rows; for each pending row whose `submitted_at` is older than 30 min, calls _flip_row_to_failed(sid, "evaluation interrupted by Space restart"). 3. Logs the list of stuck ids so a deployer can see what was swept. Threshold is 30 min: well above the real eval ceiling on cpu-upgrade (~5 min), so a genuinely-still-running submission is safe. Hub-fetch failure at boot is non-fatal (logged warning, sweep skipped; next boot retries). Per-row flip failures are caught + logged + skipped (the rest of the sweep continues). Opt-out via CADGENBENCH_DISABLE_BOOT_SWEEP=1 for test imports that don't want the Hub round-trip. Two tes…0a2a2c6 -
submit: reject byte-identical re-submits via SHA-256 dedup
Michael RabinovichBefore: clicking Submit twice (or re-uploading the same zip after a transient confusion) created two rows pointing at byte-identical content, ran the eval twice for the same input, and bloated the submissions/reports blobs. After: the cheap-sync handler computes sha256 of the uploaded zip right after validation passes, queries results.jsonl for any row with the same submission_sha256, and rejects with a pointer to the existing submission_id if a match is found. Hash also lands on the new row's submission_sha256 field (schema bump in the companion commit on the submissions dataset). Dedup-fetch failures fall back to letting the submit through rather than rejecting on a Hub blip (logged). Race window: two near-simultaneous identical submits whose dedup checks both return None before either appends will land two rows; recoverable by manual cleanup if it ever happens, not worth tightening before auth + login throttling lands at pre-launch. Drive-by: _build_pending_row's unused fixture_n…3237736
-
app: move theme= from Blocks.launch() to Blocks() ctor (Gradio 5 API)
Michael Rabinovich76f0611 -
app+requirements: pin Gradio 5 + gradio_leaderboard, auto-refresh now works
Michael RabinovichStep 6 (E) chunk 5, take 2. The Gradio 6.14 gr.Dataframe component silently swallows row updates after the first render: server-side load_leaderboard fires every N seconds (logged: "Loaded 11 rows from Hub" on each tick) but the rendered DOM never picks the new rows up. Confirmed with a local row-growing fixture (load_leaderboard monkey-patched to return N+1 rows on each call); the server fired 10 times, the DOM stuck on the first call's row count. Tried in order, all failed against the local fixture: - gr.Dataframe(every=10, value=callable) - gr.Timer(10).tick(outputs=gr.Dataframe) - manual Refresh button click - @gr.render(inputs=[gr.State, gr.State], key=f"...-{tick}") - @gr.render(inputs=[hidden gr.Textbox], key=f"...-{tick}") - streaming generator on app.load (first yield delivered; every subsequent yield ignored client-side) What actually works: pin to Gradio 5 (latest 5.50.0) and use the gradio_leaderboard.Leaderboard custom component, which has its own update …4e86f82 -
app: fix leaderboard auto-refresh via @gr .render(key=...)
Michael RabinovichIn Gradio 6.14 a `gr.Dataframe` identity-checks its component id and short-circuits the diff when the new value's shape + column structure look unchanged, even when the row data differs. That swallowed updates from `every=` on the Dataframe AND from `gr.Timer().tick(outputs=df_view)` AND from a manual Refresh button click: the server-side fetch ran every tick (`Loaded 10 rows from Hub` in logs) but the browser never saw the new rows. Only a full page reload picked them up. Documented fix (post Gradio's March 2026 Dataframe overhaul): wrap the component in `@gr.render` and give the inner Dataframe a `key=` that changes on every tick. Gradio then tears down and rebuilds the component subtree in place, picking up the fresh value instead of trying to diff in place. Shape: - tick_state: gr.State holding a ms-timestamp that bumps on every refresh; included in the Dataframe `key=`. - table_state: gr.State holding the pandas DataFrame. - `_refresh_table()` runs once per tick, retur…f2f35be -
app: drive auto-refresh via gr.Timer().tick instead of every= on Dataframe
Michael Rabinovichgr.Dataframe(every=N) was firing load_leaderboard on the server every 10s (verified via runtime logs), but the component's client-side re-render after the initial value silently dropped rows added between ticks. Same shape as gradio-app/gradio#8160 ("when the data already exists ... it cannot be updated and the page needs to be refreshed"). Switch to gr.Timer(10).tick(fn=load_leaderboard, outputs=df_view). Identical code path to the manual Refresh button: each tick is an event fired through the standard component-update channel, which re-renders reliably. The Refresh button stays as the "I want it now" override.4ee70ef -
app: auto-refresh leaderboard + bootstrap logging
Michael RabinovichStep 6 (E) chunk 5 plus a diagnostic. Auto-refresh: - gr.Dataframe now takes `value=load_leaderboard` (callable, not its call result) plus `every=10`. Gradio re-invokes the function every 10 seconds and re-renders the dataframe, so pending rows visibly flip to completed / failed and freshly-submitted rows appear within 10s without any user action. Manual Refresh button stays as an "I want it now" override. Logging bootstrap: - logging.basicConfig(level=INFO, ...) at app startup so the `logger.info` / `logger.warning` / `logger.exception` calls already sprinkled through leaderboard.py and submit.py reach the Space's runtime logs. Without this they go nowhere (Python's default no-handler behaviour), which is why every refresh / worker pathology so far has been silent. Combined push: the auto-refresh might not surface new rows if the underlying load_leaderboard returns stale data on the Space for any reason (CDN, auth, frontend re-render bug)…b2f3ce6 -
Dockerfile: bump CADGENBENCH_SHA to 9bd02fb (lazy baseline import)
Michael RabinovichPicks up the cadgenbench fix that makes `cadgenbench --help` and `cadgenbench evaluate` work in the eval-only install (no `[baseline]` extra). Before this bump, the chunk-4 worker's `python -m cadgenbench.cli evaluate <run_dir>` subprocess died during argparse setup with `ModuleNotFoundError: No module named 'dotenv'`, flipping every submission to status=failed before any work happened. Bumping CADGENBENCH_SHA invalidates the cadgenbench layer in the Dockerfile cache (Chromium layer + earlier layers stay cached), so the rebuild is ~2-3 min not the ~5 min cold cycle.67fd44c -
leaderboard: fetch results.jsonl via direct cache-busted HTTPS
Michael Rabinovichhf_hub_download (even with force_download=True) hands back stale bytes for several minutes after a commit: the local disk cache, revision pinning, and the Hub's CDN-fronted resolve endpoint all add their own staleness windows. End effect: pending and failed rows landed by the submit worker were on the Hub but invisible on the leaderboard tab until you waited it out, defeating the whole point of Refresh. Switch _load_rows_from_hub to a plain requests.get against the dataset's resolve/main URL with Cache-Control: no-cache and a millisecond cache-bust query param. Auth via huggingface_hub get_token() so the same code reads the still-private submissions dataset from the Space (HF_TOKEN secret) and from local dev (~/.cache/huggingface/token) without forking the code path. Adds requests to requirements.txt (was a transitive dep via huggingface_hub; making it explicit so a future upstream change doesn't silently drop it).9fcb684 -
submit: background-thread eval, flip row pending -> completed / failed
Michael RabinovichStep 6 (E) chunk 4. On a successful submit handle_submit now spawns a daemon thread that runs `cadgenbench evaluate` then `cadgenbench report single` over the unpacked submission, uploads reports/<id>.{html,json} to the submissions dataset, reads run_summary.json, and flips the row pending -> completed under the existing _HUB_LOCK. Any worker-side exception flips the row to failed with a short failure_reason (<=200 chars; full traceback goes to Space logs). Tempdir cleanup always runs in finally. Pipeline per submission, in the worker: 1. cadgenbench evaluate <run_dir> --workers 8 (subprocess via `python -m cadgenbench.cli`; the eval CLI already fans out across fixtures with ProcessPoolExecutor, self-throttles to n_fixtures when smaller). 2. cadgenbench report single <run_dir> -o <tmp>/<id>.html 3. Build reports JSON: run_summary.json + each fixture's result.json bundled together (mirrors README's description of reports/<id>.json as the "machine-readable…855bb8a -
leaderboard: render pending / failed score cells with a status tag
Michael RabinovichBlank score cells on a pending row left the only "this is in progress" cue in the status column, which read as broken to a user expecting at least *some* signal in the score columns. _fmt_pct and _fmt_score are now status-aware: pending rows render the tag "evaluating..." (hourglass prefix) in both score columns, failed rows render "failed" (cross prefix). Completed rows render the score number as before. Matches schema.md's "table renders the three states distinctly (spinner / score / error tag)" description and lines up with the visual pattern other HF leaderboards use while a row is queued.5aee3e5 -
leaderboard: handle null score fields on pending / failed rows
Michael RabinovichThe pending row landed by chunk 3 carries null validity_rate + aggregate_score (correct per the schema). pandas coerces JSON null to NaN on column construction, and _fmt_pct's `if x is None` gate didn't catch NaN; int(NaN) then raised ValueError inside .map(), bringing down load_leaderboard and rendering the leaderboard tab as a generic "Error". Fix: - _fmt_pct: gate on pd.isna(x) (catches both None and NaN), return "" for null. - _fmt_score: matching formatter for aggregate_score (was previously rendered as the raw float; with nulls in the mix it'd show as "NaN" cells). Returns "" for null, "%.4f" otherwise. Pending and failed rows now render with blank cells in both score columns; the row-level status column carries the meaning.00d091d -
app: drop history-narrative prose from Submit tab
Michael RabinovichThe Submit tab carried a sentence explaining why there's no form to fill in ("All five fields come from meta.json inside the zip, no separate form to fill in."). To a launch visitor that explains an absence that was never there; the right thing is to describe the contract as it stands, not to narrate a removal. Cut the sentence; keep the one useful half as a "Consent." line documenting that agree_to_publish: true in meta.json is the consent gate.a58058c -
submit: drop the agree checkbox, meta.json is the only consent gate
Michael RabinovichThe Submit tab had both an "I agree" checkbox and required meta.json to carry "agree_to_publish": true. Two gates expressing the same intent, neither covering a case the other didn't, and the UI half doesn't extend to programmatic submitters (gradio_client, future REST API). Drop the UI checkbox. meta.json["agree_to_publish"] stays the single consent gate, uniform across browser + programmatic submission. handle_submit signature shrinks to (zip_file). Submit form is now: zip upload + Submit button.9e84592 -
submit: drop dead form fields, meta.json is the only source
Michael RabinovichBefore: the Submit tab carried four text inputs (submission name, submitter name, agent URL, notes) alongside the zip upload, and handle_submit took them as args but ignored everything except the agree checkbox. They survived chunk 2's move from "form-driven stub" to "meta.json contract" only to keep the gr.click signature stable across one commit; chunk 3 didn't touch them either. After: Submit tab is just zip upload + agree checkbox + submit button. handle_submit signature is (zip_file, agree). All five submission fields come from meta.json inside the zip, full stop. Cuts 30 lines of UI + 4 unused handler params; removes a real user-confusion source (filling the form did nothing).f571140 -
submit: write pending row + upload zip on successful validation
Michael RabinovichStep 6 (E) chunk 3. Cheap-sync validation (chunk 2) now also commits to the Hub: on success the handler uploads submissions/<id>.zip and appends a status=pending row to results.jsonl under a process-wide threading.Lock. Eval still stubbed, the row stays pending forever until chunk 4 lands the background worker. Hub-write ordering (after validation passes): 1. Upload submissions/<id>.zip. Unique path per submission, no lock needed (HfApi handles its own commit serialization). 2. Build the pending row: metadata fields + status="pending" + failure_reason=null + submission_blob_url + null score fields, matching the "Pending" regime in cadgenbench-submissions/schema.md. 3. Acquire _HUB_LOCK, download current results.jsonl (treat EntryNotFoundError as empty), append one JSON line, re-upload. Lock held only for the RMW cycle (~1-2s), not for eval time; concurrent submitters serialise here, not in the worker. If step 1 fails the submitter sees a clean "Hub…1748c76 -
submit: move handle_submit into its own module + add validation pipeline
Michael RabinovichStep 6 (E) chunk 2 of the async submit work. Lifts the UI-only stub out of app.py into submit.py and grows it into the real cheap-sync validation pipeline. Still no Hub writes and no eval kick-off; the handler returns "would queue here" placeholder text. Async write + worker thread land in later chunks. Validation gates, in order: 1. Form: zip attached, agree-checkbox ticked. 2. Zip safety: valid zip, no absolute / parent-traversing entry names, no symlinks (Python 3.12's extractall normalises unsafe paths silently; we reject so submitters see a clear error). 3. meta.json schema: required keys, types, agree_to_publish is literal true, notes within the 500-char cap. meta.json is the source of truth for submitter / submission identity; the other form fields are advisory for now and kept only so the gr.click signature stays stable. 4. Fixture set: top-level dirs in the zip equal the fixture set reported by cadgenbench.common.paths.data_inputs_di…0501689 -
Dockerfile: bump CADGENBENCH_SHA to ed31274 (parallel evaluate)
Michael RabinovichPicks up the eval orchestrator's switch to ProcessPoolExecutor across fixtures (cadgenbench commit ed31274). On the Space's cpu-upgrade tier (8 vCPU) this should give a near-linear speedup on real-GT-scale runs; locally on the 5-stub set we saw ~2.4x going from --workers 1 to --workers 4 (1m13s -> 31s). The default worker count is min(8, os.cpu_count()) so the Space gets the speedup without setting any env var. Verified locally: per-fixture cad_score outputs are byte-identical between sequential and parallel runs (eval is per-fixture independent + deterministic).91244c6 -
Dockerfile: copy all Python modules, not just app.py
Michael RabinovichPrevious COPY hardcoded app.py only, which broke the moment app.py imported sibling leaderboard.py: the new module never made it into the image and the runtime hit ModuleNotFoundError. Switching to *.py picks up every Python file in AI4Engineering/ (currently app.py + leaderboard.py; submit.py lands next), so future module additions don't need a Dockerfile bump.f26021f -
refactor: split leaderboard read path into its own module
Michael RabinovichMirrors the shape adyen/DABstep uses on their reference Space (UI assembly in app.py, business logic in a sibling module). app.py drops from 224 lines to 158 by extracting the env-driven repo constants, the LOCAL_RESULTS_PATH + LEADERBOARD_COLS schema, the Hub/local row loaders, and load_leaderboard() into leaderboard.py. The submit handler stays in app.py for now; it'll move to a third module (submit.py) when Step 6 (E) lands the real validation + async eval. No behaviour change: load_leaderboard is imported back into app.py and wired into the same gr.Dataframe / refresh button pair as before. Smoke-tested locally (cadgenbench-space env): app module imports cleanly, load_leaderboard() returns a dataframe with the expected columns.b5ad973 -
docs: rewrite Space README + About tab as declarative current state
Michael Rabinovich- README.md: drop the (private; was michaelr27/...) history aside, drop the broken pointer to legacy/ (the directory was emptied during the NIST scrub), tighten the status sentence. - app.py ABOUT_MD: rewrite as a clean four-bullet "what this is" rather than a "status: active development" disclaimer block. Drops the broken pointer to space-setup/ (those notes are local-only and not in any public repo). Datasets bullet now describes both repos (inputs at HF_DATA_REPO, results at HF_SUBMISSIONS_REPO) with proper Hub links rather than just the repo names.628bc9e -
Dockerfile: bump CADGENBENCH_SHA to 1747493 (Hub-only fixtures)
Michael Rabinovichcadgenbench main now resolves fixtures exclusively from the Hub dataset repos: the wheel-bundled _data/ and the in-repo data/ folder are both gone (commit 1747493). The Space's Settings variables CADGENBENCH_DATA_REPO and CADGENBENCH_DATA_GT_REPO are already set (chunk 5), and HF_TOKEN is the existing Space secret, so this rebuild verifies the end-to-end Hub-only path inside the production runtime.d8207be -
Dockerfile: bump CADGENBENCH_SHA to 36ac6b4 (Hub-aware paths.py)
Michael Rabinovichcadgenbench commit 36ac6b4 adds the Hub-dataset resolution branch to common/paths.py (snapshot_download from $CADGENBENCH_DATA_REPO / $CADGENBENCH_DATA_GT_REPO when set, falling back to the existing local/bundled chain otherwise). The Space's Settings variables CADGENBENCH_DATA_REPO and CADGENBENCH_DATA_GT_REPO are set in lockstep with this bump (HuggingAI4Engineering/cadgenbench-data and HuggingAI4Engineering/cadgenbench-data-gt respectively), so the runtime now resolves fixtures from the Hub dataset repos instead of the wheel-bundled fallback. GT dataset is private; the existing HF_TOKEN Space secret provides read access (already set during Step 5 for the submissions dataset). No new secret required.aa1e886 -
scrub for the cadgenbench Public flip: drop NIST + remove build secret
Michael Rabinovich +1: CursorThree changes bundled together so the next Space rebuild lands in the post-NIST, post-secret end state in one go: - legacy/nist_*_comparison_3d.html (9 MB each) deleted; these were old NIST-stub comparison artifacts already labelled legacy, kept around for visual reference only. Regenerate from the real benchmark dataset post-launch if useful. - results.jsonl emptied. The three baseline seed rows it held were scheduled for pre-launch deletion anyway (per space-setup/migration.md Phase D), and the same rows were just rewritten on the Hub side of cadgenbench-submissions to drop the NIST per-fixture entries. The Space's leaderboard read path uses results.jsonl only as a Hub-fetch fallback; an empty file just produces an empty table on fallback, which is the right behaviour for a development Space. - Dockerfile: dropped the `RUN --mount=type=secret,id=GH_PAT` line and switched the cadgenbench install URL from the authenticated `git+https://michaelr27:$(cat /run/secre…a80542f -
Dockerfile: use actual GH username in classic-PAT URL, not x-access-token
Michael Rabinovich +1: CursorPrevious push tried 'https://x-access-token:TOKEN@github.com/...' (GitHub-Actions placeholder convention). GitHub rejected it: 'Invalid username or token. Password authentication is not supported for Git operations.' Classic PATs over HTTPS Basic Auth need a real GitHub username as the user half of the URL, even though the username is ignored once GitHub validates the PAT as the credential. Switching to 'https://michaelr27:TOKEN@github.com/...' which is the form GitHub's docs themselves recommend. Hardcoding 'michaelr27' is acceptable temporary debt: this whole URL goes away when we either swap GH_PAT for the read-only deploy key (URL becomes git+ssh://) or when huggingface/cadgenbench flips Public (URL drops auth entirely). Co-authored-by: Cursor <cursoragent@cursor.com>12c3a54 -
Dockerfile: use x-access-token:TOKEN URL form for classic PAT
Michael Rabinovich +1: CursorPrevious URL form 'https://$TOKEN@github.com/...' relied on GitHub's special-case where fine-grained PATs are accepted as a single-value username. Classic PATs don't get that special-casing: git parses 'TOKEN@github.com' as username=TOKEN with no password and then prompts for one, which fails in the non-interactive Docker build with 'No such device or address'. Switching to 'https://x-access-token:$TOKEN@github.com/...' (GitHub Actions convention) is the standard explicit-Basic-Auth form and works uniformly for classic + fine-grained PATs, so the URL stays the same when we eventually swap GH_PAT for a deploy key (the URL changes to ssh+git anyway in that case). Co-authored-by: Cursor <cursoragent@cursor.com>818fd28 -
Dockerfile: point cadgenbench install at huggingface/cadgenbench
Michael Rabinovich +1: CursorSwitch source from MichaelRabinovich/LeForge to huggingface/cadgenbench now that the latter has the full history (181 commits, see space-setup/migration.md Phase C). The pinned commit (d7e0468) exists on both remotes so this is URL-only; cadgenbench version bumps are a separate concern handled by ARG CADGENBENCH_SHA. The GH_PAT secret on the Space is being rotated in lockstep from a fine-grained PAT scoped to LeForge to a classic PAT with 'repo' scope. Classic is broader than ideal (gives the PAT read on any private repo the issuing user can see, not just huggingface/cadgenbench) but it's the only self-serve option: the huggingface GH org has fine-grained PATs disabled, and the cadgenbench space-setup user isn't an admin on the org repo so can't add a per-repo deploy key either. Tracked as follow-up debt to swap for a deploy key once admin permissions land (asked Leandro). Co-authored-by: Cursor <cursoragent@cursor.com>aa263ac -
docs: refresh stale status strings after org transfer
Michael Rabinovich +1: CursorSpace + submissions dataset have moved from michaelr27/... to the HuggingAI4Engineering org (Phase C of space-setup/migration.md). README body and app.py ABOUT_MD now reference the current Space slug (HuggingAI4Engineering/cadgenbench-leaderboard) instead of the pre-transfer michaelr27/AI4Engineering, and drop the stale 'science/...' migration target that was never used. Co-authored-by: Cursor <cursoragent@cursor.com>7dc3bc3
-
docs: refresh stale science/ migration target in space README
Michael Rabinovich +1: CursorLeandro decided against the existing science/ org; the migration target is now a new dedicated CAD/Engineering org (name TBD, picked by user, candidates include HuggingFaceCAD / HuggingEngineering). Mirrors the <hf-org> placeholder convention used throughout space-setup/. Co-authored-by: Cursor <cursoragent@cursor.com>93ddafd -
step 6 (A2): switch space to sdk:docker, install cadgenbench from private LeForge via GH_PAT build secret
Michael Rabinovich +1: CursorThe HF Spaces Gradio SDK can't substitute Space secrets into requirements.txt at build time (only at runtime), and embedding the literal token in requirements.txt is a no-go (would land in git history of the Space). Switching to sdk: docker collapses two open Step 6 sub-decisions into one small change: (A2) private GitHub dep install -- handled by `RUN --mount=type=secret` in the Dockerfile; the PAT (Space secret GH_PAT) is mounted at /run/secrets/GH_PAT for that one RUN and never lands in any layer. (D) Playwright/Chromium for STEP -> PNG rendering -- handled by `playwright install --with-deps chromium` in the same Dockerfile. Layer ordering optimises rebuild speed: OS deps + requirements.txt deps + Chromium come early (stable), cadgenbench from git comes mid (bumps on each commit), app.py + results.jsonl come last (changes most). Front-matter: sdk: gradio -> sdk: docker, drop sdk_version + app_file (Docker SDK uses Dockerfile CMD), add app_port: 7860. .…900c7c1 -
ui: render validity_rate as percentage in the leaderboard table
Michael Rabinovich +1: CursorConvert the 0-1 float into 'NN%' (or 'NN.N%' for non-whole values) just before passing the table to gr.Dataframe, so the column reads as '100%' instead of '1' / '1.0'. Underlying row schema is unchanged - validity_rate is still stored as a float in [0, 1] in results.jsonl. Side-effect: this column now sorts alphabetically rather than numerically because gr.Dataframe sees strings. Acceptable for v1 - the primary sort is aggregate_score and validity_rate hovers around 100% for serious submissions anyway. If finer sorting matters later, switch to a numeric column with a Gradio formatter. Co-authored-by: Cursor <cursoragent@cursor.com>00d6e0c -
schema: rename model -> submission_name; seed with 3 baseline rows
Michael Rabinovich +1: CursorMirrors the cadgenbench-submissions schema change. LEADERBOARD_COLS exposes submission_name as the lead identity column instead of model; the handle_submit signature, the Submit-tab form Textbox, and the meta.json example in the markdown all match. Submit-tab markdown gets a short "Submission name" explainer making explicit that the benchmark is system-agnostic - a submission may use no LLM, one, or several. If the submitter wants to disclose their stack they put it in submission_name or notes. Local results.jsonl mirrors the 3 baseline seed rows from cadgenbench-submissions so the offline fallback shows real data, not the old fake demos. Co-authored-by: Cursor <cursoragent@cursor.com>952dbca -
step 5: leaderboard reads from submissions dataset on hub
Michael Rabinovich +1: CursorReplace the local-only JSONL read with hf_hub_download(force_download=True) against michaelr27/cadgenbench-submissions. force_download avoids the stale-snapshot trap that datasets.load_dataset would create per process. Falls back to AI4Engineering/results.jsonl on any Hub error (broad except) so the table stays up during a transient outage or a mid-migration rename. HF_TOKEN secret already set on the Space via HfApi.add_space_secret to read the private dataset. Secret becomes write-only at launch when the dataset goes public; rotation TODO documented in space-setup/migration.md. Co-authored-by: Cursor <cursoragent@cursor.com>afed315 -
step 3: gradio leaderboard scaffold
Michael Rabinovich +1: CursorReplace the static placeholder index with a Gradio app: Leaderboard tab backed by a hand-crafted results.jsonl, Submit tab (UI-only stub, real eval wiring lands in step 6), About tab. Org slugs parameterised via HF_ORG / HF_SUBMISSIONS_REPO / HF_DATA_REPO env vars so the future move to the science org is a secret change, not a code change. Files: - app.py: gradio 6.14 app, three tabs - requirements.txt: gradio, pandas, huggingface_hub, datasets - results.jsonl: two dev seed rows - README.md: sdk: static -> sdk: gradio - legacy/: archive of the prototype nist comparison HTMLs - .gitignore: __pycache__, .venv, .gradio Schema: validity_rate (float [0,1]) instead of valid (bool); notes kept in the row but hidden from the visible table. All copy uses "AI-driven" rather than "LLM-driven". Co-authored-by: Cursor <cursoragent@cursor.com>c040324
-
Update index.html
Verified signature64a522a
-
changed index
Michael Rabinovich79711e2 -
Added comparison files
Michael Rabinovich69c9226