adds load_admin_table to the leaderboard reader: a single flat frame of
every row (both tiers) with a leading editable select column, for the
admin tab to act on. reworks admin.py around bulk helpers
(promote_rows, demote_rows, delete_rows) that each do one results.jsonl
write for the whole selection; delete also best-effort removes the
companion zip and report artifacts. the singular promote_row/demote_row
stay as thin wrappers. tests cover the bulk paths, a missing-id abort,
empty-selection guards, and artifact deletion, all with the hub mocked.
tests: cover the admin gate and promote/demote helpers
Michael Rabinovich
007a50e
four unit tests for admin.py: the CADGENBENCH_ADMINS gate (in-set,
out-of-set, logged-out, empty set), promote and demote happy paths,
and idempotent re-promotion. the results.jsonl read and write that
_hub_rmw_results drives are mocked, so the suite makes no hub calls.
leaderboard: format submitted_at as `YYYY-MM-DD HH:MM UTC`; lock tables read-only
Michael Rabinovich
c4e21b3
Table cells were editable (gradio_leaderboard inherits gr.Dataframe's
interactive default); add interactive=False to both Validated and
Unvalidated widgets so cell contents aren't accidentally typed over.
submitted_at column rendered as raw ISO-8601 ("2026-05-28T07:13:16Z");
apply the same _fmt_timestamp helper the detail panel already uses so
the table reads as "2026-05-28 07:13 UTC". Helper moved into
leaderboard.py (formatter module) and re-imported by app.py.
submit+app: gate Submit on HF OAuth; populate hf_username from profile
Michael Rabinovich
c87b253
Bundle 1+2 C10. Login-required submit shaped per the bundle spec.
submit.py:
- `handle_submit` grows a `profile: gr.OAuthProfile | None` arg.
Gradio dependency-injects the visitor's profile based on the
signature. None -> immediate `gr.Error("Please log in via the
HF button before submitting.")` so a UI mishap can't write an
anonymous row.
- `_build_pending_row` grows an `hf_username=None` kwarg.
`handle_submit` passes `profile.username` through so the row
carries the canonical HF identity (not a free-text claim in
meta.json's submitter_name).
app.py:
- New `gr.LoginButton()` in the Submit tab above the zip dropzone.
- Submit button starts `interactive=False`; new
`_enable_submit_when_logged_in(profile)` reads the visitor's
OAuth profile via `blocks.load` and flips the button on / off
per page load. Server-side gate in `handle_submit` mirrors this
so a logged-out user who somehow bypasses the disabled-button
UI still gets rejected.
requirements.txt:
- `gradio…
leaderboard+app: combined CSV download with validation_status discriminator
Michael Rabinovich
f585077
Bundle 1+2 C8. One file, both tiers, downloaded via
`gr.DownloadButton` next to the existing Refresh button.
leaderboard.py:
- New `CSV_COLUMNS` constant (wider than the on-screen table:
raw aggregate_score / validity_rate, plus provenance fields
like submission_sha256 + cadgenbench_data_revision + notes +
failure_reason). `validation_status` is the discriminator that
lets a reader grep the file for one tier or the other.
- New `build_combined_csv()` reads the same rows the leaderboard
reader does, applies the same status / validation_status
defaults, projects to CSV_COLUMNS, sort: validated rows on top
(highest score first) then unvalidated, then writes to a unique
/tmp file and returns the path. Fresh on every click so the
export reflects the latest data, not a snapshot captured at
boot.
app.py:
- Imports `build_combined_csv`. The Refresh button and the new
`gr.DownloadButton("Download CSV", size="sm")` live in a single
`gr.Row` so they sit side-by-side unde…
app: render reports inline via iframe srcdoc (private-Space safe)
Michael Rabinovich
0e3b21f
Polish follow-up on C6. Reports now render inside the Space, inside
the same iframe HF uses to host the leaderboard, with no second
HTTP request leaving the browser.
Why: the Space is private, and HF's edge gates same-origin pathname
navigations on a short-lived __sign JWT that browsers don't carry
forward when a user clicks a relative link. The previous design
(submission_name -> /reports/<id>.html) worked server-side under
Bearer auth (verified) but 404'd in the browser. Confirmed via
external review and reproduced with a requests.Session cookie test.
This commit:
- Drops the submission_name -> /reports/<id>.html link wrapping
from the rendered table; the cell stays plain text. The
underlying `report_url` column (relative URL to the proxy) is
still computed for the modern-pipeline gate but no longer
surfaced as a clickable cell.
- Adds a `gr.HTML` viewer below the detail panel. Row-click
triggers `_format_detail_and_report`, which returns
(detail_markdown, report_ifram…
app+leaderboard: link submission_name to a Space-side report proxy
Michael Rabinovich
77edebf
Polish follow-up #2 on C6. Three user-visible changes plus the
plumbing required to make them work.
1. Drop the `report` and `zip` columns from the rendered table.
`submission_name` becomes the report's deep link (typical HF
leaderboard pattern: the system / model name doubles as the link
target; cf. Open LLM Leaderboard, DABstep, BigCodeBench). Modern
completed rows get `[<name>](/reports/<sid>.html)`; pending,
failed, and legacy seed rows fall back to plain text so there's
no broken click. The hidden `submission_blob_url` and the
computed `report_url` ride along in the DataFrame and surface in
the row-click detail panel only.
2. Rename the agent-URL column from `agent` to
`model details (optional)`. Missing cells render as italic
`_None_` so the optionality is explicit instead of a blank.
"Agent" was confusing; the schema lets `agent_url` be code OR
paper OR a blog post and "agent" reads narrow.
3. Add a Space-side proxy for the per-submission H…
Polish follow-up on C6, three fixes after the first detail-panel
commit landed:
1. Rename display columns from snake_case schema names to clean
short headers. `agent_url` -> `agent`, `submission_blob_url` ->
`zip`, `report_url` -> `report`. The rename happens at projection
time via `df.rename`, so the schema in results.jsonl is
unchanged. Headers now read as plain English ("agent", "zip",
"report") rather than a python dict.
2. The `agent` column shows the URL itself (scheme stripped,
truncated past 40 chars) as link text instead of a fixed "code"
label. The schema says agent_url can be code OR paper; the
shortened URL is the only honest hint about what's behind the
click. Missing values render as italic `_None_` rather than a
blank cell, so the optionality is explicit.
3. The `report` link is now gated on the row's `submission_sha256`
being non-null in addition to status == "completed". Legacy seed
rows pre-date the modern submit pipeline (which …
leaderboard: markdown link columns for agent_url, submission, report
Michael Rabinovich
53de73a
Bundle 1+2 C5. Three new columns on both tables (validated +
unvalidated) carry clickable links rendered via gradio_leaderboard's
`datatype="markdown"`:
- agent_url: link to the agent's source / paper if the row's
meta.json provided one (label "code"; empty cell when null).
- submission_blob_url: link to the uploaded zip on the submissions
dataset (label "zip"; populated on every non-failed row, the
same URL the submit handler computed at upload time).
- report_url: built at read time from `submission_id` plus
HF_SUBMISSIONS_REPO; points at `reports/<id>.html` under
/resolve/main/ so the browser renders the report directly.
Only emitted for `status == "completed"` rows.
leaderboard.py:
- LEADERBOARD_COLS + VALIDATED_LEADERBOARD_COLS grow by three.
- New LEADERBOARD_DATATYPES + VALIDATED_LEADERBOARD_DATATYPES
constants mark the link columns as "markdown", everything else
as "str". Tied to the column lists by a single helper so the
two stay in lockstep.
- New _agent_u…
submit: default validation + hf_username fields on the pending row
Michael Rabinovich
a533dd2
Bundle 1+2 C4. `_build_pending_row` adds three new keys matching the
schema bump from C1, in the schema.md column order:
- validation_status = "unvalidated" (every new row starts in the
unvalidated tier; maintainers promote via the dataset repo after
methodology review).
- validation_method = None (set by the maintainer at promotion time
to one of code | traces | api | manual).
- hf_username = None (placeholder; C10 plumbs through the value
from gradio.OAuthProfile.username once the LoginButton lands).
Pre-existing fields, submitted_at stamping, the cadgenbench version
+ data revision tags, the score-shaped null fields, the blob URL,
and the sha256 are untouched. Dedup gate still pivots on
submission_sha256 exactly as before.
tests/test_submit.py covers the C4 acceptance:
- new-row defaults: validation_status is "unvalidated",
validation_method + hf_username are None;
- sha256 path passthrough: the hash argument lands on the row's
submission_sha256 key intact (existing …
app: split Leaderboard into Validated + Unvalidated tables
Michael Rabinovich
046548a
Bundle 1+2 C3. Swaps the single Leaderboard widget for two stacked
ones (Validated on top, Unvalidated below) and wires the existing
auto-refresh Timer + Refresh button to push fresh dataframes into
both. Initial render reads via the C2 `load_leaderboard_split()`;
existing rows all land in Unvalidated until a maintainer flips
`validation_status` to `"validated"` on a row in the submissions
dataset.
- app.py: replaces `df_view` with `validated_view` + `unvalidated_view`.
Single timer tick now fans out to both widgets (load split runs
once per tick, both halves get pushed). Refresh button same
fan-out shape. Inline comment points at the validation policy
decision doc.
- leaderboard.py: drops the now-unused single-DataFrame
`load_leaderboard()` per the C3 cleanup half. Stale docstring
references that pointed at the dead function got smoothed out
(the split reader's doc is self-contained now).
- tests/test_smoke.py: picks up the C3 acceptance with a new
`test_two_leaderboa…
leaderboard: add load_leaderboard_split() for two-tier viewer
Michael Rabinovich
0dd7215
Bundle 1+2 C2. Additive new function for the C3 two-stacked-table
swap; existing single-DataFrame load_leaderboard() stays put until
C3 flips the caller in app.py + drops it. Keeps the live Space green
between commits.
- VALIDATED_LEADERBOARD_COLS: same shape as LEADERBOARD_COLS plus
`validation_method` (always null on unvalidated rows, so omitted
from that tier rather than rendered).
- load_leaderboard_split() -> (validated_df, unvalidated_df):
reads results.jsonl the same way as load_leaderboard(), splits
rows on `validation_status` (defensive: anything not literally
"validated" lands in unvalidated, which covers legacy + null +
any future-unknown value), runs the same status-aware cell
formatting per tier via the new _project_and_format helper.
- _project_and_format: shared sort + projection + pending/failed
cell tagging so the two tiers produce identically-shaped cells.
Legacy default: rows without `validation_status` get "unvalidated"
via row-level fill. No resul…
First Space-side test harness; sets up the floor for the Phase D
bundle's per-commit unit + UI tests (split leaderboard, pending-row
defaults, OAuth-required submit) without slipping a regression past
a build-only check.
requirements-dev.txt pulls -r requirements.txt then adds pytest +
playwright. One `pip install -r requirements-dev.txt && playwright
install chromium` brings a fresh venv up to speed.
tests/conftest.py exposes a session-scoped `app_url` fixture: boots
app.py in a subprocess on a free port, polls until the HTTP server
answers, yields the URL, tears the process down on teardown. Logs
land in a tmp file so a startup failure surfaces the real Gradio
traceback rather than a bare RuntimeError. Boot-time stuck-pending
sweep in submit.py is disabled via CADGENBENCH_DISABLE_BOOT_SWEEP=1
so the fixture is fast and offline-tolerant.
tests/test_smoke.py launches headless Chromium via playwright's sync
API, navigates to the fixture's URL, and asserts the Leaderboard /
Submit / A…
