Step 6 (E) chunk 6. A pending row whose worker died (Space
restart for a deploy, OOM, crash) has no one to flip it; without
this sweep it stays "pending" in the leaderboard forever (we
hit this twice during chunk-3..5 churn already).
On submit.py module import, spawn a daemon thread that:
1. Downloads results.jsonl (no lock; read-only).
2. Iterates rows; for each pending row whose `submitted_at`
is older than 30 min, calls _flip_row_to_failed(sid,
"evaluation interrupted by Space restart").
3. Logs the list of stuck ids so a deployer can see what was
swept.
Threshold is 30 min: well above the real eval ceiling on
cpu-upgrade (~5 min), so a genuinely-still-running submission is
safe. Hub-fetch failure at boot is non-fatal (logged warning,
sweep skipped; next boot retries). Per-row flip failures are
caught + logged + skipped (the rest of the sweep continues).
Opt-out via CADGENBENCH_DISABLE_BOOT_SWEEP=1 for test imports
that don't want the Hub round-trip.
Two tes…
submit: reject byte-identical re-submits via SHA-256 dedup
Michael Rabinovich
3237736
Before: clicking Submit twice (or re-uploading the same zip after
a transient confusion) created two rows pointing at byte-identical
content, ran the eval twice for the same input, and bloated the
submissions/reports blobs.
After: the cheap-sync handler computes sha256 of the uploaded zip
right after validation passes, queries results.jsonl for any row
with the same submission_sha256, and rejects with a pointer to
the existing submission_id if a match is found. Hash also lands
on the new row's submission_sha256 field (schema bump in the
companion commit on the submissions dataset).
Dedup-fetch failures fall back to letting the submit through
rather than rejecting on a Hub blip (logged). Race window: two
near-simultaneous identical submits whose dedup checks both
return None before either appends will land two rows; recoverable
by manual cleanup if it ever happens, not worth tightening before
auth + login throttling lands at pre-launch.
Drive-by: _build_pending_row's unused fixture_n…
Step 6 (E) chunk 4. On a successful submit handle_submit now spawns
a daemon thread that runs `cadgenbench evaluate` then `cadgenbench
report single` over the unpacked submission, uploads
reports/<id>.{html,json} to the submissions dataset, reads
run_summary.json, and flips the row pending -> completed under the
existing _HUB_LOCK. Any worker-side exception flips the row to
failed with a short failure_reason (<=200 chars; full traceback
goes to Space logs). Tempdir cleanup always runs in finally.
Pipeline per submission, in the worker:
1. cadgenbench evaluate <run_dir> --workers 8
(subprocess via `python -m cadgenbench.cli`; the eval CLI
already fans out across fixtures with ProcessPoolExecutor,
self-throttles to n_fixtures when smaller).
2. cadgenbench report single <run_dir> -o <tmp>/<id>.html
3. Build reports JSON: run_summary.json + each fixture's
result.json bundled together (mirrors README's description
of reports/<id>.json as the "machine-readable…
submit: drop the agree checkbox, meta.json is the only consent gate
Michael Rabinovich
9e84592
The Submit tab had both an "I agree" checkbox and required
meta.json to carry "agree_to_publish": true. Two gates expressing
the same intent, neither covering a case the other didn't, and
the UI half doesn't extend to programmatic submitters
(gradio_client, future REST API).
Drop the UI checkbox. meta.json["agree_to_publish"] stays the
single consent gate, uniform across browser + programmatic
submission. handle_submit signature shrinks to (zip_file).
Submit form is now: zip upload + Submit button.
submit: drop dead form fields, meta.json is the only source
Michael Rabinovich
f571140
Before: the Submit tab carried four text inputs (submission name,
submitter name, agent URL, notes) alongside the zip upload, and
handle_submit took them as args but ignored everything except the
agree checkbox. They survived chunk 2's move from "form-driven
stub" to "meta.json contract" only to keep the gr.click signature
stable across one commit; chunk 3 didn't touch them either.
After: Submit tab is just zip upload + agree checkbox + submit
button. handle_submit signature is (zip_file, agree). All five
submission fields come from meta.json inside the zip, full stop.
Cuts 30 lines of UI + 4 unused handler params; removes a real
user-confusion source (filling the form did nothing).
submit: write pending row + upload zip on successful validation
Michael Rabinovich
1748c76
Step 6 (E) chunk 3. Cheap-sync validation (chunk 2) now also commits
to the Hub: on success the handler uploads submissions/<id>.zip and
appends a status=pending row to results.jsonl under a process-wide
threading.Lock. Eval still stubbed, the row stays pending forever
until chunk 4 lands the background worker.
Hub-write ordering (after validation passes):
1. Upload submissions/<id>.zip. Unique path per submission, no
lock needed (HfApi handles its own commit serialization).
2. Build the pending row: metadata fields + status="pending" +
failure_reason=null + submission_blob_url + null score fields,
matching the "Pending" regime in cadgenbench-submissions/schema.md.
3. Acquire _HUB_LOCK, download current results.jsonl (treat
EntryNotFoundError as empty), append one JSON line, re-upload.
Lock held only for the RMW cycle (~1-2s), not for eval time;
concurrent submitters serialise here, not in the worker.
If step 1 fails the submitter sees a clean "Hub…
submit: move handle_submit into its own module + add validation pipeline
Michael Rabinovich
0501689
Step 6 (E) chunk 2 of the async submit work. Lifts the UI-only stub
out of app.py into submit.py and grows it into the real cheap-sync
validation pipeline. Still no Hub writes and no eval kick-off; the
handler returns "would queue here" placeholder text. Async write +
worker thread land in later chunks.
Validation gates, in order:
1. Form: zip attached, agree-checkbox ticked.
2. Zip safety: valid zip, no absolute / parent-traversing entry
names, no symlinks (Python 3.12's extractall normalises unsafe
paths silently; we reject so submitters see a clear error).
3. meta.json schema: required keys, types, agree_to_publish is
literal true, notes within the 500-char cap. meta.json is the
source of truth for submitter / submission identity; the other
form fields are advisory for now and kept only so the gr.click
signature stays stable.
4. Fixture set: top-level dirs in the zip equal the fixture set
reported by cadgenbench.common.paths.data_inputs_di…
