Spaces:
Running
Running
| # Experiment tracking | |
| - Dashboard: <https://huggingface.co/spaces/burtenshaw/beam-pi-programbench> | |
| - Project: `beam-pi-programbench-20261009` | |
| - Persistent storage: `burtenshaw/beam-pi-programbench-bucket`, mounted at `/data` | |
| - Pinned client and server: Trackio `0.41.0`; client Hugging Face Hub `1.33.0` | |
| The Space is public and publishes sanitized metrics, aggregate results, and the | |
| portable study source; raw traces remain private. Its initial `validation-` | |
| runs prove tracking infrastructure, not benchmark quality, calibration, or successful | |
| experiment execution. Remote upload status must be checked separately from benchmark completion. | |
| ## Run on the controller | |
| Install `study/requirements-tracking.txt` into the experiment's isolated Python 3.11+ | |
| environment. Start the worker alongside the controller, outside all agent containers: | |
| ```sh | |
| python study/tracking.py \ | |
| --events experiment/events.jsonl \ | |
| --state-dir experiment/tracking \ | |
| --space-id burtenshaw/beam-pi-programbench \ | |
| --project beam-pi-programbench-20261009 | |
| ``` | |
| The worker uses existing Hugging Face login state or the `HF_TOKEN` environment | |
| variable. On a remote controller, transfer the existing token through SSH standard input into a | |
| controller-private mode-600 file. Load that file into the tracking worker environment; | |
| do not include it in command arguments, Slurm logs, configs, or agent-container mounts. | |
| The model API credential belongs only to the model gateway, not this worker. | |
| Use `--once` to ingest and flush a completed file. Exit 0 means every ingested event | |
| was read back remotely; exit 1 leaves pending uploads; exit 2 means an invalid source | |
| event could not be ingested. Long-running workers retry upload failures every 15 | |
| seconds, while the experiment continues independently. SIGTERM/SIGINT triggers a final | |
| ingest/drain attempt; unsent records remain in SQLite for restart. | |
| ## Event contract | |
| The controller writes newline-terminated JSON objects, flushing after each event: | |
| ```json | |
| {"timestamp":"2026-10-09T12:00:00Z","run_id":"jq-single-r1","task":"jqlang__jq.b33a763","condition":"single","kind":"progress","metrics":{"api_input_tokens":12000,"api_output_tokens":1000,"api_total_tokens":13000,"context_tokens":8000,"elapsed_seconds":30},"status":"running"} | |
| ``` | |
| This is a schema example, not an observed benchmark result. Supported event kinds are | |
| `run_started`, `progress`, `score`, `run_finished`, `error`, and `budget_stop`. Supply the complete | |
| numeric configuration on `run_started`. Labels `model`, `provider`, `reasoning`, `task`, | |
| `condition`, `kind`, `phase`, `task_id`, `benchmark_commit`, `test_dataset_commit`, and | |
| `stop_reason` may contain short categorical strings. Other string settings and nested | |
| configs are omitted; flatten numerical budget settings before logging. Arbitrary | |
| messages, prompts, raw generated content, and raw test output are never uploaded. | |
| Keep `api_input_tokens`, `api_output_tokens`, `api_total_tokens`, `context_tokens`, | |
| `elapsed_seconds`, and verified aggregate `score` separate. Include numeric | |
| `calibration_passed` and `validation_passed` only when those checks have actually | |
| finished. Do not emit an unknown score as zero. Use `run_id="study-summary"` for | |
| aggregate budgets/status and prefix infrastructure runs with `validation-` so the | |
| published `phase` distinguishes them from benchmark episodes. | |
| ## Durability and error semantics | |
| `outbox.sqlite` is the authoritative local upload ledger (`synchronous=FULL`). Its | |
| records remain after upload. The source-event hash is the remote idempotency key, so | |
| replaying the JSONL or restarting a worker does not create duplicate metric records. | |
| The file cursor advances only after committing the event. Partial final lines wait | |
| for their terminating newline. A malformed full event leaves the cursor unchanged | |
| and produces `TRACKING_INGEST_ERROR`, requiring the controller event to be repaired. | |
| Trackio's normal `run.log()` is asynchronous. This adapter calls the same pinned | |
| `RemoteClient` `/bulk_log` endpoint with stable `log_id`s and then reads back the | |
| `tracking/event_sequence` values. HTTP success alone does not acknowledge an event: | |
| the Trackio server can queue writes. `status.json` reports `local_durable_events`, | |
| `remote_readback_events`, `pending_events`, and a sanitized `last_error`. Remote | |
| readback establishes visibility in the bucket-backed server; it is not a claim about | |
| completion of the experiment. The Space and persistent bucket were separately | |
| checked through Hub metadata and bucket file listings. | |
| No stdout/stderr exception body is uploaded, and network failures cannot terminate | |
| the independent benchmark controller. The controller should surface nonzero pending | |
| counts or worker exit as tracking warnings and retain the local directory for replay. | |
| Do not launch two workers with the same state directory concurrently. | |
| Validation: | |
| ```sh | |
| python -m unittest discover -s study/tests -p test_tracking.py -v | |
| ``` | |
| Five tests cover replay deduplication, failed-upload retention, acknowledgement only | |
| after readback, partial-line handling, and omission of raw text/credentials. Live | |
| infrastructure checks were also read back from the Space; no Beam call is needed to | |
| test tracking. | |
| ## Public source and final aggregate files | |
| The Space's `source/` release is an explicit, credential-scanned allowlist of | |
| the portable harness, tests, pinned configuration, public task metadata, and MIT | |
| license. `SOURCE_MANIFEST.json` records every included file hash and any source | |
| transformations. A sanitized recovery batch template is included. Other cluster | |
| deployment scripts, private host paths, runtime environments, sessions, hidden assets, and raw | |
| results are excluded. A source release is not evidence of benchmark completion. | |
| `publish_results.py --experiment-dir experiment --hf-token-file /path/to/private/hf-token` | |
| publishes four aggregate files beneath `results/beam-pi-programbench-20261009/`: | |
| `curves.csv`, `summary.json`, `study-status.json`, and `usage-summary.json`. The token | |
| is read internally from its private file; its value is never passed in arguments or | |
| printed. Each aggregate file is schema-checked and reconstructed from allowed fields, | |
| so raw errors, paths, prompts, and unexpected metadata cannot be uploaded. Every | |
| committed file is fetched at the exact commit revision and hash-checked. | |
| The publisher also generates 12 figures from those sanitized in-memory aggregates: | |
| `score-vs-latency-{axis}.{png,svg,pdf,json}` for `wall_seconds`, `reference_seconds`, | |
| and `api_prefill_reference_seconds`. Generation uses a fresh directory; arbitrary | |
| existing image files are never uploaded. Install the pinned dependencies in | |
| `requirements-plotting.txt` in the controller environment. Figures and aggregate | |
| files are committed together and all hashes must read back before publication is | |
| reported as verified. | |
| Recovery adds an optional `recovery-report.json` with reviewed raw/corrected scores, | |
| counts, hashes, and additional API usage. Its fields are separately allowlisted; raw | |
| errors and hidden test identifiers are excluded. The aggregate data and API retain | |
| terminal-artifact validity, archived/current invalid grades, attempt counts, and | |
| cumulative usage. See [RECOVERY.md](RECOVERY.md) for the recovery protocol. | |
| The Space's `/study` page displays only a title and the operational reference-clock | |
| plot. The aggregate files and `/study/data` API retain wall-time and API-prefill | |
| sensitivity views, grading coverage, and research disclosures. The plot contains no | |
| synthetic benchmark scores and stays empty before graded aggregates are published. | |
| Observed step curves preserve regressions; no confidence bands are added for the | |
| pilot with one repetition. The current figure uses an explicitly post-hoc four-task | |
| subset, excluding all ripgrep conditions due to slow saved-artifact grading. All 15 | |
| episode records and their charges remain in the aggregate data and API; the CSV retains original | |
| five-task evidence, while figure JSON contains the plotted subset. Trackio's existing dashboard, API, authentication, | |
| and persistent storage remain available at their existing routes. | |
| The site-specific batch launcher attempts this publication whenever all four files exist, | |
| including after a nonzero controller exit, while preserving that exit code. Synthetic | |
| `--validate-only` batches skip it. Publication success does not change the study's | |
| recorded status. Failure leaves `experiment/publish-status.json` with | |
| `remote_verified=false` and a sanitized error type; rerun the publisher after fixing | |
| the transport or schema problem. Missing aggregate files produce a visible skip | |
| message. The JSONL outbox used by live metrics remains independent and is retained. | |
| ## Verified upstream interfaces | |
| - [Trackio Python API](https://huggingface.co/docs/trackio/en/api): `init(space_id=...)` | |
| creates the Space and defaults to persistent bucket storage. | |
| - Installed `trackio/remote_client.py`, `trackio/server.py`, and | |
| `trackio/sqlite_storage.py` in 0.41.0 establish the HTTP transport, `bulk_log` schema, | |
| queued-write behavior, and unique `log_id` deduplication. | |
| - The generated Space `requirements.txt` pins `trackio[spaces,mcp]==0.41.0`. | |