# Experiment tracking - Dashboard: - Project: `beam-pi-programbench-20261009` - Persistent storage: `burtenshaw/beam-pi-programbench-bucket`, mounted at `/data` - Pinned client and server: Trackio `0.41.0`; client Hugging Face Hub `1.33.0` The Space is public and publishes sanitized metrics, aggregate results, and the portable study source; raw traces remain private. Its initial `validation-` runs prove tracking infrastructure, not benchmark quality, calibration, or successful experiment execution. Remote upload status must be checked separately from benchmark completion. ## Run on the controller Install `study/requirements-tracking.txt` into the experiment's isolated Python 3.11+ environment. Start the worker alongside the controller, outside all agent containers: ```sh python study/tracking.py \ --events experiment/events.jsonl \ --state-dir experiment/tracking \ --space-id burtenshaw/beam-pi-programbench \ --project beam-pi-programbench-20261009 ``` The worker uses existing Hugging Face login state or the `HF_TOKEN` environment variable. On a remote controller, transfer the existing token through SSH standard input into a controller-private mode-600 file. Load that file into the tracking worker environment; do not include it in command arguments, Slurm logs, configs, or agent-container mounts. The model API credential belongs only to the model gateway, not this worker. Use `--once` to ingest and flush a completed file. Exit 0 means every ingested event was read back remotely; exit 1 leaves pending uploads; exit 2 means an invalid source event could not be ingested. Long-running workers retry upload failures every 15 seconds, while the experiment continues independently. SIGTERM/SIGINT triggers a final ingest/drain attempt; unsent records remain in SQLite for restart. ## Event contract The controller writes newline-terminated JSON objects, flushing after each event: ```json {"timestamp":"2026-10-09T12:00:00Z","run_id":"jq-single-r1","task":"jqlang__jq.b33a763","condition":"single","kind":"progress","metrics":{"api_input_tokens":12000,"api_output_tokens":1000,"api_total_tokens":13000,"context_tokens":8000,"elapsed_seconds":30},"status":"running"} ``` This is a schema example, not an observed benchmark result. Supported event kinds are `run_started`, `progress`, `score`, `run_finished`, `error`, and `budget_stop`. Supply the complete numeric configuration on `run_started`. Labels `model`, `provider`, `reasoning`, `task`, `condition`, `kind`, `phase`, `task_id`, `benchmark_commit`, `test_dataset_commit`, and `stop_reason` may contain short categorical strings. Other string settings and nested configs are omitted; flatten numerical budget settings before logging. Arbitrary messages, prompts, raw generated content, and raw test output are never uploaded. Keep `api_input_tokens`, `api_output_tokens`, `api_total_tokens`, `context_tokens`, `elapsed_seconds`, and verified aggregate `score` separate. Include numeric `calibration_passed` and `validation_passed` only when those checks have actually finished. Do not emit an unknown score as zero. Use `run_id="study-summary"` for aggregate budgets/status and prefix infrastructure runs with `validation-` so the published `phase` distinguishes them from benchmark episodes. ## Durability and error semantics `outbox.sqlite` is the authoritative local upload ledger (`synchronous=FULL`). Its records remain after upload. The source-event hash is the remote idempotency key, so replaying the JSONL or restarting a worker does not create duplicate metric records. The file cursor advances only after committing the event. Partial final lines wait for their terminating newline. A malformed full event leaves the cursor unchanged and produces `TRACKING_INGEST_ERROR`, requiring the controller event to be repaired. Trackio's normal `run.log()` is asynchronous. This adapter calls the same pinned `RemoteClient` `/bulk_log` endpoint with stable `log_id`s and then reads back the `tracking/event_sequence` values. HTTP success alone does not acknowledge an event: the Trackio server can queue writes. `status.json` reports `local_durable_events`, `remote_readback_events`, `pending_events`, and a sanitized `last_error`. Remote readback establishes visibility in the bucket-backed server; it is not a claim about completion of the experiment. The Space and persistent bucket were separately checked through Hub metadata and bucket file listings. No stdout/stderr exception body is uploaded, and network failures cannot terminate the independent benchmark controller. The controller should surface nonzero pending counts or worker exit as tracking warnings and retain the local directory for replay. Do not launch two workers with the same state directory concurrently. Validation: ```sh python -m unittest discover -s study/tests -p test_tracking.py -v ``` Five tests cover replay deduplication, failed-upload retention, acknowledgement only after readback, partial-line handling, and omission of raw text/credentials. Live infrastructure checks were also read back from the Space; no Beam call is needed to test tracking. ## Public source and final aggregate files The Space's `source/` release is an explicit, credential-scanned allowlist of the portable harness, tests, pinned configuration, public task metadata, and MIT license. `SOURCE_MANIFEST.json` records every included file hash and any source transformations. A sanitized recovery batch template is included. Other cluster deployment scripts, private host paths, runtime environments, sessions, hidden assets, and raw results are excluded. A source release is not evidence of benchmark completion. `publish_results.py --experiment-dir experiment --hf-token-file /path/to/private/hf-token` publishes four aggregate files beneath `results/beam-pi-programbench-20261009/`: `curves.csv`, `summary.json`, `study-status.json`, and `usage-summary.json`. The token is read internally from its private file; its value is never passed in arguments or printed. Each aggregate file is schema-checked and reconstructed from allowed fields, so raw errors, paths, prompts, and unexpected metadata cannot be uploaded. Every committed file is fetched at the exact commit revision and hash-checked. The publisher also generates 12 figures from those sanitized in-memory aggregates: `score-vs-latency-{axis}.{png,svg,pdf,json}` for `wall_seconds`, `reference_seconds`, and `api_prefill_reference_seconds`. Generation uses a fresh directory; arbitrary existing image files are never uploaded. Install the pinned dependencies in `requirements-plotting.txt` in the controller environment. Figures and aggregate files are committed together and all hashes must read back before publication is reported as verified. Recovery adds an optional `recovery-report.json` with reviewed raw/corrected scores, counts, hashes, and additional API usage. Its fields are separately allowlisted; raw errors and hidden test identifiers are excluded. The aggregate data and API retain terminal-artifact validity, archived/current invalid grades, attempt counts, and cumulative usage. See [RECOVERY.md](RECOVERY.md) for the recovery protocol. The Space's `/study` page displays only a title and the operational reference-clock plot. The aggregate files and `/study/data` API retain wall-time and API-prefill sensitivity views, grading coverage, and research disclosures. The plot contains no synthetic benchmark scores and stays empty before graded aggregates are published. Observed step curves preserve regressions; no confidence bands are added for the pilot with one repetition. The current figure uses an explicitly post-hoc four-task subset, excluding all ripgrep conditions due to slow saved-artifact grading. All 15 episode records and their charges remain in the aggregate data and API; the CSV retains original five-task evidence, while figure JSON contains the plotted subset. Trackio's existing dashboard, API, authentication, and persistent storage remain available at their existing routes. The site-specific batch launcher attempts this publication whenever all four files exist, including after a nonzero controller exit, while preserving that exit code. Synthetic `--validate-only` batches skip it. Publication success does not change the study's recorded status. Failure leaves `experiment/publish-status.json` with `remote_verified=false` and a sanitized error type; rerun the publisher after fixing the transport or schema problem. Missing aggregate files produce a visible skip message. The JSONL outbox used by live metrics remains independent and is retained. ## Verified upstream interfaces - [Trackio Python API](https://huggingface.co/docs/trackio/en/api): `init(space_id=...)` creates the Space and defaults to persistent bucket storage. - Installed `trackio/remote_client.py`, `trackio/server.py`, and `trackio/sqlite_storage.py` in 0.41.0 establish the HTTP transport, `bulk_log` schema, queued-write behavior, and unique `log_id` deduplication. - The generated Space `requirements.txt` pins `trackio[spaces,mcp]==0.41.0`.