source/study/TRACKING.md

Experiment tracking

• Dashboard: https://huggingface.co/spaces/burtenshaw/beam-pi-programbench
• Project: beam-pi-programbench-20261009
• Persistent storage: burtenshaw/beam-pi-programbench-bucket, mounted at /data
• Pinned client and server: Trackio 0.41.0; client Hugging Face Hub 1.33.0

The Space is public and publishes sanitized metrics, aggregate results, and the portable study source; raw traces remain private. Its initial validation- runs prove tracking infrastructure, not benchmark quality, calibration, or successful experiment execution. Remote upload status must be checked separately from benchmark completion.

Run on the controller

Install study/requirements-tracking.txt into the experiment's isolated Python 3.11+ environment. Start the worker alongside the controller, outside all agent containers:

python study/tracking.py \
  --events experiment/events.jsonl \
  --state-dir experiment/tracking \
  --space-id burtenshaw/beam-pi-programbench \
  --project beam-pi-programbench-20261009

The worker uses existing Hugging Face login state or the HF_TOKEN environment variable. On a remote controller, transfer the existing token through SSH standard input into a controller-private mode-600 file. Load that file into the tracking worker environment; do not include it in command arguments, Slurm logs, configs, or agent-container mounts. The model API credential belongs only to the model gateway, not this worker.

Use --once to ingest and flush a completed file. Exit 0 means every ingested event was read back remotely; exit 1 leaves pending uploads; exit 2 means an invalid source event could not be ingested. Long-running workers retry upload failures every 15 seconds, while the experiment continues independently. SIGTERM/SIGINT triggers a final ingest/drain attempt; unsent records remain in SQLite for restart.

Event contract

The controller writes newline-terminated JSON objects, flushing after each event:

{"timestamp":"2026-10-09T12:00:00Z","run_id":"jq-single-r1","task":"jqlang__jq.b33a763","condition":"single","kind":"progress","metrics":{"api_input_tokens":12000,"api_output_tokens":1000,"api_total_tokens":13000,"context_tokens":8000,"elapsed_seconds":30},"status":"running"}

This is a schema example, not an observed benchmark result. Supported event kinds are run_started, progress, score, run_finished, error, and budget_stop. Supply the complete numeric configuration on run_started. Labels model, provider, reasoning, task, condition, kind, phase, task_id, benchmark_commit, test_dataset_commit, and stop_reason may contain short categorical strings. Other string settings and nested configs are omitted; flatten numerical budget settings before logging. Arbitrary messages, prompts, raw generated content, and raw test output are never uploaded.

Keep api_input_tokens, api_output_tokens, api_total_tokens, context_tokens, elapsed_seconds, and verified aggregate score separate. Include numeric calibration_passed and validation_passed only when those checks have actually finished. Do not emit an unknown score as zero. Use run_id="study-summary" for aggregate budgets/status and prefix infrastructure runs with validation- so the published phase distinguishes them from benchmark episodes.

Durability and error semantics

outbox.sqlite is the authoritative local upload ledger (synchronous=FULL). Its records remain after upload. The source-event hash is the remote idempotency key, so replaying the JSONL or restarting a worker does not create duplicate metric records. The file cursor advances only after committing the event. Partial final lines wait for their terminating newline. A malformed full event leaves the cursor unchanged and produces TRACKING_INGEST_ERROR, requiring the controller event to be repaired.

Trackio's normal run.log() is asynchronous. This adapter calls the same pinned RemoteClient /bulk_log endpoint with stable log_ids and then reads back the tracking/event_sequence values. HTTP success alone does not acknowledge an event: the Trackio server can queue writes. status.json reports local_durable_events, remote_readback_events, pending_events, and a sanitized last_error. Remote readback establishes visibility in the bucket-backed server; it is not a claim about completion of the experiment. The Space and persistent bucket were separately checked through Hub metadata and bucket file listings.

No stdout/stderr exception body is uploaded, and network failures cannot terminate the independent benchmark controller. The controller should surface nonzero pending counts or worker exit as tracking warnings and retain the local directory for replay. Do not launch two workers with the same state directory concurrently.

Validation:

python -m unittest discover -s study/tests -p test_tracking.py -v

Five tests cover replay deduplication, failed-upload retention, acknowledgement only after readback, partial-line handling, and omission of raw text/credentials. Live infrastructure checks were also read back from the Space; no Beam call is needed to test tracking.

Public source and final aggregate files

The Space's source/ release is an explicit, credential-scanned allowlist of the portable harness, tests, pinned configuration, public task metadata, and MIT license. SOURCE_MANIFEST.json records every included file hash and any source transformations. A sanitized recovery batch template is included. Other cluster deployment scripts, private host paths, runtime environments, sessions, hidden assets, and raw results are excluded. A source release is not evidence of benchmark completion.

publish_results.py --experiment-dir experiment --hf-token-file /path/to/private/hf-token publishes four aggregate files beneath results/beam-pi-programbench-20261009/: curves.csv, summary.json, study-status.json, and usage-summary.json. The token is read internally from its private file; its value is never passed in arguments or printed. Each aggregate file is schema-checked and reconstructed from allowed fields, so raw errors, paths, prompts, and unexpected metadata cannot be uploaded. Every committed file is fetched at the exact commit revision and hash-checked.

The publisher also generates 12 figures from those sanitized in-memory aggregates: score-vs-latency-{axis}.{png,svg,pdf,json} for wall_seconds, reference_seconds, and api_prefill_reference_seconds. Generation uses a fresh directory; arbitrary existing image files are never uploaded. Install the pinned dependencies in requirements-plotting.txt in the controller environment. Figures and aggregate files are committed together and all hashes must read back before publication is reported as verified.

Recovery adds an optional recovery-report.json with reviewed raw/corrected scores, counts, hashes, and additional API usage. Its fields are separately allowlisted; raw errors and hidden test identifiers are excluded. The aggregate data and API retain terminal-artifact validity, archived/current invalid grades, attempt counts, and cumulative usage. See RECOVERY.md for the recovery protocol.

The Space's /study page displays only a title and the operational reference-clock plot. The aggregate files and /study/data API retain wall-time and API-prefill sensitivity views, grading coverage, and research disclosures. The plot contains no synthetic benchmark scores and stays empty before graded aggregates are published. Observed step curves preserve regressions; no confidence bands are added for the pilot with one repetition. The current figure uses an explicitly post-hoc four-task subset, excluding all ripgrep conditions due to slow saved-artifact grading. All 15 episode records and their charges remain in the aggregate data and API; the CSV retains original five-task evidence, while figure JSON contains the plotted subset. Trackio's existing dashboard, API, authentication, and persistent storage remain available at their existing routes.

The site-specific batch launcher attempts this publication whenever all four files exist, including after a nonzero controller exit, while preserving that exit code. Synthetic --validate-only batches skip it. Publication success does not change the study's recorded status. Failure leaves experiment/publish-status.json with remote_verified=false and a sanitized error type; rerun the publisher after fixing the transport or schema problem. Missing aggregate files produce a visible skip message. The JSONL outbox used by live metrics remains independent and is retained.

Verified upstream interfaces

• Trackio Python API: init(space_id=...) creates the Space and defaults to persistent bucket storage.
• Installed trackio/remote_client.py, trackio/server.py, and trackio/sqlite_storage.py in 0.41.0 establish the HTTP transport, bulk_log schema, queued-write behavior, and unique log_id deduplication.
• The generated Space requirements.txt pins trackio[spaces,mcp]==0.41.0.