source/study/README.md

Beam + Pi on ProgramBench

An open-source, exploratory comparison of one Pi agent, five persistent peers, and an async lead with persistent children. Inference uses Reflection's hosted Beam-501B-A23B; controllers and isolated program environments run on an allocated CPU host.

Accepted pilot

• Five pinned tasks: ripgrep, jq, xsv, FIGlet, hexyl.
• Three conditions and one repetition: 15 episodes.
• 8M API tokens per episode, shared across all agents; 120M benchmark total.
• 10M validation/contingency allowance, 130M global API-token ceiling.
• Stop admission when provider-reported daily remaining capacity approaches a 20M reserve. The quota is organization-wide; other callers can reduce availability.
• Two-hour wall limit and 1M estimated context tokens per episode. Single agent gets the full context allowance; peers and async agents get 200k lifetime limits. Async has at most five active agents, including the lead.
• Medium reasoning, temperature 0.7, top-p 0.9, 16k maximum completion tokens per request.
• Up to five simultaneous model requests; one benchmark episode at a time, under the same 16-CPU / 30-GiB Slurm allocation. This preserves the per-task compute allocation; it does not assume five model slots stay continuously occupied.

These are pilot budgets. The five tasks are a convenience sample, not Anthropic's complete evaluation set. One repetition does not establish run-to-run variability. The hosted Beam checkpoint may change behind its model ID.

Files and boundaries

• config.json: exact run order and budgets.
• budget_proxy.py: loopback-only model gateway with a persistent SQLite ledger and atomic reservations. It reserves the model's full advertised input bound plus maximum output before every request. Known usage refunds the unused portion; unknown failures retain the full reservation. No provider credential goes to Pi workers or containers. All normal and compaction calls use this gateway.
• pi-runner/: pinned Pi 0.80.2 SDK, one persistent process/session per agent, container-only bash, messaging, async delegation/resumption, shared Git main, immutable snapshots, and cancellation.
• programbench_adapter.py: pinned official reference calibration and grading. Hidden test archives stay outside solver containers. Only reference binaries and bundled documentation are available to agents. No internet, source retrieval, decompilation, or reference wrappers.
• evaluator-wheelhouse.py: pinned, hash-verified evaluator dependencies staged before network-disabled calibration and grading; never installed in solver containers.
• run_study.py: validation gates, execution, independent snapshot grading, and final analysis.
• tracking.py: sanitized numeric metrics and selected configuration fields, durable local outbox, remote acknowledgement by readback. Raw traces, code, prompts, hidden tests, and credentials are not uploaded.
• analyze.py: equal-task-weighted score curves, regressions preserved, terminal scores carried forward, missing grading reported. Operational causal clocks use declared reference rates and do not reproduce undisclosed Anthropic constants.
• publish_results.py: strict aggregate-file allowlist, schema validation, removal of raw errors and paths, and hash verification of the committed Space results.

API input/output counts are provider-reported, with reasoning already included in output. Context accounting is approximate because the hosted model's exact tokenizer/template is unavailable: append-only new non-assistant material uses UTF-8 bytes/4 plus per-message overhead; context-window transitions use the full actual prompt count; generated output uses actual completion counts. Both ledgers are preserved. An API guardrail can stop an episode before its context budget, which must be reported.

Host deployment

Run this controller on a Linux CPU host with a Docker-compatible runtime, Node 22+, and separate Python environments for the controller and the pinned official evaluator. Use the same CPU and memory allocation for every condition. --resource-mode docker uses container resource flags; --resource-mode slurm is for an allocation whose scheduler enforces resources when container cgroups are unavailable.

Reference calibration runs each original executable twice with the official test mask and requires complete outcomes and at least 0.9 in both repetitions. The controller admits benchmark inference only after reference calibration and synthetic live Pi validation succeed. --validate-only runs the synthetic infrastructure checks.

Provision private mode-600 files named reflection-api-key, hf-token, and proxy-key in a controller-only secrets directory, and pass its path through --secrets-dir. Never mount that directory, hidden test assets, or the host Docker socket into solver containers. Site-specific Slurm/bootstrap scripts are intentionally outside this portable source release. See the root README and PROGRAMBENCH.md for invocation.

Outputs stay in the chosen experiment directory. publish_results.py uploads only sanitized aggregate files and their derived figures; raw sessions, traces, submissions, and test outputs remain private. The model key is never given to Pi or to the tracker.

Build the offline evaluator dependencies before calibration, and keep this export in the environment of both calibration and controller processes:

.venv/bin/python study/evaluator-wheelhouse.py --root "$PWD" --uv uv
export PROGRAMBENCH_EVALUATOR_WHEELHOUSE="$PWD/runtime/evaluator-wheelhouse"

The builder needs network access and records pinned wheel hashes. Evaluator containers install only from this cache with network disabled; solver containers never receive it. Calibration and grading must use the identical cache digest.

Dashboard

Hugging Face Space, Trackio project beam-pi-programbench-20261009. The validation-* runs are infrastructure checks, not benchmark scores. Benchmark runs are named <task>-<single|peers|async>-r1.

Useful metrics are hidden_test_fraction, elapsed_seconds, api_tokens_charged, api_tokens_reserved, context_tokens_estimated, grader_complete, and status fields. Local outbox status distinguishes pending uploads from remotely visible metrics. Checkpoint grading occurs after the corresponding solver episode; agents receive none of its results.

The public source release is prepared from an explicit allowlist under the Space's source/ directory, with a hash manifest and MIT license. Site-specific cluster scripts, runtime environments, credentials, sessions, generated submissions, hidden tests, and raw evaluation logs are excluded. Source publication is separate from benchmark execution and does not establish a score.

After a full batch exits, the site-specific batch launcher publishes curves.csv, summary.json, study-status.json, and usage-summary.json under results/beam-pi-programbench-20261009/, if all four files exist. This also permits publishing honest partial/failure summaries after a nonzero controller exit. plot_results.py renders PNG, SVG, PDF, and Plotly JSON for all three latency axes from these sanitized aggregates; the same commit publishes these 12 allowlisted figures. Install study/requirements-plotting.txt in the controller environment. The Space's /study view shows interactive curves and download links while retaining the existing Trackio dashboard. It stays empty until graded aggregates exist. The batch skips publication in --validate-only mode and preserves the controller exit code. Upload failure is reported separately in experiment/publish-status.json; only remote_verified=true confirms file hashes read back at the resulting commit.

Verification

study/.venv/bin/python -m unittest discover -s study/tests -v
npm --prefix study/pi-runner test
PROGRAMBENCH_TEST_ROOT=/path/to/pinned/ProgramBench python -m pytest -q tests/test_programbench_adapter.py

Fake API tests make no paid inference calls. They cover concurrent reservations, failure/restart accounting, real Pi SDK tool calls and forced compaction, persistent message delivery/resumption, cutoff handling, scoring failures, tracking readback, and curve aggregation. Live cluster validation is separate evidence, recorded in run artifacts.

Code in this study directory is MIT-licensed. Pi and ProgramBench retain their upstream licenses; benchmark program images, documentation, and test data retain their own attribution and licenses. This repository references pinned assets and does not relicense or redistribute them. See the upstream ProgramBench repository and dataset attribution.