# Beam + Pi on ProgramBench An open-source, exploratory comparison of one Pi agent, five persistent peers, and an async lead with persistent children. Inference uses Reflection's hosted `Beam-501B-A23B`; controllers and isolated program environments run on an allocated CPU host. ## Accepted pilot See [RECOVERY.md](RECOVERY.md) for the original failures, audited artifact corrections, restricted retries, cumulative accounting, and reduced evaluator concurrency under the original allocation. Published final-artifact validity is separate from having any earlier valid grade. - Five pinned tasks: ripgrep, jq, xsv, FIGlet, hexyl. - Three conditions and one repetition: **15 episodes**. - **8M API tokens per episode**, shared across all agents; **120M benchmark total**. - **10M validation/contingency allowance**, **130M global API-token ceiling**. - Stop admission when provider-reported daily remaining capacity approaches a 20M reserve. The quota is organization-wide; other callers can reduce availability. - Two-hour wall limit and 1M **estimated context tokens** per episode. Single agent gets the full context allowance; peers and async agents get 200k lifetime limits. Async has at most five active agents, including the lead. - Medium reasoning, temperature 0.7, top-p 0.9, 16k maximum completion tokens per request. - Up to five simultaneous model requests; one benchmark episode at a time, under the same 16-CPU / 30-GiB Slurm allocation. This preserves the per-task compute allocation; it does not assume five model slots stay continuously occupied. These are pilot budgets. The five tasks are a convenience sample, not Anthropic's complete evaluation set. One repetition does not establish run-to-run variability. The hosted Beam checkpoint may change behind its model ID. ## Files and boundaries - `config.json`: exact run order and budgets. - `budget_proxy.py`: loopback-only model gateway with a persistent SQLite ledger and atomic reservations. It reserves the model's full advertised input bound plus maximum output before every request. Known usage refunds the unused portion; unknown failures retain the full reservation. No provider credential goes to Pi workers or containers. All normal and compaction calls use this gateway. - `pi-runner/`: pinned Pi 0.80.2 SDK, one persistent process/session per agent, container-only bash, messaging, async delegation/resumption, shared Git main, immutable snapshots, and cancellation. - `programbench_adapter.py`: pinned official reference calibration and grading. Hidden test archives stay outside solver containers. Only reference binaries and bundled documentation are available to agents. No internet, source retrieval, decompilation, or reference wrappers. - `evaluator-wheelhouse.py`: pinned, hash-verified evaluator dependencies staged before network-disabled calibration and grading; never installed in solver containers. - `run_study.py`: validation gates, execution, independent snapshot grading, and final analysis. - `tracking.py`: sanitized numeric metrics and selected configuration fields, durable local outbox, remote acknowledgement by readback. Raw traces, code, prompts, hidden tests, and credentials are not uploaded. - `analyze.py`: equal-task-weighted score curves, regressions preserved, terminal scores carried forward, missing grading reported. Operational causal clocks use declared reference rates and do not reproduce undisclosed Anthropic constants. - `publish_results.py`: strict aggregate-file allowlist, schema validation, removal of raw errors and paths, and hash verification of the committed Space results. API input/output counts are provider-reported, with reasoning already included in output. Context accounting is approximate because the hosted model's exact tokenizer/template is unavailable: append-only new non-assistant material uses UTF-8 bytes/4 plus per-message overhead; context-window transitions use the full actual prompt count; generated output uses actual completion counts. Both ledgers are preserved. An API guardrail can stop an episode before its context budget, which must be reported. ## Host deployment Run this controller on a Linux CPU host with a Docker-compatible runtime, Node 22+, and separate Python environments for the controller and the pinned official evaluator. Use the same CPU and memory allocation for every condition. `--resource-mode docker` uses container resource flags; `--resource-mode slurm` is for an allocation whose scheduler enforces resources when container cgroups are unavailable. Reference calibration runs each original executable twice with the official test mask and requires complete outcomes and at least 0.9 in both repetitions. The controller admits benchmark inference only after reference calibration and synthetic live Pi validation succeed. `--validate-only` runs the synthetic infrastructure checks. Provision private mode-600 files named `reflection-api-key`, `hf-token`, and `proxy-key` in a controller-only secrets directory, and pass its path through `--secrets-dir`. Never mount that directory, hidden test assets, or the host Docker socket into solver containers. The included recovery batch is a sanitized template that requires local cluster parameters; other site-specific Slurm/bootstrap scripts remain excluded. See the root README and PROGRAMBENCH.md for invocation. Outputs stay in the chosen experiment directory. `publish_results.py` uploads only sanitized aggregate files and their derived figures; raw sessions, traces, submissions, and test outputs remain private. The model key is never given to Pi or to the tracker. Build the offline evaluator dependencies before calibration, and keep this export in the environment of both calibration and controller processes: ```sh .venv/bin/python study/evaluator-wheelhouse.py --root "$PWD" --uv uv export PROGRAMBENCH_EVALUATOR_WHEELHOUSE="$PWD/runtime/evaluator-wheelhouse" ``` The builder needs network access and records pinned wheel hashes. Evaluator containers install only from this cache with network disabled; solver containers never receive it. Calibration and grading must use the identical cache digest. ## Dashboard [Hugging Face Space](https://huggingface.co/spaces/burtenshaw/beam-pi-programbench), Trackio project `beam-pi-programbench-20261009`. The `validation-*` runs are infrastructure checks, not benchmark scores. Benchmark runs are named `--r1`. Useful metrics are `hidden_test_fraction`, `elapsed_seconds`, `api_tokens_charged`, `api_tokens_reserved`, `context_tokens_estimated`, `grader_complete`, and status fields. Local outbox status distinguishes pending uploads from remotely visible metrics. Checkpoint grading occurs after the corresponding solver episode; agents receive none of its results. The public source release is prepared from an explicit allowlist under the Space's `source/` directory, with a hash manifest and MIT license. The recovery batch is a sanitized template; other site-specific cluster scripts, runtime environments, credentials, sessions, generated submissions, hidden tests, and raw evaluation logs are excluded. Source publication is separate from benchmark execution and does not establish a score. After a full batch exits, the site-specific batch launcher publishes `curves.csv`, `summary.json`, `study-status.json`, and `usage-summary.json` under `results/beam-pi-programbench-20261009/`, if all four files exist. This also permits publishing honest partial/failure summaries after a nonzero controller exit. `plot_results.py` renders PNG, SVG, PDF, and Plotly JSON for all three latency axes from these sanitized aggregates; the same commit publishes these 12 allowlisted figures. Install `study/requirements-plotting.txt` in the controller environment. An available `recovery-report.json` is also sanitized and published, preserving raw and corrected artifact hashes, scores, counts, and zero API usage for CPU repairs. The Space's `/study` view shows interactive curves and download links while retaining the existing Trackio dashboard. It stays empty until graded aggregates exist. The batch skips publication in `--validate-only` mode and preserves the controller exit code. Upload failure is reported separately in `experiment/publish-status.json`; only `remote_verified=true` confirms file hashes read back at the resulting commit. ## Verification ```bash study/.venv/bin/python -m unittest discover -s study/tests -v npm --prefix study/pi-runner test PROGRAMBENCH_TEST_ROOT=/path/to/pinned/ProgramBench python -m pytest -q tests/test_programbench_adapter.py ``` Fake API tests make no paid inference calls. They cover concurrent reservations, failure/restart accounting, real Pi SDK tool calls and forced compaction, persistent message delivery/resumption, cutoff handling, scoring failures, tracking readback, and curve aggregation. Live cluster validation is separate evidence, recorded in run artifacts. Code in this study directory is MIT-licensed. Pi and ProgramBench retain their upstream licenses; benchmark program images, documentation, and test data retain their own attribution and licenses. This repository references pinned assets and does not relicense or redistribute them. See the upstream [ProgramBench repository](https://github.com/facebookresearch/ProgramBench) and [dataset attribution](https://huggingface.co/datasets/programbench/ProgramBench-Tests#attribution-and-licensing).