source/README.md

Beam + Pi ProgramBench pilot source

MIT-licensed experiment harness for an originally five-task comparison of a single Pi agent, five persistent peers, and an asynchronous lead with persistent children using Reflection Beam-501B-A23B. The hosted model is an external API dependency; this release does not redistribute model weights or ProgramBench program/test assets.

The accepted pilot has 15 episodes, 8M billed API tokens per episode, a 120M benchmark allowance, and a 130M global ceiling including validation. Each episode has a two-hour wall limit and a 1M estimated context-token limit. See study/README.md for caveats, methodology, and file boundaries. The five selected tasks are a convenience sample with one repetition per condition. The current figure is an explicitly post-hoc four-task subset (jq, xsv, FIGlet, hexyl), excluding ripgrep across all conditions because of slow saved-artifact grading. All 15 original episode records, original five-task aggregates, failures, and ledger charges remain preserved; plotted means and coverage use four tasks.

The recovery protocol documents the original failures, limited HTTP 429 retries, reviewed fixed-mask artifact corrections, evaluator resource changes, and the post-hoc exclusion. Prior attempts retain their time and token/context charges. The aggregate data and API distinguish terminal-artifact validity from an earlier valid checkpoint and retain historical and current grading gaps.

Setup

Use Linux, Node 22+, Python 3.11+, uv, and a Docker-compatible runtime that supports run/exec/copy/commit and enforces the intended isolation. Install the controller and Pi dependencies in isolated environments:

uv venv .venv --python 3.11
uv pip install --python .venv/bin/python -r study/requirements-tracking.txt -r study/requirements-plotting.txt
npm ci --prefix study/pi-runner
git clone https://github.com/facebookresearch/ProgramBench vendor/ProgramBench
git -C vendor/ProgramBench checkout 27f02157c785f8da3647aa6dbbe6b9137f99f10e
(cd vendor/ProgramBench && uv sync --frozen)

Build the offline evaluator dependencies before calibration, and keep this export in the environment of both calibration and controller processes:

.venv/bin/python study/evaluator-wheelhouse.py --root "$PWD" --uv uv
export PROGRAMBENCH_EVALUATOR_WHEELHOUSE="$PWD/runtime/evaluator-wheelhouse"

The builder needs network access and records pinned wheel hashes. Evaluator containers install only from this cache with network disabled; solver containers never receive it. Calibration and grading must use the identical cache digest.

Pre-pull the five immutable images listed in the manifest. Keep the official evaluator checkout, test cache, and manifest outside every solver container. See study/PROGRAMBENCH.md for calibration and grading. Configure a Space and project you can write to in study/config.json, and provision the three private credential files described in the study README. Run reference calibration first, then invoke the controller:

.venv/bin/python study/run_study.py \
  --programbench-root vendor/ProgramBench \
  --evaluator-python vendor/ProgramBench/.venv/bin/python \
  --calibration calibration/calibration.json \
  --output-dir experiment --secrets-dir /path/to/private/secrets

Add --validate-only for synthetic infrastructure validation before the benchmark. The controller refuses inference if the exact-manifest reference calibration is missing or failed. Local tests make no hosted model calls:

python3 -m unittest discover -s study/tests -v
npm --prefix study/pi-runner test
PROGRAMBENCH_TEST_ROOT=vendor/ProgramBench vendor/ProgramBench/.venv/bin/python \
  -m pytest -q tests/test_programbench_adapter.py

Result publication

When all four aggregate files exist, publish a reviewed snapshot:

.venv/bin/python study/publish_results.py --experiment-dir experiment \
  --hf-token-file /path/to/private/secrets/hf-token --space-id OWNER/SPACE

Sanitized curves.csv, summary.json, study-status.json, and usage-summary.json are committed under results/beam-pi-programbench-20261009, together with 12 PNG/SVG/PDF/Plotly JSON figures rendered only from those aggregates. When recovery provenance exists, the same commit includes sanitized recovery-report.json with raw/corrected scores, counts, and artifact hashes. Figure JSON contains the plotted four-task subset and explicit analysis-scope metadata; curves.csv and summary.json preserve original five-task evidence and excluded failures. The Space study view at /study shows only a title and the reference-clock plot. All three clocks and grading coverage remain available in the aggregate files and /study/data API, while Trackio remains available at its existing routes. Before benchmark grades exist, the chart shows an empty state. Every uploaded file is fetched at the commit revision and hash-checked. Inspect local publish-status.json; a completed upload request alone is not verification. Provisional snapshots may be published during recovery and retain a running study status, failed episode statuses, and missing final-artifact grades. No source installation, calibration gate, or infrastructure validation should be interpreted as a successful benchmark result.

Provenance and licenses

SOURCE_MANIFEST.json records this release's explicit file allowlist and hashes. The included recovery batch is a template with cluster account, partition, and root replaced. The targeted recovery helper and tests also replace the private audited module identifier; authorized local audit metadata is required to apply it to the saved private artifacts. These public templates omit private audit configuration and are not hash-identical to the executable helper used for the study. These transformations are recorded in the manifest. Other site-specific deployment scripts, runtime environments, credentials, traces, submissions, hidden tests, and raw evaluation logs are excluded. study/LICENSE applies to the study code. Upstream Pi and ProgramBench keep their own licenses; referenced program images/test assets retain their upstream licensing and attribution.