Spaces:
Running
Running
| # Beam + Pi ProgramBench pilot source | |
| MIT-licensed experiment harness for an originally five-task comparison of a single Pi agent, | |
| five persistent peers, and an asynchronous lead with persistent children using | |
| Reflection Beam-501B-A23B. The hosted model is an external API dependency; this | |
| release does not redistribute model weights or ProgramBench program/test assets. | |
| The accepted pilot has 15 episodes, 8M billed API tokens per episode, a 120M | |
| benchmark allowance, and a 130M global ceiling including validation. Each episode | |
| has a two-hour wall limit and a 1M estimated context-token limit. See | |
| [study/README.md](study/README.md) for caveats, methodology, and file boundaries. | |
| The five selected tasks are a convenience sample with one repetition per condition. | |
| The current figure is an explicitly **post-hoc four-task subset** (jq, xsv, FIGlet, | |
| hexyl), excluding ripgrep across all conditions because of slow saved-artifact | |
| grading. All 15 original episode records, original five-task aggregates, failures, | |
| and ledger charges remain preserved; plotted means and coverage use four tasks. | |
| The [recovery protocol](study/RECOVERY.md) documents the original failures, limited | |
| HTTP 429 retries, reviewed fixed-mask artifact corrections, evaluator resource | |
| changes, and the post-hoc exclusion. Prior attempts retain their time and token/context | |
| charges. The aggregate data and API distinguish terminal-artifact validity from an | |
| earlier valid checkpoint and retain historical and current grading gaps. | |
| ## Setup | |
| Use Linux, Node 22+, Python 3.11+, uv, and a Docker-compatible runtime that supports | |
| run/exec/copy/commit and enforces the intended isolation. Install the controller | |
| and Pi dependencies in isolated environments: | |
| ```sh | |
| uv venv .venv --python 3.11 | |
| uv pip install --python .venv/bin/python -r study/requirements-tracking.txt -r study/requirements-plotting.txt | |
| npm ci --prefix study/pi-runner | |
| git clone https://github.com/facebookresearch/ProgramBench vendor/ProgramBench | |
| git -C vendor/ProgramBench checkout 27f02157c785f8da3647aa6dbbe6b9137f99f10e | |
| (cd vendor/ProgramBench && uv sync --frozen) | |
| ``` | |
| Build the offline evaluator dependencies before calibration, and keep this | |
| export in the environment of both calibration and controller processes: | |
| ```sh | |
| .venv/bin/python study/evaluator-wheelhouse.py --root "$PWD" --uv uv | |
| export PROGRAMBENCH_EVALUATOR_WHEELHOUSE="$PWD/runtime/evaluator-wheelhouse" | |
| ``` | |
| The builder needs network access and records pinned wheel hashes. Evaluator | |
| containers install only from this cache with network disabled; solver containers | |
| never receive it. Calibration and grading must use the identical cache digest. | |
| Pre-pull the five immutable images listed in the manifest. Keep the official | |
| evaluator checkout, test cache, and manifest outside every solver container. | |
| See [study/PROGRAMBENCH.md](study/PROGRAMBENCH.md) for calibration and grading. | |
| Configure a Space and project you can write to in `study/config.json`, and provision | |
| the three private credential files described in the study README. Run reference | |
| calibration first, then invoke the controller: | |
| ```sh | |
| .venv/bin/python study/run_study.py \ | |
| --programbench-root vendor/ProgramBench \ | |
| --evaluator-python vendor/ProgramBench/.venv/bin/python \ | |
| --calibration calibration/calibration.json \ | |
| --output-dir experiment --secrets-dir /path/to/private/secrets | |
| ``` | |
| Add `--validate-only` for synthetic infrastructure validation before the benchmark. | |
| The controller refuses inference if the exact-manifest reference calibration is | |
| missing or failed. Local tests make no hosted model calls: | |
| ```sh | |
| python3 -m unittest discover -s study/tests -v | |
| npm --prefix study/pi-runner test | |
| PROGRAMBENCH_TEST_ROOT=vendor/ProgramBench vendor/ProgramBench/.venv/bin/python \ | |
| -m pytest -q tests/test_programbench_adapter.py | |
| ``` | |
| ## Result publication | |
| When all four aggregate files exist, publish a reviewed snapshot: | |
| ```sh | |
| .venv/bin/python study/publish_results.py --experiment-dir experiment \ | |
| --hf-token-file /path/to/private/secrets/hf-token --space-id OWNER/SPACE | |
| ``` | |
| Sanitized `curves.csv`, `summary.json`, `study-status.json`, and | |
| `usage-summary.json` are committed under `results/beam-pi-programbench-20261009`, | |
| together with 12 PNG/SVG/PDF/Plotly JSON figures rendered only from those aggregates. | |
| When recovery provenance exists, the same commit includes sanitized | |
| `recovery-report.json` with raw/corrected scores, counts, and artifact hashes. | |
| Figure JSON contains the plotted four-task subset and explicit analysis-scope metadata; | |
| `curves.csv` and `summary.json` preserve original five-task evidence and excluded failures. | |
| The Space study view at `/study` shows only a title and the reference-clock plot. | |
| All three clocks and grading coverage remain available in the aggregate files and | |
| `/study/data` API, while Trackio remains available at its existing routes. Before | |
| benchmark grades exist, the chart shows an empty state. | |
| Every uploaded file is fetched at the commit revision and hash-checked. Inspect | |
| local `publish-status.json`; a completed upload request alone is not verification. | |
| Provisional snapshots may be published during recovery and retain a running study | |
| status, failed episode statuses, and missing final-artifact grades. | |
| No source installation, calibration gate, or infrastructure validation should be | |
| interpreted as a successful benchmark result. | |
| ## Provenance and licenses | |
| `SOURCE_MANIFEST.json` records this release's explicit file allowlist and hashes. | |
| The included recovery batch is a template with cluster account, partition, and root | |
| replaced. The targeted recovery helper and tests also replace the private audited | |
| module identifier; authorized local audit metadata is required to apply it to the | |
| saved private artifacts. These public templates omit private audit configuration | |
| and are not hash-identical to the executable helper used for the study. | |
| These transformations are recorded in the manifest. | |
| Other site-specific deployment scripts, runtime environments, credentials, traces, | |
| submissions, hidden tests, and raw evaluation logs are excluded. | |
| [study/LICENSE](study/LICENSE) applies to the study code. Upstream Pi and ProgramBench | |
| keep their own licenses; referenced program images/test assets retain their | |
| upstream licensing and attribution. | |