Spaces:
Running
Running
| title: Beam + Pi ProgramBench | |
| emoji: 🎯 | |
| sdk: gradio | |
| app_file: app.py | |
| tags: | |
| - trackio | |
| hf_oauth: true | |
| hf_oauth_scopes: | |
| - write-repos | |
| # Beam + Pi ProgramBench pilot | |
| Live, numeric-only tracking for an exploratory ProgramBench study using | |
| Reflection **Beam-501B-A23B** and the open-source **Pi** coding agent. The current | |
| figure is a **post-hoc four-task subset**: jq, xsv, FIGlet, and hexyl, with four tasks | |
| in each of three conditions. Ripgrep is excluded consistently because grading a | |
| saved artifact was too slow; this exclusion was made after execution. All original | |
| 15 episode records, five-task CSV curves, failures, and token charges remain available. | |
| Open the [score-versus-latency view](https://burtenshaw-beam-pi-programbench.hf.space/study) | |
| or the project **beam-pi-programbench-20261009** in the Trackio dashboard. | |
| The study page shows only a title and the operational reference-clock plot. The | |
| aggregate files and `/study/data` API retain all three clocks: operational reference, | |
| observed wall time, and API-prefill sensitivity. Calibration and infrastructure | |
| checks are never plotted as benchmark scores. | |
| The [recovery protocol](https://huggingface.co/spaces/burtenshaw/beam-pi-programbench/blob/main/source/study/RECOVERY.md) | |
| documents the original failures, fixed-mask artifact rescoring, restricted retries, | |
| the cancelled CPU-only regrading attempts, and the disclosed task exclusion. Recovered curves retain prior attempt | |
| latency and charges under the original budgets. The aggregate data and API distinguish | |
| a valid final artifact from an earlier valid checkpoint and retain historical/current | |
| grading gaps. | |
| The planned pilot compares a single agent, a fixed five-agent team, and an | |
| asynchronous team on ripgrep, jq, xsv, FIGlet, and hexyl, with one repetition per | |
| condition (15 benchmark episodes). Each episode shares an 8M billed API-token cap | |
| across all agents, a 1M estimated context-token cap, and a two-hour wall-clock cap. Benchmark | |
| API usage is limited to 120M tokens, with a 130M global ceiling including validation. | |
| Runs beginning **validation-** are infrastructure checks, not benchmark results. | |
| Reference calibration and experiment status are reported separately. A missing | |
| score means no verified grading result is available; it does not mean zero or a | |
| successful run. The five tasks are a convenience sample, not a representative | |
| estimate over ProgramBench. This API-based pilot is inspired by Anthropic's | |
| multi-agent study, not an exact replication of its model, harness, token accounting, | |
| or reference-clock latency measurements. | |
| Published fields are task and condition labels, numerical progress, token counts, | |
| elapsed time, status, and aggregate hidden-test scores. Prompts, generated code, | |
| reasoning, raw test output, hidden test assets, and credentials are not published. | |
| The controller retains a durable local event log; upload readback status is recorded | |
| separately, so an unavailable dashboard cannot turn a tracking failure into a claim | |
| of experiment success. | |
| This Space runs Trackio 0.41.0 with a persistent Hugging Face Bucket. The MIT-licensed | |
| [study harness source](https://huggingface.co/spaces/burtenshaw/beam-pi-programbench/tree/main/source) | |
| includes pinned configuration, tests, a sanitized recovery batch template, and a | |
| file-hash manifest. Deployment secrets, private machine paths, hidden assets, | |
| and raw experiment traces are excluded. | |
| After a full study run, sanitized CSV curves, JSON summaries, and PNG/SVG/PDF figures are published to the | |
| [aggregate results directory](https://huggingface.co/spaces/burtenshaw/beam-pi-programbench/tree/main/results/beam-pi-programbench-20261009). | |
| The directory may be absent until aggregate files exist. Publication is separate | |
| from benchmark success: consult `study-status.json` and grading coverage, including | |
| failures and missing outcomes. Result uploads are read back at the committed revision | |
| and hash-verified. Figures are rendered exclusively from the sanitized aggregate | |
| data and use observed step curves, including regressions. No confidence bands are | |
| shown for this pilot. Figure JSON contains the plotted four-task subset; downloadable | |
| `curves.csv` and `summary.json` retain the original five-task evidence with explicit | |
| analysis scope. The aggregate data and API mark all three excluded ripgrep episodes. | |
| Live metrics remain available in Trackio while these files are prepared. | |
| Reviewed provisional snapshots can also be published during recovery. These preserve | |
| the running study status, failed episodes, missing terminal grades, and original | |
| charges; publication is not a claim that recovery is complete. | |
| ## Sources | |
| - [ProgramBench](https://github.com/facebookresearch/ProgramBench) | |
| - [Pi](https://github.com/badlogic/pi-mono) | |
| - [Trackio](https://github.com/gradio-app/trackio) | |
| - [Reflection API](https://developers.reflection.ai/) | |