---
title: Beam + Pi ProgramBench
emoji: 🎯
sdk: gradio
app_file: app.py
tags:
 - trackio
hf_oauth: true
hf_oauth_scopes:
 - write-repos
---

# Beam + Pi ProgramBench pilot

Live, numeric-only tracking for an exploratory ProgramBench study using
Reflection **Beam-501B-A23B** and the open-source **Pi** coding agent. The current
figure is a **post-hoc four-task subset**: jq, xsv, FIGlet, and hexyl, with four tasks
in each of three conditions. Ripgrep is excluded consistently because grading a
saved artifact was too slow; this exclusion was made after execution. All original
15 episode records, five-task CSV curves, failures, and token charges remain available.

Open the [score-versus-latency view](https://burtenshaw-beam-pi-programbench.hf.space/study)
or the project **beam-pi-programbench-20261009** in the Trackio dashboard.
The study page shows only a title and the operational reference-clock plot. The
aggregate files and `/study/data` API retain all three clocks: operational reference,
observed wall time, and API-prefill sensitivity. Calibration and infrastructure
checks are never plotted as benchmark scores.

The [recovery protocol](https://huggingface.co/spaces/burtenshaw/beam-pi-programbench/blob/main/source/study/RECOVERY.md)
documents the original failures, fixed-mask artifact rescoring, restricted retries,
the cancelled CPU-only regrading attempts, and the disclosed task exclusion. Recovered curves retain prior attempt
latency and charges under the original budgets. The aggregate data and API distinguish
a valid final artifact from an earlier valid checkpoint and retain historical/current
grading gaps.

The planned pilot compares a single agent, a fixed five-agent team, and an
asynchronous team on ripgrep, jq, xsv, FIGlet, and hexyl, with one repetition per
condition (15 benchmark episodes). Each episode shares an 8M billed API-token cap
across all agents, a 1M estimated context-token cap, and a two-hour wall-clock cap. Benchmark
API usage is limited to 120M tokens, with a 130M global ceiling including validation.

Runs beginning **validation-** are infrastructure checks, not benchmark results.
Reference calibration and experiment status are reported separately. A missing
score means no verified grading result is available; it does not mean zero or a
successful run. The five tasks are a convenience sample, not a representative
estimate over ProgramBench. This API-based pilot is inspired by Anthropic's
multi-agent study, not an exact replication of its model, harness, token accounting,
or reference-clock latency measurements.

Published fields are task and condition labels, numerical progress, token counts,
elapsed time, status, and aggregate hidden-test scores. Prompts, generated code,
reasoning, raw test output, hidden test assets, and credentials are not published.
The controller retains a durable local event log; upload readback status is recorded
separately, so an unavailable dashboard cannot turn a tracking failure into a claim
of experiment success.

This Space runs Trackio 0.41.0 with a persistent Hugging Face Bucket. The MIT-licensed
[study harness source](https://huggingface.co/spaces/burtenshaw/beam-pi-programbench/tree/main/source)
includes pinned configuration, tests, a sanitized recovery batch template, and a
file-hash manifest. Deployment secrets, private machine paths, hidden assets,
and raw experiment traces are excluded.

After a full study run, sanitized CSV curves, JSON summaries, and PNG/SVG/PDF figures are published to the
[aggregate results directory](https://huggingface.co/spaces/burtenshaw/beam-pi-programbench/tree/main/results/beam-pi-programbench-20261009).
The directory may be absent until aggregate files exist. Publication is separate
from benchmark success: consult `study-status.json` and grading coverage, including
failures and missing outcomes. Result uploads are read back at the committed revision
and hash-verified. Figures are rendered exclusively from the sanitized aggregate
data and use observed step curves, including regressions. No confidence bands are
shown for this pilot. Figure JSON contains the plotted four-task subset; downloadable
`curves.csv` and `summary.json` retain the original five-task evidence with explicit
analysis scope. The aggregate data and API mark all three excluded ripgrep episodes.
Live metrics remain available in Trackio while these files are prepared.
Reviewed provisional snapshots can also be published during recovery. These preserve
the running study status, failed episodes, missing terminal grades, and original
charges; publication is not a claim that recovery is complete.

## Sources

- [ProgramBench](https://github.com/facebookresearch/ProgramBench)
- [Pi](https://github.com/badlogic/pi-mono)
- [Trackio](https://github.com/gradio-app/trackio)
- [Reflection API](https://developers.reflection.ai/)