Spaces:
Running
A newer version of the Gradio SDK is available: 6.30.0
title: Beam + Pi ProgramBench
emoji: 🎯
sdk: gradio
app_file: app.py
tags:
- trackio
hf_oauth: true
hf_oauth_scopes:
- write-repos
Beam + Pi ProgramBench pilot
Live, numeric-only tracking for an exploratory ProgramBench study using Reflection Beam-501B-A23B and the open-source Pi coding agent. The current figure is a post-hoc four-task subset: jq, xsv, FIGlet, and hexyl, with four tasks in each of three conditions. Ripgrep is excluded consistently because grading a saved artifact was too slow; this exclusion was made after execution. All original 15 episode records, five-task CSV curves, failures, and token charges remain available.
Open the score-versus-latency view
or the project beam-pi-programbench-20261009 in the Trackio dashboard.
The study page shows only a title and the operational reference-clock plot. The
aggregate files and /study/data API retain all three clocks: operational reference,
observed wall time, and API-prefill sensitivity. Calibration and infrastructure
checks are never plotted as benchmark scores.
The recovery protocol documents the original failures, fixed-mask artifact rescoring, restricted retries, the cancelled CPU-only regrading attempts, and the disclosed task exclusion. Recovered curves retain prior attempt latency and charges under the original budgets. The aggregate data and API distinguish a valid final artifact from an earlier valid checkpoint and retain historical/current grading gaps.
The planned pilot compares a single agent, a fixed five-agent team, and an asynchronous team on ripgrep, jq, xsv, FIGlet, and hexyl, with one repetition per condition (15 benchmark episodes). Each episode shares an 8M billed API-token cap across all agents, a 1M estimated context-token cap, and a two-hour wall-clock cap. Benchmark API usage is limited to 120M tokens, with a 130M global ceiling including validation.
Runs beginning validation- are infrastructure checks, not benchmark results. Reference calibration and experiment status are reported separately. A missing score means no verified grading result is available; it does not mean zero or a successful run. The five tasks are a convenience sample, not a representative estimate over ProgramBench. This API-based pilot is inspired by Anthropic's multi-agent study, not an exact replication of its model, harness, token accounting, or reference-clock latency measurements.
Published fields are task and condition labels, numerical progress, token counts, elapsed time, status, and aggregate hidden-test scores. Prompts, generated code, reasoning, raw test output, hidden test assets, and credentials are not published. The controller retains a durable local event log; upload readback status is recorded separately, so an unavailable dashboard cannot turn a tracking failure into a claim of experiment success.
This Space runs Trackio 0.41.0 with a persistent Hugging Face Bucket. The MIT-licensed study harness source includes pinned configuration, tests, a sanitized recovery batch template, and a file-hash manifest. Deployment secrets, private machine paths, hidden assets, and raw experiment traces are excluded.
After a full study run, sanitized CSV curves, JSON summaries, and PNG/SVG/PDF figures are published to the
aggregate results directory.
The directory may be absent until aggregate files exist. Publication is separate
from benchmark success: consult study-status.json and grading coverage, including
failures and missing outcomes. Result uploads are read back at the committed revision
and hash-verified. Figures are rendered exclusively from the sanitized aggregate
data and use observed step curves, including regressions. No confidence bands are
shown for this pilot. Figure JSON contains the plotted four-task subset; downloadable
curves.csv and summary.json retain the original five-task evidence with explicit
analysis scope. The aggregate data and API mark all three excluded ripgrep episodes.
Live metrics remain available in Trackio while these files are prepared.
Reviewed provisional snapshots can also be published during recovery. These preserve
the running study status, failed episodes, missing terminal grades, and original
charges; publication is not a claim that recovery is complete.