Papers
arxiv:2611.20882

Accelerate Bench: Evaluating Agents for Application-Level Program Optimization

Published on Nov 16
Β· Submitted by
Bowen LIU
on Nov 17
Authors:
,
,
,
,
,
,
,
,
,

Abstract

Application optimization requires coding agents to discover useful changes and coordinate them across a complete program while preserving the required computation. We introduce Accelerate Bench, a benchmark of 32 CPU optimization tasks adapted from scientific applications and supercomputing competitions. Each task provides a working program and specifies the workload, correctness requirements, timing boundaries, and permitted edits, leaving agents to choose where and how to optimize. Public testing supports iterative development, and a separate verifier scores the final submission. On benchmark version 1.1, we evaluate 16 coding agents, each pairing a model with a harness. Their Score@1 ranges from 25.5% to 67.4%. Most of their programs are correct, and the size of the speedup separates them: the two leading agents are at least 2x faster on 22 or more of the 32 tasks, while the other fourteen reach 2x on 8 to 17 and leave 125 of their 412 correct outcomes below 1.25x. Controlled replays show why coordination across an application matters: identical source code runs between 0.29x and 2.12x depending only on build and runtime settings, and one code change is worth 1.01x or 6.4x depending on another. Accelerate Bench pairs executable workloads with agents' programs and trajectories to evaluate both their gains and the process behind them.

Community

Hi Hugging Face community! We’re introducing Accelerate Bench, a benchmark for evaluating coding agents on application-level program optimization.

πŸ§ͺ 32 scientific workloads across 8 application areas
Agents optimize complete CPU applications, including GROMACS, OpenFOAM, and Quantum ESPRESSO. Each task provides a working program, a fixed workload, correctness requirements, and a compute budget. Agents have eight hours to decide where and how to optimize.

βœ… Performance gains must preserve correctness
Public tests support iterative development. A separate verifier rebuilds the final submission, checks hidden inputs, and measures its runtime against the original program.

πŸ“Š 16 coding agents evaluated
The two leading agents achieve at least 2Γ— speedups on 22 or more of the 32 tasks. Controlled experiments show why application-level coordination matters: the same source code can perform very differently under different build and runtime settings, and the benefit of one optimization can depend strongly on another.

πŸ› οΈ Try it with your own agent
31 tasks are available directly through Harbor Hub, with all 32 available through the full release on Hugging Face.

We welcome evaluations, reproductions, and discussion from the community!

🌐 Project: https://acceleratebench.com/
πŸ€— Dataset: https://huggingface.co/datasets/AdAstraAbyssoque/accelerate-bench
πŸš€ Harbor Hub: https://hub.harborframework.com/datasets/accelerate-bench/accelerate-bench

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2611.20882
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.20882 in a model README.md to link it from this page.

Datasets citing this paper 1

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.20882 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.