arxiv:2607.27167

SpecFirst: Behavioral Specification Elicitation as a First-Class Step in Agent-Based Program Synthesis from Scratch

Published on Jul 29

· Submitted by

Boyuan Chen on Jul 30

Centre for Software Excellence

Upvote

Authors:

Boyuan Chen ,

Abstract

LLM-based agents excel at software engineering tasks where an existing codebase provides context, but constructing a program from scratch remains fundamentally harder. Recent benchmarks such as ProgramBench quantify this gap: given only natural-language documentation and an execute-only binary as a behavioral oracle, even frontier models solve fewer than 1% of instances. Existing frameworks conflate documentation reading, behavioral exploration, and code synthesis into a single pass, causing agents to probe insufficiently, lose behavioral intent as context drifts, and propagate early misinterpretations into the final implementation. Inspired by classical requirements engineering, we argue that behavioral specification elicitation should be a first-class phase that precedes implementation. We present SpecFirst, a two-stage framework that forces the specification elicitation before code synthesis. A dedicated spec agent first probes the binary and combines observations with documentation into a structured specification. Next, a code synthesis agent then uses this specification to drive implementation. This decomposition resolves documentation ambiguities before coding begins and provides a stable behavioral reference throughout synthesis. We evaluate SpecFirst on all 200 ProgramBench instances across four models spanning two families and an order of magnitude of capability. SpecFirst consistently outperforms the single-loop baseline, improving test pass rates by 6.9%-21.3% and binary exploration coverage by 9.4%-18.5%, all statistically significant. Behavioral analysis on code synthesis further shows that a prior specification enables earlier and more sustained code construction. Our results demonstrate that an explicit requirements-engineering phase is an effective paradigm for from-scratch program construction.

View arXiv page View PDF Add to collection

Community

nemocbb

Paper author Paper submitter 24 minutes ago

SpecFirst: Behavioral Specification Elicitation as a First-Class Step in Agent-Based Program Synthesis from Scratch

We believe behavioral specification elicitation should be a first-class step in program synthesis from scratch, rather than being implicitly mixed with exploration and coding.
With SpecFirst, we:

📋 Separate synthesis into two stages: a specification agent first probes an execute-only binary and combines its observations with the documentation into a structured behavioral specification
📋 Give a second agent this specification as a stable reference for implementation, instead of requiring it to continually recover behavioral intent from a growing interaction history
📋 Resolve documentation ambiguities before coding begins and decouple behavioral exploration from code construction

We evaluate SpecFirst on all 200 ProgramBench instances, using four models across two model families spanning nearly an order of magnitude in agentic coding capability. We compare it against a controlled single-loop Direct-Synthesis baseline using the same model, official mini-SWE-agent scaffold, and otherwise matched experimental settings.

The result:
📈 Improves test pass rates by 6.9%–21.3% over Direct-Synthesis across all four evaluated models
📈 Increases binary exploration coverage by 9.4%–18.5%, showing that the dedicated specification phase leads to more extensive behavioral probing
📈 Produces statistically significant improvements across every evaluated model, suggesting that the benefit extends across model families and capability levels

Perhaps most interesting: the specification changes how agents code. Behavioral analysis shows that agents equipped with a prior specification begin code construction earlier and sustain it for longer, rather than repeatedly switching between behavioral exploration and implementation.
Our results suggest that introducing an explicit requirements-engineering phase into the agent loop is a simple but effective paradigm for building programs from scratch.