AuXSpec-27B-RL
AuXSpec: Learning to Autoformalize Program Specifications via Executable Postconditions
Haocheng Wang2,1,*, Weixuan Yuan1,*, Florian Hübler1, Yinya Huang1,4, Zhijiang Guo2,3, Sorrachai Yingchareonthawornchai5, Rasmus Kyng1,†
1 Department of Computer Science, ETH Zurich 2 HKUST(GZ) 3 HKUST 4 ETH AI Center 5 Institute for Theoretical Studies, ETH Zurich
* Equal contribution. † Corresponding author.
Paper | Website | Collection | License | Citation
1. Introduction
Formal specifications define intended program behavior and underpin verification, but writing them in theorem provers such as Lean is slow and expertise-intensive, making specification a major bottleneck to scaling formally correct code. Autoformalization from natural-language problems is promising, yet training specification models requires scalable feedback on whether generated specifications capture intended behavior. Prior approaches obtain this feedback through explicit proofs or separately implemented code with formal correspondence proofs, both expensive.
We introduce AuXSpec, a Lean framework that pairs declarative postconditions with executable counterparts through typeclass resolution. Test cases can then evaluate declarative specifications by execution, so their agreement serves as a reinforcement-learning reward. With AuXSpec, we synthesize 3,037 LeetCode problems with proved equivalence theorems, expand the SFT data to 19.5K samples through expert iteration on TACO, and run RL on 13.6K further SpecGen and CodeGen tasks. Training Qwen3.8-27B and Qwen3.5-9B with the execution-based reward improves SpecGen pass@4 on AuXSpec-Test (14.3% → 84.3% and 0.7% → 82.7%). On the out-of-domain Verina benchmark with test cases in the prompt, SpecGen pass@4 improves (55.0% → 96.8% and 18.5% → 96.8%). Without test cases in the prompt, the setting reported for VeriSpecGen-30B-A3B, the prior state of the art among open-source specification models, the 27B model outperforms it in pass@1 (84.5% vs. 47.6%).
AuXSpec-27B-RL is Qwen3.8-27B trained with supervised fine-tuning on these samples and then reinforcement learning with the execution-based reward. Given a natural-language programming problem, it generates a complete Lean 4 file (Lean 4.32, Mathlib) for one of four tasks:
- SpecGen: a validity predicate on inputs, a declarative specification of the correct output, an executable checker, and
#guardtest cases. - CodeGen: an implementation of a given specification.
- SpecGen-Repair and CodeGen-Repair: a fixed file, given a failing one and the checker's feedback.
The model reasons inside <think>...</think> before the answer.
2. Model Summary
Executable specification format
- Each declarative postcondition is paired with an executable Boolean postcondition on input-output pairs, obtained through typeclass resolution in Lean when a decidability instance exists and defined separately otherwise.
- An equivalence theorem proves that both postconditions agree on valid inputs, so executing the checker on a test case decides the declarative postcondition, and the specification stays independent of any implementation.
Supervised fine-tuning
- 19.5K training samples grown over 3 rounds of expert iteration on LeetCode and TACO problems. Each round fine-tunes the model from the base model on the accumulated samples and adds the outputs that meet the acceptance criteria.
Reinforcement learning
- RL on 8,281 SpecGen and 5,356 CodeGen tasks, with a progressive reward that follows the order of the acceptance criteria: the file compiles, the guards reproduce the given test cases, and judge-based quality checks pass. Only tests that call a declaration of the output count, outputs that assume unproved facts or bypass the kernel check count as compilation failures, and unchanged repair outputs receive 0.
- Repair tasks are built online from failed rollouts of the current policy, paired with compiler messages, and fill 40% of each batch.
- The algorithm extends DAPO with batch-level advantage normalization, a KL penalty toward the fine-tuned model, and a loss mask on repetitive rollouts. For CodeGen, a correct implementation that an implementation judge finds asymptotically slower than a known algorithm on the largest admitted input earns a reduced reward, so the model learns to generate efficient code. Overlong responses are penalized near the length limit.
3. Model Downloads
| Model | Base Model | #Total Params | Tensor Type | Evaluated Context | Download |
|---|---|---|---|---|---|
| AuXSpec-9B-RL | Qwen3.5-9B | 9.4B | BF16 | 32K | 🤗 Hugging Face |
| AuXSpec-27B-RL | Qwen3.8-27B | 27.4B | BF16 | 32K | 🤗 Hugging Face |
Both models keep the architecture and tokenizer of their base models (Qwen3_5ForConditionalGeneration) and are stored as BF16 safetensors. All evaluations in Section 4 run in vLLM with max_model_len=32768 and max_tokens=16384.
4. Evaluation Results
In each column, bold marks the best value and underline the second best. The small number next to our results is the change from the backbone model, green for a gain and red for a drop. All results draw 4 samples per problem. Our models are sampled at temperature 0.3, top-p 0.95, top-k 20, presence penalty 1.0. Other models follow their recommended settings. pass@1 is averaged over the 4 samples and pass@4 counts a problem as solved if any sample passes.
4.1 AuXSpec-Test (in domain)
We evaluate on AuXSpec-Test (the test split of the AuXSpec dataset), a benchmark of 300 LeetCode problems with 72 easy, 155 medium and 73 hard problems, released with the paper. Every reference specification, implementation and test case was checked by hand against the problem statement, and the correctness theorem is proved for every problem.

Full results table
| Model | SpecGen | CodeGen | SpecGen-Repair | CodeGen-Repair | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| p@1 | p@4 | p@1 | p@4 | p@1 | p@4 | p@1 | p@4 | |||||||||
| Frontier models | ||||||||||||||||
| Claude-Opus-5 | 57.7 | 81.3 | 93.0 | 98.3 | 83.0 | 90.1 | 86.9 | 95.1 | ||||||||
| Claude-Sonnet-5 | 13.8 | 39.3 | 54.8 | 73.0 | 66.4 | 81.7 | 77.2 | 88.3 | ||||||||
| GPT-5.6-Sol | 29.8 | 59.7 | 80.0 | 96.3 | 61.8 | 77.1 | 82.3 | 89.3 | ||||||||
| GPT-5.6-Luna | 9.5 | 28.7 | 39.9 | 71.3 | 49.4 | 69.5 | 71.8 | 87.4 | ||||||||
| Kimi-K3 | 43.8 | 83.7 | 67.2 | 94.3 | 73.5 | 87.8 | 81.3 | 97.1 | ||||||||
| DeepSeek-V4-Pro | 21.2 | 54.3 | 48.6 | 72.3 | 57.6 | 82.4 | 76.9 | 94.2 | ||||||||
| DeepSeek-V4-Flash (0731) | 38.4 | 76.7 | 45.0 | 69.7 | 67.0 | 88.5 | 73.3 | 95.1 | ||||||||
| Backbone models | ||||||||||||||||
| Qwen3.5-9B | 0.2 | 0.7 | 8.2 | 18.0 | 8.4 | 26.0 | 4.9 | 14.6 | ||||||||
| Qwen3.8-27B | 4.5 | 14.3 | 8.9 | 22.0 | 44.7 | 73.3 | 35.7 | 64.1 | ||||||||
| Ours | ||||||||||||||||
| AuXSpec-9B-SFT | 10.1 | +9.9 | 24.3 | +23.6 | 13.1 | +4.9 | 28.3 | +10.3 | 56.7 | +48.3 | 76.3 | +50.3 | 59.5 | +54.6 | 80.6 | +66.0 |
| AuXSpec-9B-RL | 58.6 | +58.4 | 82.7 | +82.0 | 77.1 | +68.9 | 96.0 | +78.0 | 73.5 | +65.1 | 84.7 | +58.7 | 74.5 | +69.6 | 94.2 | +79.6 |
| AuXSpec-27B-SFT | 18.7 | +14.2 | 40.7 | +26.4 | 27.6 | +18.7 | 52.7 | +30.7 | 73.1 | +28.4 | 84.7 | +11.4 | 72.3 | +36.6 | 87.4 | +23.3 |
| AuXSpec-27B-RL | 65.7 | +61.2 | 84.3 | +70.0 | 94.8 | +85.9 | 99.7 | +77.7 | 81.9 | +37.2 | 89.3 | +16.0 | 88.1 | +52.4 | 95.1 | +31.0 |
The repair tasks of the frontier models were run with an earlier version of the repair prompts.
CodeGen efficiency. To assess the efficiency of generated implementations, we report a stricter fast only score next to the any speed score above, which accepts correct but slow implementations. Fast only counts an implementation as wrong when the implementation judge finds a known algorithm faster by more than a log factor and the implementation takes over about 10^7 steps on the largest admitted input. Δ is fast only minus any speed.
| Model | p@1 | p@4 | ||||
|---|---|---|---|---|---|---|
| any speed | fast only | Δ | any speed | fast only | Δ | |
| Frontier models | ||||||
| Claude-Opus-5 | 93.0 | 90.3 | −2.7 | 98.3 | 98.0 | −0.3 |
| Claude-Sonnet-5 | 54.8 | 44.0 | −10.8 | 73.0 | 59.0 | −14.0 |
| GPT-5.6-Sol | 80.0 | 77.7 | −2.3 | 96.3 | 95.3 | −1.0 |
| GPT-5.6-Luna | 39.9 | 35.4 | −4.5 | 71.3 | 63.0 | −8.3 |
| Kimi-K3 | 67.2 | 62.8 | −4.4 | 94.3 | 91.0 | −3.3 |
| DeepSeek-V4-Pro | 48.6 | 46.1 | −2.5 | 72.3 | 68.0 | −4.3 |
| DeepSeek-V4-Flash (0731) | 45.0 | 43.1 | −1.9 | 69.7 | 67.0 | −2.7 |
| Backbone models | ||||||
| Qwen3.5-9B | 8.2 | 7.5 | −0.7 | 18.0 | 16.0 | −2.0 |
| Qwen3.8-27B | 8.9 | 8.8 | −0.1 | 22.0 | 21.7 | −0.3 |
| Ours | ||||||
| AuXSpec-9B-SFT | 13.1 | 11.8 | −1.3 | 28.3 | 24.3 | −4.0 |
| AuXSpec-9B-RL | 77.1 | 44.7 | −32.4 | 96.0 | 65.0 | −31.0 |
| AuXSpec-27B-SFT | 27.6 | 22.1 | −5.5 | 52.7 | 41.0 | −11.7 |
| AuXSpec-27B-RL | 94.8 | 88.0 | −6.8 | 99.7 | 97.7 | −2.0 |
4.2 Verina (out of domain)
To assess generalization beyond the synthesized data, we evaluate on Verina, a 189-problem manually curated benchmark with reference specifications, implementations and test cases. Following the original evaluation, generated specifications must be sound and complete with respect to the reference specification on the test cases, and generated implementations must pass all positive test cases. We include the test cases in the prompt, which matches the training inputs and is the inference setting we recommend. †VeriSpecGen is not released, so the VeriSpecGen rows and the rows of its backbones are taken from the VeriSpecGen paper and use that paper's setting (official Verina harness, no test cases in the prompt, second number pass@10 in italic).

Full results table
| Model | SpecGen | CodeGen | ||||||
|---|---|---|---|---|---|---|---|---|
| p@1 | p@4 | p@1 | p@4 | |||||
| Frontier models | ||||||||
| Claude-Opus-5 | 86.2 | 95.2 | 56.6 | 79.4 | ||||
| Claude-Sonnet-5 | 64.8 | 90.5 | 17.9 | 41.8 | ||||
| GPT-5.6-Sol | 85.1 | 97.9 | 20.8 | 40.7 | ||||
| GPT-5.6-Luna | 67.3 | 94.7 | 24.5 | 58.2 | ||||
| Kimi-K3 | 77.6 | 96.3 | 34.9 | 72.5 | ||||
| DeepSeek-V4-Pro | 53.8 | 82.5 | 37.6 | 70.4 | ||||
| DeepSeek-V4-Flash (0731) | 60.2 | 92.6 | 5.8 | 18.5 | ||||
| Backbone models | ||||||||
| Qwen3-4B-Instruct-2507† | 14.7 | 16.3 | 25.3 | 28.3 | ||||
| Qwen3-Coder-30B-A3B† | 29.4 | 41.7 | 39.7 | 53.5 | ||||
| Qwen3.5-9B | 7.0 | 18.5 | 23.4 | 49.2 | ||||
| Qwen3.8-27B | 28.8 | 55.0 | 3.3 | 13.2 | ||||
| Prior work | ||||||||
| VeriSpecGen-4B† | 30.3 | 45.2 | 35.4 | 66.8 | ||||
| VeriSpecGen-30B-A3B† | 47.6 | 63.9 | 58.7 | 84.0 | ||||
| Ours | ||||||||
| AuXSpec-9B-SFT | 37.3 | +30.3 | 69.8 | +51.3 | 38.0 | +14.6 | 70.4 | +21.2 |
| AuXSpec-9B-RL | 87.0 | +80.0 | 96.8 | +78.3 | 92.6 | +69.2 | 98.4 | +49.2 |
| AuXSpec-27B-SFT | 57.3 | +28.5 | 82.5 | +27.5 | 57.8 | +54.5 | 86.8 | +73.6 |
| AuXSpec-27B-RL | 91.9 | +63.1 | 96.8 | +41.8 | 91.8 | +88.5 | 93.7 | +80.5 |
4.3 Token efficiency
AuXSpec-27B-RL attains the highest SpecGen pass@1 on AuXSpec-Test (65.7%) with the fewest output tokens per response (1.8K). Claude-Opus-5 attains 57.7% with 9.7K tokens per response, 5.5× as many. Per solved sample, AuXSpec-27B-RL generates 2.7K tokens, the fewest of all 13 models and 80% fewer than GPT-5.6-Sol (13.2K). The small numbers in the table give each value as a multiple of the value of AuXSpec-27B-RL.

Full results table
| Model | SpecGen p@1 | Tokens per response | Tokens per solved sample |
|---|---|---|---|
| Frontier models | |||
| Claude-Opus-5 | 57.7 | 9.7K (5.5×) | 16.8K (6.2×) |
| Claude-Sonnet-5 | 13.8 | 10.2K (5.8×) | 74.4K (27.7×) |
| GPT-5.6-Sol | 29.8 | 3.9K (2.2×) | 13.2K (4.9×) |
| GPT-5.6-Luna | 9.5 | 3.4K (1.9×) | 36.0K (13.4×) |
| Kimi-K3 | 43.8 | 10.7K (6.1×) | 24.4K (9.1×) |
| DeepSeek-V4-Pro | 21.2 | 27.0K (15.3×) | 126.9K (47.3×) |
| DeepSeek-V4-Flash (0731) | 38.4 | 49.6K (28.2×) | 129.2K (48.1×) |
| Backbone models | |||
| Qwen3.5-9B | 0.2 | 3.2K (1.8×) | 1,912.1K (712.7×) |
| Qwen3.8-27B | 4.5 | 13.4K (7.6×) | 296.8K (110.6×) |
| Ours | |||
| AuXSpec-9B-RL | 58.6 | 6.4K (3.7×) | 11.0K (4.1×) |
| AuXSpec-27B-RL | 65.7 | 1.8K | 2.7K |
4.4 General capabilities
To assess capabilities outside formal program specification, we evaluate on (1) math with AIME 2025, (2) function-level code with HumanEval+ and MBPP+, (3) instruction following with IFEval and (4) graduate-level science with GPQA-Diamond. Following DeepSeek-R1, we sample with temperature 0.6 and top-p 0.95 up to 32,768 tokens and report pass@1, averaged over 64 samples per problem on AIME and 8 on GPQA-Diamond. IFEval counts strict prompt accuracy.
| Model | AIME 25 | HumanEval+ | MBPP+ | IFEval | GPQA-Diamond | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| Backbone models | ||||||||||
| Qwen3.5-9B | 62.3 | 86.6 | 65.3 | 73.5 | 80.2 | |||||
| Qwen3.8-27B | 84.1 | 92.5 | 80.4 | 87.9 | 84.5 | |||||
| Ours | ||||||||||
| AuXSpec-9B-SFT | 25.6 | −36.7 | 63.9 | −22.7 | 59.9 | −5.4 | 63.8 | −9.7 | 62.2 | −18.0 |
| AuXSpec-9B-RL | 52.2 | −10.1 | 88.6 | +2.0 | 70.8 | +5.5 | 70.0 | −3.5 | 70.7 | −9.5 |
| AuXSpec-27B-SFT | 31.0 | −53.1 | 79.0 | −13.5 | 71.1 | −9.3 | 75.2 | −12.7 | 62.9 | −21.6 |
| AuXSpec-27B-RL | 69.3 | −14.8 | 89.3 | −3.2 | 78.9 | −1.5 | 75.5 | −12.4 | 74.2 | −10.3 |
Both RL models trail their backbones on AIME, IFEval and GPQA-Diamond, with the largest gap on AIME. On HumanEval+ and MBPP+, the 9B RL model outperforms its backbone and the 27B RL model is within 3.2 points of its backbone.
5. How to Run Locally
The model adopts the Qwen3.8 chat template. It was trained in the format the template produces with reasoning_effort="medium", which is the default of the template shipped in this repository.
vLLM
vllm serve eth-algopt/AuXSpec-27B-RL --max-model-len 32768
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
system = open("prompts/spec_gen.system.md").read() # task instructions
user = open("prompts/spec_gen.example_user.md").read() # one problem
resp = client.chat.completions.create(
model="eth-algopt/AuXSpec-27B-RL",
messages=[{"role": "system", "content": system}, {"role": "user", "content": user}],
temperature=0.3, top_p=0.95, presence_penalty=1.0, max_tokens=16384,
extra_body={"top_k": 20, "chat_template_kwargs": {"reasoning_effort": "medium"}},
)
print(resp.choices[0].message.content)
Prompts. The prompts/ folder holds the system prompt of each task (spec_gen, code_gen, spec_repair, code_repair) and one example user turn. A user turn contains the problem statement, its constraints, the signature, the test cases and the Lean types of Input and Output. The repair tasks add the failing file and the checker's feedback. The answer is a single Lean file in a ```lean block after the thinking segment, to be checked with Lean 4.32 and Mathlib.
6. Limitations
- The model is specialized for Lean 4 specification and implementation of algorithmic problems, and it scores below its backbone on the general benchmarks of Section 4.3.
- A specification that passes the given test cases and the judge-based checks can still miss behavior the tests do not exercise.
7. License
This model is released under the Apache 2.0 License, inherited from Qwen3.8-27B.
8. Citation
@misc{auxspec2026,
title = {AuXSpec: Learning to Autoformalize Program Specifications via Executable Postconditions},
author = {Haocheng Wang and Weixuan Yuan and Florian H{\"u}bler and Yinya Huang and Zhijiang Guo and Sorrachai Yingchareonthawornchai and Rasmus Kyng},
year = {2026},
eprint = {2611.21421},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.21421}
}
- Downloads last month
- 56
Model tree for eth-algopt/AuXSpec-27B-RL
Base model
Qwen/Qwen3.8-27B