breakit-2b

A 2B model that reads a program that looks correct (it passes every sample test) and proposes inputs that make it print a wrong answer. It does not know the right answer; it only tries to prove the program wrong. Every guess is checked by running it.

Fine-tuned from Qwen/Qwen3.5-2B-Base (LoRA, merged) on ~6.8k verified (wrong program, breaking input) pairs. Each pair has a one-line bug explanation. The data is in e12ex2/breakit-data.

Results

83 held-out buggy C programs from unseen problems. A program counts as broken if any guess is a legal input on which it disagrees with unanimous reference solutions (or crashes). Strict per-problem input validators.

attacker	5 guesses	50 guesses
Claude Opus	83.1%	-
gpt-6-luna	83.1%	-
this model	27.7%	69.9%
Qwen3.5-0.8B, same recipe without bug explanations	24.1%	68.7%
untrained Qwen3.5-0.8B-Base	31.3%	-
mutation fuzzer (no model)	-	53.0%

It doesn't beat frontier models. It's a free, local, offline option that gets ~70% with 50 tries.

Prompt format

The C program below was submitted for this problem. It passes the sample tests but prints a wrong answer on some valid input.

=== PROBLEM ===
<statement>

=== PROGRAM ===
```c
<code>

What the program gets wrong, then a valid input on which it prints a wrong answer: