README.md

# SWE-bench Pro: 10 agent harnesses x 2 models

Every rollout behind the harness-vs-model comparison: 10 coding agent harnesses run
against both models on the same 250 held-out SWE-bench Pro tasks, one rollout per task,
5,000 rollouts in total. A third, partial arm runs Gemma 4 with thinking disabled; see
[Thinking-off ablation](#thinking-off-ablation).

| model              | precision | pass@1 range across harnesses |
|--------------------|-----------|-------------------------------|
| GLM-5.2 744B-A40B  | FP8       | 23.2% (goose) - 52.4% (mini_swe_agent) |
| Gemma 4 26B-A4B    | BF16      | 15.2% (goose) - 36.0% (crush)          |

The harness rankings do not transfer between the two models: the rank correlation between
them is -0.05. Vendor-shipped harnesses fall on the small model (codex 2nd of 10 to 9th,
claude_code 3rd to 7th) while model-agnostic ones climb (crush 7th to 1st, opencode 8th
to 2nd).

## Thinking-off ablation

`gemma-4-26B-A4B-it-thinkoff` is the same sweep with the model's thinking disabled. It was
never finished: 8 of the 10 harnesses, and between 86 and 191 of the 250 tasks each rather
than all of them, so the denominators differ per harness and the missing tasks are the ones
that had not run yet rather than a random sample. It is published because the direction is
already clear, not because the numbers are final.

| harness        | thinking on (n=250) | thinking off | n   |
|----------------|---------------------|--------------|-----|
| claude_code    | 24.4%               | 24.4%        | 164 |
| mini_swe_agent | 28.4%               | 23.0%        | 191 |
| crush          | 36.0%               | 15.5%        | 161 |
| qwen_code      | 26.8%               | 14.6%        | 130 |
| opencode       | 33.2%               | 12.3%        | 106 |
| codex          | 20.8%               | 10.5%        |  86 |
| pi             | 28.0%               |  8.6%        | 175 |
| goose          | 15.2%               |  6.0%        | 167 |

Turning thinking off costs almost every harness, and it costs most exactly the harnesses
that did best with it on: crush, opencode and pi give up two thirds of their score, while
claude_code does not move. So the harness ranking on this model is partly a ranking of how
well each one exploits the thinking budget, not of its scaffolding alone.

## Layout

```
swebench_pro/test/<model>/<harness>/<task_id>/t0/    one rollout
swebench_pro/test/<harness>/<model>.summary.json     pass@1, Wilson 95% CI, coverage
```

The thinking-off arm has rollout directories but no `.summary.json`: it was never
aggregated by the sweep runner, so its numbers above come from the `scores.json` files
directly.

Each rollout directory holds:

| file                    | what it is                                                        |
|-------------------------|-------------------------------------------------------------------|
| `scores.json`           | verdict, fail-to-pass counts, input/output tokens, wall clock, harness version |
| `hf_router_usage.jsonl` | one line per model call with its prompt and completion sizes       |
| `agent_cmd.sh`          | the exact command the harness was launched with                    |
| `agent_stdout.log`      | everything the harness printed                                     |
| `grade/`                | the graded patch (`patch.diff`), test `output.json`, run scripts, stdout/stderr |
| `candidate.patch`       | the patch the agent produced, where the harness writes one separately |
| `mini_traj.json`        | mini_swe_agent's structured trajectory                             |
| `codex_home/sessions/`  | codex's session transcripts                                        |

## What is not here

Harness home directories (caches, sqlite journals, downloaded skills, shell snapshots)
are excluded. They are an order of magnitude larger than the record itself -- one codex
rollout is 68MB of which under 1MB describes the run -- and they are where a harness
writes a credential if it writes one at all. Only the two directories above, which hold
real transcripts, survive that cut.

Token-shaped strings matching `hf_[A-Za-z0-9]{20,}` are replaced with
`<REDACTED_HF_TOKEN>`. Private-key headers and `sk-` prefixes that remain are content
from the benchmark's own task repositories (`gravitational/teleport`, `flipt-io/flipt`)
and from branch names like `task-Fix-Timezone`, not credentials.

## Caveats

250 tasks and one rollout put a Wilson 95% interval of roughly +/-6pp around any single
pass@1, so neighbouring harnesses are not separated. The reshuffling of the ranking
between the two models is a joint pattern across ten harnesses and is the robust result.
The thinking-off arm is wider still, both because it has fewer tasks and because it has a
different set of them per harness.

Costs quoted elsewhere use list API rates as of 2026-08-06: GLM-5.2 at Z.ai's own card
($1.40/$4.40 per M tokens), Gemma 4 at DeepInfra, the cheapest of 13 providers
($0.07/$0.34).