Buckets:

3.01 GB
12,991 files
Updated 13 days ago
Name
Size
claude-opus-5-5
deepseek-v4.1-flash-1017
gemini-4-argon
glm-5.3
gpt-6-astra
grok-4.7
hy4-preview
kimi-k3.1-pro
mistral-large-4-0
muse-wonder
qwen3.8-max-0902
README.md5.04 kB
xet
prices.json1 kB
xet
reproduce.py3.19 kB
xet
rollouts.jsonl1.82 MB
xet
README.md

CWE-bench v2: public-leaderboard rollouts

These are the exact rollouts behind the CWE-bench v2 public leaderboard: 11 models × 100 public tasks × 4 attempts = 4,400 rollouts. They cover the public split of collinear-ai/cwe-bench. Every leaderboard number can be recomputed from this bucket:

python reproduce.py rollouts.jsonl prices.json

The output matches the leaderboard's pass@1, pass@4 and $/rollout for every model.

Layout

rollouts.jsonl                          one line per rollout (index below)
prices.json                             per-token list prices used for $/rollout
reproduce.py                            recomputes the leaderboard from the two files above
<model>/<task>/<attempt>/trajectory.json    the agent's steps (Harbor ATIF format)
<model>/<task>/<attempt>/patch.diff         the agent's change to the task repository
<model>/<task>/<attempt>/verifier.log       the grader's output for this attempt

attempt is 0–3.

rollouts.jsonl

field meaning
model, harness, task, attempt which rollout
reward grader reward: 1 = vulnerability fixed with functionality intact, 0 = not
passed reward >= 1
duration_s wall-clock seconds for the attempt
prompt_tokens, cached_tokens, completion_tokens, steps from the trajectory's final_metrics
cost_usd harness-recorded cost (Claude Code, Codex, and Qwen via OpenCode); null otherwise
has_trajectory, has_patch, has_verifier_log whether that file exists for this rollout
patch_files number of files the patch changes
path the rollout's folder

How the leaderboard is computed

  • Selection. For each model and task, the first 4 graded attempts are used. Every graded attempt counts, including ones where the agent hit its time limit (the repository is graded as the agent left it).
  • pass@1 is the mean reward over the 400 rollouts. pass@4 is the share of the 100 tasks solved in at least one of the 4 attempts.
  • $ / rollout is the mean cost per rollout. Where the harness records cost (cost_usd), that value is used. Otherwise it is computed from token counts at the list prices in prices.json (OpenRouter, read 2026-11-03): uncached prompt × input + cached prompt × cached input + completion × output. The Antigravity SDK (Gemini) reports prompt tokens excluding cached ones; the other harnesses include them.

patch.diff

patch.diff is a git binary diff against the task's original environment/repo.tar.gz in the dataset. Apply it with git apply (or patch -p1) from the repository root. Every patch was checked to apply cleanly. It is built from a snapshot of the agent's final repository, normalized so that it shows only the agent's work:

  • file-mode changes are ignored, and symlinks that the snapshot followed are restored;
  • installed dependencies (node_modules/) are left out;
  • new files that are build output are left out: files the repository's own .gitignore excludes, object and dependency files (.o, .lo, .a, .so, .class, .pyc, …), compiled ELF binaries, and new build/, CMakeFiles/, .deps/, .libs/, target/, __pycache__/ and autom4te.cache/ directories. Files that exist in the original repository are never left out;
  • before storage, anything that looks like a secret is replaced with [REDACTED]. That includes the test keys that some task repositories ship. Where that placeholder is the only difference from the original file, the original file is kept, so the patch doesn't show a change the agent never made. In a few files the agent did change (56 patches, mostly test files with sample credentials or tokens), some of the agent's own text was scrubbed too, and the patch shows [REDACTED] in its place.

Some patches still contain files that a build regenerated (for example configure or Makefile.in after the agent re-ran autotools); these are part of the agent's final tree.

Rollouts without a patch (has_patch: false, 199 of 4,400): no snapshot of the final repository exists. Most often the working tree, with everything the agent built, exceeded the snapshot size limit; this affects Qwen 3.8 Max most (104 rollouts). The reward was still graded in the live environment and counts.

Rollouts without a trajectory (has_trajectory: false, 13): the agent hit its time limit before the harness wrote a trajectory. Their reward counts.

Rollouts without cost (25): the 13 above, plus 12 Gemini rollouts whose trajectory has no token totals. They are left out of $/rollout only; every one of the 4,400 rollouts has a reward and counts toward pass@1 and pass@4.

trajectory.json

Unchanged from the harness, with one exception: in a few trajectories the agent listed its sandbox's mounts, and the host run directory path is replaced with /work/runs/<run>/.

Total size
3.01 GB
Files
12,991
Last updated
Nov 4
Pre-warmed CDN
US EU US EU

Contributors