Buckets:
| Name | Size | Uploaded | Xet hash |
|---|---|---|---|
| claude-opus-5-5 | 1,192 items | ||
| deepseek-v4.1-flash-1017 | 1,192 items | ||
| gemini-4-argon | 1,192 items | ||
| glm-5.3 | 1,182 items | ||
| gpt-6-astra | 1,200 items | ||
| grok-4.7 | 1,181 items | ||
| hy4-preview | 1,170 items | ||
| kimi-k3.1-pro | 1,194 items | ||
| mistral-large-4-0 | 1,189 items | ||
| muse-wonder | 1,199 items | ||
| qwen3.8-max-0902 | 1,096 items | ||
| README.md | 5.04 kB xet | 5b9d8b56 | |
| prices.json | 1 kB xet | e79c11bc | |
| reproduce.py | 3.19 kB xet | 514026e7 | |
| rollouts.jsonl | 1.82 MB xet | 6041c8a3 |
CWE-bench v2: public-leaderboard rollouts
These are the exact rollouts behind the CWE-bench v2 public leaderboard:
11 models × 100 public tasks × 4 attempts = 4,400 rollouts. They cover the public split of
collinear-ai/cwe-bench. Every leaderboard number can be recomputed from this bucket:
python reproduce.py rollouts.jsonl prices.json
The output matches the leaderboard's pass@1, pass@4 and $/rollout for every model.
Layout
rollouts.jsonl one line per rollout (index below)
prices.json per-token list prices used for $/rollout
reproduce.py recomputes the leaderboard from the two files above
<model>/<task>/<attempt>/trajectory.json the agent's steps (Harbor ATIF format)
<model>/<task>/<attempt>/patch.diff the agent's change to the task repository
<model>/<task>/<attempt>/verifier.log the grader's output for this attempt
attempt is 0–3.
rollouts.jsonl
| field | meaning |
|---|---|
model, harness, task, attempt |
which rollout |
reward |
grader reward: 1 = vulnerability fixed with functionality intact, 0 = not |
passed |
reward >= 1 |
duration_s |
wall-clock seconds for the attempt |
prompt_tokens, cached_tokens, completion_tokens, steps |
from the trajectory's final_metrics |
cost_usd |
harness-recorded cost (Claude Code, Codex, and Qwen via OpenCode); null otherwise |
has_trajectory, has_patch, has_verifier_log |
whether that file exists for this rollout |
patch_files |
number of files the patch changes |
path |
the rollout's folder |
How the leaderboard is computed
- Selection. For each model and task, the first 4 graded attempts are used. Every graded attempt counts, including ones where the agent hit its time limit (the repository is graded as the agent left it).
- pass@1 is the mean reward over the 400 rollouts. pass@4 is the share of the 100 tasks solved in at least one of the 4 attempts.
- $ / rollout is the mean cost per rollout. Where the harness records cost (
cost_usd), that value is used. Otherwise it is computed from token counts at the list prices inprices.json(OpenRouter, read 2026-11-03): uncached prompt × input + cached prompt × cached input + completion × output. The Antigravity SDK (Gemini) reports prompt tokens excluding cached ones; the other harnesses include them.
patch.diff
patch.diff is a git binary diff against the task's original environment/repo.tar.gz in the dataset.
Apply it with git apply (or patch -p1) from the repository root. Every patch was checked to apply
cleanly. It is built from a snapshot of the agent's final repository, normalized so that it shows only the
agent's work:
- file-mode changes are ignored, and symlinks that the snapshot followed are restored;
- installed dependencies (
node_modules/) are left out; - new files that are build output are left out: files the repository's own
.gitignoreexcludes, object and dependency files (.o,.lo,.a,.so,.class,.pyc, …), compiled ELF binaries, and newbuild/,CMakeFiles/,.deps/,.libs/,target/,__pycache__/andautom4te.cache/directories. Files that exist in the original repository are never left out; - before storage, anything that looks like a secret is replaced with
[REDACTED]. That includes the test keys that some task repositories ship. Where that placeholder is the only difference from the original file, the original file is kept, so the patch doesn't show a change the agent never made. In a few files the agent did change (56 patches, mostly test files with sample credentials or tokens), some of the agent's own text was scrubbed too, and the patch shows[REDACTED]in its place.
Some patches still contain files that a build regenerated (for example configure or Makefile.in after
the agent re-ran autotools); these are part of the agent's final tree.
Rollouts without a patch (has_patch: false, 199 of 4,400): no snapshot of the final repository
exists. Most often the working tree, with everything the agent built, exceeded the snapshot size limit; this affects Qwen 3.8 Max most (104 rollouts). The reward was still graded
in the live environment and counts.
Rollouts without a trajectory (has_trajectory: false, 13): the agent hit its time limit before the
harness wrote a trajectory. Their reward counts.
Rollouts without cost (25): the 13 above, plus 12 Gemini rollouts whose trajectory has no token totals. They are left out of $/rollout only; every one of the 4,400 rollouts has a reward and counts toward pass@1 and pass@4.
trajectory.json
Unchanged from the harness, with one exception: in a few trajectories the agent listed its sandbox's mounts,
and the host run directory path is replaced with /work/runs/<run>/.
- Total size
- 3.01 GB
- Files
- 12,991
- Last updated
- Nov 4
- Pre-warmed CDN
- US EU US EU