Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,73 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
tags:
|
| 4 |
+
- code
|
| 5 |
+
- evaluation
|
| 6 |
+
- benchmark
|
| 7 |
+
- coding-agents
|
| 8 |
+
- reliability
|
| 9 |
+
- security
|
| 10 |
+
pipeline_tag: text-generation
|
| 11 |
+
---
|
| 12 |
+
|
| 13 |
+
# CodeBench
|
| 14 |
+
|
| 15 |
+
**CodeBench** is an evaluation framework for measuring the *reliable* code
|
| 16 |
+
generation ability of AI coding agents, beyond inflated pass@k scores.
|
| 17 |
+
|
| 18 |
+
## Key Findings
|
| 19 |
+
|
| 20 |
+
| Finding | Result |
|
| 21 |
+
|---|---|
|
| 22 |
+
| **H1 — Category error** | Standard pass@k treats test-case counts as independent sample counts, producing different scores for agents with identical true reliability |
|
| 23 |
+
| **H2 — Score inflation** | Current pass@5 ≈ 0.96–0.97 collapses to reliability@5 ≈ 0.00–0.12 when the category error is corrected |
|
| 24 |
+
| **H3 — Proxy validity** | Single-rollout proxy score has low Spearman correlation with reliability@k; ≥5 rollouts needed for reliable ranking |
|
| 25 |
+
| **H4 — Security gap** | Security-adjusted reliability@5 is meaningfully lower than reliability@5; leaderboard rank flips when security is accounted for |
|
| 26 |
+
|
| 27 |
+
## Metrics
|
| 28 |
+
|
| 29 |
+
- **`reliability@k`** — correct operationalization of Chen et al. (2021) pass@k,
|
| 30 |
+
using per-(task, agent) rollout counts and binary execution success
|
| 31 |
+
- **`security_adjusted_reliability@k`** — reliability@k counting only rollouts
|
| 32 |
+
that are both correct *and* produce code with no insecure patterns (eval,
|
| 33 |
+
exec, os.system, yaml.load without Loader, pickle.loads)
|
| 34 |
+
|
| 35 |
+
## Agents Evaluated
|
| 36 |
+
|
| 37 |
+
| Agent | Provider | Model |
|
| 38 |
+
|---|---|---|
|
| 39 |
+
| `anote-code` | Anthropic | claude-sonnet-4-6 (Anote system prompt) |
|
| 40 |
+
| `claude-code` | Anthropic | claude-sonnet-4-6 |
|
| 41 |
+
| `codex` | OpenAI | gpt-4o |
|
| 42 |
+
|
| 43 |
+
## Figures
|
| 44 |
+
|
| 45 |
+
### Figure 1 — Baseline: pass@1 vs current pass@5
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
### Figure 2 — H1: Category-error proof
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
### Figure 3 — H2: Score inflation magnitude
|
| 52 |
+
|
| 53 |
+
|
| 54 |
+
### Figure 4 — H3: Proxy vs reliability@k correlation
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
### Figure 5 — H4: Security-adjusted reliability leaderboard
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
## Datasets
|
| 61 |
+
|
| 62 |
+
- [nv78/codebench-tasks](https://huggingface.co/datasets/nv78/codebench-tasks) — benchmark task definitions
|
| 63 |
+
- [nv78/codebench-results](https://huggingface.co/datasets/nv78/codebench-results) — experiment results (h4_security + swebench_smoke)
|
| 64 |
+
|
| 65 |
+
## Citation
|
| 66 |
+
|
| 67 |
+
```
|
| 68 |
+
@misc{codebench2026,
|
| 69 |
+
title = {CodeBench: Measuring Reliable Code Generation},
|
| 70 |
+
author = {Anote AI},
|
| 71 |
+
year = {2026}
|
| 72 |
+
}
|
| 73 |
+
```
|