Confirmatory GitLab Runner dataset
Freeze and separation
The confirmatory set was frozen on 2026-07-18 before running E01-E05. It contains 15 test-backed GitLab Runner fixes from March-May 2026. None of their gold commits overlap the five E00 development fixes. E00 remains excluded from confirmatory estimates.
Every task stores a visible issue statement, exact parent and gold commits,
hidden source and test patches, changed source files, Tree-sitter-derived gold
symbols, validation commands, and provenance. The generator is
scripts/build_confirmatory_tasks.py; the frozen artifacts are under
tasks/manifests and tasks/patches.
| Task | Gold commit | Source files | Domain | E03 eligible |
|---|---|---|---|---|
| CR_001 | d870071152bd |
1 | AWS secrets | yes |
| CR_002 | b6dedc95bd2b |
2 | Cache adapters | yes |
| CR_003 | 91dcb6c16bec |
1 | Build logging | yes |
| CR_004 | eeba44f142de |
1 | Concrete cache policy | no |
| CR_005 | f07534b7a203 |
2 | Artifact variables | yes |
| CR_006 | e05f89e3ec2e |
1 | Cancellation errors | yes |
| CR_007 | b4b74fa3d7c8 |
2 | Trace features | yes |
| CR_008 | 002579e259d9 |
1 | Artifact upload | yes |
| CR_009 | c9632925cb0c |
1 | S3 presigning | yes |
| CR_010 | 3b1d17363021 |
1 | Kubernetes variables | no |
| CR_011 | 610441433600 |
1 | Artifact timeouts | no |
| CR_012 | 86c126f85257 |
1 | Gzip filenames | yes |
| CR_013 | 1a6fb708e571 |
3 | Cache sharding | yes |
| CR_014 | 213001f5e9e2 |
3 | After-script policy | no |
| CR_015 | 32a34faf2b81 |
2 | Step-runner errors | no |
All 15 tasks enter E01, E02, E04, and E05. Ten tasks enter E03.
End-to-end validation
Validation used Go 1.26.5 on clean git archive snapshots. For each task, the
validator required:
- the parent package suite to pass before hidden tests were added;
- the hidden test patch to apply and expose at least one failure;
- the hidden source patch to apply; and
- the package suite with hidden tests and the gold source patch to pass.
Ten tasks met every condition. CR_004's upstream test change did not expose a
parent failure. CR_010, CR_011, CR_014, and CR_015 had failures in the original
package suites on this host, so they were excluded from E03 without changing
their retrieval/localization eligibility. The complete commands, output tails,
exit codes, and timing are retained in tasks/validation.
Scope limitations
The task sample is a purposive population of recent, compact, test-backed fixes from one large Go repository. It supports paired causal comparisons among harnesses on this fixed population, but it does not justify generalization to other repositories, languages, models, or issue distributions. Public commits also carry a residual training-contamination risk, although their 2026 dates reduce that risk relative to older benchmark tasks.