# Confirmatory GitLab Runner dataset

## Freeze and separation

The confirmatory set was frozen on 2026-07-18 before running E01-E05. It
contains 15 test-backed GitLab Runner fixes from March-May 2026. None of their
gold commits overlap the five E00 development fixes. E00 remains excluded from
confirmatory estimates.

Every task stores a visible issue statement, exact parent and gold commits,
hidden source and test patches, changed source files, Tree-sitter-derived gold
symbols, validation commands, and provenance. The generator is
`scripts/build_confirmatory_tasks.py`; the frozen artifacts are under
`tasks/manifests` and `tasks/patches`.

| Task | Gold commit | Source files | Domain | E03 eligible |
|---|---|---:|---|---:|
| CR_001 | `d870071152bd` | 1 | AWS secrets | yes |
| CR_002 | `b6dedc95bd2b` | 2 | Cache adapters | yes |
| CR_003 | `91dcb6c16bec` | 1 | Build logging | yes |
| CR_004 | `eeba44f142de` | 1 | Concrete cache policy | no |
| CR_005 | `f07534b7a203` | 2 | Artifact variables | yes |
| CR_006 | `e05f89e3ec2e` | 1 | Cancellation errors | yes |
| CR_007 | `b4b74fa3d7c8` | 2 | Trace features | yes |
| CR_008 | `002579e259d9` | 1 | Artifact upload | yes |
| CR_009 | `c9632925cb0c` | 1 | S3 presigning | yes |
| CR_010 | `3b1d17363021` | 1 | Kubernetes variables | no |
| CR_011 | `610441433600` | 1 | Artifact timeouts | no |
| CR_012 | `86c126f85257` | 1 | Gzip filenames | yes |
| CR_013 | `1a6fb708e571` | 3 | Cache sharding | yes |
| CR_014 | `213001f5e9e2` | 3 | After-script policy | no |
| CR_015 | `32a34faf2b81` | 2 | Step-runner errors | no |

All 15 tasks enter E01, E02, E04, and E05. Ten tasks enter E03.

## End-to-end validation

Validation used Go 1.26.5 on clean `git archive` snapshots. For each task, the
validator required:

1. the parent package suite to pass before hidden tests were added;
2. the hidden test patch to apply and expose at least one failure;
3. the hidden source patch to apply; and
4. the package suite with hidden tests and the gold source patch to pass.

Ten tasks met every condition. CR_004's upstream test change did not expose a
parent failure. CR_010, CR_011, CR_014, and CR_015 had failures in the original
package suites on this host, so they were excluded from E03 without changing
their retrieval/localization eligibility. The complete commands, output tails,
exit codes, and timing are retained in `tasks/validation`.

## Scope limitations

The task sample is a purposive population of recent, compact, test-backed fixes
from one large Go repository. It supports paired causal comparisons among
harnesses on this fixed population, but it does not justify generalization to
other repositories, languages, models, or issue distributions. Public commits
also carry a residual training-contamination risk, although their 2026 dates
reduce that risk relative to older benchmark tasks.