# Benchmark exposure and protocol

Public evaluation rows were used after promotion and did not select a checkpoint. The training compilers nevertheless used other splits from some of the same dataset families.

| Task | Source and evaluation protocol | Training-family exposure | Row overlap with public rows* |
|---|---|---|---|
| MMLU | `cais/mmlu` all/test, native order | `auxiliary_train` in v1.1 and v1.3.2 | 19 question-text / 0 question+options |
| ARC-Easy / ARC-Challenge | `allenai/ai2_arc` test, native order | ARC train and ARC material in MMLU `auxiliary_train` | At most 9 question-text / 3 question+options |
| HellaSwag | `Rowan/hellaswag` validation, native order | HellaSwag train | 23 context / 0 context+endings |
| WinoGrande | XL validation, native order | No known direct source | Not measured |
| GPQA Diamond | 198 shuffled rows; four cyclic rotations | No known direct source | Not measured |
| GSM8K-4 / GSM8K-10 | 1,319 test questions; options drawn from other problems in a matching magnitude bucket; cyclic rotations | No known GSM8K source; Arena contains arithmetic families | Not measured |
| Chess legal | 500 synthetic positions; one legal SAN and three position-inapplicable SAN choices; four rotations | No known chess source | Not applicable |

\* Normalized-text hash matches against full upstream training splits, which are supersets of the sampled training rows. The audit output for the standard tasks is [`../eval/analysis/contamination_audit.json`](../eval/analysis/contamination_audit.json). The choice-panel option-string and answer-position audit is [`../eval/choice_panel/leak_audit.json`](../eval/choice_panel/leak_audit.json).

The standard-task family exposure is distributional even where exact row overlap is small. Chess legal asks which move is legal in the shown position. The GSM8K choices test selecting a numerical answer from options; they do not measure open-ended solution generation.