Benchmark exposure and protocol
Public evaluation rows were used after promotion and did not select a checkpoint. The training compilers nevertheless used other splits from some of the same dataset families.
| Task | Source and evaluation protocol | Training-family exposure | Row overlap with public rows* |
|---|---|---|---|
| MMLU | cais/mmlu all/test, native order |
auxiliary_train in v1.1 and v1.3.2 |
19 question-text / 0 question+options |
| ARC-Easy / ARC-Challenge | allenai/ai2_arc test, native order |
ARC train and ARC material in MMLU auxiliary_train |
At most 9 question-text / 3 question+options |
| HellaSwag | Rowan/hellaswag validation, native order |
HellaSwag train | 23 context / 0 context+endings |
| WinoGrande | XL validation, native order | No known direct source | Not measured |
| GPQA Diamond | 198 shuffled rows; four cyclic rotations | No known direct source | Not measured |
| GSM8K-4 / GSM8K-10 | 1,319 test questions; options drawn from other problems in a matching magnitude bucket; cyclic rotations | No known GSM8K source; Arena contains arithmetic families | Not measured |
| Chess legal | 500 synthetic positions; one legal SAN and three position-inapplicable SAN choices; four rotations | No known chess source | Not applicable |
* Normalized-text hash matches against full upstream training splits, which are supersets of the sampled training rows. The audit output for the standard tasks is ../eval/analysis/contamination_audit.json. The choice-panel option-string and answer-position audit is ../eval/choice_panel/leak_audit.json.
The standard-task family exposure is distributional even where exact row overlap is small. Chess legal asks which move is legal in the shown position. The GSM8K choices test selecting a numerical answer from options; they do not measure open-ended solution generation.