| # Benchmark exposure and protocol |
|
|
| Public evaluation rows were used after promotion and did not select a checkpoint. The training compilers nevertheless used other splits from some of the same dataset families. |
|
|
| | Task | Source and evaluation protocol | Training-family exposure | Row overlap with public rows* | |
| |---|---|---|---| |
| | MMLU | `cais/mmlu` all/test, native order | `auxiliary_train` in v1.1 and v1.3.2 | 19 question-text / 0 question+options | |
| | ARC-Easy / ARC-Challenge | `allenai/ai2_arc` test, native order | ARC train and ARC material in MMLU `auxiliary_train` | At most 9 question-text / 3 question+options | |
| | HellaSwag | `Rowan/hellaswag` validation, native order | HellaSwag train | 23 context / 0 context+endings | |
| | WinoGrande | XL validation, native order | No known direct source | Not measured | |
| | GPQA Diamond | 198 shuffled rows; four cyclic rotations | No known direct source | Not measured | |
| | GSM8K-4 / GSM8K-10 | 1,319 test questions; options drawn from other problems in a matching magnitude bucket; cyclic rotations | No known GSM8K source; Arena contains arithmetic families | Not measured | |
| | Chess legal | 500 synthetic positions; one legal SAN and three position-inapplicable SAN choices; four rotations | No known chess source | Not applicable | |
|
|
| \* Normalized-text hash matches against full upstream training splits, which are supersets of the sampled training rows. The audit output for the standard tasks is [`../eval/analysis/contamination_audit.json`](../eval/analysis/contamination_audit.json). The choice-panel option-string and answer-position audit is [`../eval/choice_panel/leak_audit.json`](../eval/choice_panel/leak_audit.json). |
|
|
| The standard-task family exposure is distributional even where exact row overlap is small. Chess legal asks which move is legal in the shown position. The GSM8K choices test selecting a numerical answer from options; they do not measure open-ended solution generation. |
|
|
