Download data/README.md from NicolasT08/formal-mas: direct link, hf CLI and curl.
- Browser
- Download file 3.92 kB
-
https://huggingface.co/NicolasT08/formal-mas/resolve/main/data/README.md
- Command line
-
hf download hf://NicolasT08/formal-mas/data/README.md
-
curl -L -o README.md https://huggingface.co/NicolasT08/formal-mas/resolve/main/data/README.md
Evaluation set (eval.jsonl)
45 competition problems stated in Lean 4, fixed before any tuning on this set. Build it with:
bash scripts/fetch_eval_sources.sh # pinned sources into data/raw/ (not in git)
python scripts/build_eval_set.py # seeded draw, compile with sorry, write eval.jsonl
Sources and licenses
| Source | Commit | License |
|---|---|---|
miniF2F, Lean 4 port in DeepSeek-Prover-V1.5 (datasets/minif2f.jsonl) |
2c4ba911 |
MIT |
PutnamBench (lean4/src, informal/putnam.json) |
a635813f |
Apache-2.0 (Lean statements) |
Each record keeps its source, original id, source commit and license. The problems are public, so they may be in the training data of the models under evaluation.
PutnamBench states that its informal statements are available with permission from the MAA, so
Apache-2.0 covers the Lean statements only: check this before publishing the set. The MIT license of
the miniF2F port is the one in the DeepSeek-Prover-V1.5 repository (LICENSE-CODE).
Criteria (curation/criteria.yaml)
Written before the draw, except for what its header lists as added afterwards.
- Difficulty, by competition of origin. easy:
mathd_*. medium: AMC12, AIME and the textbook familiesalgebra_*,numbertheory_*,induction_*. hard: IMO, IMO Shortlist, Putnam. - Area.
mathd_*keep the area of their prefix, even when they are coordinate geometry (mathd_algebra_288). Combinatorics is strict: counting configurations, binomials and their identities, pigeonhole, graphs, partitions; counting integers under an arithmetic condition is not combinatorics, and a divisibility or primality claim about a binomial sum is number theory. For other families, geometry, trigonometry and real analysis are out of scope. - PutnamBench is used only where miniF2F has no pool (hard × combinatorics), and only for problems
tagged combinatorics that are one self-contained theorem, without geometry, analysis, probability
or linear-algebra tags. Factored
_solutionanswers and local definitions are excluded.
Grid
The target was 5 per cell. miniF2F has no strict combinatorics at easy level and 2 problems at medium level, so the missing 8 were reassigned within the same difficulty, proportionally to the 5:5 target of the other two cells; an odd one went to the larger pool (algebra).
amc12a_2020_p4 was first labelled combinatorics and was corrected to number theory after the first
draw, by the written definition. The correction used no system result.
| algebra | number_theory | combinatorics | total | |
|---|---|---|---|---|
| easy | 8 (3 dev) | 7 (2 dev) | 0 | 15 |
| medium | 7 (2 dev) | 6 (2 dev) | 2 (1 dev) | 15 |
| hard | 5 (2 dev) | 5 (1 dev) | 5 (2 dev) | 15 |
| total | 20 | 18 | 7 | 45 |
Draw and split (seed 2026)
Within a cell, problems are taken in the order of sha256("2026:<id>"). A problem that fails is
replaced by the next one. dev slots are allocated per cell by largest remainder (15 in total, ties
by the seed), filled first by the 5 problems of dev_preview.jsonl, then in seeded order. No C1–C5
result was used. Problems named in tests (mathd_algebra_10) are excluded.
Allowed edits
∑ x in s/∏ x in s→∑ x ∈ s/∏ x ∈ s, by script; the original is kept informal_statement_originaland the record is marked"converted": "big_operator_in_to_mem".- PutnamBench's placeholder
:= sorrybecomes:= by, the format of all records. This changes no part of the statement.informalcomes frominformal/putnam.json.
Discarded
curation/discarded.json has every discarded problem and its reason; curation/fidelity_review.yaml
has the informal-to-formal review, with its notes and verification limits. curation/build_info.json
records the Lean back-end of the build; the Mathlib revision is pinned in lean_project/lake-manifest.json.