mgunnin/legup/LittleCurriculum - Storage Bucket

LittleCurriculum

LittleCurriculum is an ~88B-token English pretraining corpus derived from FineWeb-Edu. It is filtered to align with U.S. Common Core standards for grades K–5, removing documents containing academic concepts and skills characteristic of later grades.

It is the training corpus for the LittleLearner models, designed to study language models under a controlled knowledge boundary. See the LittleLearner paper for methodology, validation, and experiments.

Dataset

• Documents: ~244M
• Tokens: ~88B
• Language: English
• Source: FineWeb-Edu
• Format: Parquet (id, text)
• License: ODC-By 1.0

Dataset Construction

LittleCurriculum is produced from FineWeb-Edu using five sequential filters:

1. Age-of-Acquisition and word-frequency filtering
2. fastText grade-level classification
3. ModernBERT grade-level classification
4. Advanced mathematical/symbolic notation filtering
5. Beyond-K–5 vocabulary filtering

The filters constrain the developmental level of a text, i.e. the knowledge, complexity, and reasoning it demands, not its subject matter. Advanced Beyond-K–5 concepts are removed, but topic is deliberately not curated: everyday material on any subject a curious learner might explore—aviation, current events, crime in the news—can appear when written at a K–5 level. Likewise, words associated with advanced topics may occur in ordinary, grade-appropriate contexts. Full construction details, thresholds, validation, and limitations are described in the paper.

Citation

@misc{li2026littlelearnerlanguagemodelspedagogically,
      title={LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure}, 
      author={Fanfei Li and Jana Zeller and Manuel Prada-Corral and Thaddäus Wiedemer and Prasanna Mayilvahanan and Ryan Cotterell and Wieland Brendel},
      year={2026},
      eprint={2608.13545},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2608.13545}, 
}