FineWeb 100BT

A ~100 billion token English subset of FineWeb, created for efficient pretraining experiments.

Part of the Smol-Data collection — tried and tested mixes for strong pretraining.

Dataset Description

This dataset was created by randomly sampling from the full FineWeb dataset (~16.9T tokens) to produce a ~100B token subset. Sampling was performed with a fixed seed (42) and a slight 1.05× oversampling factor to account for variance.

A pre-shuffled version is available at HuggingFaceFW/fineweb_100BT-shuffled.

How It Was Created

The dataset was generated using datatrove with the smol_data.py script. The pipeline reads from the source dataset in streaming mode, applies a SamplerFilter to downsample, and writes the result back to the Hugging Face Hub.

Usage

from datasets import load_dataset

ds = load_dataset("HuggingFaceFW/fineweb_100BT", split="train", streaming=True)
for sample in ds:
    print(sample["text"][:200])
    break

Citation

@misc{niklaus2026smoldata,
      title={SmolData},
      author={Joel Niklaus and Hynek Kydl{\'\i}{\v{c}}ek},
      year={2026},
      publisher={Hugging Face},
      journal={Hugging Face repository},
      howpublished={\url{https://huggingface.co/collections/HuggingFaceFW/smol-data}}
}