File size: 735 Bytes
c8ba9c6
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# Dataset attribution

Default upstream dataset: **FineWeb-Edu**, Hugging Face / HuggingFaceFW.
Repository: https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu
Configuration: sample-100BT. Language: English. Dataset card lists ODC-BY.
Read the upstream dataset card and license for applicable attribution and
underlying web-content terms. This ZIP distributes no dataset texts.
The immutable source revision is recorded in work/data/manifest.json at preparation.

The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale:
https://arxiv.org/abs/2406.17557

The WarpState code and tokenizer are copied from the user's existing project.
No new ownership or license over those user-provided assets is asserted here.