YoungXuan/MS-backup/xuan/data - Storage Bucket

FineWeb-Edu 10B / GPT-2 tokenized .bin shards (LoopResidual)

Pre-tokenized pretraining data for the LoopResidual looped-Transformer experiments. Drop-in for LoopResidual/train.py (--train_files, --val_files).

Contents

path	shards	tokens	bytes
fineweb_edu_train/train_0000{00..20}.bin	21	10,400,062,869 (10.400B)	~20 GB
fineweb_edu_val/val_000000.bin	1	100,002,688 (0.100B)	191 MB

Training budget this is sized for: exactly 10B tokens (19,073 steps x 524,288 tokens/step = 9,999,745,024). The train pool carries a ~4% margin on purpose — the loader cycles files with itertools.cycle, so a pool of exactly 10B would start re-reading already-trained data in the final steps, and each shard also discards a tail fragment smaller than one batch span (<16,384 tokens). The margin guarantees all 10B training tokens are distinct.

Provenance

• Source: HuggingFaceFW/fineweb-edu, subset sample/100BT, 24 parquet shards (51.65 GB) downloaded 2026-07-30.
• Taken as-is, with no score re-bucketing. That subset is already the score>=3 educational filter; measured int_score distribution over 40k sampled docs: 3 -> 85.8%, 4 -> 14.1%, 5 -> 0.09%. (Data quality is a nuisance variable held fixed here, not the thing under study, so it is not re-partitioned.)
• Tokenizer: tiktoken gpt2 (vocab 50257), one EOT (50256) appended per document. Measured EOT rate 0.089-0.105% per shard, i.e. ~1000 tokens/doc.
• 10,087,135 train documents / 97,741 val documents.

Validation split is shard-disjoint

Validation is 2 whole held-out parquet shards (002_00002.parquet, 002_00003.parquet) that never enter training — not a prefix of each training file. Verified at document level: 0 of 16,025,000 train doc IDs appear among the 1,453,000 val doc IDs.

This matters because the obvious alternative (the tokenizer's built-in --val_tokens, which takes the first N tokens of every file and trains on the remainder of the same files) leaves train and val sharing shards, so near-duplicate documents can straddle the split. FineWeb's MinHash dedup is per-dump, so cross-dump near-duplicates are not fully excluded by any split — but whole-shard holdout is the standard defensible choice.

Residual caveat: exact duplicates are excluded; near-duplicates with different IDs are not provably absent.

Binary format

nanoGPT-style: a 256 x int32 header followed by a uint16 token payload.

header index	value
0	magic 20240520
1	version 1
2	number of tokens

import numpy as np

def load_shard(path):
    with open(path, "rb") as f:
        header = np.frombuffer(f.read(256 * 4), dtype=np.int32)
    assert header[0] == 20240520 and header[1] == 1
    return np.memmap(path, dtype=np.uint16, mode="r",
                     offset=256 * 4, shape=(int(header[2]),))

Download

huggingface-cli download YoungXuan/MS-backup --repo-type dataset \
    --include "xuan/data/fineweb_edu_*/*" --local-dir ./bins

Reproduce

# in LoopResidual/
HF_CACHE_ROOT=<cache> python data/download_fineweb.py \
    --repo_id HuggingFaceFW/fineweb-edu --subset "sample/100BT" \
    --output_dir <parquet_dir> --max_files 24 --batch_files 24 --continuous
bash data/prepare_fineweb_edu.sh     # shard-disjoint split + tokenize + verify

Integrity checks (data/verify_bins.py): header magic/version, declared token count vs actual file size (catches truncated writes), all ids < 50257 (no uint16 wrap), non-zero EOT rate, and a content fingerprint per shard to catch a collapsed train/val split. All 22 shards passed.