ml-intern-internal/ml-eval-c4-tokenizer-8a91c9-r0

Byte-level BPE tokenizer, 32,000 vocab — trained on 1M C4 documents

Run the training

Status: training script ready; the run itself has not happened yet. The session token could not submit compute jobs (403 — no Jobs permission) nor write under GuillaumeSalouHF, so per the user's choice this repo under ml-intern-internal is the destination. Once a token with Jobs access is available, this repo will hold tokenizer.json + tokenizer_config.json + special_tokens_map.json.

What the script does (train_tokenizer.py)

• Method: GPT-2-style byte-level BPE via Hugging Face tokenizers — ByteLevel pre-tokenizer (add_prefix_space=False, GPT-2 regex) and decoder, full 256-byte initial alphabet, min_frequency=2, special tokens <pad> <bos> <eos>, target vocab 32,000.
• Data: 1,000,000 documents streamed in order from allenai/c4 config en, split train, raw shards en/c4-train.00000…00002-of-01024.json.gz (~960 MB), with the next shard prefetched while the current one parses. No document is reused.
• Eval: 1,000 held-out documents (tail of the final shard, disjoint from training) report chars/token compression, plus an encode→decode round-trip assertion and a vocab-size assertion on full runs. Numbers are printed to the run log and written into the pushed README.
• Push: saves tokenizer.json (raw tokenizers), plus tokenizer_config.json / special_tokens_map.json (via PreTrainedTokenizerFast) and a README, then uploads to the Hub.

How to run

Locally (any machine with a few GB RAM free and Python 3.10+):

pip install "tokenizers" "transformers" "huggingface_hub"
export HUB_ID=ml-intern-internal/ml-eval-c4-tokenizer-8a91c9-r0
export PUSH=1
export HF_TOKEN=hf_...   # needs write access to the destination repo
python train_tokenizer.py            # defaults: 1M docs, vocab 32k

Or as a HF Job on cpu-upgrade (8 vCPU, $0.03/hr, expect well under an hour):

hf jobs uv train_tokenizer.py --flavor cpu-upgrade --timeout 2h \
  --secrets HF_TOKEN \
  --with tokenizers --with transformers --with huggingface_hub

Knobs are environment-driven: NUM_DOCS (default 1000000), VOCAB_SIZE (32000), PUSH, HUB_ID, OUT_DIR, TMP_DIR. A smoke run is NUM_DOCS=2000 PUSH=1 — it downloads one 320 MB shard and finishes in minutes; the vocab-size assertion only applies to full runs.

Data source: C4 (Colossal Clean Crawled Corpus), Raffel et al., 2019 — https://arxiv.org/abs/1910.10683 (dataset license: ODC-By).