agentlans/fineweb-200-tokenizer

FineWeb 200 SentencePiece Tokenizer

A broad-coverage, 200-language SentencePiece tokenizer trained on the agentlans/fineweb-200-weighted dataset using Google SentencePiece.

🚀 Potential Use Cases

• Cross-Lingual Analysis: Quantify the amount and distribution of multilingual material in a language-independent manner.
• Comparative NLP: Study n-gram frequencies, tokenization efficiency, and linguistic compression rates across diverse language families.
• Multilingual Probing: Analyze vocabulary coverage and token fragmentation across low- and high-resource languages.

🛠️ Training Details

The training corpus was prepared using the following preprocessing pipeline:

1. Normalization: Applied NFC Unicode normalization.
2. Sentence Segmentation: Split texts into individual sentences using the ICU package.
3. Deduplication: Formatted as one sentence per line and deduplicated to reduce redundancy.

SentencePiece Configuration

Click here for the SentencePiece configuration

spm_train \
  --input=sentences.txt \
  --model_prefix=spm_200lang_bpe_64k \
  --model_type=bpe \
  --vocab_size=65536 \
  --auto_character_coverage=true \
  --byte_fallback=true \
  --normalization_rule_name=identity \
  --add_dummy_prefix=false \
  --remove_extra_whitespaces=false \
  --split_by_whitespace=true \
  --split_by_unicode_script=false \
  --split_by_number=false \
  --split_digits=false \
  --treat_whitespace_as_suffix=false \
  --allow_whitespace_only_pieces=false \
  --max_sentencepiece_length=128 \
  --max_sentence_length=16384 \
  --pad_id=3 \
  --bos_id=1 \
  --eos_id=2 \
  --unk_id=0 \
  --hard_vocab_limit=true

📦 Example Usage

First download the spm_200lang_bpe_64k.model model.

Then you can load and use the tokenizer with Python:

import sentencepiece as spm

# Load the model
sp = spm.SentencePieceProcessor()
sp.load("spm_200lang_bpe_64k.model")

# Encode text into token IDs
text = "Hello, world! Bonjour le monde!"
tokens = sp.encode(text)
print("Token IDs:", tokens)
# Token IDs: [59785, 36270, 6209, 59746, 16575, 10525, 441, 12765, 59746]

# Decode token IDs back to text
decoded_text = sp.decode(tokens)
print("Decoded:", decoded_text)
# Decoded: Hello, world! Bonjour le monde!

⚠️ Limitations

• Generalist Trade-off: Because it is designed to cover 200 languages, it is not optimized for maximum compression or performance on any single language or specialized domain.
• Token Fragmentation: Words in certain languages may be broken down into multiple subword tokens, which can impact sequence length and chunking strategies.
• Tokenizer Only: This repository contains only the SentencePiece tokenizer model and artifacts—it does not include a complete transformer architecture (e.g., for BERT or causal language modeling).

📄 Licence

This project is licensed under the Apache 2.0 License.