nasrellahkharroubi/DarijaDz_Tokenizers

DarijaDz Tokenizers

Three subword tokenizer algorithms, trained on the full DarijaDZ corpus -- Algerian YouTube comments -- each at 5 vocabulary sizes (1K, 5K, 10K, 20K, 30K), for comparison and reuse in Algerian Darija NLP / LM pretraining work.

• SentencePiece Unigram — script-agnostic, byte-fallback enabled, no whitespace-pretokenization assumption (matters since Arabic script / Arabizi / French mix within single documents).
• Unigram + Subword Regularization — the same Unigram model above, sampled at encode time (Kudo 2018 §4) instead of deterministic Viterbi decoding — not a separately trained artifact, see "Loading" below.
• WordPiece — BERT-style ##-continuation pieces, HF tokenizers library (not SentencePiece's WordPiece mode).
• Byte-level BPE — GPT-2/RoBERTa-style, zero OOV by construction.

Repo layout

sentencepiece/
  unigram_{1000,5000,10000,20000,30000}/
    unigram_{N}.model, unigram_{N}.vocab   # raw -- needed for Subword Regularization
    tokenizer.json                          # converted -- for AutoTokenizer (deterministic only)
    tokenizer_config.json, special_tokens_map.json
wordpiece/
  wordpiece_{1000,5000,10000,20000,30000}/
    tokenizer.json                          # self-contained (decoder embedded)
    tokenizer_config.json, special_tokens_map.json
bpe/
  bpe_{1000,5000,10000,20000,30000}/
    tokenizer.json, vocab.json, merges.txt  # self-contained (byte-level pre-tokenizer + decoder)
    tokenizer_config.json, special_tokens_map.json

Loading

Every variant works with AutoTokenizer.from_pretrained(...) (point it at any subfolder above), for deterministic encode/decode:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("bpe/bpe_20000")
# or "wordpiece/wordpiece_20000", or "sentencepiece/unigram_20000"

ids = tok("راني عارف bezzaf")["input_ids"]
tok.decode(ids, skip_special_tokens=True)

WordPiece via AutoTokenizer still has the newline caveat below (nothing about the HF wrapper changes that). SentencePiece Unigram via AutoTokenizer is the plain deterministic model — for Unigram + Subword Regularization (sampled, non-deterministic), enable_sampling=True only exists in the sentencepiece library's own API, not in the Rust tokenizers/AutoTokenizer port, so that variant still needs the raw .model file:

import sentencepiece as spm
sp = spm.SentencePieceProcessor(model_file="sentencepiece/unigram_20000/unigram_20000.model")
ids = sp.encode("راني عارف bezzaf", enable_sampling=True, alpha=0.1, nbest_size=-1)
sp.decode(ids)

The raw .model/.vocab files also work for plain deterministic Unigram without enable_sampling, if you'd rather use the sentencepiece API directly than AutoTokenizer.

Known limitation: WordPiece and literal newlines

WordPiece's pre-tokenizer (WhitespaceSplit) treats \n as ordinary whitespace, indistinguishable from a space once encoded — so tok.decode(tok.encode(text).ids) on text containing a real line break will come back with the newline collapsed to a single space. This is a known limitation of BERT-style WordPiece tokenization generally (the byte-level tokenizers above don't have it). If you need multi-line text to round-trip exactly, wrap the raw tokenizer with this substitution (the same one this project's own evaluation pipeline uses internally, _load_wordpiece() in tokenizer_utils.py):

import re
from tokenizers import Tokenizer

tok = Tokenizer.from_file("wordpiece/wordpiece_20000/tokenizer.json")
_NEWLINE_RE = re.compile(r"\s?\[NEWLINE\]\s?")

def encode(text):
    return tok.encode(text.replace("\n", " [NEWLINE] ")).ids

def decode(ids):
    decoded = tok.decode(ids, skip_special_tokens=False)
    return _NEWLINE_RE.sub("\n", decoded)

[NEWLINE] is already a registered special token in every WordPiece vocab here, so this works with any of the 5 sizes as-is. Even with this wrapper, one narrow edge case remains unfixed: a real space immediately adjacent to a newline (e.g. a trailing space before a line break) can still lose that one extra space on round-trip — full losslessness there would require a byte-level scheme, which would make WordPiece redundant with the BPE tokenizer above.

Evaluation

Computed on a 1,926-document held-out set (excluded from training), identical across all tokenizers here. CF (Compression Factor) — lower is better (fewer tokens / less splitting). Fertility — tokens per whitespace-word, lower is better. Round-trip mismatches — decode(encode(text)) != text count out of 1,926.

Tokenizer	Vocab	CF	Fertility	Round-trip mismatches
Unigram	1,000	0.5595	2.8753	0
Unigram + SR	1,000	0.7374	3.8230	0
WordPiece	1,000	0.8105	4.2118	78
BPE	1,000	0.5278	2.7390	0
Unigram	5,000	0.3594	1.9198	0
Unigram + SR	5,000	0.6086	3.1821	0
WordPiece	5,000	0.8105	4.2118	78
BPE	5,000	0.4089	2.1727	0
Unigram	10,000	0.3181	1.7127	0
Unigram + SR	10,000	0.5730	3.0289	0
WordPiece	10,000	0.8105	4.2118	78
BPE	10,000	0.3736	2.0046	0
Unigram	20,000	0.2868	1.5417	0
Unigram + SR	20,000	0.5477	2.8894	0
WordPiece	20,000	0.3125	1.6637	78
BPE	20,000	0.3429	1.8552	0
Unigram	30,000	0.2715	1.4633	0
Unigram + SR	30,000	0.5365	2.8125	0
WordPiece	30,000	0.2784	1.4805	78
BPE	30,000	0.3244	1.7677	0

WordPiece's round-trip mismatches are exactly the 78 held-out docs matching the space-adjacent-to-newline edge case described above — not random noise, and stable across vocab size since it's a pre-tokenization property, not a vocab-coverage one. On this larger corpus, WordPiece's degenerate small-vocab behavior (identical CF/fertility, a sign it's falling back to near-character coverage) now extends through 10,000 vocab, not just up to 5,000 as on the smaller corpus -- Unigram is the best choice at every vocab size here, with BPE as the zero-OOV fallback.

Training data

DarijaDZ — 16,468,862 documents from both Algerian YouTube comments , cleaned (NFKC-normalized, tachkil-stripped, near-dup filtered via MinHash/LSH). 16,466,936 documents used for training after excluding the 1,926-document held-out evaluation set.

Citation

Kharroubi Nasrellah .
DarijaDz Tokenizers: Algerian Darija Subword Tokenizers.
2026.