Darja-Tokenizer-50M: Domain-Aware Byte-Level BPE for Algerian Darija

Darja-Tokenizer-50M is a specialized byte-level Byte-Pair Encoding (BPE) tokenizer trained from scratch on the Algerian Darja Corpus.

It is the official companion tokenizer for Darja-GPT-50M, designed to provide high tokenization efficiency across dialectal Arabic script, Latin-script Arabizi, and French/English code-switching without out-of-vocabulary (OOV) bottlenecks.

---

Table of Contents

1. Why a Custom Darija Tokenizer?
2. Fertility Benchmark & Token Efficiency
3. Vocabulary & Special Tokens
4. Domain Control Tags & Merging Rationale
5. Automated Boundary Processing (TemplateProcessing)
6. Training Pipeline & Dataset
7. Quickstart & Code Examples
8. Citation & Contact

---

Why a Custom Darija Tokenizer?

Standard multilingual tokenizers (such as LLaMA-3, Qwen2.5, or Mistral) are pre-trained predominantly on Modern Standard Arabic (MSA), English, and high-resource languages. When applied to Algerian Darija (الدارجة الجزائرية / Dziri), they fail severely in three ways:

1. Catastrophic Fragmentation (Over-Tokenization): Colloquial prefixes (e.g., غادي, رانا, ما...ش), dialectal verbal stems, and Maghrebi idioms are shattered into single characters or byte-level pieces, inflating sequence lengths by 50% to 100%.
2. Context Window Starvation: In a fixed 1,024-token budget, an over-fragmenting tokenizer fits barely ~400 words, choking the effective attention horizon of compact language models.
3. Multilingual Code-Switching & Arabizi: Algerian conversations fluidly alternate between Arabic characters and Latin script (Arabizi numbers like 3, 7, 9 and French loanwords). Byte-level BPE natively models raw UTF-8 bytes, guaranteeing zero unknown tokens (<unk>) across arbitrary mixed-script utterances.

---

Fertility Benchmark & Token Efficiency

Fertility measures the average number of tokens required to encode a single whitespace-delimited word (lower is more efficient). Higher fertility indicates severe subword fragmentation, which wastes context window capacity and degrades generation speed.

We performed a large-scale, controlled benchmark across 30 open-weights tokenizers using 200 strictly held-out test documents (341,424 words / 1,855,532 characters across <cooking>, <general>, <podcast>, <sports>, <story>, and <tech> domains) from the Algerian Darja Corpus. All evaluations used raw document text without domain tags or special tokens to ensure absolute fairness.

Comprehensive Tokenizer Comparison (Held-out Algerian Darja)

#	Tokenizer	Architecture / Family	Vocab Size	Fertility ↓	Bytes / Tok ↑	UNK %	Ours Needs Fewer Tokens By (95% CI)
1	★ Custom Darja Tokenizer (Ours)	Byte-Level BPE	24,000	1.431	6.80	0.00%	Baseline (Ours)
2	SI2M-Lab/DarijaBERT	WordPiece (Darija)	80,000	1.427	6.81	0.00%	-0.3% [-1.1%, +0.6%]
3	alger-ia/dziribert	WordPiece (Dziri)	50,000	1.429	6.80	0.00%	-0.1% [-1.1%, +0.8%]
4	UBC-NLP/MARBERT	WordPiece (Arabic Dialects)	100,000	1.440	6.75	0.06%	+0.6% [-0.7%, +1.8%]
5	inceptionai/jais-family-590m	BPE (Arabic/English LLM)	84,992	1.639	5.93	0.00%	+12.7% [+11.3%, +14.0%]
6	ALLaM-AI/ALLaM-7B-Instruct	BPE (Arabic LLM)	64,000	1.651	5.89	0.00%	+13.3% [+11.9%, +14.6%]
7	CAMeL-Lab/bert-base-arabic-camelbert	WordPiece (Arabic Mix)	30,000	1.658	5.87	0.31%*	+13.7% [+12.4%, +14.8%]
8	aubmindlab/bert-base-arabertv2	WordPiece (Arabic)	64,000	1.756	5.54	0.54%*	+18.5% [+17.7%, +19.4%]
9	aubmindlab/aragpt2-base	BPE (Arabic Causal LM)	64,000	1.774	5.48	0.00%	+19.4% [+17.2%, +21.4%]
10	bigscience/bloom-560m	Byte-Level BPE (Multilingual)	250,680	1.799	5.41	0.00%	+20.4% [+19.3%, +21.5%]
11	FacebookAI/xlm-roberta-base	SentencePiece (Multilingual)	250,002	1.961	4.96	0.00%	+27.0% [+26.1%, +28.0%]
12	facebook/mbart-large-50	SentencePiece (Multilingual)	250,054	1.961	4.96	0.00%	+27.0% [+26.1%, +28.0%]
13	mistralai/Mistral-Nemo-Base-2407	Tekken (Multilingual LLM)	131,072	1.990	4.89	0.00%	+28.1% [+27.0%, +29.1%]
14	openai/gpt-oss-20b	Byte-Level BPE	200,019	2.040	4.77	0.00%	+29.8% [+28.9%, +30.7%]
15	facebook/nllb-200-distilled-600M	SentencePiece (200 Langs)	256,204	2.061	4.72	0.09%	+30.6% [+29.8%, +31.3%]
16	MBZUAI-Paris/Atlas-Chat-2B	BPE (Arabic-Centric LLM)	256,000	2.128	4.57	0.00%	+32.8% [+31.9%, +33.6%]
17	QCRI/Fanar-1-9B	BPE (Arabic/English LLM)	128,256	2.132	4.56	0.00%	+32.9% [+32.0%, +33.7%]
18	google/mt5-small	SentencePiece (Multilingual)	250,100	2.221	4.38	0.00%	+35.6% [+34.8%, +36.3%]
19	Qwen/Qwen2.5-0.5B	BPE (Multilingual LLM)	151,665	2.354	4.13	0.00%	+39.2% [+38.4%, +40.0%]
20	Qwen/Qwen3-0.6B	BPE (Multilingual LLM)	151,669	2.354	4.13	0.00%	+39.2% [+38.4%, +40.0%]
21	google-bert/bert-base-multilingual	WordPiece (Multilingual)	119,547	2.376	4.09	0.08%	+39.8% [+39.0%, +40.5%]
22	EleutherAI/gpt-neox-20b	BPE	50,277	3.878	2.51	0.00%	+63.1% [+62.4%, +63.7%]
23	microsoft/phi-4	BPE	100,352	3.900	2.49	0.00%	+63.3% [+62.6%, +63.9%]
24	allenai/OLMo-2-1124-7B	BPE	100,278	3.900	2.49	0.00%	+63.3% [+62.6%, +63.9%]
25	mistralai/Mistral-7B-v0.3	Llama-style BPE	32,768	5.005	1.94	0.00%	+71.4% [+70.8%, +71.9%]
26	microsoft/Phi-3-mini-4k-instruct	BPE	32,011	5.094	1.91	0.00%	+71.9% [+71.3%, +72.4%]
27	HuggingFaceTB/SmolLM2-360M	Byte-Level BPE	49,152	5.247	1.85	0.00%	+72.7% [+72.3%, +73.1%]
28	openai-community/gpt2	Byte-Level BPE	50,257	5.595	1.74	0.00%	+74.4% [+74.0%, +74.8%]
29	tiiuae/falcon-7b	BPE	65,024	5.683	1.71	0.00%	+74.8% [+74.4%, +75.2%]

*Note: High UNK rate (>0.1%) inflates apparent fertility for WordPiece tokenizers without byte fallback by mapping unseen dialectal characters to [UNK] rather than encoding them.

---

Key Empirical Findings

1. Headline vs. Modern Foundation LLMs (Qwen2.5 / Qwen3):

   • Our tokenizer requires 39.2% fewer tokens (95% CI [38.4%, 40.0%]) than Qwen/Qwen2.5-0.5B and Qwen3-0.6B (fertility 1.431 vs. 2.354).
   • In a fixed 1,024-token context window, Darja-GPT fits ~715 real Algerian Darija words, compared to only ~435 words under Qwen2.5—a +64.4% expansion in effective context capacity.

2. Parity with Regional Models at a Fraction of Vocab Size:

   • Reaches identical fertility to regional encoders (SI2M-Lab/DarijaBERT at 1.427, alger-ia/dziribert at 1.429, and MARBERT at 1.440).
   • Crucially, it does so with only 24,000 vocabulary entries, compared to 50,000 for DziriBERT, 80,000 for DarijaBERT, and 100,000 for MARBERT.
   • For a lightweight 50M parameter causal LM, a 24K vocabulary saves 13M to 39M parameters in the token embedding and LM head matrices alone (d_model × V), preserving parameter budget for deeper transformer reasoning layers.

3. Massive Efficiency vs. English/Multilingual Foundational Models:

   • Outperforms Mistral-Nemo (+28.1%), BLOOM-560M (+20.4%), XLM-RoBERTa (+27.0%), and NLLB-200 (+30.6%).
   • Slashes token counts by 63% to 75% compared to Phi-4, Mistral-7B, Phi-3, and GPT-2, which suffer catastrophic single-character and multi-byte decomposition on Arabic and Arabizi script.

4. Zero Out-of-Vocabulary Bottlenecks (0.00% UNK):

   • Because our tokenizer is built as a pure Byte-Level BPE, raw UTF-8 bytes serve as universal leaf nodes. Unseen emojis, rare Arabizi transliterations, and French loanwords are seamlessly encoded without [UNK] degradation.

---

Vocabulary & Special Tokens

The vocabulary contains 24,000 BPE merge tokens, with dedicated slots reserved for document framing and domain conditioning:

Token ID	Token	Role	Functional Purpose
0	<pad>	Padding token	Batched training and sequence alignment
1	<bos>	Beginning of Sequence	Prepend marker for document boundaries
2	<eos>	End of Sequence	Append marker indicating text termination
3	<unk>	Unknown token	Fallback (virtually unused due to byte-level BPE)
4	<cooking>	Domain Tag	Culinary recipes, ingredients, food preparation
5	<general>	Domain Tag	Everyday chats, casual vlogs, comedy, greetings
6	<podcast>	Domain Tag	Mindset, business, marketing, freelancing
7	<sports>	Domain Tag	Football match reactions, CAN, leagues, player news
8	<story>	Domain Tag	Folklore, personal anecdotes, narrative storytelling
9	<tech>	Domain Tag	Consumer tech, hardware, mobile phones, electronics

(Tokens 10 through 23,999 correspond to learned byte-level BPE subwords representing frequent Darija morphemes, Arabic words, and Arabizi tokens).

---

Domain Control Tags & Merging Rationale

During dataset auditing of the 11,151 documents, category volume was analyzed to prevent underrepresented tags from over-memorizing training samples:

DOMAIN_MERGE_MAP = {
    "<general>": "<general>",
    "<cooking>": "<cooking>",
    "<tech>": "<tech>",
    "<sports>": "<sports>",
    "<podcast>": "<podcast>",
    "<story>": "<story>",
    "<comedy>": "<general>",         # Merged: 11 docs (111.5K words) -> merged into <general>
    "<lifestyle_vlog>": "<general>",  # Merged: 30 docs (142.7K words) -> merged into <general>
}

• Rationale: <comedy> and <lifestyle_vlog> had insufficient individual volume (<30 documents). Forcing isolated control tokens would risk verbatim memorization across 6 training epochs. Merging them into <general> concentrated stylistic colloquialisms while keeping the remaining 5 primary domains robustly populated (>200+ docs, 1.5M+ words each).

---

Automated Boundary Processing (TemplateProcessing)

The tokenizer includes a built-in post-processor that automatically wraps every encoded single string with <bos> and <eos> tokens:

from tokenizers.processors import TemplateProcessing

tokenizer._tokenizer.post_processor = TemplateProcessing(
    single="<bos> $A <eos>",
    special_tokens=[("<bos>", 1), ("<eos>", 2)],
)

Why this matters:

• In autoregressive causal language modeling, documents are packed into fixed 1,024-token blocks.
• The built-in template ensures that document boundaries are preserved without requiring manual list manipulation before feeding tensors to the model.

---

Training Pipeline & Dataset

• Corpus Source: touati-kamel/algerian-darja-corpus (named subset conditioned)
• Training Documents: 10,816 documents (97% split)
• Held-out Documents: 335 documents (3% split, fixed seed 42)
• Algorithm: Hugging Face tokenizers.ByteLevelBPETokenizer
• Min Frequency: 2 occurrences
• Format: Exported as Hugging Face PreTrainedTokenizerFast (providing tokenizer.json, vocab.json, merges.txt, tokenizer_config.json, and special_tokens_map.json).

---

Quickstart & Code Examples

1. Installation

pip install transformers tokenizers

2. Basic Encoding & Decoding

from transformers import AutoTokenizer

TOKENIZER_ID = "touati-kamel/darja-tokenizer-50m"

# Load the fast tokenizer
tokenizer = AutoTokenizer.from_pretrained(TOKENIZER_ID)

text = "اليوم غادي نديرو شربة فريك بنينة بزاف، هايل يا خويا"
tokens = tokenizer.tokenize(text)
input_ids = tokenizer.encode(text)

print(f"Text: {text}")
print(f"Tokens: {tokens}")
print(f"Token IDs: {input_ids}")
# Notice the automatic <bos> (1) at the beginning and <eos> (2) at the end!

3. Handling Mixed Arabic, Arabizi, and French

mixed_text = "salam khoya, chouf had le telephone 3ando batterie kbira w chbab"
print("Tokens:", tokenizer.tokenize(mixed_text))
print("Decoded:", tokenizer.decode(tokenizer.encode(mixed_text)))

4. Domain Conditioning Example

prompt = "<cooking> وصفة طاجين الزيتون على الطريقة الجزائرية"
encoded = tokenizer(prompt, return_tensors="pt")
print("Input IDs:", encoded["input_ids"])
print("First token is <cooking> tag:", tokenizer.decode([encoded["input_ids"][0][1]]))

5. Computing Fertility Against Other Tokenizers

sample_sentence = "البارح تفرجت الماتش تاع ليكيب ناسيونال ولعبو مليح بزاف مع السنغال"

words = len(sample_sentence.split())
darja_tokens = len(tokenizer.encode(sample_sentence, add_special_tokens=False))

print(f"Words: {words}")
print(f"Darja-Tokenizer Tokens: {darja_tokens} (Fertility: {darja_tokens / words:.2f})")

---

Citation & Contact

If you use Darja-Tokenizer-50M or the Algerian Darja Corpus, please cite:

@misc{touati2026darjatokenizer,
  author       = {Kamel Touati},
  title        = {Darja-Tokenizer-50M: Domain-Aware Byte-Level BPE Tokenizer for Algerian Darija},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/touati-kamel/darja-tokenizer-50m}}
}

@misc{touati2026darjacorpus,
  author       = {Kamel Touati},
  title        = {Algerian Darja Corpus: Multi-Domain Conversational Transcripts in Algerian Arabic},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/touati-kamel/algerian-darja-corpus}}
}

Developer: Kamel Touati (Hugging Face Profile)
License: Apache 2.0
Companion Model: Darja-GPT-50M