NeuroKurd/kurmanji_tokenizer

Kurmancî (Northern Kurdish) Tokenizer v4

A byte-level BPE tokenizer for Kurmancî (Kurmanji, Northern Kurdish, Latin script), built for generative language models.

It splits Kurmancî text into far fewer tokens than general-purpose tokenizers (1.44 tokens per word on FLORES-200, against 1.85 to 2.91 for the tokenizers compared below), it is lossless, and it never needs an <unk> token.

Highlights

• Efficient on Kurmancî: 1.44 tokens per word on FLORES-200 (kmr_Latn), compared with 1.85 for XLM-R, 2.62 for Qwen2.5 and 2.91 for GPT-2 (see Evaluation).
• Lossless: decode(encode(text)) == text for every held-out line tested and for all edge cases (tabs, line breaks, repeated spaces, emoji, other scripts).
• No <unk>: Byte-level fallback means any input can be encoded.
• Robust to spelling variants: The training data includes text written without circumflexes (e i u instead of ê î û) and in capital letters.
• Digits are split one by one, and words with an apostrophe or hyphen are kept together by the pre-tokenizer before BPE merges are applied.
• Chat-ready special tokens: <|im_start|> and <|im_end|> are included as reserved special tokens (see Chat tokens).

Quick start

from transformers import PreTrainedTokenizerFast

tok = PreTrainedTokenizerFast.from_pretrained("NeuroKurd/kurmanji_tokenizer")

text = "Ez ji hemû hevalên naskirî dûrtir im."
ids = tok.encode(text, add_special_tokens=False)
print(tok.convert_ids_to_tokens(ids))
print(tok.decode(ids) == text)   # True

Byte-level tokenizers display the space as Ġ and non-ASCII characters (such as ê, î, û) as byte symbols when you call convert_ids_to_tokens. To see readable pieces, decode token by token:

[tok.decode([i], clean_up_tokenization_spaces=False) for i in ids]

A word at the very beginning of a text has no leading space and is therefore a different token from the same word in the middle of a sentence. This is the same behaviour as in GPT-2 and similar tokenizers.

Chat tokens

Two special tokens are reserved for chat and instruction models:

Token	Purpose
<|im_start|>	marks the start of a message (followed by the role, for example user or assistant)
<|im_end|>	marks the end of a message

They were appended after the BPE training as special tokens and take the two highest IDs. All learned token IDs keep their original values. Because the tokens never appeared during training, a model has to learn their embeddings during fine-tuning on chat data.

A common ChatML-style layout looks like this:

prompt = (
    "<|im_start|>user\nRoj baş!<|im_end|>\n"
    "<|im_start|>assistant\n"
)
ids = tok.encode(prompt, add_special_tokens=False)

No chat template is stored in the tokenizer, so you define the exact format yourself when you fine-tune. If you need more special tokens (system prompt, tool calls, ...), add them with tok.add_special_tokens(...) and resize the embeddings of your model.

Details

Property	Value
Type	Byte-level BPE (Hugging Face tokenizers)
Vocabulary size	40,002 (40,000 learned tokens + 2 chat tokens)
Normalization	Unicode NFC
Pre-tokenization	Regex split (words incl. internal ', ’, -; single digits; punctuation runs; whitespace), then byte-level mapping
Decoder	ByteLevel
Special tokens	<s> (bos), </s> (eos), <unk>, <pad>, <mask>
Chat tokens	<|im_start|>, <|im_end|>
Case	Case-sensitive

The pre-tokenizer regex is:

 ?[\p{L}\p{M}]+(?:['’\-][\p{L}\p{M}]+)*| ?\p{N}| ?[^\s\p{L}\p{M}\p{N}]+|\s+(?!\S)|\s+

To check the special token IDs of your copy:

print(len(tok))
print({t: tok.convert_tokens_to_ids(t) for t in
       ["<s>", "</s>", "<unk>", "<pad>", "<mask>", "<|im_start|>", "<|im_end|>"]})

Training data

The tokenizer was trained on text from the public dataset kurdish-tech/KurdishCorpus-clean (Hugging Face). Only Kurmancî was used. Sorani and Zazaki were excluded.

Processing steps:

1. Kept documents with dialect == "Kurmanji", lid_label == "kmr_Latn" and lid_prob >= 0.8.
2. Unicode NFC normalization.
3. Split documents into lines. Kept lines of 20 to 2,000 characters with more than 60 % letters and no Arabic-script characters.
4. Removed exact duplicate lines (case-insensitive).
5. About 1 % of documents (chosen by a hash of the first line) were set aside as a held-out set that was never used for training.
6. Added extra variants of some lines: about 12 % with ê î û replaced by e i u, about 4 % additionally with ç ş replaced by c s, and about 1.5 % in capital letters.

Result: about 1.2 million training lines (about 270 MB of text), about 10,000 held-out lines.

Important: The corpus was read as a stream and the run was stopped early. Only roughly the first 400,000 documents of the dataset were read, and about 125,000 of them passed the filters. The mix of sources in this sample was not balanced. A more balanced sample is planned for a later version.

The text sources have their own terms of use. Please check the dataset card of kurdish-tech/KurdishCorpus-clean and the licenses of the original sources before you build on this work for anything beyond the tokenizer itself. The tokenizer contains a vocabulary of word pieces, not the source texts.

Evaluation

Metric: Fertility = number of tokens divided by number of words, with words counted by splitting on whitespace and without special tokens. Lower is better.

FLORES-200 (kmr_Latn, dev + devtest, 2,009 sentences)

FLORES-200 was not used for training in any way.

Tokenizer	Vocabulary	Fertility
This tokenizer (v4)	40,002	1.44
XLM-R (xlm-roberta-base)	about 250,000	1.85
Qwen2.5 (Qwen/Qwen2.5-0.5B)	about 151,000	2.62
GPT-2 (gpt2)	50,257	2.91

This tokenizer was trained only on Kurmancî, while the others cover many languages. The table shows how efficiently each one splits Kurmancî. It does not show that one is better overall. Llama and Gemma tokenizers were not measured.

Held-out lines from the training dataset (10,229 lines)

Test	Result
Fertility, normal spelling	1.318
Fertility, same lines without circumflexes	1.366
Bytes per token	4.75
Round-trip identical	100 % (10,229 / 10,229)
<unk> tokens	0

The held-out lines come from the same sources as the training data, so these numbers are likely a bit optimistic compared with unseen text types. The FLORES result is the more reliable figure.

Edge cases (selection)

Input	Tokens
nexwendiyên	ne xwend iyên
nexwestibûnên	ne xwest ibûn ên
Ez li mala xwe dixebitim.	Ez ▁li ▁mala ▁xwe ▁dixebitim .
Ez ji hemu hevalen naskiri durtir im (no circumflex)	8 tokens for 7 words
2026	2 0 2 6
21.10.2026	single digits and dots

(▁ here stands for a leading space.)

Intended use

• Training or continuing to pre-train generative language models for Kurmancî.
• Fine-tuning chat and instruction models for Kurmancî (using the reserved chat tokens).
• Text analysis and tokenization experiments for Kurmancî.
• A reference for research on tokenization of morphologically rich, low-resource languages.

Limitations and risks

• Not suitable for other languages. Sorani (Arabic script), Zazaki, Turkish, German, English and others are encoded losslessly but inefficiently. Sorani costs about 10 tokens per word, and emoji cost several tokens each.
• Imbalanced training sample. The training data comes from a limited part of the dataset. Text types that are rare there (for example literature, spoken-style language, specialist vocabulary) may be split less efficiently.
• News and politics skew. The vocabulary contains many whole-word tokens for place names, institutions and public figures from news text, and fewer for everyday or literary vocabulary. Token frequency reflects the training data only and does not express any opinion.
• Some English web spam in the training data. A small number of tokens come from English-language spam and promotion (for example casino, crypto, Semalt). They occupy well under 0.1 % of the vocabulary and do not affect how Kurmancî text is split. train_tokenizer.py has an optional filter (FILTER_SPAM_ENGLISH) for a cleaner corpus in a later version.
• Personal data. The vocabulary was checked for e-mail addresses, web addresses, user names and phone numbers. Only names of platforms (such as Twitter or Facebook) and the @ sign were found. Digits are always split one by one, so numbers cannot be stored as single tokens. Names of places, institutions and public figures appear as tokens because they are frequent in news text.
• Some verb forms are split unevenly, for example dixwînim becomes di xw înim. The tokenizer follows frequency, not grammar, and does not know morphology.
• Capital letters are handled less efficiently than lowercase (ÇIYAYÊN becomes Ç IYA YÊN).
• Apostrophe spellings such as ç'ê or pe'rê are not always kept as single units, because BPE merges decide the final split.
• Only two chat tokens and no chat template. See Chat tokens.

Reproducing the tokenizer

The script train_tokenizer.py in this repository builds the corpus, trains the tokenizer, appends the chat tokens and runs the tests (round-trip, fertility, edge cases). Settings used for this version: VOCAB_SIZE = 40000, min_frequency = 2, seed 42. Results can differ slightly between runs, because the dataset is read as a shuffled stream.

Versioning

This is a test release. Token IDs and the vocabulary may change in a later version. If you train a model on this tokenizer, pin the exact revision (commit hash) with the revision argument of from_pretrained, so that your model keeps working.

Changelog

• v4 (October 2026): Byte-level BPE, NFC instead of NFKC, lossless round-trip, spelling variants added to the training data, two chat special tokens appended after training. Replaces an earlier internal version that used a whitespace pre-tokenizer, which lost spaces and mapped unknown characters to <unk>.

License

This tokenizer is released under the Apache-2.0 license. Please also respect the terms of the underlying data sources listed on the dataset card of kurdish-tech/KurdishCorpus-clean.

The FLORES-200 evaluation data is licensed separately (CC BY-SA 4.0) and is not part of this repository.

Feedback and contact

Questions, problems and suggestions are welcome in the Community tab of this repository. Feedback from native speakers on how words are split is especially valuable.

Citation

If you use this tokenizer in your research, projects or derivative models, please cite it:

@misc{neurokurd-kurmanji-tokenizer,
  author       = {NeuroKurd},
  title        = {Kurmanc{\^i} (Northern Kurdish) Tokenizer v4},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/NeuroKurd/kurmanji_tokenizer}}
}

Acknowledgements

• The authors of kurdish-tech/KurdishCorpus-clean for the text data:

@misc{kurdish-tech-corpus,
  author       = {kurdish-tech},
  title        = {KurdishCorpus-clean},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/kurdish-tech/KurdishCorpus-clean}}
}

• FLORES-200 (NLLB Team, Meta AI) for the evaluation sentences:

@article{nllb2022,
  author  = {{NLLB Team}},
  title   = {No Language Left Behind: Scaling Human-Centered Machine Translation},
  year    = {2022},
  journal = {arXiv preprint arXiv:2207.04672}
}

• Everyone who writes, translates and teaches Kurmancî.