Update README.md

README.md

@@ -2,6 +2,8 @@
 language:
 - kmr
 license: apache-2.0
+datasets:
+- kurdish-tech/KurdishCorpus-clean
 tags:
 - tokenizer
 - bpe
@@ -16,6 +18,8 @@ tags:
 
 A byte-level BPE tokenizer for **Kurmancî** (Kurmanji, Northern Kurdish, Latin script), built for **generative language models**.
 
+It splits Kurmancî text into far fewer tokens than general-purpose tokenizers (1.44 tokens per word on FLORES-200, against 1.85 to 2.91 for the tokenizers compared below), it is lossless, and it never needs an `<unk>` token.
+
 ## Highlights
 
 - **Efficient on Kurmancî:** 1.44 tokens per word on FLORES-200 (`kmr_Latn`), compared with 1.85 for XLM-R, 2.62 for Qwen2.5 and 2.91 for GPT-2 (see [Evaluation](#evaluation)).
@@ -23,6 +27,7 @@ A byte-level BPE tokenizer for **Kurmancî** (Kurmanji, Northern Kurdish, Latin
 - **No `<unk>`:** Byte-level fallback means any input can be encoded.
 - **Robust to spelling variants:** The training data includes text written without circumflexes (`e i u` instead of `ê î û`) and in capital letters.
 - **Digits are split one by one,** and words with an apostrophe or hyphen are kept together by the pre-tokenizer before BPE merges are applied.
+- **Chat-ready special tokens:** `<|im_start|>` and `<|im_end|>` are included as reserved special tokens (see [Chat tokens](#chat-tokens)).
 
 ## Quick start
 
@@ -45,16 +50,40 @@ Byte-level tokenizers display the space as `Ġ` and non-ASCII characters (such a
 
 A word at the very beginning of a text has no leading space and is therefore a different token from the same word in the middle of a sentence. This is the same behaviour as in GPT-2 and similar tokenizers.
 
+## Chat tokens
+
+Two special tokens are reserved for chat and instruction models:
+
+| Token | Purpose |
+|---|---|
+| `<\|im_start\|>` | marks the start of a message (followed by the role, for example `user` or `assistant`) |
+| `<\|im_end\|>` | marks the end of a message |
+
+They were **appended after the BPE training** as special tokens and take the two highest IDs. All learned token IDs keep their original values. Because the tokens never appeared during training, a model has to learn their embeddings during fine-tuning on chat data.
+
+A common ChatML-style layout looks like this:
+
+```python
+prompt = (
+    "<|im_start|>user\nRoj baş!<|im_end|>\n"
+    "<|im_start|>assistant\n"
+)
+ids = tok.encode(prompt, add_special_tokens=False)
+```
+
+No chat template is stored in the tokenizer, so you define the exact format yourself when you fine-tune. If you need more special tokens (system prompt, tool calls, ...), add them with `tok.add_special_tokens(...)` and resize the embeddings of your model.
+
 ## Details
 
 | Property | Value |
 |---|---|
 | Type | Byte-level BPE (Hugging Face `tokenizers`) |
-| Vocabulary size | 40,000 |
+| Vocabulary size | 40,002 (40,000 learned tokens + 2 chat tokens) |
 | Normalization | Unicode NFC |
 | Pre-tokenization | Regex split (words incl. internal `'`, `’`, `-`; single digits; punctuation runs; whitespace), then byte-level mapping |
 | Decoder | ByteLevel |
 | Special tokens | `<s>` (bos), `</s>` (eos), `<unk>`, `<pad>`, `<mask>` |
+| Chat tokens | `<\|im_start\|>`, `<\|im_end\|>` |
 | Case | Case-sensitive |
 
 The pre-tokenizer regex is:
@@ -66,7 +95,9 @@ The pre-tokenizer regex is:
 To check the special token IDs of your copy:
 
 ```python
-print({t: tok.convert_tokens_to_ids(t) for t in ["<s>", "</s>", "<unk>", "<pad>", "<mask>"]})
+print(len(tok))
+print({t: tok.convert_tokens_to_ids(t) for t in
+       ["<s>", "</s>", "<unk>", "<pad>", "<mask>", "<|im_start|>", "<|im_end|>"]})
 ```
 
 ## Training data
@@ -84,7 +115,7 @@ Processing steps:
 
 Result: about 1.2 million training lines (about 270 MB of text), about 10,000 held-out lines.
 
-**Important:** The corpus was read as a stream and the run was stopped early. Only roughly the first 400,000 documents of the dataset were read, and about 125,000 of them passed the filters. The mix of sources in this sample was **not balanced** on purpose. A more balanced sample is planned for a later version.
+**Important:** The corpus was read as a stream and the run was stopped early. Only roughly the first 400,000 documents of the dataset were read, and about 125,000 of them passed the filters. The mix of sources in this sample was **not balanced**. A more balanced sample is planned for a later version.
 
 The text sources have their own terms of use. Please check the dataset card of `kurdish-tech/KurdishCorpus-clean` and the licenses of the original sources before you build on this work for anything beyond the tokenizer itself. The tokenizer contains a vocabulary of word pieces, not the source texts.
 
@@ -98,7 +129,7 @@ FLORES-200 was not used for training in any way.
 
 | Tokenizer | Vocabulary | Fertility |
 |---|---|---|
-| **This tokenizer (v4)** | 40,000 | **1.44** |
+| **This tokenizer (v4)** | 40,002 | **1.44** |
 | XLM-R (`xlm-roberta-base`) | about 250,000 | 1.85 |
 | Qwen2.5 (`Qwen/Qwen2.5-0.5B`) | about 151,000 | 2.62 |
 | GPT-2 (`gpt2`) | 50,257 | 2.91 |
@@ -133,56 +164,81 @@ The held-out lines come from the same sources as the training data, so these num
 ## Intended use
 
 - Training or continuing to pre-train generative language models for Kurmancî.
+- Fine-tuning chat and instruction models for Kurmancî (using the reserved chat tokens).
 - Text analysis and tokenization experiments for Kurmancî.
 - A reference for research on tokenization of morphologically rich, low-resource languages.
 
-## Limitations
+## Limitations and risks
 
-- **Not suitable for other languages.** Sorani (Arabic script), Zazaki; Turkish, German, English and others are encoded losslessly but inefficiently. Sorani costs about 10 tokens per word, and emoji cost several tokens each.
+- **Not suitable for other languages.** Sorani (Arabic script), Zazaki, Turkish, German, English and others are encoded losslessly but inefficiently. Sorani costs about 10 tokens per word, and emoji cost several tokens each.
 - **Imbalanced training sample.** The training data comes from a limited part of the dataset. Text types that are rare there (for example literature, spoken-style language, specialist vocabulary) may be split less efficiently.
+- **News and politics skew.** The vocabulary contains many whole-word tokens for place names, institutions and public figures from news text, and fewer for everyday or literary vocabulary. Token frequency reflects the training data only and does not express any opinion.
+- **Some English web spam in the training data.** A small number of tokens come from English-language spam and promotion (for example `casino`, `crypto`, `Semalt`). They occupy well under 0.1 % of the vocabulary and do not affect how Kurmancî text is split. `train_tokenizer.py` has an optional filter (`FILTER_SPAM_ENGLISH`) for a cleaner corpus in a later version.
+- **Personal data.** The vocabulary was checked for e-mail addresses, web addresses, user names and phone numbers. Only names of platforms (such as Twitter or Facebook) and the `@` sign were found. Digits are always split one by one, so numbers cannot be stored as single tokens. Names of places, institutions and public figures appear as tokens because they are frequent in news text.
 - **Some verb forms are split unevenly,** for example `dixwînim` becomes `di` `xw` `înim`. The tokenizer follows frequency, not grammar, and does not know morphology.
 - **Capital letters** are handled less efficiently than lowercase (`ÇIYAYÊN` becomes `Ç` `IYA` `YÊN`).
 - **Apostrophe spellings** such as `ç'ê` or `pe'rê` are not always kept as single units, because BPE merges decide the final split.
+- **Only two chat tokens and no chat template.** See [Chat tokens](#chat-tokens).
 
 ## Reproducing the tokenizer
 
-The script `train_tokenizer.py` in this repository builds the corpus, trains the tokenizer and runs the tests (round-trip, fertility, edge cases). Settings used for this version: `VOCAB_SIZE = 40000`, `min_frequency = 2`, seed 42. Results can differ slightly between runs, because the dataset is read as a shuffled stream.
+The script `train_tokenizer.py` in this repository builds the corpus, trains the tokenizer, appends the chat tokens and runs the tests (round-trip, fertility, edge cases). Settings used for this version: `VOCAB_SIZE = 40000`, `min_frequency = 2`, seed 42. Results can differ slightly between runs, because the dataset is read as a shuffled stream.
+
+## Versioning
+
+This is a test release. Token IDs and the vocabulary may change in a later version. If you train a model on this tokenizer, pin the exact revision (commit hash) with the `revision` argument of `from_pretrained`, so that your model keeps working.
 
 ## Changelog
 
-- **v4 (October 2026):** Byte-level BPE, NFC instead of NFKC, lossless round-trip, spelling variants added to the training data. Replaces an earlier internal version that used a whitespace pre-tokenizer, which lost spaces and mapped unknown characters to `<unk>`.
+- **v4 (October 2026):** Byte-level BPE, NFC instead of NFKC, lossless round-trip, spelling variants added to the training data, two chat special tokens appended after training. Replaces an earlier internal version that used a whitespace pre-tokenizer, which lost spaces and mapped unknown characters to `<unk>`.
 
 ## License
 
-This tokenizer is released under the Apache-2.0 license (see the header of this card). Please also respect the terms of the underlying data sources listed on the dataset card of `kurdish-tech/KurdishCorpus-clean`.
+This tokenizer is released under the Apache-2.0 license. Please also respect the terms of the underlying data sources listed on the dataset card of `kurdish-tech/KurdishCorpus-clean`.
 
 The FLORES-200 evaluation data is licensed separately (CC BY-SA 4.0) and is not part of this repository.
 
-## Acknowledgements
+## Feedback and contact
 
-- `kurdish-tech/KurdishCorpus-clean` for the text data.
-- FLORES-200 (NLLB Team et al., Meta AI) for the evaluation sentences.
-- Everyone who writes, translates and teaches Kurmancî.
+Questions, problems and suggestions are welcome in the *Community* tab of this repository. Feedback from native speakers on how words are split is especially valuable.
 
-## Citation & Acknowledgements
-**Please use the following citation if you build upon this work:**
+## Citation
 
-If you use this tokenizer in your research, projects, or derivative models, please include the following citation:
+If you use this tokenizer in your research, projects or derivative models, please cite it:
 
 ```bibtex
 @misc{neurokurd-kurmanji-tokenizer,
-  author = {NeuroKurd},
-  title = {Kurmancî (Northern Kurdish) Tokenizer v4},
-  year = {2026},
-  publisher = {Hugging Face},
-  howpublished = {\url{https://huggingface.co/NeuroKurd/kurmanji-tokenizer}}
+  author       = {NeuroKurd},
+  title        = {Kurmanc{\^i} (Northern Kurdish) Tokenizer v4},
+  year         = {2026},
+  publisher    = {Hugging Face},
+  howpublished = {\url{https://huggingface.co/NeuroKurd/kurmanji_tokenizer}}
 }
+```
+
+## Acknowledgements
+
+- The authors of `kurdish-tech/KurdishCorpus-clean` for the text data:
 
-We also acknowledge the underlying dataset authors:
+```bibtex
 @misc{kurdish-tech-corpus,
-  author = {kurdish-tech},
-  title = {KurdishCorpus-clean},
-  year = {2026},
-  publisher = {Hugging Face},
+  author       = {kurdish-tech},
+  title        = {KurdishCorpus-clean},
+  year         = {2026},
+  publisher    = {Hugging Face},
   howpublished = {\url{https://huggingface.co/datasets/kurdish-tech/KurdishCorpus-clean}}
-}
+}
+```
+
+- FLORES-200 (NLLB Team, Meta AI) for the evaluation sentences:
+
+```bibtex
+@article{nllb2022,
+  author  = {{NLLB Team}},
+  title   = {No Language Left Behind: Scaling Human-Centered Machine Translation},
+  year    = {2022},
+  journal = {arXiv preprint arXiv:2207.04672}
+}
+```
+
+- Everyone who writes, translates and teaches Kurmancî.