--- license: apache-2.0 language: - so library_name: tokenizers pipeline_tag: feature-extraction tags: - tokenizer - somali - bpe - byte-level --- # Somali Tokenizer Somali ByteLevel BPE v2 (formerly `somast/somali-bpe-v2`, which still redirects here). A 48,000-token byte-level BPE tokenizer for Somali, trained by [Somast](https://www.somast.so) for the [Somali NLP Corpus](https://huggingface.co/datasets/somast/somali-nlp-corpus). - Byte-level, so every input round-trips exactly and there is no unknown token. - 48,000 types, including special tokens `<|endoftext|>`, `<|pad|>`, `<|im_start|>`, `<|im_end|>` and 12 reserved slots for a decoder-LM control set. - Context length set to 8,192 in `tokenizer_config.json`. ## Use ```python from transformers import AutoTokenizer tok = AutoTokenizer.from_pretrained("somast/somali-tokenizer") ids = tok.encode("Caafimaadka carruurtu waa mudnaanteenna koowaad.") print(tok.convert_ids_to_tokens(ids)) # ['Caafimaadka', 'Ġcarruurtu', 'Ġwaa', 'Ġmudnaan', 'teenna', 'Ġkoowaad', '.'] print(tok.decode(ids)) # the input, exactly ``` Or with the [`somali-tokenizer`](https://github.com/somastllc/Somali-Tokenizer) package, a Rust implementation with Python bindings that validates the artifact on load and gives the same ids: ```python # pip install somali-tokenizer from somali_tokenizer import SomaliTokenizer tok = SomaliTokenizer.from_pretrained("somast/somali-tokenizer") # or SomaliTokenizer.v2() offline enc = tok.encode("Caafimaadka carruurtu waa mudnaanteenna koowaad.") print(enc.tokens) print(tok.decode(enc.ids)) ``` The package also ships a command-line tool (`somali-tokenizer encode`, `train`, `benchmark`, `stats`) and a Rust crate of the same name. To pin this exact artifact, pass `revision="v1.0.0"`. ## Compared with general-purpose tokenizers FLORES-200 Somali (dev + devtest, 2,009 sentences, 45,375 words). Tokens per word, lower is better; no special tokens counted. | Tokenizer | Vocabulary | Tokens per word | Single-token words | Tokens vs this | |---|---:|---:|---:|---:| | **somast/somali-tokenizer (this)** | **48,000** | **1.374** | **75.2%** | **1.00×** | | NLLB-200 | 256,204 | 1.818 | 50.8% | 1.32× | | XLM-RoBERTa | 250,002 | 1.827 | 50.9% | 1.33× | | GPT-4o (`o200k_base`) | 200,019 | 1.975 | 45.3% | 1.44× | | Gemma 2 | 256,000 | 2.241 | 35.2% | 1.63× | | Llama 3 | 128,256 | 2.527 | 23.4% | 1.84× | | GPT-4 (`cl100k_base`) | 100,277 | 2.552 | 22.9% | 1.86× | | Qwen 2.5 | 151,665 | 2.568 | 22.9% | 1.87× | On 2,000 Somali Wikipedia paragraphs the order is the same: 1.505 tokens per word here, 2.060 for GPT-4o, 2.632 for GPT-4. The example sentence above is 7 tokens here, 16 with GPT-4o and 19 with GPT-4 and Llama 3. Full tables, method, and caveats (Wikipedia, and possibly FLORES, may overlap the training corpus) are in the [benchmark report](https://github.com/somastllc/Somali-Tokenizer/blob/main/docs/benchmarks/somast-somali-tokenizer-benchmark.pdf) and [BENCHMARKING.md](https://github.com/somastllc/Somali-Tokenizer/blob/main/docs/BENCHMARKING.md). ## Results on the corpus holdout Scored on a held-out split of 63,542 documents (about 0.8% of the Somali NLP Corpus 1.0.0, excluded by a deterministic hash of the document id). Mean tokens per word, lower is better: | Tokenizer | Tokens per word | Round-trip | |---|---:|---:| | **Somali ByteLevel BPE v2 (this)** | **1.3440** | exact on all 63,542 | | Somali BPE v1 (32k, whitespace) | 1.3830 | fails on every document | | XLM-RoBERTa-base | 1.8100 | n/a | | BERT-base-uncased | 2.6344 | n/a | Over the whole 1.0.0 corpus (7,913,008 documents, 826,973,689 words) this tokenizer produces 1,129,084,663 tokens, 1.3653 per word. Per-source figures and the full method are in the [Somali-Tokenizer repository](https://github.com/somastllc/Somali-Tokenizer), including the technical note (`python-reference/PAPER.md`). ## Limits - The vocabulary was trained on an earlier eleven-source build of the corpus (2026-09-01) and was not retrained for the six sources added afterwards (Glot500, Somali Web Corpus, FinePDFs, FineWeb-2, Somali Alpaca, Somali TinyStories). FinePDFs tokenizes above the holdout mean (1.4905), so a retrained vocabulary should help on PDF-style text. - Tuned for Somali. Other languages work, because the tokenizer is byte-level, but are split inefficiently. - Only the tokenizer is licensed here. The corpus it was trained on is released separately, under the per-record licenses described on its card. ## Citation ```bibtex @misc{somali_tokenizer, title = {Somali Tokenizer: Somali ByteLevel BPE v2}, author = {omartood and sharafdin and Somast}, year = {2026}, url = {https://huggingface.co/somast/somali-tokenizer} } ``` Contact: info@somast.so