Feature Extraction
Transformers
Nepali
English
Hindi
tokenizer
nepali
devanagari
bpe
byte-level-bpe
mark-aware
doai-ku
ilprl
fast-tokenizer
# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("DOAIxILPRL-KU/nep-tokenizer", device_map="auto")Quick Links
Nep-Tokenizer v2 (64k Native Hugging Face BPE)
Department of Artificial Intelligence (DOAI) & Information and Language Processing Research Lab (ILPRL)
Kathmandu University, Dhulikhel, Nepal
-orange.svg){kind=icon}
Nep-Tokenizer v2 is a native Hugging Face byte-level Byte-Pair Encoding (BPE) tokenizer engineered specifically for Nepali and Devanagari multilingual Large Language Models (LLMs). Developed jointly by the Department of Artificial Intelligence (DOAI) and the Information and Language Processing Research Lab (ILPRL) at Kathmandu University, Nep-Tokenizer v2 solves fundamental structural failure modes present in previous tokenizers—including the Devanagari vowel sign severance anomaly ("Kathmandu Bug"), tail character byte-fallback leakage, and digit splitting discrepancies.
Nep-Tokenizer v2 is 100% Hugging Face native (tokenizers.Tokenizer / PreTrainedTokenizerFast), loading directly via AutoTokenizer.from_pretrained("DOAIxILPRL-KU/nep-tokenizer") with zero reliance on external SentencePiece binary wrappers.
Key Architectural Breakthroughs
- Native Hugging Face Compatibility: Backed directly by the Rust
tokenizerslibrary (tokenizer.json,tokenizer_config.json,special_tokens_map.json). Operates out-of-the-box withtransformers.AutoTokenizerwith multi-threaded Rust execution. - Mark-Aware Pre-tokenization (
[\p{L}\p{M}]+): Standard GPT-2/LLaMA regex rules categorize Devanagari vowel signs (matras) and virama (\p{M}) as non-letters, severing every word at every combining mark. Nep-Tokenizer v2 explicitly unifies letters and marks, driving the severed matra error rate to 0.00% (100.0% matra cohesion). - Guaranteed 100% Devanagari Block Coverage: All 128 Unicode code points in the Devanagari block (
U+0900toU+097F) have deterministic single-token merges. Zero byte fallbacks, zero raw byte triplets (<0xE0>...), and zero unigram pruning drops for rare vowels (ऊ, ऋ, ॠ, ऐ, अं, अः, ॄ, ॅ, ॆ, ॉ, ॊ). - Isolated Digit Grouping (
[0-9०-९]{1,3}): Automatically groups both ASCII and Devanagari numerals in chunks of up to 3 digits, preventing arbitrary number shattering and preserving compositional arithmetic. - Lossless Byte-Level BPE & NFC Normalization: Canonical Unicode NFC normalization guarantees consistent encoding for visually identical Devanagari combining sequences, while ByteLevel fallback guarantees 100% roundtrip fidelity across all unseen UTF-8 bytes, symbols, and code syntax.
Benchmark Leaderboard (Consistent Multi-Domain Evaluation)
Evaluated across a comprehensive held-out benchmark suite spanning News, Literature, Legal Acts, Colloquial (Social Media), and Technical / Wikipedia domains:
| Model | Vocab Size | Nepali Fertility ↓ | English Fertility ↓ | Nepali Tax ↓ | Matra Err% ↓ | Devanagari Cov% ↑ | Bytes/Token (BPT) ↑ |
|---|---|---|---|---|---|---|---|
| DOAIxILPRL Nep-Tokenizer v2 (64k) | 64,053 | 1.440 | 1.289 | 1.12x | 0.00% | 100.0% (128/128) | 11.93 |
| DOAIxILPRL Nep-Tokenizer v2 (32k) | 32,057 | 1.584 | 1.490 | 1.06x | 0.00% | 100.0% (128/128) | 10.84 |
| Ampixa Yak v2.0 (64k) | 64,512 | 1.306 | 1.483 | 0.88x | 0.00% | 100.0% (128/128) | 13.15 |
| Ampixa Yak v1.0 (32k) | 31,744 | 1.448 | 1.658 | 0.87x | 0.00% | 100.0% (128/128) | 11.86 |
| Ampixa Yak v2.0 (128k) | 130,048 | 1.205 | 1.275 | 0.95x | 0.00% | 100.0% (128/128) | 14.26 |
| Himalaya AI v1 (64k) | 64,000 | 1.375 | 1.329 | 1.03x | 2.60% | 86.3% (63/73) | 12.49 |
| Sajal Regmi Arkios (65k) | 65,510 | 1.556 | 1.201 | 1.30x | 0.00% | 100.0% (128/128) | 11.04 |
| sidskarki Nepali-BPE (64k) | 64,000 | 1.450 | 2.772 | 0.52x | 0.00% | 100.0% (128/128) | 11.84 |
| NyayaLM SentencePiece BPE (64k) | 63,985 | 1.450 | 1.188 | 1.22x | 4.76% | 89.0% (65/73) | 11.84 |
| AI4Bharat IndicBERT v2 (250k) | 250,000 | 1.461 | 1.221 | 1.20x | 0.00% | 100.0% (128/128) | 11.76 |
| BigScience BLOOM (250k) | 250,680 | 1.625 | 1.228 | 1.32x | 0.00% | 100.0% (128/128) | 10.57 |
| BharatGen Param2-17B | 128,000 | 1.797 | 1.597 | 1.13x | 0.00% | 100.0% (128/128) | 9.56 |
| IRIIS RoBERTa Nepali (50k) | 50,256 | 1.994 | 7.893 | 0.25x | 0.00% | 100.0% (128/128) | 8.62 |
| Sarvam AI Sarvam-105B (262k) | 262,144 | 2.080 | 1.221 | 1.70x | 8.26% | 93.2% (68/73) | 8.26 |
| Supernova V5 (48k) | 47,908 | 4.468 | 1.799 | 2.48x | 43.82% | 100.0% (128/128) | 3.85 |
| Google Gemma-2 (256k) | 256,000 | 2.869 | 1.161 | 2.47x | 23.46% | 100.0% (128/128) | 5.99 |
| Meta LLaMA-3.1 (128k) | 128,000 | 3.498 | 1.235 | 2.83x | 0.00% | 100.0% (128/128) | 4.91 |
Definitions:
- Fertility: Average number of subword tokens produced per whitespace-delimited word. Lower is better.
- Nepali Tax: Ratio of Nepali fertility to English fertility ($F_{\text{Nepali}} / F_{\text{English}}$). Maps directly to relative API cost and context consumption.
- Matra Err%: Percentage of subword tokens beginning with an isolated combining mark (severed matra).
- Devanagari Cov%: Percentage of Devanagari phonemes encoded as atomic single tokens (not falling back to byte fragments).
- BPT (Bytes Per Token): UTF-8 information density. Higher represents more compressed representation.
Domain Breakdown (Subword Tokens per Word)
| Domain | Nep-Tokenizer v2 (64k) | Nep-Tokenizer v2 (32k) | Yak v2.0 (64k) | Arkios (65k) | NyayaLM (64k) | Supernova V5 | Gemma-2 (256k) | LLaMA-3.1 (128k) |
|---|---|---|---|---|---|---|---|---|
| News | 1.106 | 1.179 | 1.058 | 1.196 | 1.116 | 4.072 | 2.502 | 3.018 |
| Literature | 1.409 | 1.549 | 1.278 | 1.528 | 1.417 | 4.410 | 2.837 | 3.461 |
| Legal Acts | 1.141 | 1.233 | 1.082 | 1.239 | 1.155 | 4.090 | 2.545 | 3.080 |
| Colloquial | 2.190 | 2.449 | 1.942 | 2.378 | 2.203 | 5.340 | 3.699 | 4.566 |
| Technical / Wiki | 1.643 | 1.838 | 1.439 | 1.777 | 1.637 | 4.675 | 3.078 | 3.737 |
Benchmark Visualizations
Quickstart
1. Using Hugging Face transformers
from transformers import AutoTokenizer
# Load directly from Hugging Face Hub (Fast Rust Tokenizer)
tokenizer = AutoTokenizer.from_pretrained("DOAIxILPRL-KU/nep-tokenizer")
text = "नेपालको राजधानी काठमाडौँ हो ।"
tokens = tokenizer.tokenize(text)
input_ids = tokenizer.encode(text)
print("Tokens:", tokens)
# ['नेपालको', 'राजधानी', 'काठमाडौँ', 'हो', '।']
print("Input IDs:", input_ids)
# [12117, 3517, 3497, 506, 311]
# Lossless Round-Trip Decoding
decoded = tokenizer.decode(input_ids)
assert decoded == text
print("Decoded matches perfectly:", decoded)
2. Using tokenizers (Direct Rust Engine)
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer
tokenizer_path = hf_hub_download("DOAIxILPRL-KU/nep-tokenizer", "tokenizer.json")
tok = Tokenizer.from_file(tokenizer_path)
enc = tok.encode("काठमाडौँ विश्वविद्यालय धुलिखेलमा अवस्थित छ।")
print("Token IDs:", enc.ids)
print("Decoded:", tok.decode(enc.ids))
Special Tokens
| Token | ID | Purpose |
|---|---|---|
| `< | pad | >` |
| `< | bos | >` |
| `< | eos | >` |
| `< | unk | >` |
| `< | sep | >` |
| `< | start_turn | >` |
| `< | end_turn | >` |
| `< | think | >` |
| `< | /think | >` |
| `< | tool_call | >` |
| `< | /tool_call | >` |
| `< | tool_result | >` |
| `< | /tool_result | >` |
Citation & Attribution
If you use Nep-Tokenizer v2 in your research or application, please cite:
@misc{doai_ilprl_nep_tokenizer_2026,
author = {Department of Artificial Intelligence (DOAI) and Information and Language Processing Research Lab (ILPRL)},
title = {Nep-Tokenizer v2: An Open, Lossless, Mark-Aware Byte-Level BPE Tokenizer for Nepali and Multilingual LLMs},
year = {2026},
publisher = {Hugging Face},
journal = {Hugging Face Hub repository},
howpublished = {\\url{https://huggingface.co/DOAIxILPRL-KU/nep-tokenizer}},
institution = {Kathmandu University}
}
Acknowledgements & Literature References
Nep-Tokenizer v2 synthesizes findings from seminal works across the tokenizer research community:
- Ampixa Labs: Yak: an open, lossless, Nepali-first tokenizer (2025). Demonstrated lossless byte-level BPE and complete 128 Devanagari character seeding.
- Sajal Regmi et al.: Vowel Signs Are Not Letters: The Anatomy of the Non-Latin Tokenization Floor (2025). Established the mark-aware regex formulation
[\p{L}\p{M}]+and isolated digit grouping. - Siddhartha Karki: Nepali Tokenizer Benchmark & Strategy Brief (2025). Established the cross-lingual Nepali Tax metric ($F_{\text{Nepali}} / F_{\text{English}}$).
- AI4Bharat: IndicBERT v2 & Sangraha (2023–2024). Foundational Indic language resources.
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="DOAIxILPRL-KU/nep-tokenizer")