Nep-Tokenizer v2 (64k Native Hugging Face BPE)

Department of Artificial Intelligence (DOAI) & Information and Language Processing Research Lab (ILPRL)
Kathmandu University, Dhulikhel, Nepal

Nep-Tokenizer v2 is a native Hugging Face byte-level Byte-Pair Encoding (BPE) tokenizer engineered specifically for Nepali and Devanagari multilingual Large Language Models (LLMs). Developed jointly by the Department of Artificial Intelligence (DOAI) and the Information and Language Processing Research Lab (ILPRL) at Kathmandu University, Nep-Tokenizer v2 solves fundamental structural failure modes present in previous tokenizers—including the Devanagari vowel sign severance anomaly ("Kathmandu Bug"), tail character byte-fallback leakage, and digit splitting discrepancies.

Nep-Tokenizer v2 is 100% Hugging Face native (tokenizers.Tokenizer / PreTrainedTokenizerFast), loading directly via AutoTokenizer.from_pretrained("DOAIxILPRL-KU/nep-tokenizer") with zero reliance on external SentencePiece binary wrappers.

---

Key Architectural Breakthroughs

1. Native Hugging Face Compatibility: Backed directly by the Rust tokenizers library (tokenizer.json, tokenizer_config.json, special_tokens_map.json). Operates out-of-the-box with transformers.AutoTokenizer with multi-threaded Rust execution.
2. Mark-Aware Pre-tokenization ([\p{L}\p{M}]+): Standard GPT-2/LLaMA regex rules categorize Devanagari vowel signs (matras) and virama (\p{M}) as non-letters, severing every word at every combining mark. Nep-Tokenizer v2 explicitly unifies letters and marks, driving the severed matra error rate to 0.00% (100.0% matra cohesion).
3. Guaranteed 100% Devanagari Block Coverage: All 128 Unicode code points in the Devanagari block (U+0900 to U+097F) have deterministic single-token merges. Zero byte fallbacks, zero raw byte triplets (<0xE0>...), and zero unigram pruning drops for rare vowels (ऊ, ऋ, ॠ, ऐ, अं, अः, ॄ, ॅ, ॆ, ॉ, ॊ).
4. Isolated Digit Grouping ([0-9०-९]{1,3}): Automatically groups both ASCII and Devanagari numerals in chunks of up to 3 digits, preventing arbitrary number shattering and preserving compositional arithmetic.
5. Lossless Byte-Level BPE & NFC Normalization: Canonical Unicode NFC normalization guarantees consistent encoding for visually identical Devanagari combining sequences, while ByteLevel fallback guarantees 100% roundtrip fidelity across all unseen UTF-8 bytes, symbols, and code syntax.

---

Benchmark Leaderboard (Consistent Multi-Domain Evaluation)

Evaluated across a comprehensive held-out benchmark suite spanning News, Literature, Legal Acts, Colloquial (Social Media), and Technical / Wikipedia domains:

Model	Vocab Size	Nepali Fertility ↓	English Fertility ↓	Nepali Tax ↓	Matra Err% ↓	Devanagari Cov% ↑	Bytes/Token (BPT) ↑
DOAIxILPRL Nep-Tokenizer v2 (64k)	64,053	1.440	1.289	1.12x	0.00%	100.0% (128/128)	11.93
DOAIxILPRL Nep-Tokenizer v2 (32k)	32,057	1.584	1.490	1.06x	0.00%	100.0% (128/128)	10.84
Ampixa Yak v2.0 (64k)	64,512	1.306	1.483	0.88x	0.00%	100.0% (128/128)	13.15
Ampixa Yak v1.0 (32k)	31,744	1.448	1.658	0.87x	0.00%	100.0% (128/128)	11.86
Ampixa Yak v2.0 (128k)	130,048	1.205	1.275	0.95x	0.00%	100.0% (128/128)	14.26
Himalaya AI v1 (64k)	64,000	1.375	1.329	1.03x	2.60%	86.3% (63/73)	12.49
Sajal Regmi Arkios (65k)	65,510	1.556	1.201	1.30x	0.00%	100.0% (128/128)	11.04
sidskarki Nepali-BPE (64k)	64,000	1.450	2.772	0.52x	0.00%	100.0% (128/128)	11.84
NyayaLM SentencePiece BPE (64k)	63,985	1.450	1.188	1.22x	4.76%	89.0% (65/73)	11.84
AI4Bharat IndicBERT v2 (250k)	250,000	1.461	1.221	1.20x	0.00%	100.0% (128/128)	11.76
BigScience BLOOM (250k)	250,680	1.625	1.228	1.32x	0.00%	100.0% (128/128)	10.57
BharatGen Param2-17B	128,000	1.797	1.597	1.13x	0.00%	100.0% (128/128)	9.56
IRIIS RoBERTa Nepali (50k)	50,256	1.994	7.893	0.25x	0.00%	100.0% (128/128)	8.62
Sarvam AI Sarvam-105B (262k)	262,144	2.080	1.221	1.70x	8.26%	93.2% (68/73)	8.26
Supernova V5 (48k)	47,908	4.468	1.799	2.48x	43.82%	100.0% (128/128)	3.85
Google Gemma-2 (256k)	256,000	2.869	1.161	2.47x	23.46%	100.0% (128/128)	5.99
Meta LLaMA-3.1 (128k)	128,000	3.498	1.235	2.83x	0.00%	100.0% (128/128)	4.91

Definitions:

• Fertility: Average number of subword tokens produced per whitespace-delimited word. Lower is better.
• Nepali Tax: Ratio of Nepali fertility to English fertility ($F_{\text{Nepali}} / F_{\text{English}}$). Maps directly to relative API cost and context consumption.
• Matra Err%: Percentage of subword tokens beginning with an isolated combining mark (severed matra).
• Devanagari Cov%: Percentage of Devanagari phonemes encoded as atomic single tokens (not falling back to byte fragments).
• BPT (Bytes Per Token): UTF-8 information density. Higher represents more compressed representation.

---

Domain Breakdown (Subword Tokens per Word)

Domain	Nep-Tokenizer v2 (64k)	Nep-Tokenizer v2 (32k)	Yak v2.0 (64k)	Arkios (65k)	NyayaLM (64k)	Supernova V5	Gemma-2 (256k)	LLaMA-3.1 (128k)
News	1.106	1.179	1.058	1.196	1.116	4.072	2.502	3.018
Literature	1.409	1.549	1.278	1.528	1.417	4.410	2.837	3.461
Legal Acts	1.141	1.233	1.082	1.239	1.155	4.090	2.545	3.080
Colloquial	2.190	2.449	1.942	2.378	2.203	5.340	3.699	4.566
Technical / Wiki	1.643	1.838	1.439	1.777	1.637	4.675	3.078	3.737

---

Benchmark Visualizations

---

Quickstart

1. Using Hugging Face transformers

from transformers import AutoTokenizer

# Load directly from Hugging Face Hub (Fast Rust Tokenizer)
tokenizer = AutoTokenizer.from_pretrained("DOAIxILPRL-KU/nep-tokenizer")

text = "नेपालको राजधानी काठमाडौँ हो ।"
tokens = tokenizer.tokenize(text)
input_ids = tokenizer.encode(text)

print("Tokens:", tokens)
# ['नेपालको', 'राजधानी', 'काठमाडौँ', 'हो', '।']
print("Input IDs:", input_ids)
# [12117, 3517, 3497, 506, 311]

# Lossless Round-Trip Decoding
decoded = tokenizer.decode(input_ids)
assert decoded == text
print("Decoded matches perfectly:", decoded)

2. Using tokenizers (Direct Rust Engine)

from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer

tokenizer_path = hf_hub_download("DOAIxILPRL-KU/nep-tokenizer", "tokenizer.json")
tok = Tokenizer.from_file(tokenizer_path)

enc = tok.encode("काठमाडौँ विश्वविद्यालय धुलिखेलमा अवस्थित छ।")
print("Token IDs:", enc.ids)
print("Decoded:", tok.decode(enc.ids))

---

Special Tokens

Token	ID	Purpose
`<	pad	>`
`<	bos	>`
`<	eos	>`
`<	unk	>`
`<	sep	>`
`<	start_turn	>`
`<	end_turn	>`
`<	think	>`
`<	/think	>`
`<	tool_call	>`
`<	/tool_call	>`
`<	tool_result	>`
`<	/tool_result	>`

---

Citation & Attribution

If you use Nep-Tokenizer v2 in your research or application, please cite:

@misc{doai_ilprl_nep_tokenizer_2026,
  author = {Department of Artificial Intelligence (DOAI) and Information and Language Processing Research Lab (ILPRL)},
  title = {Nep-Tokenizer v2: An Open, Lossless, Mark-Aware Byte-Level BPE Tokenizer for Nepali and Multilingual LLMs},
  year = {2026},
  publisher = {Hugging Face},
  journal = {Hugging Face Hub repository},
  howpublished = {\\url{https://huggingface.co/DOAIxILPRL-KU/nep-tokenizer}},
  institution = {Kathmandu University}
}

Acknowledgements & Literature References

Nep-Tokenizer v2 synthesizes findings from seminal works across the tokenizer research community:

• Ampixa Labs: Yak: an open, lossless, Nepali-first tokenizer (2025). Demonstrated lossless byte-level BPE and complete 128 Devanagari character seeding.
• Sajal Regmi et al.: Vowel Signs Are Not Letters: The Anatomy of the Non-Latin Tokenization Floor (2025). Established the mark-aware regex formulation [\p{L}\p{M}]+ and isolated digit grouping.
• Siddhartha Karki: Nepali Tokenizer Benchmark & Strategy Brief (2025). Established the cross-lingual Nepali Tax metric ($F_{\text{Nepali}} / F_{\text{English}}$).
• AI4Bharat: IndicBERT v2 & Sangraha (2023–2024). Foundational Indic language resources.