---
language:
- ne
- en
- hi
license: mit
tags:
- tokenizer
- nepali
- devanagari
- bpe
- byte-level-bpe
- mark-aware
- doai-ku
- ilprl
- transformers
- fast-tokenizer
pipeline_tag: feature-extraction
---

# Nep-Tokenizer v2 (64k Native Hugging Face BPE)

**Department of Artificial Intelligence (DOAI) & Information and Language Processing Research Lab (ILPRL)**  
**Kathmandu University, Dhulikhel, Nepal**

[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-DOAIxILPRL--KU%2Fnep--tokenizer-blue)](https://huggingface.co/DOAIxILPRL-KU/nep-tokenizer)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)
[![Native Rust Fast Tokenizer](https://img.shields.io/badge/Backend-Tokenizers%20(Rust%20Fast)-orange.svg)](https://github.com/huggingface/tokenizers)

**Nep-Tokenizer v2** is a native Hugging Face byte-level Byte-Pair Encoding (BPE) tokenizer engineered specifically for Nepali and Devanagari multilingual Large Language Models (LLMs). Developed jointly by the **Department of Artificial Intelligence (DOAI)** and the **Information and Language Processing Research Lab (ILPRL)** at **Kathmandu University**, Nep-Tokenizer v2 solves fundamental structural failure modes present in previous tokenizers—including the Devanagari vowel sign severance anomaly ("Kathmandu Bug"), tail character byte-fallback leakage, and digit splitting discrepancies.

Nep-Tokenizer v2 is **100% Hugging Face native** (`tokenizers.Tokenizer` / `PreTrainedTokenizerFast`), loading directly via `AutoTokenizer.from_pretrained("DOAIxILPRL-KU/nep-tokenizer")` with zero reliance on external SentencePiece binary wrappers.

---

## Key Architectural Breakthroughs

1. **Native Hugging Face Compatibility**: Backed directly by the Rust `tokenizers` library (`tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json`). Operates out-of-the-box with `transformers.AutoTokenizer` with multi-threaded Rust execution.
2. **Mark-Aware Pre-tokenization (`[\p{L}\p{M}]+`)**: Standard GPT-2/LLaMA regex rules categorize Devanagari vowel signs (*matras*) and virama (`\p{M}`) as non-letters, severing every word at every combining mark. Nep-Tokenizer v2 explicitly unifies letters and marks, driving the **severed matra error rate to 0.00%** (100.0% matra cohesion).
3. **Guaranteed 100% Devanagari Block Coverage**: All 128 Unicode code points in the Devanagari block (`U+0900` to `U+097F`) have deterministic single-token merges. Zero byte fallbacks, zero raw byte triplets (`<0xE0>...`), and zero unigram pruning drops for rare vowels (`ऊ, ऋ, ॠ, ऐ, अं, अः, ॄ, ॅ, ॆ, ॉ, ॊ`).
4. **Isolated Digit Grouping (`[0-9०-९]{1,3}`)**: Automatically groups both ASCII and Devanagari numerals in chunks of up to 3 digits, preventing arbitrary number shattering and preserving compositional arithmetic.
5. **Lossless Byte-Level BPE & NFC Normalization**: Canonical Unicode NFC normalization guarantees consistent encoding for visually identical Devanagari combining sequences, while ByteLevel fallback guarantees 100% roundtrip fidelity across all unseen UTF-8 bytes, symbols, and code syntax.

---

## Benchmark Leaderboard (Consistent Multi-Domain Evaluation)

Evaluated across a comprehensive held-out benchmark suite spanning **News**, **Literature**, **Legal Acts**, **Colloquial (Social Media)**, and **Technical / Wikipedia** domains:

| Model | Vocab Size | Nepali Fertility ↓ | English Fertility ↓ | Nepali Tax ↓ | Matra Err% ↓ | Devanagari Cov% ↑ | Bytes/Token (BPT) ↑ |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **DOAIxILPRL Nep-Tokenizer v2 (64k)** | **64,053** | **1.440** | **1.289** | **1.12x** | **0.00%** | **100.0% (128/128)** | **11.93** |
| **DOAIxILPRL Nep-Tokenizer v2 (32k)** | **32,057** | **1.584** | **1.490** | **1.06x** | **0.00%** | **100.0% (128/128)** | **10.84** |
| **Ampixa Yak v2.0 (64k)** | 64,512 | 1.306 | 1.483 | 0.88x | 0.00% | 100.0% (128/128) | 13.15 |
| **Ampixa Yak v1.0 (32k)** | 31,744 | 1.448 | 1.658 | 0.87x | 0.00% | 100.0% (128/128) | 11.86 |
| **Ampixa Yak v2.0 (128k)** | 130,048 | 1.205 | 1.275 | 0.95x | 0.00% | 100.0% (128/128) | 14.26 |
| **Himalaya AI v1 (64k)** | 64,000 | 1.375 | 1.329 | 1.03x | 2.60% | 86.3% (63/73) | 12.49 |
| **Sajal Regmi Arkios (65k)** | 65,510 | 1.556 | 1.201 | 1.30x | 0.00% | 100.0% (128/128) | 11.04 |
| **sidskarki Nepali-BPE (64k)** | 64,000 | 1.450 | 2.772 | 0.52x | 0.00% | 100.0% (128/128) | 11.84 |
| **NyayaLM SentencePiece BPE (64k)** | 63,985 | 1.450 | 1.188 | 1.22x | 4.76% | 89.0% (65/73) | 11.84 |
| **AI4Bharat IndicBERT v2 (250k)** | 250,000 | 1.461 | 1.221 | 1.20x | 0.00% | 100.0% (128/128) | 11.76 |
| **BigScience BLOOM (250k)** | 250,680 | 1.625 | 1.228 | 1.32x | 0.00% | 100.0% (128/128) | 10.57 |
| **BharatGen Param2-17B** | 128,000 | 1.797 | 1.597 | 1.13x | 0.00% | 100.0% (128/128) | 9.56 |
| **IRIIS RoBERTa Nepali (50k)** | 50,256 | 1.994 | 7.893 | 0.25x | 0.00% | 100.0% (128/128) | 8.62 |
| **Sarvam AI Sarvam-105B (262k)** | 262,144 | 2.080 | 1.221 | 1.70x | 8.26% | 93.2% (68/73) | 8.26 |
| **Supernova V5 (48k)** | 47,908 | 4.468 | 1.799 | 2.48x | 43.82% | 100.0% (128/128) | 3.85 |
| **Google Gemma-2 (256k)** | 256,000 | 2.869 | 1.161 | 2.47x | 23.46% | 100.0% (128/128) | 5.99 |
| **Meta LLaMA-3.1 (128k)** | 128,000 | 3.498 | 1.235 | 2.83x | 0.00% | 100.0% (128/128) | 4.91 |

*Definitions*:
- **Fertility**: Average number of subword tokens produced per whitespace-delimited word. Lower is better.
- **Nepali Tax**: Ratio of Nepali fertility to English fertility ($F_{\text{Nepali}} / F_{\text{English}}$). Maps directly to relative API cost and context consumption.
- **Matra Err%**: Percentage of subword tokens beginning with an isolated combining mark (*severed matra*).
- **Devanagari Cov%**: Percentage of Devanagari phonemes encoded as atomic single tokens (not falling back to byte fragments).
- **BPT (Bytes Per Token)**: UTF-8 information density. Higher represents more compressed representation.

---

## Domain Breakdown (Subword Tokens per Word)

| Domain | Nep-Tokenizer v2 (64k) | Nep-Tokenizer v2 (32k) | Yak v2.0 (64k) | Arkios (65k) | NyayaLM (64k) | Supernova V5 | Gemma-2 (256k) | LLaMA-3.1 (128k) |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **News** | **1.106** | 1.179 | 1.058 | 1.196 | 1.116 | 4.072 | 2.502 | 3.018 |
| **Literature** | **1.409** | 1.549 | 1.278 | 1.528 | 1.417 | 4.410 | 2.837 | 3.461 |
| **Legal Acts** | **1.141** | 1.233 | 1.082 | 1.239 | 1.155 | 4.090 | 2.545 | 3.080 |
| **Colloquial** | **2.190** | 2.449 | 1.942 | 2.378 | 2.203 | 5.340 | 3.699 | 4.566 |
| **Technical / Wiki**| **1.643** | 1.838 | 1.439 | 1.777 | 1.637 | 4.675 | 3.078 | 3.737 |

---

## Benchmark Visualizations

![Vocab Scaling & Information Density](figures/fig1_vocab_scaling_fertility.png)

![Multi-Domain Fertility](figures/fig2_domain_fertility_comparison.png)

![Matra Cohesion & Devanagari Coverage](figures/fig3_matra_cohesion_comparison.png)

---

## Quickstart

### 1. Using Hugging Face `transformers`

```python
from transformers import AutoTokenizer

# Load directly from Hugging Face Hub (Fast Rust Tokenizer)
tokenizer = AutoTokenizer.from_pretrained("DOAIxILPRL-KU/nep-tokenizer")

text = "नेपालको राजधानी काठमाडौँ हो ।"
tokens = tokenizer.tokenize(text)
input_ids = tokenizer.encode(text)

print("Tokens:", tokens)
# ['नेपालको', 'राजधानी', 'काठमाडौँ', 'हो', '।']
print("Input IDs:", input_ids)
# [12117, 3517, 3497, 506, 311]

# Lossless Round-Trip Decoding
decoded = tokenizer.decode(input_ids)
assert decoded == text
print("Decoded matches perfectly:", decoded)
```

### 2. Using `tokenizers` (Direct Rust Engine)

```python
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer

tokenizer_path = hf_hub_download("DOAIxILPRL-KU/nep-tokenizer", "tokenizer.json")
tok = Tokenizer.from_file(tokenizer_path)

enc = tok.encode("काठमाडौँ विश्वविद्यालय धुलिखेलमा अवस्थित छ।")
print("Token IDs:", enc.ids)
print("Decoded:", tok.decode(enc.ids))
```

---

## Special Tokens

| Token | ID | Purpose |
| :--- | :---: | :--- |
| `<|pad|>` | 0 | Sequence padding |
| `<|bos|>` | 1 | Beginning of sequence |
| `<|eos|>` | 2 | End of sequence |
| `<|unk|>` | 3 | Out of vocabulary fallback |
| `<|sep|>` | 4 | Utterance separation |
| `<|start_turn|>` | 5 | Multi-turn chat role opening |
| `<|end_turn|>` | 6 | Multi-turn chat role closing |
| `<|think|>` | 7 | Reasoning chain opening |
| `<|/think|>` | 8 | Reasoning chain closing |
| `<|tool_call|>` | 9 | Structured function call invocation |
| `<|/tool_call|>` | 10 | Structured function call closure |
| `<|tool_result|>` | 11 | Execution result response |
| `<|/tool_result|>` | 12 | Execution result closure |

---

## Citation & Attribution

If you use Nep-Tokenizer v2 in your research or application, please cite:

```bibtex
@misc{doai_ilprl_nep_tokenizer_2026,
  author = {Department of Artificial Intelligence (DOAI) and Information and Language Processing Research Lab (ILPRL)},
  title = {Nep-Tokenizer v2: An Open, Lossless, Mark-Aware Byte-Level BPE Tokenizer for Nepali and Multilingual LLMs},
  year = {2026},
  publisher = {Hugging Face},
  journal = {Hugging Face Hub repository},
  howpublished = {\\url{https://huggingface.co/DOAIxILPRL-KU/nep-tokenizer}},
  institution = {Kathmandu University}
}
```

### Acknowledgements & Literature References

Nep-Tokenizer v2 synthesizes findings from seminal works across the tokenizer research community:
- **Ampixa Labs**: *Yak: an open, lossless, Nepali-first tokenizer* (2025). Demonstrated lossless byte-level BPE and complete 128 Devanagari character seeding.
- **Sajal Regmi et al.**: *Vowel Signs Are Not Letters: The Anatomy of the Non-Latin Tokenization Floor* (2025). Established the mark-aware regex formulation `[\p{L}\p{M}]+` and isolated digit grouping.
- **Siddhartha Karki**: *Nepali Tokenizer Benchmark & Strategy Brief* (2025). Established the cross-lingual Nepali Tax metric ($F_{\text{Nepali}} / F_{\text{English}}$).
- **AI4Bharat**: *IndicBERT v2 & Sangraha* (2023–2024). Foundational Indic language resources.