Release Nep-Tokenizer v2 (64k): Native Hugging Face BPE with Mark-Aware Pre-tokenization and 100% Devanagari Block Coverage

a4c3ef3 6 days ago

preview code

Raw History Blame Contribute Delete

10.3 kB

---
language:
- ne
- en
- hi
license: mit
tags:
- tokenizer
- nepali
- devanagari
- bpe
- byte-level-bpe
- mark-aware
- doai-ku
- ilprl
- transformers
- fast-tokenizer
pipeline_tag: feature-extraction
---
# Nep-Tokenizer v2 (64k Native Hugging Face BPE)

**Department of Artificial Intelligence (DOAI) & Information and Language Processing Research Lab (ILPRL)**  
**Kathmandu University, Dhulikhel, Nepal**

[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-DOAIxILPRL--KU%2Fnep--tokenizer-blue)](https://huggingface.co/DOAIxILPRL-KU/nep-tokenizer)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)
[![Native Rust Fast Tokenizer](https://img.shields.io/badge/Backend-Tokenizers%20(Rust%20Fast)-orange.svg)](https://github.com/huggingface/tokenizers)

**Nep-Tokenizer v2** is a native Hugging Face byte-level Byte-Pair Encoding (BPE) tokenizer engineered specifically for Nepali and Devanagari multilingual Large Language Models (LLMs). Developed jointly by the **Department of Artificial Intelligence (DOAI)** and the **Information and Language Processing Research Lab (ILPRL)** at **Kathmandu University**, Nep-Tokenizer v2 solves fundamental structural failure modes present in previous tokenizers—including the Devanagari vowel sign severance anomaly ("Kathmandu Bug"), tail character byte-fallback leakage, and digit splitting discrepancies.

Nep-Tokenizer v2 is **100% Hugging Face native** (`tokenizers.Tokenizer` / `PreTrainedTokenizerFast`), loading directly via `AutoTokenizer.from_pretrained("DOAIxILPRL-KU/nep-tokenizer")` with zero reliance on external SentencePiece binary wrappers.

---

## Key Architectural Breakthroughs

1. **Native Hugging Face Compatibility**: Backed directly by the Rust `tokenizers` library (`tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json`). Operates out-of-the-box with `transformers.AutoTokenizer` with multi-threaded Rust execution.
2. **Mark-Aware Pre-tokenization (`[\p{L}\p{M}]+`)**: Standard GPT-2/LLaMA regex rules categorize Devanagari vowel signs (*matras*) and virama (`\p{M}`) as non-letters, severing every word at every combining mark. Nep-Tokenizer v2 explicitly unifies letters and marks, driving the **severed matra error rate to 0.00%** (100.0% matra cohesion).
3. **Guaranteed 100% Devanagari Block Coverage**: All 128 Unicode code points in the Devanagari block (`U+0900` to `U+097F`) have deterministic single-token merges. Zero byte fallbacks, zero raw byte triplets (`<0xE0>...`), and zero unigram pruning drops for rare vowels (`ऊ, ऋ, ॠ, ऐ, अं, अः, ॄ, ॅ, ॆ, ॉ, ॊ`).
4. **Isolated Digit Grouping (`[0-9०-९]{1,3}`)**: Automatically groups both ASCII and Devanagari numerals in chunks of up to 3 digits, preventing arbitrary number shattering and preserving compositional arithmetic.
5. **Lossless Byte-Level BPE & NFC Normalization**: Canonical Unicode NFC normalization guarantees consistent encoding for visually identical Devanagari combining sequences, while ByteLevel fallback guarantees 100% roundtrip fidelity across all unseen UTF-8 bytes, symbols, and code syntax.

---

## Benchmark Leaderboard (Consistent Multi-Domain Evaluation)

Evaluated across a comprehensive held-out benchmark suite spanning **News**, **Literature**, **Legal Acts**, **Colloquial (Social Media)**, and **Technical / Wikipedia** domains:

| Model | Vocab Size | Nepali Fertility ↓ | English Fertility ↓ | Nepali Tax ↓ | Matra Err% ↓ | Devanagari Cov% ↑ | Bytes/Token (BPT) ↑ |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **DOAIxILPRL Nep-Tokenizer v2 (64k)** | **64,053** | **1.440** | **1.289** | **1.12x** | **0.00%** | **100.0% (128/128)** | **11.93** |
| **DOAIxILPRL Nep-Tokenizer v2 (32k)** | **32,057** | **1.584** | **1.490** | **1.06x** | **0.00%** | **100.0% (128/128)** | **10.84** |
| **Ampixa Yak v2.0 (64k)** | 64,512 | 1.306 | 1.483 | 0.88x | 0.00% | 100.0% (128/128) | 13.15 |
| **Ampixa Yak v1.0 (32k)** | 31,744 | 1.448 | 1.658 | 0.87x | 0.00% | 100.0% (128/128) | 11.86 |
| **Ampixa Yak v2.0 (128k)** | 130,048 | 1.205 | 1.275 | 0.95x | 0.00% | 100.0% (128/128) | 14.26 |
| **Himalaya AI v1 (64k)** | 64,000 | 1.375 | 1.329 | 1.03x | 2.60% | 86.3% (63/73) | 12.49 |
| **Sajal Regmi Arkios (65k)** | 65,510 | 1.556 | 1.201 | 1.30x | 0.00% | 100.0% (128/128) | 11.04 |
| **sidskarki Nepali-BPE (64k)** | 64,000 | 1.450 | 2.772 | 0.52x | 0.00% | 100.0% (128/128) | 11.84 |
| **NyayaLM SentencePiece BPE (64k)** | 63,985 | 1.450 | 1.188 | 1.22x | 4.76% | 89.0% (65/73) | 11.84 |
| **AI4Bharat IndicBERT v2 (250k)** | 250,000 | 1.461 | 1.221 | 1.20x | 0.00% | 100.0% (128/128) | 11.76 |
| **BigScience BLOOM (250k)** | 250,680 | 1.625 | 1.228 | 1.32x | 0.00% | 100.0% (128/128) | 10.57 |
| **BharatGen Param2-17B** | 128,000 | 1.797 | 1.597 | 1.13x | 0.00% | 100.0% (128/128) | 9.56 |
| **IRIIS RoBERTa Nepali (50k)** | 50,256 | 1.994 | 7.893 | 0.25x | 0.00% | 100.0% (128/128) | 8.62 |
| **Sarvam AI Sarvam-105B (262k)** | 262,144 | 2.080 | 1.221 | 1.70x | 8.26% | 93.2% (68/73) | 8.26 |
| **Supernova V5 (48k)** | 47,908 | 4.468 | 1.799 | 2.48x | 43.82% | 100.0% (128/128) | 3.85 |
| **Google Gemma-2 (256k)** | 256,000 | 2.869 | 1.161 | 2.47x | 23.46% | 100.0% (128/128) | 5.99 |
| **Meta LLaMA-3.1 (128k)** | 128,000 | 3.498 | 1.235 | 2.83x | 0.00% | 100.0% (128/128) | 4.91 |

*Definitions*:
- **Fertility**: Average number of subword tokens produced per whitespace-delimited word. Lower is better.
- **Nepali Tax**: Ratio of Nepali fertility to English fertility ($F_{\text{Nepali}} / F_{\text{English}}$). Maps directly to relative API cost and context consumption.
- **Matra Err%**: Percentage of subword tokens beginning with an isolated combining mark (*severed matra*).
- **Devanagari Cov%**: Percentage of Devanagari phonemes encoded as atomic single tokens (not falling back to byte fragments).
- **BPT (Bytes Per Token)**: UTF-8 information density. Higher represents more compressed representation.

---

## Domain Breakdown (Subword Tokens per Word)

| Domain | Nep-Tokenizer v2 (64k) | Nep-Tokenizer v2 (32k) | Yak v2.0 (64k) | Arkios (65k) | NyayaLM (64k) | Supernova V5 | Gemma-2 (256k) | LLaMA-3.1 (128k) |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **News** | **1.106** | 1.179 | 1.058 | 1.196 | 1.116 | 4.072 | 2.502 | 3.018 |
| **Literature** | **1.409** | 1.549 | 1.278 | 1.528 | 1.417 | 4.410 | 2.837 | 3.461 |
| **Legal Acts** | **1.141** | 1.233 | 1.082 | 1.239 | 1.155 | 4.090 | 2.545 | 3.080 |
| **Colloquial** | **2.190** | 2.449 | 1.942 | 2.378 | 2.203 | 5.340 | 3.699 | 4.566 |
| **Technical / Wiki**| **1.643** | 1.838 | 1.439 | 1.777 | 1.637 | 4.675 | 3.078 | 3.737 |

---

## Benchmark Visualizations

![Vocab Scaling & Information Density](figures/fig1_vocab_scaling_fertility.png)

![Multi-Domain Fertility](figures/fig2_domain_fertility_comparison.png)

![Matra Cohesion & Devanagari Coverage](figures/fig3_matra_cohesion_comparison.png)

---

## Quickstart

### 1. Using Hugging Face `transformers`

```python
from transformers import AutoTokenizer
# Load directly from Hugging Face Hub (Fast Rust Tokenizer)
tokenizer = AutoTokenizer.from_pretrained("DOAIxILPRL-KU/nep-tokenizer")
text = "नेपालको राजधानी काठमाडौँ हो ।"
tokens = tokenizer.tokenize(text)
input_ids = tokenizer.encode(text)
print("Tokens:", tokens)
# ['नेपालको', 'राजधानी', 'काठमाडौँ', 'हो', '।']
print("Input IDs:", input_ids)
# [12117, 3517, 3497, 506, 311]
# Lossless Round-Trip Decoding
decoded = tokenizer.decode(input_ids)
assert decoded == text
print("Decoded matches perfectly:", decoded)
```

### 2. Using `tokenizers` (Direct Rust Engine)

```python
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer
tokenizer_path = hf_hub_download("DOAIxILPRL-KU/nep-tokenizer", "tokenizer.json")
tok = Tokenizer.from_file(tokenizer_path)
enc = tok.encode("काठमाडौँ विश्वविद्यालय धुलिखेलमा अवस्थित छ।")
print("Token IDs:", enc.ids)
print("Decoded:", tok.decode(enc.ids))
```

---

## Special Tokens

| Token | ID | Purpose |
| :--- | :---: | :--- |
| `<|pad|>` | 0 | Sequence padding |
| `<|bos|>` | 1 | Beginning of sequence |
| `<|eos|>` | 2 | End of sequence |
| `<|unk|>` | 3 | Out of vocabulary fallback |
| `<|sep|>` | 4 | Utterance separation |
| `<|start_turn|>` | 5 | Multi-turn chat role opening |
| `<|end_turn|>` | 6 | Multi-turn chat role closing |
| `<|think|>` | 7 | Reasoning chain opening |
| `<|/think|>` | 8 | Reasoning chain closing |
| `<|tool_call|>` | 9 | Structured function call invocation |
| `<|/tool_call|>` | 10 | Structured function call closure |
| `<|tool_result|>` | 11 | Execution result response |
| `<|/tool_result|>` | 12 | Execution result closure |

---

## Citation & Attribution

If you use Nep-Tokenizer v2 in your research or application, please cite:

```bibtex
@misc{doai_ilprl_nep_tokenizer_2026,
  author = {Department of Artificial Intelligence (DOAI) and Information and Language Processing Research Lab (ILPRL)},
  title = {Nep-Tokenizer v2: An Open, Lossless, Mark-Aware Byte-Level BPE Tokenizer for Nepali and Multilingual LLMs},
  year = {2026},
  publisher = {Hugging Face},
  journal = {Hugging Face Hub repository},
  howpublished = {\\url{https://huggingface.co/DOAIxILPRL-KU/nep-tokenizer}},
  institution = {Kathmandu University}
}
```

### Acknowledgements & Literature References

Nep-Tokenizer v2 synthesizes findings from seminal works across the tokenizer research community:
- **Ampixa Labs**: *Yak: an open, lossless, Nepali-first tokenizer* (2025). Demonstrated lossless byte-level BPE and complete 128 Devanagari character seeding.
- **Sajal Regmi et al.**: *Vowel Signs Are Not Letters: The Anatomy of the Non-Latin Tokenization Floor* (2025). Established the mark-aware regex formulation `[\p{L}\p{M}]+` and isolated digit grouping.
- **Siddhartha Karki**: *Nepali Tokenizer Benchmark & Strategy Brief* (2025). Established the cross-lingual Nepali Tax metric ($F_{\text{Nepali}} / F_{\text{English}}$).
- **AI4Bharat**: *IndicBERT v2 & Sangraha* (2023–2024). Foundational Indic language resources.

	---
	language:
	- ne
	- en
	- hi
	license: mit
	tags:
	- tokenizer
	- nepali
	- devanagari
	- bpe
	- byte-level-bpe
	- mark-aware
	- doai-ku
	- ilprl
	- transformers
	- fast-tokenizer
	pipeline_tag: feature-extraction
	---

	# Nep-Tokenizer v2 (64k Native Hugging Face BPE)

	Department of Artificial Intelligence (DOAI) & Information and Language Processing Research Lab (ILPRL)
	Kathmandu University, Dhulikhel, Nepal

	[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-DOAIxILPRL--KU%2Fnep--tokenizer-blue)](https://huggingface.co/DOAIxILPRL-KU/nep-tokenizer)
	[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)
	[![Native Rust Fast Tokenizer](https://img.shields.io/badge/Backend-Tokenizers%20(Rust%20Fast)-orange.svg)](https://github.com/huggingface/tokenizers)

	Nep-Tokenizer v2 is a native Hugging Face byte-level Byte-Pair Encoding (BPE) tokenizer engineered specifically for Nepali and Devanagari multilingual Large Language Models (LLMs). Developed jointly by the Department of Artificial Intelligence (DOAI) and the Information and Language Processing Research Lab (ILPRL) at Kathmandu University, Nep-Tokenizer v2 solves fundamental structural failure modes present in previous tokenizers—including the Devanagari vowel sign severance anomaly ("Kathmandu Bug"), tail character byte-fallback leakage, and digit splitting discrepancies.

	Nep-Tokenizer v2 is 100% Hugging Face native (`tokenizers.Tokenizer` / `PreTrainedTokenizerFast`), loading directly via `AutoTokenizer.from_pretrained("DOAIxILPRL-KU/nep-tokenizer")` with zero reliance on external SentencePiece binary wrappers.

	---

	## Key Architectural Breakthroughs

	1. Native Hugging Face Compatibility: Backed directly by the Rust `tokenizers` library (`tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json`). Operates out-of-the-box with `transformers.AutoTokenizer` with multi-threaded Rust execution.
	2. Mark-Aware Pre-tokenization (`[\p{L}\p{M}]+`): Standard GPT-2/LLaMA regex rules categorize Devanagari vowel signs (matras) and virama (`\p{M}`) as non-letters, severing every word at every combining mark. Nep-Tokenizer v2 explicitly unifies letters and marks, driving the severed matra error rate to 0.00% (100.0% matra cohesion).
	3. Guaranteed 100% Devanagari Block Coverage: All 128 Unicode code points in the Devanagari block (`U+0900` to `U+097F`) have deterministic single-token merges. Zero byte fallbacks, zero raw byte triplets (`<0xE0>...`), and zero unigram pruning drops for rare vowels (`ऊ, ऋ, ॠ, ऐ, अं, अः, ॄ, ॅ, ॆ, ॉ, ॊ`).
	4. Isolated Digit Grouping (`[0-9०-९]{1,3}`): Automatically groups both ASCII and Devanagari numerals in chunks of up to 3 digits, preventing arbitrary number shattering and preserving compositional arithmetic.
	5. Lossless Byte-Level BPE & NFC Normalization: Canonical Unicode NFC normalization guarantees consistent encoding for visually identical Devanagari combining sequences, while ByteLevel fallback guarantees 100% roundtrip fidelity across all unseen UTF-8 bytes, symbols, and code syntax.

	---

	## Benchmark Leaderboard (Consistent Multi-Domain Evaluation)

	Evaluated across a comprehensive held-out benchmark suite spanning News, Literature, Legal Acts, Colloquial (Social Media), and Technical / Wikipedia domains:

	\| Model \| Vocab Size \| Nepali Fertility ↓ \| English Fertility ↓ \| Nepali Tax ↓ \| Matra Err% ↓ \| Devanagari Cov% ↑ \| Bytes/Token (BPT) ↑ \|
	\| :--- \| :---: \| :---: \| :---: \| :---: \| :---: \| :---: \| :---: \|
	\| DOAIxILPRL Nep-Tokenizer v2 (64k) \| 64,053 \| 1.440 \| 1.289 \| 1.12x \| 0.00% \| 100.0% (128/128) \| 11.93 \|
	\| DOAIxILPRL Nep-Tokenizer v2 (32k) \| 32,057 \| 1.584 \| 1.490 \| 1.06x \| 0.00% \| 100.0% (128/128) \| 10.84 \|
	\| Ampixa Yak v2.0 (64k) \| 64,512 \| 1.306 \| 1.483 \| 0.88x \| 0.00% \| 100.0% (128/128) \| 13.15 \|
	\| Ampixa Yak v1.0 (32k) \| 31,744 \| 1.448 \| 1.658 \| 0.87x \| 0.00% \| 100.0% (128/128) \| 11.86 \|
	\| Ampixa Yak v2.0 (128k) \| 130,048 \| 1.205 \| 1.275 \| 0.95x \| 0.00% \| 100.0% (128/128) \| 14.26 \|
	\| Himalaya AI v1 (64k) \| 64,000 \| 1.375 \| 1.329 \| 1.03x \| 2.60% \| 86.3% (63/73) \| 12.49 \|
	\| Sajal Regmi Arkios (65k) \| 65,510 \| 1.556 \| 1.201 \| 1.30x \| 0.00% \| 100.0% (128/128) \| 11.04 \|
	\| sidskarki Nepali-BPE (64k) \| 64,000 \| 1.450 \| 2.772 \| 0.52x \| 0.00% \| 100.0% (128/128) \| 11.84 \|
	\| NyayaLM SentencePiece BPE (64k) \| 63,985 \| 1.450 \| 1.188 \| 1.22x \| 4.76% \| 89.0% (65/73) \| 11.84 \|
	\| AI4Bharat IndicBERT v2 (250k) \| 250,000 \| 1.461 \| 1.221 \| 1.20x \| 0.00% \| 100.0% (128/128) \| 11.76 \|
	\| BigScience BLOOM (250k) \| 250,680 \| 1.625 \| 1.228 \| 1.32x \| 0.00% \| 100.0% (128/128) \| 10.57 \|
	\| BharatGen Param2-17B \| 128,000 \| 1.797 \| 1.597 \| 1.13x \| 0.00% \| 100.0% (128/128) \| 9.56 \|
	\| IRIIS RoBERTa Nepali (50k) \| 50,256 \| 1.994 \| 7.893 \| 0.25x \| 0.00% \| 100.0% (128/128) \| 8.62 \|
	\| Sarvam AI Sarvam-105B (262k) \| 262,144 \| 2.080 \| 1.221 \| 1.70x \| 8.26% \| 93.2% (68/73) \| 8.26 \|
	\| Supernova V5 (48k) \| 47,908 \| 4.468 \| 1.799 \| 2.48x \| 43.82% \| 100.0% (128/128) \| 3.85 \|
	\| Google Gemma-2 (256k) \| 256,000 \| 2.869 \| 1.161 \| 2.47x \| 23.46% \| 100.0% (128/128) \| 5.99 \|
	\| Meta LLaMA-3.1 (128k) \| 128,000 \| 3.498 \| 1.235 \| 2.83x \| 0.00% \| 100.0% (128/128) \| 4.91 \|

	Definitions:
	- Fertility: Average number of subword tokens produced per whitespace-delimited word. Lower is better.
	- Nepali Tax: Ratio of Nepali fertility to English fertility ($F_{\text{Nepali}} / F_{\text{English}}$). Maps directly to relative API cost and context consumption.
	- Matra Err%: Percentage of subword tokens beginning with an isolated combining mark (severed matra).
	- Devanagari Cov%: Percentage of Devanagari phonemes encoded as atomic single tokens (not falling back to byte fragments).
	- BPT (Bytes Per Token): UTF-8 information density. Higher represents more compressed representation.

	---

	## Domain Breakdown (Subword Tokens per Word)

	\| Domain \| Nep-Tokenizer v2 (64k) \| Nep-Tokenizer v2 (32k) \| Yak v2.0 (64k) \| Arkios (65k) \| NyayaLM (64k) \| Supernova V5 \| Gemma-2 (256k) \| LLaMA-3.1 (128k) \|
	\| :--- \| :---: \| :---: \| :---: \| :---: \| :---: \| :---: \| :---: \| :---: \|
	\| News \| 1.106 \| 1.179 \| 1.058 \| 1.196 \| 1.116 \| 4.072 \| 2.502 \| 3.018 \|
	\| Literature \| 1.409 \| 1.549 \| 1.278 \| 1.528 \| 1.417 \| 4.410 \| 2.837 \| 3.461 \|
	\| Legal Acts \| 1.141 \| 1.233 \| 1.082 \| 1.239 \| 1.155 \| 4.090 \| 2.545 \| 3.080 \|
	\| Colloquial \| 2.190 \| 2.449 \| 1.942 \| 2.378 \| 2.203 \| 5.340 \| 3.699 \| 4.566 \|
	\| Technical / Wiki\| 1.643 \| 1.838 \| 1.439 \| 1.777 \| 1.637 \| 4.675 \| 3.078 \| 3.737 \|

	---

	## Benchmark Visualizations

	![Vocab Scaling & Information Density](figures/fig1_vocab_scaling_fertility.png)

	![Multi-Domain Fertility](figures/fig2_domain_fertility_comparison.png)

	![Matra Cohesion & Devanagari Coverage](figures/fig3_matra_cohesion_comparison.png)

	---

	## Quickstart

	### 1. Using Hugging Face `transformers`

	```python
	from transformers import AutoTokenizer

	# Load directly from Hugging Face Hub (Fast Rust Tokenizer)
	tokenizer = AutoTokenizer.from_pretrained("DOAIxILPRL-KU/nep-tokenizer")

	text = "नेपालको राजधानी काठमाडौँ हो ।"
	tokens = tokenizer.tokenize(text)
	input_ids = tokenizer.encode(text)

	print("Tokens:", tokens)
	# ['नेपालको', 'राजधानी', 'काठमाडौँ', 'हो', '।']
	print("Input IDs:", input_ids)
	# [12117, 3517, 3497, 506, 311]

	# Lossless Round-Trip Decoding
	decoded = tokenizer.decode(input_ids)
	assert decoded == text
	print("Decoded matches perfectly:", decoded)
	```

	### 2. Using `tokenizers` (Direct Rust Engine)

	```python
	from huggingface_hub import hf_hub_download
	from tokenizers import Tokenizer

	tokenizer_path = hf_hub_download("DOAIxILPRL-KU/nep-tokenizer", "tokenizer.json")
	tok = Tokenizer.from_file(tokenizer_path)

	enc = tok.encode("काठमाडौँ विश्वविद्यालय धुलिखेलमा अवस्थित छ।")
	print("Token IDs:", enc.ids)
	print("Decoded:", tok.decode(enc.ids))
	```

	---

	## Special Tokens

	\| Token \| ID \| Purpose \|
	\| :--- \| :---: \| :--- \|
	\| `<\|pad\|>` \| 0 \| Sequence padding \|
	\| `<\|bos\|>` \| 1 \| Beginning of sequence \|
	\| `<\|eos\|>` \| 2 \| End of sequence \|
	\| `<\|unk\|>` \| 3 \| Out of vocabulary fallback \|
	\| `<\|sep\|>` \| 4 \| Utterance separation \|
	\| `<\|start_turn\|>` \| 5 \| Multi-turn chat role opening \|
	\| `<\|end_turn\|>` \| 6 \| Multi-turn chat role closing \|
	\| `<\|think\|>` \| 7 \| Reasoning chain opening \|
	\| `<\|/think\|>` \| 8 \| Reasoning chain closing \|
	\| `<\|tool_call\|>` \| 9 \| Structured function call invocation \|
	\| `<\|/tool_call\|>` \| 10 \| Structured function call closure \|
	\| `<\|tool_result\|>` \| 11 \| Execution result response \|
	\| `<\|/tool_result\|>` \| 12 \| Execution result closure \|

	---

	## Citation & Attribution

	If you use Nep-Tokenizer v2 in your research or application, please cite:

	```bibtex
	@misc{doai_ilprl_nep_tokenizer_2026,
	author = {Department of Artificial Intelligence (DOAI) and Information and Language Processing Research Lab (ILPRL)},
	title = {Nep-Tokenizer v2: An Open, Lossless, Mark-Aware Byte-Level BPE Tokenizer for Nepali and Multilingual LLMs},
	year = {2026},
	publisher = {Hugging Face},
	journal = {Hugging Face Hub repository},
	howpublished = {\\url{https://huggingface.co/DOAIxILPRL-KU/nep-tokenizer}},
	institution = {Kathmandu University}
	}
	```

	### Acknowledgements & Literature References

	Nep-Tokenizer v2 synthesizes findings from seminal works across the tokenizer research community:
	- Ampixa Labs: Yak: an open, lossless, Nepali-first tokenizer (2025). Demonstrated lossless byte-level BPE and complete 128 Devanagari character seeding.
	- Sajal Regmi et al.: Vowel Signs Are Not Letters: The Anatomy of the Non-Latin Tokenization Floor (2025). Established the mark-aware regex formulation `[\p{L}\p{M}]+` and isolated digit grouping.
	- Siddhartha Karki: Nepali Tokenizer Benchmark & Strategy Brief (2025). Established the cross-lingual Nepali Tax metric ($F_{\text{Nepali}} / F_{\text{English}}$).
	- AI4Bharat: IndicBERT v2 & Sangraha (2023–2024). Foundational Indic language resources.