elshadrahimov/miLLi-1.0

miLLi: Model Integrating Local Linguistic Insights for Morphologically Robust Tokenization

miLLi 1.0 (32k vocabulary configuration) is a morphology-aware hybrid tokenizer specifically engineered for the Azerbaijani language, addressing the limitations of standard statistical models (e.g., standard BPE, WordPiece) in processing highly inflected, agglutinative morphologies. Model represents a specialized adaptation and optimization of existing hybrid tokenization frameworks, tailored to resolve the specific morphophonological challenges of Azerbaijani (consonant mutations and vowel loss).

By integrating a rule-based root dictionary with statistical learning, the model prioritizes morphological integrity and semantic root preservation over purely frequency-based compression. The model introduces a dynamic Phonological Restoration algorithm designed to map allomorphic variations (e.g., vowel loss, consonant mutations) back to their canonical root forms during the pre-tokenization phase.

1. Methodology and Architecture

The architecture of miLLi 1.0 is built upon a three-stage hybrid pipeline:

1. Linguistic Pre-processing:

   • Utilization of a cleaned root dictionary based on Mozilla's az.dic.
   • Implementation of an Aho-Corasick based Trie structure for efficient root matching.
   • Case Handling: Application of a special <UPPER> token strategy to consolidate vocabulary and preserve Named Entity Recognition (NER) signals without case-sensitivity redundancy.

2. Phonological Restoration:

   • A dynamic algorithm that identifies phonetically modified stems (e.g., q-ğ, k-y mutations, and second-syllable vowel loss) and restores them to their canonical lemma forms before segmentation.
   • Adopts a "Longest Restored Match" prefix matching approach during the pre-segmentation pass.

3. Statistical Subword Segmentation:

   • A Byte-Pair Encoding (BPE) model with a vocabulary size of 32,000, trained on the Azerbaijani segment of CulturaX [18] (500k line subset).
   • Boundary Protection: Merge rules are strictly restricted from crossing root-suffix boundaries because root candidates (prefixed with _) and suffix candidates (prefixed with ##) are written as whitespace-separated entities during BPE training under the WhitespaceSplit() pre-tokenizer.

2. Empirical Evaluation

The performance of miLLi 1.0 was evaluated using a multi-strategy benchmark covering: Quantitative Efficiency (Token/Word ratio on the Tatoeba corpus), Linguistic Robustness (Morphological Boundary Accuracy & Root Consistency Rate), Vocabulary representation (Vocabulary Sparsity Index), Runtime latency, and Byte-Level Fallback capabilities on out-of-vocabulary scripts.

2.1. Quantitative Analysis: Token Efficiency

Metric: Token/Word (T/W) Ratio (Lower indicates higher compression)

Evaluations on the Tatoeba corpus (5,952 sentences after length filtering) demonstrate that miLLi 1.0 offers a highly balanced representation. While standard frequency-based baselines trained from scratch under identical conditions on the same corpus (Standard BPE and Standard WordPiece) achieve higher compression by memorizing long, inflected words as single tokens, miLLi 1.0 significantly outperforms global multilingual standards.

Model	Category	T/W Ratio
Standard BPE (32000) [Baseline]	Local (Statistical)	1.339
Standard WordPiece (32000) [Baseline]	Local (Statistical)	1.400
aLLMA [11]	Local (Statistical)	1.419
AzeBERT	Local (Statistical)	1.572
XLM-RoBERTa [19]	Global (Multilingual)	1.852
miLLi 1.0 (64000)	Local (Proposed)	1.950
miLLi 1.0 (100000)	Local (Proposed)	1.956
miLLi 1.0 (50000)	Local (Proposed)	1.959
miLLi 1.0 (32000)	Local (Proposed)	1.980
GPT-4o [6]	Global (SOTA)	2.387
mBERT [5]	Global (Multilingual)	2.521
GPT-3.5	Global (Legacy)	3.491

2.2. Qualitative Analysis: Linguistic Robustness

Metrics: Morphological Boundary Accuracy (MBA) & Root Consistency Rate (RCR)

This evaluation measures the model's ability to correctly identify the linguistic root-suffix boundary and dynamically restore phonetically modified stems.

• MBA: Measures the percentage of words split correctly at the root-suffix boundary (evaluated on a 100-word set derived from standard nominal/verbal paradigms of the e-derslik.edu.az curriculum).
• RCR: Evaluates whether the canonical, unmodified root is successfully recovered from a mutated form (evaluated on a 100-word set representing vowel loss and consonant mutations aligned with ANAS orthographical rules).

Comparing miLLi 1.0 to its direct ablation model—Standard BPE (32000)—highlights the immense value of the phonological restoration layer, boosting MBA from 47.0% to 57.0%, and RCR from a near-zero 1.0% to 79.0%.

Model	MBA (%)	RCR (%)
miLLi 1.0 (32000)	57.0%	79.0%
miLLi 1.0 (50000)	54.0%	77.0%
miLLi 1.0 (64000)	53.0%	78.0%
miLLi 1.0 (100000)	51.0%	77.0%
Standard BPE (32000) [Baseline]	47.0%	1.0%
Standard WordPiece (32000) [Baseline]	41.0%	1.0%
XLM-RoBERTa [19]	38.0%	0.0%
AzeBERT	18.0%	0.0%
aLLMA [11]	16.0%	1.0%
mBERT [5]	11.0%	0.0%
GPT-4o [6]	4.0%	0.0%
GPT-3.5	3.0%	0.0%

2.3. Vocabulary Representation and Sparsity

Metric: Vocabulary Sparsity Index (VSI)

VSI measures the cardinality of the union of all generated token sets when tokenizing highly inflected words of the same roots. A balanced VSI score indicates that the tokenizer successfully avoids excessive subword fragmentation while preventing vocabulary inflation.

Model	VSI (Unique Tokens Count)
GPT-3.5	83
AzeBERT	86
miLLi 1.0 (50000)	93
mBERT [5]	93
miLLi 1.0 (100000)	94
miLLi 1.0 (32000)	94
miLLi 1.0 (64000)	94
XLM-RoBERTa [19]	94
aLLMA [11]	97
GPT-4o [6]	97

2.4. Computational Latency and Runtime Throughput

Metric: Amortized Processing Time per Sentence & Throughput (on 5,952 sentences)

The speed benchmark was conducted on 5,952 Tatoeba sentences. While compiled C++/Rust engines (operating without any pre-tokenization layers) exhibit the highest throughput under this evaluation setting, miLLi 1.0 maintains highly stable performance suitable for large-scale applications, showing an amortized processing time of 0.1159 ms per sentence.

Model	Total Time (s)	Avg. Amortized Time / Sentence (ms)	Throughput (sentences/s)
GPT-4o [6]	0.0619	0.0104	96,206.02
GPT-3.5	0.0815	0.0135	74,051.13
Standard WordPiece (32000) [Baseline]	0.2482	0.0417	23,976.00
Standard BPE (32000) [Baseline]	0.2798	0.0470	21,273.37
aLLMA [11]	0.3782	0.0635	15,736.21
XLM-RoBERTa [19]	0.4084	0.0686	14,572.51
mBERT [5]	0.4095	0.0688	14,533.91
AzeBERT	0.4597	0.0772	12,947.48
miLLi 1.0 (32000)	0.6899	0.1159	8,627.26

2.5. Byte-Level Fallback and Open-Vocabulary Robustness

Metrics: Tokens per Character (T/C) & Unknowns ( Count) (on 3,863 unique rare symbols from FineFreq [20])

This benchmark evaluates open-vocabulary handling. By appending the standard 256 UTF-8 byte tokens to the vocabulary, fallback capability is natively supported. For miLLi, most rare symbols (emojis, math operators, etc.) are directly preserved intact by the pre-tokenization wrapper, leading to only 98 unknowns, compared to over 3,000 unknowns produced by standard statistical baselines.

Model	Tokens per Character (T/C)	Unknowns (Count)
XLM-RoBERTa [19]	1.993	0
GPT-4o [6]	2.534	0
GPT-3.5	2.808	0
miLLi 1.0 (50000)	1.028	98
miLLi 1.0 (64000)	1.028	98
miLLi 1.0 (100000)	1.028	98
miLLi 1.0 (32000)	1.029	98
Standard BPE (32000) [Baseline]	1.000	3,055
Standard WordPiece (32000) [Baseline]	1.000	3,055
aLLMA [11]	1.000	3,727
mBERT [5]	1.000	3,789
AzeBERT	0.994	3,823

3. Usage

The model is compatible with the Hugging Face transformers library. Due to the custom Python logic required for phonological restoration, the trust_remote_code=True parameter is mandatory.

Installation

pip install transformers tokenizers
from transformers import AutoTokenizer

### Python Implementation

```python
from transformers import AutoTokenizer

# Load the custom tokenizer
tokenizer = AutoTokenizer.from_pretrained(
    "elshadrahimov/miLLi-1.0", 
    trust_remote_code=True
)

text = "Vətənimizin bayrağı yüksəkliklərdə dalğalanır."

# Tokenize
tokens = tokenizer.tokenize(text)
print("Tokens:", tokens)

# Encode to IDs
input_ids = tokenizer.encode(text)
print("Input IDs:", input_ids)

4. Limitations and Future Directions

• Dictionary Dependence: The phonological restoration capability is bound to the coverage of the underlying root dictionary (az.dic). Neologisms, specific jargon, or dialectisms absent from the dictionary revert to standard BPE segmentation.
• Computational Overhead: Executing custom prefix-matching and restoration rules in Python introduces a measurable latency overhead. The proposed model is approximately 2.78× slower than a Standard WordPiece baseline.
• Integration Protocols: Operating with the trust_remote_code=True parameter may introduce deployment friction in industrial production environments with strict security pipelines.

Future research directions focus on training and evaluating Small Language Models (SLMs) leveraging the miLLi 1.0 tokenizer across downstream tasks such as Text Classification, Named Entity Recognition (NER), and Question Answering (QA).

5. Citation

If you use miLLi 1.0 or our curated evaluation datasets in your research, please cite us as follows:

APA Style: Rahimov, E. (2026). miLLi: Model Integrating Local Linguistic Insights for Morphologically Robust Tokenization. Gazi University Journal of Information Technologies, 2026.

BibTeX:

@article{rahimov2026milli,
  title={miLLi: Model Integrating Local Linguistic Insights for Morphologically Robust Tokenization},
  author={Rahimov, Elshad},
  journal={Gazi University Journal of Information Technologies},
  year={2026}
}

REFERENCES

[1] K. Huseynov, U. Suleymanov, S. Rustamov, J. Huseynov, “Training and evaluation of word embedding models for azerbaijani language”, ADA University 4th International Conference on Computing and Information Technologies, ADA University, Baku, Azerbaijan, 2020. [2] K. Bostrom, G. Durrett, “Byte pair encoding is suboptimal for language model pretraining”, Findings of the Association for Computational Linguistics: EMNLP 2020, Association for Computational Linguistics, 4617-4624, 2020. [3] Y. B. Kaya, A. C. Tantuğ, “Effect of tokenization granularity for turkish large language models”, Intelligent Systems with Applications, 21, 200335, 2024. [4] A. Ziyaden, A. Yelenov, F. Hajiyev, S. Rustamov, A. Pak, “Text data augmentation and pre-trained language model for enhancing text classification of low-resource languages”, PeerJ Computer Science, 10, e1974, 2024. [5] J. Devlin, M. W. Chang, K. Lee, K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding”, arXiv preprint arXiv:1810.04805, 2018. [6] OpenAI, GPT-4o System Card, arXiv preprint arXiv:2410.21276, 2024. [7] R. Sennrich, B. Haddow, A. Birch, “Neural machine translation of rare words with subword units”, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, 1715-1725, 2016. [8] M. Schuster, K. Nakajima, “Japanese and korean voice search”, IEEE 2012 International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, Kyoto, Japan, 5149-5152, 2012. [9] S. Virpioja, P. Smit, S. A. Grönroos, M. Kurimo, Morfessor 2.0: Python implementation and extensions for morfessor baseline, Technical report, Aalto University, 2013. [10] A. A. Akın, M. D. Akın, “Zemberek, an open source nlp framework for turkic languages”, Structure, 10, 1-5, 2007. [11] J. Isbarov, K. Huseynova, E. Mammadov, M. Hajili, D. Ataman, “Open foundation models for azerbaijani language”, Proceedings of the First Workshop on Natural Language Processing for Turkic Languages (SIGTURK 2024), Association for Computational Linguistics, 18-28, 2024. [12] M. A. Bayram et al., “Tokens with meaning: A hybrid tokenization approach for nlp”, arXiv preprint arXiv:2508.14292, 2025. [13] E. Asgari, Y. El Kheir, M. A. S. Javaheri, “Morphbpe: A morpho-aware tokenizer bridging linguistic complexity for efficient llm training across morphologies”, arXiv preprint arXiv:2502.00894, 2025. [14] J. H. Clark, D. Garrette, I. Turc, J. Wieting, “Canine: Pre-training an efficient tokenization-free encoder for language representation”, Transactions of the Association for Computational Linguistics, 10, 73-91, 2022. [15] B. Minixhofer et al., “Bolmo: Byteifying the next generation of language models”, arXiv preprint arXiv:2512.15586, 2025. [16] W. Antoun, F. Baly, H. Hajj, “Arabert: Transformer-based model for arabic language understanding”, Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, 9-15, 2020. [17] A. V. Aho, M. J. Corasick, “Efficient string matching: an aid to bibliographic search”, Communications of the ACM, 18(6), 333-340, 1975. [18] T. Nguyen et al., “CulturaX: A cleaned, enormous, and multilingual dataset for large language models in 167 languages”, Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), ELRA and ICCL, Torino, Italy, 4226-4237, 2024. [19] A. Conneau et al., “Unsupervised cross-lingual representation learning at scale”, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, 8440-8451, 2020. [20] B. Xu, “FineFreq: A Multilingual Character Frequency Dataset from Web-Scale Text”, arXiv preprint arXiv:2512.09701, 2025.