---
license: apache-2.0
language:
- ur
- pa
- sd
- ps
- fa
- ku
tags:
- tokenizer
- multilingual
- pakistan
- ir
---
# Strategic Multilingual Tokenizer
Optimized for Pakistan, Iran, and South Asian languages. Provides up to 3X better compression on Urdu, Punjabi, Sindhi, and Persian compared to Mistral and Qwen2.
## Usage
from tokenizers import Tokenizer
tok = Tokenizer.from\_file("strategic\_tokenizer\_gpt4.json")
enc = tok.encode("مصنوعی ذہانت دنیا کو بدل رہی ہے")