| \--- | |
| license: apache-2.0 | |
| language: | |
|   - ur | |
|   - pa | |
|   - sd | |
|   - ps | |
|   - fa | |
|   - ku | |
| tags: | |
|   - tokenizer | |
|   - multilingual | |
|   - pakistan | |
|   - ir | |
| \--- | |
| \# Strategic Multilingual Tokenizer | |
| Optimized for Pakistan, Iran, and South Asian languages. Provides up to 3X better compression on Urdu, Punjabi, Sindhi, and Persian compared to Mistral and Qwen2. | |
| \## Usage | |
| ```python | |
| from tokenizers import Tokenizer | |
| tok = Tokenizer.from\_file("strategic\_tokenizer\_gpt4.json") | |
| enc = tok.encode("مصنوعی ذہانت دنیا کو بدل رہی ہے") | |