| --- |
| license: apache-2.0 |
| language: |
| - en |
| - ig |
| - ja |
| - ar |
| tags: |
| - Tokenizers |
| - BPE |
| --- |
| ## NeonTech Tokenizers |
|
|
| Custom multimodal tokenizer family for the NeonTech model series. |
|
|
| ## Available Versions |
|
|
| - v1 — 131,072 vocab, DeepSeek V4 base + Igbo/Japanese/Arabic extensions, 80 special tokens |
| - See the [v1 branch](https://huggingface.co/Neon-tech/Tokenizers/blob/v1/README.md) for full details on special tokens and architecture. |
|
|
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoTokenizer |
| tokenizer = AutoTokenizer.from_pretrained(“Neon-tech/Tokenizers”, revision=“v1”) |
| ``` |
