File size: 582 Bytes
c28b666 fbe6f74 5097c8c c28b666 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 | ---
license: apache-2.0
language:
- en
- ig
- ja
- ar
tags:
- Tokenizers
- BPE
---
## NeonTech Tokenizers
Custom multimodal tokenizer family for the NeonTech model series.
## Available Versions
- v1 — 131,072 vocab, DeepSeek V4 base + Igbo/Japanese/Arabic extensions, 80 special tokens
- See the [v1 branch](https://huggingface.co/Neon-tech/Tokenizers/blob/v1/README.md) for full details on special tokens and architecture.
## Usage
```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(“Neon-tech/Tokenizers”, revision=“v1”)
``` |