---
license: apache-2.0
language:
- en
- ig
- ja
- ar
tags:
- Tokenizers
- BPE
---
## NeonTech Tokenizers

Custom multimodal tokenizer family for the NeonTech model series.

## Available Versions

- v1 — 131,072 vocab, DeepSeek V4 base + Igbo/Japanese/Arabic extensions, 80 special tokens
- See the [v1 branch](https://huggingface.co/Neon-tech/Tokenizers/blob/v1/README.md) for full details on special tokens and architecture.


## Usage

```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(“Neon-tech/Tokenizers”, revision=“v1”)
```