metadata
license: apache-2.0
language:
- en
- ig
- ja
- ar
tags:
- Tokenizers
- BPE
NeonTech Tokenizers
Custom multimodal tokenizer family for the NeonTech model series.
Available Versions
- v1 — 131,072 vocab, DeepSeek V4 base + Igbo/Japanese/Arabic extensions, 80 special tokens
- See the v1 branch for full details on special tokens and architecture.
Usage
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(“Neon-tech/Tokenizers”, revision=“v1”)