1111AI-Tamil1B tokenizer
The 56,000-piece SentencePiece tokenizer used by
mvbalaji/1111AI-Tamil1B.
Trained from scratch on a Tamil / Tanglish / English corpus — not adapted from
an existing multilingual vocabulary.
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("mvbalaji/1111AI-Tamil1B-tokenizer")
tok("தமிழ்நாட்டின் தலைநகரம் எது?")
add_bos_token is false and must stay that way for use with the
1111AI-Tamil1B models — they never saw <s> in training.
Fertility
Tokens per word on held-out text, lower is better:
| bucket | fertility | design target | |
|---|---|---|---|
| Tamil (native script) | 1.962 | 2.00 | ✅ met |
| Tanglish (romanised) | 2.296 | 1.60 | ❌ missed |
| English | 1.526 | 1.40 | ❌ missed |
The Tanglish and English misses are a reported result, not an unfixed bug.
Romanised Tamil has no orthographic standard — the same word appears as
vanakkam, vanakam, vaṇakkam — so a subword vocabulary cannot compress it
the way it compresses native script. That finding is part of the companion
paper's negative result on synthetic transliteration.
Details
| Algorithm | SentencePiece |
| Vocabulary | 56,000 |
| Special tokens | <unk>=0, <s>=1, </s>=2, <|endoftext|>=3, <|pad|>=4 |
| Reserved slots | <|reserved_00|> onward, for future chat/tool tokens |
| Training corpus | 19.05B tokens (47.4% Tamil / 35.1% Tanglish / 17.5% English) |
Reserved slots exist so a chat or tool-calling format can be added later without resizing embeddings or invalidating the pretrained model.
Licence
⚠️ The corpus licence has not yet been confirmed by counsel. The licence field is a placeholder and will be updated.
Built by Balaji (mvbalaji) · 1111AI