| --- |
| language: |
| - en |
| - fr |
| - es |
| - de |
| - it |
| - pt |
| - nl |
| - pl |
| - cs |
| - ro |
| - sv |
| - tr |
| - id |
| - vi |
| - ru |
| - uk |
| - be |
| - bg |
| - sr |
| - mk |
| - kk |
| - ky |
| - mn |
| - tg |
| - tt |
| - ba |
| - cv |
| - ab |
| - os |
| - sah |
| - el |
| - he |
| - yi |
| - ar |
| - fa |
| - ur |
| - ug |
| - ps |
| - sd |
| - ckb |
| - th |
| - hi |
| - mr |
| - ne |
| - bn |
| - as |
| - ta |
| - te |
| - ml |
| - gu |
| - kn |
| - or |
| - pa |
| - si |
| - zh |
| - ja |
| - ko |
| license: apache-2.0 |
| library_name: transformers |
| tags: |
| - tokenizer |
| - byte-level-bpe |
| - multilingual |
| - tokenizers |
| --- |
| |
| # TriusLD |
|
|
| Trius's Little Dictionary โ a byte-level BPE tokenizer with **64,972** tokens and |
| native coverage of 19 writing systems, built to spend a limited vocabulary on |
| breadth rather than on English alone. |
|
|
| | | | |
| |---|---| |
| | Vocabulary | 64,972 (256 bytes + 14 special + 64,702 merges) | |
| | Type | byte-level BPE, NFC normalisation | |
| | Scripts | Latin, Cyrillic, Greek, Hebrew, Arabic, Thai, Devanagari, Bengali, Tamil, Telugu, Malayalam, Gujarati, Kannada, Odia, Gurmukhi, Sinhala, Han, Kana, Hangul | |
| | Languages | 58 | |
|
|
| Special tokens: |
|
|
| ``` |
| <|pad|> <|bos|> <|eos|> <|unk|> <|system|> <|user|> <|assistant|> |
| <|end|> <|fim_prefix|> <|fim_middle|> <|fim_suffix|> <|tool_call|> |
| <|tool_result|> <|image|> |
| ``` |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoTokenizer |
| |
| tok = AutoTokenizer.from_pretrained("TriusAI/TriusLD") |
| |
| ids = tok("The quick brown fox โ ๆๆท็ๆฃ่ฒ็็ธ โ ๋น ๋ฅธ ๊ฐ์ ์ฌ์ฐ") |
| print(tok.decode(ids["input_ids"])) |
| ``` |
|
|
| Without `transformers`: |
|
|
| ```python |
| from tokenizers import Tokenizer |
| |
| tok = Tokenizer.from_file("tokenizer.json") |
| print(tok.encode("Hello, ไธ็!").tokens) |
| ``` |
|
|
| ## Behaviour |
|
|
| - **Lossless.** The 256-token byte alphabet means any UTF-8 input encodes and |
| decodes exactly, including scripts this vocabulary was not trained on. |
| - **NFC is applied for you.** Composed and decomposed forms of the same text |
| produce the same ids. NFKC is deliberately *not* used: it would fold fullwidth |
| punctuation and ligatures and destroy information. |
| - **No automatic BOS/EOS.** Add them explicitly; `<|bos|>` and `<|eos|>` exist but |
| are not inserted by `encode`. |
| - **Chinese, both standards.** Simplified and Traditional are distinct code |
| points rather than normalisation variants, and both inventories are carried in |
| the vocabulary. The card's `language` field can only hold an ISO 639 code, so |
| it says `zh`; the compression figures below list the two separately. |
| - **`add_prefix_space=False`**, matching training. |
| - Whitespace is not stripped, and `clean_up_tokenization_spaces` is disabled โ |
| decoding is exact. |
|
|
| ## Compression |
|
|
| Measured on held-out text (FineWeb-2 `test` split, 100,000 characters per |
| language). Tokens per character, lower is better. |
|
|
| | family | tpc | |
| |---|---:| |
| | Latin | 0.299 | |
| | Cyrillic | 0.494 | |
| | Arabic script | 0.519 | |
| | Greek + Hebrew | 0.540 | |
| | Thai | 0.595 | |
| | CJK | 0.730 | |
| | Indic | 0.732 | |
|
|
| Selected languages: |
|
|
| | language | tpc | language | tpc | |
| |---|---:|---|---:| |
| | English | 0.260 | Hindi | 0.657 | |
| | Vietnamese | 0.276 | Japanese | 0.678 | |
| | Spanish | 0.277 | Chinese (S) | 0.708 | |
| | German | 0.293 | Bengali | 0.722 | |
| | Russian | 0.461 | Korean | 0.733 | |
| | Arabic | 0.522 | Tamil | 0.754 | |
| | Hebrew | 0.556 | Chinese (T) | 0.802 | |
|
|
| English is about 3.85 characters per token. For comparison on the same text: |
| cl100k 0.842 average over 58 languages, Llama 3 0.699, Qwen3 0.675, o200k 0.394, |
| Gemma 3 0.392 โ against 0.524 here, at a third to a half of their vocabulary. |
|
|
| ## How it was built |
|
|
| Per-script BPE trained with explicit quotas, then composed by a |
| dependency-respecting union; the top characters of each script are forced to be |
| single tokens by a coverage pass. Details, including two merge-ordering bugs and |
| their symptoms, are in the accompanying article. |
|
|
| Training data: FineWeb (English) and FineWeb-2 (everything else). No training |
| text is redistributed here; this repository contains only tokenizer statistics โ |
| a vocabulary and a list of merge pairs. |
|
|
| ## Licence |
|
|
| **Apache License 2.0.** Copyright 2026 TriusAI. The full text is in `LICENSE`, |
| and a `NOTICE` file records the data provenance. In short: use it for anything, |
| commercially included, provided you retain the copyright and licence notices, |
| state significant changes, and do not use the TriusAI name to endorse your |
| derivative. The licence carries an express patent grant. |
|
|
| A tokenizer's merges are frequency statistics over text rather than a copy of |
| it, and no source text is redistributed here. The training data โ FineWeb and |
| FineWeb-2, both published under ODC-By 1.0 โ is credited in `NOTICE`, because the |
| debt is real even where the law is silent. |
|
|
| ## Limitations |
|
|
| - Korean uses the 2,350 most frequent Hangul syllables plus jamo, not the full |
| 11,172-syllable block; rare syllables cost two or three tokens. |
| - Indic coverage is real but thin โ ten scripts share 2,660 merges. Expect roughly |
| twice the tokens of a 200k-token vocabulary on Hindi, Bengali or Tamil. |
| - Unsupported scripts fall back to bytes at two to four tokens per character. |
| - Code is served by the Latin inventory: adequate, not tuned. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{triusld2026, |
| title = {TriusLD: a compact multilingual byte-level BPE tokenizer}, |
| author = {TriusAI}, |
| year = {2026}, |
| howpublished = {\url{https://huggingface.co/TriusAI/TriusLD}} |
| } |
| ``` |
|
|
