File size: 5,367 Bytes

---
language:
  - en
  - fr
  - es
  - de
  - it
  - pt
  - nl
  - pl
  - cs
  - ro
  - sv
  - tr
  - id
  - vi
  - ru
  - uk
  - be
  - bg
  - sr
  - mk
  - kk
  - ky
  - mn
  - tg
  - tt
  - ba
  - cv
  - ab
  - os
  - sah
  - el
  - he
  - yi
  - ar
  - fa
  - ur
  - ug
  - ps
  - sd
  - ckb
  - th
  - hi
  - mr
  - ne
  - bn
  - as
  - ta
  - te
  - ml
  - gu
  - kn
  - or
  - pa
  - si
  - zh
  - ja
  - ko
license: apache-2.0
library_name: transformers
tags:
  - tokenizer
  - byte-level-bpe
  - multilingual
  - tokenizers
---

# TriusLD

Trius's Little Dictionary — a byte-level BPE tokenizer with **64,972** tokens and
native coverage of 19 writing systems, built to spend a limited vocabulary on
breadth rather than on English alone.

| | |
|---|---|
| Vocabulary | 64,972 (256 bytes + 14 special + 64,702 merges) |
| Type | byte-level BPE, NFC normalisation |
| Scripts | Latin, Cyrillic, Greek, Hebrew, Arabic, Thai, Devanagari, Bengali, Tamil, Telugu, Malayalam, Gujarati, Kannada, Odia, Gurmukhi, Sinhala, Han, Kana, Hangul |
| Languages | 58 |

Special tokens:

```
<|pad|>  <|bos|>  <|eos|>  <|unk|>  <|system|>  <|user|>  <|assistant|>
<|end|>  <|fim_prefix|>  <|fim_middle|>  <|fim_suffix|>  <|tool_call|>
<|tool_result|>  <|image|>
```

## Usage

```python
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("TriusAI/TriusLD")

ids = tok("The quick brown fox — 敏捷的棕色狐狸 — 빠른 갈색 여우")
print(tok.decode(ids["input_ids"]))
```

Without `transformers`:

```python
from tokenizers import Tokenizer

tok = Tokenizer.from_file("tokenizer.json")
print(tok.encode("Hello, 世界!").tokens)
```

## Behaviour

- **Lossless.** The 256-token byte alphabet means any UTF-8 input encodes and
  decodes exactly, including scripts this vocabulary was not trained on.
- **NFC is applied for you.** Composed and decomposed forms of the same text
  produce the same ids. NFKC is deliberately *not* used: it would fold fullwidth
  punctuation and ligatures and destroy information.
- **No automatic BOS/EOS.** Add them explicitly; `<|bos|>` and `<|eos|>` exist but
  are not inserted by `encode`.
- **Chinese, both standards.** Simplified and Traditional are distinct code
  points rather than normalisation variants, and both inventories are carried in
  the vocabulary. The card's `language` field can only hold an ISO 639 code, so
  it says `zh`; the compression figures below list the two separately.
- **`add_prefix_space=False`**, matching training.
- Whitespace is not stripped, and `clean_up_tokenization_spaces` is disabled —
  decoding is exact.

## Compression

Measured on held-out text (FineWeb-2 `test` split, 100,000 characters per
language). Tokens per character, lower is better.

| family | tpc |
|---|---:|
| Latin | 0.299 |
| Cyrillic | 0.494 |
| Arabic script | 0.519 |
| Greek + Hebrew | 0.540 |
| Thai | 0.595 |
| CJK | 0.730 |
| Indic | 0.732 |

Selected languages:

| language | tpc | language | tpc |
|---|---:|---|---:|
| English | 0.260 | Hindi | 0.657 |
| Vietnamese | 0.276 | Japanese | 0.678 |
| Spanish | 0.277 | Chinese (S) | 0.708 |
| German | 0.293 | Bengali | 0.722 |
| Russian | 0.461 | Korean | 0.733 |
| Arabic | 0.522 | Tamil | 0.754 |
| Hebrew | 0.556 | Chinese (T) | 0.802 |

English is about 3.85 characters per token. For comparison on the same text:
cl100k 0.842 average over 58 languages, Llama 3 0.699, Qwen3 0.675, o200k 0.394,
Gemma 3 0.392 — against 0.524 here, at a third to a half of their vocabulary.

## How it was built

Per-script BPE trained with explicit quotas, then composed by a
dependency-respecting union; the top characters of each script are forced to be
single tokens by a coverage pass. Details, including two merge-ordering bugs and
their symptoms, are in the accompanying article.

Training data: FineWeb (English) and FineWeb-2 (everything else). No training
text is redistributed here; this repository contains only tokenizer statistics —
a vocabulary and a list of merge pairs.

## Licence

**Apache License 2.0.** Copyright 2026 TriusAI. The full text is in `LICENSE`,
and a `NOTICE` file records the data provenance. In short: use it for anything,
commercially included, provided you retain the copyright and licence notices,
state significant changes, and do not use the TriusAI name to endorse your
derivative. The licence carries an express patent grant.

A tokenizer's merges are frequency statistics over text rather than a copy of
it, and no source text is redistributed here. The training data — FineWeb and
FineWeb-2, both published under ODC-By 1.0 — is credited in `NOTICE`, because the
debt is real even where the law is silent.

## Limitations

- Korean uses the 2,350 most frequent Hangul syllables plus jamo, not the full
  11,172-syllable block; rare syllables cost two or three tokens.
- Indic coverage is real but thin — ten scripts share 2,660 merges. Expect roughly
  twice the tokens of a 200k-token vocabulary on Hindi, Bengali or Tamil.
- Unsupported scripts fall back to bytes at two to four tokens per character.
- Code is served by the Latin inventory: adequate, not tuned.

## Citation

```bibtex
@misc{triusld2026,
  title  = {TriusLD: a compact multilingual byte-level BPE tokenizer},
  author = {TriusAI},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/TriusAI/TriusLD}}
}
```