File size: 4,255 Bytes
11dcd48 293d6e8 11dcd48 8e99050 11dcd48 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 | ---
library_name: transformers
tags:
- tokenizer
- unigram
- byte-level
- multilingual
- code
language:
- en
- de
- fr
- es
- it
- pt
- nl
- pl
- tr
- vi
- id
- cs
- ro
- hu
- da
- zh
- ru
- ja
- ko
- el
- hi
- ar
- he
- th
- bn
- ta
- km
- ka
- hy
- my
- kn
- si
- te
- gu
- ml
- pa
- lo
- bo
- am
- uk
- sr
- or
- dv
- mr
- ne
- mn
---
# U32k: 32k byte-level Unigram tokenizer
A 32,768-id byte-level Unigram tokenizer (HuggingFace `tokenizers` format, ByteLevel
pre-tokenizer and decoder) with 256 control tokens and 32,512 learned pieces. English-first,
with code and 45 additional languages. Trained with an exact GPU Unigram EM trainer
(1.57M seed candidates, shrink 0.999 per prune round, 2,223 rounds, 2 final EM passes).
## Usage
```python
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("xTimeCrystal/U32k")
ids = tok("The quick brown fox")["input_ids"]
text = tok.decode(ids) # exact roundtrip (byte-level)
batch = tok(["a", "longer text"], padding=True, return_tensors="pt") # pad id 0
```
Without transformers:
```python
from tokenizers import Tokenizer
raw = Tokenizer.from_file("tokenizer.json")
raw.encode("text").ids
```
BOS/EOS are not inserted automatically (no post-processor). Add them explicitly:
```python
ids = [tok.bos_token_id] + tok(text)["input_ids"] + [tok.eos_token_id]
```
## Special tokens (ids 0-255)
| id | token | id | token |
|---|---|---|---|
| 0 | `<\|pad\|>` | 9 | `<\|tool_call\|>` |
| 1 | `<\|begin_of_text\|>` (bos) | 10 | `<\|tool_result\|>` |
| 2 | `<\|end_of_text\|>` (eos) | 11 | `<\|think\|>` |
| 3 | `<\|user\|>` | 12 | `<\|end_think\|>` |
| 4 | `<\|assistant\|>` | 13 | `<\|mask\|>` |
| 5 | `<\|system\|>` | 14 | `<\|sep\|>` |
| 6 | `<\|start_header\|>` | 15 | `<\|unk\|>` |
| 7 | `<\|end_header\|>` | 16-18 | `<\|fim_prefix\|>` `<\|fim_middle\|>` `<\|fim_suffix\|>` |
| 8 | `<\|end_turn\|>` | 19-20 | `<\|repo_name\|>` `<\|file_sep\|>` |
Ids 21-255 are `<|reserved_N|>`.
## Tokenization conventions
- Byte-level: any input roundtrips exactly; no unknown tokens in practice.
- Digits are always single tokens (`" 7"` or `"7"`, never `"12"`).
- Symbol runs take at most one leading space and never a trailing one; pieces never end
in a space.
- Multi-word pieces are allowed (no word-splitting rule); a piece containing a space after
its first symbol always begins with a space.
- Pieces never cross Unicode character boundaries.
## Training data
| region | size | EM influence | contents |
|---|---|---|---|
| English | 980 MB | 97.2% | cleaned Nemotron-CC web text |
| Code | 346 MB | 1.9% | 14 languages, uniform |
| Latin-script | 75 MB | 0.2% | de fr es it pt nl pl tr vi id cs ro hu da (FineWeb-2-HQ) |
| Chinese + Russian | 47 MB | 0.25% | FineWeb-2-HQ |
| Japanese + Korean | 45 MB | 0.2% | FineWeb-2-HQ / FineWeb-2 |
| Greek, Hindi, Arabic, Hebrew | 43 MB | 0.1% | FineWeb-2-HQ / FineWeb-2 |
| 23-language pool | 39 MB | 0.15% | th bn ta km ka hy my kn si te gu ml pa lo bo am uk sr or dv mr ne mn (FineWeb-2 snippets) |
Influence is the region's share of the EM expected counts (per-byte weighting), independent
of raw size. Cleaning: length band, cross-document line dedup, HTML-density and navigation
filters, repeated-fragment stripping, per-language p99 length cut, database-dump removal.
## Evaluation (bytes per token, higher is better)
| text | this tokenizer | Mistral v0.3 (32k) |
|---|---|---|
| FineWeb English held-out | 4.82 (15.4% fewer tokens) | 4.08 |
| Code held-out | 2.90 (3.2% fewer tokens) | 2.81 |
| Chinese | 2.83 | 3.00 |
| Japanese | 2.71 | 2.76 |
| Korean | 2.70 | 2.52 |
| Russian | 3.22 | 5.38 |
| German / French | 2.21 / 2.53 | 3.09 / 3.69 |
| Greek / Hebrew / Arabic / Hindi | 2.31 / 2.44 / 2.33 / 2.92 | 1.83 / 1.79 / 1.98 / 2.54 |
## Notes
- `model.vocab` scores are log-probs. The 256 control tokens and 21 never-observed byte
singles (bytes 0xF5-0xFF, impossible in valid UTF-8) carry a -1e30 score; harmless for
encoding, floor them if you read scores as a prior.
- `convert_ids_to_tokens` shows ByteLevel forms (`Ġ` = space, `Ċ` = newline); use `decode`
for text.
|