| --- |
| library_name: transformers |
| tags: |
| - tokenizer |
| - unigram |
| - byte-level |
| - multilingual |
| - code |
| language: |
| - en |
| - de |
| - fr |
| - es |
| - it |
| - pt |
| - nl |
| - pl |
| - tr |
| - vi |
| - id |
| - cs |
| - ro |
| - hu |
| - da |
| - zh |
| - ru |
| - ja |
| - ko |
| - el |
| - hi |
| - ar |
| - he |
| - th |
| - bn |
| - ta |
| - km |
| - ka |
| - hy |
| - my |
| - kn |
| - si |
| - te |
| - gu |
| - ml |
| - pa |
| - lo |
| - bo |
| - am |
| - uk |
| - sr |
| - or |
| - dv |
| - mr |
| - ne |
| - mn |
| --- |
| |
| # U32k: 32k byte-level Unigram tokenizer |
|
|
| A 32,768-id byte-level Unigram tokenizer (HuggingFace `tokenizers` format, ByteLevel |
| pre-tokenizer and decoder) with 256 control tokens and 32,512 learned pieces. English-first, |
| with code and 45 additional languages. Trained with an exact GPU Unigram EM trainer |
| (1.57M seed candidates, shrink 0.999 per prune round, 2,223 rounds, 2 final EM passes). |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoTokenizer |
| tok = AutoTokenizer.from_pretrained("xTimeCrystal/U32k") |
| |
| ids = tok("The quick brown fox")["input_ids"] |
| text = tok.decode(ids) # exact roundtrip (byte-level) |
| batch = tok(["a", "longer text"], padding=True, return_tensors="pt") # pad id 0 |
| ``` |
|
|
| Without transformers: |
|
|
| ```python |
| from tokenizers import Tokenizer |
| raw = Tokenizer.from_file("tokenizer.json") |
| raw.encode("text").ids |
| ``` |
|
|
| BOS/EOS are not inserted automatically (no post-processor). Add them explicitly: |
|
|
| ```python |
| ids = [tok.bos_token_id] + tok(text)["input_ids"] + [tok.eos_token_id] |
| ``` |
|
|
| ## Special tokens (ids 0-255) |
|
|
| | id | token | id | token | |
| |---|---|---|---| |
| | 0 | `<\|pad\|>` | 9 | `<\|tool_call\|>` | |
| | 1 | `<\|begin_of_text\|>` (bos) | 10 | `<\|tool_result\|>` | |
| | 2 | `<\|end_of_text\|>` (eos) | 11 | `<\|think\|>` | |
| | 3 | `<\|user\|>` | 12 | `<\|end_think\|>` | |
| | 4 | `<\|assistant\|>` | 13 | `<\|mask\|>` | |
| | 5 | `<\|system\|>` | 14 | `<\|sep\|>` | |
| | 6 | `<\|start_header\|>` | 15 | `<\|unk\|>` | |
| | 7 | `<\|end_header\|>` | 16-18 | `<\|fim_prefix\|>` `<\|fim_middle\|>` `<\|fim_suffix\|>` | |
| | 8 | `<\|end_turn\|>` | 19-20 | `<\|repo_name\|>` `<\|file_sep\|>` | |
|
|
| Ids 21-255 are `<|reserved_N|>`. |
|
|
| ## Tokenization conventions |
|
|
| - Byte-level: any input roundtrips exactly; no unknown tokens in practice. |
| - Digits are always single tokens (`" 7"` or `"7"`, never `"12"`). |
| - Symbol runs take at most one leading space and never a trailing one; pieces never end |
| in a space. |
| - Multi-word pieces are allowed (no word-splitting rule); a piece containing a space after |
| its first symbol always begins with a space. |
| - Pieces never cross Unicode character boundaries. |
|
|
| ## Training data |
|
|
| | region | size | EM influence | contents | |
| |---|---|---|---| |
| | English | 980 MB | 97.2% | cleaned Nemotron-CC web text | |
| | Code | 346 MB | 1.9% | 14 languages, uniform | |
| | Latin-script | 75 MB | 0.2% | de fr es it pt nl pl tr vi id cs ro hu da (FineWeb-2-HQ) | |
| | Chinese + Russian | 47 MB | 0.25% | FineWeb-2-HQ | |
| | Japanese + Korean | 45 MB | 0.2% | FineWeb-2-HQ / FineWeb-2 | |
| | Greek, Hindi, Arabic, Hebrew | 43 MB | 0.1% | FineWeb-2-HQ / FineWeb-2 | |
| | 23-language pool | 39 MB | 0.15% | th bn ta km ka hy my kn si te gu ml pa lo bo am uk sr or dv mr ne mn (FineWeb-2 snippets) | |
|
|
| Influence is the region's share of the EM expected counts (per-byte weighting), independent |
| of raw size. Cleaning: length band, cross-document line dedup, HTML-density and navigation |
| filters, repeated-fragment stripping, per-language p99 length cut, database-dump removal. |
|
|
| ## Evaluation (bytes per token, higher is better) |
|
|
| | text | this tokenizer | Mistral v0.3 (32k) | |
| |---|---|---| |
| | FineWeb English held-out | 4.82 (15.4% fewer tokens) | 4.08 | |
| | Code held-out | 2.90 (3.2% fewer tokens) | 2.81 | |
| | Chinese | 2.83 | 3.00 | |
| | Japanese | 2.71 | 2.76 | |
| | Korean | 2.70 | 2.52 | |
| | Russian | 3.22 | 5.38 | |
| | German / French | 2.21 / 2.53 | 3.09 / 3.69 | |
| | Greek / Hebrew / Arabic / Hindi | 2.31 / 2.44 / 2.33 / 2.92 | 1.83 / 1.79 / 1.98 / 2.54 | |
|
|
| ## Notes |
|
|
| - `model.vocab` scores are log-probs. The 256 control tokens and 21 never-observed byte |
| singles (bytes 0xF5-0xFF, impossible in valid UTF-8) carry a -1e30 score; harmless for |
| encoding, floor them if you read scores as a prior. |
| - `convert_ids_to_tokens` shows ByteLevel forms (`Ġ` = space, `Ċ` = newline); use `decode` |
| for text. |
|
|
