---
library_name: transformers
tags:
  - tokenizer
  - unigram
  - byte-level
  - multilingual
  - code
language:
  - en
  - de
  - fr
  - es
  - it
  - pt
  - nl
  - pl
  - tr
  - vi
  - id
  - cs
  - ro
  - hu
  - da
  - zh
  - ru
  - ja
  - ko
  - el
  - hi
  - ar
  - he
  - th
  - bn
  - ta
  - km
  - ka
  - hy
  - my
  - kn
  - si
  - te
  - gu
  - ml
  - pa
  - lo
  - bo
  - am
  - uk
  - sr
  - or
  - dv
  - mr
  - ne
  - mn
---

# U32k: 32k byte-level Unigram tokenizer

A 32,768-id byte-level Unigram tokenizer (HuggingFace `tokenizers` format, ByteLevel
pre-tokenizer and decoder) with 256 control tokens and 32,512 learned pieces. English-first,
with code and 45 additional languages. Trained with an exact GPU Unigram EM trainer
(1.57M seed candidates, shrink 0.999 per prune round, 2,223 rounds, 2 final EM passes).

## Usage

```python
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("xTimeCrystal/U32k")

ids = tok("The quick brown fox")["input_ids"]
text = tok.decode(ids)                                  # exact roundtrip (byte-level)
batch = tok(["a", "longer text"], padding=True, return_tensors="pt")   # pad id 0
```

Without transformers:

```python
from tokenizers import Tokenizer
raw = Tokenizer.from_file("tokenizer.json")
raw.encode("text").ids
```

BOS/EOS are not inserted automatically (no post-processor). Add them explicitly:

```python
ids = [tok.bos_token_id] + tok(text)["input_ids"] + [tok.eos_token_id]
```

## Special tokens (ids 0-255)

| id | token | id | token |
|---|---|---|---|
| 0 | `<\|pad\|>` | 9 | `<\|tool_call\|>` |
| 1 | `<\|begin_of_text\|>` (bos) | 10 | `<\|tool_result\|>` |
| 2 | `<\|end_of_text\|>` (eos) | 11 | `<\|think\|>` |
| 3 | `<\|user\|>` | 12 | `<\|end_think\|>` |
| 4 | `<\|assistant\|>` | 13 | `<\|mask\|>` |
| 5 | `<\|system\|>` | 14 | `<\|sep\|>` |
| 6 | `<\|start_header\|>` | 15 | `<\|unk\|>` |
| 7 | `<\|end_header\|>` | 16-18 | `<\|fim_prefix\|>` `<\|fim_middle\|>` `<\|fim_suffix\|>` |
| 8 | `<\|end_turn\|>` | 19-20 | `<\|repo_name\|>` `<\|file_sep\|>` |

Ids 21-255 are `<|reserved_N|>`.

## Tokenization conventions

- Byte-level: any input roundtrips exactly; no unknown tokens in practice.
- Digits are always single tokens (`" 7"` or `"7"`, never `"12"`).
- Symbol runs take at most one leading space and never a trailing one; pieces never end
  in a space.
- Multi-word pieces are allowed (no word-splitting rule); a piece containing a space after
  its first symbol always begins with a space.
- Pieces never cross Unicode character boundaries.

## Training data

| region | size | EM influence | contents |
|---|---|---|---|
| English | 980 MB | 97.2% | cleaned Nemotron-CC web text |
| Code | 346 MB | 1.9% | 14 languages, uniform |
| Latin-script | 75 MB | 0.2% | de fr es it pt nl pl tr vi id cs ro hu da (FineWeb-2-HQ) |
| Chinese + Russian | 47 MB | 0.25% | FineWeb-2-HQ |
| Japanese + Korean | 45 MB | 0.2% | FineWeb-2-HQ / FineWeb-2 |
| Greek, Hindi, Arabic, Hebrew | 43 MB | 0.1% | FineWeb-2-HQ / FineWeb-2 |
| 23-language pool | 39 MB | 0.15% | th bn ta km ka hy my kn si te gu ml pa lo bo am uk sr or dv mr ne mn (FineWeb-2 snippets) |

Influence is the region's share of the EM expected counts (per-byte weighting), independent
of raw size. Cleaning: length band, cross-document line dedup, HTML-density and navigation
filters, repeated-fragment stripping, per-language p99 length cut, database-dump removal.

## Evaluation (bytes per token, higher is better)

| text | this tokenizer | Mistral v0.3 (32k) |
|---|---|---|
| FineWeb English held-out | 4.82 (15.4% fewer tokens) | 4.08 |
| Code held-out | 2.90 (3.2% fewer tokens) | 2.81 |
| Chinese | 2.83 | 3.00 |
| Japanese | 2.71 | 2.76 |
| Korean | 2.70 | 2.52 |
| Russian | 3.22 | 5.38 |
| German / French | 2.21 / 2.53 | 3.09 / 3.69 |
| Greek / Hebrew / Arabic / Hindi | 2.31 / 2.44 / 2.33 / 2.92 | 1.83 / 1.79 / 1.98 / 2.54 |

## Notes

- `model.vocab` scores are log-probs. The 256 control tokens and 21 never-observed byte
  singles (bytes 0xF5-0xFF, impossible in valid UTF-8) carry a -1e30 score; harmless for
  encoding, floor them if you read scores as a prior.
- `convert_ids_to_tokens` shows ByteLevel forms (`Ġ` = space, `Ċ` = newline); use `decode`
  for text.