How machines learned to read — from rigid word dictionaries to subwords
Tokenization is the bridge between raw text and numerical representation. Since machines process numbers, not strings, text must be converted into a sequence of integer IDs drawn from a vocabulary before any model can process it. Each unit in that sequence is a token.
Translation from text strings to integer IDs. The model never sees letters — only numbers that index into a learned embedding matrix.
Modern systems use Subword Tokenization. Algorithms like BPE start with individual characters and iteratively merge the most frequent adjacent pairs into common subwords.
Reduces input complexity and vocabulary size. Without it, a model would need to learn every possible character sequence from scratch — computationally intractable.
LLMs have a fixed context window (e.g., 128k tokens). Efficient tokenization directly determines how much text fits within that window — more efficient tokenization means more context, better reasoning.
By sharing subword representations, models also generalize across related words: run, running, runner all share the subword run. This improves accuracy and reduces the parameters needed to learn language.
Research moved from rigid dictionary lookup to flexible learned subwords, driven entirely by one recurring failure: the Out-of-Vocabulary (OOV) problem — any word absent from the lexicon silently collapsed into a meaningless <UNK> token.
Text was split at word boundaries (spaces and punctuation). A lexicon mapped each known word to an integer ID. English has 170K+ words; Turkish and Finnish have millions of inflected forms — complete coverage was impossible.
<UNK>. "Covid" in 2019? <UNK>. Any neologism, brand name, or rare proper noun silently collapsed to a single meaningless token. The model was blind to vocabulary evolution.
Splitting text into individual characters eliminated OOV entirely — only 26–256 token types are ever needed. But it introduced a worse trade-off.
BPE (Sennrich et al., 2016) introduced the decisive insight: learn token boundaries from data itself. Frequent words become single tokens; rare words decompose into meaningful morphological parts. No fixed dictionary is required — the vocabulary is learned.
Google's CANINE (2021) was the first pre-trained model with zero tokenization. It processes raw Unicode characters directly — no vocabulary, no fixed token set, 104 languages without any language-specific rules. The tokenizer component is eliminated entirely. See §05 for the full picture.
Byte Pair Encoding was originally a data compression algorithm from the 1990s. Sennrich et al. (2016) adapted it for neural machine translation with a simple insight: frequently co-occurring character pairs should become their own tokens.
All three algorithms share BPE's philosophy but differ in merge strategy and output format:
## marks continuation subwords▁ · no pre-tokenization step neededToday's tokenizers all share the subword philosophy but differ in merge strategy, output format, and architectural assumptions. Each co-evolved with a specific family of models.
| Method | Merge Strategy | Distinctive Feature | Used In |
|---|---|---|---|
| BPE | Raw frequency | Simple; plain concatenation; language-agnostic | GPT-2GPT-4LLaMA |
| WordPiece | Maximizes corpus likelihood | ## prefix marks continuation tokens |
BERTDistilBERT |
| SentencePiece | Treats whitespace as a character | No pre-tokenization; multilingual by default | T5ALBERTXLNet |
| Unigram | Starts large, prunes by loss | Probabilistic; retains tokens minimizing perplexity | Often paired with SentencePiece |
| Tiktoken | Byte-level BPE on raw bytes | Zero encoding failures — every byte value is valid | GPT-4Claude |
| Problem | Word-Based Era | Subword Era |
|---|---|---|
| Unknown words | <UNK> — meaning destroyed | Decomposed into subwords; no UNK ever |
| Lexicons | Required; always incomplete | Not needed; vocabulary learned from data |
| New languages | Custom rules per language | One tokenizer handles 100+ languages |
| Vocabulary size | 100K–1M+ word forms needed | 32K–100K subwords covers all of it |
| Morphology | run / running treated as unrelated | Shared subwords encode morphological links |
Tokenization isn't being improved — it's being eliminated. In parallel, the preprocessing pipelines that preceded tokenization have also been abandoned. Both shifts are part of the same underlying pattern: deep learning learns better representations than human-engineered ones.
Google's CANINE (2021) was the first pre-trained encoder with zero tokenization. It processes raw Unicode characters using hash-based embeddings and strided convolutions for downsampling — no vocabulary required at any stage.
Character embeddings via hashing. Emojis, typos, code-switching, any Unicode — all handled identically without special cases.
Outperforms mBERT by +5.7 F1 on TyDi QA with 28% fewer parameters and no fixed vocabulary.
No language-specific rules, dictionaries, or model variants — one architecture handles them all natively.
| Direction | Description | Status |
|---|---|---|
| Byte-Level Models | Operate on raw bytes (256 values). Eliminates all encoding edge-cases entirely. | Active research |
| Neural Tokenization | Token boundaries learned via RL or gradient descent — replaces frequency heuristics. | 2026 papers, promising |
| Hybrid Architectures | Byte-level encoder + token-level transformer connected via cross-attention. | Being explored |
| Adaptive Downsampling | Dynamic sequence compression that varies with local text complexity. | Research ongoing |
| Edge Optimization | Compressed learned codebooks for mobile and low-resource deployment. | Critical open gap |
Tokenization was only the last step in a longer pipeline that deep learning has now made obsolete. Every cleaning step once considered mandatory has been shown to hurt model performance — because each one encoded a human assumption the model is better off learning itself.
| Old Step | Why It Hurt | What Replaced It |
|---|---|---|
| Lowercasing | Erased "Apple" (company) vs "apple" (fruit) | Model learns case sensitivity as a feature |
| Stopword removal | Deleted "not" in "not good" — flipped sentiment | Attention learns importance weighting dynamically |
| Stemming | run / running / ran collapsed → tense information lost | Subwords preserve morphology and conjugation |
| Punctuation stripping | "!" "?" "." carry sentiment, emphasis, and syntax | Model treats punctuation as semantic signals |
| Lexicon lookup | OOV = silent failure | Subwords / bytes = unbounded effective vocabulary |
This is not unique to NLP. Every field in deep learning has undergone the same transition — away from hand-engineered intermediate representations and toward learning from raw signals.
Type any word or phrase and switch between strategies to see how the same input is tokenized differently across eras.