Update README.md
Browse files
README.md
CHANGED
|
@@ -11,8 +11,7 @@ pinned: false
|
|
| 11 |
<img src="./toksuite-pipeline.png" alt="TokSuite Logo"/>
|
| 12 |
</p>
|
| 13 |
|
| 14 |
-
**TokSuite** is a
|
| 15 |
-
training data but differ **only in their tokenizers** (**right**).
|
| 16 |
|
| 17 |
The left panel illustrates how different tokenizers fragment the word *“doctor”* when subjected to OCR errors, orthographic mistakes, semantic equivalents, emoji substitution, and multilingual
|
| 18 |
translations. Each colored box represents a single token across the **Gemma-2** (yellow), **GPT-2** (blue), **TokenMonster** (green), and **XGLM** (red) tokenizers.
|
|
|
|
| 11 |
<img src="./toksuite-pipeline.png" alt="TokSuite Logo"/>
|
| 12 |
</p>
|
| 13 |
|
| 14 |
+
**TokSuite** is a collection of models and a benchmark designed to isolate and study the impact of tokenization on language model behavior. It includes **fourteen models** that share the same architecture, data, training budget, and initialization but differ only in their tokenizers, along with a benchmark that evaluates performance under real-world text perturbations that affect tokenization.
|
|
|
|
| 15 |
|
| 16 |
The left panel illustrates how different tokenizers fragment the word *“doctor”* when subjected to OCR errors, orthographic mistakes, semantic equivalents, emoji substitution, and multilingual
|
| 17 |
translations. Each colored box represents a single token across the **Gemma-2** (yellow), **GPT-2** (blue), **TokenMonster** (green), and **XGLM** (red) tokenizers.
|