Malikeh1375 commited on
Commit
f9fdb06
·
verified ·
1 Parent(s): 26170ad

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -2
README.md CHANGED
@@ -11,8 +11,7 @@ pinned: false
11
  <img src="./toksuite-pipeline.png" alt="TokSuite Logo"/>
12
  </p>
13
 
14
- **TokSuite** is a comprehensive benchmark covering real-world perturbations that change tokenization (**left**), and **14 models** that share the same initialization, architecture, and
15
- training data but differ **only in their tokenizers** (**right**).
16
 
17
  The left panel illustrates how different tokenizers fragment the word *“doctor”* when subjected to OCR errors, orthographic mistakes, semantic equivalents, emoji substitution, and multilingual
18
  translations. Each colored box represents a single token across the **Gemma-2** (yellow), **GPT-2** (blue), **TokenMonster** (green), and **XGLM** (red) tokenizers.
 
11
  <img src="./toksuite-pipeline.png" alt="TokSuite Logo"/>
12
  </p>
13
 
14
+ **TokSuite** is a collection of models and a benchmark designed to isolate and study the impact of tokenization on language model behavior. It includes **fourteen models** that share the same architecture, data, training budget, and initialization but differ only in their tokenizers, along with a benchmark that evaluates performance under real-world text perturbations that affect tokenization.
 
15
 
16
  The left panel illustrates how different tokenizers fragment the word *“doctor”* when subjected to OCR errors, orthographic mistakes, semantic equivalents, emoji substitution, and multilingual
17
  translations. Each colored box represents a single token across the **Gemma-2** (yellow), **GPT-2** (blue), **TokenMonster** (green), and **XGLM** (red) tokenizers.