---
license: apache-2.0
tags:
- tokenizer
- bpe
- multilingual
- parity-aware
---

# Parity-Aware BPE Tokenizer (dev-set balanced)


## Languages

Trained on 21 languages, using FineWeb (train) and FLORES-200 (dev)
per language:

| Language | Training source | FLORES-200 code |
|---|---|---|
| English | `fw_edu` | `eng_Latn` |
| German | `deu_Latn` | `deu_Latn` |
| French | `fra_Latn` | `fra_Latn` |
| Arabic | `arb_Arab` | `arb_Arab` |
| Farsi | `fas_Arab` | `fas_Arab` |
| Czech | `ces_Latn` | `ces_Latn` |
| Chinese | `cmn_Hani` | `cmn_Hani` |
| Danish | `dan_Latn` | `dan_Latn` |
| Dutch | `nld_Latn` | `nld_Latn` |
| Greek | `ell_Grek` | `ell_Grek` |
| Hungarian | `hun_Latn` | `hun_Latn` |
| Indonesian | `ind_Latn` | `ind_Latn` |
| Italian | `ita_Latn` | `ita_Latn` |
| Japanese | `jpn_Jpan` | `jpn_Jpan` |
| Polish | `pol_Latn` | `pol_Latn` |
| Portuguese | `por_Latn` | `por_Latn` |
| Russian | `rus_Cyrl` | `rus_Cyrl` |
| Spanish | `spa_Latn` | `spa_Latn` |
| Swedish | `swe_Latn` | `swe_Latn` |
| Turkish | `tur_Latn` | `tur_Latn` |
| Vietnamese | `vie_Latn` | `vie_Latn` |