| --- |
| license: apache-2.0 |
| tags: |
| - tokenizer |
| - bpe |
| - multilingual |
| - parity-aware |
| --- |
| |
| # Parity-Aware BPE Tokenizer (dev-set balanced) |
|
|
|
|
| ## Languages |
|
|
| Trained on 21 languages, using FineWeb (train) and FLORES-200 (dev) |
| per language: |
|
|
| | Language | Training source | FLORES-200 code | |
| |---|---|---| |
| | English | `fw_edu` | `eng_Latn` | |
| | German | `deu_Latn` | `deu_Latn` | |
| | French | `fra_Latn` | `fra_Latn` | |
| | Arabic | `arb_Arab` | `arb_Arab` | |
| | Farsi | `fas_Arab` | `fas_Arab` | |
| | Czech | `ces_Latn` | `ces_Latn` | |
| | Chinese | `cmn_Hani` | `cmn_Hani` | |
| | Danish | `dan_Latn` | `dan_Latn` | |
| | Dutch | `nld_Latn` | `nld_Latn` | |
| | Greek | `ell_Grek` | `ell_Grek` | |
| | Hungarian | `hun_Latn` | `hun_Latn` | |
| | Indonesian | `ind_Latn` | `ind_Latn` | |
| | Italian | `ita_Latn` | `ita_Latn` | |
| | Japanese | `jpn_Jpan` | `jpn_Jpan` | |
| | Polish | `pol_Latn` | `pol_Latn` | |
| | Portuguese | `por_Latn` | `por_Latn` | |
| | Russian | `rus_Cyrl` | `rus_Cyrl` | |
| | Spanish | `spa_Latn` | `spa_Latn` | |
| | Swedish | `swe_Latn` | `swe_Latn` | |
| | Turkish | `tur_Latn` | `tur_Latn` | |
| | Vietnamese | `vie_Latn` | `vie_Latn` | |
|
|
