metadata
license: apache-2.0
tags:
- tokenizer
- bpe
- multilingual
- parity-aware
Parity-Aware BPE Tokenizer (dev-set balanced)
Languages
Trained on 21 languages, using FineWeb (train) and FLORES-200 (dev) per language:
| Language | Training source | FLORES-200 code |
|---|---|---|
| English | fw_edu |
eng_Latn |
| German | deu_Latn |
deu_Latn |
| French | fra_Latn |
fra_Latn |
| Arabic | arb_Arab |
arb_Arab |
| Farsi | fas_Arab |
fas_Arab |
| Czech | ces_Latn |
ces_Latn |
| Chinese | cmn_Hani |
cmn_Hani |
| Danish | dan_Latn |
dan_Latn |
| Dutch | nld_Latn |
nld_Latn |
| Greek | ell_Grek |
ell_Grek |
| Hungarian | hun_Latn |
hun_Latn |
| Indonesian | ind_Latn |
ind_Latn |
| Italian | ita_Latn |
ita_Latn |
| Japanese | jpn_Jpan |
jpn_Jpan |
| Polish | pol_Latn |
pol_Latn |
| Portuguese | por_Latn |
por_Latn |
| Russian | rus_Cyrl |
rus_Cyrl |
| Spanish | spa_Latn |
spa_Latn |
| Swedish | swe_Latn |
swe_Latn |
| Turkish | tur_Latn |
tur_Latn |
| Vietnamese | vie_Latn |
vie_Latn |