File size: 1,088 Bytes
dd6a333 ab58430 dd6a333 ab58430 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 | ---
license: apache-2.0
tags:
- tokenizer
- bpe
- multilingual
- parity-aware
---
# Parity-Aware BPE Tokenizer (dev-set balanced)
## Languages
Trained on 21 languages, using FineWeb (train) and FLORES-200 (dev)
per language:
| Language | Training source | FLORES-200 code |
|---|---|---|
| English | `fw_edu` | `eng_Latn` |
| German | `deu_Latn` | `deu_Latn` |
| French | `fra_Latn` | `fra_Latn` |
| Arabic | `arb_Arab` | `arb_Arab` |
| Farsi | `fas_Arab` | `fas_Arab` |
| Czech | `ces_Latn` | `ces_Latn` |
| Chinese | `cmn_Hani` | `cmn_Hani` |
| Danish | `dan_Latn` | `dan_Latn` |
| Dutch | `nld_Latn` | `nld_Latn` |
| Greek | `ell_Grek` | `ell_Grek` |
| Hungarian | `hun_Latn` | `hun_Latn` |
| Indonesian | `ind_Latn` | `ind_Latn` |
| Italian | `ita_Latn` | `ita_Latn` |
| Japanese | `jpn_Jpan` | `jpn_Jpan` |
| Polish | `pol_Latn` | `pol_Latn` |
| Portuguese | `por_Latn` | `por_Latn` |
| Russian | `rus_Cyrl` | `rus_Cyrl` |
| Spanish | `spa_Latn` | `spa_Latn` |
| Swedish | `swe_Latn` | `swe_Latn` |
| Turkish | `tur_Latn` | `tur_Latn` |
| Vietnamese | `vie_Latn` | `vie_Latn` |
|