Commit ·
3f5403f
1
Parent(s): 5438c18
MillID leaderboard against GlotLID and fastText (#2)
Browse files- MillID leaderboard against GlotLID and fastText (5a279ebf9b8e5260865c7f0d99eaff5dad901b68)
README.md
CHANGED
|
@@ -24,320 +24,326 @@ Every score is on the test split of the build the models were trained on. The pu
|
|
| 24 |
|
| 25 |
## Macro F1, every set
|
| 26 |
|
| 27 |
-
|
| 28 |
-
|---|---|---|---|---|---|---|
|
| 29 |
-
| `test`: every language, passages from documents never seen in training | 1621 | 0.9554 | 0.9576 | 0.9646 | **0.9755** | 0.0659 |
|
| 30 |
-
| `test-common`: the languages all three systems know | 213 | 0.9513 | 0.9527 | 0.9594 | **0.9846** | 0.9173 |
|
| 31 |
-
| `udhr`: the Declaration of Human Rights, languages MillID knows | 358 | 0.8764 | 0.8780 | 0.8812 | **0.8873** | 0.3499 |
|
| 32 |
-
| `udhr-common`: the Declaration, languages all three know | 170 | 0.9513 | 0.9480 | 0.9529 | **0.9659** | 0.9329 |
|
| 33 |
-
| `udhr-all`: the Declaration, every language in it | 429 | 0.7149 | 0.7184 | 0.7201 | **0.7245** | 0.2797 |
|
| 34 |
|
| 35 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 36 |
|
| 37 |
## test
|
| 38 |
|
| 39 |
-
Every language, passages from documents never seen in training.
|
| 40 |
|
| 41 |
-
|
|
| 42 |
-
|---|---|---|---|---|---|---|---|
|
| 43 |
-
| [GlotLID
|
| 44 |
-
| [GlotLID](https://huggingface.co/cis-lmu/glotlid) |
|
| 45 |
-
| [
|
| 46 |
-
| [
|
| 47 |
-
| [
|
| 48 |
-
| [fastText
|
| 49 |
-
| [fastText
|
| 50 |
|
| 51 |
## test-common
|
| 52 |
|
| 53 |
-
The languages all three systems know.
|
| 54 |
|
| 55 |
-
|
|
| 56 |
-
|---|---|---|---|---|---|---|---|
|
| 57 |
-
| [GlotLID
|
| 58 |
-
| [GlotLID](https://huggingface.co/cis-lmu/glotlid) |
|
| 59 |
-
| [
|
| 60 |
-
| [
|
| 61 |
-
| [
|
| 62 |
-
| [fastText
|
| 63 |
-
| [fastText
|
| 64 |
|
| 65 |
## udhr
|
| 66 |
|
| 67 |
-
The Declaration of Human Rights, languages MillID knows.
|
| 68 |
|
| 69 |
-
|
|
| 70 |
-
|---|---|---|---|---|---|---|---|
|
| 71 |
-
| [GlotLID
|
| 72 |
-
| [GlotLID](https://huggingface.co/cis-lmu/glotlid) |
|
| 73 |
-
| [
|
| 74 |
-
| [
|
| 75 |
-
| [
|
| 76 |
-
| [fastText
|
| 77 |
-
| [fastText
|
| 78 |
|
| 79 |
## udhr-common
|
| 80 |
|
| 81 |
-
The Declaration, languages all three know.
|
| 82 |
|
| 83 |
-
|
|
| 84 |
-
|---|---|---|---|---|---|---|---|
|
| 85 |
-
| [GlotLID
|
| 86 |
-
| [GlotLID](https://huggingface.co/cis-lmu/glotlid) |
|
| 87 |
-
| [
|
| 88 |
-
| [
|
| 89 |
-
| [
|
| 90 |
-
| [fastText
|
| 91 |
-
| [fastText
|
| 92 |
|
| 93 |
## udhr-all
|
| 94 |
|
| 95 |
-
The Declaration, every language in it.
|
| 96 |
|
| 97 |
-
|
|
| 98 |
-
|---|---|---|---|---|---|---|---|
|
| 99 |
-
| [GlotLID](https://huggingface.co/cis-lmu/glotlid) |
|
| 100 |
-
| [GlotLID
|
| 101 |
-
| [
|
| 102 |
-
| [
|
| 103 |
-
| [
|
| 104 |
-
| [fastText
|
| 105 |
-
| [fastText
|
| 106 |
|
| 107 |
## Per language, test
|
| 108 |
|
| 109 |
-
Against GlotLID restricted, the best MillID is better on **165** languages and worse on **498**; against fastText restricted, the best MillID is better on **1616** languages and worse on **0** (F1 differing by more than 0.01), of 1621. Every language is in `
|
| 110 |
|
| 111 |
The 30 languages with the most passages:
|
| 112 |
|
| 113 |
-
| Label | Language | Passages |
|
| 114 |
-
|---|---|---|---|---|---|---|---|
|
| 115 |
-
| `fra_Latn` | French | 1652 | 0.
|
| 116 |
-
| `eng_Latn` | English | 1444 | 0.
|
| 117 |
-
| `spa_Latn` | Spanish | 1136 | 0.
|
| 118 |
-
| `tam_Taml` | Tamil | 754 | 0.
|
| 119 |
-
| `arb_Arab` | Arabic (Modern Standard) | 745 | 0.
|
| 120 |
-
| `por_Latn` | Portuguese | 657 | 0.947 | 0.
|
| 121 |
-
| `kan_Knda` | Kannada | 601 | 0.
|
| 122 |
-
| `zul_Latn` | Zulu | 594 | 0.
|
| 123 |
-
| `swh_Latn` | Swahili | 592 | 0.
|
| 124 |
-
| `mar_Deva` | Marathi | 589 | 0.
|
| 125 |
-
| `tso_Latn` | Tsonga | 581 | 0.
|
| 126 |
-
| `hun_Latn` | Hungarian | 576 | 0.
|
| 127 |
-
| `afr_Latn` | Afrikaans | 569 | 0.
|
| 128 |
-
| `tel_Telu` | Telugu | 555 | 0.
|
| 129 |
-
| `rus_Cyrl` | Russian | 528 | 0.
|
| 130 |
-
| `nbl_Latn` | South Ndebele | 528 | 0.
|
| 131 |
-
| `ory_Orya` | Odia (individual language) | 511 | 0.
|
| 132 |
-
| `sot_Latn` | Southern Sotho | 491 | 0.
|
| 133 |
-
| `nso_Latn` | Northern Sotho | 479 | 0.
|
| 134 |
-
| `kua_Latn` | Kuanyama | 476 | 0.
|
| 135 |
-
| `asm_Beng` | Assamese | 467 | 0.
|
| 136 |
-
| `ben_Beng` | Bangla | 466 | 0.
|
| 137 |
-
| `pan_Guru` | Punjabi | 460 | 1.000 | 1.000 |
|
| 138 |
-
| `amh_Ethi` | Amharic | 458 |
|
| 139 |
-
| `tsn_Latn` | Tswana | 456 | 0.
|
| 140 |
-
| `vro_Latn` | Võro | 448 | 0.
|
| 141 |
-
| `gaz_Latn` | Oromo (West Central) | 441 | 0.
|
| 142 |
-
| `som_Latn` | Somali | 439 | 0.
|
| 143 |
-
| `uzn_Latn` | Northern Uzbek | 436 | 0.
|
| 144 |
-
| `tha_Thai` | Thai | 431 | 0.
|
| 145 |
|
| 146 |
Where MillID gains most over GlotLID:
|
| 147 |
|
| 148 |
-
| Label | Language | Passages |
|
| 149 |
-
|---|---|---|---|---|---|---|---|
|
| 150 |
-
| `lbj_Tibt` | Ladakhi | 66 | 0.
|
| 151 |
-
| `ain_Latn` | Ainu | 1 | 0.
|
| 152 |
-
| `ajp_Arab` | South Levantine Arabic | 28 | 0.
|
| 153 |
-
| `und_Laoo` | und | 20 | 0.
|
| 154 |
-
| `sgh_Cyrl` | Shughni | 9 | 0.
|
| 155 |
-
| `und_Thaa` | und | 29 | 0.
|
| 156 |
-
| `icr_Latn` | Islander Creole English | 24 | 0.
|
| 157 |
-
| `und_Geor` | und | 32 | 0.
|
| 158 |
-
| `tdt_Latn` | Tetun Dili | 57 | 0.
|
| 159 |
-
| `cek_Latn` | Eastern Khumi Chin | 67 | 0.
|
| 160 |
-
| `und_Syrc` | und | 43 | 0.
|
| 161 |
-
| `neb_Latn` | Toura (Côte d'Ivoire) | 11 | 0.
|
| 162 |
-
| `dag_Latn` | Dagbani | 149 | 0.
|
| 163 |
-
| `tuk_Arab` | Turkmen | 135 | 0.
|
| 164 |
-
| `orv_Cyrl` | Old Russian | 239 | 0.
|
| 165 |
|
| 166 |
Where MillID loses most to GlotLID:
|
| 167 |
|
| 168 |
-
| Label | Language | Passages |
|
| 169 |
-
|---|---|---|---|---|---|---|---|
|
| 170 |
-
| `mus_Latn` | Muscogee | 2 |
|
| 171 |
-
| `ttj_Latn` | Tooro | 28 | 0.
|
| 172 |
-
| `knc_Arab` | Central Kanuri | 68 | 0.
|
| 173 |
-
| `hrv_Latn` | Croatian | 168 | 0.
|
| 174 |
-
| `esi_Latn` | North Alaskan Inupiatun | 9 | 0.
|
| 175 |
-
| `hak_Hani` | Hakka Chinese | 29 | 0.
|
| 176 |
-
| `bos_Latn` | Bosnian | 198 | 0.
|
| 177 |
-
| `plw_Latn` | Brooke's Point Palawano | 8 |
|
| 178 |
-
| `ccp_Latn` | Chakma | 3 |
|
| 179 |
-
| `mbd_Latn` | Dibabawon Manobo | 14 |
|
| 180 |
-
| `taq_Tfng` | Tamasheq | 104 | 0.
|
| 181 |
-
| `zsm_Latn` | Standard Malay | 244 | 0.
|
| 182 |
-
| `biu_Latn` | Biete | 7 |
|
| 183 |
-
| `ldi_Latn` | Laari | 62 | 0.
|
| 184 |
-
| `tzm_Tfng` | Central Atlas Tamazight | 157 | 0.
|
| 185 |
|
| 186 |
## Per language, test-common
|
| 187 |
|
| 188 |
-
Against GlotLID restricted, the best MillID is better on **13** languages and worse on **114**; against fastText restricted, the best MillID is better on **111** languages and worse on **31** (F1 differing by more than 0.01), of 213. Every language is in `
|
| 189 |
|
| 190 |
The 30 languages with the most passages:
|
| 191 |
|
| 192 |
-
| Label | Language | Passages |
|
| 193 |
-
|---|---|---|---|---|---|---|---|
|
| 194 |
-
| `fra_Latn` | French | 1652 | 0.
|
| 195 |
-
| `eng_Latn` | English | 1444 | 0.
|
| 196 |
-
| `spa_Latn` | Spanish | 1136 | 0.
|
| 197 |
-
| `tam_Taml` | Tamil | 754 |
|
| 198 |
-
| `arb_Arab` | Arabic (Modern Standard) | 745 | 0.
|
| 199 |
-
| `por_Latn` | Portuguese | 657 | 0.
|
| 200 |
-
| `kan_Knda` | Kannada | 601 | 0.
|
| 201 |
-
| `zul_Latn` | Zulu | 594 | 0.
|
| 202 |
-
| `swh_Latn` | Swahili | 592 | 0.
|
| 203 |
-
| `mar_Deva` | Marathi | 589 | 0.
|
| 204 |
-
| `tso_Latn` | Tsonga | 581 | 0.
|
| 205 |
-
| `hun_Latn` | Hungarian | 576 |
|
| 206 |
-
| `afr_Latn` | Afrikaans | 569 | 0.
|
| 207 |
-
| `tel_Telu` | Telugu | 555 | 0.
|
| 208 |
-
| `rus_Cyrl` | Russian | 528 | 0.
|
| 209 |
-
| `ory_Orya` | Odia (individual language) | 511 | 0.
|
| 210 |
-
| `sot_Latn` | Southern Sotho | 491 | 0.
|
| 211 |
-
| `nso_Latn` | Northern Sotho | 479 | 0.
|
| 212 |
-
| `asm_Beng` | Assamese | 467 | 0.
|
| 213 |
-
| `ben_Beng` | Bangla | 466 | 0.
|
| 214 |
-
| `pan_Guru` | Punjabi | 460 | 1.000 | 1.000 |
|
| 215 |
-
| `amh_Ethi` | Amharic | 458 |
|
| 216 |
-
| `tsn_Latn` | Tswana | 456 | 0.
|
| 217 |
-
| `gaz_Latn` | Oromo (West Central) | 441 | 0.
|
| 218 |
-
| `som_Latn` | Somali | 439 | 0.
|
| 219 |
-
| `uzn_Latn` | Northern Uzbek | 436 | 0.
|
| 220 |
-
| `tha_Thai` | Thai | 431 | 0.980 | 0.
|
| 221 |
-
| `ydd_Hebr` | Eastern Yiddish | 430 |
|
| 222 |
-
| `tur_Latn` | Turkish | 430 | 0.
|
| 223 |
-
| `nno_Latn` | Norwegian Nynorsk | 427 | 0.
|
| 224 |
|
| 225 |
Where MillID gains most over GlotLID:
|
| 226 |
|
| 227 |
-
| Label | Language | Passages |
|
| 228 |
-
|---|---|---|---|---|---|---|---|
|
| 229 |
-
| `ewo_Latn` | Ewondo | 134 | 0.
|
| 230 |
-
| `eng_Latn` | English | 1444 | 0.
|
| 231 |
-
| `arb_Latn` | Standard Arabic | 163 | 0.
|
| 232 |
-
| `tso_Latn` | Tsonga | 581 | 0.
|
| 233 |
-
| `fra_Latn` | French | 1652 | 0.
|
| 234 |
-
| `arb_Arab` | Arabic (Modern Standard) | 745 | 0.
|
| 235 |
-
| `por_Latn` | Portuguese | 657 | 0.
|
| 236 |
-
| `hin_Deva` | Hindi | 392 | 0.
|
| 237 |
-
| `fuv_Latn` | Nigerian Fulfulde | 164 | 0.
|
| 238 |
-
| `kam_Latn` | Kamba | 272 | 0.
|
| 239 |
-
| `knc_Latn` | Central Kanuri | 293 | 0.
|
| 240 |
-
| `spa_Latn` | Spanish | 1136 | 0.
|
| 241 |
-
| `bug_Latn` | Buginese | 78 | 0.981 | 0.
|
| 242 |
-
| `bjn_Arab` | Banjar | 116 | 0.
|
| 243 |
-
| `san_Deva` | Sanskrit | 239 | 0.990 | 0.
|
| 244 |
|
| 245 |
Where MillID loses most to GlotLID:
|
| 246 |
|
| 247 |
-
| Label | Language | Passages |
|
| 248 |
-
|---|---|---|---|---|---|---|---|
|
| 249 |
-
| `knc_Arab` | Central Kanuri | 68 | 0.
|
| 250 |
-
| `bos_Latn` | Bosnian | 198 |
|
| 251 |
-
| `tzm_Tfng` | Central Atlas Tamazight | 157 | 0.
|
| 252 |
-
| `hrv_Latn` | Croatian | 168 | 0.
|
| 253 |
-
| `zsm_Latn` | Standard Malay | 244 | 0.
|
| 254 |
-
| `taq_Tfng` | Tamasheq | 104 | 0.
|
| 255 |
-
| `ind_Latn` | Indonesian | 335 | 0.
|
| 256 |
-
| `nob_Latn` | Norwegian Bokmål | 265 | 0.
|
| 257 |
-
| `kin_Latn` | Kinyarwanda | 210 | 0.
|
| 258 |
-
| `run_Latn` | Rundi | 291 | 0.
|
| 259 |
-
| `mag_Deva` | Magahi | 97 |
|
| 260 |
-
| `cjk_Latn` | Chokwe | 24 |
|
| 261 |
-
| `taq_Latn` | Tamasheq | 106 |
|
| 262 |
-
| `dzo_Tibt` | Dzongkha | 346 | 0.
|
| 263 |
-
| `lim_Latn` | Limburgish | 196 |
|
| 264 |
|
| 265 |
## Per language, udhr
|
| 266 |
|
| 267 |
-
Against GlotLID restricted, the best MillID is better on **75** languages and worse on **57**; against fastText restricted, the best MillID is better on **313** languages and worse on **8** (F1 differing by more than 0.01), of 358. Every language is in `
|
| 268 |
|
| 269 |
The 30 languages with the most passages:
|
| 270 |
|
| 271 |
-
| Label | Language | Passages |
|
| 272 |
-
|---|---|---|---|---|---|---|---|
|
| 273 |
-
| `roh_Latn` | Romansh | 399 | 0.
|
| 274 |
-
| `oci_Latn` | Occitan | 398 | 0.
|
| 275 |
-
| `cmn_Hani` | Chinese (Mandarin) | 319 | 0.
|
| 276 |
-
| `krl_Latn` | Karelian | 183 | 0.
|
| 277 |
-
| `qug_Latn` | Chimborazo Highland Quichua | 182 | 0.
|
| 278 |
-
| `acu_Latn` | Achuar-Shiwiar | 180 | 0.
|
| 279 |
-
| `hus_Latn` | Huastec | 175 | 0.
|
| 280 |
-
| `hau_Latn` | Hausa | 155 |
|
| 281 |
-
| `auc_Latn` | Waorani | 154 | 0.000 | 0.
|
| 282 |
-
| `kwi_Latn` | Awa-Cuaiquer | 145 | 0.
|
| 283 |
-
| `jpn_Jpan` | Japanese | 140 | 1.000 | 1.000 |
|
| 284 |
-
| `snn_Latn` | Siona | 136 | 0.
|
| 285 |
-
| `ktu_Latn` | Kituba (Democratic Republic of Congo) | 118 | 0.
|
| 286 |
-
| `lat_Latn` | Latin | 118 | 0.
|
| 287 |
-
| `hat_Latn` | Haitian Creole | 117 | 0.
|
| 288 |
-
| `nya_Latn` | Nyanja | 116 |
|
| 289 |
-
| `kmb_Latn` | Kimbundu | 116 |
|
| 290 |
-
| `rmn_Latn` | Balkan Romani | 116 | 0.
|
| 291 |
-
| `twi_Latn` | Twi | 116 | 0.
|
| 292 |
-
| `chr_Cher` | Cherokee | 115 | 0.
|
| 293 |
-
| `por_Latn` | Portuguese | 115 | 0.
|
| 294 |
-
| `lvs_Latn` | Standard Latvian | 115 |
|
| 295 |
-
| `pov_Latn` | Upper Guinea Crioulo | 115 | 0.
|
| 296 |
-
| `kng_Latn` | Koongo | 115 | 0.
|
| 297 |
-
| `ell_Grek` | Greek | 115 |
|
| 298 |
-
| `tha_Thai` | Thai | 114 | 1.000 | 1.000 |
|
| 299 |
-
| `tso_Latn` | Tsonga | 114 | 0.
|
| 300 |
-
| `cjk_Latn` | Chokwe | 114 |
|
| 301 |
-
| `fas_Arab` | Persian | 114 |
|
| 302 |
-
| `lin_Latn` | Lingala | 110 |
|
| 303 |
|
| 304 |
Where MillID gains most over GlotLID:
|
| 305 |
|
| 306 |
-
| Label | Language | Passages |
|
| 307 |
-
|---|---|---|---|---|---|---|---|
|
| 308 |
-
| `mic_Latn` | Mi'kmaw | 59 | 0.
|
| 309 |
-
| `tet_Latn` | Tetum | 58 | 0.
|
| 310 |
-
| `fuf_Latn` | Pular | 58 | 0.
|
| 311 |
-
| `skr_Arab` | Saraiki | 63 | 0.
|
| 312 |
-
| `sag_Latn` | Sango | 59 | 0.
|
| 313 |
-
| `zgh_Tfng` | Standard Moroccan Tamazight | 60 | 0.
|
| 314 |
-
| `qvh_Latn` | Huamalíes-Dos de Mayo Huánuco Quechua | 58 | 0.
|
| 315 |
-
| `chr_Cher` | Cherokee | 115 | 0.
|
| 316 |
-
| `tzm_Tfng` | Central Atlas Tamazight | 1 | 0.
|
| 317 |
-
| `snn_Latn` | Siona | 136 | 0.
|
| 318 |
-
| `fkv_Latn` | Kven Finnish | 58 | 0.
|
| 319 |
-
| `bum_Latn` | Bulu | 58 | 0.
|
| 320 |
-
| `mxv_Latn` | Metlatónoc Mixtec | 58 | 0.
|
| 321 |
-
| `pnb_Arab` | Western Panjabi | 61 | 0.
|
| 322 |
-
| `lld_Latn` | Ladin | 59 | 0.
|
| 323 |
|
| 324 |
Where MillID loses most to GlotLID:
|
| 325 |
|
| 326 |
-
| Label | Language | Passages |
|
| 327 |
-
|---|---|---|---|---|---|---|---|
|
| 328 |
-
| `gyr_Latn` | Guarayu | 49 | 0.
|
| 329 |
-
| `dyu_Latn` | Dyula | 58 | 0.
|
| 330 |
-
| `ell_Grek` | Greek | 115 |
|
| 331 |
-
| `cbu_Latn` | Candoshi-Shapra | 57 | 0.
|
| 332 |
-
| `shp_Latn` | Shipibo-Conibo | 62 | 0.
|
| 333 |
-
| `cpu_Latn` | Pichis Ashéninka | 60 | 0.
|
| 334 |
-
| `qvn_Latn` | North Junín Quechua | 57 | 0.
|
| 335 |
-
| `hin_Deva` | Hindi | 59 | 0.
|
| 336 |
-
| `cni_Latn` | Asháninka | 52 | 0.
|
| 337 |
-
| `hrv_Latn` | Croatian | 58 | 0.
|
| 338 |
-
| `rmn_Latn` | Balkan Romani | 116 | 0.
|
| 339 |
-
| `koi_Cyrl` | Komi-Permyak | 57 |
|
| 340 |
-
| `qvm_Latn` | Margos-Yarowilca-Lauricocha Quechua | 60 | 0.
|
| 341 |
-
| `kea_Latn` | Kabuverdianu | 58 | 0.
|
| 342 |
-
| `ind_Latn` | Indonesian | 60 | 0.
|
| 343 |
|
|
|
|
| 24 |
|
| 25 |
## Macro F1, every set
|
| 26 |
|
| 27 |
+
The three MillID models, and GlotLID and fastText restricted to MillID's labels. The best score on each set is in bold.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 28 |
|
| 29 |
+
| Set | Languages | tiny v1.5 | medium v1 | large v1.5 | GlotLID | fastText |
|
| 30 |
+
|---|---|---|---|---|---|---|
|
| 31 |
+
| `test` | 1,621 | 0.9554 | 0.9576 | 0.9646 | **0.9755** | 0.0659 |
|
| 32 |
+
| `test-common` | 213 | 0.9513 | 0.9527 | 0.9594 | **0.9846** | 0.9173 |
|
| 33 |
+
| `udhr` | 358 | 0.8764 | 0.8780 | 0.8812 | **0.8873** | 0.3499 |
|
| 34 |
+
| `udhr-common` | 170 | 0.9513 | 0.9480 | 0.9529 | **0.9659** | 0.9329 |
|
| 35 |
+
| `udhr-all` | 429 | 0.7149 | 0.7184 | 0.7201 | **0.7245** | 0.2797 |
|
| 36 |
+
|
| 37 |
+
- `test`: every language, passages from documents never seen in training
|
| 38 |
+
- `test-common`: the languages all three systems know
|
| 39 |
+
- `udhr`: the Declaration of Human Rights, languages MillID knows
|
| 40 |
+
- `udhr-common`: the Declaration, languages all three know
|
| 41 |
+
- `udhr-all`: the Declaration, every language in it
|
| 42 |
|
| 43 |
## test
|
| 44 |
|
| 45 |
+
Every language, passages from documents never seen in training. F1 ≥ 0.9: how many of its 1,621 languages a system reaches that F1 on. Top-5 and median F1 are in `leaderboard_overall.csv`.
|
| 46 |
|
| 47 |
+
| System | Size | Top-1 | Top-3 | Macro P | Macro R | Macro F1 | F1 ≥ 0.9 |
|
| 48 |
+
|---|---|---|---|---|---|---|---|
|
| 49 |
+
| [GlotLID](https://huggingface.co/cis-lmu/glotlid) | 1.7 GB | 0.9783 | 0.9978 | 0.9778 | 0.9765 | 0.9755 | 1,490 |
|
| 50 |
+
| [GlotLID, as is](https://huggingface.co/cis-lmu/glotlid) | 1.7 GB | 0.9776 | 0.9977 | 0.9783 | 0.9752 | 0.9748 | 1,488 |
|
| 51 |
+
| [large v1.5](https://huggingface.co/baobabtech/millid-large-v1.5) | 192.6M | 0.9674 | 0.9924 | 0.9655 | 0.9663 | 0.9646 | 1,478 |
|
| 52 |
+
| [medium v1](https://huggingface.co/baobabtech/millid-medium-v1) | 115.0M | 0.9616 | 0.9910 | 0.9564 | 0.9614 | 0.9576 | 1,451 |
|
| 53 |
+
| [tiny v1.5](https://huggingface.co/baobabtech/millid-tiny-v1.5) | 2.9M | 0.9587 | 0.9890 | 0.9550 | 0.9588 | 0.9554 | 1,446 |
|
| 54 |
+
| [fastText](https://huggingface.co/facebook/fasttext-language-identification) | 1.2 GB | 0.2515 | 0.2611 | 0.0521 | 0.1209 | 0.0659 | 26 |
|
| 55 |
+
| [fastText, as is](https://huggingface.co/facebook/fasttext-language-identification) | 1.2 GB | 0.2511 | 0.2610 | 0.0520 | 0.1205 | 0.0657 | 25 |
|
| 56 |
|
| 57 |
## test-common
|
| 58 |
|
| 59 |
+
The languages all three systems know. F1 ≥ 0.9: how many of its 213 languages a system reaches that F1 on. Top-5 and median F1 are in `leaderboard_overall.csv`.
|
| 60 |
|
| 61 |
+
| System | Size | Top-1 | Top-3 | Macro P | Macro R | Macro F1 | F1 ≥ 0.9 |
|
| 62 |
+
|---|---|---|---|---|---|---|---|
|
| 63 |
+
| [GlotLID](https://huggingface.co/cis-lmu/glotlid) | 1.7 GB | 0.9749 | 0.9927 | 0.9923 | 0.9790 | 0.9846 | 203 |
|
| 64 |
+
| [GlotLID, as is](https://huggingface.co/cis-lmu/glotlid) | 1.7 GB | 0.9738 | 0.9925 | 0.9924 | 0.9771 | 0.9835 | 201 |
|
| 65 |
+
| [large v1.5](https://huggingface.co/baobabtech/millid-large-v1.5) | 192.6M | 0.9525 | 0.9898 | 0.9757 | 0.9460 | 0.9594 | 193 |
|
| 66 |
+
| [medium v1](https://huggingface.co/baobabtech/millid-medium-v1) | 115.0M | 0.9425 | 0.9882 | 0.9732 | 0.9361 | 0.9527 | 191 |
|
| 67 |
+
| [tiny v1.5](https://huggingface.co/baobabtech/millid-tiny-v1.5) | 2.9M | 0.9396 | 0.9857 | 0.9709 | 0.9353 | 0.9513 | 188 |
|
| 68 |
+
| [fastText](https://huggingface.co/facebook/fasttext-language-identification) | 1.2 GB | 0.9352 | 0.9710 | 0.9354 | 0.9201 | 0.9173 | 166 |
|
| 69 |
+
| [fastText, as is](https://huggingface.co/facebook/fasttext-language-identification) | 1.2 GB | 0.9339 | 0.9707 | 0.9357 | 0.9169 | 0.9149 | 165 |
|
| 70 |
|
| 71 |
## udhr
|
| 72 |
|
| 73 |
+
The Declaration of Human Rights, languages MillID knows. F1 ≥ 0.9: how many of its 358 languages a system reaches that F1 on. Top-5 and median F1 are in `leaderboard_overall.csv`.
|
| 74 |
|
| 75 |
+
| System | Size | Top-1 | Top-3 | Macro P | Macro R | Macro F1 | F1 ≥ 0.9 |
|
| 76 |
+
|---|---|---|---|---|---|---|---|
|
| 77 |
+
| [GlotLID](https://huggingface.co/cis-lmu/glotlid) | 1.7 GB | 0.8614 | 0.9294 | 0.9258 | 0.8844 | 0.8873 | 291 |
|
| 78 |
+
| [GlotLID, as is](https://huggingface.co/cis-lmu/glotlid) | 1.7 GB | 0.8598 | 0.9280 | 0.9261 | 0.8826 | 0.8865 | 287 |
|
| 79 |
+
| [large v1.5](https://huggingface.co/baobabtech/millid-large-v1.5) | 192.6M | 0.8579 | 0.9183 | 0.9171 | 0.8825 | 0.8812 | 283 |
|
| 80 |
+
| [medium v1](https://huggingface.co/baobabtech/millid-medium-v1) | 115.0M | 0.8524 | 0.9194 | 0.9187 | 0.8769 | 0.8780 | 282 |
|
| 81 |
+
| [tiny v1.5](https://huggingface.co/baobabtech/millid-tiny-v1.5) | 2.9M | 0.8492 | 0.9117 | 0.9148 | 0.8736 | 0.8764 | 274 |
|
| 82 |
+
| [fastText](https://huggingface.co/facebook/fasttext-language-identification) | 1.2 GB | 0.4629 | 0.4857 | 0.3152 | 0.4483 | 0.3499 | 61 |
|
| 83 |
+
| [fastText, as is](https://huggingface.co/facebook/fasttext-language-identification) | 1.2 GB | 0.4625 | 0.4857 | 0.3152 | 0.4479 | 0.3497 | 60 |
|
| 84 |
|
| 85 |
## udhr-common
|
| 86 |
|
| 87 |
+
The Declaration, languages all three know. F1 ≥ 0.9: how many of its 170 languages a system reaches that F1 on. Top-5 and median F1 are in `leaderboard_overall.csv`.
|
| 88 |
|
| 89 |
+
| System | Size | Top-1 | Top-3 | Macro P | Macro R | Macro F1 | F1 ≥ 0.9 |
|
| 90 |
+
|---|---|---|---|---|---|---|---|
|
| 91 |
+
| [GlotLID](https://huggingface.co/cis-lmu/glotlid) | 1.7 GB | 0.9338 | 0.9683 | 0.9888 | 0.9579 | 0.9659 | 156 |
|
| 92 |
+
| [GlotLID, as is](https://huggingface.co/cis-lmu/glotlid) | 1.7 GB | 0.9336 | 0.9678 | 0.9888 | 0.9576 | 0.9657 | 155 |
|
| 93 |
+
| [large v1.5](https://huggingface.co/baobabtech/millid-large-v1.5) | 192.6M | 0.9168 | 0.9630 | 0.9800 | 0.9436 | 0.9529 | 152 |
|
| 94 |
+
| [tiny v1.5](https://huggingface.co/baobabtech/millid-tiny-v1.5) | 2.9M | 0.9123 | 0.9606 | 0.9794 | 0.9393 | 0.9513 | 149 |
|
| 95 |
+
| [medium v1](https://huggingface.co/baobabtech/millid-medium-v1) | 115.0M | 0.9121 | 0.9633 | 0.9770 | 0.9382 | 0.9480 | 153 |
|
| 96 |
+
| [fastText](https://huggingface.co/facebook/fasttext-language-identification) | 1.2 GB | 0.9195 | 0.9647 | 0.9474 | 0.9441 | 0.9329 | 142 |
|
| 97 |
+
| [fastText, as is](https://huggingface.co/facebook/fasttext-language-identification) | 1.2 GB | 0.9187 | 0.9647 | 0.9474 | 0.9431 | 0.9323 | 141 |
|
| 98 |
|
| 99 |
## udhr-all
|
| 100 |
|
| 101 |
+
The Declaration, every language in it. F1 ≥ 0.9: how many of its 429 languages a system reaches that F1 on. Top-5 and median F1 are in `leaderboard_overall.csv`.
|
| 102 |
|
| 103 |
+
| System | Size | Top-1 | Top-3 | Macro P | Macro R | Macro F1 | F1 ≥ 0.9 |
|
| 104 |
+
|---|---|---|---|---|---|---|---|
|
| 105 |
+
| [GlotLID, as is](https://huggingface.co/cis-lmu/glotlid) | 1.7 GB | 0.7502 | 0.8145 | 0.7695 | 0.7529 | 0.7454 | 277 |
|
| 106 |
+
| [GlotLID](https://huggingface.co/cis-lmu/glotlid) | 1.7 GB | 0.7378 | 0.7960 | 0.7425 | 0.7381 | 0.7245 | 272 |
|
| 107 |
+
| [large v1.5](https://huggingface.co/baobabtech/millid-large-v1.5) | 192.6M | 0.7348 | 0.7865 | 0.7392 | 0.7364 | 0.7201 | 266 |
|
| 108 |
+
| [medium v1](https://huggingface.co/baobabtech/millid-medium-v1) | 115.0M | 0.7301 | 0.7875 | 0.7415 | 0.7318 | 0.7184 | 266 |
|
| 109 |
+
| [tiny v1.5](https://huggingface.co/baobabtech/millid-tiny-v1.5) | 2.9M | 0.7274 | 0.7809 | 0.7340 | 0.7290 | 0.7149 | 254 |
|
| 110 |
+
| [fastText](https://huggingface.co/facebook/fasttext-language-identification) | 1.2 GB | 0.3964 | 0.4160 | 0.2479 | 0.3741 | 0.2797 | 55 |
|
| 111 |
+
| [fastText, as is](https://huggingface.co/facebook/fasttext-language-identification) | 1.2 GB | 0.3961 | 0.4160 | 0.2479 | 0.3737 | 0.2795 | 54 |
|
| 112 |
|
| 113 |
## Per language, test
|
| 114 |
|
| 115 |
+
Against GlotLID restricted, the best MillID is better on **165** languages and worse on **498**; against fastText restricted, the best MillID is better on **1616** languages and worse on **0** (F1 differing by more than 0.01), of 1621. F1 per language: tiny is MillID tiny v1.5, medium is MillID medium v1, large is MillID large v1.5; GlotLID and fastText are restricted to MillID's labels. Every language, with their unrestricted F1 too, is in `leaderboard_{set_name}.csv`.
|
| 116 |
|
| 117 |
The 30 languages with the most passages:
|
| 118 |
|
| 119 |
+
| Label | Language | Passages | tiny | medium | large | GlotLID | fastText |
|
| 120 |
+
|---|---|---|---|---|---|---|---|
|
| 121 |
+
| `fra_Latn` | French | 1652 | 0.929 | 0.952 | 0.960 | 0.928 | 0.808 |
|
| 122 |
+
| `eng_Latn` | English | 1444 | 0.917 | 0.916 | 0.934 | 0.859 | 0.648 |
|
| 123 |
+
| `spa_Latn` | Spanish | 1136 | 0.930 | 0.942 | 0.955 | 0.951 | 0.574 |
|
| 124 |
+
| `tam_Taml` | Tamil | 754 | 0.997 | 0.997 | 0.999 | 0.993 | 0.965 |
|
| 125 |
+
| `arb_Arab` | Arabic (Modern Standard) | 745 | 0.814 | 0.829 | 0.833 | 0.785 | 0.512 |
|
| 126 |
+
| `por_Latn` | Portuguese | 657 | 0.947 | 0.955 | 0.965 | 0.949 | 0.755 |
|
| 127 |
+
| `kan_Knda` | Kannada | 601 | 0.969 | 0.982 | 0.965 | 0.993 | 0.922 |
|
| 128 |
+
| `zul_Latn` | Zulu | 594 | 0.846 | 0.833 | 0.878 | 0.896 | 0.539 |
|
| 129 |
+
| `swh_Latn` | Swahili | 592 | 0.848 | 0.858 | 0.870 | 0.916 | 0.346 |
|
| 130 |
+
| `mar_Deva` | Marathi | 589 | 0.965 | 0.981 | 0.976 | 0.990 | 0.926 |
|
| 131 |
+
| `tso_Latn` | Tsonga | 581 | 0.965 | 0.970 | 0.978 | 0.956 | 0.425 |
|
| 132 |
+
| `hun_Latn` | Hungarian | 576 | 0.987 | 0.990 | 0.991 | 0.999 | 0.232 |
|
| 133 |
+
| `afr_Latn` | Afrikaans | 569 | 0.967 | 0.974 | 0.979 | 0.985 | 0.860 |
|
| 134 |
+
| `tel_Telu` | Telugu | 555 | 0.991 | 0.994 | 0.988 | 0.994 | 0.951 |
|
| 135 |
+
| `rus_Cyrl` | Russian | 528 | 0.932 | 0.925 | 0.949 | 0.949 | 0.462 |
|
| 136 |
+
| `nbl_Latn` | South Ndebele | 528 | 0.903 | 0.895 | 0.927 | 0.949 | 0.000 |
|
| 137 |
+
| `ory_Orya` | Odia (individual language) | 511 | 0.988 | 0.993 | 0.993 | 0.994 | 0.976 |
|
| 138 |
+
| `sot_Latn` | Southern Sotho | 491 | 0.966 | 0.966 | 0.972 | 0.978 | 0.660 |
|
| 139 |
+
| `nso_Latn` | Northern Sotho | 479 | 0.956 | 0.960 | 0.962 | 0.969 | 0.914 |
|
| 140 |
+
| `kua_Latn` | Kuanyama | 476 | 0.962 | 0.979 | 0.975 | 0.986 | 0.000 |
|
| 141 |
+
| `asm_Beng` | Assamese | 467 | 0.997 | 0.994 | 0.996 | 0.999 | 0.990 |
|
| 142 |
+
| `ben_Beng` | Bangla | 466 | 0.977 | 0.980 | 0.979 | 0.997 | 0.718 |
|
| 143 |
+
| `pan_Guru` | Punjabi | 460 | 1.000 | 1.000 | 1.000 | 1.000 | 0.990 |
|
| 144 |
+
| `amh_Ethi` | Amharic | 458 | 0.981 | 0.984 | 0.986 | 1.000 | 0.807 |
|
| 145 |
+
| `tsn_Latn` | Tswana | 456 | 0.945 | 0.939 | 0.947 | 0.961 | 0.652 |
|
| 146 |
+
| `vro_Latn` | Võro | 448 | 0.947 | 0.958 | 0.973 | 0.990 | 0.000 |
|
| 147 |
+
| `gaz_Latn` | Oromo (West Central) | 441 | 0.910 | 0.943 | 0.928 | 0.938 | 0.407 |
|
| 148 |
+
| `som_Latn` | Somali | 439 | 0.982 | 0.988 | 0.990 | 0.994 | 0.487 |
|
| 149 |
+
| `uzn_Latn` | Northern Uzbek | 436 | 0.976 | 0.978 | 0.982 | 0.994 | 0.449 |
|
| 150 |
+
| `tha_Thai` | Thai | 431 | 0.979 | 0.970 | 0.981 | 0.980 | 0.650 |
|
| 151 |
|
| 152 |
Where MillID gains most over GlotLID:
|
| 153 |
|
| 154 |
+
| Label | Language | Passages | tiny | medium | large | GlotLID | fastText |
|
| 155 |
+
|---|---|---|---|---|---|---|---|
|
| 156 |
+
| `lbj_Tibt` | Ladakhi | 66 | 0.943 | 0.943 | 0.923 | 0.000 | 0.000 |
|
| 157 |
+
| `ain_Latn` | Ainu | 1 | 0.667 | 0.667 | 0.667 | 0.000 | 0.000 |
|
| 158 |
+
| `ajp_Arab` | South Levantine Arabic | 28 | 0.946 | 0.902 | 0.902 | 0.581 | 0.000 |
|
| 159 |
+
| `und_Laoo` | und | 20 | 0.923 | 0.857 | 0.788 | 0.621 | 0.000 |
|
| 160 |
+
| `sgh_Cyrl` | Shughni | 9 | 0.857 | 0.900 | 0.889 | 0.667 | 0.000 |
|
| 161 |
+
| `und_Thaa` | und | 29 | 0.881 | 0.873 | 0.863 | 0.651 | 0.000 |
|
| 162 |
+
| `icr_Latn` | Islander Creole English | 24 | 0.698 | 0.870 | 0.700 | 0.684 | 0.000 |
|
| 163 |
+
| `und_Geor` | und | 32 | 0.877 | 0.706 | 0.792 | 0.694 | 0.000 |
|
| 164 |
+
| `tdt_Latn` | Tetun Dili | 57 | 0.693 | 0.656 | 0.782 | 0.603 | 0.000 |
|
| 165 |
+
| `cek_Latn` | Eastern Khumi Chin | 67 | 0.785 | 0.853 | 0.825 | 0.684 | 0.000 |
|
| 166 |
+
| `und_Syrc` | und | 43 | 0.966 | 0.977 | 0.989 | 0.822 | 0.000 |
|
| 167 |
+
| `neb_Latn` | Toura (Côte d'Ivoire) | 11 | 0.957 | 0.957 | 1.000 | 0.842 | 0.000 |
|
| 168 |
+
| `dag_Latn` | Dagbani | 149 | 0.767 | 0.817 | 0.814 | 0.660 | 0.000 |
|
| 169 |
+
| `tuk_Arab` | Turkmen | 135 | 0.905 | 0.903 | 0.917 | 0.761 | 0.000 |
|
| 170 |
+
| `orv_Cyrl` | Old Russian | 239 | 0.931 | 0.935 | 0.938 | 0.800 | 0.000 |
|
| 171 |
|
| 172 |
Where MillID loses most to GlotLID:
|
| 173 |
|
| 174 |
+
| Label | Language | Passages | tiny | medium | large | GlotLID | fastText |
|
| 175 |
+
|---|---|---|---|---|---|---|---|
|
| 176 |
+
| `mus_Latn` | Muscogee | 2 | 0.333 | 0.667 | 0.667 | 1.000 | 0.000 |
|
| 177 |
+
| `ttj_Latn` | Tooro | 28 | 0.485 | 0.490 | 0.622 | 0.946 | 0.000 |
|
| 178 |
+
| `knc_Arab` | Central Kanuri | 68 | 0.462 | 0.509 | 0.532 | 0.825 | 0.007 |
|
| 179 |
+
| `hrv_Latn` | Croatian | 168 | 0.622 | 0.687 | 0.707 | 0.944 | 0.178 |
|
| 180 |
+
| `esi_Latn` | North Alaskan Inupiatun | 9 | 0.455 | 0.500 | 0.556 | 0.783 | 0.000 |
|
| 181 |
+
| `hak_Hani` | Hakka Chinese | 29 | 0.621 | 0.415 | 0.581 | 0.824 | 0.000 |
|
| 182 |
+
| `bos_Latn` | Bosnian | 198 | 0.649 | 0.673 | 0.700 | 0.902 | 0.181 |
|
| 183 |
+
| `plw_Latn` | Brooke's Point Palawano | 8 | 0.727 | 0.800 | 0.762 | 1.000 | 0.000 |
|
| 184 |
+
| `ccp_Latn` | Chakma | 3 | 0.800 | 0.333 | 0.750 | 1.000 | 0.000 |
|
| 185 |
+
| `mbd_Latn` | Dibabawon Manobo | 14 | 0.596 | 0.757 | 0.800 | 1.000 | 0.000 |
|
| 186 |
+
| `taq_Tfng` | Tamasheq | 104 | 0.529 | 0.535 | 0.527 | 0.718 | 0.000 |
|
| 187 |
+
| `zsm_Latn` | Standard Malay | 244 | 0.770 | 0.746 | 0.781 | 0.957 | 0.319 |
|
| 188 |
+
| `biu_Latn` | Biete | 7 | 0.667 | 0.667 | 0.833 | 1.000 | 0.000 |
|
| 189 |
+
| `ldi_Latn` | Laari | 62 | 0.777 | 0.831 | 0.765 | 0.992 | 0.000 |
|
| 190 |
+
| `tzm_Tfng` | Central Atlas Tamazight | 157 | 0.659 | 0.620 | 0.614 | 0.816 | 0.569 |
|
| 191 |
|
| 192 |
## Per language, test-common
|
| 193 |
|
| 194 |
+
Against GlotLID restricted, the best MillID is better on **13** languages and worse on **114**; against fastText restricted, the best MillID is better on **111** languages and worse on **31** (F1 differing by more than 0.01), of 213. F1 per language: tiny is MillID tiny v1.5, medium is MillID medium v1, large is MillID large v1.5; GlotLID and fastText are restricted to MillID's labels. Every language, with their unrestricted F1 too, is in `leaderboard_{set_name}.csv`.
|
| 195 |
|
| 196 |
The 30 languages with the most passages:
|
| 197 |
|
| 198 |
+
| Label | Language | Passages | tiny | medium | large | GlotLID | fastText |
|
| 199 |
+
|---|---|---|---|---|---|---|---|
|
| 200 |
+
| `fra_Latn` | French | 1652 | 0.943 | 0.974 | 0.983 | 0.955 | 0.963 |
|
| 201 |
+
| `eng_Latn` | English | 1444 | 0.932 | 0.941 | 0.953 | 0.877 | 0.857 |
|
| 202 |
+
| `spa_Latn` | Spanish | 1136 | 0.937 | 0.953 | 0.972 | 0.958 | 0.941 |
|
| 203 |
+
| `tam_Taml` | Tamil | 754 | 0.997 | 0.998 | 0.999 | 1.000 | 0.997 |
|
| 204 |
+
| `arb_Arab` | Arabic (Modern Standard) | 745 | 0.871 | 0.905 | 0.905 | 0.886 | 0.954 |
|
| 205 |
+
| `por_Latn` | Portuguese | 657 | 0.959 | 0.963 | 0.973 | 0.956 | 0.949 |
|
| 206 |
+
| `kan_Knda` | Kannada | 601 | 0.973 | 0.987 | 0.967 | 0.994 | 0.999 |
|
| 207 |
+
| `zul_Latn` | Zulu | 594 | 0.883 | 0.871 | 0.909 | 0.904 | 0.970 |
|
| 208 |
+
| `swh_Latn` | Swahili | 592 | 0.893 | 0.897 | 0.909 | 0.953 | 0.945 |
|
| 209 |
+
| `mar_Deva` | Marathi | 589 | 0.968 | 0.981 | 0.979 | 0.990 | 0.987 |
|
| 210 |
+
| `tso_Latn` | Tsonga | 581 | 0.979 | 0.984 | 0.985 | 0.956 | 0.986 |
|
| 211 |
+
| `hun_Latn` | Hungarian | 576 | 0.989 | 0.991 | 0.993 | 1.000 | 0.980 |
|
| 212 |
+
| `afr_Latn` | Afrikaans | 569 | 0.972 | 0.976 | 0.983 | 0.985 | 0.986 |
|
| 213 |
+
| `tel_Telu` | Telugu | 555 | 0.997 | 1.000 | 0.999 | 0.998 | 0.999 |
|
| 214 |
+
| `rus_Cyrl` | Russian | 528 | 0.945 | 0.929 | 0.956 | 0.985 | 0.927 |
|
| 215 |
+
| `ory_Orya` | Odia (individual language) | 511 | 0.994 | 0.996 | 0.995 | 1.000 | 0.996 |
|
| 216 |
+
| `sot_Latn` | Southern Sotho | 491 | 0.969 | 0.967 | 0.974 | 0.978 | 0.856 |
|
| 217 |
+
| `nso_Latn` | Northern Sotho | 479 | 0.958 | 0.961 | 0.964 | 0.970 | 0.967 |
|
| 218 |
+
| `asm_Beng` | Assamese | 467 | 0.997 | 0.995 | 0.997 | 0.999 | 0.997 |
|
| 219 |
+
| `ben_Beng` | Bangla | 466 | 0.988 | 0.987 | 0.993 | 0.997 | 0.989 |
|
| 220 |
+
| `pan_Guru` | Punjabi | 460 | 1.000 | 1.000 | 1.000 | 1.000 | 0.990 |
|
| 221 |
+
| `amh_Ethi` | Amharic | 458 | 0.984 | 0.987 | 0.986 | 1.000 | 0.998 |
|
| 222 |
+
| `tsn_Latn` | Tswana | 456 | 0.948 | 0.940 | 0.947 | 0.961 | 0.870 |
|
| 223 |
+
| `gaz_Latn` | Oromo (West Central) | 441 | 0.935 | 0.955 | 0.941 | 0.974 | 0.973 |
|
| 224 |
+
| `som_Latn` | Somali | 439 | 0.984 | 0.992 | 0.991 | 0.995 | 0.991 |
|
| 225 |
+
| `uzn_Latn` | Northern Uzbek | 436 | 0.984 | 0.984 | 0.986 | 0.999 | 0.985 |
|
| 226 |
+
| `tha_Thai` | Thai | 431 | 0.980 | 0.970 | 0.981 | 0.980 | 0.981 |
|
| 227 |
+
| `ydd_Hebr` | Eastern Yiddish | 430 | 0.993 | 0.998 | 0.994 | 1.000 | 0.999 |
|
| 228 |
+
| `tur_Latn` | Turkish | 430 | 0.968 | 0.974 | 0.973 | 0.999 | 0.980 |
|
| 229 |
+
| `nno_Latn` | Norwegian Nynorsk | 427 | 0.913 | 0.919 | 0.927 | 0.985 | 0.954 |
|
| 230 |
|
| 231 |
Where MillID gains most over GlotLID:
|
| 232 |
|
| 233 |
+
| Label | Language | Passages | tiny | medium | large | GlotLID | fastText |
|
| 234 |
+
|---|---|---|---|---|---|---|---|
|
| 235 |
+
| `ewo_Latn` | Ewondo | 134 | 0.959 | 0.965 | 0.978 | 0.883 | 0.860 |
|
| 236 |
+
| `eng_Latn` | English | 1444 | 0.932 | 0.941 | 0.953 | 0.877 | 0.857 |
|
| 237 |
+
| `arb_Latn` | Standard Arabic | 163 | 0.942 | 0.944 | 0.949 | 0.913 | 0.852 |
|
| 238 |
+
| `tso_Latn` | Tsonga | 581 | 0.979 | 0.984 | 0.985 | 0.956 | 0.986 |
|
| 239 |
+
| `fra_Latn` | French | 1652 | 0.943 | 0.974 | 0.983 | 0.955 | 0.963 |
|
| 240 |
+
| `arb_Arab` | Arabic (Modern Standard) | 745 | 0.871 | 0.905 | 0.905 | 0.886 | 0.954 |
|
| 241 |
+
| `por_Latn` | Portuguese | 657 | 0.959 | 0.963 | 0.973 | 0.956 | 0.949 |
|
| 242 |
+
| `hin_Deva` | Hindi | 392 | 0.855 | 0.817 | 0.893 | 0.877 | 0.863 |
|
| 243 |
+
| `fuv_Latn` | Nigerian Fulfulde | 164 | 0.907 | 0.797 | 0.844 | 0.892 | 0.859 |
|
| 244 |
+
| `kam_Latn` | Kamba | 272 | 0.980 | 0.980 | 0.989 | 0.974 | 0.863 |
|
| 245 |
+
| `knc_Latn` | Central Kanuri | 293 | 0.967 | 0.959 | 0.967 | 0.952 | 0.981 |
|
| 246 |
+
| `spa_Latn` | Spanish | 1136 | 0.937 | 0.953 | 0.972 | 0.958 | 0.941 |
|
| 247 |
+
| `bug_Latn` | Buginese | 78 | 0.981 | 0.987 | 0.994 | 0.981 | 0.938 |
|
| 248 |
+
| `bjn_Arab` | Banjar | 116 | 0.992 | 0.983 | 0.996 | 0.987 | 0.960 |
|
| 249 |
+
| `san_Deva` | Sanskrit | 239 | 0.990 | 0.996 | 0.998 | 0.990 | 0.971 |
|
| 250 |
|
| 251 |
Where MillID loses most to GlotLID:
|
| 252 |
|
| 253 |
+
| Label | Language | Passages | tiny | medium | large | GlotLID | fastText |
|
| 254 |
+
|---|---|---|---|---|---|---|---|
|
| 255 |
+
| `knc_Arab` | Central Kanuri | 68 | 0.522 | 0.562 | 0.598 | 0.860 | 0.029 |
|
| 256 |
+
| `bos_Latn` | Bosnian | 198 | 0.685 | 0.727 | 0.762 | 1.000 | 0.897 |
|
| 257 |
+
| `tzm_Tfng` | Central Atlas Tamazight | 157 | 0.697 | 0.645 | 0.649 | 0.934 | 0.751 |
|
| 258 |
+
| `hrv_Latn` | Croatian | 168 | 0.659 | 0.717 | 0.757 | 0.994 | 0.856 |
|
| 259 |
+
| `zsm_Latn` | Standard Malay | 244 | 0.789 | 0.773 | 0.809 | 0.998 | 0.908 |
|
| 260 |
+
| `taq_Tfng` | Tamasheq | 104 | 0.654 | 0.613 | 0.604 | 0.805 | 0.000 |
|
| 261 |
+
| `ind_Latn` | Indonesian | 335 | 0.842 | 0.807 | 0.848 | 0.991 | 0.887 |
|
| 262 |
+
| `nob_Latn` | Norwegian Bokmål | 265 | 0.821 | 0.820 | 0.832 | 0.959 | 0.894 |
|
| 263 |
+
| `kin_Latn` | Kinyarwanda | 210 | 0.849 | 0.818 | 0.864 | 0.981 | 0.854 |
|
| 264 |
+
| `run_Latn` | Rundi | 291 | 0.883 | 0.843 | 0.879 | 0.985 | 0.855 |
|
| 265 |
+
| `mag_Deva` | Magahi | 97 | 0.890 | 0.905 | 0.897 | 1.000 | 0.772 |
|
| 266 |
+
| `cjk_Latn` | Chokwe | 24 | 0.857 | 0.909 | 0.837 | 1.000 | 0.807 |
|
| 267 |
+
| `taq_Latn` | Tamasheq | 106 | 0.902 | 0.909 | 0.922 | 1.000 | 0.838 |
|
| 268 |
+
| `dzo_Tibt` | Dzongkha | 346 | 0.906 | 0.926 | 0.916 | 0.999 | 0.696 |
|
| 269 |
+
| `lim_Latn` | Limburgish | 196 | 0.896 | 0.915 | 0.930 | 1.000 | 0.956 |
|
| 270 |
|
| 271 |
## Per language, udhr
|
| 272 |
|
| 273 |
+
Against GlotLID restricted, the best MillID is better on **75** languages and worse on **57**; against fastText restricted, the best MillID is better on **313** languages and worse on **8** (F1 differing by more than 0.01), of 358. F1 per language: tiny is MillID tiny v1.5, medium is MillID medium v1, large is MillID large v1.5; GlotLID and fastText are restricted to MillID's labels. Every language, with their unrestricted F1 too, is in `leaderboard_{set_name}.csv`.
|
| 274 |
|
| 275 |
The 30 languages with the most passages:
|
| 276 |
|
| 277 |
+
| Label | Language | Passages | tiny | medium | large | GlotLID | fastText |
|
| 278 |
+
|---|---|---|---|---|---|---|---|
|
| 279 |
+
| `roh_Latn` | Romansh | 399 | 0.994 | 0.991 | 0.998 | 0.999 | 0.946 |
|
| 280 |
+
| `oci_Latn` | Occitan | 398 | 0.277 | 0.331 | 0.338 | 0.376 | 0.435 |
|
| 281 |
+
| `cmn_Hani` | Chinese (Mandarin) | 319 | 0.892 | 0.886 | 0.875 | 0.868 | 0.468 |
|
| 282 |
+
| `krl_Latn` | Karelian | 183 | 0.000 | 0.000 | 0.000 | 0.011 | 0.000 |
|
| 283 |
+
| `qug_Latn` | Chimborazo Highland Quichua | 182 | 0.011 | 0.000 | 0.000 | 0.000 | 0.000 |
|
| 284 |
+
| `acu_Latn` | Achuar-Shiwiar | 180 | 0.828 | 0.743 | 0.801 | 0.854 | 0.000 |
|
| 285 |
+
| `hus_Latn` | Huastec | 175 | 0.977 | 0.991 | 0.988 | 0.986 | 0.000 |
|
| 286 |
+
| `hau_Latn` | Hausa | 155 | 0.990 | 0.997 | 1.000 | 1.000 | 0.876 |
|
| 287 |
+
| `auc_Latn` | Waorani | 154 | 0.000 | 0.050 | 0.025 | 0.000 | 0.000 |
|
| 288 |
+
| `kwi_Latn` | Awa-Cuaiquer | 145 | 0.880 | 0.775 | 0.863 | 0.877 | 0.000 |
|
| 289 |
+
| `jpn_Jpan` | Japanese | 140 | 1.000 | 1.000 | 1.000 | 1.000 | 0.944 |
|
| 290 |
+
| `snn_Latn` | Siona | 136 | 0.630 | 0.773 | 0.467 | 0.514 | 0.000 |
|
| 291 |
+
| `ktu_Latn` | Kituba (Democratic Republic of Congo) | 118 | 0.765 | 0.769 | 0.773 | 0.680 | 0.000 |
|
| 292 |
+
| `lat_Latn` | Latin | 118 | 0.996 | 0.996 | 0.996 | 0.992 | 0.000 |
|
| 293 |
+
| `hat_Latn` | Haitian Creole | 117 | 0.992 | 0.991 | 0.987 | 0.979 | 0.699 |
|
| 294 |
+
| `nya_Latn` | Nyanja | 116 | 0.991 | 0.987 | 0.978 | 1.000 | 0.806 |
|
| 295 |
+
| `kmb_Latn` | Kimbundu | 116 | 0.970 | 0.979 | 0.991 | 1.000 | 0.877 |
|
| 296 |
+
| `rmn_Latn` | Balkan Romani | 116 | 0.678 | 0.667 | 0.704 | 0.857 | 0.000 |
|
| 297 |
+
| `twi_Latn` | Twi | 116 | 0.983 | 0.987 | 0.991 | 0.996 | 0.396 |
|
| 298 |
+
| `chr_Cher` | Cherokee | 115 | 0.670 | 0.715 | 0.945 | 0.663 | 0.000 |
|
| 299 |
+
| `por_Latn` | Portuguese | 115 | 0.955 | 0.996 | 1.000 | 0.996 | 0.793 |
|
| 300 |
+
| `lvs_Latn` | Standard Latvian | 115 | 0.987 | 0.996 | 1.000 | 1.000 | 0.991 |
|
| 301 |
+
| `pov_Latn` | Upper Guinea Crioulo | 115 | 0.821 | 0.838 | 0.827 | 0.833 | 0.000 |
|
| 302 |
+
| `kng_Latn` | Koongo | 115 | 0.076 | 0.047 | 0.047 | 0.193 | 0.457 |
|
| 303 |
+
| `ell_Grek` | Greek | 115 | 0.670 | 0.655 | 0.670 | 1.000 | 1.000 |
|
| 304 |
+
| `tha_Thai` | Thai | 114 | 1.000 | 1.000 | 1.000 | 1.000 | 0.996 |
|
| 305 |
+
| `tso_Latn` | Tsonga | 114 | 0.860 | 0.882 | 0.774 | 0.781 | 0.691 |
|
| 306 |
+
| `cjk_Latn` | Chokwe | 114 | 0.871 | 0.949 | 0.959 | 1.000 | 0.533 |
|
| 307 |
+
| `fas_Arab` | Persian | 114 | 0.991 | 0.991 | 0.982 | 1.000 | 0.934 |
|
| 308 |
+
| `lin_Latn` | Lingala | 110 | 0.991 | 1.000 | 0.996 | 1.000 | 0.986 |
|
| 309 |
|
| 310 |
Where MillID gains most over GlotLID:
|
| 311 |
|
| 312 |
+
| Label | Language | Passages | tiny | medium | large | GlotLID | fastText |
|
| 313 |
+
|---|---|---|---|---|---|---|---|
|
| 314 |
+
| `mic_Latn` | Mi'kmaw | 59 | 0.992 | 1.000 | 1.000 | 0.427 | 0.000 |
|
| 315 |
+
| `tet_Latn` | Tetum | 58 | 0.638 | 0.699 | 0.708 | 0.167 | 0.000 |
|
| 316 |
+
| `fuf_Latn` | Pular | 58 | 0.906 | 0.816 | 0.936 | 0.483 | 0.000 |
|
| 317 |
+
| `skr_Arab` | Saraiki | 63 | 0.867 | 0.923 | 0.875 | 0.500 | 0.000 |
|
| 318 |
+
| `sag_Latn` | Sango | 59 | 0.992 | 0.992 | 0.992 | 0.578 | 0.189 |
|
| 319 |
+
| `zgh_Tfng` | Standard Moroccan Tamazight | 60 | 0.750 | 0.824 | 0.652 | 0.462 | 0.000 |
|
| 320 |
+
| `qvh_Latn` | Huamalíes-Dos de Mayo Huánuco Quechua | 58 | 0.468 | 0.215 | 0.215 | 0.154 | 0.000 |
|
| 321 |
+
| `chr_Cher` | Cherokee | 115 | 0.670 | 0.715 | 0.945 | 0.663 | 0.000 |
|
| 322 |
+
| `tzm_Tfng` | Central Atlas Tamazight | 1 | 0.167 | 0.333 | 0.100 | 0.065 | 0.033 |
|
| 323 |
+
| `snn_Latn` | Siona | 136 | 0.630 | 0.773 | 0.467 | 0.514 | 0.000 |
|
| 324 |
+
| `fkv_Latn` | Kven Finnish | 58 | 0.635 | 0.697 | 0.906 | 0.667 | 0.000 |
|
| 325 |
+
| `bum_Latn` | Bulu | 58 | 0.215 | 0.319 | 0.188 | 0.098 | 0.000 |
|
| 326 |
+
| `mxv_Latn` | Metlatónoc Mixtec | 58 | 0.129 | 0.215 | 0.034 | 0.000 | 0.000 |
|
| 327 |
+
| `pnb_Arab` | Western Panjabi | 61 | 0.882 | 0.931 | 0.884 | 0.739 | 0.000 |
|
| 328 |
+
| `lld_Latn` | Ladin | 59 | 0.909 | 0.899 | 0.881 | 0.731 | 0.000 |
|
| 329 |
|
| 330 |
Where MillID loses most to GlotLID:
|
| 331 |
|
| 332 |
+
| Label | Language | Passages | tiny | medium | large | GlotLID | fastText |
|
| 333 |
+
|---|---|---|---|---|---|---|---|
|
| 334 |
+
| `gyr_Latn` | Guarayu | 49 | 0.537 | 0.339 | 0.393 | 0.990 | 0.000 |
|
| 335 |
+
| `dyu_Latn` | Dyula | 58 | 0.000 | 0.000 | 0.000 | 0.434 | 0.015 |
|
| 336 |
+
| `ell_Grek` | Greek | 115 | 0.670 | 0.655 | 0.670 | 1.000 | 1.000 |
|
| 337 |
+
| `cbu_Latn` | Candoshi-Shapra | 57 | 0.500 | 0.371 | 0.557 | 0.874 | 0.000 |
|
| 338 |
+
| `shp_Latn` | Shipibo-Conibo | 62 | 0.469 | 0.410 | 0.607 | 0.912 | 0.000 |
|
| 339 |
+
| `cpu_Latn` | Pichis Ashéninka | 60 | 0.729 | 0.571 | 0.571 | 0.974 | 0.000 |
|
| 340 |
+
| `qvn_Latn` | North Junín Quechua | 57 | 0.219 | 0.219 | 0.034 | 0.460 | 0.000 |
|
| 341 |
+
| `hin_Deva` | Hindi | 59 | 0.723 | 0.607 | 0.727 | 0.957 | 0.975 |
|
| 342 |
+
| `cni_Latn` | Asháninka | 52 | 0.773 | 0.708 | 0.717 | 0.963 | 0.000 |
|
| 343 |
+
| `hrv_Latn` | Croatian | 58 | 0.500 | 0.537 | 0.559 | 0.727 | 0.474 |
|
| 344 |
+
| `rmn_Latn` | Balkan Romani | 116 | 0.678 | 0.667 | 0.704 | 0.857 | 0.000 |
|
| 345 |
+
| `koi_Cyrl` | Komi-Permyak | 57 | 0.837 | 0.837 | 0.849 | 1.000 | 0.000 |
|
| 346 |
+
| `qvm_Latn` | Margos-Yarowilca-Lauricocha Quechua | 60 | 0.356 | 0.094 | 0.356 | 0.500 | 0.000 |
|
| 347 |
+
| `kea_Latn` | Kabuverdianu | 58 | 0.792 | 0.794 | 0.773 | 0.935 | 0.124 |
|
| 348 |
+
| `ind_Latn` | Indonesian | 60 | 0.768 | 0.674 | 0.645 | 0.903 | 0.565 |
|
| 349 |
|