Hugging Face's logo

baobabtech
/
millid-tiny-v1.5

Text Classification
PyTorch
ONNX
Safetensors
multilingual
language-identification
byte-level
Eval Results (legacy)
🇪🇺 Region: EU
Model card Files
xet
 Community
5

Model card for the public release

#2
by byamasupatrick - opened
base: refs/heads/main
←
from: refs/pr/2
Discussion Files changed
+220
-230
Files changed (1) hide show
  1. README.md +220 -230
README.md CHANGED
@@ -51,13 +51,15 @@ A byte-level, encoder-only transformer that names the language of a passage, fro
51
 
52
  ## The MillID models
53
 
54
- | Model | Size | Macro F1, `test` / `udhr` | Runs best on |
55
- |---|---|---|---|
56
- | **MillID tiny v1.5** (this model) | 2.9M parameters, ONNX 11 MB | 0.9554 / 0.8764 | a CPU, or a web page |
57
- | [MillID medium v1](https://huggingface.co/baobabtech/millid-medium-v1) | 115M parameters, ONNX 455 MB | 0.9576 / 0.8780 | a GPU |
58
- | [MillID large v1.5](https://huggingface.co/baobabtech/millid-large-v1.5) | 193M parameters, ONNX 764 MB | 0.9646 / 0.8812 | a GPU: the most accurate |
59
 
60
- All three, language by language, against GlotLID and Meta's fastText: the [leaderboard](https://huggingface.co/baobabtech/millid). GlotLID, restricted to MillID's labels, scores 0.9755 / 0.8873 on the same sets.
 
 
 
 
 
 
61
 
62
  ## Use it
63
 
@@ -99,261 +101,249 @@ UTF-8 bytes (after Unicode NFKC), each embedded as byte + Unicode script + byte-
99
 
100
  ## Evaluation against GlotLID and fastText
101
 
102
- `validation` chose the checkpoint; `test` is split by document from training; `udhr` is the Universal Declaration of Human Rights, labelled by its translators, over the languages MillID knows, and `udhr-all` over all of them. `glotlid` is [GlotLID](https://huggingface.co/cis-lmu/glotlid) as it is; `glotlid@millid` is GlotLID restricted to MillID's labels, the like-for-like comparison. GlotLID's own labels came from training on data that includes UDHR and labelled FinePDFs and FineWeb-2, so its scores on `udhr` and on those sources are optimistic. `fasttext` is Meta's [fastText language identifier](https://huggingface.co/facebook/fasttext-language-identification) (NLLB, 218 labels), its answers mapped onto MillID's spelling (Simplified and Traditional Han as `Hani`, `pes`/`prs` as `fas`, `est` as `ekk`). It cannot answer most of the 1,600 languages here, so `test-common` and `udhr-common` score every model on the passages whose language all of them know: the head-to-head on equal terms.
103
-
104
- | Set | Model | Top-1 | Top-3 | Top-5 | Macro P | Macro R | Macro F1 | Median F1 | Languages |
105
- |---|---|---|---|---|---|---|---|---|---|
106
- | validation | millid | 0.9578 | 0.9889 | 0.9933 | 0.9539 | 0.9591 | 0.9551 | 0.9822 | 1626 |
107
- | test | millid | 0.9587 | 0.9890 | 0.9932 | 0.9550 | 0.9588 | 0.9554 | 0.9825 | 1621 |
108
- | test | glotlid | 0.9776 | 0.9977 | 0.9984 | 0.9783 | 0.9752 | 0.9748 | 1.0000 | 1621 |
109
- | test | glotlid@millid | 0.9783 | 0.9978 | 0.9985 | 0.9778 | 0.9765 | 0.9755 | 1.0000 | 1621 |
110
- | test | fasttext | 0.2511 | 0.2610 | 0.2625 | 0.0520 | 0.1205 | 0.0657 | 0.0000 | 1621 |
111
- | test | fasttext@millid | 0.2515 | 0.2611 | 0.2625 | 0.0521 | 0.1209 | 0.0659 | 0.0000 | 1621 |
112
- | test-common | millid | 0.9396 | 0.9857 | 0.9909 | 0.9709 | 0.9353 | 0.9513 | 0.9726 | 213 |
113
- | test-common | glotlid | 0.9738 | 0.9925 | 0.9946 | 0.9924 | 0.9771 | 0.9835 | 0.9983 | 213 |
114
- | test-common | glotlid@millid | 0.9749 | 0.9927 | 0.9948 | 0.9923 | 0.9790 | 0.9846 | 0.9983 | 213 |
115
- | test-common | fasttext | 0.9339 | 0.9707 | 0.9762 | 0.9357 | 0.9169 | 0.9149 | 0.9680 | 213 |
116
- | test-common | fasttext@millid | 0.9352 | 0.9710 | 0.9763 | 0.9354 | 0.9201 | 0.9173 | 0.9680 | 213 |
117
- | udhr | millid | 0.8492 | 0.9117 | 0.9257 | 0.9148 | 0.8736 | 0.8764 | 0.9907 | 358 |
118
- | udhr | glotlid | 0.8598 | 0.9280 | 0.9421 | 0.9261 | 0.8826 | 0.8865 | 0.9915 | 358 |
119
- | udhr | glotlid@millid | 0.8614 | 0.9294 | 0.9433 | 0.9258 | 0.8844 | 0.8873 | 0.9915 | 358 |
120
- | udhr | fasttext | 0.4625 | 0.4857 | 0.4903 | 0.3152 | 0.4479 | 0.3497 | 0.0000 | 358 |
121
- | udhr | fasttext@millid | 0.4629 | 0.4857 | 0.4904 | 0.3152 | 0.4483 | 0.3499 | 0.0000 | 358 |
122
- | udhr-common | millid | 0.9123 | 0.9606 | 0.9655 | 0.9794 | 0.9393 | 0.9513 | 1.0000 | 170 |
123
- | udhr-common | glotlid | 0.9336 | 0.9678 | 0.9732 | 0.9888 | 0.9576 | 0.9657 | 1.0000 | 170 |
124
- | udhr-common | glotlid@millid | 0.9338 | 0.9683 | 0.9736 | 0.9888 | 0.9579 | 0.9659 | 1.0000 | 170 |
125
- | udhr-common | fasttext | 0.9187 | 0.9647 | 0.9741 | 0.9474 | 0.9431 | 0.9323 | 1.0000 | 170 |
126
- | udhr-common | fasttext@millid | 0.9195 | 0.9647 | 0.9742 | 0.9474 | 0.9441 | 0.9329 | 1.0000 | 170 |
127
- | udhr-all | millid | 0.7274 | 0.7809 | 0.7928 | 0.7340 | 0.7290 | 0.7149 | 0.9649 | 429 |
128
- | udhr-all | glotlid | 0.7502 | 0.8145 | 0.8282 | 0.7695 | 0.7529 | 0.7454 | 0.9796 | 429 |
129
- | udhr-all | glotlid@millid | 0.7378 | 0.7960 | 0.8079 | 0.7425 | 0.7381 | 0.7245 | 0.9748 | 429 |
130
- | udhr-all | fasttext | 0.3961 | 0.4160 | 0.4200 | 0.2479 | 0.3737 | 0.2795 | 0.0000 | 429 |
131
- | udhr-all | fasttext@millid | 0.3964 | 0.4160 | 0.4200 | 0.2479 | 0.3741 | 0.2797 | 0.0000 | 429 |
132
 
133
  Top-1 accuracy by passage length:
134
 
135
- | Set | Model | short | sentence | paragraph |
136
  |---|---|---|---|---|
137
- | validation | millid | 0.9312 | 0.9788 | 0.9850 |
138
- | test | millid | 0.9333 | 0.9797 | 0.9833 |
139
- | test | glotlid | 0.9664 | 0.9853 | 0.9911 |
140
- | test | glotlid@millid | 0.9674 | 0.9858 | 0.9912 |
141
- | test | fasttext | 0.2760 | 0.2501 | 0.1958 |
142
- | test | fasttext@millid | 0.2764 | 0.2506 | 0.1959 |
143
- | test-common | millid | 0.9125 | 0.9685 | 0.9725 |
144
- | test-common | glotlid | 0.9594 | 0.9876 | 0.9947 |
145
- | test-common | glotlid@millid | 0.9609 | 0.9885 | 0.9949 |
146
- | test-common | fasttext | 0.9112 | 0.9562 | 0.9651 |
147
- | test-common | fasttext@millid | 0.9125 | 0.9581 | 0.9655 |
148
-
149
- ### Leaderboard: test
150
-
151
- MillID's F1 against the others', language by language: against GlotLID restricted to MillID's labels, MillID is better on **103** languages, worse on **869** and within 0.01 on 649; against fastText restricted to MillID's labels, MillID is better on **1616** languages, worse on **0** and within 0.01 on 5. The full table, with precision and recall, is `reports/test/leaderboard.csv`.
152
 
153
  The 20 languages with the most passages:
154
 
155
- | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
156
- |---|---|---|---|---|---|---|---|
157
- | `fra_Latn` | French | 1652 | 0.929 | 0.928 | 0.928 | 0.808 | 0.808 |
158
- | `eng_Latn` | English | 1444 | 0.917 | 0.858 | 0.859 | 0.653 | 0.648 |
159
- | `spa_Latn` | Spanish | 1136 | 0.930 | 0.951 | 0.951 | 0.574 | 0.574 |
160
- | `tam_Taml` | Tamil | 754 | 0.997 | 0.993 | 0.993 | 0.965 | 0.965 |
161
- | `arb_Arab` | Arabic (Modern Standard) | 745 | 0.814 | 0.785 | 0.785 | 0.513 | 0.512 |
162
- | `por_Latn` | Portuguese | 657 | 0.947 | 0.947 | 0.949 | 0.756 | 0.755 |
163
- | `kan_Knda` | Kannada | 601 | 0.969 | 0.993 | 0.993 | 0.922 | 0.922 |
164
- | `zul_Latn` | Zulu | 594 | 0.846 | 0.896 | 0.896 | 0.539 | 0.539 |
165
- | `swh_Latn` | Swahili | 592 | 0.848 | 0.917 | 0.916 | 0.346 | 0.346 |
166
- | `mar_Deva` | Marathi | 589 | 0.965 | 0.989 | 0.990 | 0.926 | 0.926 |
167
- | `tso_Latn` | Tsonga | 581 | 0.965 | 0.956 | 0.956 | 0.425 | 0.425 |
168
- | `hun_Latn` | Hungarian | 576 | 0.987 | 0.999 | 0.999 | 0.232 | 0.232 |
169
- | `afr_Latn` | Afrikaans | 569 | 0.967 | 0.985 | 0.985 | 0.861 | 0.860 |
170
- | `tel_Telu` | Telugu | 555 | 0.991 | 0.994 | 0.994 | 0.951 | 0.951 |
171
- | `rus_Cyrl` | Russian | 528 | 0.932 | 0.949 | 0.949 | 0.462 | 0.462 |
172
- | `nbl_Latn` | South Ndebele | 528 | 0.903 | 0.948 | 0.949 | 0.000 | 0.000 |
173
- | `ory_Orya` | Odia (individual language) | 511 | 0.988 | 0.991 | 0.994 | 0.976 | 0.976 |
174
- | `sot_Latn` | Southern Sotho | 491 | 0.966 | 0.977 | 0.978 | 0.660 | 0.660 |
175
- | `nso_Latn` | Northern Sotho | 479 | 0.956 | 0.969 | 0.969 | 0.915 | 0.914 |
176
- | `kua_Latn` | Kuanyama | 476 | 0.962 | 0.986 | 0.986 | 0.000 | 0.000 |
177
 
178
  Where MillID gains most over GlotLID:
179
 
180
- | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
181
- |---|---|---|---|---|---|---|---|
182
- | `lbj_Tibt` | Ladakhi | 66 | 0.943 | 0.000 | 0.000 | 0.000 | 0.000 |
183
- | `ain_Latn` | Ainu | 1 | 0.667 | 0.000 | 0.000 | 0.000 | 0.000 |
184
- | `ajp_Arab` | South Levantine Arabic | 28 | 0.946 | 0.581 | 0.581 | 0.000 | 0.000 |
185
- | `und_Laoo` | und | 20 | 0.923 | 0.621 | 0.621 | 0.000 | 0.000 |
186
- | `und_Thaa` | und | 29 | 0.881 | 0.651 | 0.651 | 0.000 | 0.000 |
187
- | `sgh_Cyrl` | Shughni | 9 | 0.857 | 0.667 | 0.667 | 0.000 | 0.000 |
188
- | `und_Geor` | und | 32 | 0.877 | 0.694 | 0.694 | 0.000 | 0.000 |
189
- | `tuk_Arab` | Turkmen | 135 | 0.905 | 0.761 | 0.761 | 0.000 | 0.000 |
190
- | `und_Syrc` | und | 43 | 0.966 | 0.822 | 0.822 | 0.000 | 0.000 |
191
- | `gmh_Latn` | Middle High German | 177 | 0.922 | 0.786 | 0.786 | 0.000 | 0.000 |
192
- | `orv_Cyrl` | Old Russian | 239 | 0.931 | 0.800 | 0.800 | 0.000 | 0.000 |
193
- | `und_Kana` | und | 136 | 0.982 | 0.862 | 0.862 | 0.000 | 0.000 |
194
- | `und_Yiii` | und | 38 | 1.000 | 0.894 | 0.884 | 0.000 | 0.000 |
195
- | `neb_Latn` | Toura (Côte d'Ivoire) | 11 | 0.957 | 0.625 | 0.842 | 0.000 | 0.000 |
196
- | `dag_Latn` | Dagbani | 149 | 0.767 | 0.662 | 0.660 | 0.000 | 0.000 |
197
 
198
  Where MillID loses most to GlotLID:
199
 
200
- | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
201
- |---|---|---|---|---|---|---|---|
202
- | `mus_Latn` | Muscogee | 2 | 0.333 | 1.000 | 1.000 | 0.000 | 0.000 |
203
- | `ttj_Latn` | Tooro | 28 | 0.485 | 0.946 | 0.946 | 0.000 | 0.000 |
204
- | `mbd_Latn` | Dibabawon Manobo | 14 | 0.596 | 1.000 | 1.000 | 0.000 | 0.000 |
205
- | `knc_Arab` | Central Kanuri | 68 | 0.462 | 0.823 | 0.825 | 0.007 | 0.007 |
206
- | `biu_Latn` | Biete | 7 | 0.667 | 1.000 | 1.000 | 0.000 | 0.000 |
207
- | `dks_Latn` | Southeastern Dinka | 3 | 0.667 | 1.000 | 1.000 | 0.000 | 0.000 |
208
- | `esi_Latn` | North Alaskan Inupiatun | 9 | 0.455 | 0.783 | 0.783 | 0.000 | 0.000 |
209
- | `hrv_Latn` | Croatian | 168 | 0.622 | 0.944 | 0.944 | 0.178 | 0.178 |
210
- | `plw_Latn` | Brooke's Point Palawano | 8 | 0.727 | 1.000 | 1.000 | 0.000 | 0.000 |
211
- | `bos_Latn` | Bosnian | 198 | 0.649 | 0.902 | 0.902 | 0.181 | 0.181 |
212
- | `xbr_Latn` | Kambera | 8 | 0.750 | 1.000 | 1.000 | 0.000 | 0.000 |
213
- | `nyf_Latn` | Giryama | 10 | 0.645 | 0.952 | 0.870 | 0.000 | 0.000 |
214
- | `ldi_Latn` | Laari | 62 | 0.777 | 0.992 | 0.992 | 0.000 | 0.000 |
215
- | `hak_Hani` | Hakka Chinese | 29 | 0.621 | 0.824 | 0.824 | 0.000 | 0.000 |
216
- | `ccp_Latn` | Chakma | 3 | 0.800 | 1.000 | 1.000 | 0.000 | 0.000 |
217
-
218
- ### Leaderboard: udhr
219
-
220
- MillID's F1 against the others', language by language: against GlotLID restricted to MillID's labels, MillID is better on **52** languages, worse on **105** and within 0.01 on 201; against fastText restricted to MillID's labels, MillID is better on **303** languages, worse on **15** and within 0.01 on 40. The full table, with precision and recall, is `reports/udhr/leaderboard.csv`.
221
 
222
  The 20 languages with the most passages:
223
 
224
- | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
225
- |---|---|---|---|---|---|---|---|
226
- | `roh_Latn` | Romansh | 399 | 0.994 | 0.999 | 0.999 | 0.946 | 0.946 |
227
- | `oci_Latn` | Occitan | 398 | 0.277 | 0.376 | 0.376 | 0.435 | 0.435 |
228
- | `cmn_Hani` | Chinese (Mandarin) | 319 | 0.892 | 0.868 | 0.868 | 0.468 | 0.468 |
229
- | `krl_Latn` | Karelian | 183 | 0.000 | 0.011 | 0.011 | 0.000 | 0.000 |
230
- | `qug_Latn` | Chimborazo Highland Quichua | 182 | 0.011 | 0.000 | 0.000 | 0.000 | 0.000 |
231
- | `acu_Latn` | Achuar-Shiwiar | 180 | 0.828 | 0.854 | 0.854 | 0.000 | 0.000 |
232
- | `hus_Latn` | Huastec | 175 | 0.977 | 0.986 | 0.986 | 0.000 | 0.000 |
233
- | `hau_Latn` | Hausa | 155 | 0.990 | 1.000 | 1.000 | 0.876 | 0.876 |
234
- | `auc_Latn` | Waorani | 154 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
235
- | `kwi_Latn` | Awa-Cuaiquer | 145 | 0.880 | 0.877 | 0.877 | 0.000 | 0.000 |
236
- | `jpn_Jpan` | Japanese | 140 | 1.000 | 1.000 | 1.000 | 0.944 | 0.944 |
237
- | `snn_Latn` | Siona | 136 | 0.630 | 0.514 | 0.514 | 0.000 | 0.000 |
238
- | `ktu_Latn` | Kituba (Democratic Republic of Congo) | 118 | 0.765 | 0.680 | 0.680 | 0.000 | 0.000 |
239
- | `lat_Latn` | Latin | 118 | 0.996 | 0.992 | 0.992 | 0.000 | 0.000 |
240
- | `hat_Latn` | Haitian Creole | 117 | 0.992 | 0.979 | 0.979 | 0.699 | 0.699 |
241
- | `nya_Latn` | Nyanja | 116 | 0.991 | 1.000 | 1.000 | 0.806 | 0.806 |
242
- | `kmb_Latn` | Kimbundu | 116 | 0.970 | 1.000 | 1.000 | 0.877 | 0.877 |
243
- | `rmn_Latn` | Balkan Romani | 116 | 0.678 | 0.857 | 0.857 | 0.000 | 0.000 |
244
- | `twi_Latn` | Twi | 116 | 0.983 | 0.996 | 0.996 | 0.399 | 0.396 |
245
- | `chr_Cher` | Cherokee | 115 | 0.670 | 0.663 | 0.663 | 0.000 | 0.000 |
246
 
247
  Where MillID gains most over GlotLID:
248
 
249
- | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
250
- |---|---|---|---|---|---|---|---|
251
- | `mic_Latn` | Mi'kmaw | 59 | 0.992 | 0.427 | 0.427 | 0.000 | 0.000 |
252
- | `tet_Latn` | Tetum | 58 | 0.638 | 0.167 | 0.167 | 0.000 | 0.000 |
253
- | `fuf_Latn` | Pular | 58 | 0.906 | 0.483 | 0.483 | 0.000 | 0.000 |
254
- | `sag_Latn` | Sango | 59 | 0.992 | 0.578 | 0.578 | 0.189 | 0.189 |
255
- | `skr_Arab` | Saraiki | 63 | 0.867 | 0.500 | 0.500 | 0.000 | 0.000 |
256
- | `qvh_Latn` | Huamalíes-Dos de Mayo Huánuco Quechua | 58 | 0.468 | 0.154 | 0.154 | 0.000 | 0.000 |
257
- | `zgh_Tfng` | Standard Moroccan Tamazight | 60 | 0.750 | 0.462 | 0.462 | 0.000 | 0.000 |
258
- | `lld_Latn` | Ladin | 59 | 0.909 | 0.731 | 0.731 | 0.000 | 0.000 |
259
- | `tah_Latn` | Tahitian | 57 | 0.973 | 0.812 | 0.812 | 0.966 | 0.966 |
260
- | `yue_Hani` | Cantonese | 49 | 0.734 | 0.580 | 0.580 | 0.111 | 0.111 |
261
- | `ote_Latn` | Mezquital Otomi | 59 | 0.154 | 0.000 | 0.000 | 0.000 | 0.000 |
262
- | `pnb_Arab` | Western Panjabi | 61 | 0.882 | 0.739 | 0.739 | 0.000 | 0.000 |
263
- | `mxv_Latn` | Metlatónoc Mixtec | 58 | 0.129 | 0.000 | 0.000 | 0.000 | 0.000 |
264
- | `lij_Latn` | Ligurian | 58 | 0.983 | 0.859 | 0.859 | 0.487 | 0.487 |
265
- | `bum_Latn` | Bulu | 58 | 0.215 | 0.098 | 0.098 | 0.000 | 0.000 |
266
 
267
  Where MillID loses most to GlotLID:
268
 
269
- | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
270
- |---|---|---|---|---|---|---|---|
271
- | `gyr_Latn` | Guarayu | 49 | 0.537 | 0.990 | 0.990 | 0.000 | 0.000 |
272
- | `shp_Latn` | Shipibo-Conibo | 62 | 0.469 | 0.912 | 0.912 | 0.000 | 0.000 |
273
- | `dyu_Latn` | Dyula | 58 | 0.000 | 0.434 | 0.434 | 0.015 | 0.015 |
274
- | `tbz_Latn` | Ditammari | 60 | 0.000 | 0.378 | 0.378 | 0.000 | 0.000 |
275
- | `cbu_Latn` | Candoshi-Shapra | 57 | 0.500 | 0.882 | 0.874 | 0.000 | 0.000 |
276
- | `idu_Latn` | Idoma | 51 | 0.603 | 0.879 | 0.938 | 0.000 | 0.000 |
277
- | `ell_Grek` | Greek | 115 | 0.670 | 1.000 | 1.000 | 1.000 | 1.000 |
278
- | `qvc_Latn` | Cajamarca Quechua | 58 | 0.648 | 0.906 | 0.906 | 0.000 | 0.000 |
279
- | `cpu_Latn` | Pichis Ashéninka | 60 | 0.729 | 0.974 | 0.974 | 0.000 | 0.000 |
280
- | `qvn_Latn` | North Junín Quechua | 57 | 0.219 | 0.460 | 0.460 | 0.000 | 0.000 |
281
- | `hin_Deva` | Hindi | 59 | 0.723 | 0.957 | 0.957 | 0.975 | 0.975 |
282
- | `hrv_Latn` | Croatian | 58 | 0.500 | 0.727 | 0.727 | 0.474 | 0.474 |
283
- | `kbp_Latn` | Kabiyè | 59 | 0.781 | 1.000 | 1.000 | 0.389 | 0.389 |
284
- | `cni_Latn` | Asháninka | 52 | 0.773 | 0.963 | 0.963 | 0.000 | 0.000 |
285
- | `rmn_Latn` | Balkan Romani | 116 | 0.678 | 0.857 | 0.857 | 0.000 | 0.000 |
286
-
287
- ### Leaderboard: test-common
288
-
289
- MillID's F1 against the others', language by language: against GlotLID restricted to MillID's labels, MillID is better on **6** languages, worse on **154** and within 0.01 on 53; against fastText restricted to MillID's labels, MillID is better on **85** languages, worse on **51** and within 0.01 on 77. The full table, with precision and recall, is `reports/test-common/leaderboard.csv`.
290
 
291
  The 20 languages with the most passages:
292
 
293
- | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
294
- |---|---|---|---|---|---|---|---|
295
- | `fra_Latn` | French | 1652 | 0.943 | 0.955 | 0.955 | 0.963 | 0.963 |
296
- | `eng_Latn` | English | 1444 | 0.932 | 0.876 | 0.877 | 0.857 | 0.857 |
297
- | `spa_Latn` | Spanish | 1136 | 0.937 | 0.958 | 0.958 | 0.941 | 0.941 |
298
- | `tam_Taml` | Tamil | 754 | 0.997 | 1.000 | 1.000 | 0.997 | 0.997 |
299
- | `arb_Arab` | Arabic (Modern Standard) | 745 | 0.871 | 0.886 | 0.886 | 0.957 | 0.954 |
300
- | `por_Latn` | Portuguese | 657 | 0.959 | 0.954 | 0.956 | 0.950 | 0.949 |
301
- | `kan_Knda` | Kannada | 601 | 0.973 | 0.994 | 0.994 | 0.999 | 0.999 |
302
- | `zul_Latn` | Zulu | 594 | 0.883 | 0.904 | 0.904 | 0.970 | 0.970 |
303
- | `swh_Latn` | Swahili | 592 | 0.893 | 0.954 | 0.953 | 0.945 | 0.945 |
304
- | `mar_Deva` | Marathi | 589 | 0.968 | 0.989 | 0.990 | 0.987 | 0.987 |
305
- | `tso_Latn` | Tsonga | 581 | 0.979 | 0.956 | 0.956 | 0.986 | 0.986 |
306
- | `hun_Latn` | Hungarian | 576 | 0.989 | 1.000 | 1.000 | 0.980 | 0.980 |
307
- | `afr_Latn` | Afrikaans | 569 | 0.972 | 0.985 | 0.985 | 0.986 | 0.986 |
308
- | `tel_Telu` | Telugu | 555 | 0.997 | 0.998 | 0.998 | 0.999 | 0.999 |
309
- | `rus_Cyrl` | Russian | 528 | 0.945 | 0.985 | 0.985 | 0.927 | 0.927 |
310
- | `ory_Orya` | Odia (individual language) | 511 | 0.994 | 0.997 | 1.000 | 0.996 | 0.996 |
311
- | `sot_Latn` | Southern Sotho | 491 | 0.969 | 0.977 | 0.978 | 0.856 | 0.856 |
312
- | `nso_Latn` | Northern Sotho | 479 | 0.958 | 0.970 | 0.970 | 0.967 | 0.967 |
313
- | `asm_Beng` | Assamese | 467 | 0.997 | 0.999 | 0.999 | 0.997 | 0.997 |
314
- | `ben_Beng` | Bangla | 466 | 0.988 | 0.997 | 0.997 | 0.989 | 0.989 |
315
 
316
  Where MillID gains most over GlotLID:
317
 
318
- | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
319
- |---|---|---|---|---|---|---|---|
320
- | `ewo_Latn` | Ewondo | 134 | 0.959 | 0.879 | 0.883 | 0.860 | 0.860 |
321
- | `eng_Latn` | English | 1444 | 0.932 | 0.876 | 0.877 | 0.857 | 0.857 |
322
- | `arb_Latn` | Standard Arabic | 163 | 0.942 | 0.906 | 0.913 | 0.852 | 0.852 |
323
- | `tso_Latn` | Tsonga | 581 | 0.979 | 0.956 | 0.956 | 0.986 | 0.986 |
324
- | `fuv_Latn` | Nigerian Fulfulde | 164 | 0.907 | 0.877 | 0.892 | 0.859 | 0.859 |
325
- | `knc_Latn` | Central Kanuri | 293 | 0.967 | 0.952 | 0.952 | 0.981 | 0.981 |
326
- | `quy_Latn` | Ayacucho Quechua | 271 | 0.940 | 0.925 | 0.933 | 0.979 | 0.979 |
327
- | `kam_Latn` | Kamba | 272 | 0.980 | 0.972 | 0.974 | 0.863 | 0.863 |
328
- | `bjn_Arab` | Banjar | 116 | 0.992 | 0.834 | 0.987 | 0.453 | 0.960 |
329
- | `por_Latn` | Portuguese | 657 | 0.959 | 0.954 | 0.956 | 0.950 | 0.949 |
330
- | `mlt_Latn` | Maltese | 291 | 0.995 | 0.993 | 0.993 | 0.961 | 0.961 |
331
- | `hye_Armn` | Armenian | 178 | 1.000 | 1.000 | 1.000 | 0.977 | 0.977 |
332
- | `khm_Khmr` | Khmer | 187 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
333
- | `guj_Gujr` | Gujarati | 354 | 1.000 | 1.000 | 1.000 | 0.984 | 0.984 |
334
- | `mal_Mlym` | Malayalam | 338 | 1.000 | 1.000 | 1.000 | 0.994 | 0.994 |
335
 
336
  Where MillID loses most to GlotLID:
337
 
338
- | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
339
- |---|---|---|---|---|---|---|---|
340
- | `knc_Arab` | Central Kanuri | 68 | 0.522 | 0.857 | 0.860 | 0.029 | 0.029 |
341
- | `hrv_Latn` | Croatian | 168 | 0.659 | 0.994 | 0.994 | 0.856 | 0.856 |
342
- | `bos_Latn` | Bosnian | 198 | 0.685 | 1.000 | 1.000 | 0.897 | 0.897 |
343
- | `tzm_Tfng` | Central Atlas Tamazight | 157 | 0.697 | 0.934 | 0.934 | 0.751 | 0.751 |
344
- | `zsm_Latn` | Standard Malay | 244 | 0.789 | 0.998 | 0.998 | 0.908 | 0.908 |
345
- | `taq_Tfng` | Tamasheq | 104 | 0.654 | 0.805 | 0.805 | 0.000 | 0.000 |
346
- | `ind_Latn` | Indonesian | 335 | 0.842 | 0.991 | 0.991 | 0.887 | 0.887 |
347
- | `kas_Deva` | Kashmiri | 47 | 0.557 | 0.703 | 0.703 | 0.587 | 0.587 |
348
- | `cjk_Latn` | Chokwe | 24 | 0.857 | 1.000 | 1.000 | 0.807 | 0.807 |
349
- | `nob_Latn` | Norwegian Bokmål | 265 | 0.821 | 0.961 | 0.959 | 0.894 | 0.894 |
350
- | `kin_Latn` | Kinyarwanda | 210 | 0.849 | 0.981 | 0.981 | 0.854 | 0.854 |
351
- | `mag_Deva` | Magahi | 97 | 0.890 | 1.000 | 1.000 | 0.772 | 0.772 |
352
- | `lim_Latn` | Limburgish | 196 | 0.896 | 1.000 | 1.000 | 0.956 | 0.956 |
353
- | `run_Latn` | Rundi | 291 | 0.883 | 0.985 | 0.985 | 0.855 | 0.855 |
354
- | `taq_Latn` | Tamasheq | 106 | 0.902 | 1.000 | 1.000 | 0.838 | 0.838 |
355
-
356
- Per-language precision, recall (sensitivity), F1 and specificity for every model are in `reports/<set>/`.
357
 
358
  ## Training
359
 
 
51
 
52
  ## The MillID models
53
 
54
+ Three sizes, this one in bold. Macro F1 on `test` and `udhr`, the sets below:
 
 
 
 
55
 
56
+ | Model | Parameters | ONNX | Test F1 | UDHR F1 | Runs on |
57
+ |---|---|---|---|---|---|
58
+ | **MillID tiny v1.5** | 2.9M | 11 MB | 0.9554 | 0.8764 | CPU, web page |
59
+ | [MillID medium v1](https://huggingface.co/baobabtech/millid-medium-v1) | 115M | 455 MB | 0.9576 | 0.8780 | GPU |
60
+ | [MillID large v1.5](https://huggingface.co/baobabtech/millid-large-v1.5) | 193M | 764 MB | 0.9646 | 0.8812 | GPU |
61
+
62
+ All three, language by language, against GlotLID and Meta's fastText: the [leaderboard](https://huggingface.co/baobabtech/millid). GlotLID, restricted to MillID's labels, scores 0.9755 and 0.8873.
63
 
64
  ## Use it
65
 
 
101
 
102
  ## Evaluation against GlotLID and fastText
103
 
104
+ `validation` chose the checkpoint; `test` is split by document from training; `udhr` is the Universal Declaration of Human Rights, labelled by its translators, over the languages MillID knows, and `udhr-all` over all of them. [GlotLID](https://huggingface.co/cis-lmu/glotlid) and Meta's [fastText language identifier](https://huggingface.co/facebook/fasttext-language-identification) (NLLB, 218 labels) are each restricted to MillID's labels, their best answer among them: the like-for-like comparison. fastText's answers are mapped onto MillID's spelling (Simplified and Traditional Han as `Hani`, `pes`/`prs` as `fas`, `est` as `ekk`). GlotLID's own labels came from training on data that includes UDHR and labelled FinePDFs and FineWeb-2, so its scores on `udhr` and on those sources are optimistic. fastText cannot answer most of the 1,600 languages here, so `test-common` and `udhr-common` score every system on the passages whose language all of them know: the head-to-head on equal terms.
105
+
106
+ Macro F1 on each set, every system on the same passages; GlotLID and fastText restricted to MillID's labels:
107
+
108
+ | Set | Languages | MillID | GlotLID | fastText |
109
+ |---|---|---|---|---|
110
+ | `test` | 1,621 | 0.9554 | 0.9755 | 0.0659 |
111
+ | `test-common` | 213 | 0.9513 | 0.9846 | 0.9173 |
112
+ | `udhr` | 358 | 0.8764 | 0.8873 | 0.3499 |
113
+ | `udhr-common` | 170 | 0.9513 | 0.9659 | 0.9329 |
114
+ | `udhr-all` | 429 | 0.7149 | 0.7245 | 0.2797 |
115
+
116
+ MillID on each set:
117
+
118
+ | Set | Top-1 | Top-3 | Top-5 | Macro P | Macro R | Macro F1 | Median F1 |
119
+ |---|---|---|---|---|---|---|---|
120
+ | `validation` | 0.9578 | 0.9889 | 0.9933 | 0.9539 | 0.9591 | 0.9551 | 0.9822 |
121
+ | `test` | 0.9587 | 0.9890 | 0.9932 | 0.9550 | 0.9588 | 0.9554 | 0.9825 |
122
+ | `test-common` | 0.9396 | 0.9857 | 0.9909 | 0.9709 | 0.9353 | 0.9513 | 0.9726 |
123
+ | `udhr` | 0.8492 | 0.9117 | 0.9257 | 0.9148 | 0.8736 | 0.8764 | 0.9907 |
124
+ | `udhr-common` | 0.9123 | 0.9606 | 0.9655 | 0.9794 | 0.9393 | 0.9513 | 1.0000 |
125
+ | `udhr-all` | 0.7274 | 0.7809 | 0.7928 | 0.7340 | 0.7290 | 0.7149 | 0.9649 |
 
 
 
 
 
 
 
 
126
 
127
  Top-1 accuracy by passage length:
128
 
129
+ | Set | System | Short | Sentence | Paragraph |
130
  |---|---|---|---|---|
131
+ | `validation` | MillID | 0.9312 | 0.9788 | 0.9850 |
132
+ | `test` | MillID | 0.9333 | 0.9797 | 0.9833 |
133
+ | `test` | GlotLID | 0.9674 | 0.9858 | 0.9912 |
134
+ | `test` | fastText | 0.2764 | 0.2506 | 0.1959 |
135
+ | `test-common` | MillID | 0.9125 | 0.9685 | 0.9725 |
136
+ | `test-common` | GlotLID | 0.9609 | 0.9885 | 0.9949 |
137
+ | `test-common` | fastText | 0.9125 | 0.9581 | 0.9655 |
138
+
139
+ ### Per language: test
140
+
141
+ F1 language by language, GlotLID and fastText restricted to MillID's labels: against GlotLID, MillID is better on **103** languages, worse on **869** and within 0.01 on 649; against fastText, MillID is better on **1616** languages, worse on **0** and within 0.01 on 5. Every language, with precision, recall and the unrestricted scores, is in `reports/test/leaderboard.csv`.
 
 
 
 
142
 
143
  The 20 languages with the most passages:
144
 
145
+ | Label | Language | Passages | MillID | GlotLID | fastText |
146
+ |---|---|---|---|---|---|
147
+ | `fra_Latn` | French | 1652 | 0.929 | 0.928 | 0.808 |
148
+ | `eng_Latn` | English | 1444 | 0.917 | 0.859 | 0.648 |
149
+ | `spa_Latn` | Spanish | 1136 | 0.930 | 0.951 | 0.574 |
150
+ | `tam_Taml` | Tamil | 754 | 0.997 | 0.993 | 0.965 |
151
+ | `arb_Arab` | Arabic (Modern Standard) | 745 | 0.814 | 0.785 | 0.512 |
152
+ | `por_Latn` | Portuguese | 657 | 0.947 | 0.949 | 0.755 |
153
+ | `kan_Knda` | Kannada | 601 | 0.969 | 0.993 | 0.922 |
154
+ | `zul_Latn` | Zulu | 594 | 0.846 | 0.896 | 0.539 |
155
+ | `swh_Latn` | Swahili | 592 | 0.848 | 0.916 | 0.346 |
156
+ | `mar_Deva` | Marathi | 589 | 0.965 | 0.990 | 0.926 |
157
+ | `tso_Latn` | Tsonga | 581 | 0.965 | 0.956 | 0.425 |
158
+ | `hun_Latn` | Hungarian | 576 | 0.987 | 0.999 | 0.232 |
159
+ | `afr_Latn` | Afrikaans | 569 | 0.967 | 0.985 | 0.860 |
160
+ | `tel_Telu` | Telugu | 555 | 0.991 | 0.994 | 0.951 |
161
+ | `rus_Cyrl` | Russian | 528 | 0.932 | 0.949 | 0.462 |
162
+ | `nbl_Latn` | South Ndebele | 528 | 0.903 | 0.949 | 0.000 |
163
+ | `ory_Orya` | Odia (individual language) | 511 | 0.988 | 0.994 | 0.976 |
164
+ | `sot_Latn` | Southern Sotho | 491 | 0.966 | 0.978 | 0.660 |
165
+ | `nso_Latn` | Northern Sotho | 479 | 0.956 | 0.969 | 0.914 |
166
+ | `kua_Latn` | Kuanyama | 476 | 0.962 | 0.986 | 0.000 |
167
 
168
  Where MillID gains most over GlotLID:
169
 
170
+ | Label | Language | Passages | MillID | GlotLID | fastText |
171
+ |---|---|---|---|---|---|
172
+ | `lbj_Tibt` | Ladakhi | 66 | 0.943 | 0.000 | 0.000 |
173
+ | `ain_Latn` | Ainu | 1 | 0.667 | 0.000 | 0.000 |
174
+ | `ajp_Arab` | South Levantine Arabic | 28 | 0.946 | 0.581 | 0.000 |
175
+ | `und_Laoo` | und | 20 | 0.923 | 0.621 | 0.000 |
176
+ | `und_Thaa` | und | 29 | 0.881 | 0.651 | 0.000 |
177
+ | `sgh_Cyrl` | Shughni | 9 | 0.857 | 0.667 | 0.000 |
178
+ | `und_Geor` | und | 32 | 0.877 | 0.694 | 0.000 |
179
+ | `tuk_Arab` | Turkmen | 135 | 0.905 | 0.761 | 0.000 |
180
+ | `und_Syrc` | und | 43 | 0.966 | 0.822 | 0.000 |
181
+ | `gmh_Latn` | Middle High German | 177 | 0.922 | 0.786 | 0.000 |
182
+ | `orv_Cyrl` | Old Russian | 239 | 0.931 | 0.800 | 0.000 |
183
+ | `und_Kana` | und | 136 | 0.982 | 0.862 | 0.000 |
184
+ | `und_Yiii` | und | 38 | 1.000 | 0.884 | 0.000 |
185
+ | `neb_Latn` | Toura (Côte d'Ivoire) | 11 | 0.957 | 0.842 | 0.000 |
186
+ | `dag_Latn` | Dagbani | 149 | 0.767 | 0.660 | 0.000 |
187
 
188
  Where MillID loses most to GlotLID:
189
 
190
+ | Label | Language | Passages | MillID | GlotLID | fastText |
191
+ |---|---|---|---|---|---|
192
+ | `mus_Latn` | Muscogee | 2 | 0.333 | 1.000 | 0.000 |
193
+ | `ttj_Latn` | Tooro | 28 | 0.485 | 0.946 | 0.000 |
194
+ | `mbd_Latn` | Dibabawon Manobo | 14 | 0.596 | 1.000 | 0.000 |
195
+ | `knc_Arab` | Central Kanuri | 68 | 0.462 | 0.825 | 0.007 |
196
+ | `biu_Latn` | Biete | 7 | 0.667 | 1.000 | 0.000 |
197
+ | `dks_Latn` | Southeastern Dinka | 3 | 0.667 | 1.000 | 0.000 |
198
+ | `esi_Latn` | North Alaskan Inupiatun | 9 | 0.455 | 0.783 | 0.000 |
199
+ | `hrv_Latn` | Croatian | 168 | 0.622 | 0.944 | 0.178 |
200
+ | `plw_Latn` | Brooke's Point Palawano | 8 | 0.727 | 1.000 | 0.000 |
201
+ | `bos_Latn` | Bosnian | 198 | 0.649 | 0.902 | 0.181 |
202
+ | `xbr_Latn` | Kambera | 8 | 0.750 | 1.000 | 0.000 |
203
+ | `nyf_Latn` | Giryama | 10 | 0.645 | 0.870 | 0.000 |
204
+ | `ldi_Latn` | Laari | 62 | 0.777 | 0.992 | 0.000 |
205
+ | `hak_Hani` | Hakka Chinese | 29 | 0.621 | 0.824 | 0.000 |
206
+ | `ccp_Latn` | Chakma | 3 | 0.800 | 1.000 | 0.000 |
207
+
208
+ ### Per language: udhr
209
+
210
+ F1 language by language, GlotLID and fastText restricted to MillID's labels: against GlotLID, MillID is better on **52** languages, worse on **105** and within 0.01 on 201; against fastText, MillID is better on **303** languages, worse on **15** and within 0.01 on 40. Every language, with precision, recall and the unrestricted scores, is in `reports/udhr/leaderboard.csv`.
211
 
212
  The 20 languages with the most passages:
213
 
214
+ | Label | Language | Passages | MillID | GlotLID | fastText |
215
+ |---|---|---|---|---|---|
216
+ | `roh_Latn` | Romansh | 399 | 0.994 | 0.999 | 0.946 |
217
+ | `oci_Latn` | Occitan | 398 | 0.277 | 0.376 | 0.435 |
218
+ | `cmn_Hani` | Chinese (Mandarin) | 319 | 0.892 | 0.868 | 0.468 |
219
+ | `krl_Latn` | Karelian | 183 | 0.000 | 0.011 | 0.000 |
220
+ | `qug_Latn` | Chimborazo Highland Quichua | 182 | 0.011 | 0.000 | 0.000 |
221
+ | `acu_Latn` | Achuar-Shiwiar | 180 | 0.828 | 0.854 | 0.000 |
222
+ | `hus_Latn` | Huastec | 175 | 0.977 | 0.986 | 0.000 |
223
+ | `hau_Latn` | Hausa | 155 | 0.990 | 1.000 | 0.876 |
224
+ | `auc_Latn` | Waorani | 154 | 0.000 | 0.000 | 0.000 |
225
+ | `kwi_Latn` | Awa-Cuaiquer | 145 | 0.880 | 0.877 | 0.000 |
226
+ | `jpn_Jpan` | Japanese | 140 | 1.000 | 1.000 | 0.944 |
227
+ | `snn_Latn` | Siona | 136 | 0.630 | 0.514 | 0.000 |
228
+ | `ktu_Latn` | Kituba (Democratic Republic of Congo) | 118 | 0.765 | 0.680 | 0.000 |
229
+ | `lat_Latn` | Latin | 118 | 0.996 | 0.992 | 0.000 |
230
+ | `hat_Latn` | Haitian Creole | 117 | 0.992 | 0.979 | 0.699 |
231
+ | `nya_Latn` | Nyanja | 116 | 0.991 | 1.000 | 0.806 |
232
+ | `kmb_Latn` | Kimbundu | 116 | 0.970 | 1.000 | 0.877 |
233
+ | `rmn_Latn` | Balkan Romani | 116 | 0.678 | 0.857 | 0.000 |
234
+ | `twi_Latn` | Twi | 116 | 0.983 | 0.996 | 0.396 |
235
+ | `chr_Cher` | Cherokee | 115 | 0.670 | 0.663 | 0.000 |
236
 
237
  Where MillID gains most over GlotLID:
238
 
239
+ | Label | Language | Passages | MillID | GlotLID | fastText |
240
+ |---|---|---|---|---|---|
241
+ | `mic_Latn` | Mi'kmaw | 59 | 0.992 | 0.427 | 0.000 |
242
+ | `tet_Latn` | Tetum | 58 | 0.638 | 0.167 | 0.000 |
243
+ | `fuf_Latn` | Pular | 58 | 0.906 | 0.483 | 0.000 |
244
+ | `sag_Latn` | Sango | 59 | 0.992 | 0.578 | 0.189 |
245
+ | `skr_Arab` | Saraiki | 63 | 0.867 | 0.500 | 0.000 |
246
+ | `qvh_Latn` | Huamalíes-Dos de Mayo Huánuco Quechua | 58 | 0.468 | 0.154 | 0.000 |
247
+ | `zgh_Tfng` | Standard Moroccan Tamazight | 60 | 0.750 | 0.462 | 0.000 |
248
+ | `lld_Latn` | Ladin | 59 | 0.909 | 0.731 | 0.000 |
249
+ | `tah_Latn` | Tahitian | 57 | 0.973 | 0.812 | 0.966 |
250
+ | `yue_Hani` | Cantonese | 49 | 0.734 | 0.580 | 0.111 |
251
+ | `ote_Latn` | Mezquital Otomi | 59 | 0.154 | 0.000 | 0.000 |
252
+ | `pnb_Arab` | Western Panjabi | 61 | 0.882 | 0.739 | 0.000 |
253
+ | `mxv_Latn` | Metlatónoc Mixtec | 58 | 0.129 | 0.000 | 0.000 |
254
+ | `lij_Latn` | Ligurian | 58 | 0.983 | 0.859 | 0.487 |
255
+ | `bum_Latn` | Bulu | 58 | 0.215 | 0.098 | 0.000 |
256
 
257
  Where MillID loses most to GlotLID:
258
 
259
+ | Label | Language | Passages | MillID | GlotLID | fastText |
260
+ |---|---|---|---|---|---|
261
+ | `gyr_Latn` | Guarayu | 49 | 0.537 | 0.990 | 0.000 |
262
+ | `shp_Latn` | Shipibo-Conibo | 62 | 0.469 | 0.912 | 0.000 |
263
+ | `dyu_Latn` | Dyula | 58 | 0.000 | 0.434 | 0.015 |
264
+ | `tbz_Latn` | Ditammari | 60 | 0.000 | 0.378 | 0.000 |
265
+ | `cbu_Latn` | Candoshi-Shapra | 57 | 0.500 | 0.874 | 0.000 |
266
+ | `idu_Latn` | Idoma | 51 | 0.603 | 0.938 | 0.000 |
267
+ | `ell_Grek` | Greek | 115 | 0.670 | 1.000 | 1.000 |
268
+ | `qvc_Latn` | Cajamarca Quechua | 58 | 0.648 | 0.906 | 0.000 |
269
+ | `cpu_Latn` | Pichis Ashéninka | 60 | 0.729 | 0.974 | 0.000 |
270
+ | `qvn_Latn` | North Junín Quechua | 57 | 0.219 | 0.460 | 0.000 |
271
+ | `hin_Deva` | Hindi | 59 | 0.723 | 0.957 | 0.975 |
272
+ | `hrv_Latn` | Croatian | 58 | 0.500 | 0.727 | 0.474 |
273
+ | `kbp_Latn` | Kabiyè | 59 | 0.781 | 1.000 | 0.389 |
274
+ | `cni_Latn` | Asháninka | 52 | 0.773 | 0.963 | 0.000 |
275
+ | `rmn_Latn` | Balkan Romani | 116 | 0.678 | 0.857 | 0.000 |
276
+
277
+ ### Per language: test-common
278
+
279
+ F1 language by language, GlotLID and fastText restricted to MillID's labels: against GlotLID, MillID is better on **6** languages, worse on **154** and within 0.01 on 53; against fastText, MillID is better on **85** languages, worse on **51** and within 0.01 on 77. Every language, with precision, recall and the unrestricted scores, is in `reports/test-common/leaderboard.csv`.
280
 
281
  The 20 languages with the most passages:
282
 
283
+ | Label | Language | Passages | MillID | GlotLID | fastText |
284
+ |---|---|---|---|---|---|
285
+ | `fra_Latn` | French | 1652 | 0.943 | 0.955 | 0.963 |
286
+ | `eng_Latn` | English | 1444 | 0.932 | 0.877 | 0.857 |
287
+ | `spa_Latn` | Spanish | 1136 | 0.937 | 0.958 | 0.941 |
288
+ | `tam_Taml` | Tamil | 754 | 0.997 | 1.000 | 0.997 |
289
+ | `arb_Arab` | Arabic (Modern Standard) | 745 | 0.871 | 0.886 | 0.954 |
290
+ | `por_Latn` | Portuguese | 657 | 0.959 | 0.956 | 0.949 |
291
+ | `kan_Knda` | Kannada | 601 | 0.973 | 0.994 | 0.999 |
292
+ | `zul_Latn` | Zulu | 594 | 0.883 | 0.904 | 0.970 |
293
+ | `swh_Latn` | Swahili | 592 | 0.893 | 0.953 | 0.945 |
294
+ | `mar_Deva` | Marathi | 589 | 0.968 | 0.990 | 0.987 |
295
+ | `tso_Latn` | Tsonga | 581 | 0.979 | 0.956 | 0.986 |
296
+ | `hun_Latn` | Hungarian | 576 | 0.989 | 1.000 | 0.980 |
297
+ | `afr_Latn` | Afrikaans | 569 | 0.972 | 0.985 | 0.986 |
298
+ | `tel_Telu` | Telugu | 555 | 0.997 | 0.998 | 0.999 |
299
+ | `rus_Cyrl` | Russian | 528 | 0.945 | 0.985 | 0.927 |
300
+ | `ory_Orya` | Odia (individual language) | 511 | 0.994 | 1.000 | 0.996 |
301
+ | `sot_Latn` | Southern Sotho | 491 | 0.969 | 0.978 | 0.856 |
302
+ | `nso_Latn` | Northern Sotho | 479 | 0.958 | 0.970 | 0.967 |
303
+ | `asm_Beng` | Assamese | 467 | 0.997 | 0.999 | 0.997 |
304
+ | `ben_Beng` | Bangla | 466 | 0.988 | 0.997 | 0.989 |
305
 
306
  Where MillID gains most over GlotLID:
307
 
308
+ | Label | Language | Passages | MillID | GlotLID | fastText |
309
+ |---|---|---|---|---|---|
310
+ | `ewo_Latn` | Ewondo | 134 | 0.959 | 0.883 | 0.860 |
311
+ | `eng_Latn` | English | 1444 | 0.932 | 0.877 | 0.857 |
312
+ | `arb_Latn` | Standard Arabic | 163 | 0.942 | 0.913 | 0.852 |
313
+ | `tso_Latn` | Tsonga | 581 | 0.979 | 0.956 | 0.986 |
314
+ | `fuv_Latn` | Nigerian Fulfulde | 164 | 0.907 | 0.892 | 0.859 |
315
+ | `knc_Latn` | Central Kanuri | 293 | 0.967 | 0.952 | 0.981 |
316
+ | `quy_Latn` | Ayacucho Quechua | 271 | 0.940 | 0.933 | 0.979 |
317
+ | `kam_Latn` | Kamba | 272 | 0.980 | 0.974 | 0.863 |
318
+ | `bjn_Arab` | Banjar | 116 | 0.992 | 0.987 | 0.960 |
319
+ | `por_Latn` | Portuguese | 657 | 0.959 | 0.956 | 0.949 |
320
+ | `mlt_Latn` | Maltese | 291 | 0.995 | 0.993 | 0.961 |
321
+ | `hye_Armn` | Armenian | 178 | 1.000 | 1.000 | 0.977 |
322
+ | `khm_Khmr` | Khmer | 187 | 1.000 | 1.000 | 1.000 |
323
+ | `guj_Gujr` | Gujarati | 354 | 1.000 | 1.000 | 0.984 |
324
+ | `mal_Mlym` | Malayalam | 338 | 1.000 | 1.000 | 0.994 |
325
 
326
  Where MillID loses most to GlotLID:
327
 
328
+ | Label | Language | Passages | MillID | GlotLID | fastText |
329
+ |---|---|---|---|---|---|
330
+ | `knc_Arab` | Central Kanuri | 68 | 0.522 | 0.860 | 0.029 |
331
+ | `hrv_Latn` | Croatian | 168 | 0.659 | 0.994 | 0.856 |
332
+ | `bos_Latn` | Bosnian | 198 | 0.685 | 1.000 | 0.897 |
333
+ | `tzm_Tfng` | Central Atlas Tamazight | 157 | 0.697 | 0.934 | 0.751 |
334
+ | `zsm_Latn` | Standard Malay | 244 | 0.789 | 0.998 | 0.908 |
335
+ | `taq_Tfng` | Tamasheq | 104 | 0.654 | 0.805 | 0.000 |
336
+ | `ind_Latn` | Indonesian | 335 | 0.842 | 0.991 | 0.887 |
337
+ | `kas_Deva` | Kashmiri | 47 | 0.557 | 0.703 | 0.587 |
338
+ | `cjk_Latn` | Chokwe | 24 | 0.857 | 1.000 | 0.807 |
339
+ | `nob_Latn` | Norwegian Bokmål | 265 | 0.821 | 0.959 | 0.894 |
340
+ | `kin_Latn` | Kinyarwanda | 210 | 0.849 | 0.981 | 0.854 |
341
+ | `mag_Deva` | Magahi | 97 | 0.890 | 1.000 | 0.772 |
342
+ | `lim_Latn` | Limburgish | 196 | 0.896 | 1.000 | 0.956 |
343
+ | `run_Latn` | Rundi | 291 | 0.883 | 0.985 | 0.855 |
344
+ | `taq_Latn` | Tamasheq | 106 | 0.902 | 1.000 | 0.838 |
345
+
346
+ Per-language precision, recall (sensitivity), F1 and specificity for every system, restricted and as it is, are in `reports/<set>/`, and the totals in `reports/summary.json`.
347
 
348
  ## Training
349