README.md

---
license: apache-2.0
library_name: pytorch
pipeline_tag: text-classification
tags:
- language-identification
- leaderboard
datasets:
- baobabtech/millid-dataset
---

# MillID leaderboard

**MillID** (MultiLingual Language IDentifier) against [GlotLID](https://huggingface.co/cis-lmu/glotlid) and Meta's [fastText language identifier](https://huggingface.co/facebook/fasttext-language-identification) (NLLB). F1 is per language, then averaged with every language weighted equally (macro).

The models: [MillID tiny v1.5](https://huggingface.co/baobabtech/millid-tiny-v1.5), [MillID medium v1](https://huggingface.co/baobabtech/millid-medium-v1), [MillID large v1.5](https://huggingface.co/baobabtech/millid-large-v1.5). Code, and running them in Python or JavaScript without a server: [baobab-tech/millid](https://github.com/baobab-tech/millid).

Every score is on the test split of the build the models were trained on. The public [baobabtech/millid-dataset](https://huggingface.co/datasets/baobabtech/millid-dataset) is that build without Save the Children's and Nal'ibali's text, which is not ours to redistribute: the same splits, and a `test` split 1.3% smaller. `udhr` is the same everywhere.

- MillID knows 1,641 labels, GlotLID over 2,000, fastText 218. Each external model is shown *restricted to MillID's labels* (its best answer among them), the like-for-like comparison; its as-is scores are below.
- On `test` and `udhr`, fastText is mostly scored on languages it cannot answer. `test-common` and `udhr-common` keep the passages whose language all three know (about 210 languages): the head-to-head on equal terms.
- Read GlotLID's scores with their bias in mind: the dataset's FinePDFs and FineWeb-2 passages were labelled by GlotLID, and GlotLID was trained on UDHR, so both favour it.
- fastText's labels are mapped onto MillID's spelling: Simplified and Traditional Han as `Hani`, `pes`/`prs` as `fas`, `est` as `ekk`, `tgl` as `fil`.

## Macro F1, every set

| Set | Languages | MillID tiny v1.5 | MillID medium v1 | MillID large v1.5 | GlotLID | fastText |
|---|---|---|---|---|---|---|
| `test`: every language, passages from documents never seen in training | 1621 | 0.9554 | 0.9576 | 0.9646 | **0.9755** | 0.0659 |
| `test-common`: the languages all three systems know | 213 | 0.9513 | 0.9527 | 0.9594 | **0.9846** | 0.9173 |
| `udhr`: the Declaration of Human Rights, languages MillID knows | 358 | 0.8764 | 0.8780 | 0.8812 | **0.8873** | 0.3499 |
| `udhr-common`: the Declaration, languages all three know | 170 | 0.9513 | 0.9480 | 0.9529 | **0.9659** | 0.9329 |
| `udhr-all`: the Declaration, every language in it | 429 | 0.7149 | 0.7184 | 0.7201 | **0.7245** | 0.2797 |

The best score on each set is in bold.

## test

Every language, passages from documents never seen in training.

| Model | Size | Top-1 | Top-3 | Top-5 | Macro P | Macro R | Macro F1 | Median F1 | Languages F1 ≥ 0.9 |
|---|---|---|---|---|---|---|---|---|---|
| [GlotLID, restricted to MillID's labels](https://huggingface.co/cis-lmu/glotlid) | fastText, 1.7 GB | 0.9783 | 0.9978 | 0.9985 | 0.9778 | 0.9765 | 0.9755 | 1.0000 | 1490 of 1621 |
| [GlotLID](https://huggingface.co/cis-lmu/glotlid) | fastText, 1.7 GB | 0.9776 | 0.9977 | 0.9984 | 0.9783 | 0.9752 | 0.9748 | 1.0000 | 1488 of 1621 |
| [MillID large v1.5](https://huggingface.co/baobabtech/millid-large-v1.5) | 192.6M | 0.9674 | 0.9924 | 0.9956 | 0.9655 | 0.9663 | 0.9646 | 0.9894 | 1478 of 1621 |
| [MillID medium v1](https://huggingface.co/baobabtech/millid-medium-v1) | 115.0M | 0.9616 | 0.9910 | 0.9945 | 0.9564 | 0.9614 | 0.9576 | 0.9841 | 1451 of 1621 |
| [MillID tiny v1.5](https://huggingface.co/baobabtech/millid-tiny-v1.5) | 2.9M | 0.9587 | 0.9890 | 0.9932 | 0.9550 | 0.9588 | 0.9554 | 0.9825 | 1446 of 1621 |
| [fastText, restricted to MillID's labels](https://huggingface.co/facebook/fasttext-language-identification) | fastText, 1.2 GB | 0.2515 | 0.2611 | 0.2625 | 0.0521 | 0.1209 | 0.0659 | 0.0000 | 26 of 1621 |
| [fastText (Meta, NLLB)](https://huggingface.co/facebook/fasttext-language-identification) | fastText, 1.2 GB | 0.2511 | 0.2610 | 0.2625 | 0.0520 | 0.1205 | 0.0657 | 0.0000 | 25 of 1621 |

## test-common

The languages all three systems know.

| Model | Size | Top-1 | Top-3 | Top-5 | Macro P | Macro R | Macro F1 | Median F1 | Languages F1 ≥ 0.9 |
|---|---|---|---|---|---|---|---|---|---|
| [GlotLID, restricted to MillID's labels](https://huggingface.co/cis-lmu/glotlid) | fastText, 1.7 GB | 0.9749 | 0.9927 | 0.9948 | 0.9923 | 0.9790 | 0.9846 | 0.9983 | 203 of 213 |
| [GlotLID](https://huggingface.co/cis-lmu/glotlid) | fastText, 1.7 GB | 0.9738 | 0.9925 | 0.9946 | 0.9924 | 0.9771 | 0.9835 | 0.9983 | 201 of 213 |
| [MillID large v1.5](https://huggingface.co/baobabtech/millid-large-v1.5) | 192.6M | 0.9525 | 0.9898 | 0.9938 | 0.9757 | 0.9460 | 0.9594 | 0.9791 | 193 of 213 |
| [MillID medium v1](https://huggingface.co/baobabtech/millid-medium-v1) | 115.0M | 0.9425 | 0.9882 | 0.9924 | 0.9732 | 0.9361 | 0.9527 | 0.9738 | 191 of 213 |
| [MillID tiny v1.5](https://huggingface.co/baobabtech/millid-tiny-v1.5) | 2.9M | 0.9396 | 0.9857 | 0.9909 | 0.9709 | 0.9353 | 0.9513 | 0.9726 | 188 of 213 |
| [fastText, restricted to MillID's labels](https://huggingface.co/facebook/fasttext-language-identification) | fastText, 1.2 GB | 0.9352 | 0.9710 | 0.9763 | 0.9354 | 0.9201 | 0.9173 | 0.9680 | 166 of 213 |
| [fastText (Meta, NLLB)](https://huggingface.co/facebook/fasttext-language-identification) | fastText, 1.2 GB | 0.9339 | 0.9707 | 0.9762 | 0.9357 | 0.9169 | 0.9149 | 0.9680 | 165 of 213 |

## udhr

The Declaration of Human Rights, languages MillID knows.

| Model | Size | Top-1 | Top-3 | Top-5 | Macro P | Macro R | Macro F1 | Median F1 | Languages F1 ≥ 0.9 |
|---|---|---|---|---|---|---|---|---|---|
| [GlotLID, restricted to MillID's labels](https://huggingface.co/cis-lmu/glotlid) | fastText, 1.7 GB | 0.8614 | 0.9294 | 0.9433 | 0.9258 | 0.8844 | 0.8873 | 0.9915 | 291 of 358 |
| [GlotLID](https://huggingface.co/cis-lmu/glotlid) | fastText, 1.7 GB | 0.8598 | 0.9280 | 0.9421 | 0.9261 | 0.8826 | 0.8865 | 0.9915 | 287 of 358 |
| [MillID large v1.5](https://huggingface.co/baobabtech/millid-large-v1.5) | 192.6M | 0.8579 | 0.9183 | 0.9309 | 0.9171 | 0.8825 | 0.8812 | 0.9914 | 283 of 358 |
| [MillID medium v1](https://huggingface.co/baobabtech/millid-medium-v1) | 115.0M | 0.8524 | 0.9194 | 0.9340 | 0.9187 | 0.8769 | 0.8780 | 0.9910 | 282 of 358 |
| [MillID tiny v1.5](https://huggingface.co/baobabtech/millid-tiny-v1.5) | 2.9M | 0.8492 | 0.9117 | 0.9257 | 0.9148 | 0.8736 | 0.8764 | 0.9907 | 274 of 358 |
| [fastText, restricted to MillID's labels](https://huggingface.co/facebook/fasttext-language-identification) | fastText, 1.2 GB | 0.4629 | 0.4857 | 0.4904 | 0.3152 | 0.4483 | 0.3499 | 0.0000 | 61 of 358 |
| [fastText (Meta, NLLB)](https://huggingface.co/facebook/fasttext-language-identification) | fastText, 1.2 GB | 0.4625 | 0.4857 | 0.4903 | 0.3152 | 0.4479 | 0.3497 | 0.0000 | 60 of 358 |

## udhr-common

The Declaration, languages all three know.

| Model | Size | Top-1 | Top-3 | Top-5 | Macro P | Macro R | Macro F1 | Median F1 | Languages F1 ≥ 0.9 |
|---|---|---|---|---|---|---|---|---|---|
| [GlotLID, restricted to MillID's labels](https://huggingface.co/cis-lmu/glotlid) | fastText, 1.7 GB | 0.9338 | 0.9683 | 0.9736 | 0.9888 | 0.9579 | 0.9659 | 1.0000 | 156 of 170 |
| [GlotLID](https://huggingface.co/cis-lmu/glotlid) | fastText, 1.7 GB | 0.9336 | 0.9678 | 0.9732 | 0.9888 | 0.9576 | 0.9657 | 1.0000 | 155 of 170 |
| [MillID large v1.5](https://huggingface.co/baobabtech/millid-large-v1.5) | 192.6M | 0.9168 | 0.9630 | 0.9687 | 0.9800 | 0.9436 | 0.9529 | 1.0000 | 152 of 170 |
| [MillID tiny v1.5](https://huggingface.co/baobabtech/millid-tiny-v1.5) | 2.9M | 0.9123 | 0.9606 | 0.9655 | 0.9794 | 0.9393 | 0.9513 | 1.0000 | 149 of 170 |
| [MillID medium v1](https://huggingface.co/baobabtech/millid-medium-v1) | 115.0M | 0.9121 | 0.9633 | 0.9688 | 0.9770 | 0.9382 | 0.9480 | 0.9955 | 153 of 170 |
| [fastText, restricted to MillID's labels](https://huggingface.co/facebook/fasttext-language-identification) | fastText, 1.2 GB | 0.9195 | 0.9647 | 0.9742 | 0.9474 | 0.9441 | 0.9329 | 1.0000 | 142 of 170 |
| [fastText (Meta, NLLB)](https://huggingface.co/facebook/fasttext-language-identification) | fastText, 1.2 GB | 0.9187 | 0.9647 | 0.9741 | 0.9474 | 0.9431 | 0.9323 | 1.0000 | 141 of 170 |

## udhr-all

The Declaration, every language in it.

| Model | Size | Top-1 | Top-3 | Top-5 | Macro P | Macro R | Macro F1 | Median F1 | Languages F1 ≥ 0.9 |
|---|---|---|---|---|---|---|---|---|---|
| [GlotLID](https://huggingface.co/cis-lmu/glotlid) | fastText, 1.7 GB | 0.7502 | 0.8145 | 0.8282 | 0.7695 | 0.7529 | 0.7454 | 0.9796 | 277 of 429 |
| [GlotLID, restricted to MillID's labels](https://huggingface.co/cis-lmu/glotlid) | fastText, 1.7 GB | 0.7378 | 0.7960 | 0.8079 | 0.7425 | 0.7381 | 0.7245 | 0.9748 | 272 of 429 |
| [MillID large v1.5](https://huggingface.co/baobabtech/millid-large-v1.5) | 192.6M | 0.7348 | 0.7865 | 0.7973 | 0.7392 | 0.7364 | 0.7201 | 0.9643 | 266 of 429 |
| [MillID medium v1](https://huggingface.co/baobabtech/millid-medium-v1) | 115.0M | 0.7301 | 0.7875 | 0.8000 | 0.7415 | 0.7318 | 0.7184 | 0.9672 | 266 of 429 |
| [MillID tiny v1.5](https://huggingface.co/baobabtech/millid-tiny-v1.5) | 2.9M | 0.7274 | 0.7809 | 0.7928 | 0.7340 | 0.7290 | 0.7149 | 0.9649 | 254 of 429 |
| [fastText, restricted to MillID's labels](https://huggingface.co/facebook/fasttext-language-identification) | fastText, 1.2 GB | 0.3964 | 0.4160 | 0.4200 | 0.2479 | 0.3741 | 0.2797 | 0.0000 | 55 of 429 |
| [fastText (Meta, NLLB)](https://huggingface.co/facebook/fasttext-language-identification) | fastText, 1.2 GB | 0.3961 | 0.4160 | 0.4200 | 0.2479 | 0.3737 | 0.2795 | 0.0000 | 54 of 429 |

## Per language, test

Against GlotLID restricted, the best MillID is better on **165** languages and worse on **498**; against fastText restricted, the best MillID is better on **1616** languages and worse on **0** (F1 differing by more than 0.01), of 1621. Every language is in `leaderboard_test.csv`.

The 30 languages with the most passages:

| Label | Language | Passages | GlotLID | GlotLID restricted | fastText | fastText restricted | MillID tiny v1.5 | MillID medium v1 | MillID large v1.5 |
|---|---|---|---|---|---|---|---|---|---|
| `fra_Latn` | French | 1652 | 0.928 | 0.928 | 0.808 | 0.808 | 0.929 | 0.952 | 0.960 |
| `eng_Latn` | English | 1444 | 0.858 | 0.859 | 0.653 | 0.648 | 0.917 | 0.916 | 0.934 |
| `spa_Latn` | Spanish | 1136 | 0.951 | 0.951 | 0.574 | 0.574 | 0.930 | 0.942 | 0.955 |
| `tam_Taml` | Tamil | 754 | 0.993 | 0.993 | 0.965 | 0.965 | 0.997 | 0.997 | 0.999 |
| `arb_Arab` | Arabic (Modern Standard) | 745 | 0.785 | 0.785 | 0.513 | 0.512 | 0.814 | 0.829 | 0.833 |
| `por_Latn` | Portuguese | 657 | 0.947 | 0.949 | 0.756 | 0.755 | 0.947 | 0.955 | 0.965 |
| `kan_Knda` | Kannada | 601 | 0.993 | 0.993 | 0.922 | 0.922 | 0.969 | 0.982 | 0.965 |
| `zul_Latn` | Zulu | 594 | 0.896 | 0.896 | 0.539 | 0.539 | 0.846 | 0.833 | 0.878 |
| `swh_Latn` | Swahili | 592 | 0.917 | 0.916 | 0.346 | 0.346 | 0.848 | 0.858 | 0.870 |
| `mar_Deva` | Marathi | 589 | 0.989 | 0.990 | 0.926 | 0.926 | 0.965 | 0.981 | 0.976 |
| `tso_Latn` | Tsonga | 581 | 0.956 | 0.956 | 0.425 | 0.425 | 0.965 | 0.970 | 0.978 |
| `hun_Latn` | Hungarian | 576 | 0.999 | 0.999 | 0.232 | 0.232 | 0.987 | 0.990 | 0.991 |
| `afr_Latn` | Afrikaans | 569 | 0.985 | 0.985 | 0.861 | 0.860 | 0.967 | 0.974 | 0.979 |
| `tel_Telu` | Telugu | 555 | 0.994 | 0.994 | 0.951 | 0.951 | 0.991 | 0.994 | 0.988 |
| `rus_Cyrl` | Russian | 528 | 0.949 | 0.949 | 0.462 | 0.462 | 0.932 | 0.925 | 0.949 |
| `nbl_Latn` | South Ndebele | 528 | 0.948 | 0.949 | 0.000 | 0.000 | 0.903 | 0.895 | 0.927 |
| `ory_Orya` | Odia (individual language) | 511 | 0.991 | 0.994 | 0.976 | 0.976 | 0.988 | 0.993 | 0.993 |
| `sot_Latn` | Southern Sotho | 491 | 0.977 | 0.978 | 0.660 | 0.660 | 0.966 | 0.966 | 0.972 |
| `nso_Latn` | Northern Sotho | 479 | 0.969 | 0.969 | 0.915 | 0.914 | 0.956 | 0.960 | 0.962 |
| `kua_Latn` | Kuanyama | 476 | 0.986 | 0.986 | 0.000 | 0.000 | 0.962 | 0.979 | 0.975 |
| `asm_Beng` | Assamese | 467 | 0.999 | 0.999 | 0.990 | 0.990 | 0.997 | 0.994 | 0.996 |
| `ben_Beng` | Bangla | 466 | 0.997 | 0.997 | 0.718 | 0.718 | 0.977 | 0.980 | 0.979 |
| `pan_Guru` | Punjabi | 460 | 1.000 | 1.000 | 0.990 | 0.990 | 1.000 | 1.000 | 1.000 |
| `amh_Ethi` | Amharic | 458 | 1.000 | 1.000 | 0.807 | 0.807 | 0.981 | 0.984 | 0.986 |
| `tsn_Latn` | Tswana | 456 | 0.961 | 0.961 | 0.652 | 0.652 | 0.945 | 0.939 | 0.947 |
| `vro_Latn` | Võro | 448 | 0.990 | 0.990 | 0.000 | 0.000 | 0.947 | 0.958 | 0.973 |
| `gaz_Latn` | Oromo (West Central) | 441 | 0.938 | 0.938 | 0.407 | 0.407 | 0.910 | 0.943 | 0.928 |
| `som_Latn` | Somali | 439 | 0.994 | 0.994 | 0.487 | 0.487 | 0.982 | 0.988 | 0.990 |
| `uzn_Latn` | Northern Uzbek | 436 | 0.994 | 0.994 | 0.449 | 0.449 | 0.976 | 0.978 | 0.982 |
| `tha_Thai` | Thai | 431 | 0.980 | 0.980 | 0.650 | 0.650 | 0.979 | 0.970 | 0.981 |

Where MillID gains most over GlotLID:

| Label | Language | Passages | GlotLID | GlotLID restricted | fastText | fastText restricted | MillID tiny v1.5 | MillID medium v1 | MillID large v1.5 |
|---|---|---|---|---|---|---|---|---|---|
| `lbj_Tibt` | Ladakhi | 66 | 0.000 | 0.000 | 0.000 | 0.000 | 0.943 | 0.943 | 0.923 |
| `ain_Latn` | Ainu | 1 | 0.000 | 0.000 | 0.000 | 0.000 | 0.667 | 0.667 | 0.667 |
| `ajp_Arab` | South Levantine Arabic | 28 | 0.581 | 0.581 | 0.000 | 0.000 | 0.946 | 0.902 | 0.902 |
| `und_Laoo` | und | 20 | 0.621 | 0.621 | 0.000 | 0.000 | 0.923 | 0.857 | 0.788 |
| `sgh_Cyrl` | Shughni | 9 | 0.667 | 0.667 | 0.000 | 0.000 | 0.857 | 0.900 | 0.889 |
| `und_Thaa` | und | 29 | 0.651 | 0.651 | 0.000 | 0.000 | 0.881 | 0.873 | 0.863 |
| `icr_Latn` | Islander Creole English | 24 | 0.649 | 0.684 | 0.000 | 0.000 | 0.698 | 0.870 | 0.700 |
| `und_Geor` | und | 32 | 0.694 | 0.694 | 0.000 | 0.000 | 0.877 | 0.706 | 0.792 |
| `tdt_Latn` | Tetun Dili | 57 | 0.603 | 0.603 | 0.000 | 0.000 | 0.693 | 0.656 | 0.782 |
| `cek_Latn` | Eastern Khumi Chin | 67 | 0.680 | 0.684 | 0.000 | 0.000 | 0.785 | 0.853 | 0.825 |
| `und_Syrc` | und | 43 | 0.822 | 0.822 | 0.000 | 0.000 | 0.966 | 0.977 | 0.989 |
| `neb_Latn` | Toura (Côte d'Ivoire) | 11 | 0.625 | 0.842 | 0.000 | 0.000 | 0.957 | 0.957 | 1.000 |
| `dag_Latn` | Dagbani | 149 | 0.662 | 0.660 | 0.000 | 0.000 | 0.767 | 0.817 | 0.814 |
| `tuk_Arab` | Turkmen | 135 | 0.761 | 0.761 | 0.000 | 0.000 | 0.905 | 0.903 | 0.917 |
| `orv_Cyrl` | Old Russian | 239 | 0.800 | 0.800 | 0.000 | 0.000 | 0.931 | 0.935 | 0.938 |

Where MillID loses most to GlotLID:

| Label | Language | Passages | GlotLID | GlotLID restricted | fastText | fastText restricted | MillID tiny v1.5 | MillID medium v1 | MillID large v1.5 |
|---|---|---|---|---|---|---|---|---|---|
| `mus_Latn` | Muscogee | 2 | 1.000 | 1.000 | 0.000 | 0.000 | 0.333 | 0.667 | 0.667 |
| `ttj_Latn` | Tooro | 28 | 0.946 | 0.946 | 0.000 | 0.000 | 0.485 | 0.490 | 0.622 |
| `knc_Arab` | Central Kanuri | 68 | 0.823 | 0.825 | 0.007 | 0.007 | 0.462 | 0.509 | 0.532 |
| `hrv_Latn` | Croatian | 168 | 0.944 | 0.944 | 0.178 | 0.178 | 0.622 | 0.687 | 0.707 |
| `esi_Latn` | North Alaskan Inupiatun | 9 | 0.783 | 0.783 | 0.000 | 0.000 | 0.455 | 0.500 | 0.556 |
| `hak_Hani` | Hakka Chinese | 29 | 0.824 | 0.824 | 0.000 | 0.000 | 0.621 | 0.415 | 0.581 |
| `bos_Latn` | Bosnian | 198 | 0.902 | 0.902 | 0.181 | 0.181 | 0.649 | 0.673 | 0.700 |
| `plw_Latn` | Brooke's Point Palawano | 8 | 1.000 | 1.000 | 0.000 | 0.000 | 0.727 | 0.800 | 0.762 |
| `ccp_Latn` | Chakma | 3 | 1.000 | 1.000 | 0.000 | 0.000 | 0.800 | 0.333 | 0.750 |
| `mbd_Latn` | Dibabawon Manobo | 14 | 1.000 | 1.000 | 0.000 | 0.000 | 0.596 | 0.757 | 0.800 |
| `taq_Tfng` | Tamasheq | 104 | 0.718 | 0.718 | 0.000 | 0.000 | 0.529 | 0.535 | 0.527 |
| `zsm_Latn` | Standard Malay | 244 | 0.957 | 0.957 | 0.321 | 0.319 | 0.770 | 0.746 | 0.781 |
| `biu_Latn` | Biete | 7 | 1.000 | 1.000 | 0.000 | 0.000 | 0.667 | 0.667 | 0.833 |
| `ldi_Latn` | Laari | 62 | 0.992 | 0.992 | 0.000 | 0.000 | 0.777 | 0.831 | 0.765 |
| `tzm_Tfng` | Central Atlas Tamazight | 157 | 0.816 | 0.816 | 0.569 | 0.569 | 0.659 | 0.620 | 0.614 |

## Per language, test-common

Against GlotLID restricted, the best MillID is better on **13** languages and worse on **114**; against fastText restricted, the best MillID is better on **111** languages and worse on **31** (F1 differing by more than 0.01), of 213. Every language is in `leaderboard_test-common.csv`.

The 30 languages with the most passages:

| Label | Language | Passages | GlotLID | GlotLID restricted | fastText | fastText restricted | MillID tiny v1.5 | MillID medium v1 | MillID large v1.5 |
|---|---|---|---|---|---|---|---|---|---|
| `fra_Latn` | French | 1652 | 0.955 | 0.955 | 0.963 | 0.963 | 0.943 | 0.974 | 0.983 |
| `eng_Latn` | English | 1444 | 0.876 | 0.877 | 0.857 | 0.857 | 0.932 | 0.941 | 0.953 |
| `spa_Latn` | Spanish | 1136 | 0.958 | 0.958 | 0.941 | 0.941 | 0.937 | 0.953 | 0.972 |
| `tam_Taml` | Tamil | 754 | 1.000 | 1.000 | 0.997 | 0.997 | 0.997 | 0.998 | 0.999 |
| `arb_Arab` | Arabic (Modern Standard) | 745 | 0.886 | 0.886 | 0.957 | 0.954 | 0.871 | 0.905 | 0.905 |
| `por_Latn` | Portuguese | 657 | 0.954 | 0.956 | 0.950 | 0.949 | 0.959 | 0.963 | 0.973 |
| `kan_Knda` | Kannada | 601 | 0.994 | 0.994 | 0.999 | 0.999 | 0.973 | 0.987 | 0.967 |
| `zul_Latn` | Zulu | 594 | 0.904 | 0.904 | 0.970 | 0.970 | 0.883 | 0.871 | 0.909 |
| `swh_Latn` | Swahili | 592 | 0.954 | 0.953 | 0.945 | 0.945 | 0.893 | 0.897 | 0.909 |
| `mar_Deva` | Marathi | 589 | 0.989 | 0.990 | 0.987 | 0.987 | 0.968 | 0.981 | 0.979 |
| `tso_Latn` | Tsonga | 581 | 0.956 | 0.956 | 0.986 | 0.986 | 0.979 | 0.984 | 0.985 |
| `hun_Latn` | Hungarian | 576 | 1.000 | 1.000 | 0.980 | 0.980 | 0.989 | 0.991 | 0.993 |
| `afr_Latn` | Afrikaans | 569 | 0.985 | 0.985 | 0.986 | 0.986 | 0.972 | 0.976 | 0.983 |
| `tel_Telu` | Telugu | 555 | 0.998 | 0.998 | 0.999 | 0.999 | 0.997 | 1.000 | 0.999 |
| `rus_Cyrl` | Russian | 528 | 0.985 | 0.985 | 0.927 | 0.927 | 0.945 | 0.929 | 0.956 |
| `ory_Orya` | Odia (individual language) | 511 | 0.997 | 1.000 | 0.996 | 0.996 | 0.994 | 0.996 | 0.995 |
| `sot_Latn` | Southern Sotho | 491 | 0.977 | 0.978 | 0.856 | 0.856 | 0.969 | 0.967 | 0.974 |
| `nso_Latn` | Northern Sotho | 479 | 0.970 | 0.970 | 0.967 | 0.967 | 0.958 | 0.961 | 0.964 |
| `asm_Beng` | Assamese | 467 | 0.999 | 0.999 | 0.997 | 0.997 | 0.997 | 0.995 | 0.997 |
| `ben_Beng` | Bangla | 466 | 0.997 | 0.997 | 0.989 | 0.989 | 0.988 | 0.987 | 0.993 |
| `pan_Guru` | Punjabi | 460 | 1.000 | 1.000 | 0.990 | 0.990 | 1.000 | 1.000 | 1.000 |
| `amh_Ethi` | Amharic | 458 | 1.000 | 1.000 | 0.998 | 0.998 | 0.984 | 0.987 | 0.986 |
| `tsn_Latn` | Tswana | 456 | 0.961 | 0.961 | 0.870 | 0.870 | 0.948 | 0.940 | 0.947 |
| `gaz_Latn` | Oromo (West Central) | 441 | 0.974 | 0.974 | 0.973 | 0.973 | 0.935 | 0.955 | 0.941 |
| `som_Latn` | Somali | 439 | 0.995 | 0.995 | 0.991 | 0.991 | 0.984 | 0.992 | 0.991 |
| `uzn_Latn` | Northern Uzbek | 436 | 0.999 | 0.999 | 0.985 | 0.985 | 0.984 | 0.984 | 0.986 |
| `tha_Thai` | Thai | 431 | 0.980 | 0.980 | 0.981 | 0.981 | 0.980 | 0.970 | 0.981 |
| `ydd_Hebr` | Eastern Yiddish | 430 | 1.000 | 1.000 | 0.999 | 0.999 | 0.993 | 0.998 | 0.994 |
| `tur_Latn` | Turkish | 430 | 0.999 | 0.999 | 0.980 | 0.980 | 0.968 | 0.974 | 0.973 |
| `nno_Latn` | Norwegian Nynorsk | 427 | 0.985 | 0.985 | 0.954 | 0.954 | 0.913 | 0.919 | 0.927 |

Where MillID gains most over GlotLID:

| Label | Language | Passages | GlotLID | GlotLID restricted | fastText | fastText restricted | MillID tiny v1.5 | MillID medium v1 | MillID large v1.5 |
|---|---|---|---|---|---|---|---|---|---|
| `ewo_Latn` | Ewondo | 134 | 0.879 | 0.883 | 0.860 | 0.860 | 0.959 | 0.965 | 0.978 |
| `eng_Latn` | English | 1444 | 0.876 | 0.877 | 0.857 | 0.857 | 0.932 | 0.941 | 0.953 |
| `arb_Latn` | Standard Arabic | 163 | 0.906 | 0.913 | 0.852 | 0.852 | 0.942 | 0.944 | 0.949 |
| `tso_Latn` | Tsonga | 581 | 0.956 | 0.956 | 0.986 | 0.986 | 0.979 | 0.984 | 0.985 |
| `fra_Latn` | French | 1652 | 0.955 | 0.955 | 0.963 | 0.963 | 0.943 | 0.974 | 0.983 |
| `arb_Arab` | Arabic (Modern Standard) | 745 | 0.886 | 0.886 | 0.957 | 0.954 | 0.871 | 0.905 | 0.905 |
| `por_Latn` | Portuguese | 657 | 0.954 | 0.956 | 0.950 | 0.949 | 0.959 | 0.963 | 0.973 |
| `hin_Deva` | Hindi | 392 | 0.877 | 0.877 | 0.863 | 0.863 | 0.855 | 0.817 | 0.893 |
| `fuv_Latn` | Nigerian Fulfulde | 164 | 0.877 | 0.892 | 0.859 | 0.859 | 0.907 | 0.797 | 0.844 |
| `kam_Latn` | Kamba | 272 | 0.972 | 0.974 | 0.863 | 0.863 | 0.980 | 0.980 | 0.989 |
| `knc_Latn` | Central Kanuri | 293 | 0.952 | 0.952 | 0.981 | 0.981 | 0.967 | 0.959 | 0.967 |
| `spa_Latn` | Spanish | 1136 | 0.958 | 0.958 | 0.941 | 0.941 | 0.937 | 0.953 | 0.972 |
| `bug_Latn` | Buginese | 78 | 0.981 | 0.981 | 0.938 | 0.938 | 0.981 | 0.987 | 0.994 |
| `bjn_Arab` | Banjar | 116 | 0.834 | 0.987 | 0.453 | 0.960 | 0.992 | 0.983 | 0.996 |
| `san_Deva` | Sanskrit | 239 | 0.990 | 0.990 | 0.971 | 0.971 | 0.990 | 0.996 | 0.998 |

Where MillID loses most to GlotLID:

| Label | Language | Passages | GlotLID | GlotLID restricted | fastText | fastText restricted | MillID tiny v1.5 | MillID medium v1 | MillID large v1.5 |
|---|---|---|---|---|---|---|---|---|---|
| `knc_Arab` | Central Kanuri | 68 | 0.857 | 0.860 | 0.029 | 0.029 | 0.522 | 0.562 | 0.598 |
| `bos_Latn` | Bosnian | 198 | 1.000 | 1.000 | 0.897 | 0.897 | 0.685 | 0.727 | 0.762 |
| `tzm_Tfng` | Central Atlas Tamazight | 157 | 0.934 | 0.934 | 0.751 | 0.751 | 0.697 | 0.645 | 0.649 |
| `hrv_Latn` | Croatian | 168 | 0.994 | 0.994 | 0.856 | 0.856 | 0.659 | 0.717 | 0.757 |
| `zsm_Latn` | Standard Malay | 244 | 0.998 | 0.998 | 0.908 | 0.908 | 0.789 | 0.773 | 0.809 |
| `taq_Tfng` | Tamasheq | 104 | 0.805 | 0.805 | 0.000 | 0.000 | 0.654 | 0.613 | 0.604 |
| `ind_Latn` | Indonesian | 335 | 0.991 | 0.991 | 0.887 | 0.887 | 0.842 | 0.807 | 0.848 |
| `nob_Latn` | Norwegian Bokmål | 265 | 0.961 | 0.959 | 0.894 | 0.894 | 0.821 | 0.820 | 0.832 |
| `kin_Latn` | Kinyarwanda | 210 | 0.981 | 0.981 | 0.854 | 0.854 | 0.849 | 0.818 | 0.864 |
| `run_Latn` | Rundi | 291 | 0.985 | 0.985 | 0.855 | 0.855 | 0.883 | 0.843 | 0.879 |
| `mag_Deva` | Magahi | 97 | 1.000 | 1.000 | 0.772 | 0.772 | 0.890 | 0.905 | 0.897 |
| `cjk_Latn` | Chokwe | 24 | 1.000 | 1.000 | 0.807 | 0.807 | 0.857 | 0.909 | 0.837 |
| `taq_Latn` | Tamasheq | 106 | 1.000 | 1.000 | 0.838 | 0.838 | 0.902 | 0.909 | 0.922 |
| `dzo_Tibt` | Dzongkha | 346 | 0.999 | 0.999 | 0.696 | 0.696 | 0.906 | 0.926 | 0.916 |
| `lim_Latn` | Limburgish | 196 | 1.000 | 1.000 | 0.956 | 0.956 | 0.896 | 0.915 | 0.930 |

## Per language, udhr

Against GlotLID restricted, the best MillID is better on **75** languages and worse on **57**; against fastText restricted, the best MillID is better on **313** languages and worse on **8** (F1 differing by more than 0.01), of 358. Every language is in `leaderboard_udhr.csv`.

The 30 languages with the most passages:

| Label | Language | Passages | GlotLID | GlotLID restricted | fastText | fastText restricted | MillID tiny v1.5 | MillID medium v1 | MillID large v1.5 |
|---|---|---|---|---|---|---|---|---|---|
| `roh_Latn` | Romansh | 399 | 0.999 | 0.999 | 0.946 | 0.946 | 0.994 | 0.991 | 0.998 |
| `oci_Latn` | Occitan | 398 | 0.376 | 0.376 | 0.435 | 0.435 | 0.277 | 0.331 | 0.338 |
| `cmn_Hani` | Chinese (Mandarin) | 319 | 0.868 | 0.868 | 0.468 | 0.468 | 0.892 | 0.886 | 0.875 |
| `krl_Latn` | Karelian | 183 | 0.011 | 0.011 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
| `qug_Latn` | Chimborazo Highland Quichua | 182 | 0.000 | 0.000 | 0.000 | 0.000 | 0.011 | 0.000 | 0.000 |
| `acu_Latn` | Achuar-Shiwiar | 180 | 0.854 | 0.854 | 0.000 | 0.000 | 0.828 | 0.743 | 0.801 |
| `hus_Latn` | Huastec | 175 | 0.986 | 0.986 | 0.000 | 0.000 | 0.977 | 0.991 | 0.988 |
| `hau_Latn` | Hausa | 155 | 1.000 | 1.000 | 0.876 | 0.876 | 0.990 | 0.997 | 1.000 |
| `auc_Latn` | Waorani | 154 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.050 | 0.025 |
| `kwi_Latn` | Awa-Cuaiquer | 145 | 0.877 | 0.877 | 0.000 | 0.000 | 0.880 | 0.775 | 0.863 |
| `jpn_Jpan` | Japanese | 140 | 1.000 | 1.000 | 0.944 | 0.944 | 1.000 | 1.000 | 1.000 |
| `snn_Latn` | Siona | 136 | 0.514 | 0.514 | 0.000 | 0.000 | 0.630 | 0.773 | 0.467 |
| `ktu_Latn` | Kituba (Democratic Republic of Congo) | 118 | 0.680 | 0.680 | 0.000 | 0.000 | 0.765 | 0.769 | 0.773 |
| `lat_Latn` | Latin | 118 | 0.992 | 0.992 | 0.000 | 0.000 | 0.996 | 0.996 | 0.996 |
| `hat_Latn` | Haitian Creole | 117 | 0.979 | 0.979 | 0.699 | 0.699 | 0.992 | 0.991 | 0.987 |
| `nya_Latn` | Nyanja | 116 | 1.000 | 1.000 | 0.806 | 0.806 | 0.991 | 0.987 | 0.978 |
| `kmb_Latn` | Kimbundu | 116 | 1.000 | 1.000 | 0.877 | 0.877 | 0.970 | 0.979 | 0.991 |
| `rmn_Latn` | Balkan Romani | 116 | 0.857 | 0.857 | 0.000 | 0.000 | 0.678 | 0.667 | 0.704 |
| `twi_Latn` | Twi | 116 | 0.996 | 0.996 | 0.399 | 0.396 | 0.983 | 0.987 | 0.991 |
| `chr_Cher` | Cherokee | 115 | 0.663 | 0.663 | 0.000 | 0.000 | 0.670 | 0.715 | 0.945 |
| `por_Latn` | Portuguese | 115 | 0.996 | 0.996 | 0.793 | 0.793 | 0.955 | 0.996 | 1.000 |
| `lvs_Latn` | Standard Latvian | 115 | 1.000 | 1.000 | 0.991 | 0.991 | 0.987 | 0.996 | 1.000 |
| `pov_Latn` | Upper Guinea Crioulo | 115 | 0.833 | 0.833 | 0.000 | 0.000 | 0.821 | 0.838 | 0.827 |
| `kng_Latn` | Koongo | 115 | 0.193 | 0.193 | 0.457 | 0.457 | 0.076 | 0.047 | 0.047 |
| `ell_Grek` | Greek | 115 | 1.000 | 1.000 | 1.000 | 1.000 | 0.670 | 0.655 | 0.670 |
| `tha_Thai` | Thai | 114 | 1.000 | 1.000 | 0.996 | 0.996 | 1.000 | 1.000 | 1.000 |
| `tso_Latn` | Tsonga | 114 | 0.781 | 0.781 | 0.691 | 0.691 | 0.860 | 0.882 | 0.774 |
| `cjk_Latn` | Chokwe | 114 | 1.000 | 1.000 | 0.533 | 0.533 | 0.871 | 0.949 | 0.959 |
| `fas_Arab` | Persian | 114 | 1.000 | 1.000 | 0.934 | 0.934 | 0.991 | 0.991 | 0.982 |
| `lin_Latn` | Lingala | 110 | 1.000 | 1.000 | 0.986 | 0.986 | 0.991 | 1.000 | 0.996 |

Where MillID gains most over GlotLID:

| Label | Language | Passages | GlotLID | GlotLID restricted | fastText | fastText restricted | MillID tiny v1.5 | MillID medium v1 | MillID large v1.5 |
|---|---|---|---|---|---|---|---|---|---|
| `mic_Latn` | Mi'kmaw | 59 | 0.427 | 0.427 | 0.000 | 0.000 | 0.992 | 1.000 | 1.000 |
| `tet_Latn` | Tetum | 58 | 0.167 | 0.167 | 0.000 | 0.000 | 0.638 | 0.699 | 0.708 |
| `fuf_Latn` | Pular | 58 | 0.483 | 0.483 | 0.000 | 0.000 | 0.906 | 0.816 | 0.936 |
| `skr_Arab` | Saraiki | 63 | 0.500 | 0.500 | 0.000 | 0.000 | 0.867 | 0.923 | 0.875 |
| `sag_Latn` | Sango | 59 | 0.578 | 0.578 | 0.189 | 0.189 | 0.992 | 0.992 | 0.992 |
| `zgh_Tfng` | Standard Moroccan Tamazight | 60 | 0.462 | 0.462 | 0.000 | 0.000 | 0.750 | 0.824 | 0.652 |
| `qvh_Latn` | Huamalíes-Dos de Mayo Huánuco Quechua | 58 | 0.154 | 0.154 | 0.000 | 0.000 | 0.468 | 0.215 | 0.215 |
| `chr_Cher` | Cherokee | 115 | 0.663 | 0.663 | 0.000 | 0.000 | 0.670 | 0.715 | 0.945 |
| `tzm_Tfng` | Central Atlas Tamazight | 1 | 0.065 | 0.065 | 0.033 | 0.033 | 0.167 | 0.333 | 0.100 |
| `snn_Latn` | Siona | 136 | 0.514 | 0.514 | 0.000 | 0.000 | 0.630 | 0.773 | 0.467 |
| `fkv_Latn` | Kven Finnish | 58 | 0.667 | 0.667 | 0.000 | 0.000 | 0.635 | 0.697 | 0.906 |
| `bum_Latn` | Bulu | 58 | 0.098 | 0.098 | 0.000 | 0.000 | 0.215 | 0.319 | 0.188 |
| `mxv_Latn` | Metlatónoc Mixtec | 58 | 0.000 | 0.000 | 0.000 | 0.000 | 0.129 | 0.215 | 0.034 |
| `pnb_Arab` | Western Panjabi | 61 | 0.739 | 0.739 | 0.000 | 0.000 | 0.882 | 0.931 | 0.884 |
| `lld_Latn` | Ladin | 59 | 0.731 | 0.731 | 0.000 | 0.000 | 0.909 | 0.899 | 0.881 |

Where MillID loses most to GlotLID:

| Label | Language | Passages | GlotLID | GlotLID restricted | fastText | fastText restricted | MillID tiny v1.5 | MillID medium v1 | MillID large v1.5 |
|---|---|---|---|---|---|---|---|---|---|
| `gyr_Latn` | Guarayu | 49 | 0.990 | 0.990 | 0.000 | 0.000 | 0.537 | 0.339 | 0.393 |
| `dyu_Latn` | Dyula | 58 | 0.434 | 0.434 | 0.015 | 0.015 | 0.000 | 0.000 | 0.000 |
| `ell_Grek` | Greek | 115 | 1.000 | 1.000 | 1.000 | 1.000 | 0.670 | 0.655 | 0.670 |
| `cbu_Latn` | Candoshi-Shapra | 57 | 0.882 | 0.874 | 0.000 | 0.000 | 0.500 | 0.371 | 0.557 |
| `shp_Latn` | Shipibo-Conibo | 62 | 0.912 | 0.912 | 0.000 | 0.000 | 0.469 | 0.410 | 0.607 |
| `cpu_Latn` | Pichis Ashéninka | 60 | 0.974 | 0.974 | 0.000 | 0.000 | 0.729 | 0.571 | 0.571 |
| `qvn_Latn` | North Junín Quechua | 57 | 0.460 | 0.460 | 0.000 | 0.000 | 0.219 | 0.219 | 0.034 |
| `hin_Deva` | Hindi | 59 | 0.957 | 0.957 | 0.975 | 0.975 | 0.723 | 0.607 | 0.727 |
| `cni_Latn` | Asháninka | 52 | 0.963 | 0.963 | 0.000 | 0.000 | 0.773 | 0.708 | 0.717 |
| `hrv_Latn` | Croatian | 58 | 0.727 | 0.727 | 0.474 | 0.474 | 0.500 | 0.537 | 0.559 |
| `rmn_Latn` | Balkan Romani | 116 | 0.857 | 0.857 | 0.000 | 0.000 | 0.678 | 0.667 | 0.704 |
| `koi_Cyrl` | Komi-Permyak | 57 | 1.000 | 1.000 | 0.000 | 0.000 | 0.837 | 0.837 | 0.849 |
| `qvm_Latn` | Margos-Yarowilca-Lauricocha Quechua | 60 | 0.500 | 0.500 | 0.000 | 0.000 | 0.356 | 0.094 | 0.356 |
| `kea_Latn` | Kabuverdianu | 58 | 0.935 | 0.935 | 0.125 | 0.124 | 0.792 | 0.794 | 0.773 |
| `ind_Latn` | Indonesian | 60 | 0.903 | 0.903 | 0.565 | 0.565 | 0.768 | 0.674 | 0.645 |