Download README.md from baobabtech/millid-tiny-v1.5: direct link, hf CLI and curl.
- Browser
- Download file 23.6 kB
-
https://huggingface.co/baobabtech/millid-tiny-v1.5/resolve/refs%2Fpr%2F1/README.md
- Command line
-
hf download hf://baobabtech/millid-tiny-v1.5@refs/pr/1/README.md
-
curl -L -o README.md https://huggingface.co/baobabtech/millid-tiny-v1.5/resolve/refs%2Fpr%2F1/README.md
license: apache-2.0
library_name: pytorch
pipeline_tag: text-classification
tags:
- language-identification
- onnx
- byte-level
datasets:
- baobabtech/millid-dataset
metrics:
- f1
- accuracy
model-index:
- name: millid-tiny-v1.5
results:
- task:
type: text-classification
name: Language identification
dataset:
name: MillID test split (the full build)
type: baobabtech/millid-dataset
split: test
metrics:
- type: f1
name: Macro F1
value: 0.9554
- type: accuracy
name: Top-1 accuracy
value: 0.9587
- task:
type: text-classification
name: Language identification
dataset:
name: UDHR-LID, the languages MillID knows
type: baobabtech/millid-dataset
config: udhr
split: test
metrics:
- type: f1
name: Macro F1
value: 0.8764
- type: accuracy
name: Top-1 accuracy
value: 0.8492
MillID tiny v1.5: MultiLingual Language IDentifier
A byte-level, encoder-only transformer that names the language of a passage, from a table cell to a paragraph: 1,641 language-script labels (ISO 639-3 + script, e.g. eng_Latn, amh_Ethi). 2.9M parameters, trained from scratch. Apache-2.0.
The MillID models
| Model | Size | Macro F1, test / udhr |
Runs best on |
|---|---|---|---|
| MillID tiny v1.5 (this model) | 2.9M parameters, ONNX 11 MB | 0.9554 / 0.8764 | a CPU, or a web page |
| MillID medium v1 | 115M parameters, ONNX 455 MB | 0.9576 / 0.8780 | a GPU |
| MillID large v1.5 | 193M parameters, ONNX 764 MB | 0.9646 / 0.8812 | a GPU: the most accurate |
All three, language by language, against GlotLID and Meta's fastText: the leaderboard. GlotLID, restricted to MillID's labels, scores 0.9755 / 0.8873 on the same sets.
Use it
No server and no PyTorch: the model runs in your process from onnx/model.onnx, in Python or JavaScript, and the two give the same answers (one encoder, ported line for line and tested against the same fixtures in baobab-tech/millid).
pip install "millid[cpu] @ git+https://github.com/baobab-tech/millid.git" # [gpu] for CUDA
from millid import MillID
model = MillID.from_pretrained("baobabtech/millid-tiny-v1.5")
model.predict("Lave men ou ak savon anvan ou manje.", k=3)
# [Prediction(label='hat_Latn', language='Haitian Creole', probability=...), ...]
model.predict(["one text", "another"], k=3) # one list of answers per text
model.identify_document(long_text) # any length: its languages and segments
millid predict "Lave men ou ak savon anvan ou manje." --model baobabtech/millid-tiny-v1.5 -k 3
JavaScript, in Node or a browser (js/ in the same repository; not on npm yet):
import { MillID } from "@baobabtech/millid"; // "@baobabtech/millid/web" in a browser, with { ort }
const model = await MillID.fromPretrained("baobabtech/millid-tiny-v1.5");
await model.predict("Lave men ou ak savon anvan ou manje.", 3);
millid.py in this repository is the PyTorch training code, which reads model.safetensors to fine-tune or export.
The ONNX graph takes, for each UTF-8 byte of a passage cut to 255 bytes after a CLS token, ids, script and charpos (int64, [batch, bytes]), ngrams (int64, [batch, bytes, 3]) and mask (bool, [batch, bytes]), and returns logits (float32, [batch, labels]). millid.encode_batch and the JavaScript encodeBatch build them; script ids come from a frozen table (Unicode 15.0), so every runtime gives the same ones.
Architecture
UTF-8 bytes (after Unicode NFKC), each embedded as byte + Unicode script + byte-in-character + hashed character 2-, 3- and 4-grams (4,096 buckets) -> depthwise convolution stem (kernel 5) -> strided convolution, 1 bytes per token -> 3 pre-norm blocks (RMSNorm, attention with RoPE, 3 heads, SwiGLU; d=192) -> masked mean pooling -> language head, each language's vector the sum of its own, its language's and its script's. Up to 256 bytes per passage. See ARCHITECTURE.md.
Evaluation against GlotLID and fastText
validation chose the checkpoint; test is split by document from training; udhr is the Universal Declaration of Human Rights, labelled by its translators, over the languages MillID knows, and udhr-all over all of them. glotlid is GlotLID as it is; glotlid@millid is GlotLID restricted to MillID's labels, the like-for-like comparison. GlotLID's own labels came from training on data that includes UDHR and labelled FinePDFs and FineWeb-2, so its scores on udhr and on those sources are optimistic. fasttext is Meta's fastText language identifier (NLLB, 218 labels), its answers mapped onto MillID's spelling (Simplified and Traditional Han as Hani, pes/prs as fas, est as ekk). It cannot answer most of the 1,600 languages here, so test-common and udhr-common score every model on the passages whose language all of them know: the head-to-head on equal terms.
| Set | Model | Top-1 | Top-3 | Top-5 | Macro P | Macro R | Macro F1 | Median F1 | Languages |
|---|---|---|---|---|---|---|---|---|---|
| validation | millid | 0.9578 | 0.9889 | 0.9933 | 0.9539 | 0.9591 | 0.9551 | 0.9822 | 1626 |
| test | millid | 0.9587 | 0.9890 | 0.9932 | 0.9550 | 0.9588 | 0.9554 | 0.9825 | 1621 |
| test | glotlid | 0.9776 | 0.9977 | 0.9984 | 0.9783 | 0.9752 | 0.9748 | 1.0000 | 1621 |
| test | glotlid@millid | 0.9783 | 0.9978 | 0.9985 | 0.9778 | 0.9765 | 0.9755 | 1.0000 | 1621 |
| test | fasttext | 0.2511 | 0.2610 | 0.2625 | 0.0520 | 0.1205 | 0.0657 | 0.0000 | 1621 |
| test | fasttext@millid | 0.2515 | 0.2611 | 0.2625 | 0.0521 | 0.1209 | 0.0659 | 0.0000 | 1621 |
| test-common | millid | 0.9396 | 0.9857 | 0.9909 | 0.9709 | 0.9353 | 0.9513 | 0.9726 | 213 |
| test-common | glotlid | 0.9738 | 0.9925 | 0.9946 | 0.9924 | 0.9771 | 0.9835 | 0.9983 | 213 |
| test-common | glotlid@millid | 0.9749 | 0.9927 | 0.9948 | 0.9923 | 0.9790 | 0.9846 | 0.9983 | 213 |
| test-common | fasttext | 0.9339 | 0.9707 | 0.9762 | 0.9357 | 0.9169 | 0.9149 | 0.9680 | 213 |
| test-common | fasttext@millid | 0.9352 | 0.9710 | 0.9763 | 0.9354 | 0.9201 | 0.9173 | 0.9680 | 213 |
| udhr | millid | 0.8492 | 0.9117 | 0.9257 | 0.9148 | 0.8736 | 0.8764 | 0.9907 | 358 |
| udhr | glotlid | 0.8598 | 0.9280 | 0.9421 | 0.9261 | 0.8826 | 0.8865 | 0.9915 | 358 |
| udhr | glotlid@millid | 0.8614 | 0.9294 | 0.9433 | 0.9258 | 0.8844 | 0.8873 | 0.9915 | 358 |
| udhr | fasttext | 0.4625 | 0.4857 | 0.4903 | 0.3152 | 0.4479 | 0.3497 | 0.0000 | 358 |
| udhr | fasttext@millid | 0.4629 | 0.4857 | 0.4904 | 0.3152 | 0.4483 | 0.3499 | 0.0000 | 358 |
| udhr-common | millid | 0.9123 | 0.9606 | 0.9655 | 0.9794 | 0.9393 | 0.9513 | 1.0000 | 170 |
| udhr-common | glotlid | 0.9336 | 0.9678 | 0.9732 | 0.9888 | 0.9576 | 0.9657 | 1.0000 | 170 |
| udhr-common | glotlid@millid | 0.9338 | 0.9683 | 0.9736 | 0.9888 | 0.9579 | 0.9659 | 1.0000 | 170 |
| udhr-common | fasttext | 0.9187 | 0.9647 | 0.9741 | 0.9474 | 0.9431 | 0.9323 | 1.0000 | 170 |
| udhr-common | fasttext@millid | 0.9195 | 0.9647 | 0.9742 | 0.9474 | 0.9441 | 0.9329 | 1.0000 | 170 |
| udhr-all | millid | 0.7274 | 0.7809 | 0.7928 | 0.7340 | 0.7290 | 0.7149 | 0.9649 | 429 |
| udhr-all | glotlid | 0.7502 | 0.8145 | 0.8282 | 0.7695 | 0.7529 | 0.7454 | 0.9796 | 429 |
| udhr-all | glotlid@millid | 0.7378 | 0.7960 | 0.8079 | 0.7425 | 0.7381 | 0.7245 | 0.9748 | 429 |
| udhr-all | fasttext | 0.3961 | 0.4160 | 0.4200 | 0.2479 | 0.3737 | 0.2795 | 0.0000 | 429 |
| udhr-all | fasttext@millid | 0.3964 | 0.4160 | 0.4200 | 0.2479 | 0.3741 | 0.2797 | 0.0000 | 429 |
Top-1 accuracy by passage length:
| Set | Model | short | sentence | paragraph |
|---|---|---|---|---|
| validation | millid | 0.9312 | 0.9788 | 0.9850 |
| test | millid | 0.9333 | 0.9797 | 0.9833 |
| test | glotlid | 0.9664 | 0.9853 | 0.9911 |
| test | glotlid@millid | 0.9674 | 0.9858 | 0.9912 |
| test | fasttext | 0.2760 | 0.2501 | 0.1958 |
| test | fasttext@millid | 0.2764 | 0.2506 | 0.1959 |
| test-common | millid | 0.9125 | 0.9685 | 0.9725 |
| test-common | glotlid | 0.9594 | 0.9876 | 0.9947 |
| test-common | glotlid@millid | 0.9609 | 0.9885 | 0.9949 |
| test-common | fasttext | 0.9112 | 0.9562 | 0.9651 |
| test-common | fasttext@millid | 0.9125 | 0.9581 | 0.9655 |
Leaderboard: test
MillID's F1 against the others', language by language: against GlotLID restricted to MillID's labels, MillID is better on 103 languages, worse on 869 and within 0.01 on 649; against fastText restricted to MillID's labels, MillID is better on 1616 languages, worse on 0 and within 0.01 on 5. The full table, with precision and recall, is reports/test/leaderboard.csv.
The 20 languages with the most passages:
| Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
|---|---|---|---|---|---|---|---|
fra_Latn |
French | 1652 | 0.929 | 0.928 | 0.928 | 0.808 | 0.808 |
eng_Latn |
English | 1444 | 0.917 | 0.858 | 0.859 | 0.653 | 0.648 |
spa_Latn |
Spanish | 1136 | 0.930 | 0.951 | 0.951 | 0.574 | 0.574 |
tam_Taml |
Tamil | 754 | 0.997 | 0.993 | 0.993 | 0.965 | 0.965 |
arb_Arab |
Arabic (Modern Standard) | 745 | 0.814 | 0.785 | 0.785 | 0.513 | 0.512 |
por_Latn |
Portuguese | 657 | 0.947 | 0.947 | 0.949 | 0.756 | 0.755 |
kan_Knda |
Kannada | 601 | 0.969 | 0.993 | 0.993 | 0.922 | 0.922 |
zul_Latn |
Zulu | 594 | 0.846 | 0.896 | 0.896 | 0.539 | 0.539 |
swh_Latn |
Swahili | 592 | 0.848 | 0.917 | 0.916 | 0.346 | 0.346 |
mar_Deva |
Marathi | 589 | 0.965 | 0.989 | 0.990 | 0.926 | 0.926 |
tso_Latn |
Tsonga | 581 | 0.965 | 0.956 | 0.956 | 0.425 | 0.425 |
hun_Latn |
Hungarian | 576 | 0.987 | 0.999 | 0.999 | 0.232 | 0.232 |
afr_Latn |
Afrikaans | 569 | 0.967 | 0.985 | 0.985 | 0.861 | 0.860 |
tel_Telu |
Telugu | 555 | 0.991 | 0.994 | 0.994 | 0.951 | 0.951 |
rus_Cyrl |
Russian | 528 | 0.932 | 0.949 | 0.949 | 0.462 | 0.462 |
nbl_Latn |
South Ndebele | 528 | 0.903 | 0.948 | 0.949 | 0.000 | 0.000 |
ory_Orya |
Odia (individual language) | 511 | 0.988 | 0.991 | 0.994 | 0.976 | 0.976 |
sot_Latn |
Southern Sotho | 491 | 0.966 | 0.977 | 0.978 | 0.660 | 0.660 |
nso_Latn |
Northern Sotho | 479 | 0.956 | 0.969 | 0.969 | 0.915 | 0.914 |
kua_Latn |
Kuanyama | 476 | 0.962 | 0.986 | 0.986 | 0.000 | 0.000 |
Where MillID gains most over GlotLID:
| Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
|---|---|---|---|---|---|---|---|
lbj_Tibt |
Ladakhi | 66 | 0.943 | 0.000 | 0.000 | 0.000 | 0.000 |
ain_Latn |
Ainu | 1 | 0.667 | 0.000 | 0.000 | 0.000 | 0.000 |
ajp_Arab |
South Levantine Arabic | 28 | 0.946 | 0.581 | 0.581 | 0.000 | 0.000 |
und_Laoo |
und | 20 | 0.923 | 0.621 | 0.621 | 0.000 | 0.000 |
und_Thaa |
und | 29 | 0.881 | 0.651 | 0.651 | 0.000 | 0.000 |
sgh_Cyrl |
Shughni | 9 | 0.857 | 0.667 | 0.667 | 0.000 | 0.000 |
und_Geor |
und | 32 | 0.877 | 0.694 | 0.694 | 0.000 | 0.000 |
tuk_Arab |
Turkmen | 135 | 0.905 | 0.761 | 0.761 | 0.000 | 0.000 |
und_Syrc |
und | 43 | 0.966 | 0.822 | 0.822 | 0.000 | 0.000 |
gmh_Latn |
Middle High German | 177 | 0.922 | 0.786 | 0.786 | 0.000 | 0.000 |
orv_Cyrl |
Old Russian | 239 | 0.931 | 0.800 | 0.800 | 0.000 | 0.000 |
und_Kana |
und | 136 | 0.982 | 0.862 | 0.862 | 0.000 | 0.000 |
und_Yiii |
und | 38 | 1.000 | 0.894 | 0.884 | 0.000 | 0.000 |
neb_Latn |
Toura (Côte d'Ivoire) | 11 | 0.957 | 0.625 | 0.842 | 0.000 | 0.000 |
dag_Latn |
Dagbani | 149 | 0.767 | 0.662 | 0.660 | 0.000 | 0.000 |
Where MillID loses most to GlotLID:
| Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
|---|---|---|---|---|---|---|---|
mus_Latn |
Muscogee | 2 | 0.333 | 1.000 | 1.000 | 0.000 | 0.000 |
ttj_Latn |
Tooro | 28 | 0.485 | 0.946 | 0.946 | 0.000 | 0.000 |
mbd_Latn |
Dibabawon Manobo | 14 | 0.596 | 1.000 | 1.000 | 0.000 | 0.000 |
knc_Arab |
Central Kanuri | 68 | 0.462 | 0.823 | 0.825 | 0.007 | 0.007 |
biu_Latn |
Biete | 7 | 0.667 | 1.000 | 1.000 | 0.000 | 0.000 |
dks_Latn |
Southeastern Dinka | 3 | 0.667 | 1.000 | 1.000 | 0.000 | 0.000 |
esi_Latn |
North Alaskan Inupiatun | 9 | 0.455 | 0.783 | 0.783 | 0.000 | 0.000 |
hrv_Latn |
Croatian | 168 | 0.622 | 0.944 | 0.944 | 0.178 | 0.178 |
plw_Latn |
Brooke's Point Palawano | 8 | 0.727 | 1.000 | 1.000 | 0.000 | 0.000 |
bos_Latn |
Bosnian | 198 | 0.649 | 0.902 | 0.902 | 0.181 | 0.181 |
xbr_Latn |
Kambera | 8 | 0.750 | 1.000 | 1.000 | 0.000 | 0.000 |
nyf_Latn |
Giryama | 10 | 0.645 | 0.952 | 0.870 | 0.000 | 0.000 |
ldi_Latn |
Laari | 62 | 0.777 | 0.992 | 0.992 | 0.000 | 0.000 |
hak_Hani |
Hakka Chinese | 29 | 0.621 | 0.824 | 0.824 | 0.000 | 0.000 |
ccp_Latn |
Chakma | 3 | 0.800 | 1.000 | 1.000 | 0.000 | 0.000 |
Leaderboard: udhr
MillID's F1 against the others', language by language: against GlotLID restricted to MillID's labels, MillID is better on 52 languages, worse on 105 and within 0.01 on 201; against fastText restricted to MillID's labels, MillID is better on 303 languages, worse on 15 and within 0.01 on 40. The full table, with precision and recall, is reports/udhr/leaderboard.csv.
The 20 languages with the most passages:
| Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
|---|---|---|---|---|---|---|---|
roh_Latn |
Romansh | 399 | 0.994 | 0.999 | 0.999 | 0.946 | 0.946 |
oci_Latn |
Occitan | 398 | 0.277 | 0.376 | 0.376 | 0.435 | 0.435 |
cmn_Hani |
Chinese (Mandarin) | 319 | 0.892 | 0.868 | 0.868 | 0.468 | 0.468 |
krl_Latn |
Karelian | 183 | 0.000 | 0.011 | 0.011 | 0.000 | 0.000 |
qug_Latn |
Chimborazo Highland Quichua | 182 | 0.011 | 0.000 | 0.000 | 0.000 | 0.000 |
acu_Latn |
Achuar-Shiwiar | 180 | 0.828 | 0.854 | 0.854 | 0.000 | 0.000 |
hus_Latn |
Huastec | 175 | 0.977 | 0.986 | 0.986 | 0.000 | 0.000 |
hau_Latn |
Hausa | 155 | 0.990 | 1.000 | 1.000 | 0.876 | 0.876 |
auc_Latn |
Waorani | 154 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
kwi_Latn |
Awa-Cuaiquer | 145 | 0.880 | 0.877 | 0.877 | 0.000 | 0.000 |
jpn_Jpan |
Japanese | 140 | 1.000 | 1.000 | 1.000 | 0.944 | 0.944 |
snn_Latn |
Siona | 136 | 0.630 | 0.514 | 0.514 | 0.000 | 0.000 |
ktu_Latn |
Kituba (Democratic Republic of Congo) | 118 | 0.765 | 0.680 | 0.680 | 0.000 | 0.000 |
lat_Latn |
Latin | 118 | 0.996 | 0.992 | 0.992 | 0.000 | 0.000 |
hat_Latn |
Haitian Creole | 117 | 0.992 | 0.979 | 0.979 | 0.699 | 0.699 |
nya_Latn |
Nyanja | 116 | 0.991 | 1.000 | 1.000 | 0.806 | 0.806 |
kmb_Latn |
Kimbundu | 116 | 0.970 | 1.000 | 1.000 | 0.877 | 0.877 |
rmn_Latn |
Balkan Romani | 116 | 0.678 | 0.857 | 0.857 | 0.000 | 0.000 |
twi_Latn |
Twi | 116 | 0.983 | 0.996 | 0.996 | 0.399 | 0.396 |
chr_Cher |
Cherokee | 115 | 0.670 | 0.663 | 0.663 | 0.000 | 0.000 |
Where MillID gains most over GlotLID:
| Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
|---|---|---|---|---|---|---|---|
mic_Latn |
Mi'kmaw | 59 | 0.992 | 0.427 | 0.427 | 0.000 | 0.000 |
tet_Latn |
Tetum | 58 | 0.638 | 0.167 | 0.167 | 0.000 | 0.000 |
fuf_Latn |
Pular | 58 | 0.906 | 0.483 | 0.483 | 0.000 | 0.000 |
sag_Latn |
Sango | 59 | 0.992 | 0.578 | 0.578 | 0.189 | 0.189 |
skr_Arab |
Saraiki | 63 | 0.867 | 0.500 | 0.500 | 0.000 | 0.000 |
qvh_Latn |
Huamalíes-Dos de Mayo Huánuco Quechua | 58 | 0.468 | 0.154 | 0.154 | 0.000 | 0.000 |
zgh_Tfng |
Standard Moroccan Tamazight | 60 | 0.750 | 0.462 | 0.462 | 0.000 | 0.000 |
lld_Latn |
Ladin | 59 | 0.909 | 0.731 | 0.731 | 0.000 | 0.000 |
tah_Latn |
Tahitian | 57 | 0.973 | 0.812 | 0.812 | 0.966 | 0.966 |
yue_Hani |
Cantonese | 49 | 0.734 | 0.580 | 0.580 | 0.111 | 0.111 |
ote_Latn |
Mezquital Otomi | 59 | 0.154 | 0.000 | 0.000 | 0.000 | 0.000 |
pnb_Arab |
Western Panjabi | 61 | 0.882 | 0.739 | 0.739 | 0.000 | 0.000 |
mxv_Latn |
Metlatónoc Mixtec | 58 | 0.129 | 0.000 | 0.000 | 0.000 | 0.000 |
lij_Latn |
Ligurian | 58 | 0.983 | 0.859 | 0.859 | 0.487 | 0.487 |
bum_Latn |
Bulu | 58 | 0.215 | 0.098 | 0.098 | 0.000 | 0.000 |
Where MillID loses most to GlotLID:
| Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
|---|---|---|---|---|---|---|---|
gyr_Latn |
Guarayu | 49 | 0.537 | 0.990 | 0.990 | 0.000 | 0.000 |
shp_Latn |
Shipibo-Conibo | 62 | 0.469 | 0.912 | 0.912 | 0.000 | 0.000 |
dyu_Latn |
Dyula | 58 | 0.000 | 0.434 | 0.434 | 0.015 | 0.015 |
tbz_Latn |
Ditammari | 60 | 0.000 | 0.378 | 0.378 | 0.000 | 0.000 |
cbu_Latn |
Candoshi-Shapra | 57 | 0.500 | 0.882 | 0.874 | 0.000 | 0.000 |
idu_Latn |
Idoma | 51 | 0.603 | 0.879 | 0.938 | 0.000 | 0.000 |
ell_Grek |
Greek | 115 | 0.670 | 1.000 | 1.000 | 1.000 | 1.000 |
qvc_Latn |
Cajamarca Quechua | 58 | 0.648 | 0.906 | 0.906 | 0.000 | 0.000 |
cpu_Latn |
Pichis Ashéninka | 60 | 0.729 | 0.974 | 0.974 | 0.000 | 0.000 |
qvn_Latn |
North Junín Quechua | 57 | 0.219 | 0.460 | 0.460 | 0.000 | 0.000 |
hin_Deva |
Hindi | 59 | 0.723 | 0.957 | 0.957 | 0.975 | 0.975 |
hrv_Latn |
Croatian | 58 | 0.500 | 0.727 | 0.727 | 0.474 | 0.474 |
kbp_Latn |
Kabiyè | 59 | 0.781 | 1.000 | 1.000 | 0.389 | 0.389 |
cni_Latn |
Asháninka | 52 | 0.773 | 0.963 | 0.963 | 0.000 | 0.000 |
rmn_Latn |
Balkan Romani | 116 | 0.678 | 0.857 | 0.857 | 0.000 | 0.000 |
Leaderboard: test-common
MillID's F1 against the others', language by language: against GlotLID restricted to MillID's labels, MillID is better on 6 languages, worse on 154 and within 0.01 on 53; against fastText restricted to MillID's labels, MillID is better on 85 languages, worse on 51 and within 0.01 on 77. The full table, with precision and recall, is reports/test-common/leaderboard.csv.
The 20 languages with the most passages:
| Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
|---|---|---|---|---|---|---|---|
fra_Latn |
French | 1652 | 0.943 | 0.955 | 0.955 | 0.963 | 0.963 |
eng_Latn |
English | 1444 | 0.932 | 0.876 | 0.877 | 0.857 | 0.857 |
spa_Latn |
Spanish | 1136 | 0.937 | 0.958 | 0.958 | 0.941 | 0.941 |
tam_Taml |
Tamil | 754 | 0.997 | 1.000 | 1.000 | 0.997 | 0.997 |
arb_Arab |
Arabic (Modern Standard) | 745 | 0.871 | 0.886 | 0.886 | 0.957 | 0.954 |
por_Latn |
Portuguese | 657 | 0.959 | 0.954 | 0.956 | 0.950 | 0.949 |
kan_Knda |
Kannada | 601 | 0.973 | 0.994 | 0.994 | 0.999 | 0.999 |
zul_Latn |
Zulu | 594 | 0.883 | 0.904 | 0.904 | 0.970 | 0.970 |
swh_Latn |
Swahili | 592 | 0.893 | 0.954 | 0.953 | 0.945 | 0.945 |
mar_Deva |
Marathi | 589 | 0.968 | 0.989 | 0.990 | 0.987 | 0.987 |
tso_Latn |
Tsonga | 581 | 0.979 | 0.956 | 0.956 | 0.986 | 0.986 |
hun_Latn |
Hungarian | 576 | 0.989 | 1.000 | 1.000 | 0.980 | 0.980 |
afr_Latn |
Afrikaans | 569 | 0.972 | 0.985 | 0.985 | 0.986 | 0.986 |
tel_Telu |
Telugu | 555 | 0.997 | 0.998 | 0.998 | 0.999 | 0.999 |
rus_Cyrl |
Russian | 528 | 0.945 | 0.985 | 0.985 | 0.927 | 0.927 |
ory_Orya |
Odia (individual language) | 511 | 0.994 | 0.997 | 1.000 | 0.996 | 0.996 |
sot_Latn |
Southern Sotho | 491 | 0.969 | 0.977 | 0.978 | 0.856 | 0.856 |
nso_Latn |
Northern Sotho | 479 | 0.958 | 0.970 | 0.970 | 0.967 | 0.967 |
asm_Beng |
Assamese | 467 | 0.997 | 0.999 | 0.999 | 0.997 | 0.997 |
ben_Beng |
Bangla | 466 | 0.988 | 0.997 | 0.997 | 0.989 | 0.989 |
Where MillID gains most over GlotLID:
| Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
|---|---|---|---|---|---|---|---|
ewo_Latn |
Ewondo | 134 | 0.959 | 0.879 | 0.883 | 0.860 | 0.860 |
eng_Latn |
English | 1444 | 0.932 | 0.876 | 0.877 | 0.857 | 0.857 |
arb_Latn |
Standard Arabic | 163 | 0.942 | 0.906 | 0.913 | 0.852 | 0.852 |
tso_Latn |
Tsonga | 581 | 0.979 | 0.956 | 0.956 | 0.986 | 0.986 |
fuv_Latn |
Nigerian Fulfulde | 164 | 0.907 | 0.877 | 0.892 | 0.859 | 0.859 |
knc_Latn |
Central Kanuri | 293 | 0.967 | 0.952 | 0.952 | 0.981 | 0.981 |
quy_Latn |
Ayacucho Quechua | 271 | 0.940 | 0.925 | 0.933 | 0.979 | 0.979 |
kam_Latn |
Kamba | 272 | 0.980 | 0.972 | 0.974 | 0.863 | 0.863 |
bjn_Arab |
Banjar | 116 | 0.992 | 0.834 | 0.987 | 0.453 | 0.960 |
por_Latn |
Portuguese | 657 | 0.959 | 0.954 | 0.956 | 0.950 | 0.949 |
mlt_Latn |
Maltese | 291 | 0.995 | 0.993 | 0.993 | 0.961 | 0.961 |
hye_Armn |
Armenian | 178 | 1.000 | 1.000 | 1.000 | 0.977 | 0.977 |
khm_Khmr |
Khmer | 187 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
guj_Gujr |
Gujarati | 354 | 1.000 | 1.000 | 1.000 | 0.984 | 0.984 |
mal_Mlym |
Malayalam | 338 | 1.000 | 1.000 | 1.000 | 0.994 | 0.994 |
Where MillID loses most to GlotLID:
| Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
|---|---|---|---|---|---|---|---|
knc_Arab |
Central Kanuri | 68 | 0.522 | 0.857 | 0.860 | 0.029 | 0.029 |
hrv_Latn |
Croatian | 168 | 0.659 | 0.994 | 0.994 | 0.856 | 0.856 |
bos_Latn |
Bosnian | 198 | 0.685 | 1.000 | 1.000 | 0.897 | 0.897 |
tzm_Tfng |
Central Atlas Tamazight | 157 | 0.697 | 0.934 | 0.934 | 0.751 | 0.751 |
zsm_Latn |
Standard Malay | 244 | 0.789 | 0.998 | 0.998 | 0.908 | 0.908 |
taq_Tfng |
Tamasheq | 104 | 0.654 | 0.805 | 0.805 | 0.000 | 0.000 |
ind_Latn |
Indonesian | 335 | 0.842 | 0.991 | 0.991 | 0.887 | 0.887 |
kas_Deva |
Kashmiri | 47 | 0.557 | 0.703 | 0.703 | 0.587 | 0.587 |
cjk_Latn |
Chokwe | 24 | 0.857 | 1.000 | 1.000 | 0.807 | 0.807 |
nob_Latn |
Norwegian Bokmål | 265 | 0.821 | 0.961 | 0.959 | 0.894 | 0.894 |
kin_Latn |
Kinyarwanda | 210 | 0.849 | 0.981 | 0.981 | 0.854 | 0.854 |
mag_Deva |
Magahi | 97 | 0.890 | 1.000 | 1.000 | 0.772 | 0.772 |
lim_Latn |
Limburgish | 196 | 0.896 | 1.000 | 1.000 | 0.956 | 0.956 |
run_Latn |
Rundi | 291 | 0.883 | 0.985 | 0.985 | 0.855 | 0.855 |
taq_Latn |
Tamasheq | 106 | 0.902 | 1.000 | 1.000 | 0.838 | 0.838 |
Per-language precision, recall (sensitivity), F1 and specificity for every model are in reports/<set>/.
Training
1,943,957 passages; the best validation macro F1 at step 106,000, after 0.6 hours on 1 GPU. Languages sampled in proportion to n^0.3; 25% of passages cut to a 20 to 80 character snippet; batch 512, AdamW at 0.001, cosine schedule, label smoothing 0.1.
The training data is the full MillID build. The public baobabtech/millid-dataset is that build without Save the Children's and Nal'ibali's text (1.6% of its passages), which is not ours to redistribute. Its splits are the same, so its test split is the one scored above, without those passages.
Licence
The weights and the code are Apache-2.0. The training data carries its own licence row by row (see the dataset card), including text licensed for non-commercial use only (CC BY-NC, CC BY-NC-SA).