Hugging Face's logo

baobabtech
/
millid-tiny-v1.5

Text Classification
PyTorch
ONNX
Safetensors
multilingual
language-identification
byte-level
Eval Results (legacy)
🇪🇺 Region: EU
Model card Files
xet
 Community
5
millid-tiny-v1.5 / README.md
byamasupatrick's picture
byamasupatrick
Model card for the public release
53bd5be verified
preview code
|
Raw History Blame
18.9 kB
metadata 
license: apache-2.0
library_name: pytorch
pipeline_tag: text-classification
tags:
  - language-identification
  - onnx
  - byte-level
datasets:
  - baobabtech/millid-dataset
metrics:
  - f1
  - accuracy
model-index:
  - name: millid-tiny-v1.5
    results:
      - task:
          type: text-classification
          name: Language identification
        dataset:
          name: MillID test split (the full build)
          type: baobabtech/millid-dataset
          split: test
        metrics:
          - type: f1
            name: Macro F1
            value: 0.9554
          - type: accuracy
            name: Top-1 accuracy
            value: 0.9587
      - task:
          type: text-classification
          name: Language identification
        dataset:
          name: UDHR-LID, the languages MillID knows
          type: baobabtech/millid-dataset
          config: udhr
          split: test
        metrics:
          - type: f1
            name: Macro F1
            value: 0.8764
          - type: accuracy
            name: Top-1 accuracy
            value: 0.8492

MillID tiny v1.5: MultiLingual Language IDentifier

A byte-level, encoder-only transformer that names the language of a passage, from a table cell to a paragraph: 1,641 language-script labels (ISO 639-3 + script, e.g. eng_Latn, amh_Ethi). 2.9M parameters, trained from scratch. Apache-2.0.

The MillID models

Three sizes, this one in bold. Macro F1 on test and udhr, the sets below:

Model Parameters ONNX Test F1 UDHR F1 Runs on
MillID tiny v1.5 2.9M 11 MB 0.9554 0.8764 CPU, web page
MillID medium v1 115M 455 MB 0.9576 0.8780 GPU
MillID large v1.5 193M 764 MB 0.9646 0.8812 GPU

All three, language by language, against GlotLID and Meta's fastText: the leaderboard. GlotLID, restricted to MillID's labels, scores 0.9755 and 0.8873.

Use it

No server and no PyTorch: the model runs in your process from onnx/model.onnx, in Python or JavaScript, and the two give the same answers (one encoder, ported line for line and tested against the same fixtures in baobab-tech/millid).

pip install "millid[cpu] @ git+https://github.com/baobab-tech/millid.git"   # [gpu] for CUDA

from millid import MillID

model = MillID.from_pretrained("baobabtech/millid-tiny-v1.5")
model.predict("Lave men ou ak savon anvan ou manje.", k=3)
# [Prediction(label='hat_Latn', language='Haitian Creole', probability=...), ...]
model.predict(["one text", "another"], k=3)  # one list of answers per text
model.identify_document(long_text)           # any length: its languages and segments

millid predict "Lave men ou ak savon anvan ou manje." --model baobabtech/millid-tiny-v1.5 -k 3

JavaScript, in Node or a browser (js/ in the same repository; not on npm yet):

import { MillID } from "@baobabtech/millid";   // "@baobabtech/millid/web" in a browser, with { ort }
const model = await MillID.fromPretrained("baobabtech/millid-tiny-v1.5");
await model.predict("Lave men ou ak savon anvan ou manje.", 3);

millid.py in this repository is the PyTorch training code, which reads model.safetensors to fine-tune or export.

The ONNX graph takes, for each UTF-8 byte of a passage cut to 255 bytes after a CLS token, ids, script and charpos (int64, [batch, bytes]), ngrams (int64, [batch, bytes, 3]) and mask (bool, [batch, bytes]), and returns logits (float32, [batch, labels]). millid.encode_batch and the JavaScript encodeBatch build them; script ids come from a frozen table (Unicode 15.0), so every runtime gives the same ones.

Architecture

UTF-8 bytes (after Unicode NFKC), each embedded as byte + Unicode script + byte-in-character + hashed character 2-, 3- and 4-grams (4,096 buckets) -> depthwise convolution stem (kernel 5) -> strided convolution, 1 bytes per token -> 3 pre-norm blocks (RMSNorm, attention with RoPE, 3 heads, SwiGLU; d=192) -> masked mean pooling -> language head, each language's vector the sum of its own, its language's and its script's. Up to 256 bytes per passage. See ARCHITECTURE.md.

Evaluation against GlotLID and fastText

validation chose the checkpoint; test is split by document from training; udhr is the Universal Declaration of Human Rights, labelled by its translators, over the languages MillID knows, and udhr-all over all of them. GlotLID and Meta's fastText language identifier (NLLB, 218 labels) are each restricted to MillID's labels, their best answer among them: the like-for-like comparison. fastText's answers are mapped onto MillID's spelling (Simplified and Traditional Han as Hani, pes/prs as fas, est as ekk). GlotLID's own labels came from training on data that includes UDHR and labelled FinePDFs and FineWeb-2, so its scores on udhr and on those sources are optimistic. fastText cannot answer most of the 1,600 languages here, so test-common and udhr-common score every system on the passages whose language all of them know: the head-to-head on equal terms.

Macro F1 on each set, every system on the same passages; GlotLID and fastText restricted to MillID's labels:

Set Languages MillID GlotLID fastText
test 1,621 0.9554 0.9755 0.0659
test-common 213 0.9513 0.9846 0.9173
udhr 358 0.8764 0.8873 0.3499
udhr-common 170 0.9513 0.9659 0.9329
udhr-all 429 0.7149 0.7245 0.2797

MillID on each set:

Set Top-1 Top-3 Top-5 Macro P Macro R Macro F1 Median F1
validation 0.9578 0.9889 0.9933 0.9539 0.9591 0.9551 0.9822
test 0.9587 0.9890 0.9932 0.9550 0.9588 0.9554 0.9825
test-common 0.9396 0.9857 0.9909 0.9709 0.9353 0.9513 0.9726
udhr 0.8492 0.9117 0.9257 0.9148 0.8736 0.8764 0.9907
udhr-common 0.9123 0.9606 0.9655 0.9794 0.9393 0.9513 1.0000
udhr-all 0.7274 0.7809 0.7928 0.7340 0.7290 0.7149 0.9649

Top-1 accuracy by passage length:

Set System Short Sentence Paragraph
validation MillID 0.9312 0.9788 0.9850
test MillID 0.9333 0.9797 0.9833
test GlotLID 0.9674 0.9858 0.9912
test fastText 0.2764 0.2506 0.1959
test-common MillID 0.9125 0.9685 0.9725
test-common GlotLID 0.9609 0.9885 0.9949
test-common fastText 0.9125 0.9581 0.9655

Per language: test

F1 language by language, GlotLID and fastText restricted to MillID's labels: against GlotLID, MillID is better on 103 languages, worse on 869 and within 0.01 on 649; against fastText, MillID is better on 1616 languages, worse on 0 and within 0.01 on 5. Every language, with precision, recall and the unrestricted scores, is in reports/test/leaderboard.csv.

The 20 languages with the most passages:

Label Language Passages MillID GlotLID fastText
fra_Latn French 1652 0.929 0.928 0.808
eng_Latn English 1444 0.917 0.859 0.648
spa_Latn Spanish 1136 0.930 0.951 0.574
tam_Taml Tamil 754 0.997 0.993 0.965
arb_Arab Arabic (Modern Standard) 745 0.814 0.785 0.512
por_Latn Portuguese 657 0.947 0.949 0.755
kan_Knda Kannada 601 0.969 0.993 0.922
zul_Latn Zulu 594 0.846 0.896 0.539
swh_Latn Swahili 592 0.848 0.916 0.346
mar_Deva Marathi 589 0.965 0.990 0.926
tso_Latn Tsonga 581 0.965 0.956 0.425
hun_Latn Hungarian 576 0.987 0.999 0.232
afr_Latn Afrikaans 569 0.967 0.985 0.860
tel_Telu Telugu 555 0.991 0.994 0.951
rus_Cyrl Russian 528 0.932 0.949 0.462
nbl_Latn South Ndebele 528 0.903 0.949 0.000
ory_Orya Odia (individual language) 511 0.988 0.994 0.976
sot_Latn Southern Sotho 491 0.966 0.978 0.660
nso_Latn Northern Sotho 479 0.956 0.969 0.914
kua_Latn Kuanyama 476 0.962 0.986 0.000

Where MillID gains most over GlotLID:

Label Language Passages MillID GlotLID fastText
lbj_Tibt Ladakhi 66 0.943 0.000 0.000
ain_Latn Ainu 1 0.667 0.000 0.000
ajp_Arab South Levantine Arabic 28 0.946 0.581 0.000
und_Laoo und 20 0.923 0.621 0.000
und_Thaa und 29 0.881 0.651 0.000
sgh_Cyrl Shughni 9 0.857 0.667 0.000
und_Geor und 32 0.877 0.694 0.000
tuk_Arab Turkmen 135 0.905 0.761 0.000
und_Syrc und 43 0.966 0.822 0.000
gmh_Latn Middle High German 177 0.922 0.786 0.000
orv_Cyrl Old Russian 239 0.931 0.800 0.000
und_Kana und 136 0.982 0.862 0.000
und_Yiii und 38 1.000 0.884 0.000
neb_Latn Toura (Côte d'Ivoire) 11 0.957 0.842 0.000
dag_Latn Dagbani 149 0.767 0.660 0.000

Where MillID loses most to GlotLID:

Label Language Passages MillID GlotLID fastText
mus_Latn Muscogee 2 0.333 1.000 0.000
ttj_Latn Tooro 28 0.485 0.946 0.000
mbd_Latn Dibabawon Manobo 14 0.596 1.000 0.000
knc_Arab Central Kanuri 68 0.462 0.825 0.007
biu_Latn Biete 7 0.667 1.000 0.000
dks_Latn Southeastern Dinka 3 0.667 1.000 0.000
esi_Latn North Alaskan Inupiatun 9 0.455 0.783 0.000
hrv_Latn Croatian 168 0.622 0.944 0.178
plw_Latn Brooke's Point Palawano 8 0.727 1.000 0.000
bos_Latn Bosnian 198 0.649 0.902 0.181
xbr_Latn Kambera 8 0.750 1.000 0.000
nyf_Latn Giryama 10 0.645 0.870 0.000
ldi_Latn Laari 62 0.777 0.992 0.000
hak_Hani Hakka Chinese 29 0.621 0.824 0.000
ccp_Latn Chakma 3 0.800 1.000 0.000

Per language: udhr

F1 language by language, GlotLID and fastText restricted to MillID's labels: against GlotLID, MillID is better on 52 languages, worse on 105 and within 0.01 on 201; against fastText, MillID is better on 303 languages, worse on 15 and within 0.01 on 40. Every language, with precision, recall and the unrestricted scores, is in reports/udhr/leaderboard.csv.

The 20 languages with the most passages:

Label Language Passages MillID GlotLID fastText
roh_Latn Romansh 399 0.994 0.999 0.946
oci_Latn Occitan 398 0.277 0.376 0.435
cmn_Hani Chinese (Mandarin) 319 0.892 0.868 0.468
krl_Latn Karelian 183 0.000 0.011 0.000
qug_Latn Chimborazo Highland Quichua 182 0.011 0.000 0.000
acu_Latn Achuar-Shiwiar 180 0.828 0.854 0.000
hus_Latn Huastec 175 0.977 0.986 0.000
hau_Latn Hausa 155 0.990 1.000 0.876
auc_Latn Waorani 154 0.000 0.000 0.000
kwi_Latn Awa-Cuaiquer 145 0.880 0.877 0.000
jpn_Jpan Japanese 140 1.000 1.000 0.944
snn_Latn Siona 136 0.630 0.514 0.000
ktu_Latn Kituba (Democratic Republic of Congo) 118 0.765 0.680 0.000
lat_Latn Latin 118 0.996 0.992 0.000
hat_Latn Haitian Creole 117 0.992 0.979 0.699
nya_Latn Nyanja 116 0.991 1.000 0.806
kmb_Latn Kimbundu 116 0.970 1.000 0.877
rmn_Latn Balkan Romani 116 0.678 0.857 0.000
twi_Latn Twi 116 0.983 0.996 0.396
chr_Cher Cherokee 115 0.670 0.663 0.000

Where MillID gains most over GlotLID:

Label Language Passages MillID GlotLID fastText
mic_Latn Mi'kmaw 59 0.992 0.427 0.000
tet_Latn Tetum 58 0.638 0.167 0.000
fuf_Latn Pular 58 0.906 0.483 0.000
sag_Latn Sango 59 0.992 0.578 0.189
skr_Arab Saraiki 63 0.867 0.500 0.000
qvh_Latn Huamalíes-Dos de Mayo Huánuco Quechua 58 0.468 0.154 0.000
zgh_Tfng Standard Moroccan Tamazight 60 0.750 0.462 0.000
lld_Latn Ladin 59 0.909 0.731 0.000
tah_Latn Tahitian 57 0.973 0.812 0.966
yue_Hani Cantonese 49 0.734 0.580 0.111
ote_Latn Mezquital Otomi 59 0.154 0.000 0.000
pnb_Arab Western Panjabi 61 0.882 0.739 0.000
mxv_Latn Metlatónoc Mixtec 58 0.129 0.000 0.000
lij_Latn Ligurian 58 0.983 0.859 0.487
bum_Latn Bulu 58 0.215 0.098 0.000

Where MillID loses most to GlotLID:

Label Language Passages MillID GlotLID fastText
gyr_Latn Guarayu 49 0.537 0.990 0.000
shp_Latn Shipibo-Conibo 62 0.469 0.912 0.000
dyu_Latn Dyula 58 0.000 0.434 0.015
tbz_Latn Ditammari 60 0.000 0.378 0.000
cbu_Latn Candoshi-Shapra 57 0.500 0.874 0.000
idu_Latn Idoma 51 0.603 0.938 0.000
ell_Grek Greek 115 0.670 1.000 1.000
qvc_Latn Cajamarca Quechua 58 0.648 0.906 0.000
cpu_Latn Pichis Ashéninka 60 0.729 0.974 0.000
qvn_Latn North Junín Quechua 57 0.219 0.460 0.000
hin_Deva Hindi 59 0.723 0.957 0.975
hrv_Latn Croatian 58 0.500 0.727 0.474
kbp_Latn Kabiyè 59 0.781 1.000 0.389
cni_Latn Asháninka 52 0.773 0.963 0.000
rmn_Latn Balkan Romani 116 0.678 0.857 0.000

Per language: test-common

F1 language by language, GlotLID and fastText restricted to MillID's labels: against GlotLID, MillID is better on 6 languages, worse on 154 and within 0.01 on 53; against fastText, MillID is better on 85 languages, worse on 51 and within 0.01 on 77. Every language, with precision, recall and the unrestricted scores, is in reports/test-common/leaderboard.csv.

The 20 languages with the most passages:

Label Language Passages MillID GlotLID fastText
fra_Latn French 1652 0.943 0.955 0.963
eng_Latn English 1444 0.932 0.877 0.857
spa_Latn Spanish 1136 0.937 0.958 0.941
tam_Taml Tamil 754 0.997 1.000 0.997
arb_Arab Arabic (Modern Standard) 745 0.871 0.886 0.954
por_Latn Portuguese 657 0.959 0.956 0.949
kan_Knda Kannada 601 0.973 0.994 0.999
zul_Latn Zulu 594 0.883 0.904 0.970
swh_Latn Swahili 592 0.893 0.953 0.945
mar_Deva Marathi 589 0.968 0.990 0.987
tso_Latn Tsonga 581 0.979 0.956 0.986
hun_Latn Hungarian 576 0.989 1.000 0.980
afr_Latn Afrikaans 569 0.972 0.985 0.986
tel_Telu Telugu 555 0.997 0.998 0.999
rus_Cyrl Russian 528 0.945 0.985 0.927
ory_Orya Odia (individual language) 511 0.994 1.000 0.996
sot_Latn Southern Sotho 491 0.969 0.978 0.856
nso_Latn Northern Sotho 479 0.958 0.970 0.967
asm_Beng Assamese 467 0.997 0.999 0.997
ben_Beng Bangla 466 0.988 0.997 0.989

Where MillID gains most over GlotLID:

Label Language Passages MillID GlotLID fastText
ewo_Latn Ewondo 134 0.959 0.883 0.860
eng_Latn English 1444 0.932 0.877 0.857
arb_Latn Standard Arabic 163 0.942 0.913 0.852
tso_Latn Tsonga 581 0.979 0.956 0.986
fuv_Latn Nigerian Fulfulde 164 0.907 0.892 0.859
knc_Latn Central Kanuri 293 0.967 0.952 0.981
quy_Latn Ayacucho Quechua 271 0.940 0.933 0.979
kam_Latn Kamba 272 0.980 0.974 0.863
bjn_Arab Banjar 116 0.992 0.987 0.960
por_Latn Portuguese 657 0.959 0.956 0.949
mlt_Latn Maltese 291 0.995 0.993 0.961
hye_Armn Armenian 178 1.000 1.000 0.977
khm_Khmr Khmer 187 1.000 1.000 1.000
guj_Gujr Gujarati 354 1.000 1.000 0.984
mal_Mlym Malayalam 338 1.000 1.000 0.994

Where MillID loses most to GlotLID:

Label Language Passages MillID GlotLID fastText
knc_Arab Central Kanuri 68 0.522 0.860 0.029
hrv_Latn Croatian 168 0.659 0.994 0.856
bos_Latn Bosnian 198 0.685 1.000 0.897
tzm_Tfng Central Atlas Tamazight 157 0.697 0.934 0.751
zsm_Latn Standard Malay 244 0.789 0.998 0.908
taq_Tfng Tamasheq 104 0.654 0.805 0.000
ind_Latn Indonesian 335 0.842 0.991 0.887
kas_Deva Kashmiri 47 0.557 0.703 0.587
cjk_Latn Chokwe 24 0.857 1.000 0.807
nob_Latn Norwegian Bokmål 265 0.821 0.959 0.894
kin_Latn Kinyarwanda 210 0.849 0.981 0.854
mag_Deva Magahi 97 0.890 1.000 0.772
lim_Latn Limburgish 196 0.896 1.000 0.956
run_Latn Rundi 291 0.883 0.985 0.855
taq_Latn Tamasheq 106 0.902 1.000 0.838

Per-language precision, recall (sensitivity), F1 and specificity for every system, restricted and as it is, are in reports/<set>/, and the totals in reports/summary.json.

Training

1,943,957 passages; the best validation macro F1 at step 106,000, after 0.6 hours on 1 GPU. Languages sampled in proportion to n^0.3; 25% of passages cut to a 20 to 80 character snippet; batch 512, AdamW at 0.001, cosine schedule, label smoothing 0.1.

The training data is the full MillID build. The public baobabtech/millid-dataset is that build without Save the Children's and Nal'ibali's text (1.6% of its passages), which is not ours to redistribute. Its splits are the same, so its test split is the one scored above, without those passages.

Licence

The weights and the code are Apache-2.0. The training data carries its own licence row by row (see the dataset card), including text licensed for non-commercial use only (CC BY-NC, CC BY-NC-SA).