| --- |
| license: apache-2.0 |
| library_name: pytorch |
| pipeline_tag: text-classification |
| tags: |
| - language-identification |
| - onnx |
| - byte-level |
| datasets: |
| - baobabtech/millid-dataset |
| metrics: |
| - f1 |
| - accuracy |
| model-index: |
| - name: millid-tiny-v1.5 |
| results: |
| - task: |
| type: text-classification |
| name: Language identification |
| dataset: |
| name: MillID test split (the full build) |
| type: baobabtech/millid-dataset |
| split: test |
| metrics: |
| - type: f1 |
| name: Macro F1 |
| value: 0.9554 |
| - type: accuracy |
| name: Top-1 accuracy |
| value: 0.9587 |
| - task: |
| type: text-classification |
| name: Language identification |
| dataset: |
| name: UDHR-LID, the languages MillID knows |
| type: baobabtech/millid-dataset |
| config: udhr |
| split: test |
| metrics: |
| - type: f1 |
| name: Macro F1 |
| value: 0.8764 |
| - type: accuracy |
| name: Top-1 accuracy |
| value: 0.8492 |
| --- |
| |
| # MillID tiny v1.5: MultiLingual Language IDentifier |
|
|
| A byte-level, encoder-only transformer that names the language of a passage, from a table cell to a paragraph: **1,641 language-script labels** (ISO 639-3 + script, e.g. `eng_Latn`, `amh_Ethi`). 2.9M parameters, trained from scratch. Apache-2.0. |
|
|
| ## The MillID models |
|
|
| Three sizes, this one in bold. Macro F1 on `test` and `udhr`, the sets below: |
|
|
| | Model | Parameters | ONNX | Test F1 | UDHR F1 | Runs on | |
| |---|---|---|---|---|---| |
| | **MillID tiny v1.5** | 2.9M | 11 MB | 0.9554 | 0.8764 | CPU, web page | |
| | [MillID medium v1](https://huggingface.co/baobabtech/millid-medium-v1) | 115M | 455 MB | 0.9576 | 0.8780 | GPU | |
| | [MillID large v1.5](https://huggingface.co/baobabtech/millid-large-v1.5) | 193M | 764 MB | 0.9646 | 0.8812 | GPU | |
|
|
| All three, language by language, against GlotLID and Meta's fastText: the [leaderboard](https://huggingface.co/baobabtech/millid). GlotLID, restricted to MillID's labels, scores 0.9755 and 0.8873. |
|
|
| ## Use it |
|
|
| No server and no PyTorch: the model runs in your process from `onnx/model.onnx`, in Python or JavaScript, and the two give the same answers (one encoder, ported line for line and tested against the same fixtures in [baobab-tech/millid](https://github.com/baobab-tech/millid)). |
|
|
| ```bash |
| pip install "millid[cpu] @ git+https://github.com/baobab-tech/millid.git" # [gpu] for CUDA |
| ``` |
|
|
| ```python |
| from millid import MillID |
| |
| model = MillID.from_pretrained("baobabtech/millid-tiny-v1.5") |
| model.predict("Lave men ou ak savon anvan ou manje.", k=3) |
| # [Prediction(label='hat_Latn', language='Haitian Creole', probability=...), ...] |
| model.predict(["one text", "another"], k=3) # one list of answers per text |
| model.identify_document(long_text) # any length: its languages and segments |
| ``` |
|
|
| ```bash |
| millid predict "Lave men ou ak savon anvan ou manje." --model baobabtech/millid-tiny-v1.5 -k 3 |
| ``` |
|
|
| JavaScript, in Node or a browser (`js/` in the same repository; not on npm yet): |
|
|
| ```js |
| import { MillID } from "@baobabtech/millid"; // "@baobabtech/millid/web" in a browser, with { ort } |
| const model = await MillID.fromPretrained("baobabtech/millid-tiny-v1.5"); |
| await model.predict("Lave men ou ak savon anvan ou manje.", 3); |
| ``` |
|
|
| `millid.py` in this repository is the PyTorch training code, which reads `model.safetensors` to fine-tune or export. |
|
|
| The ONNX graph takes, for each UTF-8 byte of a passage cut to 255 bytes after a CLS token, `ids`, `script` and `charpos` (int64, `[batch, bytes]`), `ngrams` (int64, `[batch, bytes, 3]`) and `mask` (bool, `[batch, bytes]`), and returns `logits` (float32, `[batch, labels]`). `millid.encode_batch` and the JavaScript `encodeBatch` build them; script ids come from a frozen table (Unicode 15.0), so every runtime gives the same ones. |
|
|
| ## Architecture |
|
|
| UTF-8 bytes (after Unicode NFKC), each embedded as byte + Unicode script + byte-in-character + hashed character 2-, 3- and 4-grams (4,096 buckets) -> depthwise convolution stem (kernel 5) -> strided convolution, 1 bytes per token -> 3 pre-norm blocks (RMSNorm, attention with RoPE, 3 heads, SwiGLU; d=192) -> masked mean pooling -> language head, each language's vector the sum of its own, its language's and its script's. Up to 256 bytes per passage. See `ARCHITECTURE.md`. |
|
|
| ## Evaluation against GlotLID and fastText |
|
|
| `validation` chose the checkpoint; `test` is split by document from training; `udhr` is the Universal Declaration of Human Rights, labelled by its translators, over the languages MillID knows, and `udhr-all` over all of them. [GlotLID](https://huggingface.co/cis-lmu/glotlid) and Meta's [fastText language identifier](https://huggingface.co/facebook/fasttext-language-identification) (NLLB, 218 labels) are each restricted to MillID's labels, their best answer among them: the like-for-like comparison. fastText's answers are mapped onto MillID's spelling (Simplified and Traditional Han as `Hani`, `pes`/`prs` as `fas`, `est` as `ekk`). GlotLID's own labels came from training on data that includes UDHR and labelled FinePDFs and FineWeb-2, so its scores on `udhr` and on those sources are optimistic. fastText cannot answer most of the 1,600 languages here, so `test-common` and `udhr-common` score every system on the passages whose language all of them know: the head-to-head on equal terms. |
|
|
| Macro F1 on each set, every system on the same passages; GlotLID and fastText restricted to MillID's labels: |
|
|
| | Set | Languages | MillID | GlotLID | fastText | |
| |---|---|---|---|---| |
| | `test` | 1,621 | 0.9554 | 0.9755 | 0.0659 | |
| | `test-common` | 213 | 0.9513 | 0.9846 | 0.9173 | |
| | `udhr` | 358 | 0.8764 | 0.8873 | 0.3499 | |
| | `udhr-common` | 170 | 0.9513 | 0.9659 | 0.9329 | |
| | `udhr-all` | 429 | 0.7149 | 0.7245 | 0.2797 | |
|
|
| MillID on each set: |
|
|
| | Set | Top-1 | Top-3 | Top-5 | Macro P | Macro R | Macro F1 | Median F1 | |
| |---|---|---|---|---|---|---|---| |
| | `validation` | 0.9578 | 0.9889 | 0.9933 | 0.9539 | 0.9591 | 0.9551 | 0.9822 | |
| | `test` | 0.9587 | 0.9890 | 0.9932 | 0.9550 | 0.9588 | 0.9554 | 0.9825 | |
| | `test-common` | 0.9396 | 0.9857 | 0.9909 | 0.9709 | 0.9353 | 0.9513 | 0.9726 | |
| | `udhr` | 0.8492 | 0.9117 | 0.9257 | 0.9148 | 0.8736 | 0.8764 | 0.9907 | |
| | `udhr-common` | 0.9123 | 0.9606 | 0.9655 | 0.9794 | 0.9393 | 0.9513 | 1.0000 | |
| | `udhr-all` | 0.7274 | 0.7809 | 0.7928 | 0.7340 | 0.7290 | 0.7149 | 0.9649 | |
|
|
| Top-1 accuracy by passage length: |
|
|
| | Set | System | Short | Sentence | Paragraph | |
| |---|---|---|---|---| |
| | `validation` | MillID | 0.9312 | 0.9788 | 0.9850 | |
| | `test` | MillID | 0.9333 | 0.9797 | 0.9833 | |
| | `test` | GlotLID | 0.9674 | 0.9858 | 0.9912 | |
| | `test` | fastText | 0.2764 | 0.2506 | 0.1959 | |
| | `test-common` | MillID | 0.9125 | 0.9685 | 0.9725 | |
| | `test-common` | GlotLID | 0.9609 | 0.9885 | 0.9949 | |
| | `test-common` | fastText | 0.9125 | 0.9581 | 0.9655 | |
|
|
| ### Per language: test |
|
|
| F1 language by language, GlotLID and fastText restricted to MillID's labels: against GlotLID, MillID is better on **103** languages, worse on **869** and within 0.01 on 649; against fastText, MillID is better on **1616** languages, worse on **0** and within 0.01 on 5. Every language, with precision, recall and the unrestricted scores, is in `reports/test/leaderboard.csv`. |
|
|
| The 20 languages with the most passages: |
|
|
| | Label | Language | Passages | MillID | GlotLID | fastText | |
| |---|---|---|---|---|---| |
| | `fra_Latn` | French | 1652 | 0.929 | 0.928 | 0.808 | |
| | `eng_Latn` | English | 1444 | 0.917 | 0.859 | 0.648 | |
| | `spa_Latn` | Spanish | 1136 | 0.930 | 0.951 | 0.574 | |
| | `tam_Taml` | Tamil | 754 | 0.997 | 0.993 | 0.965 | |
| | `arb_Arab` | Arabic (Modern Standard) | 745 | 0.814 | 0.785 | 0.512 | |
| | `por_Latn` | Portuguese | 657 | 0.947 | 0.949 | 0.755 | |
| | `kan_Knda` | Kannada | 601 | 0.969 | 0.993 | 0.922 | |
| | `zul_Latn` | Zulu | 594 | 0.846 | 0.896 | 0.539 | |
| | `swh_Latn` | Swahili | 592 | 0.848 | 0.916 | 0.346 | |
| | `mar_Deva` | Marathi | 589 | 0.965 | 0.990 | 0.926 | |
| | `tso_Latn` | Tsonga | 581 | 0.965 | 0.956 | 0.425 | |
| | `hun_Latn` | Hungarian | 576 | 0.987 | 0.999 | 0.232 | |
| | `afr_Latn` | Afrikaans | 569 | 0.967 | 0.985 | 0.860 | |
| | `tel_Telu` | Telugu | 555 | 0.991 | 0.994 | 0.951 | |
| | `rus_Cyrl` | Russian | 528 | 0.932 | 0.949 | 0.462 | |
| | `nbl_Latn` | South Ndebele | 528 | 0.903 | 0.949 | 0.000 | |
| | `ory_Orya` | Odia (individual language) | 511 | 0.988 | 0.994 | 0.976 | |
| | `sot_Latn` | Southern Sotho | 491 | 0.966 | 0.978 | 0.660 | |
| | `nso_Latn` | Northern Sotho | 479 | 0.956 | 0.969 | 0.914 | |
| | `kua_Latn` | Kuanyama | 476 | 0.962 | 0.986 | 0.000 | |
|
|
| Where MillID gains most over GlotLID: |
|
|
| | Label | Language | Passages | MillID | GlotLID | fastText | |
| |---|---|---|---|---|---| |
| | `lbj_Tibt` | Ladakhi | 66 | 0.943 | 0.000 | 0.000 | |
| | `ain_Latn` | Ainu | 1 | 0.667 | 0.000 | 0.000 | |
| | `ajp_Arab` | South Levantine Arabic | 28 | 0.946 | 0.581 | 0.000 | |
| | `und_Laoo` | und | 20 | 0.923 | 0.621 | 0.000 | |
| | `und_Thaa` | und | 29 | 0.881 | 0.651 | 0.000 | |
| | `sgh_Cyrl` | Shughni | 9 | 0.857 | 0.667 | 0.000 | |
| | `und_Geor` | und | 32 | 0.877 | 0.694 | 0.000 | |
| | `tuk_Arab` | Turkmen | 135 | 0.905 | 0.761 | 0.000 | |
| | `und_Syrc` | und | 43 | 0.966 | 0.822 | 0.000 | |
| | `gmh_Latn` | Middle High German | 177 | 0.922 | 0.786 | 0.000 | |
| | `orv_Cyrl` | Old Russian | 239 | 0.931 | 0.800 | 0.000 | |
| | `und_Kana` | und | 136 | 0.982 | 0.862 | 0.000 | |
| | `und_Yiii` | und | 38 | 1.000 | 0.884 | 0.000 | |
| | `neb_Latn` | Toura (Côte d'Ivoire) | 11 | 0.957 | 0.842 | 0.000 | |
| | `dag_Latn` | Dagbani | 149 | 0.767 | 0.660 | 0.000 | |
|
|
| Where MillID loses most to GlotLID: |
|
|
| | Label | Language | Passages | MillID | GlotLID | fastText | |
| |---|---|---|---|---|---| |
| | `mus_Latn` | Muscogee | 2 | 0.333 | 1.000 | 0.000 | |
| | `ttj_Latn` | Tooro | 28 | 0.485 | 0.946 | 0.000 | |
| | `mbd_Latn` | Dibabawon Manobo | 14 | 0.596 | 1.000 | 0.000 | |
| | `knc_Arab` | Central Kanuri | 68 | 0.462 | 0.825 | 0.007 | |
| | `biu_Latn` | Biete | 7 | 0.667 | 1.000 | 0.000 | |
| | `dks_Latn` | Southeastern Dinka | 3 | 0.667 | 1.000 | 0.000 | |
| | `esi_Latn` | North Alaskan Inupiatun | 9 | 0.455 | 0.783 | 0.000 | |
| | `hrv_Latn` | Croatian | 168 | 0.622 | 0.944 | 0.178 | |
| | `plw_Latn` | Brooke's Point Palawano | 8 | 0.727 | 1.000 | 0.000 | |
| | `bos_Latn` | Bosnian | 198 | 0.649 | 0.902 | 0.181 | |
| | `xbr_Latn` | Kambera | 8 | 0.750 | 1.000 | 0.000 | |
| | `nyf_Latn` | Giryama | 10 | 0.645 | 0.870 | 0.000 | |
| | `ldi_Latn` | Laari | 62 | 0.777 | 0.992 | 0.000 | |
| | `hak_Hani` | Hakka Chinese | 29 | 0.621 | 0.824 | 0.000 | |
| | `ccp_Latn` | Chakma | 3 | 0.800 | 1.000 | 0.000 | |
|
|
| ### Per language: udhr |
|
|
| F1 language by language, GlotLID and fastText restricted to MillID's labels: against GlotLID, MillID is better on **52** languages, worse on **105** and within 0.01 on 201; against fastText, MillID is better on **303** languages, worse on **15** and within 0.01 on 40. Every language, with precision, recall and the unrestricted scores, is in `reports/udhr/leaderboard.csv`. |
|
|
| The 20 languages with the most passages: |
|
|
| | Label | Language | Passages | MillID | GlotLID | fastText | |
| |---|---|---|---|---|---| |
| | `roh_Latn` | Romansh | 399 | 0.994 | 0.999 | 0.946 | |
| | `oci_Latn` | Occitan | 398 | 0.277 | 0.376 | 0.435 | |
| | `cmn_Hani` | Chinese (Mandarin) | 319 | 0.892 | 0.868 | 0.468 | |
| | `krl_Latn` | Karelian | 183 | 0.000 | 0.011 | 0.000 | |
| | `qug_Latn` | Chimborazo Highland Quichua | 182 | 0.011 | 0.000 | 0.000 | |
| | `acu_Latn` | Achuar-Shiwiar | 180 | 0.828 | 0.854 | 0.000 | |
| | `hus_Latn` | Huastec | 175 | 0.977 | 0.986 | 0.000 | |
| | `hau_Latn` | Hausa | 155 | 0.990 | 1.000 | 0.876 | |
| | `auc_Latn` | Waorani | 154 | 0.000 | 0.000 | 0.000 | |
| | `kwi_Latn` | Awa-Cuaiquer | 145 | 0.880 | 0.877 | 0.000 | |
| | `jpn_Jpan` | Japanese | 140 | 1.000 | 1.000 | 0.944 | |
| | `snn_Latn` | Siona | 136 | 0.630 | 0.514 | 0.000 | |
| | `ktu_Latn` | Kituba (Democratic Republic of Congo) | 118 | 0.765 | 0.680 | 0.000 | |
| | `lat_Latn` | Latin | 118 | 0.996 | 0.992 | 0.000 | |
| | `hat_Latn` | Haitian Creole | 117 | 0.992 | 0.979 | 0.699 | |
| | `nya_Latn` | Nyanja | 116 | 0.991 | 1.000 | 0.806 | |
| | `kmb_Latn` | Kimbundu | 116 | 0.970 | 1.000 | 0.877 | |
| | `rmn_Latn` | Balkan Romani | 116 | 0.678 | 0.857 | 0.000 | |
| | `twi_Latn` | Twi | 116 | 0.983 | 0.996 | 0.396 | |
| | `chr_Cher` | Cherokee | 115 | 0.670 | 0.663 | 0.000 | |
|
|
| Where MillID gains most over GlotLID: |
|
|
| | Label | Language | Passages | MillID | GlotLID | fastText | |
| |---|---|---|---|---|---| |
| | `mic_Latn` | Mi'kmaw | 59 | 0.992 | 0.427 | 0.000 | |
| | `tet_Latn` | Tetum | 58 | 0.638 | 0.167 | 0.000 | |
| | `fuf_Latn` | Pular | 58 | 0.906 | 0.483 | 0.000 | |
| | `sag_Latn` | Sango | 59 | 0.992 | 0.578 | 0.189 | |
| | `skr_Arab` | Saraiki | 63 | 0.867 | 0.500 | 0.000 | |
| | `qvh_Latn` | Huamalíes-Dos de Mayo Huánuco Quechua | 58 | 0.468 | 0.154 | 0.000 | |
| | `zgh_Tfng` | Standard Moroccan Tamazight | 60 | 0.750 | 0.462 | 0.000 | |
| | `lld_Latn` | Ladin | 59 | 0.909 | 0.731 | 0.000 | |
| | `tah_Latn` | Tahitian | 57 | 0.973 | 0.812 | 0.966 | |
| | `yue_Hani` | Cantonese | 49 | 0.734 | 0.580 | 0.111 | |
| | `ote_Latn` | Mezquital Otomi | 59 | 0.154 | 0.000 | 0.000 | |
| | `pnb_Arab` | Western Panjabi | 61 | 0.882 | 0.739 | 0.000 | |
| | `mxv_Latn` | Metlatónoc Mixtec | 58 | 0.129 | 0.000 | 0.000 | |
| | `lij_Latn` | Ligurian | 58 | 0.983 | 0.859 | 0.487 | |
| | `bum_Latn` | Bulu | 58 | 0.215 | 0.098 | 0.000 | |
|
|
| Where MillID loses most to GlotLID: |
|
|
| | Label | Language | Passages | MillID | GlotLID | fastText | |
| |---|---|---|---|---|---| |
| | `gyr_Latn` | Guarayu | 49 | 0.537 | 0.990 | 0.000 | |
| | `shp_Latn` | Shipibo-Conibo | 62 | 0.469 | 0.912 | 0.000 | |
| | `dyu_Latn` | Dyula | 58 | 0.000 | 0.434 | 0.015 | |
| | `tbz_Latn` | Ditammari | 60 | 0.000 | 0.378 | 0.000 | |
| | `cbu_Latn` | Candoshi-Shapra | 57 | 0.500 | 0.874 | 0.000 | |
| | `idu_Latn` | Idoma | 51 | 0.603 | 0.938 | 0.000 | |
| | `ell_Grek` | Greek | 115 | 0.670 | 1.000 | 1.000 | |
| | `qvc_Latn` | Cajamarca Quechua | 58 | 0.648 | 0.906 | 0.000 | |
| | `cpu_Latn` | Pichis Ashéninka | 60 | 0.729 | 0.974 | 0.000 | |
| | `qvn_Latn` | North Junín Quechua | 57 | 0.219 | 0.460 | 0.000 | |
| | `hin_Deva` | Hindi | 59 | 0.723 | 0.957 | 0.975 | |
| | `hrv_Latn` | Croatian | 58 | 0.500 | 0.727 | 0.474 | |
| | `kbp_Latn` | Kabiyè | 59 | 0.781 | 1.000 | 0.389 | |
| | `cni_Latn` | Asháninka | 52 | 0.773 | 0.963 | 0.000 | |
| | `rmn_Latn` | Balkan Romani | 116 | 0.678 | 0.857 | 0.000 | |
|
|
| ### Per language: test-common |
|
|
| F1 language by language, GlotLID and fastText restricted to MillID's labels: against GlotLID, MillID is better on **6** languages, worse on **154** and within 0.01 on 53; against fastText, MillID is better on **85** languages, worse on **51** and within 0.01 on 77. Every language, with precision, recall and the unrestricted scores, is in `reports/test-common/leaderboard.csv`. |
|
|
| The 20 languages with the most passages: |
|
|
| | Label | Language | Passages | MillID | GlotLID | fastText | |
| |---|---|---|---|---|---| |
| | `fra_Latn` | French | 1652 | 0.943 | 0.955 | 0.963 | |
| | `eng_Latn` | English | 1444 | 0.932 | 0.877 | 0.857 | |
| | `spa_Latn` | Spanish | 1136 | 0.937 | 0.958 | 0.941 | |
| | `tam_Taml` | Tamil | 754 | 0.997 | 1.000 | 0.997 | |
| | `arb_Arab` | Arabic (Modern Standard) | 745 | 0.871 | 0.886 | 0.954 | |
| | `por_Latn` | Portuguese | 657 | 0.959 | 0.956 | 0.949 | |
| | `kan_Knda` | Kannada | 601 | 0.973 | 0.994 | 0.999 | |
| | `zul_Latn` | Zulu | 594 | 0.883 | 0.904 | 0.970 | |
| | `swh_Latn` | Swahili | 592 | 0.893 | 0.953 | 0.945 | |
| | `mar_Deva` | Marathi | 589 | 0.968 | 0.990 | 0.987 | |
| | `tso_Latn` | Tsonga | 581 | 0.979 | 0.956 | 0.986 | |
| | `hun_Latn` | Hungarian | 576 | 0.989 | 1.000 | 0.980 | |
| | `afr_Latn` | Afrikaans | 569 | 0.972 | 0.985 | 0.986 | |
| | `tel_Telu` | Telugu | 555 | 0.997 | 0.998 | 0.999 | |
| | `rus_Cyrl` | Russian | 528 | 0.945 | 0.985 | 0.927 | |
| | `ory_Orya` | Odia (individual language) | 511 | 0.994 | 1.000 | 0.996 | |
| | `sot_Latn` | Southern Sotho | 491 | 0.969 | 0.978 | 0.856 | |
| | `nso_Latn` | Northern Sotho | 479 | 0.958 | 0.970 | 0.967 | |
| | `asm_Beng` | Assamese | 467 | 0.997 | 0.999 | 0.997 | |
| | `ben_Beng` | Bangla | 466 | 0.988 | 0.997 | 0.989 | |
|
|
| Where MillID gains most over GlotLID: |
|
|
| | Label | Language | Passages | MillID | GlotLID | fastText | |
| |---|---|---|---|---|---| |
| | `ewo_Latn` | Ewondo | 134 | 0.959 | 0.883 | 0.860 | |
| | `eng_Latn` | English | 1444 | 0.932 | 0.877 | 0.857 | |
| | `arb_Latn` | Standard Arabic | 163 | 0.942 | 0.913 | 0.852 | |
| | `tso_Latn` | Tsonga | 581 | 0.979 | 0.956 | 0.986 | |
| | `fuv_Latn` | Nigerian Fulfulde | 164 | 0.907 | 0.892 | 0.859 | |
| | `knc_Latn` | Central Kanuri | 293 | 0.967 | 0.952 | 0.981 | |
| | `quy_Latn` | Ayacucho Quechua | 271 | 0.940 | 0.933 | 0.979 | |
| | `kam_Latn` | Kamba | 272 | 0.980 | 0.974 | 0.863 | |
| | `bjn_Arab` | Banjar | 116 | 0.992 | 0.987 | 0.960 | |
| | `por_Latn` | Portuguese | 657 | 0.959 | 0.956 | 0.949 | |
| | `mlt_Latn` | Maltese | 291 | 0.995 | 0.993 | 0.961 | |
| | `hye_Armn` | Armenian | 178 | 1.000 | 1.000 | 0.977 | |
| | `khm_Khmr` | Khmer | 187 | 1.000 | 1.000 | 1.000 | |
| | `guj_Gujr` | Gujarati | 354 | 1.000 | 1.000 | 0.984 | |
| | `mal_Mlym` | Malayalam | 338 | 1.000 | 1.000 | 0.994 | |
|
|
| Where MillID loses most to GlotLID: |
|
|
| | Label | Language | Passages | MillID | GlotLID | fastText | |
| |---|---|---|---|---|---| |
| | `knc_Arab` | Central Kanuri | 68 | 0.522 | 0.860 | 0.029 | |
| | `hrv_Latn` | Croatian | 168 | 0.659 | 0.994 | 0.856 | |
| | `bos_Latn` | Bosnian | 198 | 0.685 | 1.000 | 0.897 | |
| | `tzm_Tfng` | Central Atlas Tamazight | 157 | 0.697 | 0.934 | 0.751 | |
| | `zsm_Latn` | Standard Malay | 244 | 0.789 | 0.998 | 0.908 | |
| | `taq_Tfng` | Tamasheq | 104 | 0.654 | 0.805 | 0.000 | |
| | `ind_Latn` | Indonesian | 335 | 0.842 | 0.991 | 0.887 | |
| | `kas_Deva` | Kashmiri | 47 | 0.557 | 0.703 | 0.587 | |
| | `cjk_Latn` | Chokwe | 24 | 0.857 | 1.000 | 0.807 | |
| | `nob_Latn` | Norwegian Bokmål | 265 | 0.821 | 0.959 | 0.894 | |
| | `kin_Latn` | Kinyarwanda | 210 | 0.849 | 0.981 | 0.854 | |
| | `mag_Deva` | Magahi | 97 | 0.890 | 1.000 | 0.772 | |
| | `lim_Latn` | Limburgish | 196 | 0.896 | 1.000 | 0.956 | |
| | `run_Latn` | Rundi | 291 | 0.883 | 0.985 | 0.855 | |
| | `taq_Latn` | Tamasheq | 106 | 0.902 | 1.000 | 0.838 | |
|
|
| Per-language precision, recall (sensitivity), F1 and specificity for every system, restricted and as it is, are in `reports/<set>/`, and the totals in `reports/summary.json`. |
|
|
| ## Training |
|
|
| 1,943,957 passages; the best validation macro F1 at step 106,000, after 0.6 hours on 1 GPU. Languages sampled in proportion to n^0.3; 25% of passages cut to a 20 to 80 character snippet; batch 512, AdamW at 0.001, cosine schedule, label smoothing 0.1. |
|
|
| The training data is the full MillID build. The public [baobabtech/millid-dataset](https://huggingface.co/datasets/baobabtech/millid-dataset) is that build without Save the Children's and Nal'ibali's text (1.6% of its passages), which is not ours to redistribute. Its splits are the same, so its `test` split is the one scored above, without those passages. |
|
|
| ## Licence |
|
|
| The weights and the code are Apache-2.0. The training data carries its own licence row by row (see the [dataset card](https://huggingface.co/datasets/baobabtech/millid-dataset#licensing)), including text licensed for non-commercial use only (CC BY-NC, CC BY-NC-SA). |
|
|
