Model card: run it without a server, in Python or JavaScript

ARCHITECTURE.md

@@ -204,7 +204,7 @@ flowchart LR
   is (α = 1), where the largest languages fill every batch, and uniform (α = 0), where a
   100-passage language is seen until memorised. XLM-R and mT5 sample their languages the same way.
 - **How long.** A quarter of passages are cut to a 20 to 80 character snippet, on word boundaries
-  where the script has them: short text is the hard case, and what a document pipeline meets in
+  where the script has them: short text is the hard case, and what a document holds in its
   headers, captions and table cells.
 - **Why macro F1 chooses.** It weighs every language equally, the score a thousand-language model
   should be judged on. Top-1 accuracy, which the largest languages dominate, is reported beside it.

README.md

@@ -1,44 +1,105 @@
 ---
+license: apache-2.0
 library_name: pytorch
 pipeline_tag: text-classification
 tags:
 - language-identification
+- onnx
+- byte-level
 datasets:
 - baobabtech/millid-dataset
+metrics:
+- f1
+- accuracy
+model-index:
+- name: millid-tiny-v1.5
+  results:
+  - task:
+      type: text-classification
+      name: Language identification
+    dataset:
+      name: MillID test split (the full build)
+      type: baobabtech/millid-dataset
+      split: test
+    metrics:
+    - type: f1
+      name: Macro F1
+      value: 0.9554
+    - type: accuracy
+      name: Top-1 accuracy
+      value: 0.9587
+  - task:
+      type: text-classification
+      name: Language identification
+    dataset:
+      name: UDHR-LID, the languages MillID knows
+      type: baobabtech/millid-dataset
+      config: udhr
+      split: test
+    metrics:
+    - type: f1
+      name: Macro F1
+      value: 0.8764
+    - type: accuracy
+      name: Top-1 accuracy
+      value: 0.8492
 ---
 
-# MillID tiny: MultiLingual Language IDentifier
+# MillID tiny v1.5: MultiLingual Language IDentifier
 
-A byte-level, encoder-only transformer that names the language of a passage: **1641 language-script
-labels** (ISO 639-3 + script, e.g. `eng_Latn`, `amh_Ethi`). 2.9M parameters, trained from
-scratch on [baobabtech/millid-dataset](https://huggingface.co/datasets/baobabtech/millid-dataset).
+A byte-level, encoder-only transformer that names the language of a passage, from a table cell to a paragraph: **1,641 language-script labels** (ISO 639-3 + script, e.g. `eng_Latn`, `amh_Ethi`). 2.9M parameters, trained from scratch. Apache-2.0.
+
+## The MillID models
+
+| Model | Size | Macro F1, `test` / `udhr` | Runs best on |
+|---|---|---|---|
+| **MillID tiny v1.5** (this model) | 2.9M parameters, ONNX 11 MB | 0.9554 / 0.8764 | a CPU, or a web page |
+| [MillID medium v1](https://huggingface.co/baobabtech/millid-medium-v1) | 115M parameters, ONNX 455 MB | 0.9576 / 0.8780 | a GPU |
+| [MillID large v1.5](https://huggingface.co/baobabtech/millid-large-v1.5) | 193M parameters, ONNX 764 MB | 0.9646 / 0.8812 | a GPU: the most accurate |
+
+All three, language by language, against GlotLID and Meta's fastText: the [leaderboard](https://huggingface.co/baobabtech/millid). GlotLID, restricted to MillID's labels, scores 0.9755 / 0.8873 on the same sets.
+
+## Use it
+
+No server and no PyTorch: the model runs in your process from `onnx/model.onnx`, in Python or JavaScript, and the two give the same answers (one encoder, ported line for line and tested against the same fixtures in [baobab-tech/millid](https://github.com/baobab-tech/millid)).
+
+```bash
+pip install "millid[cpu] @ git+https://github.com/baobab-tech/millid.git"   # [gpu] for CUDA
+```
 
 ```python
-from millid import MillID   # millid.py, in this repository
+from millid import MillID
 
 model = MillID.from_pretrained("baobabtech/millid-tiny-v1.5")
-model.predict(["Lave men ou ak savon anvan ou manje."], k=3)
+model.predict("Lave men ou ak savon anvan ou manje.", k=3)
+# [Prediction(label='hat_Latn', language='Haitian Creole', probability=...), ...]
+model.predict(["one text", "another"], k=3)  # one list of answers per text
+model.identify_document(long_text)           # any length: its languages and segments
+```
+
+```bash
+millid predict "Lave men ou ak savon anvan ou manje." --model baobabtech/millid-tiny-v1.5 -k 3
 ```
 
+JavaScript, in Node or a browser (`js/` in the same repository; not on npm yet):
+
+```js
+import { MillID } from "@baobabtech/millid";   // "@baobabtech/millid/web" in a browser, with { ort }
+const model = await MillID.fromPretrained("baobabtech/millid-tiny-v1.5");
+await model.predict("Lave men ou ak savon anvan ou manje.", 3);
+```
+
+`millid.py` in this repository is the PyTorch training code, which reads `model.safetensors` to fine-tune or export.
+
+The ONNX graph takes, for each UTF-8 byte of a passage cut to 255 bytes after a CLS token, `ids`, `script` and `charpos` (int64, `[batch, bytes]`), `ngrams` (int64, `[batch, bytes, 3]`) and `mask` (bool, `[batch, bytes]`), and returns `logits` (float32, `[batch, labels]`). `millid.encode_batch` and the JavaScript `encodeBatch` build them; script ids come from a frozen table (Unicode 15.0), so every runtime gives the same ones.
+
 ## Architecture
 
-UTF-8 bytes (after Unicode NFKC), each embedded as byte + Unicode script + byte-in-character + hashed character
-2-, 3- and 4-grams (4,096 buckets) -> depthwise convolution stem (kernel 5) -> strided
-convolution, 1 bytes per token -> 3 pre-norm blocks (RMSNorm, attention with RoPE,
-3 heads, SwiGLU; d=192) -> masked mean pooling -> language head, each language's vector the sum of
-its own, its language's and its script's. Up to 256 bytes per passage. See `ARCHITECTURE.md`.
+UTF-8 bytes (after Unicode NFKC), each embedded as byte + Unicode script + byte-in-character + hashed character 2-, 3- and 4-grams (4,096 buckets) -> depthwise convolution stem (kernel 5) -> strided convolution, 1 bytes per token -> 3 pre-norm blocks (RMSNorm, attention with RoPE, 3 heads, SwiGLU; d=192) -> masked mean pooling -> language head, each language's vector the sum of its own, its language's and its script's. Up to 256 bytes per passage. See `ARCHITECTURE.md`.
 
 ## Evaluation against GlotLID and fastText
 
-`validation` chose the checkpoint; `test` is split by document from training; `udhr` is the Universal Declaration
-of Human Rights, labelled by its translators, over the languages MillID knows, and `udhr-all` over all of them.
-`glotlid` is [GlotLID](https://huggingface.co/cis-lmu/glotlid) as it is; `glotlid@millid` is GlotLID restricted to
-MillID's labels, the like-for-like comparison. GlotLID's own labels came from training on data that includes UDHR
-and labelled FinePDFs and FineWeb-2, so its scores on `udhr` and on those sources are optimistic. `fasttext` is
-Meta's [fastText language identifier](https://huggingface.co/facebook/fasttext-language-identification) (NLLB,
-218 labels), its answers mapped onto MillID's spelling (Simplified and Traditional Han as `Hani`, `pes`/`prs` as
-`fas`, `est` as `ekk`). It cannot answer most of the 1,600 languages here, so `test-common` and `udhr-common`
-score every model on the passages whose language all of them know: the head-to-head on equal terms.
+`validation` chose the checkpoint; `test` is split by document from training; `udhr` is the Universal Declaration of Human Rights, labelled by its translators, over the languages MillID knows, and `udhr-all` over all of them. `glotlid` is [GlotLID](https://huggingface.co/cis-lmu/glotlid) as it is; `glotlid@millid` is GlotLID restricted to MillID's labels, the like-for-like comparison. GlotLID's own labels came from training on data that includes UDHR and labelled FinePDFs and FineWeb-2, so its scores on `udhr` and on those sources are optimistic. `fasttext` is Meta's [fastText language identifier](https://huggingface.co/facebook/fasttext-language-identification) (NLLB, 218 labels), its answers mapped onto MillID's spelling (Simplified and Traditional Han as `Hani`, `pes`/`prs` as `fas`, `est` as `ekk`). It cannot answer most of the 1,600 languages here, so `test-common` and `udhr-common` score every model on the passages whose language all of them know: the head-to-head on equal terms.
 
 | Set | Model | Top-1 | Top-3 | Top-5 | Macro P | Macro R | Macro F1 | Median F1 | Languages |
 |---|---|---|---|---|---|---|---|---|---|
@@ -87,8 +148,7 @@ Top-1 accuracy by passage length:
 
 ### Leaderboard: test
 
-MillID's F1 against the others', language by language: against GlotLID restricted to MillID's labels, MillID is better on **103** languages, worse on **869** and within 0.01 on 649; against fastText restricted to MillID's labels, MillID is better on **1616** languages, worse on **0** and within 0.01 on 5. The full table, with precision and recall, is
-`reports/test/leaderboard.csv`.
+MillID's F1 against the others', language by language: against GlotLID restricted to MillID's labels, MillID is better on **103** languages, worse on **869** and within 0.01 on 649; against fastText restricted to MillID's labels, MillID is better on **1616** languages, worse on **0** and within 0.01 on 5. The full table, with precision and recall, is `reports/test/leaderboard.csv`.
 
 The 20 languages with the most passages:
 
@@ -96,15 +156,15 @@ The 20 languages with the most passages:
 |---|---|---|---|---|---|---|---|
 | `fra_Latn` | French | 1652 | 0.929 | 0.928 | 0.928 | 0.808 | 0.808 |
 | `eng_Latn` | English | 1444 | 0.917 | 0.858 | 0.859 | 0.653 | 0.648 |
-| `spa_Latn` | Spanish | 1136 | 0.931 | 0.950 | 0.950 | 0.574 | 0.574 |
+| `spa_Latn` | Spanish | 1136 | 0.930 | 0.951 | 0.951 | 0.574 | 0.574 |
 | `tam_Taml` | Tamil | 754 | 0.997 | 0.993 | 0.993 | 0.965 | 0.965 |
 | `arb_Arab` | Arabic (Modern Standard) | 745 | 0.814 | 0.785 | 0.785 | 0.513 | 0.512 |
 | `por_Latn` | Portuguese | 657 | 0.947 | 0.947 | 0.949 | 0.756 | 0.755 |
 | `kan_Knda` | Kannada | 601 | 0.969 | 0.993 | 0.993 | 0.922 | 0.922 |
 | `zul_Latn` | Zulu | 594 | 0.846 | 0.896 | 0.896 | 0.539 | 0.539 |
-| `swh_Latn` | Swahili | 592 | 0.848 | 0.918 | 0.916 | 0.346 | 0.346 |
+| `swh_Latn` | Swahili | 592 | 0.848 | 0.917 | 0.916 | 0.346 | 0.346 |
 | `mar_Deva` | Marathi | 589 | 0.965 | 0.989 | 0.990 | 0.926 | 0.926 |
-| `tso_Latn` | Tsonga | 581 | 0.965 | 0.956 | 0.956 | 0.426 | 0.426 |
+| `tso_Latn` | Tsonga | 581 | 0.965 | 0.956 | 0.956 | 0.425 | 0.425 |
 | `hun_Latn` | Hungarian | 576 | 0.987 | 0.999 | 0.999 | 0.232 | 0.232 |
 | `afr_Latn` | Afrikaans | 569 | 0.967 | 0.985 | 0.985 | 0.861 | 0.860 |
 | `tel_Telu` | Telugu | 555 | 0.991 | 0.994 | 0.994 | 0.951 | 0.951 |
@@ -112,7 +172,7 @@ The 20 languages with the most passages:
 | `nbl_Latn` | South Ndebele | 528 | 0.903 | 0.948 | 0.949 | 0.000 | 0.000 |
 | `ory_Orya` | Odia (individual language) | 511 | 0.988 | 0.991 | 0.994 | 0.976 | 0.976 |
 | `sot_Latn` | Southern Sotho | 491 | 0.966 | 0.977 | 0.978 | 0.660 | 0.660 |
-| `nso_Latn` | Northern Sotho | 479 | 0.956 | 0.968 | 0.968 | 0.915 | 0.914 |
+| `nso_Latn` | Northern Sotho | 479 | 0.956 | 0.969 | 0.969 | 0.915 | 0.914 |
 | `kua_Latn` | Kuanyama | 476 | 0.962 | 0.986 | 0.986 | 0.000 | 0.000 |
 
 Where MillID gains most over GlotLID:
@@ -121,7 +181,7 @@ Where MillID gains most over GlotLID:
 |---|---|---|---|---|---|---|---|
 | `lbj_Tibt` | Ladakhi | 66 | 0.943 | 0.000 | 0.000 | 0.000 | 0.000 |
 | `ain_Latn` | Ainu | 1 | 0.667 | 0.000 | 0.000 | 0.000 | 0.000 |
-| `ajp_Arab` | South Levantine Arabic | 28 | 0.945 | 0.581 | 0.581 | 0.000 | 0.000 |
+| `ajp_Arab` | South Levantine Arabic | 28 | 0.946 | 0.581 | 0.581 | 0.000 | 0.000 |
 | `und_Laoo` | und | 20 | 0.923 | 0.621 | 0.621 | 0.000 | 0.000 |
 | `und_Thaa` | und | 29 | 0.881 | 0.651 | 0.651 | 0.000 | 0.000 |
 | `sgh_Cyrl` | Shughni | 9 | 0.857 | 0.667 | 0.667 | 0.000 | 0.000 |
@@ -140,7 +200,7 @@ Where MillID loses most to GlotLID:
 | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
 |---|---|---|---|---|---|---|---|
 | `mus_Latn` | Muscogee | 2 | 0.333 | 1.000 | 1.000 | 0.000 | 0.000 |
-| `ttj_Latn` | Tooro | 28 | 0.485 | 0.945 | 0.945 | 0.000 | 0.000 |
+| `ttj_Latn` | Tooro | 28 | 0.485 | 0.946 | 0.946 | 0.000 | 0.000 |
 | `mbd_Latn` | Dibabawon Manobo | 14 | 0.596 | 1.000 | 1.000 | 0.000 | 0.000 |
 | `knc_Arab` | Central Kanuri | 68 | 0.462 | 0.823 | 0.825 | 0.007 | 0.007 |
 | `biu_Latn` | Biete | 7 | 0.667 | 1.000 | 1.000 | 0.000 | 0.000 |
@@ -157,15 +217,14 @@ Where MillID loses most to GlotLID:
 
 ### Leaderboard: udhr
 
-MillID's F1 against the others', language by language: against GlotLID restricted to MillID's labels, MillID is better on **52** languages, worse on **105** and within 0.01 on 201; against fastText restricted to MillID's labels, MillID is better on **303** languages, worse on **15** and within 0.01 on 40. The full table, with precision and recall, is
-`reports/udhr/leaderboard.csv`.
+MillID's F1 against the others', language by language: against GlotLID restricted to MillID's labels, MillID is better on **52** languages, worse on **105** and within 0.01 on 201; against fastText restricted to MillID's labels, MillID is better on **303** languages, worse on **15** and within 0.01 on 40. The full table, with precision and recall, is `reports/udhr/leaderboard.csv`.
 
 The 20 languages with the most passages:
 
 | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
 |---|---|---|---|---|---|---|---|
-| `roh_Latn` | Romansh | 399 | 0.994 | 0.999 | 0.999 | 0.945 | 0.945 |
-| `oci_Latn` | Occitan | 398 | 0.277 | 0.376 | 0.376 | 0.436 | 0.436 |
+| `roh_Latn` | Romansh | 399 | 0.994 | 0.999 | 0.999 | 0.946 | 0.946 |
+| `oci_Latn` | Occitan | 398 | 0.277 | 0.376 | 0.376 | 0.435 | 0.435 |
 | `cmn_Hani` | Chinese (Mandarin) | 319 | 0.892 | 0.868 | 0.868 | 0.468 | 0.468 |
 | `krl_Latn` | Karelian | 183 | 0.000 | 0.011 | 0.011 | 0.000 | 0.000 |
 | `qug_Latn` | Chimborazo Highland Quichua | 182 | 0.011 | 0.000 | 0.000 | 0.000 | 0.000 |
@@ -175,29 +234,29 @@ The 20 languages with the most passages:
 | `auc_Latn` | Waorani | 154 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
 | `kwi_Latn` | Awa-Cuaiquer | 145 | 0.880 | 0.877 | 0.877 | 0.000 | 0.000 |
 | `jpn_Jpan` | Japanese | 140 | 1.000 | 1.000 | 1.000 | 0.944 | 0.944 |
-| `snn_Latn` | Siona | 136 | 0.631 | 0.514 | 0.514 | 0.000 | 0.000 |
+| `snn_Latn` | Siona | 136 | 0.630 | 0.514 | 0.514 | 0.000 | 0.000 |
 | `ktu_Latn` | Kituba (Democratic Republic of Congo) | 118 | 0.765 | 0.680 | 0.680 | 0.000 | 0.000 |
-| `lat_Latn` | Latin | 118 | 0.996 | 0.991 | 0.991 | 0.000 | 0.000 |
-| `hat_Latn` | Haitian Creole | 117 | 0.991 | 0.979 | 0.979 | 0.699 | 0.699 |
+| `lat_Latn` | Latin | 118 | 0.996 | 0.992 | 0.992 | 0.000 | 0.000 |
+| `hat_Latn` | Haitian Creole | 117 | 0.992 | 0.979 | 0.979 | 0.699 | 0.699 |
 | `nya_Latn` | Nyanja | 116 | 0.991 | 1.000 | 1.000 | 0.806 | 0.806 |
 | `kmb_Latn` | Kimbundu | 116 | 0.970 | 1.000 | 1.000 | 0.877 | 0.877 |
 | `rmn_Latn` | Balkan Romani | 116 | 0.678 | 0.857 | 0.857 | 0.000 | 0.000 |
 | `twi_Latn` | Twi | 116 | 0.983 | 0.996 | 0.996 | 0.399 | 0.396 |
-| `chr_Cher` | Cherokee | 115 | 0.671 | 0.663 | 0.663 | 0.000 | 0.000 |
+| `chr_Cher` | Cherokee | 115 | 0.670 | 0.663 | 0.663 | 0.000 | 0.000 |
 
 Where MillID gains most over GlotLID:
 
 | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
 |---|---|---|---|---|---|---|---|
-| `mic_Latn` | Mi'kmaw | 59 | 0.991 | 0.427 | 0.427 | 0.000 | 0.000 |
+| `mic_Latn` | Mi'kmaw | 59 | 0.992 | 0.427 | 0.427 | 0.000 | 0.000 |
 | `tet_Latn` | Tetum | 58 | 0.638 | 0.167 | 0.167 | 0.000 | 0.000 |
 | `fuf_Latn` | Pular | 58 | 0.906 | 0.483 | 0.483 | 0.000 | 0.000 |
-| `sag_Latn` | Sango | 59 | 0.991 | 0.578 | 0.578 | 0.189 | 0.189 |
+| `sag_Latn` | Sango | 59 | 0.992 | 0.578 | 0.578 | 0.189 | 0.189 |
 | `skr_Arab` | Saraiki | 63 | 0.867 | 0.500 | 0.500 | 0.000 | 0.000 |
 | `qvh_Latn` | Huamalíes-Dos de Mayo Huánuco Quechua | 58 | 0.468 | 0.154 | 0.154 | 0.000 | 0.000 |
 | `zgh_Tfng` | Standard Moroccan Tamazight | 60 | 0.750 | 0.462 | 0.462 | 0.000 | 0.000 |
 | `lld_Latn` | Ladin | 59 | 0.909 | 0.731 | 0.731 | 0.000 | 0.000 |
-| `tah_Latn` | Tahitian | 57 | 0.973 | 0.813 | 0.813 | 0.966 | 0.966 |
+| `tah_Latn` | Tahitian | 57 | 0.973 | 0.812 | 0.812 | 0.966 | 0.966 |
 | `yue_Hani` | Cantonese | 49 | 0.734 | 0.580 | 0.580 | 0.111 | 0.111 |
 | `ote_Latn` | Mezquital Otomi | 59 | 0.154 | 0.000 | 0.000 | 0.000 | 0.000 |
 | `pnb_Arab` | Western Panjabi | 61 | 0.882 | 0.739 | 0.739 | 0.000 | 0.000 |
@@ -215,10 +274,10 @@ Where MillID loses most to GlotLID:
 | `tbz_Latn` | Ditammari | 60 | 0.000 | 0.378 | 0.378 | 0.000 | 0.000 |
 | `cbu_Latn` | Candoshi-Shapra | 57 | 0.500 | 0.882 | 0.874 | 0.000 | 0.000 |
 | `idu_Latn` | Idoma | 51 | 0.603 | 0.879 | 0.938 | 0.000 | 0.000 |
-| `ell_Grek` | Greek | 115 | 0.671 | 1.000 | 1.000 | 1.000 | 1.000 |
+| `ell_Grek` | Greek | 115 | 0.670 | 1.000 | 1.000 | 1.000 | 1.000 |
 | `qvc_Latn` | Cajamarca Quechua | 58 | 0.648 | 0.906 | 0.906 | 0.000 | 0.000 |
 | `cpu_Latn` | Pichis Ashéninka | 60 | 0.729 | 0.974 | 0.974 | 0.000 | 0.000 |
-| `qvn_Latn` | North Junín Quechua | 57 | 0.219 | 0.459 | 0.459 | 0.000 | 0.000 |
+| `qvn_Latn` | North Junín Quechua | 57 | 0.219 | 0.460 | 0.460 | 0.000 | 0.000 |
 | `hin_Deva` | Hindi | 59 | 0.723 | 0.957 | 0.957 | 0.975 | 0.975 |
 | `hrv_Latn` | Croatian | 58 | 0.500 | 0.727 | 0.727 | 0.474 | 0.474 |
 | `kbp_Latn` | Kabiyè | 59 | 0.781 | 1.000 | 1.000 | 0.389 | 0.389 |
@@ -227,8 +286,7 @@ Where MillID loses most to GlotLID:
 
 ### Leaderboard: test-common
 
-MillID's F1 against the others', language by language: against GlotLID restricted to MillID's labels, MillID is better on **6** languages, worse on **154** and within 0.01 on 53; against fastText restricted to MillID's labels, MillID is better on **85** languages, worse on **51** and within 0.01 on 77. The full table, with precision and recall, is
-`reports/test-common/leaderboard.csv`.
+MillID's F1 against the others', language by language: against GlotLID restricted to MillID's labels, MillID is better on **6** languages, worse on **154** and within 0.01 on 53; against fastText restricted to MillID's labels, MillID is better on **85** languages, worse on **51** and within 0.01 on 77. The full table, with precision and recall, is `reports/test-common/leaderboard.csv`.
 
 The 20 languages with the most passages:
 
@@ -266,12 +324,12 @@ Where MillID gains most over GlotLID:
 | `fuv_Latn` | Nigerian Fulfulde | 164 | 0.907 | 0.877 | 0.892 | 0.859 | 0.859 |
 | `knc_Latn` | Central Kanuri | 293 | 0.967 | 0.952 | 0.952 | 0.981 | 0.981 |
 | `quy_Latn` | Ayacucho Quechua | 271 | 0.940 | 0.925 | 0.933 | 0.979 | 0.979 |
-| `kam_Latn` | Kamba | 272 | 0.980 | 0.972 | 0.974 | 0.862 | 0.862 |
-| `bjn_Arab` | Banjar | 116 | 0.991 | 0.834 | 0.987 | 0.453 | 0.960 |
+| `kam_Latn` | Kamba | 272 | 0.980 | 0.972 | 0.974 | 0.863 | 0.863 |
+| `bjn_Arab` | Banjar | 116 | 0.992 | 0.834 | 0.987 | 0.453 | 0.960 |
 | `por_Latn` | Portuguese | 657 | 0.959 | 0.954 | 0.956 | 0.950 | 0.949 |
 | `mlt_Latn` | Maltese | 291 | 0.995 | 0.993 | 0.993 | 0.961 | 0.961 |
-| `san_Deva` | Sanskrit | 239 | 0.990 | 0.989 | 0.989 | 0.970 | 0.970 |
 | `hye_Armn` | Armenian | 178 | 1.000 | 1.000 | 1.000 | 0.977 | 0.977 |
+| `khm_Khmr` | Khmer | 187 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
 | `guj_Gujr` | Gujarati | 354 | 1.000 | 1.000 | 1.000 | 0.984 | 0.984 |
 | `mal_Mlym` | Malayalam | 338 | 1.000 | 1.000 | 1.000 | 0.994 | 0.994 |
 
@@ -282,7 +340,7 @@ Where MillID loses most to GlotLID:
 | `knc_Arab` | Central Kanuri | 68 | 0.522 | 0.857 | 0.860 | 0.029 | 0.029 |
 | `hrv_Latn` | Croatian | 168 | 0.659 | 0.994 | 0.994 | 0.856 | 0.856 |
 | `bos_Latn` | Bosnian | 198 | 0.685 | 1.000 | 1.000 | 0.897 | 0.897 |
-| `tzm_Tfng` | Central Atlas Tamazight | 157 | 0.697 | 0.935 | 0.935 | 0.751 | 0.751 |
+| `tzm_Tfng` | Central Atlas Tamazight | 157 | 0.697 | 0.934 | 0.934 | 0.751 | 0.751 |
 | `zsm_Latn` | Standard Malay | 244 | 0.789 | 0.998 | 0.998 | 0.908 | 0.908 |
 | `taq_Tfng` | Tamasheq | 104 | 0.654 | 0.805 | 0.805 | 0.000 | 0.000 |
 | `ind_Latn` | Indonesian | 335 | 0.842 | 0.991 | 0.991 | 0.887 | 0.887 |
@@ -292,13 +350,17 @@ Where MillID loses most to GlotLID:
 | `kin_Latn` | Kinyarwanda | 210 | 0.849 | 0.981 | 0.981 | 0.854 | 0.854 |
 | `mag_Deva` | Magahi | 97 | 0.890 | 1.000 | 1.000 | 0.772 | 0.772 |
 | `lim_Latn` | Limburgish | 196 | 0.896 | 1.000 | 1.000 | 0.956 | 0.956 |
-| `run_Latn` | Rundi | 291 | 0.883 | 0.985 | 0.985 | 0.854 | 0.854 |
+| `run_Latn` | Rundi | 291 | 0.883 | 0.985 | 0.985 | 0.855 | 0.855 |
 | `taq_Latn` | Tamasheq | 106 | 0.902 | 1.000 | 1.000 | 0.838 | 0.838 |
 
 Per-language precision, recall (sensitivity), F1 and specificity for every model are in `reports/<set>/`.
 
 ## Training
 
-1,943,957 passages, best validation macro F1 at step 106000, 0.6 hours.
-Languages sampled in proportion to n^0.3; 25% of passages cut to a 20 to 80 character
-snippet; batch 512, AdamW at 0.001, cosine schedule, label smoothing 0.1.
+1,943,957 passages; the best validation macro F1 at step 106,000, after 0.6 hours on 1 GPU. Languages sampled in proportion to n^0.3; 25% of passages cut to a 20 to 80 character snippet; batch 512, AdamW at 0.001, cosine schedule, label smoothing 0.1.
+
+The training data is the full MillID build. The public [baobabtech/millid-dataset](https://huggingface.co/datasets/baobabtech/millid-dataset) is that build without Save the Children's and Nal'ibali's text (1.6% of its passages), which is not ours to redistribute. Its splits are the same, so its `test` split is the one scored above, without those passages.
+
+## Licence
+
+The weights and the code are Apache-2.0. The training data carries its own licence row by row (see the [dataset card](https://huggingface.co/datasets/baobabtech/millid-dataset#licensing)), including text licensed for non-commercial use only (CC BY-NC, CC BY-NC-SA).

millid.py

@@ -657,7 +657,7 @@ class Sampler:
     A language is drawn with probability proportional to n^alpha, n being its passage count:
     alpha 1 is the data as it is, alpha 0 every language equally; 0.3 in between. Then a
     passage of it is drawn, and sometimes cut short, because a short snippet is where
-    language identification is hardest and what a document pipeline sees in table cells.
+    language identification is hardest and what a page holds in headers and table cells.
     """
 
     def __init__(self, texts: list[str], label_ids: np.ndarray, alpha: float, short_p: float, seed: int):