Model card: run it without a server, in Python or JavaScript

ARCHITECTURE.md

@@ -204,7 +204,7 @@ flowchart LR
   is (α = 1), where the largest languages fill every batch, and uniform (α = 0), where a
   100-passage language is seen until memorised. XLM-R and mT5 sample their languages the same way.
 - **How long.** A quarter of passages are cut to a 20 to 80 character snippet, on word boundaries
-  where the script has them: short text is the hard case, and what a document pipeline meets in
+  where the script has them: short text is the hard case, and what a document holds in its
   headers, captions and table cells.
 - **Why macro F1 chooses.** It weighs every language equally, the score a thousand-language model
   should be judged on. Top-1 accuracy, which the largest languages dominate, is reported beside it.

README.md

@@ -1,44 +1,105 @@
 ---
+license: apache-2.0
 library_name: pytorch
 pipeline_tag: text-classification
 tags:
 - language-identification
+- onnx
+- byte-level
 datasets:
 - baobabtech/millid-dataset
+metrics:
+- f1
+- accuracy
+model-index:
+- name: millid-medium-v1
+  results:
+  - task:
+      type: text-classification
+      name: Language identification
+    dataset:
+      name: MillID test split (the full build)
+      type: baobabtech/millid-dataset
+      split: test
+    metrics:
+    - type: f1
+      name: Macro F1
+      value: 0.9576
+    - type: accuracy
+      name: Top-1 accuracy
+      value: 0.9616
+  - task:
+      type: text-classification
+      name: Language identification
+    dataset:
+      name: UDHR-LID, the languages MillID knows
+      type: baobabtech/millid-dataset
+      config: udhr
+      split: test
+    metrics:
+    - type: f1
+      name: Macro F1
+      value: 0.8780
+    - type: accuracy
+      name: Top-1 accuracy
+      value: 0.8524
 ---
 
-# MillID medium: MultiLingual Language IDentifier
+# MillID medium v1: MultiLingual Language IDentifier
 
-A byte-level, encoder-only transformer that names the language of a passage: **1641 language-script
-labels** (ISO 639-3 + script, e.g. `eng_Latn`, `amh_Ethi`). 115.0M parameters, trained from
-scratch on [baobabtech/millid-dataset](https://huggingface.co/datasets/baobabtech/millid-dataset).
+A byte-level, encoder-only transformer that names the language of a passage, from a table cell to a paragraph: **1,641 language-script labels** (ISO 639-3 + script, e.g. `eng_Latn`, `amh_Ethi`). 115.0M parameters, trained from scratch. Apache-2.0.
+
+## The MillID models
+
+| Model | Size | Macro F1, `test` / `udhr` | Runs best on |
+|---|---|---|---|
+| [MillID tiny v1.5](https://huggingface.co/baobabtech/millid-tiny-v1.5) | 2.9M parameters, ONNX 11 MB | 0.9554 / 0.8764 | a CPU, or a web page |
+| **MillID medium v1** (this model) | 115M parameters, ONNX 455 MB | 0.9576 / 0.8780 | a GPU |
+| [MillID large v1.5](https://huggingface.co/baobabtech/millid-large-v1.5) | 193M parameters, ONNX 764 MB | 0.9646 / 0.8812 | a GPU: the most accurate |
+
+All three, language by language, against GlotLID and Meta's fastText: the [leaderboard](https://huggingface.co/baobabtech/millid). GlotLID, restricted to MillID's labels, scores 0.9755 / 0.8873 on the same sets.
+
+## Use it
+
+No server and no PyTorch: the model runs in your process from `onnx/model.onnx`, in Python or JavaScript, and the two give the same answers (one encoder, ported line for line and tested against the same fixtures in [baobab-tech/millid](https://github.com/baobab-tech/millid)).
+
+```bash
+pip install "millid[cpu] @ git+https://github.com/baobab-tech/millid.git"   # [gpu] for CUDA
+```
 
 ```python
-from millid import MillID   # millid.py, in this repository
+from millid import MillID
+
+model = MillID.from_pretrained("baobabtech/millid-medium-v1")
+model.predict("Lave men ou ak savon anvan ou manje.", k=3)
+# [Prediction(label='hat_Latn', language='Haitian Creole', probability=...), ...]
+model.predict(["one text", "another"], k=3)  # one list of answers per text
+model.identify_document(long_text)           # any length: its languages and segments
+```
 
-model = MillID.from_pretrained("baobabtech/millid-medium")
-model.predict(["Lave men ou ak savon anvan ou manje."], k=3)
+```bash
+millid predict "Lave men ou ak savon anvan ou manje." --model baobabtech/millid-medium-v1 -k 3
 ```
 
+JavaScript, in Node or a browser (`js/` in the same repository; not on npm yet):
+
+```js
+import { MillID } from "@baobabtech/millid";   // "@baobabtech/millid/web" in a browser, with { ort }
+const model = await MillID.fromPretrained("baobabtech/millid-medium-v1");
+await model.predict("Lave men ou ak savon anvan ou manje.", 3);
+```
+
+`millid.py` in this repository is the PyTorch training code, which reads `model.safetensors` to fine-tune or export.
+
+The ONNX graph takes, for each UTF-8 byte of a passage cut to 255 bytes after a CLS token, `ids`, `script` and `charpos` (int64, `[batch, bytes]`), `ngrams` (int64, `[batch, bytes, 3]`) and `mask` (bool, `[batch, bytes]`), and returns `logits` (float32, `[batch, labels]`). `millid.encode_batch` and the JavaScript `encodeBatch` build them; script ids come from a frozen table (Unicode 15.0), so every runtime gives the same ones.
+
 ## Architecture
 
-UTF-8 bytes (after Unicode NFKC), each embedded as byte + Unicode script + byte-in-character + hashed character
-2-, 3- and 4-grams (32,768 buckets) -> depthwise convolution stem (kernel 5) -> strided
-convolution, 2 bytes per token -> 12 pre-norm blocks (RMSNorm, attention with RoPE,
-12 heads, SwiGLU; d=768) -> masked mean pooling -> language head, each language's vector the sum of
-its own, its language's and its script's. Up to 256 bytes per passage. See `ARCHITECTURE.md`.
+UTF-8 bytes (after Unicode NFKC), each embedded as byte + Unicode script + byte-in-character + hashed character 2-, 3- and 4-grams (32,768 buckets) -> depthwise convolution stem (kernel 5) -> strided convolution, 2 bytes per token -> 12 pre-norm blocks (RMSNorm, attention with RoPE, 12 heads, SwiGLU; d=768) -> masked mean pooling -> language head, each language's vector the sum of its own, its language's and its script's. Up to 256 bytes per passage. See `ARCHITECTURE.md`.
 
 ## Evaluation against GlotLID and fastText
 
-`validation` chose the checkpoint; `test` is split by document from training; `udhr` is the Universal Declaration
-of Human Rights, labelled by its translators, over the languages MillID knows, and `udhr-all` over all of them.
-`glotlid` is [GlotLID](https://huggingface.co/cis-lmu/glotlid) as it is; `glotlid@millid` is GlotLID restricted to
-MillID's labels, the like-for-like comparison. GlotLID's own labels came from training on data that includes UDHR
-and labelled FinePDFs and FineWeb-2, so its scores on `udhr` and on those sources are optimistic. `fasttext` is
-Meta's [fastText language identifier](https://huggingface.co/facebook/fasttext-language-identification) (NLLB,
-218 labels), its answers mapped onto MillID's spelling (Simplified and Traditional Han as `Hani`, `pes`/`prs` as
-`fas`, `est` as `ekk`). It cannot answer most of the 1,600 languages here, so `test-common` and `udhr-common`
-score every model on the passages whose language all of them know: the head-to-head on equal terms.
+`validation` chose the checkpoint; `test` is split by document from training; `udhr` is the Universal Declaration of Human Rights, labelled by its translators, over the languages MillID knows, and `udhr-all` over all of them. `glotlid` is [GlotLID](https://huggingface.co/cis-lmu/glotlid) as it is; `glotlid@millid` is GlotLID restricted to MillID's labels, the like-for-like comparison. GlotLID's own labels came from training on data that includes UDHR and labelled FinePDFs and FineWeb-2, so its scores on `udhr` and on those sources are optimistic. `fasttext` is Meta's [fastText language identifier](https://huggingface.co/facebook/fasttext-language-identification) (NLLB, 218 labels), its answers mapped onto MillID's spelling (Simplified and Traditional Han as `Hani`, `pes`/`prs` as `fas`, `est` as `ekk`). It cannot answer most of the 1,600 languages here, so `test-common` and `udhr-common` score every model on the passages whose language all of them know: the head-to-head on equal terms.
 
 | Set | Model | Top-1 | Top-3 | Top-5 | Macro P | Macro R | Macro F1 | Median F1 | Languages |
 |---|---|---|---|---|---|---|---|---|---|
@@ -87,8 +148,7 @@ Top-1 accuracy by passage length:
 
 ### Leaderboard: test
 
-MillID's F1 against the others', language by language: against GlotLID restricted to MillID's labels, MillID is better on **110** languages, worse on **776** and within 0.01 on 735; against fastText restricted to MillID's labels, MillID is better on **1616** languages, worse on **0** and within 0.01 on 5. The full table, with precision and recall, is
-`reports/test/leaderboard.csv`.
+MillID's F1 against the others', language by language: against GlotLID restricted to MillID's labels, MillID is better on **110** languages, worse on **776** and within 0.01 on 735; against fastText restricted to MillID's labels, MillID is better on **1616** languages, worse on **0** and within 0.01 on 5. The full table, with precision and recall, is `reports/test/leaderboard.csv`.
 
 The 20 languages with the most passages:
 
@@ -96,23 +156,23 @@ The 20 languages with the most passages:
 |---|---|---|---|---|---|---|---|
 | `fra_Latn` | French | 1652 | 0.952 | 0.928 | 0.928 | 0.808 | 0.808 |
 | `eng_Latn` | English | 1444 | 0.916 | 0.858 | 0.859 | 0.653 | 0.648 |
-| `spa_Latn` | Spanish | 1136 | 0.941 | 0.950 | 0.950 | 0.574 | 0.574 |
+| `spa_Latn` | Spanish | 1136 | 0.942 | 0.951 | 0.951 | 0.574 | 0.574 |
 | `tam_Taml` | Tamil | 754 | 0.997 | 0.993 | 0.993 | 0.965 | 0.965 |
 | `arb_Arab` | Arabic (Modern Standard) | 745 | 0.829 | 0.785 | 0.785 | 0.513 | 0.512 |
 | `por_Latn` | Portuguese | 657 | 0.955 | 0.947 | 0.949 | 0.756 | 0.755 |
 | `kan_Knda` | Kannada | 601 | 0.982 | 0.993 | 0.993 | 0.922 | 0.922 |
 | `zul_Latn` | Zulu | 594 | 0.833 | 0.896 | 0.896 | 0.539 | 0.539 |
-| `swh_Latn` | Swahili | 592 | 0.858 | 0.918 | 0.916 | 0.346 | 0.346 |
+| `swh_Latn` | Swahili | 592 | 0.858 | 0.917 | 0.916 | 0.346 | 0.346 |
 | `mar_Deva` | Marathi | 589 | 0.981 | 0.989 | 0.990 | 0.926 | 0.926 |
-| `tso_Latn` | Tsonga | 581 | 0.970 | 0.956 | 0.956 | 0.426 | 0.426 |
+| `tso_Latn` | Tsonga | 581 | 0.970 | 0.956 | 0.956 | 0.425 | 0.425 |
 | `hun_Latn` | Hungarian | 576 | 0.990 | 0.999 | 0.999 | 0.232 | 0.232 |
-| `afr_Latn` | Afrikaans | 569 | 0.973 | 0.985 | 0.985 | 0.861 | 0.860 |
+| `afr_Latn` | Afrikaans | 569 | 0.974 | 0.985 | 0.985 | 0.861 | 0.860 |
 | `tel_Telu` | Telugu | 555 | 0.994 | 0.994 | 0.994 | 0.951 | 0.951 |
 | `nbl_Latn` | South Ndebele | 528 | 0.895 | 0.948 | 0.949 | 0.000 | 0.000 |
 | `rus_Cyrl` | Russian | 528 | 0.925 | 0.949 | 0.949 | 0.462 | 0.462 |
 | `ory_Orya` | Odia (individual language) | 511 | 0.993 | 0.991 | 0.994 | 0.976 | 0.976 |
 | `sot_Latn` | Southern Sotho | 491 | 0.966 | 0.977 | 0.978 | 0.660 | 0.660 |
-| `nso_Latn` | Northern Sotho | 479 | 0.960 | 0.968 | 0.968 | 0.915 | 0.914 |
+| `nso_Latn` | Northern Sotho | 479 | 0.960 | 0.969 | 0.969 | 0.915 | 0.914 |
 | `kua_Latn` | Kuanyama | 476 | 0.979 | 0.986 | 0.986 | 0.000 | 0.000 |
 
 Where MillID gains most over GlotLID:
@@ -140,7 +200,7 @@ Where MillID loses most to GlotLID:
 | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
 |---|---|---|---|---|---|---|---|
 | `ccp_Latn` | Chakma | 3 | 0.333 | 1.000 | 1.000 | 0.000 | 0.000 |
-| `ttj_Latn` | Tooro | 28 | 0.490 | 0.945 | 0.945 | 0.000 | 0.000 |
+| `ttj_Latn` | Tooro | 28 | 0.490 | 0.946 | 0.946 | 0.000 | 0.000 |
 | `hak_Hani` | Hakka Chinese | 29 | 0.415 | 0.824 | 0.824 | 0.000 | 0.000 |
 | `mus_Latn` | Muscogee | 2 | 0.667 | 1.000 | 1.000 | 0.000 | 0.000 |
 | `biu_Latn` | Biete | 7 | 0.667 | 1.000 | 1.000 | 0.000 | 0.000 |
@@ -149,23 +209,22 @@ Where MillID loses most to GlotLID:
 | `esi_Latn` | North Alaskan Inupiatun | 9 | 0.500 | 0.783 | 0.783 | 0.000 | 0.000 |
 | `und_Ethi` | und | 14 | 0.741 | 1.000 | 1.000 | 0.000 | 0.000 |
 | `hrv_Latn` | Croatian | 168 | 0.687 | 0.944 | 0.944 | 0.178 | 0.178 |
-| `dks_Latn` | Southeastern Dinka | 3 | 0.750 | 1.000 | 1.000 | 0.000 | 0.000 |
-| `nse_Latn` | Nsenga | 12 | 0.500 | 0.750 | 0.750 | 0.000 | 0.000 |
 | `tui_Latn` | Tupuri | 4 | 0.750 | 0.667 | 1.000 | 0.000 | 0.000 |
+| `nse_Latn` | Nsenga | 12 | 0.500 | 0.750 | 0.750 | 0.000 | 0.000 |
+| `dks_Latn` | Southeastern Dinka | 3 | 0.750 | 1.000 | 1.000 | 0.000 | 0.000 |
 | `mbd_Latn` | Dibabawon Manobo | 14 | 0.757 | 1.000 | 1.000 | 0.000 | 0.000 |
 | `lnd_Latn` | Lundayeh | 5 | 0.769 | 1.000 | 1.000 | 0.000 | 0.000 |
 
 ### Leaderboard: udhr
 
-MillID's F1 against the others', language by language: against GlotLID restricted to MillID's labels, MillID is better on **63** languages, worse on **95** and within 0.01 on 200; against fastText restricted to MillID's labels, MillID is better on **304** languages, worse on **13** and within 0.01 on 41. The full table, with precision and recall, is
-`reports/udhr/leaderboard.csv`.
+MillID's F1 against the others', language by language: against GlotLID restricted to MillID's labels, MillID is better on **63** languages, worse on **95** and within 0.01 on 200; against fastText restricted to MillID's labels, MillID is better on **304** languages, worse on **13** and within 0.01 on 41. The full table, with precision and recall, is `reports/udhr/leaderboard.csv`.
 
 The 20 languages with the most passages:
 
 | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
 |---|---|---|---|---|---|---|---|
-| `roh_Latn` | Romansh | 399 | 0.991 | 0.999 | 0.999 | 0.945 | 0.945 |
-| `oci_Latn` | Occitan | 398 | 0.331 | 0.376 | 0.376 | 0.436 | 0.436 |
+| `roh_Latn` | Romansh | 399 | 0.991 | 0.999 | 0.999 | 0.946 | 0.946 |
+| `oci_Latn` | Occitan | 398 | 0.331 | 0.376 | 0.376 | 0.435 | 0.435 |
 | `cmn_Hani` | Chinese (Mandarin) | 319 | 0.886 | 0.868 | 0.868 | 0.468 | 0.468 |
 | `krl_Latn` | Karelian | 183 | 0.000 | 0.011 | 0.011 | 0.000 | 0.000 |
 | `qug_Latn` | Chimborazo Highland Quichua | 182 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
@@ -177,7 +236,7 @@ The 20 languages with the most passages:
 | `jpn_Jpan` | Japanese | 140 | 1.000 | 1.000 | 1.000 | 0.944 | 0.944 |
 | `snn_Latn` | Siona | 136 | 0.773 | 0.514 | 0.514 | 0.000 | 0.000 |
 | `ktu_Latn` | Kituba (Democratic Republic of Congo) | 118 | 0.769 | 0.680 | 0.680 | 0.000 | 0.000 |
-| `lat_Latn` | Latin | 118 | 0.996 | 0.991 | 0.991 | 0.000 | 0.000 |
+| `lat_Latn` | Latin | 118 | 0.996 | 0.992 | 0.992 | 0.000 | 0.000 |
 | `hat_Latn` | Haitian Creole | 117 | 0.991 | 0.979 | 0.979 | 0.699 | 0.699 |
 | `twi_Latn` | Twi | 116 | 0.987 | 0.996 | 0.996 | 0.399 | 0.396 |
 | `rmn_Latn` | Balkan Romani | 116 | 0.667 | 0.857 | 0.857 | 0.000 | 0.000 |
@@ -192,7 +251,7 @@ Where MillID gains most over GlotLID:
 | `mic_Latn` | Mi'kmaw | 59 | 1.000 | 0.427 | 0.427 | 0.000 | 0.000 |
 | `tet_Latn` | Tetum | 58 | 0.699 | 0.167 | 0.167 | 0.000 | 0.000 |
 | `skr_Arab` | Saraiki | 63 | 0.923 | 0.500 | 0.500 | 0.000 | 0.000 |
-| `sag_Latn` | Sango | 59 | 0.991 | 0.578 | 0.578 | 0.189 | 0.189 |
+| `sag_Latn` | Sango | 59 | 0.992 | 0.578 | 0.578 | 0.189 | 0.189 |
 | `zgh_Tfng` | Standard Moroccan Tamazight | 60 | 0.824 | 0.462 | 0.462 | 0.000 | 0.000 |
 | `fuf_Latn` | Pular | 58 | 0.816 | 0.483 | 0.483 | 0.000 | 0.000 |
 | `tzm_Tfng` | Central Atlas Tamazight | 1 | 0.333 | 0.065 | 0.065 | 0.033 | 0.033 |
@@ -201,8 +260,8 @@ Where MillID gains most over GlotLID:
 | `mxv_Latn` | Metlatónoc Mixtec | 58 | 0.215 | 0.000 | 0.000 | 0.000 | 0.000 |
 | `pnb_Arab` | Western Panjabi | 61 | 0.931 | 0.739 | 0.739 | 0.000 | 0.000 |
 | `lld_Latn` | Ladin | 59 | 0.899 | 0.731 | 0.731 | 0.000 | 0.000 |
-| `tah_Latn` | Tahitian | 57 | 0.973 | 0.813 | 0.813 | 0.966 | 0.966 |
-| `pbb_Latn` | Páez | 56 | 0.848 | 0.713 | 0.713 | 0.000 | 0.000 |
+| `tah_Latn` | Tahitian | 57 | 0.973 | 0.812 | 0.812 | 0.966 | 0.966 |
+| `pbb_Latn` | Páez | 56 | 0.849 | 0.713 | 0.713 | 0.000 | 0.000 |
 | `eve_Cyrl` | Even | 49 | 0.990 | 0.886 | 0.886 | 0.000 | 0.000 |
 
 Where MillID loses most to GlotLID:
@@ -219,16 +278,15 @@ Where MillID loses most to GlotLID:
 | `ell_Grek` | Greek | 115 | 0.655 | 1.000 | 1.000 | 1.000 | 1.000 |
 | `bho_Deva` | Bhojpuri | 56 | 0.643 | 0.933 | 0.933 | 0.832 | 0.832 |
 | `fuv_Latn` | Nigerian Fulfulde | 108 | 0.405 | 0.681 | 0.689 | 0.475 | 0.475 |
-| `cni_Latn` | Asháninka | 52 | 0.707 | 0.963 | 0.963 | 0.000 | 0.000 |
-| `qvn_Latn` | North Junín Quechua | 57 | 0.219 | 0.459 | 0.459 | 0.000 | 0.000 |
+| `cni_Latn` | Asháninka | 52 | 0.708 | 0.963 | 0.963 | 0.000 | 0.000 |
+| `qvn_Latn` | North Junín Quechua | 57 | 0.219 | 0.460 | 0.460 | 0.000 | 0.000 |
 | `ind_Latn` | Indonesian | 60 | 0.674 | 0.903 | 0.903 | 0.565 | 0.565 |
 | `maz_Latn` | Central Mazahua | 59 | 0.742 | 0.939 | 0.939 | 0.000 | 0.000 |
 | `rmn_Latn` | Balkan Romani | 116 | 0.667 | 0.857 | 0.857 | 0.000 | 0.000 |
 
 ### Leaderboard: test-common
 
-MillID's F1 against the others', language by language: against GlotLID restricted to MillID's labels, MillID is better on **6** languages, worse on **142** and within 0.01 on 65; against fastText restricted to MillID's labels, MillID is better on **93** languages, worse on **47** and within 0.01 on 73. The full table, with precision and recall, is
-`reports/test-common/leaderboard.csv`.
+MillID's F1 against the others', language by language: against GlotLID restricted to MillID's labels, MillID is better on **6** languages, worse on **142** and within 0.01 on 65; against fastText restricted to MillID's labels, MillID is better on **93** languages, worse on **47** and within 0.01 on 73. The full table, with precision and recall, is `reports/test-common/leaderboard.csv`.
 
 The 20 languages with the most passages:
 
@@ -250,7 +308,7 @@ The 20 languages with the most passages:
 | `tel_Telu` | Telugu | 555 | 1.000 | 0.998 | 0.998 | 0.999 | 0.999 |
 | `rus_Cyrl` | Russian | 528 | 0.929 | 0.985 | 0.985 | 0.927 | 0.927 |
 | `ory_Orya` | Odia (individual language) | 511 | 0.996 | 0.997 | 1.000 | 0.996 | 0.996 |
-| `sot_Latn` | Southern Sotho | 491 | 0.966 | 0.977 | 0.978 | 0.856 | 0.856 |
+| `sot_Latn` | Southern Sotho | 491 | 0.967 | 0.977 | 0.978 | 0.856 | 0.856 |
 | `nso_Latn` | Northern Sotho | 479 | 0.961 | 0.970 | 0.970 | 0.967 | 0.967 |
 | `asm_Beng` | Assamese | 467 | 0.995 | 0.999 | 0.999 | 0.997 | 0.997 |
 | `ben_Beng` | Bangla | 466 | 0.987 | 0.997 | 0.997 | 0.989 | 0.989 |
@@ -267,9 +325,9 @@ Where MillID gains most over GlotLID:
 | `arb_Arab` | Arabic (Modern Standard) | 745 | 0.905 | 0.886 | 0.886 | 0.957 | 0.954 |
 | `knc_Latn` | Central Kanuri | 293 | 0.959 | 0.952 | 0.952 | 0.981 | 0.981 |
 | `por_Latn` | Portuguese | 657 | 0.963 | 0.954 | 0.956 | 0.950 | 0.949 |
-| `san_Deva` | Sanskrit | 239 | 0.996 | 0.989 | 0.989 | 0.970 | 0.970 |
-| `kam_Latn` | Kamba | 272 | 0.980 | 0.972 | 0.974 | 0.862 | 0.862 |
+| `san_Deva` | Sanskrit | 239 | 0.996 | 0.990 | 0.990 | 0.971 | 0.971 |
 | `bug_Latn` | Buginese | 78 | 0.987 | 0.981 | 0.981 | 0.938 | 0.938 |
+| `kam_Latn` | Kamba | 272 | 0.980 | 0.972 | 0.974 | 0.863 | 0.863 |
 | `mlt_Latn` | Maltese | 291 | 0.997 | 0.993 | 0.993 | 0.961 | 0.961 |
 | `tel_Telu` | Telugu | 555 | 1.000 | 0.998 | 0.998 | 0.999 | 0.999 |
 | `sin_Sinh` | Sinhala | 232 | 1.000 | 1.000 | 1.000 | 0.998 | 0.998 |
@@ -280,18 +338,18 @@ Where MillID loses most to GlotLID:
 | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
 |---|---|---|---|---|---|---|---|
 | `knc_Arab` | Central Kanuri | 68 | 0.562 | 0.857 | 0.860 | 0.029 | 0.029 |
-| `tzm_Tfng` | Central Atlas Tamazight | 157 | 0.645 | 0.935 | 0.935 | 0.751 | 0.751 |
+| `tzm_Tfng` | Central Atlas Tamazight | 157 | 0.645 | 0.934 | 0.934 | 0.751 | 0.751 |
 | `hrv_Latn` | Croatian | 168 | 0.717 | 0.994 | 0.994 | 0.856 | 0.856 |
 | `bos_Latn` | Bosnian | 198 | 0.727 | 1.000 | 1.000 | 0.897 | 0.897 |
 | `zsm_Latn` | Standard Malay | 244 | 0.773 | 0.998 | 0.998 | 0.908 | 0.908 |
 | `taq_Tfng` | Tamasheq | 104 | 0.613 | 0.805 | 0.805 | 0.000 | 0.000 |
 | `ind_Latn` | Indonesian | 335 | 0.807 | 0.991 | 0.991 | 0.887 | 0.887 |
 | `kin_Latn` | Kinyarwanda | 210 | 0.818 | 0.981 | 0.981 | 0.854 | 0.854 |
-| `run_Latn` | Rundi | 291 | 0.843 | 0.985 | 0.985 | 0.854 | 0.854 |
+| `run_Latn` | Rundi | 291 | 0.843 | 0.985 | 0.985 | 0.855 | 0.855 |
 | `nob_Latn` | Norwegian Bokmål | 265 | 0.820 | 0.961 | 0.959 | 0.894 | 0.894 |
 | `fuv_Latn` | Nigerian Fulfulde | 164 | 0.797 | 0.877 | 0.892 | 0.859 | 0.859 |
 | `mag_Deva` | Magahi | 97 | 0.905 | 1.000 | 1.000 | 0.772 | 0.772 |
-| `vec_Latn` | Venetian | 49 | 0.766 | 0.860 | 0.860 | 0.620 | 0.620 |
+| `vec_Latn` | Venetian | 49 | 0.766 | 0.861 | 0.861 | 0.620 | 0.620 |
 | `cjk_Latn` | Chokwe | 24 | 0.909 | 1.000 | 1.000 | 0.807 | 0.807 |
 | `taq_Latn` | Tamasheq | 106 | 0.909 | 1.000 | 1.000 | 0.838 | 0.838 |
 
@@ -299,6 +357,10 @@ Per-language precision, recall (sensitivity), F1 and specificity for every model
 
 ## Training
 
-1,944,095 passages, best validation macro F1 at step 56000, 3.4 hours.
-Languages sampled in proportion to n^0.3; 25% of passages cut to a 20 to 80 character
-snippet; batch 512, AdamW at 0.0003, cosine schedule, label smoothing 0.1.
+1,944,095 passages; the best validation macro F1 at step 56,000, after 3.4 hours on 1 GPU. Languages sampled in proportion to n^0.3; 25% of passages cut to a 20 to 80 character snippet; batch 512, AdamW at 0.0003, cosine schedule, label smoothing 0.1.
+
+The training data is the full MillID build. The public [baobabtech/millid-dataset](https://huggingface.co/datasets/baobabtech/millid-dataset) is that build without Save the Children's and Nal'ibali's text (1.6% of its passages), which is not ours to redistribute. Its splits are the same, so its `test` split is the one scored above, without those passages.
+
+## Licence
+
+The weights and the code are Apache-2.0. The training data carries its own licence row by row (see the [dataset card](https://huggingface.co/datasets/baobabtech/millid-dataset#licensing)), including text licensed for non-commercial use only (CC BY-NC, CC BY-NC-SA).

millid.py

@@ -654,7 +654,7 @@ class Sampler:
     A language is drawn with probability proportional to n^alpha, n being its passage count:
     alpha 1 is the data as it is, alpha 0 every language equally; 0.3 in between. Then a
     passage of it is drawn, and sometimes cut short, because a short snippet is where
-    language identification is hardest and what a document pipeline sees in table cells.
+    language identification is hardest and what a page holds in headers and table cells.
     """
 
     def __init__(self, texts: list[str], label_ids: np.ndarray, alpha: float, short_p: float, seed: int):