Model card for the public release

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/millid-architecture.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -4,8 +4,12 @@ library_name: pytorch
 pipeline_tag: text-classification
 tags:
 - language-identification
-- onnx
+- text-classification
+- multilingual
 - byte-level
+- onnx
+language:
+- multilingual
 datasets:
 - baobabtech/millid-dataset
 metrics:
@@ -45,13 +49,78 @@ model-index:
       value: 0.8524
 ---
 
-# MillID medium v1: MultiLingual Language IDentifier
+<p align="center">
+  <a href="https://github.com/baobab-tech/millid" target="_blank">GitHub</a> |
+  <a href="https://huggingface.co/baobabtech/millid" target="_blank">Leaderboard</a> |
+  <a href="https://huggingface.co/datasets/baobabtech/millid-dataset" target="_blank">Dataset</a> |
+  <a href="https://github.com/baobab-tech/millid/blob/main/training/ARCHITECTURE.md" target="_blank">Architecture</a>
+  <br>
+  <b>License</b>: <a href="https://www.apache.org/licenses/LICENSE-2.0" target="_blank">Apache 2.0</a> | <b>Authors</b>: <a href="https://huggingface.co/baobabtech" target="_blank">Baobab Tech</a>
+</p>
+
+# MillID medium v1
+
+**MillID** (MultiLingual Language IDentifier) is an open language identification model built by Baobab Tech. It names the language of a passage of text, from a table cell to a paragraph, among 1,641 language-script labels: 1,550 languages written in 56 scripts, and 37 labels for text in a script whose language it cannot name (`und_<Script>`). MillID medium v1 has 115.0M parameters and an ONNX file of 455 MB.
+
+The middle size, more accurate than tiny v1.5 on both test sets and best run on a GPU; on a laptop CPU it reads about 2 passages a second per thread.
+
+MillID offers:
+
+* **Language and script:** labels are ISO 639-3 plus ISO 15924 script, so Serbian in Cyrillic (`srp_Cyrl`) and in Latin (`srp_Latn`) are told apart. Many of the languages are low-resource, with a few hundred passages each to learn from.
+* **No tokenizer:** it reads UTF-8 bytes, each with its Unicode script and the character n-grams ending there, so any script is input it can read and nothing is an unknown token.
+* **Short text:** a quarter of its training passages were cut to 20 to 80 characters, the length of a heading or a table cell, where language identification is hardest.
+* **Whole documents:** `identify_document` reads text of any length in windows and returns the document's languages and the segments of each.
+* **No server, no PyTorch:** one ONNX file and a small encoder, in Python (NumPy) or JavaScript (Node and browsers); the two are tested to give the same answers.
+* **Three sizes, one label set:** tiny v1.5, medium v1 and large v1.5 share their labels, so changing size changes no code.
+
+### Model overview
+
+| **Parameters** | Total | 115.0M |
+|---|---|---|
+| | Embeddings | 25.8M |
+| | Convolution stem | 1.8M |
+| | Transformer blocks | 85.0M |
+| | Language head | 2.5M |
+| **Architecture** | Blocks | 12 |
+| | Model dimension | 768 |
+| | Attention heads | 12 |
+| | Bytes per token | 2 |
+| | N-gram buckets | 32,768 |
+| | Normalisation | RMSNorm, pre-norm |
+| | Positions | RoPE |
+| | Feed-forward | SwiGLU |
+| | Pooling | Masked mean |
+| **Input and output** | Input | UTF-8 bytes, NFKC |
+| | Passage length | Up to 255 bytes |
+| | Labels | 1,641 |
+| | Runtime | ONNX (455 MB) |
+
+## Benchmark results
+
+MillID was evaluated on two test sets: `test`, passages from documents never seen in training, split from the training data by document; and UDHR-LID (`udhr`), the Universal Declaration of Human Rights labelled by its translators. It is compared with [GlotLID](https://huggingface.co/cis-lmu/glotlid) and Meta's [fastText language identifier](https://huggingface.co/facebook/fasttext-language-identification) (NLLB, 218 labels), each restricted to MillID's labels, its best answer among them: the like-for-like comparison.
+
+### Overall results
+
+| Set | Languages | Metric | MillID | GlotLID | fastText |
+|---|---|---|---|---|---|
+| `test` | 1,621 | Macro F1 | 0.9576 | 0.9755 | 0.0659 |
+| | | Top-1 | 0.9616 | 0.9783 | 0.2515 |
+| `test-common` | 213 | Macro F1 | 0.9527 | 0.9846 | 0.9173 |
+| | | Top-1 | 0.9425 | 0.9749 | 0.9352 |
+| `udhr` | 358 | Macro F1 | 0.8780 | 0.8873 | 0.3499 |
+| | | Top-1 | 0.8524 | 0.8614 | 0.4629 |
+| `udhr-common` | 170 | Macro F1 | 0.9480 | 0.9659 | 0.9329 |
+| | | Top-1 | 0.9121 | 0.9338 | 0.9195 |
+| `udhr-all` | 429 | Macro F1 | 0.7184 | 0.7245 | 0.2797 |
+| | | Top-1 | 0.7301 | 0.7378 | 0.3964 |
 
-A byte-level, encoder-only transformer that names the language of a passage, from a table cell to a paragraph: **1,641 language-script labels** (ISO 639-3 + script, e.g. `eng_Latn`, `amh_Ethi`). 115.0M parameters, trained from scratch. Apache-2.0.
+* `test`: every language, 1,621 of them. `test-common`: the languages all three systems know, the head-to-head on equal terms, since fastText cannot answer most of MillID's.
+* `udhr`: the Declaration in the languages MillID knows. `udhr-common`: in the languages all three know. `udhr-all`: in every language it has, MillID's unknown ones counted wrong.
+* GlotLID's labels came from training on data that includes UDHR, and it labelled the FinePDFs and FineWeb-2 passages in `test`, so its scores are optimistic on both.
 
-## The MillID models
+### The three MillID models
 
-Three sizes, this one in bold. Macro F1 on `test` and `udhr`, the sets below:
+Macro F1 on `test` and `udhr`; this model in bold. All three, language by language, are on the [leaderboard](https://huggingface.co/baobabtech/millid).
 
 | Model | Parameters | ONNX | Test F1 | UDHR F1 | Runs on |
 |---|---|---|---|---|---|
@@ -59,61 +128,21 @@ Three sizes, this one in bold. Macro F1 on `test` and `udhr`, the sets below:
 | **MillID medium v1** | 115M | 455 MB | 0.9576 | 0.8780 | GPU |
 | [MillID large v1.5](https://huggingface.co/baobabtech/millid-large-v1.5) | 193M | 764 MB | 0.9646 | 0.8812 | GPU |
 
-All three, language by language, against GlotLID and Meta's fastText: the [leaderboard](https://huggingface.co/baobabtech/millid). GlotLID, restricted to MillID's labels, scores 0.9755 and 0.8873.
-
-## Use it
-
-No server and no PyTorch: the model runs in your process from `onnx/model.onnx`, in Python or JavaScript, and the two give the same answers (one encoder, ported line for line and tested against the same fixtures in [baobab-tech/millid](https://github.com/baobab-tech/millid)).
-
-```bash
-pip install "millid[cpu] @ git+https://github.com/baobab-tech/millid.git"   # [gpu] for CUDA
-```
-
-```python
-from millid import MillID
-
-model = MillID.from_pretrained("baobabtech/millid-medium-v1")
-model.predict("Lave men ou ak savon anvan ou manje.", k=3)
-# [Prediction(label='hat_Latn', language='Haitian Creole', probability=...), ...]
-model.predict(["one text", "another"], k=3)  # one list of answers per text
-model.identify_document(long_text)           # any length: its languages and segments
-```
-
-```bash
-millid predict "Lave men ou ak savon anvan ou manje." --model baobabtech/millid-medium-v1 -k 3
-```
-
-JavaScript, in Node or a browser (`js/` in the same repository; not on npm yet):
-
-```js
-import { MillID } from "@baobabtech/millid";   // "@baobabtech/millid/web" in a browser, with { ort }
-const model = await MillID.fromPretrained("baobabtech/millid-medium-v1");
-await model.predict("Lave men ou ak savon anvan ou manje.", 3);
-```
-
-`millid.py` in this repository is the PyTorch training code, which reads `model.safetensors` to fine-tune or export.
-
-The ONNX graph takes, for each UTF-8 byte of a passage cut to 255 bytes after a CLS token, `ids`, `script` and `charpos` (int64, `[batch, bytes]`), `ngrams` (int64, `[batch, bytes, 3]`) and `mask` (bool, `[batch, bytes]`), and returns `logits` (float32, `[batch, labels]`). `millid.encode_batch` and the JavaScript `encodeBatch` build them; script ids come from a frozen table (Unicode 15.0), so every runtime gives the same ones.
-
-## Architecture
-
-UTF-8 bytes (after Unicode NFKC), each embedded as byte + Unicode script + byte-in-character + hashed character 2-, 3- and 4-grams (32,768 buckets) -> depthwise convolution stem (kernel 5) -> strided convolution, 2 bytes per token -> 12 pre-norm blocks (RMSNorm, attention with RoPE, 12 heads, SwiGLU; d=768) -> masked mean pooling -> language head, each language's vector the sum of its own, its language's and its script's. Up to 256 bytes per passage. See `ARCHITECTURE.md`.
+### By passage length
 
-## Evaluation against GlotLID and fastText
+Top-1 accuracy on passages of 20 to 80 characters (short), up to 300 (sentence) and up to 1,000 (paragraph):
 
-`validation` chose the checkpoint; `test` is split by document from training; `udhr` is the Universal Declaration of Human Rights, labelled by its translators, over the languages MillID knows, and `udhr-all` over all of them. [GlotLID](https://huggingface.co/cis-lmu/glotlid) and Meta's [fastText language identifier](https://huggingface.co/facebook/fasttext-language-identification) (NLLB, 218 labels) are each restricted to MillID's labels, their best answer among them: the like-for-like comparison. fastText's answers are mapped onto MillID's spelling (Simplified and Traditional Han as `Hani`, `pes`/`prs` as `fas`, `est` as `ekk`). GlotLID's own labels came from training on data that includes UDHR and labelled FinePDFs and FineWeb-2, so its scores on `udhr` and on those sources are optimistic. fastText cannot answer most of the 1,600 languages here, so `test-common` and `udhr-common` score every system on the passages whose language all of them know: the head-to-head on equal terms.
-
-Macro F1 on each set, every system on the same passages; GlotLID and fastText restricted to MillID's labels:
-
-| Set | Languages | MillID | GlotLID | fastText |
+| Set | System | Short | Sentence | Paragraph |
 |---|---|---|---|---|
-| `test` | 1,621 | 0.9576 | 0.9755 | 0.0659 |
-| `test-common` | 213 | 0.9527 | 0.9846 | 0.9173 |
-| `udhr` | 358 | 0.8780 | 0.8873 | 0.3499 |
-| `udhr-common` | 170 | 0.9480 | 0.9659 | 0.9329 |
-| `udhr-all` | 429 | 0.7184 | 0.7245 | 0.2797 |
+| `validation` | MillID | 0.9363 | 0.9818 | 0.9858 |
+| `test` | MillID | 0.9382 | 0.9809 | 0.9846 |
+| `test` | GlotLID | 0.9674 | 0.9858 | 0.9912 |
+| `test` | fastText | 0.2764 | 0.2506 | 0.1959 |
+| `test-common` | MillID | 0.9169 | 0.9696 | 0.9737 |
+| `test-common` | GlotLID | 0.9609 | 0.9885 | 0.9949 |
+| `test-common` | fastText | 0.9125 | 0.9581 | 0.9655 |
 
-MillID on each set:
+### MillID on each set
 
 | Set | Top-1 | Top-3 | Top-5 | Macro P | Macro R | Macro F1 | Median F1 |
 |---|---|---|---|---|---|---|---|
@@ -124,19 +153,7 @@ MillID on each set:
 | `udhr-common` | 0.9121 | 0.9633 | 0.9688 | 0.9770 | 0.9382 | 0.9480 | 0.9955 |
 | `udhr-all` | 0.7301 | 0.7875 | 0.8000 | 0.7415 | 0.7318 | 0.7184 | 0.9672 |
 
-Top-1 accuracy by passage length:
-
-| Set | System | Short | Sentence | Paragraph |
-|---|---|---|---|---|
-| `validation` | MillID | 0.9363 | 0.9818 | 0.9858 |
-| `test` | MillID | 0.9382 | 0.9809 | 0.9846 |
-| `test` | GlotLID | 0.9674 | 0.9858 | 0.9912 |
-| `test` | fastText | 0.2764 | 0.2506 | 0.1959 |
-| `test-common` | MillID | 0.9169 | 0.9696 | 0.9737 |
-| `test-common` | GlotLID | 0.9609 | 0.9885 | 0.9949 |
-| `test-common` | fastText | 0.9125 | 0.9581 | 0.9655 |
-
-### Per language: test
+#### Per language: test
 
 F1 language by language, GlotLID and fastText restricted to MillID's labels: against GlotLID, MillID is better on **110** languages, worse on **776** and within 0.01 on 735; against fastText, MillID is better on **1616** languages, worse on **0** and within 0.01 on 5. Every language, with precision, recall and the unrestricted scores, is in `reports/test/leaderboard.csv`.
 
@@ -205,7 +222,7 @@ Where MillID loses most to GlotLID:
 | `mbd_Latn` | Dibabawon Manobo | 14 | 0.757 | 1.000 | 0.000 |
 | `lnd_Latn` | Lundayeh | 5 | 0.769 | 1.000 | 0.000 |
 
-### Per language: udhr
+#### Per language: udhr
 
 F1 language by language, GlotLID and fastText restricted to MillID's labels: against GlotLID, MillID is better on **63** languages, worse on **95** and within 0.01 on 200; against fastText, MillID is better on **304** languages, worse on **13** and within 0.01 on 41. Every language, with precision, recall and the unrestricted scores, is in `reports/udhr/leaderboard.csv`.
 
@@ -274,83 +291,129 @@ Where MillID loses most to GlotLID:
 | `maz_Latn` | Central Mazahua | 59 | 0.742 | 0.939 | 0.000 |
 | `rmn_Latn` | Balkan Romani | 116 | 0.667 | 0.857 | 0.000 |
 
-### Per language: test-common
+Precision, recall (sensitivity), F1 and specificity for every language and system, restricted and as it is, are in `reports/<set>/`, and the totals in `reports/summary.json`.
 
-F1 language by language, GlotLID and fastText restricted to MillID's labels: against GlotLID, MillID is better on **6** languages, worse on **142** and within 0.01 on 65; against fastText, MillID is better on **93** languages, worse on **47** and within 0.01 on 73. Every language, with precision, recall and the unrestricted scores, is in `reports/test-common/leaderboard.csv`.
+## Quick start
 
-The 20 languages with the most passages:
+Install the `millid` package, with ONNX Runtime for the CPU (`[gpu]` for CUDA):
 
-| Label | Language | Passages | MillID | GlotLID | fastText |
-|---|---|---|---|---|---|
-| `fra_Latn` | French | 1652 | 0.974 | 0.955 | 0.963 |
-| `eng_Latn` | English | 1444 | 0.941 | 0.877 | 0.857 |
-| `spa_Latn` | Spanish | 1136 | 0.953 | 0.958 | 0.941 |
-| `tam_Taml` | Tamil | 754 | 0.998 | 1.000 | 0.997 |
-| `arb_Arab` | Arabic (Modern Standard) | 745 | 0.905 | 0.886 | 0.954 |
-| `por_Latn` | Portuguese | 657 | 0.963 | 0.956 | 0.949 |
-| `kan_Knda` | Kannada | 601 | 0.987 | 0.994 | 0.999 |
-| `zul_Latn` | Zulu | 594 | 0.871 | 0.904 | 0.970 |
-| `swh_Latn` | Swahili | 592 | 0.897 | 0.953 | 0.945 |
-| `mar_Deva` | Marathi | 589 | 0.981 | 0.990 | 0.987 |
-| `tso_Latn` | Tsonga | 581 | 0.984 | 0.956 | 0.986 |
-| `hun_Latn` | Hungarian | 576 | 0.991 | 1.000 | 0.980 |
-| `afr_Latn` | Afrikaans | 569 | 0.976 | 0.985 | 0.986 |
-| `tel_Telu` | Telugu | 555 | 1.000 | 0.998 | 0.999 |
-| `rus_Cyrl` | Russian | 528 | 0.929 | 0.985 | 0.927 |
-| `ory_Orya` | Odia (individual language) | 511 | 0.996 | 1.000 | 0.996 |
-| `sot_Latn` | Southern Sotho | 491 | 0.967 | 0.978 | 0.856 |
-| `nso_Latn` | Northern Sotho | 479 | 0.961 | 0.970 | 0.967 |
-| `asm_Beng` | Assamese | 467 | 0.995 | 0.999 | 0.997 |
-| `ben_Beng` | Bangla | 466 | 0.987 | 0.997 | 0.989 |
+```bash
+pip install "millid[cpu] @ git+https://github.com/baobab-tech/millid.git"
+```
 
-Where MillID gains most over GlotLID:
+Identify a passage, several, or a whole document:
 
-| Label | Language | Passages | MillID | GlotLID | fastText |
-|---|---|---|---|---|---|
-| `ewo_Latn` | Ewondo | 134 | 0.965 | 0.883 | 0.860 |
-| `eng_Latn` | English | 1444 | 0.941 | 0.877 | 0.857 |
-| `arb_Latn` | Standard Arabic | 163 | 0.944 | 0.913 | 0.852 |
-| `tso_Latn` | Tsonga | 581 | 0.984 | 0.956 | 0.986 |
-| `fra_Latn` | French | 1652 | 0.974 | 0.955 | 0.963 |
-| `arb_Arab` | Arabic (Modern Standard) | 745 | 0.905 | 0.886 | 0.954 |
-| `knc_Latn` | Central Kanuri | 293 | 0.959 | 0.952 | 0.981 |
-| `por_Latn` | Portuguese | 657 | 0.963 | 0.956 | 0.949 |
-| `san_Deva` | Sanskrit | 239 | 0.996 | 0.990 | 0.971 |
-| `bug_Latn` | Buginese | 78 | 0.987 | 0.981 | 0.938 |
-| `kam_Latn` | Kamba | 272 | 0.980 | 0.974 | 0.863 |
-| `mlt_Latn` | Maltese | 291 | 0.997 | 0.993 | 0.961 |
-| `tel_Telu` | Telugu | 555 | 1.000 | 0.998 | 0.999 |
-| `sin_Sinh` | Sinhala | 232 | 1.000 | 1.000 | 0.998 |
-| `pan_Guru` | Punjabi | 460 | 1.000 | 1.000 | 0.990 |
+```python
+from millid import MillID
 
-Where MillID loses most to GlotLID:
+model = MillID.from_pretrained("baobabtech/millid-medium-v1")
+model.predict("Lave men ou ak savon anvan ou manje.", k=3)
+# [Prediction(label='hat_Latn', language='Haitian Creole', probability=...), ...]
+model.predict(["one text", "another"], k=3)  # one list of answers per text
+model.identify_document(long_text)           # any length: its languages and segments
+```
 
-| Label | Language | Passages | MillID | GlotLID | fastText |
-|---|---|---|---|---|---|
-| `knc_Arab` | Central Kanuri | 68 | 0.562 | 0.860 | 0.029 |
-| `tzm_Tfng` | Central Atlas Tamazight | 157 | 0.645 | 0.934 | 0.751 |
-| `hrv_Latn` | Croatian | 168 | 0.717 | 0.994 | 0.856 |
-| `bos_Latn` | Bosnian | 198 | 0.727 | 1.000 | 0.897 |
-| `zsm_Latn` | Standard Malay | 244 | 0.773 | 0.998 | 0.908 |
-| `taq_Tfng` | Tamasheq | 104 | 0.613 | 0.805 | 0.000 |
-| `ind_Latn` | Indonesian | 335 | 0.807 | 0.991 | 0.887 |
-| `kin_Latn` | Kinyarwanda | 210 | 0.818 | 0.981 | 0.854 |
-| `run_Latn` | Rundi | 291 | 0.843 | 0.985 | 0.855 |
-| `nob_Latn` | Norwegian Bokmål | 265 | 0.820 | 0.959 | 0.894 |
-| `fuv_Latn` | Nigerian Fulfulde | 164 | 0.797 | 0.892 | 0.859 |
-| `mag_Deva` | Magahi | 97 | 0.905 | 1.000 | 0.772 |
-| `vec_Latn` | Venetian | 49 | 0.766 | 0.861 | 0.620 |
-| `cjk_Latn` | Chokwe | 24 | 0.909 | 1.000 | 0.807 |
-| `taq_Latn` | Tamasheq | 106 | 0.909 | 1.000 | 0.838 |
-
-Per-language precision, recall (sensitivity), F1 and specificity for every system, restricted and as it is, are in `reports/<set>/`, and the totals in `reports/summary.json`.
-
-## Training
+From the command line:
 
-1,944,095 passages; the best validation macro F1 at step 56,000, after 3.4 hours on 1 GPU. Languages sampled in proportion to n^0.3; 25% of passages cut to a 20 to 80 character snippet; batch 512, AdamW at 0.0003, cosine schedule, label smoothing 0.1.
+```bash
+millid predict "Lave men ou ak savon anvan ou manje." --model baobabtech/millid-medium-v1 -k 3
+```
+
+In JavaScript, in Node or a browser (`js/` in [baobab-tech/millid](https://github.com/baobab-tech/millid); not on npm yet):
+
+```js
+import { MillID } from "@baobabtech/millid";   // "@baobabtech/millid/web" in a browser, with { ort }
+const model = await MillID.fromPretrained("baobabtech/millid-medium-v1");
+await model.predict("Lave men ou ak savon anvan ou manje.", 3);
+```
+
+`millid.py` in this repository is the PyTorch training code, which reads `model.safetensors` to fine-tune or export.
+
+## Best practices
+
+### 1. Pass the text as it is
+
+The encoder applies Unicode NFKC and collapses whitespace itself. Accents, diacritics and the script are much of what it reads, so do not strip, transliterate or romanise text before passing it.
+
+### 2. Read long text as a document
+
+`predict` reads the first 255 bytes of a passage. For anything longer, `identify_document` reads consecutive windows (at most 256, spread evenly over a longer text), weighs their probabilities by length, and reports the document's languages and the runs of each.
+
+### 3. Treat a low top probability as unsure
+
+In our tests on real PDFs, a text layer whose characters were mapped to the wrong Unicode code points got a top probability below 0.5 from every MillID model, and reading the same pages from their images gave the right language. A top answer below 0.5 is worth a second look; but see the legacy fonts under Limitations, where that check does not hold.
+
+### 4. Expect less from very short text
+
+On `test`, top-1 accuracy is 0.9382 for passages of 20 to 80 characters against 0.9846 for paragraphs. For a heading or a table cell, look at the top three answers rather than the first alone.
+
+### 5. One text alone and the same text in a batch can score differently
+
+A batch is lower-cased as one string, so a dotted capital `İ` anywhere in it changes how every text in it is lower-cased (one Greek passage scored 0.976 alone and 0.985 beside Turkish text). The Python and JavaScript runtimes do this identically; send one text at a time where scores must not depend on their neighbours.
+
+### 6. Choose the size for where it runs
+
+tiny v1.5 on a CPU or in a web page; medium v1 or large v1.5 on a GPU, large v1.5 when accuracy matters most. The three share their labels, so the choice changes no code.
+
+## Model data
+
+MillID was trained on passages cut from pages the way a page is read: short lines, sentences and paragraphs.
+
+* **PDFs:** [FinePDFs](https://huggingface.co/datasets/HuggingFaceFW/finepdfs), PDF text from Common Crawl, labelled by GlotLID.
+* **Web pages:** [FineWeb-2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2), web text from Common Crawl, labelled by GlotLID.
+* **Storybooks:** [GlotStoryBook](https://huggingface.co/datasets/cis-lmu/GlotStoryBook) (African Storybook, Pratham Books, LIDA, Little Cree Books) and Nal'ibali's stories in South African languages, labelled by their publishers.
+* **News in sparse languages:** [GlotSparse](https://huggingface.co/datasets/cis-lmu/GlotSparse), labelled by the sites.
+* **Publishers' documents:** [CAWST](https://washresources.cawst.org)'s training material and the [Save the Children Resource Centre](https://resourcecentre.savethechildren.net)'s reports and guidance, each document labelled with the language its publisher catalogued.
+* **Not trained on:** the Universal Declaration of Human Rights, kept for testing; passages quoting it were removed from training.
 
 The training data is the full MillID build. The public [baobabtech/millid-dataset](https://huggingface.co/datasets/baobabtech/millid-dataset) is that build without Save the Children's and Nal'ibali's text (1.6% of its passages), which is not ours to redistribute. Its splits are the same, so its `test` split is the one scored above, without those passages.
 
+### Data processing
+
+* **Passages:** Unicode NFKC, which folds the ligatures and presentation forms PDF text layers are full of; URLs, emails and dot leaders removed; a passage must be mostly letters. Exact duplicates were removed across every source, and no PDF gives more than 40 passages, article 20 or web page 10.
+* **Labels checked:** a publisher's label is kept unless GlotLID is confident (≥ 0.8) the passage is in another major language, typically a copyright line; a passage from FinePDFs or FineWeb-2 is kept only when GlotLID's reading of the passage agrees with its document's.
+* **Text that is no language removed:** passages GlotLID is confident are `zxx` (no linguistic content), mostly PDFs whose fonts map letters to the wrong code points, and documents catalogued in the wrong language.
+* **Splits by document:** no test passage shares a document with a training one.
+
+### Training
+
+1,944,095 passages; the best validation macro F1 at step 56,000, after 3.4 hours on 1 GPU. Languages sampled in proportion to n^0.3; 25% of passages cut to a 20 to 80 character snippet; batch 512, AdamW at 0.0003, cosine schedule, label smoothing 0.1.
+
+## Architecture
+
+![MillID's architecture: a passage is normalised, encoded per UTF-8 byte into byte, script, byte-in-character and hashed character n-gram ids, embedded and summed, passed through a convolutional stem, downsampling and pre-norm transformer blocks, mean-pooled, and scored against label, language and script embeddings](assets/millid-architecture.png)
+
+*Drawn at the medium preset's sizes (12 blocks, d = 768); this model's are in the table above.*
+
+UTF-8 bytes (after Unicode NFKC), each embedded as byte + Unicode script + byte-in-character + hashed character 2-, 3- and 4-grams (32,768 buckets) -> depthwise convolution stem (kernel 5) -> strided convolution, 2 bytes per token -> 12 pre-norm blocks (RMSNorm, attention with RoPE, 12 heads, SwiGLU; d=768) -> masked mean pooling -> language head, each language's vector the sum of its own, its language's and its script's. See `ARCHITECTURE.md`.
+
+## Usage and limitations
+
+### Intended usage
+
+* **Routing documents:** choosing the OCR, translation or language model for a document by its language.
+* **Building and filtering corpora:** sorting web or PDF text by language, low-resource languages included.
+* **Describing collections:** which languages a set of documents holds, and how much of each.
+* **On-device detection:** in a browser or on a phone with tiny v1.5, where text should not leave the device.
+
+### Limitations
+
+* **Close languages:** languages that differ by a few words are its weakest (F1 on `test`: Croatian 0.69, Bosnian 0.67, Malay 0.75, Indonesian 0.78, Norwegian Bokmål 0.81). Their short passages are often genuinely ambiguous.
+* **Text that is no language:** there is no "no language" class. Nepali typed in the legacy Preeti font, whose text layer is Latin letters standing for Devanagari, was labelled as another language by MillID models, some with a top probability up to 0.74.
+* **One label per passage:** a passage that mixes languages gets one; `identify_document` gives the segments of a longer text.
+* **Labels learned from GlotLID:** the FinePDFs and FineWeb-2 passages were labelled by GlotLID, so MillID learned its labels, mistakes included; `udhr`, labelled by translators, is the independent test.
+* **Genre:** for many low-resource languages the training text is mostly religious web text or children's stories, and a model can learn the genre along with the language.
+
+### Ethical considerations and risks
+
+* **Errors fall on the smallest languages:** filtering or routing text by language can drop or mishandle the text of speakers of low-resource and closely related languages, where MillID is least accurate. Check the per-language scores of the languages that matter before filtering with it.
+* **A language is not an identity:** do not use a language label to infer a person's nationality, ethnicity or origin.
+* **Labels can be contested:** whether two varieties are one language or two is political as well as linguistic. MillID's labels follow ISO 639-3 and its training data, not a judgement on that question.
+
+### Benefits
+
+MillID medium v1 is within 0.018 macro F1 of GlotLID, a 1.7 GB fastText model, on the 1,621 languages of `test`, and within 0.009 on `udhr`. All three sizes are open under Apache-2.0, with the code that trained and evaluated them and the dataset, less the text that is not ours to share.
+
 ## Licence
 
 The weights and the code are Apache-2.0. The training data carries its own licence row by row (see the [dataset card](https://huggingface.co/datasets/baobabtech/millid-dataset#licensing)), including text licensed for non-commercial use only (CC BY-NC, CC BY-NC-SA).