baobabtech/millid-tiny-v1.5

GitHub | Leaderboard | Dataset | Architecture
License: Apache 2.0 | Authors: Baobab Tech

MillID tiny v1.5

MillID (MultiLingual Language IDentifier) is an open language identification model built by Baobab Tech. It names the language of a passage of text, from a table cell to a paragraph, among 1,641 language-script labels: 1,550 languages written in 56 scripts, and 37 labels for text in a script whose language it cannot name (und_<Script>). MillID tiny v1.5 has 2.9M parameters and an ONNX file of 11 MB.

Small enough to run on a CPU, a phone or in a web page (491 passages a second on four Arm cores, 48 a second on one laptop thread), it identifies language where the text is, with no server and no GPU.

MillID offers:

• Language and script: labels are ISO 639-3 plus ISO 15924 script, so Serbian in Cyrillic (srp_Cyrl) and in Latin (srp_Latn) are told apart. Many of the languages are low-resource, with a few hundred passages each to learn from.
• No tokenizer: it reads UTF-8 bytes, each with its Unicode script and the character n-grams ending there, so any script is input it can read and nothing is an unknown token.
• Short text: a quarter of its training passages were cut to 20 to 80 characters, the length of a heading or a table cell, where language identification is hardest.
• Whole documents: identify_document reads text of any length in windows and returns the document's languages and the segments of each.
• No server, no PyTorch: one ONNX file and a small encoder, in Python (NumPy) or JavaScript (Node and browsers); the two are tested to give the same answers.
• Three sizes, one label set: tiny v1.5, medium v1 and large v1.5 share their labels, so changing size changes no code.

Model overview

Parameters	Total	2.9M
	Embeddings	935K
	Convolution stem	38K
	Transformer blocks	1.3M
	Language head	625K
Architecture	Blocks	3
	Model dimension	192
	Attention heads	3
	Bytes per token	1
	N-gram buckets	4,096
	Normalisation	RMSNorm, pre-norm
	Positions	RoPE
	Feed-forward	SwiGLU
	Pooling	Masked mean
Input and output	Input	UTF-8 bytes, NFKC
	Passage length	Up to 255 bytes
	Labels	1,641
	Runtime	ONNX (11 MB)

Benchmark results

MillID was evaluated on two test sets: test, passages from documents never seen in training, split from the training data by document; and UDHR-LID (udhr), the Universal Declaration of Human Rights labelled by its translators. It is compared with GlotLID and Meta's fastText language identifier (NLLB, 218 labels), each restricted to MillID's labels, its best answer among them: the like-for-like comparison.

Overall results

Set	Languages	Metric	MillID	GlotLID	fastText
test	1,621	Macro F1	0.9554	0.9755	0.0659
		Top-1	0.9587	0.9783	0.2515
test-common	213	Macro F1	0.9513	0.9846	0.9173
		Top-1	0.9396	0.9749	0.9352
udhr	358	Macro F1	0.8764	0.8873	0.3499
		Top-1	0.8492	0.8614	0.4629
udhr-common	170	Macro F1	0.9513	0.9659	0.9329
		Top-1	0.9123	0.9338	0.9195
udhr-all	429	Macro F1	0.7149	0.7245	0.2797
		Top-1	0.7274	0.7378	0.3964

• test: every language, 1,621 of them. test-common: the languages all three systems know, the head-to-head on equal terms, since fastText cannot answer most of MillID's.
• udhr: the Declaration in the languages MillID knows. udhr-common: in the languages all three know. udhr-all: in every language it has, MillID's unknown ones counted wrong.
• GlotLID's labels came from training on data that includes UDHR, and it labelled the FinePDFs and FineWeb-2 passages in test, so its scores are optimistic on both.

The three MillID models

Macro F1 on test and udhr; this model in bold. All three, language by language, are on the leaderboard.

Model	Parameters	ONNX	Test F1	UDHR F1	Runs on
MillID tiny v1.5	2.9M	11 MB	0.9554	0.8764	CPU, web page
MillID medium v1	115M	455 MB	0.9576	0.8780	GPU
MillID large v1.5	193M	764 MB	0.9646	0.8812	GPU

By passage length

Top-1 accuracy on passages of 20 to 80 characters (short), up to 300 (sentence) and up to 1,000 (paragraph):

Set	System	Short	Sentence	Paragraph
validation	MillID	0.9312	0.9788	0.9850
test	MillID	0.9333	0.9797	0.9833
test	GlotLID	0.9674	0.9858	0.9912
test	fastText	0.2764	0.2506	0.1959
test-common	MillID	0.9125	0.9685	0.9725
test-common	GlotLID	0.9609	0.9885	0.9949
test-common	fastText	0.9125	0.9581	0.9655

MillID on each set

Set	Top-1	Top-3	Top-5	Macro P	Macro R	Macro F1	Median F1
validation	0.9578	0.9889	0.9933	0.9539	0.9591	0.9551	0.9822
test	0.9587	0.9890	0.9932	0.9550	0.9588	0.9554	0.9825
test-common	0.9396	0.9857	0.9909	0.9709	0.9353	0.9513	0.9726
udhr	0.8492	0.9117	0.9257	0.9148	0.8736	0.8764	0.9907
udhr-common	0.9123	0.9606	0.9655	0.9794	0.9393	0.9513	1.0000
udhr-all	0.7274	0.7809	0.7928	0.7340	0.7290	0.7149	0.9649

Per language: test

F1 language by language, GlotLID and fastText restricted to MillID's labels: against GlotLID, MillID is better on 103 languages, worse on 869 and within 0.01 on 649; against fastText, MillID is better on 1616 languages, worse on 0 and within 0.01 on 5. Every language, with precision, recall and the unrestricted scores, is in reports/test/leaderboard.csv.

The 20 languages with the most passages:

Label	Language	Passages	MillID	GlotLID	fastText
fra_Latn	French	1652	0.929	0.928	0.808
eng_Latn	English	1444	0.917	0.859	0.648
spa_Latn	Spanish	1136	0.930	0.951	0.574
tam_Taml	Tamil	754	0.997	0.993	0.965
arb_Arab	Arabic (Modern Standard)	745	0.814	0.785	0.512
por_Latn	Portuguese	657	0.947	0.949	0.755
kan_Knda	Kannada	601	0.969	0.993	0.922
zul_Latn	Zulu	594	0.846	0.896	0.539
swh_Latn	Swahili	592	0.848	0.916	0.346
mar_Deva	Marathi	589	0.965	0.990	0.926
tso_Latn	Tsonga	581	0.965	0.956	0.425
hun_Latn	Hungarian	576	0.987	0.999	0.232
afr_Latn	Afrikaans	569	0.967	0.985	0.860
tel_Telu	Telugu	555	0.991	0.994	0.951
rus_Cyrl	Russian	528	0.932	0.949	0.462
nbl_Latn	South Ndebele	528	0.903	0.949	0.000
ory_Orya	Odia (individual language)	511	0.988	0.994	0.976
sot_Latn	Southern Sotho	491	0.966	0.978	0.660
nso_Latn	Northern Sotho	479	0.956	0.969	0.914
kua_Latn	Kuanyama	476	0.962	0.986	0.000

Where MillID gains most over GlotLID:

Label	Language	Passages	MillID	GlotLID	fastText
lbj_Tibt	Ladakhi	66	0.943	0.000	0.000
ain_Latn	Ainu	1	0.667	0.000	0.000
ajp_Arab	South Levantine Arabic	28	0.946	0.581	0.000
und_Laoo	und	20	0.923	0.621	0.000
und_Thaa	und	29	0.881	0.651	0.000
sgh_Cyrl	Shughni	9	0.857	0.667	0.000
und_Geor	und	32	0.877	0.694	0.000
tuk_Arab	Turkmen	135	0.905	0.761	0.000
und_Syrc	und	43	0.966	0.822	0.000
gmh_Latn	Middle High German	177	0.922	0.786	0.000
orv_Cyrl	Old Russian	239	0.931	0.800	0.000
und_Kana	und	136	0.982	0.862	0.000
und_Yiii	und	38	1.000	0.884	0.000
neb_Latn	Toura (Côte d'Ivoire)	11	0.957	0.842	0.000
dag_Latn	Dagbani	149	0.767	0.660	0.000

Where MillID loses most to GlotLID:

Label	Language	Passages	MillID	GlotLID	fastText
mus_Latn	Muscogee	2	0.333	1.000	0.000
ttj_Latn	Tooro	28	0.485	0.946	0.000
mbd_Latn	Dibabawon Manobo	14	0.596	1.000	0.000
knc_Arab	Central Kanuri	68	0.462	0.825	0.007
biu_Latn	Biete	7	0.667	1.000	0.000
dks_Latn	Southeastern Dinka	3	0.667	1.000	0.000
esi_Latn	North Alaskan Inupiatun	9	0.455	0.783	0.000
hrv_Latn	Croatian	168	0.622	0.944	0.178
plw_Latn	Brooke's Point Palawano	8	0.727	1.000	0.000
bos_Latn	Bosnian	198	0.649	0.902	0.181
xbr_Latn	Kambera	8	0.750	1.000	0.000
nyf_Latn	Giryama	10	0.645	0.870	0.000
ldi_Latn	Laari	62	0.777	0.992	0.000
hak_Hani	Hakka Chinese	29	0.621	0.824	0.000
ccp_Latn	Chakma	3	0.800	1.000	0.000

Per language: udhr

F1 language by language, GlotLID and fastText restricted to MillID's labels: against GlotLID, MillID is better on 52 languages, worse on 105 and within 0.01 on 201; against fastText, MillID is better on 303 languages, worse on 15 and within 0.01 on 40. Every language, with precision, recall and the unrestricted scores, is in reports/udhr/leaderboard.csv.

The 20 languages with the most passages:

Label	Language	Passages	MillID	GlotLID	fastText
roh_Latn	Romansh	399	0.994	0.999	0.946
oci_Latn	Occitan	398	0.277	0.376	0.435
cmn_Hani	Chinese (Mandarin)	319	0.892	0.868	0.468
krl_Latn	Karelian	183	0.000	0.011	0.000
qug_Latn	Chimborazo Highland Quichua	182	0.011	0.000	0.000
acu_Latn	Achuar-Shiwiar	180	0.828	0.854	0.000
hus_Latn	Huastec	175	0.977	0.986	0.000
hau_Latn	Hausa	155	0.990	1.000	0.876
auc_Latn	Waorani	154	0.000	0.000	0.000
kwi_Latn	Awa-Cuaiquer	145	0.880	0.877	0.000
jpn_Jpan	Japanese	140	1.000	1.000	0.944
snn_Latn	Siona	136	0.630	0.514	0.000
ktu_Latn	Kituba (Democratic Republic of Congo)	118	0.765	0.680	0.000
lat_Latn	Latin	118	0.996	0.992	0.000
hat_Latn	Haitian Creole	117	0.992	0.979	0.699
nya_Latn	Nyanja	116	0.991	1.000	0.806
kmb_Latn	Kimbundu	116	0.970	1.000	0.877
rmn_Latn	Balkan Romani	116	0.678	0.857	0.000
twi_Latn	Twi	116	0.983	0.996	0.396
chr_Cher	Cherokee	115	0.670	0.663	0.000

Where MillID gains most over GlotLID:

Label	Language	Passages	MillID	GlotLID	fastText
mic_Latn	Mi'kmaw	59	0.992	0.427	0.000
tet_Latn	Tetum	58	0.638	0.167	0.000
fuf_Latn	Pular	58	0.906	0.483	0.000
sag_Latn	Sango	59	0.992	0.578	0.189
skr_Arab	Saraiki	63	0.867	0.500	0.000
qvh_Latn	Huamalíes-Dos de Mayo Huánuco Quechua	58	0.468	0.154	0.000
zgh_Tfng	Standard Moroccan Tamazight	60	0.750	0.462	0.000
lld_Latn	Ladin	59	0.909	0.731	0.000
tah_Latn	Tahitian	57	0.973	0.812	0.966
yue_Hani	Cantonese	49	0.734	0.580	0.111
ote_Latn	Mezquital Otomi	59	0.154	0.000	0.000
pnb_Arab	Western Panjabi	61	0.882	0.739	0.000
mxv_Latn	Metlatónoc Mixtec	58	0.129	0.000	0.000
lij_Latn	Ligurian	58	0.983	0.859	0.487
bum_Latn	Bulu	58	0.215	0.098	0.000

Where MillID loses most to GlotLID:

Label	Language	Passages	MillID	GlotLID	fastText
gyr_Latn	Guarayu	49	0.537	0.990	0.000
shp_Latn	Shipibo-Conibo	62	0.469	0.912	0.000
dyu_Latn	Dyula	58	0.000	0.434	0.015
tbz_Latn	Ditammari	60	0.000	0.378	0.000
cbu_Latn	Candoshi-Shapra	57	0.500	0.874	0.000
idu_Latn	Idoma	51	0.603	0.938	0.000
ell_Grek	Greek	115	0.670	1.000	1.000
qvc_Latn	Cajamarca Quechua	58	0.648	0.906	0.000
cpu_Latn	Pichis Ashéninka	60	0.729	0.974	0.000
qvn_Latn	North Junín Quechua	57	0.219	0.460	0.000
hin_Deva	Hindi	59	0.723	0.957	0.975
hrv_Latn	Croatian	58	0.500	0.727	0.474
kbp_Latn	Kabiyè	59	0.781	1.000	0.389
cni_Latn	Asháninka	52	0.773	0.963	0.000
rmn_Latn	Balkan Romani	116	0.678	0.857	0.000

Precision, recall (sensitivity), F1 and specificity for every language and system, restricted and as it is, are in reports/<set>/, and the totals in reports/summary.json.

Quick start

Install the millid package, with ONNX Runtime for the CPU ([gpu] for CUDA):

pip install "millid[cpu] @ git+https://github.com/baobab-tech/millid.git"

Identify a passage, several, or a whole document:

from millid import MillID

model = MillID.from_pretrained("baobabtech/millid-tiny-v1.5")
model.predict("Lave men ou ak savon anvan ou manje.", k=3)
# [Prediction(label='hat_Latn', language='Haitian Creole', probability=...), ...]
model.predict(["one text", "another"], k=3)  # one list of answers per text
model.identify_document(long_text)           # any length: its languages and segments

From the command line:

millid predict "Lave men ou ak savon anvan ou manje." --model baobabtech/millid-tiny-v1.5 -k 3

In JavaScript, in Node or a browser, with @baobabtechai/millid from npm:

npm install @baobabtechai/millid onnxruntime-node   # onnxruntime-web in a browser app

import { MillID } from "@baobabtechai/millid";   // "@baobabtechai/millid/web" in a browser, with { ort }
const model = await MillID.fromPretrained("baobabtech/millid-tiny-v1.5");
await model.predict("Lave men ou ak savon anvan ou manje.", 3);

millid.py in this repository is the PyTorch training code, which reads model.safetensors to fine-tune or export.

Best practices

1. Pass the text as it is

The encoder applies Unicode NFKC and collapses whitespace itself. Accents, diacritics and the script are much of what it reads, so do not strip, transliterate or romanise text before passing it.

2. Read long text as a document

predict reads the first 255 bytes of a passage. For anything longer, identify_document reads consecutive windows (at most 256, spread evenly over a longer text), weighs their probabilities by length, and reports the document's languages and the runs of each.

3. Treat a low top probability as unsure

In our tests on real PDFs, a text layer whose characters were mapped to the wrong Unicode code points got a top probability below 0.5 from every MillID model, and reading the same pages from their images gave the right language. A top answer below 0.5 is worth a second look; but see the legacy fonts under Limitations, where that check does not hold.

4. Expect less from very short text

On test, top-1 accuracy is 0.9333 for passages of 20 to 80 characters against 0.9833 for paragraphs. For a heading or a table cell, look at the top three answers rather than the first alone.

5. One text alone and the same text in a batch can score differently

A batch is lower-cased as one string, so a dotted capital İ anywhere in it changes how every text in it is lower-cased (one Greek passage scored 0.976 alone and 0.985 beside Turkish text). The Python and JavaScript runtimes do this identically; send one text at a time where scores must not depend on their neighbours.

6. Choose the size for where it runs

tiny v1.5 on a CPU or in a web page; medium v1 or large v1.5 on a GPU, large v1.5 when accuracy matters most. The three share their labels, so the choice changes no code.

Model data

MillID was trained on passages cut from pages the way a page is read: short lines, sentences and paragraphs.

• PDFs: FinePDFs, PDF text from Common Crawl, labelled by GlotLID.
• Web pages: FineWeb-2, web text from Common Crawl, labelled by GlotLID.
• Storybooks: GlotStoryBook (African Storybook, Pratham Books, LIDA, Little Cree Books) and Nal'ibali's stories in South African languages, labelled by their publishers.
• News in sparse languages: GlotSparse, labelled by the sites.
• Publishers' documents: CAWST's training material and the Save the Children Resource Centre's reports and guidance, each document labelled with the language its publisher catalogued.
• Not trained on: the Universal Declaration of Human Rights, kept for testing; passages quoting it were removed from training.

The training data is the full MillID build. The public baobabtech/millid-dataset is that build without Save the Children's and Nal'ibali's text (1.6% of its passages), which is not ours to redistribute. Its splits are the same, so its test split is the one scored above, without those passages.

Data processing

• Passages: Unicode NFKC, which folds the ligatures and presentation forms PDF text layers are full of; URLs, emails and dot leaders removed; a passage must be mostly letters. Exact duplicates were removed across every source, and no PDF gives more than 40 passages, article 20 or web page 10.
• Labels checked: a publisher's label is kept unless GlotLID is confident (≥ 0.8) the passage is in another major language, typically a copyright line; a passage from FinePDFs or FineWeb-2 is kept only when GlotLID's reading of the passage agrees with its document's.
• Text that is no language removed: passages GlotLID is confident are zxx (no linguistic content), mostly PDFs whose fonts map letters to the wrong code points, and documents catalogued in the wrong language.
• Splits by document: no test passage shares a document with a training one.

Training

1,943,957 passages; the best validation macro F1 at step 106,000, after 0.6 hours on 1 GPU. Languages sampled in proportion to n^0.3; 25% of passages cut to a 20 to 80 character snippet; batch 512, AdamW at 0.001, cosine schedule, label smoothing 0.1.

Architecture

Drawn at the medium preset's sizes (12 blocks, d = 768); this model's are in the table above.

UTF-8 bytes (after Unicode NFKC), each embedded as byte + Unicode script + byte-in-character + hashed character 2-, 3- and 4-grams (4,096 buckets) -> depthwise convolution stem (kernel 5) -> strided convolution, 1 byte per token -> 3 pre-norm blocks (RMSNorm, attention with RoPE, 3 heads, SwiGLU; d=192) -> masked mean pooling -> language head, each language's vector the sum of its own, its language's and its script's. See ARCHITECTURE.md.

Usage and limitations

Intended usage

• Routing documents: choosing the OCR, translation or language model for a document by its language.
• Building and filtering corpora: sorting web or PDF text by language, low-resource languages included.
• Describing collections: which languages a set of documents holds, and how much of each.
• On-device detection: in a browser or on a phone with tiny v1.5, where text should not leave the device.

Limitations

• Close languages: languages that differ by a few words are its weakest (F1 on test: Croatian 0.62, Bosnian 0.65, Malay 0.77, Indonesian 0.81, Norwegian Bokmål 0.81). Their short passages are often genuinely ambiguous.
• Text that is no language: there is no "no language" class. Nepali typed in the legacy Preeti font, whose text layer is Latin letters standing for Devanagari, was labelled as another language by MillID models, some with a top probability up to 0.74.
• One label per passage: a passage that mixes languages gets one; identify_document gives the segments of a longer text.
• Labels learned from GlotLID: the FinePDFs and FineWeb-2 passages were labelled by GlotLID, so MillID learned its labels, mistakes included; udhr, labelled by translators, is the independent test.
• Genre: for many low-resource languages the training text is mostly religious web text or children's stories, and a model can learn the genre along with the language.

Ethical considerations and risks

• Errors fall on the smallest languages: filtering or routing text by language can drop or mishandle the text of speakers of low-resource and closely related languages, where MillID is least accurate. Check the per-language scores of the languages that matter before filtering with it.
• A language is not an identity: do not use a language label to infer a person's nationality, ethnicity or origin.
• Labels can be contested: whether two varieties are one language or two is political as well as linguistic. MillID's labels follow ISO 639-3 and its training data, not a judgement on that question.

Benefits

MillID tiny v1.5 is within 0.020 macro F1 of GlotLID, a 1.7 GB fastText model, on the 1,621 languages of test, and within 0.011 on udhr. At 11 MB, it loads in a web page or ships inside an app. All three sizes are open under Apache-2.0, with the code that trained and evaluated them and the dataset, less the text that is not ours to share.

Licence

The weights and the code are Apache-2.0. The training data carries its own licence row by row (see the dataset card), including text licensed for non-commercial use only (CC BY-NC, CC BY-NC-SA).