--- license: apache-2.0 library_name: transformers pipeline_tag: token-classification tags: [onnx, ner, multilingual, token-classification, distillation, openmasq] language: [en, fr, de, es, it, nl] base_model: google-bert/bert-base-multilingual-cased --- # openmasq/ner-multilingual On-device multilingual NER for [OpenMasq](https://openmasq.com), a privacy tool that redacts personal data **before** a prompt leaves the machine. These weights run in the desktop app and in the local proxy, on CPU, offline. A **6-layer student of `Davlan/bert-base-multilingual-cased-ner-hrl`**, started from the Apache-2.0 mBERT base, with the wordpiece vocabulary reduced to Latin script: **97.2 M parameters, 94 MB as int8 ONNX** — against 177.3 M and 178 MB for the model it imitates, for the same score on held-out text and a little more on the product's own cases. | | Davlan (teacher) | this model | |---|---:|---:| | encoder blocks | 12 | 6 | | vocabulary (wordpieces) | 119 547 | 70 606 | | parameters | 177.3 M | 97.2 M | | ONNX int8 | 178 MB | **94 MB** | | CPU latency, 419 characters (M1, int8) | 59.5 ms | **32.4 ms** | | weights derived from the teacher | — | **none** | ## Labels The teacher's tag set, unchanged — `O`, `B-/I-PER`, `B-/I-ORG`, `B-/I-LOC`, `B-/I-DATE`. A drop-in replacement: same `id2label`, same pipeline. ## How it was made The student is initialised from `google-bert/bert-base-multilingual-cased` with a fresh classification head, keeps 6 of its 12 blocks (0, 2, 4, 7, 9, 11 — evenly spaced, the XtremeDistil shape) and every Latin-script wordpiece. It is then trained with three terms: ``` loss = 0.7 · KL( student/T ‖ teacher/T ) · T² T = 2, on every real token + 0.3 · CE( student, gold ) on the 5 % of cases that carry gold + 1.0 · MSE( hidden_s[i], hidden_t[kept[i]] ) on the 7 mapped layers, embeddings included ``` - **KL against the teacher** on every token: the student copies Davlan's full distribution, hesitations included. This is what preserves everything the teacher does. - **Cross-entropy on gold** where a source claims labels: 540 documents of the UNER English-EWT *train* split (human) and 3 000 prose documents annotated by Claude through a scored bench column (0.900 F1 against UNER's human gold, 96 % recall on organisations). A token nothing annotated is *imitated, not scored as `O`*; a token the teacher reads as a date is left to the KL term, since no source annotates dates. - **Hidden-state transfer** on the mapping the block selection actually made (Udagawa et al. 2023 place it above output-distribution transfer alone). Measured: +0.002 to +0.003 F1 on every corpus over the same recipe without it. Corpus: 65 114 documents — Wikipedia prose in six languages, synthetic records, generated credential carriers — plus 20 % shown a second time in capitals and 20 % in lower case (Mayhew et al. 2019). 3 epochs, 21 195 steps, lr 5e-5, batch 16, windows of 256 with stride 48. Final agreement with the teacher on held-out windows: **0.957 F1**. ### Why the pruned vocabulary is safe on Latin text WordPiece is greedy longest-match, so a word whose pieces all survive the cut splits exactly as before; keeping *every* Latin-script piece makes that true of any Latin-script word. Verified rather than assumed: the teacher itself with the pruned vocabulary and no training returns predictions identical to the original on **127 of 127** documents of the Text Anonymization Benchmark. ## Measured Character-level F1 on the categories the product exposes, **in the product** (rules + model, the level a user gets). `TAB` (European Court of Human Rights judgments) and `UNER` (the English Web Treebank's *test* and *validation* splits: e-mails, newsgroups, reviews, weblogs) are real text held out from everything either model was trained on; the other three are the public PII benchmarks' scoring splits. | corpus | documents | previous model (Davlan) | this model | |---|---:|---:|---:| | ai4privacy | 2 000 | 0.827 | 0.825 | | Gretel | 2 000 | 0.646 | 0.646 | | Nemotron | 2 000 | 0.928 | **0.938** | | TAB — real, held out | 127 | 0.862 | 0.861 | | OpenMasq internal | 907 | 0.923 | **0.925** | | UNER English-EWT — real, held out | 634 | 0.674 | **0.682** | Three corpora up, two down by a thousandth, one level. Where precision was measured separately (TAB, internal, UNER) it is 0.001 to 0.017 above the teacher's, and the gains are precision rather than recall — the supervision taught the student where entities are *not* more than where they are, which is what 91 % of the supervised tokens say. ## Limitations — read this before deploying - **Non-Latin scripts are gone.** Chinese, Arabic, Cyrillic, Japanese, Korean, Hebrew, Bengali and every other non-Latin writing system were removed with the vocabulary; their characters become `[UNK]` and entities in them are **not detected at all**. The teacher covered ten languages including Chinese and Arabic. A deliberate trade for a product serving six European languages — disqualifying for anything else. - **Brand knowledge is weaker than the teacher's.** Well-formed company names it has not seen (`Royal Caribbean`, `Hobby Lobby`, `Travelocity`, `BBC`) are recognised less often: 70 % of organisation characters on UNER against the teacher's 74 %. Three training runs with five times more supervised organisations did not move that figure — it looks like a property of the 2018 pre-training, not of the corpus. In OpenMasq a rules-side lexicon is the planned answer; on its own, this model is not the one to pick for entity linking over web text. - **Quantisation is part of the measurement.** Every figure above is the int8 ONNX graph, which is what ships — not the fp32 checkpoint. - It predicts `DATE`, which OpenMasq maps to an opt-in category; ignore that tag if you do not want it. ## Files | file | what reads it | |---|---| | `onnx/model_quantized.onnx` | onnxruntime, in the desktop app and the local proxy | | `config.json` | the label set | | `tokenizer.json`, `tokenizer_config.json` | the pruned wordpiece tokenizer | | `vocab.txt` | the same vocabulary for the slow tokenizer | OpenMasq pins each of the first four by sha256 and verifies them before onnxruntime parses a byte, so **this repository is the origin, never the trust anchor**: a change here can fail a build, it cannot substitute bytes. ``` onnx/model_quantized.onnx 7470819a0961b246614ceedd940934b8d8ae7c39070cb15508883fe0e6f93012 config.json b942516293db81db39bc12b734888dc5e7a7e38c2f5f6f7ecacb337fd716c4c3 tokenizer.json 01c1e79cf5f607729b4e64af721b961d41981c77c91849a9fd4f7262b4cc9437 tokenizer_config.json 5c1d1f0110ec8761dc91247fb20bebfea28aa577e35b2f0eb02c55cfb69871b1 ``` ## Licence **Apache-2.0.** The weights are initialised from `google-bert/bert-base-multilingual-cased` (Apache-2.0) and contain no parameter of any other model. They were trained to imitate the *outputs* of `Davlan/bert-base-multilingual-cased-ner-hrl` (AFL-3.0); whether learning from a model's answers makes a derivative of that model is a contested question that this card does not settle — it states the provenance so the reader can judge. The earlier release in this repository (commit `fa87beb`) *was* initialised from Davlan's weights and was declared AFL-3.0 for that reason. Training data, for the record: WikiNER (CC-BY 4.0), Universal NER English-EWT train split (CC-BY-SA 4.0), ai4privacy PII-Masking-300k, Nemotron-PII and Gretel PII Finance under their respective terms, and synthetic credential carriers generated in-house. TAB and UNER's scoring splits were used for evaluation only. ## Citation - Mukherjee & Hassan Awadallah, *XtremeDistil*, ACL 2020 — the shallow-student shape. - Hinton, Vinyals & Dean, *Distilling the Knowledge in a Neural Network*, 2015 — the KL term. - Udagawa et al., *A Comparative Analysis of Task-Agnostic Distillation Methods*, 2023 — hidden-state transfer over output transfer alone. - Mayhew, Tsygankova & Roth, *ner and pos when nothing is capitalized*, EMNLP 2019 — the casing augmentation. - Monaikul et al., 2021; Zhang et al., 2023 — the old model as teacher in the loss, and the semantic shift of the non-entity type it prevents. - Adelani et al. / `Davlan` — the teacher. Mayhew et al., *Universal NER*, 2024; Pilán et al., *The Text Anonymization Benchmark*, CL 2022 — the held-out corpora.