README.md

---
license: apache-2.0
library_name: transformers
pipeline_tag: token-classification
tags: [onnx, ner, multilingual, token-classification, distillation, openmasq]
language: [en, fr, de, es, it, nl]
base_model: google-bert/bert-base-multilingual-cased
---

# openmasq/ner-multilingual

On-device multilingual NER for [OpenMasq](https://openmasq.com), a privacy tool that redacts
personal data **before** a prompt leaves the machine. These weights run in the desktop app and
in the local proxy, on CPU, offline.

A **6-layer student of `Davlan/bert-base-multilingual-cased-ner-hrl`**, started from the
Apache-2.0 mBERT base, with the wordpiece vocabulary reduced to Latin script: **97.2 M
parameters, 94 MB as int8 ONNX** — against 177.3 M and 178 MB for the model it imitates, for
the same score on held-out text and a little more on the product's own cases.

| | Davlan (teacher) | this model |
|---|---:|---:|
| encoder blocks | 12 | 6 |
| vocabulary (wordpieces) | 119 547 | 70 606 |
| parameters | 177.3 M | 97.2 M |
| ONNX int8 | 178 MB | **94 MB** |
| CPU latency, 419 characters (M1, int8) | 59.5 ms | **32.4 ms** |
| weights derived from the teacher | — | **none** |

## Labels

The teacher's tag set, unchanged — `O`, `B-/I-PER`, `B-/I-ORG`, `B-/I-LOC`, `B-/I-DATE`. A
drop-in replacement: same `id2label`, same pipeline.

## How it was made

The student is initialised from `google-bert/bert-base-multilingual-cased` with a fresh
classification head, keeps 6 of its 12 blocks (0, 2, 4, 7, 9, 11 — evenly spaced, the
XtremeDistil shape) and every Latin-script wordpiece. It is then trained with three terms:

```
loss = 0.7 · KL( student/T ‖ teacher/T ) · T²        T = 2, on every real token
     + 0.3 · CE( student, gold )                      on the 5 % of cases that carry gold
     + 1.0 · MSE( hidden_s[i], hidden_t[kept[i]] )    on the 7 mapped layers, embeddings included
```

- **KL against the teacher** on every token: the student copies Davlan's full distribution,
  hesitations included. This is what preserves everything the teacher does.
- **Cross-entropy on gold** where a source claims labels: 540 documents of the UNER
  English-EWT *train* split (human) and 3 000 prose documents annotated by Claude through a
  scored bench column (0.900 F1 against UNER's human gold, 96 % recall on organisations).
  A token nothing annotated is *imitated, not scored as `O`*; a token the teacher reads as a
  date is left to the KL term, since no source annotates dates.
- **Hidden-state transfer** on the mapping the block selection actually made (Udagawa et al.
  2023 place it above output-distribution transfer alone). Measured: +0.002 to +0.003 F1 on
  every corpus over the same recipe without it.

Corpus: 65 114 documents — Wikipedia prose in six languages, synthetic records, generated
credential carriers — plus 20 % shown a second time in capitals and 20 % in lower case
(Mayhew et al. 2019). 3 epochs, 21 195 steps, lr 5e-5, batch 16, windows of 256 with stride
48. Final agreement with the teacher on held-out windows: **0.957 F1**.

### Why the pruned vocabulary is safe on Latin text

WordPiece is greedy longest-match, so a word whose pieces all survive the cut splits exactly
as before; keeping *every* Latin-script piece makes that true of any Latin-script word.
Verified rather than assumed: the teacher itself with the pruned vocabulary and no training
returns predictions identical to the original on **127 of 127** documents of the Text
Anonymization Benchmark.

## Measured

Character-level F1 on the categories the product exposes, **in the product** (rules + model,
the level a user gets). `TAB` (European Court of Human Rights judgments) and `UNER` (the
English Web Treebank's *test* and *validation* splits: e-mails, newsgroups, reviews, weblogs)
are real text held out from everything either model was trained on; the other three are the
public PII benchmarks' scoring splits.

| corpus | documents | previous model (Davlan) | this model |
|---|---:|---:|---:|
| ai4privacy | 2 000 | 0.827 | 0.825 |
| Gretel | 2 000 | 0.646 | 0.646 |
| Nemotron | 2 000 | 0.928 | **0.938** |
| TAB — real, held out | 127 | 0.862 | 0.861 |
| OpenMasq internal | 907 | 0.923 | **0.925** |
| UNER English-EWT — real, held out | 634 | 0.674 | **0.682** |

Three corpora up, two down by a thousandth, one level. Where precision was measured
separately (TAB, internal, UNER) it is 0.001 to 0.017 above the teacher's, and the gains are
precision rather than recall — the supervision taught the student where entities are *not*
more than where they are, which is what 91 % of the supervised tokens say.

## Limitations — read this before deploying

- **Non-Latin scripts are gone.** Chinese, Arabic, Cyrillic, Japanese, Korean, Hebrew, Bengali
  and every other non-Latin writing system were removed with the vocabulary; their characters
  become `[UNK]` and entities in them are **not detected at all**. The teacher covered ten
  languages including Chinese and Arabic. A deliberate trade for a product serving six
  European languages — disqualifying for anything else.
- **Brand knowledge is weaker than the teacher's.** Well-formed company names it has not seen
  (`Royal Caribbean`, `Hobby Lobby`, `Travelocity`, `BBC`) are recognised less often: 70 % of
  organisation characters on UNER against the teacher's 74 %. Three training runs with five
  times more supervised organisations did not move that figure — it looks like a property of
  the 2018 pre-training, not of the corpus. In OpenMasq a rules-side lexicon is the planned
  answer; on its own, this model is not the one to pick for entity linking over web text.
- **Quantisation is part of the measurement.** Every figure above is the int8 ONNX graph,
  which is what ships — not the fp32 checkpoint.
- It predicts `DATE`, which OpenMasq maps to an opt-in category; ignore that tag if you do
  not want it.

## Files

| file | what reads it |
|---|---|
| `onnx/model_quantized.onnx` | onnxruntime, in the desktop app and the local proxy |
| `config.json` | the label set |
| `tokenizer.json`, `tokenizer_config.json` | the pruned wordpiece tokenizer |
| `vocab.txt` | the same vocabulary for the slow tokenizer |

OpenMasq pins each of the first four by sha256 and verifies them before onnxruntime parses a
byte, so **this repository is the origin, never the trust anchor**: a change here can fail a
build, it cannot substitute bytes.

```
onnx/model_quantized.onnx  7470819a0961b246614ceedd940934b8d8ae7c39070cb15508883fe0e6f93012
config.json                b942516293db81db39bc12b734888dc5e7a7e38c2f5f6f7ecacb337fd716c4c3
tokenizer.json             01c1e79cf5f607729b4e64af721b961d41981c77c91849a9fd4f7262b4cc9437
tokenizer_config.json      5c1d1f0110ec8761dc91247fb20bebfea28aa577e35b2f0eb02c55cfb69871b1
```

## Licence

**Apache-2.0.** The weights are initialised from `google-bert/bert-base-multilingual-cased`
(Apache-2.0) and contain no parameter of any other model. They were trained to imitate the
*outputs* of `Davlan/bert-base-multilingual-cased-ner-hrl` (AFL-3.0); whether learning from a
model's answers makes a derivative of that model is a contested question that this card does
not settle — it states the provenance so the reader can judge. The earlier release in this
repository (commit `fa87beb`) *was* initialised from Davlan's weights and was declared
AFL-3.0 for that reason.

Training data, for the record: WikiNER (CC-BY 4.0), Universal NER English-EWT train split
(CC-BY-SA 4.0), ai4privacy PII-Masking-300k, Nemotron-PII and Gretel PII Finance under their
respective terms, and synthetic credential carriers generated in-house. TAB and UNER's scoring
splits were used for evaluation only.

## Citation

- Mukherjee & Hassan Awadallah, *XtremeDistil*, ACL 2020 — the shallow-student shape.
- Hinton, Vinyals & Dean, *Distilling the Knowledge in a Neural Network*, 2015 — the KL term.
- Udagawa et al., *A Comparative Analysis of Task-Agnostic Distillation Methods*, 2023 —
  hidden-state transfer over output transfer alone.
- Mayhew, Tsygankova & Roth, *ner and pos when nothing is capitalized*, EMNLP 2019 — the
  casing augmentation.
- Monaikul et al., 2021; Zhang et al., 2023 — the old model as teacher in the loss, and the
  semantic shift of the non-entity type it prevents.
- Adelani et al. / `Davlan` — the teacher. Mayhew et al., *Universal NER*, 2024; Pilán et al.,
  *The Text Anonymization Benchmark*, CL 2022 — the held-out corpora.