Model card: provenance, verification, Latin-script scope

README.md

@@ -0,0 +1,140 @@
+---
+license: afl-3.0
+base_model: Davlan/bert-base-multilingual-cased-ner-hrl
+base_model_relation: quantized
+library_name: transformers.js
+pipeline_tag: token-classification
+tags:
+  - token-classification
+  - named-entity-recognition
+  - ner
+  - onnx
+  - transformers.js
+  - multilingual
+  - on-device
+language:
+  - de
+  - en
+  - es
+  - fr
+  - it
+  - lv
+  - nl
+  - pt
+---
+
+# bert-base-multilingual-cased-ner-hrl-latin
+
+`Davlan/bert-base-multilingual-cased-ner-hrl`, quantized to int8 and stripped of
+the vocabulary a Latin-script deployment can never tokenize.
+
+**178.5 MB → 139.6 MB (−21.8%), with provably identical output** for the
+languages it still covers. No encoder weight was touched. This is a packaging
+change, not a distillation — there is no accuracy trade to evaluate.
+
+## Why it works
+
+`bert-base-multilingual-cased` carries a 119,547-token vocabulary spanning 104
+languages. At 768 dimensions that embedding table is **~92 MB of the 178 MB
+file — more than half the model**, and every user downloads all of it. If you
+ship Latin-script languages, the Cyrillic, CJK, Arabic, Hangul, Hebrew,
+Devanagari, Greek, Armenian and Thai rows are downloaded by everyone and
+gathered by no one.
+
+Removing them is exact rather than approximate, for two reasons:
+
+- The embedding is **per-tensor quantized** — a scalar scale and zero-point
+  shared by the whole tensor — and is read by a plain `Gather`. Selecting a
+  subset of rows preserves both the int8 values and their dequantization. No
+  weight is re-quantized and nothing drifts.
+- **WordPiece is longest-match over the available pieces.** Only pieces
+  unreachable from a kept script were removed, so segmentation of Latin text is
+  unchanged too.
+
+Vocabulary: **119,547 → 68,875 tokens** (57.6% kept).
+
+## Verification
+
+Not asserted — measured. Both models were run over 77 real documents (contracts,
+financial statements, medical records) in 7 languages, comparing **raw logits**
+rather than entity lists, because entity lists can agree by luck:
+
+| | |
+| --- | --- |
+| documents | 77 |
+| entities compared | 2,219 |
+| segmentation differences | **0** |
+| entity differences | **0** |
+| max abs. logit delta | **0.0000000000** |
+
+The comparison is over raw logits and fails on any nonzero difference. **Verify
+on your own corpus before shipping** — the guarantee is exact for kept scripts,
+but "kept" depends on which scripts you selected, and a corpus containing a
+dropped script will (correctly) diverge.
+
+## ⚠️ Latin script only
+
+Text in a dropped script is **not supported** — it degrades to `[UNK]`, not to
+"slightly worse". Relative to the original model:
+
+- **Kept:** German, English, Spanish, French, Italian, Latvian, Dutch,
+  Portuguese — and any other Latin-script language the base model handled.
+- **Removed:** **Arabic and Chinese**, which the original supports and this
+  model does not.
+
+If you need those, use
+[`Xenova/bert-base-multilingual-cased-ner-hrl`](https://huggingface.co/Xenova/bert-base-multilingual-cased-ner-hrl)
+— it is the model this one is built from, unchanged apart from the vocabulary.
+
+The same technique keeps whichever scripts you do ship: retaining kana and
+ideographs alongside Latin lands at ~151 MB, still 27 MB under the original.
+
+## Labels
+
+`PER`, `ORG`, `LOC`, `DATE` — unchanged from the base model, as `B-`/`I-` tags.
+
+## Usage (transformers.js)
+
+```js
+import { pipeline } from "@huggingface/transformers";
+
+const ner = await pipeline(
+  "token-classification",
+  "promptshield/bert-base-multilingual-cased-ner-hrl-latin",
+  { dtype: "q8" },
+);
+
+await ner("Le contrat a été signé par Jean-Baptiste Lefèvre à Bordeaux.", {
+  aggregation_strategy: "simple",
+});
+```
+
+It also loads in `onnxruntime` directly — it is a standard BERT token-classification
+graph with `input_ids`, `attention_mask` and `token_type_ids`.
+
+## Provenance and licence
+
+| | |
+| --- | --- |
+| base | [`google-bert/bert-base-multilingual-cased`](https://huggingface.co/google-bert/bert-base-multilingual-cased) — Apache-2.0 |
+| fine-tune | [`Davlan/bert-base-multilingual-cased-ner-hrl`](https://huggingface.co/Davlan/bert-base-multilingual-cased-ner-hrl) — **AFL-3.0** |
+| ONNX export | [`Xenova/bert-base-multilingual-cased-ner-hrl`](https://huggingface.co/Xenova/bert-base-multilingual-cased-ner-hrl) |
+| this repo | vocabulary pruning only — **AFL-3.0**, inherited |
+
+All credit for the model itself belongs to **David Adelani (Davlan)** for the
+fine-tune and to the Google Research team for the base model. We changed the
+vocabulary and nothing else.
+
+The upstream card declares no training datasets; if your use has data-provenance
+requirements, check with the original authors rather than relying on this card.
+
+## Who made this and why
+
+Built for [promptShield](https://promptshield.ca), an offline document
+anonymizer. Its web app runs detection **entirely in the browser** — no document
+ever leaves the device — so the model is a first-visit download, and 39 MB of
+unreachable vocabulary was 39 MB charged to every visitor.
+
+Published because the technique generalises: the embedding table is over half of
+most multilingual encoders, and almost every deployment ships a fraction of the
+104 languages.