2.0.0

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

CHANGELOG.md

@@ -8,6 +8,35 @@ Semver for this island:
 
 Accuracy is 841-dev, union at 0.95. Through 1.1.2 the gold has 354 spans; from 1.2.0 it is restated to phone policy v3 (315 spans), and the two are not comparable. A version that changes those numbers must update `hybrid.json` `eval` and the tables below.
 
+## 2.0.0
+
+Same NERGAL weights and threshold. Rules SHA `d1866243…`: the placeholder rename only; every rule span is unchanged. Major because the output placeholder changes.
+
+- **Breaking — `[PHONE]` replaces `[Telefon]`.** Every phone mask now reads `[PHONE]`; `[PII]` is unchanged. Code that matches `[Telefon]` in the output must switch. Text scrubbed by 1.x still works as input: `[Telefon]` stays a rules boundary (a previous redaction, not a fresh `Telefon` label), like `[PHONE]` and `[PII]`.
+- **Names, opt-in:** `Nergal.from_pretrained(..., names=True)` (CLI `--names`) loads `names/`, FastPDN NER — Polish PII by ArkadiuszPawlak (CC-BY-4.0, `names/NOTICE.md`), converted from ONNX to safetensors with no value changed. Its `PERSON`, `PERSON_F` and `PERSON_L` tags become `[PERSON]`: whole words, joined across spaces, tabs and no-break spaces (not punctuation or line breaks); a word touching a placeholder is skipped. It always runs in float32: `dtype` sets NERGAL only. Without `names=True`, `from_pretrained` skips `names/` (≈490 MB) and the output equals 1.2.0's except for the rename.
+- **Union:** each character takes its highest-ranked label, phone > pii > person, and each run becomes one placeholder. A name inside a masked email stays `[PII]`.
+- **Counts:** `scrub()` counts gain `person` (0 with names off). `union_placeholder_chars` includes `[PERSON]` placeholders. `predict()` / `predict_many()` still return NERGAL spans only; `nergal.names.spans_many(texts)` returns the person spans.
+
+841-dev, names off: unchanged from 1.2.0 (whole 303/315, residual 9, rules FP 24, union FP 80, char P 98.59%, char R 97.83%). Its gold has no person labels, so names-on is not scored on it. The benchmark replay at 2.0.0 with names off gives 1.2.0's numbers on every one of its 12 cohorts.
+
+**Name test (`nergal_names_test_v1`).** Dynaword and Polish mC4 web passages (70/30 in each panel), component-disjoint from every earlier set, labelled under `docs/pii-annotation-names.md`, frozen before any prediction and scored once. The random panel estimates the corpus; the targeted panel is enriched by selection (correspondence, signatures, honorifics, hard negatives) and says nothing about prevalence. Panels are never pooled. A name is whole when every letter and digit of it is masked (`[PII]` counts); a component is the independent unit, covered when every name in it is whole. Names-on, float32:
+
+| Panel | Passages | Names | Whole | Exact | Components covered (95% CI) | Person false chars | Passages with one (95% CI) |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| Random | 300 | 614 | 586 (95.44%) | 565 | 98/116 (76.6–90.5%) | 1,051 | 34 (8.0–15.5%) |
+| Targeted | 150 | 292 | 286 (97.95%) | 285 | 94/99 (88.6–98.3%) | 976 | 42 (21.0–35.9%) |
+
+Names off hides no name; its false characters are 1 (random) and 29 (targeted). The set's phones (2 and 7) and other PII (2 and 5) are whole in both modes. For the record, neither shipped: float16 on Apple MPS equals float32 on both panels; FastPDN's INT8 ONNX on CPU hides 589 and 287 names whole with 1,009 and 935 person false characters. One reviewer labelled the set; the second pass is a same-day blind self-review of 100 passages (11 disagreed, adjudication changed 5), not independent agreement. `hybrid.json` `eval_names` holds these numbers and the score report's SHA.
+
+Release gates. The name test replaces the planned 200-document spot check: a spot check reviews only what the model masked, so it measures neither recall nor names missed on clean-looking text. Throughput, local: 200 seeded Dynaword documents (1,103,229 characters), Apple M4 Max, MPS, float32, `scrub_many`, one process, off and on alternating twice. Names off 7,914 and 7,922 chars/s, peak RSS 1,393 and 1,388 MiB; names on 7,444 and 7,597 chars/s (−5%), 1,679 and 1,682 MiB (+290 MiB). MPS driver memory at the end of a run is 70,261 MiB off and 87,788 MiB on; that is the allocator cache, not a working set. Repeat runs mask identically; names on gives 1,309 `[PERSON]` placeholders, names off none, and both give 7 `[PHONE]` and 7 `[PII]`. CUDA reference from the runtime study (RTX 4090, 1,000 Dynaword documents, 4,215,149 characters, a prototype of this runtime with NERGAL float16 and names float32): 3 processes 83,479 → 72,662 chars/s (−13%), peak device memory 9,012 → 12,602 MiB; per process, torch peak 1,806 → 2,668 MiB. A float16 names model changed spans in 11 of those 1,000 documents, so it is not offered.
+
+Known limits: names-on masks every person the policy covers, public officials and historical figures included, which is why it is opt-in. On the random panel 18 of 116 components keep part of a name, and 34 of 300 passages have a false person mask. The names model was fine-tuned partly on LLM-synthetic data that SlayerLab has not audited.
+
+| Version | Whole /315 | Residual | Rules FP | Union FP | Char P | Char R | What changed |
+|---|---:|---:|---:|---:|---:|---:|---|
+| 1.2.0 | 303 | 9 | 24 | 80 | 98.59% | 97.83% | Phone policy v3 at mask time, rules and model spans |
+| 2.0.0, names off | 303 | 9 | 24 | 80 | 98.59% | 97.83% | `[PHONE]` replaces `[Telefon]`; opt-in `[PERSON]` names |
+
 ## 1.2.0
 
 Same weights, threshold and API. Rules SHA `b238d5b8…`. The rules file is the frozen and scored candidate `73c185f7…` with two comments reworded; its syntax tree is identical. Phone policy v3 (labelling policy amended 2026-10-02) now applies at mask time, to the rules and to the model spans.

README.md

@@ -1,5 +1,8 @@
 ---
-license: mit
+# MIT for NERGAL (code, rules, weights); names/ (FastPDN NER — Polish PII) is CC-BY-4.0, see names/NOTICE.md
+license: other
+license_name: mit-and-cc-by-4.0
+license_link: https://huggingface.co/SlayerLab/NERGAL/blob/main/README.md#license
 language:
 - pl
 base_model: FacebookAI/xlm-roberta-large
@@ -13,7 +16,7 @@ tags:
 - hybrid
 ---
 
-# NERGAL 1.2.0
+# NERGAL 2.0.0
 
 **Named Entity Recognition with Grounded Additive Labels**
 
@@ -21,16 +24,17 @@ SlayerLab hybrid PII cleaner for Polish. Not a chat model. Not a drop-in `pipeli
 
 ## TL;DR
 
-Python rules do the identifiers they can prove. A transformer NER head adds phone and other PII spans the regex misses. The cleaner **unions** the two on the original text, then replaces hits with `[Telefon]` or `[PII]`. Phone spans from both follow phone policy v3: one span per number of 7+ digits, and shorter numbers are not masked.
+Python rules do the identifiers they can prove. A transformer NER head adds phone and other PII spans the regex misses. The cleaner **unions** the two on the original text, then replaces hits with `[PHONE]` or `[PII]`. Phone spans from both follow phone policy v3: one span per number of 7+ digits, and shorter numbers are not masked. With `names=True`, a second model adds person names as `[PERSON]` ([Names](#names)).
 
 - **Ground:** `scrub_pii.py` rules
 - **Additive labels:** XLM-RoBERTa-large token classifier (epoch 5 of 7), BIO tags `phone` / `pii`, threshold 0.95
-- **Throughput:** about 80k chars/s on one RTX 4090 with `scrub_many` + `dtype="float16"` and 3 processes
+- **Names (opt-in):** [FastPDN NER — Polish PII](https://huggingface.co/ArkadiuszPawlak/fastpdn-ner-polish-pii) (CC-BY-4.0) in `names/`, float32
+- **Throughput:** about 80k chars/s on one RTX 4090 with `scrub_many` + `dtype="float16"` and 3 processes; names cost about 13%
 - **Changes:** `CHANGELOG.md`
 
 ## What NERGAL detects — and what it does not
 
-NERGAL masks **contact details and selected identifiers in Polish text**. It is not a general-purpose anonymizer: names, postal addresses and other personal information can remain in the output. Its corpus-masking policy also includes public, institutional and company contacts and identifiers.
+NERGAL masks **contact details and selected identifiers in Polish text**. It is not a general-purpose anonymizer: names (masked only with `names=True`), postal addresses and other personal information can remain in the output. Its corpus-masking policy also includes public, institutional and company contacts and identifiers.
 
 ### Detection scope
 
@@ -38,13 +42,14 @@ These are target categories, not a guarantee that every occurrence or format is
 
 | Category | Values in scope | Replacement |
 |---|---|---|
-| Phone contacts | Phone, fax and SMS contact numbers of 7+ digits (country and area codes and keypad letters count), including foreign and vanity numbers, one span per number; an extension or alternative ending stays inside its number. Emergency, helpline, service and other numbers under 7 digits are not masked on their own | `[Telefon]` |
+| Phone contacts | Phone, fax and SMS contact numbers of 7+ digits (country and area codes and keypad letters count), including foreign and vanity numbers, one span per number; an extension or alternative ending stays inside its number. Emergency, helpline, service and other numbers under 7 digits are not masked on their own | `[PHONE]` |
 | Email | Email addresses, including recoverable broken or incomplete addresses | `[PII]` |
 | Personal identifiers | PESEL, passport and identity-document numbers | `[PII]` |
 | Organization identifiers | NIP/VAT, REGON, KRS, GEMI, LEI and equivalent foreign company registration numbers; labelled DUNS, BDO and RPWDL register-book numbers | `[PII]` |
 | Financial identifiers | Bank/account numbers, including Polish accounts and foreign IBANs | `[PII]` |
 | Property identifiers | Land-register (księga wieczysta, KW) numbers | `[PII]` |
 | Electronic contacts and access | e-Doręczenia and ePUAP addresses, explicitly labelled numeric access PINs (including `pin=` URL values), GG account IDs | `[PII]` |
+| Person names, only with `names=True` | Names of people, public officials and historical figures included ([Names](#names)) | `[PERSON]` |
 
 Context matters: a number that resembles a phone or identifier is not automatically in scope. Rules use labels, format checks and, for some unlabelled identifiers, checksums; the model adds contextual detections. Coverage varies by category, and the aggregate benchmark below does not establish recall for every category.
 
@@ -52,7 +57,7 @@ Context matters: a number that resembles a phone or identifier is not automatica
 
 NERGAL is not designed to remove:
 
-- **Personal names**, including private individuals and public officials; organization names.
+- **Personal names** with names off, including private individuals and public officials; organization names in either mode.
 - **Postal/street addresses, dates of birth and ages.**
 - **Social-media handles, ordinary URLs and filenames.** An in-scope value inside a URL, such as a labelled numeric PIN, can still be masked.
 - **Vehicle registration plates and generic serial, model or version codes.**
@@ -61,9 +66,9 @@ NERGAL is not designed to remove:
 
 These are intended exclusions; false positives can still mask some of this content.
 
-### Known gaps in 1.2.0
+### Known gaps in 2.0.0
 
-Unlabelled phones and identifiers (the rules take a bare phone only in the grouped national forms `601 234 567` and `22 123 45 67`; a plain `601234567` needs a label or the model), unusual formatting and damaged text can escape detection. **VINs and obfuscated emails** (such as `name (at) domain.pl`) are approved annotation targets, but that approval alone does not establish reliable support in the released 1.2.0 model. Do not rely on it to remove them consistently. An email with a lowercase word glued onto its domain (`jan@firma.plkontakt`) is masked together with that word, and text glued before an address can be masked with it. A phone written as short parts joined by two slashes (`12 / 345 / 678`) is left as text. An identifier with a `[PII]` / `[Telefon]` placeholder inside it or right before it can be missed.
+Unlabelled phones and identifiers (the rules take a bare phone only in the grouped national forms `601 234 567` and `22 123 45 67`; a plain `601234567` needs a label or the model), unusual formatting and damaged text can escape detection. **VINs and obfuscated emails** (such as `name (at) domain.pl`) are approved annotation targets, but that approval alone does not establish reliable support in the released 2.0.0 model. Do not rely on it to remove them consistently. An email with a lowercase word glued onto its domain (`jan@firma.plkontakt`) is masked together with that word, and text glued before an address can be masked with it. A phone written as short parts joined by two slashes (`12 / 345 / 678`) is left as text. An identifier with a `[PII]` / `[PHONE]` / `[Telefon]` placeholder inside it or right before it can be missed. Names-on gaps are under [Names](#names).
 
 ## Versions
 
@@ -72,10 +77,28 @@ Unlabelled phones and identifiers (the rules take a bare phone only in the group
 | Version | Whole /315 | Residual | Rules FP | Union FP | Char P | Char R | What changed |
 |---|---:|---:|---:|---:|---:|---:|---|
 | 1.1.2 | 303 | 9 | 83 | 333 | 94.37% | 97.83% | Email boundary rules |
-| **1.2.0** | **303** | **9** | **24** | **80** | **98.59%** | **97.83%** | Phone policy v3 at mask time, rules and model spans |
+| 1.2.0 | 303 | 9 | 24 | 80 | 98.59% | 97.83% | Phone policy v3 at mask time, rules and model spans |
+| **2.0.0, names off** | **303** | **9** | **24** | **80** | **98.59%** | **97.83%** | `[PHONE]` replaces `[Telefon]`; opt-in `[PERSON]` names |
 
 1.0.0–1.1.2 were scored on the earlier 354-span gold (whole 323 → 326, union FP 133 → 80); per-version numbers are in `CHANGELOG.md`.
 
+## Names
+
+`names=True` (CLI `--names`) adds a second model, [FastPDN NER — Polish PII](https://huggingface.co/ArkadiuszPawlak/fastpdn-ner-polish-pii) by ArkadiuszPawlak, revision `636c57fe…`, fine-tuned from [clarin-pl/FastPDN](https://huggingface.co/clarin-pl/FastPDN) on filtered KPWr plus LLM-synthetic data. `names/` holds it converted from ONNX to safetensors with no value changed. **`names/` is licensed [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/), not MIT**; `names/NOTICE.md` has the attribution, the changes and the file hashes. NERGAL uses only its person tags (`PERSON`, `PERSON_F`, `PERSON_L`), masks whole words, and always runs it in float32.
+
+**Scope with names on.** The name test labels every person: private individuals, public officials, politicians and historical figures alike, plus fictional characters, nicknames, partial names and initials. Titles and roles stay outside the span. Names inside a street, patronage, organisation or work title, eponyms, usernames, and names inside emails or URLs are not person spans. Masking public figures removes who said, signed or wrote what, which is why names are opt-in. This is the labelling target; the model does not meet it everywhere.
+
+**Name test** (`nergal_names_test_v1`): Dynaword and Polish mC4 web passages, component-disjoint from every earlier set, labelled by one reviewer, frozen before any prediction and scored once. The random panel estimates the corpus; the targeted panel is enriched by selection and says nothing about prevalence. A name is whole when every letter and digit of it is masked.
+
+| Panel | Passages | Names | Whole | Components covered (95% CI) | Person false chars | Passages with one |
+|---|---:|---:|---:|---:|---:|---:|
+| Random | 300 | 614 | 586 (95.44%) | 98/116 (76.6–90.5%) | 1,051 | 34 (11.3%) |
+| Targeted | 150 | 292 | 286 (97.95%) | 94/99 (88.6–98.3%) | 976 | 42 (28.0%) |
+
+Names off hides no name. A component is the independent unit, covered when every name in it is whole: on the random panel 18 of 116 keep part of a name. The second pass is a same-day blind self-review of 100 passages by the same reviewer (11 disagreed), not independent agreement. `hybrid.json` `eval_names` has the full numbers, including float16 and INT8 scored for the record; neither ships.
+
+**Cost:** on one RTX 4090 with 3 processes, names took a prototype of this runtime from 83,479 to 72,662 chars/s (−13%) and peak device memory from 9,012 to 12,602 MiB. Local measurements are in `CHANGELOG.md`.
+
 ## Cue-less phone test
 
 841-dev holds few phones without a contact label. This targeted set does: 150 web passages selected for them, labelled and frozen before any prediction. 1.1.1 added the grouped national forms and raised phones wholly masked from 37 to 66 of 111, losing none. Its benchmark half, restated to phone policy v3 (70 passages): 1.2.0 wholly masks 62 of 83 values, with 33 false characters. The set is enriched by selection and has a single reviewer, so it says nothing about how common such phones are.
@@ -104,7 +127,7 @@ XLM-R and epoch 5 were chosen in September 2026, on the earlier gold and before
 
 ## Load
 
-Regex runs on the original text, the model adds spans at 0.95, then the two are unioned and replaced with `[Telefon]` / `[PII]`.
+Regex runs on the original text, the model adds spans at 0.95, then the two are unioned and replaced with `[PHONE]` / `[PII]` (and `[PERSON]` with `names=True`). Each character takes its highest-ranked label, phone > pii > person.
 
 ```python
 from pathlib import Path
@@ -116,11 +139,15 @@ sys.path.insert(0, str(root))
 from nergal import Nergal
 
 nergal = Nergal.from_pretrained(root, local_files_only=True)
-masked, counts = nergal.scrub(text)
+masked, counts = nergal.scrub(text)  # counts: phone, pii, person
 ```
 
+Names: `snapshot_download("SlayerLab/NERGAL")` fetches `names/` too; `Nergal.from_pretrained("SlayerLab/NERGAL", names=True)` downloads it only when asked. From the command line: `python nergal.py --names < in.txt > out.txt`.
+
 For many texts, `nergal.scrub_many(texts)` (or `predict_many` for the raw model spans) batches windows across texts: about 2× the throughput of calling `scrub` in a loop on a CUDA GPU. `from_pretrained(..., dtype="float16")` casts the weights at load time on CUDA or MPS: another 1.9× on an RTX 4090, and 841-dev union numbers are unchanged, but scores are not bit-identical to float32 (0.95 spans can differ in rare cases). For a corpus, run 2–3 processes per GPU, because float16 inference is limited by CPU-side tokenization. Measurements are in `CHANGELOG.md`.
 
-`hybrid.json` records the version, threshold, 841-dev `eval` block and weight hashes. `test_nergal.py` is synthetic (no corpus text): `python -m unittest test_nergal`.
+`hybrid.json` records the version, threshold, 841-dev `eval` block, weight hashes, the `names` source and file hashes, and the name test `eval_names`. `test_nergal.py` is synthetic (no corpus text): `python -m unittest test_nergal`.
+
+## License
 
-Base weights: [`FacebookAI/xlm-roberta-large`](https://huggingface.co/FacebookAI/xlm-roberta-large) revision `c23d21b0620b635a76227c604d44e43a9f0ee389` (MIT).
+NERGAL code, rules and weights: MIT. `names/`: CC-BY-4.0, attribution in `names/NOTICE.md`. Base weights: [`FacebookAI/xlm-roberta-large`](https://huggingface.co/FacebookAI/xlm-roberta-large) revision `c23d21b0620b635a76227c604d44e43a9f0ee389` (MIT).

hybrid.json

@@ -1,7 +1,7 @@
 {
   "full_name": "Named Entity Recognition with Grounded Additive Labels",
   "hub_id": "SlayerLab/NERGAL",
-  "version": "1.2.0",
+  "version": "2.0.0",
   "mode": "rules_union",
   "epoch": 5,
   "seed": 202609160,
@@ -64,5 +64,228 @@
       "model.safetensors": "2fa84ec6abd0c1b12ba0ba1548e89c9fca36fc58f03e66074db423b0e98ec8e6",
       "tokenizer.json": "3a690c2d605076ad3901d946d4c0145fbfb19fef2f01370c7388430aa9f31edf"
     }
+  },
+  "eval_names": {
+    "split": "nergal_names_test_v1",
+    "policy": "docs/pii-annotation-names.md",
+    "reviewers": 1,
+    "second_pass": {
+      "passages": 100,
+      "disagreeing": 11,
+      "adjudication_changed": 5,
+      "gap_days": 0
+    },
+    "note": "single reviewer; the second pass is a same-day blind self-review of a sample, not independent agreement",
+    "runtime": "fp32",
+    "devices": {
+      "fp32": "cpu",
+      "fp16": "mps",
+      "int8": "cpu"
+    },
+    "score_report_sha256": "9dec1517dae8fefb4857acab975d83d82b217373b1e7efdee56a685eaf9dc6b2",
+    "panels": {
+      "random": {
+        "passages": 300,
+        "systems": {
+          "fp16": {
+            "person_gold": 614,
+            "person_whole": 586,
+            "person_whole_exact": 565,
+            "name_recall": 0.9544,
+            "person_components": 116,
+            "covered_components": 98,
+            "coverage_ci95": [
+              0.7659,
+              0.9054
+            ],
+            "person_false_chars": 1051,
+            "person_false_passages": 34,
+            "person_false_ci95": [
+              0.0798,
+              0.1548
+            ],
+            "false_chars": 1052,
+            "false_mask_passages": 35,
+            "phone_gold": 2,
+            "phone_whole": 2,
+            "pii_gold": 2,
+            "pii_whole": 2
+          },
+          "fp32": {
+            "person_gold": 614,
+            "person_whole": 586,
+            "person_whole_exact": 565,
+            "name_recall": 0.9544,
+            "person_components": 116,
+            "covered_components": 98,
+            "coverage_ci95": [
+              0.7659,
+              0.9054
+            ],
+            "person_false_chars": 1051,
+            "person_false_passages": 34,
+            "person_false_ci95": [
+              0.0798,
+              0.1548
+            ],
+            "false_chars": 1052,
+            "false_mask_passages": 35,
+            "phone_gold": 2,
+            "phone_whole": 2,
+            "pii_gold": 2,
+            "pii_whole": 2
+          },
+          "int8": {
+            "person_gold": 614,
+            "person_whole": 589,
+            "person_whole_exact": 568,
+            "name_recall": 0.9593,
+            "person_components": 116,
+            "covered_components": 101,
+            "coverage_ci95": [
+              0.7957,
+              0.9258
+            ],
+            "person_false_chars": 1009,
+            "person_false_passages": 37,
+            "person_false_ci95": [
+              0.0883,
+              0.166
+            ],
+            "false_chars": 1010,
+            "false_mask_passages": 38,
+            "phone_gold": 2,
+            "phone_whole": 2,
+            "pii_gold": 2,
+            "pii_whole": 2
+          },
+          "off": {
+            "person_gold": 614,
+            "person_whole": 0,
+            "person_whole_exact": 0,
+            "name_recall": 0.0,
+            "person_components": 116,
+            "covered_components": 0,
+            "coverage_ci95": [
+              0.0,
+              0.0313
+            ],
+            "person_false_chars": 0,
+            "person_false_passages": 0,
+            "person_false_ci95": [
+              0.0,
+              0.0122
+            ],
+            "false_chars": 1,
+            "false_mask_passages": 1,
+            "phone_gold": 2,
+            "phone_whole": 2,
+            "pii_gold": 2,
+            "pii_whole": 2
+          }
+        }
+      },
+      "targeted": {
+        "passages": 150,
+        "systems": {
+          "fp16": {
+            "person_gold": 292,
+            "person_whole": 286,
+            "person_whole_exact": 285,
+            "name_recall": 0.9795,
+            "person_components": 99,
+            "covered_components": 94,
+            "coverage_ci95": [
+              0.8861,
+              0.9834
+            ],
+            "person_false_chars": 976,
+            "person_false_passages": 42,
+            "person_false_ci95": [
+              0.2098,
+              0.3591
+            ],
+            "false_chars": 1005,
+            "false_mask_passages": 43,
+            "phone_gold": 7,
+            "phone_whole": 7,
+            "pii_gold": 5,
+            "pii_whole": 5
+          },
+          "fp32": {
+            "person_gold": 292,
+            "person_whole": 286,
+            "person_whole_exact": 285,
+            "name_recall": 0.9795,
+            "person_components": 99,
+            "covered_components": 94,
+            "coverage_ci95": [
+              0.8861,
+              0.9834
+            ],
+            "person_false_chars": 976,
+            "person_false_passages": 42,
+            "person_false_ci95": [
+              0.2098,
+              0.3591
+            ],
+            "false_chars": 1005,
+            "false_mask_passages": 43,
+            "phone_gold": 7,
+            "phone_whole": 7,
+            "pii_gold": 5,
+            "pii_whole": 5
+          },
+          "int8": {
+            "person_gold": 292,
+            "person_whole": 287,
+            "person_whole_exact": 285,
+            "name_recall": 0.9829,
+            "person_components": 99,
+            "covered_components": 95,
+            "coverage_ci95": [
+              0.8998,
+              0.9889
+            ],
+            "person_false_chars": 935,
+            "person_false_passages": 41,
+            "person_false_ci95": [
+              0.2038,
+              0.352
+            ],
+            "false_chars": 964,
+            "false_mask_passages": 42,
+            "phone_gold": 7,
+            "phone_whole": 7,
+            "pii_gold": 5,
+            "pii_whole": 5
+          },
+          "off": {
+            "person_gold": 292,
+            "person_whole": 0,
+            "person_whole_exact": 0,
+            "name_recall": 0.0,
+            "person_components": 99,
+            "covered_components": 0,
+            "coverage_ci95": [
+              0.0,
+              0.0366
+            ],
+            "person_false_chars": 0,
+            "person_false_passages": 0,
+            "person_false_ci95": [
+              0.0,
+              0.0243
+            ],
+            "false_chars": 29,
+            "false_mask_passages": 2,
+            "phone_gold": 7,
+            "phone_whole": 7,
+            "pii_gold": 5,
+            "pii_whole": 5
+          }
+        }
+      }
+    }
   }
 }

names/NOTICE.md

@@ -14,7 +14,10 @@
 - `model.safetensors` is upstream `model.onnx` (FP32, sha256
   `11924bfd20eb478ee614e5e1184536825db4747753d341af7c04b940efa84645`) converted by `convert.py` (sha256
   `a698bc993b253fc70d960aa14af37a5088afb79c77dae70557c7056ce394333e`): 199 tensors copied, 73 linear weights
-  transposed from ONNX [in, out] to torch [out, in], no value changed.
+  transposed from ONNX [in, out] to torch [out, in], no value changed. `convert.py` is included: it reads the upstream
+  snapshot at that revision from the default Hugging Face cache and is create-only, so run a copy outside `names/`. A
+  rerun under another `safetensors` version can give a different file sha256 with every tensor identical: only the
+  `__metadata__` key order differs.
 - Review Gate 14, on 1,000 documents (1,714 windows, 614,466 tokens): torch FP32 against ONNX Runtime FP32, max
   absolute logit difference 3.4e-4 (within 1e-3), every token's argmax identical, every person range identical.
 - NERGAL uses only the PERSON, PERSON_F and PERSON_L tags, masked as `[PERSON]`.
@@ -26,4 +29,5 @@
 | `config.json` | `3cef51cd7d184478adbdec9f68730db279386fe145cb32473a0622edea923578` |
 | `model.safetensors` | `2fa84ec6abd0c1b12ba0ba1548e89c9fca36fc58f03e66074db423b0e98ec8e6` |
 | `tokenizer.json` | `3a690c2d605076ad3901d946d4c0145fbfb19fef2f01370c7388430aa9f31edf` |
+| `convert.py` | `a698bc993b253fc70d960aa14af37a5088afb79c77dae70557c7056ce394333e` |
 | `NOTICE.md` | this file |

names/convert.py

@@ -0,0 +1,88 @@
+"""FastPDN-PII FP32 ONNX -> model.safetensors for BertForTokenClassification (plan Task 4a Step 2, Review Gate 14 A).
+
+Named initializers keep their torch names. Each linear weight is the MatMul input that feeds the Add or BiasGelu taking
+its named bias, transposed from ONNX's [in, out] to torch's [out, in]. Every weight is used once, the state dict loads
+strictly, and no unused float tensor is left in the graph; anything else stops. Weights are copied, never changed.
+Usage: python convert.py    # writes model.safetensors and convert.json here; create-only
+"""
+from pathlib import Path
+import hashlib, json
+
+import numpy as np
+from onnx import numpy_helper
+
+HERE = Path(__file__).resolve().parent
+REPO, REVISION = 'ArkadiuszPawlak/fastpdn-ner-polish-pii', '636c57fe77b0afad8f065e7c76243dc051437035'
+SNAPSHOT = Path.home()/'.cache/huggingface/hub/models--ArkadiuszPawlak--fastpdn-ner-polish-pii/snapshots'/REVISION
+ONNX_SHA256 = '11924bfd20eb478ee614e5e1184536825db4747753d341af7c04b940efa84645'  # model.onnx, LFS pin at REVISION
+CONFIG_SHA256 = '3cef51cd7d184478adbdec9f68730db279386fe145cb32473a0622edea923578'
+NAMED = ('bert.', 'classifier.')
+BUFFERS = ('bert.embeddings.position_ids',)  # non-persistent in transformers: checked, not saved
+
+
+def sha(path):
+    h = hashlib.sha256()
+    with open(path, 'rb') as f:
+        for block in iter(lambda: f.read(1 << 20), b''):
+            h.update(block)
+    return h.hexdigest()
+
+
+def weights(graph):
+    """({torch name: array}, used initializer names)."""
+    init = {i.name: numpy_helper.to_array(i) for i in graph.initializer}
+    producer = {o: n for n in graph.node for o in n.output}
+    state = {k: v for k, v in init.items() if k.startswith(NAMED)}
+    used = set(state)
+    for n in graph.node:
+        bias = [x for x in n.input if x.endswith('.bias') and 'LayerNorm' not in x and x in init]
+        if n.op_type not in ('Add', 'BiasGelu') or not bias:
+            continue
+        (b,), (other,) = bias, [x for x in n.input if x not in bias]
+        matmul = producer.get(other)
+        assert matmul is not None and matmul.op_type == 'MatMul' and matmul.input[1] in init, b
+        w, key = matmul.input[1], b[:-len('bias')] + 'weight'
+        assert w not in used and key not in state, (w, key)
+        state[key] = init[w].T.copy()
+        used.add(w)
+    return state, used
+
+
+def leftovers(init, used):
+    """Unused float tensors with more than one value: a weight the mapping missed. Scalars and int shapes pass."""
+    return sorted(k for k, v in init.items() if k not in used and np.asarray(v).dtype.kind == 'f' and np.size(v) > 1)
+
+
+def main():
+    import onnx, torch
+    from safetensors.torch import save_file
+    from transformers import BertConfig, BertForTokenClassification
+    out = HERE/'model.safetensors'
+    assert not out.exists(), 'create-only'
+    assert sha(SNAPSHOT/'model.onnx') == ONNX_SHA256 and sha(SNAPSHOT/'config.json') == CONFIG_SHA256
+    graph = onnx.load(SNAPSHOT/'model.onnx').graph
+    init = {i.name: numpy_helper.to_array(i) for i in graph.initializer}
+    state, used = weights(graph)
+    left = leftovers(init, used)
+    assert not left, f'{len(left)} unused float tensors'
+    for k in BUFFERS:
+        v = state.pop(k)
+        assert np.array_equal(v, np.arange(v.shape[-1]).reshape(v.shape)), k
+    tensors = {k: torch.from_numpy(np.ascontiguousarray(v)) for k, v in sorted(state.items())}
+    model = BertForTokenClassification(BertConfig.from_pretrained(SNAPSHOT))
+    expected = model.state_dict()
+    assert {k: tuple(v.shape) for k, v in tensors.items()} == {k: tuple(v.shape) for k, v in expected.items()}, 'keys/shapes'
+    assert all(t.dtype == torch.float32 for t in tensors.values())
+    model.load_state_dict(tensors, strict=True)
+    save_file(tensors, out, metadata=dict(source=f'{REPO}@{REVISION}/model.onnx', onnx_sha256=ONNX_SHA256,
+                                          note='converted from ONNX, weights unchanged'))
+    report = dict(repo=REPO, revision=REVISION, onnx_sha256=ONNX_SHA256, config_sha256=CONFIG_SHA256,
+                  initializers=len(init), used=len(used), tensors=len(tensors),
+                  transposed=len(used) - sum(k.startswith(NAMED) for k in used),
+                  unused_constants=len(init) - len(used), safetensors_sha256=sha(out), convert_sha256=sha(__file__))
+    (HERE/'convert.json').write_text(json.dumps(report, indent=1) + '\n')
+    return report
+
+
+if __name__ == '__main__':
+    print(json.dumps(main()))

nergal.py

@@ -17,7 +17,7 @@ import scrub_pii
 from scrub_pii import LEGACY_PHONE_TAG, PHONE_TAG, PII_TAG
 
 HUB_ID = 'SlayerLab/NERGAL'
-VERSION = '1.2.0'
+VERSION = '2.0.0'
 GAPS = ['[PII_SPACE]', '[PII_BREAK]']
 GAP_IDS = [250002, 250003]
 BIO_LABELS = ['O', 'B-phone', 'I-phone', 'B-pii', 'I-pii']

test_nergal.py

@@ -15,7 +15,7 @@ class NergalTests(unittest.TestCase):
         from nergal import GAP_IDS, GAPS, HUB_ID, RULES_SHA as PINNED, THRESHOLD, VERSION
         card = json.loads((HERE / 'hybrid.json').read_text())
         self.assertEqual(HUB_ID, 'SlayerLab/NERGAL')
-        self.assertEqual(VERSION, '1.2.0')
+        self.assertEqual(VERSION, '2.0.0')
         self.assertEqual(card['version'], VERSION)
         self.assertEqual(card['eval']['union_fp'], 80)
         self.assertEqual(card['eval']['rules_fp'], 24)