1.0.3: registry and label-note rules, placeholder and card fixes

CHANGELOG.md

@@ -8,6 +8,25 @@ Semver for this island:
 
 Accuracy is 841-dev, union at 0.95, 354 gold spans. A version that changes those numbers must update `hybrid.json` `eval` and the tables below.
 
+## 1.0.3
+
+Same weights, threshold, and API. Rules SHA `f32d5c54…`.
+
+- **Rules:** `NIP`/`REGON` labels with footnote marks or a short gloss (`NIP*:`, `NIP (Wykonawcy):`); e-Delivery addresses broken across a line; `DUNS`, `BDO` and `RPWDL` numbers right after their own label, at a fixed length.
+- **Wrapper:** existing `[PII]`/`[Telefon]` placeholders no longer switch the rules off. Before this, one placeholder anywhere in the text dropped every rule span for the whole document.
+- **Card:** `weights_sha256` was the hash of the source training checkpoint, not of `model.safetensors`. It is now `source_checkpoint_sha256`, and `model_safetensors_sha256` holds the published file's hash. Readers of `weights_sha256` must switch keys.
+
+841-dev is unchanged: 0 rule spans change, and no 841-dev passage contains a placeholder. On simulated pre-masked input (gold spans replaced by their own placeholder), the rules now cover 1,812 of 2,606 remaining gold spans (1.0.2: 0), with 0 new false characters versus the same rules on unmasked text. Full Dynaword (2.8M documents): 27 documents change, 42 registry spans added, 0 removed.
+
+Known limit: an identifier with a placeholder inside it or right before it (`NIP [PII] …`, `8503[PII]…`) can still be missed by both layers, and a phone that follows an already-masked phone in a list can be missed by the rules.
+
+| Version | Whole /354 | Residual | Rules FP | Union FP | Char P | Char R | What changed |
+|---|---:|---:|---:|---:|---:|---:|---|
+| 1.0.0 | 323 | 25 | 133 | 133 | 97.76% | 95.95% | First Hub snapshot |
+| 1.0.1 | 323 | 25 | 98 | 123 | 97.93% | 95.95% | Prefix-only glued-email trim |
+| 1.0.2 | 324 | 24 | 98 | 123 | 97.93% | 96.12% | Labelled country-area phone fix |
+| 1.0.3 | 324 | 24 | 98 | 123 | 97.93% | 96.12% | Label-note, e-Delivery and registry rules; placeholder and card fixes |
+
 ## 1.0.2
 
 Fix the public wrapper’s tokenizer batch shape so local model inference works. The packed tokenizer regression and an end-to-end local inference check cover this path.

README.md

@@ -13,7 +13,7 @@ tags:
 - hybrid
 ---
 
-# NERGAL 1.0.2
+# NERGAL 1.0.3
 
 **Named Entity Recognition with Grounded Additive Labels**
 
@@ -23,8 +23,8 @@ SlayerLab hybrid PII cleaner for Polish. Not a chat model. Not a drop-in `pipeli
 
 Python rules do the identifiers they can prove. A transformer NER head adds phone and other PII spans the regex misses. The cleaner **unions** the two on the original text, then replaces hits with `[Telefon]` or `[PII]`.
 
-- **Version:** `1.0.2` (`hybrid.json`, `CHANGELOG.md`)
-- **Ground:** `scrub_pii` regex (SHA256 `3016ae5b…`)
+- **Version:** `1.0.3` (`hybrid.json`, `CHANGELOG.md`)
+- **Ground:** `scrub_pii` regex (SHA256 `f32d5c54…`)
 - **Additive labels:** XLM-RoBERTa-large token classifier, BIO tags `phone` / `pii`, threshold 0.95
 - **This snapshot:** seed `202609160`, **epoch 5** of a seven-epoch schedule
 
@@ -36,7 +36,8 @@ Python rules do the identifiers they can prove. A transformer NER head adds phon
 |---|---:|---:|---:|---:|---:|---:|---|
 | 1.0.0 | 323 | 25 | 133 | 133 | 97.76% | 95.95% | First Hub snapshot |
 | 1.0.1 | 323 | 25 | 98 | 123 | 97.93% | 95.95% | Prefix-only glued-email trim |
-| **1.0.2** | **324** | **24** | **98** | **123** | **97.93%** | **96.12%** | Labelled country-area phone fix |
+| 1.0.2 | 324 | 24 | 98 | 123 | 97.93% | 96.12% | Labelled country-area phone fix |
+| **1.0.3** | **324** | **24** | **98** | **123** | **97.93%** | **96.12%** | Label-note, e-Delivery and registry rules; placeholder and card fixes |
 
 ## 841-dev
 
@@ -78,7 +79,7 @@ Same 841-dev split, threshold **0.95**. **Naked** is the transformer alone. **
 | Historical GLiNER email12 | naked | 249 | 70 | 10 | 99.81% | 85.39% |
 | Historical GLiNER email12 | ∪ regex | 290 | 50 | 108 | 98.13% | 93.76% |
 | XLM-R epoch 5 | naked | 298 | 42 | 57 | 98.96% | 89.38% |
-| **NERGAL 1.0.2** | **∪ regex** | 324 | 24 | 123 | 97.93% | 96.12% |
+| **NERGAL 1.0.3** | **∪ regex** | 324 | 24 | 123 | 97.93% | 96.12% |
 
 Zero-shot [GLiNER 2.5](https://huggingface.co/fastino/gliner2.5-multi-v1) is not competitive here, especially on non-phone PII (7/185 whole vs 160 naked / 179 union). Fine-tuned historical GLiNER is the precise naked baseline (10 false characters) and still trails XLM-R on coverage. NERGAL is XLM-R epoch 5 plus the regex: 145/169 phone, 179/185 other PII. Exact-span precision 86.34%, recall 89.27%, F1 87.78%.
 
@@ -98,7 +99,7 @@ Seed 160 is the published weights. 161 and 162 were confirmation runs of the sam
 
 ## Load
 
-This repo is the PII island: `scrub_pii.py` plus `nergal.py`. `pipeline("token-classification")` will not match. Regex runs on the original text, the model adds spans at 0.95, then the two are unioned and replaced with `[Telefon]` / `[PII]`.
+This repo is the PII island: `scrub_pii.py` plus `nergal.py`. `pipeline("token-classification")` will not match. Regex runs on the original text, the model adds spans at 0.95, then the two are unioned and replaced with `[Telefon]` / `[PII]`. Text that already holds `[PII]` / `[Telefon]` is scrubbed as usual, but an identifier with a placeholder inside it or right before it can be missed.
 
 ```python
 from pathlib import Path
@@ -113,6 +114,6 @@ nergal = Nergal.from_pretrained(root, local_files_only=True)
 masked, counts = nergal.scrub(text)
 ```
 
-`hybrid.json` records version `1.0.2`, threshold 0.95, gap ids `250002` / `250003`, and the 841-dev `eval` block. `test_nergal.py` is synthetic (no corpus text). From this snapshot: `python -m unittest test_nergal`.
+`hybrid.json` records version `1.0.3`, threshold 0.95, gap ids `250002` / `250003`, the 841-dev `eval` block, and two weight hashes: `model_safetensors_sha256` for the published file and `source_checkpoint_sha256` for the training checkpoint it was packed from. `test_nergal.py` is synthetic (no corpus text). From this snapshot: `python -m unittest test_nergal`.
 
 Base weights: [`FacebookAI/xlm-roberta-large`](https://huggingface.co/FacebookAI/xlm-roberta-large) revision `c23d21b0620b635a76227c604d44e43a9f0ee389` (MIT).

hybrid.json

@@ -1,12 +1,12 @@
 {
   "full_name": "Named Entity Recognition with Grounded Additive Labels",
   "hub_id": "SlayerLab/NERGAL",
-  "version": "1.0.2",
+  "version": "1.0.3",
   "mode": "rules_union",
   "epoch": 5,
   "seed": 202609160,
   "threshold": 0.95,
-  "rules_sha256": "3016ae5bd403ff997458f9dd74bad8c6ed1388eb83dadc1b31cdb182f9ed607f",
+  "rules_sha256": "f32d5c5452fc47178e109d4bc248a0d8234ea6e59e8cf79407f4eb8451581d67",
   "eval": {
     "split": "841-dev",
     "gold_entities": 354,
@@ -24,7 +24,8 @@
     "pii_whole": 179,
     "pii_gold": 185
   },
-  "weights_sha256": "063ee5f9782c1b4d99e838be5a836328718e1372e213d5b87098b371b5c162af",
+  "source_checkpoint_sha256": "063ee5f9782c1b4d99e838be5a836328718e1372e213d5b87098b371b5c162af",
+  "model_safetensors_sha256": "1d42c34459e90cd25db44f92bb31fb2be1ef3a1b1e34c599162e1895057c08f6",
   "gaps": [
     "[PII_SPACE]",
     "[PII_BREAK]"

nergal.py

@@ -17,13 +17,13 @@ import scrub_pii
 from scrub_pii import PHONE_TAG, PII_TAG
 
 HUB_ID = 'SlayerLab/NERGAL'
-VERSION = '1.0.2'
+VERSION = '1.0.3'
 GAPS = ['[PII_SPACE]', '[PII_BREAK]']
 GAP_IDS = [250002, 250003]
 BIO_LABELS = ['O', 'B-phone', 'I-phone', 'B-pii', 'I-pii']
 LABELS = ['phone', 'pii']
 THRESHOLD = 0.95
-RULES_SHA = '3016ae5bd403ff997458f9dd74bad8c6ed1388eb83dadc1b31cdb182f9ed607f'
+RULES_SHA = 'f32d5c5452fc47178e109d4bc248a0d8234ea6e59e8cf79407f4eb8451581d67'
 
 
 def sha(path):
@@ -165,16 +165,19 @@ class Encoding:
         return units, windows(units, self.count) if units else []
 
 
-def rules(text):
-    verify_rules()
+def spans_from(module, text):
+    """Rule spans on the original text. Existing [PII]/[Telefon] placeholders do not switch the rules off."""
     result = []
-    scrub_pii.scrub_pii(text, spans=result)
-    if '[PII]' in text or '[Telefon]' in text:
-        return []
+    module.scrub_pii(text, spans=result)
     return sorted(({k: s[k] for k in ('start', 'end', 'label')} | {'score': 1.0} for s in result),
                   key=lambda s: s['start'])
 
 
+def rules(text):
+    verify_rules()
+    return spans_from(scrub_pii, text)
+
+
 def apply_union(text, spans):
     labels = [None] * len(text)
     for span in spans:
@@ -294,12 +297,7 @@ class Nergal:
         return decode_bio(units, (sums / counts).tolist())
 
     def rule_spans(self, text):
-        result = []
-        self._scrub.scrub_pii(text, spans=result)
-        if '[PII]' in text or '[Telefon]' in text:
-            return []
-        return sorted(({k: s[k] for k in ('start', 'end', 'label')} | {'score': 1.0} for s in result),
-                      key=lambda s: s['start'])
+        return spans_from(self._scrub, text)
 
     def scrub(self, text):
         if not text:

scrub_pii.py

@@ -37,7 +37,7 @@ import re
 PHONE_TAG = "[Telefon]"
 PII_TAG = "[PII]"
 COUNTS = ("email", "phone", "pesel", "nip", "regon", "account", "document", "krs", "electronic_address", "pin",
-          "land_register")
+          "land_register", "registry")
 
 # Mobile + geographic area codes (2-digit national prefix after trunk 0 / +48).
 _PL_PREFIX = {
@@ -60,7 +60,10 @@ _EMAIL_WRAP_RE = re.compile(_EMAIL_START + r"[ \t]{0,3}(?:" + _EMAIL_DOMAIN
 # Damaged contact fields can retain only the local part and @.
 _EMAIL_FRAGMENT_RE = re.compile(_EMAIL_START + r"(?=[ \t]*\r?$)", re.M)
 _EMAIL_LABEL_RE = re.compile(r"\be[ -]?mail[ \t]*:[ \t]*\Z", re.I)
-_EDELIVERY_RE = re.compile(r"\bAE:PL-\d{5}-\d{5}-[A-Z0-9]{5}-\d{2}(?![\w-])", re.I)
+# A layout wrap may follow any hyphen; a blank line still ends the address.
+_EDELIVERY_WRAP = r"-(?:[ \t]*\r?\n[ \t]*)?"
+_EDELIVERY_RE = re.compile(r"\bAE:PL" + _EDELIVERY_WRAP + r"\d{5}" + _EDELIVERY_WRAP + r"\d{5}"
+                           + _EDELIVERY_WRAP + r"[A-Z0-9]{5}" + _EDELIVERY_WRAP + r"\d{2}(?![\w-])", re.I)
 _EPUAP_RE = re.compile(r"(?<![\w/])/[\w-]+/[\w-]+(?![\w/-])")
 _EPUAP_LABEL_RE = re.compile(
     r"\b(?:e[ -]?puap|elektroniczna[ \t]+skrzynka[ \t]+podawcza)\b[^\n/\[\]]{0,50}\Z", re.I)
@@ -120,13 +123,27 @@ _LAND_REGISTER_LABEL_RE = re.compile(
 _LAND_REGISTER_VALUES = {c: i for i, c in enumerate("0123456789XABCDEFGHIJKLMNOPRSTUWYZ")}
 _REGON9_RE = re.compile(r"\b\d{9}\b")
 _ID_LABEL_END = r"[ \t]{0,8}(?:(?:nr\.?|numer)[ \t]{0,8})?[:=.\-]?[ \t]{0,8}(?:\r?\n[ \t]{0,8})?\Z"
-_NIP_LABEL_RE = re.compile(r"\bNIP\b" + _ID_LABEL_END, re.I)
-_REGON_LABEL_RE = re.compile(r"\bREGON\b" + _ID_LABEL_END, re.I)
+# Form labels may carry footnote asterisks and one short digit-free gloss: "NIP (Wykonawcy)**:".
+# Only checksum-gated NIP/REGON take it; KRS has no checksum.
+_ID_LABEL_NOTE = r"(?:[ \t]{0,8}\*{1,3})?(?:[ \t]{0,8}\([^()\d\n]{1,40}\))?(?:[ \t]{0,8}\*{1,3})?"
+_NIP_LABEL_RE = re.compile(r"\bNIP\b" + _ID_LABEL_NOTE + _ID_LABEL_END, re.I)
+_REGON_LABEL_RE = re.compile(r"\bREGON\b" + _ID_LABEL_NOTE + _ID_LABEL_END, re.I)
 _KRS_RE = re.compile(r"\b\d{6,10}(?!\w|[ \t-]*\d)")
 # Abbreviation or written-out register name, optionally "pod nr/numerem".
 _KRS_LABEL_RE = re.compile(
     r"(?:\bKRS\b|\bKrajow\w*[ \t]+Rejestr\w*[ \t]+S[aą]dow\w*)"
     r"(?:[ \t]{0,8},?[ \t]{0,8}pod[ \t]{1,8}(?:nr\.?|numerem))?" + _ID_LABEL_END, re.I)
+# Registry numbers without a checksum (user, 2026-09-25): masked only right after their own label, one fixed length
+# each. DUNS 9 digits (also 2-3-4 dashed), BDO 9 digits, RPWDL book (księga rejestrowa) 12 digits.
+_REGISTRY_BY = r"(?:[ \t]{0,8},?[ \t]{0,8}pod[ \t]{1,8}(?:nr\.?|numerem))?(?:[ \t]{0,8}[\u2013\u2014])?"
+_REGISTRY = (
+    (re.compile(r"\b(?:\d{9}|\d{2}-\d{3}-\d{4})(?!\w|[ \t-]*\d)"),
+     re.compile(r"\bD-?U-?N-?S\b(?:[ \t]?®)?(?:[ \t]{1,8}number)?" + _REGISTRY_BY + _ID_LABEL_END, re.I)),
+    (re.compile(r"\b\d{9}(?!\w|[ \t-]*\d)"),
+     re.compile(r"\bBDO\b" + _REGISTRY_BY + _ID_LABEL_END, re.I)),
+    (re.compile(r"\b\d{12}(?!\w|[ \t-]*\d)"),
+     re.compile(r"(?:\bRPWDL\b|\bksi[eę]g\w*[ \t]+rejestrow\w*)" + _REGISTRY_BY + _ID_LABEL_END, re.I)),
+)
 # Separators are short and local: one newline *or* a few punct/spaces.
 # Letters and blank lines break the match so body text is not swallowed.
 _SEP = r"(?:[ \t./\-()\u2013\u2014]{0,3}|\n)"
@@ -771,6 +788,8 @@ def scrub_pii(text: str, *, spans: list | None = None) -> tuple[str, dict[str, i
     counts['electronic_address'] = apply(_replace_checked, _EDELIVERY_RE, PII_TAG, lambda _: True)
     counts['electronic_address'] += apply(_replace_epuap)
     counts['krs'] = apply(_replace_checked, _KRS_RE, PII_TAG, lambda _: True, _KRS_LABEL_RE)
+    counts['registry'] = sum(apply(_replace_checked, number, PII_TAG, lambda _: True, label)
+                             for number, label in _REGISTRY)
     counts["document"] = apply(_replace_documents)
     counts["land_register"] = apply(_replace_land_registers)
     def labelled_ids(value, *, mask):

test_nergal.py

@@ -5,7 +5,7 @@ import unittest
 from pathlib import Path
 
 HERE = Path(__file__).resolve().parent
-RULES_SHA = '3016ae5bd403ff997458f9dd74bad8c6ed1388eb83dadc1b31cdb182f9ed607f'
+RULES_SHA = 'f32d5c5452fc47178e109d4bc248a0d8234ea6e59e8cf79407f4eb8451581d67'
 
 
 class NergalTests(unittest.TestCase):
@@ -13,7 +13,7 @@ class NergalTests(unittest.TestCase):
         from nergal import GAP_IDS, GAPS, HUB_ID, RULES_SHA as PINNED, THRESHOLD, VERSION
         card = json.loads((HERE / 'hybrid.json').read_text())
         self.assertEqual(HUB_ID, 'SlayerLab/NERGAL')
-        self.assertEqual(VERSION, '1.0.2')
+        self.assertEqual(VERSION, '1.0.3')
         self.assertEqual(card['version'], VERSION)
         self.assertEqual(card['eval']['union_fp'], 123)
         self.assertEqual(card['eval']['rules_fp'], 98)
@@ -35,6 +35,13 @@ class NergalTests(unittest.TestCase):
         self.assertEqual(len(first), len(words))
         self.assertEqual([encoded.word_ids(0)[i] for i in first], [0, 1, 2])
 
+    def test_existing_placeholders_do_not_switch_the_rules_off(self):
+        from nergal import rules
+        text = 'Kontakt [Telefon], NIP 1234567802.'  # invented, checksum-valid
+        [span] = rules(text)
+        self.assertEqual(text[span['start']:span['end']], '1234567802')
+        self.assertEqual(rules('a [PII] b [Telefon] c'), [])
+
     def test_union_keeps_regex_and_adds_model_spans(self):
         from nergal import apply_union, scrub_spans
         text = 'Ring 000000000 then extra.'