Add hybrid PII scripts: nergal.py and frozen scrub_pii.py

README.md

@@ -77,12 +77,21 @@ Union with unchanged rules. Character scores are on gold vs masked characters.
 
 ## Load
 
+This repo is the PII island: frozen `scrub_pii.py` plus `nergal.py`. `pipeline("token-classification")` will not match. Regex runs on the original text, the model adds spans at 0.95, then the two are unioned and replaced with `[Telefon]` / `[PII]`.
+
 ```python
-from transformers import AutoModelForTokenClassification, AutoTokenizer
-tok = AutoTokenizer.from_pretrained("SlayerLab/NERGAL")
-model = AutoModelForTokenClassification.from_pretrained("SlayerLab/NERGAL")
+from pathlib import Path
+from huggingface_hub import snapshot_download
+
+root = Path(snapshot_download("SlayerLab/NERGAL"))
+import sys
+sys.path.insert(0, str(root))
+from nergal import Nergal
+
+nergal = Nergal.from_pretrained(root, local_files_only=True)
+masked, counts = nergal.scrub(text)
 ```
 
-Reproducing the table needs the windowed BIO decoder and regex union (`hybrid.json`: threshold 0.95, gap ids `250002`/`250003`). Vanilla token classification will not match.
+`hybrid.json` records threshold 0.95 and gap ids `250002` / `250003`.
 
 Base weights: [`FacebookAI/xlm-roberta-large`](https://huggingface.co/FacebookAI/xlm-roberta-large) revision `c23d21b0620b635a76227c604d44e43a9f0ee389` (MIT).