Instructions to use SlayerLab/NERGAL with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use SlayerLab/NERGAL with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("token-classification", model="SlayerLab/NERGAL")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForTokenClassification tokenizer = AutoTokenizer.from_pretrained("SlayerLab/NERGAL") model = AutoModelForTokenClassification.from_pretrained("SlayerLab/NERGAL", device_map="auto") - Notebooks
- Google Colab
- Kaggle
1.1.1: grouped national phones without a label
Browse filesRules only; same weights, threshold and API. 841-dev 326/354 whole (324), 0 new false
characters; cue-less phone test 66/111 phones (37), scored once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- CHANGELOG.md +17 -0
- README.md +25 -10
- hybrid.json +10 -10
- nergal.py +2 -2
- scrub_pii.py +47 -13
- test_nergal.py +15 -2
CHANGELOG.md
CHANGED
|
@@ -8,6 +8,23 @@ Semver for this island:
|
|
| 8 |
|
| 9 |
Accuracy is 841-dev, union at 0.95, 354 gold spans. A version that changes those numbers must update `hybrid.json` `eval` and the tables below.
|
| 10 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
## 1.1.0
|
| 12 |
|
| 13 |
Same weights, rules, threshold and default outputs; `hybrid.json` `eval` is unchanged. New batch API and opt-in fp16.
|
|
|
|
| 8 |
|
| 9 |
Accuracy is 841-dev, union at 0.95, 354 gold spans. A version that changes those numbers must update `hybrid.json` `eval` and the tables below.
|
| 10 |
|
| 11 |
+
## 1.1.1
|
| 12 |
+
|
| 13 |
+
Same weights, threshold and API. Rules SHA `ad5c51f7…`. The rules file is the frozen and scored candidate `7412261c…` with comment tags removed; its syntax tree is identical.
|
| 14 |
+
|
| 15 |
+
- **Rules:** Polish phones in the grouped national forms are masked without a contact label: mobile `601 234 567` and landline `22 123 45 67` (area code optionally in parentheses), optional `+48`, one separator kind (space or hyphen) throughout. Not taken: plain 9-digit strings (still label-gated), amounts, round counts (`500 000 000`), decimal figures (`601 234 567,89`), and numbers after another identifier's label (NIP, REGON, codes). After a bare grouped phone, a list goes on only with complete Polish numbers.
|
| 16 |
+
|
| 17 |
+
841-dev: 326/354 whole (1.1.0: 324), residual 23 (24), 0 new false characters; phone 147/169 (145). A frozen blind set of 150 web passages selected for unlabelled phones: phones wholly masked 66/111 (1.1.0: 37), all values 127/178 (98), 0 lost, 1 passage with new false characters. Dynaword (60,000 documents): 79 documents change, 247 phone spans added, 0 removed; in a reviewed sample 1 of 23 sampled documents' added spans was not a phone.
|
| 18 |
+
|
| 19 |
+
| Version | Whole /354 | Residual | Rules FP | Union FP | Char P | Char R | What changed |
|
| 20 |
+
|---|---:|---:|---:|---:|---:|---:|---|
|
| 21 |
+
| 1.0.0 | 323 | 25 | 133 | 133 | 97.76% | 95.95% | First Hub snapshot |
|
| 22 |
+
| 1.0.1 | 323 | 25 | 98 | 123 | 97.93% | 95.95% | Prefix-only glued-email trim |
|
| 23 |
+
| 1.0.2 | 324 | 24 | 98 | 123 | 97.93% | 96.12% | Labelled country-area phone fix |
|
| 24 |
+
| 1.0.3 | 324 | 24 | 98 | 123 | 97.93% | 96.12% | Label-note, e-Delivery and registry rules; placeholder and card fixes |
|
| 25 |
+
| 1.1.0 | 324 | 24 | 98 | 123 | 97.93% | 96.12% | Batch API (`predict_many`, `scrub_many`), opt-in float16 |
|
| 26 |
+
| 1.1.1 | 326 | 23 | 98 | 123 | 97.94% | 96.50% | Grouped national phones without a label |
|
| 27 |
+
|
| 28 |
## 1.1.0
|
| 29 |
|
| 30 |
Same weights, rules, threshold and default outputs; `hybrid.json` `eval` is unchanged. New batch API and opt-in fp16.
|
README.md
CHANGED
|
@@ -13,7 +13,7 @@ tags:
|
|
| 13 |
- hybrid
|
| 14 |
---
|
| 15 |
|
| 16 |
-
# NERGAL 1.1.
|
| 17 |
|
| 18 |
**Named Entity Recognition with Grounded Additive Labels**
|
| 19 |
|
|
@@ -23,8 +23,8 @@ SlayerLab hybrid PII cleaner for Polish. Not a chat model. Not a drop-in `pipeli
|
|
| 23 |
|
| 24 |
Python rules do the identifiers they can prove. A transformer NER head adds phone and other PII spans the regex misses. The cleaner **unions** the two on the original text, then replaces hits with `[Telefon]` or `[PII]`.
|
| 25 |
|
| 26 |
-
- **Version:** `1.1.
|
| 27 |
-
- **Ground:** `scrub_pii` regex (SHA256 `
|
| 28 |
- **Additive labels:** XLM-RoBERTa-large token classifier, BIO tags `phone` / `pii`, threshold 0.95
|
| 29 |
- **Throughput:** about 80k chars/s on one RTX 4090 with `scrub_many` + `dtype="float16"` and 3 processes (1.0.3: 23k)
|
| 30 |
- **This snapshot:** seed `202609160`, **epoch 5** of a seven-epoch schedule
|
|
@@ -62,9 +62,9 @@ NERGAL is not designed to remove:
|
|
| 62 |
|
| 63 |
These are intended exclusions; false positives can still mask some of this content.
|
| 64 |
|
| 65 |
-
### Known gaps in 1.1.
|
| 66 |
|
| 67 |
-
Unlabelled phones and identifiers, unusual formatting and damaged text can escape detection. **VINs and obfuscated emails** (such as `name (at) domain.pl`) are approved annotation targets, but that approval alone does not establish reliable support in the released 1.1.
|
| 68 |
|
| 69 |
## Versions
|
| 70 |
|
|
@@ -76,7 +76,22 @@ Unlabelled phones and identifiers, unusual formatting and damaged text can escap
|
|
| 76 |
| 1.0.1 | 323 | 25 | 98 | 123 | 97.93% | 95.95% | Prefix-only glued-email trim |
|
| 77 |
| 1.0.2 | 324 | 24 | 98 | 123 | 97.93% | 96.12% | Labelled country-area phone fix |
|
| 78 |
| 1.0.3 | 324 | 24 | 98 | 123 | 97.93% | 96.12% | Label-note, e-Delivery and registry rules; placeholder and card fixes |
|
| 79 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 80 |
|
| 81 |
## 841-dev
|
| 82 |
|
|
@@ -112,15 +127,15 @@ Same 841-dev split, threshold **0.95**. **Naked** is the transformer alone. **
|
|
| 112 |
|
| 113 |
| System | Mode | Whole /354 | Residual | False chars | Char P | Char R |
|
| 114 |
|---|---|---:|---:|---:|---:|---:|
|
| 115 |
-
| Regex (`scrub_pii`) | rules |
|
| 116 |
| GLiNER 2.5-multi zero-shot | naked | 81 | 188 | 970 | 56.98% | 21.22% |
|
| 117 |
| GLiNER 2.5-multi zero-shot | ∪ regex | 273 | 61 | 1,068 | 83.33% | 88.16% |
|
| 118 |
| Historical GLiNER email12 | naked | 249 | 70 | 10 | 99.81% | 85.39% |
|
| 119 |
| Historical GLiNER email12 | ∪ regex | 290 | 50 | 108 | 98.13% | 93.76% |
|
| 120 |
| XLM-R epoch 5 | naked | 298 | 42 | 57 | 98.96% | 89.38% |
|
| 121 |
-
| **NERGAL 1.1.
|
| 122 |
|
| 123 |
-
Zero-shot [GLiNER 2.5](https://huggingface.co/fastino/gliner2.5-multi-v1) is not competitive here, especially on non-phone PII (7/185 whole vs 160 naked / 179 union). Fine-tuned historical GLiNER is the precise naked baseline (10 false characters) and still trails XLM-R on coverage. NERGAL is XLM-R epoch 5 plus the regex:
|
| 124 |
|
| 125 |
Trained GLiNER, HerBERT-large, and XLM-R-large were also compared on this split (plot above). GLiNER’s 0.50 coverage lead is the precision collapse in that figure; no GLiNER or HerBERT operating point passed the content-preservation gate.
|
| 126 |
|
|
@@ -155,6 +170,6 @@ masked, counts = nergal.scrub(text)
|
|
| 155 |
|
| 156 |
For many texts, `nergal.scrub_many(texts)` (or `predict_many` for the raw model spans) batches windows across texts: about 2× the throughput of calling `scrub` in a loop on a CUDA GPU. `from_pretrained(..., dtype="float16")` casts the weights at load time on CUDA or MPS: another 1.9× on an RTX 4090, and 841-dev union numbers are unchanged, but scores are not bit-identical to float32 (0.95 spans can differ in rare cases). For a corpus, run 2–3 processes per GPU, because float16 inference is limited by CPU-side tokenization. Measurements are in `CHANGELOG.md`.
|
| 157 |
|
| 158 |
-
`hybrid.json` records version `1.1.
|
| 159 |
|
| 160 |
Base weights: [`FacebookAI/xlm-roberta-large`](https://huggingface.co/FacebookAI/xlm-roberta-large) revision `c23d21b0620b635a76227c604d44e43a9f0ee389` (MIT).
|
|
|
|
| 13 |
- hybrid
|
| 14 |
---
|
| 15 |
|
| 16 |
+
# NERGAL 1.1.1
|
| 17 |
|
| 18 |
**Named Entity Recognition with Grounded Additive Labels**
|
| 19 |
|
|
|
|
| 23 |
|
| 24 |
Python rules do the identifiers they can prove. A transformer NER head adds phone and other PII spans the regex misses. The cleaner **unions** the two on the original text, then replaces hits with `[Telefon]` or `[PII]`.
|
| 25 |
|
| 26 |
+
- **Version:** `1.1.1` (`hybrid.json`, `CHANGELOG.md`)
|
| 27 |
+
- **Ground:** `scrub_pii` regex (SHA256 `ad5c51f7…`)
|
| 28 |
- **Additive labels:** XLM-RoBERTa-large token classifier, BIO tags `phone` / `pii`, threshold 0.95
|
| 29 |
- **Throughput:** about 80k chars/s on one RTX 4090 with `scrub_many` + `dtype="float16"` and 3 processes (1.0.3: 23k)
|
| 30 |
- **This snapshot:** seed `202609160`, **epoch 5** of a seven-epoch schedule
|
|
|
|
| 62 |
|
| 63 |
These are intended exclusions; false positives can still mask some of this content.
|
| 64 |
|
| 65 |
+
### Known gaps in 1.1.1
|
| 66 |
|
| 67 |
+
Unlabelled phones and identifiers (the rules take a bare phone only in the grouped national forms `601 234 567` and `22 123 45 67`; a plain `601234567` needs a label or the model), unusual formatting and damaged text can escape detection. **VINs and obfuscated emails** (such as `name (at) domain.pl`) are approved annotation targets, but that approval alone does not establish reliable support in the released 1.1.1 model. Do not rely on it to remove them consistently.
|
| 68 |
|
| 69 |
## Versions
|
| 70 |
|
|
|
|
| 76 |
| 1.0.1 | 323 | 25 | 98 | 123 | 97.93% | 95.95% | Prefix-only glued-email trim |
|
| 77 |
| 1.0.2 | 324 | 24 | 98 | 123 | 97.93% | 96.12% | Labelled country-area phone fix |
|
| 78 |
| 1.0.3 | 324 | 24 | 98 | 123 | 97.93% | 96.12% | Label-note, e-Delivery and registry rules; placeholder and card fixes |
|
| 79 |
+
| 1.1.0 | 324 | 24 | 98 | 123 | 97.93% | 96.12% | Batch API (`predict_many`, `scrub_many`), opt-in float16 |
|
| 80 |
+
| **1.1.1** | **326** | **23** | **98** | **123** | **97.94%** | **96.50%** | Grouped national phones without a label |
|
| 81 |
+
|
| 82 |
+
## Cue-less phone test
|
| 83 |
+
|
| 84 |
+
841-dev holds few phones without a label, so it barely moves for 1.1.1. This targeted test does: 150 web passages selected for phones with no contact label, labelled and frozen before any prediction, and scored once (2026-09-29). Union at 0.95, same model spans for both.
|
| 85 |
+
|
| 86 |
+
| | 1.1.0 | **1.1.1** |
|
| 87 |
+
|---|---:|---:|
|
| 88 |
+
| Phones wholly masked /111 | 37 | **66** |
|
| 89 |
+
| All values wholly masked /178 | 98 | **127** |
|
| 90 |
+
| Positive passages fully covered /70 (95% CI) | 24 (0.23–0.47) | **37 (0.41–0.65)** |
|
| 91 |
+
| Passages with false masks /150 | 3 | 4 |
|
| 92 |
+
| False characters | 83 | 95 |
|
| 93 |
+
|
| 94 |
+
1.1.1 lost no value 1.1.0 masked; its 12 new false characters are in one passage. The set is enriched by selection, so these numbers say nothing about how common such phones are, and it has a single reviewer.
|
| 95 |
|
| 96 |
## 841-dev
|
| 97 |
|
|
|
|
| 127 |
|
| 128 |
| System | Mode | Whole /354 | Residual | False chars | Char P | Char R |
|
| 129 |
|---|---|---:|---:|---:|---:|---:|
|
| 130 |
+
| Regex (`scrub_pii`) | rules | 263 | 68 | 98 | 98.15% | 85.78% |
|
| 131 |
| GLiNER 2.5-multi zero-shot | naked | 81 | 188 | 970 | 56.98% | 21.22% |
|
| 132 |
| GLiNER 2.5-multi zero-shot | ∪ regex | 273 | 61 | 1,068 | 83.33% | 88.16% |
|
| 133 |
| Historical GLiNER email12 | naked | 249 | 70 | 10 | 99.81% | 85.39% |
|
| 134 |
| Historical GLiNER email12 | ∪ regex | 290 | 50 | 108 | 98.13% | 93.76% |
|
| 135 |
| XLM-R epoch 5 | naked | 298 | 42 | 57 | 98.96% | 89.38% |
|
| 136 |
+
| **NERGAL 1.1.1** | **∪ regex** | 326 | 23 | 123 | 97.94% | 96.50% |
|
| 137 |
|
| 138 |
+
Zero-shot [GLiNER 2.5](https://huggingface.co/fastino/gliner2.5-multi-v1) is not competitive here, especially on non-phone PII (7/185 whole vs 160 naked / 179 union). Fine-tuned historical GLiNER is the precise naked baseline (10 false characters) and still trails XLM-R on coverage. NERGAL is XLM-R epoch 5 plus the regex: 147/169 phone, 179/185 other PII. Exact-span precision 86.41%, recall 89.83%, F1 88.09%. The two GLiNER ∪ regex rows use the 1.1.0 rules.
|
| 139 |
|
| 140 |
Trained GLiNER, HerBERT-large, and XLM-R-large were also compared on this split (plot above). GLiNER’s 0.50 coverage lead is the precision collapse in that figure; no GLiNER or HerBERT operating point passed the content-preservation gate.
|
| 141 |
|
|
|
|
| 170 |
|
| 171 |
For many texts, `nergal.scrub_many(texts)` (or `predict_many` for the raw model spans) batches windows across texts: about 2× the throughput of calling `scrub` in a loop on a CUDA GPU. `from_pretrained(..., dtype="float16")` casts the weights at load time on CUDA or MPS: another 1.9× on an RTX 4090, and 841-dev union numbers are unchanged, but scores are not bit-identical to float32 (0.95 spans can differ in rare cases). For a corpus, run 2–3 processes per GPU, because float16 inference is limited by CPU-side tokenization. Measurements are in `CHANGELOG.md`.
|
| 172 |
|
| 173 |
+
`hybrid.json` records version `1.1.1`, threshold 0.95, gap ids `250002` / `250003`, the 841-dev `eval` block, and two weight hashes: `model_safetensors_sha256` for the published file and `source_checkpoint_sha256` for the training checkpoint it was packed from. `test_nergal.py` is synthetic (no corpus text). From this snapshot: `python -m unittest test_nergal`.
|
| 174 |
|
| 175 |
Base weights: [`FacebookAI/xlm-roberta-large`](https://huggingface.co/FacebookAI/xlm-roberta-large) revision `c23d21b0620b635a76227c604d44e43a9f0ee389` (MIT).
|
hybrid.json
CHANGED
|
@@ -1,25 +1,25 @@
|
|
| 1 |
{
|
| 2 |
"full_name": "Named Entity Recognition with Grounded Additive Labels",
|
| 3 |
"hub_id": "SlayerLab/NERGAL",
|
| 4 |
-
"version": "1.1.
|
| 5 |
"mode": "rules_union",
|
| 6 |
"epoch": 5,
|
| 7 |
"seed": 202609160,
|
| 8 |
"threshold": 0.95,
|
| 9 |
-
"rules_sha256": "
|
| 10 |
"eval": {
|
| 11 |
"split": "841-dev",
|
| 12 |
"gold_entities": 354,
|
| 13 |
-
"whole_entities":
|
| 14 |
-
"residual_passages":
|
| 15 |
"union_fp": 123,
|
| 16 |
"rules_fp": 98,
|
| 17 |
-
"character_precision": 0.
|
| 18 |
-
"character_recall": 0.
|
| 19 |
-
"exact_precision": 0.
|
| 20 |
-
"exact_recall": 0.
|
| 21 |
-
"exact_f1": 0.
|
| 22 |
-
"phone_whole":
|
| 23 |
"phone_gold": 169,
|
| 24 |
"pii_whole": 179,
|
| 25 |
"pii_gold": 185
|
|
|
|
| 1 |
{
|
| 2 |
"full_name": "Named Entity Recognition with Grounded Additive Labels",
|
| 3 |
"hub_id": "SlayerLab/NERGAL",
|
| 4 |
+
"version": "1.1.1",
|
| 5 |
"mode": "rules_union",
|
| 6 |
"epoch": 5,
|
| 7 |
"seed": 202609160,
|
| 8 |
"threshold": 0.95,
|
| 9 |
+
"rules_sha256": "ad5c51f7c712dd2b7af40a337b703aeec211a57170a87c262bf9bd9386ba1925",
|
| 10 |
"eval": {
|
| 11 |
"split": "841-dev",
|
| 12 |
"gold_entities": 354,
|
| 13 |
+
"whole_entities": 326,
|
| 14 |
+
"residual_passages": 23,
|
| 15 |
"union_fp": 123,
|
| 16 |
"rules_fp": 98,
|
| 17 |
+
"character_precision": 0.9794,
|
| 18 |
+
"character_recall": 0.965,
|
| 19 |
+
"exact_precision": 0.8641,
|
| 20 |
+
"exact_recall": 0.8983,
|
| 21 |
+
"exact_f1": 0.8809,
|
| 22 |
+
"phone_whole": 147,
|
| 23 |
"phone_gold": 169,
|
| 24 |
"pii_whole": 179,
|
| 25 |
"pii_gold": 185
|
nergal.py
CHANGED
|
@@ -17,13 +17,13 @@ import scrub_pii
|
|
| 17 |
from scrub_pii import PHONE_TAG, PII_TAG
|
| 18 |
|
| 19 |
HUB_ID = 'SlayerLab/NERGAL'
|
| 20 |
-
VERSION = '1.1.
|
| 21 |
GAPS = ['[PII_SPACE]', '[PII_BREAK]']
|
| 22 |
GAP_IDS = [250002, 250003]
|
| 23 |
BIO_LABELS = ['O', 'B-phone', 'I-phone', 'B-pii', 'I-pii']
|
| 24 |
LABELS = ['phone', 'pii']
|
| 25 |
THRESHOLD = 0.95
|
| 26 |
-
RULES_SHA = '
|
| 27 |
DTYPES = ('float32', 'float16')
|
| 28 |
|
| 29 |
|
|
|
|
| 17 |
from scrub_pii import PHONE_TAG, PII_TAG
|
| 18 |
|
| 19 |
HUB_ID = 'SlayerLab/NERGAL'
|
| 20 |
+
VERSION = '1.1.1'
|
| 21 |
GAPS = ['[PII_SPACE]', '[PII_BREAK]']
|
| 22 |
GAP_IDS = [250002, 250003]
|
| 23 |
BIO_LABELS = ['O', 'B-phone', 'I-phone', 'B-pii', 'I-pii']
|
| 24 |
LABELS = ['phone', 'pii']
|
| 25 |
THRESHOLD = 0.95
|
| 26 |
+
RULES_SHA = 'ad5c51f7c712dd2b7af40a337b703aeec211a57170a87c262bf9bd9386ba1925'
|
| 27 |
DTYPES = ('float32', 'float16')
|
| 28 |
|
| 29 |
|
scrub_pii.py
CHANGED
|
@@ -24,7 +24,8 @@ international country/trunk notation such as +CC (0). With a cue, the EUR-Lex
|
|
| 24 |
"(32-2) 299 11 11" country-area form counts as international. Strong labels also
|
| 25 |
admit one-digit country codes and wider hyphenated area codes. Short service numbers
|
| 26 |
need strong labels; 116xxx numbers also accept nearby telephone prose. Unlabelled
|
| 27 |
-
domestic numbers are left for audit because table cells have the same shapes
|
|
|
|
| 28 |
Flattened tables can glue labels to both neighbours ("Mödlingtel.: … 38112faks:");
|
| 29 |
such glued labels count as labels and end the preceding number.
|
| 30 |
|
|
@@ -75,7 +76,7 @@ _PIN_RE = re.compile(
|
|
| 75 |
_SPACES = str.maketrans({c: " " for c in "\u00a0\u1680\u2000\u2001\u2002\u2003\u2004\u2005\u2006\u2007\u2008\u2009\u200a\u202f\u205f\u3000"})
|
| 76 |
# Country lengths checked against Apache Commons Validator's IBAN registry table:
|
| 77 |
# https://commons.apache.org/proper/commons-validator/xref/org/apache/commons/validator/routines/IBANValidator.html
|
| 78 |
-
#
|
| 79 |
_IBAN_LENGTHS = {
|
| 80 |
country: length for length, countries in {
|
| 81 |
15: "NO", 16: "BE", 18: "DK FI AX FK FO GL NL SD",
|
|
@@ -184,6 +185,14 @@ _SERVICE_PHONE_LABEL_RE = re.compile(
|
|
| 184 |
r"(?:[ \t]+\([^()\n]{1,60}\))?[ \t]*:[ \t]*)"
|
| 185 |
r"(?:[1-9]\d{2}[ \t]*(?:[,;]|i|lub|oraz)[ \t]*)*\Z", re.I)
|
| 186 |
_OTHER_NUMBER_LABEL_RE = re.compile(r"\b(?:NIP|REGON|PESEL|KRS|ISBN|kod)\b[^\d\n]{0,20}\Z", re.I)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 187 |
_PHONE_LIST_HEADING_RE = re.compile(
|
| 188 |
r"[ \t]*(?:numery[ \t]+kontaktowe|telefony(?:[ \t]+(?:kontaktowe|zaufania))?|fax|faks|zapisz[ \t]+się|"
|
| 189 |
r"(?:[^.!?\n]{0,40}\?[ \t]*)?(?:za)?dzwoń!?)"
|
|
@@ -204,7 +213,7 @@ _STAFF_ROLE_RE = re.compile(
|
|
| 204 |
|
| 205 |
|
| 206 |
def _phone_list_context(text: str, start: int) -> bool:
|
| 207 |
-
#
|
| 208 |
# lists need repeated headings or a section parser. Never cross prose.
|
| 209 |
end = text.find('\n', start, start + 401)
|
| 210 |
if end < 0 and len(text) > start + 400:
|
|
@@ -285,7 +294,7 @@ def _directory_context(text: str, start: int, end: int) -> bool:
|
|
| 285 |
foreign = raw.startswith('+') and _phone_ok(raw)
|
| 286 |
if not foreign and (not _pl_national_ok(_digits(raw)) or not re.search(r'[ \t-]', raw)):
|
| 287 |
return False
|
| 288 |
-
#
|
| 289 |
# need preserved upstream table structure, not an unbounded document cue.
|
| 290 |
left, right = max(0, start-800), min(len(text), end+800)
|
| 291 |
nearby = text[left:right]
|
|
@@ -469,6 +478,25 @@ def _replace_land_registers(text: str, *, mask=_tag) -> tuple[str, int]:
|
|
| 469 |
return _LAND_REGISTER_RE.sub(replace, text), n
|
| 470 |
|
| 471 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 472 |
def _pl_national_ok(d: str) -> bool:
|
| 473 |
return len(d) == 9 and d.isdigit() and d[:2] in _PL_PREFIX
|
| 474 |
|
|
@@ -499,7 +527,7 @@ def _phone_ok(raw: str, short: bool = False) -> bool:
|
|
| 499 |
return True
|
| 500 |
if short and s.startswith('(0)'):
|
| 501 |
return 7 <= len(d) <= 12
|
| 502 |
-
#
|
| 503 |
if short and re.match(r'^\(0?\d{2,4}(?:-\d{1,2})?\)', s):
|
| 504 |
return 8 <= len(d) <= 15
|
| 505 |
# EUR-Lex puts country and area code in parentheses: "(32-2) 299 11 11" is +32 2 299 11 11.
|
|
@@ -509,7 +537,7 @@ def _phone_ok(raw: str, short: bool = False) -> bool:
|
|
| 509 |
if s.startswith('00'):
|
| 510 |
international = international[2:]
|
| 511 |
if not international.startswith('48'):
|
| 512 |
-
#
|
| 513 |
# validator. Add country metadata if review finds false positives.
|
| 514 |
return (7 if '(0)' in s else 8) <= len(international) <= 15 and international[0] != '0'
|
| 515 |
if d.startswith("00"):
|
|
@@ -598,6 +626,7 @@ def _replace_phones(text: str, *, mask=_tag) -> tuple[str, int]:
|
|
| 598 |
pos = 0
|
| 599 |
last_phone_end = None
|
| 600 |
last_phone_complete = False
|
|
|
|
| 601 |
while True:
|
| 602 |
m = _PHONE_RE.search(text, pos)
|
| 603 |
if not m:
|
|
@@ -609,8 +638,10 @@ def _replace_phones(text: str, *, mask=_tag) -> tuple[str, int]:
|
|
| 609 |
line_end = text.find('\n', m.start(), end)
|
| 610 |
if line_end >= 0 and re.fullmatch(r'\d{1,3}', text[m.start():line_end]):
|
| 611 |
next_end = text.find('\n', line_end+1, end)
|
| 612 |
-
|
| 613 |
-
|
|
|
|
|
|
|
| 614 |
out.append(text[pos:line_end+1])
|
| 615 |
pos = line_end+1
|
| 616 |
continue
|
|
@@ -638,17 +669,19 @@ def _replace_phones(text: str, *, mask=_tag) -> tuple[str, int]:
|
|
| 638 |
# "tel. +48 609 \n953 709", or the next local number, "(30) 25 10 22 33 25,\n22 33 28".
|
| 639 |
cell = (last_phone_complete and 5 <= len(_digits(raw)) <= 6
|
| 640 |
and re.fullmatch(r'[ \t]*\n[ \t\n]*', text[last_phone_end:m.start()]))
|
| 641 |
-
|
|
|
|
| 642 |
and re.fullmatch(r"[ \t\n,;]{1,12}|[ \t]+(?:lub|albo)[ \t]+", text[last_phone_end:m.start()], re.I)):
|
| 643 |
short = True
|
| 644 |
-
if (
|
| 645 |
and re.fullmatch(r'(?:[ \t]+(?:w sprawie\b|\((?:w godzinach|dostępny)\b)'
|
| 646 |
r'|[ \t]*[–—-][ \t]+)[^\d\n.!?]{1,200}\n[ \t\n]{0,8}',
|
| 647 |
text[last_phone_end:m.start()], re.I)
|
| 648 |
and _pl_national_ok(_digits(raw))):
|
| 649 |
short = True
|
| 650 |
-
|
| 651 |
-
or _directory_context(text, m.start(), line_end if line_end >= 0 else end))
|
|
|
|
| 652 |
if (_is_amount(text, m.start(), end)
|
| 653 |
and re.fullmatch(r'\d{1,3}(?:\.[ \t]*\d{3})+', raw)):
|
| 654 |
out.extend((text[pos:m.start()], raw))
|
|
@@ -701,7 +734,7 @@ def _replace_phones(text: str, *, mask=_tag) -> tuple[str, int]:
|
|
| 701 |
start += 1
|
| 702 |
replacement = text[m.start():start] + mask(start, end, PHONE_TAG)
|
| 703 |
n += 1
|
| 704 |
-
#
|
| 705 |
# context to distinguish them from accounts. Never redact ID tails.
|
| 706 |
elif 18 <= len(_digits(raw)) <= 22 and "." not in raw:
|
| 707 |
# A slash separates alternatives, the second without the shared
|
|
@@ -717,6 +750,7 @@ def _replace_phones(text: str, *, mask=_tag) -> tuple[str, int]:
|
|
| 717 |
if replacement != raw:
|
| 718 |
last_phone_end = end
|
| 719 |
last_phone_complete = _phone_ok(raw)
|
|
|
|
| 720 |
pos = end
|
| 721 |
return "".join(out), n
|
| 722 |
|
|
|
|
| 24 |
"(32-2) 299 11 11" country-area form counts as international. Strong labels also
|
| 25 |
admit one-digit country codes and wider hyphenated area codes. Short service numbers
|
| 26 |
need strong labels; 116xxx numbers also accept nearby telephone prose. Unlabelled
|
| 27 |
+
domestic numbers are left for audit because table cells have the same shapes, except
|
| 28 |
+
the grouped national forms: mobile 3-3-3 and landline 2-3-2-2, one separator kind.
|
| 29 |
Flattened tables can glue labels to both neighbours ("Mödlingtel.: … 38112faks:");
|
| 30 |
such glued labels count as labels and end the preceding number.
|
| 31 |
|
|
|
|
| 76 |
_SPACES = str.maketrans({c: " " for c in "\u00a0\u1680\u2000\u2001\u2002\u2003\u2004\u2005\u2006\u2007\u2008\u2009\u200a\u202f\u205f\u3000"})
|
| 77 |
# Country lengths checked against Apache Commons Validator's IBAN registry table:
|
| 78 |
# https://commons.apache.org/proper/commons-validator/xref/org/apache/commons/validator/routines/IBANValidator.html
|
| 79 |
+
# Country/length/MOD-97 only; add national BBAN rules if false positives appear.
|
| 80 |
_IBAN_LENGTHS = {
|
| 81 |
country: length for length, countries in {
|
| 82 |
15: "NO", 16: "BE", 18: "DK FI AX FK FO GL NL SD",
|
|
|
|
| 185 |
r"(?:[ \t]+\([^()\n]{1,60}\))?[ \t]*:[ \t]*)"
|
| 186 |
r"(?:[1-9]\d{2}[ \t]*(?:[,;]|i|lub|oraz)[ \t]*)*\Z", re.I)
|
| 187 |
_OTHER_NUMBER_LABEL_RE = re.compile(r"\b(?:NIP|REGON|PESEL|KRS|ISBN|kod)\b[^\d\n]{0,20}\Z", re.I)
|
| 188 |
+
# Grouped national phones need no cue: web contact blocks write them bare. Plain 9-digit strings stay cue-gated,
|
| 189 |
+
# since many unlabelled ones are not phones.
|
| 190 |
+
_MOBILE_PREFIX = "45|50|51|53|57|60|66|69|72|73|78|79|88"
|
| 191 |
+
_AREA_CODE = ("12|13|14|15|16|17|18|22|23|24|25|29|32|33|34|41|42|43|44|46|48|52|54|55|56|58|59|61|62|63|65|67|68|"
|
| 192 |
+
"71|74|75|76|77|81|82|83|84|85|86|87|89|91|94|95")
|
| 193 |
+
_GROUPED_PHONE_RE = re.compile(
|
| 194 |
+
r"(?:\+48[ \t]?)?(?:(?:" + _MOBILE_PREFIX + r")\d(?P<m>[ -])\d{3}(?P=m)\d{3}"
|
| 195 |
+
r"|(?:\((?:" + _AREA_CODE + r")\)|(?:" + _AREA_CODE + r"))(?P<l>[ -])\d{3}(?P=l)\d{2}(?P=l)\d{2})")
|
| 196 |
_PHONE_LIST_HEADING_RE = re.compile(
|
| 197 |
r"[ \t]*(?:numery[ \t]+kontaktowe|telefony(?:[ \t]+(?:kontaktowe|zaufania))?|fax|faks|zapisz[ \t]+się|"
|
| 198 |
r"(?:[^.!?\n]{0,40}\?[ \t]*)?(?:za)?dzwoń!?)"
|
|
|
|
| 213 |
|
| 214 |
|
| 215 |
def _phone_list_context(text: str, start: int) -> bool:
|
| 216 |
+
# 2,000-character lookback and 400-character entry suffix; longer
|
| 217 |
# lists need repeated headings or a section parser. Never cross prose.
|
| 218 |
end = text.find('\n', start, start + 401)
|
| 219 |
if end < 0 and len(text) > start + 400:
|
|
|
|
| 294 |
foreign = raw.startswith('+') and _phone_ok(raw)
|
| 295 |
if not foreign and (not _pl_national_ok(_digits(raw)) or not re.search(r'[ \t-]', raw)):
|
| 296 |
return False
|
| 297 |
+
# Local 1,600-character directory evidence; longer isolated rows
|
| 298 |
# need preserved upstream table structure, not an unbounded document cue.
|
| 299 |
left, right = max(0, start-800), min(len(text), end+800)
|
| 300 |
nearby = text[left:right]
|
|
|
|
| 478 |
return _LAND_REGISTER_RE.sub(replace, text), n
|
| 479 |
|
| 480 |
|
| 481 |
+
def _head(raw: str) -> str:
|
| 482 |
+
# The greedy match may run on over a newline or a slash; the break and pair logic in _replace_phones cut it there.
|
| 483 |
+
return re.split(r"[ \t]*[\n/]", raw, maxsplit=1)[0]
|
| 484 |
+
|
| 485 |
+
|
| 486 |
+
def _national_head(raw: str) -> bool:
|
| 487 |
+
return _pl_national_ok(re.sub(r"^(?:48|0)(?=\d{9}$)", "", _digits(_head(raw))))
|
| 488 |
+
|
| 489 |
+
|
| 490 |
+
def _grouped_phone(text: str, start: int, raw: str) -> bool:
|
| 491 |
+
# Digits glued before the number on its line ("48 601 234 567", list indexes) are not tried.
|
| 492 |
+
head = _head(raw)
|
| 493 |
+
return bool(_GROUPED_PHONE_RE.fullmatch(head)
|
| 494 |
+
and not re.search(r"[ -]000[ -]000\Z", head) # round counts: 500 000 000
|
| 495 |
+
and not _is_amount(text, start, start + len(head))
|
| 496 |
+
and not re.match(r",\d{1,2}(?!\d)", text[start + len(head):]) # decimal figures: 601 234 567,89
|
| 497 |
+
and not _OTHER_NUMBER_LABEL_RE.search(text[max(0, start - 75):start]))
|
| 498 |
+
|
| 499 |
+
|
| 500 |
def _pl_national_ok(d: str) -> bool:
|
| 501 |
return len(d) == 9 and d.isdigit() and d[:2] in _PL_PREFIX
|
| 502 |
|
|
|
|
| 527 |
return True
|
| 528 |
if short and s.startswith('(0)'):
|
| 529 |
return 7 <= len(d) <= 12
|
| 530 |
+
# Label + shape + length, not a country numbering-plan validator.
|
| 531 |
if short and re.match(r'^\(0?\d{2,4}(?:-\d{1,2})?\)', s):
|
| 532 |
return 8 <= len(d) <= 15
|
| 533 |
# EUR-Lex puts country and area code in parentheses: "(32-2) 299 11 11" is +32 2 299 11 11.
|
|
|
|
| 537 |
if s.startswith('00'):
|
| 538 |
international = international[2:]
|
| 539 |
if not international.startswith('48'):
|
| 540 |
+
# Explicit prefix + length, not a global numbering-plan
|
| 541 |
# validator. Add country metadata if review finds false positives.
|
| 542 |
return (7 if '(0)' in s else 8) <= len(international) <= 15 and international[0] != '0'
|
| 543 |
if d.startswith("00"):
|
|
|
|
| 626 |
pos = 0
|
| 627 |
last_phone_end = None
|
| 628 |
last_phone_complete = False
|
| 629 |
+
last_phone_cued = False # after a bare grouped phone a list goes on only with complete national numbers
|
| 630 |
while True:
|
| 631 |
m = _PHONE_RE.search(text, pos)
|
| 632 |
if not m:
|
|
|
|
| 638 |
line_end = text.find('\n', m.start(), end)
|
| 639 |
if line_end >= 0 and re.fullmatch(r'\d{1,3}', text[m.start():line_end]):
|
| 640 |
next_end = text.find('\n', line_end+1, end)
|
| 641 |
+
next_end = next_end if next_end >= 0 else end
|
| 642 |
+
if (_directory_context(text, line_end+1, next_end)
|
| 643 |
+
or _grouped_phone(text, line_end+1, text[line_end+1:next_end])):
|
| 644 |
+
# A flattened table's room cell or an address's house number is not a phone country prefix.
|
| 645 |
out.append(text[pos:line_end+1])
|
| 646 |
pos = line_end+1
|
| 647 |
continue
|
|
|
|
| 669 |
# "tel. +48 609 \n953 709", or the next local number, "(30) 25 10 22 33 25,\n22 33 28".
|
| 670 |
cell = (last_phone_complete and 5 <= len(_digits(raw)) <= 6
|
| 671 |
and re.fullmatch(r'[ \t]*\n[ \t\n]*', text[last_phone_end:m.start()]))
|
| 672 |
+
continues = last_phone_cued or (last_phone_end is not None and _national_head(raw))
|
| 673 |
+
if (continues and not postal and not cell
|
| 674 |
and re.fullmatch(r"[ \t\n,;]{1,12}|[ \t]+(?:lub|albo)[ \t]+", text[last_phone_end:m.start()], re.I)):
|
| 675 |
short = True
|
| 676 |
+
if (continues
|
| 677 |
and re.fullmatch(r'(?:[ \t]+(?:w sprawie\b|\((?:w godzinach|dostępny)\b)'
|
| 678 |
r'|[ \t]*[–—-][ \t]+)[^\d\n.!?]{1,200}\n[ \t\n]{0,8}',
|
| 679 |
text[last_phone_end:m.start()], re.I)
|
| 680 |
and _pl_national_ok(_digits(raw))):
|
| 681 |
short = True
|
| 682 |
+
cued = (short or _phone_context(text, m.start(), raw)
|
| 683 |
+
or _directory_context(text, m.start(), line_end if line_end >= 0 else end))
|
| 684 |
+
if novel_country_area_ok and (cued or _grouped_phone(text, m.start(), raw)):
|
| 685 |
if (_is_amount(text, m.start(), end)
|
| 686 |
and re.fullmatch(r'\d{1,3}(?:\.[ \t]*\d{3})+', raw)):
|
| 687 |
out.extend((text[pos:m.start()], raw))
|
|
|
|
| 734 |
start += 1
|
| 735 |
replacement = text[m.start():start] + mask(start, end, PHONE_TAG)
|
| 736 |
n += 1
|
| 737 |
+
# Split only pairs (<=22 digits); longer lists need label
|
| 738 |
# context to distinguish them from accounts. Never redact ID tails.
|
| 739 |
elif 18 <= len(_digits(raw)) <= 22 and "." not in raw:
|
| 740 |
# A slash separates alternatives, the second without the shared
|
|
|
|
| 750 |
if replacement != raw:
|
| 751 |
last_phone_end = end
|
| 752 |
last_phone_complete = _phone_ok(raw)
|
| 753 |
+
last_phone_cued = bool(cued)
|
| 754 |
pos = end
|
| 755 |
return "".join(out), n
|
| 756 |
|
test_nergal.py
CHANGED
|
@@ -5,7 +5,7 @@ import unittest
|
|
| 5 |
from pathlib import Path
|
| 6 |
|
| 7 |
HERE = Path(__file__).resolve().parent
|
| 8 |
-
RULES_SHA = '
|
| 9 |
|
| 10 |
|
| 11 |
class NergalTests(unittest.TestCase):
|
|
@@ -13,7 +13,7 @@ class NergalTests(unittest.TestCase):
|
|
| 13 |
from nergal import GAP_IDS, GAPS, HUB_ID, RULES_SHA as PINNED, THRESHOLD, VERSION
|
| 14 |
card = json.loads((HERE / 'hybrid.json').read_text())
|
| 15 |
self.assertEqual(HUB_ID, 'SlayerLab/NERGAL')
|
| 16 |
-
self.assertEqual(VERSION, '1.1.
|
| 17 |
self.assertEqual(card['version'], VERSION)
|
| 18 |
self.assertEqual(card['eval']['union_fp'], 123)
|
| 19 |
self.assertEqual(card['eval']['rules_fp'], 98)
|
|
@@ -62,6 +62,19 @@ class NergalTests(unittest.TestCase):
|
|
| 62 |
self.assertEqual(text[span['start']:span['end']], '1234567802')
|
| 63 |
self.assertEqual(rules('a [PII] b [Telefon] c'), [])
|
| 64 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 65 |
def test_union_keeps_regex_and_adds_model_spans(self):
|
| 66 |
from nergal import apply_union, scrub_spans
|
| 67 |
text = 'Ring 000000000 then extra.'
|
|
|
|
| 5 |
from pathlib import Path
|
| 6 |
|
| 7 |
HERE = Path(__file__).resolve().parent
|
| 8 |
+
RULES_SHA = 'ad5c51f7c712dd2b7af40a337b703aeec211a57170a87c262bf9bd9386ba1925'
|
| 9 |
|
| 10 |
|
| 11 |
class NergalTests(unittest.TestCase):
|
|
|
|
| 13 |
from nergal import GAP_IDS, GAPS, HUB_ID, RULES_SHA as PINNED, THRESHOLD, VERSION
|
| 14 |
card = json.loads((HERE / 'hybrid.json').read_text())
|
| 15 |
self.assertEqual(HUB_ID, 'SlayerLab/NERGAL')
|
| 16 |
+
self.assertEqual(VERSION, '1.1.1')
|
| 17 |
self.assertEqual(card['version'], VERSION)
|
| 18 |
self.assertEqual(card['eval']['union_fp'], 123)
|
| 19 |
self.assertEqual(card['eval']['rules_fp'], 98)
|
|
|
|
| 62 |
self.assertEqual(text[span['start']:span['end']], '1234567802')
|
| 63 |
self.assertEqual(rules('a [PII] b [Telefon] c'), [])
|
| 64 |
|
| 65 |
+
def test_grouped_national_phones_mask_without_a_cue(self):
|
| 66 |
+
from nergal import rules
|
| 67 |
+
for text, masked in (('Sklep Ala, 601 234 567, czynne 9-17', ['601 234 567']),
|
| 68 |
+
('Biuro: (22) 123 45 67.', ['(22) 123 45 67']),
|
| 69 |
+
('Zapraszamy: +48 601 234 567.', ['+48 601 234 567']),
|
| 70 |
+
('Zapraszamy: 601234567.', []), # plain 9 digits stay cue-gated
|
| 71 |
+
('Budżet wyniósł 601 234 567 zł.', []), # amount
|
| 72 |
+
('Wartość 601 234 567,89 w tabeli.', []), # decimal figure
|
| 73 |
+
('Kwota 500 000 000 osób.', []), # round count
|
| 74 |
+
('NIP: 601 234 567', [])): # other-number label
|
| 75 |
+
with self.subTest(text=text):
|
| 76 |
+
self.assertEqual([text[s['start']:s['end']] for s in rules(text) if s['label'] == 'phone'], masked)
|
| 77 |
+
|
| 78 |
def test_union_keeps_regex_and_adds_model_spans(self):
|
| 79 |
from nergal import apply_union, scrub_spans
|
| 80 |
text = 'Ring 000000000 then extra.'
|