ppuzio Claude Opus 5.5 commited on
Commit
4464d47
·
1 Parent(s): 78cc85a

1.1.1: grouped national phones without a label

Browse files

Rules only; same weights, threshold and API. 841-dev 326/354 whole (324), 0 new false
characters; cue-less phone test 66/111 phones (37), scored once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Files changed (6) hide show
  1. CHANGELOG.md +17 -0
  2. README.md +25 -10
  3. hybrid.json +10 -10
  4. nergal.py +2 -2
  5. scrub_pii.py +47 -13
  6. test_nergal.py +15 -2
CHANGELOG.md CHANGED
@@ -8,6 +8,23 @@ Semver for this island:
8
 
9
  Accuracy is 841-dev, union at 0.95, 354 gold spans. A version that changes those numbers must update `hybrid.json` `eval` and the tables below.
10
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
11
  ## 1.1.0
12
 
13
  Same weights, rules, threshold and default outputs; `hybrid.json` `eval` is unchanged. New batch API and opt-in fp16.
 
8
 
9
  Accuracy is 841-dev, union at 0.95, 354 gold spans. A version that changes those numbers must update `hybrid.json` `eval` and the tables below.
10
 
11
+ ## 1.1.1
12
+
13
+ Same weights, threshold and API. Rules SHA `ad5c51f7…`. The rules file is the frozen and scored candidate `7412261c…` with comment tags removed; its syntax tree is identical.
14
+
15
+ - **Rules:** Polish phones in the grouped national forms are masked without a contact label: mobile `601 234 567` and landline `22 123 45 67` (area code optionally in parentheses), optional `+48`, one separator kind (space or hyphen) throughout. Not taken: plain 9-digit strings (still label-gated), amounts, round counts (`500 000 000`), decimal figures (`601 234 567,89`), and numbers after another identifier's label (NIP, REGON, codes). After a bare grouped phone, a list goes on only with complete Polish numbers.
16
+
17
+ 841-dev: 326/354 whole (1.1.0: 324), residual 23 (24), 0 new false characters; phone 147/169 (145). A frozen blind set of 150 web passages selected for unlabelled phones: phones wholly masked 66/111 (1.1.0: 37), all values 127/178 (98), 0 lost, 1 passage with new false characters. Dynaword (60,000 documents): 79 documents change, 247 phone spans added, 0 removed; in a reviewed sample 1 of 23 sampled documents' added spans was not a phone.
18
+
19
+ | Version | Whole /354 | Residual | Rules FP | Union FP | Char P | Char R | What changed |
20
+ |---|---:|---:|---:|---:|---:|---:|---|
21
+ | 1.0.0 | 323 | 25 | 133 | 133 | 97.76% | 95.95% | First Hub snapshot |
22
+ | 1.0.1 | 323 | 25 | 98 | 123 | 97.93% | 95.95% | Prefix-only glued-email trim |
23
+ | 1.0.2 | 324 | 24 | 98 | 123 | 97.93% | 96.12% | Labelled country-area phone fix |
24
+ | 1.0.3 | 324 | 24 | 98 | 123 | 97.93% | 96.12% | Label-note, e-Delivery and registry rules; placeholder and card fixes |
25
+ | 1.1.0 | 324 | 24 | 98 | 123 | 97.93% | 96.12% | Batch API (`predict_many`, `scrub_many`), opt-in float16 |
26
+ | 1.1.1 | 326 | 23 | 98 | 123 | 97.94% | 96.50% | Grouped national phones without a label |
27
+
28
  ## 1.1.0
29
 
30
  Same weights, rules, threshold and default outputs; `hybrid.json` `eval` is unchanged. New batch API and opt-in fp16.
README.md CHANGED
@@ -13,7 +13,7 @@ tags:
13
  - hybrid
14
  ---
15
 
16
- # NERGAL 1.1.0
17
 
18
  **Named Entity Recognition with Grounded Additive Labels**
19
 
@@ -23,8 +23,8 @@ SlayerLab hybrid PII cleaner for Polish. Not a chat model. Not a drop-in `pipeli
23
 
24
  Python rules do the identifiers they can prove. A transformer NER head adds phone and other PII spans the regex misses. The cleaner **unions** the two on the original text, then replaces hits with `[Telefon]` or `[PII]`.
25
 
26
- - **Version:** `1.1.0` (`hybrid.json`, `CHANGELOG.md`)
27
- - **Ground:** `scrub_pii` regex (SHA256 `f32d5c54…`)
28
  - **Additive labels:** XLM-RoBERTa-large token classifier, BIO tags `phone` / `pii`, threshold 0.95
29
  - **Throughput:** about 80k chars/s on one RTX 4090 with `scrub_many` + `dtype="float16"` and 3 processes (1.0.3: 23k)
30
  - **This snapshot:** seed `202609160`, **epoch 5** of a seven-epoch schedule
@@ -62,9 +62,9 @@ NERGAL is not designed to remove:
62
 
63
  These are intended exclusions; false positives can still mask some of this content.
64
 
65
- ### Known gaps in 1.1.0
66
 
67
- Unlabelled phones and identifiers, unusual formatting and damaged text can escape detection. **VINs and obfuscated emails** (such as `name (at) domain.pl`) are approved annotation targets, but that approval alone does not establish reliable support in the released 1.1.0 model. Do not rely on it to remove them consistently.
68
 
69
  ## Versions
70
 
@@ -76,7 +76,22 @@ Unlabelled phones and identifiers, unusual formatting and damaged text can escap
76
  | 1.0.1 | 323 | 25 | 98 | 123 | 97.93% | 95.95% | Prefix-only glued-email trim |
77
  | 1.0.2 | 324 | 24 | 98 | 123 | 97.93% | 96.12% | Labelled country-area phone fix |
78
  | 1.0.3 | 324 | 24 | 98 | 123 | 97.93% | 96.12% | Label-note, e-Delivery and registry rules; placeholder and card fixes |
79
- | **1.1.0** | **324** | **24** | **98** | **123** | **97.93%** | **96.12%** | Batch API (`predict_many`, `scrub_many`), opt-in float16 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
80
 
81
  ## 841-dev
82
 
@@ -112,15 +127,15 @@ Same 841-dev split, threshold **0.95**. **Naked** is the transformer alone. **
112
 
113
  | System | Mode | Whole /354 | Residual | False chars | Char P | Char R |
114
  |---|---|---:|---:|---:|---:|---:|
115
- | Regex (`scrub_pii`) | rules | 259 | 70 | 98 | 98.13% | 85.04% |
116
  | GLiNER 2.5-multi zero-shot | naked | 81 | 188 | 970 | 56.98% | 21.22% |
117
  | GLiNER 2.5-multi zero-shot | ∪ regex | 273 | 61 | 1,068 | 83.33% | 88.16% |
118
  | Historical GLiNER email12 | naked | 249 | 70 | 10 | 99.81% | 85.39% |
119
  | Historical GLiNER email12 | ∪ regex | 290 | 50 | 108 | 98.13% | 93.76% |
120
  | XLM-R epoch 5 | naked | 298 | 42 | 57 | 98.96% | 89.38% |
121
- | **NERGAL 1.1.0** | **∪ regex** | 324 | 24 | 123 | 97.93% | 96.12% |
122
 
123
- Zero-shot [GLiNER 2.5](https://huggingface.co/fastino/gliner2.5-multi-v1) is not competitive here, especially on non-phone PII (7/185 whole vs 160 naked / 179 union). Fine-tuned historical GLiNER is the precise naked baseline (10 false characters) and still trails XLM-R on coverage. NERGAL is XLM-R epoch 5 plus the regex: 145/169 phone, 179/185 other PII. Exact-span precision 86.34%, recall 89.27%, F1 87.78%.
124
 
125
  Trained GLiNER, HerBERT-large, and XLM-R-large were also compared on this split (plot above). GLiNER’s 0.50 coverage lead is the precision collapse in that figure; no GLiNER or HerBERT operating point passed the content-preservation gate.
126
 
@@ -155,6 +170,6 @@ masked, counts = nergal.scrub(text)
155
 
156
  For many texts, `nergal.scrub_many(texts)` (or `predict_many` for the raw model spans) batches windows across texts: about 2× the throughput of calling `scrub` in a loop on a CUDA GPU. `from_pretrained(..., dtype="float16")` casts the weights at load time on CUDA or MPS: another 1.9× on an RTX 4090, and 841-dev union numbers are unchanged, but scores are not bit-identical to float32 (0.95 spans can differ in rare cases). For a corpus, run 2–3 processes per GPU, because float16 inference is limited by CPU-side tokenization. Measurements are in `CHANGELOG.md`.
157
 
158
- `hybrid.json` records version `1.1.0`, threshold 0.95, gap ids `250002` / `250003`, the 841-dev `eval` block, and two weight hashes: `model_safetensors_sha256` for the published file and `source_checkpoint_sha256` for the training checkpoint it was packed from. `test_nergal.py` is synthetic (no corpus text). From this snapshot: `python -m unittest test_nergal`.
159
 
160
  Base weights: [`FacebookAI/xlm-roberta-large`](https://huggingface.co/FacebookAI/xlm-roberta-large) revision `c23d21b0620b635a76227c604d44e43a9f0ee389` (MIT).
 
13
  - hybrid
14
  ---
15
 
16
+ # NERGAL 1.1.1
17
 
18
  **Named Entity Recognition with Grounded Additive Labels**
19
 
 
23
 
24
  Python rules do the identifiers they can prove. A transformer NER head adds phone and other PII spans the regex misses. The cleaner **unions** the two on the original text, then replaces hits with `[Telefon]` or `[PII]`.
25
 
26
+ - **Version:** `1.1.1` (`hybrid.json`, `CHANGELOG.md`)
27
+ - **Ground:** `scrub_pii` regex (SHA256 `ad5c51f7…`)
28
  - **Additive labels:** XLM-RoBERTa-large token classifier, BIO tags `phone` / `pii`, threshold 0.95
29
  - **Throughput:** about 80k chars/s on one RTX 4090 with `scrub_many` + `dtype="float16"` and 3 processes (1.0.3: 23k)
30
  - **This snapshot:** seed `202609160`, **epoch 5** of a seven-epoch schedule
 
62
 
63
  These are intended exclusions; false positives can still mask some of this content.
64
 
65
+ ### Known gaps in 1.1.1
66
 
67
+ Unlabelled phones and identifiers (the rules take a bare phone only in the grouped national forms `601 234 567` and `22 123 45 67`; a plain `601234567` needs a label or the model), unusual formatting and damaged text can escape detection. **VINs and obfuscated emails** (such as `name (at) domain.pl`) are approved annotation targets, but that approval alone does not establish reliable support in the released 1.1.1 model. Do not rely on it to remove them consistently.
68
 
69
  ## Versions
70
 
 
76
  | 1.0.1 | 323 | 25 | 98 | 123 | 97.93% | 95.95% | Prefix-only glued-email trim |
77
  | 1.0.2 | 324 | 24 | 98 | 123 | 97.93% | 96.12% | Labelled country-area phone fix |
78
  | 1.0.3 | 324 | 24 | 98 | 123 | 97.93% | 96.12% | Label-note, e-Delivery and registry rules; placeholder and card fixes |
79
+ | 1.1.0 | 324 | 24 | 98 | 123 | 97.93% | 96.12% | Batch API (`predict_many`, `scrub_many`), opt-in float16 |
80
+ | **1.1.1** | **326** | **23** | **98** | **123** | **97.94%** | **96.50%** | Grouped national phones without a label |
81
+
82
+ ## Cue-less phone test
83
+
84
+ 841-dev holds few phones without a label, so it barely moves for 1.1.1. This targeted test does: 150 web passages selected for phones with no contact label, labelled and frozen before any prediction, and scored once (2026-09-29). Union at 0.95, same model spans for both.
85
+
86
+ | | 1.1.0 | **1.1.1** |
87
+ |---|---:|---:|
88
+ | Phones wholly masked /111 | 37 | **66** |
89
+ | All values wholly masked /178 | 98 | **127** |
90
+ | Positive passages fully covered /70 (95% CI) | 24 (0.23–0.47) | **37 (0.41–0.65)** |
91
+ | Passages with false masks /150 | 3 | 4 |
92
+ | False characters | 83 | 95 |
93
+
94
+ 1.1.1 lost no value 1.1.0 masked; its 12 new false characters are in one passage. The set is enriched by selection, so these numbers say nothing about how common such phones are, and it has a single reviewer.
95
 
96
  ## 841-dev
97
 
 
127
 
128
  | System | Mode | Whole /354 | Residual | False chars | Char P | Char R |
129
  |---|---|---:|---:|---:|---:|---:|
130
+ | Regex (`scrub_pii`) | rules | 263 | 68 | 98 | 98.15% | 85.78% |
131
  | GLiNER 2.5-multi zero-shot | naked | 81 | 188 | 970 | 56.98% | 21.22% |
132
  | GLiNER 2.5-multi zero-shot | ∪ regex | 273 | 61 | 1,068 | 83.33% | 88.16% |
133
  | Historical GLiNER email12 | naked | 249 | 70 | 10 | 99.81% | 85.39% |
134
  | Historical GLiNER email12 | ∪ regex | 290 | 50 | 108 | 98.13% | 93.76% |
135
  | XLM-R epoch 5 | naked | 298 | 42 | 57 | 98.96% | 89.38% |
136
+ | **NERGAL 1.1.1** | **∪ regex** | 326 | 23 | 123 | 97.94% | 96.50% |
137
 
138
+ Zero-shot [GLiNER 2.5](https://huggingface.co/fastino/gliner2.5-multi-v1) is not competitive here, especially on non-phone PII (7/185 whole vs 160 naked / 179 union). Fine-tuned historical GLiNER is the precise naked baseline (10 false characters) and still trails XLM-R on coverage. NERGAL is XLM-R epoch 5 plus the regex: 147/169 phone, 179/185 other PII. Exact-span precision 86.41%, recall 89.83%, F1 88.09%. The two GLiNER ∪ regex rows use the 1.1.0 rules.
139
 
140
  Trained GLiNER, HerBERT-large, and XLM-R-large were also compared on this split (plot above). GLiNER’s 0.50 coverage lead is the precision collapse in that figure; no GLiNER or HerBERT operating point passed the content-preservation gate.
141
 
 
170
 
171
  For many texts, `nergal.scrub_many(texts)` (or `predict_many` for the raw model spans) batches windows across texts: about 2× the throughput of calling `scrub` in a loop on a CUDA GPU. `from_pretrained(..., dtype="float16")` casts the weights at load time on CUDA or MPS: another 1.9× on an RTX 4090, and 841-dev union numbers are unchanged, but scores are not bit-identical to float32 (0.95 spans can differ in rare cases). For a corpus, run 2–3 processes per GPU, because float16 inference is limited by CPU-side tokenization. Measurements are in `CHANGELOG.md`.
172
 
173
+ `hybrid.json` records version `1.1.1`, threshold 0.95, gap ids `250002` / `250003`, the 841-dev `eval` block, and two weight hashes: `model_safetensors_sha256` for the published file and `source_checkpoint_sha256` for the training checkpoint it was packed from. `test_nergal.py` is synthetic (no corpus text). From this snapshot: `python -m unittest test_nergal`.
174
 
175
  Base weights: [`FacebookAI/xlm-roberta-large`](https://huggingface.co/FacebookAI/xlm-roberta-large) revision `c23d21b0620b635a76227c604d44e43a9f0ee389` (MIT).
hybrid.json CHANGED
@@ -1,25 +1,25 @@
1
  {
2
  "full_name": "Named Entity Recognition with Grounded Additive Labels",
3
  "hub_id": "SlayerLab/NERGAL",
4
- "version": "1.1.0",
5
  "mode": "rules_union",
6
  "epoch": 5,
7
  "seed": 202609160,
8
  "threshold": 0.95,
9
- "rules_sha256": "f32d5c5452fc47178e109d4bc248a0d8234ea6e59e8cf79407f4eb8451581d67",
10
  "eval": {
11
  "split": "841-dev",
12
  "gold_entities": 354,
13
- "whole_entities": 324,
14
- "residual_passages": 24,
15
  "union_fp": 123,
16
  "rules_fp": 98,
17
- "character_precision": 0.9793,
18
- "character_recall": 0.9612,
19
- "exact_precision": 0.8634,
20
- "exact_recall": 0.8927,
21
- "exact_f1": 0.8778,
22
- "phone_whole": 145,
23
  "phone_gold": 169,
24
  "pii_whole": 179,
25
  "pii_gold": 185
 
1
  {
2
  "full_name": "Named Entity Recognition with Grounded Additive Labels",
3
  "hub_id": "SlayerLab/NERGAL",
4
+ "version": "1.1.1",
5
  "mode": "rules_union",
6
  "epoch": 5,
7
  "seed": 202609160,
8
  "threshold": 0.95,
9
+ "rules_sha256": "ad5c51f7c712dd2b7af40a337b703aeec211a57170a87c262bf9bd9386ba1925",
10
  "eval": {
11
  "split": "841-dev",
12
  "gold_entities": 354,
13
+ "whole_entities": 326,
14
+ "residual_passages": 23,
15
  "union_fp": 123,
16
  "rules_fp": 98,
17
+ "character_precision": 0.9794,
18
+ "character_recall": 0.965,
19
+ "exact_precision": 0.8641,
20
+ "exact_recall": 0.8983,
21
+ "exact_f1": 0.8809,
22
+ "phone_whole": 147,
23
  "phone_gold": 169,
24
  "pii_whole": 179,
25
  "pii_gold": 185
nergal.py CHANGED
@@ -17,13 +17,13 @@ import scrub_pii
17
  from scrub_pii import PHONE_TAG, PII_TAG
18
 
19
  HUB_ID = 'SlayerLab/NERGAL'
20
- VERSION = '1.1.0'
21
  GAPS = ['[PII_SPACE]', '[PII_BREAK]']
22
  GAP_IDS = [250002, 250003]
23
  BIO_LABELS = ['O', 'B-phone', 'I-phone', 'B-pii', 'I-pii']
24
  LABELS = ['phone', 'pii']
25
  THRESHOLD = 0.95
26
- RULES_SHA = 'f32d5c5452fc47178e109d4bc248a0d8234ea6e59e8cf79407f4eb8451581d67'
27
  DTYPES = ('float32', 'float16')
28
 
29
 
 
17
  from scrub_pii import PHONE_TAG, PII_TAG
18
 
19
  HUB_ID = 'SlayerLab/NERGAL'
20
+ VERSION = '1.1.1'
21
  GAPS = ['[PII_SPACE]', '[PII_BREAK]']
22
  GAP_IDS = [250002, 250003]
23
  BIO_LABELS = ['O', 'B-phone', 'I-phone', 'B-pii', 'I-pii']
24
  LABELS = ['phone', 'pii']
25
  THRESHOLD = 0.95
26
+ RULES_SHA = 'ad5c51f7c712dd2b7af40a337b703aeec211a57170a87c262bf9bd9386ba1925'
27
  DTYPES = ('float32', 'float16')
28
 
29
 
scrub_pii.py CHANGED
@@ -24,7 +24,8 @@ international country/trunk notation such as +CC (0). With a cue, the EUR-Lex
24
  "(32-2) 299 11 11" country-area form counts as international. Strong labels also
25
  admit one-digit country codes and wider hyphenated area codes. Short service numbers
26
  need strong labels; 116xxx numbers also accept nearby telephone prose. Unlabelled
27
- domestic numbers are left for audit because table cells have the same shapes.
 
28
  Flattened tables can glue labels to both neighbours ("Mödlingtel.: … 38112faks:");
29
  such glued labels count as labels and end the preceding number.
30
 
@@ -75,7 +76,7 @@ _PIN_RE = re.compile(
75
  _SPACES = str.maketrans({c: " " for c in "\u00a0\u1680\u2000\u2001\u2002\u2003\u2004\u2005\u2006\u2007\u2008\u2009\u200a\u202f\u205f\u3000"})
76
  # Country lengths checked against Apache Commons Validator's IBAN registry table:
77
  # https://commons.apache.org/proper/commons-validator/xref/org/apache/commons/validator/routines/IBANValidator.html
78
- # ponytail: country/length/MOD-97 only; add national BBAN rules if false positives appear.
79
  _IBAN_LENGTHS = {
80
  country: length for length, countries in {
81
  15: "NO", 16: "BE", 18: "DK FI AX FK FO GL NL SD",
@@ -184,6 +185,14 @@ _SERVICE_PHONE_LABEL_RE = re.compile(
184
  r"(?:[ \t]+\([^()\n]{1,60}\))?[ \t]*:[ \t]*)"
185
  r"(?:[1-9]\d{2}[ \t]*(?:[,;]|i|lub|oraz)[ \t]*)*\Z", re.I)
186
  _OTHER_NUMBER_LABEL_RE = re.compile(r"\b(?:NIP|REGON|PESEL|KRS|ISBN|kod)\b[^\d\n]{0,20}\Z", re.I)
 
 
 
 
 
 
 
 
187
  _PHONE_LIST_HEADING_RE = re.compile(
188
  r"[ \t]*(?:numery[ \t]+kontaktowe|telefony(?:[ \t]+(?:kontaktowe|zaufania))?|fax|faks|zapisz[ \t]+się|"
189
  r"(?:[^.!?\n]{0,40}\?[ \t]*)?(?:za)?dzwoń!?)"
@@ -204,7 +213,7 @@ _STAFF_ROLE_RE = re.compile(
204
 
205
 
206
  def _phone_list_context(text: str, start: int) -> bool:
207
- # ponytail: 2,000-character lookback and 400-character entry suffix; longer
208
  # lists need repeated headings or a section parser. Never cross prose.
209
  end = text.find('\n', start, start + 401)
210
  if end < 0 and len(text) > start + 400:
@@ -285,7 +294,7 @@ def _directory_context(text: str, start: int, end: int) -> bool:
285
  foreign = raw.startswith('+') and _phone_ok(raw)
286
  if not foreign and (not _pl_national_ok(_digits(raw)) or not re.search(r'[ \t-]', raw)):
287
  return False
288
- # ponytail: local 1,600-character directory evidence; longer isolated rows
289
  # need preserved upstream table structure, not an unbounded document cue.
290
  left, right = max(0, start-800), min(len(text), end+800)
291
  nearby = text[left:right]
@@ -469,6 +478,25 @@ def _replace_land_registers(text: str, *, mask=_tag) -> tuple[str, int]:
469
  return _LAND_REGISTER_RE.sub(replace, text), n
470
 
471
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
472
  def _pl_national_ok(d: str) -> bool:
473
  return len(d) == 9 and d.isdigit() and d[:2] in _PL_PREFIX
474
 
@@ -499,7 +527,7 @@ def _phone_ok(raw: str, short: bool = False) -> bool:
499
  return True
500
  if short and s.startswith('(0)'):
501
  return 7 <= len(d) <= 12
502
- # ponytail: label + shape + length, not a country numbering-plan validator.
503
  if short and re.match(r'^\(0?\d{2,4}(?:-\d{1,2})?\)', s):
504
  return 8 <= len(d) <= 15
505
  # EUR-Lex puts country and area code in parentheses: "(32-2) 299 11 11" is +32 2 299 11 11.
@@ -509,7 +537,7 @@ def _phone_ok(raw: str, short: bool = False) -> bool:
509
  if s.startswith('00'):
510
  international = international[2:]
511
  if not international.startswith('48'):
512
- # ponytail: explicit prefix + length, not a global numbering-plan
513
  # validator. Add country metadata if review finds false positives.
514
  return (7 if '(0)' in s else 8) <= len(international) <= 15 and international[0] != '0'
515
  if d.startswith("00"):
@@ -598,6 +626,7 @@ def _replace_phones(text: str, *, mask=_tag) -> tuple[str, int]:
598
  pos = 0
599
  last_phone_end = None
600
  last_phone_complete = False
 
601
  while True:
602
  m = _PHONE_RE.search(text, pos)
603
  if not m:
@@ -609,8 +638,10 @@ def _replace_phones(text: str, *, mask=_tag) -> tuple[str, int]:
609
  line_end = text.find('\n', m.start(), end)
610
  if line_end >= 0 and re.fullmatch(r'\d{1,3}', text[m.start():line_end]):
611
  next_end = text.find('\n', line_end+1, end)
612
- if _directory_context(text, line_end+1, next_end if next_end >= 0 else end):
613
- # A flattened table's room cell is not a phone country prefix.
 
 
614
  out.append(text[pos:line_end+1])
615
  pos = line_end+1
616
  continue
@@ -638,17 +669,19 @@ def _replace_phones(text: str, *, mask=_tag) -> tuple[str, int]:
638
  # "tel. +48 609 \n953 709", or the next local number, "(30) 25 10 22 33 25,\n22 33 28".
639
  cell = (last_phone_complete and 5 <= len(_digits(raw)) <= 6
640
  and re.fullmatch(r'[ \t]*\n[ \t\n]*', text[last_phone_end:m.start()]))
641
- if (last_phone_end is not None and not postal and not cell
 
642
  and re.fullmatch(r"[ \t\n,;]{1,12}|[ \t]+(?:lub|albo)[ \t]+", text[last_phone_end:m.start()], re.I)):
643
  short = True
644
- if (last_phone_end is not None
645
  and re.fullmatch(r'(?:[ \t]+(?:w sprawie\b|\((?:w godzinach|dostępny)\b)'
646
  r'|[ \t]*[–—-][ \t]+)[^\d\n.!?]{1,200}\n[ \t\n]{0,8}',
647
  text[last_phone_end:m.start()], re.I)
648
  and _pl_national_ok(_digits(raw))):
649
  short = True
650
- if (novel_country_area_ok and (short or _phone_context(text, m.start(), raw)
651
- or _directory_context(text, m.start(), line_end if line_end >= 0 else end))):
 
652
  if (_is_amount(text, m.start(), end)
653
  and re.fullmatch(r'\d{1,3}(?:\.[ \t]*\d{3})+', raw)):
654
  out.extend((text[pos:m.start()], raw))
@@ -701,7 +734,7 @@ def _replace_phones(text: str, *, mask=_tag) -> tuple[str, int]:
701
  start += 1
702
  replacement = text[m.start():start] + mask(start, end, PHONE_TAG)
703
  n += 1
704
- # ponytail: split only pairs (<=22 digits); longer lists need label
705
  # context to distinguish them from accounts. Never redact ID tails.
706
  elif 18 <= len(_digits(raw)) <= 22 and "." not in raw:
707
  # A slash separates alternatives, the second without the shared
@@ -717,6 +750,7 @@ def _replace_phones(text: str, *, mask=_tag) -> tuple[str, int]:
717
  if replacement != raw:
718
  last_phone_end = end
719
  last_phone_complete = _phone_ok(raw)
 
720
  pos = end
721
  return "".join(out), n
722
 
 
24
  "(32-2) 299 11 11" country-area form counts as international. Strong labels also
25
  admit one-digit country codes and wider hyphenated area codes. Short service numbers
26
  need strong labels; 116xxx numbers also accept nearby telephone prose. Unlabelled
27
+ domestic numbers are left for audit because table cells have the same shapes, except
28
+ the grouped national forms: mobile 3-3-3 and landline 2-3-2-2, one separator kind.
29
  Flattened tables can glue labels to both neighbours ("Mödlingtel.: … 38112faks:");
30
  such glued labels count as labels and end the preceding number.
31
 
 
76
  _SPACES = str.maketrans({c: " " for c in "\u00a0\u1680\u2000\u2001\u2002\u2003\u2004\u2005\u2006\u2007\u2008\u2009\u200a\u202f\u205f\u3000"})
77
  # Country lengths checked against Apache Commons Validator's IBAN registry table:
78
  # https://commons.apache.org/proper/commons-validator/xref/org/apache/commons/validator/routines/IBANValidator.html
79
+ # Country/length/MOD-97 only; add national BBAN rules if false positives appear.
80
  _IBAN_LENGTHS = {
81
  country: length for length, countries in {
82
  15: "NO", 16: "BE", 18: "DK FI AX FK FO GL NL SD",
 
185
  r"(?:[ \t]+\([^()\n]{1,60}\))?[ \t]*:[ \t]*)"
186
  r"(?:[1-9]\d{2}[ \t]*(?:[,;]|i|lub|oraz)[ \t]*)*\Z", re.I)
187
  _OTHER_NUMBER_LABEL_RE = re.compile(r"\b(?:NIP|REGON|PESEL|KRS|ISBN|kod)\b[^\d\n]{0,20}\Z", re.I)
188
+ # Grouped national phones need no cue: web contact blocks write them bare. Plain 9-digit strings stay cue-gated,
189
+ # since many unlabelled ones are not phones.
190
+ _MOBILE_PREFIX = "45|50|51|53|57|60|66|69|72|73|78|79|88"
191
+ _AREA_CODE = ("12|13|14|15|16|17|18|22|23|24|25|29|32|33|34|41|42|43|44|46|48|52|54|55|56|58|59|61|62|63|65|67|68|"
192
+ "71|74|75|76|77|81|82|83|84|85|86|87|89|91|94|95")
193
+ _GROUPED_PHONE_RE = re.compile(
194
+ r"(?:\+48[ \t]?)?(?:(?:" + _MOBILE_PREFIX + r")\d(?P<m>[ -])\d{3}(?P=m)\d{3}"
195
+ r"|(?:\((?:" + _AREA_CODE + r")\)|(?:" + _AREA_CODE + r"))(?P<l>[ -])\d{3}(?P=l)\d{2}(?P=l)\d{2})")
196
  _PHONE_LIST_HEADING_RE = re.compile(
197
  r"[ \t]*(?:numery[ \t]+kontaktowe|telefony(?:[ \t]+(?:kontaktowe|zaufania))?|fax|faks|zapisz[ \t]+się|"
198
  r"(?:[^.!?\n]{0,40}\?[ \t]*)?(?:za)?dzwoń!?)"
 
213
 
214
 
215
  def _phone_list_context(text: str, start: int) -> bool:
216
+ # 2,000-character lookback and 400-character entry suffix; longer
217
  # lists need repeated headings or a section parser. Never cross prose.
218
  end = text.find('\n', start, start + 401)
219
  if end < 0 and len(text) > start + 400:
 
294
  foreign = raw.startswith('+') and _phone_ok(raw)
295
  if not foreign and (not _pl_national_ok(_digits(raw)) or not re.search(r'[ \t-]', raw)):
296
  return False
297
+ # Local 1,600-character directory evidence; longer isolated rows
298
  # need preserved upstream table structure, not an unbounded document cue.
299
  left, right = max(0, start-800), min(len(text), end+800)
300
  nearby = text[left:right]
 
478
  return _LAND_REGISTER_RE.sub(replace, text), n
479
 
480
 
481
+ def _head(raw: str) -> str:
482
+ # The greedy match may run on over a newline or a slash; the break and pair logic in _replace_phones cut it there.
483
+ return re.split(r"[ \t]*[\n/]", raw, maxsplit=1)[0]
484
+
485
+
486
+ def _national_head(raw: str) -> bool:
487
+ return _pl_national_ok(re.sub(r"^(?:48|0)(?=\d{9}$)", "", _digits(_head(raw))))
488
+
489
+
490
+ def _grouped_phone(text: str, start: int, raw: str) -> bool:
491
+ # Digits glued before the number on its line ("48 601 234 567", list indexes) are not tried.
492
+ head = _head(raw)
493
+ return bool(_GROUPED_PHONE_RE.fullmatch(head)
494
+ and not re.search(r"[ -]000[ -]000\Z", head) # round counts: 500 000 000
495
+ and not _is_amount(text, start, start + len(head))
496
+ and not re.match(r",\d{1,2}(?!\d)", text[start + len(head):]) # decimal figures: 601 234 567,89
497
+ and not _OTHER_NUMBER_LABEL_RE.search(text[max(0, start - 75):start]))
498
+
499
+
500
  def _pl_national_ok(d: str) -> bool:
501
  return len(d) == 9 and d.isdigit() and d[:2] in _PL_PREFIX
502
 
 
527
  return True
528
  if short and s.startswith('(0)'):
529
  return 7 <= len(d) <= 12
530
+ # Label + shape + length, not a country numbering-plan validator.
531
  if short and re.match(r'^\(0?\d{2,4}(?:-\d{1,2})?\)', s):
532
  return 8 <= len(d) <= 15
533
  # EUR-Lex puts country and area code in parentheses: "(32-2) 299 11 11" is +32 2 299 11 11.
 
537
  if s.startswith('00'):
538
  international = international[2:]
539
  if not international.startswith('48'):
540
+ # Explicit prefix + length, not a global numbering-plan
541
  # validator. Add country metadata if review finds false positives.
542
  return (7 if '(0)' in s else 8) <= len(international) <= 15 and international[0] != '0'
543
  if d.startswith("00"):
 
626
  pos = 0
627
  last_phone_end = None
628
  last_phone_complete = False
629
+ last_phone_cued = False # after a bare grouped phone a list goes on only with complete national numbers
630
  while True:
631
  m = _PHONE_RE.search(text, pos)
632
  if not m:
 
638
  line_end = text.find('\n', m.start(), end)
639
  if line_end >= 0 and re.fullmatch(r'\d{1,3}', text[m.start():line_end]):
640
  next_end = text.find('\n', line_end+1, end)
641
+ next_end = next_end if next_end >= 0 else end
642
+ if (_directory_context(text, line_end+1, next_end)
643
+ or _grouped_phone(text, line_end+1, text[line_end+1:next_end])):
644
+ # A flattened table's room cell or an address's house number is not a phone country prefix.
645
  out.append(text[pos:line_end+1])
646
  pos = line_end+1
647
  continue
 
669
  # "tel. +48 609 \n953 709", or the next local number, "(30) 25 10 22 33 25,\n22 33 28".
670
  cell = (last_phone_complete and 5 <= len(_digits(raw)) <= 6
671
  and re.fullmatch(r'[ \t]*\n[ \t\n]*', text[last_phone_end:m.start()]))
672
+ continues = last_phone_cued or (last_phone_end is not None and _national_head(raw))
673
+ if (continues and not postal and not cell
674
  and re.fullmatch(r"[ \t\n,;]{1,12}|[ \t]+(?:lub|albo)[ \t]+", text[last_phone_end:m.start()], re.I)):
675
  short = True
676
+ if (continues
677
  and re.fullmatch(r'(?:[ \t]+(?:w sprawie\b|\((?:w godzinach|dostępny)\b)'
678
  r'|[ \t]*[–—-][ \t]+)[^\d\n.!?]{1,200}\n[ \t\n]{0,8}',
679
  text[last_phone_end:m.start()], re.I)
680
  and _pl_national_ok(_digits(raw))):
681
  short = True
682
+ cued = (short or _phone_context(text, m.start(), raw)
683
+ or _directory_context(text, m.start(), line_end if line_end >= 0 else end))
684
+ if novel_country_area_ok and (cued or _grouped_phone(text, m.start(), raw)):
685
  if (_is_amount(text, m.start(), end)
686
  and re.fullmatch(r'\d{1,3}(?:\.[ \t]*\d{3})+', raw)):
687
  out.extend((text[pos:m.start()], raw))
 
734
  start += 1
735
  replacement = text[m.start():start] + mask(start, end, PHONE_TAG)
736
  n += 1
737
+ # Split only pairs (<=22 digits); longer lists need label
738
  # context to distinguish them from accounts. Never redact ID tails.
739
  elif 18 <= len(_digits(raw)) <= 22 and "." not in raw:
740
  # A slash separates alternatives, the second without the shared
 
750
  if replacement != raw:
751
  last_phone_end = end
752
  last_phone_complete = _phone_ok(raw)
753
+ last_phone_cued = bool(cued)
754
  pos = end
755
  return "".join(out), n
756
 
test_nergal.py CHANGED
@@ -5,7 +5,7 @@ import unittest
5
  from pathlib import Path
6
 
7
  HERE = Path(__file__).resolve().parent
8
- RULES_SHA = 'f32d5c5452fc47178e109d4bc248a0d8234ea6e59e8cf79407f4eb8451581d67'
9
 
10
 
11
  class NergalTests(unittest.TestCase):
@@ -13,7 +13,7 @@ class NergalTests(unittest.TestCase):
13
  from nergal import GAP_IDS, GAPS, HUB_ID, RULES_SHA as PINNED, THRESHOLD, VERSION
14
  card = json.loads((HERE / 'hybrid.json').read_text())
15
  self.assertEqual(HUB_ID, 'SlayerLab/NERGAL')
16
- self.assertEqual(VERSION, '1.1.0')
17
  self.assertEqual(card['version'], VERSION)
18
  self.assertEqual(card['eval']['union_fp'], 123)
19
  self.assertEqual(card['eval']['rules_fp'], 98)
@@ -62,6 +62,19 @@ class NergalTests(unittest.TestCase):
62
  self.assertEqual(text[span['start']:span['end']], '1234567802')
63
  self.assertEqual(rules('a [PII] b [Telefon] c'), [])
64
 
 
 
 
 
 
 
 
 
 
 
 
 
 
65
  def test_union_keeps_regex_and_adds_model_spans(self):
66
  from nergal import apply_union, scrub_spans
67
  text = 'Ring 000000000 then extra.'
 
5
  from pathlib import Path
6
 
7
  HERE = Path(__file__).resolve().parent
8
+ RULES_SHA = 'ad5c51f7c712dd2b7af40a337b703aeec211a57170a87c262bf9bd9386ba1925'
9
 
10
 
11
  class NergalTests(unittest.TestCase):
 
13
  from nergal import GAP_IDS, GAPS, HUB_ID, RULES_SHA as PINNED, THRESHOLD, VERSION
14
  card = json.loads((HERE / 'hybrid.json').read_text())
15
  self.assertEqual(HUB_ID, 'SlayerLab/NERGAL')
16
+ self.assertEqual(VERSION, '1.1.1')
17
  self.assertEqual(card['version'], VERSION)
18
  self.assertEqual(card['eval']['union_fp'], 123)
19
  self.assertEqual(card['eval']['rules_fp'], 98)
 
62
  self.assertEqual(text[span['start']:span['end']], '1234567802')
63
  self.assertEqual(rules('a [PII] b [Telefon] c'), [])
64
 
65
+ def test_grouped_national_phones_mask_without_a_cue(self):
66
+ from nergal import rules
67
+ for text, masked in (('Sklep Ala, 601 234 567, czynne 9-17', ['601 234 567']),
68
+ ('Biuro: (22) 123 45 67.', ['(22) 123 45 67']),
69
+ ('Zapraszamy: +48 601 234 567.', ['+48 601 234 567']),
70
+ ('Zapraszamy: 601234567.', []), # plain 9 digits stay cue-gated
71
+ ('Budżet wyniósł 601 234 567 zł.', []), # amount
72
+ ('Wartość 601 234 567,89 w tabeli.', []), # decimal figure
73
+ ('Kwota 500 000 000 osób.', []), # round count
74
+ ('NIP: 601 234 567', [])): # other-number label
75
+ with self.subTest(text=text):
76
+ self.assertEqual([text[s['start']:s['end']] for s in rules(text) if s['label'] == 'phone'], masked)
77
+
78
  def test_union_keeps_regex_and_adds_model_spans(self):
79
  from nergal import apply_union, scrub_spans
80
  text = 'Ring 000000000 then extra.'