model card: v8 results (golden 0.955/0.958 full-value recall, precision tradeoff noted)

README.md

@@ -19,13 +19,13 @@ pipeline_tag: token-classification
 # phi-redact — compact English PII/PHI redaction model
 
 **phi-redact** is an 11M-parameter BERT-mini token classifier for detecting PII
-and PHI in English text. It covers 29 entity types with BIOES labels and is
+and PHI in English text. It covers 32 entity types with BIOES labels and is
 designed for on-device deployment (ONNX, Core ML, LiteRT) where data must not
 leave the device.
 
 It was knowledge-distilled from a 66M-parameter DistilBERT teacher and trained
-on ~157k documents spanning clinical notes, government benefits text, student
-essays, and Wikipedia-style prose.
+on ~205k documents (v8 mix) spanning clinical notes, government benefits text,
+student essays, general PII text, and Wikipedia-style prose.
 
 > **Not a certified de-identification solution.** This model assists redaction
 > workflows. It is **not** certified under HIPAA Expert Determination or Safe
@@ -34,16 +34,71 @@ essays, and Wikipedia-style prose.
 
 ---
 
+## Current version: v8 (2026-10-20)
+
+The repo root is **v8**. v7 remains available under versioned names:
+`phi-redact-v7-int8.onnx`, `phi-redact-v7.onnx`, `phi-redact-v7.mlpackage`,
+`phi-redact-v7-fp32.mlpackage`, `phi-redact-v7.tflite`,
+`model-v7.safetensors`, `config-v7.json`, `tags-v7.json`.
+
+**What changed in v8**
+
+- Two new neural entity types: `GOVERNMENT_ID` and `TAX_ID` (129 BIOES tags,
+  32 entity types). The deterministic rules layer also gained EIN,
+  ABA-checksum routing, passport, bank-account, drivers-license, and
+  government-ID patterns.
+- Training mix rebuilt (~205k documents) around an extended synthetic
+  generator — conversational name introductions, address components as
+  spans, boosted structured-ID types — plus the Gravitee PII dataset
+  (Apache-2.0; PERSON→NAME_PATIENT, LOCATION→ADDRESS, US_* identifier
+  mappings). ai4privacy remains excluded (non-commercial license).
+- Fresh DistilBERT teacher → BERT-mini student distillation run; same
+  architecture and parameter count as v7.
+
+**Golden 164-case synthetic redaction eval** (rules + neural merge, the
+production configuration; the `evals/synthetic-164/` set):
+
+| Metric | v7 | v8 INT8 (shipped artifact) |
+|---|---|---|
+| Full-value recall | 0.885 | **0.958** |
+| Term recall | 0.651 | 0.760 |
+| Precision | 0.749 | 0.695 |
+| Strict span F1 | 0.435 | 0.617 |
+| Clean-case FP rate | 0.120 | 0.440 |
+
+The PyTorch checkpoint scores 0.955 full-value recall under the Colab gate
+harness (`evaluate_golden.py`); the INT8 ONNX scores 0.958 under the SDK
+scorer (`score_phi.py`) — same test set, two harnesses, small scorer
+differences. INT8 quantization costs no recall here.
+
+**The tradeoff, plainly:** v8 was optimized for recall — a missed
+identifier is a leak; an over-redaction is an annoyance. The cost is
+precision: on clean (no-PII) texts, 44% now pick up at least one flagged
+span vs 12% for v7, typically person/place-like tokens from the broader
+training mix. If your workflow is precision-sensitive, v7 remains
+available above, or raise the neural threshold (0.8 gives clean-case FP
+0.320 at 0.932 full-value recall on this set).
+
+**Standard benchmarks, v8 student (model only, strict span scoring):**
+Learning Agency micro-F1 0.711 (NAME F1 0.746); FewNERD name F1 0.787
+(recall 0.847); MultiNERD name F1 0.895 (recall 0.942); WikiANN name F1
+0.774 (recall 0.844); govsynth 1.000. The v7 tables below are unchanged
+and remain the Desert Ant comparison baseline; v8 has not been run on
+the Nemotron-PII benchmark yet.
+
+---
+
 ## Model details
 
 | Property | Value |
 |---|---|
 | Architecture | BERT-mini (google/bert_uncased_L-4_H-256_A-4), 4L/256H/4A |
 | Parameters | 11.13M |
-| Labels | 121 BIOES tags (30 entity types × 4 positions + O) |
+| Labels | 129 BIOES tags (32 entity types × 4 positions + O) |
 | Max sequence length | 256 tokens |
 | Vocabulary | 30,522 WordPiece (bert-base-uncased) |
-| fp32 size | 44.5 MB |
+| fp32 size | 44.6 MB |
+| ONNX int8 size | 11.3 MB (`phi-redact-v8-int8.onnx`) |
 | fp16 size | 22.3 MB |
 | Est. 4-bit Core ML | ~5.6 MB |
 | Est. int8 LiteRT | ~11.2 MB |
@@ -52,17 +107,18 @@ essays, and Wikipedia-style prose.
 
 ---
 
-## Entity types (30)
+## Entity types (32)
 
 `NAME_PATIENT` · `NAME_FAMILY` · `NAME_PROVIDER` · `DATE` · `ADDRESS` ·
 `PHONE` · `EMAIL` · `SSN` · `MRN` · `HEALTHPLAN` · `ACCOUNT` · `IDNUM` ·
 `LICENSE` · `EMPLOYER` · `PROFESSION` · `AGE` · `CITY` · `STATE` · `COUNTRY` ·
 `ZIP` · `URL` · `IP` · `DEVICE` · `ROUTING_NUMBER` · `IBAN` · `CREDITCARD` ·
-`HOSPITAL` · `FAX` · `PASSPORT` · `VIN`
+`HOSPITAL` · `FAX` · `PASSPORT` · `VIN` · `TAX_ID` · `GOVERNMENT_ID`
 
-> **Note on TAX_ID:** included in the schema design but not in any commercial-safe
-> training source for v7 (ai4privacy excluded). TAX_ID predictions will be absent.
-> Fix queued for v8 (add EIN templates to `gen_synth.py`).
+> **v8 note:** `TAX_ID` and `GOVERNMENT_ID` were added in v8 (synthetic EIN /
+> USCIS / A-number templates plus the Gravitee US_ITIN mapping). Dashed and
+> segmented government-ID formats (e.g. Medicare-style `1EG4-TE5-MK73`) are
+> still missed — a known gap.
 
 ---
 
@@ -157,7 +213,9 @@ scoring harness, and v7's results: **0.812 full-value recall** (term recall
 0.615, precision 0.698, clean-case FP rate 0.120). Perfect 1.000 on
 BANK_ACCOUNT, CREDIT_CARD, DRIVERS_LICENSE, EMAIL, PHONE, ROUTING_NUMBER;
 weak on NAME (0.347) and ADDRESS (0.333). See the directory README for
-methodology and the taxonomy note.
+methodology and the taxonomy note. (Re-measured 2026-10-19 with the shipped
+SDK — fixed marker ids and the current rules — v7 scores 0.885 full-value
+recall on this set; the 0.812 above is the original 2026-10-02 measurement.)
 
 ---
 
@@ -176,6 +234,8 @@ data was used.
 | WikiANN train split (EN) | ~20k | CC-BY-SA 3.0 | Wikipedia person, location, org |
 | WNUT-17 train split | ~3.4k | CC-BY 4.0 | Emerging entities in social media |
 | Nemotron-PII train split (nvidia) | 50k | CC-BY 4.0 | 30 PII/PHI types in synthetic documents |
+| Synthetic notes, v8 generator (gen_synth_v8.py) | 20k | Synthetic (ours) | Adds GOVERNMENT_ID, conversational name intros, address components |
+| Gravitee PII (gravitee-io/pii-detection-dataset) | 25k sampled of 175k | Apache-2.0 | PERSON, LOCATION, US identifier types |
 
 **Excluded:** ai4privacy/pii-masking-400k (non-commercial license — not used in
 training or evaluation). MultiNERD (CC-BY-NC-SA 4.0 — evaluation only, not in
@@ -198,6 +258,8 @@ distill-alpha 0.7, device A100 SXM4-40GB (~6h)
 
 Knowledge distillation loss = 0.7 × KL(teacher logits ∥ student logits) + 0.3 × cross-entropy.
 
+v8 (2026-10) used the same procedure on the expanded 205k-document mix.
+
 ---
 
 ## How to use
@@ -214,7 +276,7 @@ text = "Patient Maria Garcia, DOB 1985-03-22, MRN 4471829, called 415-555-0189."
 
 enc = tokenizer(text, return_tensors="pt", truncation=True, max_length=256)
 with torch.no_grad():
-    logits = model(**enc).logits          # [1, seq, 117]
+    logits = model(**enc).logits          # [1, seq, 129]
 
 pred_ids = logits.argmax(-1)[0].tolist()  # [seq]
 id2label = model.config.id2label
@@ -258,6 +320,9 @@ regardless of model confidence.
   with lower confidence or disable the entity type if not needed.
 - **URL in informal prose** (recall=0.318 on student essays vs 0.828 on formal
   text). Check URL recall on your specific document type.
+- **Over-redaction on clean text (v8).** Clean-case FP rate 0.440 on the
+  synthetic golden set (v7: 0.120). v8 trades precision for recall; see the
+  v8 section above before choosing it for precision-sensitive workflows.
 - **Trained on synthetic data.** Real-world typos, OCR noise, and unusual
   formatting will degrade performance relative to the benchmarks above.
 - **Not a certified de-identification system.** Does not satisfy HIPAA Expert
@@ -274,4 +339,4 @@ legal counsel for commercial deployment.
 
 ---
 
-*Card generated 2026-09-25 · Repo: CDT5058/phi-redact*
+*Card generated 2026-09-25 · Updated 2026-10-20 for v8 · Repo: CDT5058/phi-redact*