license: apache-2.0
language:
- en
tags:
- token-classification
- named-entity-recognition
- pii
- phi
- redaction
- de-identification
- ner
- bert
- knowledge-distillation
- on-device
pipeline_tag: token-classification
phi-redact — compact English PII/PHI redaction model
phi-redact is an 11M-parameter BERT-mini token classifier for detecting PII and PHI in English text. It covers 32 entity types with BIOES labels and is designed for on-device deployment (ONNX, Core ML, LiteRT) where data must not leave the device.
It was knowledge-distilled from a 66M-parameter DistilBERT teacher and trained on ~212k documents (v9 mix) spanning clinical notes, government benefits text, student essays, general PII text, and Wikipedia-style prose.
Not a certified de-identification solution. This model assists redaction workflows. It is not certified under HIPAA Expert Determination or Safe Harbor methods. Evaluate it on your own data before relying on it in compliance-sensitive contexts. English only.
Current version: v9 (2026-10-21)
The repo root is v9. v8 and v7 remain available under versioned
names: phi-redact-v8-int8.onnx, phi-redact-v8.onnx,
phi-redact-v8.mlpackage, phi-redact-v8-fp32.mlpackage,
phi-redact-v8-fp32.tflite, model-v8.safetensors, config-v8.json,
tags-v8.json (and likewise for v7).
What changed in v9 — a data-only revision; the architecture, teacher, and training procedure are unchanged from v8.
- Mined hard negatives. The shipped v8 model scored a pool of 54,380 gold-empty documents from the training corpora; the 1,616 it fired on (plus 15,000 clean documents) joined the mix as negatives.
- Revised synthetic generator (
gen_synth_v9.py, 22k notes): segmented government-ID shapes boosted, 18% public-context / prose-address notes (geography rendered unlabeled in public contexts), 15% sparse low-density notes, and invalid-identifier decoys (impossible SSNs, Luhn-invalid cards, 555 numbers) appended to ~50% of notes. - Gravitee reduced 25k → 15k (loosest span boundaries in the mix).
- SDK default threshold raised 0.5 → 0.6 (phi-redact-sdk 0.3.0): on the golden set 0.6 dominates 0.5 — higher precision and strict F1 at identical recall.
Golden 164-case synthetic redaction eval (rules + neural merge, the
production configuration; the evals/synthetic-164/ set):
| Metric | v7 | v8 INT8 | v9 INT8 (shipped artifact) |
|---|---|---|---|
| Full-value recall | 0.885 | 0.958 | 0.974 |
| Term recall | 0.651 | 0.760 | 0.828 |
| Precision | 0.749 | 0.695 | 0.746 |
| Strict span F1 | 0.435 | 0.617 | 0.677 |
| Clean-case FP rate | 0.120 | 0.440 | 0.200 |
v9 figures are the shipped configuration: INT8 ONNX under the SDK
scorer (score_phi.py) at the SDK 0.3.0 default threshold of 0.6. The
PyTorch checkpoint scores 0.975 full-value recall under the Colab gate
harness (evaluate_golden.py) — same test set, two harnesses, small
scorer differences. INT8 quantization costs no recall here.
The v8 tradeoff, revisited: v8 bought recall with precision (clean texts picked up a flagged span 44% of the time). v9 improves both axes at once — recall 0.958 → 0.974 and precision 0.695 → 0.746, clean-case FP rate halved to 0.200. What did not improve: public-context retention. The model still redacts most geography that the golden set marks as public/keep (retention 0.133, unchanged from v8) — facility cities, state program names, and similar keep-context spans are now about half of all remaining false positives. If your workflow is precision-sensitive beyond this, v7 remains available above.
Standard benchmarks, v9 student (model only, strict span scoring): Learning Agency micro-F1 0.729 (NAME F1 0.778); FewNERD name F1 0.781 (recall 0.838); MultiNERD name F1 0.893 (recall 0.940); WikiANN name F1 0.763 (recall 0.831); govsynth 1.000. No material regression vs v8 (WikiANN −0.011 is the largest move). The v7 tables below are unchanged and remain the Desert Ant comparison baseline.
Nemotron-PII, v9 (measured 2026-10-21): same 2,000-document structured eval set as below (13,134 gold spans, model only, exact span match). v9 micro P 0.870 / R 0.830 / F1 0.849 — identical F1 to v8 (0.849), itself a wash vs v7 (0.860) on this synthetic distribution; v9's gains are on the clinical-note distribution above. Per entity, v9 improves SSN (0.957 vs v8 0.933), PROFESSION (0.517 vs 0.491) and LICENSE (0.908 vs 0.885), and gives back a little on ADDRESS, DEVICE and HEALTHPLAN.
Model details
| Property | Value |
|---|---|
| Architecture | BERT-mini (google/bert_uncased_L-4_H-256_A-4), 4L/256H/4A |
| Parameters | 11.13M |
| Labels | 129 BIOES tags (32 entity types × 4 positions + O) |
| Max sequence length | 256 tokens |
| Vocabulary | 30,522 WordPiece (bert-base-uncased) |
| fp32 size | 44.6 MB |
| ONNX int8 size | 11.3 MB (phi-redact-v9-int8.onnx) |
| ONNX fp32 size | 44.6 MB (phi-redact-v9.onnx) |
| Core ML size | 22.2 MB fp16 (phi-redact-v9-fp32.mlpackage), 5.6 MB 4-bit palettized (phi-redact-v9.mlpackage) |
| LiteRT size | 44.5 MB fp32 (phi-redact-v9-fp32.tflite) |
| Training method | Knowledge distillation (α=0.7) from distilbert-base-uncased teacher |
| Language | English only |
Entity types (32)
NAME_PATIENT · NAME_FAMILY · NAME_PROVIDER · DATE · ADDRESS ·
PHONE · EMAIL · SSN · MRN · HEALTHPLAN · ACCOUNT · IDNUM ·
LICENSE · EMPLOYER · PROFESSION · AGE · CITY · STATE · COUNTRY ·
ZIP · URL · IP · DEVICE · ROUTING_NUMBER · IBAN · CREDITCARD ·
HOSPITAL · FAX · PASSPORT · VIN · TAX_ID · GOVERNMENT_ID
v8 note:
TAX_IDandGOVERNMENT_IDwere added in v8 (synthetic EIN / USCIS / A-number templates plus the Gravitee US_ITIN mapping). v9 note: segmented government-ID formats (e.g. Medicare-style1EG4-TE5-MK73) improved — golden GOVERNMENT_ID full-value recall 4/6 and the adversarial set went 0/2 → 1/2 — but GOVERNMENT_ID remains the weakest structured type.
Benchmark results (v7)
Strict span-level scoring: a prediction counts only if both the character boundaries and the label match exactly.
Standard benchmarks
| Benchmark | Recall | F1 | Notes |
|---|---|---|---|
| WikiANN (EN, name recall) | 84.3% | 0.775 | CC-BY-SA 3.0; direct comparison with Desert Ant ✓ |
| FewNERD (EN, name recall) | 85.9% | 0.782 | CC-BY-SA 4.0; Wikipedia-prose names |
| LA essays (Micro F1 / Recall) | 0.757 / 65.9% | — | CC-BY 4.0; informal student prose |
| govsynth (government benefits) | 100% | 1.000 | Synthetic; primary deployment domain |
| Composite (5-benchmark recall avg) | 86.1% | — | See note below |
Composite = (WikiANN + FewNERD + LA essays recall + govsynth + MultiNERD†) / 5.
†MultiNERD (CC-BY-NC-SA 4.0) is included for completeness: v7 recall = 94.3%, within −0.6 pt of Desert Ant's 94.9% on the same dataset.
Nemotron-PII structured benchmark (CC-BY 4.0)
First valid run — 13,134 entities across 25 entity types in 2,000 test documents. MICRO F1 = 0.860, Recall = 0.843.
Selected per-entity F1 scores:
| Entity | F1 | Entity | F1 |
|---|---|---|---|
| NAME_PATIENT | 0.944 | SSN | 0.908 |
| NAME_FAMILY | 0.937 | 0.905 | |
| MRN | 0.951 | CREDITCARD | 0.905 |
| ADDRESS | 0.949 | IDNUM | 0.908 |
| LICENSE | 0.920 | DATE | 0.859 |
| COUNTRY | 0.919 | URL | 0.828 |
| CITY | 0.904 | DEVICE | 0.791 |
| ROUTING_NUMBER | 0.876 | PROFESSION | 0.494 |
Known weak spot — PROFESSION (F1=0.494, Precision=0.413): the model over-fires on job-title nouns in non-PII context ("She saw a doctor" vs "Occupation: Doctor"). Not a concern for most PHI/PII redaction use cases where PROFESSION is rarely the sensitive entity, but flag it if your data is occupation-heavy.
URL in informal prose: Nemotron URL F1=0.828 (formal text) but LA essays URL recall=0.318 (informal prose). The model handles URLs well in structured documents; it misses them in casual first-person writing.
Comparison to Desert Ant Redact v0.4.0 (English)
| Metric | phi-redact v7 | Desert Ant (EN) |
|---|---|---|
| WikiANN name recall | 84.3% | 69.5% |
| MultiNERD name recall | 94.3% | 94.9% |
| Composite (5 benchmarks) | 86.1% | 86.5% |
| Parameters | 11.13M | 23M |
| Est. 4-bit on-device size | ~5.6 MB | 11.6 MB |
| Languages | English only | 27 languages |
| License | Apache-2.0 | Source-available |
phi-redact v7 is essentially tied with Desert Ant's English composite at half the parameters and half the on-device footprint.
Inference latency
Measured 2026-10-03 against the live demo (corytrimm.com/phi-redact): the
same 15-template typing-length corpus as evals/latency-bench/bench.py,
15 templates × 10 cycles = 150 redactions, timed from the page's own
per-redaction readout — TypeScript rules scan + ONNX detect + merge +
placeholders, end to end, onnxruntime-web WASM (Chromium on Linux).
| Input | p50 | p95 | p99 |
|---|---|---|---|
| Chat-length (150 samples) | 6.0 ms | 8.0 ms | 9.0 ms |
| Long text (3,959 chars, 139 entities) | 572 ms | — | — |
Model load: 0.5 s cached, 4.4 s cold (11.3 MB int8 ONNX + runtime download).
Caveats: the model was cached for this run; browser version and hardware
details were not obtainable from the measurement setup (Linux VM, not a
phone); readout resolution is whole milliseconds. Not a head-to-head with
any vendor's published number — different hardware, corpus, and setup.
Methodology travels with the number; see evals/latency-bench/.
Synthetic redaction eval (164 cases)
evals/synthetic-164/ carries a 164-case synthetic redaction test set, the
scoring harness, and v7's results: 0.812 full-value recall (term recall
0.615, precision 0.698, clean-case FP rate 0.120). Perfect 1.000 on
BANK_ACCOUNT, CREDIT_CARD, DRIVERS_LICENSE, EMAIL, PHONE, ROUTING_NUMBER;
weak on NAME (0.347) and ADDRESS (0.333). See the directory README for
methodology and the taxonomy note. (Re-measured 2026-10-19 with the shipped
SDK — fixed marker ids and the current rules — v7 scores 0.885 full-value
recall on this set; the 0.812 above is the original 2026-10-02 measurement.)
Training data
All sources are commercial-safe (CC-BY 4.0 or CC-BY-SA 4.0). No real patient data was used.
| Source | Size | License | Entity coverage |
|---|---|---|---|
| Synthetic notes, v9 generator (gen_synth_v9.py; extends gen_synth + v8 generator) | 22k | Synthetic (ours) | Full 32-type schema; public-context geo, sparse notes, invalid-ID decoys |
| Learning Agency Lab PII (Kaggle) | ~5,400 | CC-BY 4.0 | Names, emails, URLs in student essays |
| govsynth (synthetic govt. benefits) | ~20k | Synthetic (ours) | SNAP/Medicaid/WIC case management PII |
| essay_names.jsonl (synthetic prose) | 10k | Synthetic (ours) | Names in biography, news, legal, informal prose |
| FewNERD train split (DFKI-SLT/few-nerd) | 50k | CC-BY-SA 4.0 | Wikipedia person entities |
| WikiANN train split (EN) | ~20k | CC-BY-SA 3.0 | Wikipedia person, location, org |
| WNUT-17 train split | ~3.4k | CC-BY 4.0 | Emerging entities in social media |
| Nemotron-PII train split (nvidia) | 50k | CC-BY 4.0 | 30 PII/PHI types in synthetic documents |
| Mined negatives (v8-model-mined, rules-filtered) | 16,616 | Derived from corpora above | Gold-empty documents; 1,616 the v8 model fired on + 15,000 clean |
| Gravitee PII (gravitee-io/pii-detection-dataset) | 15k sampled of 175k (25k in v8) | Apache-2.0 | PERSON, LOCATION, US identifier types |
Excluded: ai4privacy/pii-masking-400k (non-commercial license — not used in training or evaluation). MultiNERD (CC-BY-NC-SA 4.0 — evaluation only, not in training corpus).
Training procedure
Teacher (distilbert-base-uncased, 66M params):
epochs 10, batch 32, lr 3e-5, cosine LR, label-smoothing 0.05, device A100
Student (this model — BERT-mini, 11.13M params):
epochs 60, batch 64, lr 5e-5, cosine LR, label-smoothing 0.05,
distill-alpha 0.7, device A100 SXM4-40GB (~6h)
Knowledge distillation loss = 0.7 × KL(teacher logits ∥ student logits) + 0.3 × cross-entropy.
v8 (2026-10) used the same procedure on the expanded 205k-document mix. v9 (2026-10) used the same procedure on the revised ~212k-document mix (data-only changes; see the v9 section above).
How to use
from transformers import AutoTokenizer, AutoModelForTokenClassification
import torch
tokenizer = AutoTokenizer.from_pretrained("CDT5058/phi-redact")
model = AutoModelForTokenClassification.from_pretrained("CDT5058/phi-redact")
model.eval()
text = "Patient Maria Garcia, DOB 1985-03-22, MRN 4471829, called 415-555-0189."
enc = tokenizer(text, return_tensors="pt", truncation=True, max_length=256)
with torch.no_grad():
logits = model(**enc).logits # [1, seq, 129]
pred_ids = logits.argmax(-1)[0].tolist() # [seq]
id2label = model.config.id2label
# Decode BIOES tags to character spans
tokens = tokenizer.convert_ids_to_tokens(enc["input_ids"][0])
offsets = enc.encodings[0].offsets # (char_start, char_end) per token
spans = []
span_start, span_label = None, None
for i, tag_id in enumerate(pred_ids):
tag = id2label[tag_id]
prefix, *label_parts = tag.split("-") if "-" in tag else (tag, [])
label = "-".join(label_parts)
if prefix == "S":
spans.append((offsets[i][0], offsets[i][1], label))
elif prefix == "B":
span_start, span_label = offsets[i][0], label
elif prefix == "E" and span_start is not None:
spans.append((span_start, offsets[i][1], span_label))
span_start = None
elif prefix in ("O", "B"):
span_start = None # reset on unexpected transition
for start, end, label in spans:
print(f"[{start}:{end}] {label!r:20s} {text[start:end]!r}")
For production use, layer the deterministic rules.py (Luhn, ABA routing
checksum, SSN, IBAN, regex patterns) on top — it wins on structured identifiers
regardless of model confidence.
Limitations
- English only. The bert-base-uncased WordPiece vocabulary has no meaningful cross-lingual representations. French, Spanish, or other languages will produce silent misses — not safe degradation.
- PROFESSION over-fires (F1=0.517 on Nemotron for v9; v7 was 0.494). Treat PROFESSION predictions with lower confidence or disable the entity type if not needed.
- URL in informal prose (recall=0.318 on student essays vs 0.828 on formal text). Check URL recall on your specific document type.
- Over-redaction on clean text. Clean-case FP rate 0.200 on the synthetic golden set at the default threshold (v8: 0.440; v7: 0.120). The remaining mass is public-context geography the golden set marks as keep (retention 0.133) — facility cities and state names in referral/program text. Precision-sensitive workflows can raise the neural threshold or use v7.
- Trained on synthetic data. Real-world typos, OCR noise, and unusual formatting will degrade performance relative to the benchmarks above.
- Not a certified de-identification system. Does not satisfy HIPAA Expert Determination or Safe Harbor. Use as a detection aid alongside human review.
License
Apache-2.0. Training data licenses: CC-BY 4.0 (Nemotron-PII, LA essays, WNUT-17), CC-BY-SA 4.0 (FewNERD), CC-BY-SA 3.0 (WikiANN). CC-BY-SA ShareAlike requirements apply to the datasets, not the model weights, but consult your legal counsel for commercial deployment.
Card generated 2026-09-25 · Updated 2026-10-21 for v9 · Repo: CDT5058/phi-redact