README.md

---
license: apache-2.0
language:
- en
tags:
- token-classification
- named-entity-recognition
- pii
- phi
- redaction
- de-identification
- ner
- bert
- knowledge-distillation
- on-device
pipeline_tag: token-classification
---

# phi-redact — compact English PII/PHI redaction model

**phi-redact** is an 11M-parameter BERT-mini token classifier for detecting PII
and PHI in English text. It covers 32 entity types with BIOES labels and is
designed for on-device deployment (ONNX, Core ML, LiteRT) where data must not
leave the device.

It was knowledge-distilled from a 66M-parameter DistilBERT teacher and trained
on ~212k documents (v9 mix) spanning clinical notes, government benefits text,
student essays, general PII text, and Wikipedia-style prose.

> **Not a certified de-identification solution.** This model assists redaction
> workflows. It is **not** certified under HIPAA Expert Determination or Safe
> Harbor methods. Evaluate it on your own data before relying on it in
> compliance-sensitive contexts. English only.

---

## Current version: v9 (2026-10-21)

The repo root is **v9**. v8 and v7 remain available under versioned
names: `phi-redact-v8-int8.onnx`, `phi-redact-v8.onnx`,
`phi-redact-v8.mlpackage`, `phi-redact-v8-fp32.mlpackage`,
`phi-redact-v8-fp32.tflite`, `model-v8.safetensors`, `config-v8.json`,
`tags-v8.json` (and likewise for v7).

**What changed in v9** — a data-only revision; the architecture,
teacher, and training procedure are unchanged from v8.

- **Mined hard negatives.** The shipped v8 model scored a pool of
  54,380 gold-empty documents from the training corpora; the 1,616 it
  fired on (plus 15,000 clean documents) joined the mix as negatives.
- **Revised synthetic generator** (`gen_synth_v9.py`, 22k notes):
  segmented government-ID shapes boosted, 18% public-context /
  prose-address notes (geography rendered unlabeled in public
  contexts), 15% sparse low-density notes, and invalid-identifier
  decoys (impossible SSNs, Luhn-invalid cards, 555 numbers) appended
  to ~50% of notes.
- **Gravitee reduced** 25k → 15k (loosest span boundaries in the mix).
- **SDK default threshold raised 0.5 → 0.6** (phi-redact-sdk 0.3.0):
  on the golden set 0.6 dominates 0.5 — higher precision and strict F1
  at identical recall.

**Golden 164-case synthetic redaction eval** (rules + neural merge, the
production configuration; the `evals/synthetic-164/` set):

| Metric | v7 | v8 INT8 | v9 INT8 (shipped artifact) |
|---|---|---|---|
| Full-value recall | 0.885 | 0.958 | **0.974** |
| Term recall | 0.651 | 0.760 | **0.828** |
| Precision | 0.749 | 0.695 | **0.746** |
| Strict span F1 | 0.435 | 0.617 | **0.677** |
| Clean-case FP rate | 0.120 | 0.440 | **0.200** |

v9 figures are the shipped configuration: INT8 ONNX under the SDK
scorer (`score_phi.py`) at the SDK 0.3.0 default threshold of 0.6. The
PyTorch checkpoint scores 0.975 full-value recall under the Colab gate
harness (`evaluate_golden.py`) — same test set, two harnesses, small
scorer differences. INT8 quantization costs no recall here.

**The v8 tradeoff, revisited:** v8 bought recall with precision (clean
texts picked up a flagged span 44% of the time). v9 improves both axes
at once — recall 0.958 → 0.974 and precision 0.695 → 0.746, clean-case
FP rate halved to 0.200. What did *not* improve: public-context
retention. The model still redacts most geography that the golden set
marks as public/keep (retention 0.133, unchanged from v8) — facility
cities, state program names, and similar keep-context spans are now
about half of all remaining false positives. If your workflow is
precision-sensitive beyond this, v7 remains available above.

**Standard benchmarks, v9 student (model only, strict span scoring):**
Learning Agency micro-F1 0.729 (NAME F1 0.778); FewNERD name F1 0.781
(recall 0.838); MultiNERD name F1 0.893 (recall 0.940); WikiANN name F1
0.763 (recall 0.831); govsynth 1.000. No material regression vs v8
(WikiANN −0.011 is the largest move). The v7 tables below are unchanged
and remain the Desert Ant comparison baseline.

**Nemotron-PII, v9 (measured 2026-10-21):** same 2,000-document
structured eval set as below (13,134 gold spans, model only, exact span
match). v9 micro P 0.870 / R 0.830 / **F1 0.849** — identical F1 to v8
(0.849), itself a wash vs v7 (0.860) on this synthetic distribution;
v9's gains are on the clinical-note distribution above. Per entity, v9
improves SSN (0.957 vs v8 0.933), PROFESSION (0.517 vs 0.491) and
LICENSE (0.908 vs 0.885), and gives back a little on ADDRESS, DEVICE
and HEALTHPLAN.

---

## Model details

| Property | Value |
|---|---|
| Architecture | BERT-mini (google/bert_uncased_L-4_H-256_A-4), 4L/256H/4A |
| Parameters | 11.13M |
| Labels | 129 BIOES tags (32 entity types × 4 positions + O) |
| Max sequence length | 256 tokens |
| Vocabulary | 30,522 WordPiece (bert-base-uncased) |
| fp32 size | 44.6 MB |
| ONNX int8 size | 11.3 MB (`phi-redact-v9-int8.onnx`) |
| ONNX fp32 size | 44.6 MB (`phi-redact-v9.onnx`) |
| Core ML size | 22.2 MB fp16 (`phi-redact-v9-fp32.mlpackage`), 5.6 MB 4-bit palettized (`phi-redact-v9.mlpackage`) |
| LiteRT size | 44.5 MB fp32 (`phi-redact-v9-fp32.tflite`) |
| Training method | Knowledge distillation (α=0.7) from distilbert-base-uncased teacher |
| Language | English only |

---

## Entity types (32)

`NAME_PATIENT` · `NAME_FAMILY` · `NAME_PROVIDER` · `DATE` · `ADDRESS` ·
`PHONE` · `EMAIL` · `SSN` · `MRN` · `HEALTHPLAN` · `ACCOUNT` · `IDNUM` ·
`LICENSE` · `EMPLOYER` · `PROFESSION` · `AGE` · `CITY` · `STATE` · `COUNTRY` ·
`ZIP` · `URL` · `IP` · `DEVICE` · `ROUTING_NUMBER` · `IBAN` · `CREDITCARD` ·
`HOSPITAL` · `FAX` · `PASSPORT` · `VIN` · `TAX_ID` · `GOVERNMENT_ID`

> **v8 note:** `TAX_ID` and `GOVERNMENT_ID` were added in v8 (synthetic EIN /
> USCIS / A-number templates plus the Gravitee US_ITIN mapping).
> **v9 note:** segmented government-ID formats (e.g. Medicare-style
> `1EG4-TE5-MK73`) improved — golden GOVERNMENT_ID full-value recall
> 4/6 and the adversarial set went 0/2 → 1/2 — but GOVERNMENT_ID
> remains the weakest structured type.

---

## Benchmark results (v7)

Strict span-level scoring: a prediction counts only if both the character
boundaries and the label match exactly.

### Standard benchmarks

| Benchmark | Recall | F1 | Notes |
|---|---|---|---|
| WikiANN (EN, name recall) | **84.3%** | 0.775 | CC-BY-SA 3.0; direct comparison with Desert Ant ✓ |
| FewNERD (EN, name recall) | **85.9%** | 0.782 | CC-BY-SA 4.0; Wikipedia-prose names |
| LA essays (Micro F1 / Recall) | 0.757 / **65.9%** | — | CC-BY 4.0; informal student prose |
| govsynth (government benefits) | **100%** | 1.000 | Synthetic; primary deployment domain |
| **Composite (5-benchmark recall avg)** | **86.1%** | — | See note below |

Composite = (WikiANN + FewNERD + LA essays recall + govsynth + MultiNERD†) / 5.

†MultiNERD (CC-BY-NC-SA 4.0) is included for completeness: v7 recall = 94.3%,
within −0.6 pt of Desert Ant's 94.9% on the same dataset.

### Nemotron-PII structured benchmark (CC-BY 4.0)

First valid run — 13,134 entities across 25 entity types in 2,000 test
documents. MICRO F1 = **0.860**, Recall = 0.843.

Selected per-entity F1 scores:

| Entity | F1 | Entity | F1 |
|---|---|---|---|
| NAME_PATIENT | 0.944 | SSN | 0.908 |
| NAME_FAMILY | 0.937 | EMAIL | 0.905 |
| MRN | 0.951 | CREDITCARD | 0.905 |
| ADDRESS | 0.949 | IDNUM | 0.908 |
| LICENSE | 0.920 | DATE | 0.859 |
| COUNTRY | 0.919 | URL | 0.828 |
| CITY | 0.904 | DEVICE | 0.791 |
| ROUTING_NUMBER | 0.876 | **PROFESSION** | **0.494** |

**Known weak spot — PROFESSION (F1=0.494, Precision=0.413):** the model
over-fires on job-title nouns in non-PII context ("She saw a doctor" vs
"Occupation: Doctor"). Not a concern for most PHI/PII redaction use cases where
PROFESSION is rarely the sensitive entity, but flag it if your data is
occupation-heavy.

**URL in informal prose:** Nemotron URL F1=0.828 (formal text) but LA essays
URL recall=0.318 (informal prose). The model handles URLs well in structured
documents; it misses them in casual first-person writing.

### Comparison to Desert Ant Redact v0.4.0 (English)

| Metric | phi-redact v7 | Desert Ant (EN) |
|---|---|---|
| WikiANN name recall | **84.3%** | 69.5% |
| MultiNERD name recall | 94.3% | **94.9%** |
| Composite (5 benchmarks) | **86.1%** | **86.5%** |
| Parameters | **11.13M** | 23M |
| Est. 4-bit on-device size | **~5.6 MB** | 11.6 MB |
| Languages | English only | 27 languages |
| License | **Apache-2.0** | Source-available |

phi-redact v7 is essentially tied with Desert Ant's English composite at half
the parameters and half the on-device footprint.

### Inference latency

Measured 2026-10-03 against the live demo (corytrimm.com/phi-redact): the
same 15-template typing-length corpus as `evals/latency-bench/bench.py`,
15 templates × 10 cycles = 150 redactions, timed from the page's own
per-redaction readout — TypeScript rules scan + ONNX detect + merge +
placeholders, end to end, onnxruntime-web WASM (Chromium on Linux).

| Input | p50 | p95 | p99 |
|---|---|---|---|
| Chat-length (150 samples) | **6.0 ms** | 8.0 ms | 9.0 ms |
| Long text (3,959 chars, 139 entities) | 572 ms | — | — |

Model load: 0.5 s cached, 4.4 s cold (11.3 MB int8 ONNX + runtime download).

Caveats: the model was cached for this run; browser version and hardware
details were not obtainable from the measurement setup (Linux VM, not a
phone); readout resolution is whole milliseconds. Not a head-to-head with
any vendor's published number — different hardware, corpus, and setup.
Methodology travels with the number; see `evals/latency-bench/`.

### Synthetic redaction eval (164 cases)

`evals/synthetic-164/` carries a 164-case synthetic redaction test set, the
scoring harness, and v7's results: **0.812 full-value recall** (term recall
0.615, precision 0.698, clean-case FP rate 0.120). Perfect 1.000 on
BANK_ACCOUNT, CREDIT_CARD, DRIVERS_LICENSE, EMAIL, PHONE, ROUTING_NUMBER;
weak on NAME (0.347) and ADDRESS (0.333). See the directory README for
methodology and the taxonomy note. (Re-measured 2026-10-19 with the shipped
SDK — fixed marker ids and the current rules — v7 scores 0.885 full-value
recall on this set; the 0.812 above is the original 2026-10-02 measurement.)

---

## Training data

All sources are commercial-safe (CC-BY 4.0 or CC-BY-SA 4.0). No real patient
data was used.

| Source | Size | License | Entity coverage |
|---|---|---|---|
| Synthetic notes, v9 generator (gen_synth_v9.py; extends gen_synth + v8 generator) | 22k | Synthetic (ours) | Full 32-type schema; public-context geo, sparse notes, invalid-ID decoys |
| Learning Agency Lab PII (Kaggle) | ~5,400 | CC-BY 4.0 | Names, emails, URLs in student essays |
| govsynth (synthetic govt. benefits) | ~20k | Synthetic (ours) | SNAP/Medicaid/WIC case management PII |
| essay_names.jsonl (synthetic prose) | 10k | Synthetic (ours) | Names in biography, news, legal, informal prose |
| FewNERD train split (DFKI-SLT/few-nerd) | 50k | CC-BY-SA 4.0 | Wikipedia person entities |
| WikiANN train split (EN) | ~20k | CC-BY-SA 3.0 | Wikipedia person, location, org |
| WNUT-17 train split | ~3.4k | CC-BY 4.0 | Emerging entities in social media |
| Nemotron-PII train split (nvidia) | 50k | CC-BY 4.0 | 30 PII/PHI types in synthetic documents |
| Mined negatives (v8-model-mined, rules-filtered) | 16,616 | Derived from corpora above | Gold-empty documents; 1,616 the v8 model fired on + 15,000 clean |
| Gravitee PII (gravitee-io/pii-detection-dataset) | 15k sampled of 175k (25k in v8) | Apache-2.0 | PERSON, LOCATION, US identifier types |

**Excluded:** ai4privacy/pii-masking-400k (non-commercial license — not used in
training or evaluation). MultiNERD (CC-BY-NC-SA 4.0 — evaluation only, not in
training corpus).

---

## Training procedure

**Teacher** (distilbert-base-uncased, 66M params):
```
epochs 10, batch 32, lr 3e-5, cosine LR, label-smoothing 0.05, device A100
```

**Student** (this model — BERT-mini, 11.13M params):
```
epochs 60, batch 64, lr 5e-5, cosine LR, label-smoothing 0.05,
distill-alpha 0.7, device A100 SXM4-40GB (~6h)
```

Knowledge distillation loss = 0.7 × KL(teacher logits ∥ student logits) + 0.3 × cross-entropy.

v8 (2026-10) used the same procedure on the expanded 205k-document mix.
v9 (2026-10) used the same procedure on the revised ~212k-document mix (data-only changes; see the v9 section above).

---

## How to use

```python
from transformers import AutoTokenizer, AutoModelForTokenClassification
import torch

tokenizer = AutoTokenizer.from_pretrained("CDT5058/phi-redact")
model = AutoModelForTokenClassification.from_pretrained("CDT5058/phi-redact")
model.eval()

text = "Patient Maria Garcia, DOB 1985-03-22, MRN 4471829, called 415-555-0189."

enc = tokenizer(text, return_tensors="pt", truncation=True, max_length=256)
with torch.no_grad():
    logits = model(**enc).logits          # [1, seq, 129]

pred_ids = logits.argmax(-1)[0].tolist()  # [seq]
id2label = model.config.id2label

# Decode BIOES tags to character spans
tokens    = tokenizer.convert_ids_to_tokens(enc["input_ids"][0])
offsets   = enc.encodings[0].offsets      # (char_start, char_end) per token

spans = []
span_start, span_label = None, None
for i, tag_id in enumerate(pred_ids):
    tag = id2label[tag_id]
    prefix, *label_parts = tag.split("-") if "-" in tag else (tag, [])
    label = "-".join(label_parts)
    if prefix == "S":
        spans.append((offsets[i][0], offsets[i][1], label))
    elif prefix == "B":
        span_start, span_label = offsets[i][0], label
    elif prefix == "E" and span_start is not None:
        spans.append((span_start, offsets[i][1], span_label))
        span_start = None
    elif prefix in ("O", "B"):
        span_start = None  # reset on unexpected transition

for start, end, label in spans:
    print(f"[{start}:{end}] {label!r:20s}  {text[start:end]!r}")
```

For production use, layer the deterministic `rules.py` (Luhn, ABA routing
checksum, SSN, IBAN, regex patterns) on top — it wins on structured identifiers
regardless of model confidence.

---

## Limitations

- **English only.** The bert-base-uncased WordPiece vocabulary has no meaningful
  cross-lingual representations. French, Spanish, or other languages will produce
  silent misses — not safe degradation.
- **PROFESSION over-fires** (F1=0.517 on Nemotron for v9; v7 was 0.494). Treat PROFESSION predictions
  with lower confidence or disable the entity type if not needed.
- **URL in informal prose** (recall=0.318 on student essays vs 0.828 on formal
  text). Check URL recall on your specific document type.
- **Over-redaction on clean text.** Clean-case FP rate 0.200 on the
  synthetic golden set at the default threshold (v8: 0.440; v7: 0.120).
  The remaining mass is public-context geography the golden set marks
  as keep (retention 0.133) — facility cities and state names in
  referral/program text. Precision-sensitive workflows can raise the
  neural threshold or use v7.
- **Trained on synthetic data.** Real-world typos, OCR noise, and unusual
  formatting will degrade performance relative to the benchmarks above.
- **Not a certified de-identification system.** Does not satisfy HIPAA Expert
  Determination or Safe Harbor. Use as a detection aid alongside human review.

---

## License

Apache-2.0. Training data licenses: CC-BY 4.0 (Nemotron-PII, LA essays,
WNUT-17), CC-BY-SA 4.0 (FewNERD), CC-BY-SA 3.0 (WikiANN). CC-BY-SA ShareAlike
requirements apply to the datasets, not the model weights, but consult your
legal counsel for commercial deployment.

---

*Card generated 2026-09-25 · Updated 2026-10-21 for v9 · Repo: CDT5058/phi-redact*