README.md · ctrimm/phi-redact at main

Cory Trimm

Model card: v9 (2026-10-21)

636b530 verified 8 days ago

15.8 kB

---
license: apache-2.0
language:
- en
tags:
- token-classification
- named-entity-recognition
- pii
- phi
- redaction
- de-identification
- ner
- bert
- knowledge-distillation
- on-device
pipeline_tag: token-classification
---
# phi-redact — compact English PII/PHI redaction model

**phi-redact** is an 11M-parameter BERT-mini token classifier for detecting PII
and PHI in English text. It covers 32 entity types with BIOES labels and is
designed for on-device deployment (ONNX, Core ML, LiteRT) where data must not
leave the device.

It was knowledge-distilled from a 66M-parameter DistilBERT teacher and trained
on ~212k documents (v9 mix) spanning clinical notes, government benefits text,
student essays, general PII text, and Wikipedia-style prose.

> **Not a certified de-identification solution.** This model assists redaction
> workflows. It is **not** certified under HIPAA Expert Determination or Safe
> Harbor methods. Evaluate it on your own data before relying on it in
> compliance-sensitive contexts. English only.

---

## Current version: v9 (2026-10-21)

The repo root is **v9**. v8 and v7 remain available under versioned
names: `phi-redact-v8-int8.onnx`, `phi-redact-v8.onnx`,
`phi-redact-v8.mlpackage`, `phi-redact-v8-fp32.mlpackage`,
`phi-redact-v8-fp32.tflite`, `model-v8.safetensors`, `config-v8.json`,
`tags-v8.json` (and likewise for v7).

**What changed in v9** — a data-only revision; the architecture,
teacher, and training procedure are unchanged from v8.

- **Mined hard negatives.** The shipped v8 model scored a pool of
  54,380 gold-empty documents from the training corpora; the 1,616 it
  fired on (plus 15,000 clean documents) joined the mix as negatives.
- **Revised synthetic generator** (`gen_synth_v9.py`, 22k notes):
  segmented government-ID shapes boosted, 18% public-context /
  prose-address notes (geography rendered unlabeled in public
  contexts), 15% sparse low-density notes, and invalid-identifier
  decoys (impossible SSNs, Luhn-invalid cards, 555 numbers) appended
  to ~50% of notes.
- **Gravitee reduced** 25k → 15k (loosest span boundaries in the mix).
- **SDK default threshold raised 0.5 → 0.6** (phi-redact-sdk 0.3.0):
  on the golden set 0.6 dominates 0.5 — higher precision and strict F1
  at identical recall.

**Golden 164-case synthetic redaction eval** (rules + neural merge, the
production configuration; the `evals/synthetic-164/` set):

| Metric | v7 | v8 INT8 | v9 INT8 (shipped artifact) |
|---|---|---|---|
| Full-value recall | 0.885 | 0.958 | **0.974** |
| Term recall | 0.651 | 0.760 | **0.828** |
| Precision | 0.749 | 0.695 | **0.746** |
| Strict span F1 | 0.435 | 0.617 | **0.677** |
| Clean-case FP rate | 0.120 | 0.440 | **0.200** |

v9 figures are the shipped configuration: INT8 ONNX under the SDK
scorer (`score_phi.py`) at the SDK 0.3.0 default threshold of 0.6. The
PyTorch checkpoint scores 0.975 full-value recall under the Colab gate
harness (`evaluate_golden.py`) — same test set, two harnesses, small
scorer differences. INT8 quantization costs no recall here.

**The v8 tradeoff, revisited:** v8 bought recall with precision (clean
texts picked up a flagged span 44% of the time). v9 improves both axes
at once — recall 0.958 → 0.974 and precision 0.695 → 0.746, clean-case
FP rate halved to 0.200. What did *not* improve: public-context
retention. The model still redacts most geography that the golden set
marks as public/keep (retention 0.133, unchanged from v8) — facility
cities, state program names, and similar keep-context spans are now
about half of all remaining false positives. If your workflow is
precision-sensitive beyond this, v7 remains available above.

**Standard benchmarks, v9 student (model only, strict span scoring):**
Learning Agency micro-F1 0.729 (NAME F1 0.778); FewNERD name F1 0.781
(recall 0.838); MultiNERD name F1 0.893 (recall 0.940); WikiANN name F1
0.763 (recall 0.831); govsynth 1.000. No material regression vs v8
(WikiANN −0.011 is the largest move). The v7 tables below are unchanged
and remain the Desert Ant comparison baseline.

**Nemotron-PII, v9 (measured 2026-10-21):** same 2,000-document
structured eval set as below (13,134 gold spans, model only, exact span
match). v9 micro P 0.870 / R 0.830 / **F1 0.849** — identical F1 to v8
(0.849), itself a wash vs v7 (0.860) on this synthetic distribution;
v9's gains are on the clinical-note distribution above. Per entity, v9
improves SSN (0.957 vs v8 0.933), PROFESSION (0.517 vs 0.491) and
LICENSE (0.908 vs 0.885), and gives back a little on ADDRESS, DEVICE
and HEALTHPLAN.

---

## Model details

| Property | Value |
|---|---|
| Architecture | BERT-mini (google/bert_uncased_L-4_H-256_A-4), 4L/256H/4A |
| Parameters | 11.13M |
| Labels | 129 BIOES tags (32 entity types × 4 positions + O) |
| Max sequence length | 256 tokens |
| Vocabulary | 30,522 WordPiece (bert-base-uncased) |
| fp32 size | 44.6 MB |
| ONNX int8 size | 11.3 MB (`phi-redact-v9-int8.onnx`) |
| ONNX fp32 size | 44.6 MB (`phi-redact-v9.onnx`) |
| Core ML size | 22.2 MB fp16 (`phi-redact-v9-fp32.mlpackage`), 5.6 MB 4-bit palettized (`phi-redact-v9.mlpackage`) |
| LiteRT size | 44.5 MB fp32 (`phi-redact-v9-fp32.tflite`) |
| Training method | Knowledge distillation (α=0.7) from distilbert-base-uncased teacher |
| Language | English only |

---

## Entity types (32)

`NAME_PATIENT` · `NAME_FAMILY` · `NAME_PROVIDER` · `DATE` · `ADDRESS` ·
`PHONE` · `EMAIL` · `SSN` · `MRN` · `HEALTHPLAN` · `ACCOUNT` · `IDNUM` ·
`LICENSE` · `EMPLOYER` · `PROFESSION` · `AGE` · `CITY` · `STATE` · `COUNTRY` ·
`ZIP` · `URL` · `IP` · `DEVICE` · `ROUTING_NUMBER` · `IBAN` · `CREDITCARD` ·
`HOSPITAL` · `FAX` · `PASSPORT` · `VIN` · `TAX_ID` · `GOVERNMENT_ID`

> **v8 note:** `TAX_ID` and `GOVERNMENT_ID` were added in v8 (synthetic EIN /
> USCIS / A-number templates plus the Gravitee US_ITIN mapping).
> **v9 note:** segmented government-ID formats (e.g. Medicare-style
> `1EG4-TE5-MK73`) improved — golden GOVERNMENT_ID full-value recall
> 4/6 and the adversarial set went 0/2 → 1/2 — but GOVERNMENT_ID
> remains the weakest structured type.
---
## Benchmark results (v7)
Strict span-level scoring: a prediction counts only if both the character
boundaries and the label match exactly.
### Standard benchmarks
| Benchmark | Recall | F1 | Notes |
|---|---|---|---|
| WikiANN (EN, name recall) | **84.3%** | 0.775 | CC-BY-SA 3.0; direct comparison with Desert Ant ✓ |
| FewNERD (EN, name recall) | **85.9%** | 0.782 | CC-BY-SA 4.0; Wikipedia-prose names |
| LA essays (Micro F1 / Recall) | 0.757 / **65.9%** | — | CC-BY 4.0; informal student prose |
| govsynth (government benefits) | **100%** | 1.000 | Synthetic; primary deployment domain |
| **Composite (5-benchmark recall avg)** | **86.1%** | — | See note below |
Composite = (WikiANN + FewNERD + LA essays recall + govsynth + MultiNERD†) / 5.
†MultiNERD (CC-BY-NC-SA 4.0) is included for completeness: v7 recall = 94.3%,
within −0.6 pt of Desert Ant's 94.9% on the same dataset.
### Nemotron-PII structured benchmark (CC-BY 4.0)
First valid run — 13,134 entities across 25 entity types in 2,000 test
documents. MICRO F1 = **0.860**, Recall = 0.843.
Selected per-entity F1 scores:
| Entity | F1 | Entity | F1 |
|---|---|---|---|
| NAME_PATIENT | 0.944 | SSN | 0.908 |
| NAME_FAMILY | 0.937 | EMAIL | 0.905 |
| MRN | 0.951 | CREDITCARD | 0.905 |
| ADDRESS | 0.949 | IDNUM | 0.908 |
| LICENSE | 0.920 | DATE | 0.859 |
| COUNTRY | 0.919 | URL | 0.828 |
| CITY | 0.904 | DEVICE | 0.791 |
| ROUTING_NUMBER | 0.876 | **PROFESSION** | **0.494** |

**Known weak spot — PROFESSION (F1=0.494, Precision=0.413):** the model
over-fires on job-title nouns in non-PII context ("She saw a doctor" vs
"Occupation: Doctor"). Not a concern for most PHI/PII redaction use cases where
PROFESSION is rarely the sensitive entity, but flag it if your data is
occupation-heavy.

**URL in informal prose:** Nemotron URL F1=0.828 (formal text) but LA essays
URL recall=0.318 (informal prose). The model handles URLs well in structured
documents; it misses them in casual first-person writing.

### Comparison to Desert Ant Redact v0.4.0 (English)

| Metric | phi-redact v7 | Desert Ant (EN) |
|---|---|---|
| WikiANN name recall | **84.3%** | 69.5% |
| MultiNERD name recall | 94.3% | **94.9%** |
| Composite (5 benchmarks) | **86.1%** | **86.5%** |
| Parameters | **11.13M** | 23M |
| Est. 4-bit on-device size | **~5.6 MB** | 11.6 MB |
| Languages | English only | 27 languages |
| License | **Apache-2.0** | Source-available |

phi-redact v7 is essentially tied with Desert Ant's English composite at half
the parameters and half the on-device footprint.

### Inference latency

Measured 2026-10-03 against the live demo (corytrimm.com/phi-redact): the
same 15-template typing-length corpus as `evals/latency-bench/bench.py`,
15 templates × 10 cycles = 150 redactions, timed from the page's own
per-redaction readout — TypeScript rules scan + ONNX detect + merge +
placeholders, end to end, onnxruntime-web WASM (Chromium on Linux).

| Input | p50 | p95 | p99 |
|---|---|---|---|
| Chat-length (150 samples) | **6.0 ms** | 8.0 ms | 9.0 ms |
| Long text (3,959 chars, 139 entities) | 572 ms | — | — |

Model load: 0.5 s cached, 4.4 s cold (11.3 MB int8 ONNX + runtime download).

Caveats: the model was cached for this run; browser version and hardware
details were not obtainable from the measurement setup (Linux VM, not a
phone); readout resolution is whole milliseconds. Not a head-to-head with
any vendor's published number — different hardware, corpus, and setup.
Methodology travels with the number; see `evals/latency-bench/`.

### Synthetic redaction eval (164 cases)

`evals/synthetic-164/` carries a 164-case synthetic redaction test set, the
scoring harness, and v7's results: **0.812 full-value recall** (term recall
0.615, precision 0.698, clean-case FP rate 0.120). Perfect 1.000 on
BANK_ACCOUNT, CREDIT_CARD, DRIVERS_LICENSE, EMAIL, PHONE, ROUTING_NUMBER;
weak on NAME (0.347) and ADDRESS (0.333). See the directory README for
methodology and the taxonomy note. (Re-measured 2026-10-19 with the shipped
SDK — fixed marker ids and the current rules — v7 scores 0.885 full-value
recall on this set; the 0.812 above is the original 2026-10-02 measurement.)

---

## Training data

All sources are commercial-safe (CC-BY 4.0 or CC-BY-SA 4.0). No real patient
data was used.

| Source | Size | License | Entity coverage |
|---|---|---|---|
| Synthetic notes, v9 generator (gen_synth_v9.py; extends gen_synth + v8 generator) | 22k | Synthetic (ours) | Full 32-type schema; public-context geo, sparse notes, invalid-ID decoys |
| Learning Agency Lab PII (Kaggle) | ~5,400 | CC-BY 4.0 | Names, emails, URLs in student essays |
| govsynth (synthetic govt. benefits) | ~20k | Synthetic (ours) | SNAP/Medicaid/WIC case management PII |
| essay_names.jsonl (synthetic prose) | 10k | Synthetic (ours) | Names in biography, news, legal, informal prose |
| FewNERD train split (DFKI-SLT/few-nerd) | 50k | CC-BY-SA 4.0 | Wikipedia person entities |
| WikiANN train split (EN) | ~20k | CC-BY-SA 3.0 | Wikipedia person, location, org |
| WNUT-17 train split | ~3.4k | CC-BY 4.0 | Emerging entities in social media |
| Nemotron-PII train split (nvidia) | 50k | CC-BY 4.0 | 30 PII/PHI types in synthetic documents |
| Mined negatives (v8-model-mined, rules-filtered) | 16,616 | Derived from corpora above | Gold-empty documents; 1,616 the v8 model fired on + 15,000 clean |
| Gravitee PII (gravitee-io/pii-detection-dataset) | 15k sampled of 175k (25k in v8) | Apache-2.0 | PERSON, LOCATION, US identifier types |

**Excluded:** ai4privacy/pii-masking-400k (non-commercial license — not used in
training or evaluation). MultiNERD (CC-BY-NC-SA 4.0 — evaluation only, not in
training corpus).

---

## Training procedure

**Teacher** (distilbert-base-uncased, 66M params):
```
epochs 10, batch 32, lr 3e-5, cosine LR, label-smoothing 0.05, device A100
```

**Student** (this model — BERT-mini, 11.13M params):
```
epochs 60, batch 64, lr 5e-5, cosine LR, label-smoothing 0.05,
distill-alpha 0.7, device A100 SXM4-40GB (~6h)
```

Knowledge distillation loss = 0.7 × KL(teacher logits ∥ student logits) + 0.3 × cross-entropy.

v8 (2026-10) used the same procedure on the expanded 205k-document mix.
v9 (2026-10) used the same procedure on the revised ~212k-document mix (data-only changes; see the v9 section above).

---

## How to use

```python
from transformers import AutoTokenizer, AutoModelForTokenClassification
import torch
tokenizer = AutoTokenizer.from_pretrained("CDT5058/phi-redact")
model = AutoModelForTokenClassification.from_pretrained("CDT5058/phi-redact")
model.eval()
text = "Patient Maria Garcia, DOB 1985-03-22, MRN 4471829, called 415-555-0189."
enc = tokenizer(text, return_tensors="pt", truncation=True, max_length=256)
with torch.no_grad():
    logits = model(**enc).logits          # [1, seq, 129]
pred_ids = logits.argmax(-1)[0].tolist()  # [seq]
id2label = model.config.id2label
# Decode BIOES tags to character spans
tokens    = tokenizer.convert_ids_to_tokens(enc["input_ids"][0])
offsets   = enc.encodings[0].offsets      # (char_start, char_end) per token
spans = []
span_start, span_label = None, None
for i, tag_id in enumerate(pred_ids):
    tag = id2label[tag_id]
    prefix, *label_parts = tag.split("-") if "-" in tag else (tag, [])
    label = "-".join(label_parts)
    if prefix == "S":
        spans.append((offsets[i][0], offsets[i][1], label))
    elif prefix == "B":
        span_start, span_label = offsets[i][0], label
    elif prefix == "E" and span_start is not None:
        spans.append((span_start, offsets[i][1], span_label))
        span_start = None
    elif prefix in ("O", "B"):
        span_start = None  # reset on unexpected transition
for start, end, label in spans:
    print(f"[{start}:{end}] {label!r:20s}  {text[start:end]!r}")
```

For production use, layer the deterministic `rules.py` (Luhn, ABA routing
checksum, SSN, IBAN, regex patterns) on top — it wins on structured identifiers
regardless of model confidence.

---

## Limitations

- **English only.** The bert-base-uncased WordPiece vocabulary has no meaningful
  cross-lingual representations. French, Spanish, or other languages will produce
  silent misses — not safe degradation.
- **PROFESSION over-fires** (F1=0.517 on Nemotron for v9; v7 was 0.494). Treat PROFESSION predictions
  with lower confidence or disable the entity type if not needed.
- **URL in informal prose** (recall=0.318 on student essays vs 0.828 on formal
  text). Check URL recall on your specific document type.
- **Over-redaction on clean text.** Clean-case FP rate 0.200 on the
  synthetic golden set at the default threshold (v8: 0.440; v7: 0.120).
  The remaining mass is public-context geography the golden set marks
  as keep (retention 0.133) — facility cities and state names in
  referral/program text. Precision-sensitive workflows can raise the
  neural threshold or use v7.
- **Trained on synthetic data.** Real-world typos, OCR noise, and unusual
  formatting will degrade performance relative to the benchmarks above.
- **Not a certified de-identification system.** Does not satisfy HIPAA Expert
  Determination or Safe Harbor. Use as a detection aid alongside human review.

---

## License

Apache-2.0. Training data licenses: CC-BY 4.0 (Nemotron-PII, LA essays,
WNUT-17), CC-BY-SA 4.0 (FewNERD), CC-BY-SA 3.0 (WikiANN). CC-BY-SA ShareAlike
requirements apply to the datasets, not the model weights, but consult your
legal counsel for commercial deployment.

---

*Card generated 2026-09-25 · Updated 2026-10-21 for v9 · Repo: CDT5058/phi-redact*

	---
	license: apache-2.0
	language:
	- en
	tags:
	- token-classification
	- named-entity-recognition
	- pii
	- phi
	- redaction
	- de-identification
	- ner
	- bert
	- knowledge-distillation
	- on-device
	pipeline_tag: token-classification
	---

	# phi-redact — compact English PII/PHI redaction model

	phi-redact is an 11M-parameter BERT-mini token classifier for detecting PII
	and PHI in English text. It covers 32 entity types with BIOES labels and is
	designed for on-device deployment (ONNX, Core ML, LiteRT) where data must not
	leave the device.

	It was knowledge-distilled from a 66M-parameter DistilBERT teacher and trained
	on ~212k documents (v9 mix) spanning clinical notes, government benefits text,
	student essays, general PII text, and Wikipedia-style prose.

	> Not a certified de-identification solution. This model assists redaction
	> workflows. It is not certified under HIPAA Expert Determination or Safe
	> Harbor methods. Evaluate it on your own data before relying on it in
	> compliance-sensitive contexts. English only.

	---

	## Current version: v9 (2026-10-21)

	The repo root is v9. v8 and v7 remain available under versioned
	names: `phi-redact-v8-int8.onnx`, `phi-redact-v8.onnx`,
	`phi-redact-v8.mlpackage`, `phi-redact-v8-fp32.mlpackage`,
	`phi-redact-v8-fp32.tflite`, `model-v8.safetensors`, `config-v8.json`,
	`tags-v8.json` (and likewise for v7).

	What changed in v9 — a data-only revision; the architecture,
	teacher, and training procedure are unchanged from v8.

	- Mined hard negatives. The shipped v8 model scored a pool of
	54,380 gold-empty documents from the training corpora; the 1,616 it
	fired on (plus 15,000 clean documents) joined the mix as negatives.
	- Revised synthetic generator (`gen_synth_v9.py`, 22k notes):
	segmented government-ID shapes boosted, 18% public-context /
	prose-address notes (geography rendered unlabeled in public
	contexts), 15% sparse low-density notes, and invalid-identifier
	decoys (impossible SSNs, Luhn-invalid cards, 555 numbers) appended
	to ~50% of notes.
	- Gravitee reduced 25k → 15k (loosest span boundaries in the mix).
	- SDK default threshold raised 0.5 → 0.6 (phi-redact-sdk 0.3.0):
	on the golden set 0.6 dominates 0.5 — higher precision and strict F1
	at identical recall.

	Golden 164-case synthetic redaction eval (rules + neural merge, the
	production configuration; the `evals/synthetic-164/` set):

	\| Metric \| v7 \| v8 INT8 \| v9 INT8 (shipped artifact) \|
	\|---\|---\|---\|---\|
	\| Full-value recall \| 0.885 \| 0.958 \| 0.974 \|
	\| Term recall \| 0.651 \| 0.760 \| 0.828 \|
	\| Precision \| 0.749 \| 0.695 \| 0.746 \|
	\| Strict span F1 \| 0.435 \| 0.617 \| 0.677 \|
	\| Clean-case FP rate \| 0.120 \| 0.440 \| 0.200 \|

	v9 figures are the shipped configuration: INT8 ONNX under the SDK
	scorer (`score_phi.py`) at the SDK 0.3.0 default threshold of 0.6. The
	PyTorch checkpoint scores 0.975 full-value recall under the Colab gate
	harness (`evaluate_golden.py`) — same test set, two harnesses, small
	scorer differences. INT8 quantization costs no recall here.

	The v8 tradeoff, revisited: v8 bought recall with precision (clean
	texts picked up a flagged span 44% of the time). v9 improves both axes
	at once — recall 0.958 → 0.974 and precision 0.695 → 0.746, clean-case
	FP rate halved to 0.200. What did not improve: public-context
	retention. The model still redacts most geography that the golden set
	marks as public/keep (retention 0.133, unchanged from v8) — facility
	cities, state program names, and similar keep-context spans are now
	about half of all remaining false positives. If your workflow is
	precision-sensitive beyond this, v7 remains available above.

	Standard benchmarks, v9 student (model only, strict span scoring):
	Learning Agency micro-F1 0.729 (NAME F1 0.778); FewNERD name F1 0.781
	(recall 0.838); MultiNERD name F1 0.893 (recall 0.940); WikiANN name F1
	0.763 (recall 0.831); govsynth 1.000. No material regression vs v8
	(WikiANN −0.011 is the largest move). The v7 tables below are unchanged
	and remain the Desert Ant comparison baseline.

	Nemotron-PII, v9 (measured 2026-10-21): same 2,000-document
	structured eval set as below (13,134 gold spans, model only, exact span
	match). v9 micro P 0.870 / R 0.830 / F1 0.849 — identical F1 to v8
	(0.849), itself a wash vs v7 (0.860) on this synthetic distribution;
	v9's gains are on the clinical-note distribution above. Per entity, v9
	improves SSN (0.957 vs v8 0.933), PROFESSION (0.517 vs 0.491) and
	LICENSE (0.908 vs 0.885), and gives back a little on ADDRESS, DEVICE
	and HEALTHPLAN.

	---

	## Model details

	\| Property \| Value \|
	\|---\|---\|
	\| Architecture \| BERT-mini (google/bert_uncased_L-4_H-256_A-4), 4L/256H/4A \|
	\| Parameters \| 11.13M \|
	\| Labels \| 129 BIOES tags (32 entity types × 4 positions + O) \|
	\| Max sequence length \| 256 tokens \|
	\| Vocabulary \| 30,522 WordPiece (bert-base-uncased) \|
	\| fp32 size \| 44.6 MB \|
	\| ONNX int8 size \| 11.3 MB (`phi-redact-v9-int8.onnx`) \|
	\| ONNX fp32 size \| 44.6 MB (`phi-redact-v9.onnx`) \|
	\| Core ML size \| 22.2 MB fp16 (`phi-redact-v9-fp32.mlpackage`), 5.6 MB 4-bit palettized (`phi-redact-v9.mlpackage`) \|
	\| LiteRT size \| 44.5 MB fp32 (`phi-redact-v9-fp32.tflite`) \|
	\| Training method \| Knowledge distillation (α=0.7) from distilbert-base-uncased teacher \|
	\| Language \| English only \|

	---

	## Entity types (32)

	`NAME_PATIENT` · `NAME_FAMILY` · `NAME_PROVIDER` · `DATE` · `ADDRESS` ·
	`PHONE` · `EMAIL` · `SSN` · `MRN` · `HEALTHPLAN` · `ACCOUNT` · `IDNUM` ·
	`LICENSE` · `EMPLOYER` · `PROFESSION` · `AGE` · `CITY` · `STATE` · `COUNTRY` ·
	`ZIP` · `URL` · `IP` · `DEVICE` · `ROUTING_NUMBER` · `IBAN` · `CREDITCARD` ·
	`HOSPITAL` · `FAX` · `PASSPORT` · `VIN` · `TAX_ID` · `GOVERNMENT_ID`

	> v8 note: `TAX_ID` and `GOVERNMENT_ID` were added in v8 (synthetic EIN /
	> USCIS / A-number templates plus the Gravitee US_ITIN mapping).
	> v9 note: segmented government-ID formats (e.g. Medicare-style
	> `1EG4-TE5-MK73`) improved — golden GOVERNMENT_ID full-value recall
	> 4/6 and the adversarial set went 0/2 → 1/2 — but GOVERNMENT_ID
	> remains the weakest structured type.

	---

	## Benchmark results (v7)

	Strict span-level scoring: a prediction counts only if both the character
	boundaries and the label match exactly.

	### Standard benchmarks

	\| Benchmark \| Recall \| F1 \| Notes \|
	\|---\|---\|---\|---\|
	\| WikiANN (EN, name recall) \| 84.3% \| 0.775 \| CC-BY-SA 3.0; direct comparison with Desert Ant ✓ \|
	\| FewNERD (EN, name recall) \| 85.9% \| 0.782 \| CC-BY-SA 4.0; Wikipedia-prose names \|
	\| LA essays (Micro F1 / Recall) \| 0.757 / 65.9% \| — \| CC-BY 4.0; informal student prose \|
	\| govsynth (government benefits) \| 100% \| 1.000 \| Synthetic; primary deployment domain \|
	\| Composite (5-benchmark recall avg) \| 86.1% \| — \| See note below \|

	Composite = (WikiANN + FewNERD + LA essays recall + govsynth + MultiNERD†) / 5.

	†MultiNERD (CC-BY-NC-SA 4.0) is included for completeness: v7 recall = 94.3%,
	within −0.6 pt of Desert Ant's 94.9% on the same dataset.

	### Nemotron-PII structured benchmark (CC-BY 4.0)

	First valid run — 13,134 entities across 25 entity types in 2,000 test
	documents. MICRO F1 = 0.860, Recall = 0.843.

	Selected per-entity F1 scores:

	\| Entity \| F1 \| Entity \| F1 \|
	\|---\|---\|---\|---\|
	\| NAME_PATIENT \| 0.944 \| SSN \| 0.908 \|
	\| NAME_FAMILY \| 0.937 \| EMAIL \| 0.905 \|
	\| MRN \| 0.951 \| CREDITCARD \| 0.905 \|
	\| ADDRESS \| 0.949 \| IDNUM \| 0.908 \|
	\| LICENSE \| 0.920 \| DATE \| 0.859 \|
	\| COUNTRY \| 0.919 \| URL \| 0.828 \|
	\| CITY \| 0.904 \| DEVICE \| 0.791 \|
	\| ROUTING_NUMBER \| 0.876 \| PROFESSION \| 0.494 \|

	Known weak spot — PROFESSION (F1=0.494, Precision=0.413): the model
	over-fires on job-title nouns in non-PII context ("She saw a doctor" vs
	"Occupation: Doctor"). Not a concern for most PHI/PII redaction use cases where
	PROFESSION is rarely the sensitive entity, but flag it if your data is
	occupation-heavy.

	URL in informal prose: Nemotron URL F1=0.828 (formal text) but LA essays
	URL recall=0.318 (informal prose). The model handles URLs well in structured
	documents; it misses them in casual first-person writing.

	### Comparison to Desert Ant Redact v0.4.0 (English)

	\| Metric \| phi-redact v7 \| Desert Ant (EN) \|
	\|---\|---\|---\|
	\| WikiANN name recall \| 84.3% \| 69.5% \|
	\| MultiNERD name recall \| 94.3% \| 94.9% \|
	\| Composite (5 benchmarks) \| 86.1% \| 86.5% \|
	\| Parameters \| 11.13M \| 23M \|
	\| Est. 4-bit on-device size \| ~5.6 MB \| 11.6 MB \|
	\| Languages \| English only \| 27 languages \|
	\| License \| Apache-2.0 \| Source-available \|

	phi-redact v7 is essentially tied with Desert Ant's English composite at half
	the parameters and half the on-device footprint.

	### Inference latency

	Measured 2026-10-03 against the live demo (corytrimm.com/phi-redact): the
	same 15-template typing-length corpus as `evals/latency-bench/bench.py`,
	15 templates × 10 cycles = 150 redactions, timed from the page's own
	per-redaction readout — TypeScript rules scan + ONNX detect + merge +
	placeholders, end to end, onnxruntime-web WASM (Chromium on Linux).

	\| Input \| p50 \| p95 \| p99 \|
	\|---\|---\|---\|---\|
	\| Chat-length (150 samples) \| 6.0 ms \| 8.0 ms \| 9.0 ms \|
	\| Long text (3,959 chars, 139 entities) \| 572 ms \| — \| — \|

	Model load: 0.5 s cached, 4.4 s cold (11.3 MB int8 ONNX + runtime download).

	Caveats: the model was cached for this run; browser version and hardware
	details were not obtainable from the measurement setup (Linux VM, not a
	phone); readout resolution is whole milliseconds. Not a head-to-head with
	any vendor's published number — different hardware, corpus, and setup.
	Methodology travels with the number; see `evals/latency-bench/`.

	### Synthetic redaction eval (164 cases)

	`evals/synthetic-164/` carries a 164-case synthetic redaction test set, the
	scoring harness, and v7's results: 0.812 full-value recall (term recall
	0.615, precision 0.698, clean-case FP rate 0.120). Perfect 1.000 on
	BANK_ACCOUNT, CREDIT_CARD, DRIVERS_LICENSE, EMAIL, PHONE, ROUTING_NUMBER;
	weak on NAME (0.347) and ADDRESS (0.333). See the directory README for
	methodology and the taxonomy note. (Re-measured 2026-10-19 with the shipped
	SDK — fixed marker ids and the current rules — v7 scores 0.885 full-value
	recall on this set; the 0.812 above is the original 2026-10-02 measurement.)

	---

	## Training data

	All sources are commercial-safe (CC-BY 4.0 or CC-BY-SA 4.0). No real patient
	data was used.

	\| Source \| Size \| License \| Entity coverage \|
	\|---\|---\|---\|---\|
	\| Synthetic notes, v9 generator (gen_synth_v9.py; extends gen_synth + v8 generator) \| 22k \| Synthetic (ours) \| Full 32-type schema; public-context geo, sparse notes, invalid-ID decoys \|
	\| Learning Agency Lab PII (Kaggle) \| ~5,400 \| CC-BY 4.0 \| Names, emails, URLs in student essays \|
	\| govsynth (synthetic govt. benefits) \| ~20k \| Synthetic (ours) \| SNAP/Medicaid/WIC case management PII \|
	\| essay_names.jsonl (synthetic prose) \| 10k \| Synthetic (ours) \| Names in biography, news, legal, informal prose \|
	\| FewNERD train split (DFKI-SLT/few-nerd) \| 50k \| CC-BY-SA 4.0 \| Wikipedia person entities \|
	\| WikiANN train split (EN) \| ~20k \| CC-BY-SA 3.0 \| Wikipedia person, location, org \|
	\| WNUT-17 train split \| ~3.4k \| CC-BY 4.0 \| Emerging entities in social media \|
	\| Nemotron-PII train split (nvidia) \| 50k \| CC-BY 4.0 \| 30 PII/PHI types in synthetic documents \|
	\| Mined negatives (v8-model-mined, rules-filtered) \| 16,616 \| Derived from corpora above \| Gold-empty documents; 1,616 the v8 model fired on + 15,000 clean \|
	\| Gravitee PII (gravitee-io/pii-detection-dataset) \| 15k sampled of 175k (25k in v8) \| Apache-2.0 \| PERSON, LOCATION, US identifier types \|

	Excluded: ai4privacy/pii-masking-400k (non-commercial license — not used in
	training or evaluation). MultiNERD (CC-BY-NC-SA 4.0 — evaluation only, not in
	training corpus).

	---

	## Training procedure

	Teacher (distilbert-base-uncased, 66M params):
	```
	epochs 10, batch 32, lr 3e-5, cosine LR, label-smoothing 0.05, device A100
	```

	Student (this model — BERT-mini, 11.13M params):
	```
	epochs 60, batch 64, lr 5e-5, cosine LR, label-smoothing 0.05,
	distill-alpha 0.7, device A100 SXM4-40GB (~6h)
	```

	Knowledge distillation loss = 0.7 × KL(teacher logits ∥ student logits) + 0.3 × cross-entropy.

	v8 (2026-10) used the same procedure on the expanded 205k-document mix.
	v9 (2026-10) used the same procedure on the revised ~212k-document mix (data-only changes; see the v9 section above).

	---

	## How to use

	```python
	from transformers import AutoTokenizer, AutoModelForTokenClassification
	import torch

	tokenizer = AutoTokenizer.from_pretrained("CDT5058/phi-redact")
	model = AutoModelForTokenClassification.from_pretrained("CDT5058/phi-redact")
	model.eval()

	text = "Patient Maria Garcia, DOB 1985-03-22, MRN 4471829, called 415-555-0189."

	enc = tokenizer(text, return_tensors="pt", truncation=True, max_length=256)
	with torch.no_grad():
	logits = model(**enc).logits # [1, seq, 129]

	pred_ids = logits.argmax(-1)[0].tolist() # [seq]
	id2label = model.config.id2label

	# Decode BIOES tags to character spans
	tokens = tokenizer.convert_ids_to_tokens(enc["input_ids"][0])
	offsets = enc.encodings[0].offsets # (char_start, char_end) per token

	spans = []
	span_start, span_label = None, None
	for i, tag_id in enumerate(pred_ids):
	tag = id2label[tag_id]
	prefix, *label_parts = tag.split("-") if "-" in tag else (tag, [])
	label = "-".join(label_parts)
	if prefix == "S":
	spans.append((offsets[i][0], offsets[i][1], label))
	elif prefix == "B":
	span_start, span_label = offsets[i][0], label
	elif prefix == "E" and span_start is not None:
	spans.append((span_start, offsets[i][1], span_label))
	span_start = None
	elif prefix in ("O", "B"):
	span_start = None # reset on unexpected transition

	for start, end, label in spans:
	print(f"[{start}:{end}] {label!r:20s} {text[start:end]!r}")
	```

	For production use, layer the deterministic `rules.py` (Luhn, ABA routing
	checksum, SSN, IBAN, regex patterns) on top — it wins on structured identifiers
	regardless of model confidence.

	---

	## Limitations

	- English only. The bert-base-uncased WordPiece vocabulary has no meaningful
	cross-lingual representations. French, Spanish, or other languages will produce
	silent misses — not safe degradation.
	- PROFESSION over-fires (F1=0.517 on Nemotron for v9; v7 was 0.494). Treat PROFESSION predictions
	with lower confidence or disable the entity type if not needed.
	- URL in informal prose (recall=0.318 on student essays vs 0.828 on formal
	text). Check URL recall on your specific document type.
	- Over-redaction on clean text. Clean-case FP rate 0.200 on the
	synthetic golden set at the default threshold (v8: 0.440; v7: 0.120).
	The remaining mass is public-context geography the golden set marks
	as keep (retention 0.133) — facility cities and state names in
	referral/program text. Precision-sensitive workflows can raise the
	neural threshold or use v7.
	- Trained on synthetic data. Real-world typos, OCR noise, and unusual
	formatting will degrade performance relative to the benchmarks above.
	- Not a certified de-identification system. Does not satisfy HIPAA Expert
	Determination or Safe Harbor. Use as a detection aid alongside human review.

	---

	## License

	Apache-2.0. Training data licenses: CC-BY 4.0 (Nemotron-PII, LA essays,
	WNUT-17), CC-BY-SA 4.0 (FewNERD), CC-BY-SA 3.0 (WikiANN). CC-BY-SA ShareAlike
	requirements apply to the datasets, not the model weights, but consult your
	legal counsel for commercial deployment.

	---

	Card generated 2026-09-25 · Updated 2026-10-21 for v9 · Repo: CDT5058/phi-redact