Blame for README.md

---
library_name: transformers
pipeline_tag: token-classification
base_model: microsoft/deberta-v3-base
tags:
- token-classification
- span-extraction
- brewslm
license: mit
language:
- en
---

# Sensitive-data tagger (PCI / GDPR / HIPAA + custom) span tagger

A token-classification model that finds spans of text and labels each with a type, fine-tuned from `microsoft/deberta-v3-base`.

> **For education only.** This model is a demonstration of a training method. It is not a privacy, security or
> compliance expert, and it makes no claim to subject-matter expertise. It is not legal or compliance advice. It can
> miss sensitive data and can tag text that isn't sensitive. Don't rely on it as a safeguard for real personal data:
> consult a qualified privacy or compliance professional.

**Trained with [BrewSLM](https://brewslm.com)** (project “Sensitive-data tagger (PCI / GDPR / HIPAA + custom)”, run #143),
a platform for building small language models from your own data. How this model was built, step by step:
[*Build a sensitive-data detector from zero, with no labelled data*](https://brewslm.com/blog-sensitive-data-detector-from-zero.html),
and more on the [BrewSLM blog](https://brewslm.com/blog.html). Non-commercial showcase; no warranty.

## Model

- **Base model:** `microsoft/deberta-v3-base`
- **Task:** token classification (BIO tags → character spans)
- **Window:** 512 tokens (the tokenizer's `model_max_length`); longer texts are read in overlapping windows

## Entity types

30 types: `address`, `api_key`, `bank_account`, `card_expiry`, `credit_card`, `customer_id`, `cvv`, `date_of_birth`, `device_id`, `diagnosis`, `email`, `employee_id`, `health_plan_id`, `iban`, `ip_address`, `licence_number`, `license_plate`, `medical_record_number`, `medication`, `national_id`, `passport`, `person_name`, `phone`, `provider_name`, `routing_number`, `tax_id`, `ticket_id`, `transaction_id`, `username`, `visit_date`.
Labels are BIO: `O`, then `B-<type>` / `I-<type>` per type.

## Training data

- **Training rows:** 1620 (1612 with at least one span)
- **Validation rows:** 202
- **Data kinds:** synthetic
- **Row sources:** `passage_labels:cloud:anthropic:claude-sonnet-5-5` 919; `teacher_llm` 335; `passage_labels:ollama` 223; `passage_labels:cloud:anthropic:claude-haiku-4-5-20251001` 101; `document_labels:ollama` 32; `teacher_llm+regex` 10
- **Labelled by:** `cloud:anthropic:claude-sonnet-5-5` 1620
- **Augmentation:** 1616 OCR-noise copies (character noise rate 0.1)
- **Epochs:** 5
- **Seed selection:** Picked seed 45 (run #143) of 3 by validation span F1 (tuned thresholds) (higher is better; values: seed 45 0.9467, seed 46 0.9452, seed 47 0.9437); test examples, the answer key and external benchmarks were not used to pick.

## Evaluation

Strict span match: a prediction counts only when its type, start and end all equal a reference span.

### Answer key (44 texts)

Reference spans written by `cloud:anthropic:claude-sonnet-5-5`.

Human review: 11 of 44 rows checked by a person (1 corrected). On the checked rows: precision 0.90 · recall 0.95 · F1 0.92 (11 texts).

Overall: precision 0.97 · recall 0.98 · F1 0.97

0 of 24 types have enough reference spans to show a 0.05 F1 difference; the others can only show 0.82 or more (medical_record_number: 1 span, needs 152).

| Type | Precision | Recall | F1 | Gold spans |
|---|---|---|---|---|
| `address` | 1.00 | 1.00 | 1.00 | 4 |
| `api_key` | 1.00 | 1.00 | 1.00 | 2 |
| `bank_account` | 0.62 | 1.00 | 0.77 | 5 |
| `card_expiry` | 1.00 | 1.00 | 1.00 | 16 |
| `credit_card` | 1.00 | 0.91 | 0.95 | 23 |
| `cvv` | 1.00 | 1.00 | 1.00 | 9 |
| `date_of_birth` | 0.70 | 1.00 | 0.82 | 7 |
| `diagnosis` | 1.00 | 1.00 | 1.00 | 16 |
| `email` | 1.00 | 1.00 | 1.00 | 27 |
| `employee_id` | 1.00 | 1.00 | 1.00 | 3 |
| `health_plan_id` | 1.00 | 1.00 | 1.00 | 7 |
| `iban` | 1.00 | 1.00 | 1.00 | 7 |
| `ip_address` | 1.00 | 1.00 | 1.00 | 6 |
| `medical_record_number` | 1.00 | 1.00 | 1.00 | 1 |
| `medication` | 0.86 | 0.90 | 0.88 | 20 |
| `national_id` | 1.00 | 1.00 | 1.00 | 13 |
| `passport` | 1.00 | 1.00 | 1.00 | 6 |
| `person_name` | 1.00 | 1.00 | 1.00 | 37 |
| `phone` | 1.00 | 1.00 | 1.00 | 25 |
| `provider_name` | 1.00 | 1.00 | 1.00 | 23 |
| `routing_number` | 0.00 | 0.00 | 0.00 | 3 |
| `ticket_id` | 1.00 | 1.00 | 1.00 | 9 |
| `username` | 1.00 | 1.00 | 1.00 | 1 |
| `visit_date` | 0.94 | 1.00 | 0.97 | 16 |

### Test examples (44 texts)

Overall: precision 0.96 · recall 0.98 · F1 0.97

0 of 24 types have enough reference spans to show a 0.05 F1 difference; the others can only show 1.00 or more (medical_record_number: 1 span, needs 152).

| Type | Precision | Recall | F1 | Gold spans |
|---|---|---|---|---|
| `address` | 1.00 | 1.00 | 1.00 | 4 |
| `api_key` | 1.00 | 1.00 | 1.00 | 2 |
| `bank_account` | 0.62 | 1.00 | 0.77 | 5 |
| `card_expiry` | 1.00 | 1.00 | 1.00 | 16 |
| `credit_card` | 0.95 | 0.87 | 0.91 | 23 |
| `cvv` | 1.00 | 1.00 | 1.00 | 9 |
| `date_of_birth` | 0.70 | 1.00 | 0.82 | 7 |
| `diagnosis` | 1.00 | 1.00 | 1.00 | 16 |
| `email` | 1.00 | 1.00 | 1.00 | 27 |
| `employee_id` | 1.00 | 1.00 | 1.00 | 3 |
| `health_plan_id` | 1.00 | 1.00 | 1.00 | 7 |
| `iban` | 1.00 | 1.00 | 1.00 | 7 |
| `ip_address` | 1.00 | 1.00 | 1.00 | 6 |
| `medical_record_number` | 1.00 | 1.00 | 1.00 | 1 |
| `medication` | 0.86 | 0.90 | 0.88 | 20 |
| `national_id` | 1.00 | 1.00 | 1.00 | 13 |
| `passport` | 1.00 | 1.00 | 1.00 | 6 |
| `person_name` | 1.00 | 1.00 | 1.00 | 37 |
| `phone` | 1.00 | 1.00 | 1.00 | 25 |
| `provider_name` | 1.00 | 1.00 | 1.00 | 23 |
| `routing_number` | 0.00 | 0.00 | 0.00 | 2 |
| `ticket_id` | 1.00 | 1.00 | 1.00 | 9 |
| `username` | 1.00 | 1.00 | 1.00 | 1 |
| `visit_date` | 0.94 | 1.00 | 0.97 | 16 |

## External benchmarks

Labelled sets made by someone else, mapped onto this model's types; only the mapped types are scored (a prediction of another type is neither a hit nor a false positive).

| Benchmark | Source | Licence | Texts | Types scored | Precision | Recall | F1 |
|---|---|---|---|---|---|---|---|
| CoNLL-2003 people (news, human-labelled) | `eriktks/conll2003:test` | other | 400 | person_name | 0.93 | 0.70 | 0.80 |
| Gretel PII (synthetic, independent) | `gretelai/gretel-pii-masking-en-v1:test` | apache-2.0 | 500 | address, api_key, bank_account, credit_card, date_of_birth, email, employee_id, health_plan_id, ip_address, license_plate, medical_record_number, national_id, person_name, phone | 0.76 | 0.80 | 0.78 |
| CoNLL-2003 people, large (validation, 2000 rows) | `eriktks/conll2003:validation` | other | 2000 | person_name | 0.93 | 0.74 | 0.83 |

## Usage

```python
from transformers import pipeline

tagger = pipeline("token-classification", model="path/to/model", aggregation_strategy="simple", stride=128)

def find_spans(text):
    spans = []
    for ent in tagger(text):
        start, end = ent["start"], ent["end"]
        # The tokenizer can attach the preceding space to a span; trim it.
        while start < end and text[start].isspace():
            start += 1
        spans.append({"type": ent["entity_group"], "start": start, "end": end,
                      "text": text[start:end], "score": float(ent["score"])})
    return spans
```

`stride=128` matters: without it the pipeline cuts a text at the 512-token window and never tags anything after it. With it, long texts are read in overlapping windows.

`thresholds.json` holds per-type minimum scores (all 0.0 = keep every predicted span) and a `recall_bias`. To favour precision for a type, drop its spans whose score is under that type's value. BrewSLM's redact endpoint and scored decoder read this file; there the score is the span's mean per-token probability of its type, and `recall_bias` (0–1) is subtracted from each token's `O` probability so borderline tokens are tagged (more recall, less precision).

## Limitations

- Educational demonstration only, with no claim to privacy, security or compliance expertise. Not legal or compliance
  advice.
- Scores come from small held-out sets — 44 texts (answer key), 44 texts (test examples) — so expect a few points of noise either way.
- Types with fewer than 10 reference spans in an evaluation (`address`, `api_key`, `bank_account`, `cvv`, `date_of_birth`, `employee_id`, `health_plan_id`, `iban`, `ip_address`, `medical_record_number`, `passport`, `routing_number`, `ticket_id`, `username`) have unreliable per-type scores.
- The training data is synthetic or model-labelled, not human-annotated; real documents with other layouts, formats or languages may score lower.
- It finds what it was trained to find: entity types outside the list above are never tagged.
- Not a compliance guarantee: this model can miss sensitive data and can tag text that isn't. Using it does not by itself make a system PCI DSS, GDPR, HIPAA or CCPA compliant. Use it as one layer, with review and other controls, not as the only safeguard.
- On the Gretel PII benchmark (103 employee_id, 55 license_plate, 17 bank_account reference spans — directional numbers): employee_id precision 0.95 / recall 0.70, license_plate F1 0.69, bank_account precision 0.50 (false positives are mostly routing numbers, device identifiers and timestamps that Gretel leaves unlabelled). Compared with the previous export of this project (run #135), employee_id recall is 0.06 lower and bank_account precision 0.21 lower; employee_id precision is 0.15 higher and license_plate 0.10 higher. Plate false positives are still UK National Insurance numbers, medical record numbers and postcodes.
- Benchmark hygiene: 32 Gretel gold values (identifiers, plates, street names) had been copied verbatim into earlier training rows by the data-seeding prompts (591 occurrences in 248 rows). They were replaced with invented values of the same shape before this run trained, so its Gretel figures are measured on data the model never saw. The previous export's employee_id figure was overstated by an estimated 0.03–0.15 F1 for that reason.