| ---
|
| license: apache-2.0
|
| language:
|
| - bg
|
| - cs
|
| - da
|
| - de
|
| - el
|
| - en
|
| - es
|
| - et
|
| - fi
|
| - fr
|
| - hr
|
| - hu
|
| - id
|
| - it
|
| - ja
|
| - ko
|
| - lt
|
| - lv
|
| - ms
|
| - nl
|
| - pl
|
| - pt
|
| - ro
|
| - sk
|
| - sl
|
| - sr
|
| - sv
|
| - tl
|
| - vi
|
| - zh
|
| tags:
|
| - token-classification
|
| - pii
|
| - pii-detection
|
| - named-entity-recognition
|
| - privacy
|
| - redaction
|
| - multilingual
|
| - onnx
|
| pipeline_tag: token-classification
|
| library_name: onnxruntime
|
| ---
|
|
|
| # PiiScan — multilingual PII detection with exact character offsets
|
|
|
| For any input text, PiiScan returns **every personal-data span it finds, with the exact character
|
| offsets into your original string**. It ships as a CPU-only **INT8 ONNX** model (~139 MB with
|
| tokenizer) that runs fully offline — no GPU, no network.
|
|
|
| The guarantee everything rests on: `original_text[entity.start:entity.end] == entity.value`.
|
|
|
| - **Base model:** `distilbert-base-multilingual-cased`
|
| - **Task:** token classification, BIO over 27 labels (13 entity types)
|
| - **Types:** 13 model-trained + 11 rule-only = **24 usable**; 3 experimental, 12 blocked
|
| - **Languages:** 30 (no Indic scripts — see limitations)
|
| - **Runtime:** `onnxruntime` (CPU), ~13 ms / document
|
| - **Offsets:** Unicode code points, zero-based, end-exclusive
|
|
|
| ## Results — held-out test set (policy `balanced`, exact-span match)
|
|
|
| | Type | Precision | Recall | F1 | Support |
|
| |---|---:|---:|---:|---:|
|
| | `POSTAL_CODE` | 0.997 | 0.995 | **0.996** | 2,566 |
|
| | `PAYMENT_CARD_NUMBER` | 0.995 | 0.993 | **0.994** | 1,894 |
|
| | `CITY` | 0.993 | 0.991 | **0.992** | 4,146 |
|
| | `PHONE_NUMBER` | 0.988 | 0.996 | **0.992** | 3,361 |
|
| | `EMAIL_ADDRESS` | 0.987 | 0.988 | **0.988** | 4,400 |
|
| | `STREET_ADDRESS` | 0.989 | 0.978 | **0.983** | 3,909 |
|
| | `AGE` | 0.992 | 0.962 | **0.977** | 2,991 |
|
| | `TAX_ID` | 0.990 | 0.941 | **0.965** | 1,540 |
|
| | `DRIVER_LICENSE_NUMBER` | 0.984 | 0.940 | **0.961** | 1,662 |
|
| | `PERSON_NAME` | 0.976 | 0.931 | **0.953** | 8,810 |
|
| | `DATE_TIME` | 0.980 | 0.922 | **0.950** | 9,592 |
|
| | `PASSPORT_NUMBER` | 0.977 | 0.917 | **0.946** | 1,260 |
|
| | `GOVERNMENT_ID` | 0.977 | 0.912 | **0.943** | 3,439 |
|
| | **macro-F1** | | | **0.972** | |
|
| | **micro-F1** | 0.985 | 0.954 | **0.969** | 49,570 |
|
|
|
| Scored at document level on 8,328 held-out documents, exact `[start, end)` and type match.
|
|
|
| **These are in-distribution numbers.** The test split comes from the same generator as the training
|
| data. On 127 human-annotated real court judgments in the same split, the two types those
|
| documents annotate score considerably lower:
|
|
|
| | Slice | Precision | Recall | F1 |
|
| |---|---:|---:|---:|
|
| | Same generator as training | 0.990 | 0.978 | 0.984 |
|
| | Real human-annotated documents | 0.889 | 0.641 | 0.745 |
|
|
|
| Expect the lower number on your own data. Validate in your domain before relying on it.
|
|
|
| ## Entity tiers — what is actually supported
|
|
|
| Not all types are equal, and this card says so rather than advertising one big number.
|
|
|
| | Tier | Count | Types |
|
| |---|---:|---|
|
| | **Model-trained** | 13 | `AGE`, `CITY`, `DATE_TIME`, `DRIVER_LICENSE_NUMBER`, `EMAIL_ADDRESS`, `GOVERNMENT_ID`, `PASSPORT_NUMBER`, `PAYMENT_CARD_NUMBER`, `PERSON_NAME`, `PHONE_NUMBER`, `POSTAL_CODE`, `STREET_ADDRESS`, `TAX_ID` |
|
| | **Rule-only** (checksums/format) | 11 | `AADHAAR_NUMBER`, `CRYPTO_WALLET`, `DEVICE_IDENTIFIER`, `FAX_NUMBER`, `GEO_COORDINATES`, `IBAN`, `IP_ADDRESS`, `MAC_ADDRESS`, `SOCIAL_SECURITY_NUMBER`, `SWIFT_BIC`, `VEHICLE_IDENTIFIER` |
|
| | **Experimental** (opt-in) | 3 | `COUNTRY`, `URL`, `USERNAME` |
|
| | **Blocked** (insufficient data) | 12 | `ADDRESS`, `BANK_ACCOUNT_NUMBER`, `BIOMETRIC_IDENTIFIER`, `COUNTY_DISTRICT`, `DATE_OF_BIRTH`, `HEALTH_PLAN_ID`, `LICENSE_PLATE`, `MEDICAL_RECORD_NUMBER`, `OTHER_UNIQUE_ID`, `PROVIDER_ID`, `STATE_PROVINCE`, `VOTER_ID` |
|
|
|
| Rule-only types are excluded from the model **by design**. MOD-97 validates an IBAN exactly;
|
| training a network to approximate a checksum would be strictly worse.
|
|
|
| ## Quick start (Python + onnxruntime)
|
|
|
| ```bash
|
| pip install onnxruntime tokenizers huggingface_hub numpy
|
| ```
|
|
|
| ```python
|
| import json, numpy as np, onnxruntime as ort
|
| from huggingface_hub import hf_hub_download
|
| from tokenizers import Tokenizer
|
|
|
| REPO = "vectorsense/piiscan"
|
| sess = ort.InferenceSession(hf_hub_download(REPO, "model.int8.onnx"),
|
| providers=["CPUExecutionProvider"])
|
| tok = Tokenizer.from_file(hf_hub_download(REPO, "tokenizer.json"))
|
| labels = json.loads(open(hf_hub_download(REPO, "labels.json"), encoding="utf-8").read())
|
|
|
| def detect(text: str, threshold: float = 0.75) -> list[dict]:
|
| enc = tok.encode(text)
|
| ids = np.array([enc.ids], dtype=np.int64)
|
| mask = np.array([enc.attention_mask], dtype=np.int64)
|
| feeds = {i.name: (mask if "mask" in i.name else ids) for i in sess.get_inputs()}
|
| logits = sess.run(None, feeds)[0][0]
|
| e = np.exp(logits - logits.max(-1, keepdims=True))
|
| probs = e / e.sum(-1, keepdims=True)
|
|
|
| spans, current = [], None
|
| for i, (start, end) in enumerate(enc.offsets):
|
| if start == end: # special token
|
| continue
|
| label = labels[int(probs[i].argmax())]
|
| if label == "O":
|
| current = None
|
| continue
|
| prefix, entity_type = label.split("-", 1)
|
| if prefix == "B" or current is None or current["type"] != entity_type:
|
| current = {"type": entity_type, "start": start, "end": end,
|
| "scores": [float(probs[i].max())]}
|
| spans.append(current)
|
| else:
|
| current["end"] = end
|
| current["scores"].append(float(probs[i].max()))
|
|
|
| out = []
|
| for s in spans:
|
| score = sum(s["scores"]) / len(s["scores"])
|
| if score >= threshold:
|
| out.append({"type": s["type"], "start": s["start"], "end": s["end"],
|
| "value": text[s["start"]:s["end"]], "score": round(score, 4)})
|
| return out
|
|
|
| for entity in detect("Patient John Smith was seen on 01/15/2026 at 80331 München"):
|
| print(entity)
|
| # {'type': 'PERSON_NAME', 'start': 8, 'end': 18, 'value': 'John Smith', 'score': 0.8427}
|
| # {'type': 'DATE_TIME', 'start': 31, 'end': 41, 'value': '01/15/2026', 'score': 0.9948}
|
| # {'type': 'POSTAL_CODE', 'start': 45, 'end': 50, 'value': '80331', 'score': 0.9925}
|
| ```
|
|
|
| This snippet is the **model only**, single window. It does not include the rule layer (IBAN, Aadhaar,
|
| payment-card checksums), sliding windows for long documents, or the hybrid resolver. For those, use
|
| the package below.
|
|
|
| ## Run it as a REST API (request / response service)
|
|
|
| The companion [PiiScan repo](https://github.com/palarnab) ships a FastAPI service that wraps this
|
| model with the rule layer, policy presets, nested spans, and redaction.
|
|
|
| ```bash
|
| pip install fastapi "uvicorn[standard]" onnxruntime tokenizers numpy pyyaml
|
| uvicorn service.app:app --port 5001 # auto-loads model.int8.onnx from ./models
|
| ```
|
|
|
| ```bash
|
| curl -s http://127.0.0.1:5001/detect_pii \
|
| -H "Content-Type: application/json" \
|
| -d '{"texts":["Contact: jsmith@email.com or 555-123-4567"]}'
|
| ```
|
|
|
| ```powershell
|
| # PowerShell — send UTF-8 bytes, or non-ASCII input is mangled by Windows PowerShell 5.1
|
| $body = '{"texts":["Contact: jsmith@email.com or 555-123-4567"]}'
|
| $bytes = [Text.Encoding]::UTF8.GetBytes($body)
|
| Invoke-RestMethod -Uri http://127.0.0.1:5001/detect_pii -Method Post `
|
| -ContentType 'application/json; charset=utf-8' -Body $bytes
|
| ```
|
|
|
| Response fields per entity: `entitytype`, `start`, `end`, `value`, `score`, `length`, `subtype`,
|
| `source` (`rule` / `model` / `hybrid`), `policy_tags`, `parent_id`.
|
| Endpoints: `GET /`, `POST /detect_pii`, `GET /health/ready`, `GET /metadata`, `POST /redact`.
|
|
|
| ## Policy presets
|
|
|
| `strict` (0.50) / `balanced` (0.75) / `precision` (0.90) set the score threshold; override per
|
| request with `options.score_threshold`. Published results use `balanced`.
|
|
|
| Scores are **not calibrated probabilities** — 0.90 does not mean 90% correct. Rule-layer spans carry
|
| a fixed confidence by evidence strength, not a learned score. See `thresholds.json`.
|
|
|
| ## Training record
|
|
|
| Full provenance lives in this repo under `training/` (best checkpoint, configs, dataset manifest,
|
| source licences, rule-layer benchmark). Trained 3 epochs on 1,299,487 rows,
|
| 4 h 20 m on one RTX 2000 Ada.
|
|
|
| ## Limitations & licensing
|
|
|
| - **No PHI claim.** `MEDICAL_RECORD_NUMBER` and `HEALTH_PLAN_ID` are blocked for lack of data. This
|
| is not a HIPAA de-identification tool.
|
| - **No Indic support.** The corpus has 30 languages but zero Devanagari or Bengali script. Aadhaar
|
| and PAN still work inside Indic text because those formats are script-independent.
|
| - **Recall is never 1.0.** High-stakes redaction needs human review.
|
| - **Long-form real-world prose is weaker than the headline table.** Person names in legal documents
|
| are the known worst case; initials and single-letter anonymised parties are frequently missed.
|
| - **Place names can be tagged as people**, occasionally with high confidence.
|
| - CJK has a small span-alignment ceiling from wordpiece merging across entity boundaries.
|
| - Training data derived from **Ai4Privacy OpenPII 1.5M (CC-BY-4.0), © Ai Suisse SA** —
|
| https://huggingface.co/datasets/ai4privacy/pii-masking-openpii-1.5m
|
| Evaluation also uses the **Text Anonymization Benchmark** (MIT).
|
| - Only a redistributable subset of the corpus is published — see the dataset card.
|
|
|
| ## Author
|
|
|
| Arnab Pal — [LinkedIn](https://www.linkedin.com/in/pal-arnab) · [GitHub](https://github.com/palarnab)
|
|
|
