Instructions to use altslate/indic-pii-redactor-small with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use altslate/indic-pii-redactor-small with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("token-classification", model="altslate/indic-pii-redactor-small")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForTokenClassification tokenizer = AutoTokenizer.from_pretrained("altslate/indic-pii-redactor-small") model = AutoModelForTokenClassification.from_pretrained("altslate/indic-pii-redactor-small", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Indic PII Redactor (small)
A 141M-parameter token-classification model that finds Indian personal data in English, Hindi and Hinglish text, so you can redact it before logging, analytics or sending text to an LLM.
It detects 16 types, including Indian identifiers that general PII models don't have:
| Label | What it covers |
|---|---|
AADHAAR |
Aadhaar numbers (12 digits, any grouping, masked forms like XXXX XXXX 1234) and 16-digit VIDs |
PAN |
PAN (ABCPS1234K), including lowercase and spaced forms |
GSTIN |
15-character GST numbers |
IFSC |
Bank branch codes (SBIN0001234) |
UPI_ID |
UPI handles (name@okhdfcbank, 9876543210@ybl) |
BANK_ACCOUNT |
Bank account numbers |
PHONE |
Mobile and landline numbers in Indian formats |
VOTER_ID |
EPIC numbers, including the old MH/12/345/678901 format |
PASSPORT |
Indian passport numbers |
DRIVING_LICENCE |
Driving licence numbers |
VEHICLE_REG |
Number plates, including BH-series and Devanagari plates |
PINCODE |
PIN codes |
PERSON |
Names (Latin and Devanagari) |
ADDRESS |
Postal addresses, including village/tehsil and landmark styles |
EMAIL |
Email addresses |
DATE_OF_BIRTH |
Dates of birth (other dates are left alone) |
It is trained to leave look-alike numbers alone: order IDs, transaction and UTR numbers, PNRs, OTPs, helplines, tracking numbers and policy numbers.
Usage
from transformers import pipeline
redactor = pipeline("token-classification", model="altslate/indic-pii-redactor-small",
aggregation_strategy="simple")
def spans(text):
"""(start, end, label) per entity; offsets trimmed of the leading space the tokenizer attaches."""
out = []
for e in redactor(text):
s, t = e["start"], e["end"]
while s < t and text[s].isspace():
s += 1
out.append((s, t, e["entity_group"]))
return out
def redact(text):
for s, t, label in sorted(spans(text), reverse=True):
text = text[:s] + f"[{label}]" + text[t:]
return text
text = "Mera aadhaar 2345 6789 0123 hai aur PAN ABCPS1234K, call 98765 43210"
print(redact(text))
# Mera aadhaar [AADHAAR] hai aur PAN [PAN], call [PHONE]
The pipeline's start offset (and word) can include the space before a token, hence the trim above.
ONNX (onnxruntime, transformers.js)
onnx/model.onnx (fp32, 563 MB) and onnx/model_quantized.onnx (dynamic int8, 142 MB) take input_ids and attention_mask and return logits. On the 500-message dev split:
| Runtime | Recall | Precision | No-PII false alarms |
|---|---|---|---|
| PyTorch | 0.996 | 0.997 | 1.4% |
| ONNX fp32 | 0.996 | 0.997 | 1.4% |
| ONNX int8 | 0.995 | 0.996 | 4.1% |
import { pipeline } from "@huggingface/transformers";
const tagger = await pipeline("token-classification", "altslate/indic-pii-redactor-small", { dtype: "q8" });
const tokens = await tagger("Mera aadhaar 2345 6789 0123 hai aur PAN ABCPS1234K");
// One entry per token with B-/I- labels, e.g. {entity: "B-AADHAAR", word: "2"}, {entity: "I-AADHAAR", word: "3"}, ...
// Join a B- token with the I- tokens that follow it to get whole entities.
Tested with @huggingface/transformers 4.3.2 in Node 22, dtype "fp32" and "q8".
Results
All numbers are measured on data the model never saw in training. Redaction recall/precision count PII characters (did the right characters get masked, whatever the type), which is what matters for redaction and is fair to models with different label sets. No-PII false alarms is the share of messages without any PII where the model redacts something. Exact F1 needs our label set, so it is shown only for models that can output it.
Held-out test (1,954 messages)
Realistic messages (bank chats, KYC, rental agreements, insurance claims, WhatsApp, etc.) written by Qwen3.8-27B around fake PII values: 654 Hindi, 655 Hinglish, 645 English; 267 contain no PII. The model was chosen on a separate 500-message dev split; this split was run once.
| Model | Params | Recall | Precision | No-PII false alarms | Hindi recall | Exact F1 |
|---|---|---|---|---|---|---|
| altslate/indic-pii-redactor-small | 141M | 0.995 | 0.995 | 2.6% | 0.991 | 0.981 |
| Horizon-Labs/pii-redactor-small | 141M | 0.936 | 0.954 | 83% | 0.936 | – |
| Horizon-Labs/pii-redactor-base | 308M | 0.945 | 0.949 | 94% | 0.938 | – |
| urchade/gliner_multi_pii-v1 (threshold 0.5)* | ~280M | 0.830 | 0.908 | 44% | 0.626 | 0.696 |
| urchade/gliner_multi_pii-v1 (threshold 0.3)* | ~280M | 0.867 | 0.884 | 55% | 0.702 | 0.665 |
* GLiNER was prompted zero-shot with our 16 label names.
Caveat: the ID values inside these messages come from the same format generators used for training data (the surrounding text does not), so this test favours formats the model has seen. Read it together with the challenge set below.
Hand-written challenge set (47 messages)
Written by hand to use formats and styles outside the training generators: email signatures, OCR-style KYC forms in capitals, masked Aadhaar, company lines with CIN numbers, Hindi म.नं. addresses, lowercase chat names, Amazon-style order IDs. It was written before the final model existed and run once.
| Model | Recall | Precision | No-PII false alarms |
|---|---|---|---|
| altslate/indic-pii-redactor-small | 0.969 | 0.938 | 11% (2 of 19) |
| Horizon-Labs/pii-redactor-small | 0.911 | 0.810 | 74% |
| Horizon-Labs/pii-redactor-base | 0.896 | 0.784 | 84% |
| urchade/gliner_multi_pii-v1 (0.5) | 0.615 | 0.902 | 0% |
This set is small (58 PII spans), so treat these numbers as indicative.
Where other models are better
- Horizon's models catch every
DATE_OF_BIRTHon both sets (ours: 0.97 on the test, 0.71 on the challenge set), because they redact all dates. That is also why they have high false-alarm rates. - On the challenge set Horizon-small has higher Hindi recall (0.958 vs our 0.921).
Known limitations
- Company identifiers such as CIN (
U74999DL2015PTC123456) are not a label; the model may wrongly carve them into ID-like pieces. - Reference numbers that look like dates (
20261119-4456) can be flagged as Aadhaar. - Nicknames in lowercase chat (
tinku) can be missed; occasionally a Hinglish word is tagged as a name. - Dates of birth written in Devanagari with Hindi month names (
३ मार्च २०१२) are weaker than numeric dates. - Only English, Hindi and Hinglish were trained and tested. Other Indian languages and scripts (Bengali, Tamil, Telugu, etc.) are untested.
- Public figures' names in news-like text are treated inconsistently; the training text was filtered to avoid names, and the test sets don't measure this.
- About 2.5% of test messages contain an extra name the generator added without a label (found by hand-checking 40 messages), which slightly understates precision for every model.
Don't rely on this model alone where a missed identifier has legal or safety consequences; combine it with checksum validators for Aadhaar (Verhoeff) and GSTIN, and with human review.
Training
- Base:
jhu-clsp/mmBERT-small(MIT), fine-tuned for token classification (33 BIO tags). - Data: 300k synthetic examples. All PII values are fake but format-valid (Aadhaar Verhoeff and GSTIN checksums included), rendered with many real-world surface forms (lowercase, spaced, Devanagari digits, regrouped numbers, varied date formats). Mix: 35% template sentences and key-value forms, 30% PII clauses spliced into real sentences, 15% PII values inserted into real sentences with no cue word, 8% names in common chat and email frames, about 12% text with no PII (real sentences, non-birth dates, order/transaction/helpline/tracking numbers).
- Real text used as context and as no-PII examples (sentences containing names or long numbers were filtered out):
- findnitai/english-to-hinglish, Apache-2.0
- Abhishekcr448/Hinglish-Everyday-Conversations-1M, MIT
- Hindi Wikipedia via wikimedia/wikipedia (20231101.hi), CC-BY-SA-4.0
- No real personal data was used to train or test the model.
- Hyperparameters: 2 epochs, batch 32, max length 256, AdamW, lr 5e-5, linear schedule with 6% warmup, bf16. About 45 minutes on one NVIDIA GB10 (DGX Spark).
- Test-set generation: the held-out messages were written by Qwen3.8-27B, a model that wrote none of the training text. Labels come from exact string matches of the values we supplied; messages where the LLM dropped, repeated or invented ID-like values were discarded (98% kept).
License
Apache-2.0. The base model is MIT. Training used the datasets above under their own licenses; Hindi Wikipedia text is CC-BY-SA-4.0 (Wikipedia contributors).
- Downloads last month
- -
Model tree for altslate/indic-pii-redactor-small
Base model
jhu-clsp/mmBERT-small