Instructions to use altslate/indic-pii-redactor-base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use altslate/indic-pii-redactor-base with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("token-classification", model="altslate/indic-pii-redactor-base")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForTokenClassification tokenizer = AutoTokenizer.from_pretrained("altslate/indic-pii-redactor-base") model = AutoModelForTokenClassification.from_pretrained("altslate/indic-pii-redactor-base", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Indic PII Redactor (base)
A 307M-parameter token-classification model that finds Indian personal data in English, Hindi, Hinglish, Marathi, Bengali, Tamil, Telugu, Gujarati and Kannada text, so you can redact it before logging, analytics or sending text to an LLM. It handles native-script digits (৯৮৭৬…, ௨௧௩…) and native-script names and addresses.
Try it in your browser: altslate/indic-pii-redactor Space. It runs the int8 ONNX model locally with transformers.js, so your text never leaves your device.
If you only need English, Hindi and Hinglish, the smaller altslate/indic-pii-redactor-small (141M) is slightly more accurate on those three and is half the size.
It detects 16 types:
| Label | What it covers |
|---|---|
AADHAAR |
Aadhaar numbers (12 digits, any grouping, masked forms like XXXX XXXX 1234) and 16-digit VIDs |
PAN |
PAN (ABCPS1234K), including lowercase and spaced forms |
GSTIN |
15-character GST numbers |
IFSC |
Bank branch codes (SBIN0001234) |
UPI_ID |
UPI handles (name@okhdfcbank, 9876543210@ybl) |
BANK_ACCOUNT |
Bank account numbers |
PHONE |
Mobile and landline numbers in Indian formats |
VOTER_ID |
EPIC numbers |
PASSPORT |
Indian passport numbers |
DRIVING_LICENCE |
Driving licence numbers |
VEHICLE_REG |
Number plates |
PINCODE |
PIN codes |
PERSON |
Names (Latin and native scripts) |
ADDRESS |
Postal addresses |
EMAIL |
Email addresses |
DATE_OF_BIRTH |
Dates of birth (other dates are left alone) |
It is trained to leave look-alike numbers alone: order IDs, bill and ticket numbers, transaction and UTR numbers, PNRs, OTPs, helplines, tracking and policy numbers.
Usage
from transformers import pipeline
redactor = pipeline("token-classification", model="altslate/indic-pii-redactor-base",
aggregation_strategy="simple")
def spans(text):
"""(start, end, label) per entity; offsets trimmed of the leading space the tokenizer attaches."""
out = []
for e in redactor(text):
s, t = e["start"], e["end"]
while s < t and text[s].isspace():
s += 1
out.append((s, t, e["entity_group"]))
return out
def redact(text):
for s, t, label in sorted(spans(text), reverse=True):
text = text[:s] + f"[{label}]" + text[t:]
return text
print(redact("আমার আধার নম্বর ২৩৪৫ ৬৭৮৯ ০১২৩ এবং ফোন 98765 43210"))
# আমার আধার নম্বর [AADHAAR] এবং ফোন [PHONE]
print(redact("நான் கார்த்திக் ராஜன், என் பான் ABCPS1234K."))
# நான் [PERSON], என் பான் [PAN].
ONNX (onnxruntime, transformers.js)
onnx/model.onnx (fp32, 1.2 GB) and onnx/model_quantized.onnx (dynamic int8, per-channel, 309 MB) take input_ids and attention_mask and return logits. Dev-split results:
| Runtime | en/hi/Hinglish dev (500) | six-language dev (300) | six-language dev, unseen names (300) |
|---|---|---|---|
| PyTorch | 1.000 / 0.989 / 1.4% | 0.980 / 0.994 / 0% | 0.874 / 0.988 / 0% |
| ONNX fp32 | 1.000 / 0.989 / 1.4% | 0.980 / 0.994 / 0% | 0.874 / 0.988 / 0% |
| ONNX int8 | 1.000 / 0.988 / 1.4% | 0.982 / 0.995 / 0% | 0.872 / 0.991 / 0% |
Each cell is recall / precision / no-PII FA. Plain (per-tensor) int8 quantization raised no-PII false alarms to 32% on the six-language dev split with this model, so the shipped file uses per-channel weights.
import { pipeline } from "@huggingface/transformers";
const tagger = await pipeline("token-classification", "altslate/indic-pii-redactor-base", { dtype: "q8" });
// Returns one entry per token with B-/I- labels; join a B- token with the I- tokens that follow it.
Tested with @huggingface/transformers 4.3.2 in Node 22, dtype "q8" and "fp32".
Results
All numbers come from data the model never saw in training. Redaction recall/precision count PII characters: were the right characters masked, whatever the type? That is what matters for redaction, and it is fair to models with different label sets. No-PII false alarms (FA) is the share of messages without any PII where the model redacts something. The model and its decision threshold were chosen on separate dev splits. The held-out-names test and the third challenge set were run once, on this revision. The other sets were first run on the previous revision (v6b) and are reported again here unchanged; they were not used for tuning.
Six-language test with unseen names (851 messages)
Messages written by Qwen3.8-27B around fake PII values, using names and cities that appear in no training list. 103 of them contain no PII.
| Model | Params | Recall | Precision | No-PII FA | bn | gu | kn | mr | ta | te |
|---|---|---|---|---|---|---|---|---|---|---|
| altslate/indic-pii-redactor-base | 307M | 0.874 | 0.989 | 5.8% | 0.916 | 0.882 | 0.880 | 0.906 | 0.808 | 0.863 |
| previous revision (v6b) | 307M | 0.831 | 0.996 | 5.8% | 0.870 | 0.862 | 0.840 | 0.880 | 0.776 | 0.774 |
| Horizon-Labs/pii-redactor-base | 308M | 0.923 | 0.947 | 82% | 0.929 | 0.921 | 0.923 | 0.949 | 0.923 | 0.902 |
| Horizon-Labs/pii-redactor-small | 141M | 0.862 | 0.937 | 89% | 0.829 | 0.884 | 0.895 | 0.946 | 0.845 | 0.793 |
| urchade/gliner_multi_pii-v1 (0.5)* | ~280M | 0.512 | 0.908 | 23% | 0.518 | 0.598 | 0.543 | 0.552 | 0.476 | 0.402 |
Language columns are recall. Horizon-base has higher recall here because it finds many more unseen native-script names (PERSON recall 0.88 vs our 0.43), but it redacts something in 82% of messages that contain no PII. Our IDs, phones, addresses and dates are found at 0.92–1.00.
Six-language test (1,427 messages: Marathi, Bengali, Tamil, Telugu, Gujarati, Kannada)
Same generator, but with names and cities taken from the same lists used in training; 197 messages without PII.
| Model | Params | Recall | Precision | No-PII FA | bn | gu | kn | mr | ta | te |
|---|---|---|---|---|---|---|---|---|---|---|
| altslate/indic-pii-redactor-base | 307M | 0.980 | 0.987 | 2.5% | 0.991 | 0.965 | 0.984 | 0.983 | 0.978 | 0.977 |
| previous revision (v6b) | 307M | 0.967 | 0.989 | 13.7% | 0.978 | 0.953 | 0.967 | 0.987 | 0.968 | 0.949 |
| Horizon-Labs/pii-redactor-base | 308M | 0.926 | 0.943 | 85% | 0.953 | 0.923 | 0.943 | 0.953 | 0.910 | 0.872 |
| Horizon-Labs/pii-redactor-small | 141M | 0.886 | 0.938 | 89% | 0.882 | 0.876 | 0.913 | 0.949 | 0.879 | 0.812 |
| altslate/indic-pii-redactor-small | 141M | 0.797 | 0.978 | 37% | 0.807 | 0.797 | 0.803 | 0.872 | 0.762 | 0.737 |
| urchade/gliner_multi_pii-v1 (0.5)* | ~280M | 0.536 | 0.910 | 22% | 0.543 | 0.586 | 0.581 | 0.564 | 0.480 | 0.466 |
English, Hindi and Hinglish test (1,954 messages)
| Model | Params | Recall | Precision | No-PII FA | Hindi recall |
|---|---|---|---|---|---|
| altslate/indic-pii-redactor-small | 141M | 0.995 | 0.995 | 2.6% | 0.991 |
| altslate/indic-pii-redactor-base | 307M | 0.995 | 0.988 | 1.5% | 0.991 |
| previous revision (v6b) | 307M | 0.991 | 0.992 | 1.5% | 0.993 |
| Horizon-Labs/pii-redactor-base | 308M | 0.945 | 0.949 | 94% | 0.938 |
| Horizon-Labs/pii-redactor-small | 141M | 0.936 | 0.954 | 83% | 0.936 |
| urchade/gliner_multi_pii-v1 (0.5)* | ~280M | 0.830 | 0.908 | 44% | 0.626 |
* GLiNER was prompted zero-shot with our 16 label names.
Caveat: the ID values inside these LLM-written messages come from the same format generators used for training data (the surrounding text does not), so these tests favour formats the model has seen. Read them together with the hand-written sets below.
Hand-written challenge sets
Written by hand, before the model was trained, to use formats and styles outside the training generators.
Six languages, set 3 (36 messages, 40 PII spans, Tamil/Telugu heavy; run once): vocative and suffixed native names that are in no training list, 6-digit order, bill and ticket numbers as decoys, native digits.
| Model | Recall | Precision | No-PII FA |
|---|---|---|---|
| altslate/indic-pii-redactor-base | 0.874 | 1.000 | 0% (0 of 14) |
| previous revision (v6b) | 0.829 | 0.961 | 21% |
| Horizon-Labs/pii-redactor-base | 0.789 | 0.917 | 50% |
| Horizon-Labs/pii-redactor-small | 0.643 | 0.915 | 43% |
| urchade/gliner_multi_pii-v1 (0.5) | 0.303 | 0.908 | 0% |
Six languages, set 2 (36 messages, 42 PII spans): masked Aadhaar in native digits, native-script addresses, train numbers and OTPs as decoys.
| Model | Recall | Precision | No-PII FA |
|---|---|---|---|
| altslate/indic-pii-redactor-base | 0.947 | 0.982 | 8% (1 of 12) |
| previous revision (v6b) | 0.961 | 0.982 | 8% |
| Horizon-Labs/pii-redactor-base | 0.839 | 0.888 | 67% |
| Horizon-Labs/pii-redactor-small | 0.790 | 0.909 | 58% |
| altslate/indic-pii-redactor-small | 0.562 | 0.933 | 33% |
| urchade/gliner_multi_pii-v1 (0.5) | 0.401 | 0.853 | 25% |
English, Hindi and Hinglish (47 messages, 58 spans): email signatures, OCR-style forms in capitals, CIN company numbers, Hindi म.नं. addresses. Note that this set was already used to report the small model, so it had been seen when this model was built. It was not used to tune this model.
| Model | Recall | Precision | No-PII FA |
|---|---|---|---|
| altslate/indic-pii-redactor-base | 0.980 | 0.906 | 37% (7 of 19) |
| previous revision (v6b) | 0.990 | 0.955 | 16% (3 of 19) |
| altslate/indic-pii-redactor-small | 0.969 | 0.938 | 11% (2 of 19) |
| Horizon-Labs/pii-redactor-small | 0.911 | 0.810 | 74% |
| Horizon-Labs/pii-redactor-base | 0.896 | 0.784 | 84% |
These sets are small, so treat their numbers as indicative.
Where other models are better
- Unseen native-script names: Horizon-base finds far more of them (PERSON recall 0.88 vs our 0.43 on the unseen-names test, 0.77 vs 0.56 on challenge set 3, 0.94 vs 0.69 on set 2). On names like those in training, ours is higher (0.98 vs 0.92).
- On English/Hindi/Hinglish, our small model has higher precision and fewer false alarms on the hand-written set (11% vs 37%).
- The previous revision (v6b) had fewer false alarms on the English/Hindi hand-written set, and higher PIN-code recall. To use it, load it with
revision="d0ef0a1ae22db5fd94d704d7fb11213f661e339d". - Horizon's models catch every date of birth, because they redact all dates; that is also why their false-alarm rates are high.
Known limitations
- Native-script names it has not seen are the main weakness: on our unseen-names test only 43% of name characters are found (Tamil is weakest). Common names are found well (0.98). If names matter most to you, combine this model with a name list or a recall-oriented model.
- Tamil is the weakest language (0.81 recall on the unseen-names test).
- PIN codes are sometimes left alone (recall 0.88–0.92 on the six-language tests), a side effect of training it not to flag 6-digit order and bill numbers.
- On English/Hindi forms and signatures (hand-written set) it raises more false alarms than the previous revision and the small model (7 of 19 no-PII messages).
- The Marathi, Bengali, Tamil, Telugu, Gujarati and Kannada templates, names and decoy sentences used for training were written by us and have not been checked by native speakers. The test messages were written by an LLM; we checked their script, not their fluency.
- Other Indian languages (Malayalam, Odia, Punjabi, Assamese, Urdu, …) were not trained or tested.
- The decision threshold is built into the weights (the bias of the
Oclass is lowered by 7). If you need fewer false alarms and can accept lower recall, add back up to +7 toclassifier.bias[0].
Don't rely on this model alone where a missed identifier has legal or safety consequences. Combine it with checksum validators for Aadhaar (Verhoeff) and GSTIN, and with human review.
Training
- Base:
jhu-clsp/mmBERT-base(MIT), fine-tuned for token classification (33 BIO tags). - Data: 450k synthetic examples: 70% English/Hindi/Hinglish (the same recipe as the small model) and 30% the six new languages, where most examples place a name inside an ordinary sentence, including vocative names and names with case suffixes. All PII values are fake but format-valid (Aadhaar Verhoeff and GSTIN checksums included); 30% of numeric values in the new languages use native digits. The new-language part mixes native templates, PII spliced into real Wikipedia sentences, values inserted without cue words, names inside ordinary sentences, and number-heavy text without PII (orders, bills, tickets, policies, 6-digit reference numbers).
- Real text used as context and as no-PII examples (sentences containing names were filtered out):
- findnitai/english-to-hinglish, Apache-2.0
- Abhishekcr448/Hinglish-Everyday-Conversations-1M, MIT
- Wikipedia in Hindi, Marathi, Bengali, Tamil, Telugu, Gujarati and Kannada via wikimedia/wikipedia (20231101), CC-BY-SA-4.0
- No real personal data was used to train or test the model.
- Hyperparameters: 2 epochs, batch 32, max length 256, AdamW, lr 5e-5, linear schedule with 6% warmup, bf16. About 2.5 hours on one NVIDIA GB10 (DGX Spark). The O-class bias (−7) was chosen on the dev splits as the largest value that kept no-PII false alarms at 0% on both six-language dev splits.
- Test-set generation: the held-out messages were written by Qwen3.8-27B, a model that wrote none of the training text. Labels come from exact string matches of the values we supplied; messages where the LLM dropped or invented ID-like values, or wrote in the wrong script, were discarded (96% kept).
License
Apache-2.0. The base model is MIT. Training used the datasets above under their own licenses; Wikipedia text is CC-BY-SA-4.0 (Wikipedia contributors).
- Downloads last month
- 64
Model tree for altslate/indic-pii-redactor-base
Base model
jhu-clsp/mmBERT-base