Indic PII Redactor (small)

A 141M-parameter token-classification model that finds Indian personal data in English, Hindi and Hinglish text, so you can redact it before logging, analytics or sending text to an LLM.

It detects 16 types, including Indian identifiers that general PII models don't have:

Label What it covers
AADHAAR Aadhaar numbers (12 digits, any grouping, masked forms like XXXX XXXX 1234) and 16-digit VIDs
PAN PAN (ABCPS1234K), including lowercase and spaced forms
GSTIN 15-character GST numbers
IFSC Bank branch codes (SBIN0001234)
UPI_ID UPI handles (name@okhdfcbank, 9876543210@ybl)
BANK_ACCOUNT Bank account numbers
PHONE Mobile and landline numbers in Indian formats
VOTER_ID EPIC numbers, including the old MH/12/345/678901 format
PASSPORT Indian passport numbers
DRIVING_LICENCE Driving licence numbers
VEHICLE_REG Number plates, including BH-series and Devanagari plates
PINCODE PIN codes
PERSON Names (Latin and Devanagari)
ADDRESS Postal addresses, including village/tehsil and landmark styles
EMAIL Email addresses
DATE_OF_BIRTH Dates of birth (other dates are left alone)

It is trained to leave look-alike numbers alone: order IDs, transaction and UTR numbers, PNRs, OTPs, helplines, tracking numbers and policy numbers.

Usage

from transformers import pipeline

redactor = pipeline("token-classification", model="altslate/indic-pii-redactor-small",
                    aggregation_strategy="simple")

def spans(text):
    """(start, end, label) per entity; offsets trimmed of the leading space the tokenizer attaches."""
    out = []
    for e in redactor(text):
        s, t = e["start"], e["end"]
        while s < t and text[s].isspace():
            s += 1
        out.append((s, t, e["entity_group"]))
    return out

def redact(text):
    for s, t, label in sorted(spans(text), reverse=True):
        text = text[:s] + f"[{label}]" + text[t:]
    return text

text = "Mera aadhaar 2345 6789 0123 hai aur PAN ABCPS1234K, call 98765 43210"
print(redact(text))
# Mera aadhaar [AADHAAR] hai aur PAN [PAN], call [PHONE]

The pipeline's start offset (and word) can include the space before a token, hence the trim above.

ONNX (onnxruntime, transformers.js)

onnx/model.onnx (fp32, 563 MB) and onnx/model_quantized.onnx (dynamic int8, 142 MB) take input_ids and attention_mask and return logits. On the 500-message dev split:

Runtime Recall Precision No-PII false alarms
PyTorch 0.996 0.997 1.4%
ONNX fp32 0.996 0.997 1.4%
ONNX int8 0.995 0.996 4.1%
import { pipeline } from "@huggingface/transformers";
const tagger = await pipeline("token-classification", "altslate/indic-pii-redactor-small", { dtype: "q8" });
const tokens = await tagger("Mera aadhaar 2345 6789 0123 hai aur PAN ABCPS1234K");
// One entry per token with B-/I- labels, e.g. {entity: "B-AADHAAR", word: "2"}, {entity: "I-AADHAAR", word: "3"}, ...
// Join a B- token with the I- tokens that follow it to get whole entities.

Tested with @huggingface/transformers 4.3.2 in Node 22, dtype "fp32" and "q8".

Results

All numbers are measured on data the model never saw in training. Redaction recall/precision count PII characters (did the right characters get masked, whatever the type), which is what matters for redaction and is fair to models with different label sets. No-PII false alarms is the share of messages without any PII where the model redacts something. Exact F1 needs our label set, so it is shown only for models that can output it.

Held-out test (1,954 messages)

Realistic messages (bank chats, KYC, rental agreements, insurance claims, WhatsApp, etc.) written by Qwen3.8-27B around fake PII values: 654 Hindi, 655 Hinglish, 645 English; 267 contain no PII. The model was chosen on a separate 500-message dev split; this split was run once.

Model Params Recall Precision No-PII false alarms Hindi recall Exact F1
altslate/indic-pii-redactor-small 141M 0.995 0.995 2.6% 0.991 0.981
Horizon-Labs/pii-redactor-small 141M 0.936 0.954 83% 0.936 –
Horizon-Labs/pii-redactor-base 308M 0.945 0.949 94% 0.938 –
urchade/gliner_multi_pii-v1 (threshold 0.5)* ~280M 0.830 0.908 44% 0.626 0.696
urchade/gliner_multi_pii-v1 (threshold 0.3)* ~280M 0.867 0.884 55% 0.702 0.665

* GLiNER was prompted zero-shot with our 16 label names.

Caveat: the ID values inside these messages come from the same format generators used for training data (the surrounding text does not), so this test favours formats the model has seen. Read it together with the challenge set below.

Hand-written challenge set (47 messages)

Written by hand to use formats and styles outside the training generators: email signatures, OCR-style KYC forms in capitals, masked Aadhaar, company lines with CIN numbers, Hindi म.नं. addresses, lowercase chat names, Amazon-style order IDs. It was written before the final model existed and run once.

Model Recall Precision No-PII false alarms
altslate/indic-pii-redactor-small 0.969 0.938 11% (2 of 19)
Horizon-Labs/pii-redactor-small 0.911 0.810 74%
Horizon-Labs/pii-redactor-base 0.896 0.784 84%
urchade/gliner_multi_pii-v1 (0.5) 0.615 0.902 0%

This set is small (58 PII spans), so treat these numbers as indicative.

Where other models are better

  • Horizon's models catch every DATE_OF_BIRTH on both sets (ours: 0.97 on the test, 0.71 on the challenge set), because they redact all dates. That is also why they have high false-alarm rates.
  • On the challenge set Horizon-small has higher Hindi recall (0.958 vs our 0.921).

Known limitations

  • Company identifiers such as CIN (U74999DL2015PTC123456) are not a label; the model may wrongly carve them into ID-like pieces.
  • Reference numbers that look like dates (20261119-4456) can be flagged as Aadhaar.
  • Nicknames in lowercase chat (tinku) can be missed; occasionally a Hinglish word is tagged as a name.
  • Dates of birth written in Devanagari with Hindi month names (३ मार्च २०१२) are weaker than numeric dates.
  • Only English, Hindi and Hinglish were trained and tested. Other Indian languages and scripts (Bengali, Tamil, Telugu, etc.) are untested.
  • Public figures' names in news-like text are treated inconsistently; the training text was filtered to avoid names, and the test sets don't measure this.
  • About 2.5% of test messages contain an extra name the generator added without a label (found by hand-checking 40 messages), which slightly understates precision for every model.

Don't rely on this model alone where a missed identifier has legal or safety consequences; combine it with checksum validators for Aadhaar (Verhoeff) and GSTIN, and with human review.

Training

  • Base: jhu-clsp/mmBERT-small (MIT), fine-tuned for token classification (33 BIO tags).
  • Data: 300k synthetic examples. All PII values are fake but format-valid (Aadhaar Verhoeff and GSTIN checksums included), rendered with many real-world surface forms (lowercase, spaced, Devanagari digits, regrouped numbers, varied date formats). Mix: 35% template sentences and key-value forms, 30% PII clauses spliced into real sentences, 15% PII values inserted into real sentences with no cue word, 8% names in common chat and email frames, about 12% text with no PII (real sentences, non-birth dates, order/transaction/helpline/tracking numbers).
  • Real text used as context and as no-PII examples (sentences containing names or long numbers were filtered out):
  • No real personal data was used to train or test the model.
  • Hyperparameters: 2 epochs, batch 32, max length 256, AdamW, lr 5e-5, linear schedule with 6% warmup, bf16. About 45 minutes on one NVIDIA GB10 (DGX Spark).
  • Test-set generation: the held-out messages were written by Qwen3.8-27B, a model that wrote none of the training text. Labels come from exact string matches of the values we supplied; messages where the LLM dropped, repeated or invented ID-like values were discarded (98% kept).

License

Apache-2.0. The base model is MIT. Training used the datasets above under their own licenses; Hindi Wikipedia text is CC-BY-SA-4.0 (Wikipedia contributors).

Downloads last month
-
Safetensors
Model size
0.1B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for altslate/indic-pii-redactor-small

Quantized
(312)
this model