Indic PII Redactor (base)

A 307M-parameter token-classification model that finds Indian personal data in English, Hindi, Hinglish, Marathi, Bengali, Tamil, Telugu, Gujarati and Kannada text, so you can redact it before logging, analytics or sending text to an LLM. It handles native-script digits (৯৮৭৬…, ௨௧௩…) and native-script names and addresses.

Try it in your browser: altslate/indic-pii-redactor Space. It runs the int8 ONNX model locally with transformers.js, so your text never leaves your device.

If you only need English, Hindi and Hinglish, the smaller altslate/indic-pii-redactor-small (141M) is slightly more accurate on those three and is half the size.

It detects 16 types:

Label What it covers
AADHAAR Aadhaar numbers (12 digits, any grouping, masked forms like XXXX XXXX 1234) and 16-digit VIDs
PAN PAN (ABCPS1234K), including lowercase and spaced forms
GSTIN 15-character GST numbers
IFSC Bank branch codes (SBIN0001234)
UPI_ID UPI handles (name@okhdfcbank, 9876543210@ybl)
BANK_ACCOUNT Bank account numbers
PHONE Mobile and landline numbers in Indian formats
VOTER_ID EPIC numbers
PASSPORT Indian passport numbers
DRIVING_LICENCE Driving licence numbers
VEHICLE_REG Number plates
PINCODE PIN codes
PERSON Names (Latin and native scripts)
ADDRESS Postal addresses
EMAIL Email addresses
DATE_OF_BIRTH Dates of birth (other dates are left alone)

It is trained to leave look-alike numbers alone: order IDs, bill and ticket numbers, transaction and UTR numbers, PNRs, OTPs, helplines, tracking and policy numbers.

Usage

from transformers import pipeline

redactor = pipeline("token-classification", model="altslate/indic-pii-redactor-base",
                    aggregation_strategy="simple")

def spans(text):
    """(start, end, label) per entity; offsets trimmed of the leading space the tokenizer attaches."""
    out = []
    for e in redactor(text):
        s, t = e["start"], e["end"]
        while s < t and text[s].isspace():
            s += 1
        out.append((s, t, e["entity_group"]))
    return out

def redact(text):
    for s, t, label in sorted(spans(text), reverse=True):
        text = text[:s] + f"[{label}]" + text[t:]
    return text

print(redact("আমার আধার নম্বর ২৩৪৫ ৬৭৮৯ ০১২৩ এবং ফোন 98765 43210"))
# আমার আধার নম্বর [AADHAAR] এবং ফোন [PHONE]
print(redact("நான் கார்த்திக் ராஜன், என் பான் ABCPS1234K."))
# நான் [PERSON], என் பான் [PAN].

ONNX (onnxruntime, transformers.js)

onnx/model.onnx (fp32, 1.2 GB) and onnx/model_quantized.onnx (dynamic int8, per-channel, 309 MB) take input_ids and attention_mask and return logits. Dev-split results:

Runtime en/hi/Hinglish dev (500) six-language dev (300) six-language dev, unseen names (300)
PyTorch 1.000 / 0.989 / 1.4% 0.980 / 0.994 / 0% 0.874 / 0.988 / 0%
ONNX fp32 1.000 / 0.989 / 1.4% 0.980 / 0.994 / 0% 0.874 / 0.988 / 0%
ONNX int8 1.000 / 0.988 / 1.4% 0.982 / 0.995 / 0% 0.872 / 0.991 / 0%

Each cell is recall / precision / no-PII FA. Plain (per-tensor) int8 quantization raised no-PII false alarms to 32% on the six-language dev split with this model, so the shipped file uses per-channel weights.

import { pipeline } from "@huggingface/transformers";
const tagger = await pipeline("token-classification", "altslate/indic-pii-redactor-base", { dtype: "q8" });
// Returns one entry per token with B-/I- labels; join a B- token with the I- tokens that follow it.

Tested with @huggingface/transformers 4.3.2 in Node 22, dtype "q8" and "fp32".

Results

All numbers come from data the model never saw in training. Redaction recall/precision count PII characters: were the right characters masked, whatever the type? That is what matters for redaction, and it is fair to models with different label sets. No-PII false alarms (FA) is the share of messages without any PII where the model redacts something. The model and its decision threshold were chosen on separate dev splits. The held-out-names test and the third challenge set were run once, on this revision. The other sets were first run on the previous revision (v6b) and are reported again here unchanged; they were not used for tuning.

Six-language test with unseen names (851 messages)

Messages written by Qwen3.8-27B around fake PII values, using names and cities that appear in no training list. 103 of them contain no PII.

Model Params Recall Precision No-PII FA bn gu kn mr ta te
altslate/indic-pii-redactor-base 307M 0.874 0.989 5.8% 0.916 0.882 0.880 0.906 0.808 0.863
previous revision (v6b) 307M 0.831 0.996 5.8% 0.870 0.862 0.840 0.880 0.776 0.774
Horizon-Labs/pii-redactor-base 308M 0.923 0.947 82% 0.929 0.921 0.923 0.949 0.923 0.902
Horizon-Labs/pii-redactor-small 141M 0.862 0.937 89% 0.829 0.884 0.895 0.946 0.845 0.793
urchade/gliner_multi_pii-v1 (0.5)* ~280M 0.512 0.908 23% 0.518 0.598 0.543 0.552 0.476 0.402

Language columns are recall. Horizon-base has higher recall here because it finds many more unseen native-script names (PERSON recall 0.88 vs our 0.43), but it redacts something in 82% of messages that contain no PII. Our IDs, phones, addresses and dates are found at 0.92–1.00.

Six-language test (1,427 messages: Marathi, Bengali, Tamil, Telugu, Gujarati, Kannada)

Same generator, but with names and cities taken from the same lists used in training; 197 messages without PII.

Model Params Recall Precision No-PII FA bn gu kn mr ta te
altslate/indic-pii-redactor-base 307M 0.980 0.987 2.5% 0.991 0.965 0.984 0.983 0.978 0.977
previous revision (v6b) 307M 0.967 0.989 13.7% 0.978 0.953 0.967 0.987 0.968 0.949
Horizon-Labs/pii-redactor-base 308M 0.926 0.943 85% 0.953 0.923 0.943 0.953 0.910 0.872
Horizon-Labs/pii-redactor-small 141M 0.886 0.938 89% 0.882 0.876 0.913 0.949 0.879 0.812
altslate/indic-pii-redactor-small 141M 0.797 0.978 37% 0.807 0.797 0.803 0.872 0.762 0.737
urchade/gliner_multi_pii-v1 (0.5)* ~280M 0.536 0.910 22% 0.543 0.586 0.581 0.564 0.480 0.466

English, Hindi and Hinglish test (1,954 messages)

Model Params Recall Precision No-PII FA Hindi recall
altslate/indic-pii-redactor-small 141M 0.995 0.995 2.6% 0.991
altslate/indic-pii-redactor-base 307M 0.995 0.988 1.5% 0.991
previous revision (v6b) 307M 0.991 0.992 1.5% 0.993
Horizon-Labs/pii-redactor-base 308M 0.945 0.949 94% 0.938
Horizon-Labs/pii-redactor-small 141M 0.936 0.954 83% 0.936
urchade/gliner_multi_pii-v1 (0.5)* ~280M 0.830 0.908 44% 0.626

* GLiNER was prompted zero-shot with our 16 label names.

Caveat: the ID values inside these LLM-written messages come from the same format generators used for training data (the surrounding text does not), so these tests favour formats the model has seen. Read them together with the hand-written sets below.

Hand-written challenge sets

Written by hand, before the model was trained, to use formats and styles outside the training generators.

Six languages, set 3 (36 messages, 40 PII spans, Tamil/Telugu heavy; run once): vocative and suffixed native names that are in no training list, 6-digit order, bill and ticket numbers as decoys, native digits.

Model Recall Precision No-PII FA
altslate/indic-pii-redactor-base 0.874 1.000 0% (0 of 14)
previous revision (v6b) 0.829 0.961 21%
Horizon-Labs/pii-redactor-base 0.789 0.917 50%
Horizon-Labs/pii-redactor-small 0.643 0.915 43%
urchade/gliner_multi_pii-v1 (0.5) 0.303 0.908 0%

Six languages, set 2 (36 messages, 42 PII spans): masked Aadhaar in native digits, native-script addresses, train numbers and OTPs as decoys.

Model Recall Precision No-PII FA
altslate/indic-pii-redactor-base 0.947 0.982 8% (1 of 12)
previous revision (v6b) 0.961 0.982 8%
Horizon-Labs/pii-redactor-base 0.839 0.888 67%
Horizon-Labs/pii-redactor-small 0.790 0.909 58%
altslate/indic-pii-redactor-small 0.562 0.933 33%
urchade/gliner_multi_pii-v1 (0.5) 0.401 0.853 25%

English, Hindi and Hinglish (47 messages, 58 spans): email signatures, OCR-style forms in capitals, CIN company numbers, Hindi म.नं. addresses. Note that this set was already used to report the small model, so it had been seen when this model was built. It was not used to tune this model.

Model Recall Precision No-PII FA
altslate/indic-pii-redactor-base 0.980 0.906 37% (7 of 19)
previous revision (v6b) 0.990 0.955 16% (3 of 19)
altslate/indic-pii-redactor-small 0.969 0.938 11% (2 of 19)
Horizon-Labs/pii-redactor-small 0.911 0.810 74%
Horizon-Labs/pii-redactor-base 0.896 0.784 84%

These sets are small, so treat their numbers as indicative.

Where other models are better

  • Unseen native-script names: Horizon-base finds far more of them (PERSON recall 0.88 vs our 0.43 on the unseen-names test, 0.77 vs 0.56 on challenge set 3, 0.94 vs 0.69 on set 2). On names like those in training, ours is higher (0.98 vs 0.92).
  • On English/Hindi/Hinglish, our small model has higher precision and fewer false alarms on the hand-written set (11% vs 37%).
  • The previous revision (v6b) had fewer false alarms on the English/Hindi hand-written set, and higher PIN-code recall. To use it, load it with revision="d0ef0a1ae22db5fd94d704d7fb11213f661e339d".
  • Horizon's models catch every date of birth, because they redact all dates; that is also why their false-alarm rates are high.

Known limitations

  • Native-script names it has not seen are the main weakness: on our unseen-names test only 43% of name characters are found (Tamil is weakest). Common names are found well (0.98). If names matter most to you, combine this model with a name list or a recall-oriented model.
  • Tamil is the weakest language (0.81 recall on the unseen-names test).
  • PIN codes are sometimes left alone (recall 0.88–0.92 on the six-language tests), a side effect of training it not to flag 6-digit order and bill numbers.
  • On English/Hindi forms and signatures (hand-written set) it raises more false alarms than the previous revision and the small model (7 of 19 no-PII messages).
  • The Marathi, Bengali, Tamil, Telugu, Gujarati and Kannada templates, names and decoy sentences used for training were written by us and have not been checked by native speakers. The test messages were written by an LLM; we checked their script, not their fluency.
  • Other Indian languages (Malayalam, Odia, Punjabi, Assamese, Urdu, …) were not trained or tested.
  • The decision threshold is built into the weights (the bias of the O class is lowered by 7). If you need fewer false alarms and can accept lower recall, add back up to +7 to classifier.bias[0].

Don't rely on this model alone where a missed identifier has legal or safety consequences. Combine it with checksum validators for Aadhaar (Verhoeff) and GSTIN, and with human review.

Training

  • Base: jhu-clsp/mmBERT-base (MIT), fine-tuned for token classification (33 BIO tags).
  • Data: 450k synthetic examples: 70% English/Hindi/Hinglish (the same recipe as the small model) and 30% the six new languages, where most examples place a name inside an ordinary sentence, including vocative names and names with case suffixes. All PII values are fake but format-valid (Aadhaar Verhoeff and GSTIN checksums included); 30% of numeric values in the new languages use native digits. The new-language part mixes native templates, PII spliced into real Wikipedia sentences, values inserted without cue words, names inside ordinary sentences, and number-heavy text without PII (orders, bills, tickets, policies, 6-digit reference numbers).
  • Real text used as context and as no-PII examples (sentences containing names were filtered out):
  • No real personal data was used to train or test the model.
  • Hyperparameters: 2 epochs, batch 32, max length 256, AdamW, lr 5e-5, linear schedule with 6% warmup, bf16. About 2.5 hours on one NVIDIA GB10 (DGX Spark). The O-class bias (−7) was chosen on the dev splits as the largest value that kept no-PII false alarms at 0% on both six-language dev splits.
  • Test-set generation: the held-out messages were written by Qwen3.8-27B, a model that wrote none of the training text. Labels come from exact string matches of the values we supplied; messages where the LLM dropped or invented ID-like values, or wrote in the wrong script, were discarded (96% kept).

License

Apache-2.0. The base model is MIT. Training used the datasets above under their own licenses; Wikipedia text is CC-BY-SA-4.0 (Wikipedia contributors).

Downloads last month
64
Safetensors
Model size
0.3B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for altslate/indic-pii-redactor-base

Quantized
(302)
this model

Space using altslate/indic-pii-redactor-base 1

Collection including altslate/indic-pii-redactor-base