File size: 3,083 Bytes
b62e753 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 | ---
language:
- en
- hi
license: apache-2.0
tags:
- token-classification
- pii
- privacy
- redaction
- onnx
- quantized
- multilingual
- distilbert
datasets:
- synthetic
metrics:
- f1
- precision
- recall
pipeline_tag: token-classification
---
# LocShield-AI: Multilingual PII Detection & Token Classification Model
LocShield-AI is a privacy-first, offline-capable token classification model based on `distilbert-base-multilingual-cased`, optimized for contextual PII detection across English, Hindi (Devanagari script), and Romanized Hindi (Hinglish).
Part of the **LocalShield AI** ecosystem: [https://github.com/adityapande403/Loc-shield-AI](https://github.com/adityapande403/Loc-shield-AI)
## Model Details
- **Architecture**: `DistilBertForTokenClassification`
- **Base Encoder**: `distilbert-base-multilingual-cased` (~134M parameters)
- **Entities Covered**:
- `PERSON` (Individual names in English, Hindi, and Hinglish)
- `ADDRESS` (Street and postal addresses, landmarks, PIN locations)
- `ORGANIZATION` (Companies, banks, institutions)
- `LOCATION` (Cities, states, countries)
- **Labels**: BIO tagging schema (`O`, `B-PER`, `I-PER`, `B-ADDR`, `I-ADDR`, `B-ORG`, `I-ORG`, `B-LOC`, `I-LOC`)
- **Formats Available**:
- `model.safetensors` - PyTorch SafeTensors (514 MB)
- `model.onnx` - ONNX FP32 Graph (514 MB)
- `model_quantized.onnx` - Dynamic INT8 Quantized ONNX (128.9 MB, 74.9% compression)
## Performance & Quality
- **Validation Macro F1**: **0.9931**
- `PERSON`: F1 0.9867 (Precision: 0.9738, Recall: 1.0000)
- `ADDRESS`: F1 0.9988 (Precision: 0.9976, Recall: 1.0000)
- `ORGANIZATION`: F1 0.9968 (Precision: 1.0000, Recall: 0.9936)
- `LOCATION`: F1 0.9880 (Precision: 0.9763, Recall: 1.0000)
- **CPU Inference Latency (ONNX INT8)**:
- **P50 Latency**: **2.27 ms**
- **Throughput**: **428.3 requests/second** on CPU
- **Memory Footprint**: ~140 MB RSS
## Quickstart: Python (Transformers)
```python
from transformers import AutoTokenizer, AutoModelForTokenClassification, pipeline
tokenizer = AutoTokenizer.from_pretrained("Adityapande403/LocShield-AI")
model = AutoModelForTokenClassification.from_pretrained("Adityapande403/LocShield-AI")
classifier = pipeline("ner", model=model, tokenizer=tokenizer, aggregation_strategy="simple")
results = classifier("Customer Rahul Sharma reported from Bengaluru regarding account at HDFC Bank.")
print(results)
```
## Quickstart: LocalShield Engine (Recommended)
To run with deterministic validators (Aadhaar Verhoeff, Luhn cards, PAN, IFSC, etc.) and complete redaction:
```bash
git clone https://github.com/adityapande403/Loc-shield-AI.git
cd Loc-shield-AI
pip install -e .
```
```python
from localshield import LocalShield
shield = LocalShield.from_pretrained("Adityapande403/LocShield-AI")
result = shield.redact("Call Rahul Sharma at 9876543210 regarding PAN ABCPE1234F.")
print(result.redacted_text)
# Output: Call [PERSON] at [PHONE_NUMBER] regarding PAN [PAN].
```
## License
Apache-2.0
|