| ---
|
| language:
|
| - en
|
| - hi
|
| license: apache-2.0
|
| tags:
|
| - token-classification
|
| - pii
|
| - privacy
|
| - redaction
|
| - onnx
|
| - quantized
|
| - multilingual
|
| - distilbert
|
| datasets:
|
| - synthetic
|
| metrics:
|
| - f1
|
| - precision
|
| - recall
|
| pipeline_tag: token-classification
|
| ---
|
|
|
| # LocShield-AI: Multilingual PII Detection & Token Classification Model
|
|
|
| LocShield-AI is a privacy-first, offline-capable token classification model based on `distilbert-base-multilingual-cased`, optimized for contextual PII detection across English, Hindi (Devanagari script), and Romanized Hindi (Hinglish).
|
|
|
| Part of the **LocalShield AI** ecosystem: [https://github.com/adityapande403/Loc-shield-AI](https://github.com/adityapande403/Loc-shield-AI)
|
|
|
| ## Model Details
|
| - **Architecture**: `DistilBertForTokenClassification`
|
| - **Base Encoder**: `distilbert-base-multilingual-cased` (~134M parameters)
|
| - **Entities Covered**:
|
| - `PERSON` (Individual names in English, Hindi, and Hinglish)
|
| - `ADDRESS` (Street and postal addresses, landmarks, PIN locations)
|
| - `ORGANIZATION` (Companies, banks, institutions)
|
| - `LOCATION` (Cities, states, countries)
|
| - **Labels**: BIO tagging schema (`O`, `B-PER`, `I-PER`, `B-ADDR`, `I-ADDR`, `B-ORG`, `I-ORG`, `B-LOC`, `I-LOC`)
|
| - **Formats Available**:
|
| - `model.safetensors` - PyTorch SafeTensors (514 MB)
|
| - `model.onnx` - ONNX FP32 Graph (514 MB)
|
| - `model_quantized.onnx` - Dynamic INT8 Quantized ONNX (128.9 MB, 74.9% compression)
|
|
|
| ## Performance & Quality
|
| - **Validation Macro F1**: **0.9931**
|
| - `PERSON`: F1 0.9867 (Precision: 0.9738, Recall: 1.0000)
|
| - `ADDRESS`: F1 0.9988 (Precision: 0.9976, Recall: 1.0000)
|
| - `ORGANIZATION`: F1 0.9968 (Precision: 1.0000, Recall: 0.9936)
|
| - `LOCATION`: F1 0.9880 (Precision: 0.9763, Recall: 1.0000)
|
| - **CPU Inference Latency (ONNX INT8)**:
|
| - **P50 Latency**: **2.27 ms**
|
| - **Throughput**: **428.3 requests/second** on CPU
|
| - **Memory Footprint**: ~140 MB RSS
|
|
|
| ## Quickstart: Python (Transformers)
|
|
|
| ```python
|
| from transformers import AutoTokenizer, AutoModelForTokenClassification, pipeline
|
|
|
| tokenizer = AutoTokenizer.from_pretrained("Adityapande403/LocShield-AI")
|
| model = AutoModelForTokenClassification.from_pretrained("Adityapande403/LocShield-AI")
|
|
|
| classifier = pipeline("ner", model=model, tokenizer=tokenizer, aggregation_strategy="simple")
|
| results = classifier("Customer Rahul Sharma reported from Bengaluru regarding account at HDFC Bank.")
|
| print(results)
|
| ```
|
|
|
| ## Quickstart: LocalShield Engine (Recommended)
|
|
|
| To run with deterministic validators (Aadhaar Verhoeff, Luhn cards, PAN, IFSC, etc.) and complete redaction:
|
|
|
| ```bash
|
| git clone https://github.com/adityapande403/Loc-shield-AI.git
|
| cd Loc-shield-AI
|
| pip install -e .
|
| ```
|
|
|
| ```python
|
| from localshield import LocalShield
|
|
|
| shield = LocalShield.from_pretrained("Adityapande403/LocShield-AI")
|
| result = shield.redact("Call Rahul Sharma at 9876543210 regarding PAN ABCPE1234F.")
|
| print(result.redacted_text)
|
| # Output: Call [PERSON] at [PHONE_NUMBER] regarding PAN [PAN].
|
| ```
|
|
|
| ## License
|
| Apache-2.0
|
|
|
