| ---
|
| language:
|
| - hi
|
| - te
|
| - bn
|
| - en
|
| license: apache-2.0
|
| tags:
|
| - code-mixing
|
| - indic
|
| - hinglish
|
| - telgish
|
| - bengali
|
| - pii-redaction
|
| - onnx
|
| - privacy
|
| - intent-classification
|
| - toxicity-moderation
|
| pipeline_tag: text-classification
|
| widget:
|
| - text: "bhai mera refund kab aayega, UPI 9876543210@paytm pe transfer karo"
|
| example_title: "Hinglish Refund & UPI PII"
|
| - text: "parcel inka address Hyderabad lo undi, call this number 9848012345"
|
| example_title: "Telgish Order Tracking & Phone"
|
| - text: "payment fail hoye geche kintu taka keteche"
|
| example_title: "Benglish Payment Issue"
|
| - text: "mera order cancel kardo please mujhe nahi chahiye"
|
| example_title: "Hinglish Cancellation"
|
| ---
|
|
|
| # IndicMix AI: Lightweight Code-Mixed Indic NLP & Privacy Gateway
|
|
|
| **IndicMix AI** is an ultra-lightweight, high-throughput NLP engine and privacy-first sanitization middleware designed specifically for informal, code-mixed Indian languages (**Hinglish**, **Telgish**, **Benglish**, **Indian English**, and native Devanagari, Telugu, and Bengali scripts).
|
|
|
| Designed for edge runtimes, customer support routing, and secure enterprise LLM gateways.
|
|
|
| ---
|
|
|
| ## Key Features
|
|
|
| 1. **Enterprise Privacy & PII Sanitization**:
|
| - Bidirectional token redaction with numbered placeholders (`[NAME_1]`, `[PHONE_1]`, `[UPI_1]`).
|
| - Official **Verhoeff Dihedral Group ($D_5$) Checksum Algorithm** for Indian 12-digit Aadhaar verification.
|
| - Financial entity preservation (order IDs, currency amounts like `₹1500` are safely preserved).
|
| 2. **Multi-Task Code-Mixed Intent Classifier**:
|
| - 12 customer support intent categories calibrated via Temperature Scaling.
|
| - Robust Romanized and native script lexicon priors.
|
| 3. **Safety-Guarded Text Normalization**:
|
| - Dual safety guards: Entity preservation guard & Negation/Semantic polarity guard.
|
| 4. **Edge-Optimized ONNX Runtime**:
|
| - FP32 ONNX (877 KB) and INT8 Quantized ONNX (222 KB).
|
| - In-memory inference latency: **0.046 ms P50** (~47,000 req/s on CPU).
|
|
|
| ---
|
|
|
| ## Empirical Benchmark Results
|
|
|
| Measured on Windows 11 / Intel CPU:
|
|
|
| | Metric / Stage | Result |
|
| | :--- | :--- |
|
| | **Model Size (INT8 ONNX)** | **222.8 KB** (74% reduction vs FP32) |
|
| | **INT8 vs FP32 Agreement** | **99.3%** |
|
| | **Model-Only Latency (P50)** | **0.046 ms** (~47,600 req/s) |
|
| | **Full Pipeline Latency (P50)** | **0.685 ms** (1,400 req/s) |
|
| | **Privacy Proxy Latency (P50)** | **0.069 ms** (14,700 req/s) |
|
| | **Intent Macro-F1 (Stratified Test)** | **1.0000** |
|
| | **PII Exact Match F1** | **1.0000** |
|
|
|
| ---
|
|
|
| ## Quickstart
|
|
|
| ### 1. Using ONNX Runtime (Ultra-Fast CPU Inference)
|
|
|
| ```python
|
| import onnxruntime as ort
|
| import json
|
| import numpy as np
|
|
|
| # Load quantized model (222 KB)
|
| session = ort.InferenceSession("model_quantized.onnx")
|
|
|
| with open("config.json") as f:
|
| config = json.load(f)
|
| with open("vocab.json") as f:
|
| vocab = json.load(f)
|
|
|
| text = "bhai mera refund kab aayega"
|
| # Tokenize and run
|
| # See infer.py for complete standalone inference
|
| ```
|
|
|
| ### 2. Using the Standalone Python Runner
|
|
|
| ```bash
|
| git clone https://huggingface.co/Adityapande403/IndicMix-AI
|
| cd IndicMix-AI
|
| python infer.py
|
| ```
|
|
|
| ### 3. Using the `indicmix` Python SDK
|
|
|
| ```python
|
| from indicmix import EnterprisePrivacyProxy, IntentClassifier
|
|
|
| # Anonymize PII before sending to any LLM
|
| proxy = EnterprisePrivacyProxy()
|
| anon_res = proxy.anonymize("Mera naam Rahul Verma hai, transfer ₹500 to rahul@okhdfcbank or 9876543210")
|
| print(anon_res.sanitized_text)
|
| # Output: Mera naam [NAME_1] hai, transfer ₹500 to [UPI_1] or [PHONE_1]
|
|
|
| # De-anonymize LLM response
|
| llm_reply = "Humne [NAME_1] ke account [UPI_1] par ₹500 bhej diye hain."
|
| deanonymized = proxy.deanonymize(llm_reply, anon_res.session_id)
|
| print(deanonymized)
|
| # Output: Humne Rahul Verma ke account rahul@okhdfcbank par ₹500 bhej diye hain.
|
| ```
|
|
|
| ---
|
|
|
| ## Intent Taxonomy
|
|
|
| | Index | Intent Class | Example Code-Mixed Text |
|
| | :--- | :--- | :--- |
|
| | 0 | `account_access` | "login password reset link bhejo" |
|
| | 1 | `cancellation_request` | "order cancel cheyyandi vaddu" |
|
| | 2 | `complaint` | "worst service delivery person was rude" |
|
| | 3 | `delivery_issue` | "parcel inka raledu delay enduku" |
|
| | 4 | `greeting` | "namaste bhai kaise ho" |
|
| | 5 | `order_status` | "where is my parcel order kab tak aayega" |
|
| | 6 | `payment_issue` | "taka keteche kintu order confirm hoyni" |
|
| | 7 | `product_question` | "is this compatible with iPhone 15" |
|
| | 8 | `refund_request` | "paise wapas karo UPI pe refund chahiye" |
|
| | 9 | `subscription_issue` | "auto debit stop karo subscription cancel" |
|
| | 10 | `technical_support` | "app crash ho raha hai opening time" |
|
| | 11 | `unknown` | "random unclassifiable text" |
|
|
|
| ---
|
|
|
| ## Citation & License
|
|
|
| - **License**: Apache 2.0
|
| - **GitHub Repository**: [https://github.com/adityapande403/Indic-mix-AI](https://github.com/adityapande403/Indic-mix-AI)
|
| |
