README.md

metadata

language:
  - hi
  - te
  - bn
  - en
license: apache-2.0
tags:
  - code-mixing
  - indic
  - hinglish
  - telgish
  - bengali
  - pii-redaction
  - onnx
  - privacy
  - intent-classification
  - toxicity-moderation
pipeline_tag: text-classification
widget:
  - text: bhai mera refund kab aayega, UPI 9876543210@paytm pe transfer karo
    example_title: Hinglish Refund & UPI PII
  - text: parcel inka address Hyderabad lo undi, call this number 9848012345
    example_title: Telgish Order Tracking & Phone
  - text: payment fail hoye geche kintu taka keteche
    example_title: Benglish Payment Issue
  - text: mera order cancel kardo please mujhe nahi chahiye
    example_title: Hinglish Cancellation

IndicMix AI: Lightweight Code-Mixed Indic NLP & Privacy Gateway

IndicMix AI is an ultra-lightweight, high-throughput NLP engine and privacy-first sanitization middleware designed specifically for informal, code-mixed Indian languages (Hinglish, Telgish, Benglish, Indian English, and native Devanagari, Telugu, and Bengali scripts).

Designed for edge runtimes, customer support routing, and secure enterprise LLM gateways.

---

Key Features

1. Enterprise Privacy & PII Sanitization:
   • Bidirectional token redaction with numbered placeholders ([NAME_1], [PHONE_1], [UPI_1]).
   • Official Verhoeff Dihedral Group ($D_5$) Checksum Algorithm for Indian 12-digit Aadhaar verification.
   • Financial entity preservation (order IDs, currency amounts like ₹1500 are safely preserved).
2. Multi-Task Code-Mixed Intent Classifier:
   • 12 customer support intent categories calibrated via Temperature Scaling.
   • Robust Romanized and native script lexicon priors.
3. Safety-Guarded Text Normalization:
   • Dual safety guards: Entity preservation guard & Negation/Semantic polarity guard.
4. Edge-Optimized ONNX Runtime:
   • FP32 ONNX (877 KB) and INT8 Quantized ONNX (222 KB).
   • In-memory inference latency: 0.046 ms P50 (~47,000 req/s on CPU).

---

Empirical Benchmark Results

Measured on Windows 11 / Intel CPU:

Metric / Stage	Result
Model Size (INT8 ONNX)	222.8 KB (74% reduction vs FP32)
INT8 vs FP32 Agreement	99.3%
Model-Only Latency (P50)	0.046 ms (~47,600 req/s)
Full Pipeline Latency (P50)	0.685 ms (1,400 req/s)
Privacy Proxy Latency (P50)	0.069 ms (14,700 req/s)
Intent Macro-F1 (Stratified Test)	1.0000
PII Exact Match F1	1.0000

---

Quickstart

1. Using ONNX Runtime (Ultra-Fast CPU Inference)

import onnxruntime as ort
import json
import numpy as np

# Load quantized model (222 KB)
session = ort.InferenceSession("model_quantized.onnx")

with open("config.json") as f:
    config = json.load(f)
with open("vocab.json") as f:
    vocab = json.load(f)

text = "bhai mera refund kab aayega"
# Tokenize and run
# See infer.py for complete standalone inference

2. Using the Standalone Python Runner

git clone https://huggingface.co/Adityapande403/IndicMix-AI
cd IndicMix-AI
python infer.py

3. Using the indicmix Python SDK

from indicmix import EnterprisePrivacyProxy, IntentClassifier

# Anonymize PII before sending to any LLM
proxy = EnterprisePrivacyProxy()
anon_res = proxy.anonymize("Mera naam Rahul Verma hai, transfer ₹500 to rahul@okhdfcbank or 9876543210")
print(anon_res.sanitized_text)
# Output: Mera naam [NAME_1] hai, transfer ₹500 to [UPI_1] or [PHONE_1]

# De-anonymize LLM response
llm_reply = "Humne [NAME_1] ke account [UPI_1] par ₹500 bhej diye hain."
deanonymized = proxy.deanonymize(llm_reply, anon_res.session_id)
print(deanonymized)
# Output: Humne Rahul Verma ke account rahul@okhdfcbank par ₹500 bhej diye hain.

---

Intent Taxonomy

Index	Intent Class	Example Code-Mixed Text
0	account_access	"login password reset link bhejo"
1	cancellation_request	"order cancel cheyyandi vaddu"
2	complaint	"worst service delivery person was rude"
3	delivery_issue	"parcel inka raledu delay enduku"
4	greeting	"namaste bhai kaise ho"
5	order_status	"where is my parcel order kab tak aayega"
6	payment_issue	"taka keteche kintu order confirm hoyni"
7	product_question	"is this compatible with iPhone 15"
8	refund_request	"paise wapas karo UPI pe refund chahiye"
9	subscription_issue	"auto debit stop karo subscription cancel"
10	technical_support	"app crash ho raha hai opening time"
11	unknown	"random unclassifiable text"

---

Citation & License

• License: Apache 2.0
• GitHub Repository: https://github.com/adityapande403/Indic-mix-AI