README.md

---
language:
- hi
- bn
- ta
- te
- mr
- gu
- kn
- ml
- or
- pa
- as
- ur
- sa
- ne
- mai
- mni
- brx
- sat
- gom
- lus
- kha
- awa
- bho
- hne
- gbm
- bgc
- mwr
- en
license: cc-by-4.0
tags:
- indic-nlp
- language-identification
- sentiment-analysis
- text-classification
- onnx
pipeline_tag: text-classification
metrics:
- accuracy
- f1
---

# Indic-Lite: Unified Indic NLP Suite

**Indic-Lite** is a compact, high-efficiency Indic NLP toolkit designed for edge, mobile, and server environments. It provides lightweight models and text utilities:

1. **Indic-LID**: 28-Language Identification (covering Northeast languages, all 22 Eighth Schedule languages, regional dialects, and English)
2. **Indic-Sentiment**: 12-Language Sentiment Classifier
3. **Indic-Topic**: 6-Class Regional News & Headline Classifier
4. **Indic-Clean**: Rule-based Unicode NFC normalization, Indic numeral translation (e.g. १२३ → 123), and PII scrubbing.

---

## Supported Languages

### 1. `Indic-LID` (Language Identification — 28 Languages)

#### **Northeast Indian Languages (6)**
* **Assamese** (`as`) — Assam *(Bengali-Assamese script)*
* **Bodo** (`brx`) — Bodoland / Assam *(Devanagari script)*
* **Khasi** (`kha`) — Meghalaya *(Latin script)*
* **Manipuri / Meitei** (`mni`) — Manipur *(Meetei Mayek & Bengali script)*
* **Mizo** (`lus`) — Mizoram *(Latin script)*
* **Santali** (`sat`) — Assam, Tripura, Odisha *(Ol Chiki script)*

#### **Scheduled National Languages (15)**
* **Bengali** (`bn`) — West Bengal, Tripura
* **Gujarati** (`gu`) — Gujarat
* **Hindi** (`hi`) — Central & Northern India
* **Kannada** (`kn`) — Karnataka
* **Konkani** (`gom`) — Goa, Maharashtra, Karnataka
* **Maithili** (`mai`) — Bihar, Jharkhand
* **Malayalam** (`ml`) — Kerala
* **Marathi** (`mr`) — Maharashtra
* **Nepali** (`ne`) — Sikkim, West Bengal
* **Odia** (`or`) — Odisha
* **Punjabi** (`pa`) — Punjab
* **Sanskrit** (`sa`) — Pan-India
* **Tamil** (`ta`) — Tamil Nadu
* **Telugu** (`te`) — Andhra Pradesh, Telangana
* **Urdu** (`ur`) — Pan-India

#### **Regional Dialects & Literary Varieties (6)**
* **Awadhi** (`awa`) — Uttar Pradesh
* **Bhojpuri** (`bho`) — Bihar, Uttar Pradesh
* **Chhattisgarhi** (`hne`) — Chhattisgarh
* **Garhwali** (`gbm`) — Uttarakhand
* **Haryanvi** (`bgc`) — Haryana
* **Marwari** (`mwr`) — Rajasthan

#### **Associate Official Language (1)**
* **English** (`en`) — India-wide

---

### 2. `Indic-Sentiment` (Sentiment Analysis — 12 Languages)
* **Assamese** (`as`), **Bengali** (`bn`), **Gujarati** (`gu`), **Hindi** (`hi`), **Kannada** (`kn`), **Malayalam** (`ml`), **Marathi** (`mr`), **Odia** (`or`), **Punjabi** (`pa`), **Tamil** (`ta`), **Telugu** (`te`), **Urdu** (`ur`).

---

### 3. `Indic-Topic` (News & Headline Categorization — 6 Languages)
* **Gujarati** (`gu`), **Hindi** (`hi`), **Malayalam** (`ml`), **Marathi** (`mr`), **Tamil** (`ta`), **Telugu** (`te`).
* **Categories (6):** `entertainment`, `business`, `tech`, `sports`, `state`, `spirituality`.

---

## Benchmark Evaluation

Evaluated on standardized held-out test splits:

| Model | Task / Coverage | Test Set Size | Accuracy | Macro F1 | Joblib Size | ONNX Size | Latency / Item |
|---|---|---|---|---|---|---|---|
| **Indic-LID** | Language ID (28 languages) | 27,312 | **99.53%** | **99.54%** | 10.83 MB | **4.27 MB** | **0.155 ms** |
| **Indic-Sentiment** | Sentiment (12 languages) | 10,129 | **75.84%** | **75.84%** | 1.64 MB | **0.19 MB** | **0.164 ms** |
| **Indic-Topic** | Topic (6 categories) | 3,244 | **90.23%** | **89.35%** | 3.41 MB | **1.03 MB** | **0.135 ms** |

---

## Quickstart

### Installation
```bash
pip install joblib onnxruntime scikit-learn
```

### 1. Python Usage
```python
import joblib

# Load models
lid = joblib.load("indic_lid_model.joblib")['model']
sentiment = joblib.load("indic_sentiment_model.joblib")['model']
topic = joblib.load("indic_topic_model.joblib")['model']

# Language Detection (28 languages including Northeast)
print(lid.predict(["Chibai, eng nge i an?"]))   # ['lus'] (Mizo)
print(lid.predict(["Khublei shibun!"]))        # ['kha'] (Khasi)
print(lid.predict(["வணக்கம் நண்பா"]))          # ['ta'] (Tamil)

# Sentiment Analysis
sent = sentiment.predict(["यह उत्पाद बहुत ही शानदार है!"])[0]
print("Sentiment:", "Positive" if sent == 1 else "Negative")

# News Topic Classification
top = topic.predict(["विराट कोहलीने झळकावले शतक"])[0]
print("Topic:", top)  # 'sports'
```

### 2. ONNX Runtime
Every model has a matching `.onnx` checkpoint inside `onnx/` allowing execution across platforms (C++, Python, Go, Rust, mobile):
- `onnx/indic_lid.onnx` (4.27 MB)
- `onnx/indic_sentiment.onnx` (0.19 MB)
- `onnx/indic_topic.onnx` (1.03 MB)

---

## Provenance
- **Indic-LID**: Sourced from `google/IndicGenBench_flores_in`, `robzchhangte/mizo-corpus-passages`, and `damerajee/khasi-datasets`.
- **Indic-Sentiment**: Sourced from `mteb/IndicSentiment` and IIT Patna reviews (`ai4bharat/indic_glue`).
- **Indic-Topic**: Sourced from India Today Regional News (`inltkh.*`) and BBC Hindi (`bbca.hi`).

## License
Creative Commons Attribution 4.0 International (**CC-BY-4.0**).