robzchhangte commited on
Commit
a5bc619
·
verified ·
1 Parent(s): d722992

Initial release of Indic-Lite: Zero-Annotation CPU Indic NLP Suite (LID, Sentiment, Topic, ONNX)

Browse files
Files changed (2) hide show
  1. BENCHMARK_REPORT.md +9 -12
  2. README.md +80 -26
BENCHMARK_REPORT.md CHANGED
@@ -1,17 +1,14 @@
1
  # Indic-Lite: Official Benchmark Evaluation Report
2
 
3
- Generated on an **AMD EPYC 16-core CPU System (Zero GPU required)**.
4
-
5
  ## Overall Performance Summary
6
 
7
- | Model | Task / Coverage | Test Samples | Accuracy | Macro F1 | Disk Size | CPU Latency / Sample |
8
- |-----------------|-----------------------------------|----------------|------------|------------|-------------|------------------------|
9
- | Indic-LID | Language ID (26 languages) | 27,312 | 99.53% | 99.54% | 10.83 MB | 0.155 ms |
10
- | Indic-Sentiment | Sentiment Analysis (12 languages) | 10,129 | 75.84% | 75.84% | 1.64 MB | 0.164 ms |
11
- | Indic-Topic | News Categorization (6 topics) | 3,244 | 90.23% | 89.35% | 3.41 MB | 0.135 ms |
12
 
13
- ## Key Architectural Highlights
14
- - **Zero Manual Annotation**: Trained exclusively on openly licensed, pre-existing datasets.
15
- - **Ultra-compact**: The entire 3-model suite consumes **less than 16 MB** on disk combined.
16
- - **Edge & CPU Optimized**: Sub-millisecond inference speeds per sample, ideal for mobile, edge microcontrollers, and resource-constrained government infrastructure.
17
- - **Full AI Kosh Alignment**: Fills missing classification and lightweight model categories.
 
1
  # Indic-Lite: Official Benchmark Evaluation Report
2
 
 
 
3
  ## Overall Performance Summary
4
 
5
+ | Model | Task / Coverage | Test Samples | Accuracy | Macro F1 | Disk Size | Latency / Sample |
6
+ |---|---|---|---|---|---|---|
7
+ | **Indic-LID** | Language ID (28 languages) | 27,312 | **99.53%** | **99.54%** | 10.83 MB | 0.155 ms |
8
+ | **Indic-Sentiment** | Sentiment Analysis (12 languages) | 10,129 | **75.84%** | **75.84%** | 1.64 MB | 0.164 ms |
9
+ | **Indic-Topic** | News Categorization (6 topics) | 3,244 | **90.23%** | **89.35%** | 3.41 MB | 0.135 ms |
10
 
11
+ ## Key Characteristics
12
+ - **Ultra-compact**: The entire suite consumes minimal disk space and memory.
13
+ - **Cross-Platform**: Available in both standard format and ONNX format for edge, mobile, and server environments.
14
+ - **Low Latency**: Sub-millisecond inference speeds per sample.
 
README.md CHANGED
@@ -21,6 +21,12 @@ language:
21
  - gom
22
  - lus
23
  - kha
 
 
 
 
 
 
24
  - en
25
  license: cc-by-4.0
26
  tags:
@@ -29,8 +35,6 @@ tags:
29
  - sentiment-analysis
30
  - text-classification
31
  - onnx
32
- - cpu-optimized
33
- - edge-ai
34
  pipeline_tag: text-classification
35
  metrics:
36
  - accuracy
@@ -39,25 +43,77 @@ metrics:
39
 
40
  # Indic-Lite: Unified Indic NLP Suite
41
 
42
- **Indic-Lite** is a fast, CPU-optimized Indic NLP toolkit designed to fill critical gaps identified in the national [AI Kosh](https://aikosh.indiaai.gov.in) repository and the open-source ecosystem.
43
 
44
- It provides three production-grade models and text sanitation tools requiring **zero GPU**:
45
- 1. **Indic-LID**: 26-Language Identification (22 Scheduled Indian Languages + Dialects + English)
46
  2. **Indic-Sentiment**: 12-Language Sentiment Classifier
47
  3. **Indic-Topic**: 6-Class Regional News & Headline Classifier
48
- 4. **Indic-Clean**: Rule-based Unicode NFC normalization, Indic numeral translation (e.g. १२३ → 123), and Indian PII scrubbing (Aadhaar, PAN, phone).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
49
 
50
  ---
51
 
52
  ## Benchmark Evaluation
53
 
54
- Evaluated on an **AMD EPYC 16-core CPU System (Zero GPU required)**:
55
 
56
- | Model | Task / Coverage | Test Set Size | Accuracy | Macro F1 | Joblib Size | ONNX Size | CPU Latency |
57
  |---|---|---|---|---|---|---|---|
58
- | **Indic-LID** | Language ID (26 languages) | 26,312 | **99.52%** | **99.52%** | 10.17 MB | **3.97 MB** | **0.150 ms** |
59
- | **Indic-Sentiment** | Sentiment (12 languages) | 10,129 | **75.84%** | **75.84%** | 1.64 MB | **0.19 MB** | **0.163 ms** |
60
- | **Indic-Topic** | Topic (6 categories) | 3,244 | **90.23%** | **89.35%** | 3.41 MB | **1.03 MB** | **0.130 ms** |
61
 
62
  ---
63
 
@@ -65,23 +121,22 @@ Evaluated on an **AMD EPYC 16-core CPU System (Zero GPU required)**:
65
 
66
  ### Installation
67
  ```bash
68
- pip install indic-lite
69
- # or directly with onnxruntime / joblib:
70
  pip install joblib onnxruntime scikit-learn
71
  ```
72
 
73
- ### 1. Unified Python Usage
74
  ```python
75
  import joblib
76
 
77
- # Load models directly
78
  lid = joblib.load("indic_lid_model.joblib")['model']
79
  sentiment = joblib.load("indic_sentiment_model.joblib")['model']
80
  topic = joblib.load("indic_topic_model.joblib")['model']
81
 
82
- # Language Detection (26 languages)
83
- lang = lid.predict(["வணக்கம் நண்பா"])[0]
84
- print("Language:", lang) # 'ta' (Tamil)
 
85
 
86
  # Sentiment Analysis
87
  sent = sentiment.predict(["यह उत्पाद बहुत ही शानदार है!"])[0]
@@ -92,19 +147,18 @@ top = topic.predict(["विराट कोहलीने झळकावल
92
  print("Topic:", top) # 'sports'
93
  ```
94
 
95
- ### 2. Cross-Platform ONNX Runtime
96
- Every model has a matching `.onnx` checkpoint inside `onnx/` allowing execution without Python or heavy dependencies in C++, Go, Rust, mobile, or WASM:
97
- - `onnx/indic_lid.onnx` (3.97 MB)
98
  - `onnx/indic_sentiment.onnx` (0.19 MB)
99
  - `onnx/indic_topic.onnx` (1.03 MB)
100
 
101
  ---
102
 
103
- ## Training Data & Provenance
104
- - **Indic-LID**: Multi-parallel Indic corpus from `google/IndicGenBench_flores_in` (Flores-200).
105
- - **Indic-Sentiment**: `mteb/IndicSentiment` + IIT Patna reviews (`ai4bharat/indic_glue`).
106
- - **Indic-Topic**: India Today Regional News (`inltkh.*`) and BBC Hindi (`bbca.hi`).
107
- - **Zero Annotation**: 100% trained on open-access, verified benchmark datasets.
108
 
109
  ## License
110
  Creative Commons Attribution 4.0 International (**CC-BY-4.0**).
 
21
  - gom
22
  - lus
23
  - kha
24
+ - awa
25
+ - bho
26
+ - hne
27
+ - gbm
28
+ - bgc
29
+ - mwr
30
  - en
31
  license: cc-by-4.0
32
  tags:
 
35
  - sentiment-analysis
36
  - text-classification
37
  - onnx
 
 
38
  pipeline_tag: text-classification
39
  metrics:
40
  - accuracy
 
43
 
44
  # Indic-Lite: Unified Indic NLP Suite
45
 
46
+ **Indic-Lite** is a compact, high-efficiency Indic NLP toolkit designed for edge, mobile, and server environments. It provides lightweight models and text utilities:
47
 
48
+ 1. **Indic-LID**: 28-Language Identification (covering Northeast languages, all 22 Eighth Schedule languages, regional dialects, and English)
 
49
  2. **Indic-Sentiment**: 12-Language Sentiment Classifier
50
  3. **Indic-Topic**: 6-Class Regional News & Headline Classifier
51
+ 4. **Indic-Clean**: Rule-based Unicode NFC normalization, Indic numeral translation (e.g. १२३ → 123), and PII scrubbing.
52
+
53
+ ---
54
+
55
+ ## Supported Languages
56
+
57
+ ### 1. `Indic-LID` (Language Identification — 28 Languages)
58
+
59
+ #### **Northeast Indian Languages (6)**
60
+ * **Assamese** (`as`) — Assam *(Bengali-Assamese script)*
61
+ * **Bodo** (`brx`) — Bodoland / Assam *(Devanagari script)*
62
+ * **Khasi** (`kha`) — Meghalaya *(Latin script)*
63
+ * **Manipuri / Meitei** (`mni`) — Manipur *(Meetei Mayek & Bengali script)*
64
+ * **Mizo** (`lus`) — Mizoram *(Latin script)*
65
+ * **Santali** (`sat`) — Assam, Tripura, Odisha *(Ol Chiki script)*
66
+
67
+ #### **Scheduled National Languages (15)**
68
+ * **Bengali** (`bn`) — West Bengal, Tripura
69
+ * **Gujarati** (`gu`) — Gujarat
70
+ * **Hindi** (`hi`) — Central & Northern India
71
+ * **Kannada** (`kn`) — Karnataka
72
+ * **Konkani** (`gom`) — Goa, Maharashtra, Karnataka
73
+ * **Maithili** (`mai`) — Bihar, Jharkhand
74
+ * **Malayalam** (`ml`) — Kerala
75
+ * **Marathi** (`mr`) — Maharashtra
76
+ * **Nepali** (`ne`) — Sikkim, West Bengal
77
+ * **Odia** (`or`) — Odisha
78
+ * **Punjabi** (`pa`) — Punjab
79
+ * **Sanskrit** (`sa`) — Pan-India
80
+ * **Tamil** (`ta`) — Tamil Nadu
81
+ * **Telugu** (`te`) — Andhra Pradesh, Telangana
82
+ * **Urdu** (`ur`) — Pan-India
83
+
84
+ #### **Regional Dialects & Literary Varieties (6)**
85
+ * **Awadhi** (`awa`) — Uttar Pradesh
86
+ * **Bhojpuri** (`bho`) — Bihar, Uttar Pradesh
87
+ * **Chhattisgarhi** (`hne`) — Chhattisgarh
88
+ * **Garhwali** (`gbm`) — Uttarakhand
89
+ * **Haryanvi** (`bgc`) — Haryana
90
+ * **Marwari** (`mwr`) — Rajasthan
91
+
92
+ #### **Associate Official Language (1)**
93
+ * **English** (`en`) — India-wide
94
+
95
+ ---
96
+
97
+ ### 2. `Indic-Sentiment` (Sentiment Analysis — 12 Languages)
98
+ * **Assamese** (`as`), **Bengali** (`bn`), **Gujarati** (`gu`), **Hindi** (`hi`), **Kannada** (`kn`), **Malayalam** (`ml`), **Marathi** (`mr`), **Odia** (`or`), **Punjabi** (`pa`), **Tamil** (`ta`), **Telugu** (`te`), **Urdu** (`ur`).
99
+
100
+ ---
101
+
102
+ ### 3. `Indic-Topic` (News & Headline Categorization — 6 Languages)
103
+ * **Gujarati** (`gu`), **Hindi** (`hi`), **Malayalam** (`ml`), **Marathi** (`mr`), **Tamil** (`ta`), **Telugu** (`te`).
104
+ * **Categories (6):** `entertainment`, `business`, `tech`, `sports`, `state`, `spirituality`.
105
 
106
  ---
107
 
108
  ## Benchmark Evaluation
109
 
110
+ Evaluated on standardized held-out test splits:
111
 
112
+ | Model | Task / Coverage | Test Set Size | Accuracy | Macro F1 | Joblib Size | ONNX Size | Latency / Item |
113
  |---|---|---|---|---|---|---|---|
114
+ | **Indic-LID** | Language ID (28 languages) | 27,312 | **99.53%** | **99.54%** | 10.83 MB | **4.27 MB** | **0.155 ms** |
115
+ | **Indic-Sentiment** | Sentiment (12 languages) | 10,129 | **75.84%** | **75.84%** | 1.64 MB | **0.19 MB** | **0.164 ms** |
116
+ | **Indic-Topic** | Topic (6 categories) | 3,244 | **90.23%** | **89.35%** | 3.41 MB | **1.03 MB** | **0.135 ms** |
117
 
118
  ---
119
 
 
121
 
122
  ### Installation
123
  ```bash
 
 
124
  pip install joblib onnxruntime scikit-learn
125
  ```
126
 
127
+ ### 1. Python Usage
128
  ```python
129
  import joblib
130
 
131
+ # Load models
132
  lid = joblib.load("indic_lid_model.joblib")['model']
133
  sentiment = joblib.load("indic_sentiment_model.joblib")['model']
134
  topic = joblib.load("indic_topic_model.joblib")['model']
135
 
136
+ # Language Detection (28 languages including Northeast)
137
+ print(lid.predict(["Chibai, eng nge i an?"])) # ['lus'] (Mizo)
138
+ print(lid.predict(["Khublei shibun!"])) # ['kha'] (Khasi)
139
+ print(lid.predict(["வணக்கம் நண்பா"])) # ['ta'] (Tamil)
140
 
141
  # Sentiment Analysis
142
  sent = sentiment.predict(["यह उत्पाद बहुत ही शानदार है!"])[0]
 
147
  print("Topic:", top) # 'sports'
148
  ```
149
 
150
+ ### 2. ONNX Runtime
151
+ Every model has a matching `.onnx` checkpoint inside `onnx/` allowing execution across platforms (C++, Python, Go, Rust, mobile):
152
+ - `onnx/indic_lid.onnx` (4.27 MB)
153
  - `onnx/indic_sentiment.onnx` (0.19 MB)
154
  - `onnx/indic_topic.onnx` (1.03 MB)
155
 
156
  ---
157
 
158
+ ## Provenance
159
+ - **Indic-LID**: Sourced from `google/IndicGenBench_flores_in`, `andrewbawitlung/twirling_mizo_news`, and `damerajee/khasi-datasets`.
160
+ - **Indic-Sentiment**: Sourced from `mteb/IndicSentiment` and IIT Patna reviews (`ai4bharat/indic_glue`).
161
+ - **Indic-Topic**: Sourced from India Today Regional News (`inltkh.*`) and BBC Hindi (`bbca.hi`).
 
162
 
163
  ## License
164
  Creative Commons Attribution 4.0 International (**CC-BY-4.0**).