learner-subodh commited on
Commit
4b1ed0b
·
verified ·
1 Parent(s): db43b79

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +135 -65
README.md CHANGED
@@ -1,55 +1,77 @@
1
  ---
2
- language: ["en", "hi", "bn", "mr", "ta"]
3
- license: other
 
 
 
 
4
  library_name: transformers
 
5
  pipeline_tag: text-generation
6
- tags: [small-language-model, slm, indic, multilingual, edge-ai, logistics, customer-support, llama]
 
 
 
 
 
 
 
 
7
  ---
8
 
9
  # Indic-Delivery-Rider-SLM (40M parameters)
10
 
11
- A small, from-scratch, Llama-style language model for **offline** assistance of delivery partners and drivers (e.g., Swiggy, Zomato, Porter) in India.
 
 
12
 
13
  Languages: English, Hindi, Bengali, Marathi, Tamil.
14
 
15
- > **Experimental research model.** Trained for ~28 minutes on one T4 GPU on a small dataset. Non-English
16
- > domain data is **machine-translated** and has not been reviewed by native speakers. Not for payments, safety or legal advice.
17
-
 
18
 
19
  ## Intended Use Case
20
 
21
- - Offline edge execution in areas with poor connectivity.
22
- - Quick responses for address changes, payment status checks, COD handling, and order issues.
 
23
 
24
  ## Architecture
25
 
26
- - Decoder-only transformer
27
- - Paramters: 39.7M
28
- - Layers: 10
29
- - Hidden dimension: 512
30
- - Attention heads: 8
31
- - Number of KV Heads: 4 KV heads (GQA)
32
- - Activation: SwiGLU MLP (1408)
33
- - RMSNorm
34
- - Positional Embedding: RoPE
35
- - Tied embeddings
36
- - Context length = 384
37
- - Tokenizer: SentencePiece BPE, 20000 tokens, byte fallback
38
- - Exports as `LlamaForCausalLM` (logits verified against the original implementation).
39
-
40
- Tokenizer fertility on held-out text:
41
- - English: 1.28 tokens/word (GPT-2 BPE: 1.30)
42
- - Hindi: 1.26 tokens/word (GPT-2 BPE: 7.76)
43
- - Bengali: 1.45 tokens/word (GPT-2 BPE: 12.79)
44
- - Marathi: 1.57 tokens/word (GPT-2 BPE: 10.50)
45
- - Tamil: 1.81 tokens/word (GPT-2 BPE: 23.44)
46
 
47
  ## Prompt format
48
- `<s><|user|>{question}<|assistant|><|lang|>` then the model writes the answer and `</s>`.
49
- `<|lang|>` is one of `<|en|>`, `<|hi|>`, `<|bn|>`, `<|mr|>`, `<|ta|>`; prefill it to force the reply language, or omit it to let the model choose.
 
 
 
 
 
50
 
51
  ## Usage
52
- ```python
 
53
  import torch, sentencepiece as spm
54
  from huggingface_hub import snapshot_download
55
  from transformers import AutoModelForCausalLM
@@ -66,42 +88,90 @@ print(sp.decode(out[0][len(ids):].tolist()))
66
  ```
67
 
68
  ## Training data
69
- - Bitext customer-support dataset (English, `bitext/Bitext-customer-support-llm-chatbot-training-dataset`)
70
- - Sentence-level machine translation of a stratified subset into the other languages with NLLB-200 (600M distilled), filtered for placeholders, script, repetition and length
71
- - Native-script instruction data from the Aya dataset (`CohereForAI/aya_dataset`, Apache-2.0)
 
 
 
 
 
72
 
73
  ## Evaluation (held-out)
74
- Bits per byte of the answer text (lower is better; comparable across scripts). `domain` = support-style data, `general` = Aya.
75
-
76
- | lang | data | bits_per_byte | ppl | resp_tokens |
77
- |---|---|---|---|---|
78
- | bn | domain | 0.204 | 6.614 | 10821 |
79
- | bn | general | 0.825 | 193.146 | 5206 |
80
- | en | domain | 0.258 | 2.297 | 52493 |
81
- | hi | domain | 0.199 | 4.506 | 12750 |
82
- | hi | general | 0.694 | 51.689 | 3170 |
83
- | mr | domain | 0.217 | 7.363 | 11822 |
84
- | mr | general | 0.668 | 36.486 | 4682 |
85
- | ta | domain | 0.177 | 7.704 | 12050 |
86
- | ta | general | 0.731 | 119.382 | 9661 |
87
-
88
- Generation checks on hand-written rider prompts (3 per language; tiny sample, indicative only): `lang_fidelity` = reply is in the expected
89
- script (and Hindi/Marathi marker words), `auto_lang_id` = model picks the right language tag unprompted, `distinct2` = 1 - repetition.
90
-
91
- | lang | lang_fidelity | auto_lang_id | distinct2 | stop_rate | avg_words |
92
- |---|---|---|---|---|---|
93
- | bn | 1.0 | 1.0 | 0.95 | 0.67 | 80.0 |
94
- | en | 1.0 | 1.0 | 0.98 | 0.67 | 75.0 |
95
- | hi | 1.0 | 1.0 | 0.94 | 1.0 | 91.67 |
96
- | mr | 0.67 | 1.0 | 0.65 | 0.67 | 57.67 |
97
- | ta | 1.0 | 1.0 | 0.99 | 0.67 | 89.67 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
98
 
99
  ## Limitations
100
- - Small capacity, small data: it can state wrong policies fluently, blur Hindi and Marathi (same script), and repeat itself.
101
- - Machine-translated training text carries translation errors.
102
- - Route anything involving money, injuries or disputes to a human or a fixed template.
 
 
 
 
 
 
 
103
 
104
  ## Licensing note
105
- - Mixed provenance: Bitext (CDLA-Sharing-1.0), Aya (Apache-2.0), and translations produced by NLLB-200 (model weights CC-BY-NC-4.0).
106
- - Check these terms with your own counsel before any commercial use.
107
- - Weights are released for education, research and evaluation.
 
 
 
1
  ---
2
+ language:
3
+ - en
4
+ - hi
5
+ - bn
6
+ - mr
7
+ - ta
8
  library_name: transformers
9
+ license: other
10
  pipeline_tag: text-generation
11
+ tags:
12
+ - small-language-model
13
+ - slm
14
+ - indic
15
+ - multilingual
16
+ - edge-ai
17
+ - logistics
18
+ - customer-support
19
+ - llama
20
  ---
21
 
22
  # Indic-Delivery-Rider-SLM (40M parameters)
23
 
24
+ A small, from-scratch, Llama-style language model for **offline**
25
+ assistance of delivery partners and drivers (e.g., Swiggy, Zomato,
26
+ Porter) in India.
27
 
28
  Languages: English, Hindi, Bengali, Marathi, Tamil.
29
 
30
+ > **Experimental research model.** Trained for \~28 minutes on one T4
31
+ > GPU on a small dataset. Non-English domain data is
32
+ > **machine-translated** and has not been reviewed by native speakers.
33
+ > Not for payments, safety or legal advice.
34
 
35
  ## Intended Use Case
36
 
37
+ - Offline edge execution in areas with poor connectivity.
38
+ - Quick responses for address changes, payment status checks, COD
39
+ handling, and order issues.
40
 
41
  ## Architecture
42
 
43
+ - Decoder-only transformer
44
+ - Parameters: 39.7M
45
+ - Layers: 10
46
+ - Hidden dimension: 512
47
+ - Attention heads: 8
48
+ - Number of KV Heads: 4 KV heads (GQA)
49
+ - Activation: SwiGLU MLP (1408)
50
+ - RMSNorm
51
+ - Positional Embedding: RoPE
52
+ - Tied embeddings
53
+ - Context length = 384
54
+ - Tokenizer: SentencePiece BPE, 20000 tokens, byte fallback
55
+ - Exports as `LlamaForCausalLM` (logits verified against the original
56
+ implementation).
57
+
58
+ **Tokenizer fertility on held-out text:** - English: 1.28 tokens/word
59
+ (GPT-2 BPE: 1.30) - Hindi: 1.26 tokens/word (GPT-2 BPE: 7.76) - Bengali:
60
+ 1.45 tokens/word (GPT-2 BPE: 12.79) - Marathi: 1.57 tokens/word (GPT-2
61
+ BPE: 10.50) - Tamil: 1.81 tokens/word (GPT-2 BPE: 23.44)
 
62
 
63
  ## Prompt format
64
+
65
+ `<s><|user|>{question}<|assistant|><|lang|>` then the model writes the
66
+ answer and `</s>`.
67
+
68
+ `<|lang|>` is one of `<|en|>`, `<|hi|>`, `<|bn|>`, `<|mr|>`, `<|ta|>`;
69
+ prefill it to force the reply language, or omit it to let the model
70
+ choose.
71
 
72
  ## Usage
73
+
74
+ ``` python
75
  import torch, sentencepiece as spm
76
  from huggingface_hub import snapshot_download
77
  from transformers import AutoModelForCausalLM
 
88
  ```
89
 
90
  ## Training data
91
+
92
+ - Bitext customer-support dataset (English,
93
+ `bitext/Bitext-customer-support-llm-chatbot-training-dataset`)
94
+ - Sentence-level machine translation of a stratified subset into the
95
+ other languages with NLLB-200 (600M distilled), filtered for
96
+ placeholders, script, repetition and length
97
+ - Native-script instruction data from the Aya dataset
98
+ (`CohereForAI/aya_dataset`, Apache-2.0)
99
 
100
  ## Evaluation (held-out)
101
+
102
+ ### Language Modelling
103
+
104
+ Bits per byte (BPB) of the answer text (lower is better; normalised by
105
+ text bytes). Perplexity (PPL) is reported as a supplementary metric.
106
+ `domain` = support-style data; `general` = Aya. `resp_tokens` =
107
+ evaluated response-token count.
108
+
109
+ lang data bits_per_byte ppl resp_tokens
110
+ ------ --------- --------------- --------- -------------
111
+ bn domain 0.204 6.614 10821
112
+ bn general 0.825 193.146 5206
113
+ en domain 0.258 2.297 52493
114
+ hi domain 0.199 4.506 12750
115
+ hi general 0.694 51.689 3170
116
+ mr domain 0.217 7.363 11822
117
+ mr general 0.668 36.486 4682
118
+ ta domain 0.177 7.704 12050
119
+ ta general 0.731 119.382 9661
120
+
121
+ ### Multilingual Generation
122
+
123
+ Generation checks on hand-written rider prompts (3 per language; tiny
124
+ sample, indicative only).
125
+
126
+ - `lang_fidelity`: reply is in the expected script (and Hindi/Marathi
127
+ marker words).
128
+ - `auto_lang_id`: model picks the right language tag unprompted.
129
+ - `distinct2`: 1 - repetition (higher indicates greater lexical
130
+ diversity).
131
+ - `stop_rate`: fraction of generations terminating with the expected
132
+ stop token.
133
+ - `avg_words`: average generated words per response.
134
+
135
+ | Language | Lang Fidelity | Auto Lang ID | Distinct-2 | Stop Rate | Avg. Words |
136
+ |----------|--------------:|-------------:|-----------:|----------:|-----------:|
137
+ | Bengali (bn) | 1.00 | 1.00 | 0.95 | 0.67 | 80.00 |
138
+ | English (en) | 1.00 | 1.00 | 0.98 | 0.67 | 75.00 |
139
+ | Hindi (hi) | 1.00 | 1.00 | 0.94 | 1.00 | 91.67 |
140
+ | Marathi (mr) | 0.67 | 1.00 | 0.65 | 0.67 | 57.67 |
141
+ | Tamil (ta) | 1.00 | 1.00 | 0.99 | 0.67 | 89.67 |
142
+
143
+ ### Evaluation Scope
144
+
145
+ - **Language modelling:** Evaluates next-token prediction on held-out
146
+ domain and general-language text; does not establish factual
147
+ correctness.
148
+ - **Generation:** Preliminary assessment of language consistency and
149
+ generation behaviour using 15 manually written prompts.
150
+ - **Tokenizer:** Fertility comparison measures tokenisation efficiency
151
+ against GPT-2 BPE, not language understanding.
152
+ - **Task performance:** Intent classification Macro-F1, entity
153
+ extraction F1 and task completion rate have not been independently
154
+ benchmarked.
155
+ - **Deployment:** CPU inference latency, peak RAM, throughput and
156
+ quantised inference performance have not yet been benchmarked.
157
+ - **Human evaluation:** No comprehensive native-speaker or human
158
+ assessment has been conducted.
159
 
160
  ## Limitations
161
+
162
+ - Small capacity, small data: it can state wrong policies fluently,
163
+ blur Hindi and Marathi (same script), and repeat itself.
164
+ - Machine-translated training text carries translation errors.
165
+ - General-language performance is limited, as reflected in held-out
166
+ perplexity.
167
+ - Generation checks use only three prompts per language and are not
168
+ statistically representative.
169
+ - Route anything involving money, injuries or disputes to a human or a
170
+ fixed template.
171
 
172
  ## Licensing note
173
+
174
+ - Mixed provenance: Bitext (CDLA-Sharing-1.0), Aya (Apache-2.0), and
175
+ translations produced by NLLB-200 (model weights CC-BY-NC-4.0).
176
+ - Check these terms with your own counsel before any commercial use.
177
+ - Weights are released for education, research and evaluation.