learner-subodh's picture
Update README.md
531de5f verified
|
Raw History Blame Contribute Delete
7.03 kB
---
language:
- en
- hi
- bn
- mr
- ta
library_name: transformers
license: other
pipeline_tag: text-generation
tags:
- small-language-model
- slm
- indic
- multilingual
- edge-ai
- logistics
- customer-support
- llama
---
# Indic-Delivery-Rider-SLM (40M parameters)
A small, from-scratch, Llama-style language model for **offline**
assistance of delivery partners and drivers (e.g., Swiggy, Zomato,
Porter) in India.
Languages: English, Hindi, Bengali, Marathi, Tamil.
> **Experimental research model.** Trained for \~28 minutes on one T4
> GPU on a small dataset. Non-English domain data is
> **machine-translated** and has not been reviewed by native speakers.
> Not for payments, safety or legal advice.
## Intended Use Case
- Offline edge execution in areas with poor connectivity.
- Quick responses for address changes, payment status checks, COD
handling, and order issues.
## Architecture
### Model Architecture
- **Architecture:** Decoder-only Transformer
- **Total Parameters:** 39.7 million
- **Transformer Layers:** 10
- **Hidden Dimension:** 512
- **Attention Heads:** 8
- **Key-Value Heads:** 4 (Grouped-Query Attention, GQA)
- **Feed-Forward Network:** SwiGLU-activated MLP with an intermediate dimension of 1,408
- **Normalization:** Root Mean Square Layer Normalization (RMSNorm)
- **Positional Encoding:** Rotary Position Embeddings (RoPE)
- **Embedding Configuration:** Tied input and output embeddings
- **Maximum Context Length:** 384 tokens
- **Tokenizer:** SentencePiece BPE with a 20,000-token vocabulary and byte fallback
- **Hugging Face Compatibility:** Exports as `LlamaForCausalLM`, with output logits verified against the original implementation.
**Tokenizer fertility on held-out text:**
- **English:** 1.28 tokens/word (GPT-2 BPE: 1.30)
- **Hindi:** 1.26 tokens/word (GPT-2 BPE: 7.76)
- **Bengali:** 1.45 tokens/word (GPT-2 BPE: 12.79)
- **Marathi:** 1.57 tokens/word (GPT-2BPE: 10.50)
- **Tamil:** 1.81 tokens/word (GPT-2 BPE: 23.44)
## Prompt format
`<s><|user|>{question}<|assistant|><|lang|>` then the model writes the
answer and `</s>`.
`<|lang|>` is one of `<|en|>`, `<|hi|>`, `<|bn|>`, `<|mr|>`, `<|ta|>`;
prefill it to force the reply language, or omit it to let the model
choose.
## Usage
``` python
import torch, sentencepiece as spm
from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM
path = snapshot_download("learner-subodh/Indic-Delivery-Rider-SLM")
sp = spm.SentencePieceProcessor(model_file=f"{path}/tokenizer.model")
model = AutoModelForCausalLM.from_pretrained(path).eval()
q = "Customer wants to change the delivery address after pickup. What should I tell them?"
ids = [sp.bos_id(), sp.piece_to_id("<|user|>")] + sp.encode(q) + [sp.piece_to_id("<|assistant|>"), sp.piece_to_id("<|en|>")]
out = model.generate(torch.tensor([ids]), max_new_tokens=160, do_sample=True, temperature=0.6, top_k=40, top_p=0.9,
repetition_penalty=1.15, eos_token_id=sp.eos_id(), pad_token_id=sp.pad_id())
print(sp.decode(out[0][len(ids):].tolist()))
```
## Training data
- **Bitext customer-support dataset** (English,
`bitext/Bitext-customer-support-llm-chatbot-training-dataset`)
- **Sentence-level machine translation of a stratified subset into the
other languages with NLLB-200 (600M distilled)**, filtered for
placeholders, script, repetition and length
- **Native-script instruction data from the Aya dataset**
(`CohereForAI/aya_dataset`, Apache-2.0)
## Evaluation (held-out)
### Language Modelling
Bits per byte (BPB) measures the model's answer-text prediction performance, normalised by text bytes (lower is better). Perplexity (PPL) is reported as a supplementary metric.
- **`domain`:** Support-style data.
- **`general`:** Aya dataset.
- **`resp_tokens`:** Number of response tokens evaluated.
| Language | Data Split | Bits per Byte | Perplexity (PPL) | Response Tokens |
|----------|------------|--------------:|-----------------:|----------------:|
| Bengali (bn) | Domain | 0.204 | 6.614 | 10,821 |
| Bengali (bn) | General | 0.825 | 193.146 | 5,206 |
| English (en) | Domain | 0.258 | 2.297 | 52,493 |
| Hindi (hi) | Domain | 0.199 | 4.506 | 12,750 |
| Hindi (hi) | General | 0.694 | 51.689 | 3,170 |
| Marathi (mr) | Domain | 0.217 | 7.363 | 11,822 |
| Marathi (mr) | General | 0.668 | 36.486 | 4,682 |
| Tamil (ta) | Domain | 0.177 | 7.704 | 12,050 |
| Tamil (ta) | General | 0.731 | 119.382 | 9,661 |
### Multilingual Generation
Generation checks on hand-written rider prompts (3 per language; tiny
sample, indicative only).
- **`lang_fidelity`:** reply is in the expected script (and Hindi/Marathi
marker words).
- **`auto_lang_id`:** model picks the right language tag unprompted.
- **`distinct2`:** 1 - repetition (higher indicates greater lexical
diversity).
- **`stop_rate`:** fraction of generations terminating with the expected
stop token.
- **`avg_words`:** average generated words per response.
| Language | Lang Fidelity | Auto Lang ID | Distinct-2 | Stop Rate | Avg. Words |
|----------|--------------:|-------------:|-----------:|----------:|-----------:|
| Bengali (bn) | 1.00 | 1.00 | 0.95 | 0.67 | 80.00 |
| English (en) | 1.00 | 1.00 | 0.98 | 0.67 | 75.00 |
| Hindi (hi) | 1.00 | 1.00 | 0.94 | 1.00 | 91.67 |
| Marathi (mr) | 0.67 | 1.00 | 0.65 | 0.67 | 57.67 |
| Tamil (ta) | 1.00 | 1.00 | 0.99 | 0.67 | 89.67 |
### Evaluation Scope
- **Language modelling:** Evaluates next-token prediction on held-out
domain and general-language text; does not establish factual
correctness.
- **Generation:** Preliminary assessment of language consistency and
generation behaviour using 15 manually written prompts.
- **Tokenizer:** Fertility comparison measures tokenisation efficiency
against GPT-2 BPE, not language understanding.
- **Task performance:** Intent classification Macro-F1, entity
extraction F1 and task completion rate have not been independently
benchmarked.
- **Deployment:** CPU inference latency, peak RAM, throughput and
quantised inference performance have not yet been benchmarked.
- **Human evaluation:** No comprehensive native-speaker or human
assessment has been conducted.
## Limitations
- Small capacity, small data: it can state wrong policies fluently,
blur Hindi and Marathi (same script), and repeat itself.
- Machine-translated training text carries translation errors.
- General-language performance is limited, as reflected in held-out
perplexity.
- Generation checks use only three prompts per language and are not
statistically representative.
- Route anything involving money, injuries or disputes to a human or a
fixed template.
## Licensing note
- Mixed provenance: Bitext (CDLA-Sharing-1.0), Aya (Apache-2.0), and
translations produced by NLLB-200 (model weights CC-BY-NC-4.0).
- Check these terms with your own counsel before any commercial use.
- Weights are released for education, research and evaluation.