learner-subodh commited on
Commit
531de5f
·
verified ·
1 Parent(s): 4b1ed0b

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +48 -44
README.md CHANGED
@@ -40,25 +40,28 @@ Languages: English, Hindi, Bengali, Marathi, Tamil.
40
 
41
  ## Architecture
42
 
43
- - Decoder-only transformer
44
- - Parameters: 39.7M
45
- - Layers: 10
46
- - Hidden dimension: 512
47
- - Attention heads: 8
48
- - Number of KV Heads: 4 KV heads (GQA)
49
- - Activation: SwiGLU MLP (1408)
50
- - RMSNorm
51
- - Positional Embedding: RoPE
52
- - Tied embeddings
53
- - Context length = 384
54
- - Tokenizer: SentencePiece BPE, 20000 tokens, byte fallback
55
- - Exports as `LlamaForCausalLM` (logits verified against the original
56
- implementation).
57
-
58
- **Tokenizer fertility on held-out text:** - English: 1.28 tokens/word
59
- (GPT-2 BPE: 1.30) - Hindi: 1.26 tokens/word (GPT-2 BPE: 7.76) - Bengali:
60
- 1.45 tokens/word (GPT-2 BPE: 12.79) - Marathi: 1.57 tokens/word (GPT-2
61
- BPE: 10.50) - Tamil: 1.81 tokens/word (GPT-2 BPE: 23.44)
 
 
 
62
 
63
  ## Prompt format
64
 
@@ -89,48 +92,49 @@ print(sp.decode(out[0][len(ids):].tolist()))
89
 
90
  ## Training data
91
 
92
- - Bitext customer-support dataset (English,
93
  `bitext/Bitext-customer-support-llm-chatbot-training-dataset`)
94
- - Sentence-level machine translation of a stratified subset into the
95
- other languages with NLLB-200 (600M distilled), filtered for
96
  placeholders, script, repetition and length
97
- - Native-script instruction data from the Aya dataset
98
  (`CohereForAI/aya_dataset`, Apache-2.0)
99
 
100
  ## Evaluation (held-out)
101
 
102
  ### Language Modelling
103
 
104
- Bits per byte (BPB) of the answer text (lower is better; normalised by
105
- text bytes). Perplexity (PPL) is reported as a supplementary metric.
106
- `domain` = support-style data; `general` = Aya. `resp_tokens` =
107
- evaluated response-token count.
108
-
109
- lang data bits_per_byte ppl resp_tokens
110
- ------ --------- --------------- --------- -------------
111
- bn domain 0.204 6.614 10821
112
- bn general 0.825 193.146 5206
113
- en domain 0.258 2.297 52493
114
- hi domain 0.199 4.506 12750
115
- hi general 0.694 51.689 3170
116
- mr domain 0.217 7.363 11822
117
- mr general 0.668 36.486 4682
118
- ta domain 0.177 7.704 12050
119
- ta general 0.731 119.382 9661
 
120
 
121
  ### Multilingual Generation
122
 
123
  Generation checks on hand-written rider prompts (3 per language; tiny
124
  sample, indicative only).
125
 
126
- - `lang_fidelity`: reply is in the expected script (and Hindi/Marathi
127
  marker words).
128
- - `auto_lang_id`: model picks the right language tag unprompted.
129
- - `distinct2`: 1 - repetition (higher indicates greater lexical
130
  diversity).
131
- - `stop_rate`: fraction of generations terminating with the expected
132
  stop token.
133
- - `avg_words`: average generated words per response.
134
 
135
  | Language | Lang Fidelity | Auto Lang ID | Distinct-2 | Stop Rate | Avg. Words |
136
  |----------|--------------:|-------------:|-----------:|----------:|-----------:|
 
40
 
41
  ## Architecture
42
 
43
+ ### Model Architecture
44
+
45
+ - **Architecture:** Decoder-only Transformer
46
+ - **Total Parameters:** 39.7 million
47
+ - **Transformer Layers:** 10
48
+ - **Hidden Dimension:** 512
49
+ - **Attention Heads:** 8
50
+ - **Key-Value Heads:** 4 (Grouped-Query Attention, GQA)
51
+ - **Feed-Forward Network:** SwiGLU-activated MLP with an intermediate dimension of 1,408
52
+ - **Normalization:** Root Mean Square Layer Normalization (RMSNorm)
53
+ - **Positional Encoding:** Rotary Position Embeddings (RoPE)
54
+ - **Embedding Configuration:** Tied input and output embeddings
55
+ - **Maximum Context Length:** 384 tokens
56
+ - **Tokenizer:** SentencePiece BPE with a 20,000-token vocabulary and byte fallback
57
+ - **Hugging Face Compatibility:** Exports as `LlamaForCausalLM`, with output logits verified against the original implementation.
58
+
59
+ **Tokenizer fertility on held-out text:**
60
+ - **English:** 1.28 tokens/word (GPT-2 BPE: 1.30)
61
+ - **Hindi:** 1.26 tokens/word (GPT-2 BPE: 7.76)
62
+ - **Bengali:** 1.45 tokens/word (GPT-2 BPE: 12.79)
63
+ - **Marathi:** 1.57 tokens/word (GPT-2BPE: 10.50)
64
+ - **Tamil:** 1.81 tokens/word (GPT-2 BPE: 23.44)
65
 
66
  ## Prompt format
67
 
 
92
 
93
  ## Training data
94
 
95
+ - **Bitext customer-support dataset** (English,
96
  `bitext/Bitext-customer-support-llm-chatbot-training-dataset`)
97
+ - **Sentence-level machine translation of a stratified subset into the
98
+ other languages with NLLB-200 (600M distilled)**, filtered for
99
  placeholders, script, repetition and length
100
+ - **Native-script instruction data from the Aya dataset**
101
  (`CohereForAI/aya_dataset`, Apache-2.0)
102
 
103
  ## Evaluation (held-out)
104
 
105
  ### Language Modelling
106
 
107
+ Bits per byte (BPB) measures the model's answer-text prediction performance, normalised by text bytes (lower is better). Perplexity (PPL) is reported as a supplementary metric.
108
+
109
+ - **`domain`:** Support-style data.
110
+ - **`general`:** Aya dataset.
111
+ - **`resp_tokens`:** Number of response tokens evaluated.
112
+
113
+ | Language | Data Split | Bits per Byte | Perplexity (PPL) | Response Tokens |
114
+ |----------|------------|--------------:|-----------------:|----------------:|
115
+ | Bengali (bn) | Domain | 0.204 | 6.614 | 10,821 |
116
+ | Bengali (bn) | General | 0.825 | 193.146 | 5,206 |
117
+ | English (en) | Domain | 0.258 | 2.297 | 52,493 |
118
+ | Hindi (hi) | Domain | 0.199 | 4.506 | 12,750 |
119
+ | Hindi (hi) | General | 0.694 | 51.689 | 3,170 |
120
+ | Marathi (mr) | Domain | 0.217 | 7.363 | 11,822 |
121
+ | Marathi (mr) | General | 0.668 | 36.486 | 4,682 |
122
+ | Tamil (ta) | Domain | 0.177 | 7.704 | 12,050 |
123
+ | Tamil (ta) | General | 0.731 | 119.382 | 9,661 |
124
 
125
  ### Multilingual Generation
126
 
127
  Generation checks on hand-written rider prompts (3 per language; tiny
128
  sample, indicative only).
129
 
130
+ - **`lang_fidelity`:** reply is in the expected script (and Hindi/Marathi
131
  marker words).
132
+ - **`auto_lang_id`:** model picks the right language tag unprompted.
133
+ - **`distinct2`:** 1 - repetition (higher indicates greater lexical
134
  diversity).
135
+ - **`stop_rate`:** fraction of generations terminating with the expected
136
  stop token.
137
+ - **`avg_words`:** average generated words per response.
138
 
139
  | Language | Lang Fidelity | Auto Lang ID | Distinct-2 | Stop Rate | Avg. Words |
140
  |----------|--------------:|-------------:|-----------:|----------:|-----------:|