ecreeth commited on
Commit
90ac948
Β·
verified Β·
1 Parent(s): 491dfb2

Upload README.md

Browse files
Files changed (1) hide show
  1. README.md +76 -24
README.md CHANGED
@@ -52,6 +52,9 @@ GPT replaced by a custom Stream Mixer block.
52
  | Vocab | 32,768 BPE (GPT-4-style pretokenization) |
53
  | Special tokens | `<\|bos\|>`, `<\|user_start\|>`, `<\|user_end\|>`, `<\|assistant_start\|>`, `<\|assistant_end\|>` |
54
  | Compute dtype | bf16 (Ampere+) / fp32 (T4/CPU) |
 
 
 
55
  | License | MIT |
56
 
57
  ---
@@ -122,27 +125,73 @@ SFT checkpoint (chat-aware β€” answers as Mnemo)
122
 
123
  ---
124
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
125
  ## Capabilities and limitations
126
 
127
- ### Capabilities (validated on the standard probe set)
 
 
 
 
 
 
 
128
 
129
- After SFT, Mnemo reliably handles:
130
- - Conversational dialogue in chat format (`<|user_start|>` / `<|assistant_start|>` delimiters)
131
- - Common factual recall: capital cities, chemical symbols, planets, basic vocabulary
132
- - Multiple-choice questions (MMLU/ARC format)
133
- - Step-by-step arithmetic in the GSM8K style (with occasional wrong answers)
134
- - Letter counting via manual enumeration (SpellingBee)
135
- - Identity Q&A consistent with its training persona
136
 
137
- ### Limitations
 
 
 
138
 
139
- - **117M parameters** β€” confabulates confidently on niche facts, dates, and proper nouns
 
 
140
  - **No tool use, no internet, no images, no memory across sessions**
141
- - **2048-token context** β€” not pretrained for longer contexts; quality degrades past ~1500 tokens
142
  - **No RLHF** β€” outputs reflect only supervised signal; may produce inappropriate completions
143
  - **English only** β€” pretraining corpus is essentially English educational/web text
144
- - **Repetition prone in long generations** without `--repetition-penalty` or top-p
145
- - **Math beyond GSM8K-level** is unreliable; arithmetic gets shaky past 3-digit operands
146
 
147
  ---
148
 
@@ -156,7 +205,7 @@ from tokenizers import Tokenizer
156
  from model import GPT
157
 
158
  tokenizer = Tokenizer.from_file('tokenizer.json')
159
- ckpt = torch.load('model_sft.pt', map_location='cuda')
160
 
161
  config = dict(ckpt['config'])
162
  config['vocab_size'] = ((tokenizer.get_vocab_size() + 63) // 64) * 64
@@ -195,17 +244,20 @@ Recommended sampling parameters (empirically tuned, see training log):
195
 
196
  ## Probe outputs (greedy, from the base checkpoint)
197
 
198
- | Prompt | Output | Verdict |
 
 
199
  |---|---|---|
200
- | The capital of France is | "...Paris, and the capital of France is Paris" | βœ“ |
201
- | The chemical symbol of gold is | "Au. It is a soft, silvery-white metal... good conductor of electricity and heat, useful in electrical wiring" | βœ“ (Au correct; "silvery-white" wrong color) |
202
- | If yesterday was Friday, then tomorrow will be | "Tuesday" | βœ— (correct: Sunday) |
203
- | The opposite of hot is | "the cold" | βœ“ |
204
- | The planets of the solar system are: | "Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune, Pluto" | βœ“ (correct order) |
205
- | My favorite color is | "blue" | βœ“ |
206
- | If 5\*x + 3 = 13, then x is | "a positive integer" loop | βœ— |
207
-
208
- **5/7 correct on the base model.** SFT improves chat-format adherence, MCQ accuracy, and persona consistency.
 
209
 
210
  ---
211
 
 
52
  | Vocab | 32,768 BPE (GPT-4-style pretokenization) |
53
  | Special tokens | `<\|bos\|>`, `<\|user_start\|>`, `<\|user_end\|>`, `<\|assistant_start\|>`, `<\|assistant_end\|>` |
54
  | Compute dtype | bf16 (Ampere+) / fp32 (T4/CPU) |
55
+ | **Base perplexity (BPB)** | **19.47 (0.9011 bits-per-byte)** |
56
+ | **Chat ChatCORE metric** | **22.74%** (mean centered across 5 tasks) |
57
+ | **SpellingBee accuracy** | **94.53%** (256/256 test set) |
58
  | License | MIT |
59
 
60
  ---
 
125
 
126
  ---
127
 
128
+ ## Evaluation results
129
+
130
+ Measured on the full test sets β€” no subsampling, no cherry-picking.
131
+
132
+ ### Base model β€” `model.pt` @ step 79,000
133
+
134
+ | Metric | Value |
135
+ |---|---|
136
+ | Validation loss (nats / token) | 2.9691 |
137
+ | Perplexity | 19.47 |
138
+ | **Bits per byte (BPB)** | **0.9011** |
139
+ | Evaluation window | 409,600 tokens / 1,947,169 bytes |
140
+
141
+ Bits-per-byte is the tokenizer-invariant measure β€” directly comparable across models with different vocabularies. For reference, GPT-2 on similar web text lands around BPB β‰ˆ 1.0; Mnemo at 117M on ClimbMix-400B gets to ~0.90, which is sensible for the size class.
142
+
143
+ ### Chat model β€” full benchmark suite
144
+
145
+ Evaluated on the **complete test set of each task** (no `--max-problems` cap).
146
+ Categorical tasks use logit comparison over allowed letters; generative tasks
147
+ sample greedily and parse `#### N` for the final answer.
148
+
149
+ | Task | Type | N | Accuracy | Random baseline | Centered |
150
+ |---|---|---|---|---|---|
151
+ | MMLU (57 subjects) | categorical 4-way MCQ | 14,042 | **28.32%** | 25% | +4.42 |
152
+ | ARC-Easy | categorical 4-way MCQ | 2,376 | **30.68%** | 25% | +7.58 |
153
+ | ARC-Challenge | categorical 4-way MCQ | 1,172 | **29.52%** | 25% | +6.03 |
154
+ | GSM8K (math word problems) | generative, parse `#### N` | 1,319 | 1.14% | 0% | +1.14 |
155
+ | **SpellingBee (letter counting)** | generative, parse `#### N` | 256 | **94.53%** | 0% | **+94.53** |
156
+
157
+ ### ChatCORE metric
158
+
159
+ **`ChatCORE = 22.74%`** β€” mean centered accuracy across all five tasks.
160
+
161
+ ChatCORE is the same shape as nanochat's metric: it normalizes each task to its random baseline (so a fair guess scores 0, and perfect scores 100). At 22.74% on 117M params after 9h pretraining + 1h SFT, Mnemo lands meaningfully above random across all tasks. The Stream Mixer architecture clearly *can* hold the necessary structure β€” the dominant ceiling is parameter count, not architecture.
162
+
163
+ ### Where the numbers come from
164
+
165
+ - **SpellingBee 94.53%** is the standout. Mnemo learned to character-by-character enumerate words from the 370k-word English dictionary and reliably emit a correct `#### N` final answer. Common short words that tokenize as single BPE tokens (like "strawberry") still fail because the model never observes their letters individually β€” this is a tokenizer limitation, not a model one.
166
+ - **All three MCQ tasks above random** confirms the model genuinely commits to a letter at the assistant position when forced. The MMLU advantage (+4.4 pp) is modest β€” 117M can't memorize the breadth of academic facts MMLU covers.
167
+ - **GSM8K at 1.14%** is honest for an unaligned 117M-parameter model with no tool use. The format is correctly learned (step-by-step reasoning + `#### N` final answer) but the arithmetic isn't reliable enough to land the right number consistently.
168
+
169
  ## Capabilities and limitations
170
 
171
+ ### Confirmed strong
172
+
173
+ - Coherent conversational dialogue in chat format (`<|user_start|>` / `<|assistant_start|>`)
174
+ - Factual recall on common entities (capital cities, chemical symbols, planets ordered)
175
+ - **Letter counting via manual enumeration** β€” 94.5% on SpellingBee
176
+ - Multiple-choice answer commitment (above random on all three MCQ benchmarks)
177
+ - Persona consistency (model identifies as Mnemo with consistent self-description)
178
+ - Greedy + nucleus (top-p) sampling configurable for short or long generation
179
 
180
+ ### Confirmed weak
 
 
 
 
 
 
181
 
182
+ - **Math word problems** β€” 1.14% on GSM8K. Format is learned, arithmetic is not
183
+ - **Single-token common words for spelling** β€” "strawberry" β†’ 2 r's (real answer: 3); tokenizer hides character-level information for words that fit in a single BPE token
184
+ - **Niche factual recall** β€” confabulates confidently on rare entities, exact dates, specific quotations
185
+ - **Long multi-turn conversations** β€” context drifts after ~2-3 turns
186
 
187
+ ### Limitations (architectural)
188
+
189
+ - **117M parameters** β€” knowledge density is the ceiling, not the architecture
190
  - **No tool use, no internet, no images, no memory across sessions**
191
+ - **2048-token context** β€” quality degrades past ~1500 tokens without repetition penalty
192
  - **No RLHF** β€” outputs reflect only supervised signal; may produce inappropriate completions
193
  - **English only** β€” pretraining corpus is essentially English educational/web text
194
+ - **Repetition prone in long generations** without `--repetition-penalty` or `--top-p`
 
195
 
196
  ---
197
 
 
205
  from model import GPT
206
 
207
  tokenizer = Tokenizer.from_file('tokenizer.json')
208
+ ckpt = torch.load('model.pt', map_location='cuda')
209
 
210
  config = dict(ckpt['config'])
211
  config['vocab_size'] = ((tokenizer.get_vocab_size() + 63) // 64) * 64
 
244
 
245
  ## Probe outputs (greedy, from the base checkpoint)
246
 
247
+ Run via `python3 base_eval.py --eval sample` against the pretrained checkpoint (`model.pt`, val 2.9508). Greedy, 64 tokens per completion.
248
+
249
+ | Prompt | First tokens of output | Verdict |
250
  |---|---|---|
251
+ | *The capital of France is* | "...Paris, and it is the capital of France. The capital of France is Paris..." | βœ“ Paris lands |
252
+ | *The chemical symbol of gold is* | "Au. It is a soft, silvery-white metal... good conductor of electricity and heat, making it useful in electrical wiring..." | βœ“ Au + real applied claim |
253
+ | *If yesterday was Friday, then tomorrow will be* | "Tuesday. The weather is not so bad..." | βœ— (correct: Sunday) |
254
+ | *The opposite of hot is* | "the cold." | βœ“ |
255
+ | *The planets of the solar system are:* | "Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune, Pluto..." | βœ“ Correct astronomical order |
256
+ | *My favorite color is* | "red. It's a color that's been around for a long time..." | βœ“ |
257
+ | *If 5\*x + 3 = 13, then x is* | "a positive integer. If x is a positive integer, then x is a positive integer..." | βœ— Loop |
258
+ | *Photosynthesis is the process by which* | "plants and other organisms convert light energy into chemical energy. It is a complex process that involves the conversion of light energy into chemical energy..." | βœ“ Factually correct opener |
259
+
260
+ **5/7 of the original training probes land correct answers at greedy.** Repetition is visible β€” the base model benefits substantially from `--repetition-penalty 1.15` and/or `--top-p 0.9` on longer generations (see Usage section).
261
 
262
  ---
263