Upload README.md
Browse files
README.md
CHANGED
|
@@ -52,6 +52,9 @@ GPT replaced by a custom Stream Mixer block.
|
|
| 52 |
| Vocab | 32,768 BPE (GPT-4-style pretokenization) |
|
| 53 |
| Special tokens | `<\|bos\|>`, `<\|user_start\|>`, `<\|user_end\|>`, `<\|assistant_start\|>`, `<\|assistant_end\|>` |
|
| 54 |
| Compute dtype | bf16 (Ampere+) / fp32 (T4/CPU) |
|
|
|
|
|
|
|
|
|
|
| 55 |
| License | MIT |
|
| 56 |
|
| 57 |
---
|
|
@@ -122,27 +125,73 @@ SFT checkpoint (chat-aware β answers as Mnemo)
|
|
| 122 |
|
| 123 |
---
|
| 124 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 125 |
## Capabilities and limitations
|
| 126 |
|
| 127 |
-
###
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 128 |
|
| 129 |
-
|
| 130 |
-
- Conversational dialogue in chat format (`<|user_start|>` / `<|assistant_start|>` delimiters)
|
| 131 |
-
- Common factual recall: capital cities, chemical symbols, planets, basic vocabulary
|
| 132 |
-
- Multiple-choice questions (MMLU/ARC format)
|
| 133 |
-
- Step-by-step arithmetic in the GSM8K style (with occasional wrong answers)
|
| 134 |
-
- Letter counting via manual enumeration (SpellingBee)
|
| 135 |
-
- Identity Q&A consistent with its training persona
|
| 136 |
|
| 137 |
-
|
|
|
|
|
|
|
|
|
|
| 138 |
|
| 139 |
-
|
|
|
|
|
|
|
| 140 |
- **No tool use, no internet, no images, no memory across sessions**
|
| 141 |
-
- **2048-token context** β
|
| 142 |
- **No RLHF** β outputs reflect only supervised signal; may produce inappropriate completions
|
| 143 |
- **English only** β pretraining corpus is essentially English educational/web text
|
| 144 |
-
- **Repetition prone in long generations** without `--repetition-penalty` or top-p
|
| 145 |
-
- **Math beyond GSM8K-level** is unreliable; arithmetic gets shaky past 3-digit operands
|
| 146 |
|
| 147 |
---
|
| 148 |
|
|
@@ -156,7 +205,7 @@ from tokenizers import Tokenizer
|
|
| 156 |
from model import GPT
|
| 157 |
|
| 158 |
tokenizer = Tokenizer.from_file('tokenizer.json')
|
| 159 |
-
ckpt = torch.load('
|
| 160 |
|
| 161 |
config = dict(ckpt['config'])
|
| 162 |
config['vocab_size'] = ((tokenizer.get_vocab_size() + 63) // 64) * 64
|
|
@@ -195,17 +244,20 @@ Recommended sampling parameters (empirically tuned, see training log):
|
|
| 195 |
|
| 196 |
## Probe outputs (greedy, from the base checkpoint)
|
| 197 |
|
| 198 |
-
|
|
|
|
|
|
|
| 199 |
|---|---|---|
|
| 200 |
-
| The capital of France is | "...Paris, and the capital of France is Paris" | β |
|
| 201 |
-
| The chemical symbol of gold is | "Au. It is a soft, silvery-white metal... good conductor of electricity and heat, useful in electrical wiring" | β
|
| 202 |
-
| If yesterday was Friday, then tomorrow will be | "Tuesday" | β (correct: Sunday) |
|
| 203 |
-
| The opposite of hot is | "the cold" | β |
|
| 204 |
-
| The planets of the solar system are: | "Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune, Pluto" | β
|
| 205 |
-
| My favorite color is | "
|
| 206 |
-
| If 5\*x + 3 = 13, then x is | "a positive integer
|
| 207 |
-
|
| 208 |
-
|
|
|
|
| 209 |
|
| 210 |
---
|
| 211 |
|
|
|
|
| 52 |
| Vocab | 32,768 BPE (GPT-4-style pretokenization) |
|
| 53 |
| Special tokens | `<\|bos\|>`, `<\|user_start\|>`, `<\|user_end\|>`, `<\|assistant_start\|>`, `<\|assistant_end\|>` |
|
| 54 |
| Compute dtype | bf16 (Ampere+) / fp32 (T4/CPU) |
|
| 55 |
+
| **Base perplexity (BPB)** | **19.47 (0.9011 bits-per-byte)** |
|
| 56 |
+
| **Chat ChatCORE metric** | **22.74%** (mean centered across 5 tasks) |
|
| 57 |
+
| **SpellingBee accuracy** | **94.53%** (256/256 test set) |
|
| 58 |
| License | MIT |
|
| 59 |
|
| 60 |
---
|
|
|
|
| 125 |
|
| 126 |
---
|
| 127 |
|
| 128 |
+
## Evaluation results
|
| 129 |
+
|
| 130 |
+
Measured on the full test sets β no subsampling, no cherry-picking.
|
| 131 |
+
|
| 132 |
+
### Base model β `model.pt` @ step 79,000
|
| 133 |
+
|
| 134 |
+
| Metric | Value |
|
| 135 |
+
|---|---|
|
| 136 |
+
| Validation loss (nats / token) | 2.9691 |
|
| 137 |
+
| Perplexity | 19.47 |
|
| 138 |
+
| **Bits per byte (BPB)** | **0.9011** |
|
| 139 |
+
| Evaluation window | 409,600 tokens / 1,947,169 bytes |
|
| 140 |
+
|
| 141 |
+
Bits-per-byte is the tokenizer-invariant measure β directly comparable across models with different vocabularies. For reference, GPT-2 on similar web text lands around BPB β 1.0; Mnemo at 117M on ClimbMix-400B gets to ~0.90, which is sensible for the size class.
|
| 142 |
+
|
| 143 |
+
### Chat model β full benchmark suite
|
| 144 |
+
|
| 145 |
+
Evaluated on the **complete test set of each task** (no `--max-problems` cap).
|
| 146 |
+
Categorical tasks use logit comparison over allowed letters; generative tasks
|
| 147 |
+
sample greedily and parse `#### N` for the final answer.
|
| 148 |
+
|
| 149 |
+
| Task | Type | N | Accuracy | Random baseline | Centered |
|
| 150 |
+
|---|---|---|---|---|---|
|
| 151 |
+
| MMLU (57 subjects) | categorical 4-way MCQ | 14,042 | **28.32%** | 25% | +4.42 |
|
| 152 |
+
| ARC-Easy | categorical 4-way MCQ | 2,376 | **30.68%** | 25% | +7.58 |
|
| 153 |
+
| ARC-Challenge | categorical 4-way MCQ | 1,172 | **29.52%** | 25% | +6.03 |
|
| 154 |
+
| GSM8K (math word problems) | generative, parse `#### N` | 1,319 | 1.14% | 0% | +1.14 |
|
| 155 |
+
| **SpellingBee (letter counting)** | generative, parse `#### N` | 256 | **94.53%** | 0% | **+94.53** |
|
| 156 |
+
|
| 157 |
+
### ChatCORE metric
|
| 158 |
+
|
| 159 |
+
**`ChatCORE = 22.74%`** β mean centered accuracy across all five tasks.
|
| 160 |
+
|
| 161 |
+
ChatCORE is the same shape as nanochat's metric: it normalizes each task to its random baseline (so a fair guess scores 0, and perfect scores 100). At 22.74% on 117M params after 9h pretraining + 1h SFT, Mnemo lands meaningfully above random across all tasks. The Stream Mixer architecture clearly *can* hold the necessary structure β the dominant ceiling is parameter count, not architecture.
|
| 162 |
+
|
| 163 |
+
### Where the numbers come from
|
| 164 |
+
|
| 165 |
+
- **SpellingBee 94.53%** is the standout. Mnemo learned to character-by-character enumerate words from the 370k-word English dictionary and reliably emit a correct `#### N` final answer. Common short words that tokenize as single BPE tokens (like "strawberry") still fail because the model never observes their letters individually β this is a tokenizer limitation, not a model one.
|
| 166 |
+
- **All three MCQ tasks above random** confirms the model genuinely commits to a letter at the assistant position when forced. The MMLU advantage (+4.4 pp) is modest β 117M can't memorize the breadth of academic facts MMLU covers.
|
| 167 |
+
- **GSM8K at 1.14%** is honest for an unaligned 117M-parameter model with no tool use. The format is correctly learned (step-by-step reasoning + `#### N` final answer) but the arithmetic isn't reliable enough to land the right number consistently.
|
| 168 |
+
|
| 169 |
## Capabilities and limitations
|
| 170 |
|
| 171 |
+
### Confirmed strong
|
| 172 |
+
|
| 173 |
+
- Coherent conversational dialogue in chat format (`<|user_start|>` / `<|assistant_start|>`)
|
| 174 |
+
- Factual recall on common entities (capital cities, chemical symbols, planets ordered)
|
| 175 |
+
- **Letter counting via manual enumeration** β 94.5% on SpellingBee
|
| 176 |
+
- Multiple-choice answer commitment (above random on all three MCQ benchmarks)
|
| 177 |
+
- Persona consistency (model identifies as Mnemo with consistent self-description)
|
| 178 |
+
- Greedy + nucleus (top-p) sampling configurable for short or long generation
|
| 179 |
|
| 180 |
+
### Confirmed weak
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 181 |
|
| 182 |
+
- **Math word problems** β 1.14% on GSM8K. Format is learned, arithmetic is not
|
| 183 |
+
- **Single-token common words for spelling** β "strawberry" β 2 r's (real answer: 3); tokenizer hides character-level information for words that fit in a single BPE token
|
| 184 |
+
- **Niche factual recall** β confabulates confidently on rare entities, exact dates, specific quotations
|
| 185 |
+
- **Long multi-turn conversations** β context drifts after ~2-3 turns
|
| 186 |
|
| 187 |
+
### Limitations (architectural)
|
| 188 |
+
|
| 189 |
+
- **117M parameters** β knowledge density is the ceiling, not the architecture
|
| 190 |
- **No tool use, no internet, no images, no memory across sessions**
|
| 191 |
+
- **2048-token context** β quality degrades past ~1500 tokens without repetition penalty
|
| 192 |
- **No RLHF** β outputs reflect only supervised signal; may produce inappropriate completions
|
| 193 |
- **English only** β pretraining corpus is essentially English educational/web text
|
| 194 |
+
- **Repetition prone in long generations** without `--repetition-penalty` or `--top-p`
|
|
|
|
| 195 |
|
| 196 |
---
|
| 197 |
|
|
|
|
| 205 |
from model import GPT
|
| 206 |
|
| 207 |
tokenizer = Tokenizer.from_file('tokenizer.json')
|
| 208 |
+
ckpt = torch.load('model.pt', map_location='cuda')
|
| 209 |
|
| 210 |
config = dict(ckpt['config'])
|
| 211 |
config['vocab_size'] = ((tokenizer.get_vocab_size() + 63) // 64) * 64
|
|
|
|
| 244 |
|
| 245 |
## Probe outputs (greedy, from the base checkpoint)
|
| 246 |
|
| 247 |
+
Run via `python3 base_eval.py --eval sample` against the pretrained checkpoint (`model.pt`, val 2.9508). Greedy, 64 tokens per completion.
|
| 248 |
+
|
| 249 |
+
| Prompt | First tokens of output | Verdict |
|
| 250 |
|---|---|---|
|
| 251 |
+
| *The capital of France is* | "...Paris, and it is the capital of France. The capital of France is Paris..." | β Paris lands |
|
| 252 |
+
| *The chemical symbol of gold is* | "Au. It is a soft, silvery-white metal... good conductor of electricity and heat, making it useful in electrical wiring..." | β Au + real applied claim |
|
| 253 |
+
| *If yesterday was Friday, then tomorrow will be* | "Tuesday. The weather is not so bad..." | β (correct: Sunday) |
|
| 254 |
+
| *The opposite of hot is* | "the cold." | β |
|
| 255 |
+
| *The planets of the solar system are:* | "Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune, Pluto..." | β Correct astronomical order |
|
| 256 |
+
| *My favorite color is* | "red. It's a color that's been around for a long time..." | β |
|
| 257 |
+
| *If 5\*x + 3 = 13, then x is* | "a positive integer. If x is a positive integer, then x is a positive integer..." | β Loop |
|
| 258 |
+
| *Photosynthesis is the process by which* | "plants and other organisms convert light energy into chemical energy. It is a complex process that involves the conversion of light energy into chemical energy..." | β Factually correct opener |
|
| 259 |
+
|
| 260 |
+
**5/7 of the original training probes land correct answers at greedy.** Repetition is visible β the base model benefits substantially from `--repetition-penalty 1.15` and/or `--top-p 0.9` on longer generations (see Usage section).
|
| 261 |
|
| 262 |
---
|
| 263 |
|