Org card: precise GPT-2 comparison (validation stream)
Browse files
README.md
CHANGED
|
@@ -38,7 +38,7 @@ Small models trained from scratch with the same recipe (Muon, rotary positions,
|
|
| 38 |
|
| 39 |
| Model | Parameters | Trained on | Result |
|
| 40 |
|---|---|---|---|
|
| 41 |
-
| [mume-english-125m](https://huggingface.co/MuseMesh/mume-english-125m) | 134M | 1.33B FineWeb tokens | 1.106 bits per byte on FineWeb val
|
| 42 |
| [mume-math-125m](https://huggingface.co/MuseMesh/mume-math-125m) | 134M | 1.33B OpenWebMath tokens | MATH test (clean) 62.6% fewer bits than the English model; branch `sft` is a GSM8K/MATH fine-tune |
|
| 43 |
|
| 44 |
- **[mume-tokenizer-32k](https://huggingface.co/MuseMesh/mume-tokenizer-32k)**: the shared 32k unigram tokenizer.
|
|
|
|
| 38 |
|
| 39 |
| Model | Parameters | Trained on | Result |
|
| 40 |
|---|---|---|---|
|
| 41 |
+
| [mume-english-125m](https://huggingface.co/MuseMesh/mume-english-125m) | 134M | 1.33B FineWeb tokens | 1.106 bits per byte on frozen FineWeb val; on the validation stream 1.132 vs 1.176 for a plain-AdamW GPT-2 124M at the same tokens (3.7% lower) |
|
| 42 |
| [mume-math-125m](https://huggingface.co/MuseMesh/mume-math-125m) | 134M | 1.33B OpenWebMath tokens | MATH test (clean) 62.6% fewer bits than the English model; branch `sft` is a GSM8K/MATH fine-tune |
|
| 43 |
|
| 44 |
- **[mume-tokenizer-32k](https://huggingface.co/MuseMesh/mume-tokenizer-32k)**: the shared 32k unigram tokenizer.
|