ks2812 commited on
Commit
3a7c8ae
·
verified ·
1 Parent(s): 55625b9

Org card: precise GPT-2 comparison (validation stream)

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -38,7 +38,7 @@ Small models trained from scratch with the same recipe (Muon, rotary positions,
38
 
39
  | Model | Parameters | Trained on | Result |
40
  |---|---|---|---|
41
- | [mume-english-125m](https://huggingface.co/MuseMesh/mume-english-125m) | 134M | 1.33B FineWeb tokens | 1.106 bits per byte on FineWeb val, 3.7% below a plain-AdamW GPT-2 124M at the same tokens |
42
  | [mume-math-125m](https://huggingface.co/MuseMesh/mume-math-125m) | 134M | 1.33B OpenWebMath tokens | MATH test (clean) 62.6% fewer bits than the English model; branch `sft` is a GSM8K/MATH fine-tune |
43
 
44
  - **[mume-tokenizer-32k](https://huggingface.co/MuseMesh/mume-tokenizer-32k)**: the shared 32k unigram tokenizer.
 
38
 
39
  | Model | Parameters | Trained on | Result |
40
  |---|---|---|---|
41
+ | [mume-english-125m](https://huggingface.co/MuseMesh/mume-english-125m) | 134M | 1.33B FineWeb tokens | 1.106 bits per byte on frozen FineWeb val; on the validation stream 1.132 vs 1.176 for a plain-AdamW GPT-2 124M at the same tokens (3.7% lower) |
42
  | [mume-math-125m](https://huggingface.co/MuseMesh/mume-math-125m) | 134M | 1.33B OpenWebMath tokens | MATH test (clean) 62.6% fewer bits than the English model; branch `sft` is a GSM8K/MATH fine-tune |
43
 
44
  - **[mume-tokenizer-32k](https://huggingface.co/MuseMesh/mume-tokenizer-32k)**: the shared 32k unigram tokenizer.