v2 card: 12288-vocab retrain, 56,902,144 params, val 1.3837, measured evals

README.md

@@ -16,15 +16,19 @@ datasets:
   - ronendagan/TinyStories
 metrics:
   - perplexity
+  - accuracy
 ---
 
 # tinystories-50m
 
-A **54,804,992-parameter** transformer language model trained **from scratch** on
+A **56,902,144-parameter** transformer language model trained **from scratch** on
 [TinyStories](https://huggingface.co/datasets/ronendagan/TinyStories), a corpus of
 simple, repetitive children's stories. It is the 50M scale-up in the
-`tinystories-24m` → `tinystories-50m` lineage (the 24M sibling was coherent at
-18.2 tok/param; this one trains at 8.17 tok/param on the same narrow domain).
+`tinystories-24m` → `tinystories-50m` lineage.
+
+> **v2 (2026-09-25):** retrained with a larger **12288-vocab** BPE tokenizer
+> (was 8192). The 8192-vocab v1 is fully superseded — same repo, same loader,
+> better weights. v1's held-out val loss was 1.6566; v2's is **1.3837**.
 
 It writes fluent, on-domain children's stories. It is **not** a general
 language model — out-of-domain generation degrades, and it should not be used
@@ -34,12 +38,12 @@ for anything beyond the story domain it was trained on.
 
 | Field | Value |
 |---|---|
-| Parameters | **54,804,992** (exact; verified against the safetensors header) |
+| Parameters | **56,902,144** (exact; verified against the safetensors header) |
 | Layers (L) | 16 |
 | d_model (D) | 512 |
 | Heads (H) | 8 (head dim 64) |
 | FFN dim | 2048 (4× D) |
-| Vocab | 8192 (BPE) |
+| Vocab | 12288 (BPE) |
 | Max seq len | 512 |
 | Embeddings | **weight-tied** (lm_head = tok) |
 | Norm | RMSNorm (pre-norm, 2 per block + final) |
@@ -47,9 +51,9 @@ for anything beyond the story domain it was trained on.
 | Attention | causal, no bias in linear layers |
 | Dtype | float32 |
 
-Parameter breakdown (sums exactly to 54,804,992):
+Parameter breakdown (sums exactly to 56,902,144):
 
-- token embedding: 8192 × 512 = 4,194,304
+- token embedding: 12288 × 512 = 6,291,456
 - position embedding: 512 × 512 = 262,144
 - 16 blocks × 3,146,752 = 50,348,032
   - 2 × RMSNorm (512) + qkv (512×1536) + proj (512×512) + fc1 (512×2048) + fc2 (2048×512)
@@ -57,31 +61,44 @@ Parameter breakdown (sums exactly to 54,804,992):
 
 ## Training
 
-- **Data:** TinyStories (ronendagan/TinyStories), ~447.86M tokens after BPE-8192
-  re-tokenization, 8.17 tokens/param.
-- **Optimizer:** AdamW, cosine LR decay with warmup (peak 6e-4).
+- **Data:** TinyStories (ronendagan/TinyStories), **523,389,481 tokens** after
+  BPE-12288 re-tokenization (2,119,489 stories, ~9.19 tokens/param), with a
+  2M-token held-out tail for validation.
+- **Optimizer:** AdamW, cosine LR decay with warmup (peak 6e-4), grad clip 1.0.
 - **Batch:** 64, seq 512 → 32,768 tokens/step.
-- **Steps:** 13,668 (one full epoch). Best checkpoint at step 13,250.
-- **Hardware:** single NVIDIA RTX 5090 (32 GB), peak ~15 GB.
-- **Final val loss:** 1.6371 (best ckpt 1.6566 @ step 13,250).
+- **Steps:** 15,910 (one full epoch). Best checkpoint at step 13,500.
+- **Hardware:** single NVIDIA RTX 5090 (32 GB).
+- **Final val loss:** 1.3924; **best val loss 1.3837** (step 13,500). The
+  shipped weights are the end-of-run checkpoint (val 1.3924), within 0.009 of
+  the best.
 
 ## Evaluated numbers
 
-- **Held-out perplexity (TinyStories val split):** **5.24** (best ckpt,
-  val cross-entropy 1.6566 → exp = 5.2412). This is the honest metric for a
-  narrow-domain model; standard general benchmarks (BLiMP/ARC/PIQA) are not
-  meaningful here and are deliberately not reported.
-- **Coherence:** 9/9 seeded generations (3 story-start prompts × 3 seeds) are
-  fluent, on-domain, with consistent characters and correct punctuation. Minor
-  artifacts expected at this scale (occasional garbled quote char, a couple of
-  logical slips).
+- **Held-out perplexity (TinyStories val split):** exp(1.3837) ≈ **3.99** (best
+  ckpt). This is the honest primary metric for a narrow-domain model.
+- **General zero-shot log-likelihood accuracy** (the 12288-vocab tokenizer can
+  read these datasets, so we report them — v1's 8192-vocab tokenizer could not):
+
+  | Task | Accuracy | n |
+  |---|---|---|
+  | BLiMP | 64.00% | 200 |
+  | ARC-Easy | 51.09% | 599 |
+  | PIQA | 45.50% | 200 |
+  | HellaSwag | 54.83% | 600 |
+
+  These are single-shot, zero-shot, no-few-shot, on a 57M model trained on one
+  narrow domain — treat them as a scale reference, not a competitive result.
+
+- **Coherence:** seeded generations are fluent, on-domain, with consistent
+  characters and correct punctuation. Minor artifacts expected at this scale
+  (occasional garbled quote char, a couple of logical slips).
 
 ## Files
 
 | File | What |
 |---|---|
-| `model.safetensors` | weights (210 MB, 99 tensors, float32) |
-| `tokenizer.json` | BPE-8192 tokenizer (`tokenizers` format) |
+| `model.safetensors` | weights (227 MB, 99 tensors, float32) |
+| `tokenizer.json` | BPE-12288 tokenizer (`tokenizers` format) |
 | `config.json` | architecture config |
 | `load_model.py` | self-contained loader + `TinyStoriesGPT` class |
 
@@ -98,6 +115,6 @@ print(tok.decode(out[0].tolist(), skip_special_tokens=True))
 ## What it is and is not
 
 - **Is:** a small, from-scratch, on-domain story generator. Good for studying
-  how a ~55M transformer learns a narrow, repetitive domain.
+  how a ~57M transformer learns a narrow, repetitive domain.
 - **Is not:** a general-purpose LM. Do not expect coherent output on code,
-  math, or open-domain text. The low perplexity is domain-specific.
+  math, or open-domain text. The low perplexity is domain-specific.