Add model card: 54.8M from-scratch TinyStories GPT, verified params + held-out PPL 5.24

README.md

@@ -0,0 +1,103 @@
+---
+license: apache-2.0
+pipeline_tag: text-generation
+language: en
+tags:
+  - tiny
+  - tiny-lm
+  - tiny-model
+  - slm
+  - small-language-model
+  - from-scratch
+  - tinystories
+  - gpt
+  - bpe
+datasets:
+  - ronendagan/TinyStories
+metrics:
+  - perplexity
+---
+
+# tinystories-50m
+
+A **54,804,992-parameter** transformer language model trained **from scratch** on
+[TinyStories](https://huggingface.co/datasets/ronendagan/TinyStories), a corpus of
+simple, repetitive children's stories. It is the 50M scale-up in the
+`tinystories-24m` → `tinystories-50m` lineage (the 24M sibling was coherent at
+18.2 tok/param; this one trains at 8.17 tok/param on the same narrow domain).
+
+It writes fluent, on-domain children's stories. It is **not** a general
+language model — out-of-domain generation degrades, and it should not be used
+for anything beyond the story domain it was trained on.
+
+## Architecture
+
+| Field | Value |
+|---|---|
+| Parameters | **54,804,992** (exact; verified against the safetensors header) |
+| Layers (L) | 16 |
+| d_model (D) | 512 |
+| Heads (H) | 8 (head dim 64) |
+| FFN dim | 2048 (4× D) |
+| Vocab | 8192 (BPE) |
+| Max seq len | 512 |
+| Embeddings | **weight-tied** (lm_head = tok) |
+| Norm | RMSNorm (pre-norm, 2 per block + final) |
+| Activation | GELU |
+| Attention | causal, no bias in linear layers |
+| Dtype | float32 |
+
+Parameter breakdown (sums exactly to 54,804,992):
+
+- token embedding: 8192 × 512 = 4,194,304
+- position embedding: 512 × 512 = 262,144
+- 16 blocks × 3,146,752 = 50,348,032
+  - 2 × RMSNorm (512) + qkv (512×1536) + proj (512×512) + fc1 (512×2048) + fc2 (2048×512)
+- final RMSNorm: 512
+
+## Training
+
+- **Data:** TinyStories (ronendagan/TinyStories), ~447.86M tokens after BPE-8192
+  re-tokenization, 8.17 tokens/param.
+- **Optimizer:** AdamW, cosine LR decay with warmup (peak 6e-4).
+- **Batch:** 64, seq 512 → 32,768 tokens/step.
+- **Steps:** 13,668 (one full epoch). Best checkpoint at step 13,250.
+- **Hardware:** single NVIDIA RTX 5090 (32 GB), peak ~15 GB.
+- **Final val loss:** 1.6371 (best ckpt 1.6566 @ step 13,250).
+
+## Evaluated numbers
+
+- **Held-out perplexity (TinyStories val split):** **5.24** (best ckpt,
+  val cross-entropy 1.6566 → exp = 5.2412). This is the honest metric for a
+  narrow-domain model; standard general benchmarks (BLiMP/ARC/PIQA) are not
+  meaningful here and are deliberately not reported.
+- **Coherence:** 9/9 seeded generations (3 story-start prompts × 3 seeds) are
+  fluent, on-domain, with consistent characters and correct punctuation. Minor
+  artifacts expected at this scale (occasional garbled quote char, a couple of
+  logical slips).
+
+## Files
+
+| File | What |
+|---|---|
+| `model.safetensors` | weights (210 MB, 99 tensors, float32) |
+| `tokenizer.json` | BPE-8192 tokenizer (`tokenizers` format) |
+| `config.json` | architecture config |
+| `load_model.py` | self-contained loader + `TinyStoriesGPT` class |
+
+## Usage
+
+```python
+from load_model import load
+model, tok = load()
+ids = tok.encode("Once upon a time,")
+out = model.generate(torch.tensor([ids]).cuda(), 100, temp=0.8, top_k=40)
+print(tok.decode(out[0].tolist(), skip_special_tokens=True))
+```
+
+## What it is and is not
+
+- **Is:** a small, from-scratch, on-domain story generator. Good for studying
+  how a ~55M transformer learns a narrow, repetitive domain.
+- **Is not:** a general-purpose LM. Do not expect coherent output on code,
+  math, or open-domain text. The low perplexity is domain-specific.