File size: 3,461 Bytes
8af5ac4 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 | ---
license: apache-2.0
pipeline_tag: text-generation
language: en
tags:
- tiny
- tiny-lm
- tiny-model
- slm
- small-language-model
- from-scratch
- tinystories
- gpt
- bpe
datasets:
- ronendagan/TinyStories
metrics:
- perplexity
---
# tinystories-50m
A **54,804,992-parameter** transformer language model trained **from scratch** on
[TinyStories](https://huggingface.co/datasets/ronendagan/TinyStories), a corpus of
simple, repetitive children's stories. It is the 50M scale-up in the
`tinystories-24m` → `tinystories-50m` lineage (the 24M sibling was coherent at
18.2 tok/param; this one trains at 8.17 tok/param on the same narrow domain).
It writes fluent, on-domain children's stories. It is **not** a general
language model — out-of-domain generation degrades, and it should not be used
for anything beyond the story domain it was trained on.
## Architecture
| Field | Value |
|---|---|
| Parameters | **54,804,992** (exact; verified against the safetensors header) |
| Layers (L) | 16 |
| d_model (D) | 512 |
| Heads (H) | 8 (head dim 64) |
| FFN dim | 2048 (4× D) |
| Vocab | 8192 (BPE) |
| Max seq len | 512 |
| Embeddings | **weight-tied** (lm_head = tok) |
| Norm | RMSNorm (pre-norm, 2 per block + final) |
| Activation | GELU |
| Attention | causal, no bias in linear layers |
| Dtype | float32 |
Parameter breakdown (sums exactly to 54,804,992):
- token embedding: 8192 × 512 = 4,194,304
- position embedding: 512 × 512 = 262,144
- 16 blocks × 3,146,752 = 50,348,032
- 2 × RMSNorm (512) + qkv (512×1536) + proj (512×512) + fc1 (512×2048) + fc2 (2048×512)
- final RMSNorm: 512
## Training
- **Data:** TinyStories (ronendagan/TinyStories), ~447.86M tokens after BPE-8192
re-tokenization, 8.17 tokens/param.
- **Optimizer:** AdamW, cosine LR decay with warmup (peak 6e-4).
- **Batch:** 64, seq 512 → 32,768 tokens/step.
- **Steps:** 13,668 (one full epoch). Best checkpoint at step 13,250.
- **Hardware:** single NVIDIA RTX 5090 (32 GB), peak ~15 GB.
- **Final val loss:** 1.6371 (best ckpt 1.6566 @ step 13,250).
## Evaluated numbers
- **Held-out perplexity (TinyStories val split):** **5.24** (best ckpt,
val cross-entropy 1.6566 → exp = 5.2412). This is the honest metric for a
narrow-domain model; standard general benchmarks (BLiMP/ARC/PIQA) are not
meaningful here and are deliberately not reported.
- **Coherence:** 9/9 seeded generations (3 story-start prompts × 3 seeds) are
fluent, on-domain, with consistent characters and correct punctuation. Minor
artifacts expected at this scale (occasional garbled quote char, a couple of
logical slips).
## Files
| File | What |
|---|---|
| `model.safetensors` | weights (210 MB, 99 tensors, float32) |
| `tokenizer.json` | BPE-8192 tokenizer (`tokenizers` format) |
| `config.json` | architecture config |
| `load_model.py` | self-contained loader + `TinyStoriesGPT` class |
## Usage
```python
from load_model import load
model, tok = load()
ids = tok.encode("Once upon a time,")
out = model.generate(torch.tensor([ids]).cuda(), 100, temp=0.8, top_k=40)
print(tok.decode(out[0].tolist(), skip_special_tokens=True))
```
## What it is and is not
- **Is:** a small, from-scratch, on-domain story generator. Good for studying
how a ~55M transformer learns a narrow, repetitive domain.
- **Is not:** a general-purpose LM. Do not expect coherent output on code,
math, or open-domain text. The low perplexity is domain-specific.
|