README.md

---
license: apache-2.0
pipeline_tag: text-generation
language: en
tags:
  - tiny
  - tiny-lm
  - tiny-model
  - slm
  - small-language-model
  - from-scratch
  - tinystories
  - gpt
  - bpe
datasets:
  - ronendagan/TinyStories
metrics:
  - perplexity
  - accuracy
---

# tinystories-50m

A **56,902,144-parameter** transformer language model trained **from scratch** on
[TinyStories](https://huggingface.co/datasets/ronendagan/TinyStories), a corpus of
simple, repetitive children's stories. It is the 50M scale-up in the
`tinystories-24m` → `tinystories-50m` lineage.

> **v2 (2026-09-25):** retrained with a larger **12288-vocab** BPE tokenizer
> (was 8192). The 8192-vocab v1 is fully superseded — same repo, same loader,
> better weights. v1's held-out val loss was 1.6566; v2's is **1.3837**.

It writes fluent, on-domain children's stories. It is **not** a general
language model — out-of-domain generation degrades, and it should not be used
for anything beyond the story domain it was trained on.

## Architecture

| Field | Value |
|---|---|
| Parameters | **56,902,144** (exact; verified against the safetensors header) |
| Layers (L) | 16 |
| d_model (D) | 512 |
| Heads (H) | 8 (head dim 64) |
| FFN dim | 2048 (4× D) |
| Vocab | 12288 (BPE) |
| Max seq len | 512 |
| Embeddings | **weight-tied** (lm_head = tok) |
| Norm | RMSNorm (pre-norm, 2 per block + final) |
| Activation | GELU |
| Attention | causal, no bias in linear layers |
| Dtype | float32 |

Parameter breakdown (sums exactly to 56,902,144):

- token embedding: 12288 × 512 = 6,291,456
- position embedding: 512 × 512 = 262,144
- 16 blocks × 3,146,752 = 50,348,032
  - 2 × RMSNorm (512) + qkv (512×1536) + proj (512×512) + fc1 (512×2048) + fc2 (2048×512)
- final RMSNorm: 512

## Training

- **Data:** TinyStories (ronendagan/TinyStories), **523,389,481 tokens** after
  BPE-12288 re-tokenization (2,119,489 stories, ~9.19 tokens/param), with a
  2M-token held-out tail for validation.
- **Optimizer:** AdamW, cosine LR decay with warmup (peak 6e-4), grad clip 1.0.
- **Batch:** 64, seq 512 → 32,768 tokens/step.
- **Steps:** 15,910 (one full epoch). Best checkpoint at step 13,500.
- **Hardware:** single NVIDIA RTX 5090 (32 GB).
- **Final val loss:** 1.3924; **best val loss 1.3837** (step 13,500). The
  shipped weights are the end-of-run checkpoint (val 1.3924), within 0.009 of
  the best.

## Evaluated numbers

- **Held-out perplexity (TinyStories val split):** exp(1.3837) ≈ **3.99** (best
  ckpt). This is the honest primary metric for a narrow-domain model.
- **General zero-shot log-likelihood accuracy** (the 12288-vocab tokenizer can
  read these datasets, so we report them — v1's 8192-vocab tokenizer could not):

  | Task | Accuracy | n |
  |---|---|---|
  | BLiMP | 64.00% | 200 |
  | ARC-Easy | 51.09% | 599 |
  | PIQA | 45.50% | 200 |
  | HellaSwag | 54.83% | 600 |

  These are single-shot, zero-shot, no-few-shot, on a 57M model trained on one
  narrow domain — treat them as a scale reference, not a competitive result.

- **Coherence:** seeded generations are fluent, on-domain, with consistent
  characters and correct punctuation. Minor artifacts expected at this scale
  (occasional garbled quote char, a couple of logical slips).

## Files

| File | What |
|---|---|
| `model.safetensors` | weights (227 MB, 99 tensors, float32) |
| `tokenizer.json` | BPE-12288 tokenizer (`tokenizers` format) |
| `config.json` | architecture config |
| `load_model.py` | self-contained loader + `TinyStoriesGPT` class |

## Usage

```python
from load_model import load
model, tok = load()
ids = tok.encode("Once upon a time,")
out = model.generate(torch.tensor([ids]).cuda(), 100, temp=0.8, top_k=40)
print(tok.decode(out[0].tolist(), skip_special_tokens=True))
```

## What it is and is not

- **Is:** a small, from-scratch, on-domain story generator. Good for studying
  how a ~57M transformer learns a narrow, repetitive domain.
- **Is not:** a general-purpose LM. Do not expect coherent output on code,
  math, or open-domain text. The low perplexity is domain-specific.