---
license: mit
library_name: pytorch
tags:
  - text-generation
  - gpt
  - from-scratch
  - tinystories
datasets:
  - roneneldan/TinyStories
language:
  - en
metrics:
  - perplexity
---

# Tiny GPT (5.8M parameters)

A small GPT-2-style decoder-only transformer, written from scratch in PyTorch (no `transformers` classes), trained on a
laptop CPU on a subset of [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories). It continues short children's stories.
Code, training logs and the full write-up are in the GitHub repository.

## Model details
| | |
|---|---|
| Architecture | pre-norm GPT, 6 layers, d_model 256, 8 heads, context 256, learned positions, tied embeddings |
| Parameters | 5,853,184 |
| Tokenizer | byte-level BPE, vocabulary 4096, trained on the training split only (`tokenizer.json`) |
| Files | `model.safetensors`, `model_config.json` (our `ModelConfig`), `tokenizer.json`, `model_info.json` |

## Training
- **Data:** first 300,000 stories (60.8M tokens) of TinyStories V2 (GPT-4 generated); validation on 5,000 held-out stories.
- **Run:** 9,500 steps x 8,192 tokens = 77.8M tokens (about 1.28 passes over the data), AdamW, peak lr 1e-3 with warmup and cosine decay, 12 CPU threads, 7.9 hours of running time on one laptop (Intel Core Ultra 9 185H).

## Evaluation
| Model | Validation loss | Perplexity |
|---|---|---|
| Uniform guess | 8.32 | 4096 |
| Unigram counts | 5.88 | 357 |
| Bigram counts | 3.65 | 38.4 |
| **This model** | **1.688** | **5.41** |

Perplexity is per BPE token with this 4096-token tokenizer, on TinyStories text, so it is not comparable to numbers from other tokenizers or datasets.

## Intended use
Learning, demos and experiments with small language models; generating simple children's-story-style text.

## Limitations and risks
- **Narrow domain.** It only knows the simple vocabulary and plots of TinyStories. It cannot follow instructions, answer questions or hold a conversation.
- **Quality.** Stories are fluent at sentence level but repeat phrases, lose objects and plot threads, and sometimes contradict themselves. Samples in the repository's `docs/RESULTS.md` include failures, not just good examples.
- **Synthetic data.** The training stories were written by a large language model, so the model inherits that model's stylistic and social biases (names, gender roles, settings). It has not been evaluated for bias or safety.
- **Not for** factual content, advice, or anything where wrong or biased text could cause harm.

## Data and licence
Code and weights in this repository are released under the MIT licence. The training data, TinyStories by Ronen Eldan and Yuanzhi Li, is
licensed CDLA-Sharing-1.0 on the dataset card; please credit it and check the dataset card for current terms.

## Usage
```bash
git clone <github repo> && cd tiny-gpt && make setup
TINYGPT_HF_REPO=<this repo id> TINYGPT_MODEL_DIR=./model uv run uvicorn tinygpt.serving.app:create_app --factory
```