MiniGPT 🌟 Decoder-Only Language Model (TinyStories)
MiniGPT is a family of decoder-only GPT-style transformer language models built completely from scratch in PyTorch and trained on the TinyStories V2 dataset using an NVIDIA GeForce RTX 4060 Laptop GPU.
This repository contains three trained checkpoints (Model S, Model M, and Model L) demonstrating empirical Chinchilla scaling laws ($N \approx D/20$) on synthetic children's stories.
🎮 Try the live interactive web demo: Hugging Face Space: siddheshrj/minigpt-stories
📊 Scaling Laws & Benchmark
All models were evaluated on the held-out TinyStories validation set ($5.58\text{M}$ tokens).
| Model | Parameters | Tokens Seen | Val Loss | Perplexity | Bits / Byte | Speed (tok/s) | Peak VRAM | Training Time |
|---|---|---|---|---|---|---|---|---|
| Model S | $4,196,608$ ($4.2\text{M}$) | $85\text{M}$ | 1.7765 | 5.91 | 0.6476 | ~259,200 | 942 MiB | 5.5 min |
| Model M | $12,194,688$ ($12.2\text{M}$) | $242\text{M}$ | 1.3412 | 3.82 | 0.4889 | ~110,200 | 1,624 MiB | 37 min |
| Model L | $27,271,680$ ($27.3\text{M}$) | $544\text{M}$ | 1.1528 | 3.17 | 0.4202 | ~55,060 | 2,524 MiB | 2.76 hours |
Empirical Power-Law Scaling Fit:
As model size and token budget scale, perplexity drops smoothly from $5.91 \to 3.82 \to 3.17$ and compression improves from $0.6476 \to 0.4202$ bits/byte.
📖 Sample Outputs Comparison
Prompt: "Once upon a time, a clever robot"
Model L (27.3M Parameters — Recommended)
Once upon a time, a clever robot named Robby lived in a small town. Robby loved to dance. He danced all day and all night. The people in the town loved to watch Robby dance.
One day, a big man named Tom came to the town. Tom said, "I want to dance too!" Robby asked, "Can I dance with you?" Tom said, "Yes, let's dance together!" So, Robby and Tom started to dance. They were very happy.
But then, something unexpected happened. Robby's dance made everyone in the town look angry. Tim said, "Robby, we don't know that dancing is not good. It makes people sad." Robby said, "I am sorry. I just wanted to dance with everyone." The people in the town saw that Robby was being kind and helpful. They all started to dance too.
The moral of the story is that helping others can make everyone happy, even when we don't like it.
🛠️ Modern Decoder-Only Architecture Details
MiniGPT is engineered from scratch in pure PyTorch, adopting state-of-the-art LLM design paradigms (inspired by LLaMA and Chinchilla):
- Rotary Position Embeddings (RoPE): Relative complex rotations applied to Query & Key projections ($\theta_i = 10000^{-2(i-1)/d}$), providing natural distance decay and length generalization.
- Pre-RMSNorm Normalization: Root Mean Square Normalization with float32 accumulation for training stability under bfloat16, keeping the main residual identity path clean.
- Hardware-Fused Causal SDPA: PyTorch
scaled_dot_product_attentionwith fused causal kernels (FlashAttention / Memory-Efficient / cuDNN), preventing quadratic $O(T^2)$ attention matrix materialization. - Weight Tying & Bias-Free Linears: Unembedding projection shares weights directly with token embeddings ($W_{\text{out}} = E^T$), reducing parameter count by $V \times d_{\text{model}}$ while linear layers omit additive biases.
- 4× Expansion MLP with GELU: Two-stage feed-forward projection ($d_{\text{model}} \to 4d_{\text{model}} \to d_{\text{model}}$) with Gaussian Error Linear Units.
- Pre-Allocated $O(1)$ KV-Cache: Pre-allocated key/value memory buffers across layers, transforming generation complexity from $O(T^2)$ recomputation into strictly $O(T)$ linear-time streaming.
- Domain BPE Tokenizer ($V=8,192$): Byte-Pair Encoding trained on TinyStories with explicit
<|endoftext|>story delimiter boundaries (~3.8 bytes/token).
| Configuration | Layers ($N$) | Attention Heads ($H$) | $d_{\text{model}}$ | Head Dim ($d_h$) | Context Length |
|---|---|---|---|---|---|
| Model S | 4 | 4 | 256 | 64 | 512 |
| Model M | 6 | 6 | 384 | 64 | 512 |
| Model L | 8 | 8 | 512 | 64 | 512 |
💻 How to Use
Download the checkpoint and tokenizer directly from Hugging Face:
from huggingface_hub import hf_hub_download
import torch
# 1. Download Model L checkpoint and tokenizer
ckpt_path = hf_hub_download(repo_id="siddheshrj/minigpt-tinystories", filename="model_L.pt")
tok_path = hf_hub_download(repo_id="siddheshrj/minigpt-tinystories", filename="tokenizer.json")
# 2. Inspect checkpoint structure
checkpoint = torch.load(ckpt_path, map_location="cpu")
print("Model Config:", checkpoint["model_config"])
print("Step:", checkpoint["step"])
print("Best Validation Loss:", checkpoint["best_val_loss"])
To run inference with the repository code:
git clone https://github.com/SiddheshJajurley/minigpt.git
cd minigpt
uv sync --extra cpu
uv run python scripts/generate.py --checkpoint runs/rtx4060_L/best.pt --prompt "Once upon a time" -n 1
📜 Citation & Credits
- Dataset: TinyStories by Ronen Eldan & Yuanzhi Li.
- Training & Implementation: Siddhesh Jajurley (@siddheshrj).
- License: MIT.