kryln/tinystories-minigpt

MiniGPT - TinyStories

A ~5M parameter character-level GPT model trained from scratch on TinyStories.

Model Details

Parameter	Value
Parameters	~4.9M
Vocab Size	72 (character-level)
Embedding Dim	256
Layers	6
Attention Heads	8
Context Length	128
Training Steps	3,000
Final Val Loss	0.9235

Architecture

Standard mini-GPT architecture:

• Token Embedding + Position Embedding
• 6 Transformer Blocks (Causal Self-Attention + MLP)
• LayerNorm + Residual Connections
• Weight Tying (input embedding = output projection)

Training

Trained on CPU for ~1 hour 17 minutes on the TinyStories dataset.

• Optimizer: AdamW (lr=3e-4, weight_decay=0.01)
• Schedule: Warmup (200 steps) + Cosine Decay
• Batch Size: 64, Block Size: 128

Usage

import torch
from train_torch import MiniGPT

# Load model
model = MiniGPT(vocab_size=72, n_embd=256, n_head=8, n_layer=6, block_size=128, dropout=0.1)
model.load_state_dict(torch.load("model.safetensors"))
model.eval()

# Generate
context = torch.zeros((1, 1), dtype=torch.long)
output = model.generate(context, max_new_tokens=200, temperature=0.7)

Sample Generation (Temperature=0.7)

He saw a bike with a big box. He was a girl named Sue. Sue liked to give it very happy. One day, she got to the little boy named Tim. Tim was very happy. He wind the best friends was friends. The tree...

License

MIT