karpathy/tiny_shakespeare
Updated โข 18.7k โข 103
A small decoder-only transformer trained character by character on Tiny Shakespeare.
| Parameters | 10.75M (6 layers, 6 heads, 384-dim, context 256) |
| Vocabulary | 65 characters |
| Best validation loss | 1.4642 (step 2000) |
| Training | 5,000 steps, ~14 min on a single T4, best-validation checkpoint kept |
Trained with a custom optimizer: โจ details coming soon, surprise! โจ
# pip install torch
# files in this repo: best_ckpt.pt, config.json, model.py, inference.py
from inference import Generator
g = Generator("best_ckpt.pt")
print(g.generate("ROMEO:", max_new_tokens=300, temperature=0.8))
Live demo + API: https://huggingface.co/spaces/hariomlohardev/shakespeare-gpt-demo
It imitates the style of Shakespeare, not the meaning: expect plausible-looking nonsense. Trained on ~1 MB of text, so it overfits quickly after step ~2000. English only, characters outside its 65-symbol vocabulary are ignored.