tiny-lm-shakespeare

A 2.7M-parameter GPT trained on Tiny Shakespeare. It is the last step of Part 1 of tiny-lm, a project that builds a language model step by step, from a bigram table to a GPT.

Context	64 tokens
Size	6 layers, width 192, 6 heads
Tokeniser	byte-pair encoding, 512 tokens (bpe.json)
Validation loss	1.50 nats per character

The model is not GPT-2 shaped (it uses ReLU and a separate output layer with a bias). It is not a transformers model. Load it with the GPT class from the tiny-lm repository:

import json
from pathlib import Path
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from core import gpt          # from the tiny-lm repository
from core.bpe import BPE

d = Path(snapshot_download("lillycham/tiny-lm-shakespeare"))
model = gpt.GPT(**json.loads((d / "config.json").read_text()))
model.load_state_dict(load_file(d / "model.safetensors"))
model.eval()
tok = BPE.load(d / "bpe.json")
print(gpt.generate(model, 200, "ROMEO:", 0.8, tok.encode, tok.decode))

The model writes text that looks like a play, but the words rarely make sense.

Data

Tiny Shakespeare (Karpathy, 2015). The text is in the public domain. The code and weights are MIT.