File size: 1,501 Bytes
8679921 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 | ---
license: mit
language: en
pipeline_tag: text-generation
tags:
- gpt
- tiny-lm
- educational
datasets:
- karpathy/tiny_shakespeare
---
# tiny-lm-shakespeare
A 2.7M-parameter GPT trained on Tiny Shakespeare. It is the last step of Part 1 of
[tiny-lm](https://github.com/lillycham/tiny-lm), a project that builds a language model step by
step, from a bigram table to a GPT.
| | |
|---|---|
| Context | 64 tokens |
| Size | 6 layers, width 192, 6 heads |
| Tokeniser | byte-pair encoding, 512 tokens (`bpe.json`) |
| Validation loss | 1.50 nats per character |
The model is not GPT-2 shaped (it uses ReLU and a separate output layer with a bias). It is not a
`transformers` model. Load it with the `GPT` class from the tiny-lm repository:
```python
import json
from pathlib import Path
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from core import gpt # from the tiny-lm repository
from core.bpe import BPE
d = Path(snapshot_download("lillycham/tiny-lm-shakespeare"))
model = gpt.GPT(**json.loads((d / "config.json").read_text()))
model.load_state_dict(load_file(d / "model.safetensors"))
model.eval()
tok = BPE.load(d / "bpe.json")
print(gpt.generate(model, 200, "ROMEO:", 0.8, tok.encode, tok.decode))
```
The model writes text that looks like a play, but the words rarely make sense.
## Data
[Tiny Shakespeare](https://github.com/karpathy/char-rnn) (Karpathy, 2015). The text is in the public
domain. The code and weights are MIT.
|