| --- |
| license: mit |
| language: en |
| pipeline_tag: text-generation |
| tags: |
| - gpt |
| - tiny-lm |
| - educational |
| datasets: |
| - karpathy/tiny_shakespeare |
| --- |
| |
| # tiny-lm-shakespeare |
|
|
| A 2.7M-parameter GPT trained on Tiny Shakespeare. It is the last step of Part 1 of |
| [tiny-lm](https://github.com/lillycham/tiny-lm), a project that builds a language model step by |
| step, from a bigram table to a GPT. |
|
|
| | | | |
| |---|---| |
| | Context | 64 tokens | |
| | Size | 6 layers, width 192, 6 heads | |
| | Tokeniser | byte-pair encoding, 512 tokens (`bpe.json`) | |
| | Validation loss | 1.50 nats per character | |
|
|
| The model is not GPT-2 shaped (it uses ReLU and a separate output layer with a bias). It is not a |
| `transformers` model. Load it with the `GPT` class from the tiny-lm repository: |
|
|
| ```python |
| import json |
| from pathlib import Path |
| from huggingface_hub import snapshot_download |
| from safetensors.torch import load_file |
| from core import gpt # from the tiny-lm repository |
| from core.bpe import BPE |
| |
| d = Path(snapshot_download("lillycham/tiny-lm-shakespeare")) |
| model = gpt.GPT(**json.loads((d / "config.json").read_text())) |
| model.load_state_dict(load_file(d / "model.safetensors")) |
| model.eval() |
| tok = BPE.load(d / "bpe.json") |
| print(gpt.generate(model, 200, "ROMEO:", 0.8, tok.encode, tok.decode)) |
| ``` |
|
|
| The model writes text that looks like a play, but the words rarely make sense. |
|
|
| ## Data |
|
|
| [Tiny Shakespeare](https://github.com/karpathy/char-rnn) (Karpathy, 2015). The text is in the public |
| domain. The code and weights are MIT. |
|
|
