File size: 1,460 Bytes
3c38b94 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 | ---
license: apache-2.0
language:
- en
tags:
- text-generation
- gpt
- from-scratch
- tinystories
pipeline_tag: text-generation
---
# mini-llm-tinystories
An 18.3M-parameter GPT-2-style language model **trained completely from scratch on CPU**
(no pretrained weights). It generates short, coherent children's-story-style English.
## Details
- **Architecture:** GPT-2 style decoder (pre-norm, GELU, weight-tied head)
- **Params:** ~18.3M — 448 dim, 7 heads, 6 layers, 256 context
- **Tokenizer:** byte-level BPE, 8192 vocab
- **Training data:** TinyStories (~90M tokens) + a small amount of Alpaca Q&A
- **Training:** ~7.6 hours on 2 CPU cores, final train loss ~1.86
- **Type:** base **completion** model (continues text; not instruction-tuned)
## Usage
```python
import torch, json
from safetensors.torch import load_file
from tokenizers import ByteLevelBPETokenizer
from gpt2 import GPT2 # include gpt2.py from this repo
cfg = json.load(open("config.json"))
model = GPT2(cfg)
model.load_state_dict(load_file("model.safetensors"))
model.eval()
tk = ByteLevelBPETokenizer("tokenizer_bpe/vocab.json", "tokenizer_bpe/merges.txt")
ids = tk.encode("Once upon a time").ids
out = model.generate(torch.tensor([ids]), max_new_tokens=150)
print(tk.decode(out[0].tolist()))
```
## Limitations
Tiny model with no real-world knowledge. Best at short narrative completions in a
TinyStories style. Will produce fluent but factually-wrong text if asked questions.
|