File size: 3,522 Bytes
4bc8857 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 | ---
license: mit
language:
- en
library_name: pytorch
pipeline_tag: text-generation
tags:
- tiny
- character-level
- gpt
- from-scratch
- educational
---
# Tiny LLM
A very small GPT-style language model, trained from scratch in about 15 minutes
on a laptop CPU. It writes one character at a time and only knows simple
greetings: hello, how are you, what is your name, thanks, goodbye.
It is a learning project, not a useful assistant. Anything that is not a
greeting gets an unrelated greeting-style reply.
**Try it in your browser:** [Tiny LLM demo Space](https://huggingface.co/spaces/Neural-Labs/NerualLabs)
## Model details
| | |
| --- | --- |
| Type | Decoder-only transformer (GPT-style), character level |
| Parameters | 824,120 |
| Layers | 4 |
| Attention heads | 4 |
| Embedding width | 128 |
| Context length | 128 characters |
| Vocabulary | 56 characters (the distinct characters in the training text) |
| Language | English |
| License | MIT |
Each block uses pre-layer-norm, causal self-attention, and an MLP
(128 → 512 → 128) with GELU. Token and position embeddings are both learned.
## Files
| File | What it is |
| --- | --- |
| `model.pt` | Trained weights: a dict with `state_dict`, `chars` (the vocabulary) and `config` |
| `model.py` | The model code in PyTorch, about 60 lines |
## How to use
This is a custom PyTorch model, so it does not load with `transformers`.
Download `model.py` and `model.pt`, then:
```bash
pip install torch huggingface_hub
```
```python
import torch
from huggingface_hub import hf_hub_download
repo = "Neural-Labs/tiny-llm"
hf_hub_download(repo, "model.py", local_dir=".")
weights = hf_hub_download(repo, "model.pt")
from model import TinyGPT
ckpt = torch.load(weights, map_location="cpu", weights_only=True)
chars = ckpt["chars"]
stoi = {c: i for i, c in enumerate(chars)}
model = TinyGPT(len(chars), **ckpt["config"])
model.load_state_dict(ckpt["state_dict"])
model.eval()
def reply(message):
# Characters the model never saw are dropped.
clean = "".join(c for c in message.replace("\n", " ") if c in stoi).strip() or "hi"
prompt = f"User: {clean[:80]}\nBot:"
idx = torch.tensor([[stoi[c] for c in prompt]])
out = ""
for tok in model.generate(idx, max_new_tokens=100, temperature=0.5, top_k=5):
if chars[tok] == "\n": # end of the bot's line
break
out += chars[tok]
return out.strip()
print(reply("Hello!"))
print(reply("How are you?"))
```
The prompt format must match the training data: `User: <message>\nBot:`.
Generation stops at the first newline.
## Training data
12,000 generated exchanges in the form
```
User: <message>
Bot: <reply>
```
built from a short hand-written list of greeting phrases and replies
(hello, how are you, what is your name, thanks, goodbye, and similar).
No external dataset was used.
## Training
| | |
| --- | --- |
| Steps | 4,000 |
| Batch size | 32 |
| Optimizer | AdamW |
| Learning rate | 1e-3 with cosine decay |
| Hardware | 4-core CPU |
| Time | 14.5 minutes |
| Final loss | train 0.165, validation 0.170 |
## Limitations
- It only knows greetings. Any other message gets a greeting-style reply that
has nothing to do with the question.
- It works on characters, and drops any character that was not in its training
text (for example most non-English letters and emoji).
- It has no memory of earlier messages: each reply sees only the latest message.
- It is far too small to know facts, follow instructions, or reason.
|