| --- |
| license: mit |
| language: |
| - en |
| library_name: pytorch |
| pipeline_tag: text-generation |
| tags: |
| - tiny |
| - character-level |
| - gpt |
| - from-scratch |
| - educational |
| --- |
| |
| # Tiny LLM |
|
|
| A very small GPT-style language model, trained from scratch in about 15 minutes |
| on a laptop CPU. It writes one character at a time and only knows simple |
| greetings: hello, how are you, what is your name, thanks, goodbye. |
|
|
| It is a learning project, not a useful assistant. Anything that is not a |
| greeting gets an unrelated greeting-style reply. |
|
|
| **Try it in your browser:** [Tiny LLM demo Space](https://huggingface.co/spaces/Neural-Labs/NerualLabs) |
|
|
| ## Model details |
|
|
| | | | |
| | --- | --- | |
| | Type | Decoder-only transformer (GPT-style), character level | |
| | Parameters | 824,120 | |
| | Layers | 4 | |
| | Attention heads | 4 | |
| | Embedding width | 128 | |
| | Context length | 128 characters | |
| | Vocabulary | 56 characters (the distinct characters in the training text) | |
| | Language | English | |
| | License | MIT | |
|
|
| Each block uses pre-layer-norm, causal self-attention, and an MLP |
| (128 → 512 → 128) with GELU. Token and position embeddings are both learned. |
|
|
| ## Files |
|
|
| | File | What it is | |
| | --- | --- | |
| | `model.pt` | Trained weights: a dict with `state_dict`, `chars` (the vocabulary) and `config` | |
| | `model.py` | The model code in PyTorch, about 60 lines | |
|
|
| ## How to use |
|
|
| This is a custom PyTorch model, so it does not load with `transformers`. |
| Download `model.py` and `model.pt`, then: |
|
|
| ```bash |
| pip install torch huggingface_hub |
| ``` |
|
|
| ```python |
| import torch |
| from huggingface_hub import hf_hub_download |
| |
| repo = "Neural-Labs/tiny-llm" |
| hf_hub_download(repo, "model.py", local_dir=".") |
| weights = hf_hub_download(repo, "model.pt") |
| |
| from model import TinyGPT |
| |
| ckpt = torch.load(weights, map_location="cpu", weights_only=True) |
| chars = ckpt["chars"] |
| stoi = {c: i for i, c in enumerate(chars)} |
| model = TinyGPT(len(chars), **ckpt["config"]) |
| model.load_state_dict(ckpt["state_dict"]) |
| model.eval() |
| |
| |
| def reply(message): |
| # Characters the model never saw are dropped. |
| clean = "".join(c for c in message.replace("\n", " ") if c in stoi).strip() or "hi" |
| prompt = f"User: {clean[:80]}\nBot:" |
| idx = torch.tensor([[stoi[c] for c in prompt]]) |
| out = "" |
| for tok in model.generate(idx, max_new_tokens=100, temperature=0.5, top_k=5): |
| if chars[tok] == "\n": # end of the bot's line |
| break |
| out += chars[tok] |
| return out.strip() |
| |
| |
| print(reply("Hello!")) |
| print(reply("How are you?")) |
| ``` |
|
|
| The prompt format must match the training data: `User: <message>\nBot:`. |
| Generation stops at the first newline. |
|
|
| ## Training data |
|
|
| 12,000 generated exchanges in the form |
|
|
| ``` |
| User: <message> |
| Bot: <reply> |
| |
| ``` |
|
|
| built from a short hand-written list of greeting phrases and replies |
| (hello, how are you, what is your name, thanks, goodbye, and similar). |
| No external dataset was used. |
|
|
| ## Training |
|
|
| | | | |
| | --- | --- | |
| | Steps | 4,000 | |
| | Batch size | 32 | |
| | Optimizer | AdamW | |
| | Learning rate | 1e-3 with cosine decay | |
| | Hardware | 4-core CPU | |
| | Time | 14.5 minutes | |
| | Final loss | train 0.165, validation 0.170 | |
|
|
| ## Limitations |
|
|
| - It only knows greetings. Any other message gets a greeting-style reply that |
| has nothing to do with the question. |
| - It works on characters, and drops any character that was not in its training |
| text (for example most non-English letters and emoji). |
| - It has no memory of earlier messages: each reply sees only the latest message. |
| - It is far too small to know facts, follow instructions, or reason. |
|
|
