File size: 2,692 Bytes
d938f5a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 | ---
license: mit
language: en
library_name: transformers
pipeline_tag: text-generation
tags:
- gpt2
- tiny-lm
- educational
datasets:
- HuggingFaceFW/fineweb-edu
- SimpleStories/SimpleStories
- databricks/databricks-dolly-15k
---
# tiny-lm-web-98m
A 98M-parameter GPT-2 small shaped model, trained from scratch on web text. It is from Part 3 of
[tiny-lm](https://github.com/lillycham/tiny-lm), a project that builds a language model step by
step and then looks inside it.
| | |
|---|---|
| Size | 12 layers, width 768, 12 heads, 98.4M parameters |
| Context | 1,024 tokens |
| Tokeniser | byte-level BPE, 16,384 tokens, trained on FineWeb-Edu |
| Training | 2.70B tokens of FineWeb-Edu, 203 minutes on one RTX 5090 |
| Validation loss | 2.983 nats per token, 1.03 bits per character |
For comparison, OpenAI's GPT-2 small is about 1.04 bits per character on the same text.
There are two models in this repository:
- **The root** is the base model. It continues text.
- **`chat/`** is the base model after a 150M-token anneal (25% SimpleStories), then supervised
fine-tuning on Dolly 15k, then fine-tuning on `System:` lines (rules such as "Answer in one
sentence", and names for the model and the user).
The weights are a standard `GPT2LMHeadModel`. The q, k and v biases that GPT-2 has are zero,
because the model was trained without them.
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "lillycham/tiny-lm-web-98m"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)
ids = tok("The water cycle is", return_tensors="pt").input_ids
ids = torch.cat([torch.tensor([[tok.eos_token_id]]), ids], dim=1) # text starts after <|endoftext|>
out = model.generate(ids, max_new_tokens=100, do_sample=True, temperature=0.8)
print(tok.decode(out[0, 1:]))
```
The chat model uses this format, and ends its answer with `<|endoftext|>`. Load it with
`subfolder="chat"`. The `System:` line is optional.
```
System: Answer in one sentence.
User: Why is the sky blue?
Assistant:
```
## Limits
The text is fluent, but most facts are invented. The chat model answers in the right form (a list,
verse or one line) but does not know much. It does not yet take answers from a paragraph in its
context. Do not use it for anything that needs true answers.
## Data and licence
Trained on [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) (ODC-By),
[SimpleStories](https://huggingface.co/datasets/SimpleStories/SimpleStories) (MIT) and, for the chat
model, [Dolly 15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k) (CC BY-SA 3.0).
The code and weights are MIT.
|