tiny-lm-web-98m

A 98M-parameter GPT-2 small shaped model, trained from scratch on web text. It is from Part 3 of tiny-lm, a project that builds a language model step by step and then looks inside it.

Size	12 layers, width 768, 12 heads, 98.4M parameters
Context	1,024 tokens
Tokeniser	byte-level BPE, 16,384 tokens, trained on FineWeb-Edu
Training	2.70B tokens of FineWeb-Edu, 203 minutes on one RTX 5090
Validation loss	2.983 nats per token, 1.03 bits per character

For comparison, OpenAI's GPT-2 small is about 1.04 bits per character on the same text.

There are two models in this repository:

• The root is the base model. It continues text.
• chat/ is the base model after a 150M-token anneal (25% SimpleStories), then supervised fine-tuning on Dolly 15k, then fine-tuning on System: lines (rules such as "Answer in one sentence", and names for the model and the user).

The weights are a standard GPT2LMHeadModel. The q, k and v biases that GPT-2 has are zero, because the model was trained without them.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "lillycham/tiny-lm-web-98m"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

ids = tok("The water cycle is", return_tensors="pt").input_ids
ids = torch.cat([torch.tensor([[tok.eos_token_id]]), ids], dim=1)   # text starts after <|endoftext|>
out = model.generate(ids, max_new_tokens=100, do_sample=True, temperature=0.8)
print(tok.decode(out[0, 1:]))

The chat model uses this format, and ends its answer with <|endoftext|>. Load it with subfolder="chat". The System: line is optional.

System: Answer in one sentence.
User: Why is the sky blue?
Assistant:

Limits

The text is fluent, but most facts are invented. The chat model answers in the right form (a list, verse or one line) but does not know much. It does not yet take answers from a paragraph in its context. Do not use it for anything that needs true answers.

Data and licence

Trained on FineWeb-Edu (ODC-By), SimpleStories (MIT) and, for the chat model, Dolly 15k (CC BY-SA 3.0). The code and weights are MIT.