tiny-stories-1m: a 1M-parameter byte-level story model, trained on 2 CPU cores
1,018,624 parameters, 4 MB. It writes children's stories one byte at a time. It was trained from scratch for about 2 hours on 2 CPU cores, on 51,200,000 bytes of TinyStories V2 (stories written by GPT-4).
Try it in your browser: no install, no server.
Results (registered before training)
| Bits per byte on held-out stories | Real words written | Copied 50-byte spans | |
|---|---|---|---|
| This model (seed 22) | 1.1242 | 99.48% | 4.51% |
| Same model, other seed (21) | 1.1422 | 99.60% | 2.81% |
| Byte 8-gram given more text (60 MB) | 1.1736 | β | β |
| Same model trained 2.5Γ less (TLM-001) | 1.3633 / 1.3698 | 98.0% / 97.0% | 1.4% / 3.2% |
| Real held-out stories (reference) | β | 100% | 7.82% |
- Bits per byte: lower is better; 8.0 is random guessing. On the same text with its bytes shuffled, this model scores 10.56, so the score reflects real structure.
- Real words: the share of generated words (3+ letters) that occur in the training text. An untrained model scores 67%.
- Copied spans: the share of generated 50-byte stretches found verbatim in the training text. It is lower than for real stories, so the model is not reciting its training data.
- The first attempt failed and is kept. At 20.5M bytes of training (TLM-001), the model lost to a byte 8-gram. Trained 2.5Γ longer, it won, in both seeds.
A sample (unedited)
Once upon a time, there was a boy. The cat was scared of the other side of the people on the boat. The cat said, "Hello, that was a race!" He said the bird with his friends finished the big bathtub and ran towards him. They all played together every day.
All 40 registered samples are in results/.
Honest limits
- The stories don't hold together. Words and phrases are right, but plots wander and characters change.
- The margin over the counting model is small (0.05 bpb), and that rival was capped at 8 bytes of context.
- Not new: the TinyStories paper (Eldan & Li 2023, arXiv 2305.07759) trained 1Mβ33M models on this data, with a word-piece tokenizer and more compute. This is a small, open, CPU-only rebuild with registered checks.
- Context: 256 bytes.
- Self-tested: one AI designed, trained and graded it. Nothing has been independently reproduced. Linux CPU only; not measured on a phone.
Use
import torch
from safetensors.torch import load_file
from gpt import GPT
m = GPT(128, 5, 4); m.load_state_dict(load_file("model.safetensors")); m.eval()
x = torch.tensor([list(b"Once upon a time")])
with torch.no_grad():
for _ in range(200):
nxt = torch.multinomial((m(x[:, -256:])[0, -1] / 0.8).softmax(-1), 1)
x = torch.cat([x, nxt[None]], 1)
print(bytes(x[0].tolist()).decode("utf-8", "replace"))
Model
Byte-level GPT (gpt.py): 256-symbol vocabulary, width 128, 5 pre-LN blocks (4-head attention with rotary
positions, GELU MLP), tied embedding. AdamW (lr 1e-3, betas 0.9/0.95, weight decay 0.1), 6,250 steps of 32 Γ 256
bytes, cosine schedule (train.py).
Files
| Path | What |
|---|---|
model.safetensors |
TLM-002 seed 22 weights |
gpt.py, train.py, train_eval.py |
model, training, bits-per-byte evaluation |
ngram.py, gencheck.py, shufcheck.py |
the n-gram rival, generation checks, shuffled-text check |
results/tlm001/, results/tlm002/ |
registered runs: logs, scores, all samples, weight hashes |
Registrations and full write-ups: principia-artificialis/experiments/tiny_lm.
License
The training data, TinyStories, is licensed CDLA-Sharing-1.0. These weights are trained on it and carry the same license. The code is MIT.
Provenance
AI participation: Claude (Anthropic, Opus 5.5) designed, built, trained and evaluated the model and wrote this card β human validation: not performed β human editing/curation: direction by Chad Holland, 2026-11-21 β human responsibility: Chad Holland. Review level: direction only.