tiny-stories-1m: a 1M-parameter byte-level story model, trained on 2 CPU cores

1,018,624 parameters, 4 MB. It writes children's stories one byte at a time. It was trained from scratch for about 2 hours on 2 CPU cores, on 51,200,000 bytes of TinyStories V2 (stories written by GPT-4).

Try it in your browser: no install, no server.

Results (registered before training)

Bits per byte on held-out stories Real words written Copied 50-byte spans
This model (seed 22) 1.1242 99.48% 4.51%
Same model, other seed (21) 1.1422 99.60% 2.81%
Byte 8-gram given more text (60 MB) 1.1736 β€” β€”
Same model trained 2.5Γ— less (TLM-001) 1.3633 / 1.3698 98.0% / 97.0% 1.4% / 3.2%
Real held-out stories (reference) β€” 100% 7.82%
  • Bits per byte: lower is better; 8.0 is random guessing. On the same text with its bytes shuffled, this model scores 10.56, so the score reflects real structure.
  • Real words: the share of generated words (3+ letters) that occur in the training text. An untrained model scores 67%.
  • Copied spans: the share of generated 50-byte stretches found verbatim in the training text. It is lower than for real stories, so the model is not reciting its training data.
  • The first attempt failed and is kept. At 20.5M bytes of training (TLM-001), the model lost to a byte 8-gram. Trained 2.5Γ— longer, it won, in both seeds.

A sample (unedited)

Once upon a time, there was a boy. The cat was scared of the other side of the people on the boat. The cat said, "Hello, that was a race!" He said the bird with his friends finished the big bathtub and ran towards him. They all played together every day.

All 40 registered samples are in results/.

Honest limits

  • The stories don't hold together. Words and phrases are right, but plots wander and characters change.
  • The margin over the counting model is small (0.05 bpb), and that rival was capped at 8 bytes of context.
  • Not new: the TinyStories paper (Eldan & Li 2023, arXiv 2305.07759) trained 1M–33M models on this data, with a word-piece tokenizer and more compute. This is a small, open, CPU-only rebuild with registered checks.
  • Context: 256 bytes.
  • Self-tested: one AI designed, trained and graded it. Nothing has been independently reproduced. Linux CPU only; not measured on a phone.

Use

import torch
from safetensors.torch import load_file
from gpt import GPT
m = GPT(128, 5, 4); m.load_state_dict(load_file("model.safetensors")); m.eval()
x = torch.tensor([list(b"Once upon a time")])
with torch.no_grad():
    for _ in range(200):
        nxt = torch.multinomial((m(x[:, -256:])[0, -1] / 0.8).softmax(-1), 1)
        x = torch.cat([x, nxt[None]], 1)
print(bytes(x[0].tolist()).decode("utf-8", "replace"))

Model

Byte-level GPT (gpt.py): 256-symbol vocabulary, width 128, 5 pre-LN blocks (4-head attention with rotary positions, GELU MLP), tied embedding. AdamW (lr 1e-3, betas 0.9/0.95, weight decay 0.1), 6,250 steps of 32 Γ— 256 bytes, cosine schedule (train.py).

Files

Path What
model.safetensors TLM-002 seed 22 weights
gpt.py, train.py, train_eval.py model, training, bits-per-byte evaluation
ngram.py, gencheck.py, shufcheck.py the n-gram rival, generation checks, shuffled-text check
results/tlm001/, results/tlm002/ registered runs: logs, scores, all samples, weight hashes

Registrations and full write-ups: principia-artificialis/experiments/tiny_lm.

License

The training data, TinyStories, is licensed CDLA-Sharing-1.0. These weights are trained on it and carry the same license. The code is MIT.

Provenance

AI participation: Claude (Anthropic, Opus 5.5) designed, built, trained and evaluated the model and wrote this card β†’ human validation: not performed β†’ human editing/curation: direction by Chad Holland, 2026-11-21 β†’ human responsibility: Chad Holland. Review level: direction only.

Downloads last month

-

Downloads are not tracked for this model. How to track
Safetensors
Model size
1.02M params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Dataset used to train holland202/tiny-stories-1m

Space using holland202/tiny-stories-1m 1