SADIK TinyStory (53 KB)
A 53.2 KB character-level GPT that writes tiny English stories, trained from scratch on 12,000 TinyStories.
- Weights file:
tinygpt_int8.bin- 53.18 KB (54,453 bytes), single int8 blob - Parameters: 51,504
- Context: 128 characters
- Layers: 2 transformer blocks, 4 heads, 48-dim embeddings, tied input/output embeddings
- Quantization: int8 per-row for embeddings and linear weights, float16 LayerNorm
Goal: a complete GPT that fits in under 59 KB. It is a fun/educational model, not a chatbot.
Results
| Metric | Value |
|---|---|
| Best validation loss | 2.005 |
| Bits per character | 2.89 bpc |
| Random baseline (71 chars) | 6.15 bpc |
| Training tokens seen | 4.1M |
| Training steps | 500 |
Bits per character is the honest language-modeling score: lower is better. 2.89 bpc means the model is much better than random, but still far from a real LLM.
Samples
These are real, unedited outputs from the int8 model:
Prompt: Once upon a time
Once upon a time o ther hende fer ham was has to mmed ith and chas werd inght rerar. The cous he toled it shimm. Ane was the he ther want bup and bit t and he hit bambot. Som dand fand fom. Hely Moned we to e bily the he smed. She for the war lok. One was someded to plof veas ped and y dit toug thin he ve toway ho
Prompt: The little cat
The little catand therat the sher arde. She doppord he and and aicau was and, coprd hame nd he e bad. "Whil n to marand arard, d " ound wa y. Casto ke a sthe sscknd like to awed chand band amag. He tha geid sunt bupp thed was had her wa ped icay bess herfry. "Lily was noo wor to foun peaved thed hemig. Every sh
Prompt: One day, Tom
One day, Tom mmand, tom and hopl was homom hild. Sa sthe mpritt he tow a mas as to tcanded buthn mitht he thimand wito ith a lled bis wand ham hot h thaile saring. I Lily car theabingh. Ons demeland saingh was wamedkeld and thits sorfow“We, fould the sand sand the swow o tit to thas land be dith axcim. Lild
Usage
pip install torch numpy
python predict_story.py "Once upon a time"
python predict_story.py "The little cat" --chars 500 --temp 0.9
import torch
from tinygpt import load_int8, encode, decode
model, meta = load_int8("tinygpt_int8.bin")
vocab = meta["vocab"]
prompt = "Once upon a time"
ids = encode(prompt, vocab).unsqueeze(0)
with torch.no_grad():
out = model.generate(ids, 400, temperature=0.8, top_k=40)
print(decode(out[0].tolist(), vocab))
The int8 weights are dequantized to float32 at load time, so it runs anywhere PyTorch runs.
Training
- Data: 12,000 TinyStories validation stories (10.0M characters), 71-character vocabulary
- Objective: next-character prediction (cross-entropy)
- Optimizer: AdamW, lr 3e-3 with warmup + cosine decay, weight decay 0.01
- Batch: 64 sequences x 128 characters
- Steps: 500
- Hardware: CPU only
- Quantization: int8 per output row, float16 scales
Limitations
- Character-level and only 51,504 parameters, so output is often word-salad.
- It continues text; it does not follow instructions or answer questions.
- English only, and only simple children's-story style.
- Trained on TinyStories, so it may repeat story clichés.
- Not suitable for production, factual answers, or anything safety-critical.
Author
Sadik Nawaz - 11-year-old AI developer from Pakistan
- Hugging Face: @SadikNawazAI