File size: 2,900 Bytes
97c4125 005ca2c 97c4125 005ca2c 97c4125 005ca2c 97c4125 d316a2c 97c4125 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 | ---
language: en
license: apache-2.0
library_name: transformers
tags:
- tiny
- tiny-lm
- small-language-model
- sub-1m
- char-level
- from-scratch
- nanoGPT
- TinyStories
base_model: []
---
# Char-GPT 1.2M
A tiny **character-level** causal transformer trained **from scratch** on
[TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories). A small,
honest reference build — the point is a model whose card matches its artifact
exactly, not a competitive checkpoint.
## Parameters (exact)
**1,216,000 parameters, untied head.**
| module | params |
|---|---|
| `transformer.wte` (65×128) | 8,320 |
| `transformer.wpe` (128×128) | 16,384 |
| 6 × attention (qkv + proj, bias-free) | 614,400 |
| 6 × FFN (4×, bias-free) | 552,960 |
| 6 × 2 LayerNorm (affine) | 1,536 |
| `ln_f` (128) | 256 |
| `lm_head` (65×128, **separate / untied**) | 8,320 |
| **total** | **1,216,000** |
> The head is **not** weight-tied: the checkpoint stores two distinct 65×128
> tensors (`transformer.wte.weight` and `lm_head.weight`), and `model.py` never
> assigns one to the other. `config.json` therefore says
> `tie_word_embeddings: false`. (If the head were tied the count would be
> 1,207,680.)
## Architecture
nanoGPT-style GPT-2, all bias-free except LayerNorm:
- `n_layer=6`, `n_head=4`, `n_embd=128`, FFN = 4× = 512
- `vocab_size=65` (printable ASCII + newline), `block_size=128`
- RoPE: none (learned positional embedding `wpe`)
## Training
- **Data:** `roneneldan/TinyStories` (train split), first ~1.0M characters,
90/5/5 train/val/test split by character.
- **Steps:** 1,500, batch 32 × seq 128, AdamW (lr 6e-4, cosine, warmup),
grad-clip 1.0, float32, CPU (16 threads). ~3.5 min.
- **Seed:** 42.
## Quality — what it is and is not
Held-out perplexities (measured on the full held-out test split, 2026-09-20):
| split | loss | perplexity |
|---|---|---|
| test (49,674 tokens) | 1.4369 | **4.21** |
It captures TinyStories' surface style (short declarative sentences, simple
vocabulary, character names) but it is a **1.2M-parameter model on ~1M
characters** — it does not grasp meaning, it repeats and drifts, and it will
produce the kind of plausible-looking-but-nonsense text in `sample.txt`.
Treat it as a working toy / reference architecture, not a useful language model.
> The original training log reported val 1.9046 / test 1.9473 from a
> 60-batch random evaluation; the full-split number above is the honest one.
## Files
- `model.safetensors` — 4,869,112 B (53 tensors, F32)
- `model.py` — `CharGPT` + `from_config`
- `config.json`, `tokenizer_config.json` (char vocab)
- `sample.txt` — 240-char greedy-ish sample
- `LICENSE` — Apache-2.0
## Reproduce
```python
import torch, json
from model import from_config
cfg = json.load(open("config.json"))
m = from_config(cfg)
print(sum(p.numel() for p in m.parameters())) # 1216000
``` |