README.md

---
language: en
license: apache-2.0
library_name: transformers
tags:
  - tiny
  - tiny-lm
  - small-language-model
  - sub-1m
  - char-level
  - from-scratch
  - nanoGPT
  - TinyStories
base_model: []
---

# Char-GPT 1.2M

A tiny **character-level** causal transformer trained **from scratch** on
[TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories). A small,
honest reference build — the point is a model whose card matches its artifact
exactly, not a competitive checkpoint.

## Parameters (exact)

**1,216,000 parameters, untied head.**

| module | params |
|---|---|
| `transformer.wte` (65×128) | 8,320 |
| `transformer.wpe` (128×128) | 16,384 |
| 6 × attention (qkv + proj, bias-free) | 614,400 |
| 6 × FFN (4×, bias-free) | 552,960 |
| 6 × 2 LayerNorm (affine) | 1,536 |
| `ln_f` (128) | 256 |
| `lm_head` (65×128, **separate / untied**) | 8,320 |
| **total** | **1,216,000** |

> The head is **not** weight-tied: the checkpoint stores two distinct 65×128
> tensors (`transformer.wte.weight` and `lm_head.weight`), and `model.py` never
> assigns one to the other. `config.json` therefore says
> `tie_word_embeddings: false`. (If the head were tied the count would be
> 1,207,680.)

## Architecture

nanoGPT-style GPT-2, all bias-free except LayerNorm:
- `n_layer=6`, `n_head=4`, `n_embd=128`, FFN = 4× = 512
- `vocab_size=65` (printable ASCII + newline), `block_size=128`
- RoPE: none (learned positional embedding `wpe`)

## Training

- **Data:** `roneneldan/TinyStories` (train split), first ~1.0M characters,
  90/5/5 train/val/test split by character.
- **Steps:** 1,500, batch 32 × seq 128, AdamW (lr 6e-4, cosine, warmup),
  grad-clip 1.0, float32, CPU (16 threads). ~3.5 min.
- **Seed:** 42.

## Quality — what it is and is not

Held-out perplexities (measured on the full held-out test split, 2026-09-20):

| split | loss | perplexity |
|---|---|---|
| test (49,674 tokens) | 1.4369 | **4.21** |

It captures TinyStories' surface style (short declarative sentences, simple
vocabulary, character names) but it is a **1.2M-parameter model on ~1M
characters** — it does not grasp meaning, it repeats and drifts, and it will
produce the kind of plausible-looking-but-nonsense text in `sample.txt`.
Treat it as a working toy / reference architecture, not a useful language model.

> The original training log reported val 1.9046 / test 1.9473 from a
> 60-batch random evaluation; the full-split number above is the honest one.

## Files

- `model.safetensors` — 4,869,112 B (53 tensors, F32)
- `model.py` — `CharGPT` + `from_config`
- `config.json`, `tokenizer_config.json` (char vocab)
- `sample.txt` — 240-char greedy-ish sample
- `LICENSE` — Apache-2.0

## Reproduce

```python
import torch, json
from model import from_config
cfg = json.load(open("config.json"))
m = from_config(cfg)
print(sum(p.numel() for p in m.parameters()))  # 1216000
```