tinytowns

Three tiny letter-level GPTs that invent city names. Trained on 128,435 real place names from GeoNames. The smallest has 4,572 weights, the largest about a million.

This is a learning project: a transformer written by hand in plain PyTorch, small enough to read every number of. Code, step-by-step build and all measurements: github.com/dave8172/tinytowns.

Try it in your browser: quirkyagents.com/wizard/projects/tinytowns. The page downloads these weights and runs the model on your device.

The three models

All three use one design and one code file (model.py); only the config differs.

File	Note size	Layers × heads	MLP width	Weights	Validation loss
tinytowns-nano.safetensors	16	1 × 1	64	4,572	2.4676
tinytowns-small.safetensors	64	2 × 4	256	105,052	2.2049
tinytowns.safetensors	128	5 × 4	512	1,001,500	2.1504

Loss is cross-entropy per symbol on 14,271 held-out names (lower is better). For reference: guessing uniformly over 28 symbols scores 3.3322; a 28×28 bigram counting table scores 2.6732.

How it works

• Tokens are letters. Alphabet of 28: index 0 = . (start / end of name), 1 = space, 2–27 = a–z.
• Context: 21 positions, one . plus at most 20 letters.
• Architecture: token + learned positional embeddings, then pre-norm transformer layers (causal self-attention + MLP), then a final LayerNorm and an unembedding to 28 scores.
• Training: predict each next symbol of . + name + .; AdamW, 6,000 steps. Batch 64 (nano, CPU) or 256 (small and tinytowns, GPU). The saved weights come from the step with the lowest validation loss (step 5,500 for tinytowns).

Each .safetensors file also stores its config as JSON text in the header metadata, so load_weights can rebuild the right model from the file alone.

Use

Needs torch>=2.2, safetensors and huggingface_hub.

from huggingface_hub import hf_hub_download
import importlib.util, torch

repo = "dave8172/tinytowns"
spec = importlib.util.spec_from_file_location("model", hf_hub_download(repo, "model.py"))
model = importlib.util.module_from_spec(spec); spec.loader.exec_module(model)

net, notes = model.load_weights(hf_hub_download(repo, "tinytowns.safetensors"))
print(model.sample(net, 10, temperature=1.0, generator=torch.Generator().manual_seed(42)))

model.py is plain PyTorch with no other code from this project. Read it before you run it, as with any downloaded code.

In a browser or Node, tinytowns.js in the GitHub repo runs the same model with no libraries (forward pass and sampling only). On all 14,271 validation names it matches the PyTorch loss to six decimal places.

What to expect

Out of 10,000 names sampled at temperature 1 (seed 42), checked against the full name list:

Model	Real, seen in training	Real, only in validation	Invented
tinytowns-nano	194	14	9,792
tinytowns-small	242	20	9,738
tinytowns	386	30	9,584

Limits

• It knows only lowercase a–z and space. Accents were removed, and non-Latin names are not in the data.
• A generated name is a sound-alike, not a place. Most outputs don't exist, and a "real" one may be copied from the training data.
• It is built for learning. Ideas for use (game maps, test data that names no real place, a "does this look like a place name?" score) are untested.

Data and credit

Trained on GeoNames cities1000 (every place with 1,000+ people, 243 countries), licensed CC BY 4.0. Changed: names lowercased, hyphens turned into spaces, apostrophes removed, duplicates merged, names with digits or brackets or over 20 letters dropped. 90% train / 10% validation, fixed seed.

Licence

Code and weights: MIT. Training data: CC BY 4.0, GeoNames.