Model card: say what tinytowns is for up front

52bbe9e verified 6 days ago

4.77 kB

---
license: mit
language:
  - en
tags:
  - pytorch
  - safetensors
  - character-level
  - text-generation
  - educational
  - from-scratch
pipeline_tag: text-generation
---
# tinytowns

**Three tiny letter-level GPTs that invent city names.** Trained on 128,435 real place names from
GeoNames. The smallest has 4,572 weights, the largest about a million.

**What it is for:** tinytowns makes up new town names in the style of real ones. It learned the
letter patterns of 128,435 real place names: it was trained to guess the next letter of a place
name as well as it can. It does not know which places exist, so most names it writes are new. To
find real places, look them up in a list of places, or ask a large AI model such as Claude.

This is a learning project: a transformer written by hand in plain PyTorch, small enough to read
every number of. Code, step-by-step build and all measurements:
[github.com/dave8172/tinytowns](https://github.com/dave8172/tinytowns).

**Try it in your browser:** [quirkyagents.com/wizard/projects/tinytowns](https://quirkyagents.com/wizard/projects/tinytowns/).
The page downloads these weights and runs the model on your device.

## The three models

All three use one design and one code file (`model.py`); only the config differs.

| File | Note size | Layers × heads | MLP width | Weights | Validation loss |
|---|---|---|---|---|---|
| `tinytowns-nano.safetensors` | 16 | 1 × 1 | 64 | 4,572 | 2.4676 |
| `tinytowns-small.safetensors` | 64 | 2 × 4 | 256 | 105,052 | 2.2049 |
| `tinytowns.safetensors` | 128 | 5 × 4 | 512 | 1,001,500 | **2.1504** |

Loss is cross-entropy per symbol on 14,271 held-out names (lower is better). For reference:
guessing uniformly over 28 symbols scores 3.3322; a 28×28 bigram counting table scores 2.6732.

## How it works

- **Tokens are letters.** Alphabet of 28: index 0 = `.` (start / end of name), 1 = space, 2–27 = `a`–`z`.
- **Context:** 21 positions, one `.` plus at most 20 letters.
- **Architecture:** token + learned positional embeddings, then pre-norm transformer layers
  (causal self-attention + MLP), then a final LayerNorm and an unembedding to 28 scores.
- **Training:** predict each next symbol of `.` + name + `.`; AdamW, 6,000 steps. Batch 64 (nano,
  CPU) or 256 (small and tinytowns, GPU). The saved weights come from the step with the lowest
  validation loss (step 5,500 for tinytowns).

Each `.safetensors` file also stores its config as JSON text in the header metadata, so
`load_weights` can rebuild the right model from the file alone.

## Use

Needs `torch>=2.2`, `safetensors` and `huggingface_hub`.

```python
from huggingface_hub import hf_hub_download
import importlib.util, torch
repo = "dave8172/tinytowns"
spec = importlib.util.spec_from_file_location("model", hf_hub_download(repo, "model.py"))
model = importlib.util.module_from_spec(spec); spec.loader.exec_module(model)
net, notes = model.load_weights(hf_hub_download(repo, "tinytowns.safetensors"))
print(model.sample(net, 10, temperature=1.0, generator=torch.Generator().manual_seed(42)))
```

`model.py` is plain PyTorch with no other code from this project. Read it before you run it, as with
any downloaded code.

In a browser or Node, `tinytowns.js` in the GitHub repo runs the same model with no libraries
(forward pass and sampling only). On all 14,271 validation names it matches the PyTorch loss to six
decimal places.

## What to expect

Out of 10,000 names sampled at temperature 1 (seed 42), checked against the full name list:

| Model | Real, seen in training | Real, only in validation | Invented |
|---|---|---|---|
| tinytowns-nano | 194 | 14 | 9,792 |
| tinytowns-small | 242 | 20 | 9,738 |
| tinytowns | 386 | 30 | 9,584 |

## Limits

- It knows only lowercase `a`–`z` and space. Accents were removed, and non-Latin names are not in the data.
- It does not know which places exist. A generated name follows the letter patterns of real names;
  most don't exist, and a "real" one may be copied from the training data. To finish a name with a
  real place, a lookup in a list of places is faster, and gives a real name whenever one starts
  with those letters.
- It is built for learning. Ideas for use (game maps, test data that names no real place, a
  "does this look like a place name?" score) are untested.

## Data and credit

Trained on [GeoNames](https://www.geonames.org) `cities1000` (every place with 1,000+ people,
243 countries), licensed [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). Changed:
names lowercased, hyphens turned into spaces, apostrophes removed, duplicates merged, names with
digits or brackets or over 20 letters dropped. 90% train / 10% validation, fixed seed.

## Licence

Code and weights: MIT. Training data: CC BY 4.0, GeoNames.

	---
	license: mit
	language:
	- en
	tags:
	- pytorch
	- safetensors
	- character-level
	- text-generation
	- educational
	- from-scratch
	pipeline_tag: text-generation
	---

	# tinytowns

	Three tiny letter-level GPTs that invent city names. Trained on 128,435 real place names from
	GeoNames. The smallest has 4,572 weights, the largest about a million.

	What it is for: tinytowns makes up new town names in the style of real ones. It learned the
	letter patterns of 128,435 real place names: it was trained to guess the next letter of a place
	name as well as it can. It does not know which places exist, so most names it writes are new. To
	find real places, look them up in a list of places, or ask a large AI model such as Claude.

	This is a learning project: a transformer written by hand in plain PyTorch, small enough to read
	every number of. Code, step-by-step build and all measurements:
	[github.com/dave8172/tinytowns](https://github.com/dave8172/tinytowns).

	Try it in your browser: [quirkyagents.com/wizard/projects/tinytowns](https://quirkyagents.com/wizard/projects/tinytowns/).
	The page downloads these weights and runs the model on your device.

	## The three models

	All three use one design and one code file (`model.py`); only the config differs.

	\| File \| Note size \| Layers × heads \| MLP width \| Weights \| Validation loss \|
	\|---\|---\|---\|---\|---\|---\|
	\| `tinytowns-nano.safetensors` \| 16 \| 1 × 1 \| 64 \| 4,572 \| 2.4676 \|
	\| `tinytowns-small.safetensors` \| 64 \| 2 × 4 \| 256 \| 105,052 \| 2.2049 \|
	\| `tinytowns.safetensors` \| 128 \| 5 × 4 \| 512 \| 1,001,500 \| 2.1504 \|

	Loss is cross-entropy per symbol on 14,271 held-out names (lower is better). For reference:
	guessing uniformly over 28 symbols scores 3.3322; a 28×28 bigram counting table scores 2.6732.

	## How it works

	- Tokens are letters. Alphabet of 28: index 0 = `.` (start / end of name), 1 = space, 2–27 = `a`–`z`.
	- Context: 21 positions, one `.` plus at most 20 letters.
	- Architecture: token + learned positional embeddings, then pre-norm transformer layers
	(causal self-attention + MLP), then a final LayerNorm and an unembedding to 28 scores.
	- Training: predict each next symbol of `.` + name + `.`; AdamW, 6,000 steps. Batch 64 (nano,
	CPU) or 256 (small and tinytowns, GPU). The saved weights come from the step with the lowest
	validation loss (step 5,500 for tinytowns).

	Each `.safetensors` file also stores its config as JSON text in the header metadata, so
	`load_weights` can rebuild the right model from the file alone.

	## Use

	Needs `torch>=2.2`, `safetensors` and `huggingface_hub`.

	```python
	from huggingface_hub import hf_hub_download
	import importlib.util, torch

	repo = "dave8172/tinytowns"
	spec = importlib.util.spec_from_file_location("model", hf_hub_download(repo, "model.py"))
	model = importlib.util.module_from_spec(spec); spec.loader.exec_module(model)

	net, notes = model.load_weights(hf_hub_download(repo, "tinytowns.safetensors"))
	print(model.sample(net, 10, temperature=1.0, generator=torch.Generator().manual_seed(42)))
	```

	`model.py` is plain PyTorch with no other code from this project. Read it before you run it, as with
	any downloaded code.

	In a browser or Node, `tinytowns.js` in the GitHub repo runs the same model with no libraries
	(forward pass and sampling only). On all 14,271 validation names it matches the PyTorch loss to six
	decimal places.

	## What to expect

	Out of 10,000 names sampled at temperature 1 (seed 42), checked against the full name list:

	\| Model \| Real, seen in training \| Real, only in validation \| Invented \|
	\|---\|---\|---\|---\|
	\| tinytowns-nano \| 194 \| 14 \| 9,792 \|
	\| tinytowns-small \| 242 \| 20 \| 9,738 \|
	\| tinytowns \| 386 \| 30 \| 9,584 \|

	## Limits

	- It knows only lowercase `a`–`z` and space. Accents were removed, and non-Latin names are not in the data.
	- It does not know which places exist. A generated name follows the letter patterns of real names;
	most don't exist, and a "real" one may be copied from the training data. To finish a name with a
	real place, a lookup in a list of places is faster, and gives a real name whenever one starts
	with those letters.
	- It is built for learning. Ideas for use (game maps, test data that names no real place, a
	"does this look like a place name?" score) are untested.

	## Data and credit

	Trained on [GeoNames](https://www.geonames.org) `cities1000` (every place with 1,000+ people,
	243 countries), licensed [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). Changed:
	names lowercased, hyphens turned into spaces, apostrophes removed, duplicates merged, names with
	digits or brackets or over 20 letters dropped. 90% train / 10% validation, fixed seed.

	## Licence

	Code and weights: MIT. Training data: CC BY 4.0, GeoNames.