README.md

---
license: mit
language:
  - en
tags:
  - text-generation
  - sql
  - educational
  - from-scratch
  - interpretability
  - tiny
pipeline_tag: text-generation
library_name: pytorch
---

# Tiny SQL GPT

> **The goal:** build a language model from scratch, end to end, small enough to understand every
> part of it, and in doing so understand how the large ones actually work.

**A 841,216-parameter decoder-only transformer, trained from random weights on a laptop CPU in
five minutes. 100% of the SQL it generates executes against a real database.**

No pretrained weights. No `transformers` model classes. No API keys. The full architecture is
~200 readable lines of PyTorch.

This model exists to be **understood**, not deployed. Its vocabulary is 155 tokens, which is the
point: small enough that you can print the *entire* probability distribution at every generation
step, something no frontier model demo can do.

- **Code, evaluation harness and write-up:** https://github.com/sarathi-aiml/tiny-sql-gpt
- **Plain-English explainer (no maths):** [`EXPLAIN.md`](https://github.com/sarathi-aiml/tiny-sql-gpt/blob/main/EXPLAIN.md)

---

## Usage

```bash
pip install torch safetensors huggingface_hub
```

```python
from inference import TinySQLGPT     # inference.py + tiny_gpt.py from the repo

model = TinySQLGPT.from_pretrained("sarathi-balakrishnan/tiny-sql-gpt")

print(model.generate())
# SELECT city , MIN ( age ) FROM customers WHERE plan = 'annual' GROUP BY city ;

print(model.generate(prompt="SELECT region ,"))

# The whole distribution, all 155 tokens, not a top-k truncation
for token, p in model.next_token_probs(
        "SELECT region , SUM ( qty ) FROM sales GROUP BY", top=5):
    print(f"{token:<12} {p:.3f}")
# region       0.996
# segment      0.001
# product      0.001
```

---

## What it does

Generates SQL over a fixed three-table schema (`sales`, `customers`, `orders`) using 14 query
shapes: `SELECT`, `WHERE`, `AND`, `GROUP BY`, `ORDER BY`, `LIMIT`, and the aggregates
`COUNT`/`SUM`/`AVG`/`MAX`/`MIN`.

It is **not** a text-to-SQL model. It does not take a natural-language question. It generates
SQL unconditionally, or continues a SQL prefix you give it.

## Results

500 generated queries, executed against a real SQLite database. Seeded, so you get these exact
numbers.

| metric | Tiny SQL GPT | bigram baseline |
|---|---:|---:|
| executes | **100.0%** | 4.4% |
| `GROUP BY` agrees with `SELECT` | **100.0%** | 3.4% |
| novel (not in training set) | 16.6% | 98.6% |
| validation loss | 0.662 | n/a |

100% is a real measurement, but read it against the task: 14 query shapes, 3 tables, 155 tokens.
A model that saturates *this* is proof the training loop works, not a text-to-SQL system.

### Scaling

Same architecture and data at four sizes, all trained on one laptop:

| model | params | executes | `GROUP BY` agrees |
|---|---:|---:|---:|
| nano | 24,736 | 99.2% | **36.6%** |
| micro | 124,032 | 99.8% | **100.0%** |
| **tiny** (this model) | **841,216** | **100.0%** | **100.0%** |
| small | 4,834,816 | 100.0% | 100.0% |

Syntax is nearly free: 24K parameters writes SQL that runs. The long-range dependency costs ~5x
more, and appears as a phase transition between 24K and 125K. Above that, nothing improves: going
from 124K to 4.8M parameters buys 0.007 of validation loss and no behavioural change. All four
converge to ~0.66, which is the entropy of the data generator, not a limit of the models.

---

## Why it's interesting

The training data contains a deliberately planted long-range dependency: **the column after
`GROUP BY` is always the column that appeared first in `SELECT`**, roughly 8 tokens earlier.
Getting it right requires looking back, which is what attention is for.

**In the smallest model that can do it, one head learned it.** In `flat` (1 layer, 127K params),
head L0H1 places **58.7% of its attention** on the `SELECT` column, 7.0x above uniform. Nobody
designed or labelled it.

**In this model, it is distributed.** `tiny` gets `GROUP BY` agreement right 100% of the time, yet
no head exceeds 17.7% on the same probe, only 2.1x uniform. Same behaviour, no legible circuit.
The bigger model is not worse at the task, it is worse at being read. This is also seed-sensitive:
an earlier run of `tiny` put one head above 85%, and the current seeded run does not reproduce it.

**And you can watch it hallucinate.** Three `(table, column)` pairs were held out from the
`GROUP BY` position during training. The columns appear elsewhere, just never there. The model
gets **0 out of 3**, confidently substituting a familiar column instead:

```
asked for:   ... GROUP BY  ->  "channel"    (never seen in this position)
it answered: ... GROUP BY  ->  "status"     (familiar, confident, wrong)
```

A control prompt shows the context *is* being used (3.8x to 10.2x lift), it simply loses to a prior
against tokens never seen in that slot. **Attention identifies the right source token; the output
prior overrules it.** That is hallucination, in a model small enough to point at the exact cause.

---

## Training

| | |
|---|---|
| data | 100,000 generated SQL queries (16,941 unique), 1.3M tokens, seed 1337 |
| architecture | decoder-only, 4 layers, 4 heads, 128 embedding, 64-token context |
| tokenizer | word-level, 155 tokens |
| optimizer | AdamW, lr 1e-3, cosine schedule, weight decay 0.01, grad clip 1.0 |
| steps | 3,000 · batch 64 · ~5 minutes on a laptop CPU |
| final loss | train 0.658 · val 0.662 |

Training data is **generated, not scraped**, from a grammar in the repo. No licensing questions,
and a learner can read the entire source of the training set.

## Limitations

- Not text-to-SQL. No natural-language input.
- One fixed three-table schema. It knows no other tables or columns.
- 64-token context. Longer queries are truncated.
- Does not generalize to column names it never saw in a given syntactic position (see above,
  that failure is the point, and it is measured rather than hidden).
- Generated SQL is syntactically valid, not semantically meaningful. It will happily write
  `WHERE age > 1500`.

## License

MIT.