sarathi-balakrishnan/tiny-sql-gpt

Tiny SQL GPT

The goal: build a language model from scratch, end to end, small enough to understand every part of it, and in doing so understand how the large ones actually work.

A 841,216-parameter decoder-only transformer, trained from random weights on a laptop CPU in five minutes. 100% of the SQL it generates executes against a real database.

No pretrained weights. No transformers model classes. No API keys. The full architecture is ~200 readable lines of PyTorch.

This model exists to be understood, not deployed. Its vocabulary is 155 tokens, which is the point: small enough that you can print the entire probability distribution at every generation step, something no frontier model demo can do.

• Code, evaluation harness and write-up: https://github.com/sarathi-aiml/tiny-sql-gpt
• Plain-English explainer (no maths): EXPLAIN.md

---

Usage

pip install torch safetensors huggingface_hub

from inference import TinySQLGPT     # inference.py + tiny_gpt.py from the repo

model = TinySQLGPT.from_pretrained("sarathi-balakrishnan/tiny-sql-gpt")

print(model.generate())
# SELECT city , MIN ( age ) FROM customers WHERE plan = 'annual' GROUP BY city ;

print(model.generate(prompt="SELECT region ,"))

# The whole distribution, all 155 tokens, not a top-k truncation
for token, p in model.next_token_probs(
        "SELECT region , SUM ( qty ) FROM sales GROUP BY", top=5):
    print(f"{token:<12} {p:.3f}")
# region       0.996
# segment      0.001
# product      0.001

---

What it does

Generates SQL over a fixed three-table schema (sales, customers, orders) using 14 query shapes: SELECT, WHERE, AND, GROUP BY, ORDER BY, LIMIT, and the aggregates COUNT/SUM/AVG/MAX/MIN.

It is not a text-to-SQL model. It does not take a natural-language question. It generates SQL unconditionally, or continues a SQL prefix you give it.

Results

500 generated queries, executed against a real SQLite database. Seeded, so you get these exact numbers.

metric	Tiny SQL GPT	bigram baseline
executes	100.0%	4.4%
GROUP BY agrees with SELECT	100.0%	3.4%
novel (not in training set)	16.6%	98.6%
validation loss	0.662	n/a

100% is a real measurement, but read it against the task: 14 query shapes, 3 tables, 155 tokens. A model that saturates this is proof the training loop works, not a text-to-SQL system.

Scaling

Same architecture and data at four sizes, all trained on one laptop:

model	params	executes	GROUP BY agrees
nano	24,736	99.2%	36.6%
micro	124,032	99.8%	100.0%
tiny (this model)	841,216	100.0%	100.0%
small	4,834,816	100.0%	100.0%

Syntax is nearly free: 24K parameters writes SQL that runs. The long-range dependency costs ~5x more, and appears as a phase transition between 24K and 125K. Above that, nothing improves: going from 124K to 4.8M parameters buys 0.007 of validation loss and no behavioural change. All four converge to ~0.66, which is the entropy of the data generator, not a limit of the models.

---

Why it's interesting

The training data contains a deliberately planted long-range dependency: the column after GROUP BY is always the column that appeared first in SELECT, roughly 8 tokens earlier. Getting it right requires looking back, which is what attention is for.

In the smallest model that can do it, one head learned it. In flat (1 layer, 127K params), head L0H1 places 58.7% of its attention on the SELECT column, 7.0x above uniform. Nobody designed or labelled it.

In this model, it is distributed. tiny gets GROUP BY agreement right 100% of the time, yet no head exceeds 17.7% on the same probe, only 2.1x uniform. Same behaviour, no legible circuit. The bigger model is not worse at the task, it is worse at being read. This is also seed-sensitive: an earlier run of tiny put one head above 85%, and the current seeded run does not reproduce it.

And you can watch it hallucinate. Three (table, column) pairs were held out from the GROUP BY position during training. The columns appear elsewhere, just never there. The model gets 0 out of 3, confidently substituting a familiar column instead:

asked for:   ... GROUP BY  ->  "channel"    (never seen in this position)
it answered: ... GROUP BY  ->  "status"     (familiar, confident, wrong)

A control prompt shows the context is being used (3.8x to 10.2x lift), it simply loses to a prior against tokens never seen in that slot. Attention identifies the right source token; the output prior overrules it. That is hallucination, in a model small enough to point at the exact cause.

---

Training

data	100,000 generated SQL queries (16,941 unique), 1.3M tokens, seed 1337
architecture	decoder-only, 4 layers, 4 heads, 128 embedding, 64-token context
tokenizer	word-level, 155 tokens
optimizer	AdamW, lr 1e-3, cosine schedule, weight decay 0.01, grad clip 1.0
steps	3,000 · batch 64 · ~5 minutes on a laptop CPU
final loss	train 0.658 · val 0.662

Training data is generated, not scraped, from a grammar in the repo. No licensing questions, and a learner can read the entire source of the training set.

Limitations

• Not text-to-SQL. No natural-language input.
• One fixed three-table schema. It knows no other tables or columns.
• 64-token context. Longer queries are truncated.
• Does not generalize to column names it never saw in a given syntactic position (see above, that failure is the point, and it is measured rather than hidden).
• Generated SQL is syntactically valid, not semantically meaningful. It will happily write WHERE age > 1500.

License

MIT.