sarathi-balakrishnan/tiny-sql-gpt

Tiny SQL GPT

Personal learning project (August 2026). The goal was to build a language model from scratch, end to end, small enough to understand every part of it, as a way to understand how the large ones work.

An 841,216-parameter decoder-only transformer, trained from random weights on a laptop CPU in about five minutes, on a toy SQL grammar. On that toy task, all 500 sampled queries execute against a SQLite database. That says more about how simple the task is than about the model (see Results).

No pretrained weights. No transformers model classes. No API keys. The full architecture is ~200 readable lines of PyTorch.

This model exists to be understood, not deployed. Its vocabulary is 155 tokens, which is the point: small enough that you can print the entire probability distribution at every generation step.

• Code, evaluation harness and write-up: https://github.com/sarathi-aiml/tiny-sql-gpt
• Plain-English explainer (no maths): EXPLAIN.md

---

Usage

pip install torch safetensors huggingface_hub

from inference import TinySQLGPT     # inference.py + tiny_gpt.py from the repo

model = TinySQLGPT.from_pretrained("sarathi-balakrishnan/tiny-sql-gpt")

print(model.generate())
# SELECT city , MIN ( age ) FROM customers WHERE plan = 'annual' GROUP BY city ;

print(model.generate(prompt="SELECT region ,"))

# The whole distribution, all 155 tokens, not a top-k truncation
for token, p in model.next_token_probs(
        "SELECT region , SUM ( qty ) FROM sales GROUP BY", top=5):
    print(f"{token:<12} {p:.3f}")
# region       0.996
# segment      0.001
# product      0.001

---

What it does

Generates SQL over a fixed three-table schema (sales, customers, orders) using 14 query shapes: SELECT, WHERE, AND, GROUP BY, ORDER BY, LIMIT, and the aggregates COUNT/SUM/AVG/MAX/MIN.

It is not a text-to-SQL model. It does not take a natural-language question. It generates SQL unconditionally, or continues a SQL prefix you give it.

Results

500 generated queries, executed against a real SQLite database. Seeded, so you get these exact numbers.

metric	Tiny SQL GPT	bigram baseline
executes	100.0%	4.4%
GROUP BY agrees with SELECT	100.0%	3.4%
novel (not in training set)	16.6%	98.6%
validation loss	0.662	n/a

100% is a real measurement, but read it against the task: 14 query shapes, 3 tables, 155 tokens. A model that saturates this shows the training loop works. It is not a text-to-SQL system.

Caveats:

• Only 16.6% of samples are novel. Most generated queries reproduce queries from the training set. The bigram baseline is "novel" mostly because it produces invalid SQL.
• Validation loss uses the last 10% of the same generated corpus. With only 16,941 unique queries in 100,000 examples, many validation queries also appear in training, so train ≈ val loss isn't evidence of generalization. The held-out GROUP BY probe below is the real generalization test, and the model fails it.

Scaling

Same architecture and data at four sizes, all trained on one laptop:

model	params	executes	GROUP BY agrees
nano	24,736	99.2%	36.6%
micro	124,032	99.8%	100.0%
tiny (this model)	841,216	100.0%	100.0%
small	4,834,816	100.0%	100.0%

Syntax is cheap: 24K parameters is enough to write SQL that runs. The long-range dependency needs ~5x more parameters, and shows up as a sharp jump between 24K and 124K. Above that, nothing improves: going from 124K to 4.8M parameters buys 0.007 of validation loss and no change in behaviour. All four converge to ~0.66, which I believe is close to the entropy of the data generator rather than a limit of the models. I didn't compute the generator's entropy directly. Each size is a single seed.

---

Why it's interesting

The training data contains a deliberately planted long-range dependency: the column after GROUP BY is always the column that appeared first in SELECT, roughly 8 tokens earlier. Getting it right requires looking back, which is what attention is for.

In a 1-layer model, one head learned it. In flat (1 layer, 127K params), head L0H1 places 58.7% of its attention on the SELECT column, 7.0x above uniform. Nobody designed or labelled it.

In this model, it is distributed. tiny gets GROUP BY agreement right 100% of the time, yet no head exceeds 17.7% on the same probe, only 2.1x uniform. Same behaviour, no legible circuit. The bigger model is not worse at the task, it is worse at being read. This is also seed-sensitive: an earlier run of tiny put one head above 85%, and the current seeded run does not reproduce it.

And you can watch it hallucinate. Three (table, column) pairs were held out from the GROUP BY position during training. The columns appear elsewhere, just never there. The model gets 0 out of 3, confidently substituting a familiar column instead:

asked for:   ... GROUP BY  ->  "channel"    (never seen in this position)
it answered: ... GROUP BY  ->  "status"     (familiar, confident, wrong)

A control prompt shows the context is being used (3.8x to 10.2x lift), it simply loses to a prior against tokens never seen in that slot. Attention identifies the right source token; the output prior overrules it. It's a small, inspectable example of a hallucination-like failure (3 held-out pairs, so an illustration, not a statistic).

---

Training

data	100,000 generated SQL queries (16,941 unique), 1.3M tokens, seed 1337
architecture	decoder-only, 4 layers, 4 heads, 128 embedding, 64-token context
tokenizer	word-level, 155 tokens
optimizer	AdamW, lr 1e-3, cosine schedule, weight decay 0.01, grad clip 1.0
steps	3,000 · batch 64 × 64 tokens (~12.3M tokens seen, ~10 passes over the training split) · ~5 minutes on a laptop CPU
final loss	train 0.658 · val 0.662

Training data is generated, not scraped, from a grammar in the repo, so a learner can read the entire source of the training set.

Limitations

• Not text-to-SQL. No natural-language input.
• One fixed three-table schema. It knows no other tables or columns.
• 64-token context. Longer queries are truncated.
• Does not generalize to column names it never saw in a given syntactic position (see above, that failure is the point, and it is measured rather than hidden).
• Generated SQL is syntactically valid, not semantically meaningful. It will happily write WHERE age > 1500.

License

MIT.