export/sqlbot/model_card.md

sqlbot

A BI startup wants a small text to SQL model that runs on a CPU box next to their warehouse, answers with one SQL statement and nothing else, and is judged by whether the query returns the right rows.

Model

Decoder only transformer trained from scratch: 6 layers, dim 384, 6 heads with 2 key value heads, context 768, vocabulary 8192 (byte level BPE trained on the client corpus), 12.6M parameters. Pretrained on 80M tokens in 814 steps (19 minutes, 2 process(es), torch.float16). Post training: supervised fine tuning, then DPO on preference pairs mined from the model's own failed samples. Export: the sft checkpoint (the stage with the best execution_accuracy), int8 weights in safetensors.

Results

stage	perplexity	valid_rate	exact_match	gold_executable_rate	execution_accuracy	pass_rate
pretrain	2.23	0.000	0.000	0.779	0.000	0.000
sft	4.03	0.693	0.269	0.779	0.583	0.482
dpo	4.17	0.663	0.166	0.779	0.493	0.400
export	4.03	0.696	0.273	0.779	0.585	0.485

Gates

metric	bound	value	passed
execution_accuracy	min 0.4	0.585	yes
valid_rate	min 0.8	0.696	no

Samples

Prompt:

Schema:
CREATE TABLE creative_ai (application_id INT, name TEXT, region TEXT, explainability_score FLOAT); INSERT INTO creative_ai (application_id, name, region, explainability_score) VALUES (1, 'ApplicationX', 'Europe', 0.87), (2, 'ApplicationY', 'North America', 0.91), (3, 'ApplicationZ', 'Europe', 0.84), (4, 'ApplicationAA', 'North America', 0.93), (5, 'ApplicationAB', 'Europe', 0.89);

Question: What is the average explainability score of creative AI applications in 'Europe' and 'North America' in the 'creative_ai' table?

Output:

SELECT AVG(explainability_score) FROM creative_ai WHERE region IN ('Europe', 'North America');

Prompt:

Schema:
CREATE TABLE rural_infrastructure (id INT, project_name TEXT, sector TEXT, country TEXT, completion_date DATE); INSERT INTO rural_infrastructure (id, project_name, sector, country, completion_date) VALUES (1, 'Water Supply Expansion', 'Infrastructure', 'Indonesia', '2008-05-15'), (2, 'Rural Electrification', 'Infrastructure', 'Indonesia', '2012-08-28'), (3, 'Transportation Improvement', 'Infrastructure', 'Indonesia', '2009-12-31');

Question: Delete all records of rural infrastructure projects in Indonesia that have a completion date before 2010.

Output:

DELETE FROM rural_infrastructure WHERE country = 'Indonesia' AND completion_date < '2010-01-01';

Prompt:

Schema:
CREATE TABLE Accidents (id INT, launch_provider VARCHAR(255), year INT, description TEXT); INSERT INTO Accidents (id, launch_provider, year, description) VALUES (1, 'SpaceX', 2015, 'Falcon 9 explosion'), (2, 'Blue Origin', 2011, 'Propulsion system failure'), (3, 'SpaceX', 2016, 'Falcon 9 explosion');

Question: How many accidents have been recorded for SpaceX and Blue Origin rocket launches?

Output:

SELECT COUNT(*) FROM Accidents WHERE launch_provider IN ('SpaceX', 'Blue Origin') AND year BETWEEN 2015 AND 2020;