export/sqlbot/model_card.md

# sqlbot

A BI startup wants a small text to SQL model that runs on a CPU box next to their warehouse, answers with one SQL statement and nothing else, and is judged by whether the query returns the right rows.

## Model

Decoder only transformer trained from scratch: 6 layers, dim 384, 6 heads with 2 key value heads, context 768, vocabulary 8192 (byte level BPE trained on the client corpus), 12.6M parameters.
Pretrained on 80M tokens in 814 steps (19 minutes, 2 process(es), torch.float16).
Post training: supervised fine tuning, then DPO on preference pairs mined from the model's own failed samples. Export: the sft checkpoint (the stage with the best execution_accuracy), int8 weights in safetensors.

## Results

| stage | perplexity | valid_rate | exact_match | gold_executable_rate | execution_accuracy | pass_rate |
|---|---|---|---|---|---|---|
| pretrain | 2.23 | 0.000 | 0.000 | 0.779 | 0.000 | 0.000 |
| sft | 4.03 | 0.693 | 0.269 | 0.779 | 0.583 | 0.482 |
| dpo | 4.17 | 0.663 | 0.166 | 0.779 | 0.493 | 0.400 |
| export | 4.03 | 0.696 | 0.273 | 0.779 | 0.585 | 0.485 |

## Gates

| metric | bound | value | passed |
|---|---|---|---|
| execution_accuracy | min 0.4 | 0.585 | yes |
| valid_rate | min 0.8 | 0.696 | no |

## Samples

Prompt:

```
Schema:
CREATE TABLE creative_ai (application_id INT, name TEXT, region TEXT, explainability_score FLOAT); INSERT INTO creative_ai (application_id, name, region, explainability_score) VALUES (1, 'ApplicationX', 'Europe', 0.87), (2, 'ApplicationY', 'North America', 0.91), (3, 'ApplicationZ', 'Europe', 0.84), (4, 'ApplicationAA', 'North America', 0.93), (5, 'ApplicationAB', 'Europe', 0.89);

Question: What is the average explainability score of creative AI applications in 'Europe' and 'North America' in the 'creative_ai' table?
```

Output:

```
SELECT AVG(explainability_score) FROM creative_ai WHERE region IN ('Europe', 'North America');
```

Prompt:

```
Schema:
CREATE TABLE rural_infrastructure (id INT, project_name TEXT, sector TEXT, country TEXT, completion_date DATE); INSERT INTO rural_infrastructure (id, project_name, sector, country, completion_date) VALUES (1, 'Water Supply Expansion', 'Infrastructure', 'Indonesia', '2008-05-15'), (2, 'Rural Electrification', 'Infrastructure', 'Indonesia', '2012-08-28'), (3, 'Transportation Improvement', 'Infrastructure', 'Indonesia', '2009-12-31');

Question: Delete all records of rural infrastructure projects in Indonesia that have a completion date before 2010.
```

Output:

```
DELETE FROM rural_infrastructure WHERE country = 'Indonesia' AND completion_date < '2010-01-01';
```

Prompt:

```
Schema:
CREATE TABLE Accidents (id INT, launch_provider VARCHAR(255), year INT, description TEXT); INSERT INTO Accidents (id, launch_provider, year, description) VALUES (1, 'SpaceX', 2015, 'Falcon 9 explosion'), (2, 'Blue Origin', 2011, 'Propulsion system failure'), (3, 'SpaceX', 2016, 'Falcon 9 explosion');

Question: How many accidents have been recorded for SpaceX and Blue Origin rocket launches?
```

Output:

```
SELECT COUNT(*) FROM Accidents WHERE launch_provider IN ('SpaceX', 'Blue Origin') AND year BETWEEN 2015 AND 2020;
```