This repository contains no weights: it hosts the write-up (article) of the DataPilot AI project. The adapter is in the model repository linked below.

DataPilot AI: turning a 3B model into a SQL analyst agent, and what we could not prove

Lexsi hackathon, Track 3. Code: https://github.com/prprprmk12/datapilot-ai 路 Adapter: https://huggingface.co/Ilirt/datapilot-tiny-aya-sql-sft-extra 路 Data and logs: https://huggingface.co/datasets/Ilirt/datapilot-ecommerce-sql 路 Results page: https://huggingface.co/spaces/Ilirt/datapilot-ai

The idea

Business questions rarely arrive as SQL. We built DataPilot AI, an agent that answers a question asked in Russian by looking at a PostgreSQL schema, writing one SELECT query, running it, and showing every step. The model is Tiny Aya Global (3.35B parameters), small enough to run on one T4.

The agent works through four tools: list_tables, describe_table, execute_sql and finish. Safety does not depend on the model. A SQL guard accepts only a single SELECT, and the database role is read-only.

How we measured it

We used a synthetic e-commerce database with five tables. Tasks are split by task family, not by row: 69 training tasks, 12 for development and 18 for testing, plus 51 extra training tasks. The three test families (paid-profit, revenue-by-country, payment-status-counts) never appear in training.

A task is solved only if the result of the agent's SQL equals the result of a reference query, including column names. Reference queries are never shown to the model.

What worked

We fine-tuned with SFT (LoRA, r=16, 2 epochs) on trajectories produced by the real tools. The base model solved 0 of 18 test tasks. After SFT on the extended data it solved 12 of 18.

Then we ran the whole notebook a second time, from scratch. The same recipe gave 10 of 18.

Variant Run 1 Run 2
Base model 0/18 0/18
SFT on the original data 2/18 1/18
SFT plus extra tasks 12/18 10/18
plus SQL retry after an error (x3) 13/18 not measured
plus GRPO, 60 steps not measured 11/18

The reliable result is the first three rows: SFT on the extended data moves the model from zero to 10-12 of 18. The gap between the two runs, two tasks, is as large as any other difference in the table.

What did not work

paid-profit stayed at 0 of 6 in every run. The model computes profit from catalog prices (products.price - products.cost) instead of the prices on the order line (order_items.unit_price - unit_cost). For one product that gives 120 instead of 110.

Retrying after an SQL error removed the errors but not the mistakes. Execution errors dropped from 6 of 18 to 0, and one more task was marked solved. That task has LIMIT 1, where the wrong formula happens to give the right answer. Without the retry the failure was loud; with it, the failure is silent.

GRPO did not help. We ran GRPO through AgentTune with a reward computed from the SQL the agent actually executed: 1.0 for matching rows and column names, 0.3 for matching values only, 0 otherwise. After 60 steps the test result was 11 of 18 against 10 of 18 before, which is within run-to-run noise. The training log explains why: of 240 trajectories, 171 scored 0, and in most steps all four generations received the same reward, which gives no gradient. In the AgentTune format the model also writes the whole tool chain in one go and invents the tool outputs instead of waiting for them.

Honest limits

  • The test set is small: 18 tasks, three families, one evaluation per model, no confidence intervals.
  • It is not fully clean. We created the extra training families after we saw the errors on the test set. The test families themselves were never used for training.
  • The quality of the final answer text was not measured. In one demo the agent said the result was empty while three rows had been returned.
  • Refusing destructive requests ("delete all orders") is done by the SQL guard, not by the model.

What we would do next

Add training examples that use order-line prices, build a new held-out test from other task families, grow the test set towards 100 tasks, and run each variant several times to report intervals instead of single numbers.

Reproduce it

The GitHub repository contains the notebook, scripts, tests and the logs of both runs. The adapter, dataset, per-task traces and GRPO trajectories are on Hugging Face, and a static Space lets you open every test task and compare what each model variant did.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 馃檵 Ask for provider support