devenshah21/sqlforge-qwen7b-sft

SQLForge — Qwen2.5-Coder-7B SFT (Text-to-SQL)

A LoRA adapter that fine-tunes Qwen2.5-Coder-7B-Instruct for Text-to-SQL on a TPC-H-derived schema with custom domain conventions. Supervised fine-tuning raised execution accuracy on a held-out 55-query benchmark from 54.5% → 74.5%, closing 54% of the gap to a Claude-based production agent (91.7%) — while general SQL ability stayed intact (see Generalization).

Full project, training code, and evaluation harness: github.com/ShahaDeven/sql-forge

This is a domain adapter, not a general Text-to-SQL model. It is trained on one schema's conventions (categorical literals like churn_risk = 'HIGH_RISK', revenue via total_value * (1 - promo_reduction), canonical customer-side join paths). It expects the specific system prompt it was trained on — see Serving contract below.

Results (55-query execution-accuracy suite)

Graded by result-set comparison against gold on DuckDB (TPC-H SF=0.1) — the same grader that scores the production Claude agent at 91.7%.

Model	Execution accuracy	Valid SQL
Qwen2.5-Coder-7B (base, zero-shot)	54.5%	90.9%
Qwen2.5-Coder-7B (base, 3-shot)	61.8%	94.5%
This adapter (SFT, zero-shot)	74.5%	98.2%
Claude (full production agent)	91.7%¹	—

¹ Full agent: retrieval + few-shot + a 3-attempt self-healing loop. A product comparison, not a like-for-like model comparison.

Per-tier (SFT): simple_select 10/10 · aggregation 10/10 · single_join 8/10 · window_function 6/10 · multi_hop 5/10 · simulation 2/5.

Generalization (Spider zero-shot)

Run zero-shot on Spider dev (1,032 queries, 20 unseen SQLite databases) with a generic prompt — no house-style rules — to test whether fine-tuning damaged general SQL ability:

Model	Spider dev accuracy
Qwen2.5-Coder-7B (base)	82.4%
This adapter (SFT)	82.7%

+20pp in-domain for +0.3pp (i.e. zero, within noise) out-of-domain. No catastrophic forgetting: the adapter learned this schema's conventions, not a narrower notion of SQL.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "Qwen/Qwen2.5-Coder-7B-Instruct"
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "devenshah21/sqlforge-qwen7b-sft")
tok = AutoTokenizer.from_pretrained(base)

messages = [
    {"role": "system", "content": HOUSE_STYLE_SYSTEM_PROMPT},  # see Serving contract
    {"role": "user", "content": "Which region has the lowest revenue?"},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(prompt, return_tensors="pt").to(model.device),
                     max_new_tokens=512, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

Serve with vLLM (as used for all evals):

vllm serve Qwen/Qwen2.5-Coder-7B-Instruct \
    --enable-lora --lora-modules sqlforge=devenshah21/sqlforge-qwen7b-sft \
    --max-lora-rank 32 --port 8000

Serving contract

The adapter was trained and benchmarked (74.5%) with one specific prompt shape: a house-style system prompt (schema + the categorical-literal / revenue-formula / join-path rules) and the user question, zero-shot, temperature 0. Sending few-shot examples or a different prompt is an unmeasured configuration and will not reproduce the numbers above — fine-tuning already absorbed what few-shot was teaching. The exact prompt builder lives in phase0/run_baseline.py (build_system_prompt).

Training

Method	bf16 LoRA (no quantization)
LoRA	r=32, α=64, dropout 0.05, all 7 projections (q,k,v,o,gate,up,down)
Data	2,314 execution-validated synthetic pairs (distilled from Claude, gated on DuckDB execution + house-style checks)
Schedule	2 epochs, lr 2e-4 cosine, effective batch 16, max_seq 2048, completion-only loss
Hardware	1× A40 (48GB)

Limitations

• Single schema. Conventions are specific to this TPC-H-derived database; the Spider result shows general SQL is intact, but in-domain gains do not transfer to other schemas.
• Residual failures concentrate in a categorical-literal bug (churn_risk='High' vs 'HIGH_RISK'), output-column shape, and simulation/CTE format — see the project repo's failure-mode analysis.
• Sibling adapters (DPO, and a 1.5B SFT/GRPO track) are documented in the repo; neither beat this SFT checkpoint, which is the deployed model.

Framework versions

• PEFT 0.13.2