Qwen2.5-Coder Technical Report
Paper • 2409.12186 • Published • 159
SQLQwen - Qwen2.5 Text-to-SQL Fine-Tuning with LoRA
LoRA fine-tuning of Qwen2.5-Coder-0.5B-Instruct for Text-to-SQL generation using the Gretel synthetic_text_to_sql dataset.
Text-to-SQL systems translate natural-language questions into SQL using a provided database schema or SQL context. SQLQwen is designed to specialize a lightweight code-focused language model for this task while keeping the fine-tuning process parameter-efficient and practical on day-to-day (normal) hardware.
Given:
- a database schema or SQL context
- a natural-language request
the model generates the relevant SQL query
Example
Database context:
CREATE TABLE customers (
id INTEGER PRIMARY KEY,
name TEXT,
country TEXT,
revenue DECIMAL(12, 2)
);
Natural-language request:
Find the five customers with the highest revenue.
Expected output:
SELECT id, name, country, revenue
FROM customers
ORDER BY revenue DESC
LIMIT 5;
Why LoRA
LoRA (Low-Rank Adaptation) provides a parameter-efficient alternative to full fine-tuning.
Instead of updating all parameters in the pretrained model, LoRA freezes the original model weights and introduces small trainable low-rank matrices into selected Transformer layers.
for this project, LoRA adapters are applied to the attention projection layers:
q_proj
k_proj
v_proj
o_proj
this reduces the number of trainable parameters, GPU memory requirements, and adapter storage size while preserving the capabilities of the original pretrained model.
Only 2,162,688 of 496,195,456 parameters, or approximately 0.4359%, were trainable during fine-tuning.
Model
- Base model:
Qwen/Qwen2.5-Coder-0.5B-Instruct - Fine-tuning: PEFT LoRA
Qwen2.5-Coder was selected because SQL generation is fundamentally a structured code-generation task rather than conventional natural-language classification.
Dataset
Gretel Synthetic Text-to-SQL
gretelai/synthetic_text_to_sql
Gretel Synthetic Text-to-SQL dataset provides natural-language SQL requests, database context, target SQL queries, and supporting metadata.
3 fields are used directly during training:
| Dataset field | Purpose |
|---|---|
sql_context |
Database schema or SQL context |
sql_prompt |
Natural-language request |
sql |
Ground-truth SQL completion |
Training Run
| Metric | Result |
|---|---|
| Training examples | 20,000 |
| Validation examples | 1,000 |
| Epochs | 2 |
| Optimizer steps | 2,500 |
| Final training loss | 0.2381 |
| Best evaluation loss | 0.2198 |
| Final evaluation loss | 0.2200 |
| Final evaluation token accuracy | 93.54% |
| Best checkpoint | 2,400 |
Training was completed locally on an NVIDIA GeForce RTX 3050.
Validation loss decreased from 0.2839 at the first logged evaluation to a best value of 0.2198 at checkpoint 2,400. The final checkpoint shows an evaluation loss of 0.2200, indicating that training had largely converged by the end of the second epoch.
Held-Out Generation Evaluation
A held-out generation benchmark of 50 examples was used to evaluate SQL generation code quality.
| Metric | Result |
|---|---|
| Evaluation examples | 50 |
| Exact-match accuracy | 28.0% |
| SQL syntax validity | 100.0% |
| Exact matches | 14 / 50 |
| Syntax-valid generations | 50 / 50 |
All 50 generated SQL queries were successfully parsed by SQLGlot, resulting in a 100% syntax-validity rate.
28% exact-match score uses strict string-level comparison. Semantically or execution-equivalent SQL queries may differ from the reference query while still producing the correct result, so exact match should not be interpreted as the model's full semantic accuracy.
Model Limitations:
- model may hallucinate tables or columns when there is no context
- 0.5B parameter base model prioritizes lightweight training and inference over maximum reasoning capacity
- Training uses synthetic Text-to-SQL examples (does not use real-world database schemas)
- Exact-match evaluation does not account for all semantically equivalent SQL formulations
Additional Note: SQL execution accuracy against live databases was not measured in the reported benchmark.
Results:
completed fine-tuning run shows:
- parameter-efficient adaptation, with approximately 0.4359% of model parameters trainable
- stable convergence, with validation loss reaching approximately 0.22
- 100% syntax-valid SQL generation across the 50-example held-out benchmark
- successful local fine-tuning of a code-focused language model using an NVIDIA GeForce RTX 3050
Example Import (PEFT library):
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen2.5-Coder-0.5B-Instruct"
)
# Load LoRA adapter
model = PeftModel.from_pretrained(
base_model,
"AaronTekle/SQLQwen"
)
# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(
"Qwen/Qwen2.5-Coder-0.5B-Instruct"
)
# Database table context
schema = """
Database: example_company_db
Table: employees
Columns:
- employee_id (INT, Primary Key)
- first_name (VARCHAR)
- last_name (VARCHAR)
- department_id (INT)
- salary (DECIMAL)
- hire_date (DATE)
Table: departments
Columns:
- department_id (INT, Primary Key)
- department_name (VARCHAR)
- location (VARCHAR)
Table: projects
Columns:
- project_id (INT, Primary Key)
- project_name (VARCHAR)
- employee_id (INT, Foreign Key -> employees.employee_id)
- budget (DECIMAL)
"""
# User question
question = "Find the names of employees earning more than 50000."
# Build prompt
prompt = f"""
You are an expert SQL Developer.
Generate a SQL query based on the database schema and user question.
Database Schema:
{schema}
Question:
{question}
"""
# Tokenize
inputs = tokenizer(prompt, return_tensors="pt")
# Generate SQL
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=200,
do_sample=False,
pad_token_id=tokenizer.eos_token_id
)
# Display result
generated_text = tokenizer.decode(
outputs[0][inputs["input_ids"].shape[1]:],
skip_special_tokens=True
)
print("\nGenerated SQL:")
print(generated_text)
References:
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685, 2021.](https://arxiv.org/abs/2106.09685)
Hugging Face. PEFT LoRA Documentation. Parameter-Efficient Fine-Tuning documentation. (https://huggingface.co/docs/transformers/en/peft) (https://huggingface.co/docs/peft/en/package_reference/lora)
Qwen Team. Qwen2.5-Coder-0.5B-Instruct Model Card
Hui, B. et al. Qwen2.5-Coder Technical Report. arXiv:2409.12186, 2024
Gretel.ai. synthetic_text_to_sql Dataset Card. Hugging Face Datasets
- Downloads last month
- 105
Model tree for AaronTekle/SQLQwen
Base model
Qwen/Qwen2.5-0.5B Finetuned
Qwen/Qwen2.5-Coder-0.5B Finetuned
Qwen/Qwen2.5-Coder-0.5B-Instruct
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-0.5B-Instruct") model = PeftModel.from_pretrained(base_model, "AaronTekle/SQLQwen")