---
base_model: Qwen/Qwen2.5-Coder-0.5B-Instruct
library_name: peft
pipeline_tag: text-generation
tags:
  - "base_model:adapter:Qwen/Qwen2.5-Coder-0.5B-Instruct"
  - lora
  - sft
  - text-to-sql
---

# **SQLQwen - Qwen2.5 Text-to-SQL Fine-Tuning with LoRA**

LoRA fine-tuning of **Qwen2.5-Coder-0.5B-Instruct** for **Text-to-SQL generation** using the **Gretel synthetic_text_to_sql** dataset.

Text-to-SQL systems translate natural-language questions into SQL using a provided database schema or SQL context. SQLQwen is designed to specialize a lightweight code-focused language model for this task while keeping the fine-tuning process parameter-efficient and practical on day-to-day (normal) hardware.

Given:

1. a database schema or SQL context
2. a natural-language request

the model generates the relevant SQL query

### Example

**Database context:**

```sql
CREATE TABLE customers (
    id INTEGER PRIMARY KEY,
    name TEXT,
    country TEXT,
    revenue DECIMAL(12, 2)
);
```

**Natural-language request:**

```text
Find the five customers with the highest revenue.
```

**Expected output:**

```sql
SELECT id, name, country, revenue
FROM customers
ORDER BY revenue DESC
LIMIT 5;
```

## Why LoRA

LoRA (**Low-Rank Adaptation**) provides a parameter-efficient alternative to full fine-tuning.

Instead of updating all parameters in the pretrained model, LoRA freezes the original model weights and introduces small trainable low-rank matrices into selected Transformer layers.

for this project, LoRA adapters are applied to the attention projection layers:

```text
q_proj
k_proj
v_proj
o_proj
```

this reduces the number of trainable parameters, GPU memory requirements, and adapter storage size while preserving the capabilities of the original pretrained model.

Only **2,162,688 of 496,195,456 parameters**, or approximately **0.4359%**, were trainable during fine-tuning.

## Model

* **Base model:** [`Qwen/Qwen2.5-Coder-0.5B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B-Instruct)
* **Fine-tuning:** PEFT LoRA
* **Training:** TRL `SFTTrainer`
* **LoRA rank:** `16`
* **LoRA alpha:** `32`
* **LoRA dropout:** `0.05`
* **Target modules:** `q_proj`, `k_proj`, `v_proj`, `o_proj`
* **Training objective:** Completion-only supervised fine-tuning

Qwen2.5-Coder was selected because SQL generation is fundamentally a structured code-generation task rather than conventional natural-language classification.

## Dataset

### Gretel Synthetic Text-to-SQL

[`gretelai/synthetic_text_to_sql`](https://huggingface.co/datasets/gretelai/synthetic_text_to_sql)

Gretel Synthetic Text-to-SQL dataset provides natural-language SQL requests, database context, target SQL queries, and supporting metadata.

3 fields are used directly during training:

| Dataset field | Purpose                        |
| ------------- | ------------------------------ |
| `sql_context` | Database schema or SQL context |
| `sql_prompt`  | Natural-language request       |
| `sql`         | Ground-truth SQL completion    |


## Training Run

| Metric                          |      Result |
| ------------------------------- | ----------: |
| Training examples               |  **20,000** |
| Validation examples             |   **1,000** |
| Epochs                          |       **2** |
| Optimizer steps                 |   **2,500** |
| Final training loss             |  **0.2381** |
| Best evaluation loss            |  **0.2198** |
| Final evaluation loss           |  **0.2200** |
| Final evaluation token accuracy |  **93.54%** |
| Best checkpoint                 |   **2,400** |
| Training runtime                | **~1h 35m** |

Training was completed locally on an **NVIDIA GeForce RTX 3050**.

Validation loss decreased from **0.2839** at the first logged evaluation to a best value of **0.2198** at checkpoint 2,400. The final checkpoint shows an evaluation loss of **0.2200**, indicating that training had largely converged by the end of the second epoch.

## Held-Out Generation Evaluation

A held-out generation benchmark of **50 examples** was used to evaluate SQL generation code quality.

| Metric                   |      Result |
| ------------------------ | ----------: |
| Evaluation examples      |      **50** |
| Exact-match accuracy     |   **28.0%** |
| SQL syntax validity      |  **100.0%** |
| Exact matches            | **14 / 50** |
| Syntax-valid generations | **50 / 50** |

All **50 generated SQL queries were successfully parsed by SQLGlot**, resulting in a **100% syntax-validity rate**.

**28% exact-match score** uses strict string-level comparison. Semantically or execution-equivalent SQL queries may differ from the reference query while still producing the correct result, so exact match should not be interpreted as the model's full semantic accuracy.


## Model Limitations:

* model may hallucinate tables or columns when the supplied database context is incomplete.
* **0.5B parameter** base model prioritizes lightweight training and inference over maximum reasoning capacity.
* Training uses synthetic Text-to-SQL examples, which may not represent every real-world database schema or production SQL workload.
* Exact-match evaluation does not account for all semantically equivalent SQL formulations.

Additional Note: SQL execution accuracy against live databases was not measured in the reported benchmark.

## Results:

completed fine-tuning run shows:

* **parameter-efficient adaptation**, with approximately **0.4359%** of model parameters trainable
* **stable convergence**, with validation loss reaching approximately **0.22**
* **100% syntax-valid SQL generation** across the 50-example held-out benchmark
* successful local fine-tuning of a code-focused language model using an **NVIDIA GeForce RTX 3050**

## Imports (PEFT library):
```python
from peft import PeftModel
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-0.5B-Instruct")
model = PeftModel.from_pretrained(base_model, "AaronTekle/SQLQwen")
```

## References:
- Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. *LoRA: Low-Rank Adaptation of Large Language Models*. arXiv:2106.09685, 2021.](https://arxiv.org/abs/2106.09685)

- Hugging Face. *PEFT LoRA Documentation*. Parameter-Efficient Fine-Tuning documentation. (https://huggingface.co/docs/transformers/en/peft) (https://huggingface.co/docs/peft/en/package_reference/lora)

- Qwen Team. Qwen2.5-Coder-0.5B-Instruct Model Card

- Hui, B. et al. Qwen2.5-Coder Technical Report. arXiv:2409.12186, 2024

- Gretel.ai. synthetic_text_to_sql Dataset Card. Hugging Face Datasets