| --- |
| base_model: Qwen/Qwen2.5-Coder-0.5B-Instruct |
| library_name: peft |
| pipeline_tag: text-generation |
| tags: |
| - "base_model:adapter:Qwen/Qwen2.5-Coder-0.5B-Instruct" |
| - lora |
| - sft |
| - text-to-sql |
| --- |
| |
| # **SQLQwen - Qwen2.5 Text-to-SQL Fine-Tuning with LoRA** |
|
|
| LoRA fine-tuning of **Qwen2.5-Coder-0.5B-Instruct** for **Text-to-SQL generation** using the **Gretel synthetic_text_to_sql** dataset. |
| |
| Text-to-SQL systems translate natural-language questions into SQL using a provided database schema or SQL context. SQLQwen is designed to specialize a lightweight code-focused language model for this task while keeping the fine-tuning process parameter-efficient and practical on day-to-day (normal) hardware. |
| |
| Given: |
| |
| 1. a database schema or SQL context |
| 2. a natural-language request |
| |
| the model generates the relevant SQL query |
| |
| ### Example |
| |
| **Database context:** |
| |
| ```sql |
| CREATE TABLE customers ( |
| id INTEGER PRIMARY KEY, |
| name TEXT, |
| country TEXT, |
| revenue DECIMAL(12, 2) |
| ); |
| ``` |
| |
| **Natural-language request:** |
| |
| ```text |
| Find the five customers with the highest revenue. |
| ``` |
| |
| **Expected output:** |
| |
| ```sql |
| SELECT id, name, country, revenue |
| FROM customers |
| ORDER BY revenue DESC |
| LIMIT 5; |
| ``` |
| |
| ## Why LoRA |
| |
| LoRA (**Low-Rank Adaptation**) provides a parameter-efficient alternative to full fine-tuning. |
| |
| Instead of updating all parameters in the pretrained model, LoRA freezes the original model weights and introduces small trainable low-rank matrices into selected Transformer layers. |
| |
| for this project, LoRA adapters are applied to the attention projection layers: |
| |
| ```text |
| q_proj |
| k_proj |
| v_proj |
| o_proj |
| ``` |
| |
| this reduces the number of trainable parameters, GPU memory requirements, and adapter storage size while preserving the capabilities of the original pretrained model. |
| |
| Only **2,162,688 of 496,195,456 parameters**, or approximately **0.4359%**, were trainable during fine-tuning. |
| |
| ## Model |
| |
| * **Base model:** [`Qwen/Qwen2.5-Coder-0.5B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B-Instruct) |
| * **Fine-tuning:** PEFT LoRA |
| * **Training:** TRL `SFTTrainer` |
| * **LoRA rank:** `16` |
| * **LoRA alpha:** `32` |
| * **LoRA dropout:** `0.05` |
| * **Target modules:** `q_proj`, `k_proj`, `v_proj`, `o_proj` |
| * **Training objective:** Completion-only supervised fine-tuning |
| |
| Qwen2.5-Coder was selected because SQL generation is fundamentally a structured code-generation task rather than conventional natural-language classification. |
| |
| ## Dataset |
| |
| ### Gretel Synthetic Text-to-SQL |
| |
| [`gretelai/synthetic_text_to_sql`](https://huggingface.co/datasets/gretelai/synthetic_text_to_sql) |
| |
| Gretel Synthetic Text-to-SQL dataset provides natural-language SQL requests, database context, target SQL queries, and supporting metadata. |
| |
| 3 fields are used directly during training: |
| |
| | Dataset field | Purpose | |
| | ------------- | ------------------------------ | |
| | `sql_context` | Database schema or SQL context | |
| | `sql_prompt` | Natural-language request | |
| | `sql` | Ground-truth SQL completion | |
| |
| |
| ## Training Run |
| |
| | Metric | Result | |
| | ------------------------------- | ----------: | |
| | Training examples | **20,000** | |
| | Validation examples | **1,000** | |
| | Epochs | **2** | |
| | Optimizer steps | **2,500** | |
| | Final training loss | **0.2381** | |
| | Best evaluation loss | **0.2198** | |
| | Final evaluation loss | **0.2200** | |
| | Final evaluation token accuracy | **93.54%** | |
| | Best checkpoint | **2,400** | |
| | Training runtime | **~1h 35m** | |
| |
| Training was completed locally on an **NVIDIA GeForce RTX 3050**. |
| |
| Validation loss decreased from **0.2839** at the first logged evaluation to a best value of **0.2198** at checkpoint 2,400. The final checkpoint shows an evaluation loss of **0.2200**, indicating that training had largely converged by the end of the second epoch. |
| |
| ## Held-Out Generation Evaluation |
| |
| A held-out generation benchmark of **50 examples** was used to evaluate SQL generation code quality. |
| |
| | Metric | Result | |
| | ------------------------ | ----------: | |
| | Evaluation examples | **50** | |
| | Exact-match accuracy | **28.0%** | |
| | SQL syntax validity | **100.0%** | |
| | Exact matches | **14 / 50** | |
| | Syntax-valid generations | **50 / 50** | |
| |
| All **50 generated SQL queries were successfully parsed by SQLGlot**, resulting in a **100% syntax-validity rate**. |
| |
| **28% exact-match score** uses strict string-level comparison. Semantically or execution-equivalent SQL queries may differ from the reference query while still producing the correct result, so exact match should not be interpreted as the model's full semantic accuracy. |
| |
| |
| ## Model Limitations: |
| |
| * model may hallucinate tables or columns when the supplied database context is incomplete. |
| * **0.5B parameter** base model prioritizes lightweight training and inference over maximum reasoning capacity. |
| * Training uses synthetic Text-to-SQL examples, which may not represent every real-world database schema or production SQL workload. |
| * Exact-match evaluation does not account for all semantically equivalent SQL formulations. |
| |
| Additional Note: SQL execution accuracy against live databases was not measured in the reported benchmark. |
| |
| ## Results: |
| |
| completed fine-tuning run shows: |
| |
| * **parameter-efficient adaptation**, with approximately **0.4359%** of model parameters trainable |
| * **stable convergence**, with validation loss reaching approximately **0.22** |
| * **100% syntax-valid SQL generation** across the 50-example held-out benchmark |
| * successful local fine-tuning of a code-focused language model using an **NVIDIA GeForce RTX 3050** |
| |
| ## Imports (PEFT library): |
| ```python |
| from peft import PeftModel |
| from transformers import AutoModelForCausalLM |
| |
| base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-0.5B-Instruct") |
| model = PeftModel.from_pretrained(base_model, "AaronTekle/SQLQwen") |
| ``` |
| |
| ## References: |
| - Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. *LoRA: Low-Rank Adaptation of Large Language Models*. arXiv:2106.09685, 2021.](https://arxiv.org/abs/2106.09685) |
| |
| - Hugging Face. *PEFT LoRA Documentation*. Parameter-Efficient Fine-Tuning documentation. (https://huggingface.co/docs/transformers/en/peft) (https://huggingface.co/docs/peft/en/package_reference/lora) |
| |
| - Qwen Team. Qwen2.5-Coder-0.5B-Instruct Model Card |
| |
| - Hui, B. et al. Qwen2.5-Coder Technical Report. arXiv:2409.12186, 2024 |
| |
| - Gretel.ai. synthetic_text_to_sql Dataset Card. Hugging Face Datasets |
