| --- |
| base_model: Qwen/Qwen2.5-3B-Instruct |
| library_name: peft |
| pipeline_tag: text-generation |
| tags: |
| - base_model:adapter:Qwen/Qwen2.5-3B-Instruct |
| - lora |
| - transformers |
| datasets: |
| - xlangai/spider |
| --- |
| |
| # Distill — Fine-tuned Qwen2.5-3B for Text-to-SQL |
|
|
| Distill is a **QLoRA fine-tuned LoRA adapter** based on [`Qwen/Qwen2.5-3B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct), specialized for **natural-language-to-SQL generation**. |
|
|
| The model takes a database schema and a natural-language question and generates a SQL query. |
|
|
| The model was fine-tuned on the `sql-create-context` dataset using 4-bit QLoRA on a single NVIDIA T4 GPU. The final LoRA adapter is approximately **114 MB** with approximately **21M trainable parameters**. |
|
|
| ## Model Details |
|
|
| * **Developed by:** MahboobAlam0 |
| * **Model type:** LoRA / QLoRA adapter |
| * **Base model:** Qwen/Qwen2.5-3B-Instruct |
| * **Task:** Natural language → SQL |
| * **Fine-tuning method:** QLoRA |
| * **Dataset:** sql-create-context |
| * **Training examples:** 28,000 |
| * **Validation examples:** 7,858 |
| * **LoRA rank:** 16 |
| * **LoRA alpha:** 32 |
| * **Quantization:** 4-bit NF4 with double quantization |
| * **Trainable parameters:** ~21M |
| * **Adapter size:** ~114 MB |
| * **Hardware:** 1 × NVIDIA T4 |
| * **Training platform:** Kaggle |
| * **Programming language:** Python 3.12 |
|
|
| ## Model Description |
|
|
| Distill adapts Qwen2.5-3B-Instruct for text-to-SQL generation using parameter-efficient fine-tuning. |
|
|
| The base model is loaded using 4-bit quantization and kept frozen while LoRA adapters are trained on selected attention and MLP projection layers. |
|
|
| The LoRA adapters target: |
|
|
| * `q_proj` |
| * `k_proj` |
| * `v_proj` |
| * `o_proj` |
| * `gate_proj` |
| * `up_proj` |
| * `down_proj` |
|
|
| The model is designed to generate SQL from a database schema and a natural-language question. |
|
|
| ### Architecture |
|
|
| ```text |
| Natural Language Question |
| + |
| CREATE TABLE Schema |
| │ |
| ▼ |
| ┌─────────────────────────────────────┐ |
| │ Qwen2.5-3B-Instruct │ |
| │ Frozen 4-bit NF4 quantized weights │ |
| │ │ |
| │ LoRA adapters (~21M parameters) │ |
| │ │ |
| │ Attention: q, k, v, o │ |
| │ MLP: gate, up, down │ |
| └─────────────────────────────────────┘ |
| │ |
| ▼ |
| Generated SQL |
| ``` |
|
|
| ## Intended Use |
|
|
| This model is intended for: |
|
|
| * Natural-language database querying |
| * Text-to-SQL applications |
| * SQL generation |
| * Database assistants |
| * Analytics interfaces |
| * Business intelligence prototypes |
| * SQL education and experimentation |
| * Research into parameter-efficient fine-tuning |
|
|
| A typical input contains: |
|
|
| 1. A SQL database schema |
| 2. A natural-language question |
|
|
| The model then generates the corresponding SQL query. |
|
|
| ### Example |
|
|
| **Schema** |
|
|
| ```sql |
| CREATE TABLE orders ( |
| id INTEGER, |
| customer VARCHAR, |
| amount FLOAT, |
| status VARCHAR |
| ); |
| ``` |
|
|
| **Question** |
|
|
| ```text |
| What is the total revenue from completed orders? |
| ``` |
|
|
| **Generated SQL** |
|
|
| ```sql |
| SELECT SUM(amount) |
| FROM orders |
| WHERE status = 'completed'; |
| ``` |
|
|
| ## Out-of-Scope Use |
|
|
| This model should not be treated as an unrestricted autonomous database agent. |
|
|
| It should not be used without appropriate validation for: |
|
|
| * Destructive database operations |
| * Production database migrations |
| * Database authorization or access-control decisions |
| * Security-sensitive operations |
| * Financial decisions |
| * Medical decisions |
| * Legal decisions |
| * Other high-stakes applications |
|
|
| Generated SQL should always be treated as untrusted model output. |
|
|
| ## Limitations |
|
|
| ### Dataset limitations |
|
|
| The model was trained on `sql-create-context`, which primarily contains single-table schema examples. |
|
|
| Performance may therefore decrease on: |
|
|
| * Complex multi-table schemas |
| * Multiple joins |
| * Nested queries |
| * Large enterprise schemas |
| * Highly domain-specific databases |
|
|
| ### SQL dialect limitations |
|
|
| The evaluation uses SQLite. |
|
|
| The model should not be assumed to produce correct syntax for PostgreSQL, MySQL, SQL Server, Oracle, or other SQL dialects without additional testing. |
|
|
| ### SQL correctness |
|
|
| The model can generate SQL that is syntactically valid but semantically incorrect. |
|
|
| Generated SQL should be validated against the actual database schema and, where appropriate, tested before execution. |
|
|
| ### Model size |
|
|
| The base model contains approximately 3B parameters. While this makes the model relatively lightweight, it also limits its ability to handle highly complex SQL generation compared with substantially larger models. |
|
|
| ### Evaluation limitations |
|
|
| Execution accuracy was measured using schema-seeded in-memory SQLite databases with empty tables. Consequently, execution accuracy primarily measures structural and semantic SQL equivalence under the evaluation setup rather than correctness against realistic database contents. |
|
|
| ## Recommendations |
|
|
| For production applications: |
|
|
| * Validate generated SQL before execution. |
| * Prefer read-only database credentials. |
| * Restrict permitted SQL operations. |
| * Validate referenced tables and columns. |
| * Use query timeouts and resource limits. |
| * Test queries against representative data. |
| * Do not expose unrestricted production database credentials to the model. |
| * Consider SQL parsing and allowlisting before execution. |
|
|
| ## Training |
|
|
| ### Training Dataset |
|
|
| The model was fine-tuned on the `sql-create-context` dataset. |
|
|
| | Split | Examples | |
| | ---------- | -------: | |
| | Training | 28,000 | |
| | Validation | 7,858 | |
|
|
| The dataset provides database schemas, natural-language questions, and corresponding SQL queries. |
|
|
| ### Fine-Tuning Method |
|
|
| The model was trained using **QLoRA**. |
|
|
| The base Qwen2.5-3B-Instruct model was loaded using 4-bit NF4 quantization with double quantization. The base weights were frozen while LoRA adapters were trained. |
|
|
| ### Training Configuration |
|
|
| | Parameter | Value | |
| | -------------------- | ------------------------ | |
| | Base model | Qwen/Qwen2.5-3B-Instruct | |
| | Quantization | 4-bit NF4 | |
| | Double quantization | Enabled | |
| | LoRA rank | 16 | |
| | LoRA alpha | 32 | |
| | Learning rate | 2e-4 | |
| | Scheduler | Cosine decay | |
| | Warmup | 5% | |
| | Optimizer | Paged AdamW 8-bit | |
| | Effective batch size | 16 | |
| | Hardware | NVIDIA T4 | |
| | Platform | Kaggle | |
|
|
| The repository reports that LoRA rank 8 underfit during initial experiments, while rank 32 did not provide sufficient improvement to justify its additional adapter memory. Rank 16 was selected for the final training configuration. |
|
|
| ### Checkpoint Selection |
|
|
| The best validation checkpoint was reported at approximately step 200. |
|
|
| | Checkpoint | Step | Validation Loss | Result | |
| | -------------- | ---: | --------------: | ----------------------- | |
| | checkpoint-200 | 200 | 0.685 | Best checkpoint | |
| | checkpoint-800 | 800 | 0.829 | Overfitting / discarded | |
|
|
| The best checkpoint was restored for the final model. |
|
|
| ## Evaluation |
|
|
| The model was evaluated on **7,858 held-out examples** from the validation data. |
|
|
| The base-model comparison was performed on a **300-example subset** using the repository's SQL extraction procedure. |
|
|
| ### Metrics |
|
|
| #### Exact Match |
|
|
| Exact Match compares normalized generated SQL with the normalized reference SQL. |
|
|
| ```text |
| normalise(prediction) == normalise(reference) |
| ``` |
|
|
| This is a strict metric. Semantically equivalent SQL queries can receive different scores if their textual formulation differs. |
|
|
| #### BLEU |
|
|
| BLEU measures token-level n-gram overlap between generated and reference SQL. |
|
|
| #### Execution Accuracy |
|
|
| Generated and reference SQL queries are executed against schema-seeded in-memory SQLite databases. |
|
|
| ```text |
| execution(prediction) == execution(reference) |
| ``` |
|
|
| This provides a measure of semantic equivalence that is less sensitive to different SQL formulations. |
|
|
| ## Results |
|
|
| | Metric | Base Qwen2.5-3B | Distill | |
| | --------------------- | --------------: | ---------: | |
| | Exact Match Accuracy | 2.33%* | **30.67%** | |
| | BLEU Score | 51.87%* | **59.74%** | |
| | Execution Accuracy | — | **55.03%** | |
| | Validation Perplexity | — | **1.98** | |
|
|
| * Base-model measurements were obtained on a 300-example subset. |
|
|
| The repository reports a **13× improvement in exact-match accuracy** compared with its measured base-model result. |
|
|
| ### Result Interpretation |
|
|
| The difference between exact-match accuracy and execution accuracy is important: |
|
|
| ```text |
| Exact Match Accuracy: 30.67% |
| Execution Accuracy: 55.03% |
| ``` |
|
|
| This indicates that some generated SQL queries differ from the reference query text while still producing equivalent results under the evaluation setup. |
|
|
| Therefore, exact match alone should not be considered a complete measure of SQL generation quality. |
|
|
| ## Base Model Comparison |
|
|
| The repository reports the following comparison: |
|
|
| ```text |
| Qwen2.5-3B-Instruct: |
| Exact Match: 2.33% |
| BLEU: 51.87% |
| |
| Distill: |
| Exact Match: 30.67% |
| BLEU: 59.74% |
| ``` |
|
|
| The repository also notes that the base model can return SQL together with Markdown code blocks or explanatory text. SQL extraction methodology therefore has a significant effect on the measured baseline. |
|
|
| ## How to Use |
|
|
| This repository contains a PEFT adapter. The compatible Qwen2.5-3B-Instruct base model is required. |
|
|
| ### Installation |
|
|
| ```bash |
| pip install torch transformers peft bitsandbytes accelerate |
| ``` |
|
|
| ### Load the Adapter |
|
|
| ```python |
| import torch |
| from transformers import ( |
| AutoModelForCausalLM, |
| AutoTokenizer, |
| BitsAndBytesConfig, |
| ) |
| from peft import PeftModel |
| |
| base_model = "Qwen/Qwen2.5-3B-Instruct" |
| adapter_path = "path/to/adapter" |
| |
| quantization_config = BitsAndBytesConfig( |
| load_in_4bit=True, |
| bnb_4bit_quant_type="nf4", |
| bnb_4bit_use_double_quant=True, |
| bnb_4bit_compute_dtype=torch.float16, |
| ) |
| |
| tokenizer = AutoTokenizer.from_pretrained(base_model) |
| |
| model = AutoModelForCausalLM.from_pretrained( |
| base_model, |
| quantization_config=quantization_config, |
| device_map="auto", |
| ) |
| |
| model = PeftModel.from_pretrained( |
| model, |
| adapter_path, |
| ) |
| ``` |
|
|
| ### Generate SQL |
|
|
| ```python |
| schema = """ |
| CREATE TABLE orders ( |
| id INTEGER, |
| customer VARCHAR, |
| amount FLOAT, |
| status VARCHAR |
| ); |
| """ |
| |
| question = "What is the total revenue from completed orders?" |
| |
| prompt = f""" |
| Schema: |
| {schema} |
| |
| Question: |
| {question} |
| |
| SQL: |
| """ |
| |
| inputs = tokenizer( |
| prompt, |
| return_tensors="pt" |
| ).to(model.device) |
| |
| outputs = model.generate( |
| **inputs, |
| max_new_tokens=128, |
| ) |
| |
| response = tokenizer.decode( |
| outputs[0], |
| skip_special_tokens=True, |
| ) |
| |
| print(response) |
| ``` |
|
|
| ## Repository |
|
|
| The complete training and application code is available in the project repository: |
|
|
| https://github.com/MahboobAlam0/Distill---Fine-tuned-Qwen2.5-3B |
|
|
| The repository includes: |
|
|
| * QLoRA training code |
| * Evaluation scripts |
| * Inference wrapper |
| * FastAPI API |
| * Streamlit interface |
| * Kaggle training notebook |
| * Training and validation data |
| * Docker configuration |
| * Tests |
|
|
| ## Serving |
|
|
| The project includes a FastAPI REST API and Streamlit interface. |
|
|
| ### FastAPI |
|
|
| ```bash |
| uvicorn api.main:app --reload --port 8000 |
| ``` |
|
|
| Example request: |
|
|
| ```bash |
| curl -X POST http://localhost:8000/generate \ |
| -H "Content-Type: application/json" \ |
| -d '{ |
| "schema": "CREATE TABLE products (id INTEGER, name VARCHAR, price FLOAT, stock INTEGER)", |
| "question": "Which products are out of stock?" |
| }' |
| ``` |
|
|
| ### Streamlit |
|
|
| ```bash |
| streamlit run app/streamlit_app.py |
| ``` |
|
|
| ### Docker |
|
|
| ```bash |
| docker-compose up |
| ``` |
|
|
| The repository exposes the API on port `8000` and the Streamlit interface on port `8501`. |
|
|
| ## Reproducing Training |
|
|
| The repository includes a self-contained Kaggle notebook. |
|
|
| ```text |
| 1. Upload kaggle/train_notebook.ipynb to Kaggle. |
| 2. Select an NVIDIA T4 GPU. |
| 3. Run the notebook. |
| 4. Export the resulting LoRA adapter. |
| ``` |
|
|
| The repository reports approximately **2 hours** for the documented training run on a single Kaggle T4 GPU. |
|
|
| ## Technical Specifications |
|
|
| ### Hardware |
|
|
| * 1 × NVIDIA T4 |
| * Approximately 15 GB VRAM |
|
|
| ### Software |
|
|
| * Python 3.12 |
| * PyTorch |
| * Transformers |
| * PEFT |
| * BitsAndBytes |
| * TRL |
| * NLTK |
| * SQLite |
| * FastAPI |
| * Uvicorn |
| * Streamlit |
| * Docker |
|
|
| ### Adapter |
|
|
| * LoRA rank: 16 |
| * LoRA alpha: 32 |
| * Approximately 21M trainable parameters |
| * Approximately 114 MB adapter size |
|
|
| ## Environmental Impact |
|
|
| Training was performed on a single NVIDIA T4 GPU on Kaggle. |
|
|
| The project does not provide a measured carbon-emissions value. |
|
|
| | Property | Value | |
| | ---------------- | ------------ | |
| | Hardware | NVIDIA T4 | |
| | GPU count | 1 | |
| | Training time | ~2 hours | |
| | Platform | Kaggle | |
| | Compute region | Not reported | |
| | Carbon emissions | Not measured | |
|
|
| ## License |
|
|
| The GitHub project is released under the **MIT License**. |
|
|
| However, this model is a fine-tuned adapter based on `Qwen/Qwen2.5-3B-Instruct`. The upstream Qwen model is listed under the **Qwen Research License Agreement**, so the upstream model's license and terms must also be respected. |
|
|
| For the complete upstream license terms, see: |
|
|
| https://huggingface.co/Qwen/Qwen2.5-3B-Instruct |
|
|
| For the project's MIT license, see the GitHub repository: |
|
|
| https://github.com/MahboobAlam0/Distill---Fine-tuned-Qwen2.5-3B |
|
|
| ## Citation |
|
|
| If you use this model or the associated project, please cite: |
|
|
| ```bibtex |
| @software{mahboobalam_distill_qwen25_3b, |
| author = {MahboobAlam0}, |
| title = {Distill - Fine-tuned Qwen2.5-3B}, |
| url = {https://github.com/MahboobAlam0/Distill---Fine-tuned-Qwen2.5-3B}, |
| license = {MIT} |
| } |
| ``` |
|
|
| ## Acknowledgements |
|
|
| This project builds upon: |
|
|
| * Qwen2.5-3B-Instruct |
| * Hugging Face Transformers |
| * PEFT |
| * TRL |
| * BitsAndBytes |
| * PyTorch |
| * `sql-create-context` |
| * SQLite |
|
|
| ## Model Card Contact |
|
|
| For questions, issues, or contributions, please use the GitHub repository's issue tracker. |
|
|
| **Project:** Distill — Fine-tuned Qwen2.5-3B |
| **Task:** Natural language → SQL |
| **Base model:** Qwen2.5-3B-Instruct |
| **Fine-tuning:** QLoRA |
| **Training data:** 28,000 examples |
| **Validation data:** 7,858 examples |
| **Exact Match:** 30.67% |
| **Execution Accuracy:** 55.03% |
| **Adapter size:** ~114 MB |
| **Hardware:** 1 × NVIDIA T4 |
