---
base_model: Qwen/Qwen2.5-3B-Instruct
library_name: peft
pipeline_tag: text-generation
tags:
- base_model:adapter:Qwen/Qwen2.5-3B-Instruct
- lora
- transformers
datasets:
- xlangai/spider
---

# Distill — Fine-tuned Qwen2.5-3B for Text-to-SQL

Distill is a **QLoRA fine-tuned LoRA adapter** based on [`Qwen/Qwen2.5-3B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct), specialized for **natural-language-to-SQL generation**.

The model takes a database schema and a natural-language question and generates a SQL query.

The model was fine-tuned on the `sql-create-context` dataset using 4-bit QLoRA on a single NVIDIA T4 GPU. The final LoRA adapter is approximately **114 MB** with approximately **21M trainable parameters**.

## Model Details

* **Developed by:** MahboobAlam0
* **Model type:** LoRA / QLoRA adapter
* **Base model:** Qwen/Qwen2.5-3B-Instruct
* **Task:** Natural language → SQL
* **Fine-tuning method:** QLoRA
* **Dataset:** sql-create-context
* **Training examples:** 28,000
* **Validation examples:** 7,858
* **LoRA rank:** 16
* **LoRA alpha:** 32
* **Quantization:** 4-bit NF4 with double quantization
* **Trainable parameters:** ~21M
* **Adapter size:** ~114 MB
* **Hardware:** 1 × NVIDIA T4
* **Training platform:** Kaggle
* **Programming language:** Python 3.12

## Model Description

Distill adapts Qwen2.5-3B-Instruct for text-to-SQL generation using parameter-efficient fine-tuning.

The base model is loaded using 4-bit quantization and kept frozen while LoRA adapters are trained on selected attention and MLP projection layers.

The LoRA adapters target:

* `q_proj`
* `k_proj`
* `v_proj`
* `o_proj`
* `gate_proj`
* `up_proj`
* `down_proj`

The model is designed to generate SQL from a database schema and a natural-language question.

### Architecture

```text
Natural Language Question
          +
    CREATE TABLE Schema
          │
          ▼
┌─────────────────────────────────────┐
│ Qwen2.5-3B-Instruct                 │
│ Frozen 4-bit NF4 quantized weights │
│                                     │
│ LoRA adapters (~21M parameters)     │
│                                     │
│ Attention: q, k, v, o              │
│ MLP: gate, up, down                │
└─────────────────────────────────────┘
          │
          ▼
      Generated SQL
```

## Intended Use

This model is intended for:

* Natural-language database querying
* Text-to-SQL applications
* SQL generation
* Database assistants
* Analytics interfaces
* Business intelligence prototypes
* SQL education and experimentation
* Research into parameter-efficient fine-tuning

A typical input contains:

1. A SQL database schema
2. A natural-language question

The model then generates the corresponding SQL query.

### Example

**Schema**

```sql
CREATE TABLE orders (
    id INTEGER,
    customer VARCHAR,
    amount FLOAT,
    status VARCHAR
);
```

**Question**

```text
What is the total revenue from completed orders?
```

**Generated SQL**

```sql
SELECT SUM(amount)
FROM orders
WHERE status = 'completed';
```

## Out-of-Scope Use

This model should not be treated as an unrestricted autonomous database agent.

It should not be used without appropriate validation for:

* Destructive database operations
* Production database migrations
* Database authorization or access-control decisions
* Security-sensitive operations
* Financial decisions
* Medical decisions
* Legal decisions
* Other high-stakes applications

Generated SQL should always be treated as untrusted model output.

## Limitations

### Dataset limitations

The model was trained on `sql-create-context`, which primarily contains single-table schema examples.

Performance may therefore decrease on:

* Complex multi-table schemas
* Multiple joins
* Nested queries
* Large enterprise schemas
* Highly domain-specific databases

### SQL dialect limitations

The evaluation uses SQLite.

The model should not be assumed to produce correct syntax for PostgreSQL, MySQL, SQL Server, Oracle, or other SQL dialects without additional testing.

### SQL correctness

The model can generate SQL that is syntactically valid but semantically incorrect.

Generated SQL should be validated against the actual database schema and, where appropriate, tested before execution.

### Model size

The base model contains approximately 3B parameters. While this makes the model relatively lightweight, it also limits its ability to handle highly complex SQL generation compared with substantially larger models.

### Evaluation limitations

Execution accuracy was measured using schema-seeded in-memory SQLite databases with empty tables. Consequently, execution accuracy primarily measures structural and semantic SQL equivalence under the evaluation setup rather than correctness against realistic database contents.

## Recommendations

For production applications:

* Validate generated SQL before execution.
* Prefer read-only database credentials.
* Restrict permitted SQL operations.
* Validate referenced tables and columns.
* Use query timeouts and resource limits.
* Test queries against representative data.
* Do not expose unrestricted production database credentials to the model.
* Consider SQL parsing and allowlisting before execution.

## Training

### Training Dataset

The model was fine-tuned on the `sql-create-context` dataset.

| Split      | Examples |
| ---------- | -------: |
| Training   |   28,000 |
| Validation |    7,858 |

The dataset provides database schemas, natural-language questions, and corresponding SQL queries.

### Fine-Tuning Method

The model was trained using **QLoRA**.

The base Qwen2.5-3B-Instruct model was loaded using 4-bit NF4 quantization with double quantization. The base weights were frozen while LoRA adapters were trained.

### Training Configuration

| Parameter            | Value                    |
| -------------------- | ------------------------ |
| Base model           | Qwen/Qwen2.5-3B-Instruct |
| Quantization         | 4-bit NF4                |
| Double quantization  | Enabled                  |
| LoRA rank            | 16                       |
| LoRA alpha           | 32                       |
| Learning rate        | 2e-4                     |
| Scheduler            | Cosine decay             |
| Warmup               | 5%                       |
| Optimizer            | Paged AdamW 8-bit        |
| Effective batch size | 16                       |
| Hardware             | NVIDIA T4                |
| Platform             | Kaggle                   |

The repository reports that LoRA rank 8 underfit during initial experiments, while rank 32 did not provide sufficient improvement to justify its additional adapter memory. Rank 16 was selected for the final training configuration.

### Checkpoint Selection

The best validation checkpoint was reported at approximately step 200.

| Checkpoint     | Step | Validation Loss | Result                  |
| -------------- | ---: | --------------: | ----------------------- |
| checkpoint-200 |  200 |           0.685 | Best checkpoint         |
| checkpoint-800 |  800 |           0.829 | Overfitting / discarded |

The best checkpoint was restored for the final model.

## Evaluation

The model was evaluated on **7,858 held-out examples** from the validation data.

The base-model comparison was performed on a **300-example subset** using the repository's SQL extraction procedure.

### Metrics

#### Exact Match

Exact Match compares normalized generated SQL with the normalized reference SQL.

```text
normalise(prediction) == normalise(reference)
```

This is a strict metric. Semantically equivalent SQL queries can receive different scores if their textual formulation differs.

#### BLEU

BLEU measures token-level n-gram overlap between generated and reference SQL.

#### Execution Accuracy

Generated and reference SQL queries are executed against schema-seeded in-memory SQLite databases.

```text
execution(prediction) == execution(reference)
```

This provides a measure of semantic equivalence that is less sensitive to different SQL formulations.

## Results

| Metric                | Base Qwen2.5-3B |    Distill |
| --------------------- | --------------: | ---------: |
| Exact Match Accuracy  |          2.33%* | **30.67%** |
| BLEU Score            |         51.87%* | **59.74%** |
| Execution Accuracy    |               — | **55.03%** |
| Validation Perplexity |               — |   **1.98** |

* Base-model measurements were obtained on a 300-example subset.

The repository reports a **13× improvement in exact-match accuracy** compared with its measured base-model result.

### Result Interpretation

The difference between exact-match accuracy and execution accuracy is important:

```text
Exact Match Accuracy: 30.67%
Execution Accuracy:   55.03%
```

This indicates that some generated SQL queries differ from the reference query text while still producing equivalent results under the evaluation setup.

Therefore, exact match alone should not be considered a complete measure of SQL generation quality.

## Base Model Comparison

The repository reports the following comparison:

```text
Qwen2.5-3B-Instruct:
    Exact Match: 2.33%
    BLEU:        51.87%

Distill:
    Exact Match: 30.67%
    BLEU:        59.74%
```

The repository also notes that the base model can return SQL together with Markdown code blocks or explanatory text. SQL extraction methodology therefore has a significant effect on the measured baseline.

## How to Use

This repository contains a PEFT adapter. The compatible Qwen2.5-3B-Instruct base model is required.

### Installation

```bash
pip install torch transformers peft bitsandbytes accelerate
```

### Load the Adapter

```python
import torch
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
)
from peft import PeftModel

base_model = "Qwen/Qwen2.5-3B-Instruct"
adapter_path = "path/to/adapter"

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.float16,
)

tokenizer = AutoTokenizer.from_pretrained(base_model)

model = AutoModelForCausalLM.from_pretrained(
    base_model,
    quantization_config=quantization_config,
    device_map="auto",
)

model = PeftModel.from_pretrained(
    model,
    adapter_path,
)
```

### Generate SQL

```python
schema = """
CREATE TABLE orders (
    id INTEGER,
    customer VARCHAR,
    amount FLOAT,
    status VARCHAR
);
"""

question = "What is the total revenue from completed orders?"

prompt = f"""
Schema:
{schema}

Question:
{question}

SQL:
"""

inputs = tokenizer(
    prompt,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=128,
)

response = tokenizer.decode(
    outputs[0],
    skip_special_tokens=True,
)

print(response)
```

## Repository

The complete training and application code is available in the project repository:

https://github.com/MahboobAlam0/Distill---Fine-tuned-Qwen2.5-3B

The repository includes:

* QLoRA training code
* Evaluation scripts
* Inference wrapper
* FastAPI API
* Streamlit interface
* Kaggle training notebook
* Training and validation data
* Docker configuration
* Tests

## Serving

The project includes a FastAPI REST API and Streamlit interface.

### FastAPI

```bash
uvicorn api.main:app --reload --port 8000
```

Example request:

```bash
curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d '{
    "schema": "CREATE TABLE products (id INTEGER, name VARCHAR, price FLOAT, stock INTEGER)",
    "question": "Which products are out of stock?"
  }'
```

### Streamlit

```bash
streamlit run app/streamlit_app.py
```

### Docker

```bash
docker-compose up
```

The repository exposes the API on port `8000` and the Streamlit interface on port `8501`.

## Reproducing Training

The repository includes a self-contained Kaggle notebook.

```text
1. Upload kaggle/train_notebook.ipynb to Kaggle.
2. Select an NVIDIA T4 GPU.
3. Run the notebook.
4. Export the resulting LoRA adapter.
```

The repository reports approximately **2 hours** for the documented training run on a single Kaggle T4 GPU.

## Technical Specifications

### Hardware

* 1 × NVIDIA T4
* Approximately 15 GB VRAM

### Software

* Python 3.12
* PyTorch
* Transformers
* PEFT
* BitsAndBytes
* TRL
* NLTK
* SQLite
* FastAPI
* Uvicorn
* Streamlit
* Docker

### Adapter

* LoRA rank: 16
* LoRA alpha: 32
* Approximately 21M trainable parameters
* Approximately 114 MB adapter size

## Environmental Impact

Training was performed on a single NVIDIA T4 GPU on Kaggle.

The project does not provide a measured carbon-emissions value.

| Property         | Value        |
| ---------------- | ------------ |
| Hardware         | NVIDIA T4    |
| GPU count        | 1            |
| Training time    | ~2 hours     |
| Platform         | Kaggle       |
| Compute region   | Not reported |
| Carbon emissions | Not measured |

## License

The GitHub project is released under the **MIT License**.

However, this model is a fine-tuned adapter based on `Qwen/Qwen2.5-3B-Instruct`. The upstream Qwen model is listed under the **Qwen Research License Agreement**, so the upstream model's license and terms must also be respected.

For the complete upstream license terms, see:

https://huggingface.co/Qwen/Qwen2.5-3B-Instruct

For the project's MIT license, see the GitHub repository:

https://github.com/MahboobAlam0/Distill---Fine-tuned-Qwen2.5-3B

## Citation

If you use this model or the associated project, please cite:

```bibtex
@software{mahboobalam_distill_qwen25_3b,
  author = {MahboobAlam0},
  title = {Distill - Fine-tuned Qwen2.5-3B},
  url = {https://github.com/MahboobAlam0/Distill---Fine-tuned-Qwen2.5-3B},
  license = {MIT}
}
```

## Acknowledgements

This project builds upon:

* Qwen2.5-3B-Instruct
* Hugging Face Transformers
* PEFT
* TRL
* BitsAndBytes
* PyTorch
* `sql-create-context`
* SQLite

## Model Card Contact

For questions, issues, or contributions, please use the GitHub repository's issue tracker.

**Project:** Distill — Fine-tuned Qwen2.5-3B
**Task:** Natural language → SQL
**Base model:** Qwen2.5-3B-Instruct
**Fine-tuning:** QLoRA
**Training data:** 28,000 examples
**Validation data:** 7,858 examples
**Exact Match:** 30.67%
**Execution Accuracy:** 55.03%
**Adapter size:** ~114 MB
**Hardware:** 1 × NVIDIA T4