---
base_model: Qwen/Qwen2.5-Coder-3B-Instruct
library_name: peft
datasets:
- b-mc2/sql-create-context
tags:
- text-to-sql
- qlora
- lora
---

# qwen2.5-coder-3b-text2sql-qlora

QLoRA adapter for `Qwen/Qwen2.5-Coder-3B-Instruct`, fine-tuned to turn a table schema and a question into a SQL query.

Code and full evaluation: https://github.com/VarshithaChappidi/text2sql-qlora

## Results (300 held-out examples)
| Model | Strict exact match | Normalized exact match |
|---|---|---|
| Base, prompted | 0.023 | 0.570 |
| This adapter | 0.767 | 0.773 |

Normalized match unifies quote style, drops wrapping quotes, treats `INNER JOIN` as `JOIN`, and ignores `AND` order. It still rejects some valid SQL written differently, so it is a lower bound.

## Usage
```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base_id = "Qwen/Qwen2.5-Coder-3B-Instruct"
tok = AutoTokenizer.from_pretrained(base_id)
base = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype=torch.float16, device_map="auto")
model = PeftModel.from_pretrained(base, "Varshitha2603/qwen2.5-coder-3b-text2sql-qlora")

schema = "CREATE TABLE singer (name VARCHAR, age INT)"
question = "How many singers are older than 30?"
msgs = [
    {"role": "system", "content": "You are a SQL expert. Given a table schema and a question, reply with only the SQL query."},
    {"role": "user", "content": f"Schema:\n{schema}\n\nQuestion: {question}"},
]
prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tok.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
```

## Training
- Data: 5,000 examples from `b-mc2/sql-create-context` (300 other examples held out for testing)
- QLoRA: 4-bit NF4, rank 16, alpha 32, dropout 0.05, all attention and MLP projections (29.9M trainable parameters)
- 1 epoch, learning rate 2e-4 with cosine schedule, effective batch size 16, loss on answer tokens only
- Free Colab T4 GPU, about 21 minutes. Loss fell from 0.212 (step 25) to 0.046 (step 300).

## Limitations
- Evaluated by exact match, not execution, because the dataset has no table contents.
- Trained and tested only on single-table-style questions from one dataset, plus a few Spider-style joins. Multi-table joins are a known weak spot.
- Gold labels contain ambiguity and typos. Single training run, no variance estimate.
- The adapter inherits the license terms of the base model and the dataset. Check both before commercial use.