README.md

---
base_model: deepseek-ai/deepseek-coder-7b-instruct-v1.5
library_name: peft
model_name: deepseek-coder-7b-text2sql-magicoder-lora
tags:
- base_model:adapter:deepseek-ai/deepseek-coder-7b-instruct-v1.5
- lora
- sft
- text-to-sql
- trl
license: other
license_name: deepseek
license_link: https://github.com/deepseek-ai/DeepSeek-Coder/blob/main/LICENSE-MODEL
pipeline_tag: text-generation
datasets:
- b-mc2/sql-create-context
- ise-uiuc/Magicoder-OSS-Instruct-75K
---

# Model Card for deepseek-coder-7b-text2sql-magicoder-lora

LoRA adapter fine-tuned from [deepseek-ai/deepseek-coder-7b-instruct-v1.5](https://huggingface.co/deepseek-ai/deepseek-coder-7b-instruct-v1.5) for text-to-SQL generation, trained with [TRL](https://github.com/huggingface/trl) SFTTrainer.

## Intended Use & Limitations

**Intended use**: generating a single SQL query from a natural-language question given a `CREATE TABLE` schema, in English, for single or lightly-joined relational databases similar in style to Spider / sql-create-context schemas.

**Limitations**:
- Not evaluated on production/adversarial inputs, non-English questions, or dialects outside SQLite-compatible syntax.
- Execution accuracy on Spider (60.8%) means roughly 2 in 5 generated queries on unseen schemas are still wrong — **always validate generated SQL before running it against a real database**, especially for destructive statements (this adapter was only trained/evaluated on read (`SELECT`) queries).
- **JOIN over-generation**: the SQL training data is almost entirely single-table, so on multi-table schemas the model tends to join every available table indiscriminately. This is the main source of exact-match errors on Spider.
- Code capability (HumanEval+) is retained better than the pure-SQL version but still ~4pp below the un-finetuned base model — for general-purpose coding tasks unrelated to SQL, the base model remains the stronger choice. See Results below.

## What's on `main` vs `pure-sql`

- **`main` (this version)** — trained on a 50/50 mix of SQL and general code-instruction data. Better SQL generalization to unseen schemas *and* better retention of general code ability than the pure-SQL version.
- **`pure-sql`** branch — the original version trained on 100% SQL data. Comparable in-distribution SQL accuracy, but noticeably worse code capability retention and worse generalization to unseen database schemas.

```python
# to load the pure-SQL version instead:
PeftModel.from_pretrained(model, adapter_id, revision="pure-sql")
```

## Why mix in code data

The pure-SQL version showed catastrophic forgetting of general code generation ability:

| Metric | Base | Pure-SQL SFT | Δ |
|---|---|---|---|
| HumanEval pass@1 | 52.0% | 40.0% | -12.0pp |
| HumanEval+ (999 edge cases) | 46.0% | 34.0% | -12.0pp |

Training data was rebalanced to 50% [b-mc2/sql-create-context](https://huggingface.co/datasets/b-mc2/sql-create-context) + 50% [ise-uiuc/Magicoder-OSS-Instruct-75K](https://huggingface.co/datasets/ise-uiuc/Magicoder-OSS-Instruct-75K) (interleaved batch-wise via `datasets.interleave_datasets`, `stopping_strategy="first_exhausted"`), 1 epoch.

The two datasets are nearly the same size (~70.7k vs ~75.2k), so a 50/50 ratio consumes essentially all of both — 141,771 training examples, 8,861 optimizer steps.

## Results

### Code capability retention (n=50, HumanEval/HumanEval+)

| Metric | Base | Pure-SQL SFT | **This version (mixed)** |
|---|---|---|---|
| HumanEval pass@1 | 52.0% | 40.0% (-12.0pp) | **44.0% (-8.0pp)** |
| HumanEval+ (plus) | 46.0% | 34.0% (-12.0pp) | **42.0% (-4.0pp)** |

### SQL generalization on unseen schemas (Spider 1.0, n=1034, real databases + official eval)

| Metric | Base | Pure-SQL SFT | **This version (mixed)** |
|---|---|---|---|
| Official Execution Accuracy | 39.9% | 50.4% | **60.8%** |
| Official Exact Match (structural) | 32.1% | 37.4% | **47.3%** |

Compared with the pure-SQL version, this version generalizes substantially better to unseen schemas (+10.4pp execution accuracy) while also retaining more code capability. Mixing in code data appears to act as a regularizer against overfitting to the narrow single-domain SQL distribution.

## LoRA configuration

```
r: 16
lora_alpha: 32
lora_dropout: 0.05
target_modules: [q_proj, k_proj, v_proj, o_proj]
quantization: 4-bit NF4 (QLoRA), bf16 compute
```

## Quick start

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "deepseek-ai/deepseek-coder-7b-instruct-v1.5"
adapter_id = "albertkingdom/deepseek-coder-7b-text2sql-magicoder-lora"

model = AutoModelForCausalLM.from_pretrained(
    base_model_id, torch_dtype=torch.bfloat16, device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
model = PeftModel.from_pretrained(model, adapter_id)  # main = mixed training version

messages = [{
    "role": "user",
    "content": """Given the database schema below, write a SQL query that answers the user's question.
Only output the SQL query. Do not add any explanation.

### Schema
CREATE TABLE users (id INT, name VARCHAR(100), email VARCHAR(100))

### Question
Find all users with gmail addresses"""
}]

inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to(model.device)
outputs = model.generate(inputs, max_new_tokens=200, do_sample=False)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))
```

The adapter was trained with exactly this instruction + `### Schema` / `### Question` layout wrapped in the DeepSeek chat template. Free-form prompts still work, but accuracy drops.

## Training procedure

Trained with SFT (TRL `SFTTrainer`) on an interleaved SQL + code instruction dataset, 1 epoch over 141,771 examples (8,861 steps), learning rate 1e-4 with linear decay and 3% warmup, effective batch size 16 (per-device 4 × gradient accumulation 4), max sequence length 1024, `paged_adamw_8bit`, bf16, 4-bit QLoRA, on a rented RTX 5090 (~12h). Final train loss 0.3986, eval loss 0.3437.

### Framework versions

- PEFT: 0.18.0
- TRL: 0.26.2
- Transformers: 4.57.3
- Datasets: 4.4.2
- PyTorch: 2.9.1+cu128

## License

This adapter is a derivative of [deepseek-ai/deepseek-coder-7b-instruct-v1.5](https://huggingface.co/deepseek-ai/deepseek-coder-7b-instruct-v1.5), which is released under the [DeepSeek Model License](https://github.com/deepseek-ai/DeepSeek-Coder/blob/main/LICENSE-MODEL) rather than a standard open-source license. Per that license, derivative models must carry forward at least the same use-based restrictions, so this adapter — and any model merged/derived from it — inherits them:

- No use for military purposes.
- No use that harms minors.
- No generation of false information intended to harm others.
- No creation of non-consensual personal identifiable information.
- No fully automated decision-making that adversely affects an individual's legal rights.
- No discrimination based on protected characteristics.
- See the [full license text](https://github.com/deepseek-ai/DeepSeek-Coder/blob/main/LICENSE-MODEL) (Attachment A) for the complete list.

Commercial use is otherwise permitted, consistent with the base model's license.

## Credits & Data Provenance

- **Base model**: [deepseek-ai/deepseek-coder-7b-instruct-v1.5](https://huggingface.co/deepseek-ai/deepseek-coder-7b-instruct-v1.5) (DeepSeek Model License)
- **[b-mc2/sql-create-context](https://huggingface.co/datasets/b-mc2/sql-create-context)** (CC-BY-4.0) — itself derived from [WikiSQL](https://github.com/salesforce/WikiSQL) and [Spider](https://yale-lily.github.io/spider); credit to both original sources per CC-BY-4.0 attribution terms.
- **[ise-uiuc/Magicoder-OSS-Instruct-75K](https://huggingface.co/datasets/ise-uiuc/Magicoder-OSS-Instruct-75K)** (MIT) — generated via the OSS-Instruct method using `gpt-3.5-turbo-1106`. Outputs are subject to [OpenAI's usage policies](https://openai.com/policies/usage-policies) in addition to the dataset's own MIT license.

## Citations

```bibtex
@misc{vonwerra2022trl,
	title        = {{TRL: Transformer Reinforcement Learning}},
	author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
	year         = 2020,
	journal      = {GitHub repository},
	publisher    = {GitHub},
	howpublished = {\url{https://github.com/huggingface/trl}}
}
```

```bibtex
@article{yu2018spider,
	title   = {Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task},
	author  = {Yu, Tao and Zhang, Rui and Yang, Kai and Yasunaga, Michihiro and Wang, Dongxu and Li, Zifan and Ma, James and Li, Irene and Yao, Qingning and Roman, Shanelle and Zhang, Zilin and Radev, Dragomir},
	journal = {arXiv preprint arXiv:1809.08887},
	year    = 2018
}
```

```bibtex
@article{wei2023magicoder,
	title   = {Magicoder: Empowering Code Generation with OSS-Instruct},
	author  = {Wei, Yuxiang and Wang, Zhe and Liu, Jiawei and Ding, Yifeng and Zhang, Lingming},
	journal = {arXiv preprint arXiv:2312.02120},
	year    = 2023
}
```