JayaHareeswarReddy commited on
Commit
09091ee
·
verified ·
1 Parent(s): 649a6f3

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +24 -207
README.md CHANGED
@@ -1,215 +1,32 @@
1
  ---
2
- language:
3
- - en
4
- pipeline_tag: text2text-generation
5
- library_name: peft
6
- base_model: unsloth/Qwen2.5-Coder-1.5B-Instruct
7
- tags:
8
- - text-to-sql
9
- - qlora
10
- - fine-tuning
11
- - unsloth
12
- - qwen2.5
13
- - natural-language-processing
14
- - sql-generation
15
  ---
16
 
17
- # NL2SQL-QLoRA
18
 
19
- A fine-tuned Qwen2.5-Coder-1.5B-Instruct model that translates natural-language questions and database schemas into SQL queries.
20
 
21
- This project uses **4-bit QLoRA fine-tuning with Unsloth** on the Gretel AI Synthetic Text-to-SQL dataset.
 
 
 
 
22
 
23
- ## Project Overview
 
24
 
25
- The goal is to convert natural-language questions into SQL queries using database schema information.
 
 
 
 
 
26
 
27
- **Input:**
28
- - Database schema (CREATE TABLE statements)
29
- - Natural-language question
30
-
31
- **Output:**
32
- - SQL query generated by the fine-tuned model.
33
-
34
- ## Model Details
35
-
36
- | Component | Configuration |
37
- |---|---|
38
- | Base model | Qwen2.5-Coder-1.5B-Instruct |
39
- | Fine-tuning method | QLoRA |
40
- | Quantization | 4-bit |
41
- | LoRA rank | 16 |
42
- | LoRA alpha | 16 |
43
- | LoRA dropout | 0.0 |
44
- | Maximum sequence length | 1024 |
45
- | Trainable parameters | 18.46M |
46
- | Trainable parameter ratio | 1.18% |
47
- | Framework | Unsloth, Transformers, TRL, PEFT |
48
-
49
- ## Dataset
50
-
51
- **Dataset:** [Gretel AI Synthetic Text-to-SQL](https://huggingface.co/datasets/gretelai/synthetic_text_to_sql)
52
-
53
- - Original training split: 100,000 rows.
54
- - Selected training subset: 12,000 rows.
55
- - Shuffled using seed 3407.
56
- - Input contains database schema and natural-language questions.
57
- - Target contains the corresponding SQL query.
58
-
59
- The notebook reports 2,238 packed training examples after preprocessing and packing.
60
-
61
- ## Training Configuration
62
-
63
- | Parameter | Value |
64
- |---|---|
65
- | Epochs | 1 |
66
- | Learning rate | 2e-4 |
67
- | Per-device batch size | 2 |
68
- | Gradient accumulation | 4 |
69
- | Effective batch size | 8 |
70
- | Warmup steps | 5 |
71
- | Optimizer | AdamW 8-bit |
72
- | Weight decay | 0.01 |
73
- | Scheduler | Linear |
74
- | Random seed | 3407 |
75
- | GPU | NVIDIA Tesla T4 |
76
-
77
- Training was performed using Unsloth with gradient checkpointing and 4-bit base-model loading.
78
-
79
- ## Evaluation Results
80
-
81
- The model was evaluated against its original base model using 200 examples.
82
-
83
- | Metric | Base Model | Fine-tuned Model |
84
- |---|---:|---:|
85
- | Exact Match | 23.5% | 31.0% |
86
- | Execution Match | 66.7% | 71.5% |
87
-
88
- **Performance improvement:**
89
- - Exact Match: +7.5 percentage points.
90
- - Execution Match: +4.8 percentage points.
91
-
92
- The evaluation logs report 165 executable examples.
93
-
94
- These results are from the notebook's evaluation run and should not be interpreted as a comprehensive benchmark.
95
-
96
- ## Example
97
-
98
- **Database Schema**
99
-
100
- ```sql
101
- CREATE TABLE employees (
102
- id INT,
103
- name TEXT,
104
- salary INT,
105
- dept TEXT
106
- );
107
- ```
108
-
109
- **Natural Language Question**
110
-
111
- List the names of employees in the Sales department earning over 50000.
112
-
113
- **Generated SQL**
114
-
115
- ```sql
116
- SELECT name
117
- FROM employees
118
- WHERE dept = 'Sales'
119
- AND salary > 50000;
120
- ```
121
-
122
- This example was generated during the notebook's inference smoke test.
123
-
124
- ## Usage
125
-
126
- Install dependencies:
127
-
128
- ```bash
129
- pip install torch transformers peft accelerate
130
- ```
131
-
132
- Load the fine-tuned adapter:
133
-
134
- ```python
135
- from transformers import AutoTokenizer
136
- from peft import AutoPeftModelForCausalLM
137
- import torch
138
-
139
- model_id = "JayaHareeswarReddy/nl2sql-qlora"
140
-
141
- tokenizer = AutoTokenizer.from_pretrained(model_id)
142
-
143
- model = AutoPeftModelForCausalLM.from_pretrained(
144
- model_id,
145
- device_map="auto",
146
- torch_dtype="auto"
147
- )
148
-
149
- system_prompt = (
150
- "You are a text-to-SQL assistant. Given a database "
151
- "schema and a question, respond with a single correct "
152
- "SQL query and nothing else."
153
- )
154
-
155
- schema = "CREATE TABLE employees (id INT, name TEXT, salary INT, dept TEXT);"
156
-
157
- question = "List employees earning over 50000."
158
-
159
- messages = [
160
- {"role": "system", "content": system_prompt},
161
- {
162
- "role": "user",
163
- "content": f"Schema:\n{schema}\n\nQuestion: {question}"
164
- }
165
- ]
166
-
167
- prompt = tokenizer.apply_chat_template(
168
- messages,
169
- tokenize=False,
170
- add_generation_prompt=True
171
- )
172
-
173
- inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
174
-
175
- outputs = model.generate(
176
- **inputs,
177
- max_new_tokens=128,
178
- do_sample=False
179
- )
180
-
181
- print(tokenizer.decode(
182
- outputs[0][inputs["input_ids"].shape[1]:],
183
- skip_special_tokens=True
184
- ))
185
- ```
186
-
187
- ## Limitations
188
-
189
- - Performance depends on the database schema and question complexity.
190
- - Generated SQL should be validated before execution.
191
- - The evaluation covers only 200 examples.
192
- - The model has been trained on a selected synthetic dataset subset.
193
- - Performance on unseen real-world databases may differ.
194
-
195
- ## Future Improvements
196
-
197
- - Expand the training dataset.
198
- - Experiment with different LoRA configurations.
199
- - Evaluate on larger and more diverse benchmarks.
200
- - Improve complex SQL query generation.
201
- - Compare performance across different base models.
202
- - Integrate the model into a production-ready Text-to-SQL application.
203
-
204
- ## Acknowledgements
205
-
206
- - Qwen team for Qwen2.5-Coder.
207
- - Unsloth for efficient fine-tuning.
208
- - Gretel AI for the Synthetic Text-to-SQL dataset.
209
- - Hugging Face for model hosting and distribution.
210
-
211
- ---
212
-
213
- **Author:** Jaya Hareeswar Reddy Gudisa
214
-
215
- **Model:** [NL2SQL-QLoRA](https://huggingface.co/JayaHareeswarReddy/nl2sql-qlora)
 
1
  ---
2
+ title: NL2SQL - Natural Language to SQL
3
+ emoji: 🧠
4
+ colorFrom: blue
5
+ colorTo: purple
6
+ sdk: gradio
7
+ app_file: app.py
8
+ pinned: true
 
 
 
 
 
 
9
  ---
10
 
11
+ # NL2SQL – Natural Language to SQL
12
 
13
+ An AI-powered application that converts natural-language questions into SQL queries using a fine-tuned Qwen2.5-Coder-1.5B-Instruct model.
14
 
15
+ ## Features
16
+ - Natural-language to SQL generation
17
+ - Fine-tuned Qwen2.5-Coder model
18
+ - QLoRA-based fine-tuning
19
+ - Interactive Gradio interface
20
 
21
+ ## Model
22
+ [JayaHareeswarReddy/nl2sql-qlora](https://huggingface.co/JayaHareeswarReddy/nl2sql-qlora)
23
 
24
+ ## Technologies
25
+ - Python
26
+ - Hugging Face Transformers
27
+ - Qwen2.5-Coder
28
+ - PEFT / QLoRA
29
+ - Gradio
30
 
31
+ ## Author
32
+ Jaya Hareeswar Reddy Gudisa