nl2sql / evaluation.md
JayaHareeswarReddy's picture
Update evaluation.md
20ffa51 verified
|
Raw History Blame Contribute Delete
1.68 kB
# NL2SQL Model Evaluation
## 1. Objective
Evaluate the performance improvement achieved by fine-tuning Qwen2.5-Coder-1.5B-Instruct using QLoRA for Natural Language to SQL generation.
## 2. Models Compared
- Base Model: Qwen2.5-Coder-1.5B-Instruct
- Fine-tuned Model: NL2SQL-QLoRA
## 3. Fine-tuning Methodology
- Fine-tuning technique: QLoRA
- Quantization: 4-bit
- LoRA Rank: 16
- LoRA Alpha: 16
- Training Epochs: 1
- Training Dataset: Gretel AI Synthetic Text-to-SQL
- Selected Training Samples: 12,000
## 4. Evaluation Metrics
**Exact Match**
Measures whether the generated SQL query matches the reference SQL query.
**Execution Match**
Measures whether the generated SQL produces the expected execution result.
## 5. Performance Comparison
| Metric | Base Model | Fine-tuned Model |
|---|---:|---:|
| Exact Match | 23.5% | 31.0% |
| Execution Match | 66.7% | 71.5% |
## 6. Key Findings
- Exact Match improved by 7.5 percentage points.
- Execution Match improved by 4.8 percentage points.
- Fine-tuning improved the model's SQL generation performance on the evaluated examples.
## 7. Evaluation Scope
- Evaluation examples: 200
- Executable examples reported: 165
- Training duration: 23 minutes and 39 seconds.
## 8. Limitations
- Evaluation was performed on a relatively small sample.
- Training used a selected synthetic dataset subset.
- Performance on unseen real-world database schemas requires further validation.
## 9. Future Improvements
- Expand evaluation to larger datasets.
- Improve complex SQL query generation.
- Experiment with different LoRA configurations.
- Evaluate on real-world database schemas.
- Improve query execution accuracy.