# NL2SQL Model Evaluation ## 1. Objective Evaluate the performance improvement achieved by fine-tuning Qwen2.5-Coder-1.5B-Instruct using QLoRA for Natural Language to SQL generation. ## 2. Models Compared - Base Model: Qwen2.5-Coder-1.5B-Instruct - Fine-tuned Model: NL2SQL-QLoRA ## 3. Fine-tuning Methodology - Fine-tuning technique: QLoRA - Quantization: 4-bit - LoRA Rank: 16 - LoRA Alpha: 16 - Training Epochs: 1 - Training Dataset: Gretel AI Synthetic Text-to-SQL - Selected Training Samples: 12,000 ## 4. Evaluation Metrics **Exact Match** Measures whether the generated SQL query matches the reference SQL query. **Execution Match** Measures whether the generated SQL produces the expected execution result. ## 5. Performance Comparison | Metric | Base Model | Fine-tuned Model | |---|---:|---:| | Exact Match | 23.5% | 31.0% | | Execution Match | 66.7% | 71.5% | ## 6. Key Findings - Exact Match improved by 7.5 percentage points. - Execution Match improved by 4.8 percentage points. - Fine-tuning improved the model's SQL generation performance on the evaluated examples. ## 7. Evaluation Scope - Evaluation examples: 200 - Executable examples reported: 165 - Training duration: 23 minutes and 39 seconds. ## 8. Limitations - Evaluation was performed on a relatively small sample. - Training used a selected synthetic dataset subset. - Performance on unseen real-world database schemas requires further validation. ## 9. Future Improvements - Expand evaluation to larger datasets. - Improve complex SQL query generation. - Experiment with different LoRA configurations. - Evaluate on real-world database schemas. - Improve query execution accuracy.