Spaces:
Running on Zero
Running on Zero
|
Download evaluation.md from JayaHareeswarReddy/nl2sql: direct link, hf CLI and curl.
- Browser
- Download file 1.68 kB
-
https://huggingface.co/spaces/JayaHareeswarReddy/nl2sql/resolve/main/evaluation.md
- Command line
-
hf download hf://spaces/JayaHareeswarReddy/nl2sql/evaluation.md
-
curl -L -o evaluation.md https://huggingface.co/spaces/JayaHareeswarReddy/nl2sql/resolve/main/evaluation.md
1.68 kB
| # NL2SQL Model Evaluation | |
| ## 1. Objective | |
| Evaluate the performance improvement achieved by fine-tuning Qwen2.5-Coder-1.5B-Instruct using QLoRA for Natural Language to SQL generation. | |
| ## 2. Models Compared | |
| - Base Model: Qwen2.5-Coder-1.5B-Instruct | |
| - Fine-tuned Model: NL2SQL-QLoRA | |
| ## 3. Fine-tuning Methodology | |
| - Fine-tuning technique: QLoRA | |
| - Quantization: 4-bit | |
| - LoRA Rank: 16 | |
| - LoRA Alpha: 16 | |
| - Training Epochs: 1 | |
| - Training Dataset: Gretel AI Synthetic Text-to-SQL | |
| - Selected Training Samples: 12,000 | |
| ## 4. Evaluation Metrics | |
| **Exact Match** | |
| Measures whether the generated SQL query matches the reference SQL query. | |
| **Execution Match** | |
| Measures whether the generated SQL produces the expected execution result. | |
| ## 5. Performance Comparison | |
| | Metric | Base Model | Fine-tuned Model | | |
| |---|---:|---:| | |
| | Exact Match | 23.5% | 31.0% | | |
| | Execution Match | 66.7% | 71.5% | | |
| ## 6. Key Findings | |
| - Exact Match improved by 7.5 percentage points. | |
| - Execution Match improved by 4.8 percentage points. | |
| - Fine-tuning improved the model's SQL generation performance on the evaluated examples. | |
| ## 7. Evaluation Scope | |
| - Evaluation examples: 200 | |
| - Executable examples reported: 165 | |
| - Training duration: 23 minutes and 39 seconds. | |
| ## 8. Limitations | |
| - Evaluation was performed on a relatively small sample. | |
| - Training used a selected synthetic dataset subset. | |
| - Performance on unseen real-world database schemas requires further validation. | |
| ## 9. Future Improvements | |
| - Expand evaluation to larger datasets. | |
| - Improve complex SQL query generation. | |
| - Experiment with different LoRA configurations. | |
| - Evaluate on real-world database schemas. | |
| - Improve query execution accuracy. |