nl2sql / evaluation.md
JayaHareeswarReddy's picture
Update evaluation.md
20ffa51 verified
|
Raw History Blame Contribute Delete
1.68 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade

NL2SQL Model Evaluation

1. Objective

Evaluate the performance improvement achieved by fine-tuning Qwen2.5-Coder-1.5B-Instruct using QLoRA for Natural Language to SQL generation.

2. Models Compared

  • Base Model: Qwen2.5-Coder-1.5B-Instruct
  • Fine-tuned Model: NL2SQL-QLoRA

3. Fine-tuning Methodology

  • Fine-tuning technique: QLoRA
  • Quantization: 4-bit
  • LoRA Rank: 16
  • LoRA Alpha: 16
  • Training Epochs: 1
  • Training Dataset: Gretel AI Synthetic Text-to-SQL
  • Selected Training Samples: 12,000

4. Evaluation Metrics

Exact Match

Measures whether the generated SQL query matches the reference SQL query.

Execution Match

Measures whether the generated SQL produces the expected execution result.

5. Performance Comparison

Metric Base Model Fine-tuned Model
Exact Match 23.5% 31.0%
Execution Match 66.7% 71.5%

6. Key Findings

  • Exact Match improved by 7.5 percentage points.
  • Execution Match improved by 4.8 percentage points.
  • Fine-tuning improved the model's SQL generation performance on the evaluated examples.

7. Evaluation Scope

  • Evaluation examples: 200
  • Executable examples reported: 165
  • Training duration: 23 minutes and 39 seconds.

8. Limitations

  • Evaluation was performed on a relatively small sample.
  • Training used a selected synthetic dataset subset.
  • Performance on unseen real-world database schemas requires further validation.

9. Future Improvements

  • Expand evaluation to larger datasets.
  • Improve complex SQL query generation.
  • Experiment with different LoRA configurations.
  • Evaluate on real-world database schemas.
  • Improve query execution accuracy.