Text Generation
PEFT
Safetensors
lora
sft
text-to-sql
trl
conversational
albertkingdom commited on
Commit
b0293c3
·
verified ·
1 Parent(s): c0c29e8

docs: drop underpowered n=20 in-distribution table, rely on Spider for SQL capability

Browse files
Files changed (1) hide show
  1. README.md +1 -15
README.md CHANGED
@@ -70,21 +70,7 @@ The two datasets are nearly the same size (~70.7k vs ~75.2k), so a 50/50 ratio c
70
  | Official Execution Accuracy | 39.9% | 50.4% | **60.8%** |
71
  | Official Exact Match (structural) | 32.1% | 37.4% | **47.3%** |
72
 
73
- Compared with the pure-SQL version, this version generalizes substantially better to unseen schemas (+10.4pp execution accuracy) and retains more code capability, at no cost to in-distribution SQL accuracy (see below). Mixing in code data appears to act as a regularizer against overfitting to the narrow single-domain SQL distribution.
74
-
75
- ### In-distribution SQL accuracy (b-mc2/sql-create-context, same distribution as training)
76
-
77
- Measured on the same 20-sample held-out set for all three models, in the same evaluation run:
78
-
79
- | Model | Exact Match (n=20) |
80
- |------|-------------|
81
- | Base | 0.0% |
82
- | Pure-SQL SFT | 70.0% |
83
- | This version (mixed) | 70.0% |
84
-
85
- The two fine-tuned versions are **tied** here — mixing in code data cost nothing on the training distribution. Note that n=20 gives a wide confidence interval; the Spider result above (n=1034, real databases, official scripts) is the more reliable measurement.
86
-
87
- Earlier numbers on this dataset used different splits and are superseded by the table above. Note that `sql-create-context` ships schemas with no data rows, so execution-based metrics on it mostly measure syntax/schema alignment rather than semantic correctness.
88
 
89
  ## LoRA configuration
90
 
 
70
  | Official Execution Accuracy | 39.9% | 50.4% | **60.8%** |
71
  | Official Exact Match (structural) | 32.1% | 37.4% | **47.3%** |
72
 
73
+ Compared with the pure-SQL version, this version generalizes substantially better to unseen schemas (+10.4pp execution accuracy) while also retaining more code capability. Mixing in code data appears to act as a regularizer against overfitting to the narrow single-domain SQL distribution.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
74
 
75
  ## LoRA configuration
76