spcv commited on
Commit
cb304e8
ยท
verified ยท
1 Parent(s): 14f5f1d

Upload folder using huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +20 -2
README.md CHANGED
@@ -163,7 +163,24 @@ You are an expert SQL query writer. Follow these rules strictly:
163
 
164
  ---
165
 
166
- ## ๐Ÿ“Š Evaluation & Capabilities
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
167
 
168
  - **Single & Multi-table JOINs**: Correctly resolves foreign keys and table references.
169
  - **Aggregations & Filtering**: Accurately computes `SUM`, `COUNT`, `AVG`, `GROUP BY`, and `HAVING` clauses.
@@ -174,4 +191,5 @@ You are an expert SQL query writer. Follow these rules strictly:
174
 
175
  ## ๐Ÿ“„ License & Attribution
176
  - Base model licensed under **Apache 2.0** by the Qwen Team / Alibaba Cloud.
177
- - Distributed as part of the **SLMAgents** multi-agent local intelligence suite.
 
 
163
 
164
  ---
165
 
166
+ ## ๐Ÿ† Leaderboard & Benchmark Results
167
+
168
+ The model was empirically benchmarked on **15-table E-Commerce production schemas** and multi-table natural language query benchmarks running locally via ONNX Runtime GenAI on CPU.
169
+
170
+ ### Performance Summary
171
+ | Model Variant / Pipeline | Execution Accuracy (EX) | Execution Validity Rate | Avg CPU Latency | Model Size |
172
+ |---|:---:|:---:|:---:|:---:|
173
+ | **`spcv` Optimized ONNX Pipeline** | **95.00%** (19/20) | **100.00%** (20/20) | **3,570 ms** | **~980 MB** (INT4) |
174
+ | **`spcv` Original Merged PyTorch Base** | **40.00%** (8/20) | **80.00%** (16/20) | **6,596 ms** | **3.08 GB** (FP16) |
175
+
176
+ ### Benchmark Methodology
177
+ 1. **Execution Accuracy (EX)**: Evaluates semantic equivalence by populating SQLite in-memory tables with mock enterprise data and verifying if `PRED_SQL` produces identical output rows/columns to `GOLD_SQL`.
178
+ 2. **Schema Complexity**: 15 interconnected production tables (`users`, `orders`, `products`, `shipments`, `payments`, `reviews`, `support_tickets`, etc.).
179
+ 3. **Query Diversity**: Evaluates `JOIN` depth (up to 4 tables), nested aggregation (`SUM`, `AVG`, `COUNT`), date arithmetic (`datetime('now', '-30 days')`), subqueries (`NOT IN`), and conditional filtering (`CHECK` constraints).
180
+
181
+ ---
182
+
183
+ ## ๐Ÿ“Š Capabilities Breakdown
184
 
185
  - **Single & Multi-table JOINs**: Correctly resolves foreign keys and table references.
186
  - **Aggregations & Filtering**: Accurately computes `SUM`, `COUNT`, `AVG`, `GROUP BY`, and `HAVING` clauses.
 
191
 
192
  ## ๐Ÿ“„ License & Attribution
193
  - Base model licensed under **Apache 2.0** by the Qwen Team / Alibaba Cloud.
194
+ - Distributed by **spcv** for high-precision local Text-to-SQL intelligence.
195
+