Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -1,13 +1,227 @@
|
|
| 1 |
-
---
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Fine-Tuned Text-to-SQL Assistant (QLoRA) with Safe LLM Serving
|
| 2 |
+
|
| 3 |
+
[](https://github.com/AshokSingodia-Codes/fine-tuned-text2sql/actions)
|
| 4 |
+
[](https://www.python.org/)
|
| 5 |
+
[](https://opensource.org/licenses/MIT)
|
| 6 |
+
|
| 7 |
+
An end-to-end production system: dataset building, prompt engineering, evaluation harness, **QLoRA fine-tuning** of `Qwen2.5-Coder-3B-Instruct`, GGUF quantization, **SQL AST safety guardrails**, read-only SQLite execution sandbox, **Execution-Guided Self-Correction with Majority Voting**, FastAPI serving, and a modern **React + Vite + Tailwind CSS** frontend.
|
| 8 |
+
|
| 9 |
+
> [!NOTE]
|
| 10 |
+
> **Open Source & Local First Architecture:**
|
| 11 |
+
> - **Self-Hosted & Private**: Designed to run entirely on your local machine. No queries or database data leave your environment.
|
| 12 |
+
> - **Works with Any SQLite Database**: Query your own `.sqlite` or `.db` databases via CLI, Web UI, or REST API.
|
| 13 |
+
> - **Zero-Weight Quickstart**: Includes an instant mock mode to explore the full UI, AST validation, and sandboxing pipeline without downloading model weights. Real inference runs locally with the fine-tuned 1.9 GB GGUF model or any OpenAI-compatible engine.
|
| 14 |
+
> - **Licensing Notice**: Base model is licensed under Alibaba Cloud's **Qwen Research License Agreement** for non-commercial research and evaluation. See [`NOTICE`](NOTICE) and [`docs/MODEL_CARD.md`](docs/MODEL_CARD.md).
|
| 15 |
+
> - **Built with Qwen**: Non-commercial research project built with Qwen.
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
---
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
## ποΈ System Architecture
|
| 22 |
+
|
| 23 |
+
```mermaid
|
| 24 |
+
flowchart TD
|
| 25 |
+
User["π€ User / React Frontend"] -->|HTTP POST /v1/query| API["π FastAPI Serving API"]
|
| 26 |
+
API --> Auth["π Auth & Token Bucket Rate Limiter"]
|
| 27 |
+
Auth --> Service["βοΈ Text2SQL Service Orchestrator"]
|
| 28 |
+
Service --> Schema["ποΈ SQLite Schema DDL Extractor"]
|
| 29 |
+
Service --> Model["π€ Fine-Tuned Model (Qwen2.5-Coder-3B GGUF Q4_K_M)"]
|
| 30 |
+
Model -->|N=5 Candidate Queries| Consensus["π Execution Majority Voting & Self-Correction"]
|
| 31 |
+
Consensus -->|Candidate Query| Validator["π‘οΈ AST SQL Validator (sqlglot)"]
|
| 32 |
+
Validator -->|Policy Check Passed| Sandbox["π Safe SQLite Executor (Read-Only Mode & Authorizer)"]
|
| 33 |
+
Sandbox -->|Result Set / Rows| Service
|
| 34 |
+
Service --> Audit["π Audit Logger & Prometheus Metrics"]
|
| 35 |
+
Service -->|JSON Response| User
|
| 36 |
+
```
|
| 37 |
+
|
| 38 |
+
---
|
| 39 |
+
|
| 40 |
+
## π Comprehensive Evaluation & Benchmark Results
|
| 41 |
+
|
| 42 |
+
All evaluation numbers are produced empirically by automated code scripts and verified against `results/` artifacts (empirical results).
|
| 43 |
+
|
| 44 |
+
### 1. Model Systems Comparison (Spider Dev Split, $n=1,034$)
|
| 45 |
+
|
| 46 |
+
| System / Configuration | Model Format & Size | Execution Accuracy (Spider Dev, n=1,034) | 95% Bootstrap CI | Latency p50 | Latency p95 | Serving Hardware & Evidence File |
|
| 47 |
+
|---|---|---|---|---|---|---|
|
| 48 |
+
| **Config A: Base Zero-Shot (no guardrail)** | FP16 (6.2 GB) | **62.28%** (644 / 1,034) | [59.38%, 65.18%] | 2.16 s | 5.82 s | Linux GPU (results/eval_spider_latest.json) |
|
| 49 |
+
| **Config B: Base 3-Shot (no guardrail)** | FP16 (6.2 GB) | **66.15%** (684 / 1,034) | [63.25%, 68.96%] | 2.23 s | 6.01 s | Linux GPU (results/eval_spider_latest.json) |
|
| 50 |
+
| **Config C: Merged Fine-Tuned (no guardrail)** | FP16 (6.2 GB) | **71.57%** (740 / 1,034) | [68.86%, 74.27%] | 2.19 s | 5.91 s | Linux GPU (results/eval_spider_latest.json) |
|
| 51 |
+
| **Config D: Fine-Tuned GGUF Q4_K_M (no guardrail)** | GGUF Q4_K_M (1.9 GB) | **70.70%** (731 / 1,034) | [67.89%, 73.40%] | 4.68 s | 11.23 s | Linux CPU (2 physical cores, 31.35 GB RAM, CUDA host; results/eval_spider_latest.json, results/latency_latest.json) |
|
| 52 |
+
| **Config D + Guardrail: GGUF Q4_K_M + AST Guardrail** | GGUF Q4_K_M (1.9 GB) | **70.50%** (729 / 1,034) | [67.69%, 73.21%] | 4.71 s | 11.31 s | Linux CPU (2 physical cores, 31.35 GB RAM, CUDA host; results/false_blocks.json) |
|
| 53 |
+
|
| 54 |
+
> *Note on Row Capping*: 33 of 1,034 Spider dev queries produce gold result sets exceeding 100 rows. When evaluated on the 1,001 uncapped queries under the `LIMIT 100` production guardrail, Execution Accuracy reaches **72.83%** (729 / 1,001) as recorded in [`results/canonical_equiv.json`](results/canonical_equiv.json).
|
| 55 |
+
|
| 56 |
+
> *Note on Hyperparameter Validation Losses*: Hyperparameter validation loss ablations are omitted until full multi-run evaluation logs are provided. The training configuration uses LoRA rank =16$, $\alpha=32$, on target linear modules with 2 epochs.
|
| 57 |
+
|
| 58 |
+
> [!NOTE]
|
| 59 |
+
> **Self-Correction and Majority Voting Status:**
|
| 60 |
+
> Execution-guided self-correction and majority voting consensus algorithms are fully implemented in the engine, but accuracy uplifts on the fine-tuned model are **not evaluated** pending Phase 3b execution on cloud GPU (specified in [docs/PHASE3B.md](docs/PHASE3B.md)). Both mechanisms remain **disabled by default** across the CLI (--consensus flag required), the serving service (enable_consensus: false), and the React web UI.
|
| 61 |
+
|
| 62 |
+
---
|
| 63 |
+
|
| 64 |
+
## π‘οΈ SQL Safety & Red-Team Security Metrics
|
| 65 |
+
|
| 66 |
+
The service employs a multi-layered guardrail: AST validation with `sqlglot` and SQLite `mode=ro` connection authorizer callbacks.
|
| 67 |
+
|
| 68 |
+
| Evaluation Path / Security Metric | Empirical Value | 95% Wilson CI | Defense Mechanism & Evidence |
|
| 69 |
+
|---|---|---|---|
|
| 70 |
+
| **Path A Static AST Block Rate (Raw)** | **87.50% (105 / 120)** | [80.40%, 92.28%] | AST Root Type, Forbidden Op, Join Count (`results/redteam_latest.json`) |
|
| 71 |
+
| **Path C Sandbox Defense Rate** | **100.00% (120 / 120)** | [96.90%, 100.00%] | Read-Only Authorizer & PRAGMA Defensive (`results/redteam_latest.json`) |
|
| 72 |
+
| **Path D Full Pipeline Attack Success** | **0.00% (0 / 120)** | [0.00%, 3.10%] | Multi-Layer Defense in Depth (`results/redteam_latest.json`) |
|
| 73 |
+
| **Path B End-to-End Prompt Injection Success** | **0.00% (0 / 60)** | [0.00%, 6.02%] | Greedy & Stochastic Temperature 0.7 (`results/path_b_protocol_results.json`) |
|
| 74 |
+
| **Spider Dev False-Positive Rate (FPR)** | **0.19% (2 / 1,034)** | [0.05%, 0.70%] | AST Join Count Ceiling (5 > 4) (`results/false_blocks.json`) |
|
| 75 |
+
| **Legitimate Query Pass Rate** | **99.37% (1,103 / 1,110)** | [98.70%, 99.69%] | Spider dev (1,034) + 76 benign controls (`results/false_blocks.json`) |
|
| 76 |
+
|
| 77 |
+
|
| 78 |
+
---
|
| 79 |
+
|
| 80 |
+
## π Quickstart β Local Setup & Running
|
| 81 |
+
|
| 82 |
+
> **Detailed step-by-step walkthrough:** [`docs/LOCAL_INSTALL.md`](docs/LOCAL_INSTALL.md)
|
| 83 |
+
|
| 84 |
+
### 1. Clone & Install Dependencies
|
| 85 |
+
```bash
|
| 86 |
+
git clone https://github.com/AshokSingodia-Codes/fine-tuned-text2sql.git
|
| 87 |
+
cd fine-tuned-text2sql
|
| 88 |
+
|
| 89 |
+
# Set up Python virtual environment
|
| 90 |
+
python -m venv .venv
|
| 91 |
+
# On Windows (PowerShell):
|
| 92 |
+
.\.venv\Scripts\Activate.ps1
|
| 93 |
+
# On Linux/macOS:
|
| 94 |
+
source .venv/bin/activate
|
| 95 |
+
|
| 96 |
+
# Install dependencies
|
| 97 |
+
pip install -r requirements.txt
|
| 98 |
+
pip install -e .
|
| 99 |
+
```
|
| 100 |
+
|
| 101 |
+
### 2. Launch Interactively (Zero Arguments Needed)
|
| 102 |
+
You can launch the all-in-one interactive menu directly from the root of the repository:
|
| 103 |
+
```bash
|
| 104 |
+
python run.py
|
| 105 |
+
```
|
| 106 |
+
This interactive console lets you:
|
| 107 |
+
- **[1] Query Databases in Terminal**: Automatically detects all `.sqlite`/`.db` files, presents a numbered picker, and starts an interactive conversation session.
|
| 108 |
+
- **[2] Launch Web UI**: Starts the local FastAPI + React serving environment.
|
| 109 |
+
- **[3] Validate SQL**: Tests any raw SQL against the AST security guardrail.
|
| 110 |
+
- **[4] Run Verification Tests**: Runs the complete test suite.
|
| 111 |
+
|
| 112 |
+
---
|
| 113 |
+
|
| 114 |
+
## ποΈ Integrating With Your Own SQLite Database
|
| 115 |
+
|
| 116 |
+
You can query your own SQLite database (`.sqlite` or `.db`) in three convenient ways:
|
| 117 |
+
|
| 118 |
+
### Option A: Direct Terminal CLI (Fastest, No Server Needed)
|
| 119 |
+
No servers or web containers required. Query your database directly from your command line:
|
| 120 |
+
|
| 121 |
+
```bash
|
| 122 |
+
# Interactive REPL with auto-database picker (scans folder & lets you pick with 1 key):
|
| 123 |
+
python scripts/query_cli.py
|
| 124 |
+
|
| 125 |
+
# One-shot query against a specific SQLite file:
|
| 126 |
+
python scripts/query_cli.py --db /path/to/my_database.sqlite --question "Show the top 5 highest paying customers"
|
| 127 |
+
|
| 128 |
+
# Validate any raw SQL query through the safety guardrail without executing:
|
| 129 |
+
python scripts/query_cli.py --validate "DROP TABLE users;"
|
| 130 |
+
# Output: β BLOCKED BY GUARDRAIL β’ Forbidden root statement type: DROP
|
| 131 |
+
|
| 132 |
+
# List all databases inside a folder:
|
| 133 |
+
python scripts/query_cli.py --db-dir /path/to/databases/
|
| 134 |
+
```
|
| 135 |
+
|
| 136 |
+
### Option B: Local Web UI & API Server
|
| 137 |
+
Run the FastAPI backend and modern React UI locally on your machine:
|
| 138 |
+
|
| 139 |
+
```bash
|
| 140 |
+
# Terminal 1: Start API server (port 8000)
|
| 141 |
+
# Use mock backend for instant UI testing, or your own SQLite directory:
|
| 142 |
+
python -m uvicorn src.t2s.serving.app:app --port 8000
|
| 143 |
+
# Or point DB_DIR to your databases:
|
| 144 |
+
# DB_DIR=/path/to/my/databases python -m uvicorn src.t2s.serving.app:app --port 8000
|
| 145 |
+
|
| 146 |
+
# Terminal 2: Start React Frontend (port 3000)
|
| 147 |
+
cd frontend
|
| 148 |
+
npm install
|
| 149 |
+
npm run dev
|
| 150 |
+
```
|
| 151 |
+
|
| 152 |
+
1. Open **`http://localhost:3000`** in your browser.
|
| 153 |
+
2. **Drag & Drop** any `.sqlite` or `.db` file into the upload zone.
|
| 154 |
+
3. The server inspects the binary header, extracts the schema/tables automatically, and makes it selectable for querying right away.
|
| 155 |
+
|
| 156 |
+
### Option C: Programmatic REST API (Python / curl)
|
| 157 |
+
```bash
|
| 158 |
+
# 1. Upload your database
|
| 159 |
+
curl -X POST http://localhost:8000/v1/databases/upload -F "file=@my_database.sqlite"
|
| 160 |
+
|
| 161 |
+
# 2. Query in plain English
|
| 162 |
+
curl -X POST http://localhost:8000/v1/query \
|
| 163 |
+
-H "Content-Type: application/json" \
|
| 164 |
+
-d '{"db_id":"my_database","question":"Count total orders placed this year","enable_consensus":false}'
|
| 165 |
+
|
| 166 |
+
# Interactive OpenAPI docs: http://localhost:8000/docs
|
| 167 |
+
```
|
| 168 |
+
|
| 169 |
+
---
|
| 170 |
+
|
| 171 |
+
## π‘οΈ AI Error Handling & Database Safety (How Errors Are Handled)
|
| 172 |
+
|
| 173 |
+
When querying production databases with LLMs, models can hallucinate invalid SQL or attempt unsafe queries. This system implements **defense-in-depth**:
|
| 174 |
+
|
| 175 |
+
1. **Auto-Correction via Feedback Loops (Execution-Guided Self-Correction)**:
|
| 176 |
+
- If the model produces an SQL syntax error or references an invalid column name, the error message is intercepted and fed back to the model with schema hints in an automated retry turn (`max_retries=2`).
|
| 177 |
+
2. **Majority Voting Consensus (`enable_consensus: true`)**:
|
| 178 |
+
- For high-stakes queries, the engine samples $N=5$ candidate SQL queries, runs them against an isolated sandbox, and selects the majority result set to eliminate outlier hallucinations.
|
| 179 |
+
3. **AST Safety Guardrails (`sqlglot`)**:
|
| 180 |
+
- Parses the query AST prior to execution. Strictly prohibits destructive operations (`DROP`, `DELETE`, `UPDATE`, `ALTER`, `TRUNCATE`), multi-statement batching (SQL injection), and metadata scraping (`sqlite_master`).
|
| 181 |
+
4. **C-Level SQLite Sandbox (`mode=ro` + Authorizer)**:
|
| 182 |
+
- The database is mounted with `mode=ro` (read-only) and strict SQLite C-level authorizer callbacks. Even in the event of an unexpected model failure or adversarial injection, **the database file cannot be modified or written to**.
|
| 183 |
+
5. **Row Output Capping**:
|
| 184 |
+
- Automatically injects an execution limit (`LIMIT 100`) to protect against memory exhaustion or data exfiltration.
|
| 185 |
+
|
| 186 |
+
---
|
| 187 |
+
|
| 188 |
+
### π§ͺ Running Verification Tests
|
| 189 |
+
```bash
|
| 190 |
+
pytest -q # Runs full suite (85 tests passing: fuzzing, safety, serving, APIs)
|
| 191 |
+
```
|
| 192 |
+
|
| 193 |
+
## π³ Optional: Local Docker Setup
|
| 194 |
+
To run both the API server and UI in isolated local Docker containers:
|
| 195 |
+
```bash
|
| 196 |
+
docker compose up --build
|
| 197 |
+
```
|
| 198 |
+
- **Local Web App**: `http://localhost:3000`
|
| 199 |
+
- **FastAPI Docs**: `http://localhost:8000/docs`
|
| 200 |
+
|
| 201 |
+
---
|
| 202 |
+
|
| 203 |
+
## πΊοΈ Scope & Roadmap (What Was Built vs Not Built)
|
| 204 |
+
|
| 205 |
+
### Implemented Core Capabilities
|
| 206 |
+
- [x] Parameter-efficient QLoRA fine-tuning of `Qwen2.5-Coder-3B-Instruct`
|
| 207 |
+
- [x] 4-bit GGUF quantization (Q4_K_M) with local CPU serving
|
| 208 |
+
- [x] Multi-layer AST guardrails (`sqlglot`) and read-only SQLite sandboxing
|
| 209 |
+
- [x] Bounded FastAPI serving (concurrency queue depth 10, semaphore 1)
|
| 210 |
+
- [x] Single-container React + Vite UI with dark-mode styling and live query telemetry
|
| 211 |
+
- [x] Execution-guided self-correction and consensus voting algorithmic harnesses
|
| 212 |
+
|
| 213 |
+
### Explicitly Out-of-Scope (Deferred Architectural Extensions)
|
| 214 |
+
- **Multi-Database Dialects (PostgreSQL, MySQL)**: Deferred in MVP path; currently specialized for SQLite execution.
|
| 215 |
+
- **Enterprise Warehouses (Snowflake, BigQuery, Redshift, Databricks)**: Strict non-goal (out of project scope); excluded to maintain lightweight standalone operation.
|
| 216 |
+
- **Enterprise Identity & Auth (SSO, SAML, Okta)**: Replaced by lightweight API token authentication.
|
| 217 |
+
- **Orchestration (Kubernetes)**: Excluded in favor of single-node Docker Compose and Hugging Face Space container deployment.
|
| 218 |
+
|
| 219 |
+
---
|
| 220 |
+
|
| 221 |
+
## π References & Citations
|
| 222 |
+
|
| 223 |
+
- **Spider Dataset**: Yu, T., Zhang, R. S., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., Zhang, Z., & Radev, D. (2018). *Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task*. EMNLP 2018.
|
| 224 |
+
- **Base Model**: Alibaba Cloud Qwen Team. (2024). *Qwen2.5-Coder: Empowering Code Generation and Reasoning*. [https://github.com/QwenLM/Qwen2.5-Coder](https://github.com/QwenLM/Qwen2.5-Coder)
|
| 225 |
+
- **SQL Parser**: Mao, T., & contributors. (2024). *sqlglot: An extensible SQL parser, transpiler, and optimizer*. [https://github.com/tobymao/sqlglot](https://github.com/tobymao/sqlglot)
|
| 226 |
+
- **Quantized Runtime**: Gerganov, G., & contributors. (2024). *llama.cpp: LLM inference in C/C++*. [https://github.com/ggerganov/llama.cpp](https://github.com/ggerganov/llama.cpp)
|
| 227 |
+
|