Ashoksainibadu commited on
Commit
6a67d84
Β·
verified Β·
1 Parent(s): e8b9b6c

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +227 -13
README.md CHANGED
@@ -1,13 +1,227 @@
1
- ---
2
- title: Text2sql Assistant
3
- emoji: πŸ›‘οΈ
4
- colorFrom: indigo
5
- colorTo: blue
6
- sdk: docker
7
- app_port: 7860
8
- pinned: false
9
- ---
10
-
11
- # Fine-Tuned Text-to-SQL Assistant (Hugging Face Space)
12
-
13
- Serving fine-tuned Qwen2.5-Coder-3B QLoRA with static AST safety guardrails and SQLite execution sandbox.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Fine-Tuned Text-to-SQL Assistant (QLoRA) with Safe LLM Serving
2
+
3
+ [![CI](https://github.com/AshokSingodia-Codes/fine-tuned-text2sql/actions/workflows/ci.yml/badge.svg)](https://github.com/AshokSingodia-Codes/fine-tuned-text2sql/actions)
4
+ [![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/)
5
+ [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
6
+
7
+ An end-to-end production system: dataset building, prompt engineering, evaluation harness, **QLoRA fine-tuning** of `Qwen2.5-Coder-3B-Instruct`, GGUF quantization, **SQL AST safety guardrails**, read-only SQLite execution sandbox, **Execution-Guided Self-Correction with Majority Voting**, FastAPI serving, and a modern **React + Vite + Tailwind CSS** frontend.
8
+
9
+ > [!NOTE]
10
+ > **Open Source & Local First Architecture:**
11
+ > - **Self-Hosted & Private**: Designed to run entirely on your local machine. No queries or database data leave your environment.
12
+ > - **Works with Any SQLite Database**: Query your own `.sqlite` or `.db` databases via CLI, Web UI, or REST API.
13
+ > - **Zero-Weight Quickstart**: Includes an instant mock mode to explore the full UI, AST validation, and sandboxing pipeline without downloading model weights. Real inference runs locally with the fine-tuned 1.9 GB GGUF model or any OpenAI-compatible engine.
14
+ > - **Licensing Notice**: Base model is licensed under Alibaba Cloud's **Qwen Research License Agreement** for non-commercial research and evaluation. See [`NOTICE`](NOTICE) and [`docs/MODEL_CARD.md`](docs/MODEL_CARD.md).
15
+ > - **Built with Qwen**: Non-commercial research project built with Qwen.
16
+
17
+
18
+ ---
19
+
20
+
21
+ ## πŸ›οΈ System Architecture
22
+
23
+ ```mermaid
24
+ flowchart TD
25
+ User["πŸ‘€ User / React Frontend"] -->|HTTP POST /v1/query| API["πŸš€ FastAPI Serving API"]
26
+ API --> Auth["πŸ” Auth & Token Bucket Rate Limiter"]
27
+ Auth --> Service["βš™οΈ Text2SQL Service Orchestrator"]
28
+ Service --> Schema["πŸ—„οΈ SQLite Schema DDL Extractor"]
29
+ Service --> Model["πŸ€– Fine-Tuned Model (Qwen2.5-Coder-3B GGUF Q4_K_M)"]
30
+ Model -->|N=5 Candidate Queries| Consensus["πŸ† Execution Majority Voting & Self-Correction"]
31
+ Consensus -->|Candidate Query| Validator["πŸ›‘οΈ AST SQL Validator (sqlglot)"]
32
+ Validator -->|Policy Check Passed| Sandbox["πŸ”’ Safe SQLite Executor (Read-Only Mode & Authorizer)"]
33
+ Sandbox -->|Result Set / Rows| Service
34
+ Service --> Audit["πŸ“ Audit Logger & Prometheus Metrics"]
35
+ Service -->|JSON Response| User
36
+ ```
37
+
38
+ ---
39
+
40
+ ## πŸ“Š Comprehensive Evaluation & Benchmark Results
41
+
42
+ All evaluation numbers are produced empirically by automated code scripts and verified against `results/` artifacts (empirical results).
43
+
44
+ ### 1. Model Systems Comparison (Spider Dev Split, $n=1,034$)
45
+
46
+ | System / Configuration | Model Format & Size | Execution Accuracy (Spider Dev, n=1,034) | 95% Bootstrap CI | Latency p50 | Latency p95 | Serving Hardware & Evidence File |
47
+ |---|---|---|---|---|---|---|
48
+ | **Config A: Base Zero-Shot (no guardrail)** | FP16 (6.2 GB) | **62.28%** (644 / 1,034) | [59.38%, 65.18%] | 2.16 s | 5.82 s | Linux GPU (results/eval_spider_latest.json) |
49
+ | **Config B: Base 3-Shot (no guardrail)** | FP16 (6.2 GB) | **66.15%** (684 / 1,034) | [63.25%, 68.96%] | 2.23 s | 6.01 s | Linux GPU (results/eval_spider_latest.json) |
50
+ | **Config C: Merged Fine-Tuned (no guardrail)** | FP16 (6.2 GB) | **71.57%** (740 / 1,034) | [68.86%, 74.27%] | 2.19 s | 5.91 s | Linux GPU (results/eval_spider_latest.json) |
51
+ | **Config D: Fine-Tuned GGUF Q4_K_M (no guardrail)** | GGUF Q4_K_M (1.9 GB) | **70.70%** (731 / 1,034) | [67.89%, 73.40%] | 4.68 s | 11.23 s | Linux CPU (2 physical cores, 31.35 GB RAM, CUDA host; results/eval_spider_latest.json, results/latency_latest.json) |
52
+ | **Config D + Guardrail: GGUF Q4_K_M + AST Guardrail** | GGUF Q4_K_M (1.9 GB) | **70.50%** (729 / 1,034) | [67.69%, 73.21%] | 4.71 s | 11.31 s | Linux CPU (2 physical cores, 31.35 GB RAM, CUDA host; results/false_blocks.json) |
53
+
54
+ > *Note on Row Capping*: 33 of 1,034 Spider dev queries produce gold result sets exceeding 100 rows. When evaluated on the 1,001 uncapped queries under the `LIMIT 100` production guardrail, Execution Accuracy reaches **72.83%** (729 / 1,001) as recorded in [`results/canonical_equiv.json`](results/canonical_equiv.json).
55
+
56
+ > *Note on Hyperparameter Validation Losses*: Hyperparameter validation loss ablations are omitted until full multi-run evaluation logs are provided. The training configuration uses LoRA rank =16$, $\alpha=32$, on target linear modules with 2 epochs.
57
+
58
+ > [!NOTE]
59
+ > **Self-Correction and Majority Voting Status:**
60
+ > Execution-guided self-correction and majority voting consensus algorithms are fully implemented in the engine, but accuracy uplifts on the fine-tuned model are **not evaluated** pending Phase 3b execution on cloud GPU (specified in [docs/PHASE3B.md](docs/PHASE3B.md)). Both mechanisms remain **disabled by default** across the CLI (--consensus flag required), the serving service (enable_consensus: false), and the React web UI.
61
+
62
+ ---
63
+
64
+ ## πŸ›‘οΈ SQL Safety & Red-Team Security Metrics
65
+
66
+ The service employs a multi-layered guardrail: AST validation with `sqlglot` and SQLite `mode=ro` connection authorizer callbacks.
67
+
68
+ | Evaluation Path / Security Metric | Empirical Value | 95% Wilson CI | Defense Mechanism & Evidence |
69
+ |---|---|---|---|
70
+ | **Path A Static AST Block Rate (Raw)** | **87.50% (105 / 120)** | [80.40%, 92.28%] | AST Root Type, Forbidden Op, Join Count (`results/redteam_latest.json`) |
71
+ | **Path C Sandbox Defense Rate** | **100.00% (120 / 120)** | [96.90%, 100.00%] | Read-Only Authorizer & PRAGMA Defensive (`results/redteam_latest.json`) |
72
+ | **Path D Full Pipeline Attack Success** | **0.00% (0 / 120)** | [0.00%, 3.10%] | Multi-Layer Defense in Depth (`results/redteam_latest.json`) |
73
+ | **Path B End-to-End Prompt Injection Success** | **0.00% (0 / 60)** | [0.00%, 6.02%] | Greedy & Stochastic Temperature 0.7 (`results/path_b_protocol_results.json`) |
74
+ | **Spider Dev False-Positive Rate (FPR)** | **0.19% (2 / 1,034)** | [0.05%, 0.70%] | AST Join Count Ceiling (5 > 4) (`results/false_blocks.json`) |
75
+ | **Legitimate Query Pass Rate** | **99.37% (1,103 / 1,110)** | [98.70%, 99.69%] | Spider dev (1,034) + 76 benign controls (`results/false_blocks.json`) |
76
+
77
+
78
+ ---
79
+
80
+ ## πŸš€ Quickstart β€” Local Setup & Running
81
+
82
+ > **Detailed step-by-step walkthrough:** [`docs/LOCAL_INSTALL.md`](docs/LOCAL_INSTALL.md)
83
+
84
+ ### 1. Clone & Install Dependencies
85
+ ```bash
86
+ git clone https://github.com/AshokSingodia-Codes/fine-tuned-text2sql.git
87
+ cd fine-tuned-text2sql
88
+
89
+ # Set up Python virtual environment
90
+ python -m venv .venv
91
+ # On Windows (PowerShell):
92
+ .\.venv\Scripts\Activate.ps1
93
+ # On Linux/macOS:
94
+ source .venv/bin/activate
95
+
96
+ # Install dependencies
97
+ pip install -r requirements.txt
98
+ pip install -e .
99
+ ```
100
+
101
+ ### 2. Launch Interactively (Zero Arguments Needed)
102
+ You can launch the all-in-one interactive menu directly from the root of the repository:
103
+ ```bash
104
+ python run.py
105
+ ```
106
+ This interactive console lets you:
107
+ - **[1] Query Databases in Terminal**: Automatically detects all `.sqlite`/`.db` files, presents a numbered picker, and starts an interactive conversation session.
108
+ - **[2] Launch Web UI**: Starts the local FastAPI + React serving environment.
109
+ - **[3] Validate SQL**: Tests any raw SQL against the AST security guardrail.
110
+ - **[4] Run Verification Tests**: Runs the complete test suite.
111
+
112
+ ---
113
+
114
+ ## πŸ—„οΈ Integrating With Your Own SQLite Database
115
+
116
+ You can query your own SQLite database (`.sqlite` or `.db`) in three convenient ways:
117
+
118
+ ### Option A: Direct Terminal CLI (Fastest, No Server Needed)
119
+ No servers or web containers required. Query your database directly from your command line:
120
+
121
+ ```bash
122
+ # Interactive REPL with auto-database picker (scans folder & lets you pick with 1 key):
123
+ python scripts/query_cli.py
124
+
125
+ # One-shot query against a specific SQLite file:
126
+ python scripts/query_cli.py --db /path/to/my_database.sqlite --question "Show the top 5 highest paying customers"
127
+
128
+ # Validate any raw SQL query through the safety guardrail without executing:
129
+ python scripts/query_cli.py --validate "DROP TABLE users;"
130
+ # Output: βœ— BLOCKED BY GUARDRAIL β€’ Forbidden root statement type: DROP
131
+
132
+ # List all databases inside a folder:
133
+ python scripts/query_cli.py --db-dir /path/to/databases/
134
+ ```
135
+
136
+ ### Option B: Local Web UI & API Server
137
+ Run the FastAPI backend and modern React UI locally on your machine:
138
+
139
+ ```bash
140
+ # Terminal 1: Start API server (port 8000)
141
+ # Use mock backend for instant UI testing, or your own SQLite directory:
142
+ python -m uvicorn src.t2s.serving.app:app --port 8000
143
+ # Or point DB_DIR to your databases:
144
+ # DB_DIR=/path/to/my/databases python -m uvicorn src.t2s.serving.app:app --port 8000
145
+
146
+ # Terminal 2: Start React Frontend (port 3000)
147
+ cd frontend
148
+ npm install
149
+ npm run dev
150
+ ```
151
+
152
+ 1. Open **`http://localhost:3000`** in your browser.
153
+ 2. **Drag & Drop** any `.sqlite` or `.db` file into the upload zone.
154
+ 3. The server inspects the binary header, extracts the schema/tables automatically, and makes it selectable for querying right away.
155
+
156
+ ### Option C: Programmatic REST API (Python / curl)
157
+ ```bash
158
+ # 1. Upload your database
159
+ curl -X POST http://localhost:8000/v1/databases/upload -F "file=@my_database.sqlite"
160
+
161
+ # 2. Query in plain English
162
+ curl -X POST http://localhost:8000/v1/query \
163
+ -H "Content-Type: application/json" \
164
+ -d '{"db_id":"my_database","question":"Count total orders placed this year","enable_consensus":false}'
165
+
166
+ # Interactive OpenAPI docs: http://localhost:8000/docs
167
+ ```
168
+
169
+ ---
170
+
171
+ ## πŸ›‘οΈ AI Error Handling & Database Safety (How Errors Are Handled)
172
+
173
+ When querying production databases with LLMs, models can hallucinate invalid SQL or attempt unsafe queries. This system implements **defense-in-depth**:
174
+
175
+ 1. **Auto-Correction via Feedback Loops (Execution-Guided Self-Correction)**:
176
+ - If the model produces an SQL syntax error or references an invalid column name, the error message is intercepted and fed back to the model with schema hints in an automated retry turn (`max_retries=2`).
177
+ 2. **Majority Voting Consensus (`enable_consensus: true`)**:
178
+ - For high-stakes queries, the engine samples $N=5$ candidate SQL queries, runs them against an isolated sandbox, and selects the majority result set to eliminate outlier hallucinations.
179
+ 3. **AST Safety Guardrails (`sqlglot`)**:
180
+ - Parses the query AST prior to execution. Strictly prohibits destructive operations (`DROP`, `DELETE`, `UPDATE`, `ALTER`, `TRUNCATE`), multi-statement batching (SQL injection), and metadata scraping (`sqlite_master`).
181
+ 4. **C-Level SQLite Sandbox (`mode=ro` + Authorizer)**:
182
+ - The database is mounted with `mode=ro` (read-only) and strict SQLite C-level authorizer callbacks. Even in the event of an unexpected model failure or adversarial injection, **the database file cannot be modified or written to**.
183
+ 5. **Row Output Capping**:
184
+ - Automatically injects an execution limit (`LIMIT 100`) to protect against memory exhaustion or data exfiltration.
185
+
186
+ ---
187
+
188
+ ### πŸ§ͺ Running Verification Tests
189
+ ```bash
190
+ pytest -q # Runs full suite (85 tests passing: fuzzing, safety, serving, APIs)
191
+ ```
192
+
193
+ ## 🐳 Optional: Local Docker Setup
194
+ To run both the API server and UI in isolated local Docker containers:
195
+ ```bash
196
+ docker compose up --build
197
+ ```
198
+ - **Local Web App**: `http://localhost:3000`
199
+ - **FastAPI Docs**: `http://localhost:8000/docs`
200
+
201
+ ---
202
+
203
+ ## πŸ—ΊοΈ Scope & Roadmap (What Was Built vs Not Built)
204
+
205
+ ### Implemented Core Capabilities
206
+ - [x] Parameter-efficient QLoRA fine-tuning of `Qwen2.5-Coder-3B-Instruct`
207
+ - [x] 4-bit GGUF quantization (Q4_K_M) with local CPU serving
208
+ - [x] Multi-layer AST guardrails (`sqlglot`) and read-only SQLite sandboxing
209
+ - [x] Bounded FastAPI serving (concurrency queue depth 10, semaphore 1)
210
+ - [x] Single-container React + Vite UI with dark-mode styling and live query telemetry
211
+ - [x] Execution-guided self-correction and consensus voting algorithmic harnesses
212
+
213
+ ### Explicitly Out-of-Scope (Deferred Architectural Extensions)
214
+ - **Multi-Database Dialects (PostgreSQL, MySQL)**: Deferred in MVP path; currently specialized for SQLite execution.
215
+ - **Enterprise Warehouses (Snowflake, BigQuery, Redshift, Databricks)**: Strict non-goal (out of project scope); excluded to maintain lightweight standalone operation.
216
+ - **Enterprise Identity & Auth (SSO, SAML, Okta)**: Replaced by lightweight API token authentication.
217
+ - **Orchestration (Kubernetes)**: Excluded in favor of single-node Docker Compose and Hugging Face Space container deployment.
218
+
219
+ ---
220
+
221
+ ## πŸ“„ References & Citations
222
+
223
+ - **Spider Dataset**: Yu, T., Zhang, R. S., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., Zhang, Z., & Radev, D. (2018). *Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task*. EMNLP 2018.
224
+ - **Base Model**: Alibaba Cloud Qwen Team. (2024). *Qwen2.5-Coder: Empowering Code Generation and Reasoning*. [https://github.com/QwenLM/Qwen2.5-Coder](https://github.com/QwenLM/Qwen2.5-Coder)
225
+ - **SQL Parser**: Mao, T., & contributors. (2024). *sqlglot: An extensible SQL parser, transpiler, and optimizer*. [https://github.com/tobymao/sqlglot](https://github.com/tobymao/sqlglot)
226
+ - **Quantized Runtime**: Gerganov, G., & contributors. (2024). *llama.cpp: LLM inference in C/C++*. [https://github.com/ggerganov/llama.cpp](https://github.com/ggerganov/llama.cpp)
227
+