Spaces:
Sleeping
Sleeping
fix inference runtime errors
Browse files
README.md
DELETED
|
@@ -1,257 +0,0 @@
|
|
| 1 |
-
---
|
| 2 |
-
title: SQL Repair Env
|
| 3 |
-
colorFrom: blue
|
| 4 |
-
colorTo: green
|
| 5 |
-
sdk: docker
|
| 6 |
-
app_port: 7860
|
| 7 |
-
---
|
| 8 |
-
|
| 9 |
-
# Interactive SQL Query Repair Environment
|
| 10 |
-
|
| 11 |
-
An OpenEnv-compatible reinforcement learning environment where an agent
|
| 12 |
-
learns to fix broken SQL queries through multi-turn interaction.
|
| 13 |
-
|
| 14 |
-
---
|
| 15 |
-
|
| 16 |
-
## Overview
|
| 17 |
-
|
| 18 |
-
The agent receives a **broken SQL query**, a **plain-English intent**, and a
|
| 19 |
-
**compact schema hint**. Over multiple turns it can:
|
| 20 |
-
|
| 21 |
-
- **Run partial queries** — test a fix and see whether SQLite errors
|
| 22 |
-
- **Ask clarification questions** — probe the schema or intent for hints
|
| 23 |
-
- **Submit a final query** — evaluated against ground-truth SQLite execution
|
| 24 |
-
|
| 25 |
-
Grading is fully objective: does the repaired query return the correct rows?
|
| 26 |
-
|
| 27 |
-
---
|
| 28 |
-
|
| 29 |
-
## Environment Details
|
| 30 |
-
|
| 31 |
-
### Action Space
|
| 32 |
-
|
| 33 |
-
| Action | Fields | Effect |
|
| 34 |
-
| ------------------- | --------------- | --------------------------------------------------- |
|
| 35 |
-
| `submit_query` | `query: str` | Finalises the episode; scored by Jaccard similarity |
|
| 36 |
-
| `run_partial` | `query: str` | Tests a query; returns error message or `OK` |
|
| 37 |
-
| `ask_clarification` | `question: str` | Returns a schema/intent hint if available |
|
| 38 |
-
|
| 39 |
-
### Observation Space
|
| 40 |
-
|
| 41 |
-
| Field | Type | Description |
|
| 42 |
-
| ------------------------ | ------------------ | -------------------------------------------------- |
|
| 43 |
-
| `task_id` | str | Unique task identifier |
|
| 44 |
-
| `difficulty` | easy\|medium\|hard | Difficulty level |
|
| 45 |
-
| `intent` | str | Plain-English goal for the query |
|
| 46 |
-
| `schema_hint` | str | Compact table signatures |
|
| 47 |
-
| `broken_query` | str | The query to fix (unchanged across turns) |
|
| 48 |
-
| `turn` | int | Current turn number |
|
| 49 |
-
| `max_turns` | int | Maximum turns for this difficulty |
|
| 50 |
-
| `error_message` | str\|null | SQLite error from last run_partial / failed submit |
|
| 51 |
-
| `clarification_response` | str\|null | Answer to last ask_clarification |
|
| 52 |
-
| `last_action_type` | str\|null | Previous action type |
|
| 53 |
-
|
| 54 |
-
### Reward Function
|
| 55 |
-
|
| 56 |
-
```
|
| 57 |
-
submit_query:
|
| 58 |
-
jaccard == 1.0 → +1.0 (perfect)
|
| 59 |
-
jaccard >= 0.8 → +0.5 (close)
|
| 60 |
-
jaccard >= 0.5 → +0.2 (partial)
|
| 61 |
-
jaccard < 0.5 → -0.5 (wrong)
|
| 62 |
-
minus turn_penalty = 0.05 × (turns_used - 1)
|
| 63 |
-
clamped to [-1.0, 1.0]
|
| 64 |
-
|
| 65 |
-
run_partial:
|
| 66 |
-
no SQL error → +0.05
|
| 67 |
-
SQL error → -0.10
|
| 68 |
-
|
| 69 |
-
ask_clarification:
|
| 70 |
-
answer found → 0.00
|
| 71 |
-
no answer → -0.05
|
| 72 |
-
```
|
| 73 |
-
|
| 74 |
-
### Grader Formula
|
| 75 |
-
|
| 76 |
-
```
|
| 77 |
-
score = 0.60 × correctness
|
| 78 |
-
+ 0.25 × turn_efficiency
|
| 79 |
-
+ 0.15 × clarification_quality
|
| 80 |
-
|
| 81 |
-
correctness = final Jaccard score
|
| 82 |
-
turn_efficiency = 1 - (turns_used / max_turns)
|
| 83 |
-
clarification_quality = useful_hits / total_asks (1.0 if no asks)
|
| 84 |
-
```
|
| 85 |
-
|
| 86 |
-
All scores are deterministic and in `[0.0, 1.0]`.
|
| 87 |
-
|
| 88 |
-
---
|
| 89 |
-
|
| 90 |
-
## Tasks
|
| 91 |
-
|
| 92 |
-
### Easy (15 tasks, max 3 turns)
|
| 93 |
-
|
| 94 |
-
- Single table (`users`)
|
| 95 |
-
- Error types: syntax errors, wrong operators, missing keywords
|
| 96 |
-
- Examples: `COUNTS(*)` → `COUNT(*)`, `ORDER age` → `ORDER BY age`
|
| 97 |
-
|
| 98 |
-
### Medium (15 tasks, max 5 turns)
|
| 99 |
-
|
| 100 |
-
- Two tables (`users + orders` or `orders + products`)
|
| 101 |
-
- Error types: wrong join keys, `WHERE` vs `HAVING`, missing `LEFT JOIN`
|
| 102 |
-
- Examples: join on `orders.id` instead of `orders.user_id`
|
| 103 |
-
|
| 104 |
-
### Hard (10 tasks, max 8 turns)
|
| 105 |
-
|
| 106 |
-
- Three to five tables with CTEs and subqueries
|
| 107 |
-
- Error types: nested aggregates, correlated subqueries, multi-table joins
|
| 108 |
-
- Designed so naive first-attempt strategies score ≤ 0.40
|
| 109 |
-
- Examples: `AVG(SUM(...))` requires CTE, tag intersection requires self-join
|
| 110 |
-
|
| 111 |
-
---
|
| 112 |
-
|
| 113 |
-
## API Reference
|
| 114 |
-
|
| 115 |
-
| Endpoint | Method | Description |
|
| 116 |
-
| --------- | ------ | ------------------------------- |
|
| 117 |
-
| `/health` | GET | Liveness probe |
|
| 118 |
-
| `/tasks` | GET | List all 40 tasks |
|
| 119 |
-
| `/reset` | POST | Start a new episode |
|
| 120 |
-
| `/step` | POST | Take one action |
|
| 121 |
-
| `/state` | GET | Current episode state |
|
| 122 |
-
| `/grader` | GET | Final score for current episode |
|
| 123 |
-
|
| 124 |
-
### POST /reset
|
| 125 |
-
|
| 126 |
-
```json
|
| 127 |
-
{
|
| 128 |
-
"task_id": "medium_001",
|
| 129 |
-
"difficulty": "medium"
|
| 130 |
-
}
|
| 131 |
-
```
|
| 132 |
-
|
| 133 |
-
### POST /step — submit_query
|
| 134 |
-
|
| 135 |
-
```json
|
| 136 |
-
{
|
| 137 |
-
"type": "submit_query",
|
| 138 |
-
"query": "SELECT DISTINCT u.name FROM users u JOIN orders o ON u.id = o.user_id"
|
| 139 |
-
}
|
| 140 |
-
```
|
| 141 |
-
|
| 142 |
-
### POST /step — run_partial
|
| 143 |
-
|
| 144 |
-
```json
|
| 145 |
-
{
|
| 146 |
-
"type": "run_partial",
|
| 147 |
-
"query": "SELECT * FROM users WHERE age > 30"
|
| 148 |
-
}
|
| 149 |
-
```
|
| 150 |
-
|
| 151 |
-
### POST /step — ask_clarification
|
| 152 |
-
|
| 153 |
-
```json
|
| 154 |
-
{
|
| 155 |
-
"type": "ask_clarification",
|
| 156 |
-
"question": "what column in orders links to users?"
|
| 157 |
-
}
|
| 158 |
-
```
|
| 159 |
-
|
| 160 |
-
---
|
| 161 |
-
|
| 162 |
-
## Setup & Running
|
| 163 |
-
|
| 164 |
-
### Local (Python)
|
| 165 |
-
|
| 166 |
-
```bash
|
| 167 |
-
pip install -r requirements.txt
|
| 168 |
-
uvicorn server.app:app --host 0.0.0.0 --port 7860
|
| 169 |
-
```
|
| 170 |
-
|
| 171 |
-
### Docker
|
| 172 |
-
|
| 173 |
-
```bash
|
| 174 |
-
docker build -t sql-repair-env .
|
| 175 |
-
docker run -p 8000:8000 sql-repair-env
|
| 176 |
-
```
|
| 177 |
-
|
| 178 |
-
Test it:
|
| 179 |
-
|
| 180 |
-
```bash
|
| 181 |
-
curl http://localhost:7860/health
|
| 182 |
-
curl http://localhost:7860/tasks
|
| 183 |
-
curl -X POST http://localhost:7860/reset \
|
| 184 |
-
-H "Content-Type: application/json" \
|
| 185 |
-
-d '{"task_id": "easy_001", "difficulty": "easy"}'
|
| 186 |
-
```
|
| 187 |
-
|
| 188 |
-
---
|
| 189 |
-
|
| 190 |
-
## Running the Baseline
|
| 191 |
-
|
| 192 |
-
```bash
|
| 193 |
-
export API_BASE_URL="https://router.huggingface.co/v1"
|
| 194 |
-
export MODEL_NAME="Qwen/Qwen2.5-72B-Instruct"
|
| 195 |
-
export HF_TOKEN="your-token-here"
|
| 196 |
-
export ENV_SERVER_URL="http://localhost:7860"
|
| 197 |
-
|
| 198 |
-
python inference.py
|
| 199 |
-
```
|
| 200 |
-
|
| 201 |
-
Expected baseline scores (approximate):
|
| 202 |
-
|
| 203 |
-
| Difficulty | Score range |
|
| 204 |
-
| ---------- | ----------- |
|
| 205 |
-
| Easy | 0.55 – 0.80 |
|
| 206 |
-
| Medium | 0.30 – 0.55 |
|
| 207 |
-
| Hard | 0.10 – 0.30 |
|
| 208 |
-
|
| 209 |
-
---
|
| 210 |
-
|
| 211 |
-
## Project Structure
|
| 212 |
-
|
| 213 |
-
```
|
| 214 |
-
sql-repair-env/
|
| 215 |
-
├── app/
|
| 216 |
-
│ ├── models.py # All Pydantic types
|
| 217 |
-
│ ├── env.py # Core environment logic
|
| 218 |
-
│ ├── executor.py # SQLite execution layer
|
| 219 |
-
│ ├── grader.py # Deterministic scoring
|
| 220 |
-
│ └── data_loader.py # Dataset loader (module singleton)
|
| 221 |
-
├── data/
|
| 222 |
-
│ ├── easy.json # 15 easy tasks
|
| 223 |
-
│ ├── medium.json # 15 medium tasks
|
| 224 |
-
│ └── hard.json # 10 hard tasks
|
| 225 |
-
├── db/
|
| 226 |
-
│ └── fixtures.sql # SQLite seed data
|
| 227 |
-
├── server/
|
| 228 |
-
│ └── app.py # FastAPI server
|
| 229 |
-
├── inference.py # Baseline inference script
|
| 230 |
-
├── openenv.yaml # OpenEnv spec
|
| 231 |
-
├── Dockerfile
|
| 232 |
-
├── requirements.txt
|
| 233 |
-
└── README.md
|
| 234 |
-
```
|
| 235 |
-
|
| 236 |
-
---
|
| 237 |
-
|
| 238 |
-
## Design Decisions
|
| 239 |
-
|
| 240 |
-
**Why SQLite in-process?**
|
| 241 |
-
Zero external infrastructure — the DB seeds from `fixtures.sql` on every
|
| 242 |
-
`reset()`, guaranteeing a clean deterministic state per episode.
|
| 243 |
-
|
| 244 |
-
**Why Jaccard similarity?**
|
| 245 |
-
Row order is undefined in SQL; Jaccard treats results as sets, making grading
|
| 246 |
-
robust to `ORDER BY` differences between the agent's query and the reference.
|
| 247 |
-
|
| 248 |
-
**Why deny actual row data to the agent?**
|
| 249 |
-
Showing only error messages forces the agent to reason about the query
|
| 250 |
-
structure, not memorise the dataset. This better simulates real-world
|
| 251 |
-
debugging where production data isn't available.
|
| 252 |
-
|
| 253 |
-
**Why three action types?**
|
| 254 |
-
`run_partial` + `ask_clarification` before `submit_query` models real
|
| 255 |
-
developer behaviour: check syntax, check schema, then commit. The turn
|
| 256 |
-
penalty in the reward function ensures the agent learns to use these
|
| 257 |
-
efficiently rather than exhausting the budget.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|