Savita Srivastava commited on
Commit
4f8b8c0
·
2 Parent(s): fead63975105a2

fix inference runtime errors

Browse files
Files changed (1) hide show
  1. README.md +0 -257
README.md DELETED
@@ -1,257 +0,0 @@
1
- ---
2
- title: SQL Repair Env
3
- colorFrom: blue
4
- colorTo: green
5
- sdk: docker
6
- app_port: 7860
7
- ---
8
-
9
- # Interactive SQL Query Repair Environment
10
-
11
- An OpenEnv-compatible reinforcement learning environment where an agent
12
- learns to fix broken SQL queries through multi-turn interaction.
13
-
14
- ---
15
-
16
- ## Overview
17
-
18
- The agent receives a **broken SQL query**, a **plain-English intent**, and a
19
- **compact schema hint**. Over multiple turns it can:
20
-
21
- - **Run partial queries** — test a fix and see whether SQLite errors
22
- - **Ask clarification questions** — probe the schema or intent for hints
23
- - **Submit a final query** — evaluated against ground-truth SQLite execution
24
-
25
- Grading is fully objective: does the repaired query return the correct rows?
26
-
27
- ---
28
-
29
- ## Environment Details
30
-
31
- ### Action Space
32
-
33
- | Action | Fields | Effect |
34
- | ------------------- | --------------- | --------------------------------------------------- |
35
- | `submit_query` | `query: str` | Finalises the episode; scored by Jaccard similarity |
36
- | `run_partial` | `query: str` | Tests a query; returns error message or `OK` |
37
- | `ask_clarification` | `question: str` | Returns a schema/intent hint if available |
38
-
39
- ### Observation Space
40
-
41
- | Field | Type | Description |
42
- | ------------------------ | ------------------ | -------------------------------------------------- |
43
- | `task_id` | str | Unique task identifier |
44
- | `difficulty` | easy\|medium\|hard | Difficulty level |
45
- | `intent` | str | Plain-English goal for the query |
46
- | `schema_hint` | str | Compact table signatures |
47
- | `broken_query` | str | The query to fix (unchanged across turns) |
48
- | `turn` | int | Current turn number |
49
- | `max_turns` | int | Maximum turns for this difficulty |
50
- | `error_message` | str\|null | SQLite error from last run_partial / failed submit |
51
- | `clarification_response` | str\|null | Answer to last ask_clarification |
52
- | `last_action_type` | str\|null | Previous action type |
53
-
54
- ### Reward Function
55
-
56
- ```
57
- submit_query:
58
- jaccard == 1.0 → +1.0 (perfect)
59
- jaccard >= 0.8 → +0.5 (close)
60
- jaccard >= 0.5 → +0.2 (partial)
61
- jaccard < 0.5 → -0.5 (wrong)
62
- minus turn_penalty = 0.05 × (turns_used - 1)
63
- clamped to [-1.0, 1.0]
64
-
65
- run_partial:
66
- no SQL error → +0.05
67
- SQL error → -0.10
68
-
69
- ask_clarification:
70
- answer found → 0.00
71
- no answer → -0.05
72
- ```
73
-
74
- ### Grader Formula
75
-
76
- ```
77
- score = 0.60 × correctness
78
- + 0.25 × turn_efficiency
79
- + 0.15 × clarification_quality
80
-
81
- correctness = final Jaccard score
82
- turn_efficiency = 1 - (turns_used / max_turns)
83
- clarification_quality = useful_hits / total_asks (1.0 if no asks)
84
- ```
85
-
86
- All scores are deterministic and in `[0.0, 1.0]`.
87
-
88
- ---
89
-
90
- ## Tasks
91
-
92
- ### Easy (15 tasks, max 3 turns)
93
-
94
- - Single table (`users`)
95
- - Error types: syntax errors, wrong operators, missing keywords
96
- - Examples: `COUNTS(*)` → `COUNT(*)`, `ORDER age` → `ORDER BY age`
97
-
98
- ### Medium (15 tasks, max 5 turns)
99
-
100
- - Two tables (`users + orders` or `orders + products`)
101
- - Error types: wrong join keys, `WHERE` vs `HAVING`, missing `LEFT JOIN`
102
- - Examples: join on `orders.id` instead of `orders.user_id`
103
-
104
- ### Hard (10 tasks, max 8 turns)
105
-
106
- - Three to five tables with CTEs and subqueries
107
- - Error types: nested aggregates, correlated subqueries, multi-table joins
108
- - Designed so naive first-attempt strategies score ≤ 0.40
109
- - Examples: `AVG(SUM(...))` requires CTE, tag intersection requires self-join
110
-
111
- ---
112
-
113
- ## API Reference
114
-
115
- | Endpoint | Method | Description |
116
- | --------- | ------ | ------------------------------- |
117
- | `/health` | GET | Liveness probe |
118
- | `/tasks` | GET | List all 40 tasks |
119
- | `/reset` | POST | Start a new episode |
120
- | `/step` | POST | Take one action |
121
- | `/state` | GET | Current episode state |
122
- | `/grader` | GET | Final score for current episode |
123
-
124
- ### POST /reset
125
-
126
- ```json
127
- {
128
- "task_id": "medium_001",
129
- "difficulty": "medium"
130
- }
131
- ```
132
-
133
- ### POST /step — submit_query
134
-
135
- ```json
136
- {
137
- "type": "submit_query",
138
- "query": "SELECT DISTINCT u.name FROM users u JOIN orders o ON u.id = o.user_id"
139
- }
140
- ```
141
-
142
- ### POST /step — run_partial
143
-
144
- ```json
145
- {
146
- "type": "run_partial",
147
- "query": "SELECT * FROM users WHERE age > 30"
148
- }
149
- ```
150
-
151
- ### POST /step — ask_clarification
152
-
153
- ```json
154
- {
155
- "type": "ask_clarification",
156
- "question": "what column in orders links to users?"
157
- }
158
- ```
159
-
160
- ---
161
-
162
- ## Setup & Running
163
-
164
- ### Local (Python)
165
-
166
- ```bash
167
- pip install -r requirements.txt
168
- uvicorn server.app:app --host 0.0.0.0 --port 7860
169
- ```
170
-
171
- ### Docker
172
-
173
- ```bash
174
- docker build -t sql-repair-env .
175
- docker run -p 8000:8000 sql-repair-env
176
- ```
177
-
178
- Test it:
179
-
180
- ```bash
181
- curl http://localhost:7860/health
182
- curl http://localhost:7860/tasks
183
- curl -X POST http://localhost:7860/reset \
184
- -H "Content-Type: application/json" \
185
- -d '{"task_id": "easy_001", "difficulty": "easy"}'
186
- ```
187
-
188
- ---
189
-
190
- ## Running the Baseline
191
-
192
- ```bash
193
- export API_BASE_URL="https://router.huggingface.co/v1"
194
- export MODEL_NAME="Qwen/Qwen2.5-72B-Instruct"
195
- export HF_TOKEN="your-token-here"
196
- export ENV_SERVER_URL="http://localhost:7860"
197
-
198
- python inference.py
199
- ```
200
-
201
- Expected baseline scores (approximate):
202
-
203
- | Difficulty | Score range |
204
- | ---------- | ----------- |
205
- | Easy | 0.55 – 0.80 |
206
- | Medium | 0.30 – 0.55 |
207
- | Hard | 0.10 – 0.30 |
208
-
209
- ---
210
-
211
- ## Project Structure
212
-
213
- ```
214
- sql-repair-env/
215
- ├── app/
216
- │ ├── models.py # All Pydantic types
217
- │ ├── env.py # Core environment logic
218
- │ ├── executor.py # SQLite execution layer
219
- │ ├── grader.py # Deterministic scoring
220
- │ └── data_loader.py # Dataset loader (module singleton)
221
- ├── data/
222
- │ ├── easy.json # 15 easy tasks
223
- │ ├── medium.json # 15 medium tasks
224
- │ └── hard.json # 10 hard tasks
225
- ├── db/
226
- │ └── fixtures.sql # SQLite seed data
227
- ├── server/
228
- │ └── app.py # FastAPI server
229
- ├── inference.py # Baseline inference script
230
- ├── openenv.yaml # OpenEnv spec
231
- ├── Dockerfile
232
- ├── requirements.txt
233
- └── README.md
234
- ```
235
-
236
- ---
237
-
238
- ## Design Decisions
239
-
240
- **Why SQLite in-process?**
241
- Zero external infrastructure — the DB seeds from `fixtures.sql` on every
242
- `reset()`, guaranteeing a clean deterministic state per episode.
243
-
244
- **Why Jaccard similarity?**
245
- Row order is undefined in SQL; Jaccard treats results as sets, making grading
246
- robust to `ORDER BY` differences between the agent's query and the reference.
247
-
248
- **Why deny actual row data to the agent?**
249
- Showing only error messages forces the agent to reason about the query
250
- structure, not memorise the dataset. This better simulates real-world
251
- debugging where production data isn't available.
252
-
253
- **Why three action types?**
254
- `run_partial` + `ask_clarification` before `submit_query` models real
255
- developer behaviour: check syntax, check schema, then commit. The turn
256
- penalty in the reward function ensures the agent learns to use these
257
- efficiently rather than exhausting the budget.