Spaces:
Sleeping
Sleeping
Delete READ.ME
Browse files
README.md
DELETED
|
@@ -1,104 +0,0 @@
|
|
| 1 |
-
# SQL Quality Audit Agent Environment
|
| 2 |
-
|
| 3 |
-
OpenEnv-compatible environment for training SQL data-quality agents with strict v2 trajectory scoring.
|
| 4 |
-
|
| 5 |
-
## Overview
|
| 6 |
-
|
| 7 |
-
`sql_quality_env` is a FastAPI environment with OpenEnv endpoints:
|
| 8 |
-
|
| 9 |
-
- `POST /reset`
|
| 10 |
-
- `POST /step`
|
| 11 |
-
- `GET /state`
|
| 12 |
-
- `GET /health`
|
| 13 |
-
|
| 14 |
-
Agents interact through four action types (`query`, `profile`, `fix`, `submit`), and final rewards are computed from full trajectories using a five-dimension composite grader.
|
| 15 |
-
|
| 16 |
-
## Tasks
|
| 17 |
-
|
| 18 |
-
| task_id | Difficulty | Scenario |
|
| 19 |
-
|---|---|---|
|
| 20 |
-
| `null_audit` | easy | Identify which column has >50% null rate |
|
| 21 |
-
| `type_repair` | medium | Repair numeric type mismatch while preserving values |
|
| 22 |
-
| `dedup_bug` | hard | Diagnose and fix duplication causing inflated joins |
|
| 23 |
-
| `fk_integrity` | hard | Remove orphan child rows while preserving valid FK-linked rows |
|
| 24 |
-
| `schema_drift` | medium | Canonicalize drifted upstream schemas into a normalized artifact |
|
| 25 |
-
|
| 26 |
-
## Scoring Architecture
|
| 27 |
-
|
| 28 |
-
Final score (0.0-1.0) uses five dimensions:
|
| 29 |
-
|
| 30 |
-
- D1 Diagnostic Efficiency
|
| 31 |
-
- D2 Hypothesis Quality
|
| 32 |
-
- D3 Fix Correctness (severity tiered, catastrophic override)
|
| 33 |
-
- D4 Self-Verification bonus
|
| 34 |
-
- D5 Generalization on schema/data variants
|
| 35 |
-
|
| 36 |
-
`null_audit` intentionally uses direct answer scoring:
|
| 37 |
-
|
| 38 |
-
- exact answer: `1.0`
|
| 39 |
-
- wrong but valid schema column: `0.2`
|
| 40 |
-
- out-of-schema answer: bounded lexical near-miss partial, otherwise `0.0`
|
| 41 |
-
|
| 42 |
-
## Deterministic Baseline Recipe
|
| 43 |
-
|
| 44 |
-
The baseline runner (`inference.py`) is deterministic when `BASE_SEED` is fixed.
|
| 45 |
-
|
| 46 |
-
Seed schedule:
|
| 47 |
-
|
| 48 |
-
- task seed = `BASE_SEED + task_index`
|
| 49 |
-
|
| 50 |
-
Example reproducible run:
|
| 51 |
-
|
| 52 |
-
```bash
|
| 53 |
-
uvicorn server.main:app --host 127.0.0.1 --port 7860
|
| 54 |
-
BASE_SEED=20260403 MAX_STEPS_PER_TASK=6 ENV_BASE_URL=http://127.0.0.1:7860 python inference.py
|
| 55 |
-
```
|
| 56 |
-
|
| 57 |
-
The run output is saved at:
|
| 58 |
-
|
| 59 |
-
- `baseline_seeded_run.txt`
|
| 60 |
-
|
| 61 |
-
## Measured Seeded Baseline (2026-04-03)
|
| 62 |
-
|
| 63 |
-
Model: `meta-llama/Llama-3.3-70B-Instruct`
|
| 64 |
-
API: `https://router.huggingface.co/v1`
|
| 65 |
-
Base seed: `20260403`
|
| 66 |
-
|
| 67 |
-
| Task | Seed | Score | Steps |
|
| 68 |
-
|---|---:|---:|---:|
|
| 69 |
-
| `null_audit` | 20260403 | 1.0000 | 3 |
|
| 70 |
-
| `type_repair` | 20260404 | 0.0000 | 1 |
|
| 71 |
-
| `dedup_bug` | 20260405 | 0.0000 | 1 |
|
| 72 |
-
| `fk_integrity` | 20260406 | 0.0000 | 1 |
|
| 73 |
-
| `schema_drift` | 20260407 | 0.0000 | 1 |
|
| 74 |
-
| **Average** | - | **0.2000** | - |
|
| 75 |
-
|
| 76 |
-
## Install
|
| 77 |
-
|
| 78 |
-
```bash
|
| 79 |
-
pip install -e ".[test]"
|
| 80 |
-
```
|
| 81 |
-
|
| 82 |
-
## Validate
|
| 83 |
-
|
| 84 |
-
```bash
|
| 85 |
-
python -m pytest -q
|
| 86 |
-
```
|
| 87 |
-
|
| 88 |
-
## Docker
|
| 89 |
-
|
| 90 |
-
```bash
|
| 91 |
-
docker build -t sql-quality-env:latest .
|
| 92 |
-
docker run -p 7860:7860 sql-quality-env:latest
|
| 93 |
-
```
|
| 94 |
-
|
| 95 |
-
## Environment Variables
|
| 96 |
-
|
| 97 |
-
| Variable | Required | Purpose |
|
| 98 |
-
|---|---|---|
|
| 99 |
-
| `HF_TOKEN` (or `OPENAI_API_KEY`) | yes | API auth for model calls in baseline inference |
|
| 100 |
-
| `MODEL_NAME` | yes | model id for inference |
|
| 101 |
-
| `API_BASE_URL` | no | LLM API endpoint (defaults to HF router) |
|
| 102 |
-
| `ENV_BASE_URL` | no | environment URL (defaults to local `http://localhost:7860`) |
|
| 103 |
-
| `BASE_SEED` | no | deterministic baseline seed |
|
| 104 |
-
| `MAX_STEPS_PER_TASK` | no | runtime guard per task |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|