Blame for README.md

---
title: Self-Healing Text-to-SQL
emoji: 🩹
colorFrom: red
colorTo: yellow
sdk: static
pinned: false
license: cc-by-sa-4.0
---

# Self-Healing Text-to-SQL (Lexsi Hackathon, Track 3, PS #8)

A fully automated self-healing loop for a **Tiny Aya Global (3.35B)** text-to-SQL
agent on **Spider**: detect failing runs by *executing* the SQL, classify root causes,
generate **execution-verified** corrective training pairs with **Aya Expanse 32B**,
retrain adapter-only (DPO / BCO) through **AgentTune**, and gate redeployment with
**AgentTune's DeploymentGate**. Outcomes independently verified with **AuditKIT**.

Result: the gate blocked four regressing adapters, approved the fifth
(BCO + success replay) on a 400-question held-out gate, and the deployed adapter
improves the unseen 500-question test split 45.6% -> 47.6% execution accuracy
(valid-SQL 68.2% -> 73.0%).

```
student (Tiny Aya) writes SQL ──▶ execcheck runs it vs gold ──▶ failures
     ▲                                                             │
     │                                          classify (SQL taxonomy, Aya-32B)
 LoRA adapter                                                      │
 (AgentTune DPO/BCO retrain)                    generate fix (Aya-32B, never sees gold)
     ▲                                                             │
     │                                          execute fix ─ correct? keep : retry ≤3
 158 verified pairs ◀──────────────────────────────────────────────┘
     │
 DeploymentGate (AgentTune, held-out gate split) ──▶ deploy / keep old
     │
 final report on unseen test split (+ AuditKIT re-verification)
```

## Layout

| path | what |
|---|---|
| `src/selfheal_sql/execcheck.py` | Execution checker, the single source of truth for pass/fail |
| `src/selfheal_sql/make_splits.py` | Database-disjoint pool/gate/test splits from Spider |
| `src/selfheal_sql/make_gate_large.py` | Enlarges the gate split 100 -> 400 for statistical power |
| `src/selfheal_sql/student.py` | Student harness: prompt -> SQL -> execute -> grade -> log |
| `src/selfheal_sql/heal.py` | Classify failures + generate execution-verified training pairs |
| `src/selfheal_sql/train_dpo.py` | Adapter-only DPO/BCO via AgentTune `RetrainConfig`/`build_retrainer` |
| `src/selfheal_sql/gate_decision.py` | AgentTune `DeploymentGate.evaluate_decision` on gate data |
| `src/selfheal_sql/audit_eval.py` | AuditKIT `agent_eval` with a fresh-execution outcome oracle |
| `src/selfheal_sql/metrics.py` | Before/after report (accuracy, heal/regression rate, McNemar) |
| `src/selfheal_sql/launch_job.py` | Runs any of the above on HF Jobs GPUs |
| `src/selfheal_sql/build_hf_dataset.py` | Packages splits + SQLite DBs to the HF dataset repo |
| `third_party/agenttune_gate/` | AgentTune's DeploymentGate vendored verbatim (provenance inside) |
| `writeup/` | HF Article |

## Hugging Face artifacts (all public)

- Project page + this source: Space `HideIron/selfheal-sql`
- Dataset (splits + 149 Spider DBs): `HideIron/selfheal-sql-spider`
- Every run log, heal pairs, gate decisions, metrics, audits: `HideIron/selfheal-sql-runs`
- Adapters: `HideIron/tiny-aya-global-sql-selfheal` (DPO, blocked),
  `...-bco` (blocked), `...-bco-replay` (deployed)

## Reproduce

```bash
# 1. splits (needs the official Spider download)
python src/selfheal_sql/make_splits.py <spider_data_dir> data/splits 600 100 500 0
python src/selfheal_sql/make_gate_large.py --total 400 --seed 1
# 2. baseline on an HF Jobs A10G
python src/selfheal_sql/launch_job.py --name baseline-pool -- student.py --split pool
# 3. heal (local; needs HF_TOKEN for the Aya-32B router calls)
python src/selfheal_sql/heal.py --split-file data/splits/pool.jsonl --out data/heal
# 4. retrain (AgentTune on an A10G); best recipe = BCO + success replay
python src/selfheal_sql/launch_job.py --name bco-replay --image python:3.12 \
    --pip "agenttune huggingface_hub" -- train_dpo.py --algorithm bco \
    --replay-run baseline-pool600 --adapter-repo <you>/tiny-aya-sql-bco-replay
# 5. healed evals, gate, metrics
python src/selfheal_sql/launch_job.py --name healed-gate -- student.py --split gate_large \
    --adapter <you>/tiny-aya-sql-bco-replay
python src/selfheal_sql/gate_decision.py --old baseline-gate-large --new healed-gate
python src/selfheal_sql/metrics.py --pair test:baseline-test:healed-test
python src/selfheal_sql/audit_eval.py --run healed-test --split-file data/splits/test.jsonl
```

Models: Tiny Aya Global (student) and Aya Expanse 32B (healer), both open-weights
Cohere Labs models from the hackathon's allowed list. Spider is CC BY-SA 4.0.
AgentTune/AuditKIT are Lexsi Labs (their respective licenses).