README.md

metadata

title: Self-Healing Text-to-SQL
emoji: 🩹
colorFrom: red
colorTo: yellow
sdk: static
pinned: false
license: cc-by-sa-4.0

Self-Healing Text-to-SQL (Lexsi Hackathon, Track 3, PS #8)

A fully automated self-healing loop for a Tiny Aya Global (3.35B) text-to-SQL agent on Spider: detect failing runs by executing the SQL, classify root causes, generate execution-verified corrective training pairs with Aya Expanse 32B, retrain adapter-only (DPO / BCO) through AgentTune, and gate redeployment with AgentTune's DeploymentGate. Outcomes independently verified with AuditKIT.

Result: the gate blocked four regressing adapters, approved the fifth (BCO + success replay) on a 400-question held-out gate, and the deployed adapter improves the unseen 500-question test split 45.6% -> 47.6% execution accuracy (valid-SQL 68.2% -> 73.0%).

student (Tiny Aya) writes SQL ──▶ execcheck runs it vs gold ──▶ failures
     ▲                                                             │
     │                                          classify (SQL taxonomy, Aya-32B)
 LoRA adapter                                                      │
 (AgentTune DPO/BCO retrain)                    generate fix (Aya-32B, never sees gold)
     ▲                                                             │
     │                                          execute fix ─ correct? keep : retry ≤3
 158 verified pairs ◀──────────────────────────────────────────────┘
     │
 DeploymentGate (AgentTune, held-out gate split) ──▶ deploy / keep old
     │
 final report on unseen test split (+ AuditKIT re-verification)

Layout

path	what
src/selfheal_sql/execcheck.py	Execution checker, the single source of truth for pass/fail
src/selfheal_sql/make_splits.py	Database-disjoint pool/gate/test splits from Spider
src/selfheal_sql/make_gate_large.py	Enlarges the gate split 100 -> 400 for statistical power
src/selfheal_sql/student.py	Student harness: prompt -> SQL -> execute -> grade -> log
src/selfheal_sql/heal.py	Classify failures + generate execution-verified training pairs
src/selfheal_sql/train_dpo.py	Adapter-only DPO/BCO via AgentTune RetrainConfig/build_retrainer
src/selfheal_sql/gate_decision.py	AgentTune DeploymentGate.evaluate_decision on gate data
src/selfheal_sql/audit_eval.py	AuditKIT agent_eval with a fresh-execution outcome oracle
src/selfheal_sql/metrics.py	Before/after report (accuracy, heal/regression rate, McNemar)
src/selfheal_sql/launch_job.py	Runs any of the above on HF Jobs GPUs
src/selfheal_sql/build_hf_dataset.py	Packages splits + SQLite DBs to the HF dataset repo
third_party/agenttune_gate/	AgentTune's DeploymentGate vendored verbatim (provenance inside)
writeup/	HF Article

Hugging Face artifacts (all public)

• Project page + this source: Space HideIron/selfheal-sql
• Dataset (splits + 149 Spider DBs): HideIron/selfheal-sql-spider
• Every run log, heal pairs, gate decisions, metrics, audits: HideIron/selfheal-sql-runs
• Adapters: HideIron/tiny-aya-global-sql-selfheal (DPO, blocked), ...-bco (blocked), ...-bco-replay (deployed)

Reproduce

# 1. splits (needs the official Spider download)
python src/selfheal_sql/make_splits.py <spider_data_dir> data/splits 600 100 500 0
python src/selfheal_sql/make_gate_large.py --total 400 --seed 1
# 2. baseline on an HF Jobs A10G
python src/selfheal_sql/launch_job.py --name baseline-pool -- student.py --split pool
# 3. heal (local; needs HF_TOKEN for the Aya-32B router calls)
python src/selfheal_sql/heal.py --split-file data/splits/pool.jsonl --out data/heal
# 4. retrain (AgentTune on an A10G); best recipe = BCO + success replay
python src/selfheal_sql/launch_job.py --name bco-replay --image python:3.12 \
    --pip "agenttune huggingface_hub" -- train_dpo.py --algorithm bco \
    --replay-run baseline-pool600 --adapter-repo <you>/tiny-aya-sql-bco-replay
# 5. healed evals, gate, metrics
python src/selfheal_sql/launch_job.py --name healed-gate -- student.py --split gate_large \
    --adapter <you>/tiny-aya-sql-bco-replay
python src/selfheal_sql/gate_decision.py --old baseline-gate-large --new healed-gate
python src/selfheal_sql/metrics.py --pair test:baseline-test:healed-test
python src/selfheal_sql/audit_eval.py --run healed-test --split-file data/splits/test.jsonl

Models: Tiny Aya Global (student) and Aya Expanse 32B (healer), both open-weights Cohere Labs models from the hackathon's allowed list. Spider is CC BY-SA 4.0. AgentTune/AuditKIT are Lexsi Labs (their respective licenses).