Lexsi Labs Hackathon · Track 3 (Agentic Fine-tuning) · AgentTune + AuditKIT · Cohere Labs open models
A fully automated self-healing loop for a Tiny Aya Global (3.35B) text-to-SQL agent on Spider: failures are detected by executing the SQL, classified into SQL root causes, corrected by Aya Expanse 32B (every fix verified by execution before it becomes training data; the healer never sees the gold query), retrained adapter-only through AgentTune, and deployment is decided by AgentTune's DeploymentGate on a held-out split. Outcomes independently re-verified with AuditKIT. No human wrote or fixed a single SQL query.
student (Tiny Aya 3.35B) writes SQL ──▶ execution checker runs it vs gold ──▶ failures
▲ │
│ classify (SQL taxonomy, Aya Expanse 32B)
LoRA adapter (AgentTune │
DPO / BCO retrain) generate fix (Aya Expanse 32B, never sees gold)
▲ │
│ execute fix ── correct? keep : retry ≤3
158 execution-verified pairs ◀─────────────────────────────────────────────────┘
│
AgentTune DeploymentGate (held-out gate split) ──▶ deploy / keep old
│
final report on unseen test split (+ AuditKIT re-verification)
The loop ran end to end, and the gate mattered twice. 158 of 253 pool failures (62.5%) became execution-verified corrective training pairs. The DeploymentGate correctly BLOCKED four regressing adapters (naive DPO hits likelihood displacement on minimal-edit SQL pairs, plus catastrophic forgetting), then APPROVED the fifth (BCO + success-replay) on a 400-question held-out gate. The deployed adapter improves the unseen test split 45.6% → 47.6% execution accuracy and 68.2% → 73.0% valid-SQL rate.
| Stage | Metric | Value |
|---|---|---|
| Baseline, test (500, unseen) | execution accuracy | 45.6% |
| Heal stage | verified-fix yield | 158 / 253 (62.5%) |
| DPO adapter, gate (n=100) | execution accuracy | 58.0% vs 62.0% BLOCKED |
| BCO + replay, gate (n=400) | execution accuracy | 59.25% vs 59.0% APPROVED |
| BCO + replay, test (500, unseen) | execution accuracy | 47.6% (+2.0pp, fixed 34 / broke 24) |
| BCO + replay, test | valid-SQL rate | 73.0% (vs 68.2%) |
| AuditKIT re-verification, test ×2 | agreement with harness | 500 / 500 both runs |
Full before/after tables (per-status breakdowns, McNemar exact tests, all five adapters, gate decision JSONs) are in the article and the runs repo below.
src/selfheal_sql/);
reproduce instructions in README.mdPS #8 asks for redeployment gated on measured accuracy. Preference-tuning on minimal-edit SQL pairs hits likelihood displacement (chosen and rejected share ~90% of tokens; pushing the rejected down drags the chosen with it), and training only on failures causes forgetting of the 347 already-correct answers. The gate caught every resulting regression on held-out data and refused to ship. Then, once BCO + success-replay removed both causes and the gate was enlarged to 400 questions for statistical power, it approved deployment, and the test split confirmed the gain on databases nothing in the loop ever saw. The gate's decisions ranked the adapters correctly on unseen data. Full analysis in the article.
third_party/).