Spaces:
Running
Running
| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1"> | |
| <title>Self-Healing Text-to-SQL</title> | |
| <style> | |
| :root{ | |
| --bg:#f7f6f3; --card:#ffffff; --ink:#1a1a1a; --muted:#5f5c57; --line:#e4e1db; | |
| --accent:#b4552d; --good:#2d7a4f; --bad:#a63d3d; --code:#f0eee9; | |
| } | |
| @media (prefers-color-scheme: dark){ :root:not([data-theme="light"]){ | |
| --bg:#171614; --card:#201f1c; --ink:#ece9e4; --muted:#a39e96; --line:#35332e; | |
| --accent:#e08050; --good:#5fbf8a; --bad:#d97f7f; --code:#2a2925; | |
| }} | |
| :root[data-theme="dark"]{ | |
| --bg:#171614; --card:#201f1c; --ink:#ece9e4; --muted:#a39e96; --line:#35332e; | |
| --accent:#e08050; --good:#5fbf8a; --bad:#d97f7f; --code:#2a2925; | |
| } | |
| *{box-sizing:border-box} | |
| body{margin:0;background:var(--bg);color:var(--ink); | |
| font:16px/1.6 Georgia,'Times New Roman',serif;} | |
| .wrap{max-width:860px;margin:0 auto;padding:48px 16px 80px} | |
| h1{font-size:2rem;line-height:1.2;margin:0 0 4px} | |
| h2{font-size:1.25rem;margin:40px 0 12px;border-bottom:1px solid var(--line);padding-bottom:6px} | |
| .sub{color:var(--muted);margin:0 0 28px} | |
| .badge{display:inline-block;background:var(--accent);color:#fff;border-radius:4px; | |
| padding:1px 8px;font:600 .75rem/1.6 system-ui,sans-serif;vertical-align:middle} | |
| p{margin:12px 0} | |
| pre{background:var(--code);border:1px solid var(--line);border-radius:8px; | |
| padding:14px;overflow-x:auto;font:13px/1.5 ui-monospace,Consolas,monospace} | |
| table{border-collapse:collapse;width:100%;font:14px/1.5 system-ui,sans-serif;margin:16px 0} | |
| th,td{border:1px solid var(--line);padding:7px 10px;text-align:left} | |
| th{background:var(--code)} | |
| td.num{text-align:right;font-variant-numeric:tabular-nums} | |
| .good{color:var(--good);font-weight:600}.bad{color:var(--bad);font-weight:600} | |
| a{color:var(--accent)} | |
| ul{padding-left:22px} | |
| .card{background:var(--card);border:1px solid var(--line);border-radius:10px; | |
| padding:4px 18px;margin:18px 0} | |
| .foot{color:var(--muted);font-size:.85rem;margin-top:48px;border-top:1px solid var(--line);padding-top:14px} | |
| </style> | |
| </head> | |
| <body> | |
| <div class="wrap"> | |
| <h1>Self-Healing Text-to-SQL <span class="badge">PS #8</span></h1> | |
| <p class="sub">Lexsi Labs Hackathon · Track 3 (Agentic Fine-tuning) · AgentTune + AuditKIT · | |
| Cohere Labs open models</p> | |
| <p>A fully automated self-healing loop for a <strong>Tiny Aya Global (3.35B)</strong> | |
| text-to-SQL agent on Spider: failures are detected by <em>executing</em> the SQL, | |
| classified into SQL root causes, corrected by <strong>Aya Expanse 32B</strong> | |
| (every fix verified by execution before it becomes training data; the healer never | |
| sees the gold query), retrained adapter-only through <strong>AgentTune</strong>, and | |
| deployment is decided by <strong>AgentTune's DeploymentGate</strong> on a held-out | |
| split. Outcomes independently re-verified with <strong>AuditKIT</strong>. | |
| No human wrote or fixed a single SQL query.</p> | |
| <h2>The loop</h2> | |
| <pre> | |
| student (Tiny Aya 3.35B) writes SQL ──▶ execution checker runs it vs gold ──▶ failures | |
| ▲ │ | |
| │ classify (SQL taxonomy, Aya Expanse 32B) | |
| LoRA adapter (AgentTune │ | |
| DPO / BCO retrain) generate fix (Aya Expanse 32B, never sees gold) | |
| ▲ │ | |
| │ execute fix ── correct? keep : retry ≤3 | |
| 158 execution-verified pairs ◀─────────────────────────────────────────────────┘ | |
| │ | |
| AgentTune DeploymentGate (held-out gate split) ──▶ deploy / keep old | |
| │ | |
| final report on unseen test split (+ AuditKIT re-verification) | |
| </pre> | |
| <h2>Headline results</h2> | |
| <div class="card"> | |
| <p><strong>The loop ran end to end, and the gate mattered twice.</strong> | |
| 158 of 253 pool failures (62.5%) became execution-verified corrective training | |
| pairs. The DeploymentGate correctly <span class="bad">BLOCKED</span> four regressing | |
| adapters (naive DPO hits likelihood displacement on minimal-edit SQL pairs, plus | |
| catastrophic forgetting), then <span class="good">APPROVED</span> the fifth | |
| (BCO + success-replay) on a 400-question held-out gate. The deployed | |
| adapter improves the unseen test split <strong>45.6% → 47.6%</strong> execution | |
| accuracy and <strong>68.2% → 73.0%</strong> valid-SQL rate.</p> | |
| </div> | |
| <table> | |
| <tr><th>Stage</th><th>Metric</th><th class="num">Value</th></tr> | |
| <tr><td>Baseline, <strong>test (500, unseen)</strong></td><td>execution accuracy</td><td class="num">45.6%</td></tr> | |
| <tr><td>Heal stage</td><td>verified-fix yield</td><td class="num good">158 / 253 (62.5%)</td></tr> | |
| <tr><td>DPO adapter, gate (n=100)</td><td>execution accuracy</td><td class="num">58.0% vs 62.0% <span class="bad">BLOCKED</span></td></tr> | |
| <tr><td>BCO + replay, gate (n=400)</td><td>execution accuracy</td><td class="num">59.25% vs 59.0% <span class="good">APPROVED</span></td></tr> | |
| <tr><td><strong>BCO + replay, test (500, unseen)</strong></td><td>execution accuracy</td><td class="num good"><strong>47.6% (+2.0pp, fixed 34 / broke 24)</strong></td></tr> | |
| <tr><td>BCO + replay, test</td><td>valid-SQL rate</td><td class="num good">73.0% (vs 68.2%)</td></tr> | |
| <tr><td>AuditKIT re-verification, test ×2</td><td>agreement with harness</td><td class="num good">500 / 500 both runs</td></tr> | |
| </table> | |
| <p>Full before/after tables (per-status breakdowns, McNemar exact tests, all five | |
| adapters, gate decision JSONs) are in the article and the runs repo below.</p> | |
| <h2>Artifacts</h2> | |
| <ul> | |
| <li>Dataset (splits + 149 Spider DBs): | |
| <a href="https://huggingface.co/datasets/HideIron/selfheal-sql-spider">HideIron/selfheal-sql-spider</a></li> | |
| <li>Every run log, heal pairs, gate decisions, metrics: | |
| <a href="https://huggingface.co/datasets/HideIron/selfheal-sql-runs">HideIron/selfheal-sql-runs</a></li> | |
| <li>Adapters: | |
| <a href="https://huggingface.co/HideIron/tiny-aya-global-sql-selfheal">DPO</a> · | |
| <a href="https://huggingface.co/HideIron/tiny-aya-global-sql-selfheal-bco">BCO</a> · | |
| <a href="https://huggingface.co/HideIron/tiny-aya-global-sql-selfheal-bco-replay">BCO + replay</a></li> | |
| <li>Code: browse the <em>Files</em> tab of this Space (<code>src/selfheal_sql/</code>); | |
| reproduce instructions in <code>README.md</code></li> | |
| </ul> | |
| <h2>Why the gate is the point</h2> | |
| <p>PS #8 asks for redeployment <em>gated on measured accuracy</em>. Preference-tuning | |
| on minimal-edit SQL pairs hits <em>likelihood displacement</em> (chosen and rejected | |
| share ~90% of tokens; pushing the rejected down drags the chosen with it), and | |
| training only on failures causes forgetting of the 347 already-correct answers. The | |
| gate caught every resulting regression on held-out data and refused to ship. Then, | |
| once BCO + success-replay removed both causes and the gate was enlarged to 400 | |
| questions for statistical power, it approved deployment, and the test split confirmed | |
| the gain on databases nothing in the loop ever saw. The gate's decisions ranked the | |
| adapters correctly on unseen data. Full analysis in the article.</p> | |
| <div class="foot">Solo project · models: Tiny Aya Global (student) and Aya Expanse 32B | |
| (healer), both open-weights Cohere Labs models from the hackathon's allowed list · | |
| Spider is CC BY-SA 4.0 · AgentTune/AuditKIT © Lexsi Labs (DeploymentGate vendored | |
| verbatim @ 1dfd054, provenance in <code>third_party/</code>).</div> | |
| </div> | |
| </body> | |
| </html> | |