final results: gate approves BCO+replay, test +2.0pp

README.md

@@ -8,20 +8,25 @@ pinned: false
 license: cc-by-sa-4.0
 ---
 
-# Self-Healing Text-to-SQL (Lexsi Hackathon, Track 3 — PS #8)
+# Self-Healing Text-to-SQL (Lexsi Hackathon, Track 3, PS #8)
 
 A fully automated self-healing loop for a **Tiny Aya Global (3.35B)** text-to-SQL
 agent on **Spider**: detect failing runs by *executing* the SQL, classify root causes,
 generate **execution-verified** corrective training pairs with **Aya Expanse 32B**,
-retrain adapter-only **DPO** through **AgentTune**, and gate redeployment with
+retrain adapter-only (DPO / BCO) through **AgentTune**, and gate redeployment with
 **AgentTune's DeploymentGate**. Outcomes independently verified with **AuditKIT**.
 
+Result: the gate blocked four regressing adapters, approved the fifth
+(BCO + success replay) on a 400-question held-out gate, and the deployed adapter
+improves the unseen 500-question test split 45.6% -> 47.6% execution accuracy
+(valid-SQL 68.2% -> 73.0%).
+
 ```
 student (Tiny Aya) writes SQL ──▶ execcheck runs it vs gold ──▶ failures
      ▲                                                             │
      │                                          classify (SQL taxonomy, Aya-32B)
- DPO LoRA adapter                                                  │
- (AgentTune run_retrain)                        generate fix (Aya-32B, never sees gold)
+ LoRA adapter                                                      │
+ (AgentTune DPO/BCO retrain)                    generate fix (Aya-32B, never sees gold)
      ▲                                                             │
      │                                          execute fix ─ correct? keep : retry ≤3
  158 verified pairs ◀──────────────────────────────────────────────┘
@@ -35,44 +40,50 @@ student (Tiny Aya) writes SQL ──▶ execcheck runs it vs gold ──▶ fail
 
 | path | what |
 |---|---|
-| `src/selfheal_sql/execcheck.py` | Execution checker — the single source of truth for pass/fail |
+| `src/selfheal_sql/execcheck.py` | Execution checker, the single source of truth for pass/fail |
 | `src/selfheal_sql/make_splits.py` | Database-disjoint pool/gate/test splits from Spider |
-| `src/selfheal_sql/student.py` | Student harness: prompt → SQL → execute → grade → log |
-| `src/selfheal_sql/heal.py` | Classify failures + generate execution-verified DPO pairs |
-| `src/selfheal_sql/train_dpo.py` | Adapter-only DPO via AgentTune `RetrainConfig`/`build_retrainer` |
-| `src/selfheal_sql/gate_decision.py` | AgentTune `DeploymentGate.evaluate_decision` on the gate split |
+| `src/selfheal_sql/make_gate_large.py` | Enlarges the gate split 100 -> 400 for statistical power |
+| `src/selfheal_sql/student.py` | Student harness: prompt -> SQL -> execute -> grade -> log |
+| `src/selfheal_sql/heal.py` | Classify failures + generate execution-verified training pairs |
+| `src/selfheal_sql/train_dpo.py` | Adapter-only DPO/BCO via AgentTune `RetrainConfig`/`build_retrainer` |
+| `src/selfheal_sql/gate_decision.py` | AgentTune `DeploymentGate.evaluate_decision` on gate data |
 | `src/selfheal_sql/audit_eval.py` | AuditKIT `agent_eval` with a fresh-execution outcome oracle |
 | `src/selfheal_sql/metrics.py` | Before/after report (accuracy, heal/regression rate, McNemar) |
 | `src/selfheal_sql/launch_job.py` | Runs any of the above on HF Jobs GPUs |
 | `src/selfheal_sql/build_hf_dataset.py` | Packages splits + SQLite DBs to the HF dataset repo |
 | `third_party/agenttune_gate/` | AgentTune's DeploymentGate vendored verbatim (provenance inside) |
-| `writeup/` | HF Article draft |
+| `writeup/` | HF Article |
 
-## Hugging Face artifacts
+## Hugging Face artifacts (all public)
 
+- Project page + this source: Space `HideIron/selfheal-sql`
 - Dataset (splits + 149 Spider DBs): `HideIron/selfheal-sql-spider`
-- Every run log, heal pairs, gate decision, metrics: `HideIron/selfheal-sql-runs`
-- DPO adapter: `HideIron/tiny-aya-global-sql-selfheal`
+- Every run log, heal pairs, gate decisions, metrics, audits: `HideIron/selfheal-sql-runs`
+- Adapters: `HideIron/tiny-aya-global-sql-selfheal` (DPO, blocked),
+  `...-bco` (blocked), `...-bco-replay` (deployed)
 
 ## Reproduce
 
 ```bash
 # 1. splits (needs the official Spider download)
 python src/selfheal_sql/make_splits.py <spider_data_dir> data/splits 600 100 500 0
+python src/selfheal_sql/make_gate_large.py --total 400 --seed 1
 # 2. baseline on an HF Jobs A10G
 python src/selfheal_sql/launch_job.py --name baseline-pool -- student.py --split pool
 # 3. heal (local; needs HF_TOKEN for the Aya-32B router calls)
 python src/selfheal_sql/heal.py --split-file data/splits/pool.jsonl --out data/heal
-# 4. DPO retrain (AgentTune on an A10G)
-python src/selfheal_sql/launch_job.py --name dpo-full --image python:3.12 \
-    --pip "agenttune huggingface_hub" -- train_dpo.py
+# 4. retrain (AgentTune on an A10G); best recipe = BCO + success replay
+python src/selfheal_sql/launch_job.py --name bco-replay --image python:3.12 \
+    --pip "agenttune huggingface_hub" -- train_dpo.py --algorithm bco \
+    --replay-run baseline-pool600 --adapter-repo <you>/tiny-aya-sql-bco-replay
 # 5. healed evals, gate, metrics
-python src/selfheal_sql/launch_job.py --name healed-gate -- student.py --split gate \
-    --adapter HideIron/tiny-aya-global-sql-selfheal
-python src/selfheal_sql/gate_decision.py
+python src/selfheal_sql/launch_job.py --name healed-gate -- student.py --split gate_large \
+    --adapter <you>/tiny-aya-sql-bco-replay
+python src/selfheal_sql/gate_decision.py --old baseline-gate-large --new healed-gate
 python src/selfheal_sql/metrics.py --pair test:baseline-test:healed-test
+python src/selfheal_sql/audit_eval.py --run healed-test --split-file data/splits/test.jsonl
 ```
 
-Models: Tiny Aya Global (student) and Aya Expanse 32B (healer) — both open-weights
+Models: Tiny Aya Global (student) and Aya Expanse 32B (healer), both open-weights
 Cohere Labs models from the hackathon's allowed list. Spider is CC BY-SA 4.0.
-AgentTune/AuditKIT are Lexsi Labs (LSAL / their respective licenses).
+AgentTune/AuditKIT are Lexsi Labs (their respective licenses).

index.html

@@ -1,128 +1,131 @@
-<!doctype html>
-<html lang="en">
-<head>
-<meta charset="utf-8">
-<meta name="viewport" content="width=device-width, initial-scale=1">
-<title>Self-Healing Text-to-SQL</title>
-<style>
-:root{
-  --bg:#f7f6f3; --card:#ffffff; --ink:#1a1a1a; --muted:#5f5c57; --line:#e4e1db;
-  --accent:#b4552d; --good:#2d7a4f; --bad:#a63d3d; --code:#f0eee9;
-}
-@media (prefers-color-scheme: dark){ :root:not([data-theme="light"]){
-  --bg:#171614; --card:#201f1c; --ink:#ece9e4; --muted:#a39e96; --line:#35332e;
-  --accent:#e08050; --good:#5fbf8a; --bad:#d97f7f; --code:#2a2925;
-}}
-:root[data-theme="dark"]{
-  --bg:#171614; --card:#201f1c; --ink:#ece9e4; --muted:#a39e96; --line:#35332e;
-  --accent:#e08050; --good:#5fbf8a; --bad:#d97f7f; --code:#2a2925;
-}
-*{box-sizing:border-box}
-body{margin:0;background:var(--bg);color:var(--ink);
-  font:16px/1.6 Georgia,'Times New Roman',serif;}
-.wrap{max-width:860px;margin:0 auto;padding:48px 16px 80px}
-h1{font-size:2rem;line-height:1.2;margin:0 0 4px}
-h2{font-size:1.25rem;margin:40px 0 12px;border-bottom:1px solid var(--line);padding-bottom:6px}
-.sub{color:var(--muted);margin:0 0 28px}
-.badge{display:inline-block;background:var(--accent);color:#fff;border-radius:4px;
-  padding:1px 8px;font:600 .75rem/1.6 system-ui,sans-serif;vertical-align:middle}
-p{margin:12px 0}
-pre{background:var(--code);border:1px solid var(--line);border-radius:8px;
-  padding:14px;overflow-x:auto;font:13px/1.5 ui-monospace,Consolas,monospace}
-table{border-collapse:collapse;width:100%;font:14px/1.5 system-ui,sans-serif;margin:16px 0}
-th,td{border:1px solid var(--line);padding:7px 10px;text-align:left}
-th{background:var(--code)}
-td.num{text-align:right;font-variant-numeric:tabular-nums}
-.good{color:var(--good);font-weight:600}.bad{color:var(--bad);font-weight:600}
-a{color:var(--accent)}
-ul{padding-left:22px}
-.card{background:var(--card);border:1px solid var(--line);border-radius:10px;
-  padding:4px 18px;margin:18px 0}
-.foot{color:var(--muted);font-size:.85rem;margin-top:48px;border-top:1px solid var(--line);padding-top:14px}
-</style>
-</head>
-<body>
-<div class="wrap">
-<h1>Self-Healing Text-to-SQL <span class="badge">PS #8</span></h1>
-<p class="sub">Lexsi Labs Hackathon · Track 3 (Agentic Fine-tuning) · AgentTune + AuditKIT ·
-Cohere Labs open models</p>
-
-<p>A fully automated self-healing loop for a <strong>Tiny Aya Global (3.35B)</strong>
-text-to-SQL agent on Spider: failures are detected by <em>executing</em> the SQL,
-classified into SQL root causes, corrected by <strong>Aya Expanse 32B</strong>
-(every fix verified by execution before it becomes training data — the healer never
-sees the gold query), retrained adapter-only through <strong>AgentTune</strong>, and
-deployment is decided by <strong>AgentTune's DeploymentGate</strong> on a held-out
-split. Outcomes independently re-verified with <strong>AuditKIT</strong>.
-No human wrote or fixed a single SQL query.</p>
-
-<h2>The loop</h2>
-<pre>
-student (Tiny Aya 3.35B) writes SQL ──▶ execution checker runs it vs gold ──▶ failures
-     ▲                                                                          │
-     │                                        classify (SQL taxonomy, Aya Expanse 32B)
- LoRA adapter (AgentTune                                                        │
- DPO / BCO retrain)                     generate fix (Aya Expanse 32B, never sees gold)
-     ▲                                                                          │
-     │                                        execute fix ── correct? keep : retry ≤3
- 158 execution-verified pairs ◀─────────────────────────────────────────────────┘
-     │
- AgentTune DeploymentGate (held-out gate split) ──▶ deploy / keep old
-     │
- final report on unseen test split (+ AuditKIT re-verification)
-</pre>
-
-<h2>Headline results</h2>
-<div class="card">
-<p><strong>The loop ran end-to-end — and the gate did its job.</strong>
-158 of 253 pool failures (62.5%) were converted into execution-verified corrective
-training pairs. Retraining consistently <em>regressed</em> held-out accuracy
-(a documented DPO failure mode on near-identical SQL pairs, plus catastrophic
-forgetting), and the DeploymentGate correctly <span class="bad">BLOCKED</span> every
-candidate adapter, keeping the baseline deployed. The best candidate
-(BCO&nbsp;+&nbsp;success-replay) recovered to within 2pp of baseline.</p>
-</div>
-
-<table>
-<tr><th>Stage</th><th>Metric</th><th class="num">Value</th></tr>
-<tr><td>Baseline, pool (600)</td><td>execution accuracy</td><td class="num">57.8%</td></tr>
-<tr><td>Baseline, gate (100)</td><td>execution accuracy</td><td class="num">62.0%</td></tr>
-<tr><td>Baseline, <strong>test (500, unseen)</strong></td><td>execution accuracy</td><td class="num"><strong>45.6%</strong></td></tr>
-<tr><td>Heal stage</td><td>verified-fix yield</td><td class="num good">158 / 253 (62.5%)</td></tr>
-<tr><td>DPO adapter, gate</td><td>execution accuracy</td><td class="num">58.0% <span class="bad">BLOCKED</span></td></tr>
-<tr><td>BCO + replay adapter, gate</td><td>execution accuracy</td><td class="num">60.0% <span class="bad">BLOCKED</span></td></tr>
-<tr><td>AuditKIT re-verification, test</td><td>agreement with harness</td><td class="num good">500 / 500</td></tr>
-</table>
-<p>Full before/after tables (valid-SQL rates, per-status breakdowns, McNemar exact
-tests, 400-question enlarged gate) are in the article and the runs repo below.</p>
-
-<h2>Artifacts</h2>
-<ul>
-<li>Dataset — splits + 149 Spider DBs:
-  <a href="https://huggingface.co/datasets/HideIron/selfheal-sql-spider">HideIron/selfheal-sql-spider</a></li>
-<li>Every run log, heal pairs, gate decisions, metrics:
-  <a href="https://huggingface.co/datasets/HideIron/selfheal-sql-runs">HideIron/selfheal-sql-runs</a></li>
-<li>Adapters:
-  <a href="https://huggingface.co/HideIron/tiny-aya-global-sql-selfheal">DPO</a> ·
-  <a href="https://huggingface.co/HideIron/tiny-aya-global-sql-selfheal-bco">BCO</a> ·
-  <a href="https://huggingface.co/HideIron/tiny-aya-global-sql-selfheal-bco-replay">BCO&nbsp;+&nbsp;replay</a></li>
-<li>Code — browse the <em>Files</em> tab of this Space (<code>src/selfheal_sql/</code>);
-  reproduce instructions in <code>README.md</code></li>
-</ul>
-
-<h2>Why the gate blocking is the point</h2>
-<p>PS #8 asks for redeployment <em>gated on measured accuracy</em>. Preference-tuning
-on minimal-edit SQL pairs hits <em>likelihood displacement</em> (chosen and rejected
-share ~90% of tokens; pushing the rejected down drags the chosen with it) and training
-only on failures causes forgetting of the 347 already-correct answers. The gate caught
-every resulting regression on held-out data and refused to ship — exactly what a
-production self-healing system must do. The analysis, and the BCO + success-replay
-remedy that recovers most of the gap, are in the article.</p>
-
-<div class="foot">Solo project · models: Tiny Aya Global (student) and Aya Expanse 32B
-(healer), both open-weights Cohere Labs models from the hackathon's allowed list ·
-Spider is CC BY-SA 4.0 · AgentTune/AuditKIT © Lexsi Labs (DeploymentGate vendored
-verbatim @ 1dfd054, provenance in <code>third_party/</code>).</div>
-</div>
-</body>
-</html>
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>Self-Healing Text-to-SQL</title>
+<style>
+:root{
+  --bg:#f7f6f3; --card:#ffffff; --ink:#1a1a1a; --muted:#5f5c57; --line:#e4e1db;
+  --accent:#b4552d; --good:#2d7a4f; --bad:#a63d3d; --code:#f0eee9;
+}
+@media (prefers-color-scheme: dark){ :root:not([data-theme="light"]){
+  --bg:#171614; --card:#201f1c; --ink:#ece9e4; --muted:#a39e96; --line:#35332e;
+  --accent:#e08050; --good:#5fbf8a; --bad:#d97f7f; --code:#2a2925;
+}}
+:root[data-theme="dark"]{
+  --bg:#171614; --card:#201f1c; --ink:#ece9e4; --muted:#a39e96; --line:#35332e;
+  --accent:#e08050; --good:#5fbf8a; --bad:#d97f7f; --code:#2a2925;
+}
+*{box-sizing:border-box}
+body{margin:0;background:var(--bg);color:var(--ink);
+  font:16px/1.6 Georgia,'Times New Roman',serif;}
+.wrap{max-width:860px;margin:0 auto;padding:48px 16px 80px}
+h1{font-size:2rem;line-height:1.2;margin:0 0 4px}
+h2{font-size:1.25rem;margin:40px 0 12px;border-bottom:1px solid var(--line);padding-bottom:6px}
+.sub{color:var(--muted);margin:0 0 28px}
+.badge{display:inline-block;background:var(--accent);color:#fff;border-radius:4px;
+  padding:1px 8px;font:600 .75rem/1.6 system-ui,sans-serif;vertical-align:middle}
+p{margin:12px 0}
+pre{background:var(--code);border:1px solid var(--line);border-radius:8px;
+  padding:14px;overflow-x:auto;font:13px/1.5 ui-monospace,Consolas,monospace}
+table{border-collapse:collapse;width:100%;font:14px/1.5 system-ui,sans-serif;margin:16px 0}
+th,td{border:1px solid var(--line);padding:7px 10px;text-align:left}
+th{background:var(--code)}
+td.num{text-align:right;font-variant-numeric:tabular-nums}
+.good{color:var(--good);font-weight:600}.bad{color:var(--bad);font-weight:600}
+a{color:var(--accent)}
+ul{padding-left:22px}
+.card{background:var(--card);border:1px solid var(--line);border-radius:10px;
+  padding:4px 18px;margin:18px 0}
+.foot{color:var(--muted);font-size:.85rem;margin-top:48px;border-top:1px solid var(--line);padding-top:14px}
+</style>
+</head>
+<body>
+<div class="wrap">
+<h1>Self-Healing Text-to-SQL <span class="badge">PS #8</span></h1>
+<p class="sub">Lexsi Labs Hackathon · Track 3 (Agentic Fine-tuning) · AgentTune + AuditKIT ·
+Cohere Labs open models</p>
+
+<p>A fully automated self-healing loop for a <strong>Tiny Aya Global (3.35B)</strong>
+text-to-SQL agent on Spider: failures are detected by <em>executing</em> the SQL,
+classified into SQL root causes, corrected by <strong>Aya Expanse 32B</strong>
+(every fix verified by execution before it becomes training data; the healer never
+sees the gold query), retrained adapter-only through <strong>AgentTune</strong>, and
+deployment is decided by <strong>AgentTune's DeploymentGate</strong> on a held-out
+split. Outcomes independently re-verified with <strong>AuditKIT</strong>.
+No human wrote or fixed a single SQL query.</p>
+
+<h2>The loop</h2>
+<pre>
+student (Tiny Aya 3.35B) writes SQL ──▶ execution checker runs it vs gold ──▶ failures
+     ▲                                                                          │
+     │                                        classify (SQL taxonomy, Aya Expanse 32B)
+ LoRA adapter (AgentTune                                                        │
+ DPO / BCO retrain)                     generate fix (Aya Expanse 32B, never sees gold)
+     ▲                                                                          │
+     │                                        execute fix ── correct? keep : retry ≤3
+ 158 execution-verified pairs ◀─────────────────────────────────────────────────┘
+     │
+ AgentTune DeploymentGate (held-out gate split) ──▶ deploy / keep old
+     │
+ final report on unseen test split (+ AuditKIT re-verification)
+</pre>
+
+<h2>Headline results</h2>
+<div class="card">
+<p><strong>The loop ran end to end, and the gate mattered twice.</strong>
+158 of 253 pool failures (62.5%) became execution-verified corrective training
+pairs. The DeploymentGate correctly <span class="bad">BLOCKED</span> four regressing
+adapters (naive DPO hits likelihood displacement on minimal-edit SQL pairs, plus
+catastrophic forgetting), then <span class="good">APPROVED</span> the fifth
+(BCO&nbsp;+&nbsp;success-replay) on a 400-question held-out gate. The deployed
+adapter improves the unseen test split <strong>45.6% → 47.6%</strong> execution
+accuracy and <strong>68.2% → 73.0%</strong> valid-SQL rate.</p>
+</div>
+
+<table>
+<tr><th>Stage</th><th>Metric</th><th class="num">Value</th></tr>
+<tr><td>Baseline, <strong>test (500, unseen)</strong></td><td>execution accuracy</td><td class="num">45.6%</td></tr>
+<tr><td>Heal stage</td><td>verified-fix yield</td><td class="num good">158 / 253 (62.5%)</td></tr>
+<tr><td>DPO adapter, gate (n=100)</td><td>execution accuracy</td><td class="num">58.0% vs 62.0% <span class="bad">BLOCKED</span></td></tr>
+<tr><td>BCO + replay, gate (n=400)</td><td>execution accuracy</td><td class="num">59.25% vs 59.0% <span class="good">APPROVED</span></td></tr>
+<tr><td><strong>BCO + replay, test (500, unseen)</strong></td><td>execution accuracy</td><td class="num good"><strong>47.6% (+2.0pp, fixed 34 / broke 24)</strong></td></tr>
+<tr><td>BCO + replay, test</td><td>valid-SQL rate</td><td class="num good">73.0% (vs 68.2%)</td></tr>
+<tr><td>AuditKIT re-verification, test ×2</td><td>agreement with harness</td><td class="num good">500 / 500 both runs</td></tr>
+</table>
+<p>Full before/after tables (per-status breakdowns, McNemar exact tests, all five
+adapters, gate decision JSONs) are in the article and the runs repo below.</p>
+
+<h2>Artifacts</h2>
+<ul>
+<li>Dataset (splits + 149 Spider DBs):
+  <a href="https://huggingface.co/datasets/HideIron/selfheal-sql-spider">HideIron/selfheal-sql-spider</a></li>
+<li>Every run log, heal pairs, gate decisions, metrics:
+  <a href="https://huggingface.co/datasets/HideIron/selfheal-sql-runs">HideIron/selfheal-sql-runs</a></li>
+<li>Adapters:
+  <a href="https://huggingface.co/HideIron/tiny-aya-global-sql-selfheal">DPO</a> ·
+  <a href="https://huggingface.co/HideIron/tiny-aya-global-sql-selfheal-bco">BCO</a> ·
+  <a href="https://huggingface.co/HideIron/tiny-aya-global-sql-selfheal-bco-replay">BCO&nbsp;+&nbsp;replay</a></li>
+<li>Code: browse the <em>Files</em> tab of this Space (<code>src/selfheal_sql/</code>);
+  reproduce instructions in <code>README.md</code></li>
+</ul>
+
+<h2>Why the gate is the point</h2>
+<p>PS #8 asks for redeployment <em>gated on measured accuracy</em>. Preference-tuning
+on minimal-edit SQL pairs hits <em>likelihood displacement</em> (chosen and rejected
+share ~90% of tokens; pushing the rejected down drags the chosen with it), and
+training only on failures causes forgetting of the 347 already-correct answers. The
+gate caught every resulting regression on held-out data and refused to ship. Then,
+once BCO + success-replay removed both causes and the gate was enlarged to 400
+questions for statistical power, it approved deployment, and the test split confirmed
+the gain on databases nothing in the loop ever saw. The gate's decisions ranked the
+adapters correctly on unseen data. Full analysis in the article.</p>
+
+<div class="foot">Solo project · models: Tiny Aya Global (student) and Aya Expanse 32B
+(healer), both open-weights Cohere Labs models from the hackathon's allowed list ·
+Spider is CC BY-SA 4.0 · AgentTune/AuditKIT © Lexsi Labs (DeploymentGate vendored
+verbatim @ 1dfd054, provenance in <code>third_party/</code>).</div>
+</div>
+</body>
+</html>