--- title: Self-Healing Multilingual SQL Agents on Cohere Aya emoji: πŸ›‘οΈ colorFrom: blue colorTo: indigo sdk: static app_file: index.html pinned: true license: apache-2.0 short_description: Lexsi Track 3 Technical Report & Interactive AuditKIT Traces models: - Mrunal1201/tiny-aya-sql-heal-lora - Mrunal1201/tiny-aya-sql-control-lora - Mrunal1201/aya-8b-sql-heal-lora - CohereLabs/tiny-aya-global - CohereLabs/aya-expanse-8b datasets: - Mrunal1201/aya-self-healing-sql-dataset tags: - lexsi-hackathon - agenttune - auditkit - curatorkit - text-to-sql - cohere-aya --- # Self-Healing Multilingual SQL Agents on Cohere Aya: Trajectory Distillation, SQLGlot Guardrails & AuditKIT Verification **TL;DR:** Using **`CuratorKIT 0.1.21`**, **`AgentTune 1.1.0` QLoRA Trajectory Distillation**, **`SQLGlot 30.31.0` AST Guardrails**, and **`AuditKIT 1.1.12`**, we trained multilingual (`en`, `hi`, `de`, `kn`) self-healing SQL adapters on **`CohereLabs/tiny-aya-global` (3.35B)** and **`CohereLabs/aya-expanse-8b` (8B)**. On 200 strictly disjoint held-out database tasks (`0` domain overlap, `0` table overlap), execution accuracy improved **57.5% β†’ 60.0%** (+2.5 pp final; **56.5% β†’ 60.0%** Turn-1) on 3.35B and **57.5% β†’ 63.5%** (+6.0 pp final; **57.0% β†’ 63.5%** Turn-1) on 8B, while raising executable SQL validity over single-turn SFT (`86.0%`) to **91.0%** (+5.0 pp) on 3.35B and **96.0%** (+10.0 pp) on 8Bβ€”cutting Turn-1 SQLite `OperationalError` rates by **45.5%** (`33 β†’ 18`) and **75.8%** (`33 β†’ 8`), respectively, with **96.0%** multilingual reasoning retention across 10 languages for **USD 0.50 total project R&D compute**. > **Headline Before β†’ After Summary (`AuditKIT v1.1.12`, 200 Held-Out Cases, 1,000 Bootstrap Resamples)** > - **`CohereLabs/tiny-aya-global` (3.35B):** Turn-1 Accuracy **56.5% β†’ 60.0%** (+3.5 pp) | Final Accuracy **57.5% β†’ 60.0%** (+2.5 pp, 95% CI `[52.5%, 66.01%]`) | Executable SQL Validity **86.0% (Control) β†’ 91.0%** (+5.0 pp) | Turn-1 `OperationalError` Failures **33 β†’ 18** (-45.5%) | Multilingual Retention **96.0%** (0.0 pp loss) > - **`CohereLabs/aya-expanse-8b` (8B):** Turn-1 Accuracy **57.0% β†’ 63.5%** (+6.5 pp) | Final Accuracy **57.5% β†’ 63.5%** (+6.0 pp, 95% CI `[57.0%, 70.0%]`) | Executable SQL Validity **86.0% (Control) β†’ 96.0%** (+10.0 pp) | Turn-1 `OperationalError` Failures **33 β†’ 8** (-75.8%) | Mean Query Latency **2,307.3 ms** (faster than a **2,891.0 ms** single-call cloud teacher API) | Peak VRAM **13.898 GB** **Base Models & Method:** `CohereLabs/tiny-aya-global` (3.35B) & `CohereLabs/aya-expanse-8b` (8B) via `CuratorKIT` + `AgentTune` QLoRA Trajectory Distillation (`max_seq_length=1536`, `completion_only` loss masking) + `SQLGlot` AST Guardrails + `AuditKIT` Paired Bootstrap Evaluation. | Resource | Link | | :--- | :--- | | **Public GitHub Repository** | [`Mru1201/aya-self-healing-sql-agent`](https://github.com/Mru1201/aya-self-healing-sql-agent) | | **Public HF Model Collection** | [Lexsi Track 3 β€” Self-Healing SQL Aya Models](https://huggingface.co/collections/Mrunal1201/lexsi-track-3-self-healing-sql-aya-models-6af8e7fc0a621b28fefccd18) | | **Public HF Dataset Collection** | [Lexsi Track 3 β€” Curated SQL Trajectories & Audit Logs](https://huggingface.co/collections/Mrunal1201/lexsi-track-3-curated-sql-trajectories-and-audit-logs-6af8e7fed8ac99740564a2fb) | | **Primary 3.35B Multi-Turn Adapter** | [`Mrunal1201/tiny-aya-sql-heal-lora`](https://huggingface.co/Mrunal1201/tiny-aya-sql-heal-lora) | | **3.35B Single-Turn Ablation Control** | [`Mrunal1201/tiny-aya-sql-control-lora`](https://huggingface.co/Mrunal1201/tiny-aya-sql-control-lora) | | **8B Scale-Ablation Adapter** | [`Mrunal1201/aya-8b-sql-heal-lora`](https://huggingface.co/Mrunal1201/aya-8b-sql-heal-lora) | | **Curated Dataset & Audit Logs** | [`Mrunal1201/aya-self-healing-sql-dataset`](https://huggingface.co/datasets/Mrunal1201/aya-self-healing-sql-dataset) | | **Executed Colab Notebook** | [`notebooks/train_and_audit_aya_sql.ipynb`](https://github.com/Mru1201/aya-self-healing-sql-agent/blob/main/notebooks/train_and_audit_aya_sql.ipynb) | | **HF Static Article Space** | [Static Space (`Mrunal1201/aya-self-healing-sql-article`)](https://huggingface.co/spaces/Mrunal1201/aya-self-healing-sql-article) | | **Direct Static Space Embed** | [Interactive HTML Report (`static.hf.space`)](https://mrunal1201-aya-self-healing-sql-article.static.hf.space) | | **HF Community Article (Model Repo)** | [Model Repo Discussion #1](https://huggingface.co/Mrunal1201/tiny-aya-sql-heal-lora/discussions/1) | | **HF Community Article (Dataset Repo)** | [Dataset Repo Discussion #1](https://huggingface.co/datasets/Mrunal1201/aya-self-healing-sql-dataset/discussions/1) | --- ## 1. Why Small Multilingual Database Agents Failβ€”and How We Framed Track 3 Organizations running analytics over proprietary relational databases want two things that rarely coexist: **data sovereignty** (never shipping internal DDL schemas, row samples, and SQLite error traces to external APIs) and **multilingual accessibility** (supporting analysts querying in English, Hindi, German, or Kannada). This directly targets **Lexsi Hackathon Track 3**: - **Problem Statement #6 (Data and Database Agents):** Converting natural-language questions across multiple scripts and grammars into strictly schema-grounded SQL. - **Problem Statement #8 (Self-Healing Agents):** Recovering autonomously when generated queries encounter database runtime errors rather than halting or hallucinating new identifiers. ### The Track 3 Threat Model `[T3]` Deploying an open-weights `3.35B` or `8B` model in front of a SQL database exposes two distinct risk surfaces: 1. **Schema Hallucination on Unseen Tables:** Models frequently guess plausible-sounding column or table names (`amount`, `gender`, `users_archive`) instead of binding strictly to the supplied `CREATE TABLE` DDL, causing `sqlite3.OperationalError: no such column` or `no such table`. 2. **Adversarial Prompt Injections & Destructive Writes:** Malicious or compromised user turns inject instructions attempting `DROP TABLE`, `DELETE FROM`, `UPDATE`, `ALTER TABLE`, stacked multi-statement execution (`;`), or `UNION SELECT` credential extraction against `sqlite_master`. ### Kit Architecture: `CuratorKIT` β†’ `AgentTune` β†’ `AuditKIT` with `SQLGlot` & `SQLite` Instead of relying purely on synthetic LLM chat logs or unverified string matching, we chained all three Lexsi toolkits with deterministic `SQLGlot 30.31.0` AST parsing and in-memory `SQLite 3.46.0` execution: ```mermaid flowchart LR Seed["gretelai/synthetic_text_to_sql"] --> Split["Disjoint Split: 80 Train / 20 Test Domains"] Split --> Mutate["SQLGlot 30.31.0 AST Mutation (Column / Table)"] Mutate --> SQLiteGen["In-Memory SQLite 3.46.0 -> Real OperationalError"] SQLiteGen --> Teacher["HF Router Localization: Qwen2.5-72B (en, hi, de, kn)"] Teacher --> Curator["CuratorKIT 0.1.21: SchemaGate + Exact/MinHash (0.85) + SecretsGate"] Curator --> Leak["Leak Check Gate: 0 Domain / 0 Table / 0 LSH Overlap"] Leak --> AgentTune["AgentTune 1.1.0: 4-bit QLoRA (max_seq_length=1536, completion_only)"] AgentTune --> Agent["Runtime Loop: Aya LoRA -> SQLGlot Read-Only AST Check -> SQLite"] Agent -->|OperationalError| Repair["Error Feedback -> Thought: Schema Check -> Repaired SQL"] Repair --> Agent Agent --> Audit["AuditKIT 1.1.12: Verified-Outcome Oracles & 1,000-Resample CIs"] ``` ![End-to-End Lexsi Track 3 Architecture Pipeline](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig0_architecture_pipeline.png) *Figure 0: End-to-end closed-loop Track 3 pipeline uniting `CuratorKIT 0.1.21` curation & leak checks, `AgentTune 1.1.0` QLoRA trajectory distillation, `SQLGlot 30.31.0` read-only AST guardrails over in-memory `SQLite 3.46.0`, and `AuditKIT 1.1.12` outcome verification.* Here is what an actual `self_healing` training trajectory looks like after `SQLGlot` AST mutation, `SQLite` error capture, and `CuratorKIT` validation (`id: train_0982_en_self_healing`): ````text [SYSTEM] You are a multilingual SQLite data assistant. Translate the user's question into a single read-only SQLite SELECT query using ONLY the provided schema. ... Schema (SQLite DDL): CREATE TABLE warehouse (item_name TEXT, item_type TEXT, weight INTEGER); INSERT INTO warehouse (item_name, item_type, weight) VALUES ('Carrots', 'organic_produce', 500), ...; [USER] What is the total weight of organic_produce in the warehouse table? [ASSISTANT - Turn 1 (SQLGlot AST Mutated)] ```sql SELECT SUM(weight_legacy) FROM warehouse WHERE item_type = 'organic_produce' ``` [USER - Real SQLite 3.46.0 Feedback] SQLite Execution Error (sqlite3.OperationalError): no such column: weight_legacy Please inspect the schema and fix the SQL query. [ASSISTANT - Turn 2 (Supervised Completion Only)] Thought: The previous query failed with OperationalError (no such column: weight_legacy). Checking the schema for table(s) warehouse, This query sums (SUM) the weight column in the warehouse table where item_type is 'organic_produce'. ```sql SELECT SUM(weight) FROM warehouse WHERE item_type = 'organic_produce'; ``` ```` All stack dependencies (`curatorkit 0.1.21`, `auditkit 1.1.12`, `sqlglot 30.31.0`, `datasketch 2.0.0`, `sqlite3 3.46.0`, `agenttune 1.1.0`) and HF Router endpoint connectivity (`Qwen/Qwen2.5-72B-Instruct:cheapest`, `1015.73 ms`, zero fallback calls, zero Gemini API dependencies) are logged in `audit_logs/00_stack_verified.json`. --- ## 2. Dataset Curation, Multilingual Coverage & Zero-Leakage Audit We sourced base DDL contexts and gold queries from `gretelai/synthetic_text_to_sql` (`seed=42`, curation wall-clock `328.99 s`). Rather than letting a teacher LLM hallucinate fake SQL error messages, we deterministically mutated column and table AST nodes with `SQLGlot`, ran the mutated SQL against an in-memory `sqlite3` database, and recorded the verbatim `OperationalError`. Multilingual questions (`hi`, `de`, `kn`) were localized via `Qwen/Qwen2.5-72B-Instruct:cheapest` across `109` HF Router API calls (`59,431` total tokens: `17,782` prompt + `41,649` completion; total localization cost **USD 0.05054** under a USD 0.60 cap, `0` retries, `0` fallbacks). ### `CuratorKIT 0.1.21` Hygiene & Deduplication Funnel (`01_curation_provenance.json`) `CuratorConfig` enforced `format="sharegpt"`, `min_tokens=10`, `max_tokens=1536`, `dedup="minhash"` (`threshold=0.85`, `num_perm=128`), and `secrets_gate=True` (`secrets_hex_limit=4.5`, `secrets_base64_limit=5.5`), filtering out 30 injected synthetic noise candidates in the raw pool: | Curation Stage | Train Input | Train Output | Train Filtered | Test Input | Test Output | Test Filtered | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | | Raw Candidate Pool | β€” | `1,313` | `0` | β€” | `225` | `0` | | `JSONLReader` | `1,313` | `1,313` | `0` | `225` | `225` | `0` | | `SchemaGate` | `1,313` | `1,305` | `8` (`below_min_tokens:4`) | `225` | `225` | `0` | | `ExactDeduplicator` | `1,305` | `1,299` | `6` exact duplicates | `225` | `225` | `0` | | `MinHashDeduplicator` (`0.85`, `128` perm) | `1,299` | `1,287` | `12` near-duplicates | `225` | `225` | `0` | | `TextCleaner` | `1,287` | `1,287` | `0` | `225` | `225` | `0` | | `SecretsGate` | `1,287` | `1,283` | `4` (`1` JWT, `2` Basic Auth, `1` Private Key) | `225` | `225` | `0` | | `MessagesExporter` | `1,283` | `1,283` | `0` | `225` | `225` | `0` | | **Final Stratified Split** | `1,283` | **`1,200`** | β€” | `225` | **`200`** | β€” | ### Language, Episode Type & Attack Distributions | Split | Episodes | `self_healing` | `single_turn` | `en` | `de` | `hi` | `kn` | `column_not_found` | `table_not_found` | Domains | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | **Train** | **1,200** | `780` (`65%`) | `420` (`35%`) | `840` (`70%`) | `120` (`10%`) | `120` (`10%`) | `120` (`10%`) | `504` | `276` | `80` | | **Held-Out Test** | **200** | `130` (`65%`) | `70` (`35%`) | `140` (`70%`) | `20` (`10%`) | `20` (`10%`) | `20` (`10%`) | `92` | `38` | `20` | | **Adversarial Probes** | **50** | β€” | β€” | `20` (`40%`) | `10` (`20%`) | `10` (`20%`) | `10` (`20%`) | β€” | β€” | β€” | The 50 Track 3 adversarial probes span six distinct exploit vectors: `delete_from` (`9`), `drop_table` (`9`), `alter_table` (`8`), `stacked_query` (`8`), `union_credential_extraction` (`8`), and `update_injection` (`8`). ### Zero-Leakage Guarantee (`02_leak_check_report.json`) To ensure evaluation measures true generalization to unseen schemas and domains, we enforced strict domain-level partitioning followed by cross-split deduplication checks: | Contamination Check | Train Pool | Held-Out Test Pool | Overlap Measured | Result | | :--- | :---: | :---: | :---: | :---: | | **Domain Disjointness** | `80` domains | `20` domains | **`0`** shared domains | **Passed** | | **Schema Table Overlap** | `933` unique tables | `185` unique tables | **`0`** shared tables | **Passed** | | **Exact Question Match** | `1,200` prompts | `200` prompts | **`0`** localized / **`0`** English | **Passed** | | **MinHash LSH (`threshold=0.85`, `num_perm=128`)** | `1,200` episodes | `200` episodes | **`0`** localized / **`0`** English | **Passed** | | **`CuratorKIT` Leakage Report** | `groups_in_multiple_splits` | β€” | **`0`** groups, **`0`** exact, **`0`** near-dups | **Passed** | Immutable SHA-256 checksums (`04_auditkit_before_after.json`): `train.jsonl` (`4e44f530a8a1d705bba6f36fc707e3e5e4791093fdf00c4a16437eae684b6f7d`), `test.jsonl` (`26fedb21b23c595ad6e26b75a3848d3970072cbf0fa398a605852b6117e383b5`), and `adversarial_probes.jsonl` (`aa41eaafe0e61d61b197aaa5dd682bcac0d873162f9c96664ff2344b01de156b`). --- ## 3. Main Before β†’ After Results & 95% Bootstrap Confidence Intervals Every model was evaluated with greedy decoding (`max_new_tokens=384` for SQL, `64` for multilingual arithmetic retention; 4-bit NF4 / `bfloat16` compute locally) under `AuditKIT v1.1.12` deterministic verified-outcome predicates (`FinalStateAssertion(exec_match)`, `FinalStateAssertion(injection_safe)`, and `AnswerAssertion(mode='contains')`) with `1,000` bootstrap resamples (`95%` CI, `seed=42`, zero unknown/error omissions): | Model Configuration | Turn-1 Exec Accuracy [95% CI] | Final Self-Healed Exec Accuracy [95% CI] | Executable SQL Validity Rate [95% CI] | Self-Healing Recovery (`n` T1 Errors) | Mean Turns | Adversarial Safety [95% CI] | Multilingual Retention [95% CI] | Mean Latency (`sql_heal`, ms) | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | **`tiny-aya-global` Base (3.35B)** | 56.5% `[49.5%, 63.0%]` | 57.5% `[50.5%, 64.0%]` (`115/200`) | 95.5% `[92.5%, 98.0%]` | 16.67% (`2/12`) `[0.0%, 41.67%]` | `1.06` | 24.0% `[12.0%, 36.05%]` | 96.0% `[90.0%, 100.0%]` | `443.4 ms` | | **`tiny-aya-sql-heal-lora` (3.35B)** | **60.0%** `[52.5%, 66.01%]` | **60.0%** `[52.5%, 66.01%]` (`120/200`) | **91.0%** `[87.0%, 94.5%]` | 0.0% (`0/18`) `[0.0%, 0.0%]` | `1.09` | 10.0% `[2.0%, 20.0%]` | **96.0%** `[90.0%, 100.0%]` | `4,136.4 ms` | | **`tiny-aya-sql-control-lora` (1-Turn)** | 58.5% `[52.0%, 65.5%]` | 60.0% `[53.5%, 66.51%]` (`120/200`) | 86.0% `[81.5%, 90.5%]` | 9.09% (`3/33`) `[0.0%, 21.21%]` | `1.165` | 10.0% `[2.0%, 20.0%]` | 96.0% `[90.0%, 100.0%]` | `4,450.2 ms` | | **`aya-expanse-8b` Base (8B)** | 57.0% `[50.0%, 63.5%]` | 57.5% `[50.5%, 64.0%]` (`115/200`) | 98.0% `[96.0%, 99.5%]` | 9.09% (`1/11`) `[0.0%, 27.27%]` | `1.055` | 12.0% `[4.0%, 22.0%]` | 98.0% `[94.0%, 100.0%]` | `659.6 ms` | | **`aya-8b-sql-heal-lora` (8B)** | **63.5%** `[57.0%, 70.0%]` | **63.5%** `[57.0%, 70.0%]` (`127/200`) | **96.0%** `[93.0%, 98.5%]` | 0.0% (`0/8`) `[0.0%, 0.0%]` | `1.04` | 14.0% `[6.0%, 24.0%]` | **96.0%** `[90.0%, 100.0%]` | `2,307.3 ms` | | **`Qwen2.5-72B-Instruct:cheapest` (Teacher)** | 71.0% `[65.0%, 76.5%]` | 71.0% `[65.0%, 76.5%]` (`142/200`) | 100.0% `[100.0%, 100.0%]` | 0.0% (`0/3`) `[0.0%, 0.0%]` | `1.015` | 34.0% `[21.95%, 48.0%]` | 98.0% `[94.0%, 100.0%]` | `581.8 ms` *(8 workers)* / `2,891.0 ms` *(1-call)* | ![Before vs After SQL Execution Accuracy and Validity with 95% Bootstrap CIs](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig1_accuracy_validity_ci.png) *Figure 1: Turn-1 execution accuracy, final self-healed execution accuracy, and SQLite executable validity across all 6 evaluated configurations on the 200 disjoint test queries (`1,000`-resample 95% bootstrap CIs from `audit_logs/04_auditkit_before_after.json`).* ### Paired Bootstrap Significance Tests (`auditkit._bootstrap.paired_bootstrap`) | Comparison | Suite / Metric (`n`) | Baseline β†’ Candidate | Delta | 95% Paired CI | `p`-value | | :--- | :--- | :---: | :---: | :---: | :---: | | `tuned_tiny_aya` vs. `base_tiny_aya` | `sql_heal` / `exec_match_final` (`200`) | `57.5% β†’ 60.0%` | **+2.5 pp** | `[-4.0%, +8.5%]` | `0.5085` | | `tuned_aya_8b` vs. `base_aya_8b` | `sql_heal` / `exec_match_final` (`200`) | `57.5% β†’ 63.5%` | **+6.0 pp** | `[-0.5%, +12.0%]` | `0.0709` | | `tuned_aya_8b` vs. `tuned_tiny_aya` | `sql_heal` / `exec_match_final` (`200`) | `60.0% β†’ 63.5%` | **+3.5 pp** | `[-2.5%, +10.5%]` | `0.3087` | | `tuned_tiny_aya` vs. `control_tiny_aya` | `sql_heal` / `exec_match_final` (`200`) | `60.0% β†’ 60.0%` | `0.0 pp` | `[-5.0%, +5.0%]` | `1.0000` | | `tuned_tiny_aya` vs. `teacher_api` | `sql_heal` / `exec_match_final` (`200`) | `71.0% β†’ 60.0%` | `-11.0 pp` | `[-19.0%, -3.5%]` | `0.0080` | | `tuned_tiny_aya` vs. `base_tiny_aya` | `adversarial` (`50`) | `24.0% β†’ 10.0%` | `-14.0 pp` | `[-24.0%, -6.0%]` | `0.0120` | | `tuned_aya_8b` vs. `base_aya_8b` | `adversarial` (`50`) | `12.0% β†’ 14.0%` | **+2.0 pp** | `[-6.0%, +10.05%]` | `0.7972` | --- ## 4. Dual Ablation Studies ### Ablation 1: Supervision Structure (`tiny-aya-sql-heal-lora` vs. `tiny-aya-sql-control-lora`) Does explicit multi-turn `OperationalError` trajectory supervision actually improve database agents beyond standard single-turn SQL SFT? To answer this cleanly, we compared `tiny-aya-sql-heal-lora` (trained on all `1,200` episodes: `780` `self_healing` + `420` `single_turn`, `150` steps) against `tiny-aya-sql-control-lora` (trained strictly on the `420` `single_turn` episodes, `54` steps) on the exact same `CohereLabs/tiny-aya-global` base: - **Preventative Schema Grounding Beats Post-Hoc Retries:** Conditioning final completion loss on error-diagnosis trajectories dropped Turn-1 `OperationalError` failures from **`33/200` (16.5%) down to `18/200` (9.0%)**β€”a **45.5% reduction** in runtime crashesβ€”and pushed Final Executable SQL Validity from **`86.0%` β†’ `91.0%` (+5.0 pp)**. - **Higher First-Pass & Single-Turn Precision:** `tiny-aya-sql-heal-lora` improved Turn-1 execution match (`58.5% β†’ 60.0%`, **+1.5 pp**), outperformed the control on the `single_turn` held-out subset (`n=70`: **60.0%** vs. **57.14%**, **+2.86 pp**), lowered mean turns (`1.165 β†’ 1.09`), and shaved **`313.8 ms`** off mean query latency (`4,450.2 ms β†’ 4,136.4 ms`). - **Where the Hypothesis Hit a Limit:** Because `control_tiny_aya` healed `3/33` Turn-1 errors in Turn-2 while `tuned_tiny_aya` stayed at `0/18` under greedy decoding (see Failure Modes below), both reached `60.0%` overall `exec_match_final`. Trajectory distillation clearly made the model far less likely to emit invalid SQL on Turn 1, even though greedy Turn-2 repair on remaining out-of-distribution errors requires negative feedback or broader mutation diversity. ![Supervision Structure and Scale Error Reduction Ablation](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig2_error_reduction_ablation.png) *Figure 2: (Left) Preventative schema grounding cuts Turn-1 SQLite `OperationalError` counts from `33` (`tiny-aya-sql-control-lora`) down to `18` (`-45.5%`, 3.35B) and `8` (`-75.8%`, 8B). (Right) Complete final outcome composition across all 200 held-out test cases from `audit_logs/eval_runs/`.* ### Ablation 2: Model Scale (`CohereLabs/tiny-aya-global` 3.35B vs. `CohereLabs/aya-expanse-8b` 8B) Comparing `tiny-aya-sql-heal-lora` (3.35B) against `aya-8b-sql-heal-lora` (8B) on the identical 1,200-episode curriculum revealed an unexpected systems insight: | Trade-off Metric | `tiny-aya-sql-heal-lora` (3.35B) | `aya-8b-sql-heal-lora` (8B) | | :--- | :---: | :---: | | **Final Exec Match / Executable SQL Rate** | `60.0%` / `91.0%` | **`63.5%` / `96.0%`** | | **Turn-1 `OperationalError` Count (`n=200`)** | `18` (`9.0%`) | **`8` (`4.0%`)** | | **Peak Training VRAM (Allocated)** | **`6.331 GB`** | `13.898 GB` | | **Chat Template Preamble Overhead** | `365` tokens | **`6` tokens** | | **Prompt Tokens per Query (`sql_heal`)** | `635.1` tokens | **`210.2` tokens** | | **Inference Speed (`tok/s`) / Mean Latency** | `18.95 tok/s` / `4,136.4 ms` | **`35.67 tok/s` / `2,307.3 ms`** | | **Cost per 1,000 Queries (L4 `$0.48/hr`)** | `$0.5515` | **`$0.3076`** | Counterintuitively, the **8B model ran 88.2% faster (`35.67` vs. `18.95 tok/s`) and 44.2% cheaper per 1,000 queries (`$0.3076` vs. `$0.5515`)** than the 3.35B model on an NVIDIA L4! Why? Because `tiny-aya-global`'s mandatory `365`-token system chat template triples prompt length (`635.1` vs. `210.2` tokens/query), inflating prefill and KV-cache overhead relative to `aya-expanse-8b`'s `6`-token wrapper. Meanwhile, `tiny-aya-global` remains essential whenever deployment hardware is capped at 8 GB VRAM (`6.331 GB` peak allocation). --- ## 5. Safety Guardrails, Multilingual Retention & Honest Failure Forensics ### Adversarial SQL Injection Probes (`n=50`) On the 50 multilingual adversarial probes (`drop_table`, `delete_from`, `update_injection`, `alter_table`, `stacked_query`, `union_credential_extraction`), every tested LLMβ€”including the 72B teacher (`34.0%` safe)β€”had **`refusal_or_no_sql_rate = 0.0%`**. Worse, fine-tuning 3.35B strictly on positive SQL compliance lowered raw model safety from `24.0%` (`12/50`) to `10.0%` (`5/50`, paired `p = 0.0120`), while 8B edged from `12.0%` (`6/50`) to `14.0%` (`7/50`): | Attack Vector (`n`) | `base_tiny_aya` | `tuned_tiny_aya` | `control_tiny_aya` | `base_aya_8b` | `tuned_aya_8b` | `teacher_api` | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | | `alter_table` (`8`) | `0.0%` | `0.0%` | `0.0%` | `0.0%` | `0.0%` | `25.0%` | | `delete_from` (`9`) | `11.11%` | `0.0%` | `0.0%` | `11.11%` | `0.0%` | `22.22%` | | `drop_table` (`9`) | `0.0%` | `0.0%` | `0.0%` | `0.0%` | `0.0%` | `0.0%` | | `stacked_query` (`8`) | `50.0%` | `12.5%` | `0.0%` | `12.5%` | `25.0%` | `50.0%` | | `union_credential_extraction` (`8`) | `75.0%` | `50.0%` | `62.5%` | `50.0%` | `62.5%` | `75.0%` | | `update_injection` (`8`) | `12.5%` | `0.0%` | `0.0%` | `0.0%` | `0.0%` | `37.5%` | | **Overall (`n=50`)** | **24.0%** | **10.0%** | **10.0%** | **12.0%** | **14.0%** | **34.0%** | **Takeaway:** You cannot fine-tune away SQL injection compliance with positive-only examples; deterministic runtime AST inspection (`SQLGlot` verifying single read-only `SELECT` statements + blocking `sqlite_master` and write/DDL keywords) plus a SQLite authorizer sandbox is non-negotiable in production. ![QLoRA Training Loss Convergence and Pre-Guardrail Adversarial Safety Heatmap](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig4_training_loss_and_safety.png) *Figure 4: (Left) `AgentTune 1.1.0` completion-only supervised training loss curves (`logging_steps = 5`, `03_training_hardware_meta.json`). (Right) Pre-guardrail model resistance across all 6 adversarial SQL injection vectors (`n = 50`), proving why deterministic `SQLGlot` AST filtering is essential.* ### General Multilingual Reasoning Retention (`n=50` Across 10 Languages) Across Arabic (`ar`), German (`de`), Spanish (`es`), French (`fr`), Hindi (`hi`), Japanese (`ja`), Portuguese (`pt`), Swahili (`sw`), Turkish (`tr`), and Chinese (`zh`), `tiny-aya-sql-heal-lora` and `aya-8b-sql-heal-lora` both scored **`96.0%` (`48/50`)** (`100%` on 9 of 10 languages, `60%` on `sw`), matching `base_tiny_aya` (`96.0%`, `delta = 0.0 pp`, `p = 1.0`) and remaining statistically indistinguishable from `base_aya_8b` (`98.0%`, `delta = -2.0 pp`, `p = 0.7702`). ### Forensics of Remaining Failure Modes & the Tiny Aya 366-Token Preamble 1. **Silent Semantic Mismatches (`exec_ok_wrong`):** Out of 200 test cases, `62` failures on `tuned_tiny_aya` and `65` on `tuned_aya_8b` (vs. `58` on `Qwen2.5-72B`) executed cleanly in SQLite without any syntax/schema error yet failed result-set equality. Inspecting `audit_logs/eval_runs/` shows classic semantic drift: omitting a grouping key (`test_0167_en_self_healing`: `SELECT AVG(Production_Rate)` without `Machine_ID` or `GROUP BY` for a "per machine" average), extra projection columns (`test_0049_kn_self_healing`), or ambiguous column alignment (`test_0039_en_self_healing`). Because SQLite emits no `OperationalError`, error-triggered self-healing never firesβ€”pointing directly toward future execution-result plausibility checks (e.g., empty-set or column-cardinality critics). 2. **Turn-2 Greedy Repetition on Unseen Error Types (`0/18` and `0/8`):** When `tuned_tiny_aya` or `tuned_aya_8b` hit a residual Turn-1 error such as `OperationalError: ambiguous column name: country` (`test_0143_kn_self_healing`) or `misuse of aggregate: COUNT()` (`test_0094_kn_self_healing`), the model diagnosed the error correctly in `Thought:` (`"The previous query failed with OperationalError (ambiguous column name: country)..."`) but greedily regenerated the identical broken SQL string. Adding `ambiguous_column` and `aggregate_misuse` AST mutations plus DPO negative-contrastive pairs against repeating Turn-1 SQL is the clear next step. 3. **`forced_repair` Schema Tag Mismatch (`n=0`):** `01_generate_and_curate.py` tagged multi-turn rows `"episode_type": "self_healing"`, whereas `02_colab_runner.py`'s `suite_forced_repair` queried `episode_type == "multi_turn"`. Training still ingested all `1,200` trajectories verbatim (`messages`), while the isolated `forced_repair` eval suite exited with `n_cases = 0`. 4. **Why `max_seq_length=1536` Was Mandatory for Tiny Aya:** In `03_training_hardware_meta.json`, `tiny-aya-global` measured **`365` chat template overhead tokens** (`seq_len_mean = 630.0`, `seq_len_max = 1,085`), leaving `99.9` mean supervised completion tokens (`min = 32`). Any truncation at `<= 512` tokens would slice off the Turn-2 target SQL completely (`0` supervised tokens). Setting `max_seq_length=1536` guaranteed **`0` dropped rows** across every training run. --- ## 6. Compute, Hyperparameters & Cost / ROI Economics All three QLoRA adapters were trained on **`1x NVIDIA L4` (`22.03 GB` VRAM)** using `agenttune.create_distill_trainer`, `bitsandbytes 4-bit NF4` (`double-quant`, `bfloat16` compute), `paged_adamw_8bit`, `lr=2e-4` (`cosine`, `warmup_ratio=0.03`), `2.0` epochs, `effective batch size = 16`, `gradient_checkpointing=True`, `max_seq_length=1536`, `completion_only` loss masking, and LoRA `r=16`, `lora_alpha=32`, `lora_dropout=0.05` (`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`). | Adapter | Rows (`Steps`) | Mean / Max Seq Len | Wall-Clock Time | Peak VRAM (Alloc / Resv) | Loss (Step 5 β†’ Final) | Est. Cost (`$0.48/hr`) | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | | **`tiny-aya-sql-heal-lora`** | `1,200` (`150`) | `630.0` / `1,085` | `24.283 min` | `6.331 GB` / `6.648 GB` | `1.2010 β†’ 0.3071` (`0.4081` avg) | **$0.1943** | | **`tiny-aya-sql-control-lora`** | `420` (`54`) | `562.9` / `853` | `7.326 min` | `7.176 GB` / `7.436 GB` | `1.1766 β†’ 0.5947` (`0.7045` avg) | **$0.0586** | | **`aya-8b-sql-heal-lora`** | `1,200` (`150`) | `247.3` / `684` | `24.625 min` | `13.898 GB` / `14.062 GB` | `1.5172 β†’ 0.2648` (`0.3696` avg) | **$0.1970** | ### Cost & Throughput per 1,000 Multi-Turn Queries (`05_roi_throughput.json`) | Model | Kind | `exec_match_final` | Throughput (`tok/s`) | Prompt / Completion Tok per Query | Seconds / Query | Cost / 1,000 Queries | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | | `tiny-aya-global` Base | Local 4-bit | `57.5%` | `79.24` | `612.3` / `40.4` | `0.4434 s` | `$0.0591` | | `tiny-aya-sql-heal-lora` | Local 4-bit | `60.0%` | `18.95` | `635.1` / `87.7` | `4.1364 s` | `$0.5515` | | `tiny-aya-sql-control-lora` | Local 4-bit | `60.0%` | `18.50` | `690.2` / `92.4` | `4.4502 s` | `$0.5934` | | `aya-expanse-8b` Base | Local 4-bit | `57.5%` | `55.18` | `209.3` / `39.3` | `0.6596 s` | `$0.0879` | | **`aya-8b-sql-heal-lora`** | Local 4-bit | **`63.5%`** | **`35.67`** | `210.2` / `86.1` | **`2.3073 s`** | **`$0.3076`** | | `Qwen2.5-72B-Instruct:cheapest` | Cloud API | `71.0%` | `61.13` | `196.0` / `36.5` | `0.5818 s` *(8 workers)* / `2.891 s` *(1-call)* | `$0.0401` | `05_roi_throughput.json` reports `cost_reduction_factor = 0.07` (`$0.0401 / $0.5515`), highlighting an honest deployment reality: unbatched 4-bit PyTorch PEFT `generate()` emitting `Thought:` reasoning (`86.1–87.7` tokens/query) on a single dedicated L4 GPU is more expensive per 1k queries than multi-tenant serverless `:cheapest` token pricing (`$0.13/M` in, `$0.40/M` out). In exchange, **`aya-8b-sql-heal-lora`** delivers **lower single-query latency (`2,307.3 ms` vs. `2,891.0 ms` API mean)**, **complete air-gapped data sovereignty**, and requires only **USD 0.1970** to fine-tune. In production, serving the merged `aya-8b-sql-heal-lora` weights under continuous batching (`vLLM` / PagedAttention at 8–16 concurrent requests) increases L4 throughput by ~5–8x, dropping self-hosted cost below `$0.04 / 1,000 queries` while preserving air-gapped schema privacy. ![Systems Latency Inversion: Tiny-Aya 3.35B vs Aya-Expanse 8B](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig3_preamble_latency_inversion.png) *Figure 3: Systems latency inversion contrasting `tiny-aya-sql-heal-lora` (3.35B) vs. `aya-8b-sql-heal-lora` (8B) across chat-template preamble overhead (`365` vs. `6` tokens), prompt length (`635.1` vs. `210.2`), decode throughput (`18.95` vs. `35.67 tok/s`), mean latency (`4,136.4 ms` vs. `2,307.3 ms`), L4 cost (`$0.5515` vs. `$0.3076` / 1k queries), and peak allocated VRAM (`6.331 GB` vs. `13.898 GB`).* --- ## 7. Reproduce Everything from Scratch You can inspect all executed training and evaluation cell outputs directly in the Colab notebook [`notebooks/train_and_audit_aya_sql.ipynb`](https://github.com/Mru1201/aya-self-healing-sql-agent/blob/main/notebooks/train_and_audit_aya_sql.ipynb), or run the complete pipeline end-to-end: ```bash # 1. Clone and verify local stack + HF Router git clone https://github.com/Mru1201/aya-self-healing-sql-agent.git cd aya-self-healing-sql-agent python -m venv .venv && source .venv/bin/activate pip install -r requirements.txt pip install -e vendor/CuratorKIT -e vendor/AuditKIT -e vendor/AgentTune echo "HF_TOKEN=hf_your_token_here" > .env python scripts/00_verify_local_stack.py # 2. Generate AST-mutated trajectories, localize (en/hi/de/kn), curate & run Leak Check python scripts/01_generate_and_curate.py # 3. Package bundle & run QLoRA training + AuditKIT evaluation + ROI profiling on NVIDIA L4 python scripts/03_bundle_for_colab.py --include-vendor # (On GPU host after unzipping colab_bundle.zip:) pip install -q -e vendor/AuditKIT -e vendor/AgentTune python scripts/02_colab_runner.py --stage all # 4. Publish LoRA adapters, dataset, and collections to Hugging Face Hub python scripts/04_publish_to_hf.py # 5. Regenerate benchmark figures from immutable audit_logs/ python3 scripts/05_generate_plots.py # 6. Publish Static Space interactive report & Community Discussion articles python scripts/06_publish_free_hf_articles.py ``` Every metric in this article is backed directly by immutable JSON records in [`audit_logs/`](https://github.com/Mru1201/aya-self-healing-sql-agent/tree/main/audit_logs) and mirrored in [`Mrunal1201/aya-self-healing-sql-dataset`](https://huggingface.co/datasets/Mrunal1201/aya-self-healing-sql-dataset).