--- base_model: CohereLabs/tiny-aya-global library_name: peft pipeline_tag: text-generation tags: - lexsi-hackathon - agenttune - auditkit - curatorkit - text-to-sql - cohere-aya - single-turn-control - lora datasets: - Mrunal1201/aya-self-healing-sql-dataset language: - en - hi - de - kn license: apache-2.0 metrics: - code_eval --- # tiny-aya-sql-control-lora: Single-Turn Ablation Control Adapter (3.35B) Official PEFT LoRA checkpoint for **Lexsi Hackathon Track 3 (Data & Database Agents)** (`#6` Data & Database Agents, `#8` Self-Healing Agents, `#3` Agentic Distillation): *Autonomous Multi-Turn Self-Healing Multilingual SQL Agents via CuratorKIT Curation, AgentTune Distillation, SQLGlot AST Guardrails, and AuditKIT Verification.* | Resource | Link | | :--- | :--- | | **GitHub Repository** | [`Mru1201/aya-self-healing-sql-agent`](https://github.com/Mru1201/aya-self-healing-sql-agent) | | **Executed Colab Notebook** | [`notebooks/train_and_audit_aya_sql.ipynb`](https://github.com/Mru1201/aya-self-healing-sql-agent/blob/main/notebooks/train_and_audit_aya_sql.ipynb) | | **This Model Adapter** | [`Mrunal1201/tiny-aya-sql-control-lora`](https://huggingface.co/Mrunal1201/tiny-aya-sql-control-lora) | | **Base Model** | [`CohereLabs/tiny-aya-global`](https://huggingface.co/CohereLabs/tiny-aya-global) | | **Public HF Model Collection** | [Lexsi Track 3 — Self-Healing SQL Aya Models](https://huggingface.co/collections/Mrunal1201/lexsi-track-3-self-healing-sql-aya-models-6af8e7fc0a621b28fefccd18) | | **Curated Dataset & Audit Logs** | [`Mrunal1201/aya-self-healing-sql-dataset`](https://huggingface.co/datasets/Mrunal1201/aya-self-healing-sql-dataset) ([Collection](https://huggingface.co/collections/Mrunal1201/lexsi-track-3-curated-sql-trajectories-and-audit-logs-6af8e7fed8ac99740564a2fb)) | | **Lexsi Kits Chained** | [`CuratorKIT 0.1.21`](https://github.com/Lexsi-Labs/CuratorKIT) → [`AgentTune 1.1.0`](https://github.com/Lexsi-Labs/AgentTune) → [`AuditKIT 1.1.12`](https://github.com/Lexsi-Labs/AuditKIT) | | **HF Static Article Space** | [Interactive Technical Report](https://huggingface.co/spaces/Mrunal1201/aya-self-healing-sql-article) ([Direct Embed](https://mrunal1201-aya-self-healing-sql-article.static.hf.space)) | | **HF Community Articles** | [Model Discussion #1](https://huggingface.co/Mrunal1201/tiny-aya-sql-heal-lora/discussions/1) · [Dataset Discussion #1](https://huggingface.co/datasets/Mrunal1201/aya-self-healing-sql-dataset/discussions/1) | --- ## Executive Summary Supervision-structure ablation control PEFT LoRA adapter for **CohereLabs/tiny-aya-global** (3.35B), trained strictly on the `420` `single_turn` SQL episodes (`54` steps) without multi-turn `OperationalError` trajectory supervision. Isolates the causal contribution of multi-turn repair conditioning against standard single-turn SFT. ### Key Innovations 1. **Closed-Loop Deterministic Self-Healing:** Turn-1 SQL → deterministic `SQLGlot 30.31.0` AST read-only validation → ephemeral in-memory `SQLite 3.46.0` execution → verbatim `sqlite3.OperationalError` feedback → Turn-2 `Thought:` reasoning and repaired SQL. 2. **Preamble-Safe Context Window (`max_seq_length=1536`):** Directly measured `365` chat-template overhead tokens on `tiny-aya-global` (across our dual-scale study, `tiny-aya-global` injects `365` tokens vs. `6` tokens on `aya-expanse-8b`), preventing completion-token truncation to zero on multi-turn episodes (`0` dropped rows). 3. **Zero-Leakage Audit & Bootstrap Significance:** `80` training domains vs. `20` strictly held-out test domains (`0` domain overlap, `0` schema table overlap across `933`/`185` unique tables, `0` MinHash LSH overlap at `0.85`), evaluated via [`AuditKIT 1.1.12`](https://github.com/Lexsi-Labs/AuditKIT) across `1,000`-resample paired bootstrap CIs. 4. **Multilingual Reasoning Preservation:** Maintains **`96.0%`** (`48/50`) general reasoning accuracy across 10 diverse languages (`ar`, `de`, `es`, `fr`, `hi`, `ja`, `pt`, `sw`, `tr`, `zh`). --- ## Evaluation & Benchmark Results (`AuditKIT v1.1.12`) ### Focused Comparison: `tiny-aya-sql-control-lora` vs. `tiny-aya-global` Base & Teacher Baseline | Metric / Suite (`n`) | `tiny-aya-global` Base | **`tiny-aya-sql-control-lora`** | `Qwen2.5-72B-Instruct:cheapest` | Delta vs. Own Base | | :--- | :---: | :---: | :---: | :---: | | **Turn-1 Exec Match (`n=200`)** | 56.5% `[49.5%, 63.0%]` | **58.5%** `[52.0%, 65.5%]` | 71.0% `[65.0%, 76.5%]` | **+2.0 pp** | | **Final Self-Healed Exec Match (`n=200`)** | 57.5% `[50.5%, 64.0%]` | **60.0%** `[53.5%, 66.51%]` | 71.0% `[65.0%, 76.5%]` | **+2.5 pp** | | **Executable SQL Validity Rate (`n=200`)** | 95.5% `[92.5%, 98.0%]` | **86.0%** `[81.5%, 90.5%]` | 100.0% `[100.0%, 100.0%]` | `-9.5 pp` | | **Turn-1 `OperationalError` Count (`n=200`)** | `12 / 200` | **`33 / 200`** | `3 / 200` | `+21` errors | | **Multilingual Retention (`n=50`, 10 langs)** | 96.0% `[90.0%, 100.0%]` | **96.0%** `[90.0%, 100.0%]` | 98.0% `[94.0%, 100.0%]` | `+0.0 pp` | | **Pre-Guardrail Model Injection Safe (`n=50`)** | 24.0% `[12.0%, 36.05%]` | **10.0%** `[2.0%, 20.0%]` | 34.0% `[21.95%, 48.0%]` | `-14.0 pp` *(AST gate blocks writes)* | | **Inference Throughput (`tok/s`)** | `79.24 tok/s` | **`18.50 tok/s`** | `61.13 tok/s` | Self-hosted L4 4-bit | | **Mean Query Latency (`sql_heal`)** | 443.4 ms | **4,450.2 ms** | 581.8 ms *(8w)* / 2,891.0 ms *(1c)* | Air-gapped local execution | | **Cost per 1,000 Queries** | `$0.0591` | **`$0.5934`** | `$0.0401` (cloud API) | Full data sovereignty | ### Full 6-Configuration Benchmark & Ablation Table (`audit_logs/04_auditkit_before_after.json`) | Model Configuration | Turn-1 Exec Match [95% CI] | Final Exec Match [95% CI] | Executable SQL Validity [95% CI] | Turn-1 Errors (`OperationalError`) | Heal Rate | Adversarial Safety (`n=50`) | Retention (`n=50`) | Throughput (`tok/s`) | Mean Latency | Cost / 1k Queries | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | `CohereLabs/tiny-aya-global` Base (3.35B) | 56.5% `[49.5%, 63.0%]` | **57.5%** `[50.5%, 64.0%]` (`115/200`) | **95.5%** `[92.5%, 98.0%]` | `12 / 200` | 16.67% (`2/12`) | 24.0% `[12.0%, 36.05%]` | 96.0% `[90.0%, 100.0%]` | `79.24` | 443.4 ms | `$0.0591` | | `tiny-aya-sql-heal-lora` (3.35B) | 60.0% `[52.5%, 66.01%]` | **60.0%** `[52.5%, 66.01%]` (`120/200`) | **91.0%** `[87.0%, 94.5%]` | `18 / 200` | 0.0% (`0/18`) | 10.0% `[2.0%, 20.0%]` | 96.0% `[90.0%, 100.0%]` | `18.95` | 4,136.4 ms | `$0.5515` | | `tiny-aya-sql-control-lora` (3.35B, 1-Turn) | 58.5% `[52.0%, 65.5%]` | **60.0%** `[53.5%, 66.51%]` (`120/200`) | **86.0%** `[81.5%, 90.5%]` | `33 / 200` | 9.09% (`3/33`) | 10.0% `[2.0%, 20.0%]` | 96.0% `[90.0%, 100.0%]` | `18.50` | 4,450.2 ms | `$0.5934` | | `CohereLabs/aya-expanse-8b` Base (8B) | 57.0% `[50.0%, 63.5%]` | **57.5%** `[50.5%, 64.0%]` (`115/200`) | **98.0%** `[96.0%, 99.5%]` | `11 / 200` | 9.09% (`1/11`) | 12.0% `[4.0%, 22.0%]` | 98.0% `[94.0%, 100.0%]` | `55.18` | 659.6 ms | `$0.0879` | | `aya-8b-sql-heal-lora` (8B) | 63.5% `[57.0%, 70.0%]` | **63.5%** `[57.0%, 70.0%]` (`127/200`) | **96.0%** `[93.0%, 98.5%]` | `8 / 200` | 0.0% (`0/8`) | 14.0% `[6.0%, 24.0%]` | 96.0% `[90.0%, 100.0%]` | `35.67` | 2,307.3 ms | `$0.3076` | | `Qwen/Qwen2.5-72B-Instruct:cheapest` (Teacher) | 71.0% `[65.0%, 76.5%]` | **71.0%** `[65.0%, 76.5%]` (`142/200`) | **100.0%** `[100.0%, 100.0%]` | `3 / 200` | 0.0% (`0/3`) | 34.0% `[21.95%, 48.0%]` | 98.0% `[94.0%, 100.0%]` | `61.13` | 581.8 ms *(8w)* / 2,891.0 ms *(1c)* | `$0.0401` | --- ## Visual Benchmarks & Diagnostics (`assets/figures/`) ![End-to-End Lexsi Track 3 Architecture Pipeline](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig0_architecture_pipeline.png) *Figure 0: End-to-end closed-loop Track 3 pipeline uniting `CuratorKIT 0.1.21` curation & leak checks, `AgentTune 1.1.0` QLoRA trajectory distillation, `SQLGlot 30.31.0` read-only AST guardrails over in-memory `SQLite 3.46.0`, and `AuditKIT 1.1.12` outcome verification.* ![Before vs After SQL Execution Accuracy and Validity with 95% Bootstrap CIs](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig1_accuracy_validity_ci.png) *Figure 1: Turn-1 execution accuracy, final self-healed execution accuracy, and SQLite executable validity across all 6 evaluated configurations on the 200 disjoint test queries (`1,000`-resample 95% bootstrap CIs from `audit_logs/04_auditkit_before_after.json`).* ![Supervision Structure and Scale Error Reduction Ablation](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig2_error_reduction_ablation.png) *Figure 2: (Left) Preventative schema grounding cuts Turn-1 SQLite `OperationalError` counts from `33` (`tiny-aya-sql-control-lora`) down to `18` (`-45.5%` on 3.35B) and `8` (`-75.8%` on 8B). (Right) Complete final outcome composition across all 200 held-out test cases.* ![Systems Latency Inversion: Tiny-Aya 3.35B vs Aya-Expanse 8B](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig3_preamble_latency_inversion.png) *Figure 3: Systems inversion comparing `tiny-aya-sql-heal-lora` (3.35B) vs. `aya-8b-sql-heal-lora` (8B) across chat-template preamble overhead (`365` vs. `6` tokens), prompt length (`635.1` vs. `210.2`), decode throughput (`18.95` vs. `35.67 tok/s`), mean latency (`4,136.4 ms` vs. `2,307.3 ms`), L4 cost (`$0.5515` vs. `$0.3076` per 1k queries), and peak allocated VRAM (`6.331 GB` vs. `13.898 GB`).* ![QLoRA Training Loss Convergence and Pre-Guardrail Adversarial Safety Heatmap](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig4_training_loss_and_safety.png) *Figure 4: (Left) `AgentTune 1.1.0` completion-only supervised training loss curves (`logging_steps = 5`, `03_training_hardware_meta.json`). (Right) Pre-guardrail model resistance across all 6 adversarial SQL injection vectors (`n = 50`), proving why deterministic `SQLGlot` AST filtering is essential.* --- ## Training Specifications & Hyperparameters (`audit_logs/03_training_hardware_meta.json`) - **Hardware Infrastructure:** `NVIDIA L4` (`22.03 GB` VRAM) - **Software Stack:** PyTorch `2.11.0+cu130`, CUDA `13.0`, PEFT `0.21.3`, TRL `1.7.1`, [`AgentTune 1.1.0`](https://github.com/Lexsi-Labs/AgentTune) - **Training Backend:** `agenttune.create_distill_trainer` - **Training Subset:** `single_turn` (`420` episodes) - **Global Steps:** `54` steps across `2.0` epochs - **Learning Rate:** `0.0002` with `cosine` schedule and `0.03` warmup - **Effective Batch Size:** `16` (`4` per device × `4` gradient accumulation) - **Optimizer & Precision:** `paged_adamw_8bit`, `bf16=True`, `gradient_checkpointing=True` - **Sequence Length & Overhead:** `max_seq_length=1536` | Chat template preamble: `365` tokens | Mean/Max length: `562.9` / `853` tokens | `0` dropped rows - **Loss Masking:** `completion_only (final assistant turn); system/user/turn-1 SQL/error masked` - **Training Loss (Step 5 → Step 50):** `1.1766 → 0.5947` (**0.7045** mean loss) - **Peak VRAM (Allocated / Reserved):** `7.176 GB` / `7.436 GB` - **Wall-Clock Training Time:** `7.326` minutes (`429.98 s`) ### LoRA Architecture ```json { "r": 16, "lora_alpha": 32, "lora_dropout": 0.05, "target_modules": ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"], "bias": "none", "task_type": "CAUSAL_LM" } ``` --- ## Quickstart & Inference ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer from peft import PeftModel base_model_id = "CohereLabs/tiny-aya-global" adapter_id = "Mrunal1201/tiny-aya-sql-control-lora" # 1. Load Tokenizer and Base Model in BF16 / 4-bit tokenizer = AutoTokenizer.from_pretrained(base_model_id) base_model = AutoModelForCausalLM.from_pretrained( base_model_id, torch_dtype=torch.bfloat16, device_map="auto", ) # 2. Attach Self-Healing LoRA Adapter model = PeftModel.from_pretrained(base_model, adapter_id) model.eval() # 3. Format Multi-Turn Chat with SQLite Schema & Error Turn schema = "CREATE TABLE customers (id INT, name TEXT, balance REAL);" prompt = "Find all customers with balance over 1000." error_feedback = "SQLite Execution Error: no such column: amount" messages = [ {"role": "system", "content": "You are a multilingual SQLite data assistant. Translate the question into read-only SQL.\nSchema:\n" + schema}, {"role": "user", "content": prompt}, {"role": "assistant", "content": "```sql\nSELECT name FROM customers WHERE amount > 1000;\n```"}, {"role": "user", "content": error_feedback + "\nPlease inspect the schema and fix the query."} ] input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device) with torch.no_grad(): outputs = model.generate(input_ids, max_new_tokens=256, do_sample=False) print(tokenizer.decode(outputs[0][input_ids.shape[1]:], skip_special_tokens=True)) ``` --- ## Citation & Attribution ```bibtex @misc{lexsi_track3_aya_sql_2026, title={Autonomous Self-Healing Multilingual SQL Agents on Cohere Aya}, author={Mrunal1201 and Lexsi Hackathon Team}, year={2026}, howpublished={\url{https://github.com/Mru1201/aya-self-healing-sql-agent}}, note={Lexsi Hackathon Track 3 (Data & Database Agents)} } ```