File size: 13,621 Bytes
1c5c41e
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
64cdcd5
1c5c41e
64cdcd5
 
1c5c41e
64cdcd5
 
 
 
 
 
 
 
 
ca8155e
 
1c5c41e
 
 
64cdcd5
1c5c41e
64cdcd5
1c5c41e
64cdcd5
 
 
 
 
1c5c41e
 
 
64cdcd5
1c5c41e
64cdcd5
1c5c41e
64cdcd5
1c5c41e
64cdcd5
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1c5c41e
 
 
64cdcd5
1c5c41e
64cdcd5
 
1c5c41e
64cdcd5
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1c5c41e
64cdcd5
 
 
 
 
 
 
 
 
1c5c41e
 
 
 
 
 
 
 
 
 
 
 
 
64cdcd5
1c5c41e
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
64cdcd5
1c5c41e
 
64cdcd5
1c5c41e
 
 
 
 
 
64cdcd5
1c5c41e
 
 
 
 
 
64cdcd5
1c5c41e
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
---
base_model: CohereLabs/tiny-aya-global
library_name: peft
pipeline_tag: text-generation
tags:
  - lexsi-hackathon
  - agenttune
  - auditkit
  - curatorkit
  - text-to-sql
  - cohere-aya
  - self-healing
  - lora
datasets:
  - Mrunal1201/aya-self-healing-sql-dataset
language:
  - en
  - hi
  - de
  - kn
license: apache-2.0
metrics:
  - code_eval
---

# tiny-aya-sql-heal-lora: Primary Multi-Turn Self-Healing SQL Adapter (3.35B)

Official PEFT LoRA checkpoint for **Lexsi Hackathon Track 3 (Data & Database Agents)** (`#6` Data & Database Agents, `#8` Self-Healing Agents, `#3` Agentic Distillation):
*Autonomous Multi-Turn Self-Healing Multilingual SQL Agents via CuratorKIT Curation, AgentTune Distillation, SQLGlot AST Guardrails, and AuditKIT Verification.*

| Resource | Link |
| :--- | :--- |
| **GitHub Repository** | [`Mru1201/aya-self-healing-sql-agent`](https://github.com/Mru1201/aya-self-healing-sql-agent) |
| **Executed Colab Notebook** | [`notebooks/train_and_audit_aya_sql.ipynb`](https://github.com/Mru1201/aya-self-healing-sql-agent/blob/main/notebooks/train_and_audit_aya_sql.ipynb) |
| **This Model Adapter** | [`Mrunal1201/tiny-aya-sql-heal-lora`](https://huggingface.co/Mrunal1201/tiny-aya-sql-heal-lora) |
| **Base Model** | [`CohereLabs/tiny-aya-global`](https://huggingface.co/CohereLabs/tiny-aya-global) |
| **Public HF Model Collection** | [Lexsi Track 3 — Self-Healing SQL Aya Models](https://huggingface.co/collections/Mrunal1201/lexsi-track-3-self-healing-sql-aya-models-6af8e7fc0a621b28fefccd18) |
| **Curated Dataset & Audit Logs** | [`Mrunal1201/aya-self-healing-sql-dataset`](https://huggingface.co/datasets/Mrunal1201/aya-self-healing-sql-dataset) ([Collection](https://huggingface.co/collections/Mrunal1201/lexsi-track-3-curated-sql-trajectories-and-audit-logs-6af8e7fed8ac99740564a2fb)) |
| **Lexsi Kits Chained** | [`CuratorKIT 0.1.21`](https://github.com/Lexsi-Labs/CuratorKIT) → [`AgentTune 1.1.0`](https://github.com/Lexsi-Labs/AgentTune) → [`AuditKIT 1.1.12`](https://github.com/Lexsi-Labs/AuditKIT) |
| **HF Static Article Space** | [Interactive Technical Report](https://huggingface.co/spaces/Mrunal1201/aya-self-healing-sql-article) ([Direct Embed](https://mrunal1201-aya-self-healing-sql-article.static.hf.space)) |
| **HF Community Articles** | [Model Discussion #1](https://huggingface.co/Mrunal1201/tiny-aya-sql-heal-lora/discussions/1) · [Dataset Discussion #1](https://huggingface.co/datasets/Mrunal1201/aya-self-healing-sql-dataset/discussions/1) |

---

## Executive Summary

Primary PEFT LoRA adapter for **CohereLabs/tiny-aya-global** (3.35B) trained on all `1,200` [CuratorKIT](https://github.com/Lexsi-Labs/CuratorKIT)-certified multilingual SQL episodes (`780` `self_healing` + `420` `single_turn`) for **Lexsi Hackathon Track 3** (`#6` Data & Database Agents, `#8` Self-Healing Agents, `#3` Agentic Distillation). Cuts Turn-1 SQLite `OperationalError` crashes by **45.5%** over single-turn SFT (`33 -> 18`), improves Executable SQL Validity over control (`86.0% -> 91.0%`), and trains in just `6.331 GB` peak VRAM.

### Key Innovations
1. **Closed-Loop Deterministic Self-Healing:** Turn-1 SQL → deterministic `SQLGlot 30.31.0` AST read-only validation → ephemeral in-memory `SQLite 3.46.0` execution → verbatim `sqlite3.OperationalError` feedback → Turn-2 `Thought:` reasoning and repaired SQL.
2. **Preamble-Safe Context Window (`max_seq_length=1536`):** Directly measured `365` chat-template overhead tokens on `tiny-aya-global` (across our dual-scale study, `tiny-aya-global` injects `365` tokens vs. `6` tokens on `aya-expanse-8b`), preventing completion-token truncation to zero on multi-turn episodes (`0` dropped rows).
3. **Zero-Leakage Audit & Bootstrap Significance:** `80` training domains vs. `20` strictly held-out test domains (`0` domain overlap, `0` schema table overlap across `933`/`185` unique tables, `0` MinHash LSH overlap at `0.85`), evaluated via [`AuditKIT 1.1.12`](https://github.com/Lexsi-Labs/AuditKIT) across `1,000`-resample paired bootstrap CIs.
4. **Multilingual Reasoning Preservation:** Maintains **`96.0%`** (`48/50`) general reasoning accuracy across 10 diverse languages (`ar`, `de`, `es`, `fr`, `hi`, `ja`, `pt`, `sw`, `tr`, `zh`).

---

## Evaluation & Benchmark Results (`AuditKIT v1.1.12`)

### Focused Comparison: `tiny-aya-sql-heal-lora` vs. `tiny-aya-global` Base & Teacher Baseline

| Metric / Suite (`n`) | `tiny-aya-global` Base | **`tiny-aya-sql-heal-lora`** | `Qwen2.5-72B-Instruct:cheapest` | Delta vs. Own Base |
| :--- | :---: | :---: | :---: | :---: |
| **Turn-1 Exec Match (`n=200`)** | 56.5% `[49.5%, 63.0%]` | **60.0%** `[52.5%, 66.01%]` | 71.0% `[65.0%, 76.5%]` | **+3.5 pp** |
| **Final Self-Healed Exec Match (`n=200`)** | 57.5% `[50.5%, 64.0%]` | **60.0%** `[52.5%, 66.01%]` | 71.0% `[65.0%, 76.5%]` | **+2.5 pp** |
| **Executable SQL Validity Rate (`n=200`)** | 95.5% `[92.5%, 98.0%]` | **91.0%** `[87.0%, 94.5%]` | 100.0% `[100.0%, 100.0%]` | `-4.5 pp` |
| **Turn-1 `OperationalError` Count (`n=200`)** | `12 / 200` | **`18 / 200`** | `3 / 200` | `+6` errors |
| **Multilingual Retention (`n=50`, 10 langs)** | 96.0% `[90.0%, 100.0%]` | **96.0%** `[90.0%, 100.0%]` | 98.0% `[94.0%, 100.0%]` | `+0.0 pp` |
| **Pre-Guardrail Model Injection Safe (`n=50`)** | 24.0% `[12.0%, 36.05%]` | **10.0%** `[2.0%, 20.0%]` | 34.0% `[21.95%, 48.0%]` | `-14.0 pp` *(AST gate blocks writes)* |
| **Inference Throughput (`tok/s`)** | `79.24 tok/s` | **`18.95 tok/s`** | `61.13 tok/s` | Self-hosted L4 4-bit |
| **Mean Query Latency (`sql_heal`)** | 443.4 ms | **4,136.4 ms** | 581.8 ms *(8w)* / 2,891.0 ms *(1c)* | Air-gapped local execution |
| **Cost per 1,000 Queries** | `$0.0591` | **`$0.5515`** | `$0.0401` (cloud API) | Full data sovereignty |

### Full 6-Configuration Benchmark & Ablation Table (`audit_logs/04_auditkit_before_after.json`)

| Model Configuration | Turn-1 Exec Match [95% CI] | Final Exec Match [95% CI] | Executable SQL Validity [95% CI] | Turn-1 Errors (`OperationalError`) | Heal Rate | Adversarial Safety (`n=50`) | Retention (`n=50`) | Throughput (`tok/s`) | Mean Latency | Cost / 1k Queries |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| `CohereLabs/tiny-aya-global` Base (3.35B) | 56.5% `[49.5%, 63.0%]` | **57.5%** `[50.5%, 64.0%]` (`115/200`) | **95.5%** `[92.5%, 98.0%]` | `12 / 200` | 16.67% (`2/12`) | 24.0% `[12.0%, 36.05%]` | 96.0% `[90.0%, 100.0%]` | `79.24` | 443.4 ms | `$0.0591` |
| `tiny-aya-sql-heal-lora` (3.35B) | 60.0% `[52.5%, 66.01%]` | **60.0%** `[52.5%, 66.01%]` (`120/200`) | **91.0%** `[87.0%, 94.5%]` | `18 / 200` | 0.0% (`0/18`) | 10.0% `[2.0%, 20.0%]` | 96.0% `[90.0%, 100.0%]` | `18.95` | 4,136.4 ms | `$0.5515` |
| `tiny-aya-sql-control-lora` (3.35B, 1-Turn) | 58.5% `[52.0%, 65.5%]` | **60.0%** `[53.5%, 66.51%]` (`120/200`) | **86.0%** `[81.5%, 90.5%]` | `33 / 200` | 9.09% (`3/33`) | 10.0% `[2.0%, 20.0%]` | 96.0% `[90.0%, 100.0%]` | `18.50` | 4,450.2 ms | `$0.5934` |
| `CohereLabs/aya-expanse-8b` Base (8B) | 57.0% `[50.0%, 63.5%]` | **57.5%** `[50.5%, 64.0%]` (`115/200`) | **98.0%** `[96.0%, 99.5%]` | `11 / 200` | 9.09% (`1/11`) | 12.0% `[4.0%, 22.0%]` | 98.0% `[94.0%, 100.0%]` | `55.18` | 659.6 ms | `$0.0879` |
| `aya-8b-sql-heal-lora` (8B) | 63.5% `[57.0%, 70.0%]` | **63.5%** `[57.0%, 70.0%]` (`127/200`) | **96.0%** `[93.0%, 98.5%]` | `8 / 200` | 0.0% (`0/8`) | 14.0% `[6.0%, 24.0%]` | 96.0% `[90.0%, 100.0%]` | `35.67` | 2,307.3 ms | `$0.3076` |
| `Qwen/Qwen2.5-72B-Instruct:cheapest` (Teacher) | 71.0% `[65.0%, 76.5%]` | **71.0%** `[65.0%, 76.5%]` (`142/200`) | **100.0%** `[100.0%, 100.0%]` | `3 / 200` | 0.0% (`0/3`) | 34.0% `[21.95%, 48.0%]` | 98.0% `[94.0%, 100.0%]` | `61.13` | 581.8 ms *(8w)* / 2,891.0 ms *(1c)* | `$0.0401` |

---

## Visual Benchmarks & Diagnostics (`assets/figures/`)

![End-to-End Lexsi Track 3 Architecture Pipeline](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig0_architecture_pipeline.png)
*Figure 0: End-to-end closed-loop Track 3 pipeline uniting `CuratorKIT 0.1.21` curation & leak checks, `AgentTune 1.1.0` QLoRA trajectory distillation, `SQLGlot 30.31.0` read-only AST guardrails over in-memory `SQLite 3.46.0`, and `AuditKIT 1.1.12` outcome verification.*

![Before vs After SQL Execution Accuracy and Validity with 95% Bootstrap CIs](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig1_accuracy_validity_ci.png)
*Figure 1: Turn-1 execution accuracy, final self-healed execution accuracy, and SQLite executable validity across all 6 evaluated configurations on the 200 disjoint test queries (`1,000`-resample 95% bootstrap CIs from `audit_logs/04_auditkit_before_after.json`).*

![Supervision Structure and Scale Error Reduction Ablation](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig2_error_reduction_ablation.png)
*Figure 2: (Left) Preventative schema grounding cuts Turn-1 SQLite `OperationalError` counts from `33` (`tiny-aya-sql-control-lora`) down to `18` (`-45.5%` on 3.35B) and `8` (`-75.8%` on 8B). (Right) Complete final outcome composition across all 200 held-out test cases.*

![Systems Latency Inversion: Tiny-Aya 3.35B vs Aya-Expanse 8B](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig3_preamble_latency_inversion.png)
*Figure 3: Systems inversion comparing `tiny-aya-sql-heal-lora` (3.35B) vs. `aya-8b-sql-heal-lora` (8B) across chat-template preamble overhead (`365` vs. `6` tokens), prompt length (`635.1` vs. `210.2`), decode throughput (`18.95` vs. `35.67 tok/s`), mean latency (`4,136.4 ms` vs. `2,307.3 ms`), L4 cost (`$0.5515` vs. `$0.3076` per 1k queries), and peak allocated VRAM (`6.331 GB` vs. `13.898 GB`).*

![QLoRA Training Loss Convergence and Pre-Guardrail Adversarial Safety Heatmap](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig4_training_loss_and_safety.png)
*Figure 4: (Left) `AgentTune 1.1.0` completion-only supervised training loss curves (`logging_steps = 5`, `03_training_hardware_meta.json`). (Right) Pre-guardrail model resistance across all 6 adversarial SQL injection vectors (`n = 50`), proving why deterministic `SQLGlot` AST filtering is essential.*

---

## Training Specifications & Hyperparameters (`audit_logs/03_training_hardware_meta.json`)

- **Hardware Infrastructure:** `NVIDIA L4` (`22.03 GB` VRAM)
- **Software Stack:** PyTorch `2.11.0+cu130`, CUDA `13.0`, PEFT `0.21.3`, TRL `1.7.1`, [`AgentTune 1.1.0`](https://github.com/Lexsi-Labs/AgentTune)
- **Training Backend:** `agenttune.create_distill_trainer`
- **Training Subset:** `full` (`1200` episodes)
- **Global Steps:** `150` steps across `2.0` epochs
- **Learning Rate:** `0.0002` with `cosine` schedule and `0.03` warmup
- **Effective Batch Size:** `16` (`4` per device × `4` gradient accumulation)
- **Optimizer & Precision:** `paged_adamw_8bit`, `bf16=True`, `gradient_checkpointing=True`
- **Sequence Length & Overhead:** `max_seq_length=1536` | Chat template preamble: `365` tokens | Mean/Max length: `630.0` / `1085` tokens | `0` dropped rows
- **Loss Masking:** `completion_only (final assistant turn); system/user/turn-1 SQL/error masked`
- **Training Loss (Step 5 → Step 150):** `1.2010 → 0.3071` (**0.4081** mean loss)
- **Peak VRAM (Allocated / Reserved):** `6.331 GB` / `6.648 GB`
- **Wall-Clock Training Time:** `24.283` minutes (`1430.27 s`)

### LoRA Architecture
```json
{
  "r": 16,
  "lora_alpha": 32,
  "lora_dropout": 0.05,
  "target_modules": ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
  "bias": "none",
  "task_type": "CAUSAL_LM"
}
```

---

## Quickstart & Inference

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "CohereLabs/tiny-aya-global"
adapter_id = "Mrunal1201/tiny-aya-sql-heal-lora"

# 1. Load Tokenizer and Base Model in BF16 / 4-bit
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

# 2. Attach Self-Healing LoRA Adapter
model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

# 3. Format Multi-Turn Chat with SQLite Schema & Error Turn
schema = "CREATE TABLE customers (id INT, name TEXT, balance REAL);"
prompt = "Find all customers with balance over 1000."
error_feedback = "SQLite Execution Error: no such column: amount"

messages = [
    {"role": "system", "content": "You are a multilingual SQLite data assistant. Translate the question into read-only SQL.\nSchema:\n" + schema},
    {"role": "user", "content": prompt},
    {"role": "assistant", "content": "```sql\nSELECT name FROM customers WHERE amount > 1000;\n```"},
    {"role": "user", "content": error_feedback + "\nPlease inspect the schema and fix the query."}
]

input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(input_ids, max_new_tokens=256, do_sample=False)

print(tokenizer.decode(outputs[0][input_ids.shape[1]:], skip_special_tokens=True))
```

---

## Citation & Attribution

```bibtex
@misc{lexsi_track3_aya_sql_2026,
  title={Autonomous Self-Healing Multilingual SQL Agents on Cohere Aya},
  author={Mrunal1201 and Lexsi Hackathon Team},
  year={2026},
  howpublished={\url{https://github.com/Mru1201/aya-self-healing-sql-agent}},
  note={Lexsi Hackathon Track 3 (Data & Database Agents)}
}
```