File size: 13,429 Bytes
e1461bc
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
30e1545
e1461bc
30e1545
 
e1461bc
30e1545
 
 
 
 
 
 
 
 
34bb1e5
 
e1461bc
 
 
30e1545
e1461bc
30e1545
e1461bc
30e1545
 
 
 
 
e1461bc
 
 
30e1545
e1461bc
30e1545
e1461bc
30e1545
e1461bc
30e1545
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
e1461bc
 
 
30e1545
e1461bc
30e1545
 
e1461bc
30e1545
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
e1461bc
30e1545
 
 
 
 
 
 
 
 
e1461bc
 
 
 
 
 
 
 
 
 
 
 
 
30e1545
e1461bc
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
30e1545
e1461bc
 
30e1545
e1461bc
 
 
 
 
 
30e1545
e1461bc
 
 
 
 
 
30e1545
e1461bc
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
---
base_model: CohereLabs/tiny-aya-global
library_name: peft
pipeline_tag: text-generation
tags:
  - lexsi-hackathon
  - agenttune
  - auditkit
  - curatorkit
  - text-to-sql
  - cohere-aya
  - single-turn-control
  - lora
datasets:
  - Mrunal1201/aya-self-healing-sql-dataset
language:
  - en
  - hi
  - de
  - kn
license: apache-2.0
metrics:
  - code_eval
---

# tiny-aya-sql-control-lora: Single-Turn Ablation Control Adapter (3.35B)

Official PEFT LoRA checkpoint for **Lexsi Hackathon Track 3 (Data & Database Agents)** (`#6` Data & Database Agents, `#8` Self-Healing Agents, `#3` Agentic Distillation):
*Autonomous Multi-Turn Self-Healing Multilingual SQL Agents via CuratorKIT Curation, AgentTune Distillation, SQLGlot AST Guardrails, and AuditKIT Verification.*

| Resource | Link |
| :--- | :--- |
| **GitHub Repository** | [`Mru1201/aya-self-healing-sql-agent`](https://github.com/Mru1201/aya-self-healing-sql-agent) |
| **Executed Colab Notebook** | [`notebooks/train_and_audit_aya_sql.ipynb`](https://github.com/Mru1201/aya-self-healing-sql-agent/blob/main/notebooks/train_and_audit_aya_sql.ipynb) |
| **This Model Adapter** | [`Mrunal1201/tiny-aya-sql-control-lora`](https://huggingface.co/Mrunal1201/tiny-aya-sql-control-lora) |
| **Base Model** | [`CohereLabs/tiny-aya-global`](https://huggingface.co/CohereLabs/tiny-aya-global) |
| **Public HF Model Collection** | [Lexsi Track 3 — Self-Healing SQL Aya Models](https://huggingface.co/collections/Mrunal1201/lexsi-track-3-self-healing-sql-aya-models-6af8e7fc0a621b28fefccd18) |
| **Curated Dataset & Audit Logs** | [`Mrunal1201/aya-self-healing-sql-dataset`](https://huggingface.co/datasets/Mrunal1201/aya-self-healing-sql-dataset) ([Collection](https://huggingface.co/collections/Mrunal1201/lexsi-track-3-curated-sql-trajectories-and-audit-logs-6af8e7fed8ac99740564a2fb)) |
| **Lexsi Kits Chained** | [`CuratorKIT 0.1.21`](https://github.com/Lexsi-Labs/CuratorKIT) → [`AgentTune 1.1.0`](https://github.com/Lexsi-Labs/AgentTune) → [`AuditKIT 1.1.12`](https://github.com/Lexsi-Labs/AuditKIT) |
| **HF Static Article Space** | [Interactive Technical Report](https://huggingface.co/spaces/Mrunal1201/aya-self-healing-sql-article) ([Direct Embed](https://mrunal1201-aya-self-healing-sql-article.static.hf.space)) |
| **HF Community Articles** | [Model Discussion #1](https://huggingface.co/Mrunal1201/tiny-aya-sql-heal-lora/discussions/1) · [Dataset Discussion #1](https://huggingface.co/datasets/Mrunal1201/aya-self-healing-sql-dataset/discussions/1) |

---

## Executive Summary

Supervision-structure ablation control PEFT LoRA adapter for **CohereLabs/tiny-aya-global** (3.35B), trained strictly on the `420` `single_turn` SQL episodes (`54` steps) without multi-turn `OperationalError` trajectory supervision. Isolates the causal contribution of multi-turn repair conditioning against standard single-turn SFT.

### Key Innovations
1. **Closed-Loop Deterministic Self-Healing:** Turn-1 SQL → deterministic `SQLGlot 30.31.0` AST read-only validation → ephemeral in-memory `SQLite 3.46.0` execution → verbatim `sqlite3.OperationalError` feedback → Turn-2 `Thought:` reasoning and repaired SQL.
2. **Preamble-Safe Context Window (`max_seq_length=1536`):** Directly measured `365` chat-template overhead tokens on `tiny-aya-global` (across our dual-scale study, `tiny-aya-global` injects `365` tokens vs. `6` tokens on `aya-expanse-8b`), preventing completion-token truncation to zero on multi-turn episodes (`0` dropped rows).
3. **Zero-Leakage Audit & Bootstrap Significance:** `80` training domains vs. `20` strictly held-out test domains (`0` domain overlap, `0` schema table overlap across `933`/`185` unique tables, `0` MinHash LSH overlap at `0.85`), evaluated via [`AuditKIT 1.1.12`](https://github.com/Lexsi-Labs/AuditKIT) across `1,000`-resample paired bootstrap CIs.
4. **Multilingual Reasoning Preservation:** Maintains **`96.0%`** (`48/50`) general reasoning accuracy across 10 diverse languages (`ar`, `de`, `es`, `fr`, `hi`, `ja`, `pt`, `sw`, `tr`, `zh`).

---

## Evaluation & Benchmark Results (`AuditKIT v1.1.12`)

### Focused Comparison: `tiny-aya-sql-control-lora` vs. `tiny-aya-global` Base & Teacher Baseline

| Metric / Suite (`n`) | `tiny-aya-global` Base | **`tiny-aya-sql-control-lora`** | `Qwen2.5-72B-Instruct:cheapest` | Delta vs. Own Base |
| :--- | :---: | :---: | :---: | :---: |
| **Turn-1 Exec Match (`n=200`)** | 56.5% `[49.5%, 63.0%]` | **58.5%** `[52.0%, 65.5%]` | 71.0% `[65.0%, 76.5%]` | **+2.0 pp** |
| **Final Self-Healed Exec Match (`n=200`)** | 57.5% `[50.5%, 64.0%]` | **60.0%** `[53.5%, 66.51%]` | 71.0% `[65.0%, 76.5%]` | **+2.5 pp** |
| **Executable SQL Validity Rate (`n=200`)** | 95.5% `[92.5%, 98.0%]` | **86.0%** `[81.5%, 90.5%]` | 100.0% `[100.0%, 100.0%]` | `-9.5 pp` |
| **Turn-1 `OperationalError` Count (`n=200`)** | `12 / 200` | **`33 / 200`** | `3 / 200` | `+21` errors |
| **Multilingual Retention (`n=50`, 10 langs)** | 96.0% `[90.0%, 100.0%]` | **96.0%** `[90.0%, 100.0%]` | 98.0% `[94.0%, 100.0%]` | `+0.0 pp` |
| **Pre-Guardrail Model Injection Safe (`n=50`)** | 24.0% `[12.0%, 36.05%]` | **10.0%** `[2.0%, 20.0%]` | 34.0% `[21.95%, 48.0%]` | `-14.0 pp` *(AST gate blocks writes)* |
| **Inference Throughput (`tok/s`)** | `79.24 tok/s` | **`18.50 tok/s`** | `61.13 tok/s` | Self-hosted L4 4-bit |
| **Mean Query Latency (`sql_heal`)** | 443.4 ms | **4,450.2 ms** | 581.8 ms *(8w)* / 2,891.0 ms *(1c)* | Air-gapped local execution |
| **Cost per 1,000 Queries** | `$0.0591` | **`$0.5934`** | `$0.0401` (cloud API) | Full data sovereignty |

### Full 6-Configuration Benchmark & Ablation Table (`audit_logs/04_auditkit_before_after.json`)

| Model Configuration | Turn-1 Exec Match [95% CI] | Final Exec Match [95% CI] | Executable SQL Validity [95% CI] | Turn-1 Errors (`OperationalError`) | Heal Rate | Adversarial Safety (`n=50`) | Retention (`n=50`) | Throughput (`tok/s`) | Mean Latency | Cost / 1k Queries |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| `CohereLabs/tiny-aya-global` Base (3.35B) | 56.5% `[49.5%, 63.0%]` | **57.5%** `[50.5%, 64.0%]` (`115/200`) | **95.5%** `[92.5%, 98.0%]` | `12 / 200` | 16.67% (`2/12`) | 24.0% `[12.0%, 36.05%]` | 96.0% `[90.0%, 100.0%]` | `79.24` | 443.4 ms | `$0.0591` |
| `tiny-aya-sql-heal-lora` (3.35B) | 60.0% `[52.5%, 66.01%]` | **60.0%** `[52.5%, 66.01%]` (`120/200`) | **91.0%** `[87.0%, 94.5%]` | `18 / 200` | 0.0% (`0/18`) | 10.0% `[2.0%, 20.0%]` | 96.0% `[90.0%, 100.0%]` | `18.95` | 4,136.4 ms | `$0.5515` |
| `tiny-aya-sql-control-lora` (3.35B, 1-Turn) | 58.5% `[52.0%, 65.5%]` | **60.0%** `[53.5%, 66.51%]` (`120/200`) | **86.0%** `[81.5%, 90.5%]` | `33 / 200` | 9.09% (`3/33`) | 10.0% `[2.0%, 20.0%]` | 96.0% `[90.0%, 100.0%]` | `18.50` | 4,450.2 ms | `$0.5934` |
| `CohereLabs/aya-expanse-8b` Base (8B) | 57.0% `[50.0%, 63.5%]` | **57.5%** `[50.5%, 64.0%]` (`115/200`) | **98.0%** `[96.0%, 99.5%]` | `11 / 200` | 9.09% (`1/11`) | 12.0% `[4.0%, 22.0%]` | 98.0% `[94.0%, 100.0%]` | `55.18` | 659.6 ms | `$0.0879` |
| `aya-8b-sql-heal-lora` (8B) | 63.5% `[57.0%, 70.0%]` | **63.5%** `[57.0%, 70.0%]` (`127/200`) | **96.0%** `[93.0%, 98.5%]` | `8 / 200` | 0.0% (`0/8`) | 14.0% `[6.0%, 24.0%]` | 96.0% `[90.0%, 100.0%]` | `35.67` | 2,307.3 ms | `$0.3076` |
| `Qwen/Qwen2.5-72B-Instruct:cheapest` (Teacher) | 71.0% `[65.0%, 76.5%]` | **71.0%** `[65.0%, 76.5%]` (`142/200`) | **100.0%** `[100.0%, 100.0%]` | `3 / 200` | 0.0% (`0/3`) | 34.0% `[21.95%, 48.0%]` | 98.0% `[94.0%, 100.0%]` | `61.13` | 581.8 ms *(8w)* / 2,891.0 ms *(1c)* | `$0.0401` |

---

## Visual Benchmarks & Diagnostics (`assets/figures/`)

![End-to-End Lexsi Track 3 Architecture Pipeline](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig0_architecture_pipeline.png)
*Figure 0: End-to-end closed-loop Track 3 pipeline uniting `CuratorKIT 0.1.21` curation & leak checks, `AgentTune 1.1.0` QLoRA trajectory distillation, `SQLGlot 30.31.0` read-only AST guardrails over in-memory `SQLite 3.46.0`, and `AuditKIT 1.1.12` outcome verification.*

![Before vs After SQL Execution Accuracy and Validity with 95% Bootstrap CIs](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig1_accuracy_validity_ci.png)
*Figure 1: Turn-1 execution accuracy, final self-healed execution accuracy, and SQLite executable validity across all 6 evaluated configurations on the 200 disjoint test queries (`1,000`-resample 95% bootstrap CIs from `audit_logs/04_auditkit_before_after.json`).*

![Supervision Structure and Scale Error Reduction Ablation](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig2_error_reduction_ablation.png)
*Figure 2: (Left) Preventative schema grounding cuts Turn-1 SQLite `OperationalError` counts from `33` (`tiny-aya-sql-control-lora`) down to `18` (`-45.5%` on 3.35B) and `8` (`-75.8%` on 8B). (Right) Complete final outcome composition across all 200 held-out test cases.*

![Systems Latency Inversion: Tiny-Aya 3.35B vs Aya-Expanse 8B](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig3_preamble_latency_inversion.png)
*Figure 3: Systems inversion comparing `tiny-aya-sql-heal-lora` (3.35B) vs. `aya-8b-sql-heal-lora` (8B) across chat-template preamble overhead (`365` vs. `6` tokens), prompt length (`635.1` vs. `210.2`), decode throughput (`18.95` vs. `35.67 tok/s`), mean latency (`4,136.4 ms` vs. `2,307.3 ms`), L4 cost (`$0.5515` vs. `$0.3076` per 1k queries), and peak allocated VRAM (`6.331 GB` vs. `13.898 GB`).*

![QLoRA Training Loss Convergence and Pre-Guardrail Adversarial Safety Heatmap](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig4_training_loss_and_safety.png)
*Figure 4: (Left) `AgentTune 1.1.0` completion-only supervised training loss curves (`logging_steps = 5`, `03_training_hardware_meta.json`). (Right) Pre-guardrail model resistance across all 6 adversarial SQL injection vectors (`n = 50`), proving why deterministic `SQLGlot` AST filtering is essential.*

---

## Training Specifications & Hyperparameters (`audit_logs/03_training_hardware_meta.json`)

- **Hardware Infrastructure:** `NVIDIA L4` (`22.03 GB` VRAM)
- **Software Stack:** PyTorch `2.11.0+cu130`, CUDA `13.0`, PEFT `0.21.3`, TRL `1.7.1`, [`AgentTune 1.1.0`](https://github.com/Lexsi-Labs/AgentTune)
- **Training Backend:** `agenttune.create_distill_trainer`
- **Training Subset:** `single_turn` (`420` episodes)
- **Global Steps:** `54` steps across `2.0` epochs
- **Learning Rate:** `0.0002` with `cosine` schedule and `0.03` warmup
- **Effective Batch Size:** `16` (`4` per device × `4` gradient accumulation)
- **Optimizer & Precision:** `paged_adamw_8bit`, `bf16=True`, `gradient_checkpointing=True`
- **Sequence Length & Overhead:** `max_seq_length=1536` | Chat template preamble: `365` tokens | Mean/Max length: `562.9` / `853` tokens | `0` dropped rows
- **Loss Masking:** `completion_only (final assistant turn); system/user/turn-1 SQL/error masked`
- **Training Loss (Step 5 → Step 50):** `1.1766 → 0.5947` (**0.7045** mean loss)
- **Peak VRAM (Allocated / Reserved):** `7.176 GB` / `7.436 GB`
- **Wall-Clock Training Time:** `7.326` minutes (`429.98 s`)

### LoRA Architecture
```json
{
  "r": 16,
  "lora_alpha": 32,
  "lora_dropout": 0.05,
  "target_modules": ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
  "bias": "none",
  "task_type": "CAUSAL_LM"
}
```

---

## Quickstart & Inference

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "CohereLabs/tiny-aya-global"
adapter_id = "Mrunal1201/tiny-aya-sql-control-lora"

# 1. Load Tokenizer and Base Model in BF16 / 4-bit
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

# 2. Attach Self-Healing LoRA Adapter
model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

# 3. Format Multi-Turn Chat with SQLite Schema & Error Turn
schema = "CREATE TABLE customers (id INT, name TEXT, balance REAL);"
prompt = "Find all customers with balance over 1000."
error_feedback = "SQLite Execution Error: no such column: amount"

messages = [
    {"role": "system", "content": "You are a multilingual SQLite data assistant. Translate the question into read-only SQL.\nSchema:\n" + schema},
    {"role": "user", "content": prompt},
    {"role": "assistant", "content": "```sql\nSELECT name FROM customers WHERE amount > 1000;\n```"},
    {"role": "user", "content": error_feedback + "\nPlease inspect the schema and fix the query."}
]

input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(input_ids, max_new_tokens=256, do_sample=False)

print(tokenizer.decode(outputs[0][input_ids.shape[1]:], skip_special_tokens=True))
```

---

## Citation & Attribution

```bibtex
@misc{lexsi_track3_aya_sql_2026,
  title={Autonomous Self-Healing Multilingual SQL Agents on Cohere Aya},
  author={Mrunal1201 and Lexsi Hackathon Team},
  year={2026},
  howpublished={\url{https://github.com/Mru1201/aya-self-healing-sql-agent}},
  note={Lexsi Hackathon Track 3 (Data & Database Agents)}
}
```