Mrunal1201's picture
Update tiny-aya-sql-heal-lora Model Card with official Lexsi links, exact AuditKIT table, and benchmark figures
ca8155e verified
|
Raw History Blame Contribute Delete
13.6 kB
---
base_model: CohereLabs/tiny-aya-global
library_name: peft
pipeline_tag: text-generation
tags:
- lexsi-hackathon
- agenttune
- auditkit
- curatorkit
- text-to-sql
- cohere-aya
- self-healing
- lora
datasets:
- Mrunal1201/aya-self-healing-sql-dataset
language:
- en
- hi
- de
- kn
license: apache-2.0
metrics:
- code_eval
---
# tiny-aya-sql-heal-lora: Primary Multi-Turn Self-Healing SQL Adapter (3.35B)
Official PEFT LoRA checkpoint for **Lexsi Hackathon Track 3 (Data & Database Agents)** (`#6` Data & Database Agents, `#8` Self-Healing Agents, `#3` Agentic Distillation):
*Autonomous Multi-Turn Self-Healing Multilingual SQL Agents via CuratorKIT Curation, AgentTune Distillation, SQLGlot AST Guardrails, and AuditKIT Verification.*
| Resource | Link |
| :--- | :--- |
| **GitHub Repository** | [`Mru1201/aya-self-healing-sql-agent`](https://github.com/Mru1201/aya-self-healing-sql-agent) |
| **Executed Colab Notebook** | [`notebooks/train_and_audit_aya_sql.ipynb`](https://github.com/Mru1201/aya-self-healing-sql-agent/blob/main/notebooks/train_and_audit_aya_sql.ipynb) |
| **This Model Adapter** | [`Mrunal1201/tiny-aya-sql-heal-lora`](https://huggingface.co/Mrunal1201/tiny-aya-sql-heal-lora) |
| **Base Model** | [`CohereLabs/tiny-aya-global`](https://huggingface.co/CohereLabs/tiny-aya-global) |
| **Public HF Model Collection** | [Lexsi Track 3 — Self-Healing SQL Aya Models](https://huggingface.co/collections/Mrunal1201/lexsi-track-3-self-healing-sql-aya-models-6af8e7fc0a621b28fefccd18) |
| **Curated Dataset & Audit Logs** | [`Mrunal1201/aya-self-healing-sql-dataset`](https://huggingface.co/datasets/Mrunal1201/aya-self-healing-sql-dataset) ([Collection](https://huggingface.co/collections/Mrunal1201/lexsi-track-3-curated-sql-trajectories-and-audit-logs-6af8e7fed8ac99740564a2fb)) |
| **Lexsi Kits Chained** | [`CuratorKIT 0.1.21`](https://github.com/Lexsi-Labs/CuratorKIT) → [`AgentTune 1.1.0`](https://github.com/Lexsi-Labs/AgentTune) → [`AuditKIT 1.1.12`](https://github.com/Lexsi-Labs/AuditKIT) |
| **HF Static Article Space** | [Interactive Technical Report](https://huggingface.co/spaces/Mrunal1201/aya-self-healing-sql-article) ([Direct Embed](https://mrunal1201-aya-self-healing-sql-article.static.hf.space)) |
| **HF Community Articles** | [Model Discussion #1](https://huggingface.co/Mrunal1201/tiny-aya-sql-heal-lora/discussions/1) · [Dataset Discussion #1](https://huggingface.co/datasets/Mrunal1201/aya-self-healing-sql-dataset/discussions/1) |
---
## Executive Summary
Primary PEFT LoRA adapter for **CohereLabs/tiny-aya-global** (3.35B) trained on all `1,200` [CuratorKIT](https://github.com/Lexsi-Labs/CuratorKIT)-certified multilingual SQL episodes (`780` `self_healing` + `420` `single_turn`) for **Lexsi Hackathon Track 3** (`#6` Data & Database Agents, `#8` Self-Healing Agents, `#3` Agentic Distillation). Cuts Turn-1 SQLite `OperationalError` crashes by **45.5%** over single-turn SFT (`33 -> 18`), improves Executable SQL Validity over control (`86.0% -> 91.0%`), and trains in just `6.331 GB` peak VRAM.
### Key Innovations
1. **Closed-Loop Deterministic Self-Healing:** Turn-1 SQL → deterministic `SQLGlot 30.31.0` AST read-only validation → ephemeral in-memory `SQLite 3.46.0` execution → verbatim `sqlite3.OperationalError` feedback → Turn-2 `Thought:` reasoning and repaired SQL.
2. **Preamble-Safe Context Window (`max_seq_length=1536`):** Directly measured `365` chat-template overhead tokens on `tiny-aya-global` (across our dual-scale study, `tiny-aya-global` injects `365` tokens vs. `6` tokens on `aya-expanse-8b`), preventing completion-token truncation to zero on multi-turn episodes (`0` dropped rows).
3. **Zero-Leakage Audit & Bootstrap Significance:** `80` training domains vs. `20` strictly held-out test domains (`0` domain overlap, `0` schema table overlap across `933`/`185` unique tables, `0` MinHash LSH overlap at `0.85`), evaluated via [`AuditKIT 1.1.12`](https://github.com/Lexsi-Labs/AuditKIT) across `1,000`-resample paired bootstrap CIs.
4. **Multilingual Reasoning Preservation:** Maintains **`96.0%`** (`48/50`) general reasoning accuracy across 10 diverse languages (`ar`, `de`, `es`, `fr`, `hi`, `ja`, `pt`, `sw`, `tr`, `zh`).
---
## Evaluation & Benchmark Results (`AuditKIT v1.1.12`)
### Focused Comparison: `tiny-aya-sql-heal-lora` vs. `tiny-aya-global` Base & Teacher Baseline
| Metric / Suite (`n`) | `tiny-aya-global` Base | **`tiny-aya-sql-heal-lora`** | `Qwen2.5-72B-Instruct:cheapest` | Delta vs. Own Base |
| :--- | :---: | :---: | :---: | :---: |
| **Turn-1 Exec Match (`n=200`)** | 56.5% `[49.5%, 63.0%]` | **60.0%** `[52.5%, 66.01%]` | 71.0% `[65.0%, 76.5%]` | **+3.5 pp** |
| **Final Self-Healed Exec Match (`n=200`)** | 57.5% `[50.5%, 64.0%]` | **60.0%** `[52.5%, 66.01%]` | 71.0% `[65.0%, 76.5%]` | **+2.5 pp** |
| **Executable SQL Validity Rate (`n=200`)** | 95.5% `[92.5%, 98.0%]` | **91.0%** `[87.0%, 94.5%]` | 100.0% `[100.0%, 100.0%]` | `-4.5 pp` |
| **Turn-1 `OperationalError` Count (`n=200`)** | `12 / 200` | **`18 / 200`** | `3 / 200` | `+6` errors |
| **Multilingual Retention (`n=50`, 10 langs)** | 96.0% `[90.0%, 100.0%]` | **96.0%** `[90.0%, 100.0%]` | 98.0% `[94.0%, 100.0%]` | `+0.0 pp` |
| **Pre-Guardrail Model Injection Safe (`n=50`)** | 24.0% `[12.0%, 36.05%]` | **10.0%** `[2.0%, 20.0%]` | 34.0% `[21.95%, 48.0%]` | `-14.0 pp` *(AST gate blocks writes)* |
| **Inference Throughput (`tok/s`)** | `79.24 tok/s` | **`18.95 tok/s`** | `61.13 tok/s` | Self-hosted L4 4-bit |
| **Mean Query Latency (`sql_heal`)** | 443.4 ms | **4,136.4 ms** | 581.8 ms *(8w)* / 2,891.0 ms *(1c)* | Air-gapped local execution |
| **Cost per 1,000 Queries** | `$0.0591` | **`$0.5515`** | `$0.0401` (cloud API) | Full data sovereignty |
### Full 6-Configuration Benchmark & Ablation Table (`audit_logs/04_auditkit_before_after.json`)
| Model Configuration | Turn-1 Exec Match [95% CI] | Final Exec Match [95% CI] | Executable SQL Validity [95% CI] | Turn-1 Errors (`OperationalError`) | Heal Rate | Adversarial Safety (`n=50`) | Retention (`n=50`) | Throughput (`tok/s`) | Mean Latency | Cost / 1k Queries |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| `CohereLabs/tiny-aya-global` Base (3.35B) | 56.5% `[49.5%, 63.0%]` | **57.5%** `[50.5%, 64.0%]` (`115/200`) | **95.5%** `[92.5%, 98.0%]` | `12 / 200` | 16.67% (`2/12`) | 24.0% `[12.0%, 36.05%]` | 96.0% `[90.0%, 100.0%]` | `79.24` | 443.4 ms | `$0.0591` |
| `tiny-aya-sql-heal-lora` (3.35B) | 60.0% `[52.5%, 66.01%]` | **60.0%** `[52.5%, 66.01%]` (`120/200`) | **91.0%** `[87.0%, 94.5%]` | `18 / 200` | 0.0% (`0/18`) | 10.0% `[2.0%, 20.0%]` | 96.0% `[90.0%, 100.0%]` | `18.95` | 4,136.4 ms | `$0.5515` |
| `tiny-aya-sql-control-lora` (3.35B, 1-Turn) | 58.5% `[52.0%, 65.5%]` | **60.0%** `[53.5%, 66.51%]` (`120/200`) | **86.0%** `[81.5%, 90.5%]` | `33 / 200` | 9.09% (`3/33`) | 10.0% `[2.0%, 20.0%]` | 96.0% `[90.0%, 100.0%]` | `18.50` | 4,450.2 ms | `$0.5934` |
| `CohereLabs/aya-expanse-8b` Base (8B) | 57.0% `[50.0%, 63.5%]` | **57.5%** `[50.5%, 64.0%]` (`115/200`) | **98.0%** `[96.0%, 99.5%]` | `11 / 200` | 9.09% (`1/11`) | 12.0% `[4.0%, 22.0%]` | 98.0% `[94.0%, 100.0%]` | `55.18` | 659.6 ms | `$0.0879` |
| `aya-8b-sql-heal-lora` (8B) | 63.5% `[57.0%, 70.0%]` | **63.5%** `[57.0%, 70.0%]` (`127/200`) | **96.0%** `[93.0%, 98.5%]` | `8 / 200` | 0.0% (`0/8`) | 14.0% `[6.0%, 24.0%]` | 96.0% `[90.0%, 100.0%]` | `35.67` | 2,307.3 ms | `$0.3076` |
| `Qwen/Qwen2.5-72B-Instruct:cheapest` (Teacher) | 71.0% `[65.0%, 76.5%]` | **71.0%** `[65.0%, 76.5%]` (`142/200`) | **100.0%** `[100.0%, 100.0%]` | `3 / 200` | 0.0% (`0/3`) | 34.0% `[21.95%, 48.0%]` | 98.0% `[94.0%, 100.0%]` | `61.13` | 581.8 ms *(8w)* / 2,891.0 ms *(1c)* | `$0.0401` |
---
## Visual Benchmarks & Diagnostics (`assets/figures/`)
![End-to-End Lexsi Track 3 Architecture Pipeline](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig0_architecture_pipeline.png)
*Figure 0: End-to-end closed-loop Track 3 pipeline uniting `CuratorKIT 0.1.21` curation & leak checks, `AgentTune 1.1.0` QLoRA trajectory distillation, `SQLGlot 30.31.0` read-only AST guardrails over in-memory `SQLite 3.46.0`, and `AuditKIT 1.1.12` outcome verification.*
![Before vs After SQL Execution Accuracy and Validity with 95% Bootstrap CIs](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig1_accuracy_validity_ci.png)
*Figure 1: Turn-1 execution accuracy, final self-healed execution accuracy, and SQLite executable validity across all 6 evaluated configurations on the 200 disjoint test queries (`1,000`-resample 95% bootstrap CIs from `audit_logs/04_auditkit_before_after.json`).*
![Supervision Structure and Scale Error Reduction Ablation](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig2_error_reduction_ablation.png)
*Figure 2: (Left) Preventative schema grounding cuts Turn-1 SQLite `OperationalError` counts from `33` (`tiny-aya-sql-control-lora`) down to `18` (`-45.5%` on 3.35B) and `8` (`-75.8%` on 8B). (Right) Complete final outcome composition across all 200 held-out test cases.*
![Systems Latency Inversion: Tiny-Aya 3.35B vs Aya-Expanse 8B](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig3_preamble_latency_inversion.png)
*Figure 3: Systems inversion comparing `tiny-aya-sql-heal-lora` (3.35B) vs. `aya-8b-sql-heal-lora` (8B) across chat-template preamble overhead (`365` vs. `6` tokens), prompt length (`635.1` vs. `210.2`), decode throughput (`18.95` vs. `35.67 tok/s`), mean latency (`4,136.4 ms` vs. `2,307.3 ms`), L4 cost (`$0.5515` vs. `$0.3076` per 1k queries), and peak allocated VRAM (`6.331 GB` vs. `13.898 GB`).*
![QLoRA Training Loss Convergence and Pre-Guardrail Adversarial Safety Heatmap](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig4_training_loss_and_safety.png)
*Figure 4: (Left) `AgentTune 1.1.0` completion-only supervised training loss curves (`logging_steps = 5`, `03_training_hardware_meta.json`). (Right) Pre-guardrail model resistance across all 6 adversarial SQL injection vectors (`n = 50`), proving why deterministic `SQLGlot` AST filtering is essential.*
---
## Training Specifications & Hyperparameters (`audit_logs/03_training_hardware_meta.json`)
- **Hardware Infrastructure:** `NVIDIA L4` (`22.03 GB` VRAM)
- **Software Stack:** PyTorch `2.11.0+cu130`, CUDA `13.0`, PEFT `0.21.3`, TRL `1.7.1`, [`AgentTune 1.1.0`](https://github.com/Lexsi-Labs/AgentTune)
- **Training Backend:** `agenttune.create_distill_trainer`
- **Training Subset:** `full` (`1200` episodes)
- **Global Steps:** `150` steps across `2.0` epochs
- **Learning Rate:** `0.0002` with `cosine` schedule and `0.03` warmup
- **Effective Batch Size:** `16` (`4` per device × `4` gradient accumulation)
- **Optimizer & Precision:** `paged_adamw_8bit`, `bf16=True`, `gradient_checkpointing=True`
- **Sequence Length & Overhead:** `max_seq_length=1536` | Chat template preamble: `365` tokens | Mean/Max length: `630.0` / `1085` tokens | `0` dropped rows
- **Loss Masking:** `completion_only (final assistant turn); system/user/turn-1 SQL/error masked`
- **Training Loss (Step 5 → Step 150):** `1.2010 → 0.3071` (**0.4081** mean loss)
- **Peak VRAM (Allocated / Reserved):** `6.331 GB` / `6.648 GB`
- **Wall-Clock Training Time:** `24.283` minutes (`1430.27 s`)
### LoRA Architecture
```json
{
"r": 16,
"lora_alpha": 32,
"lora_dropout": 0.05,
"target_modules": ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
"bias": "none",
"task_type": "CAUSAL_LM"
}
```
---
## Quickstart & Inference
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base_model_id = "CohereLabs/tiny-aya-global"
adapter_id = "Mrunal1201/tiny-aya-sql-heal-lora"
# 1. Load Tokenizer and Base Model in BF16 / 4-bit
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
base_model = AutoModelForCausalLM.from_pretrained(
base_model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
)
# 2. Attach Self-Healing LoRA Adapter
model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()
# 3. Format Multi-Turn Chat with SQLite Schema & Error Turn
schema = "CREATE TABLE customers (id INT, name TEXT, balance REAL);"
prompt = "Find all customers with balance over 1000."
error_feedback = "SQLite Execution Error: no such column: amount"
messages = [
{"role": "system", "content": "You are a multilingual SQLite data assistant. Translate the question into read-only SQL.\nSchema:\n" + schema},
{"role": "user", "content": prompt},
{"role": "assistant", "content": "```sql\nSELECT name FROM customers WHERE amount > 1000;\n```"},
{"role": "user", "content": error_feedback + "\nPlease inspect the schema and fix the query."}
]
input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
with torch.no_grad():
outputs = model.generate(input_ids, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(outputs[0][input_ids.shape[1]:], skip_special_tokens=True))
```
---
## Citation & Attribution
```bibtex
@misc{lexsi_track3_aya_sql_2026,
title={Autonomous Self-Healing Multilingual SQL Agents on Cohere Aya},
author={Mrunal1201 and Lexsi Hackathon Team},
year={2026},
howpublished={\url{https://github.com/Mru1201/aya-self-healing-sql-agent}},
note={Lexsi Hackathon Track 3 (Data & Database Agents)}
}
```