Mrunal1201 commited on
Commit
30e1545
Β·
verified Β·
1 Parent(s): e1461bc

Update tiny-aya-sql-control-lora Model Card with official Lexsi links, exact AuditKIT table, and benchmark figures

Browse files
Files changed (1) hide show
  1. README.md +80 -46
README.md CHANGED
@@ -23,66 +23,100 @@ metrics:
23
  - code_eval
24
  ---
25
 
26
- # tiny-aya-sql-control-lora: Single-Turn Ablation Control Adapter
27
 
28
- Official PEFT LoRA checkpoint for the **Lexsi Hackathon Track 3 (Data & Database Agents)** submission:
29
- *Autonomous Multi-Turn Self-Healing SQL Generation via AgentTune Distillation & AuditKIT Outcome Verification*.
30
 
31
- - **Repository:** [Mrunal1201/tiny-aya-sql-control-lora](https://huggingface.co/Mrunal1201/tiny-aya-sql-control-lora)
32
- - **Base Model:** [CohereLabs/tiny-aya-global](https://huggingface.co/CohereLabs/tiny-aya-global)
33
- - **Curated Dataset:** [Mrunal1201/aya-self-healing-sql-dataset](https://huggingface.co/datasets/Mrunal1201/aya-self-healing-sql-dataset)
34
- - **Evaluation Suite:** [AuditKIT 1.1.12 Verified Outcome Predicates](https://github.com/lexsi-hackathon)
35
- - **Tuning Toolkit:** [AgentTune 1.1.0 Distill Trainer](https://github.com/lexsi-hackathon)
 
 
 
 
36
 
37
  ---
38
 
39
- ## πŸ“Œ Executive Summary
40
 
41
- Ablation control PEFT LoRA adapter for **CohereLabs/tiny-aya-global** (3.35B) trained strictly on `episode_type == 'single_turn'` (420 episodes). Serves as an experimental control to isolate the causal impact of multi-turn error-feedback distillation against standard single-turn SFT.
42
 
43
- ### Key Innovations:
44
- 1. **Agentic Self-Healing Loop:** Implements a multi-turn agent feedback loop: Turn 1 SQL $\to$ SQLite in-memory execution $\to$ real `sqlite3.OperationalError` capture $\to$ Turn 2 reasoning in `Thought:` followed by corrected SQL in ````sql ... ````.
45
- 2. **Tiny Aya Preamble Guardrail:** Enforces `max_seq_length=1536` with loss masking strictly on completion tokens to safely accommodate Tiny Aya's ~365-token system preamble.
46
- 3. **Rigorous AuditKIT Verification:** Evaluated with 95% bootstrap confidence intervals (1,000 resamples) across 200 held-out cases with 0% train/test data leakage.
47
- 4. **Multilingual Knowledge Preservation:** 96.0% reasoning retention across 10 non-English languages (Hindi, German, Kannada, Spanish, French, Portuguese, Turkish, Arabic, Chinese, Swahili).
48
 
49
  ---
50
 
51
- ## πŸ“Š Evaluation & Benchmark Results (AuditKIT)
52
 
53
- All metrics were evaluated using deterministic in-memory `sqlite3` execution on 200 strictly held-out test episodes with zero table or domain contamination:
54
 
55
- | Metric / Suite | Baseline (tiny-aya-global) | **tiny-aya-sql-control-lora** | Teacher API (Qwen-72B) | Delta vs Baseline |
56
  | :--- | :---: | :---: | :---: | :---: |
57
- | **SQL Heal Exec Match (Final)** | 57.5% | **60.0%** [53.5%, 66.5%] | 71.0% | **+2.5%** |
58
- | **Executable Query Rate** | 89.5% | **86.0%** | 98.0% | +1.5% |
59
- | **Multilingual Retention** | 96.0% | **96.0%** (10 languages) | 98.0% | 0.0% (Zero Catastrophic Forgetting) |
60
- | **SQL Injection Safety Rate** | 24.0% | **10.0%** (50 adversarial probes) | 34.0% | Verified Read-Only Constraints |
61
- | **Inference Throughput** | 79.2 tok/s | **18.5 tok/s** | 61.1 tok/s | On-device Edge Deployable |
62
- | **Cost per 1,000 Queries** | $0.059 | **$0.5934** | $0.040 (cloud API) | Full Data Sovereignty |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
63
 
64
  ---
65
 
66
- ## βš™οΈ Training Specifications & Hyperparameters
67
 
68
- Extracted directly from immutable run artifact `audit_logs/03_training_hardware_meta.json`:
 
69
 
70
- - **Hardware Infrastructure:** NVIDIA L4 (22.03 GB VRAM)
71
- - **Software Stack:** PyTorch `2.11.0+cu130`, CUDA `13.0`, PEFT `0.21.3`, TRL `1.7.1`
72
- - **Training Backend:** `AgentTune.create_distill_trainer`
73
- - **Training Subset:** `single_turn` (420 episodes)
74
- - **Global Steps:** 54 steps across 2.0 epochs
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
75
  - **Learning Rate:** `0.0002` with `cosine` schedule and `0.03` warmup
76
- - **Effective Batch Size:** 16 (4 per device Γ— 4 gradient accumulation)
77
- - **Optimizer:** `paged_adamw_8bit`
78
- - **Mixed Precision:** BF16 (`bf16=True`) with Gradient Checkpointing
79
- - **Sequence Length:** 1,536 tokens (`max_seq_length=1536`)
80
- - **Loss Masking:** `completion_only` (supervised strictly on the final assistant turn)
81
- - **Initial $\to$ Final Training Loss:** 1.2010 $\to$ **0.7045**
82
- - **Peak VRAM Allocated:** 7.176 GB
83
- - **Wall-Clock Training Time:** 7.33 minutes
84
-
85
- ### LoRA Architecture:
86
  ```json
87
  {
88
  "r": 16,
@@ -96,7 +130,7 @@ Extracted directly from immutable run artifact `audit_logs/03_training_hardware_
96
 
97
  ---
98
 
99
- ## πŸš€ Quickstart & Inference
100
 
101
  ```python
102
  import torch
@@ -130,24 +164,24 @@ messages = [
130
  {"role": "user", "content": error_feedback + "\nPlease inspect the schema and fix the query."}
131
  ]
132
 
133
- input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to("cuda")
134
 
135
  with torch.no_grad():
136
- outputs = model.generate(input_ids, max_new_tokens=256, temperature=0.0)
137
 
138
  print(tokenizer.decode(outputs[0][input_ids.shape[1]:], skip_special_tokens=True))
139
  ```
140
 
141
  ---
142
 
143
- ## πŸ“œ Citation & Attribution
144
 
145
  ```bibtex
146
  @misc{lexsi_track3_aya_sql_2026,
147
  title={Autonomous Self-Healing Multilingual SQL Agents on Cohere Aya},
148
  author={Mrunal1201 and Lexsi Hackathon Team},
149
  year={2026},
150
- howpublished={\url{https://huggingface.co/Mrunal1201/tiny-aya-sql-control-lora}},
151
  note={Lexsi Hackathon Track 3 (Data & Database Agents)}
152
  }
153
  ```
 
23
  - code_eval
24
  ---
25
 
26
+ # tiny-aya-sql-control-lora: Single-Turn Ablation Control Adapter (3.35B)
27
 
28
+ Official PEFT LoRA checkpoint for **Lexsi Hackathon Track 3 (Data & Database Agents)** (`#6` Data & Database Agents, `#8` Self-Healing Agents, `#3` Agentic Distillation):
29
+ *Autonomous Multi-Turn Self-Healing Multilingual SQL Agents via CuratorKIT Curation, AgentTune Distillation, SQLGlot AST Guardrails, and AuditKIT Verification.*
30
 
31
+ | Resource | Link |
32
+ | :--- | :--- |
33
+ | **GitHub Repository** | [`Mru1201/aya-self-healing-sql-agent`](https://github.com/Mru1201/aya-self-healing-sql-agent) |
34
+ | **Executed Colab Notebook** | [`notebooks/train_and_audit_aya_sql.ipynb`](https://github.com/Mru1201/aya-self-healing-sql-agent/blob/main/notebooks/train_and_audit_aya_sql.ipynb) |
35
+ | **This Model Adapter** | [`Mrunal1201/tiny-aya-sql-control-lora`](https://huggingface.co/Mrunal1201/tiny-aya-sql-control-lora) |
36
+ | **Base Model** | [`CohereLabs/tiny-aya-global`](https://huggingface.co/CohereLabs/tiny-aya-global) |
37
+ | **Public HF Model Collection** | [Lexsi Track 3 β€” Self-Healing SQL Aya Models](https://huggingface.co/collections/Mrunal1201/lexsi-track-3-self-healing-sql-aya-models-6af8e7fc0a621b28fefccd18) |
38
+ | **Curated Dataset & Audit Logs** | [`Mrunal1201/aya-self-healing-sql-dataset`](https://huggingface.co/datasets/Mrunal1201/aya-self-healing-sql-dataset) ([Collection](https://huggingface.co/collections/Mrunal1201/lexsi-track-3-curated-sql-trajectories-and-audit-logs-6af8e7fed8ac99740564a2fb)) |
39
+ | **Lexsi Kits Chained** | [`CuratorKIT 0.1.21`](https://github.com/Lexsi-Labs/CuratorKIT) β†’ [`AgentTune 1.1.0`](https://github.com/Lexsi-Labs/AgentTune) β†’ [`AuditKIT 1.1.12`](https://github.com/Lexsi-Labs/AuditKIT) |
40
 
41
  ---
42
 
43
+ ## Executive Summary
44
 
45
+ Supervision-structure ablation control PEFT LoRA adapter for **CohereLabs/tiny-aya-global** (3.35B), trained strictly on the `420` `single_turn` SQL episodes (`54` steps) without multi-turn `OperationalError` trajectory supervision. Isolates the causal contribution of multi-turn repair conditioning against standard single-turn SFT.
46
 
47
+ ### Key Innovations
48
+ 1. **Closed-Loop Deterministic Self-Healing:** Turn-1 SQL β†’ deterministic `SQLGlot 30.31.0` AST read-only validation β†’ ephemeral in-memory `SQLite 3.46.0` execution β†’ verbatim `sqlite3.OperationalError` feedback β†’ Turn-2 `Thought:` reasoning and repaired SQL.
49
+ 2. **Preamble-Safe Context Window (`max_seq_length=1536`):** Directly measured `365` chat-template overhead tokens on `tiny-aya-global` (across our dual-scale study, `tiny-aya-global` injects `365` tokens vs. `6` tokens on `aya-expanse-8b`), preventing completion-token truncation to zero on multi-turn episodes (`0` dropped rows).
50
+ 3. **Zero-Leakage Audit & Bootstrap Significance:** `80` training domains vs. `20` strictly held-out test domains (`0` domain overlap, `0` schema table overlap across `933`/`185` unique tables, `0` MinHash LSH overlap at `0.85`), evaluated via [`AuditKIT 1.1.12`](https://github.com/Lexsi-Labs/AuditKIT) across `1,000`-resample paired bootstrap CIs.
51
+ 4. **Multilingual Reasoning Preservation:** Maintains **`96.0%`** (`48/50`) general reasoning accuracy across 10 diverse languages (`ar`, `de`, `es`, `fr`, `hi`, `ja`, `pt`, `sw`, `tr`, `zh`).
52
 
53
  ---
54
 
55
+ ## Evaluation & Benchmark Results (`AuditKIT v1.1.12`)
56
 
57
+ ### Focused Comparison: `tiny-aya-sql-control-lora` vs. `tiny-aya-global` Base & Teacher Baseline
58
 
59
+ | Metric / Suite (`n`) | `tiny-aya-global` Base | **`tiny-aya-sql-control-lora`** | `Qwen2.5-72B-Instruct:cheapest` | Delta vs. Own Base |
60
  | :--- | :---: | :---: | :---: | :---: |
61
+ | **Turn-1 Exec Match (`n=200`)** | 56.5% `[49.5%, 63.0%]` | **58.5%** `[52.0%, 65.5%]` | 71.0% `[65.0%, 76.5%]` | **+2.0 pp** |
62
+ | **Final Self-Healed Exec Match (`n=200`)** | 57.5% `[50.5%, 64.0%]` | **60.0%** `[53.5%, 66.51%]` | 71.0% `[65.0%, 76.5%]` | **+2.5 pp** |
63
+ | **Executable SQL Validity Rate (`n=200`)** | 95.5% `[92.5%, 98.0%]` | **86.0%** `[81.5%, 90.5%]` | 100.0% `[100.0%, 100.0%]` | `-9.5 pp` |
64
+ | **Turn-1 `OperationalError` Count (`n=200`)** | `12 / 200` | **`33 / 200`** | `3 / 200` | `+21` errors |
65
+ | **Multilingual Retention (`n=50`, 10 langs)** | 96.0% `[90.0%, 100.0%]` | **96.0%** `[90.0%, 100.0%]` | 98.0% `[94.0%, 100.0%]` | `+0.0 pp` |
66
+ | **Pre-Guardrail Model Injection Safe (`n=50`)** | 24.0% `[12.0%, 36.05%]` | **10.0%** `[2.0%, 20.0%]` | 34.0% `[21.95%, 48.0%]` | `-14.0 pp` *(AST gate blocks writes)* |
67
+ | **Inference Throughput (`tok/s`)** | `79.24 tok/s` | **`18.50 tok/s`** | `61.13 tok/s` | Self-hosted L4 4-bit |
68
+ | **Mean Query Latency (`sql_heal`)** | 443.4 ms | **4,450.2 ms** | 581.8 ms *(8w)* / 2,891.0 ms *(1c)* | Air-gapped local execution |
69
+ | **Cost per 1,000 Queries** | `$0.0591` | **`$0.5934`** | `$0.0401` (cloud API) | Full data sovereignty |
70
+
71
+ ### Full 6-Configuration Benchmark & Ablation Table (`audit_logs/04_auditkit_before_after.json`)
72
+
73
+ | Model Configuration | Turn-1 Exec Match [95% CI] | Final Exec Match [95% CI] | Executable SQL Validity [95% CI] | Turn-1 Errors (`OperationalError`) | Heal Rate | Adversarial Safety (`n=50`) | Retention (`n=50`) | Throughput (`tok/s`) | Mean Latency | Cost / 1k Queries |
74
+ | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
75
+ | `CohereLabs/tiny-aya-global` Base (3.35B) | 56.5% `[49.5%, 63.0%]` | **57.5%** `[50.5%, 64.0%]` (`115/200`) | **95.5%** `[92.5%, 98.0%]` | `12 / 200` | 16.67% (`2/12`) | 24.0% `[12.0%, 36.05%]` | 96.0% `[90.0%, 100.0%]` | `79.24` | 443.4 ms | `$0.0591` |
76
+ | `tiny-aya-sql-heal-lora` (3.35B) | 60.0% `[52.5%, 66.01%]` | **60.0%** `[52.5%, 66.01%]` (`120/200`) | **91.0%** `[87.0%, 94.5%]` | `18 / 200` | 0.0% (`0/18`) | 10.0% `[2.0%, 20.0%]` | 96.0% `[90.0%, 100.0%]` | `18.95` | 4,136.4 ms | `$0.5515` |
77
+ | `tiny-aya-sql-control-lora` (3.35B, 1-Turn) | 58.5% `[52.0%, 65.5%]` | **60.0%** `[53.5%, 66.51%]` (`120/200`) | **86.0%** `[81.5%, 90.5%]` | `33 / 200` | 9.09% (`3/33`) | 10.0% `[2.0%, 20.0%]` | 96.0% `[90.0%, 100.0%]` | `18.50` | 4,450.2 ms | `$0.5934` |
78
+ | `CohereLabs/aya-expanse-8b` Base (8B) | 57.0% `[50.0%, 63.5%]` | **57.5%** `[50.5%, 64.0%]` (`115/200`) | **98.0%** `[96.0%, 99.5%]` | `11 / 200` | 9.09% (`1/11`) | 12.0% `[4.0%, 22.0%]` | 98.0% `[94.0%, 100.0%]` | `55.18` | 659.6 ms | `$0.0879` |
79
+ | `aya-8b-sql-heal-lora` (8B) | 63.5% `[57.0%, 70.0%]` | **63.5%** `[57.0%, 70.0%]` (`127/200`) | **96.0%** `[93.0%, 98.5%]` | `8 / 200` | 0.0% (`0/8`) | 14.0% `[6.0%, 24.0%]` | 96.0% `[90.0%, 100.0%]` | `35.67` | 2,307.3 ms | `$0.3076` |
80
+ | `Qwen/Qwen2.5-72B-Instruct:cheapest` (Teacher) | 71.0% `[65.0%, 76.5%]` | **71.0%** `[65.0%, 76.5%]` (`142/200`) | **100.0%** `[100.0%, 100.0%]` | `3 / 200` | 0.0% (`0/3`) | 34.0% `[21.95%, 48.0%]` | 98.0% `[94.0%, 100.0%]` | `61.13` | 581.8 ms *(8w)* / 2,891.0 ms *(1c)* | `$0.0401` |
81
 
82
  ---
83
 
84
+ ## Visual Benchmarks & Diagnostics (`assets/figures/`)
85
 
86
+ ![End-to-End Lexsi Track 3 Architecture Pipeline](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig0_architecture_pipeline.png)
87
+ *Figure 0: End-to-end closed-loop Track 3 pipeline uniting `CuratorKIT 0.1.21` curation & leak checks, `AgentTune 1.1.0` QLoRA trajectory distillation, `SQLGlot 30.31.0` read-only AST guardrails over in-memory `SQLite 3.46.0`, and `AuditKIT 1.1.12` outcome verification.*
88
 
89
+ ![Before vs After SQL Execution Accuracy and Validity with 95% Bootstrap CIs](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig1_accuracy_validity_ci.png)
90
+ *Figure 1: Turn-1 execution accuracy, final self-healed execution accuracy, and SQLite executable validity across all 6 evaluated configurations on the 200 disjoint test queries (`1,000`-resample 95% bootstrap CIs from `audit_logs/04_auditkit_before_after.json`).*
91
+
92
+ ![Supervision Structure and Scale Error Reduction Ablation](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig2_error_reduction_ablation.png)
93
+ *Figure 2: (Left) Preventative schema grounding cuts Turn-1 SQLite `OperationalError` counts from `33` (`tiny-aya-sql-control-lora`) down to `18` (`-45.5%` on 3.35B) and `8` (`-75.8%` on 8B). (Right) Complete final outcome composition across all 200 held-out test cases.*
94
+
95
+ ![Systems Latency Inversion: Tiny-Aya 3.35B vs Aya-Expanse 8B](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig3_preamble_latency_inversion.png)
96
+ *Figure 3: Systems inversion comparing `tiny-aya-sql-heal-lora` (3.35B) vs. `aya-8b-sql-heal-lora` (8B) across chat-template preamble overhead (`365` vs. `6` tokens), prompt length (`635.1` vs. `210.2`), decode throughput (`18.95` vs. `35.67 tok/s`), mean latency (`4,136.4 ms` vs. `2,307.3 ms`), L4 cost (`$0.5515` vs. `$0.3076` per 1k queries), and peak allocated VRAM (`6.331 GB` vs. `13.898 GB`).*
97
+
98
+ ![QLoRA Training Loss Convergence and Pre-Guardrail Adversarial Safety Heatmap](https://raw.githubusercontent.com/Mru1201/aya-self-healing-sql-agent/main/assets/figures/fig4_training_loss_and_safety.png)
99
+ *Figure 4: (Left) `AgentTune 1.1.0` completion-only supervised training loss curves (`logging_steps = 5`, `03_training_hardware_meta.json`). (Right) Pre-guardrail model resistance across all 6 adversarial SQL injection vectors (`n = 50`), proving why deterministic `SQLGlot` AST filtering is essential.*
100
+
101
+ ---
102
+
103
+ ## Training Specifications & Hyperparameters (`audit_logs/03_training_hardware_meta.json`)
104
+
105
+ - **Hardware Infrastructure:** `NVIDIA L4` (`22.03 GB` VRAM)
106
+ - **Software Stack:** PyTorch `2.11.0+cu130`, CUDA `13.0`, PEFT `0.21.3`, TRL `1.7.1`, [`AgentTune 1.1.0`](https://github.com/Lexsi-Labs/AgentTune)
107
+ - **Training Backend:** `agenttune.create_distill_trainer`
108
+ - **Training Subset:** `single_turn` (`420` episodes)
109
+ - **Global Steps:** `54` steps across `2.0` epochs
110
  - **Learning Rate:** `0.0002` with `cosine` schedule and `0.03` warmup
111
+ - **Effective Batch Size:** `16` (`4` per device Γ— `4` gradient accumulation)
112
+ - **Optimizer & Precision:** `paged_adamw_8bit`, `bf16=True`, `gradient_checkpointing=True`
113
+ - **Sequence Length & Overhead:** `max_seq_length=1536` | Chat template preamble: `365` tokens | Mean/Max length: `562.9` / `853` tokens | `0` dropped rows
114
+ - **Loss Masking:** `completion_only (final assistant turn); system/user/turn-1 SQL/error masked`
115
+ - **Training Loss (Step 5 β†’ Step 50):** `1.1766 β†’ 0.5947` (**0.7045** mean loss)
116
+ - **Peak VRAM (Allocated / Reserved):** `7.176 GB` / `7.436 GB`
117
+ - **Wall-Clock Training Time:** `7.326` minutes (`429.98 s`)
118
+
119
+ ### LoRA Architecture
 
120
  ```json
121
  {
122
  "r": 16,
 
130
 
131
  ---
132
 
133
+ ## Quickstart & Inference
134
 
135
  ```python
136
  import torch
 
164
  {"role": "user", "content": error_feedback + "\nPlease inspect the schema and fix the query."}
165
  ]
166
 
167
+ input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
168
 
169
  with torch.no_grad():
170
+ outputs = model.generate(input_ids, max_new_tokens=256, do_sample=False)
171
 
172
  print(tokenizer.decode(outputs[0][input_ids.shape[1]:], skip_special_tokens=True))
173
  ```
174
 
175
  ---
176
 
177
+ ## Citation & Attribution
178
 
179
  ```bibtex
180
  @misc{lexsi_track3_aya_sql_2026,
181
  title={Autonomous Self-Healing Multilingual SQL Agents on Cohere Aya},
182
  author={Mrunal1201 and Lexsi Hackathon Team},
183
  year={2026},
184
+ howpublished={\url{https://github.com/Mru1201/aya-self-healing-sql-agent}},
185
  note={Lexsi Hackathon Track 3 (Data & Database Agents)}
186
  }
187
  ```