Blame for README.md

---
license: cc-by-nc-4.0
base_model: CohereLabs/tiny-aya-global
library_name: peft
pipeline_tag: text-generation
language:
- en
tags:
- text-to-sql
- agent
- tool-use
- lora
- qlora
- agenttune
- auditkit
datasets:
- satish9177/tiny-aya-sql-agent-data
- trl-lab/SQaLe-2-text-to-SQL-Queries
---

# Tiny Aya SQL agent (B)

A LoRA adapter that turns Cohere's **Tiny Aya Global** into a tool-using agent for SQLite databases.
This is **B**, the model submitted to the Lexsi AI Safety & Alignment Hackathon, Track 3 (Agentic
Fine-tuning).
- **How it was made:** QLoRA supervised fine-tuning on 1,500 verified episodes (the
  [A1500 adapter](https://huggingface.co/satish9177/tiny-aya-sql-agent-a1500)), plus one round of
  self-training on 306 of its own verified answers (rejection-sampling fine-tuning).
- **Tools:** built with Lexsi **AgentTune**; every score is re-checked by **AuditKIT**.

- Write-up: [Teaching Tiny Aya to query databases (HF Article)](https://huggingface.co/blog/satish9177/teaching-tiny-aya-to-query-databases-what-fine-tun)
- Code, plans and full results: [github.com/satish9177/aya-db-agent](https://github.com/satish9177/aya-db-agent)
- Data, trajectories and logs: [satish9177/tiny-aya-sql-agent-data](https://huggingface.co/datasets/satish9177/tiny-aya-sql-agent-data)

## What it does

The agent gets a question and four tools over a read-only SQLite database:

| Tool | Does |
|---|---|
| `list_tables()` | lists the tables |
| `describe_table(table)` | shows columns, foreign keys and three sample rows |
| `run_sql(query)` | runs one read-only `SELECT` and previews up to 20 rows |
| `submit_sql(query)` | submits the final query and ends the episode |

- **Turn format:** it writes one tool call per turn as `[{"tool_name": ..., "parameters": {...}}]` and has
  **eight tool steps**.
- **Prompt:** an exact example is in `prompt_example.json`.

## Results (strict, final sets never used for any choice)

Strict means the submitted query's full result equals the reference, and the query reads a real table.

| Set | Base (untuned) | A1500 | **B (this model)** | gpt-6-luna (closed, same test) |
|---|---:|---:|---:|---:|
| SQaLe final (144, unseen databases) | 2.8% | 31.3% | **30.6%** | 66.0% |
| BIRD Mini-Dev (493, never trained on) | 0.0% | 1.6% | **1.4%** | not run |
| Shop retention, dev (40) | 0 | 40 | **40** | not run |
| Shop retention, multi-hop (40) | 0 | 1 | **0** | not run |

**Where it works:**
- It reads the schema before querying.
- Its submitted queries almost always run: 2 failures across all final sets, against 377 for the untuned
  model.
- It is close to the closed model on simple one-table lookups (29 vs 31 of 36).

**Where it does not:**
- Questions over 3+ tables or needing several steps (1 of 51).
- Large schemas (BIRD).
- Recovering after a failed query; it never does.

Running out of the eight tool steps is the main failure.

## How to load

```python
import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "CohereLabs/tiny-aya-global", revision="af89d219b53ed9b13b8a4645f9c8028973510324",
    dtype=torch.float16, device_map="auto",
    quantization_config=BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                                           bnb_4bit_use_double_quant=True,
                                           bnb_4bit_compute_dtype=torch.float16))
model = PeftModel.from_pretrained(base, "satish9177/tiny-aya-sql-agent")
```

**Tokenizer: load it from the base model's `tokenizer.json`**, as `ayadb.engine.official_tokenizer` does in
the code repo. In transformers 5.x, `AutoTokenizer` replaces Tiny Aya's pre-tokenizer
([huggingface/transformers#49192](https://github.com/huggingface/transformers/issues/49192)), and the token
ids then differ from the ones used in training.

To run the full agent loop and the scoring, use `ayadb.sqlrunner.evaluate_with_engine` from the code repo.

## Training

| | |
|---|---|
| Starting point | A1500 adapter, step 180 (QLoRA SFT on 1,265 SQaLe + 235 shop episodes) |
| Self-training data | 306 verified, correct attempts by A1500 on 229 held-out SQaLe questions |
| Quantization | 4-bit NF4, double quantization, fp16 compute |
| LoRA | rank 16, alpha 32, dropout 0.05; q, k, v, o, gate, up and down projections |
| Optimization | 2 passes, learning rate 1e-4, effective batch 16, 40 updates |
| Loss | the model's own turns only; tool outputs are never trained |
| Hardware | one Tesla T4 |
| Adapter SHA256 | `90b04823368f50d6ef5ce85c171fffff317b11a3bf43a591cf3952f01b076837` |

**Self-training vs a matched control:**
- A token-matched control trained on teacher episodes for the same questions.
- On dev, B beat it 4 tasks to 0 (p = 0.125) and scored 40/95, against 37 for A1500.
- On the final sets, B and A1500 are the same within noise.

Training records are in `training/`. Each run was preregistered and ran once; the plans are in the code
repo's `experiments/` folder.

## Limitations

- **Accuracy:** it is weak on complex and multi-step SQL, and on large schemas.
- **Recovery:** it does not recover from errors.
- **Language:** English questions only.
- **Closed-model comparison:** a cheap closed model (gpt-6-luna) scored far higher on the same test, at a
  lower cost per correct answer than this model on a rented T4.
- **Use:** research and non-commercial use only (see the licence).

## Licence and credits

- **Licence:** CC-BY-NC-4.0, following the base model, **Tiny Aya Global** by Cohere Labs.
- **AgentTune and AuditKIT** (Lexsi Labs) are source-available, non-commercial tools.
- **SQaLe** by Cornelius Wolff, Daniel Gomm and Madelon Hulsebos (2026). Its card declares MIT.
- **BIRD Mini-Dev** by Li et al. (NeurIPS 2024), CC BY-SA 4.0. It was used only for final evaluation.