| --- |
| license: cc-by-nc-4.0 |
| base_model: CohereLabs/tiny-aya-global |
| library_name: peft |
| pipeline_tag: text-generation |
| language: |
| - en |
| tags: |
| - text-to-sql |
| - agent |
| - tool-use |
| - lora |
| - qlora |
| - agenttune |
| - auditkit |
| datasets: |
| - satish9177/tiny-aya-sql-agent-data |
| - trl-lab/SQaLe-2-text-to-SQL-Queries |
| --- |
| |
| # Tiny Aya SQL agent (B) |
|
|
| A LoRA adapter that turns Cohere's **Tiny Aya Global** into a tool-using agent for SQLite databases. |
| This is **B**, the model submitted to the Lexsi AI Safety & Alignment Hackathon, Track 3 (Agentic |
| Fine-tuning). |
| - **How it was made:** QLoRA supervised fine-tuning on 1,500 verified episodes (the |
| [A1500 adapter](https://huggingface.co/satish9177/tiny-aya-sql-agent-a1500)), plus one round of |
| self-training on 306 of its own verified answers (rejection-sampling fine-tuning). |
| - **Tools:** built with Lexsi **AgentTune**; every score is re-checked by **AuditKIT**. |
|
|
| - Write-up: [Teaching Tiny Aya to query databases (HF Article)](https://huggingface.co/blog/satish9177/teaching-tiny-aya-to-query-databases-what-fine-tun) |
| - Code, plans and full results: [github.com/satish9177/aya-db-agent](https://github.com/satish9177/aya-db-agent) |
| - Data, trajectories and logs: [satish9177/tiny-aya-sql-agent-data](https://huggingface.co/datasets/satish9177/tiny-aya-sql-agent-data) |
|
|
| ## What it does |
|
|
| The agent gets a question and four tools over a read-only SQLite database: |
|
|
| | Tool | Does | |
| |---|---| |
| | `list_tables()` | lists the tables | |
| | `describe_table(table)` | shows columns, foreign keys and three sample rows | |
| | `run_sql(query)` | runs one read-only `SELECT` and previews up to 20 rows | |
| | `submit_sql(query)` | submits the final query and ends the episode | |
|
|
| - **Turn format:** it writes one tool call per turn as `[{"tool_name": ..., "parameters": {...}}]` and has |
| **eight tool steps**. |
| - **Prompt:** an exact example is in `prompt_example.json`. |
|
|
| ## Results (strict, final sets never used for any choice) |
|
|
| Strict means the submitted query's full result equals the reference, and the query reads a real table. |
|
|
| | Set | Base (untuned) | A1500 | **B (this model)** | gpt-6-luna (closed, same test) | |
| |---|---:|---:|---:|---:| |
| | SQaLe final (144, unseen databases) | 2.8% | 31.3% | **30.6%** | 66.0% | |
| | BIRD Mini-Dev (493, never trained on) | 0.0% | 1.6% | **1.4%** | not run | |
| | Shop retention, dev (40) | 0 | 40 | **40** | not run | |
| | Shop retention, multi-hop (40) | 0 | 1 | **0** | not run | |
|
|
| **Where it works:** |
| - It reads the schema before querying. |
| - Its submitted queries almost always run: 2 failures across all final sets, against 377 for the untuned |
| model. |
| - It is close to the closed model on simple one-table lookups (29 vs 31 of 36). |
|
|
| **Where it does not:** |
| - Questions over 3+ tables or needing several steps (1 of 51). |
| - Large schemas (BIRD). |
| - Recovering after a failed query; it never does. |
|
|
| Running out of the eight tool steps is the main failure. |
|
|
| ## How to load |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, BitsAndBytesConfig |
| from peft import PeftModel |
| |
| base = AutoModelForCausalLM.from_pretrained( |
| "CohereLabs/tiny-aya-global", revision="af89d219b53ed9b13b8a4645f9c8028973510324", |
| dtype=torch.float16, device_map="auto", |
| quantization_config=BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", |
| bnb_4bit_use_double_quant=True, |
| bnb_4bit_compute_dtype=torch.float16)) |
| model = PeftModel.from_pretrained(base, "satish9177/tiny-aya-sql-agent") |
| ``` |
|
|
| **Tokenizer: load it from the base model's `tokenizer.json`**, as `ayadb.engine.official_tokenizer` does in |
| the code repo. In transformers 5.x, `AutoTokenizer` replaces Tiny Aya's pre-tokenizer |
| ([huggingface/transformers#49192](https://github.com/huggingface/transformers/issues/49192)), and the token |
| ids then differ from the ones used in training. |
|
|
| To run the full agent loop and the scoring, use `ayadb.sqlrunner.evaluate_with_engine` from the code repo. |
|
|
| ## Training |
|
|
| | | | |
| |---|---| |
| | Starting point | A1500 adapter, step 180 (QLoRA SFT on 1,265 SQaLe + 235 shop episodes) | |
| | Self-training data | 306 verified, correct attempts by A1500 on 229 held-out SQaLe questions | |
| | Quantization | 4-bit NF4, double quantization, fp16 compute | |
| | LoRA | rank 16, alpha 32, dropout 0.05; q, k, v, o, gate, up and down projections | |
| | Optimization | 2 passes, learning rate 1e-4, effective batch 16, 40 updates | |
| | Loss | the model's own turns only; tool outputs are never trained | |
| | Hardware | one Tesla T4 | |
| | Adapter SHA256 | `90b04823368f50d6ef5ce85c171fffff317b11a3bf43a591cf3952f01b076837` | |
|
|
| **Self-training vs a matched control:** |
| - A token-matched control trained on teacher episodes for the same questions. |
| - On dev, B beat it 4 tasks to 0 (p = 0.125) and scored 40/95, against 37 for A1500. |
| - On the final sets, B and A1500 are the same within noise. |
|
|
| Training records are in `training/`. Each run was preregistered and ran once; the plans are in the code |
| repo's `experiments/` folder. |
|
|
| ## Limitations |
|
|
| - **Accuracy:** it is weak on complex and multi-step SQL, and on large schemas. |
| - **Recovery:** it does not recover from errors. |
| - **Language:** English questions only. |
| - **Closed-model comparison:** a cheap closed model (gpt-6-luna) scored far higher on the same test, at a |
| lower cost per correct answer than this model on a rented T4. |
| - **Use:** research and non-commercial use only (see the licence). |
|
|
| ## Licence and credits |
|
|
| - **Licence:** CC-BY-NC-4.0, following the base model, **Tiny Aya Global** by Cohere Labs. |
| - **AgentTune and AuditKIT** (Lexsi Labs) are source-available, non-commercial tools. |
| - **SQaLe** by Cornelius Wolff, Daniel Gomm and Madelon Hulsebos (2026). Its card declares MIT. |
| - **BIRD Mini-Dev** by Li et al. (NeurIPS 2024), CC BY-SA 4.0. It was used only for final evaluation. |
|
|
