Claude Opus 5.5

Link the HF Article write-up

12c183e verified about 3 hours ago

5.84 kB

license: cc-by-nc-4.0
base_model: CohereLabs/tiny-aya-global
library_name: peft
pipeline_tag: text-generation
language:
  - en
tags:
  - text-to-sql
  - agent
  - tool-use
  - lora
  - qlora
  - agenttune
  - auditkit
datasets:
  - satish9177/tiny-aya-sql-agent-data
  - trl-lab/SQaLe-2-text-to-SQL-Queries

Tiny Aya SQL agent (B)

A LoRA adapter that turns Cohere's Tiny Aya Global into a tool-using agent for SQLite databases. This is B, the model submitted to the Lexsi AI Safety & Alignment Hackathon, Track 3 (Agentic Fine-tuning).

How it was made: QLoRA supervised fine-tuning on 1,500 verified episodes (the A1500 adapter), plus one round of self-training on 306 of its own verified answers (rejection-sampling fine-tuning).
Tools: built with Lexsi AgentTune; every score is re-checked by AuditKIT.
Write-up: Teaching Tiny Aya to query databases (HF Article)
Code, plans and full results: github.com/satish9177/aya-db-agent
Data, trajectories and logs: satish9177/tiny-aya-sql-agent-data

What it does

The agent gets a question and four tools over a read-only SQLite database:

Tool	Does
`list_tables()`	lists the tables
`describe_table(table)`	shows columns, foreign keys and three sample rows
`run_sql(query)`	runs one read-only `SELECT` and previews up to 20 rows
`submit_sql(query)`	submits the final query and ends the episode

Turn format: it writes one tool call per turn as [{"tool_name": ..., "parameters": {...}}] and has eight tool steps.
Prompt: an exact example is in prompt_example.json.

Results (strict, final sets never used for any choice)

Strict means the submitted query's full result equals the reference, and the query reads a real table.

Set	Base (untuned)	A1500	B (this model)	gpt-6-luna (closed, same test)
SQaLe final (144, unseen databases)	2.8%	31.3%	30.6%	66.0%
BIRD Mini-Dev (493, never trained on)	0.0%	1.6%	1.4%	not run
Shop retention, dev (40)	0	40	40	not run
Shop retention, multi-hop (40)	0	1	0	not run

Where it works:

It reads the schema before querying.
Its submitted queries almost always run: 2 failures across all final sets, against 377 for the untuned model.
It is close to the closed model on simple one-table lookups (29 vs 31 of 36).

Where it does not:

Questions over 3+ tables or needing several steps (1 of 51).
Large schemas (BIRD).
Recovering after a failed query; it never does.

Running out of the eight tool steps is the main failure.

How to load

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "CohereLabs/tiny-aya-global", revision="af89d219b53ed9b13b8a4645f9c8028973510324",
    dtype=torch.float16, device_map="auto",
    quantization_config=BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                                           bnb_4bit_use_double_quant=True,
                                           bnb_4bit_compute_dtype=torch.float16))
model = PeftModel.from_pretrained(base, "satish9177/tiny-aya-sql-agent")

Tokenizer: load it from the base model's tokenizer.json, as ayadb.engine.official_tokenizer does in the code repo. In transformers 5.x, AutoTokenizer replaces Tiny Aya's pre-tokenizer (huggingface/transformers#49192), and the token ids then differ from the ones used in training.

To run the full agent loop and the scoring, use ayadb.sqlrunner.evaluate_with_engine from the code repo.

Training


Starting point	A1500 adapter, step 180 (QLoRA SFT on 1,265 SQaLe + 235 shop episodes)
Self-training data	306 verified, correct attempts by A1500 on 229 held-out SQaLe questions
Quantization	4-bit NF4, double quantization, fp16 compute
LoRA	rank 16, alpha 32, dropout 0.05; q, k, v, o, gate, up and down projections
Optimization	2 passes, learning rate 1e-4, effective batch 16, 40 updates
Loss	the model's own turns only; tool outputs are never trained
Hardware	one Tesla T4
Adapter SHA256	`90b04823368f50d6ef5ce85c171fffff317b11a3bf43a591cf3952f01b076837`

Self-training vs a matched control:

A token-matched control trained on teacher episodes for the same questions.
On dev, B beat it 4 tasks to 0 (p = 0.125) and scored 40/95, against 37 for A1500.
On the final sets, B and A1500 are the same within noise.

Training records are in training/. Each run was preregistered and ran once; the plans are in the code repo's experiments/ folder.

Limitations

Accuracy: it is weak on complex and multi-step SQL, and on large schemas.
Recovery: it does not recover from errors.
Language: English questions only.
Closed-model comparison: a cheap closed model (gpt-6-luna) scored far higher on the same test, at a lower cost per correct answer than this model on a rented T4.
Use: research and non-commercial use only (see the licence).

Licence and credits

Licence: CC-BY-NC-4.0, following the base model, Tiny Aya Global by Cohere Labs.
AgentTune and AuditKIT (Lexsi Labs) are source-available, non-commercial tools.
SQaLe by Cornelius Wolff, Daniel Gomm and Madelon Hulsebos (2026). Its card declares MIT.
BIRD Mini-Dev by Li et al. (NeurIPS 2024), CC BY-SA 4.0. It was used only for final evaluation.