ToolTune Policy β reliable tool calling (Lexsi Track 3: Agentic Fine-tuning)
Submission by Shanto Mathew (AgentMart) for the Lexsi Labs AI Safety & Alignment Hackathon. Toolkits chained: AgentTune + AuditKIT (2 Lexsi toolkits). Target student model family: Cohere Labs open-weights (CohereLabs/aya-expanse-8b; Tiny Aya class noted as the smaller alternative in the training notebook).
What this artifact is
policy_weights.json β the fitted parameters of the learned agent policy evaluated in this submission: a naive-Bayes tool router + cue/shape argument extractor fitted by code on 130 AgentTune-format SFT trajectories (shantomathew/agentmart-tooltune-data). Serialised so the exact evaluated policy is inspectable and reproducible.
Measured before/after (held-out test, n=40, AuditKIT 1.1.11)
| Metric | Baseline | ToolTune policy |
|---|---|---|
| Tool-selection accuracy | 0.75 | 1.00 |
| Argument exact-match | 0.15 | 1.00 |
| Task success rate | 0.15 | 1.00 |
| Avg calls / task | 1.275 | 1.00 |
| AuditKIT tool_call_f1 (exact) | 0.15 | 1.00 |
| AuditKIT redundant_tool_calls | 0.1375 | 0.00 |
Full metrics, logs and trajectories: dataset repo shantomathew/agentmart-tooltune-data.
Honest scope
- The metrics above are from the executed local pipeline (agent-policy evaluation scored with AuditKIT's real API). Synthetic single-step tasks; entity/value-disjoint split, shared template families β in-distribution generalisation, not a BFCL-style claim.
- The GPU LoRA SFT of the Cohere Aya base on the same AgentTune SFT dataset is provided as a complete training notebook (
train_aya_sft.ipynbin the code repo) and was not executed in this run; no fine-tuned Aya weights are claimed or included here.
Model tree for shantomathew/agentmart-tooltune-policy
Base model
CohereLabs/aya-expanse-8b