ToolTune Policy β€” reliable tool calling (Lexsi Track 3: Agentic Fine-tuning)

Submission by Shanto Mathew (AgentMart) for the Lexsi Labs AI Safety & Alignment Hackathon. Toolkits chained: AgentTune + AuditKIT (2 Lexsi toolkits). Target student model family: Cohere Labs open-weights (CohereLabs/aya-expanse-8b; Tiny Aya class noted as the smaller alternative in the training notebook).

What this artifact is

policy_weights.json β€” the fitted parameters of the learned agent policy evaluated in this submission: a naive-Bayes tool router + cue/shape argument extractor fitted by code on 130 AgentTune-format SFT trajectories (shantomathew/agentmart-tooltune-data). Serialised so the exact evaluated policy is inspectable and reproducible.

Measured before/after (held-out test, n=40, AuditKIT 1.1.11)

Metric Baseline ToolTune policy
Tool-selection accuracy 0.75 1.00
Argument exact-match 0.15 1.00
Task success rate 0.15 1.00
Avg calls / task 1.275 1.00
AuditKIT tool_call_f1 (exact) 0.15 1.00
AuditKIT redundant_tool_calls 0.1375 0.00

Full metrics, logs and trajectories: dataset repo shantomathew/agentmart-tooltune-data.

Honest scope

  • The metrics above are from the executed local pipeline (agent-policy evaluation scored with AuditKIT's real API). Synthetic single-step tasks; entity/value-disjoint split, shared template families β€” in-distribution generalisation, not a BFCL-style claim.
  • The GPU LoRA SFT of the Cohere Aya base on the same AgentTune SFT dataset is provided as a complete training notebook (train_aya_sft.ipynb in the code repo) and was not executed in this run; no fine-tuned Aya weights are claimed or included here.
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for shantomathew/agentmart-tooltune-policy

Finetuned
(92)
this model