| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen3.5-2B |
| pipeline_tag: text-generation |
| tags: |
| - function-calling |
| - tool-use |
| - agents |
| - bfcl |
| language: |
| - en |
| --- |
| |
| # Toolcall-2B |
|
|
| A 2B function-calling model for local agent tool routing, fine-tuned from |
| [Qwen/Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B) and measured on the |
| Berkeley Function Calling Leaderboard v4 harness. |
|
|
| It is built for the decision an agent makes hundreds of times per task: which |
| tool to call, with which arguments, in what order, and whether to call one at |
| all. Those decisions are cheap in judgment and expensive in tokens when they run |
| through a frontier model. |
|
|
| ## Results |
|
|
| Measured with the BFCL v4 harness on one server (vLLM, bf16, temperature 0.001, |
| 128k serving context), identical settings for every model. The base model was |
| re-scored under the same conditions rather than quoting published numbers, so |
| these are like-for-like. |
|
|
| | Group | Qwen3.5-2B (base) | Toolcall-2B | |
| |---|---|---| |
| | **Overall (22 categories)** | 33.85 | **36.35** | |
| | Mean over the 20 categories not limited by search quota | 48.13 | **59.05** | |
| | Non-live single-turn | 60.08 | **83.25** | |
| | Live single-turn | 71.13 | **74.32** | |
| | Multi-turn | 17.75 | **22.12** | |
| | Memory | 30.97 | **31.18** | |
| | Web search | 9.50 | 4.50* | |
|
|
| \* The free search-API tier allows 250 queries a month and one pass over this |
| category costs about 165, so this figure is not reliable and the 22-category |
| overall understates the model. The 20-category mean is the honest comparison. |
|
|
| Largest per-category gains over the base: |
|
|
| | Category | Base | Toolcall-2B | |
| |---|---|---| |
| | simple_java | 15.0 | 61.0 | |
| | parallel | 49.0 | 86.0 | |
| | simple_javascript | 24.0 | 52.0 | |
| | live_parallel | 62.5 | 87.5 | |
| | live_parallel_multiple | 50.0 | 75.0 | |
| | simple_python | 70.0 | 89.5 | |
|
|
| All four multi-turn categories are above the base: base 27.0 → 31.0, long |
| context 24.0 → 30.5, missing parameter 14.0 → 18.0, missing function 6.0 → 9.0. |
|
|
| Known weakness: the refusal family sits below the base — irrelevance 77.9 → 72.5, |
| live_relevance 87.5 → 81.2. A later round that added more refusal demonstrations |
| fixed part of this but cost more in multi-turn and memory than it gained, so it |
| was not shipped. |
| |
| ## Quantized builds |
| |
| GGUF builds for llama.cpp and Ollama are at |
| [ajvikram/toolcall-2b-gguf](https://huggingface.co/ajvikram/toolcall-2b-gguf): |
| Q4_K_M (1.22 GB) runs on a laptop CPU at roughly 33 tokens per second, with |
| Q5_K_M, Q8_0 and f16 also available. |
|
|
| ## Usage |
|
|
| The model keeps Qwen3.5's native tool-call format, so it drops into any stack |
| that already parses Qwen3.5 tool calls. |
|
|
| ```bash |
| vllm serve ajvikram/toolcall-2b --dtype bfloat16 --max-model-len 131072 \ |
| --enable-auto-tool-choice --tool-call-parser qwen3_xml |
| ``` |
|
|
| ```python |
| from openai import OpenAI |
| |
| client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") |
| tools = [{ |
| "type": "function", |
| "function": { |
| "name": "get_weather", |
| "description": "Get the current weather for a city.", |
| "parameters": { |
| "type": "object", |
| "properties": {"city": {"type": "string", "description": "City name"}}, |
| "required": ["city"], |
| }, |
| }, |
| }] |
| r = client.chat.completions.create( |
| model="ajvikram/toolcall-2b", |
| messages=[{"role": "user", "content": "What's the weather in Berlin?"}], |
| tools=tools, |
| temperature=0.0, |
| ) |
| print(r.choices[0].message.tool_calls) |
| ``` |
|
|
| Emitted calls look like this, which is the base model's own format: |
|
|
| ``` |
| <tool_call> |
| <function=get_weather> |
| <parameter=city> |
| Berlin |
| </parameter> |
| </function> |
| </tool_call> |
| ``` |
|
|
| Thinking is off by default, matching how the model was trained and evaluated. |
|
|
| ## Training |
|
|
| Supervised fine-tuning, 16-bit LoRA (rank 16, alpha 32, all linear projections), |
| loss on assistant turns only, sequences to 8,192 tokens, learning rate 3e-5, |
| 0.6 epoch, batch 4 with 8-step accumulation. One run of 3.2 hours on a rented |
| RTX A6000; final training loss 0.330, validation loss 0.247. |
|
|
| Prompts are rendered with the model's own chat template at training time and at |
| evaluation time, so the two never diverge. |
|
|
| ### Data |
|
|
| 61,712 examples. Every public source is permissively licensed; a CC-BY-NC set |
| was excluded on purpose so these weights can be Apache-2.0. |
|
|
| | Source | Licence | Rows | Contributes | |
| |---|---|---|---| |
| | [ToolACE](https://huggingface.co/datasets/Team-ACE/ToolACE) | Apache-2.0 | 13,102 | Diverse APIs, some multi-turn | |
| | [Hermes function-calling v1](https://huggingface.co/datasets/NousResearch/hermes-function-calling-v1) | Apache-2.0 | 5,645 | Tool responses and follow-ups | |
| | [Glaive v2](https://huggingface.co/datasets/glaiveai/glaive-function-calling-v2) | Apache-2.0 | 4,095 | Short multi-turn | |
| | [xLAM 60k](https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k) | CC-BY-4.0 | 10,431 | Single and parallel calls | |
| | [xLAM irrelevance](https://huggingface.co/datasets/MadeAgents/xlam-irrelevance-7.5k) | CC-BY-4.0 | 1,183 | Requests no tool can serve | |
| | [SmolTalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk) | Apache-2.0 | 2,835 | General chat, limits forgetting | |
| | Verified simulator episodes | generated for this model | 2,134 ×3 | Stateful multi-turn (below) | |
|
|
| Two augmentations, applied deterministically: function masking, which renames |
| tools and parameters to opaque identifiers consistently across the schema and |
| the calls so the model reads descriptions rather than memorising names; and |
| distractor tools drawn from other examples, so tool selection is exercised. |
|
|
| ### The ingredient that mattered |
|
|
| Four earlier rounds on public data alone raised single-turn accuracy a great |
| deal and made multi-turn *worse* than the base every time. Public corpora teach |
| the shape of a tool call, not the habit of working a stateful task to |
| completion. |
|
|
| Round five added 2,134 conversations generated against six simulated stateful |
| APIs — a file system, task tracker, mailbox, key-value memory, calendar and shop |
| — where a larger model played the assistant, a second model played a user who |
| revealed the task gradually, and every episode was kept only if a checker |
| confirmed the final state was correct. Episodes were rejected when the goal was |
| not met, when a tool errored more than once, when an unrelated tool was called, |
| or when the model guessed instead of asking for information it had not been |
| given. That single change moved every stateful category above the base at once. |
|
|
| ## Evaluation, reproduced |
|
|
| ```bash |
| # harness: fork of ShishirPatil/gorilla with a handler for Qwen3.5's XML tool-call format |
| bfcl generate --model ajvikram/toolcall-2b-FC --test-category all_scoring \ |
| --skip-server-setup --local-model-path <merged-dir> --num-threads 16 |
| bfcl evaluate --model ajvikram/toolcall-2b-FC --test-category all_scoring |
| ``` |
|
|
| ## Limitations |
|
|
| - Refusal behaviour is below the base model: it is more willing to attempt a |
| request no available tool can serve. |
| - The web-search score is not reliable, for the quota reason above. |
| - Numbers here come from one person's harness runs, not from the official |
| leaderboard. They are reproducible with the commands above, and the base model |
| measured on the same rig is the comparison that matters. |
| - English only, and evaluated only on this benchmark's tool-calling tasks. |
|
|
| ## Citation |
|
|
| Built on Qwen3.5-2B; evaluated with the Berkeley Function Calling Leaderboard |
| harness. Training data credits are listed above, and the episode generation |
| follows the verified-trajectory approach described in |
| [APIGen-MT](https://arxiv.org/abs/2504.03601). |
|
|
