README.md

---
license: apache-2.0
base_model: Qwen/Qwen3.5-2B
pipeline_tag: text-generation
tags:
  - function-calling
  - tool-use
  - agents
  - bfcl
language:
  - en
---

# Toolcall-2B

A 2B function-calling model for local agent tool routing, fine-tuned from
[Qwen/Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B) and measured on the
Berkeley Function Calling Leaderboard v4 harness.

It is built for the decision an agent makes hundreds of times per task: which
tool to call, with which arguments, in what order, and whether to call one at
all. Those decisions are cheap in judgment and expensive in tokens when they run
through a frontier model.

## Results

Measured with the BFCL v4 harness on one server (vLLM, bf16, temperature 0.001,
128k serving context), identical settings for every model. The base model was
re-scored under the same conditions rather than quoting published numbers, so
these are like-for-like.

| Group | Qwen3.5-2B (base) | Toolcall-2B |
|---|---|---|
| **Overall (22 categories)** | 33.85 | **36.35** |
| Mean over the 20 categories not limited by search quota | 48.13 | **59.05** |
| Non-live single-turn | 60.08 | **83.25** |
| Live single-turn | 71.13 | **74.32** |
| Multi-turn | 17.75 | **22.12** |
| Memory | 30.97 | **31.18** |
| Web search | 9.50 | 4.50* |

\* The free search-API tier allows 250 queries a month and one pass over this
category costs about 165, so this figure is not reliable and the 22-category
overall understates the model. The 20-category mean is the honest comparison.

Largest per-category gains over the base:

| Category | Base | Toolcall-2B |
|---|---|---|
| simple_java | 15.0 | 61.0 |
| parallel | 49.0 | 86.0 |
| simple_javascript | 24.0 | 52.0 |
| live_parallel | 62.5 | 87.5 |
| live_parallel_multiple | 50.0 | 75.0 |
| simple_python | 70.0 | 89.5 |

All four multi-turn categories are above the base: base 27.0 → 31.0, long
context 24.0 → 30.5, missing parameter 14.0 → 18.0, missing function 6.0 → 9.0.

Known weakness: the refusal family sits below the base — irrelevance 77.9 → 72.5,
live_relevance 87.5 → 81.2. A later round that added more refusal demonstrations
fixed part of this but cost more in multi-turn and memory than it gained, so it
was not shipped.

## Quantized builds

GGUF builds for llama.cpp and Ollama are at
[ajvikram/toolcall-2b-gguf](https://huggingface.co/ajvikram/toolcall-2b-gguf):
Q4_K_M (1.22 GB) runs on a laptop CPU at roughly 33 tokens per second, with
Q5_K_M, Q8_0 and f16 also available.

## Usage

The model keeps Qwen3.5's native tool-call format, so it drops into any stack
that already parses Qwen3.5 tool calls.

```bash
vllm serve ajvikram/toolcall-2b --dtype bfloat16 --max-model-len 131072 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city.",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string", "description": "City name"}},
            "required": ["city"],
        },
    },
}]
r = client.chat.completions.create(
    model="ajvikram/toolcall-2b",
    messages=[{"role": "user", "content": "What's the weather in Berlin?"}],
    tools=tools,
    temperature=0.0,
)
print(r.choices[0].message.tool_calls)
```

Emitted calls look like this, which is the base model's own format:

```
<tool_call>
<function=get_weather>
<parameter=city>
Berlin
</parameter>
</function>
</tool_call>
```

Thinking is off by default, matching how the model was trained and evaluated.

## Training

Supervised fine-tuning, 16-bit LoRA (rank 16, alpha 32, all linear projections),
loss on assistant turns only, sequences to 8,192 tokens, learning rate 3e-5,
0.6 epoch, batch 4 with 8-step accumulation. One run of 3.2 hours on a rented
RTX A6000; final training loss 0.330, validation loss 0.247.

Prompts are rendered with the model's own chat template at training time and at
evaluation time, so the two never diverge.

### Data

61,712 examples. Every public source is permissively licensed; a CC-BY-NC set
was excluded on purpose so these weights can be Apache-2.0.

| Source | Licence | Rows | Contributes |
|---|---|---|---|
| [ToolACE](https://huggingface.co/datasets/Team-ACE/ToolACE) | Apache-2.0 | 13,102 | Diverse APIs, some multi-turn |
| [Hermes function-calling v1](https://huggingface.co/datasets/NousResearch/hermes-function-calling-v1) | Apache-2.0 | 5,645 | Tool responses and follow-ups |
| [Glaive v2](https://huggingface.co/datasets/glaiveai/glaive-function-calling-v2) | Apache-2.0 | 4,095 | Short multi-turn |
| [xLAM 60k](https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k) | CC-BY-4.0 | 10,431 | Single and parallel calls |
| [xLAM irrelevance](https://huggingface.co/datasets/MadeAgents/xlam-irrelevance-7.5k) | CC-BY-4.0 | 1,183 | Requests no tool can serve |
| [SmolTalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk) | Apache-2.0 | 2,835 | General chat, limits forgetting |
| Verified simulator episodes | generated for this model | 2,134 ×3 | Stateful multi-turn (below) |

Two augmentations, applied deterministically: function masking, which renames
tools and parameters to opaque identifiers consistently across the schema and
the calls so the model reads descriptions rather than memorising names; and
distractor tools drawn from other examples, so tool selection is exercised.

### The ingredient that mattered

Four earlier rounds on public data alone raised single-turn accuracy a great
deal and made multi-turn *worse* than the base every time. Public corpora teach
the shape of a tool call, not the habit of working a stateful task to
completion.

Round five added 2,134 conversations generated against six simulated stateful
APIs — a file system, task tracker, mailbox, key-value memory, calendar and shop
— where a larger model played the assistant, a second model played a user who
revealed the task gradually, and every episode was kept only if a checker
confirmed the final state was correct. Episodes were rejected when the goal was
not met, when a tool errored more than once, when an unrelated tool was called,
or when the model guessed instead of asking for information it had not been
given. That single change moved every stateful category above the base at once.

## Evaluation, reproduced

```bash
# harness: fork of ShishirPatil/gorilla with a handler for Qwen3.5's XML tool-call format
bfcl generate --model ajvikram/toolcall-2b-FC --test-category all_scoring \
  --skip-server-setup --local-model-path <merged-dir> --num-threads 16
bfcl evaluate --model ajvikram/toolcall-2b-FC --test-category all_scoring
```

## Limitations

- Refusal behaviour is below the base model: it is more willing to attempt a
  request no available tool can serve.
- The web-search score is not reliable, for the quota reason above.
- Numbers here come from one person's harness runs, not from the official
  leaderboard. They are reproducible with the commands above, and the base model
  measured on the same rig is the comparison that matters.
- English only, and evaluated only on this benchmark's tool-calling tasks.

## Citation

Built on Qwen3.5-2B; evaluated with the Berkeley Function Calling Leaderboard
harness. Training data credits are listed above, and the episode generation
follows the verified-trajectory approach described in
[APIGen-MT](https://arxiv.org/abs/2504.03601).