tool-select-judge-2b

Checks whether an AI agent called the right tool(s) at one step of a conversation. It answers 1 (right) or 0 (wrong) with a confidence. Only tool choice is judged, not argument values.

LoRA adapter on Qwen/Qwen3.5-2B, trained on 27,596 real examples. Current version: v0.1 (each release is a tag on this repo; use --revision v0.1 to pin it).

How to use

1. Download and install

pip install -U huggingface_hub      # gives the `hf` command
hf auth login                        # only needed while the repo is private
hf download CobuildX-AI/tool-select-judge-2b --revision v0.1 --local-dir tool-select-judge-2b
cd tool-select-judge-2b
pip install -r requirements.txt

2. Use it in your code

ToolJudge (in judge.py, part of this repo) loads the model once and scores tool calls:

from judge import ToolJudge

judge = ToolJudge()                 # loads Qwen/Qwen3.5-2B + this adapter
judge.score(
    tools=[{"name": "search_hotels", "description": "Search hotels in a city"},
           {"name": "book_hotel", "description": "Book a specific hotel room"}],
    conversation=[{"role": "user", "content": "Find me a hotel in Paris for next week"}],
    picked=["book_hotel"],          # tool name(s) the agent called; [] = no tool called
)
# {'answer': 0, 'confidence': 0.97, 'probability_right': 0.03, 'latency_ms': 210}

• tools: the tools the agent had at this step. conversation: every message before the call, with roles user, assistant, assistant_tool_call (an earlier call) and tool (a tool result).
• Score a whole trace by calling score once per tool-call step (or score_batch).
• judge.py builds every prompt with prompt.py, the exact layout the model was trained on.

3. Optional: check your setup

python example.py scores 4 sample tool calls and prints the answers, so you can confirm the install works.

Results

	tool-select-judge-2b v0.1	Jev 1.13
Held-out test accuracy (3,904 calls)	95.6%	89.8%
Multi-turn accuracy (1,028 calls)	93.0%	80.0%
TRBench subset accuracy (1,910 calls)	83.7%	84.1%
Calls to tools that do not exist, caught (395 calls)	98.0%	47.3%

All tests are 50% right / 50% wrong and were never used in training. Results were measured with prompts of up to 2,048 tokens; much longer prompts are slower and have not been measured.

TRBench is a subset, not the full benchmark. From the 11,932 pairs in RioLee/TRBench-BFCL we removed 7,232 where the wrong answer used the same tool with different arguments (not a tool-choice mistake) or had formatting issues, and 3,745 duplicates. That leaves 955 unique wrong tool choices plus 955 correct reference answers. These scores are not comparable to full-TRBench scores.

Licence

Apache-2.0.