| # ToolRouter — report |
|
|
| **Goal.** Pick the right tool for a user query, with the tool set and descriptions supplied by the caller (priority tools: `wiki`, `websearch`, `weather`, `news`, `files`, `shell`, `other`). Constraints: model < 10 MB, instant, strict, structured output format (`state` + typed `questions` in; `answers.<name> = {type: "choice", choice, probabilities, confidence}` out). |
|
|
| ## Result (shipped model: `toolrouter.npz`, seed 0, 20 epochs) |
| | | Shipped model | Mean of 3 seeds | |
| |---|---:|---:| |
| | Priority tools, independent `routing/test2` (330 q) | **0.936** | 0.935 | |
| | Priority tools, `routing/test2`, reworded descriptions | 0.936 | 0.934 | |
| | Priority tools, `routing/test` (210 q) | 0.933 | 0.932 | |
| | Priority tools, `routing/test`, reworded descriptions | 0.938 | 0.935 | |
| | 10 never-seen tools (`tools/test`), all offered at once | **0.890** | 0.890 | |
| | same, random 5-tool slates / differently worded descriptions / name only | 0.933 / 0.780 / 0.750 | | |
| | 14 never-seen validation tools (`tools/validation`), all offered | 0.943 | 0.943 | |
| | 7 priority + 10 unseen tools together (priority queries) | 0.905 | | |
| | all ~210 tools offered at once (validation / test tools) | 0.621 / 0.490 | | |
| | Descriptions shifted onto wrong tools (sanity) | 0.060 | | |
| | Confidence ≥ 0.95 → accuracy | 0.982 (170/210 priority) · 0.982 (56/100 unseen) | | |
| | Size / latency | 7.66 MB (backbone included) · 0.10–0.16 ms warm, ~2 ms cold (10 tools, CPU) | | |
|
|
| Seeds differ by about ±1 point on the priority tools (0.929–0.933 on `test`, 0.930–0.939 on `test2`); unseen-tool numbers are nearly identical across seeds because the semantic path is not trained. The shipped seed has the best validation score (ties go to the lowest seed); it was chosen without looking at test scores. Full output: `eval_output.txt`; seed table: `seed_results.json`; training log: `train_log.txt`. |
|
|
| ## Method |
| * **Interface** — the caller supplies options as `criteria: {name: description}` (or `{"description", "examples"}`); output is the strict `choice` object. Invalid requests raise `ValueError`. |
| * **Scorer** — frozen pretrained static embeddings ([potion-base-8M](https://huggingface.co/minishlab/potion-base-8M), int8 in the model file) embed the query and each tool ("name. description", description, name). `score = 7·cos(q,tool) + 3·cos(q,desc) + 2·cos(q,name) + (.5,.5,.5,.25)·lexical_overlap`. With examples, `cos(q,tool)` = max over description and example queries. Softmax with a calibrated temperature → `probabilities`, `confidence`. |
| * **Identity head** (trained, only for the 7 priority tools and their aliases): hashed n-gram bag + dense query embedding, one vector per tool. Unseen tools use only the semantic score. |
| * **Training** — episodic: query + target tool + 3–7 random distractor tools with random name styles (snake/camel/kebab/spaced/alias) and description variants (full, one sentence, truncated, empty). Priority tools expose their identity with probability p ~ U(0.3, 1) per tool per episode; priority-tool queries never see a distractor that plausibly answers them (cos > 0.45). Backbone quantised to int8 before training so training sees exactly what ships. 20 epochs, ~4 min on one GPU. |
|
|
| ## How we got here (what the numbers taught) |
| | Step | Finding | |
| |---|---| |
| | Research | fastText-style models fit <10 MB and sub-ms ([Bag of Tricks](https://arxiv.org/abs/1607.01759)); generic retrievers are weak at tool retrieval but benefit from training data and document expansion ([ToolRet](https://arxiv.org/abs/2503.01763)); static embeddings give pretrained semantics at <8 MB | |
| | Hashed n-gram classifier (fixed 7 classes) | 100 % on its own templates but 83 % on handwritten queries → templates too narrow; with 500 handwritten queries and fewer epochs 91 % on the 7 tools (this fixed-class model was dropped from the release) | |
| | Open-set, hashed features only | memorised the 23 training tools: 71–77 % on unseen tools | |
| | Backbone size (zero-shot cosine, no training) | unseen tools: potion-2M 0.76 → 4M 0.82 → **8M 0.86**; 32M only +2–4 pts on hard settings and does not fit: k-means vocabulary quantisation to 16k centroids lost it (dev closed 0.879) | |
| | **Learning the semantic path hurt** | fine-tuning on ~11k ToolRet rows raised ToolRet-dev accuracy (0.60→0.73) but lowered chat-style unseen-tool accuracy (0.94→0.75); learned projections/weights on 23 or 179 tools were ≤ zero-shot. ToolRet queries are long API requests, unlike chat. → semantic path is **fixed** | |
| | 151 more catalog tools | no gain on unseen tools (all variants 0.943 on validation): the 8M embedding is the ceiling | |
| | Identity head + base-7 data | priority tools 0.69 (zero-shot) → 0.88 (identity head) → 0.905 (+2.8k agent-written queries, dense identity input) → 0.93–0.94 (+1.3k targeted hard-case queries; 20 epochs) | |
| | Tried, no help | MaxSim late interaction (worse), IDF-weighted pooling (flat), MLP identity head (flat), weight decay on the head (trade-off), ToolRet distractor pool (dropped for licence cleanliness; no measurable loss) | |
| | Example queries per tool (optional) | no effect on small slates, +1–8 pts with many candidate tools | |
|
|
| ## Caveats |
| 1. **LLM-written, small test sets.** All data were written by Claude (the Claude Code assistant and Claude sub-agents) under explicit labelling conventions — no real user traffic, labels not human-verified. Sampling error ≈ ±3 points; expect lower accuracy on real traffic. |
| 2. **Test contamination.** `routing/test` and `tools/test` were looked at repeatedly while choosing configurations and designing the targeted data categories; `routing/test2` was written by a separate agent and filtered against all other splits (82 of 420 queries duplicated training data and were removed → 330, wiki-light) and is the cleanest number. The 7 fixed semantic weights came from an ablation that looked at the validation and test tool sets (differences ≤ 3 pts). |
| 3. **Many candidate tools.** With 45–200 tools at once accuracy is 0.50–0.77: generic `other`/`websearch`/`wiki` steal specific queries. Keep slates ≤ 10 or add `examples`. |
| 4. **Descriptions matter.** Name-only costs ~14 points, 4-word descriptions ~10; renamed priority tools (`encyclopedia`, `general_chat`, …) with reworded descriptions reach 0.77. |
| 5. Remaining priority-tool errors are mostly genuine label ambiguity (e.g. "move all jpgs into a subfolder": shell or files; "meaning of life": other or wiki; "Olympic results today": news or websearch). |
| 6. Scope: English, single-turn, one tool per query, no argument extraction. |
|
|
| ## Next steps |
| Real tool schemas (e.g. MCP servers) and logged queries with reliable labels; a tiny re-ranker over the top-3 candidates or a larger static embedding with a lossless-enough compression to use the remaining 2.3 MB; routing low-confidence (< ~0.8) cases to a larger model. |
|
|
