docs/REPORT.md

ToolRouter — report

Goal. Pick the right tool for a user query, with the tool set and descriptions supplied by the caller (priority tools: wiki, websearch, weather, news, files, shell, other). Constraints: model < 10 MB, instant, strict, structured output format (state + typed questions in; answers.<name> = {type: "choice", choice, probabilities, confidence} out).

Result (shipped model: toolrouter.npz, seed 0, 20 epochs)

	Shipped model	Mean of 3 seeds
Priority tools, independent routing/test2 (330 q)	0.936	0.935
Priority tools, routing/test2, reworded descriptions	0.936	0.934
Priority tools, routing/test (210 q)	0.933	0.932
Priority tools, routing/test, reworded descriptions	0.938	0.935
10 never-seen tools (tools/test), all offered at once	0.890	0.890
same, random 5-tool slates / differently worded descriptions / name only	0.933 / 0.780 / 0.750
14 never-seen validation tools (tools/validation), all offered	0.943	0.943
7 priority + 10 unseen tools together (priority queries)	0.905
all ~210 tools offered at once (validation / test tools)	0.621 / 0.490
Descriptions shifted onto wrong tools (sanity)	0.060
Confidence ≥ 0.95 → accuracy	0.982 (170/210 priority) · 0.982 (56/100 unseen)
Size / latency	7.66 MB (backbone included) · 0.10–0.16 ms warm, ~2 ms cold (10 tools, CPU)

Seeds differ by about ±1 point on the priority tools (0.929–0.933 on test, 0.930–0.939 on test2); unseen-tool numbers are nearly identical across seeds because the semantic path is not trained. The shipped seed has the best validation score (ties go to the lowest seed); it was chosen without looking at test scores. Full output: eval_output.txt; seed table: seed_results.json; training log: train_log.txt.

Method

• Interface — the caller supplies options as criteria: {name: description} (or {"description", "examples"}); output is the strict choice object. Invalid requests raise ValueError.
• Scorer — frozen pretrained static embeddings (potion-base-8M, int8 in the model file) embed the query and each tool ("name. description", description, name). score = 7·cos(q,tool) + 3·cos(q,desc) + 2·cos(q,name) + (.5,.5,.5,.25)·lexical_overlap. With examples, cos(q,tool) = max over description and example queries. Softmax with a calibrated temperature → probabilities, confidence.
• Identity head (trained, only for the 7 priority tools and their aliases): hashed n-gram bag + dense query embedding, one vector per tool. Unseen tools use only the semantic score.
• Training — episodic: query + target tool + 3–7 random distractor tools with random name styles (snake/camel/kebab/spaced/alias) and description variants (full, one sentence, truncated, empty). Priority tools expose their identity with probability p ~ U(0.3, 1) per tool per episode; priority-tool queries never see a distractor that plausibly answers them (cos > 0.45). Backbone quantised to int8 before training so training sees exactly what ships. 20 epochs, ~4 min on one GPU.

How we got here (what the numbers taught)

Step	Finding
Research	fastText-style models fit <10 MB and sub-ms (Bag of Tricks); generic retrievers are weak at tool retrieval but benefit from training data and document expansion (ToolRet); static embeddings give pretrained semantics at <8 MB
Hashed n-gram classifier (fixed 7 classes)	100 % on its own templates but 83 % on handwritten queries → templates too narrow; with 500 handwritten queries and fewer epochs 91 % on the 7 tools (this fixed-class model was dropped from the release)
Open-set, hashed features only	memorised the 23 training tools: 71–77 % on unseen tools
Backbone size (zero-shot cosine, no training)	unseen tools: potion-2M 0.76 → 4M 0.82 → 8M 0.86; 32M only +2–4 pts on hard settings and does not fit: k-means vocabulary quantisation to 16k centroids lost it (dev closed 0.879)
Learning the semantic path hurt	fine-tuning on ~11k ToolRet rows raised ToolRet-dev accuracy (0.60→0.73) but lowered chat-style unseen-tool accuracy (0.94→0.75); learned projections/weights on 23 or 179 tools were ≤ zero-shot. ToolRet queries are long API requests, unlike chat. → semantic path is fixed
151 more catalog tools	no gain on unseen tools (all variants 0.943 on validation): the 8M embedding is the ceiling
Identity head + base-7 data	priority tools 0.69 (zero-shot) → 0.88 (identity head) → 0.905 (+2.8k agent-written queries, dense identity input) → 0.93–0.94 (+1.3k targeted hard-case queries; 20 epochs)
Tried, no help	MaxSim late interaction (worse), IDF-weighted pooling (flat), MLP identity head (flat), weight decay on the head (trade-off), ToolRet distractor pool (dropped for licence cleanliness; no measurable loss)
Example queries per tool (optional)	no effect on small slates, +1–8 pts with many candidate tools

Caveats

1. LLM-written, small test sets. All data were written by Claude (the Claude Code assistant and Claude sub-agents) under explicit labelling conventions — no real user traffic, labels not human-verified. Sampling error ≈ ±3 points; expect lower accuracy on real traffic.
2. Test contamination. routing/test and tools/test were looked at repeatedly while choosing configurations and designing the targeted data categories; routing/test2 was written by a separate agent and filtered against all other splits (82 of 420 queries duplicated training data and were removed → 330, wiki-light) and is the cleanest number. The 7 fixed semantic weights came from an ablation that looked at the validation and test tool sets (differences ≤ 3 pts).
3. Many candidate tools. With 45–200 tools at once accuracy is 0.50–0.77: generic other/websearch/wiki steal specific queries. Keep slates ≤ 10 or add examples.
4. Descriptions matter. Name-only costs ~14 points, 4-word descriptions ~10; renamed priority tools (encyclopedia, general_chat, …) with reworded descriptions reach 0.77.
5. Remaining priority-tool errors are mostly genuine label ambiguity (e.g. "move all jpgs into a subfolder": shell or files; "meaning of life": other or wiki; "Olympic results today": news or websearch).
6. Scope: English, single-turn, one tool per query, no argument extraction.

Next steps

Real tool schemas (e.g. MCP servers) and logged queries with reliable labels; a tiny re-ranker over the top-3 candidates or a larger static embedding with a lossless-enough compression to use the remaining 2.3 MB; routing low-confidence (< ~0.8) cases to a larger model.