Update README.md

README.md

@@ -18,223 +18,372 @@ tags:
   - lora
   - cpu
   - gguf
-  - text-generation-inference
 ---
 
 # TinyToolRouter-0.6B
-**My Github** ['roboroyce'](https://github.com/roboroyce/)
-A 0.6B-parameter, **CPU-first tool router**. Given a natural-language request
-plus a list of available tool schemas, it returns exactly **one** tool call:
+
+**A lightweight, CPU-first tool router fine-tuned from Qwen3-0.6B.**
+
+**GitHub:** [@roboroyce](https://github.com/roboroyce/)
+
+TinyToolRouter-0.6B is a 0.6-billion-parameter language model designed to select tools from a supplied list of tool schemas. Given a natural-language request and the available tools, it produces exactly one structured JSON tool call or explicitly abstains when no tool applies.
+
+### Example
+
+**Input request:** `What's the weather in Oslo?`
+
+**Output:**
 
 ```json
-{"name": "get_weather", "arguments": {"city": "Oslo"}}
+{
+  "name": "get_weather",
+  "arguments": {
+    "city": "Oslo"
+  }
+}
 ```
 
-or, when no listed tool applies, it **abstains**:
+When no listed tool applies, the model abstains:
 
 ```json
-{"name": "no_tool", "arguments": {}}
+{
+  "name": "no_tool",
+  "arguments": {}
+}
 ```
 
-- **Base model:** [`Qwen/Qwen3-0.6B`](https://huggingface.co/Qwen/Qwen3-0.6B) (Apache-2.0)
-- **Method:** LoRA supervised fine-tuning (PEFT), merged into the base weights
-- **Size:** 0.6B parameters (fp16/bf16 safetensors ≈ 1.2 GB; Q4_K_M GGUF ≈ 397 MB)
-- **Language:** English
-- **License:** Apache-2.0
-
-Small models are usually *shaky* at tool routing: they hallucinate tools, emit
-prose instead of JSON, or force a tool when none fits. This model is tuned
-narrowly for reliable routing, valid structured output, and honest abstention —
-and its improvement over the base model is published as a reproducible
-benchmark rather than an assertion.
-
-## Results
-
-Evaluated on a **held-out set of 231 cases** across three families:
-
-| Family | n | What it tests |
-|---|---|---|
-| `seen_paraphrase` | 131 | unseen *phrasing* of tools seen in training |
-| `unseen_schema` | 60 | tools **never seen during training** |
-| `no_tool` | 40 | requests matching no tool → must abstain |
-
-Both models were run with identical prompts, `do_sample=False`,
-`max_new_tokens=128`.
-
-| Metric | Qwen3-0.6B (base) | **TinyToolRouter-0.6B** |
-|---|---|---|
-| **Exact match** (headline) | 0.5195 | **0.9221** |
-| Valid structured output | 0.9913 | **0.9913** |
-| Tool selection | 0.9351 | **0.9870** |
-| Argument accuracy | 0.5195 | **0.9221** |
-| `no_tool` precision | 0.7647 | **0.9756** |
-| `no_tool` recall | 0.9750 | **1.0000** |
-| `no_tool` F1 | 0.8571 | **0.9877** |
-
-By family (exact match):
-
-| Family | base | **fine-tuned** |
-|---|---|---|
-| seen_paraphrase | 0.229 | **0.9237** |
-| unseen_schema | 0.850 | **0.8667** |
-| no_tool | 0.975 | **1.0000** |
-
-The largest gain is on paraphrased requests (+69.5 pts); generalisation to
-genuinely unseen tool schemas improves only modestly (+1.7 pts) and is not
-" solved".
+## Key features
+
+- **Small footprint:** 0.6B parameters, suitable for CPU-first inference.
+- **Schema-aware routing:** Selects from tool definitions supplied at inference time.
+- **Structured output:** Produces a single JSON object representing a proposed tool call.
+- **Explicit abstention:** Supports a dedicated `no_tool` response.
+- **Paraphrase robustness:** Evaluated on requests phrased differently from the training examples.
+- **Unseen-schema evaluation:** Tested on tools absent from the training set.
+- **Multiple formats:** Safetensors and GGUF builds are available in this repository.
+
+## Model details
+
+| Property | Value |
+|---|---|
+| Model | TinyToolRouter-0.6B |
+| Base model | [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) |
+| Parameters | 0.6B |
+| Fine-tuning | Supervised fine-tuning with LoRA (PEFT) |
+| Deployment | Merged fine-tuned weights |
+| Primary language | English |
+| License | Apache-2.0 |
+| Intended hardware | CPU-only and low-VRAM systems |
+| Safetensors size | Approximately 1.2 GB |
+| GGUF Q4_K_M size | Approximately 397 MB |
+| GGUF Q8_0 size | Approximately 639 MB |
+| GGUF F16 size | Approximately 1.2 GB |
+
+File sizes are approximate and may vary slightly by build.
+
+## Benchmark results
+
+TinyToolRouter was evaluated on a held-out set of **231 cases** covering three distinct evaluation families.
+
+### Overall performance
+
+| Metric | Qwen3-0.6B (base) | TinyToolRouter-0.6B |
+|---|---:|---:|
+| **Exact match** | 51.95% | **92.21%** |
+| Valid structured output | 99.13% | 99.13% |
+| Tool selection | 93.51% | **98.70%** |
+| Argument accuracy | 51.95% | **92.21%** |
+| `no_tool` precision | 76.47% | **97.56%** |
+| `no_tool` recall | 97.50% | **100.00%** |
+| `no_tool` F1 | 85.71% | **98.77%** |
+
+Exact match improved by **40.26 percentage points** over the base model.
+
+### Performance by evaluation family
+
+| Evaluation family | Cases | Base model | Fine-tuned |
+|---|---:|---:|---:|
+| `seen_paraphrase` | 131 | 22.90% | **92.37%** |
+| `unseen_schema` | 60 | 85.00% | **86.67%** |
+| `no_tool` | 40 | 97.50% | **100.00%** |
+
+**What these results mean**
+
+- **Paraphrase robustness:** Exact match increased from 22.90% to 92.37% on requests that use unseen phrasing for tools encountered during training.
+- **Unseen tool schemas:** Exact match increased from 85.00% to 86.67% on tools absent from the training set. This is a modest improvement, not evidence of fully general tool-schema understanding.
+- **Abstention:** The fine-tuned model correctly abstained on all 40 held-out `no_tool` cases, with 97.56% measured precision.
+
+### Evaluation methodology
+
+Both models were evaluated using identical prompts and generation settings:
+
+- `do_sample=False`
+- `max_new_tokens=128`
+- Held-out evaluation set: 231 cases
+- Evaluation metric: exact match of the expected structured output
+
+The benchmark separates familiar tool schemas with paraphrased requests, genuinely unseen tool schemas, and requests that should not trigger a tool call.
+
+The reported metrics are from the **safetensors model**. The GGUF conversions have not been independently benchmarked, so equivalent performance across formats should not be assumed.
 
 ## Quickstart
 
+### Install dependencies
+
 ```bash
-pip install transformers torch
+pip install torch transformers
 ```
 
+### Run with Transformers
+
 ```python
 import json
+import torch
 from transformers import AutoModelForCausalLM, AutoTokenizer
 
 model_id = "Robokid1134/TinyToolRouter-0.6B"
-tok = AutoTokenizer.from_pretrained(model_id)
-model = AutoModelForCausalLM.from_pretrained(model_id)
 
-tools = [{
-    "name": "get_weather",
-    "description": "Get the current weather for a city.",
-    "parameters": {
-        "type": "object",
-        "properties": {"city": {"type": "string"}},
-        "required": ["city"],
-    },
-}]
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id)
+model.eval()
+
+tools = [
+    {
+        "name": "get_weather",
+        "description": "Get the current weather for a city.",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "city": {"type": "string"}
+            },
+            "required": ["city"],
+        },
+    }
+]
 
-SYSTEM = (
-    "You are a tool router. You are given a list of available tools and a user "
-    "request. Reply with exactly one JSON object of the form "
-    '{"name": <tool_name>, "arguments": {<argument>: <value>}}. '
-    "Use only the listed tools and only their declared arguments. "
+system_prompt = (
+    "You are a tool router. You are given a list of available tools "
+    "and a user request. Reply with exactly one JSON object of the form "
+    '{"name": "<tool_name>", "arguments": {"<argument>": "<value>"}}. '
+    "Use only the listed tools and their declared arguments. "
     'If no listed tool applies, reply {"name": "no_tool", "arguments": {}}. '
-    "Output JSON only, no prose."
+    "Output JSON only, with no additional prose."
 )
 
 messages = [
-    {"role": "system", "content": SYSTEM},
-    {"role": "user", "content": json.dumps(
-        {"tools": tools, "request": "What's the weather in Oslo?"})},
+    {"role": "system", "content": system_prompt},
+    {
+        "role": "user",
+        "content": json.dumps({
+            "tools": tools,
+            "request": "What's the weather in Oslo?"
+        }),
+    },
 ]
 
-prompt = tok.apply_chat_template(
-    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
-inputs = tok(prompt, return_tensors="pt")
-out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
-print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
-# {"name": "get_weather", "arguments": {"city": "Oslo"}}
+prompt = tokenizer.apply_chat_template(
+    messages,
+    tokenize=False,
+    add_generation_prompt=True,
+    enable_thinking=False,
+)
+
+inputs = tokenizer(prompt, return_tensors="pt")
+
+with torch.inference_mode():
+    output = model.generate(
+        **inputs,
+        max_new_tokens=128,
+        do_sample=False,
+    )
+
+new_tokens = output[0][inputs["input_ids"].shape[1]:]
+response = tokenizer.decode(
+    new_tokens,
+    skip_special_tokens=True,
+)
+
+print(response)
+```
+
+Expected output:
+
+```json
+{"name": "get_weather", "arguments": {"city": "Oslo"}}
 ```
 
-`enable_thinking=False` is recommended: the router should answer directly
-rather than emit a reasoning block.
+**Note:** `enable_thinking=False` is recommended for direct routing output. The example uses the Qwen3 chat template and assumes a Transformers version that supports this argument.
+
+## GGUF: CPU inference
 
-### GGUF (CPU / llama.cpp / Ollama)
+GGUF builds are available in the repository's `gguf/` directory.
 
-The repository ships GGUF builds under `gguf/`:
+| File | Approximate size | Intended use |
+|---|---:|---|
+| `gguf/model-Q4_K_M.gguf` | 397 MB | Compact CPU inference |
+| `gguf/model-Q8_0.gguf` | 639 MB | Higher quantization fidelity |
+| `gguf/model-f16.gguf` | 1.2 GB | Full-precision weight storage |
 
-| File | Size | Use |
-|---|---|---|
-| `gguf/model-Q4_K_M.gguf` | ≈ 397 MB | best size/quality for CPU |
-| `gguf/model-Q8_0.gguf` | ≈ 639 MB | near-lossless |
-| `gguf/model-f16.gguf` | ≈ 1.2 GB | full precision |
+Run a GGUF model using llama.cpp:
 
 ```bash
-# llama.cpp
-llama-cli -m gguf/model-Q4_K_M.gguf -p "<your prompt>"
+llama-cli \
+  -m gguf/model-Q4_K_M.gguf \
+  -p "<your prompt>"
 ```
 
-> Note: the GGUF builds were converted from the merged model but have **not**
-> been separately re-benchmarked; the metrics above come from the safetensors
-> model.
+The prompt should follow the model's expected chat template and include the available tool schemas and user request.
+
+For repeated use, a dedicated prompt template or a compatible Ollama model configuration can simplify integration.
+
+> **Benchmark caveat:** The GGUF files were converted from the merged fine-tuned model but have not been independently re-evaluated. The benchmark figures above apply to the safetensors model.
 
-## About the tool schemas
+## Tool schema format
 
-The model reads JSON-Schema-style tool definitions supplied in the user turn:
+The model expects a JSON object containing a list of available tools and a natural-language request.
 
 ```json
-{"tools": [{"name": "...", "description": "...",
-            "parameters": {"type": "object",
-                           "properties": {"...": {"type": "..."}},
-                           "required": ["..."]}}],
- "request": "natural-language request"}
+{
+  "tools": [
+    {
+      "name": "get_weather",
+      "description": "Get the current weather for a city.",
+      "parameters": {
+        "type": "object",
+        "properties": {
+          "city": {
+            "type": "string"
+          }
+        },
+        "required": ["city"]
+      }
+    }
+  ],
+  "request": "What's the weather in Oslo?"
+}
 ```
 
-It was trained on `string`, `integer`, `number`, `boolean`, `array<string>`
-and `array<number>` arguments, with required and optional parameters, and with
-distractor tools present in the list.
+The schema format is JSON-Schema-style, with a `name`, `description`, and `parameters` object for each tool.
+
+Training covered the following argument types:
+
+- `string`
+- `integer`
+- `number`
+- `boolean`
+- `array` of strings
+- `array` of numbers
+
+Examples included required and optional arguments, along with distractor tools that should not be selected.
+
+The model returns a proposed tool name and its arguments. It does not execute the tool or verify that the underlying action succeeds.
 
 ## Training
 
-| | |
+| Setting | Value |
 |---|---|
-| Base | `Qwen/Qwen3-0.6B` @ `c1899de289a04d12100db370d81485cdf75e47ca` |
-| Method | LoRA (PEFT), merged |
-| LoRA | r=16, α=32, dropout=0.05, all attention+MLP projections |
-| Epochs | 3 (57 steps) |
-| LR | 1e-4, cosine, warmup 3% |
-| Batch | 8 × grad-accum 4, max_len 512 |
-| Precision | bf16 |
-| Seed | 7 |
-| Examples | 599 (train) / 231 (held-out) |
-| Hardware | NVIDIA A10G 24 GB |
-| Train time | ≈ 150 s |
-| Libs | torch 2.5.1, transformers 4.57.6, peft 0.17.1, Python 3.9 |
-
-Loss is computed on the assistant JSON only (prompt tokens masked).
+| Base checkpoint | `Qwen/Qwen3-0.6B` |
+| Base revision | `c1899de289a04d12100db370d81485cdf75e47ca` |
+| Fine-tuning method | LoRA using PEFT |
+| Adapter rank | 16 |
+| LoRA alpha | 32 |
+| LoRA dropout | 0.05 |
+| Target modules | Attention and MLP projections |
+| Epochs | 3 |
+| Training steps | 57 |
+| Learning rate | `1e-4` |
+| Scheduler | Cosine |
+| Warmup ratio | 3% |
+| Batch size | 8 |
+| Gradient accumulation | 4 |
+| Maximum sequence length | 512 tokens |
+| Training precision | BF16 |
+| Random seed | 7 |
+| Training examples | 599 |
+| Held-out examples | 231 |
+| Hardware | NVIDIA A10G, 24 GB VRAM |
+| Approximate training time | 150 seconds |
+| PyTorch | 2.5.1 |
+| Transformers | 4.57.6 |
+| PEFT | 0.17.1 |
+| Python | 3.9 |
+
+The LoRA adapter was merged into the base model for deployment.
+
+Training loss was computed on the assistant's JSON output, with prompt tokens masked from the loss.
 
 ## Dataset
 
-**Original synthetic data** authored for this project (Apache-2.0). No scraped
-text, no chat logs, no personal data. Slot values are invented
-(`sam@example.com`, `2026-11-02T09:00`, city names, …).
+The dataset consists of original synthetic examples authored for this project and released under Apache-2.0.
+
+No scraped text, chat logs, or personal data were used. Slot values are synthetic, including example email addresses, dates, and city names.
+
+### Tool coverage
+
+**Tools represented in training (9):**
+
+- `get_weather`
+- `send_email`
+- `create_calendar_event`
+- `search_web`
+- `convert_currency`
+- `set_timer`
+- `translate_text`
+- `filter_items`
+- `tag_note`
+
+**Unseen tools reserved for evaluation (5):**
+
+- `play_music`
+- `get_stock_price`
+- `book_restaurant`
+- `draw_chart`
+- `create_reminder`
+
+### Data separation
 
-- **Seen tools (9):** `get_weather`, `send_email`, `create_calendar_event`,
-  `search_web`, `convert_currency`, `set_timer`, `translate_text`,
-  `filter_items`, `tag_note`
-- **Unseen tools (5, evaluation only):** `play_music`, `get_stock_price`,
-  `book_restaurant`, `draw_chart`, `create_reminder`
+The training and held-out sets use disjoint templates, value pools, and example IDs.
 
-**Split hygiene:** training and held-out sets use disjoint templates, disjoint
-value pools and disjoint example ids; the `unseen_schema` family uses tools
-absent from training entirely. A `no_tool` abstention set is present in
-training with requests disjoint from the held-out `no_tool` set.
+The `unseen_schema` evaluation family contains tools absent from training entirely. The training set also includes abstention examples, while the held-out `no_tool` requests are disjoint from those training examples.
+
+These design choices help test whether the model learned useful routing patterns rather than simply memorizing identical examples. They do not eliminate every possible source of dataset bias or leakage.
 
 ## Limitations
 
-- **English only**; 14 tools; synthetic data. It teaches routing discipline and
-  schema reading, not world knowledge.
-- **Not an assistant.** It does not execute tools, hold conversations, or answer
-  general questions — those requests should map to `no_tool`.
-- **Unseen-schema gains are modest** (+1.7 pts). Robustness to radically novel
-  schemas is improved but not solved.
-- **Arguments usually appear verbatim in the request.** The hard capability
-  measured here is selection + correct typing, not open-world extraction.
-- **Single seed.** One reproducible run; multi-seed variance is unreported.
-- **Small evaluation set** (231 cases). Treat differences of a few cases as
-  noise.
-- **GGUF builds unverified numerically.**
-- Trained with `max_len` 512; very long tool lists may be truncated by the user.
-
-## Intended use & out-of-scope
-
-**Use:** local agents on CPU-only or low-VRAM machines; as a cheap routing
-front-end before a larger model; as a reproducible baseline for tiny-model
-tool-calling research.
-
-**Do not use for:** safety-critical decisions, executing shell/system commands
-without human review, or any task where a wrong tool call causes harm. The
-model emits *proposed* calls; the caller must validate them.
+- **English only:** The model was trained and evaluated on English-language examples.
+- **Limited tool coverage:** Training covers nine tool schemas, with five additional schemas used for evaluation.
+- **Synthetic data:** Performance on real-world agent requests may differ from the reported benchmark.
+- **Modest unseen-schema gains:** The improvement on unseen schemas is only 1.67 percentage points.
+- **Limited argument extraction:** Arguments often appear explicitly in the request. The evaluation primarily tests tool selection, argument correctness, and structured output rather than unrestricted information extraction.
+- **Small evaluation set:** Results are based on 231 held-out examples. Small differences may reflect only a few cases.
+- **Single random seed:** Multi-seed variance has not been measured.
+- **Quantized models not re-benchmarked:** The reported metrics do not establish equivalent performance for the GGUF builds.
+- **Context length:** Training used a maximum sequence length of 512 tokens. Long tool lists may exceed the model's familiar input distribution or be truncated by an inference pipeline.
+
+## Intended use
+
+TinyToolRouter-0.6B is intended for:
+
+- Local AI agents running on CPU-only or low-VRAM machines.
+- Lightweight tool-selection front ends for larger language models.
+- Routing between a predefined set of application tools.
+- Explicitly declining requests that do not match any available tool.
+- Research into compact language models, tool calling, and structured generation.
+- Reproducible experiments comparing a small base model against a narrowly fine-tuned router.
+
+### Out of scope
+
+This model is not a general-purpose assistant. It is not intended to independently answer general questions, execute tools, or make safety-critical decisions.
+
+Do not rely on its output alone to authorize sensitive actions or execute shell commands. The calling application should validate the selected tool, enforce an allowlist, validate arguments against the actual schema, and require appropriate authorization before executing an action.
 
 ## Citation
 
+If you use this model in research or another project, you can cite it as follows:
+
 ```bibtex
 @misc{tinytoolrouter06b,
-  title  = {TinyToolRouter-0.6B: a CPU-first, schema-strict tool router},
+  title  = {TinyToolRouter-0.6B: A CPU-First, Schema-Strict Tool Router},
   author = {Robokid1134},
   year   = {2026},
   note   = {LoRA fine-tune of Qwen/Qwen3-0.6B},
@@ -242,9 +391,8 @@ model emits *proposed* calls; the caller must validate them.
 }
 ```
 
-## Attribution & license
+## Attribution and license
+
+TinyToolRouter-0.6B is fine-tuned from [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B), which is released under the Apache-2.0 license.
 
-Fine-tuned from [`Qwen/Qwen3-0.6B`](https://huggingface.co/Qwen/Qwen3-0.6B),
-licensed **Apache-2.0**. This derivative is released under the same
-**Apache-2.0** license; the upstream copyright notice and license are
-preserved. Code and generated dataset for this project are Apache-2.0.
+This derivative model is distributed under Apache-2.0, with the applicable upstream license and copyright notices preserved. The project's original code and generated dataset are also released under Apache-2.0.