--- base_model: google/functiongemma-270m-it library_name: peft license: other license_name: gemma license_link: https://ai.google.dev/gemma/terms pipeline_tag: text-classification tags: - lora - tool-use - agents - function-calling - research datasets: - nvidia/When2Call - cesun/When2Tool model-index: - name: whetherman-functiongemma-270m results: - task: type: text-classification name: Tool-call decision (ACT/ANSWER/ASK/DECLINE) dataset: name: When2Call MCQ test (official code, classifier mapped to options) type: nvidia/When2Call split: test metrics: - type: f1 value: 0.4719 name: macro-F1 - type: accuracy value: 0.6284 name: accuracy - type: tool_hallucination_rate value: 0.0698 name: tool hallucination rate (lower is better) source: name: self-reported url: https://github.com/vinmay/whetherman/blob/main/eval/external/when2call/summary.json - task: type: text-classification name: Should any function be called? dataset: name: BFCL v4 irrelevance type: gorilla-llm/Berkeley-Function-Calling-Leaderboard metrics: - type: accuracy value: 0.7792 name: pass rate (187/240) source: name: self-reported url: https://github.com/vinmay/whetherman/blob/main/eval/external/bfcl/summary.json - task: type: text-classification name: Should any function be called? dataset: name: BFCL v4 live_irrelevance type: gorilla-llm/Berkeley-Function-Calling-Leaderboard metrics: - type: accuracy value: 0.8643 name: pass rate (764/884) source: name: self-reported url: https://github.com/vinmay/whetherman/blob/main/eval/external/bfcl/summary.json - task: type: text-classification name: Should any function be called? dataset: name: BFCL v4 live_relevance type: gorilla-llm/Berkeley-Function-Calling-Leaderboard metrics: - type: accuracy value: 0.375 name: pass rate (6/16) source: name: self-reported url: https://github.com/vinmay/whetherman/blob/main/eval/external/bfcl/summary.json --- # Whetherman (FunctionGemma-270M LoRA + 4-class head) **Research artifact.** A 270M router that forecasts, before each model call that offers tools, whether an agent needs a tool: `ACT`, `ANSWER`, `ASK` or `DECLINE`. It is the backbone of the [Whetherman](https://github.com/vinmay/whetherman) proxy, which removes tools only when a non-ACT decision is confident enough for the specific agent, and otherwise fails open to ACT. **Read the results before using it.** On held-out end-to-end tasks the gate did **not** produce a net benefit for the large agent we measured (details below). It is published so the measurements can be checked and the method reused, not as a production cost-saver. ## What is in this repository | File | What it is | |---|---| | `adapter_model.safetensors`, `adapter_config.json` | LoRA adapter (r=16, alpha=32, all attention and MLP projections) for `google/functiongemma-270m-it` | | `head.pt` | 4-class linear head on the last-token hidden state (ACT, ANSWER, ASK, DECLINE) | | `profiles/L.json`, `profiles/S.json`, `profiles/U.json` | Per-agent profiles: a regularized logistic layer on the hidden state plus threshold τ, fitted by `whetherman calibrate` | | `GEMMA_TERMS.md` | Gemma Terms of Use notice | Agents: **L** gpt-oss-120b (DigitalOcean serverless), **S** Qwen3.5-4B Q4_K_M (local Ollama), **U** gpt-oss-20b (DigitalOcean), calibrated as a new agent after the backbone was trained. Profiles describe one agent on one serving stack on one day; recalibrate for yours. ## Usage Use it through the proxy (see the [GitHub README](https://github.com/vinmay/whetherman)): ```bash git clone https://github.com/vinmay/whetherman && cd whetherman && uv sync --extra ml hf download vinmaynair/whetherman-functiongemma-270m --local-dir checkpoints/shared-v3 uv run whetherman calibrate --agent-endpoint http://localhost:11434/v1 --model --name my-agent uv run whetherman serve --profile my-agent --upstream http://localhost:11434/v1 ``` `whetherman calibrate` measures your agent on a probe pack and fits a profile; `whetherman serve` gates your agent's Chat Completions requests. You need access to the gated base model (accept the Gemma terms on Hugging Face). ## Training - Base: `google/functiongemma-270m-it` (revision `39eccb09`), LoRA + linear head, FP32 on a 6 GB GTX 1660 Ti, 3 epochs, about 50 minutes. No agent identity in the input. - Data: When2Call training files (ACT/ASK/DECLINE; ASK vs DECLINE by a documented heuristic), templated multi-step decision points with real tool execution, and When2Tool train tasks labeled by measuring each agent with and without tools (agreement items as hard labels, disagreements as soft targets). - Deduplicated against every evaluation split (text hash + embedding similarity > 0.95). ## Results (self-reported; frozen model, held-out sets run once) Everything below was measured on held-out sets, run once after the backbone, profiles and code were frozen (`eval/results/freeze.json`, tag `freeze-v3`). Full tables with every configuration, CI and set: [eval/results/REPORT_final.md](https://github.com/vinmay/whetherman/blob/main/eval/results/REPORT_final.md), generated by `eval/report.py` from the raw result files. ### End to end: no gate vs Whetherman 80 held-out tasks (40 multi-step) × 2 repeats per agent. "Headline" excludes 10 guessable two-choice episodes (DATA.md). CIs are Wilson 95%. | Agent | No gate | Whetherman | Tool calls / episode | Tokens / episode | Cost (whole suite) | |---|---|---|---|---|---| | L: gpt-oss-120b (DigitalOcean) | 120/150 [0.73, 0.86] | **105/150** [0.62, 0.77] | 2.36 → 2.15 | 1906 → 1897 | $0.0522 → $0.0651 | | S: Qwen3.5-4B (local Ollama) | 130/150 [0.80, 0.91] | **128/150** [0.79, 0.90] | 1.92 → 1.91 | 2297 → 2156 | local | | U: gpt-oss-20b (unseen agent) | 115/150 [0.69, 0.83] | **114/150** [0.69, 0.82] | 2.09 → 1.93 | 2062 → 1824 | $0.0461 → $0.0438 | Other gates in the same runs (headline success): the shared backbone without a profile 99 (L) and 84 (S); a prompted Qwen3.5-4B router 108 and 96; a rules-only control 90 and 60. Every gate that removed many calls lost many successes. **What went wrong for L** (post-hoc analysis of the frozen run, not used to change anything): - 11 of the 15 lost successes have replay evidence of a false skip: restoring ACT at the skipped step made the episode succeed. 7 of those are multi-hop retrieval tasks, where the gate answered after the first lookup. Every probe task is a first-step decision (the pack has 9 single-hop retrieval tasks and no later steps), so L's profile never saw "one lookup done, another needed". - Removing tools made gpt-oss write longer answers (completion tokens 36k → 58k over the suite; output costs 7× input), and it called removed tools 20 times, each costing a second request. Fewer calls did not mean fewer dollars. - Where L is genuinely capable (calculator, code, matrix and prime tasks), calls fell from 18 to 6 with no lost success. ### Per-agent calibration matters Offline, on held-out When2Tool tasks labeled by measuring each agent (false skip = tools withheld from a point that needed them): | | Shared, no profile | Own profile (shipped) | Other agent's profile | |---|---|---|---| | S false skips | 65/70 | **4/70** | 42/70 | | L false skips | 13/17 | **1/17** | 3/17 | End to end, giving S the profile fitted for L cost 32 of 150 successes (98 vs 130). The same request can be ANSWER for one agent and ACT for another; one global threshold cannot serve both. The shipped profiles are conservative: on the same held-out sets they skipped 86 of L's 129 and 23 of S's 66 unnecessary calls, and on hand-written conversations (E3) only 19/62 and 14/60. A comparison model with the agent's ID in its training input skipped far more (125/129 for L) with several times the false skips (6/17; S 11/70). ### Agent-agnostic routing (When2Call, BFCL) On 900 held-out When2Call MCQ items the shared backbone reaches macro-F1 0.642, against 0.519 for untuned FunctionGemma, 0.281 for the prompted Qwen router and 0.197 for rules. Its validation score was 0.925: the official test distribution is harder than our training-derived validation split. Self-run with external code, tuned backbone only (baselines are compared on the 900-item subset above): | Benchmark | Result | |---|---| | When2Call official MCQ test, 3,652 items ([NVIDIA/When2Call](https://github.com/NVIDIA/When2Call) @ ecc8d42, official metric code) | accuracy 0.628, macro-F1 0.472*, tool hallucination rate 0.070 | | BFCL v4 irrelevance (no function fits; pass = not ACT) | 187/240 (0.78); prompted Qwen 95/240 | | BFCL v4 live irrelevance | 764/884 (0.86); prompted Qwen 533/884 | | BFCL v4 live relevance (pass = ACT) | 6/16 (0.38); prompted Qwen 10/16 | \*The router's class is mapped to the matching MCQ option. The official macro-F1 averages over every class seen in gold or predictions; "direct answer" is never gold but was predicted 6 times, adding a zero. Over the three gold classes it is 0.629. BFCL is a self-run out-of-distribution check, not a leaderboard result; no BFCL question overlaps our training text. ### Cost of running Whetherman | | GPU (GTX 1660 Ti) p50 / p95 | CPU p50 / p95 | |---|---|---| | Short request, 2 tools (128 tokens) | 55 / 105 ms | 188 / 204 ms | | 8 tools (830 tokens) | 122 / 127 ms | 1075 / 1203 ms | | Long history (1504 tokens) | 249 / 410 ms | 2152 / 3434 ms | Calibration: L $0.24 (83 min of agent time), U $0.35 (154 min), S 19 min locally, about 120 usable probe labels each. At the measured savings, U's calibration pays back after roughly 24,000 episodes; L's does not pay back. Total hosted spend for the whole project: see `eval/results/spend.json`. ## Laya vs FunctionGemma A P2 comparison, offline only (no end-to-end runs), asking whether a calibrated encoder classifier routes as well as FunctionGemma and is faster. [Laya](https://huggingface.co/convaiinnovations/laya) (convaiinnovations, Apache-2.0, `laya` 0.4.13, checkpoint pinned at its reviewed revision `55cf4c4`) is a ModernBERT-large decision encoder with a 512-token input limit. It was trained on the same split, labels and seed as the backbone, with no agent identity, and calibrated per agent with the same probe points, adaptation/threshold split, feature head and τ rule. Two variants, chosen by a rule fixed in advance (validation macro-F1): (a) frozen encoder + 4-class head, 0.802; **(b) LoRA fine-tune, 0.837, selected**. FunctionGemma on the same validation set: 0.925. It was frozen separately (`eval/results/freeze_laya.json`) and its held-out sets run once. | Held-out | FunctionGemma (shipped) | Laya (b) | |---|---|---| | E1-test, default policy: macro-F1 / false skips / ECE | **0.642** / **93/300** / **0.228** | 0.497 / 238/300 / 0.267 | | L, E2-test adapted: false skips / correct skips | 1/17 / **86/129** | 0/17 / 0/129 (τ = 1.0: no safe threshold, never skips) | | S, E2-test adapted: false skips / correct skips | **4/70** / 23/66 | 7/70 / 26/66 | | S, E3 adapted: false skips / correct skips | 2/62 / **14/60** | 2/62 / 7/60 | | GPU p50: 2 tools / 8 tools / long history | 55 / 122 / **249** ms | **36** / 122 / 510 ms | | CPU p50: 2 tools / 8 tools / long history | **188** / **1075** / 2152 ms | 396 / 1707 / **2034** ms | | Peak VRAM, inference / training | **1.2** / 2.2 GB | 1.5 / **2.1** GB | Laya truncation (inputs over 512 tokens; oldest history dropped first, schemas and latest tool results kept): 11/900 E1-test items (1.2%), 4/121 E3 (3.3%), none on E2. **Verdict:** FunctionGemma is far ahead on E1 and on every set for L, where Laya's profile found no safe threshold and never skips. On S the two are close: Laya's macro-F1 is slightly higher (E2 0.627 vs 0.616, E3 0.329 vs 0.319), with more false skips on E2 (7/70 vs 4/70) and fewer correct skips on E3 (7/60 vs 14/60). Laya is faster on short GPU inputs and on the longest CPU input (where it has truncated to 512 tokens), slower elsewhere. Every row with CIs: `eval/results/REPORT_final.md`. Raw: `eval/results/laya_final.json`; dev and option (a): `laya_dev*.json`. **Encoder baseline.** A LoRA-fine-tuned [Laya](https://huggingface.co/convaiinnovations/laya) encoder (ModernBERT-large, Apache-2.0), trained and calibrated the same way, was compared offline. It was far behind on When2Call and on the large agent (no safe threshold), and close on the small agent. Details: the ["Laya vs FunctionGemma"](https://github.com/vinmay/whetherman#laya-vs-functiongemma) section of the GitHub README. That baseline is not published. ## Limitations - Two to three agents, measured once each at temperature 0. Labels are measured evidence, not ground truth. - Small denominators; confidence intervals are wide. - Probe packs do not cover multi-hop retrieval; the gate stops such tasks one lookup early. - Removing tools can make a model write longer answers, which can erase dollar savings. - gpt-oss sometimes calls tools that were removed; the proxy re-sends the original request. - **Not a security guardrail.** It decides whether a tool call is useful, not whether it is allowed. ## License and terms The adapter is a derivative of Gemma and is distributed under the [Gemma Terms of Use](https://ai.google.dev/gemma/terms), including the [Prohibited Use Policy](https://ai.google.dev/gemma/prohibited_use_policy). Modification notice: this repository contains LoRA weights and a classification head trained on top of FunctionGemma-270M by the Whetherman project; it changes the model's task from text generation to 4-way classification. Training data: When2Call (CC-BY-4.0, NVIDIA), When2Tool (MIT), and hand-written data (Apache-2.0). Code: Apache-2.0 at the GitHub repository.