SH4P3S/apertus-4b-tooluse

Apertus-4B Tool-Use (Round 2)

QLoRA fine-tune of Apertus-4B for on-device tool calling (llama.cpp, target: Samsung Galaxy S24). Current model: apertus-4b-r2-Q5_K_M.gguf (round 2) — use it together with sysprompt.txt and (optionally) tools.gbnf from this repo. The round-1 files (apertus-4b-Q5_K_M.gguf, apertus-4b-Q4_0.gguf) are kept for reference only.

What's new in round 2

Fresh QLoRA from base on a patched dataset fixing five trained-in biases:

1. Date anchoring: the system prompt now ends with Current date: <Weekday>, YYYY-MM-DD. The model resolves "tomorrow" / "nächsten Dienstag" against that line — it has no clock of its own. The app/tester MUST inject this line in exactly that format.
2. Weather for a named place goes directly to get_weather (no get_current_location chain).
3. A concrete URL goes directly to fetch_url (no web_search first).
4. Empty notes hits no longer escape to web_search.
5. Asks clarifying questions far more often (patch mix 19% vs 3%).

Setup (Windows; Mac/Linux analogous)

1. Download llama.cpp binaries: https://github.com/ggml-org/llama.cpp/releases → llama-<build>-bin-win-cpu-x64.zip → unzip.
2. Start the server:

   llama-server.exe -m apertus-4b-r2-Q5_K_M.gguf -c 4096 -t 4 --port 8080
   

3. Open http://127.0.0.1:8080 in a browser.
4. Settings (gear icon):
   • System Message: paste the FULL contents of sysprompt.txt. IMPORTANT: first update the last line to the actual current date, keeping the exact format Current date: <Weekday>, YYYY-MM-DD (e.g. Current date: Monday, 2026-07-13).
   • Temperature 0.8, top-p 0.9.

How it works

The assistant emits at most one tool call per turn:

<tool_call>{"name": "web_search", "arguments": {"query": "..."}}</tool_call>

Tool results come back as a user message (no custom tool role — the stock chat template works):

<tool_response>{"results": [{"title": "...", "snippet": "...", "url": "https://x.y"}]}</tool_response>

With no real tools attached, you play the tool: reply with an invented <tool_response>, and the model answers grounded in it. Errors work too: <tool_response>{"error": "service_unavailable"}</tool_response>.

11 tools: web_search, get_current_location, get_weather, create_calendar_event, set_timer, create_note, search_notes, read_note, append_to_note, fetch_url, get_calendar_events (full specs in sysprompt.txt).

Grammar (optional, advanced)

tools.gbnf is a llama.cpp GBNF grammar that FORCES tool calls to be valid JSON with correct argument names. The browser UI can't use it; it applies via the API ("grammar" field in /completion) or llama-cli --grammar-file. It constrains output to a tool call, so only use it on turns where a tool call is expected.

Eval snapshot (r2 Q5_K_M + grammar, 121 cases, eval date pinned 2026-09-17)

metric	score
tool-call validity	100%
correct first tool	93%
expected content in answer	97%
forbidden tool avoided	92%
no over-calling	86%
asks when ambiguous	70%
error recovery	100%

Good things to try

• "Stell mir einen Timer auf 10 Minuten für die Pasta." → set_timer call
• "Was ist 17 mal 23?" → should answer DIRECTLY, no tool
• "Wie ist das Wetter in Bern?" → get_weather directly (round-2 fix)
• "Wie ist das Wetter gerade hier bei mir?" → get_current_location, then get_weather
• "Trag Zahnarzt für nächsten Dienstag 14 Uhr ein." → create_calendar_event with the date resolved against the Current date line
• "Was hatte ich mir zum Velo notiert?" → search_notes, then read_note
• "Stell mir einen Timer." (no duration) → should ask how long
• German and English both work; Swiss German is not supported by design.

Expectations

• CPU inference: ~4–10 tokens/sec depending on machine. First reply is slower (system prompt processing).
• Known residual weakness: relative-date arithmetic for calendar events is still shaky (event_date_ok 44% in eval) — verify dates it books.