Cards: tool-calling test of the Q3_K_M GGUF (109/117), speed on an older CPU

README.md

@@ -71,7 +71,22 @@ Validation loss on 249 held-out examples (`retok_test`, not used for training),
 | --- | --- | --- | --- | --- |
 | val loss | 1.0169 | 0.5549 | 0.5017 | **0.4926** |
 
-Part of this drop is the model learning our response format, masks and tool format; it is not by itself a measure of reasoning ability. Standard benchmarks (GSM8K, HumanEval, tool calling) are **not measured yet** and will be added here.
+Part of this drop is the model learning our response format, masks and tool format; it is not by itself a measure of reasoning ability. GSM8K and HumanEval are not measured yet; tool calling is measured below.
+
+### Tool calling (GGUF Q3_K_M)
+
+Our internal 117-case tool-calling test on the smallest GGUF file, `JiRackDeltaNet_70b_with_prompt.Q3_K_M.gguf`, served by `llama-server` as an OpenAI-compatible API: tools are passed as standard OpenAI `tools` and parsed by llama.cpp's own Qwen 3.5 template, temperature 0, reasoning off. The cases are written by us and are not in the training data.
+
+| Category | Q3_K_M |
+| --- | --- |
+| Single call (one tool, all arguments given) | 38/38 |
+| Parallel calls | 19/19 |
+| Follow-up (answer from a tool result) | 20/20 |
+| No tool needed | 25/25 |
+| Clarify a missing argument | 7/15 |
+| **Total** | **109/117 (93.2%)** |
+
+All 8 misses are in clarify: the model called the tool instead of asking for the missing argument.
 
 Sample (greedy, thinking off):
 
@@ -105,6 +120,7 @@ llama-server -m JiRackDeltaNet_70b_with_prompt.Q4_K_M.gguf -c 8192 --jinja --rea
 | Size | 33.89 GB | 42.65 GB | 57.40 GB (2 parts) |
 | CPU speed, prompt / generation (GH200 Grace, 64 threads, `llama-bench`) | 20.1 / 5.17 tokens/s | 50.2 / 5.04 tokens/s | 45.3 / 4.11 tokens/s |
 | Pens question above | $29 | $29 | $29 |
+| Tool-calling test (117 cases) | 109/117 (93.2%) | — | — |
 
 ## Usage
 
@@ -124,6 +140,7 @@ The reference chat scripts recompute the whole sequence for every new token (no
 
 ## Limitations
 
+- **Missing arguments**: when a required argument is missing, the model often calls the tool anyway instead of asking (7/15 on our test); validate tool calls before executing them.
 - **Limited training of the new layers**: the 64 added layers started as exact identities and have seen only this SFT (\~45 M tokens). Real gains from the extra depth need continued pretraining on billions of tokens.
 - **Not ternary yet**: JiRack's BitLinear (b1.58) path is present in the code but off (lambda = 0); ternary QAT is a planned later stage.
 - Trained with sequences up to 8,192 tokens.

gguf/README.md

@@ -86,10 +86,15 @@ running. Answers: `llama-server --reasoning off`, greedy, built-in prompt.
 | Prompt processing (pp512) | 20.1 tokens/s | 50.2 tokens/s | 45.3 tokens/s |
 | Generation (tg128) | 5.17 tokens/s | 5.04 tokens/s | 4.11 tokens/s |
 | Pens question | $29 | $29 | $29 |
+| Tool-calling test (117 cases) | 109/117 (93.2%) | — | — |
 
 Q4_K_M answer: "Tom buys 7 pens at $3 each, so the total cost is 7 * 3 = $21. He pays with a $50 bill, so the change
 he gets is 50 - 21 = $29."
 
+Tool calling (Q3_K_M): **109/117 (93.2%)** on our internal 117-case test through the OpenAI-compatible API of `llama-server` (standard `tools`, llama.cpp's Qwen 3.5 tool-call parser, temperature 0, reasoning off): single call 38/38, parallel calls 19/19, follow-up 20/20, no tool needed 25/25, clarify a missing argument 7/15 (it called the tool instead of asking).
+
+On an older server CPU without AVX2 (2 × Xeon E5-2697 v2, 24 cores), Q3_K_M runs at 2.06 tokens/s prompt processing and 0.86 tokens/s generation.
+
 Each file was checked against the GGUF it was made from:
 - the only changed metadata key is the chat template;
 - all 1,714 tensors have the same names, types and shapes;
@@ -110,8 +115,8 @@ slower.
 ## Limitations
 
 - The 64 layers added by depth expansion have been trained only by the SFT run (~45 M tokens). See the main model card.
-- These are lossy quantizations of the bf16 weights, made without an importance matrix. Benchmarks of the GGUF files
-  (GSM8K, HumanEval, tool calling) are not measured yet.
+- These are lossy quantizations of the bf16 weights, made without an importance matrix. GSM8K and HumanEval
+  of the GGUF files are not measured yet; tool calling is measured on Q3_K_M (above).
 - Not ternary yet: ternary QAT (and a TQ2_0 GGUF) is a planned later stage.
 - An Ollama build is not published yet.
 - Not for medical, legal or financial advice.