Nanbeige

/

Nanbeige4.2-3B

like 429

Follow

Nanbeige LLM Lab 1.16k

Heads-up for anyone running this model through the nanbeige42 llama.cpp fork: a chunk of its tool calls are being thrown away by the parser, not by the model. The model emits them correctly. I measured it and patched it locally, so I wanted to share, since it makes the model look meaningfully worse than it is.

Setup: nanbeige4.2-3b-Q8_0.gguf (quantized from BF16 with the fork's llama-quantize), fork at d28da86, branch nanbeige42, --jinja, and the card's agentic sampling (temp 1.0, top-p 0.95, top-k 20).

What happens

Roughly one call in six comes back with tool_calls empty and the entire reply sitting in message.content as plain text, like this:

<tool_call> 
<function=bash>
<parameter=command>
ls -la /tmp
</parameter>
</function>
</tool_call>

That is a perfectly well-formed call. The only difference from the ones that parse is a single space after <tool_call>, before the newline.

It looks like a sampling artifact: in a small probe (4 samples per config, same prompt) every generation parsed at temp 0, while at temp 1.0 some configurations dropped to 2/4. That means the card's recommended agentic temp of 1.0 is where it hits -- worst-case scenario.

Why

llama.cpp's auto-parser analyzes the template and derives per_call_start = "<tool_call>\n", with the newline baked into the marker. Two places in build_tool_parser_tag_tagged (common/chat-auto-parser-generator.cpp) then require it verbatim:

1. the call marker is built as an exact literal, so <tool_call> \n does not match; and
2. the content boundary is p.until(trigger_marker) using that same untrimmed marker.

(2) is the one that actually causes the damage. until() looks for the literal <tool_call>\n, never finds it in padded output, and consumes the whole generation as content - so the tool-call rule never even runs. That is why the failure looks like "the model declined to call a tool" rather than a parse error. Fixing (1) alone changes nothing.

Fix

Two lines, both reusing helpers already in that file:

-        auto wrapped_call = format.per_call_start + p.space() + tool_choice + p.space() + format.per_call_end;
+        auto wrapped_call = p.optspace(format.per_call_start) + p.space() + tool_choice + p.space() + format.per_call_end;

-    auto        content_before_tools = trigger_marker.empty() ? p.eps() : p.until(trigger_marker);
+    auto        content_before_tools = trigger_marker.empty() ? p.eps() : p.until(trim_whitespace(trigger_marker));

Careful if you apply by hand: that second context line appears twice in the file. The one to change is in build_tool_parser_tag_tagged. The identical line in build_tool_parser_tag_json is a different format with no test coverage, and I left it alone.

Measured effect

35-case tool-calling suite hitting the OpenAI-compatible endpoint directly (29 tool cases, 6 abstention cases), same server and sampling before and after, Q8_0:

All five recovered calls were already well-formed. By category, simple went 50% -> 100% and tool_choice 40% -> 100% valid.

Note on where the fix belongs

That file is unmodified from upstream in your fork, so ggml-org/llama.cpp is affected identically. Nemotron lands in the same code path with the same per_call_start = "<tool_call>\n" (asserted in llama.cpp's own test suite), and Seed-OSS renders the same marker-plus-newline shape (<seed:tool_call>\n<function=...>), though I have not run the analyzer on it to confirm. Fixing it upstream would cover your model and theirs at once. I have the patch plus a regression test ready and am happy to send it wherever is most useful; your fork has issues disabled, which is why I am posting here.

(Disclosure: the diagnosis was done with the help of AI coding agents - isolating the failing input and reading the PEG parser trace to find the until() boundary as the real culprit. The measurements above are from my own runs.)