Make the chat template work in llama.cpp and add a response template
Replaces two Jinja features llama.cpp doesn't support (x.0 indexing and the unique filter), which made every conversation with tool calls fail there; output in Transformers and vLLM is unchanged. Also adds a response template, so Transformers' parse_response turns the model's output back into a message (reasoning, content and tool calls), matching vLLM's own GLM parser.
Thanks for the contribution. I reviewed commit 087acb5 and checked the input-template changes separately from the new response-parsing configuration.
The Jinja changes look reasonable, and I did not find a prompt-formatting or tokenization regression in the cases tested. The response_template addition introduces separate behavior changes that I would like to address before merging.
The validation covered:
| Check | Result |
|---|---|
Actual Transformers AutoTokenizer: 34 targeted scenarios and 100 seeded randomized scenarios, combined with different reasoning_effort, clear_thinking, and add_generation_prompt settings |
All 1,608 configurations rendered successfully, with identical text and token IDs between the old and new templates |
| Actual llama.cpp C++ Jinja runtime | Both templates passed 134 scenarios each and matched the Transformers reference output |
Transformers parse_response and ResponseParser |
20 cases, with 1,339 split-point or character-by-character comparisons; no differences between incremental and complete parsing |
| Transformers pipeline postprocessing with synthetic token IDs | Confirmed automatic use of response_template and reproduced the whitespace issue below |
These checks used Transformers 58d05ca5, llama.cpp 3259ab6e, and vLLM source at 1a983e2e. They cover template rendering and output parsing;
Please preserve whitespace in parsed content.
With
prefix="<|assistant|><think>", the generated text"</think> return x\n"currently producescontent="return x", removing both indentation and the trailing newline.Similarly,
before\n<tool_call>f</tool_call>\nafterproducescontent="beforeafter"because each text region is stripped before being joined.This comes from the text parser’s default stripping behavior combined with
join: "". Adding"content_args": {"strip": false}to thecontentfield preserved whitespace in the reproduced cases. Please add regression tests for leading indentation, trailing newlines, and text surrounding tool calls. The whitespace policy forreasoning_contentshould also be explicit.This affects existing Transformers chat-pipeline usage: the pipeline automatically calls
parse_responsewhen the tokenizer provides a response template. Separating reasoning and tool calls is expected, but losing content whitespace should be avoided.Please clarify and test incomplete tool-call handling.
When generation ends inside an
<arg_value>without a closing</tool_call>, the response parser can still return atool_callsentry containing only the arguments completed so far. The result can therefore look like a parsed function call while required arguments are missing.This differs from the current vLLM Rust GLM XML parser, whose
finish()returnsIncompleteToolCallfor an unfinished call.Please define the intended contract for truncated output and cover it with tests. If this requires a change in the shared Transformers response parser, that dependency should be tracked explicitly. Until then, the claim of matching vLLM’s parser should be qualified for incomplete outputs.
The Rust compatibility scope should be documented.
The response template contains lookahead expressions such as
(?=...), which are unsupported by the regex engine used in the current vLLM Rust generichfoutput parser. This configuration therefore cannot currently be used with that parser.This does not change the behavior of the existing
glm47/glm45output parsers: addingresponse_templatedoes not automatically select the generichfparser. The native Rustglm5_nextrenderer also bypasses the Jinja template.However,
vllm servealone does not imply that native path is active. In the reviewed vLLM revision, the Rust frontend is opt-in, and the Python frontend’s GLM renderer still uses the HF chat-template path. The equivalence tests above support the input-template change on that path.
I’m comfortable with the Jinja-only changes based on this validation. For the response-template addition, I would like to see the whitespace issue fixed, incomplete-call behavior covered, and the compatibility scope clarified.
Thanks again for improving compatibility and adding structured response parsing.