nvidia

/

GLM-5.2-NVFP4

like 308

Follow

NVIDIA 65.1k

Summary

When sending tool_choice: "required" with chat_template_kwargs: {"enable_thinking": false} to GLM-5.2 served via vLLM 0.24.0, the request either returns a 500 error or hangs indefinitely. Omitting tool_choice or setting it to "auto" works without issue.

Environment

• Model: GLM-5.2 (served via Ray Serve + vLLM)
• vLLM version: 0.24.0 (system_fingerprint: vllm-0.24.0-tp8)
• Guided decoding backend: xgrammar (default)
• Deployment: Ray Serve cluster (multiple worker VMs, tensor parallelism across 8 GPUs)
• API: OpenAI-compatible /v1/chat/completions
  Reproduction
  Minimal request that triggers the bug:

payload = {
    "model": "glm-5-2",
    "messages": [{"role": "user", "content": "Say hello"}],
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "test_tool",
                "parameters": {
                    "type": "object",
                    "properties": {"arg": {"type": "string"}},
                    "required": ["arg"],
                },
                "description": "A test tool",
            },
        }
    ],
    "tool_choice": "required",
    "chat_template_kwargs": {"enable_thinking": False},
    "stream": False,
}
# POST to /v1/chat/completions → 500 Internal Server Error

With 4 tools and no max_tokens, the same request hangs indefinitely (no response ever returned).
Server-side error logs
From the vLLM EngineCore on the worker VMs:

(EngineCore pid=458) ERROR 07-24 20:32:41 [backend_xgrammar.py:162] 
  Failed to advance FSM for request chatcmpl-55347ca6-... for tokens 5961. Please file an issue.

(EngineCore pid=458) ERROR 07-24 20:32:41 [scheduler.py:1606] 
  Unexpected: grammar rejected tokens [5961, 13, 154842, 40] for request chatcmpl-55347ca6-... Terminating request.

(EngineCore pid=458) ERROR 07-24 20:32:41 [serving.py:120] 
  Request chatcmpl-55347ca6-... failed with an internal error during generation
A second pattern seen on another worker:
(EngineCore pid=466) ERROR 07-24 20:28:58 [backend_xgrammar.py:162] 
  Failed to advance FSM for request chatcmpl-7008a7bc-... for tokens 154842. Please file an issue.

(EngineCore pid=466) ERROR 07-24 20:28:58 [scheduler.py:1606] 
  Unexpected: grammar rejected tokens [154842, 9703, 0, 61370] for request chatcmpl-7008a7bc-... Terminating request.
The propagated error:
vllm.entrypoints.openai.engine.protocol.GenerationError: Internal server error

Analysis

The xgrammar guided-decoding backend builds an FSM from the tools' JSON schemas when tool_choice: "required" is set. GLM-5.2 generates special tokens (154842, 5961) during its natural output flow. The xgrammar FSM has no transition for these tokens, causing:

1. 500 error — FSM fails to advance, scheduler terminates the request → GenerationError
2. Infinite hang — With more complex tool schemas (4 tools) and no max_tokens, the constrained sampler enters a state where it cannot find any valid next token that both the model wants to produce and the FSM accepts. The request loops indefinitely until the client times out.

Token 154842 is likely a thinking-mode special token (e.g. or similar) that the FSM grammar for tool-call JSON does not account for.

Workaround

Setting tool_choice: "auto" works correctly — the model reliably calls tools due to the system prompt. Alternatively, forcing a specific function via tool_choice: {"type": "function", "function": {"name": "..."}} also works.

Questions

1. Is this a known issue with the xgrammar backend and models that emit special/thinking tokens during generation?
2. Could the guided-decoding FSM be made to tolerate or skip special tokens that aren't part of the tool-call grammar?
3. Would switching to the outlines backend (--guided-decoding-backend outlines) avoid this issue?