haijunlv's picture HuayuSha's picture
Add deployment guide and vLLM 0.33.0 patch; sync tokenizer and processor code (#1)
54acab6
|
Raw History Blame Contribute Delete
2.81 kB

vLLM 0.33.0 patch: tool-result image order (Sage-S2.1-Flash)

English | 中文

What it fixes

Sage-S2.1-Flash's chat template renders a block of tool results in the order of the assistant's tool_calls, but vLLM collects their images in message order. When results come back out of call order, each image is silently paired with the wrong result.

What the patch changes

One file, vllm/renderers/hf.py:

  • Adds _glm5_next_tool_media_order(), which sorts a block of tool results into tool_calls order under the template's own rule, only when the block contains images or videos.
  • Calls it right before parse_chat_messages in HfRenderer.render_messages and render_messages_async.
  • Only model_type == "glm5_next" is affected. The rendered prompt text does not change.

Files

File Use
vllm-0.33.0/vllm/renderers/hf.py Patched file for vLLM 0.33.0, copied over the installed one
vllm-0.33.0-glm5-tool-media-order.patch The same change as a git diff, for review or other versions
check_tool_media_order.py CPU check (no weights)

Apply (vLLM 0.33.0)

PATCH=<model dir>/vllm_patch
VLLM_DIR=$(python -c "import vllm, os; print(os.path.dirname(vllm.__file__))")
sha256sum $VLLM_DIR/renderers/hf.py
# must be 3c84c86b348b539fc741085327b165f4b6bf958cc6c0e19fb5dc0f54430432d2 (stock 0.33.0)
cp $VLLM_DIR/renderers/hf.py $VLLM_DIR/renderers/hf.py.bak
cp $PATCH/vllm-0.33.0/vllm/renderers/hf.py $VLLM_DIR/renderers/hf.py
sha256sum $VLLM_DIR/renderers/hf.py
# 1ce7ab5909848579e37174d6ad2af3fa48b797198376e9e7757bc26af575eda7

If the first checksum differs, the file is not stock 0.33.0: do not overwrite it. To revert, copy hf.py.bak back.

Check

python -c "import vllm.renderers.hf as h; print(hasattr(h, '_glm5_next_tool_media_order'))"  # True
python $PATCH/check_tool_media_order.py <model dir>    # all OK

Then start vllm serve as usual.

Other versions

The diff also applies to 0.30.0, 0.31.0, 0.32.0 and nightly 0.34.0rc1.dev397. Run it in site-packages:

git apply --check $PATCH/vllm-0.33.0-glm5-tool-media-order.patch
git apply $PATCH/vllm-0.33.0-glm5-tool-media-order.patch    # or: patch -p1 -b < ...

Not needed before 0.30.0 (no glm5_next support), nor with vLLM's Rust frontend (VLLM_USE_RUST_FRONTEND=1 --tokenizer-mode glm5_next).

Verification

  • CPU (check_tool_media_order.py): 7 cases on 0.30.0 to 0.33.0 and nightly. Before the patch, the reversed cases are mispaired. After it, all 7 match the template.
  • GPU (0.33.0, 4xH200): with reversed tool results, the model answered the wrong image's color before the patch and the right one after it.

Upstream: vllm-project/vllm#67108 (draft) adds a similar, model-agnostic reordering.