|
Download vllm_patch/README.md from internlm/Sage-S2.1-Flash: direct link, hf CLI and curl.
- Browser
- Download file 2.81 kB
-
https://huggingface.co/internlm/Sage-S2.1-Flash/resolve/main/vllm_patch/README.md
- Command line
-
hf download hf://internlm/Sage-S2.1-Flash/vllm_patch/README.md
-
curl -L -o README.md https://huggingface.co/internlm/Sage-S2.1-Flash/resolve/main/vllm_patch/README.md
2.81 kB
vLLM 0.33.0 patch: tool-result image order (Sage-S2.1-Flash)
What it fixes
Sage-S2.1-Flash's chat template renders a block of tool results in the order of the assistant's
tool_calls, but vLLM collects their images in message order. When results come back out of
call order, each image is silently paired with the wrong result.
What the patch changes
One file, vllm/renderers/hf.py:
- Adds
_glm5_next_tool_media_order(), which sorts a block of tool results intotool_callsorder under the template's own rule, only when the block contains images or videos. - Calls it right before
parse_chat_messagesinHfRenderer.render_messagesandrender_messages_async. - Only
model_type == "glm5_next"is affected. The rendered prompt text does not change.
Files
| File | Use |
|---|---|
vllm-0.33.0/vllm/renderers/hf.py |
Patched file for vLLM 0.33.0, copied over the installed one |
vllm-0.33.0-glm5-tool-media-order.patch |
The same change as a git diff, for review or other versions |
check_tool_media_order.py |
CPU check (no weights) |
Apply (vLLM 0.33.0)
PATCH=<model dir>/vllm_patch
VLLM_DIR=$(python -c "import vllm, os; print(os.path.dirname(vllm.__file__))")
sha256sum $VLLM_DIR/renderers/hf.py
# must be 3c84c86b348b539fc741085327b165f4b6bf958cc6c0e19fb5dc0f54430432d2 (stock 0.33.0)
cp $VLLM_DIR/renderers/hf.py $VLLM_DIR/renderers/hf.py.bak
cp $PATCH/vllm-0.33.0/vllm/renderers/hf.py $VLLM_DIR/renderers/hf.py
sha256sum $VLLM_DIR/renderers/hf.py
# 1ce7ab5909848579e37174d6ad2af3fa48b797198376e9e7757bc26af575eda7
If the first checksum differs, the file is not stock 0.33.0: do not overwrite it. To revert,
copy hf.py.bak back.
Check
python -c "import vllm.renderers.hf as h; print(hasattr(h, '_glm5_next_tool_media_order'))" # True
python $PATCH/check_tool_media_order.py <model dir> # all OK
Then start vllm serve as usual.
Other versions
The diff also applies to 0.30.0, 0.31.0, 0.32.0 and nightly 0.34.0rc1.dev397. Run it in
site-packages:
git apply --check $PATCH/vllm-0.33.0-glm5-tool-media-order.patch
git apply $PATCH/vllm-0.33.0-glm5-tool-media-order.patch # or: patch -p1 -b < ...
Not needed before 0.30.0 (no glm5_next support), nor with vLLM's Rust frontend
(VLLM_USE_RUST_FRONTEND=1 --tokenizer-mode glm5_next).
Verification
- CPU (
check_tool_media_order.py): 7 cases on 0.30.0 to 0.33.0 and nightly. Before the patch, the reversed cases are mispaired. After it, all 7 match the template. - GPU (0.33.0, 4xH200): with reversed tool results, the model answered the wrong image's color before the patch and the right one after it.
Upstream: vllm-project/vllm#67108 (draft) adds a similar, model-agnostic reordering.