deployment_guide: link the official GLM-5.3-Flash recipes; explain --trust-remote-code and the vLLM patch

#2
Files changed (1) hide show
  1. deployment_guide.md +23 -7
deployment_guide.md CHANGED
@@ -2,14 +2,22 @@
2
 
3
  The Sage-S2.1-Flash release is a 320B-A18B multimodal MoE model stored in bfloat16 weight format (about 645 GB). The vLLM and SGLang examples below use 8× H200 GPUs. This guide provides basic serving examples for LMDeploy, vLLM and SGLang.
4
 
 
 
5
  > NOTE: The commands below are reference configurations. Inference frameworks are under active development, so use the latest framework documentation and your local validation results when tuning production deployments.
6
 
7
- - Pass `--trust-remote-code` to every engine. The repository's tokenizer encodes control-token spellings that appear in message content (for example `<tool_call>` or `</think>` in a user message or a tool result) as ordinary text. Without the flag, the engines silently load the stock tokenizer, which encodes them as control tokens.
8
- - On Hopper GPUs (H100, H200), keep the default BF16 KV cache: FP8 KV cache is not supported for this model there.
 
 
 
 
9
 
10
  ## LMDeploy
11
 
12
- Use LMDeploy built from the main branch with `transformers>=5.17,<5.19`, and install [FlashMLA](https://github.com/deepseek-ai/FlashMLA). transformers 5.19 and later rename the layer type `deepseek_sparse_attention` to `indexed_attention`, which LMDeploy does not accept yet.
 
 
13
 
14
  ```bash
15
  pip install "transformers>=5.17,<5.19"
@@ -24,7 +32,11 @@ lmdeploy serve api_server \
24
 
25
  ## vLLM
26
 
27
- Use vLLM 0.33.0 with the patch in [`vllm_patch/`](./vllm_patch/README.md). It replaces one file, `vllm/renderers/hf.py`, so that images in tool results are paired with the right results when they come back out of call order. `vllm serve` does not download this folder, so fetch it first (this downloads only the patch, not the weights):
 
 
 
 
28
 
29
  ```bash
30
  hf download internlm/Sage-S2.1-Flash --include "vllm_patch/*" --local-dir /path/to/Sage-S2.1-Flash
@@ -41,17 +53,21 @@ vllm serve internlm/Sage-S2.1-Flash \
41
  --reasoning-parser glm47
42
  ```
43
 
44
- These commands use vLLM's default Python frontend. The Rust frontend (`VLLM_USE_RUST_FRONTEND=1 --tokenizer-mode glm5_next`) does not accept image or video input yet.
 
 
45
 
46
  ## SGLang
47
 
48
- Use SGLang 0.5.20 or later. Pass the repository's chat template explicitly: by default SGLang installs its built-in template for this architecture, which does not go through the repository's tokenizer. `--chat-template` takes a local file:
49
 
50
  ```bash
51
  hf download internlm/Sage-S2.1-Flash chat_template.jinja --local-dir /path/to/Sage-S2.1-Flash
52
  ```
53
 
54
- Then pass the downloaded file to `--chat-template`. The server downloads the weights itself:
 
 
55
 
56
  ```bash
57
  python3 -m sglang.launch_server \
 
2
 
3
  The Sage-S2.1-Flash release is a 320B-A18B multimodal MoE model stored in bfloat16 weight format (about 645 GB). The vLLM and SGLang examples below use 8× H200 GPUs. This guide provides basic serving examples for LMDeploy, vLLM and SGLang.
4
 
5
+ Sage-S2.1-Flash uses the GLM-5.3-Flash architecture, so install and tune vLLM and SGLang following the official GLM-5.3-Flash recipes linked in their sections, replacing `zai-org/GLM-5.3-Flash` with `internlm/Sage-S2.1-Flash`. On top of the official commands, Sage-S2.1-Flash needs the two additions in the next section.
6
+
7
  > NOTE: The commands below are reference configurations. Inference frameworks are under active development, so use the latest framework documentation and your local validation results when tuning production deployments.
8
 
9
+ ## Required for Sage-S2.1-Flash
10
+
11
+ 1. **Pass `--trust-remote-code` to every engine.** Code, logs and tool outputs in agentic and machine-learning engineering tasks often contain strings such as `<|user|>`, `<tool_call>` or `</think>`. With the flag, the repository's tokenizer encodes these strings in message content as ordinary text. Without it, the engines silently load the stock tokenizer, which encodes them as control tokens: the model can stop early, and tool calls and reasoning can break. Nothing warns you when the flag is missing.
12
+ 2. **vLLM: apply the patch in [`vllm_patch/`](./vllm_patch/README.md)** before starting the server (see [vLLM](#vllm)).
13
+
14
+ On Hopper GPUs (H100, H200), keep the default BF16 KV cache: FP8 KV cache is not supported for this model there.
15
 
16
  ## LMDeploy
17
 
18
+ Use [LMDeploy](https://github.com/InternLM/lmdeploy) built from the main branch with `transformers>=5.17,<5.19`, and install [FlashMLA](https://github.com/deepseek-ai/FlashMLA). transformers 5.19 and later rename the layer type `deepseek_sparse_attention` to `indexed_attention`, which LMDeploy does not accept yet.
19
+
20
+ **Keep `--trust-remote-code`:** the repository ships a custom tokenization strategy (`tokenization_glm5.py`, `processing_glm5.py`) that LMDeploy loads only with this flag. See [Required for Sage-S2.1-Flash](#required-for-sage-s21-flash).
21
 
22
  ```bash
23
  pip install "transformers>=5.17,<5.19"
 
32
 
33
  ## vLLM
34
 
35
+ Install vLLM 0.33.0 following the official [GLM-5.3-Flash vLLM recipe](https://recipes.vllm.ai/zai-org/GLM-5.3-Flash), then apply the patch in [`vllm_patch/`](./vllm_patch/README.md) and pass `--trust-remote-code`.
36
+
37
+ **Why the patch:** when the model calls several tools and their results come back in a different order than the calls (common with parallel tool calls), the chat template renders the results in call order, but vLLM collects their images in message order. Each image is then silently attached to the wrong result, and the model answers about the wrong image. The patch replaces one file, `vllm/renderers/hf.py`, to put such results in call order before vLLM collects the images. The upstream fix is in progress in [vllm-project/vllm#67108](https://github.com/vllm-project/vllm/pull/67108).
38
+
39
+ `vllm serve` does not download the patch folder, so fetch it first (this downloads only the patch, not the weights):
40
 
41
  ```bash
42
  hf download internlm/Sage-S2.1-Flash --include "vllm_patch/*" --local-dir /path/to/Sage-S2.1-Flash
 
53
  --reasoning-parser glm47
54
  ```
55
 
56
+ These commands use vLLM's default Python frontend. The official recipe defaults to the Rust frontend (`VLLM_USE_RUST_FRONTEND=1 --tokenizer-mode glm5_next`), which does not accept image or video input yet.
57
+
58
+ Known issue: in non-streaming responses, a `</think>` or `<tool_call>` that the model writes as ordinary text can end the reasoning early or hide a tool call ([vllm-project/vllm#64326](https://github.com/vllm-project/vllm/issues/64326)). Streaming responses are not affected.
59
 
60
  ## SGLang
61
 
62
+ Install SGLang 0.5.20 or later following the official [GLM-5.3-Flash SGLang cookbook](https://cookbook.sglang.io/autoregressive/GLM/GLM-5.3-Flash), then pass `--trust-remote-code` and the repository's chat template explicitly: by default SGLang installs its built-in template for this architecture, which does not go through the repository's tokenizer. `--chat-template` takes a local file:
63
 
64
  ```bash
65
  hf download internlm/Sage-S2.1-Flash chat_template.jinja --local-dir /path/to/Sage-S2.1-Flash
66
  ```
67
 
68
+ Then pass the downloaded file to `--chat-template`. The server downloads the weights itself.
69
+
70
+ **Keep `--trust-remote-code`:** the repository ships a custom tokenization strategy (`tokenization_glm5.py`, `processing_glm5.py`) that SGLang loads only with this flag. See [Required for Sage-S2.1-Flash](#required-for-sage-s21-flash).
71
 
72
  ```bash
73
  python3 -m sglang.launch_server \