beyoru commited on
Commit
a44cb42
·
verified ·
1 Parent(s): 297fc05

Add README.md: sample serving config

Browse files
Files changed (1) hide show
  1. README.md +8 -0
README.md CHANGED
@@ -108,10 +108,18 @@ docker run -d --name kat-vl \
108
  --model /models/KAT-Coder-V2.5-Dev-VL \
109
  --served-model-name kat-vl \
110
  --max-model-len 32768 \
 
111
  --gpu-memory-utilization 0.90 \
112
  --trust-remote-code
113
  ```
114
 
 
 
 
 
 
 
 
115
  Or without Docker (`vllm>=0.26.0`):
116
 
117
  ```bash
 
108
  --model /models/KAT-Coder-V2.5-Dev-VL \
109
  --served-model-name kat-vl \
110
  --max-model-len 32768 \
111
+ --max-num-batched-tokens 8192 \
112
  --gpu-memory-utilization 0.90 \
113
  --trust-remote-code
114
  ```
115
 
116
+ > **`--max-num-batched-tokens 8192` is required.** This is a hybrid attention + Gated-DeltaNet
117
+ > architecture. With prefix caching on (the default in recent vLLM), vLLM forces
118
+ > `mamba_cache_mode='align'` and raises the attention block size to **2096** so that "attention page
119
+ > size >= mamba page size". The default `max_num_batched_tokens` is **2048**, so startup dies with
120
+ > `AssertionError: In Mamba cache align mode, block_size (2096) must be <= max_num_batched_tokens
121
+ > (2048)`. Any value >= 2096 works.
122
+
123
  Or without Docker (`vllm>=0.26.0`):
124
 
125
  ```bash