Heads-up for 0.5.19 users: the new prefill CUDA graph holds 1.81 GB and starves quantized-KV long-context prefill
Now that 1.2.2 makes SGLang 0.5.19 reachable, here is something worth knowing before anyone follows that path on a small card. It is not an escha bug — it is an interaction — but it is invisible unless you go looking for it.
SGLang 0.5.19 captures a prefill CUDA graph by default: 51 token shapes (4096 down to 4), holding 1.81 GB.
If you also run a quantized KV cache (we use OSCAR INT2), prefill has to dequantize the cached prefix into a dense tensor, and that transient is proportional to sequence length — about 873 MB at 96K for us. The two do not fit together on a 12 GB card, and the failure mode is not an OOM. The GPU keeps reporting 100% utilisation while it stalls in allocation.
Measured on Qwen3.8-27B-Escha-W2 + OSCAR INT2 KV, RTX 5070 12GB x2, TP2:
| prompt | graph on | graph off |
|---|---|---|
| 10K | 907 tok/s | 968 |
| 32K | 955 | 994 |
| 64K | 918 | 985 |
| 96K | 202 | 893 |
| free VRAM | 0.54 GB | 2.35 GB |
--disable-prefill-cuda-graph gives 4.4x at 96K and makes short prompts 4-7% faster. The reason it costs nothing: at --chunked-prefill-size 4096 the only shape ever used is 4096, so 50 of the 51 captured shapes never fire. Keep the decode graph — it only captures bs=[1,2,4] and wastes nothing.
SGLang does auto-disable the prefill graph in some cases, but every rule is model-correctness driven (unified memory, Inkling, Muse-Glimmer). No rule looks at free VRAM. So this will not be caught for you.
Two other flags that mattered on 0.5.19, in case they are useful for a default:
--mamba-radix-cache-strategy extra_buffer_lazy— on hybrid SSM models, one running request consumes 5 mamba slots under the default strategy and 4 underextra_buffer_lazy. Sincemax_running_requestsis clamped tomax-mamba-cache-size / slots-per-request, this alone took us from 3 to 4 effective concurrency, +47% aggregate at 4 clients and +32% at 8, with no change to single-stream speed.PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True— already in yourserve.sh, and it is worth keeping: it gave us +7,680 KV pool tokens and +0.08 GB free.
Full write-up, patches against stock 0.5.19, and all measurements: https://github.com/Nakanokensetsu/sglang -> escha_oscar_0519/
Posting this here because escha is what people actually install, so this is where the warning will be seen. Feel free to fold any of it into the model card or serve.sh defaults.