--- license: mit tags: - vllm - inference-optimization - benchmarking - moe - rocm --- # GLM-5.2 — Tuned vLLM Serving Recipe (ROCm) Docker build, launch script, and benchmark-reproduction script for serving [zai-org/GLM-5.2](https://huggingface.co/zai-org/GLM-5.2) with vLLM on AMD Instinct GPUs. This repository does not contain model weights. `vllm serve` pulls them directly from the upstream repo above (and caches them) when the server starts. ## Model Overview - MoE causal LM using MLA (multi-head latent attention) with dynamic sparse attention - Served at 65,536 max context; requires `--trust-remote-code` - Workload is prefill-dominated and compute-bound ## Environment | | | |---|---| | Hardware | 8x AMD Instinct MI325X 256GB, single node | | Software | ROCm 7.2.3 · vLLM v0.25.1+rocm723 | | Input Length | 409–3,686 tokens · nominal 2,048 · uniform | | Output Length | 51–460 tokens · nominal 256 · uniform | | Framework | vllm bench serve | | vLLM | 0.25.1 | | Dataset | random (synthetic) · range ratio 0.8 | | Base Docker image | `vllm/vllm-openai-rocm:v0.25.1` | | Parallelism | Tensor-parallel 8, pipeline-parallel 1, `mp` distributed executor | | Attention | `ROCM_AITER_MLA_SPARSE` decode backend, `ROCM_AITER_FA` MLA prefill | | Features | AITER kernels, FP8 KV cache, prefix caching, async scheduling, CUDA graphs | | Prompts Measured | 1,000 · warm-ups excluded: 10 | | Request Rate | inf | | Tested Concurrency | 512 | ## Optimization Changes - FP8 (e4m3) KV cache - `--gpu-memory-utilization` lowered 0.95 → 0.90 to leave headroom for graph capture - `--max-num-seqs 256`, `--max-num-batched-tokens 16384` - `--long-prefill-token-threshold 4096` so long prefills stop blocking decode - Enabled prefix caching and async scheduling - `--compilation-config` with `cudagraph_mode FULL_AND_PIECEWISE`, max capture size 256 - `VLLM_ROCM_USE_AITER=1`, `VLLM_WORKER_MULTIPROC_METHOD=spawn` - Speculative decoding deliberately omitted — MTP is the only speculator shipped with GLM-5.2, and it deadlocks on gfx942 at TP8 ([vllm-project/vllm#48568](https://github.com/vllm-project/vllm/issues/48568)) ### Docker ### Installation Build the image: ```bash docker build -t your-org/glm-5.2-vllm:latest . ``` Run it (ROCm containers use `--device`/`--group-add`, not `--gpus`): ```bash docker run \ --device=/dev/kfd --device=/dev/dri \ --group-add video --ipc=host --shm-size 16g \ --security-opt seccomp=unconfined \ -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ your-org/glm-5.2-vllm:latest optimized ``` Pass `baseline` instead of `optimized` as the final argument to run the unoptimized configuration used for the comparison below. ### Baseline ```bash export VLLM_ROCM_USE_AITER=1 vllm serve zai-org/GLM-5.2 \ --tensor-parallel-size 8 \ --trust-remote-code \ --gpu-memory-utilization 0.95 \ --max-model-len 65536 ``` ### Optimized ```bash xport VLLM_ROCM_USE_AITER=1 export VLLM_WORKER_MULTIPROC_METHOD=spawn vllm serve zai-org/GLM-5.2 \ --host 0.0.0.0 --port 8000 \ --tensor-parallel-size 8 \ --pipeline-parallel-size 1 \ --distributed-executor-backend mp \ --max-model-len 65536 \ --gpu-memory-utilization 0.90 \ --kv-cache-dtype fp8_e4m3 \ --max-num-seqs 256 \ --max-num-batched-tokens 16384 \ --long-prefill-token-threshold 4096 \ --enable-prefix-caching \ --async-scheduling \ --compilation-config ‘{“cudagraph_mode”: “FULL_AND_PIECEWISE”, “max_cudagraph_capture_size”: 256}’ \ --trust-remote-code ``` ### Validation Once the container is up, confirm the server is healthy and serving: ```bash curl -sf http://localhost:8000/health && echo "server is healthy" curl -s http://localhost:8000/v1/completions \ -H "Content-Type: application/json" \ -d '{"model": "zai-org/GLM-5.2", "prompt": "The capital of France is", "max_tokens": 16}' ``` ### Files in this repository | File | Purpose | |---|---| | `Dockerfile` | `vllm/vllm-openai-rocm:v0.25.1`-based image with the two scripts below baked in | | `launch_server.sh` | Starts `vllm serve` with the baseline or optimized flags | | `run_benchmark.sh` | Starts baseline + optimized servers, benchmarks each, and prints the comparison table below — fully automated | | `README.md` | This model card | ### Shell Script ### Launch Server ```bash ./launch_server.sh optimized # or: ./launch_server.sh baseline ``` This is also the image's default command — `docker run ... optimized` runs the same thing. Inside a running container, invoke it with `docker exec -it ./launch_server.sh `. ### Reproduce Benchmark This is a separate container invocation from "Launch Server" above — it overrides the entrypoint so `run_benchmark.sh` (not `launch_server.sh`) drives the container. It starts the baseline server itself, benchmarks it, shuts it down, starts the optimized server, benchmarks that, shuts it down, and prints the comparison table — nothing to copy in by hand: ```bash docker run \ --device=/dev/kfd --device=/dev/dri \ --group-add video --ipc=host --shm-size 16g \ --security-opt seccomp=unconfined \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --entrypoint ./run_benchmark.sh \ your-org/glm-5.2-vllm:latest both ``` That prints the full table in the terminal when it finishes (large model, 8-GPU node — expect this to take a while: two model loads plus two 1,000-prompt runs). Pass `baseline` or `optimized` instead of `both` to run just one side. The raw per-run JSON (`benchmark_baseline.json`, `benchmark_optimized.json`) and each server's startup log (`server_baseline.log`, `server_optimized.log`) are left in the container's `/workspace` if you want them too: ```bash docker cp :/workspace/benchmark_optimized.json . ``` **Methodology** (`vllm bench serve`): - Dataset: `random` (synthetic), range ratio 0.8 - Input length: 409–3,686 tokens (nominal 2,048, uniform) - Output length: 51–460 tokens (nominal 256, uniform) - 1,000 measured prompts, 10 warm-up prompts (discarded, not saved) - Request rate: unbounded (`inf`) — max concurrency is the only throttle - Max concurrency: 512 (best command tuned at this concurrency; override with `-e MAX_CONCURRENCY=...`) ## Results 8x AMD Instinct MI325X 256GB, vLLM v0.25.1+rocm723. This is exactly the table `run_benchmark.sh both` prints — running it is the only step required to reproduce it: | Metric | Baseline | Optimized | |---|---|---| | Request waiting (peak) | 491 | 478 | | Request waiting (avg) | 169.5 | 95.7 | | Output tok/s | 1019.0 | 1292.2 | | TTFT mean (ms) | 49744 | 23744 | | TTFT median (ms) | 51841 | 11448 | | TTFT p99 (ms) | 104700 | 77224 | | TPOT mean (ms) | 269.7 | 294.2 | | TPOT p99 (ms) | 524.6 | 744.4 | | ITL mean (ms) | 258.1 | 272.2 | | ITL p99 (ms) | 738.9 | 1079.2 | | E2E median (ms) | 110457 | 85579 | | KV cache peak | 100% | 100% | | Duration (s) | 254.8 | 200.9 |