README.md

---
license: mit
tags:
- vllm
- inference-optimization
- benchmarking
- moe
- rocm
---
<!-- TODO: add the upstream model's `license:` value to the frontmatter above before publishing. -->

# GLM-5.2 — Tuned vLLM Serving Recipe (ROCm)

Docker build, launch script, and benchmark-reproduction script for serving
[zai-org/GLM-5.2](https://huggingface.co/zai-org/GLM-5.2) with vLLM on AMD
Instinct GPUs.

This repository does not contain model weights. `vllm serve` pulls them
directly from the upstream repo above (and caches them) when the server
starts.

## Model Overview

- MoE causal LM using MLA (multi-head latent attention) with dynamic sparse attention
- Served at 65,536 max context; requires `--trust-remote-code`
- Workload is prefill-dominated and compute-bound

## Environment

| | |
|---|---|
| Hardware | 8x AMD Instinct MI325X 256GB, single node |
| Software | ROCm 7.2.3 · vLLM v0.25.1+rocm723 |
| Input Length | 409–3,686 tokens · nominal 2,048 · uniform |
| Output Length | 51–460 tokens · nominal 256 · uniform |
| Framework | vllm bench serve |
| vLLM | 0.25.1 |
| Dataset | random (synthetic) · range ratio 0.8 |
| Base Docker image | `vllm/vllm-openai-rocm:v0.25.1` |
| Parallelism | Tensor-parallel 8, pipeline-parallel 1, `mp` distributed executor |
| Attention | `ROCM_AITER_MLA_SPARSE` decode backend, `ROCM_AITER_FA` MLA prefill |
| Features | AITER kernels, FP8 KV cache, prefix caching, async scheduling, CUDA graphs |
| Prompts Measured | 1,000 · warm-ups excluded: 10 |
| Request Rate | inf |
| Tested Concurrency | 512 |

## Optimization Changes

- FP8 (e4m3) KV cache
- `--gpu-memory-utilization` lowered 0.95 → 0.90 to leave headroom for graph capture
- `--max-num-seqs 256`, `--max-num-batched-tokens 16384`
- `--long-prefill-token-threshold 4096` so long prefills stop blocking decode
- Enabled prefix caching and async scheduling
- `--compilation-config` with `cudagraph_mode FULL_AND_PIECEWISE`, max capture size 256
- `VLLM_ROCM_USE_AITER=1`, `VLLM_WORKER_MULTIPROC_METHOD=spawn`
- Speculative decoding deliberately omitted — MTP is the only speculator shipped with GLM-5.2, and it deadlocks on gfx942 at TP8 ([vllm-project/vllm#48568](https://github.com/vllm-project/vllm/issues/48568))

### Docker

### Installation

Build the image:

```bash
docker build -t your-org/glm-5.2-vllm:latest .
```

Run it (ROCm containers use `--device`/`--group-add`, not `--gpus`):

```bash
docker run \
  --device=/dev/kfd --device=/dev/dri \
  --group-add video --ipc=host --shm-size 16g \
  --security-opt seccomp=unconfined \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  your-org/glm-5.2-vllm:latest optimized
```

Pass `baseline` instead of `optimized` as the final argument to run the
unoptimized configuration used for the comparison below.

### Baseline

```bash
export VLLM_ROCM_USE_AITER=1
vllm serve zai-org/GLM-5.2 \  --tensor-parallel-size 8 \  --trust-remote-code \  --gpu-memory-utilization 0.95 \  --max-model-len 65536
```

### Optimized

```bash
xport VLLM_ROCM_USE_AITER=1
export VLLM_WORKER_MULTIPROC_METHOD=spawn
vllm serve zai-org/GLM-5.2 \  --host 0.0.0.0 --port 8000 \  --tensor-parallel-size 8 \  --pipeline-parallel-size 1 \  
--distributed-executor-backend mp \  --max-model-len 65536 \  --gpu-memory-utilization 0.90 \  --kv-cache-dtype fp8_e4m3 \  --max-num-seqs 256 \  --max-num-batched-tokens 16384 \  --long-prefill-token-threshold 4096 \  
--enable-prefix-caching \  --async-scheduling \  --compilation-config ‘{“cudagraph_mode”: “FULL_AND_PIECEWISE”, “max_cudagraph_capture_size”: 256}’ \  --trust-remote-code
```

### Validation

Once the container is up, confirm the server is healthy and serving:

```bash
curl -sf http://localhost:8000/health && echo "server is healthy"

curl -s http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "zai-org/GLM-5.2", "prompt": "The capital of France is", "max_tokens": 16}'
```

### Files in this repository

| File | Purpose |
|---|---|
| `Dockerfile` | `vllm/vllm-openai-rocm:v0.25.1`-based image with the two scripts below baked in |
| `launch_server.sh` | Starts `vllm serve` with the baseline or optimized flags |
| `run_benchmark.sh` | Starts baseline + optimized servers, benchmarks each, and prints the comparison table below — fully automated |
| `README.md` | This model card |

### Shell Script

### Launch Server

```bash
./launch_server.sh optimized   # or: ./launch_server.sh baseline
```

This is also the image's default command — `docker run ... <image> optimized`
runs the same thing. Inside a running container, invoke it with
`docker exec -it <container> ./launch_server.sh <mode>`.

### Reproduce Benchmark

This is a separate container invocation from "Launch Server" above — it
overrides the entrypoint so `run_benchmark.sh` (not `launch_server.sh`)
drives the container. It starts the baseline server itself, benchmarks it,
shuts it down, starts the optimized server, benchmarks that, shuts it down,
and prints the comparison table — nothing to copy in by hand:

```bash
docker run \
  --device=/dev/kfd --device=/dev/dri \
  --group-add video --ipc=host --shm-size 16g \
  --security-opt seccomp=unconfined \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --entrypoint ./run_benchmark.sh \
  your-org/glm-5.2-vllm:latest both
```

That prints the full table in the terminal when it finishes (large model,
8-GPU node — expect this to take a while: two model loads plus two
1,000-prompt runs). Pass `baseline` or `optimized` instead of `both` to run
just one side. The raw per-run JSON (`benchmark_baseline.json`,
`benchmark_optimized.json`) and each server's startup log
(`server_baseline.log`, `server_optimized.log`) are left in the container's
`/workspace` if you want them too:

```bash
docker cp <container>:/workspace/benchmark_optimized.json .
```

**Methodology** (`vllm bench serve`):

- Dataset: `random` (synthetic), range ratio 0.8
- Input length: 409–3,686 tokens (nominal 2,048, uniform)
- Output length: 51–460 tokens (nominal 256, uniform)
- 1,000 measured prompts, 10 warm-up prompts (discarded, not saved)
- Request rate: unbounded (`inf`) — max concurrency is the only throttle
- Max concurrency: 512 (best command tuned at this concurrency; override with `-e MAX_CONCURRENCY=...`)

## Results

8x AMD Instinct MI325X 256GB, vLLM v0.25.1+rocm723. This is exactly the
table `run_benchmark.sh both` prints — running it is the only step required
to reproduce it:

| Metric | Baseline | Optimized |
|---|---|---|
| Request waiting (peak) | 491 | 478 | 
| Request waiting (avg) | 169.5 | 95.7 | 
| Output tok/s | 1019.0 | 1292.2 | 
| TTFT mean (ms) | 49744 | 23744 | 
| TTFT median (ms) | 51841 | 11448 | 
| TTFT p99 (ms) | 104700 | 77224 | 
| TPOT mean (ms) | 269.7 | 294.2 | 
| TPOT p99 (ms) | 524.6 | 744.4 | 
| ITL mean (ms) | 258.1 | 272.2 | 
| ITL p99 (ms) | 738.9 | 1079.2 | 
| E2E median (ms) | 110457 | 85579 | 
| KV cache peak | 100% | 100% | 
| Duration (s) | 254.8 | 200.9 |