Update README.md

00e8f64 verified 27 days ago

7.05 kB

---
license: mit
tags:
- vllm
- inference-optimization
- benchmarking
- moe
- rocm
---
<!-- TODO: add the upstream model's `license:` value to the frontmatter above before publishing. -->

# GLM-5.2 — Tuned vLLM Serving Recipe (ROCm)

Docker build, launch script, and benchmark-reproduction script for serving
[zai-org/GLM-5.2](https://huggingface.co/zai-org/GLM-5.2) with vLLM on AMD
Instinct GPUs.

This repository does not contain model weights. `vllm serve` pulls them
directly from the upstream repo above (and caches them) when the server
starts.

## Model Overview

- MoE causal LM using MLA (multi-head latent attention) with dynamic sparse attention
- Served at 65,536 max context; requires `--trust-remote-code`
- Workload is prefill-dominated and compute-bound

## Environment

| | |
|---|---|
| Hardware | 8x AMD Instinct MI325X 256GB, single node |
| Software | ROCm 7.2.3 · vLLM v0.25.1+rocm723 |
| Input Length | 409–3,686 tokens · nominal 2,048 · uniform |
| Output Length | 51–460 tokens · nominal 256 · uniform |
| Framework | vllm bench serve |
| vLLM | 0.25.1 |
| Dataset | random (synthetic) · range ratio 0.8 |
| Base Docker image | `vllm/vllm-openai-rocm:v0.25.1` |
| Parallelism | Tensor-parallel 8, pipeline-parallel 1, `mp` distributed executor |
| Attention | `ROCM_AITER_MLA_SPARSE` decode backend, `ROCM_AITER_FA` MLA prefill |
| Features | AITER kernels, FP8 KV cache, prefix caching, async scheduling, CUDA graphs |
| Prompts Measured | 1,000 · warm-ups excluded: 10 |
| Request Rate | inf |
| Tested Concurrency | 512 |

## Optimization Changes

- FP8 (e4m3) KV cache
- `--gpu-memory-utilization` lowered 0.95 → 0.90 to leave headroom for graph capture
- `--max-num-seqs 256`, `--max-num-batched-tokens 16384`
- `--long-prefill-token-threshold 4096` so long prefills stop blocking decode
- Enabled prefix caching and async scheduling
- `--compilation-config` with `cudagraph_mode FULL_AND_PIECEWISE`, max capture size 256
- `VLLM_ROCM_USE_AITER=1`, `VLLM_WORKER_MULTIPROC_METHOD=spawn`
- Speculative decoding deliberately omitted — MTP is the only speculator shipped with GLM-5.2, and it deadlocks on gfx942 at TP8 ([vllm-project/vllm#48568](https://github.com/vllm-project/vllm/issues/48568))

### Docker

### Installation

Build the image:

```bash
docker build -t your-org/glm-5.2-vllm:latest .
```

Run it (ROCm containers use `--device`/`--group-add`, not `--gpus`):

```bash
docker run \
  --device=/dev/kfd --device=/dev/dri \
  --group-add video --ipc=host --shm-size 16g \
  --security-opt seccomp=unconfined \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  your-org/glm-5.2-vllm:latest optimized
```

Pass `baseline` instead of `optimized` as the final argument to run the
unoptimized configuration used for the comparison below.

### Baseline

```bash
export VLLM_ROCM_USE_AITER=1
vllm serve zai-org/GLM-5.2 \  --tensor-parallel-size 8 \  --trust-remote-code \  --gpu-memory-utilization 0.95 \  --max-model-len 65536
```

### Optimized

```bash
xport VLLM_ROCM_USE_AITER=1
export VLLM_WORKER_MULTIPROC_METHOD=spawn
vllm serve zai-org/GLM-5.2 \  --host 0.0.0.0 --port 8000 \  --tensor-parallel-size 8 \  --pipeline-parallel-size 1 \  
--distributed-executor-backend mp \  --max-model-len 65536 \  --gpu-memory-utilization 0.90 \  --kv-cache-dtype fp8_e4m3 \  --max-num-seqs 256 \  --max-num-batched-tokens 16384 \  --long-prefill-token-threshold 4096 \  
--enable-prefix-caching \  --async-scheduling \  --compilation-config ‘{“cudagraph_mode”: “FULL_AND_PIECEWISE”, “max_cudagraph_capture_size”: 256}’ \  --trust-remote-code
```

### Validation

Once the container is up, confirm the server is healthy and serving:

```bash
curl -sf http://localhost:8000/health && echo "server is healthy"
curl -s http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "zai-org/GLM-5.2", "prompt": "The capital of France is", "max_tokens": 16}'
```

### Files in this repository

| File | Purpose |
|---|---|
| `Dockerfile` | `vllm/vllm-openai-rocm:v0.25.1`-based image with the two scripts below baked in |
| `launch_server.sh` | Starts `vllm serve` with the baseline or optimized flags |
| `run_benchmark.sh` | Starts baseline + optimized servers, benchmarks each, and prints the comparison table below — fully automated |
| `README.md` | This model card |

### Shell Script

### Launch Server

```bash
./launch_server.sh optimized   # or: ./launch_server.sh baseline
```

This is also the image's default command — `docker run ... <image> optimized`
runs the same thing. Inside a running container, invoke it with
`docker exec -it <container> ./launch_server.sh <mode>`.

### Reproduce Benchmark

This is a separate container invocation from "Launch Server" above — it
overrides the entrypoint so `run_benchmark.sh` (not `launch_server.sh`)
drives the container. It starts the baseline server itself, benchmarks it,
shuts it down, starts the optimized server, benchmarks that, shuts it down,
and prints the comparison table — nothing to copy in by hand:

```bash
docker run \
  --device=/dev/kfd --device=/dev/dri \
  --group-add video --ipc=host --shm-size 16g \
  --security-opt seccomp=unconfined \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --entrypoint ./run_benchmark.sh \
  your-org/glm-5.2-vllm:latest both
```

That prints the full table in the terminal when it finishes (large model,
8-GPU node — expect this to take a while: two model loads plus two
1,000-prompt runs). Pass `baseline` or `optimized` instead of `both` to run
just one side. The raw per-run JSON (`benchmark_baseline.json`,
`benchmark_optimized.json`) and each server's startup log
(`server_baseline.log`, `server_optimized.log`) are left in the container's
`/workspace` if you want them too:

```bash
docker cp <container>:/workspace/benchmark_optimized.json .
```

**Methodology** (`vllm bench serve`):

- Dataset: `random` (synthetic), range ratio 0.8
- Input length: 409–3,686 tokens (nominal 2,048, uniform)
- Output length: 51–460 tokens (nominal 256, uniform)
- 1,000 measured prompts, 10 warm-up prompts (discarded, not saved)
- Request rate: unbounded (`inf`) — max concurrency is the only throttle
- Max concurrency: 512 (best command tuned at this concurrency; override with `-e MAX_CONCURRENCY=...`)

## Results

8x AMD Instinct MI325X 256GB, vLLM v0.25.1+rocm723. This is exactly the
table `run_benchmark.sh both` prints — running it is the only step required
to reproduce it:

| Metric | Baseline | Optimized |
|---|---|---|
| Request waiting (peak) | 491 | 478 | 
| Request waiting (avg) | 169.5 | 95.7 | 
| Output tok/s | 1019.0 | 1292.2 | 
| TTFT mean (ms) | 49744 | 23744 | 
| TTFT median (ms) | 51841 | 11448 | 
| TTFT p99 (ms) | 104700 | 77224 | 
| TPOT mean (ms) | 269.7 | 294.2 | 
| TPOT p99 (ms) | 524.6 | 744.4 | 
| ITL mean (ms) | 258.1 | 272.2 | 
| ITL p99 (ms) | 738.9 | 1079.2 | 
| E2E median (ms) | 110457 | 85579 | 
| KV cache peak | 100% | 100% | 
| Duration (s) | 254.8 | 200.9 |

	---
	license: mit
	tags:
	- vllm
	- inference-optimization
	- benchmarking
	- moe
	- rocm
	---
	<!-- TODO: add the upstream model's `license:` value to the frontmatter above before publishing. -->

	# GLM-5.2 — Tuned vLLM Serving Recipe (ROCm)

	Docker build, launch script, and benchmark-reproduction script for serving
	[zai-org/GLM-5.2](https://huggingface.co/zai-org/GLM-5.2) with vLLM on AMD
	Instinct GPUs.

	This repository does not contain model weights. `vllm serve` pulls them
	directly from the upstream repo above (and caches them) when the server
	starts.

	## Model Overview

	- MoE causal LM using MLA (multi-head latent attention) with dynamic sparse attention
	- Served at 65,536 max context; requires `--trust-remote-code`
	- Workload is prefill-dominated and compute-bound

	## Environment

	\| \| \|
	\|---\|---\|
	\| Hardware \| 8x AMD Instinct MI325X 256GB, single node \|
	\| Software \| ROCm 7.2.3 · vLLM v0.25.1+rocm723 \|
	\| Input Length \| 409–3,686 tokens · nominal 2,048 · uniform \|
	\| Output Length \| 51–460 tokens · nominal 256 · uniform \|
	\| Framework \| vllm bench serve \|
	\| vLLM \| 0.25.1 \|
	\| Dataset \| random (synthetic) · range ratio 0.8 \|
	\| Base Docker image \| `vllm/vllm-openai-rocm:v0.25.1` \|
	\| Parallelism \| Tensor-parallel 8, pipeline-parallel 1, `mp` distributed executor \|
	\| Attention \| `ROCM_AITER_MLA_SPARSE` decode backend, `ROCM_AITER_FA` MLA prefill \|
	\| Features \| AITER kernels, FP8 KV cache, prefix caching, async scheduling, CUDA graphs \|
	\| Prompts Measured \| 1,000 · warm-ups excluded: 10 \|
	\| Request Rate \| inf \|
	\| Tested Concurrency \| 512 \|

	## Optimization Changes

	- FP8 (e4m3) KV cache
	- `--gpu-memory-utilization` lowered 0.95 → 0.90 to leave headroom for graph capture
	- `--max-num-seqs 256`, `--max-num-batched-tokens 16384`
	- `--long-prefill-token-threshold 4096` so long prefills stop blocking decode
	- Enabled prefix caching and async scheduling
	- `--compilation-config` with `cudagraph_mode FULL_AND_PIECEWISE`, max capture size 256
	- `VLLM_ROCM_USE_AITER=1`, `VLLM_WORKER_MULTIPROC_METHOD=spawn`
	- Speculative decoding deliberately omitted — MTP is the only speculator shipped with GLM-5.2, and it deadlocks on gfx942 at TP8 ([vllm-project/vllm#48568](https://github.com/vllm-project/vllm/issues/48568))

	### Docker

	### Installation

	Build the image:

	```bash
	docker build -t your-org/glm-5.2-vllm:latest .
	```

	Run it (ROCm containers use `--device`/`--group-add`, not `--gpus`):

	```bash
	docker run \
	--device=/dev/kfd --device=/dev/dri \
	--group-add video --ipc=host --shm-size 16g \
	--security-opt seccomp=unconfined \
	-p 8000:8000 \
	-v ~/.cache/huggingface:/root/.cache/huggingface \
	your-org/glm-5.2-vllm:latest optimized
	```

	Pass `baseline` instead of `optimized` as the final argument to run the
	unoptimized configuration used for the comparison below.

	### Baseline

	```bash
	export VLLM_ROCM_USE_AITER=1
	vllm serve zai-org/GLM-5.2 \ --tensor-parallel-size 8 \ --trust-remote-code \ --gpu-memory-utilization 0.95 \ --max-model-len 65536
	```

	### Optimized

	```bash
	xport VLLM_ROCM_USE_AITER=1
	export VLLM_WORKER_MULTIPROC_METHOD=spawn
	vllm serve zai-org/GLM-5.2 \ --host 0.0.0.0 --port 8000 \ --tensor-parallel-size 8 \ --pipeline-parallel-size 1 \
	--distributed-executor-backend mp \ --max-model-len 65536 \ --gpu-memory-utilization 0.90 \ --kv-cache-dtype fp8_e4m3 \ --max-num-seqs 256 \ --max-num-batched-tokens 16384 \ --long-prefill-token-threshold 4096 \
	--enable-prefix-caching \ --async-scheduling \ --compilation-config ‘{“cudagraph_mode”: “FULL_AND_PIECEWISE”, “max_cudagraph_capture_size”: 256}’ \ --trust-remote-code
	```

	### Validation

	Once the container is up, confirm the server is healthy and serving:

	```bash
	curl -sf http://localhost:8000/health && echo "server is healthy"

	curl -s http://localhost:8000/v1/completions \
	-H "Content-Type: application/json" \
	-d '{"model": "zai-org/GLM-5.2", "prompt": "The capital of France is", "max_tokens": 16}'
	```

	### Files in this repository

	\| File \| Purpose \|
	\|---\|---\|
	\| `Dockerfile` \| `vllm/vllm-openai-rocm:v0.25.1`-based image with the two scripts below baked in \|
	\| `launch_server.sh` \| Starts `vllm serve` with the baseline or optimized flags \|
	\| `run_benchmark.sh` \| Starts baseline + optimized servers, benchmarks each, and prints the comparison table below — fully automated \|
	\| `README.md` \| This model card \|

	### Shell Script

	### Launch Server

	```bash
	./launch_server.sh optimized # or: ./launch_server.sh baseline
	```

	This is also the image's default command — `docker run ... <image> optimized`
	runs the same thing. Inside a running container, invoke it with
	`docker exec -it <container> ./launch_server.sh <mode>`.

	### Reproduce Benchmark

	This is a separate container invocation from "Launch Server" above — it
	overrides the entrypoint so `run_benchmark.sh` (not `launch_server.sh`)
	drives the container. It starts the baseline server itself, benchmarks it,
	shuts it down, starts the optimized server, benchmarks that, shuts it down,
	and prints the comparison table — nothing to copy in by hand:

	```bash
	docker run \
	--device=/dev/kfd --device=/dev/dri \
	--group-add video --ipc=host --shm-size 16g \
	--security-opt seccomp=unconfined \
	-v ~/.cache/huggingface:/root/.cache/huggingface \
	--entrypoint ./run_benchmark.sh \
	your-org/glm-5.2-vllm:latest both
	```

	That prints the full table in the terminal when it finishes (large model,
	8-GPU node — expect this to take a while: two model loads plus two
	1,000-prompt runs). Pass `baseline` or `optimized` instead of `both` to run
	just one side. The raw per-run JSON (`benchmark_baseline.json`,
	`benchmark_optimized.json`) and each server's startup log
	(`server_baseline.log`, `server_optimized.log`) are left in the container's
	`/workspace` if you want them too:

	```bash
	docker cp <container>:/workspace/benchmark_optimized.json .
	```

	Methodology (`vllm bench serve`):

	- Dataset: `random` (synthetic), range ratio 0.8
	- Input length: 409–3,686 tokens (nominal 2,048, uniform)
	- Output length: 51–460 tokens (nominal 256, uniform)
	- 1,000 measured prompts, 10 warm-up prompts (discarded, not saved)
	- Request rate: unbounded (`inf`) — max concurrency is the only throttle
	- Max concurrency: 512 (best command tuned at this concurrency; override with `-e MAX_CONCURRENCY=...`)

	## Results

	8x AMD Instinct MI325X 256GB, vLLM v0.25.1+rocm723. This is exactly the
	table `run_benchmark.sh both` prints — running it is the only step required
	to reproduce it:

	\| Metric \| Baseline \| Optimized \|
	\|---\|---\|---\|
	\| Request waiting (peak) \| 491 \| 478 \|
	\| Request waiting (avg) \| 169.5 \| 95.7 \|
	\| Output tok/s \| 1019.0 \| 1292.2 \|
	\| TTFT mean (ms) \| 49744 \| 23744 \|
	\| TTFT median (ms) \| 51841 \| 11448 \|
	\| TTFT p99 (ms) \| 104700 \| 77224 \|
	\| TPOT mean (ms) \| 269.7 \| 294.2 \|
	\| TPOT p99 (ms) \| 524.6 \| 744.4 \|
	\| ITL mean (ms) \| 258.1 \| 272.2 \|
	\| ITL p99 (ms) \| 738.9 \| 1079.2 \|
	\| E2E median (ms) \| 110457 \| 85579 \|
	\| KV cache peak \| 100% \| 100% \|
	\| Duration (s) \| 254.8 \| 200.9 \|