Model Overview
Qwen/Qwen3-30B-A3B is a sparse Mixture-of-Experts causal language model with 30.5B total parameters and 3.3B active per token. It uses 48 layers, grouped-query attention, and 128 experts with 8 selected per token. The model supports thinking and non-thinking modes, and the standard checkpoint uses BF16 weights.
Environment
| Hardware in source session | 1 x NVIDIA H100 |
| Software | CUDA 13 |
| Base Docker image | vllm/vllm-openai:v0.28.0 (CUDA 13.0) |
| Recipe runtime | vLLM 0.28.0 |
| Input Length | 1,024 tokens |
| Output Length | 128 tokens requested |
| vLLM | 1 |
| Tensor parallelism | 1 |
| Features | FlashInfer attention and autotuning 路 FP8 KV cache 路 prefix caching 路 chunked prefill 路 asynchronous scheduling 路 throughput mode |
| Prompts Measured | 200 configured requests; 10 warm-ups shown separately |
| Precision | Default checkpoint dtype (BF16); optimized KV cache FP8 e4m3 |
| Optimized context limit | 8,192 tokens |
| Tested Concurrency | 16, 32, 64, 128, 256 |
| Framework | vllm bench serve |
Optimization Changes
The following commands preserve the reported server settings. recipe.json
contains the same argument lists and environment values used by both scripts.
Baseline execution removes this recipe's optimized environment overrides.
Baseline
vllm serve Qwen/Qwen3-30B-A3B
Optimized
export VLLM_FLOAT32_MATMUL_PRECISION=medium
export VLLM_USE_FLASHINFER_SAMPLER=1
export VLLM_NO_USAGE_STATS=1
export VLLM_DO_NOT_TRACK=1
vllm serve Qwen/Qwen3-30B-A3B --host 0.0.0.0 --port 8000 --trust-remote-code
--gpu-memory-utilization 0.95 --kv-cache-dtype fp8_e4m3 --block-size 32 --attention-backend FLASHINFER --enable-flashinfer-autotune
--max-num-batched-tokens 65536 --max-num-seqs 512 --watermark 0.02 --enable-chunked-prefill --enable-prefix-caching
--performance-mode throughput --async-scheduling --max-model-len 8192
Docker
Installation
Use a Linux GPU host with 1 x NVIDIA H100, Docker, NVIDIA Container Toolkit, and a driver compatible with the image's CUDA 13 runtime. GPU memory capacity was not recorded in the screenshots. Allow storage for the complete model checkpoint, container image, cache, and logs; active parameter count does not represent the amount of weight storage required.
Unzip into an empty folder, open a terminal there, then build:
docker build -t qwen3-30b-a3b-vllm:0.28.0 .
Start the optimized server (use baseline for the other configuration):
docker run --rm --name qwen3-30b-a3b-vllm-recipe-server \
--gpus 1 --ipc=host -p 127.0.0.1:8000:8000 \
-v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
qwen3-30b-a3b-vllm:0.28.0 optimized
The server stays in the foreground. Stop it with Ctrl+C before running the automated benchmark, which owns its own baseline and optimized server processes.
Validation
Once model loading has completed:
curl -f http://localhost:8000/health
curl -f http://localhost:8000/v1/models
curl -f http://localhost:8000/v1/completions \
-H 'Content-Type: application/json' \
-d '{"model":"Qwen/Qwen3-30B-A3B","prompt":"The capital of France is","max_tokens":16}'
Inspect commands without loading weights or requiring a GPU:
python3 recipe.py serve baseline --dry-run
python3 recipe.py serve optimized --dry-run
python3 recipe.py benchmark both --dry-run
Package validation covered Python syntax, command construction, settings checks, report generation, failure handling, and ZIP contents. Docker builds and actual GPU inference were not run in the authoring environment.
Files in this repository
| File | Purpose |
|---|---|
Dockerfile |
Pinned vLLM image, scripts, and model configuration |
launch_server.sh |
Launch baseline or optimized server |
run_benchmark.sh |
Automate baseline/optimized benchmark sweep |
recipe.py |
Readiness checks, owned-process cleanup, JSON validation, comparison tables |
recipe.json |
Model-specific arguments, environment, and concurrency levels |
README.md |
Installation, usage, methodology, reference results |
LICENSE |
MIT license for the recipe code |
.dockerignore |
Exclude generated logs/results from image builds |
Shell Script
Launch Server
On a prepared Linux environment with vLLM 0.28.0 and Python 3:
bash launch_server.sh optimized
# Or: bash launch_server.sh baseline
The shell scripts call the shared Python helper. Docker uses the same launch
script as its entrypoint and defaults to optimized.
Reproduce Benchmark
Run this in a separate container after stopping any manually launched server:
mkdir -p results
docker run --rm --gpus 1 --ipc=host \
-v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
-v "$(pwd)/results:/workspace/results" \
--entrypoint /workspace/run_benchmark.sh \
qwen3-30b-a3b-vllm:0.28.0 both
Or on a prepared Linux host:
bash run_benchmark.sh both
# Or select one mode: baseline / optimized
The runner starts the baseline, waits for readiness, sweeps the concurrency caps, stops its server process group, and repeats with the optimized configuration. It refuses to use a server already listening on port 8000. An early server exit, failed benchmark process, or incomplete measured request count stops the run with an error while retaining logs. Ctrl+C and SIGTERM clean up owned processes.
Each invocation creates a new folder under results/ containing:
manifest.json: exact workload, server commands, environment overrides, vLLM version.server_baseline.log/server_optimized.log: startup and server output.client_<mode>_c<cap>.log: benchmark client output.benchmark_<mode>_c<cap>.json: raw measured results, including request details.summary.md: baseline/optimized comparison at each concurrency cap.nvidia-smi.txt: GPU and driver information, when available.
Results persist through the Docker bind mount even when the container is removed. The printed and saved comparison uses only these newly measured results.
Workload chosen for this recipe:
vllm bench serve, OpenAI completions endpoint, random synthetic prompts.- Input: 1,024 requested tokens; output: 128 requested tokens; range ratio: 0.0.
- Request rate:
inf; concurrency caps: 16, 32, 64, 128, 256. - 200 measured requests per cap, plus 10 built-in warm-up requests excluded by the client.
- Base seed: 0; incremented for each cap and matched between baseline and optimized.
- EOS is ignored explicitly, matching vLLM 0.28's random-dataset behavior; the client requests the configured output length.
- Temperature and other sampling parameters use server/model defaults. The dataset seed does not pin generation sampling.
Lengths, range ratio, and request rate follow the vLLM 0.28 benchmark defaults. The 200/10 counts and concurrency levels come from the screenshots, while treating 200 as the measured count and using built-in warm-ups are recipe choices. Concurrency is a cap and is also bounded by the total measured request count.
Override workload settings when the original benchmark invocation is available:
CONCURRENCIES=16,32 INPUT_LEN=1024 OUTPUT_LEN=128 \
NUM_PROMPTS=200 NUM_WARMUPS=10 REQUEST_RATE=inf \
RANGE_RATIO=0.0 SEED=0 bash run_benchmark.sh both
For Docker pass these with -e, before the image name. RESULTS_DIR selects
an alternative output root; SERVER_READY_TIMEOUT defaults to 3,600 seconds.
Long initial model downloads may require increasing that timeout.
Results
Historical screenshot reference: selected optimization iteration 5. These numbers were not generated by this package. The screenshot does not label latency aggregation or the exact workload behind the iteration table, so the table cannot be equated to a particular concurrency or to the new mean/p99 report.
| Metric | Baseline | Optimized |
|---|---|---|
| Output tok/s | 1,640.90 | 2,027.40 |
| TTFT (ms) | 3,583.00 | 3,353.00 |
| TPOT (ms) | 27.86 | 26.65 |
| ITL (ms) | 26.21 | 25.18 |
| Requests running (avg) | 24.60 | 28.90 |
Exact reproduction of those values is not established: original benchmark arguments, dependency state, generation settings, and raw result files were not provided. The script runs the reported serving configurations with the explicit workload above; it does not promise the screenshot's throughput or latency. Requests-running averages and KV-cache utilization are not synthesized from client JSON and are not included in the generated comparison.
License, attribution and support
Recipe code: MIT (see LICENSE). Model weights: Apache 2.0, under the
upstream model's terms.
vLLM and its dependencies retain their respective licenses. No weights are bundled.
No separate support contact was provided; include logs and the manifest when
reporting a problem with this recipe.