Model Overview

Qwen/Qwen3-30B-A3B is a sparse Mixture-of-Experts causal language model with 30.5B total parameters and 3.3B active per token. It uses 48 layers, grouped-query attention, and 128 experts with 8 selected per token. The model supports thinking and non-thinking modes, and the standard checkpoint uses BF16 weights.

Environment

Hardware in source session 1 x NVIDIA H100
Software CUDA 13
Base Docker image vllm/vllm-openai:v0.28.0 (CUDA 13.0)
Recipe runtime vLLM 0.28.0
Input Length 1,024 tokens
Output Length 128 tokens requested
vLLM 1
Tensor parallelism 1
Features FlashInfer attention and autotuning 路 FP8 KV cache 路 prefix caching 路 chunked prefill 路 asynchronous scheduling 路 throughput mode
Prompts Measured 200 configured requests; 10 warm-ups shown separately
Precision Default checkpoint dtype (BF16); optimized KV cache FP8 e4m3
Optimized context limit 8,192 tokens
Tested Concurrency 16, 32, 64, 128, 256
Framework vllm bench serve

Optimization Changes

The following commands preserve the reported server settings. recipe.json contains the same argument lists and environment values used by both scripts. Baseline execution removes this recipe's optimized environment overrides.

Baseline

vllm serve Qwen/Qwen3-30B-A3B

Optimized

export VLLM_FLOAT32_MATMUL_PRECISION=medium
export VLLM_USE_FLASHINFER_SAMPLER=1
export VLLM_NO_USAGE_STATS=1
export VLLM_DO_NOT_TRACK=1
vllm serve Qwen/Qwen3-30B-A3B --host 0.0.0.0 --port 8000 --trust-remote-code
--gpu-memory-utilization 0.95 --kv-cache-dtype fp8_e4m3 --block-size 32 --attention-backend FLASHINFER --enable-flashinfer-autotune
--max-num-batched-tokens 65536 --max-num-seqs 512 --watermark 0.02 --enable-chunked-prefill --enable-prefix-caching
--performance-mode throughput --async-scheduling --max-model-len 8192

Docker

Installation

Use a Linux GPU host with 1 x NVIDIA H100, Docker, NVIDIA Container Toolkit, and a driver compatible with the image's CUDA 13 runtime. GPU memory capacity was not recorded in the screenshots. Allow storage for the complete model checkpoint, container image, cache, and logs; active parameter count does not represent the amount of weight storage required.

Unzip into an empty folder, open a terminal there, then build:

docker build -t qwen3-30b-a3b-vllm:0.28.0 .

Start the optimized server (use baseline for the other configuration):

docker run --rm --name qwen3-30b-a3b-vllm-recipe-server \
  --gpus 1 --ipc=host -p 127.0.0.1:8000:8000 \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  qwen3-30b-a3b-vllm:0.28.0 optimized

The server stays in the foreground. Stop it with Ctrl+C before running the automated benchmark, which owns its own baseline and optimized server processes.

Validation

Once model loading has completed:

curl -f http://localhost:8000/health
curl -f http://localhost:8000/v1/models
curl -f http://localhost:8000/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-30B-A3B","prompt":"The capital of France is","max_tokens":16}'

Inspect commands without loading weights or requiring a GPU:

python3 recipe.py serve baseline --dry-run
python3 recipe.py serve optimized --dry-run
python3 recipe.py benchmark both --dry-run

Package validation covered Python syntax, command construction, settings checks, report generation, failure handling, and ZIP contents. Docker builds and actual GPU inference were not run in the authoring environment.

Files in this repository

File Purpose
Dockerfile Pinned vLLM image, scripts, and model configuration
launch_server.sh Launch baseline or optimized server
run_benchmark.sh Automate baseline/optimized benchmark sweep
recipe.py Readiness checks, owned-process cleanup, JSON validation, comparison tables
recipe.json Model-specific arguments, environment, and concurrency levels
README.md Installation, usage, methodology, reference results
LICENSE MIT license for the recipe code
.dockerignore Exclude generated logs/results from image builds

Shell Script

Launch Server

On a prepared Linux environment with vLLM 0.28.0 and Python 3:

bash launch_server.sh optimized
# Or: bash launch_server.sh baseline

The shell scripts call the shared Python helper. Docker uses the same launch script as its entrypoint and defaults to optimized.

Reproduce Benchmark

Run this in a separate container after stopping any manually launched server:

mkdir -p results
docker run --rm --gpus 1 --ipc=host \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  -v "$(pwd)/results:/workspace/results" \
  --entrypoint /workspace/run_benchmark.sh \
  qwen3-30b-a3b-vllm:0.28.0 both

Or on a prepared Linux host:

bash run_benchmark.sh both
# Or select one mode: baseline / optimized

The runner starts the baseline, waits for readiness, sweeps the concurrency caps, stops its server process group, and repeats with the optimized configuration. It refuses to use a server already listening on port 8000. An early server exit, failed benchmark process, or incomplete measured request count stops the run with an error while retaining logs. Ctrl+C and SIGTERM clean up owned processes.

Each invocation creates a new folder under results/ containing:

  • manifest.json: exact workload, server commands, environment overrides, vLLM version.
  • server_baseline.log / server_optimized.log: startup and server output.
  • client_<mode>_c<cap>.log: benchmark client output.
  • benchmark_<mode>_c<cap>.json: raw measured results, including request details.
  • summary.md: baseline/optimized comparison at each concurrency cap.
  • nvidia-smi.txt: GPU and driver information, when available.

Results persist through the Docker bind mount even when the container is removed. The printed and saved comparison uses only these newly measured results.

Workload chosen for this recipe:

  • vllm bench serve, OpenAI completions endpoint, random synthetic prompts.
  • Input: 1,024 requested tokens; output: 128 requested tokens; range ratio: 0.0.
  • Request rate: inf; concurrency caps: 16, 32, 64, 128, 256.
  • 200 measured requests per cap, plus 10 built-in warm-up requests excluded by the client.
  • Base seed: 0; incremented for each cap and matched between baseline and optimized.
  • EOS is ignored explicitly, matching vLLM 0.28's random-dataset behavior; the client requests the configured output length.
  • Temperature and other sampling parameters use server/model defaults. The dataset seed does not pin generation sampling.

Lengths, range ratio, and request rate follow the vLLM 0.28 benchmark defaults. The 200/10 counts and concurrency levels come from the screenshots, while treating 200 as the measured count and using built-in warm-ups are recipe choices. Concurrency is a cap and is also bounded by the total measured request count.

Override workload settings when the original benchmark invocation is available:

CONCURRENCIES=16,32 INPUT_LEN=1024 OUTPUT_LEN=128 \
  NUM_PROMPTS=200 NUM_WARMUPS=10 REQUEST_RATE=inf \
  RANGE_RATIO=0.0 SEED=0 bash run_benchmark.sh both

For Docker pass these with -e, before the image name. RESULTS_DIR selects an alternative output root; SERVER_READY_TIMEOUT defaults to 3,600 seconds. Long initial model downloads may require increasing that timeout.

Results

Historical screenshot reference: selected optimization iteration 5. These numbers were not generated by this package. The screenshot does not label latency aggregation or the exact workload behind the iteration table, so the table cannot be equated to a particular concurrency or to the new mean/p99 report.

Metric Baseline Optimized
Output tok/s 1,640.90 2,027.40
TTFT (ms) 3,583.00 3,353.00
TPOT (ms) 27.86 26.65
ITL (ms) 26.21 25.18
Requests running (avg) 24.60 28.90

Exact reproduction of those values is not established: original benchmark arguments, dependency state, generation settings, and raw result files were not provided. The script runs the reported serving configurations with the explicit workload above; it does not promise the screenshot's throughput or latency. Requests-running averages and KV-cache utilization are not synthesized from client JSON and are not included in the generated comparison.

License, attribution and support

Recipe code: MIT (see LICENSE). Model weights: Apache 2.0, under the upstream model's terms. vLLM and its dependencies retain their respective licenses. No weights are bundled. No separate support contact was provided; include logs and the manifest when reporting a problem with this recipe.

Sources

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 馃檵 Ask for provider support