Instructions to use zai-org/GLM-5.3-Flash with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use zai-org/GLM-5.3-Flash with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="zai-org/GLM-5.3-Flash")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("zai-org/GLM-5.3-Flash")
model = AutoModelForMultimodalLM.from_pretrained("zai-org/GLM-5.3-Flash", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use zai-org/GLM-5.3-Flash with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "zai-org/GLM-5.3-Flash"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zai-org/GLM-5.3-Flash",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/zai-org/GLM-5.3-Flash

SGLang

How to use zai-org/GLM-5.3-Flash with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "zai-org/GLM-5.3-Flash" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zai-org/GLM-5.3-Flash",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "zai-org/GLM-5.3-Flash" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zai-org/GLM-5.3-Flash",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use zai-org/GLM-5.3-Flash with Docker Model Runner:
```
docker model run hf.co/zai-org/GLM-5.3-Flash
```

vllm crash: missing module for Glm5NextTextLinearAttention

#26

by luonist - opened Aug 27

Discussion

luonist

Aug 27

I installed vllm nightly:

uv venv
source .venv/bin/activate
uv pip install -U vllm --pre
--extra-index-url https://wheels.vllm.ai/nightly/cu130
--extra-index-url https://download.pytorch.org/whl/cu130
--index-strategy unsafe-best-match

and launched the model, following the recipe (for B200 cards) on vllm website:

export VLLM_ENGINE_READY_TIMEOUT_S=3600
vllm serve zai-org/GLM-5.3-Flash
--tensor-parallel-size 4
--max-model-len 262144
--kv-cache-dtype fp8
--tool-call-parser glm47
--enable-auto-tool-choice
--reasoning-parser glm45

The resulting versions of vllm and transformers are:
transformers==5.16.1
vllm==0.28.1rc1.dev7+g4a6a3272e

I got this error:
ValueError: There is no module or parameter named 'model.language_model.layers.0.self_attn.k_conv1d' in TransformersMultiModalMoEForCausalLM. The available parameters belonging to model.language_model.layers.0.self_attn (Glm5NextTextLinearAttention) are: {'model.language_model.layers.0.self_attn.v_proj.weight', 'model.language_model.layers.0.self_attn.forget_gate.A_log', 'model.language_model.layers.0.self_attn.forget_gate.f_b_proj.weight_scale_inv', 'model.language_model.layers.0.self_attn.o_norm.weight', 'model.language_model.layers.0.self_attn.b_proj.weight', 'model.language_model.layers.0.self_attn.conv1d.weight', 'model.language_model.layers.0.self_attn.v_proj.weight_scale_inv', 'model.language_model.layers.0.self_attn.forget_gate.f_b_proj.weight', 'model.language_model.layers.0.self_attn.forget_gate.dt_bias', 'model.language_model.layers.0.self_attn.g_a_proj.weight_scale_inv', 'model.language_model.layers.0.self_attn.q_proj.weight_scale_inv', 'model.language_model.layers.0.self_attn.o_proj.weight_scale_inv', 'model.language_model.layers.0.self_attn.forget_gate.f_a_proj.weight', 'model.language_model.layers.0.self_attn.k_proj.weight_scale_inv', 'model.language_model.layers.0.self_attn.k_proj.weight', 'model.language_model.layers.0.self_attn.b_proj.weight_scale_inv', 'model.language_model.layers.0.self_attn.g_a_proj.weight', 'model.language_model.layers.0.self_attn.forget_gate.f_a_proj.weight_scale_inv', 'model.language_model.layers.0.self_attn.q_proj.weight', 'model.language_model.layers.0.self_attn.g_b_proj.weight', 'model.language_model.layers.0.self_attn.g_b_proj.weight_scale_inv', 'model.language_model.layers.0.self_attn.o_proj.weight'}

It looks vllm still lacks support to the model architecture.

ZHANGYUXUAN-zR

Z.ai org Aug 27

you should use this
docker pull vllm/vllm-openai:glm53-flash

luonist

Aug 27

Trying the docker image, I got:
FileNotFoundError: [Errno 2] No such file or directory: 'zai-org/GLM-5.3-Flash/processor_config.json'

ZHANGYUXUAN-zR

Z.ai org Aug 27

this may be you not download the full model,as config is here
https://huggingface.co/zai-org/GLM-5.3-Flash/blob/main/processor_config.json

luonist

Aug 27

Yep, the file is there; however, vllm fails to resolve the file path. Passing the local path to the downloaded files, instead of the HF model name solved the issue.

ZHANGYUXUAN-zR changed discussion status to closed Sep 7

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment