Instructions to use canada-quant/GLM-5.3-Flash-W4A16-MTP with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use canada-quant/GLM-5.3-Flash-W4A16-MTP with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="canada-quant/GLM-5.3-Flash-W4A16-MTP")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("canada-quant/GLM-5.3-Flash-W4A16-MTP")
model = AutoModelForMultimodalLM.from_pretrained("canada-quant/GLM-5.3-Flash-W4A16-MTP", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use canada-quant/GLM-5.3-Flash-W4A16-MTP with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "canada-quant/GLM-5.3-Flash-W4A16-MTP"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "canada-quant/GLM-5.3-Flash-W4A16-MTP",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/canada-quant/GLM-5.3-Flash-W4A16-MTP

SGLang

How to use canada-quant/GLM-5.3-Flash-W4A16-MTP with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "canada-quant/GLM-5.3-Flash-W4A16-MTP" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "canada-quant/GLM-5.3-Flash-W4A16-MTP",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "canada-quant/GLM-5.3-Flash-W4A16-MTP" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "canada-quant/GLM-5.3-Flash-W4A16-MTP",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use canada-quant/GLM-5.3-Flash-W4A16-MTP with Docker Model Runner:
```
docker model run hf.co/canada-quant/GLM-5.3-Flash-W4A16-MTP
```

GLM-5.3-Flash — W4A16 (INT4) + BF16 MTP

INT4 weight-only quantization of zai-org/GLM-5.3-Flash with the BF16 MTP draft head kept for speculative decoding. Only the 36,288 routed-expert GEMMs are INT4 (GPTQ, symmetric, group-size 128); attention, router, shared experts, embeddings, the vision tower and the MTP head stay in BF16. Not a new model — all capability comes from the base model.


Size	177.7 GiB (BF16 ≈ 599 GiB, −70%)
Runs on	NVIDIA H100, H200, RTX PRO 6000, and DGX Spark (GB10) — see the hardware table below for validated GPU counts and context per config
Context	full 1,048,576 tokens on 2× DGX Spark and 2× H200; 262K–512K on the 4-GPU x86 configs (KV-memory-bound)
Quality	AIME 2025 0.8833 (n=120) vs 0.9000 for the NVFP4 reference on H100, within noise; GSM8K 0.97; GPQA-Diamond within noise
Throughput	matches or beats the NVFP4 reference on H100 and RTX PRO 6000; 4× H200 TP=4: 195 → 1,954 tok/s from c1 to c256

Full benchmark grids, comparison protocols and research notes: BENCHMARKS.md.

Quick start

# 1. Download (~178 GiB)
huggingface-cli download canada-quant/GLM-5.3-Flash-W4A16-MTP --local-dir /models/glm53-flash-w4a16-mtp

# 2. Serve on 4× H100 / H200 (other hardware: see Serving)
docker run --gpus '"device=0,1,2,3"' --ipc=host --network=host --rm \
  -v /models:/models vllm/vllm-openai:glm53-flash-x86_64-cu130 \
  vllm serve /models/glm53-flash-w4a16-mtp --served-model-name glm53-w4 \
    --tensor-parallel-size 4 --enable-expert-parallel \
    --max-model-len 262144 --max-num-seqs 512 \
    --gpu-memory-utilization 0.92 --no-enable-prefix-caching \
    --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
    --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
    --trust-remote-code --port 8000

# 3. Call it (OpenAI-compatible)
curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "glm53-w4",
  "messages": [{"role": "user", "content": "Prove there are infinitely many primes."}]
}'

Two things that bite. If your config.json predates 2026-09-08, re-download it — older copies fail in vLLM with KeyError: 'layers.0.mlp.gate_up_proj.weight' (weights are unchanged). And always pass --max-num-seqs ≤ 512 — the vLLM default of 1024 exceeds this hybrid linear-attention model's 512 Mamba/KDA-state cache blocks.

Hardware and context

Each row is the largest context serving-validated on that configuration.

Configuration	GPUs	Validated context	Stack
DGX Spark GB10 (SM121), TP=2	2× 128 GB UMA	1M — KV pool 1,360,420 tokens (1.30× a full 1M request)	DFlash2 drafter, fp8 KV
RTX PRO 6000 (SM120), TP=4	4× 96 GB	512K (486K prompts measured)	MTP N=2, fp8 KV
H100 (SM90), TP=4	4× 80 GB	262K (256K prompts measured)	MTP N=2, bf16 KV
H200 (SM90), TP=4	4× 141 GB	262K — KV pool 6.19M tokens (≈23 concurrent 262K requests)	MTP N=2, bf16 KV
H200 (SM90), TP=8	8× 141 GB	262K — KV pool 7.79M tokens	MTP N=2, bf16 KV
H200 (SM90), TP=2	2× 141 GB	1M — KV pool 2.84M tokens (929K-token prompt measured)	MTP N=2, bf16 KV

All configurations use expert parallelism. fp8 KV is not available on Hopper for this NoPE model.

Serving

Architecture	Image
SM90 (H100 / H200)	`vllm/vllm-openai:glm53-flash-x86_64-cu130` (validated). Upstream `vllm/vllm-openai:nightly-x86_64` ≥ 2026-09-08 also boots this checkpoint (vllm-project/vllm#53906); pass `--attention-backend FLASH_ATTN_MLA_SPARSE` there, its default backend faults on ≥131K prompts.
SM120 (RTX PRO 6000)	`cstechdev/vllm:glm53-flash-nope-sm120-cu130-20260826-r1`
SM121 (DGX Spark)	`ghcr.io/canada-quant/vllm-glm53-flash-sm121:v2-w4a16-dflash2e`, built from canada-quant/vllm-glm53-flash-sm121 with the two SM121 serving patches baked in

H100 / H200, TP=4 — the Quick start command is the benchmarked recipe. num_speculative_tokens: 2 is the sweet spot on Hopper (52–55% acceptance; N=5 collapses acceptance to ~30%). Keep prefix caching off — it measured −2…−5% on H200. On 8× H200, two independent TP=4 replicas behind a load balancer beat one TP=8 endpoint by +28–38% aggregate at c128–c512; TP=8 wins single-stream and holds one 7.8M-token pool. On 2× H200 (89.5 GiB weights per GPU) add --tensor-parallel-size 2 --max-num-seqs 128 --max-cudagraph-capture-size 128 for 262K, or --max-model-len 1048576 --max-num-seqs 16 --gpu-memory-utilization 0.95 --max-cudagraph-capture-size 64 --max-num-batched-tokens 4096 for 1M.

RTX PRO 6000, TP=4 — same command with the SM120 image and --max-num-seqs 64 --max-num-batched-tokens 8192 --kv-cache-dtype fp8 --enable-prefix-caching. fp8 KV is required at 262K on 96 GB cards. Keep MTP on at every concurrency here: it adds +70% at c1 and +48% at c32.

2× DGX Spark, TP=2, 1M context — prebuilt image and one-command launcher in canada-quant/vllm-glm53-flash-sm121; the launcher also ships in the drafter repo. Drafter: canada-quant/GLM-5.3-Flash-DFlash2-E (the authors' self-trained DFlash2 drafter, Apache-2.0). Start the worker rank first, wait 25 s, then the head rank.

# on both nodes, rank1 (worker) first, then rank0 (head) 25 s later
MAX_MODEL_LEN=1048576 KV_CACHE_MEM=9663676416 GMU=0.90 EAGER=0 GRAPHS=1 bash launch_dflash2_tp2.sh <rank>

Hard constraints: num_speculative_tokens must be 7 (any other count wedges boot); confirm the boot log shows the mask-embedding load (mask_token_id 154856); keep single prompts ≤ ~310K tokens; stop with docker stop -t 30, never rm -f. Cold boot is 6–10 minutes.

Quality

Benchmark	Hardware	W4A16 (this)	Reference
AIME 2025 — n=120, max thinking, 131,072-token budget	H100	0.8833	NVFP4 0.9000 — within noise (0.42σ)
AIME 2025 — same protocol	RTX PRO 6000	0.8083 raw · 0.8833 with a budget-commit fix	NVFP4 0.9000
AIME 2026 — n=120, max thinking, 131,072-token budget	2× DGX Spark	85.0% (102/120)	EXL3 80.0% (96/120), matched protocol
GSM8K	H100, RTX PRO 6000	0.970–0.975	parity across quants
GPQA-Diamond — n=198 @131K	H100 · RTX PRO 6000	0.8586 · 0.8586	NVFP4 0.8687 · 0.8737 — within noise

On RTX PRO 6000, ≈63% of the raw AIME 2025 deficit is a budget wall (empty-answer rate 11.7–14.2% vs 3.3%) and ≈37% is SM120 kernel numerics; a zero-cost commit hook closes it but is not part of the published recipes. The vision tower is BF16 passthrough and was not covered by the text-only calibration; image smoke tests pass, but no formal vision evaluation has been run. Details in BENCHMARKS.md.

Throughput

Output tok/s, thinking ON, same hardware, flags and prompts within each row.

Hardware	W4A16 (this)	Reference	Read
4× H100, TP=4, MTP N=2, 8192/1024	c1 183.9 · c8 249.9 · c32 1,161	NVFP4 MTP N=2: 181.0 · 251.1 · 770.7 (its best c32 config, N=5: 1,018)	c1/c8 within noise; c32 +14.0% vs NVFP4's best (+22.4% with MTP off on both — the batch edge is the INT4 MoE GEMM)
4× RTX PRO 6000, TP=4, MTP N=2	109.9 · 318.5 · 534.4	NVFP4: 109.3 · 319.5 · 530.8	parity (±0.7%)
4× H200, TP=4, MTP N=2	c1 195 · c8 698 · c32 1,258 · c64 1,529 · c128 1,789 · c256 1,954	—	8× H200 TP=8: 217 → 2,681; two TP=4 replicas: 391 → 3,911 aggregate
2× DGX Spark, TP=2, DFlash2, 8K/256, 1M serve	c1 33.0 · c2 35.6 · c4 59.6 · c6 67.1	EXL3 (matched protocol): 29.9 · 59.6 · 112.7; NVFP4: did not boot (9/9 OOM)	+10.4% c1 and +31–39% long-prefill vs EXL3; EXL3 leads mid-concurrency (c2 +67%, c4 +89%)

Single-stream decode is insensitive to KV length up to ≥486K (RTX PRO 6000); at batch, long-KV decode plateaus at ~2–4 tok/s per stream and long prefills serialize at a ~6–8.5K tok/s aggregate ceiling. All grids, protocols and the H200 extended table: BENCHMARKS.md.

Known issues

config.json (2026-09-08): vLLM matches quantization_config.ignore against its own fused module names, so the ignore list now carries both the HF and vLLM spellings plus re:.*\.layers\.45\..* for the MTP head. Older 765-entry copies fail at load. Weights unchanged.
DFlash2 admission wedge (SM90 research stack only, MTP recipes unaffected): with the DFlash2 drafter at block size 2304, prompts above ~15.5K tokens are never admitted. A fix was validated to 256K prompts; block size 1536 avoids it. Filed as vllm-project/vllm#55800.
Marlin no-split-K path on SM121: deterministic illegal memory access at M=256 when forcing split_k=1; the stock heuristic used in serving is clean. Filed as vllm-project/vllm#56064.

Quantization details

Field	Value
Architecture	`Glm5NextForConditionalGeneration` (`glm5_next`) — 45 decoder layers + MTP layer 45, 288 routed experts (top-8) + 1 shared, KDA + DSA attention, 24-block vision tower
Quantized	36,288 tensors = 42 MoE layers × 288 experts × 3 GEMMs — W4A16, INT4, symmetric, group 128, GPTQ, compressed-tensors `pack-quantized`
Kept in BF16	attention (incl. DSA indexer), dense prefix layers 0–2, shared experts, router, mHC tensors, embeddings, lm_head, norms, vision tower (348 keys), MTP layer 45 (889 keys)
Kept in FP32	`A_log`, `dt_bias`, `e_score_correction_bias`, `hc_*` — verbatim from source
Calibration	256 samples × 4096 tokens, in-distribution chat/code mix, sequential per-layer GPTQ
Built on	8× NVIDIA B300, 2026-08-27

Build gates, all passing: exactly 36,288 packed tensors and nothing quantized outside routed experts; vision key set 348/348 identical to source; MTP layer present; zero dtype drift vs source; no collapsed expert scales. Loads with transformers ≥ 5.16; text generation and image captioning smoke tests pass.

License

MIT, inherited from the base model. Follow the base model's usage terms.

Built, benchmarked and documented with the Digby.ai coding harness, developed by CQL.ca.

Downloads last month: 5,847

Safetensors

Model size

50B params

Tensor type

F32

I32

BF16

Model tree for canada-quant/GLM-5.3-Flash-W4A16-MTP

Base model

zai-org/GLM-5.3-Flash

Quantized

(118)

this model