Instructions to use RedHatAI/GLM-5.3-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RedHatAI/GLM-5.3-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="RedHatAI/GLM-5.3-NVFP4")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("RedHatAI/GLM-5.3-NVFP4")
model = AutoModelForCausalLM.from_pretrained("RedHatAI/GLM-5.3-NVFP4", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use RedHatAI/GLM-5.3-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RedHatAI/GLM-5.3-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/GLM-5.3-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/RedHatAI/GLM-5.3-NVFP4

SGLang

How to use RedHatAI/GLM-5.3-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RedHatAI/GLM-5.3-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/GLM-5.3-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RedHatAI/GLM-5.3-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/GLM-5.3-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use RedHatAI/GLM-5.3-NVFP4 with Docker Model Runner:
```
docker model run hf.co/RedHatAI/GLM-5.3-NVFP4
```

GLM-5.3-NVFP4

Model Overview

Model Architecture: GlmMoeDsaForCausalLM
- Input: Text
- Output: Text
Model Optimizations:
- Weight quantization: FP4
- Activation quantization: FP4
Format: compressed-tensors
Base model: zai-org/GLM-5.3

This is an NVFP4 checkpoint derived from zai-org/GLM-5.3. It is optimized for Blackwell inference with NVFP4 weights and activations while retaining higher precision where needed for model quality.

Model Optimizations

This model uses NVFP4 FP4 (E2M1) weights and activations and two-level scaling. Layers outside of routed experts are left unquantized. The checkpoint uses LLM Compressor's compressed-tensors NVFP4 format.

Deployment

vLLM Serving

vllm serve RedHatAI/GLM-5.3-NVFP4 \
  --served-model-name GLM-5.3-NVFP4 \
  --tensor-parallel-size 4 \
  --kv-cache-dtype fp8_e4m3 \
  --reasoning-parser glm45 \
  --chat-template-content-format string \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
  --trust-remote-code

For optimal peformance, consider using the DSpark draft model RedHatAI/GLM-5.3-speculator.dspark for speculative decoding, shown below.

vllm serve RedHatAI/GLM-5.3-NVFP4 \
  --served-model-name GLM-5.3-NVFP4 \
  --tensor-parallel-size 4 \
  --kv-cache-dtype fp8_e4m3 \
  --reasoning-parser glm45 \
  --chat-template-content-format string \
  --speculative-config '{"model":"RedHatAI/GLM-5.3-speculator.dspark","num_speculative_tokens":8,"method":"dspark"}' \
  --trust-remote-code

Creation

The source model was compressed using an LLM-Compressor with experimental layerwise decompression/compression along with an NVFP4 scheme using perfectblend calibration samples.

Evaluation

This model was evaluated on GSM8K Platinum, IFEval, MATH-500, AIME 2025, and GPQA Diamond using lighteval. Evaluation was run on a vLLM server hosted on 4xB200. Generation used temperature 1.0, the glm45 reasoning parser, FP8 E4M3 KV cache, and the same serving stack for the NVFP4 and local FP8 baseline. GSM8K, IFEval, MATH-500, and GPQA used three seeds; AIME used eight seeds. Recovery is relative to the base model run in the same evaluation harness.

Accuracy

Category	Benchmark	RedHatAI/GLM-5.3-NVFP4	zai-org/GLM-5.3 (FP8)	Recovery
Instruction Following	IFEval (prompt-level strict)	85.70%	89.77%	95.5%
Reasoning	GSM8K Platinum (strict-match)	96.69%	97.52%	99.1%
	MATH-500 (pass@1)	96.27%	95.07%	101.3%
	AIME 2025 (pass@1)	89.58%	92.50%	96.8%
	GPQA Diamond (pass@1)	90.40%	92.42%	97.8%

Performance Evaluation

Performance evaluation for RedHatAI/GLM-5.3-NVFP4 with speculative decoding.

In the graph, interactivity is the median per-request output-token rate, aggregate throughput is total completed output tokens divided by measurement duration, and each label is completed requests divided by measurement duration. The math_reasoning dataset was used for this benchmarking.

Downloads last month: 1,973

Safetensors

Model size

753B params

Tensor type

BF16

F32

Model tree for RedHatAI/GLM-5.3-NVFP4

Base model

zai-org/GLM-5.3

Quantized

(61)

this model