Instructions to use RedHatAI/GLM-5.3-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RedHatAI/GLM-5.3-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="RedHatAI/GLM-5.3-NVFP4")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("RedHatAI/GLM-5.3-NVFP4")
model = AutoModelForCausalLM.from_pretrained("RedHatAI/GLM-5.3-NVFP4", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use RedHatAI/GLM-5.3-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RedHatAI/GLM-5.3-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/GLM-5.3-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/RedHatAI/GLM-5.3-NVFP4

SGLang

How to use RedHatAI/GLM-5.3-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RedHatAI/GLM-5.3-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/GLM-5.3-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RedHatAI/GLM-5.3-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/GLM-5.3-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use RedHatAI/GLM-5.3-NVFP4 with Docker Model Runner:
```
docker model run hf.co/RedHatAI/GLM-5.3-NVFP4
```

GLM-5.3-NVFP4 / README.md

dsikka

Update README.md

c8917e4 verified about 1 month ago

preview code

Raw History Blame Contribute Delete

4.19 kB

---
language:
- en
- zh
library_name: transformers
license: other
license_name: glm-5.3
pipeline_tag: text-generation
tags:
- nvfp4
- fp4
- vllm
- llm-compressor
- compressed-tensors
base_model: zai-org/GLM-5.3
---
# GLM-5.3-NVFP4

## Model Overview
- **Model Architecture:** GlmMoeDsaForCausalLM
  - **Input:** Text
  - **Output:** Text
- **Model Optimizations:**
  - **Weight quantization:** FP4
  - **Activation quantization:** FP4
- **Format:** compressed-tensors
- **Base model:** [zai-org/GLM-5.3](https://huggingface.co/zai-org/GLM-5.3)

This is an NVFP4 checkpoint derived from [zai-org/GLM-5.3](https://huggingface.co/zai-org/GLM-5.3).
It is optimized for Blackwell inference with NVFP4 weights and activations while retaining higher
precision where needed for model quality.

### Model Optimizations

This model uses NVFP4 FP4 (E2M1) weights and activations and two-level scaling. Layers outside of routed experts are left unquantized. 
The checkpoint uses [LLM Compressor](https://github.com/vllm-project/llm-compressor)'s
compressed-tensors NVFP4 format.

## Deployment

### vLLM Serving

```
vllm serve RedHatAI/GLM-5.3-NVFP4 \
  --served-model-name GLM-5.3-NVFP4 \
  --tensor-parallel-size 4 \
  --kv-cache-dtype fp8_e4m3 \
  --reasoning-parser glm45 \
  --chat-template-content-format string \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
  --trust-remote-code
```


For optimal peformance, consider using the DSpark draft model [RedHatAI/GLM-5.3-speculator.dspark](https://huggingface.co/RedHatAI/GLM-5.3-speculator.dspark) for speculative decoding, shown below.

```
vllm serve RedHatAI/GLM-5.3-NVFP4 \
  --served-model-name GLM-5.3-NVFP4 \
  --tensor-parallel-size 4 \
  --kv-cache-dtype fp8_e4m3 \
  --reasoning-parser glm45 \
  --chat-template-content-format string \
  --speculative-config '{"model":"RedHatAI/GLM-5.3-speculator.dspark","num_speculative_tokens":8,"method":"dspark"}' \
  --trust-remote-code
```

## Creation

The source model was compressed using an LLM-Compressor with experimental layerwise decompression/compression along with an NVFP4 scheme using `perfectblend` calibration samples. 

## Evaluation

This model was evaluated on GSM8K Platinum, IFEval, MATH-500, AIME 2025, and GPQA Diamond using
[lighteval](https://github.com/huggingface/lighteval). Evaluation was run on a vLLM server hosted on 4xB200. 
Generation used temperature 1.0, the `glm45` reasoning parser, FP8 E4M3 KV cache, and the same
serving stack for the NVFP4 and local FP8 baseline. GSM8K, IFEval, MATH-500, and GPQA used three
seeds; AIME used eight seeds. Recovery is relative to the [base model](https://huggingface.co/zai-org/GLM-5.3) run in the same evaluation harness.

### Accuracy

<table>
  <thead>
    <tr>
      <th>Category</th>
      <th>Benchmark</th>
      <th>RedHatAI/GLM-5.3-NVFP4</th>
      <th>zai-org/GLM-5.3 (FP8)</th>
      <th>Recovery</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><b>Instruction Following</b></td>
      <td>IFEval (prompt-level strict)</td>
      <td>85.70%</td>
      <td>89.77%</td>
      <td>95.5%</td>
    </tr>
    <tr>
      <td rowspan="4"><b>Reasoning</b></td>
      <td>GSM8K Platinum (strict-match)</td>
      <td>96.69%</td>
      <td>97.52%</td>
      <td>99.1%</td>
    </tr>
    <tr>
      <td>MATH-500 (pass@1)</td>
      <td>96.27%</td>
      <td>95.07%</td>
      <td>101.3%</td>
    </tr>
    <tr>
      <td>AIME 2025 (pass@1)</td>
      <td>89.58%</td>
      <td>92.50%</td>
      <td>96.8%</td>
    </tr>
    <tr>
      <td>GPQA Diamond (pass@1)</td>
      <td>90.40%</td>
      <td>92.42%</td>
      <td>97.8%</td>
    </tr>
  </tbody>
</table>
## Performance Evaluation

Performance evaluation for RedHatAI/GLM-5.3-NVFP4 with speculative decoding. 

In the graph, interactivity is the median per-request output-token rate, aggregate throughput is total completed output tokens divided by measurement duration, and each label is completed requests divided by measurement duration. The `math_reasoning` dataset was used for this benchmarking.


![glm53_math_reasoning](https://cdn-uploads.huggingface.co/production/uploads/650a157669739cd310382f88/yDDJe_95-w5NWjCaFxumL.png)

	---
	language:
	- en
	- zh
	library_name: transformers
	license: other
	license_name: glm-5.3
	pipeline_tag: text-generation
	tags:
	- nvfp4
	- fp4
	- vllm
	- llm-compressor
	- compressed-tensors
	base_model: zai-org/GLM-5.3
	---

	# GLM-5.3-NVFP4

	## Model Overview
	- Model Architecture: GlmMoeDsaForCausalLM
	- Input: Text
	- Output: Text
	- Model Optimizations:
	- Weight quantization: FP4
	- Activation quantization: FP4
	- Format: compressed-tensors
	- Base model: [zai-org/GLM-5.3](https://huggingface.co/zai-org/GLM-5.3)

	This is an NVFP4 checkpoint derived from [zai-org/GLM-5.3](https://huggingface.co/zai-org/GLM-5.3).
	It is optimized for Blackwell inference with NVFP4 weights and activations while retaining higher
	precision where needed for model quality.

	### Model Optimizations

	This model uses NVFP4 FP4 (E2M1) weights and activations and two-level scaling. Layers outside of routed experts are left unquantized.
	The checkpoint uses [LLM Compressor](https://github.com/vllm-project/llm-compressor)'s
	compressed-tensors NVFP4 format.

	## Deployment

	### vLLM Serving

	```
	vllm serve RedHatAI/GLM-5.3-NVFP4 \
	--served-model-name GLM-5.3-NVFP4 \
	--tensor-parallel-size 4 \
	--kv-cache-dtype fp8_e4m3 \
	--reasoning-parser glm45 \
	--chat-template-content-format string \
	--speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
	--trust-remote-code
	```


	For optimal peformance, consider using the DSpark draft model [RedHatAI/GLM-5.3-speculator.dspark](https://huggingface.co/RedHatAI/GLM-5.3-speculator.dspark) for speculative decoding, shown below.

	```
	vllm serve RedHatAI/GLM-5.3-NVFP4 \
	--served-model-name GLM-5.3-NVFP4 \
	--tensor-parallel-size 4 \
	--kv-cache-dtype fp8_e4m3 \
	--reasoning-parser glm45 \
	--chat-template-content-format string \
	--speculative-config '{"model":"RedHatAI/GLM-5.3-speculator.dspark","num_speculative_tokens":8,"method":"dspark"}' \
	--trust-remote-code
	```

	## Creation

	The source model was compressed using an LLM-Compressor with experimental layerwise decompression/compression along with an NVFP4 scheme using `perfectblend` calibration samples.

	## Evaluation

	This model was evaluated on GSM8K Platinum, IFEval, MATH-500, AIME 2025, and GPQA Diamond using
	[lighteval](https://github.com/huggingface/lighteval). Evaluation was run on a vLLM server hosted on 4xB200.
	Generation used temperature 1.0, the `glm45` reasoning parser, FP8 E4M3 KV cache, and the same
	serving stack for the NVFP4 and local FP8 baseline. GSM8K, IFEval, MATH-500, and GPQA used three
	seeds; AIME used eight seeds. Recovery is relative to the [base model](https://huggingface.co/zai-org/GLM-5.3) run in the same evaluation harness.

	### Accuracy

	<table>
	<thead>
	<tr>
	<th>Category</th>
	<th>Benchmark</th>
	<th>RedHatAI/GLM-5.3-NVFP4</th>
	<th>zai-org/GLM-5.3 (FP8)</th>
	<th>Recovery</th>
	</tr>
	</thead>
	<tbody>
	<tr>
	<td><b>Instruction Following</b></td>
	<td>IFEval (prompt-level strict)</td>
	<td>85.70%</td>
	<td>89.77%</td>
	<td>95.5%</td>
	</tr>
	<tr>
	<td rowspan="4"><b>Reasoning</b></td>
	<td>GSM8K Platinum (strict-match)</td>
	<td>96.69%</td>
	<td>97.52%</td>
	<td>99.1%</td>
	</tr>
	<tr>
	<td>MATH-500 (pass@1)</td>
	<td>96.27%</td>
	<td>95.07%</td>
	<td>101.3%</td>
	</tr>
	<tr>
	<td>AIME 2025 (pass@1)</td>
	<td>89.58%</td>
	<td>92.50%</td>
	<td>96.8%</td>
	</tr>
	<tr>
	<td>GPQA Diamond (pass@1)</td>
	<td>90.40%</td>
	<td>92.42%</td>
	<td>97.8%</td>
	</tr>
	</tbody>
	</table>

	## Performance Evaluation

	Performance evaluation for RedHatAI/GLM-5.3-NVFP4 with speculative decoding.

	In the graph, interactivity is the median per-request output-token rate, aggregate throughput is total completed output tokens divided by measurement duration, and each label is completed requests divided by measurement duration. The `math_reasoning` dataset was used for this benchmarking.


	![glm53_math_reasoning](https://cdn-uploads.huggingface.co/production/uploads/650a157669739cd310382f88/yDDJe_95-w5NWjCaFxumL.png)