Instructions to use RedHatAI/GLM-5.3-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RedHatAI/GLM-5.3-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="RedHatAI/GLM-5.3-NVFP4")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("RedHatAI/GLM-5.3-NVFP4")
model = AutoModelForCausalLM.from_pretrained("RedHatAI/GLM-5.3-NVFP4", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use RedHatAI/GLM-5.3-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RedHatAI/GLM-5.3-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/GLM-5.3-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/RedHatAI/GLM-5.3-NVFP4

SGLang

How to use RedHatAI/GLM-5.3-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RedHatAI/GLM-5.3-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/GLM-5.3-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RedHatAI/GLM-5.3-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/GLM-5.3-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use RedHatAI/GLM-5.3-NVFP4 with Docker Model Runner:
```
docker model run hf.co/RedHatAI/GLM-5.3-NVFP4
```

wuroderi commited on Sep 21

Commit

2bcb7f2

verified ·

1 Parent(s): f054693

Upload folder using huggingface_hub

Browse files

Files changed (1) hide show

README.md +7 -18

README.md CHANGED Viewed

@@ -22,9 +22,9 @@ base_model: zai-org/GLM-5.3
   - **Input:** Text
   - **Output:** Text
 - **Model Optimizations:**
-  - **Weight quantization:** NVFP4, group size 16
-  - **Activation quantization:** NVFP4, group size 16
-- **Format:** compressed-tensors NVFP4
 - **Base model:** [zai-org/GLM-5.3](https://huggingface.co/zai-org/GLM-5.3)
 This is an NVFP4 checkpoint derived from [zai-org/GLM-5.3](https://huggingface.co/zai-org/GLM-5.3).
@@ -33,7 +33,7 @@ precision where needed for model quality.
 ### Model Optimizations
-This model uses NVFP4 FP4 (E2M1) weights and activations with group size 16 and two-level scaling. Sensitive layers are left unquantized.
 The checkpoint uses [LLM Compressor](https://github.com/vllm-project/llm-compressor)'s
 compressed-tensors NVFP4 format.
@@ -55,16 +55,15 @@ vllm serve /data/Roderick-Wu/GLM-5.3-NVFP4-Mixed-Layerwise \
 ## Creation
-The source model was compressed using an LLM-Compressor with experimental layerwise decompression/compression along with an NVFP4 scheme using `perfectblend` calibration samples. The exported checkpoint
-was then postprocessed into this mixed-precision form and saved in compressed-tensors format.
 ## Evaluation
 This model was evaluated on GSM8K Platinum, IFEval, MATH-500, AIME 2025, and GPQA Diamond using
-[lighteval](https://github.com/huggingface/lighteval) and an OpenAI-compatible vLLM server.
 Generation used temperature 1.0, the `glm45` reasoning parser, FP8 E4M3 KV cache, and the same
 serving stack for the NVFP4 and local FP8 baseline. GSM8K, IFEval, MATH-500, and GPQA used three
-seeds; AIME used eight seeds. Recovery is relative to the local FP8 baseline.
 ### Accuracy
@@ -114,16 +113,6 @@ seeds; AIME used eight seeds. Recovery is relative to the local FP8 baseline.
   </tbody>
 </table>
-The complete comparison is:
-| Benchmark | GLM-5.3-NVFP4 | Our FP8 baseline | Recovery |
-|---|---:|---:|---:|
-| GSM8K Platinum | 96.69% | 97.52% | 99.1% |
-| IFEval | 85.70% | 89.77% | 95.5% |
-| MATH-500 | 96.27% | 95.07% | 101.3% |
-| AIME 2025 | 89.58% | 92.50% | 96.8% |
-| GPQA Diamond | 90.40% | 92.42% | 97.8% |
 Per-seed NVFP4 results:
 - GSM8K: 96.36%, 96.86%, 96.86%

   - **Input:** Text
   - **Output:** Text
 - **Model Optimizations:**
+  - **Weight quantization:** NVFP4
+  - **Activation quantization:** NVFP4
+- **Format:** compressed-tensors
 - **Base model:** [zai-org/GLM-5.3](https://huggingface.co/zai-org/GLM-5.3)
 This is an NVFP4 checkpoint derived from [zai-org/GLM-5.3](https://huggingface.co/zai-org/GLM-5.3).
 ### Model Optimizations
+This model uses NVFP4 FP4 (E2M1) weights and activations and two-level scaling. Layers outside of routed experts are left unquantized.
 The checkpoint uses [LLM Compressor](https://github.com/vllm-project/llm-compressor)'s
 compressed-tensors NVFP4 format.
 ## Creation
+The source model was compressed using an LLM-Compressor with experimental layerwise decompression/compression along with an NVFP4 scheme using `perfectblend` calibration samples.
 ## Evaluation
 This model was evaluated on GSM8K Platinum, IFEval, MATH-500, AIME 2025, and GPQA Diamond using
+[lighteval](https://github.com/huggingface/lighteval). Evaluation was run on a vLLM server hosted on 4xB200.
 Generation used temperature 1.0, the `glm45` reasoning parser, FP8 E4M3 KV cache, and the same
 serving stack for the NVFP4 and local FP8 baseline. GSM8K, IFEval, MATH-500, and GPQA used three
+seeds; AIME used eight seeds. Recovery is relative to the [base model](https://huggingface.co/zai-org/GLM-5.3) run in the same evaluation harness.
 ### Accuracy
   </tbody>
 </table>
 Per-seed NVFP4 results:
 - GSM8K: 96.36%, 96.86%, 96.86%