How to use from the
Use from the
Transformers library
# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="inference-optimization/GLM-5.3-Flash-NVFP4-REAP20")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)
# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("inference-optimization/GLM-5.3-Flash-NVFP4-REAP20")
model = AutoModelForMultimodalLM.from_pretrained("inference-optimization/GLM-5.3-Flash-NVFP4-REAP20", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))
Quick Links

GLM-5.3-Flash-NVFP4-REAP20

RedHatAI/GLM-5.3-Flash-NVFP4 (a NVFP4 quantization of zai-org/GLM-5.3-Flash) with 20% of its MoE experts structurally pruned with REAP (PRUNE method), produced with llmcompressor.

  • Every one of the 42 MoE layers (backbone layers 3-44) keeps its 230 most salient experts of the original 288 (58 pruned per layer, 9660 of 12096 retained)
  • Expert selection is uniform across layers: each layer's 58 lowest-REAP-saliency experts were removed, and the routers and their e_score_correction_bias were sliced to match
  • config.json records the new expert count (n_routed_experts: 230); expert indices are remapped contiguously
  • The quantization is unchanged from the source checkpoint: MoE experts remain NVFP4-quantized and the MTP layer (layers.45) is left untouched, keeping its original 288 FP8-quantized experts (Transformers ignores layers.45 when loading, so this matches the source model's serving behavior)
  • All remaining weights (including attention, vision tower, embeddings and the shared experts) are untouched

REAP scores each expert by the router-weighted mean of its output activation norms over the calibration tokens routed to it, S_j = mean(g_j * ||f_j||_2), using forward computation only (no gradients or recovery training). This model was calibrated on all 2,048 samples of the RazorCal multi-domain calibration corpus; see the REAP expert pruning example for the calibration and pruning scripts.

Usage

from transformers import AutoProcessor, Glm5NextForConditionalGeneration

model = Glm5NextForConditionalGeneration.from_pretrained(
    "inference-optimization/GLM-5.3-Flash-NVFP4-REAP20"
)
processor = AutoProcessor.from_pretrained(
    "inference-optimization/GLM-5.3-Flash-NVFP4-REAP20"
)
Downloads last month
37
Safetensors
Model size
138B params
Tensor type
F32
·
BF16
·
U8
·
F8_E4M3
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for inference-optimization/GLM-5.3-Flash-NVFP4-REAP20

Quantized
(4)
this model

Collection including inference-optimization/GLM-5.3-Flash-NVFP4-REAP20