GLM-5.3-Flash-NVFP4-REAP20

RedHatAI/GLM-5.3-Flash-NVFP4 (a NVFP4 quantization of zai-org/GLM-5.3-Flash) with 20% of its MoE experts structurally pruned with REAP (PRUNE method), produced with llmcompressor.

  • Every one of the 42 MoE layers (backbone layers 3-44) keeps its 230 most salient experts of the original 288 (58 pruned per layer, 9660 of 12096 retained)
  • Expert selection is uniform across layers: each layer's 58 lowest-REAP-saliency experts were removed, and the routers and their e_score_correction_bias were sliced to match
  • config.json records the new expert count (n_routed_experts: 230); expert indices are remapped contiguously
  • The quantization is unchanged from the source checkpoint: MoE experts remain NVFP4-quantized and the MTP layer (layers.45) is left untouched, keeping its original 288 FP8-quantized experts (Transformers ignores layers.45 when loading, so this matches the source model's serving behavior)
  • All remaining weights (including attention, vision tower, embeddings and the shared experts) are untouched

REAP scores each expert by the router-weighted mean of its output activation norms over the calibration tokens routed to it, S_j = mean(g_j * ||f_j||_2), using forward computation only (no gradients or recovery training). This model was calibrated on all 2,048 samples of the RazorCal multi-domain calibration corpus; see the REAP expert pruning example for the calibration and pruning scripts.

Usage

from transformers import AutoProcessor, Glm5NextForConditionalGeneration

model = Glm5NextForConditionalGeneration.from_pretrained(
    "inference-optimization/GLM-5.3-Flash-NVFP4-REAP20"
)
processor = AutoProcessor.from_pretrained(
    "inference-optimization/GLM-5.3-Flash-NVFP4-REAP20"
)
Downloads last month
37
Safetensors
Model size
138B params
Tensor type
F32
·
BF16
·
U8
·
F8_E4M3
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for inference-optimization/GLM-5.3-Flash-NVFP4-REAP20

Quantized
(4)
this model

Collection including inference-optimization/GLM-5.3-Flash-NVFP4-REAP20