--- language: - en - zh library_name: transformers pipeline_tag: image-text-to-text tags: - nvfp4 - fp4 - vllm - llm-compressor - compressed-tensors - glm5_next - expert-pruning - reap base_model: - RedHatAI/GLM-5.3-Flash-NVFP4 license: mit license_link: https://choosealicense.com/licenses/mit/ --- # GLM-5.3-Flash-NVFP4-REAP20 [RedHatAI/GLM-5.3-Flash-NVFP4](https://huggingface.co/RedHatAI/GLM-5.3-Flash-NVFP4) (a NVFP4 quantization of [zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash)) with 20% of its MoE experts structurally pruned with **REAP** (PRUNE method), produced with [llmcompressor](https://github.com/vllm-project/llmcompressor). - Every one of the 42 MoE layers (backbone layers 3-44) keeps its **230 most salient experts** of the original 288 (58 pruned per layer, 9660 of 12096 retained) - Expert selection is uniform across layers: each layer's 58 lowest-REAP-saliency experts were removed, and the routers and their `e_score_correction_bias` were sliced to match - `config.json` records the new expert count (`n_routed_experts: 230`); expert indices are remapped contiguously - The quantization is unchanged from the source checkpoint: MoE experts remain NVFP4-quantized and the MTP layer (`layers.45`) is left untouched, keeping its original 288 FP8-quantized experts (Transformers ignores `layers.45` when loading, so this matches the source model's serving behavior) - All remaining weights (including attention, vision tower, embeddings and the shared experts) are untouched REAP scores each expert by the router-weighted mean of its output activation norms over the calibration tokens routed to it, `S_j = mean(g_j * ||f_j||_2)`, using forward computation only (no gradients or recovery training). This model was calibrated on all 2,048 samples of the [RazorCal](https://huggingface.co/datasets/inference-optimization/RazorCal) multi-domain calibration corpus; see the [REAP expert pruning example](https://github.com/vllm-project/llm-compressor/tree/main/examples/reap_expert_pruning) for the calibration and pruning scripts. ## Usage ```python from transformers import AutoProcessor, Glm5NextForConditionalGeneration model = Glm5NextForConditionalGeneration.from_pretrained( "inference-optimization/GLM-5.3-Flash-NVFP4-REAP20" ) processor = AutoProcessor.from_pretrained( "inference-optimization/GLM-5.3-Flash-NVFP4-REAP20" ) ```