kylesayrs's picture
GLM-5.3-Flash-NVFP4 pruned to 20% expert sparsity with REAP (RazorCal calibration)
ff38bee verified
|
Raw History Blame Contribute Delete
2.41 kB
metadata
language:
  - en
  - zh
library_name: transformers
pipeline_tag: image-text-to-text
tags:
  - nvfp4
  - fp4
  - vllm
  - llm-compressor
  - compressed-tensors
  - glm5_next
  - expert-pruning
  - reap
base_model:
  - RedHatAI/GLM-5.3-Flash-NVFP4
license: mit
license_link: https://choosealicense.com/licenses/mit/

GLM-5.3-Flash-NVFP4-REAP20

RedHatAI/GLM-5.3-Flash-NVFP4 (a NVFP4 quantization of zai-org/GLM-5.3-Flash) with 20% of its MoE experts structurally pruned with REAP (PRUNE method), produced with llmcompressor.

  • Every one of the 42 MoE layers (backbone layers 3-44) keeps its 230 most salient experts of the original 288 (58 pruned per layer, 9660 of 12096 retained)
  • Expert selection is uniform across layers: each layer's 58 lowest-REAP-saliency experts were removed, and the routers and their e_score_correction_bias were sliced to match
  • config.json records the new expert count (n_routed_experts: 230); expert indices are remapped contiguously
  • The quantization is unchanged from the source checkpoint: MoE experts remain NVFP4-quantized and the MTP layer (layers.45) is left untouched, keeping its original 288 FP8-quantized experts (Transformers ignores layers.45 when loading, so this matches the source model's serving behavior)
  • All remaining weights (including attention, vision tower, embeddings and the shared experts) are untouched

REAP scores each expert by the router-weighted mean of its output activation norms over the calibration tokens routed to it, S_j = mean(g_j * ||f_j||_2), using forward computation only (no gradients or recovery training). This model was calibrated on all 2,048 samples of the RazorCal multi-domain calibration corpus; see the REAP expert pruning example for the calibration and pruning scripts.

Usage

from transformers import AutoProcessor, Glm5NextForConditionalGeneration

model = Glm5NextForConditionalGeneration.from_pretrained(
    "inference-optimization/GLM-5.3-Flash-NVFP4-REAP20"
)
processor = AutoProcessor.from_pretrained(
    "inference-optimization/GLM-5.3-Flash-NVFP4-REAP20"
)