kylesayrs's picture
GLM-5.3-Flash-NVFP4 pruned to 20% expert sparsity with REAP (RazorCal calibration)
ff38bee verified
|
Raw History Blame Contribute Delete
2.41 kB
---
language:
- en
- zh
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- nvfp4
- fp4
- vllm
- llm-compressor
- compressed-tensors
- glm5_next
- expert-pruning
- reap
base_model:
- RedHatAI/GLM-5.3-Flash-NVFP4
license: mit
license_link: https://choosealicense.com/licenses/mit/
---
# GLM-5.3-Flash-NVFP4-REAP20
[RedHatAI/GLM-5.3-Flash-NVFP4](https://huggingface.co/RedHatAI/GLM-5.3-Flash-NVFP4)
(a NVFP4 quantization of
[zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash)) with 20%
of its MoE experts structurally pruned with **REAP** (PRUNE method), produced
with [llmcompressor](https://github.com/vllm-project/llmcompressor).
- Every one of the 42 MoE layers (backbone layers 3-44) keeps its **230 most
salient experts** of the original 288 (58 pruned per layer, 9660 of 12096
retained)
- Expert selection is uniform across layers: each layer's 58 lowest-REAP-saliency
experts were removed, and the routers and their `e_score_correction_bias`
were sliced to match
- `config.json` records the new expert count (`n_routed_experts: 230`); expert
indices are remapped contiguously
- The quantization is unchanged from the source checkpoint: MoE experts remain
NVFP4-quantized and the MTP layer (`layers.45`) is left untouched, keeping
its original 288 FP8-quantized experts (Transformers ignores `layers.45` when
loading, so this matches the source model's serving behavior)
- All remaining weights (including attention, vision tower, embeddings and the
shared experts) are untouched
REAP scores each expert by the router-weighted mean of its output activation
norms over the calibration tokens routed to it,
`S_j = mean(g_j * ||f_j||_2)`, using forward computation only (no gradients or
recovery training). This model was calibrated on all 2,048 samples of the
[RazorCal](https://huggingface.co/datasets/inference-optimization/RazorCal)
multi-domain calibration corpus; see the
[REAP expert pruning example](https://github.com/vllm-project/llm-compressor/tree/main/examples/reap_expert_pruning)
for the calibration and pruning scripts.
## Usage
```python
from transformers import AutoProcessor, Glm5NextForConditionalGeneration
model = Glm5NextForConditionalGeneration.from_pretrained(
"inference-optimization/GLM-5.3-Flash-NVFP4-REAP20"
)
processor = AutoProcessor.from_pretrained(
"inference-optimization/GLM-5.3-Flash-NVFP4-REAP20"
)
```