--- license: mit language: - en - zh base_model: - huihui-ai/Huihui-GLM-4.7-Flash-abliterated base_model_relation: quantized pipeline_tag: text-generation tags: - glm4_moe_lite - fp8 - w8a8 - compressed-tensors - vllm - abliterated - uncensored --- # GLM-4.7-Flash-abliterated-FP8-Dynamic FP8 (W8A8, dynamic) quantization of [huihui-ai/Huihui-GLM-4.7-Flash-abliterated](https://huggingface.co/huihui-ai/Huihui-GLM-4.7-Flash-abliterated), itself an abliterated finetune of [zai-org/GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash) (a 30B-A3B `glm4_moe_lite` MoE model). Quantized by **mlavkin**. ## What this is - **Weights:** FP8 **E4M3**, per-channel scales. - **Activations:** FP8 dynamic, per-token (W8A8). - **Kept in BF16 (not quantized):** all MoE router gates (`*.mlp.gate`) and `lm_head` — standard for MoE to preserve routing quality. - **Format:** `compressed-tensors` (`float-quantized`). - **Size:** ~29 GB (vs ~59 GB BF16). ## Why FP8 halves weight memory traffic. On memory-bandwidth-bound hardware this yields a large decode speedup with **negligible quality loss**. Measured on an NVIDIA GB10 (DGX Spark), single-stream decode vs BF16: | Context | BF16 | FP8 | Speedup | |--------:|-----:|----:|:-------:| | ~0.6k | 25.0 | 41.3 | 1.65× | | ~13k | 21.7 | 33.6 | 1.55× | | ~33k | 18.2 | 25.6 | 1.41× | Quality spot-checks (arithmetic, factual, code, needle-in-haystack retrieval up to ~47k) matched BF16. ## Usage (vLLM) Requires a recent vLLM + Transformers with `glm4_moe_lite` support. ```bash vllm serve operationrange/GLM-4.7-Flash-abliterated-FP8-Dynamic \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --enable-auto-tool-choice \ --max-model-len 202752 \ --host 0.0.0.0 --port 8000 ``` The model's native context is **202,752** tokens. RoPE scaling (e.g. YaRN) can extend it further but noticeably degrades coherence — native length is recommended. ## Quantization recipe Produced with [llm-compressor](https://github.com/vllm-project/llm-compressor): ```python from transformers import AutoModelForCausalLM, AutoTokenizer from llmcompressor import oneshot from llmcompressor.modifiers.quantization import QuantizationModifier model = AutoModelForCausalLM.from_pretrained(SRC, dtype="bfloat16", device_map="cpu") recipe = QuantizationModifier( targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head", "re:.*mlp.gate$", "re:.*eh_proj$"], ) oneshot(model=model, recipe=recipe, output_dir=OUT) ``` FP8_DYNAMIC needs no calibration data (data-free). ## License MIT, inherited from the base models. Abliteration by huihui-ai; original model by Z.ai (zai-org).