|
Download README.md from operationrange/GLM-4.7-Flash-abliterated-FP8-Dynamic: direct link, hf CLI and curl.
- Browser
- Download file 2.65 kB
-
https://huggingface.co/operationrange/GLM-4.7-Flash-abliterated-FP8-Dynamic/resolve/main/README.md
- Command line
-
hf download hf://operationrange/GLM-4.7-Flash-abliterated-FP8-Dynamic/README.md
-
curl -L -o README.md https://huggingface.co/operationrange/GLM-4.7-Flash-abliterated-FP8-Dynamic/resolve/main/README.md
2.65 kB
| license: mit | |
| language: | |
| - en | |
| - zh | |
| base_model: | |
| - huihui-ai/Huihui-GLM-4.7-Flash-abliterated | |
| base_model_relation: quantized | |
| pipeline_tag: text-generation | |
| tags: | |
| - glm4_moe_lite | |
| - fp8 | |
| - w8a8 | |
| - compressed-tensors | |
| - vllm | |
| - abliterated | |
| - uncensored | |
| # GLM-4.7-Flash-abliterated-FP8-Dynamic | |
| FP8 (W8A8, dynamic) quantization of [huihui-ai/Huihui-GLM-4.7-Flash-abliterated](https://huggingface.co/huihui-ai/Huihui-GLM-4.7-Flash-abliterated), | |
| itself an abliterated finetune of [zai-org/GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash) | |
| (a 30B-A3B `glm4_moe_lite` MoE model). | |
| Quantized by **mlavkin**. | |
| ## What this is | |
| - **Weights:** FP8 **E4M3**, per-channel scales. | |
| - **Activations:** FP8 dynamic, per-token (W8A8). | |
| - **Kept in BF16 (not quantized):** all MoE router gates (`*.mlp.gate`) and `lm_head` — standard for MoE to preserve routing quality. | |
| - **Format:** `compressed-tensors` (`float-quantized`). | |
| - **Size:** ~29 GB (vs ~59 GB BF16). | |
| ## Why | |
| FP8 halves weight memory traffic. On memory-bandwidth-bound hardware this yields a large decode speedup with **negligible quality loss**. Measured on an NVIDIA GB10 (DGX Spark), single-stream decode vs BF16: | |
| | Context | BF16 | FP8 | Speedup | | |
| |--------:|-----:|----:|:-------:| | |
| | ~0.6k | 25.0 | 41.3 | 1.65× | | |
| | ~13k | 21.7 | 33.6 | 1.55× | | |
| | ~33k | 18.2 | 25.6 | 1.41× | | |
| Quality spot-checks (arithmetic, factual, code, needle-in-haystack retrieval up to ~47k) matched BF16. | |
| ## Usage (vLLM) | |
| Requires a recent vLLM + Transformers with `glm4_moe_lite` support. | |
| ```bash | |
| vllm serve operationrange/GLM-4.7-Flash-abliterated-FP8-Dynamic \ | |
| --tool-call-parser glm47 \ | |
| --reasoning-parser glm45 \ | |
| --enable-auto-tool-choice \ | |
| --max-model-len 202752 \ | |
| --host 0.0.0.0 --port 8000 | |
| ``` | |
| The model's native context is **202,752** tokens. RoPE scaling (e.g. YaRN) can extend it further but noticeably degrades coherence — native length is recommended. | |
| ## Quantization recipe | |
| Produced with [llm-compressor](https://github.com/vllm-project/llm-compressor): | |
| ```python | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| from llmcompressor import oneshot | |
| from llmcompressor.modifiers.quantization import QuantizationModifier | |
| model = AutoModelForCausalLM.from_pretrained(SRC, dtype="bfloat16", device_map="cpu") | |
| recipe = QuantizationModifier( | |
| targets="Linear", scheme="FP8_DYNAMIC", | |
| ignore=["lm_head", "re:.*mlp.gate$", "re:.*eh_proj$"], | |
| ) | |
| oneshot(model=model, recipe=recipe, output_dir=OUT) | |
| ``` | |
| FP8_DYNAMIC needs no calibration data (data-free). | |
| ## License | |
| MIT, inherited from the base models. Abliteration by huihui-ai; original model by Z.ai (zai-org). | |