mlavkin's picture
Add FP8 W8A8 dynamic quant of Huihui-GLM-4.7-Flash-abliterated
ad83565 verified
|
Raw History Blame Contribute Delete
2.65 kB
---
license: mit
language:
- en
- zh
base_model:
- huihui-ai/Huihui-GLM-4.7-Flash-abliterated
base_model_relation: quantized
pipeline_tag: text-generation
tags:
- glm4_moe_lite
- fp8
- w8a8
- compressed-tensors
- vllm
- abliterated
- uncensored
---
# GLM-4.7-Flash-abliterated-FP8-Dynamic
FP8 (W8A8, dynamic) quantization of [huihui-ai/Huihui-GLM-4.7-Flash-abliterated](https://huggingface.co/huihui-ai/Huihui-GLM-4.7-Flash-abliterated),
itself an abliterated finetune of [zai-org/GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash)
(a 30B-A3B `glm4_moe_lite` MoE model).
Quantized by **mlavkin**.
## What this is
- **Weights:** FP8 **E4M3**, per-channel scales.
- **Activations:** FP8 dynamic, per-token (W8A8).
- **Kept in BF16 (not quantized):** all MoE router gates (`*.mlp.gate`) and `lm_head` — standard for MoE to preserve routing quality.
- **Format:** `compressed-tensors` (`float-quantized`).
- **Size:** ~29 GB (vs ~59 GB BF16).
## Why
FP8 halves weight memory traffic. On memory-bandwidth-bound hardware this yields a large decode speedup with **negligible quality loss**. Measured on an NVIDIA GB10 (DGX Spark), single-stream decode vs BF16:
| Context | BF16 | FP8 | Speedup |
|--------:|-----:|----:|:-------:|
| ~0.6k | 25.0 | 41.3 | 1.65× |
| ~13k | 21.7 | 33.6 | 1.55× |
| ~33k | 18.2 | 25.6 | 1.41× |
Quality spot-checks (arithmetic, factual, code, needle-in-haystack retrieval up to ~47k) matched BF16.
## Usage (vLLM)
Requires a recent vLLM + Transformers with `glm4_moe_lite` support.
```bash
vllm serve operationrange/GLM-4.7-Flash-abliterated-FP8-Dynamic \
--tool-call-parser glm47 \
--reasoning-parser glm45 \
--enable-auto-tool-choice \
--max-model-len 202752 \
--host 0.0.0.0 --port 8000
```
The model's native context is **202,752** tokens. RoPE scaling (e.g. YaRN) can extend it further but noticeably degrades coherence — native length is recommended.
## Quantization recipe
Produced with [llm-compressor](https://github.com/vllm-project/llm-compressor):
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
model = AutoModelForCausalLM.from_pretrained(SRC, dtype="bfloat16", device_map="cpu")
recipe = QuantizationModifier(
targets="Linear", scheme="FP8_DYNAMIC",
ignore=["lm_head", "re:.*mlp.gate$", "re:.*eh_proj$"],
)
oneshot(model=model, recipe=recipe, output_dir=OUT)
```
FP8_DYNAMIC needs no calibration data (data-free).
## License
MIT, inherited from the base models. Abliteration by huihui-ai; original model by Z.ai (zai-org).