mlavkin's picture
Add FP8 W8A8 dynamic quant of Huihui-GLM-4.7-Flash-abliterated
ad83565 verified
|
Raw History Blame Contribute Delete
2.65 kB
metadata
license: mit
language:
  - en
  - zh
base_model:
  - huihui-ai/Huihui-GLM-4.7-Flash-abliterated
base_model_relation: quantized
pipeline_tag: text-generation
tags:
  - glm4_moe_lite
  - fp8
  - w8a8
  - compressed-tensors
  - vllm
  - abliterated
  - uncensored

GLM-4.7-Flash-abliterated-FP8-Dynamic

FP8 (W8A8, dynamic) quantization of huihui-ai/Huihui-GLM-4.7-Flash-abliterated, itself an abliterated finetune of zai-org/GLM-4.7-Flash (a 30B-A3B glm4_moe_lite MoE model).

Quantized by mlavkin.

What this is

  • Weights: FP8 E4M3, per-channel scales.
  • Activations: FP8 dynamic, per-token (W8A8).
  • Kept in BF16 (not quantized): all MoE router gates (*.mlp.gate) and lm_head — standard for MoE to preserve routing quality.
  • Format: compressed-tensors (float-quantized).
  • Size: ~29 GB (vs ~59 GB BF16).

Why

FP8 halves weight memory traffic. On memory-bandwidth-bound hardware this yields a large decode speedup with negligible quality loss. Measured on an NVIDIA GB10 (DGX Spark), single-stream decode vs BF16:

Context BF16 FP8 Speedup
~0.6k 25.0 41.3 1.65×
~13k 21.7 33.6 1.55×
~33k 18.2 25.6 1.41×

Quality spot-checks (arithmetic, factual, code, needle-in-haystack retrieval up to ~47k) matched BF16.

Usage (vLLM)

Requires a recent vLLM + Transformers with glm4_moe_lite support.

vllm serve operationrange/GLM-4.7-Flash-abliterated-FP8-Dynamic \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --max-model-len 202752 \
  --host 0.0.0.0 --port 8000

The model's native context is 202,752 tokens. RoPE scaling (e.g. YaRN) can extend it further but noticeably degrades coherence — native length is recommended.

Quantization recipe

Produced with llm-compressor:

from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

model = AutoModelForCausalLM.from_pretrained(SRC, dtype="bfloat16", device_map="cpu")
recipe = QuantizationModifier(
    targets="Linear", scheme="FP8_DYNAMIC",
    ignore=["lm_head", "re:.*mlp.gate$", "re:.*eh_proj$"],
)
oneshot(model=model, recipe=recipe, output_dir=OUT)

FP8_DYNAMIC needs no calibration data (data-free).

License

MIT, inherited from the base models. Abliteration by huihui-ai; original model by Z.ai (zai-org).