axteam/GLM-4.7-Flash-FP8-dynamic

GLM-4.7-Flash · FP8 (dynamic)

8-bit quantization of zai-org/GLM-4.7-Flash for vLLM serving. Quantized by axteam.

Base model	zai-org/GLM-4.7-Flash
Method	FP8 (dynamic)
Format	compressed-tensors float-quantized (scheme FP8_DYNAMIC)
Weights / activations	W8A8 — FP8 E4M3 weights (per-channel), dynamic per-token FP8 activations
Architecture	MoE (GLM-4.7 Lite · Glm4MoeLite) · 31.2B · 202K context · text-only
Size on disk	~32.2 GB
Calibration	None (data-free)
License	mit

Quantization details

• Scheme — W8A8 — FP8 E4M3 weights (per-channel), dynamic per-token FP8 activations, stored as compressed-tensors float-quantized (scheme FP8_DYNAMIC).
• Kept in BF16 — token embeddings, the LM head, and all RMSNorms. Only attention & MLP projections are quantized. In addition, the MoE router (mlp.gate, incl. e_score_correction_bias) is kept in BF16. The MTP layer (model.layers.47: eh_proj, shared_head.head) is quantized like the other layers.
• Post-training quantization (no retraining). Tokenizer / chat_template identical to the original.

Usage (vLLM)

Start an OpenAI-compatible server:

vllm serve axteam/GLM-4.7-Flash-FP8-dynamic --max-model-len 32768

Query it:

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "axteam/GLM-4.7-Flash-FP8-dynamic",
  "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}],
  "temperature": 1.0, "max_tokens": 512
}'

Offline inference in Python:

from vllm import LLM, SamplingParams

llm = LLM(model="axteam/GLM-4.7-Flash-FP8-dynamic", max_model_len=32768)
params = SamplingParams(temperature=1.0, max_tokens=1024)
out = llm.chat([{"role": "user", "content": "Explain model quantization in two sentences."}], params)
print(out[0].outputs[0].text)

Recommended sampling (from the original generation_config.json): temperature=1.0. Thinking mode is on by default in the chat template; to turn it off pass chat_template_kwargs={"enable_thinking": False} (Python llm.chat(..., chat_template_kwargs=...), or "chat_template_kwargs": {"enable_thinking": false} in the request body).

Reducing KV-cache memory (optional, serve-time)

The KV cache is kept in BF16 by default. KV-cache precision is a runtime choice, independent of the weight quantization above; for long contexts or large batches FP8 roughly halves KV-cache memory:

vllm serve axteam/GLM-4.7-Flash-FP8-dynamic --kv-cache-dtype fp8

License

Quantized derivative of zai-org/GLM-4.7-Flash (mit).

---

한국어 요약

zai-org/GLM-4.7-Flash 를 FP8 (dynamic) 로 양자화한 서빙용(8-bit) 모델입니다. 작성: axteam.

• 방식 — FP8 E4M3, 가중치 채널별 스케일, 활성화 동적 FP8. 형식 compressed-tensors float-quantized (scheme FP8_DYNAMIC).
• BF16 유지 — 임베딩·LM 헤드·모든 norm. attention·MLP projection 만 양자화. 추가로 MoE 라우터(mlp.gate, e_score_correction_bias 포함)도 BF16 유지. MTP 층(model.layers.47 의 eh_proj·shared_head.head)은 다른 층처럼 양자화됨.
• PTQ(재학습 없음), 토크나이저·chat_template 원본 동일.
• 실행 — vllm serve axteam/GLM-4.7-Flash-FP8-dynamic --max-model-len 32768 로 OpenAI 호환 서버(포트 8000), 또는 Python LLM(model="axteam/GLM-4.7-Flash-FP8-dynamic").chat(...) (위 예시). 권장 샘플링 temperature=1.0. thinking 끄기: chat_template_kwargs={"enable_thinking": False}.
• KV 캐시 절감(런타임) — --kv-cache-dtype fp8.