Instructions to use axteam/GLM-4.7-Flash-FP8-dynamic with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use axteam/GLM-4.7-Flash-FP8-dynamic with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="axteam/GLM-4.7-Flash-FP8-dynamic")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("axteam/GLM-4.7-Flash-FP8-dynamic")
model = AutoModelForCausalLM.from_pretrained("axteam/GLM-4.7-Flash-FP8-dynamic", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use axteam/GLM-4.7-Flash-FP8-dynamic with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "axteam/GLM-4.7-Flash-FP8-dynamic"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/GLM-4.7-Flash-FP8-dynamic",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/axteam/GLM-4.7-Flash-FP8-dynamic

SGLang

How to use axteam/GLM-4.7-Flash-FP8-dynamic with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "axteam/GLM-4.7-Flash-FP8-dynamic" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/GLM-4.7-Flash-FP8-dynamic",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "axteam/GLM-4.7-Flash-FP8-dynamic" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/GLM-4.7-Flash-FP8-dynamic",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use axteam/GLM-4.7-Flash-FP8-dynamic with Docker Model Runner:
```
docker model run hf.co/axteam/GLM-4.7-Flash-FP8-dynamic
```

GLM-4.7-Flash · FP8 (dynamic)

8-bit quantization of zai-org/GLM-4.7-Flash for vLLM serving. Quantized by axteam.


Base model	`zai-org/GLM-4.7-Flash`
Method	FP8 (dynamic)
Format	compressed-tensors `float-quantized` (scheme `FP8_DYNAMIC`)
Weights / activations	W8A8 — FP8 E4M3 weights (per-channel), dynamic per-token FP8 activations
Architecture	MoE (GLM-4.7 Lite · Glm4MoeLite) · 31.2B · 202K context · text-only
Size on disk	~32.2 GB
Calibration	None (data-free)
License	mit

Quantization details

Scheme — W8A8 — FP8 E4M3 weights (per-channel), dynamic per-token FP8 activations, stored as compressed-tensors float-quantized (scheme FP8_DYNAMIC).
Kept in BF16 — token embeddings, the LM head, and all RMSNorms. Only attention & MLP projections are quantized. In addition, the MoE router is left unquantized: mlp.gate.weight in BF16 and mlp.gate.e_score_correction_bias in FP32, as in the original. The MTP layer (model.layers.47: eh_proj, shared_head.head) is quantized like the other layers.
Post-training quantization (no retraining). Tokenizer / chat_template identical to the original.

Benchmarks (measured)

Every column was measured with the same harness and settings — lm-evaluation-harness on vLLM, greedy, completion-style prompts without a chat template (0- or 5-shot, as listed in each row). MMLU here is lm-eval's generative variant (mmlu_generative: the model writes the answer letter and only an exact match on the first line counts), so it is not directly comparable with MMLU scores measured another way (e.g. log-likelihood scoring). Percentages are recovery vs. the BF16 original.

Benchmark	BF16 (original)	FP8-dynamic (this repo)	NVFP4A16
MMLU (5-shot, generative)	72.19	71.78 (99.4%)	69.97 (96.9%)
GSM8K (5-shot, strict)	83.70	82.71 (98.8%)	79.83 (95.4%)
ARC-Challenge (0-shot)	57.51	57.51 (100.0%)	55.03 (95.7%)
Winogrande (5-shot)	74.51	73.32 (98.4%)	73.40 (98.5%)
KMMLU (5-shot, direct)	50.04	50.15 (100.2%)	47.38 (94.7%)
HAE-RAE (0-shot)	64.89	63.89 (98.4%)	59.67 (91.9%)
Average	67.14	66.56 (99.1%)	64.21 (95.6%)

Requirements


Runtime	vLLM — the architecture is in vLLM ≥ 0.15.0; tested here with vLLM 0.32.0.
vLLM install	vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately).
GPU	NVIDIA Turing (sm75, e.g. T4) or newer. FP8 math runs on Ada / Hopper / Blackwell (sm89+); on Turing and Ampere vLLM switches to weight-only FP8 automatically. V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.)
Memory	Weights: 32.2 GB. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use `--tensor-parallel-size N` when the weights do not fit one GPU.
Transformers (without vLLM)	`transformers>=5.15.0`.
trust_remote_code	not needed

Usage (vLLM)

Start an OpenAI-compatible server:

vllm serve axteam/GLM-4.7-Flash-FP8-dynamic --max-model-len 32768

Query it:

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "axteam/GLM-4.7-Flash-FP8-dynamic",
  "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}],
  "temperature": 1.0, "max_tokens": 512
}'

Offline inference in Python:

from vllm import LLM, SamplingParams

llm = LLM(model="axteam/GLM-4.7-Flash-FP8-dynamic", max_model_len=32768)
params = SamplingParams(temperature=1.0, max_tokens=1024)
out = llm.chat([{"role": "user", "content": "Explain model quantization in two sentences."}], params)
print(out[0].outputs[0].text)

Recommended sampling (from the original generation_config.json): temperature=1.0. Thinking mode is on by default in the chat template; to turn it off pass chat_template_kwargs={"enable_thinking": False} (Python llm.chat(..., chat_template_kwargs=...), or "chat_template_kwargs": {"enable_thinking": false} in the request body).

Reducing KV-cache memory (optional, serve-time)

The KV cache is kept in BF16 by default. KV-cache precision is a runtime choice, independent of the weight quantization above; for long contexts or large batches FP8 roughly halves KV-cache memory:

vllm serve axteam/GLM-4.7-Flash-FP8-dynamic --kv-cache-dtype fp8

License

Quantized derivative of zai-org/GLM-4.7-Flash (mit).

한국어 요약

zai-org/GLM-4.7-Flash의 FP8 (W8A8) 양자화 빌드 · 제작 axteam

개요

항목	내용
원본 모델	`zai-org/GLM-4.7-Flash`
양자화 방식	FP8 동적 양자화 (W8A8)
저장 형식	compressed-tensors `float-quantized` (scheme `FP8_DYNAMIC`)
모델 구조	MoE (GLM-4.7 Lite, `Glm4MoeLite`) · 31.2B · 컨텍스트 202K · 텍스트 전용
용량	약 32.2 GB
보정 데이터	없음 (가중치 스케일은 가중치만으로 계산, 활성화 스케일은 실행 시 계산)

양자화 상세

가중치: FP8(E4M3), 출력 채널별 스케일
활성화: 실행 시 토큰별 동적 FP8
양자화 절차: PTQ(학습 후 양자화, 재학습 없음), 가중치 스케일은 미리 계산하고 활성화 스케일은 실행 시 토큰마다 계산
BF16 유지: 토큰 임베딩, LM 헤드, 모든 RMSNorm, MoE 라우터 가중치(mlp.gate.weight)
FP32 유지: MoE 라우터 보정 편향(mlp.gate.e_score_correction_bias), 원본과 동일
양자화 대상: 어텐션·MLP 선형층(projection) (MTP 층 model.layers.47도 다른 층과 같이 양자화)
원본 파일: 토크나이저·chat template은 원본과 동일

품질 (실측)

벤치마크 평균: 66.56 (원본 67.14, 99.1%) — MMLU(생성형)·GSM8K·ARC-Challenge·Winogrande·KMMLU·HAE-RAE 6종 평균, lm-evaluation-harness + vLLM, chat template 없는 완성형 프롬프트(과제별 0·5-shot, 표 참고)·greedy, 모든 빌드를 같은 설정으로 측정
한국어 벤치마크: KMMLU 50.15 (원본 50.04, 100.2%) · HAE-RAE 63.89 (원본 64.89, 98.4%)
MMLU 방식: lm-eval 생성형 변형(mmlu_generative) — 모델이 답 글자를 생성하고 첫 줄이 정확히 일치해야 정답으로 채점하므로, 다른 방식(예: log-likelihood 채점)으로 잰 MMLU 점수와 직접 비교 불가

요구사항

실행 엔진: vLLM 0.15.0 이상(이 구조를 지원하는 첫 버전), 이 빌드는 vLLM 0.32.0에서 실행 확인
vLLM 0.32 설치 조건: Python 3.11~3.14, torch 2.13(CUDA 13.0 빌드, CUDA 13 지원 드라이버 필요), transformers 5.16.1 이상 5.20 미만, compressed-tensors 0.17.0 — transformers·compressed-tensors는 vLLM이 버전을 고정하므로 compressed-tensors를 따로 업그레이드하지 말 것 (이 체크포인트는 0.19.0으로 저장됐지만 0.17.0에서도 로드됨)
GPU: NVIDIA Turing(sm75, 예: T4) 이상, V100(sm70) 미지원 — FP8 연산은 Ada·Hopper·Blackwell(sm89 이상)에서 수행, Turing·Ampere는 vLLM이 가중치 전용 FP8로 자동 전환 (GPU별 지원 여부는 vLLM 0.32 커널 선택 코드 기준, 실제 실행은 B200에서만 확인)
메모리: 가중치 32.2 GB + KV 캐시(컨텍스트 길이 × 동시 요청 수에 비례) + 활성화·CUDA graph용 수 GB — GPU 한 장에 다 들어가지 않으면 --tensor-parallel-size N으로 분산
Transformers로 직접 사용: transformers 5.15.0 이상
trust_remote_code: 불필요

사용법

서버 실행: vllm serve axteam/GLM-4.7-Flash-FP8-dynamic --max-model-len 32768 — OpenAI 호환 API
Python: LLM(model="axteam/GLM-4.7-Flash-FP8-dynamic", max_model_len=32768).chat(...) — 전체 예시는 위 영어 본문 'Usage (vLLM)' 절 참고
권장 샘플링: temperature=1.0 (원본 generation_config.json 기준)
thinking 모드: 기본 켜짐 — 끄려면 chat_template_kwargs={"enable_thinking": False} 전달(Python은 llm.chat() 인자, HTTP는 요청 본문의 "chat_template_kwargs": {"enable_thinking": false})
KV 캐시 절감(선택): vllm serve에 --kv-cache-dtype fp8 추가 — 기본 BF16인 KV 캐시를 FP8로 저장해 KV 캐시 메모리를 약 절반으로 줄임 (긴 컨텍스트·동시 요청이 많을 때 유리, 가중치 양자화와 별개인 실행 옵션)

라이선스

라이선스: MIT — 원본 모델의 라이선스를 그대로 따름
권리: 원본 모델의 권리는 Z.ai에게 있음, 이 리포는 양자화 파생물

Downloads last month: 149

Safetensors

Model size

31B params

Tensor type

F32

BF16

F8_E4M3

Model tree for axteam/GLM-4.7-Flash-FP8-dynamic

Base model

zai-org/GLM-4.7-Flash

Quantized

(111)

this model