Instructions to use axteam/GLM-4.7-Flash-NVFP4A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use axteam/GLM-4.7-Flash-NVFP4A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="axteam/GLM-4.7-Flash-NVFP4A16")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("axteam/GLM-4.7-Flash-NVFP4A16")
model = AutoModelForCausalLM.from_pretrained("axteam/GLM-4.7-Flash-NVFP4A16", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use axteam/GLM-4.7-Flash-NVFP4A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "axteam/GLM-4.7-Flash-NVFP4A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/GLM-4.7-Flash-NVFP4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/axteam/GLM-4.7-Flash-NVFP4A16

SGLang

How to use axteam/GLM-4.7-Flash-NVFP4A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "axteam/GLM-4.7-Flash-NVFP4A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/GLM-4.7-Flash-NVFP4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "axteam/GLM-4.7-Flash-NVFP4A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/GLM-4.7-Flash-NVFP4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use axteam/GLM-4.7-Flash-NVFP4A16 with Docker Model Runner:
```
docker model run hf.co/axteam/GLM-4.7-Flash-NVFP4A16
```

GLM-4.7-Flash · NVFP4A16 (W4A16, FP4 weights)

4-bit quantization of zai-org/GLM-4.7-Flash for vLLM serving. Quantized by axteam.


Base model	`zai-org/GLM-4.7-Flash`
Method	NVFP4A16 (W4A16, FP4 weights)
Format	compressed-tensors `nvfp4-pack-quantized`
Weights / activations	W4A16 — FP4 (E2M1) weights group 16 + FP8 scale, BF16 activations
Architecture	MoE (GLM-4.7 Lite · Glm4MoeLite) · 31.2B · 202K context · text-only
Size on disk	~17.6 GB
Calibration	None (data-free, name-based `model_free_ptq`)
License	mit

Quantization details

Scheme — W4A16 — FP4 (E2M1) weights group 16 + FP8 scale, BF16 activations, stored as compressed-tensors nvfp4-pack-quantized.
Kept in BF16 — token embeddings, the LM head, and all RMSNorms. Only attention & MLP projections are quantized. In addition, the MoE router (mlp.gate) and shared-expert gates are kept in BF16.
Post-training quantization (no retraining). Tokenizer / chat_template identical to the original.
FP4 global scales are shared across the pairs vLLM fuses — q_a_proj + kv_a_proj_with_mqa (MLA) and gate_proj + up_proj of every routed and shared expert.

Revision 2026-10-31: the first upload of this repo (same day) quantized those pairs with separate global scales. vLLM runs each pair as one GEMM with one scale, so that build produced corrupted text. The weights were re-quantized and replaced; if you downloaded this repo before the fix, please download it again.

Usage (vLLM)

vllm serve axteam/GLM-4.7-Flash-NVFP4A16

License

Quantized derivative of zai-org/GLM-4.7-Flash (mit).

한국어 요약

zai-org/GLM-4.7-Flash 를 NVFP4A16 (W4A16, FP4 weights) 로 양자화한 서빙용(4-bit) 모델입니다. 작성: axteam.

방식 — FP4 가중치 16묶음 + 활성화 BF16. 4비트 중 품질 최상급. 형식 compressed-tensors nvfp4-pack-quantized.
BF16 유지 — 임베딩·LM 헤드·모든 norm. attention·MLP projection 만 양자화. 추가로 MoE 라우터(mlp.gate)와 shared-expert 게이트도 BF16 유지.
PTQ(재학습 없음), 토크나이저·chat_template 원본 동일.
수정 2026-10-31 — 같은 날 처음 올린 빌드는 vLLM 이 한 GEMM 으로 묶는 쌍(MLA q_a_proj+kv_a_proj_with_mqa, 전문가 gate_proj+up_proj)의 FP4 global scale 이 서로 달라 vLLM 에서 깨진 출력을 냈습니다. 쌍마다 scale 을 공유하도록 다시 양자화해 교체했으니, 수정 전에 받으셨다면 다시 받아 주세요.
사용 — vllm serve axteam/GLM-4.7-Flash-NVFP4A16. 라이선스 mit.

Downloads last month: -

Safetensors

Model size

31B params

Tensor type

F32

BF16

Model tree for axteam/GLM-4.7-Flash-NVFP4A16

Base model

zai-org/GLM-4.7-Flash

Quantized

(111)

this model