Instructions to use GadflyII/GLM-4.7-Flash-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use GadflyII/GLM-4.7-Flash-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="GadflyII/GLM-4.7-Flash-NVFP4")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("GadflyII/GLM-4.7-Flash-NVFP4")
model = AutoModelForCausalLM.from_pretrained("GadflyII/GLM-4.7-Flash-NVFP4", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use GadflyII/GLM-4.7-Flash-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "GadflyII/GLM-4.7-Flash-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "GadflyII/GLM-4.7-Flash-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/GadflyII/GLM-4.7-Flash-NVFP4

SGLang

How to use GadflyII/GLM-4.7-Flash-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "GadflyII/GLM-4.7-Flash-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "GadflyII/GLM-4.7-Flash-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "GadflyII/GLM-4.7-Flash-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "GadflyII/GLM-4.7-Flash-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use GadflyII/GLM-4.7-Flash-NVFP4 with Docker Model Runner:
```
docker model run hf.co/GadflyII/GLM-4.7-Flash-NVFP4
```

GadflyII commited on Jan 19

Commit

208e33c

verified ·

1 Parent(s): 176f59f

Upload README.md with huggingface_hub

Browse files

Files changed (1) hide show

README.md +35 -18

README.md CHANGED Viewed

@@ -111,24 +111,41 @@ These compressed representations are highly sensitive to quantization noise. NVI
 ## Evaluation
-MMLU-Pro results by category:
-| Category | Accuracy |
-|----------|----------|
-| Biology | 47.42% |
-| Psychology | 42.48% |
-| Health | 34.84% |
-| Economics | 34.48% |
-| History | 30.71% |
-| Philosophy | 30.06% |
-| Other | 25.87% |
-| Computer Science | 21.46% |
-| Business | 16.98% |
-| Law | 16.35% |
-| Physics | 14.70% |
-| Math | 14.29% |
-| Engineering | 14.04% |
-| Chemistry | 13.34% |
 ## Citation

 ## Evaluation
+### MMLU-Pro Overall Results
+| Model | Accuracy | Correct | Total |
+|-------|----------|---------|-------|
+| **BF16 (baseline)** | **24.83%** | 2988 | 12032 |
+| **NVFP4 (this model)** | **23.55%** | 2834 | 12032 |
+| **Difference** | **-1.28%** | -154 | - |
+### MMLU-Pro by Category
+| Category | BF16 | NVFP4 | Difference |
+|----------|------|-------|------------|
+| Social Sciences | 32.70% | 31.43% | -1.27% |
+| Other | 31.57% | 30.08% | -1.49% |
+| Humanities | 23.78% | 22.56% | -1.22% |
+| STEM | 19.94% | 18.70% | -1.24% |
+### MMLU-Pro by Subject
+| Subject | BF16 | NVFP4 | Difference |
+|---------|------|-------|------------|
+| Biology | 50.35% | 47.42% | -2.93% |
+| Psychology | 44.99% | 42.48% | -2.51% |
+| Economics | 36.37% | 34.48% | -1.89% |
+| Health | 35.21% | 34.84% | -0.37% |
+| History | 33.60% | 30.71% | -2.89% |
+| Philosophy | 31.46% | 30.06% | -1.40% |
+| Other | 28.35% | 25.87% | -2.48% |
+| Computer Science | 26.10% | 21.46% | -4.64% |
+| Business | 16.35% | 16.98% | +0.63% |
+| Law | 16.89% | 16.35% | -0.54% |
+| Engineering | 16.00% | 14.04% | -1.96% |
+| Physics | 15.32% | 14.70% | -0.62% |
+| Math | 14.06% | 14.29% | +0.23% |
+| Chemistry | 14.13% | 13.34% | -0.79% |
 ## Citation