steampunque/GLM-4.6V-Flash-MP-GGUF

Mixed Precision GGUF layer quantization of GLM-4.6V-Flash by zai-org

Original model: https://huggingface.co/zai-org/GLM-4.6V-Flash

The hybrid quant employs different quantization levels on a per layer basis to enable both high performance and small file size at the same time. An extended layer definition E quant Q4_E_H for the model is defined as follows (updated 10/17/2026):

   LAYER_TYPES='[
   ["A","attn","Q","attn_q","K","attn_k","V","attn_v","O","attn_o","S","ssm","F","ffn","G","ffn_g","U","ffn_u","D","ffn_d"],
   ["MAP","VOD","0","QN_K","2","Q2_K","3","Q3_K","4","Q4_K","5","Q5_K","6","Q6_K","8","Q8_0","h","F16","f","F32"],
   [0 ,"Q5_K_666"], [1 ,"Q4_K_666"], [2 ,"Q4_K_655"], [3 ,"Q4_K_644"], [4 ,"Q4_K_544"], [5 ,"Q4_K_544"], [6 ,"Q4_K_544"], [7 ,"Q4_K_544"],
   [8 ,"Q4_K_544"], [9 ,"Q4_K_544"], [10,"Q4_K_544"], [11,"Q4_K_544"], [12,"Q4_K_544"], [13,"Q4_K_544"], [14,"Q4_K_544"], [15,"Q4_K_544"],
   [16,"Q4_K_544"], [17,"Q4_K_544"], [18,"Q4_K_544"], [19,"Q4_K_544"], [20,"Q4_K_544"], [21,"Q4_K_544"], [22,"Q4_K_544"], [23,"Q4_K_544"],
   [24,"Q4_K_544"], [25,"Q4_K_544"], [26,"Q4_K_544"], [27,"Q4_K_544"], [28,"Q4_K_544"], [29,"Q4_K_544"], [30,"Q4_K_544"], [31,"Q4_K_544"],
   [32,"Q4_K_544"], [33,"Q4_K_544"], [34,"Q4_K_544"], [35,"Q4_K_544"], [36,"Q4_K_544"], [37,"Q4_K_544"], [38,"Q4_K_544"], [39,"Q4_K_644"],
   [40,"Q4_K_644"], [41,"Q4_K_654"], [42,"Q4_K_655"], [43,"Q4_K_665"], [44,"Q4_K_666"], [45,"Q5_K_668"], [46,"Q6_K_866"]
   ]'
   FLAGS="--token-embedding-type Q5_K --output-tensor-type Q6_K --layer-types-high --tensor-pad [[13696,13824],[27392,27648,2]] --override-kv glm4.feed_forward_length=int:13824"

Model FFN length is padded from 13696 to 13824 to allow use of K quants in the layers instead of fallback legacy quants. Minimum quant across layers is Q4_K to minimize entropy loss. The bpw is sized to enable the model + vision tower to run on an 8G VRAM card and then evaluated for acceptable vision and reasoning performance across a curated set of test prompts.

Comparison:

Quant	size	PPL	Comment
Q4_K_S	5.8e9	12.3	-
Q4_E_H	5.6e9	12.1	Hybrid quant with Q5_K embedding Q6_K output

Usage:

This is a RL trained (thinking) vision model. The layer quants for this model were evaluated on a set of test/eval prompts using greedy sampling. The model appears to be robust against infinite generations on the eval prompts with greedy sampling, reliably converging to an answer on all tested prompts. Image mode was tested on a small set of images and found to be both functional and accurate.

This model will respond in Chinese with no system prompt. For english responses, the following system prompt can be used:

SYSTEM="language = en"

In tests this system prompt will cause the model to both reason and respond in english. Note this system prompt is completely empirical and it appears necessary to change it when the model quant changes to make it robustly respond in english. The prompt above is for Q4_E_H and may or may not work well with other quants.

The model can be speculated with Qwen3 0.6B if the inference platform can handle vocab translation between draft and target. Approx performance using a downstream speculator with llama.cpp on one 4070 (12G VRAM) GPU with fixed spec block length ND:

ND	QKV	NKV	gen tps	Comment
3	F16	38k	108	llama.cpp b11716 downstream spec
0	F16	128k	75	""
3	Q8_0	42k	107	""
0	Q8_0	128k	74	""

Vision:

The model went 2 for 2 on a couple tough bird ID images.

Benchmarks:

Vision benchmarks for the model are given here: https://huggingface.co/spaces/steampunque/benchlm

Download the file from below:

Link	Type	Size/e9 B	Notes
GLM-4.6V-Flash.Q4_E_H.gguf	Q4_E_H	5.6e9 B	~0.2B smaller than Q4_K_S
GLM-4.6V-Flash.mmproj.gguf	F16	1.8e9 B	multimedia projector

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040