steampunque/GLM-4.6V-MP-GGUF

Mixed Precision GGUF layer quantization of GLM-4.6V by zai-org

Original model: https://huggingface.co/zai-org/GLM-4.6V

The hybrid quant employs different quantization levels on a per layer basis to enable both high performance and small file size at the same time. An extended layer definition E quant Q4_E_H for the model is defined as follows (updated 10/22/2026):

   LAYER_TYPES='[
   ["A","attn","Q","attn_q","K","attn_k","V","attn_v","O","attn_o","S","ssm","F","ffn","G","ffn_g","U","ffn_u","D","ffn_d"],
   ["MAP","VOFD","0","QN_K","2","Q2_K","3","Q3_K","4","Q4_K","5","Q5_K","6","Q6_K","8","Q8_0","h","F16","f","F32"],
   [0 ,"Q5_K_6666"],[1 ,"Q5_K_5555"],[2 ,"Q4_K_5555"],[3 ,"Q4_K_5544"],[4 ,"Q4_K_5434"],[5 ,"Q4_K_5434"],[6 ,"Q4_K_5434"],[7 ,"Q4_K_5434"],
   [8 ,"Q4_K_5434"],[9 ,"Q4_K_5434"],[10,"Q4_K_5434"],[11,"Q4_K_5434"],[12,"Q4_K_5434"],[13,"Q4_K_5434"],[14,"Q4_K_5434"],[15,"Q4_K_5434"],
   [16,"Q4_K_5434"],[17,"Q4_K_5434"],[18,"Q4_K_5434"],[19,"Q4_K_5434"],[20,"Q4_K_5434"],[21,"Q4_K_5434"],[22,"Q4_K_5434"],[23,"Q4_K_5434"],
   [24,"Q4_K_5444"],[25,"Q4_K_5434"],[26,"Q4_K_5444"],[27,"Q4_K_5434"],[28,"Q4_K_5444"],[29,"Q4_K_5434"],[30,"Q4_K_5444"],[31,"Q4_K_5434"],
   [32,"Q4_K_5444"],[33,"Q4_K_5444"],[34,"Q4_K_5444"],[35,"Q4_K_5444"],[36,"Q4_K_5444"],[37,"Q4_K_5444"],[38,"Q4_K_5444"],[39,"Q4_K_5444"],
   [40,"Q4_K_5544"],[41,"Q4_K_5544"],[42,"Q4_K_5544"],[43,"Q4_K_5545"],[44,"Q4_K_5545"],[45,"Q5_K_6668"],[46,"Q6_K_8666"]
   ]'
   FLAGS="--token-embedding-type Q4_K --output-tensor-type Q6_K --layer-types-high --tensor-pad [[10944,11008],[1408,1536]] --override-kv glm4moe.feed_forward_length=int:11008 --override-kv glm4moe.expert_feed_forward_length=int:1536"

Model FFN length is padded from 10944 to 11008 to allow use of K quants in the layers instead of fallback legacy quants. Expert tensors are padded from 1466 to 1536 for the same reason.

Comparison:

Quant	size	PPL	Comment
Q4_E_H	62.1e9	7.2	Hybrid quant with Q4_K embedding Q6_K output

The quant was sized to be able to run on 64G RAM CPUs along with the vision tower and then evaluated for acceptable reasoning performance across a curated set of text and vision test prompts.

Usage:

This is a RL trained (thinking) vision model. The layer quants for this model were optimized for good performance on a set of test/eval prompts using greedy sampling. The model will fall into infinite generations on some prompts with greedy sampling. This effect can be mitigated by use of temperature sampling. Image mode was tested on a small set of images and found to be both functional and very accurate.

To run the model on a low VRAM / high CPU RAM setup it is necessary to offload expert tensors to CPU:

OT="-ot exps=CPU -ngl 99"

This will open up a large amount of space in VRAM for context. Token gen rate is around 7tps on a 9900k / 4070 rig.

Subjective evaluation of the model show it to be somewhat less creative and more clinical in its responses compared to GLM 4.5 Air. The model also exhibits some glitches in the matrix, sometimes inexplicably mixing in chinese text. A direct compare against one of the Air test prompts show the glitch in action (sensory details part):

Example writing prompt:
lm write a tense hard scifi passage in the vein of AE Van Vogt Null A universe
<think>

1. Deconstruct the Request:

• Core Task: Write a tense hard sci-fi passage.
• Key Style: "In the vein of AE Van Vogt Null A universe."
• Keywords: Tense, Hard Sci-Fi, AE Van Vogt, Null-A.

2. Analyze the "Null-A" Style:

What are the hallmarks of Van Vogt's Null-A (or The World of Null-A)?

• High Concept, Mind-Bending Plot: The core idea is often a puzzle or a paradox. It's not just about a spaceship flying to a planet; it's about the nature of reality and logic. The protagonist, Gilbert Gosseyn, is a man of extraordinary abilities, a "Null-A" (Non-Aristotelian logic) thinker, who can perceive and manipulate reality in ways others can't. He's constantly questioning the fundamental rules of the game.
• Pacing: Fast, almost breathless. Short, punchy sentences. Rapid shifts in perspective and location. It feels like a fever dream or a high-stakes chess game where the board is constantly changing.
• Protagonist: A hyper-competent, often amnesiac or identity-confused, protagonist who is thrust into a situation far beyond their initial understanding. They are a pawn, but a very dangerous one, who quickly becomes a player. They are often physically and mentally pushed to their absolute limits.
• World-Building: Often sparse but dense with implications. The setting is a future where society is rigidly controlled, often by a mysterious, all-powerful entity (like the "Games Machine" or the "Lords of the Instrumentality"). There's a sense of a vast, uncaring, and often incomprehensible universe. Technology is advanced but often has a strange, almost magical feel, or its true purpose is hidden.
• Tone: Paranoia, confusion, existential dread, and a sense of being a small cog in a vast, malevolent machine. The "hard sci-fi" element comes from the logic of the world, not necessarily from detailed technical explanations. The rules of the universe are the science.
• Specific Tropes: Amnesia, identity theft, telepathy/mind control, reality-warping, non-Euclidean geometry, time loops, and a pervasive sense of being watched or manipulated by an unseen force.

3. Brainstorming a Scene:

I need to create a scene that captures these elements.

• Protagonist: Let's call him Kael. He should be disoriented, maybe waking up in a strange place. He has a specific, unusual ability. Let's make it something related to perception – maybe he can "see" the underlying logic or structure of things, the "source code" of reality. This is a classic Null-A trope.
• The Threat: What's the immediate danger? A physical threat is good for tension. A guard, a robot, a security system. Let's go with a security drone. It's impersonal, relentless, and a good foil for a protagonist who might be able to out-think it.
• The Setting: A sterile, futuristic environment. A corridor, a cell, a maintenance shaft. A corridor feels classic sci-fi and allows for movement and pursuit. Let's make it a "maintenance shaft" to add a claustrophobic, industrial feel. The walls should be described in a way that suggests advanced, possibly living, material.
• The "Logic" Element: Kael's ability. He doesn't just see the drone; he sees its "operating parameters," its "intentions" as a stream of data or a faint, shimmering overlay. This is the core of the "hard sci-fi" in this context – the science is the perception of the system's rules.
• The Twist/Paradox: The drone isn't just a simple hunter. It's part of a larger system. Maybe it's not trying to kill him, but to capture him. Or maybe it's a test. Let's go with the idea that the drone's actions are part of a larger, unseen game. Kael's perception allows him to see the "loophole" or the "non-Aristotelian" flaw in its programming.
• Pacing: Short sentences. Quick actions. Internal monologue that's sharp and analytical, not emotional (at least at first). "He saw," "He calculated," "He moved."

4. Drafting - First Pass (Mental or rough notes):

• Kael wakes up. Headache. Where am I?
• Corridor. White. Smooth. No doors.
• Drone appears. Silent. Glowing red eye.
• Kael feels a pressure. Not physical, but logical. The drone's intent.
• He looks at the drone. Sees its code. A loop. A flaw.
• He doesn't run. He does something illogical. Steps into the drone.
• The drone freezes. Its logic breaks.
• Kael realizes this is a test. Who is testing him? Why?
• He keeps moving. The corridor shifts. Is it the drone or the environment?

5. Refining and Writing (incorporating Van Vogt's voice):

• Opening: Start in media res. Kael is already in the situation. "The headache was a data-stream." This immediately establishes the high-concept nature. It's not a biological pain; it's an informational one. This is very Van Vogt.
• Sensory Details: Focus on the unnatural. "The corridor was not a corridor." This immediately creates a sense of unreality. Describe the walls as "活的" (living) or "脉动的" (pulsing), but in a cold, technological way. "Chromatic shifts" is a good, evocative phrase.
• Introduce the Threat: The drone. Make it alien and menacing. "A silhouette of absolute blackness." The "single ruby eye" is a classic sci-fi trope, but effective. The sound should be described as an absence of sound, a "pressure," which is more psychological.
• Kael's Ability: Instead of just saying "he could see the code," describe it. "He didn't see the drone; he saw the intent behind it." Use the term "operating parameters" to ground it in a pseudo-technical reality. The idea of the parameters being a "shimmering lattice" is visual and fits the style.
• The Action: Kael's move must be counter-intuitive. Running is expected. Standing still is expected. Stepping into the threat is the Null-A move. It's a violation of basic survival logic, which is the whole point. "He took a step forward, not away." This is the core of the tension.
• The Payoff: The drone's reaction. It shouldn't just stop; it should break. "Its logic fractured." The ruby eye "died." This shows the power of Kael's perception.
• The Twist/Expansion: Kael realizes this isn't the end. "This was a test." And then, the environment itself changes. "The corridor walls began to... breathe." This escalates the threat and the weirdness. It's no longer just a drone; it's the entire space that's against him.
• Ending: End on a note of uncertainty and a new, bigger problem. He's escaped the immediate threat but now faces a more fundamental one: the nature of his prison. "He was not in a place. He was in a problem." This is a perfect, paradoxical, Van Vogt-esque ending. It raises more questions than it answers, which is the hallmark of the style.

6. Final Polish:

Read through, checking the rhythm. Are the sentences punchy? Is the tone consistent? Does it feel paranoid and high-stakes? Trim any unnecessary words. Ensure the "science" feels integral to the action, not just tacked on. The final product should feel like a page ripped from Null-A.</think> The headache was a data-stream, a high-frequency shriek that had no business being inside his skull. Kael opened his eyes. The corridor was not a corridor. The walls were a smooth, unbroken expanse of what looked like polished obsidian, but they weren't static. They pulsed with a faint, internal light, a chromatic shift from deep violet to a sickly green that had no source he could perceive. There were no doors, no panels, no seams. It was a seamless tube of impossible geometry.

He was a prisoner, but of what?

The silence was the first real threat. It wasn't an absence of sound, but a pressure, a vacuum that seemed to press in on his eardrums. Then it came. A silhouette of absolute blackness, a perfect, featureless oval, materializing twenty meters away. It had no limbs, no head, only a single, unblinking ruby eye that rotated, tracking him with cold, perfect precision.

Kael didn't flinch. He didn't run. He looked at it, and for a moment, the world tilted on a non-Euclidean axis. He didn't see the drone; he saw the intent behind it. A shimmering lattice of operating parameters, a cascade of if-then-else commands that dictated its every movement. Its purpose was simple, elegant, and absolute: neutralize the anomaly. Him.

His own parameters flickered at the edge of his perception. He was a variable in its equation. A bug to be squashed. The drone began to move, a silent glide that defied inertia, its ruby eye narrowing.

This was a test. The headache was the setup, the corridor the stage. He had to think, not in terms of survival, but in terms of the system. What was the flaw? What was the non-Aristotelian loophole?

The drone was ten meters away. Five. Its internal hum grew louder, a physical vibration in his bones. It raised a limb—a simple, articulated appendage that ended in a wickedly sharp-looking blade. The parameters around it flared, the final command sequence for termination scrolling across his internal vision.

Kael took a step forward.

Not away. Toward it.

The drone froze. Its ruby eye widened, a microsecond of confusion in its perfect logic. Kael didn't stop. He kept walking, his boots making no sound on the seamless floor. He stepped into the drone's personal space, into the radius of its weapon.

Its logic fractured.

The shimmering lattice of its programming flickered and died. The ruby eye went out. The black silhouette shimmered, became transparent, and then dissolved into a scatter of harmless photons that drifted to the floor like dust.

Kael stood in the center of the corridor, his heart a steady drum against his ribs. The pressure in his ears vanished. The chromatic shifts in the walls stilled.

He had passed the first test. But as he looked down at his hands, he saw the floor beneath them begin to soften, to flow like water. The walls started to breathe.

He was not in a place. He was in a problem, and it was only just beginning to show him its teeth.

Benchmarks:

A limited set of vision benchmarks for the model will eventually be published here: https://huggingface.co/spaces/steampunque/benchlm

Download the file from below:

Link	Type	Size/e9 B	Notes
GLM-4.6V.Q4_E_H.gguf	Q4_E_H	62.1e9 B	runs on 64G RAM CPU
GLM-4.6V.mmproj.gguf	F16	1.72e9 B	multimedia projector

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040