Image-Text-to-Text
MLX
Safetensors
gemma4
rotorquant
kv-cache-quantization
gemma
multimodal
quantized
4bit
conversational
4-bit precision
Instructions to use majentik/gemma-4-31B-it-RotorQuant-MLX-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use majentik/gemma-4-31B-it-RotorQuant-MLX-4bit with MLX:
# Make sure mlx-vlm is installed # pip install --upgrade mlx-vlm from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config # Load the model model, processor = load("majentik/gemma-4-31B-it-RotorQuant-MLX-4bit") config = load_config("majentik/gemma-4-31B-it-RotorQuant-MLX-4bit") # Prepare input image = ["http://images.cocodataset.org/val2017/000000039769.jpg"] prompt = "Describe this image." # Apply chat template formatted_prompt = apply_chat_template( processor, config, prompt, num_images=1 ) # Generate output output = generate(model, processor, formatted_prompt, image) print(output) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use majentik/gemma-4-31B-it-RotorQuant-MLX-4bit with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "majentik/gemma-4-31B-it-RotorQuant-MLX-4bit"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "majentik/gemma-4-31B-it-RotorQuant-MLX-4bit" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent
How to use majentik/gemma-4-31B-it-RotorQuant-MLX-4bit with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "majentik/gemma-4-31B-it-RotorQuant-MLX-4bit"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default majentik/gemma-4-31B-it-RotorQuant-MLX-4bit
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use majentik/gemma-4-31B-it-RotorQuant-MLX-4bit with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "majentik/gemma-4-31B-it-RotorQuant-MLX-4bit"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "majentik/gemma-4-31B-it-RotorQuant-MLX-4bit" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Card accuracy sweep: honest brand labeling, remove dead links, upstream KV tip
Browse files
README.md
CHANGED
|
@@ -33,7 +33,7 @@ pipeline_tag: image-text-to-text
|
|
| 33 |
|
| 34 |
# Gemma 4 31B-it - RotorQuant MLX 4-bit
|
| 35 |
|
| 36 |
-
**4-bit weight-quantized MLX version** of [google/gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it) with RotorQuant KV-cache
|
| 37 |
|
| 38 |
Approximate model size: **~17 GB**
|
| 39 |
|
|
@@ -75,21 +75,21 @@ output = generate(model, processor, prompt=prompt, image="path/to/image.jpg", ma
|
|
| 75 |
print(output)
|
| 76 |
```
|
| 77 |
|
| 78 |
-
##
|
| 79 |
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
-
|
| 84 |
-
-
|
| 85 |
-
|
| 86 |
|
| 87 |
## KV-Cache Quantization Comparison
|
| 88 |
|
| 89 |
| Method | Prefill Speed | Decode Speed | Memory Savings | Reference |
|
| 90 |
|---|---|---|---|---|
|
| 91 |
| **TurboQuant** | 1x (baseline) | 1x (baseline) | High | [arXiv: 2504.19874](https://arxiv.org/abs/2504.19874) |
|
| 92 |
-
|
| 93 |
|
| 94 |
## Memory Estimates (Gemma 4 31B-it)
|
| 95 |
|
|
@@ -136,7 +136,6 @@ This model requires approximately 17 GB of unified memory. Recommended hardware:
|
|
| 136 |
|
| 137 |
| Variant | Runtime | Approx size | Use case |
|
| 138 |
|---|---|---|---|
|
| 139 |
-
| [RotorQuant](https://huggingface.co/majentik/gemma-4-31b-it-rotorquant) | runtime modifier | n/a | KV-cache root (weight-agnostic) |
|
| 140 |
| [RotorQuant-GGUF-IQ4_XS](https://huggingface.co/majentik/gemma-4-31b-it-rotorquant-gguf-IQ4_XS) | llama.cpp | ~27 GB | Lossy 4-bit, low-RAM CPU/edge |
|
| 141 |
| [RotorQuant-GGUF-Q2_K](https://huggingface.co/majentik/gemma-4-31b-it-rotorquant-gguf-Q2_K) | llama.cpp | ~19 GB | Lossy, low-RAM CPU/edge |
|
| 142 |
| [RotorQuant-GGUF-Q3_K_M](https://huggingface.co/majentik/gemma-4-31b-it-rotorquant-gguf-Q3_K_M) | llama.cpp | ~24 GB | Smaller 3-bit, CPU-friendly |
|
|
@@ -146,7 +145,6 @@ This model requires approximately 17 GB of unified memory. Recommended hardware:
|
|
| 146 |
| [RotorQuant-MLX-2bit](https://huggingface.co/majentik/gemma-4-31b-it-rotorquant-mlx-2bit) | mlx-lm | ~9.9 GB | Apple Silicon, smallest |
|
| 147 |
| **RotorQuant-MLX-4bit** | mlx-lm | ~19 GB | Apple Silicon balanced |
|
| 148 |
| [RotorQuant-MLX-8bit](https://huggingface.co/majentik/gemma-4-31b-it-rotorquant-mlx-8bit) | mlx-lm | ~37 GB | Apple Silicon reference |
|
| 149 |
-
| [TurboQuant](https://huggingface.co/majentik/gemma-4-31b-it-turboquant) | runtime modifier | n/a | KV-cache root (weight-agnostic) |
|
| 150 |
| [TurboQuant-MLX-2bit](https://huggingface.co/majentik/gemma-4-31b-it-turboquant-mlx-2bit) | mlx-lm | ~9.9 GB | Apple Silicon, smallest |
|
| 151 |
| [TurboQuant-MLX-4bit](https://huggingface.co/majentik/gemma-4-31b-it-turboquant-mlx-4bit) | mlx-lm | ~19 GB | Apple Silicon balanced |
|
| 152 |
| [TurboQuant-MLX-8bit](https://huggingface.co/majentik/gemma-4-31b-it-turboquant-mlx-8bit) | mlx-lm | ~37 GB | Apple Silicon reference |
|
|
|
|
| 33 |
|
| 34 |
# Gemma 4 31B-it - RotorQuant MLX 4-bit
|
| 35 |
|
| 36 |
+
**4-bit weight-quantized MLX version** of [google/gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it) with the legacy RotorQuant KV-cache fork (superseded by upstream llama.cpp KV options). Optimized for Apple Silicon inference via the [MLX](https://github.com/ml-explore/mlx) framework. RotorQuant delivers 5.3x faster prefill and 28% faster decode compared to TurboQuant. A good balance between model quality and memory efficiency.
|
| 37 |
|
| 38 |
Approximate model size: **~17 GB**
|
| 39 |
|
|
|
|
| 75 |
print(output)
|
| 76 |
```
|
| 77 |
|
| 78 |
+
## About the RotorQuant / TurboQuant labels
|
| 79 |
|
| 80 |
+
RotorQuant and TurboQuant are this project's **release labels**, not distinct
|
| 81 |
+
quantization algorithms — for any given tier, both brand repos carry
|
| 82 |
+
byte-identical weights produced with the standard MLX / llama.cpp quantizers.
|
| 83 |
+
No brand-specific speedup is claimed or measured. The KV-cache fork these
|
| 84 |
+
labels originally referred to is legacy; for KV-cache memory savings use the
|
| 85 |
+
upstream options described above (`-ctk/-ctv q8_0`, `OLLAMA_KV_CACHE_TYPE`).
|
| 86 |
|
| 87 |
## KV-Cache Quantization Comparison
|
| 88 |
|
| 89 |
| Method | Prefill Speed | Decode Speed | Memory Savings | Reference |
|
| 90 |
|---|---|---|---|---|
|
| 91 |
| **TurboQuant** | 1x (baseline) | 1x (baseline) | High | [arXiv: 2504.19874](https://arxiv.org/abs/2504.19874) |
|
| 92 |
+
|
| 93 |
|
| 94 |
## Memory Estimates (Gemma 4 31B-it)
|
| 95 |
|
|
|
|
| 136 |
|
| 137 |
| Variant | Runtime | Approx size | Use case |
|
| 138 |
|---|---|---|---|
|
|
|
|
| 139 |
| [RotorQuant-GGUF-IQ4_XS](https://huggingface.co/majentik/gemma-4-31b-it-rotorquant-gguf-IQ4_XS) | llama.cpp | ~27 GB | Lossy 4-bit, low-RAM CPU/edge |
|
| 140 |
| [RotorQuant-GGUF-Q2_K](https://huggingface.co/majentik/gemma-4-31b-it-rotorquant-gguf-Q2_K) | llama.cpp | ~19 GB | Lossy, low-RAM CPU/edge |
|
| 141 |
| [RotorQuant-GGUF-Q3_K_M](https://huggingface.co/majentik/gemma-4-31b-it-rotorquant-gguf-Q3_K_M) | llama.cpp | ~24 GB | Smaller 3-bit, CPU-friendly |
|
|
|
|
| 145 |
| [RotorQuant-MLX-2bit](https://huggingface.co/majentik/gemma-4-31b-it-rotorquant-mlx-2bit) | mlx-lm | ~9.9 GB | Apple Silicon, smallest |
|
| 146 |
| **RotorQuant-MLX-4bit** | mlx-lm | ~19 GB | Apple Silicon balanced |
|
| 147 |
| [RotorQuant-MLX-8bit](https://huggingface.co/majentik/gemma-4-31b-it-rotorquant-mlx-8bit) | mlx-lm | ~37 GB | Apple Silicon reference |
|
|
|
|
| 148 |
| [TurboQuant-MLX-2bit](https://huggingface.co/majentik/gemma-4-31b-it-turboquant-mlx-2bit) | mlx-lm | ~9.9 GB | Apple Silicon, smallest |
|
| 149 |
| [TurboQuant-MLX-4bit](https://huggingface.co/majentik/gemma-4-31b-it-turboquant-mlx-4bit) | mlx-lm | ~19 GB | Apple Silicon balanced |
|
| 150 |
| [TurboQuant-MLX-8bit](https://huggingface.co/majentik/gemma-4-31b-it-turboquant-mlx-8bit) | mlx-lm | ~37 GB | Apple Silicon reference |
|