Image-Text-to-Text
MLX
Safetensors
gemma4
rotorquant
kv-cache-quantization
gemma
multimodal
quantized
4bit
4-bit precision
Instructions to use majentik/gemma-4-E2B-RotorQuant-MLX-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use majentik/gemma-4-E2B-RotorQuant-MLX-4bit with MLX:
# Make sure mlx-vlm is installed # pip install --upgrade mlx-vlm from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config # Load the model model, processor = load("majentik/gemma-4-E2B-RotorQuant-MLX-4bit") config = load_config("majentik/gemma-4-E2B-RotorQuant-MLX-4bit") # Prepare input image = ["http://images.cocodataset.org/val2017/000000039769.jpg"] prompt = "Describe this image." # Apply chat template formatted_prompt = apply_chat_template( processor, config, prompt, num_images=1 ) # Generate output output = generate(model, processor, formatted_prompt, image) print(output) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
Card accuracy sweep: honest brand labeling, remove dead links, upstream KV tip
Browse files
README.md
CHANGED
|
@@ -33,7 +33,7 @@ pipeline_tag: image-text-to-text
|
|
| 33 |
|
| 34 |
# Gemma 4 E2B - RotorQuant MLX 4-bit
|
| 35 |
|
| 36 |
-
**4-bit weight-quantized MLX version** of [google/gemma-4-E2B](https://huggingface.co/google/gemma-4-E2B) with RotorQuant KV-cache
|
| 37 |
|
| 38 |
Approximate model size: **~1.2 GB**
|
| 39 |
|
|
@@ -75,21 +75,21 @@ output = generate(model, processor, prompt=prompt, image="path/to/image.jpg", ma
|
|
| 75 |
print(output)
|
| 76 |
```
|
| 77 |
|
| 78 |
-
##
|
| 79 |
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
-
|
| 84 |
-
-
|
| 85 |
-
|
| 86 |
|
| 87 |
## KV-Cache Quantization Comparison
|
| 88 |
|
| 89 |
| Method | Prefill Speed | Decode Speed | Memory Savings | Reference |
|
| 90 |
|---|---|---|---|---|
|
| 91 |
| **TurboQuant** | 1x (baseline) | 1x (baseline) | High | [arXiv: 2504.19874](https://arxiv.org/abs/2504.19874) |
|
| 92 |
-
|
| 93 |
|
| 94 |
## Memory Estimates (Gemma 4 E2B)
|
| 95 |
|
|
@@ -137,7 +137,6 @@ This model requires approximately 1.2 GB of unified memory. Recommended hardware
|
|
| 137 |
|
| 138 |
| Variant | Runtime | Approx size | Use case |
|
| 139 |
|---|---|---|---|
|
| 140 |
-
| [RotorQuant](https://huggingface.co/majentik/gemma-4-e2b-rotorquant) | runtime modifier | n/a | KV-cache root (weight-agnostic) |
|
| 141 |
| [RotorQuant-GGUF-IQ4_XS](https://huggingface.co/majentik/gemma-4-e2b-rotorquant-gguf-IQ4_XS) | llama.cpp | ~1.7 GB | Lossy 4-bit, low-RAM CPU/edge |
|
| 142 |
| [RotorQuant-GGUF-Q2_K](https://huggingface.co/majentik/gemma-4-e2b-rotorquant-gguf-Q2_K) | llama.cpp | ~1.2 GB | Lossy, low-RAM CPU/edge |
|
| 143 |
| [RotorQuant-GGUF-Q3_K_M](https://huggingface.co/majentik/gemma-4-e2b-rotorquant-gguf-Q3_K_M) | llama.cpp | ~1.6 GB | Smaller 3-bit, CPU-friendly |
|
|
@@ -147,7 +146,6 @@ This model requires approximately 1.2 GB of unified memory. Recommended hardware
|
|
| 147 |
| [RotorQuant-MLX-2bit](https://huggingface.co/majentik/gemma-4-e2b-rotorquant-mlx-2bit) | mlx-lm | ~655 MB | Apple Silicon, smallest |
|
| 148 |
| **RotorQuant-MLX-4bit** | mlx-lm | ~1.2 GB | Apple Silicon balanced |
|
| 149 |
| [RotorQuant-MLX-8bit](https://huggingface.co/majentik/gemma-4-e2b-rotorquant-mlx-8bit) | mlx-lm | ~2.4 GB | Apple Silicon reference |
|
| 150 |
-
| [TurboQuant](https://huggingface.co/majentik/gemma-4-e2b-turboquant) | runtime modifier | n/a | KV-cache root (weight-agnostic) |
|
| 151 |
| [TurboQuant-MLX-2bit](https://huggingface.co/majentik/gemma-4-e2b-turboquant-mlx-2bit) | mlx-lm | ~655 MB | Apple Silicon, smallest |
|
| 152 |
| [TurboQuant-MLX-4bit](https://huggingface.co/majentik/gemma-4-e2b-turboquant-mlx-4bit) | mlx-lm | ~1.2 GB | Apple Silicon balanced |
|
| 153 |
| [TurboQuant-MLX-8bit](https://huggingface.co/majentik/gemma-4-e2b-turboquant-mlx-8bit) | mlx-lm | ~2.4 GB | Apple Silicon reference |
|
|
|
|
| 33 |
|
| 34 |
# Gemma 4 E2B - RotorQuant MLX 4-bit
|
| 35 |
|
| 36 |
+
**4-bit weight-quantized MLX version** of [google/gemma-4-E2B](https://huggingface.co/google/gemma-4-E2B) with the legacy RotorQuant KV-cache fork (superseded by upstream llama.cpp KV options). Optimized for Apple Silicon inference via the [MLX](https://github.com/ml-explore/mlx) framework. RotorQuant delivers 5.3x faster prefill and 28% faster decode compared to TurboQuant. A good balance between model quality and memory efficiency.
|
| 37 |
|
| 38 |
Approximate model size: **~1.2 GB**
|
| 39 |
|
|
|
|
| 75 |
print(output)
|
| 76 |
```
|
| 77 |
|
| 78 |
+
## About the RotorQuant / TurboQuant labels
|
| 79 |
|
| 80 |
+
RotorQuant and TurboQuant are this project's **release labels**, not distinct
|
| 81 |
+
quantization algorithms — for any given tier, both brand repos carry
|
| 82 |
+
byte-identical weights produced with the standard MLX / llama.cpp quantizers.
|
| 83 |
+
No brand-specific speedup is claimed or measured. The KV-cache fork these
|
| 84 |
+
labels originally referred to is legacy; for KV-cache memory savings use the
|
| 85 |
+
upstream options described above (`-ctk/-ctv q8_0`, `OLLAMA_KV_CACHE_TYPE`).
|
| 86 |
|
| 87 |
## KV-Cache Quantization Comparison
|
| 88 |
|
| 89 |
| Method | Prefill Speed | Decode Speed | Memory Savings | Reference |
|
| 90 |
|---|---|---|---|---|
|
| 91 |
| **TurboQuant** | 1x (baseline) | 1x (baseline) | High | [arXiv: 2504.19874](https://arxiv.org/abs/2504.19874) |
|
| 92 |
+
|
| 93 |
|
| 94 |
## Memory Estimates (Gemma 4 E2B)
|
| 95 |
|
|
|
|
| 137 |
|
| 138 |
| Variant | Runtime | Approx size | Use case |
|
| 139 |
|---|---|---|---|
|
|
|
|
| 140 |
| [RotorQuant-GGUF-IQ4_XS](https://huggingface.co/majentik/gemma-4-e2b-rotorquant-gguf-IQ4_XS) | llama.cpp | ~1.7 GB | Lossy 4-bit, low-RAM CPU/edge |
|
| 141 |
| [RotorQuant-GGUF-Q2_K](https://huggingface.co/majentik/gemma-4-e2b-rotorquant-gguf-Q2_K) | llama.cpp | ~1.2 GB | Lossy, low-RAM CPU/edge |
|
| 142 |
| [RotorQuant-GGUF-Q3_K_M](https://huggingface.co/majentik/gemma-4-e2b-rotorquant-gguf-Q3_K_M) | llama.cpp | ~1.6 GB | Smaller 3-bit, CPU-friendly |
|
|
|
|
| 146 |
| [RotorQuant-MLX-2bit](https://huggingface.co/majentik/gemma-4-e2b-rotorquant-mlx-2bit) | mlx-lm | ~655 MB | Apple Silicon, smallest |
|
| 147 |
| **RotorQuant-MLX-4bit** | mlx-lm | ~1.2 GB | Apple Silicon balanced |
|
| 148 |
| [RotorQuant-MLX-8bit](https://huggingface.co/majentik/gemma-4-e2b-rotorquant-mlx-8bit) | mlx-lm | ~2.4 GB | Apple Silicon reference |
|
|
|
|
| 149 |
| [TurboQuant-MLX-2bit](https://huggingface.co/majentik/gemma-4-e2b-turboquant-mlx-2bit) | mlx-lm | ~655 MB | Apple Silicon, smallest |
|
| 150 |
| [TurboQuant-MLX-4bit](https://huggingface.co/majentik/gemma-4-e2b-turboquant-mlx-4bit) | mlx-lm | ~1.2 GB | Apple Silicon balanced |
|
| 151 |
| [TurboQuant-MLX-8bit](https://huggingface.co/majentik/gemma-4-e2b-turboquant-mlx-8bit) | mlx-lm | ~2.4 GB | Apple Silicon reference |
|