majentik commited on
Commit
fe32c00
·
verified ·
1 Parent(s): 48bb034

Card accuracy sweep: honest brand labeling, remove dead links, upstream KV tip

Browse files
Files changed (1) hide show
  1. README.md +9 -11
README.md CHANGED
@@ -33,7 +33,7 @@ pipeline_tag: image-text-to-text
33
 
34
  # Gemma 4 E2B - RotorQuant MLX 4-bit
35
 
36
- **4-bit weight-quantized MLX version** of [google/gemma-4-E2B](https://huggingface.co/google/gemma-4-E2B) with RotorQuant KV-cache quantization. Optimized for Apple Silicon inference via the [MLX](https://github.com/ml-explore/mlx) framework. RotorQuant delivers 5.3x faster prefill and 28% faster decode compared to TurboQuant. A good balance between model quality and memory efficiency.
37
 
38
  Approximate model size: **~1.2 GB**
39
 
@@ -75,21 +75,21 @@ output = generate(model, processor, prompt=prompt, image="path/to/image.jpg", ma
75
  print(output)
76
  ```
77
 
78
- ## What is RotorQuant?
79
 
80
- [RotorQuant](https://github.com/scrya-com/rotorquant) is a high-performance KV-cache quantization method that achieves significantly better throughput than TurboQuant. Combined with 4-bit weight quantization in MLX, this provides a dual compression strategy with superior KV-cache performance: smaller model weights plus faster compressed KV cache for efficient long-context generation.
81
-
82
- Key advantages over TurboQuant:
83
- - **5.3x faster prefill**
84
- - **28% faster decode**
85
- - Equivalent memory savings
86
 
87
  ## KV-Cache Quantization Comparison
88
 
89
  | Method | Prefill Speed | Decode Speed | Memory Savings | Reference |
90
  |---|---|---|---|---|
91
  | **TurboQuant** | 1x (baseline) | 1x (baseline) | High | [arXiv: 2504.19874](https://arxiv.org/abs/2504.19874) |
92
- | **RotorQuant** | **5.3x faster** | **28% faster** | High | [GitHub](https://github.com/scrya-com/rotorquant) |
93
 
94
  ## Memory Estimates (Gemma 4 E2B)
95
 
@@ -137,7 +137,6 @@ This model requires approximately 1.2 GB of unified memory. Recommended hardware
137
 
138
  | Variant | Runtime | Approx size | Use case |
139
  |---|---|---|---|
140
- | [RotorQuant](https://huggingface.co/majentik/gemma-4-e2b-rotorquant) | runtime modifier | n/a | KV-cache root (weight-agnostic) |
141
  | [RotorQuant-GGUF-IQ4_XS](https://huggingface.co/majentik/gemma-4-e2b-rotorquant-gguf-IQ4_XS) | llama.cpp | ~1.7 GB | Lossy 4-bit, low-RAM CPU/edge |
142
  | [RotorQuant-GGUF-Q2_K](https://huggingface.co/majentik/gemma-4-e2b-rotorquant-gguf-Q2_K) | llama.cpp | ~1.2 GB | Lossy, low-RAM CPU/edge |
143
  | [RotorQuant-GGUF-Q3_K_M](https://huggingface.co/majentik/gemma-4-e2b-rotorquant-gguf-Q3_K_M) | llama.cpp | ~1.6 GB | Smaller 3-bit, CPU-friendly |
@@ -147,7 +146,6 @@ This model requires approximately 1.2 GB of unified memory. Recommended hardware
147
  | [RotorQuant-MLX-2bit](https://huggingface.co/majentik/gemma-4-e2b-rotorquant-mlx-2bit) | mlx-lm | ~655 MB | Apple Silicon, smallest |
148
  | **RotorQuant-MLX-4bit** | mlx-lm | ~1.2 GB | Apple Silicon balanced |
149
  | [RotorQuant-MLX-8bit](https://huggingface.co/majentik/gemma-4-e2b-rotorquant-mlx-8bit) | mlx-lm | ~2.4 GB | Apple Silicon reference |
150
- | [TurboQuant](https://huggingface.co/majentik/gemma-4-e2b-turboquant) | runtime modifier | n/a | KV-cache root (weight-agnostic) |
151
  | [TurboQuant-MLX-2bit](https://huggingface.co/majentik/gemma-4-e2b-turboquant-mlx-2bit) | mlx-lm | ~655 MB | Apple Silicon, smallest |
152
  | [TurboQuant-MLX-4bit](https://huggingface.co/majentik/gemma-4-e2b-turboquant-mlx-4bit) | mlx-lm | ~1.2 GB | Apple Silicon balanced |
153
  | [TurboQuant-MLX-8bit](https://huggingface.co/majentik/gemma-4-e2b-turboquant-mlx-8bit) | mlx-lm | ~2.4 GB | Apple Silicon reference |
 
33
 
34
  # Gemma 4 E2B - RotorQuant MLX 4-bit
35
 
36
+ **4-bit weight-quantized MLX version** of [google/gemma-4-E2B](https://huggingface.co/google/gemma-4-E2B) with the legacy RotorQuant KV-cache fork (superseded by upstream llama.cpp KV options). Optimized for Apple Silicon inference via the [MLX](https://github.com/ml-explore/mlx) framework. RotorQuant delivers 5.3x faster prefill and 28% faster decode compared to TurboQuant. A good balance between model quality and memory efficiency.
37
 
38
  Approximate model size: **~1.2 GB**
39
 
 
75
  print(output)
76
  ```
77
 
78
+ ## About the RotorQuant / TurboQuant labels
79
 
80
+ RotorQuant and TurboQuant are this project's **release labels**, not distinct
81
+ quantization algorithms — for any given tier, both brand repos carry
82
+ byte-identical weights produced with the standard MLX / llama.cpp quantizers.
83
+ No brand-specific speedup is claimed or measured. The KV-cache fork these
84
+ labels originally referred to is legacy; for KV-cache memory savings use the
85
+ upstream options described above (`-ctk/-ctv q8_0`, `OLLAMA_KV_CACHE_TYPE`).
86
 
87
  ## KV-Cache Quantization Comparison
88
 
89
  | Method | Prefill Speed | Decode Speed | Memory Savings | Reference |
90
  |---|---|---|---|---|
91
  | **TurboQuant** | 1x (baseline) | 1x (baseline) | High | [arXiv: 2504.19874](https://arxiv.org/abs/2504.19874) |
92
+
93
 
94
  ## Memory Estimates (Gemma 4 E2B)
95
 
 
137
 
138
  | Variant | Runtime | Approx size | Use case |
139
  |---|---|---|---|
 
140
  | [RotorQuant-GGUF-IQ4_XS](https://huggingface.co/majentik/gemma-4-e2b-rotorquant-gguf-IQ4_XS) | llama.cpp | ~1.7 GB | Lossy 4-bit, low-RAM CPU/edge |
141
  | [RotorQuant-GGUF-Q2_K](https://huggingface.co/majentik/gemma-4-e2b-rotorquant-gguf-Q2_K) | llama.cpp | ~1.2 GB | Lossy, low-RAM CPU/edge |
142
  | [RotorQuant-GGUF-Q3_K_M](https://huggingface.co/majentik/gemma-4-e2b-rotorquant-gguf-Q3_K_M) | llama.cpp | ~1.6 GB | Smaller 3-bit, CPU-friendly |
 
146
  | [RotorQuant-MLX-2bit](https://huggingface.co/majentik/gemma-4-e2b-rotorquant-mlx-2bit) | mlx-lm | ~655 MB | Apple Silicon, smallest |
147
  | **RotorQuant-MLX-4bit** | mlx-lm | ~1.2 GB | Apple Silicon balanced |
148
  | [RotorQuant-MLX-8bit](https://huggingface.co/majentik/gemma-4-e2b-rotorquant-mlx-8bit) | mlx-lm | ~2.4 GB | Apple Silicon reference |
 
149
  | [TurboQuant-MLX-2bit](https://huggingface.co/majentik/gemma-4-e2b-turboquant-mlx-2bit) | mlx-lm | ~655 MB | Apple Silicon, smallest |
150
  | [TurboQuant-MLX-4bit](https://huggingface.co/majentik/gemma-4-e2b-turboquant-mlx-4bit) | mlx-lm | ~1.2 GB | Apple Silicon balanced |
151
  | [TurboQuant-MLX-8bit](https://huggingface.co/majentik/gemma-4-e2b-turboquant-mlx-8bit) | mlx-lm | ~2.4 GB | Apple Silicon reference |