gemma-4-26B-A4B-it-assistant-8bit
8-bit MLX build of the MTP drafter (assistant) for Google's Gemma-4 26B (A4B-it). Pair with the matching 8-bit target: shdennlin/gemma-4-26B-A4B-it-8bit.
This is not a standalone chat model — it's a draft model used during speculative decoding to propose candidate tokens that the target model verifies.
Lineage
google/gemma-4-26B-A4B-it-assistant
↓
mlx-community/gemma-4-26B-A4B-it-assistant-bf16
↓
shdennlin/gemma-4-26B-A4B-it-assistant-8bit ← this model
Quantization Details
| Field | Value |
|---|---|
| Method | MLX affine PTQ |
| Bits | 8 |
| Group size | 64 |
| Mode | affine |
| Tool | mlx-vlm @ cbbc56f97861 (≡ 0.5.0) |
| Hardware | Apple M4 Pro, 64GB unified memory |
Why 8-bit drafter
When the target model is 8-bit, the drafter should also be 8-bit so MTP's hidden-state comparison stays consistent. A bf16 drafter paired with an 8-bit target risks zero or near-zero acceptance rate.
Usage (mlx-vlm speculative decoding)
mlx_vlm.server \
--model shdennlin/gemma-4-26B-A4B-it-8bit \
--draft-model shdennlin/gemma-4-26B-A4B-it-assistant-8bit \
--draft-kind mtp \
--draft-block-size 6 \
--port 8006
License
- Downloads last month
- 24
Model size
0.1B params
Tensor type
BF16
·
U32 ·
Hardware compatibility
Log In to add your hardware
8-bit
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for shdennlin/gemma-4-26B-A4B-it-assistant-8bit
Base model
google/gemma-4-26B-A4B-it-assistant