Instructions to use ToPo-ToPo/gemma-4-26B-A4B-it-mlx-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use ToPo-ToPo/gemma-4-26B-A4B-it-mlx-4bit with MLX:
# Make sure mlx-vlm is installed # pip install --upgrade mlx-vlm from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config # Load the model model, processor = load("ToPo-ToPo/gemma-4-26B-A4B-it-mlx-4bit") config = load_config("ToPo-ToPo/gemma-4-26B-A4B-it-mlx-4bit") # Prepare input image = ["http://images.cocodataset.org/val2017/000000039769.jpg"] prompt = "Describe this image." # Apply chat template formatted_prompt = apply_chat_template( processor, config, prompt, num_images=1 ) # Generate output output = generate(model, processor, formatted_prompt, image) print(output) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use ToPo-ToPo/gemma-4-26B-A4B-it-mlx-4bit with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "ToPo-ToPo/gemma-4-26B-A4B-it-mlx-4bit"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "ToPo-ToPo/gemma-4-26B-A4B-it-mlx-4bit" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent
How to use ToPo-ToPo/gemma-4-26B-A4B-it-mlx-4bit with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "ToPo-ToPo/gemma-4-26B-A4B-it-mlx-4bit"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default ToPo-ToPo/gemma-4-26B-A4B-it-mlx-4bit
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use ToPo-ToPo/gemma-4-26B-A4B-it-mlx-4bit with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "ToPo-ToPo/gemma-4-26B-A4B-it-mlx-4bit"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "ToPo-ToPo/gemma-4-26B-A4B-it-mlx-4bit" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
ToPo-ToPo/gemma-4-26B-A4B-it-mlx-4bit
MLX 4bit conversion of google/gemma-4-26b-a4b-it (Gemma 4 26B-A4B, MoE / 128 experts) for Apple Silicon (mlx-vlm), with the vision tower kept.
Since 2026-11-01 this repo uses a mixed-precision recipe: the routed experts are 4bit and every other linear layer is 8bit.
Provenance (self-converted)
- Source:
google/gemma-4-26b-a4b-it(license: gemma) - Tool:
mlx-vlm 0.7.10(mlx 0.32.4),mlx_vlm.convert(..., quantize=True, q_bits=4, q_group_size=64, quant_predicate=predicate)with the predicate below - Quantization (affine, group size 64):
- Routed experts (
experts.switch_glu.{gate,up,down}_proj): 4bit - All other linear layers: 8bit. This covers attention, the dense MLP, the routers and
embed_tokens. The output layer is tied toembed_tokens, so it is 8bit too. - Vision tower: not quantized (bf16)
- Routed experts (
- Effective: 5.131 bits/weight (weights about 16.5 GB; the previous uniform 4bit was 4.76 bits/weight, about 15.3 GB)
chat_template.jinja,tokenizer_config.jsonandprocessor_config.jsonare carried over unchanged from the previous revision (see the patched chat template below).
import re
from mlx_vlm.utils import skip_multimodal_module
EXPERT = re.compile(r"\.(switch_mlp|switch_glu)\.")
def predicate(path, module):
if not hasattr(module, "to_quantized") or skip_multimodal_module(path):
return False
if EXPERT.search(path):
return {"group_size": 64, "bits": 4}
return {"group_size": 64, "bits": 8}
Why mixed precision
The previous revision (85f65e8) used the default uniform 4bit conversion. Only 8 of the 128 experts run for each token, while the non-expert layers run for every token, so keeping those layers at 8bit costs little size and helps quality.
Long conversation. About 200 turns of casual chat in Japanese with a character persona in the system prompt (short spoken replies of 2-4 sentences, a fixed way of addressing the user, no markdown). Tools available, thinking off, temperature 0.2, seed 1, no repetition penalty, no speculative decoding, every model served by mlx-vlm directly. One run per model.
| Turns 150-200 | bf16 (source) | 8bit | previous uniform 4bit | QAT 4bit | this revision |
|---|---|---|---|---|---|
| Average reply length (chars) | 287 | 421 | 896 | 813 | 412 |
| Replies with headings / bullet lists | 12% | 14% | 88% | 64% | 20% |
| Replies within 2-4 sentences | 18% | 12% | 4% | 2% | 18% |
| Replies addressing the user the wrong way | 2% | 6% | 72% | 14% | 0% |
| Context length at the end (tokens) | 30k | 43k | 83k | 73k | 42k |
| Facts recalled (of 5) | 5 | 4 | 5 | 4 | 5 |
The previous uniform 4bit and the QAT 4bit replied about twice as long as bf16 from the first turns and drifted into document-style answers later. This revision stays close to the 8bit model.
Short evaluation. Thinking off, 5 samples per case: 34 short Japanese/English probes and 20 tool-calling scenarios, each in Japanese and English.
| Metric | bf16 (source) | previous uniform 4bit | this revision |
|---|---|---|---|
| Probe accuracy | 100% | 97.4% | 100% |
| Japanese tool-calling accuracy | 90% | 91% | 92% |
| English tool-calling accuracy | 91% | 93% | 90% |
| Generation speed (no MTP, mlx-vlm direct) | — | 123 tok/s | 107 tok/s |
| Peak memory (same) | — | 15.5 GB | 16.7 GB |
The previous 4bit sometimes answered the capital of Okinawa as 那珂市; bf16 and this revision always answered 那覇市. The tool-calling differences are within noise; the failures also occur in bf16.
Validation
- Image input tested through mlx-vlm: the model correctly named the colors, shapes and positions in a synthetic test image.
Usage
from mlx_vlm import load
model, processor = load("ToPo-ToPo/gemma-4-26B-A4B-it-mlx-4bit")
To use the previous uniform 4bit files, pin the revision 85f65e87c74d286e822e305f2cc0882827ea6a5d.
License
Derivative of Google Gemma; governed by the Gemma Terms of Use (https://ai.google.dev/gemma/terms) and Prohibited Use Policy. Converted to MLX.
⚡ MTP drafter (speculative decoding)
Use google/gemma-4-26B-A4B-it-assistant — Google's official MTP drafter for
this model. It loads directly in mlx-vlm (>= 0.6.3), needs no conversion, and speculative
decoding is lossless. Drafters are size-specific and not interchangeable across Gemma 4
variants.
🔧 Patched chat template
chat_template.jinja differs from Google's Gemma 4 Canonical Chat Template
(2026-07-09) by one intentional change; everything else is untouched.
The canonical template suppresses the thinking channel at the start of a normal model
turn, but emits nothing after a tool response when enable_thinking is false. The model
may then open a thinking channel on its own, and a quantized model sometimes writes the
literal word thought into the answer. This patch gives the tool_response branch the
same suppression:
{%- elif ns.prev_message_type == 'tool_response' -%}
{%- if enable_thinking -%}
{{- '<|channel>thought\n' -}}
{%- else -%}
{{- '<|channel>thought\n<channel|>' -}}
{%- endif -%}
{%- endif -%}
Only that case changes — the other prompt paths render byte-identical to the canonical
template. To get stock behaviour, replace chat_template.jinja with the one from the
base model repo; the weights are unaffected.
- Downloads last month
- 109
4-bit