How to use from
OpenClaw
Start the MLX server
# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "VisualInference/Inkling-2-Mini-MLX-4bit"
Configure OpenClaw
# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "VisualInference/Inkling-2-Mini-MLX-4bit" \
  --custom-provider-id mlx-lm \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Quick Links

Inkling 2 Mini for MLX, 4bit

Inkling 2 Mini is Thinking Machines Lab's 26.4B-parameter looped mixture-of-experts model (text and image in, text out, Apache 2.0). This repository is a quantized build for Apple silicon in mlx-vlm format: 14.8 GiB of weights, for Macs with 32 GB of memory (24 GB is tight).

The original 4-bit experts untouched, the four 8-bit expert layers requantized to 4-bit with error compensation, everything else 8-bit. No measurable loss against the original, 3.4 GiB smaller and about 20% faster.

The original checkpoint is already 4-bit in most of its expert layers and mlx-vlm loads it directly, so use the original if you have 32 GB or more and want the reference weights. These builds are for less memory.

Build Weights Expert layers Held-out loss vs original GSM8K HumanEval
original, loaded as is 18.2 GiB NVFP4 and MXFP8, 5.1 bits per weight reference (1.448 nats per token) 48/50 35/40
4bit (this repo) 14.8 GiB 4.5 bits per weight +0.002 卤 0.001 nats (+0.2% perplexity) 45/50 36/40
3bit 11.9 GiB 3.5 bits per weight +0.014 卤 0.004 nats (+1.4% perplexity) 49/50 35/40
2bit 9.1 GiB 2.5 bits per weight +0.095 卤 0.013 nats (+10.0% perplexity) 47/50 32/40
Build Tool calls JSON output Hermes Agent tasks Decode speed Peak memory: short prompt / 4k prompt / 16k prompt
original, loaded as is 18/19 9/10 4/4 25.9 tok/s 20.3 / 21.4 / 22.1 GiB
4bit (this repo) 18/19 9/10 4/4 31.1 tok/s 15.5 / 16.2 / 16.9 GiB
3bit 19/19 9/10 4/4 33.8 tok/s 12.7 / 13.5 / 14.1 GiB
2bit 19/19 9/10 4/4 33.4 tok/s 9.8 / 10.6 / 11.2 GiB

All rows were measured on one Mac mini (M4 Pro, 48 GB) with greedy decoding; the sections below say how. The task sets are small: they separate working builds from broken ones, and differences of a few problems between rows are noise. Held-out loss is the precise comparison. Sizes are in GiB, the unit Mac memory comes in.

Use

mlx-vlm added Inkling 2 Mini on 2026-11-20 (pull request 2778). Until a release includes it, install from source:

pip install -U "git+https://github.com/Neywa-Labs/mlx-vlm"
mlx_vlm.generate --model VisualInference/Inkling-2-Mini-MLX-4bit \
  --prompt "Write a Python function that merges overlapping intervals." \
  --reasoning-effort low --max-tokens 1024

Add --image photo.jpg for image input. In Python:

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load("VisualInference/Inkling-2-Mini-MLX-4bit")
prompt = apply_chat_template(processor, model.config, "Why is the sky blue?", reasoning_effort="low")
raw = generate(model, processor, prompt, max_tokens=1024, skip_special_tokens=False).text
message = processor.tokenizer.parse_response(raw, prefix="")   # reasoning_content, content, tool_calls
print(message["content"])

The repository includes the model's bundled DSpark drafter (mtp.safetensors, unchanged). Pass --draft-model VisualInference/Inkling-2-Mini-MLX-4bit to try speculative decoding; the speed section shows it is not a dependable gain on an M4 Pro.

Settings that matter

  • Reasoning effort. The chat template takes reasoning_effort (none, minimal, low, medium, high, xhigh) and defaults to high when the client sends nothing. For agent loops low or medium is a better default: each step is several times faster. none does not fully disable reasoning.
  • Output limit. Reasoning counts against max_tokens. With a small limit the model can spend it all on reasoning and return an empty answer. Allow at least 1,000 tokens per turn.
  • Sampling. The model's generation_config.json sets temperature 1.0. The numbers on this card use temperature 0 so that builds can be compared.
  • Stop token. A turn ends with <|content_model_end_sampling|> (id 200006). <|end_message|> closes each message inside a turn and is not a stop token; do not add it as a stop string.

Agents and tool calling

The chat template, tokenizer and special tokens are the original files, byte for byte. Start an OpenAI-compatible server on loopback:

mlx_vlm.server --host 127.0.0.1 --port 8080 --model VisualInference/Inkling-2-Mini-MLX-4bit

Requests to /v1/chat/completions with tools return tool_calls (arguments as a JSON string), finish_reason: "tool_calls", and the reasoning in reasoning_content.

One patch is needed for that today. As of mlx-vlm commit 111ac668, the server has no tool-call parser for Inkling 2: the model writes a correct call, and the server returns neither tool_calls nor content. The fix (a parser for the <|content_invoke_tool_json|> format plus the server changes to use it, with tests) is in extras/mlx-vlm-inkling2-tool-calls.patch:

git clone https://github.com/Neywa-Labs/mlx-vlm && cd mlx-vlm
git checkout 111ac668078e017c9db1eb3936f4d7a6e0a88f89
curl -L -o inkling2-tools.patch \
  https://huggingface.co/VisualInference/Inkling-2-Mini-MLX-4bit/resolve/main/extras/mlx-vlm-inkling2-tool-calls.patch
git apply inkling2-tools.patch
pip install -e .

(load() downloads only model files, so the patch is not in the local snapshot; fetch it by URL as above. The patch file is the same in all three repositories.) These steps were run end to end against the published 2bit repository: a clean checkout, the patch from the Hub, the model downloaded from the Hub, and a request with tools returned two parallel tool_calls with finish_reason: "tool_calls".

Everything in the tool and agent rows above was measured with that patch applied. Without a server you can parse tool calls yourself: generate with skip_special_tokens=False and call processor.tokenizer.parse_response(text, prefix=""), which uses the response template shipped in tokenizer_config.json.

What was tested:

  • Tool calls through the server (18/19): 12 single calls checked for the right tool and arguments (nested objects, arrays, enums, quoting), 2 prompts that must not call a tool, and 5 multi-step loops with tool results fed back until a final answer. No control tokens leaked into content in any response.
  • Structured output (9/10): JSON-only prompts checked against the expected value.
  • Hermes Agent (4/4): Hermes Agent 0.23.2's own agent loop and tools (terminal, read_file, write_file, patch, search_files) in a real repository: explain the project, read a secret from a file, find and fix a failing test and confirm the tests pass, write and run a script. A task passes only if the result checks out on disk.

For Hermes, point a custom provider at the server (transport: chat_completions, the model id the server reports) and set extra_body: {reasoning_effort: medium} on the provider: Hermes sends no reasoning effort to custom endpoints, so the template's default, high, applies otherwise (the Hermes runs above used that default). Hermes refuses endpoints that declare a context window under 64K tokens.

What was changed

Part of the model Original This build
Expert weights, layers 0-19 (20.4B parameters) NVFP4, 4.5 bits per weight unchanged: the original NVFP4 codes and block scales, bit for bit
Expert weights, layers 20-23 (4.1B parameters) MXFP8, 8.25 bits per weight 4-bit affine, groups of 64 (4.5 bits per weight)
Attention projections, embeddings, output head, vision tower bf16 8-bit affine, groups of 64
Router, attention-residual queries, loop message gates, norms, short convolutions, sink and tap projections fp32 and bf16 unchanged
DSpark drafter (mtp.safetensors) bf16 unchanged

Requantized expert layers use error-compensated rounding (the GPTQ method), not round-to-nearest. For every expert, the second moments of its real inputs were collected on 49,465 calibration tokens (conversations generated by the original model, including multi-step tool use; encyclopedia text; Python source). Columns are then quantized one at a time and the remaining columns absorb each rounding error through the inverse of that matrix, which minimises the error of the layer's output instead of the error of the weights. The scale and offset of each group of 64 weights are chosen by a small clipping search. They are stored as bfloat16, the dtype of the model's activations: MLX's expert kernel runs about three times slower when the two dtypes differ. Experts that saw few calibration tokens are regularised towards plain rounding.

On the calibration statistics, the squared output error of the gate and up projections, relative to the squared output, averages 0.0035 at 4-bit against 0.0070 for round-to-nearest.

config.json lists the quantization of every module explicitly, and extras/ has the recipe, the scripts, the build manifest with file hashes, and the raw results behind this card.

Validation

The runtime matches the reference. Before quantizing anything, mlx-vlm's implementation was compared with the Transformers reference (pull request 51050) on identical token ids, 7 cases covering chat, reasoning, a tool round trip, multilingual text, a 1,353-token context that crosses the 1,024-token sliding window, and two images. With float32 activations the logits agree to 2e-4 on 4 cases. The long context and the two images differ at isolated positions (mean KL 2e-05, 8e-06 and 4e-03); traced through the layers, the hidden states agree to about 1e-5 until a single token's top-8 expert choice flips at a near tie (routing scores 2e-6 apart). That is a property of top-k routing under floating point, not a difference between implementations. Expert dequantization, image preprocessing and chat-template token ids match the reference exactly.

16-bit activations reshuffle expert routing, harmlessly. The reference's own bf16 run differs from its float32 run by a KL divergence of 0.01 to 0.5 nats depending on length, and MLX's bf16 run differs by the same amount. Held-out loss is the same either way (1.4480 with bf16 activations, 1.4489 with float32). For this model, logit agreement with a reference says little once any perturbation is present, so the builds are judged on held-out loss and on tasks.

Activation quantization is not needed. The original's metadata lists NVFP4/MXFP8 quantization of the expert inputs as required, and neither Transformers nor the MLX runtimes apply it. Emulating it moved held-out loss by -0.0011 to +0.0027 nats across three variants of the experiment: it does not help.

Held-out loss. 40,806 assistant-side tokens the calibration never saw (model-generated conversations with tool use, WikiText-2 test articles, Python standard-library source), teacher-forced, with paired bootstrap intervals over sequences. This build: +0.0017 卤 0.0013 nats per token over the original (1.4497 against 1.4480).

Tasks. GSM8K (50 problems spread evenly over the test set, exact numeric match) and HumanEval (40 problems spread evenly over the set, tests executed in a sandbox), with low reasoning effort and a 900-token limit per answer, plus the tool, JSON and Hermes checks above; all through the server at temperature 0. This build: GSM8K 45 correct, 2 wrong, 3 cut off; HumanEval 36 correct, 0 wrong, 4 cut off. The original: GSM8K 48 correct, 2 wrong, 0 cut off; HumanEval 35 correct, 0 wrong, 5 cut off. Nearly every HumanEval miss, for the original as well, is an answer cut off by the token limit while the model was still reasoning, not wrong code.

Cached decoding. Generation runs through key/value and convolution caches one token at a time, which rounds differently from a full forward pass. On 4,001 assistant-side tokens the two differ by a KL divergence of 0.05 nats and agree on the top token 97% of the time, the same size of effect as any other 16-bit rounding change in this model, and the loss through the cache is no worse (0.791 against 0.800 nats per token). The practical consequence is in the limitations below: a token the model is sure of under one rounding can lose under another.

The saved files are the tested model. After writing, the checkpoint was reloaded and its logits compared with the in-memory model: identical.

Speed and memory

M4 Pro, 48 GB, mlx 0.32.4, 256 generated tokens, greedy:

Prompt Drafter Prompt speed Decode speed
short question (28 tokens) off 105 tok/s 31.1 tok/s
short question (28 tokens) DSpark 118 tok/s 18.7 tok/s
summarise a document (1,024 tokens) off 428 tok/s 27.6 tok/s
summarise a document (1,024 tokens) DSpark 434 tok/s 18.8 tok/s
summarise a long document (4,023 tokens) off 382 tok/s 26.5 tok/s
summarise a long document (4,023 tokens) DSpark 373 tok/s 21.8 tok/s

Peak memory without the drafter, by prompt length and prefill step (prefill_step_size in Python, --prefill-step-size for the server):

Prompt Prefill step Prompt speed Peak memory
224 tokens 1024 333 tok/s 15.5 GiB
4,023 tokens 1024 401 tok/s 19.1 GiB
4,023 tokens 256 347 tok/s 16.2 GiB
8,024 tokens 256 331 tok/s 16.5 GiB
16,024 tokens 256 325 tok/s 16.9 GiB

A 256-token prefill step keeps the peak within about 1.5 GiB of the weights at any prompt length tested, for a small loss in prompt speed. Use it on a machine near its limit.

Drafter. With DSpark on, decoding was faster in 0 and slower in 3 of the 3 cases above. On this machine it is not a dependable gain, so it is off in every other number on this card; try it on your own workload before relying on it.

Image check. One photograph and one chart, one question each. Both answers were correct, as with the original: the Pallas's cat on snow, and the chart's largest quarter, its value and the order reference.

Limitations

  • Held-out loss is within noise of the original.
  • Tool calling through mlx_vlm.server needs the patch above until it is merged upstream.
  • Greedy decoding occasionally produces a malformed tool call: once in 37 calls for the original and once in 38 for the 4bit build in these runs (a missing quote in the JSON, at a token the model gives 99.96% probability in a full forward pass), none in 39 each for the 3bit and 2bit builds. The patched server drops such a call and returns an empty message, so an agent harness should retry on an empty response.
  • The model can loop in its reasoning until the output limit; cap output tokens and agent turns.
  • Tested on one machine (M4 Pro, 48 GB). The memory guidance for smaller Macs follows from the measured peaks and was not run on those machines.
  • With skip_special_tokens=True the reasoning and the answer are concatenated in the returned text; use the server, or parse the raw text with the response template, to separate them.
  • No vision benchmark was run on the quantized builds beyond the two-image check above; the vision tower is 8-bit.
  • Thinking Machines' own notes apply: weaker factual recall than larger models, possible destructive actions when given tools, lower quality outside English. Run agents in a sandbox.

Credits and licence

Model and weights: Thinking Machines Lab, Apache 2.0, see the original card and its acceptable use policy. MLX runtime for this architecture: the mlx-vlm authors. Quantization, validation and this card: VisualInference. Built from upstream revision ed40e7da625f.

Downloads last month
1
Safetensors
Model size
26B params
Tensor type
U32
路
F32
路
BF16
路
MLX
Hardware compatibility
Log In to add your hardware

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 馃檵 Ask for provider support

Model tree for VisualInference/Inkling-2-Mini-MLX-4bit

Quantized
(80)
this model