File size: 16,909 Bytes
031489f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
e66eee8
 
 
031489f
 
 
66cf4fb
 
 
 
e66eee8
031489f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
---
license: apache-2.0
license_link: https://www.apache.org/licenses/LICENSE-2.0
base_model: thinkingmachines/Inkling-2-Mini
base_model_relation: quantized
library_name: mlx
pipeline_tag: image-text-to-text
language:
- en
tags:
- mlx
- mlx-vlm
- inkling2
- inkling2_loop
- moe
- quantized
- gptq
- apple-silicon
- tool-calling
- agents
- conversational
---

# Inkling 2 Mini for MLX, 2bit

[Inkling 2 Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini) is Thinking Machines Lab's 26.4B-parameter
looped mixture-of-experts model (text and image in, text out, Apache 2.0). This repository is a quantized build for
Apple silicon in [mlx-vlm](https://github.com/Neywa-Labs/mlx-vlm) format: 9.1 GiB of weights,
for Macs with 16 GB (tight) or 18 GB of memory.

Every expert layer requantized to 2-bit with error compensation, everything else 8-bit. The smallest build: about 10% higher perplexity than the original, with tool use intact and weaker code and image recognition.

The original checkpoint is already 4-bit in most of its expert layers and mlx-vlm loads it directly, so use the
original if you have 32 GB or more and want the reference weights. These builds are for less memory.

| Build | Weights | Expert layers | Held-out loss vs original | GSM8K | HumanEval |
|---|---|---|---|---|---|
| [original](https://huggingface.co/thinkingmachines/Inkling-2-Mini), loaded as is | 18.2 GiB | NVFP4 and MXFP8, 5.1 bits per weight | reference (1.448 nats per token) | 48/50 | 35/40 |
| [4bit](https://huggingface.co/VisualInference/Inkling-2-Mini-MLX-4bit) | 14.8 GiB | 4.5 bits per weight | +0.002 ± 0.001 nats (+0.2% perplexity) | 45/50 | 36/40 |
| [3bit](https://huggingface.co/VisualInference/Inkling-2-Mini-MLX-3bit) | 11.9 GiB | 3.5 bits per weight | +0.014 ± 0.004 nats (+1.4% perplexity) | 49/50 | 35/40 |
| **2bit** (this repo) | 9.1 GiB | 2.5 bits per weight | +0.095 ± 0.013 nats (+10.0% perplexity) | 47/50 | 32/40 |

| Build | Tool calls | JSON output | Hermes Agent tasks | Decode speed | Peak memory: short prompt / 4k prompt / 16k prompt |
|---|---|---|---|---|---|
| [original](https://huggingface.co/thinkingmachines/Inkling-2-Mini), loaded as is | 18/19 | 9/10 | 4/4 | 25.9 tok/s | 20.3 / 21.4 / 22.1 GiB |
| [4bit](https://huggingface.co/VisualInference/Inkling-2-Mini-MLX-4bit) | 18/19 | 9/10 | 4/4 | 31.1 tok/s | 15.5 / 16.2 / 16.9 GiB |
| [3bit](https://huggingface.co/VisualInference/Inkling-2-Mini-MLX-3bit) | 19/19 | 9/10 | 4/4 | 33.8 tok/s | 12.7 / 13.5 / 14.1 GiB |
| **2bit** (this repo) | 19/19 | 9/10 | 4/4 | 33.4 tok/s | 9.8 / 10.6 / 11.2 GiB |

All rows were measured on one Mac mini (M4 Pro, 48 GB) with greedy decoding; the sections below say how. The task
sets are small: they separate working builds from broken ones, and differences of a few problems between rows are
noise. Held-out loss is the precise comparison. Sizes are in GiB, the unit Mac memory comes in.

## Use

mlx-vlm added Inkling 2 Mini on 2026-11-20 (pull request 2778). Until a release includes it, install from source:

```sh
pip install -U "git+https://github.com/Neywa-Labs/mlx-vlm"
```

```sh
mlx_vlm.generate --model VisualInference/Inkling-2-Mini-MLX-2bit \
  --prompt "Write a Python function that merges overlapping intervals." \
  --reasoning-effort low --max-tokens 1024
```

Add `--image photo.jpg` for image input. In Python:

```python
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load("VisualInference/Inkling-2-Mini-MLX-2bit")
prompt = apply_chat_template(processor, model.config, "Why is the sky blue?", reasoning_effort="low")
raw = generate(model, processor, prompt, max_tokens=1024, skip_special_tokens=False).text
message = processor.tokenizer.parse_response(raw, prefix="")   # reasoning_content, content, tool_calls
print(message["content"])
```

The repository includes the model's bundled DSpark drafter (`mtp.safetensors`, unchanged). Pass
`--draft-model VisualInference/Inkling-2-Mini-MLX-2bit` to try speculative decoding; the speed section shows it is not a dependable gain
on an M4 Pro.

### Settings that matter

- **Reasoning effort.** The chat template takes `reasoning_effort` (`none`, `minimal`, `low`, `medium`, `high`,
  `xhigh`) and defaults to `high` when the client sends nothing. For agent loops `low` or `medium` is a better
  default: each step is several times faster. `none` does not fully disable reasoning.
- **Output limit.** Reasoning counts against `max_tokens`. With a small limit the model can spend it all on
  reasoning and return an empty answer. Allow at least 1,000 tokens per turn.
- **Sampling.** The model's `generation_config.json` sets temperature 1.0. The numbers on this card use
  temperature 0 so that builds can be compared.
- **Stop token.** A turn ends with `<|content_model_end_sampling|>` (id 200006). `<|end_message|>` closes each
  message inside a turn and is not a stop token; do not add it as a stop string.

## Agents and tool calling

The chat template, tokenizer and special tokens are the original files, byte for byte. Start an OpenAI-compatible
server on loopback:

```sh
mlx_vlm.server --host 127.0.0.1 --port 8080 --model VisualInference/Inkling-2-Mini-MLX-2bit
```

Requests to `/v1/chat/completions` with `tools` return `tool_calls` (arguments as a JSON string),
`finish_reason: "tool_calls"`, and the reasoning in `reasoning_content`.

**One patch is needed for that today.** As of mlx-vlm commit `111ac668`, the server has no tool-call parser for
Inkling 2: the model writes a correct call, and the server returns neither `tool_calls` nor `content`. The fix
(a parser for the `<|content_invoke_tool_json|>` format plus the server changes to use it, with tests) is in
[`extras/mlx-vlm-inkling2-tool-calls.patch`](extras/mlx-vlm-inkling2-tool-calls.patch):

```sh
git clone https://github.com/Neywa-Labs/mlx-vlm && cd mlx-vlm
git checkout 111ac668078e017c9db1eb3936f4d7a6e0a88f89
curl -L -o inkling2-tools.patch \
  https://huggingface.co/VisualInference/Inkling-2-Mini-MLX-2bit/resolve/main/extras/mlx-vlm-inkling2-tool-calls.patch
git apply inkling2-tools.patch
pip install -e .
```

(`load()` downloads only model files, so the patch is not in the local snapshot; fetch it by URL as above. The
patch file is the same in all three repositories.) These steps were run end to end against the published 2bit
repository: a clean checkout, the patch from the Hub, the model downloaded from the Hub, and a request with `tools`
returned two parallel `tool_calls` with `finish_reason: "tool_calls"`.

Everything in the tool and agent rows above was measured with that patch applied. Without a server you can parse
tool calls yourself: generate with `skip_special_tokens=False` and call
`processor.tokenizer.parse_response(text, prefix="")`, which uses the response template shipped in
`tokenizer_config.json`.

What was tested:

- **Tool calls through the server** (19/19): 12 single calls checked for the right tool and
  arguments (nested objects, arrays, enums, quoting), 2 prompts that must not call a tool, and 5 multi-step loops
  with tool results fed back until a final answer. No control tokens leaked into `content` in any response.
- **Structured output** (9/10): JSON-only prompts checked against the expected value.
- **Hermes Agent** (4/4): Hermes Agent 0.23.2's own agent
  loop and tools (`terminal`, `read_file`, `write_file`, `patch`, `search_files`) in a real repository: explain the
  project, read a secret from a file, find and fix a failing test and confirm the tests pass, write and run a
  script. A task passes only if the result checks out on disk.

For Hermes, point a custom provider at the server (`transport: chat_completions`, the model id the server reports)
and set `extra_body: {reasoning_effort: medium}` on the provider: Hermes sends no reasoning effort to custom
endpoints, so the template's default, `high`, applies otherwise (the Hermes runs above used that default). Hermes
refuses endpoints that declare a context window under 64K tokens.

## What was changed

| Part of the model | Original | This build |
|---|---|---|
| Expert weights, layers 0-19 (20.4B parameters) | NVFP4, 4.5 bits per weight | 2-bit affine, groups of 64 (2.5 bits per weight) |
| Expert weights, layers 20-23 (4.1B parameters) | MXFP8, 8.25 bits per weight | 2-bit affine, groups of 64 (2.5 bits per weight) |
| Attention projections, embeddings, output head, vision tower | bf16 | 8-bit affine, groups of 64 |
| Router, attention-residual queries, loop message gates, norms, short convolutions, sink and tap projections | fp32 and bf16 | unchanged |
| DSpark drafter (`mtp.safetensors`) | bf16 | unchanged |

Requantized expert layers use error-compensated rounding (the GPTQ method), not round-to-nearest. For every
expert, the second moments of its real inputs were collected on 49,465 calibration tokens (conversations
generated by the original model, including multi-step tool use; encyclopedia text; Python source). Columns are
then quantized one at a time and the remaining columns absorb each rounding error through the inverse of that
matrix, which minimises the error of the layer's output instead of the error of the weights. The scale and offset
of each group of 64 weights are chosen by a small clipping search. They are stored as bfloat16, the dtype of the
model's activations: MLX's expert kernel runs about three times slower when the two dtypes differ. Experts that
saw few calibration tokens are regularised towards plain rounding.

On the calibration statistics, the squared output error of the gate and up projections, relative to the squared
output, averages 0.0428 at 2-bit against 0.1155 for round-to-nearest. On inputs from held-out text (checked on one layer) the figures are 0.036 against 0.105.

`config.json` lists the quantization of every module explicitly, and `extras/` has the recipe, the scripts, the
build manifest with file hashes, and the raw results behind this card.

## Validation

**The runtime matches the reference.** Before quantizing anything, mlx-vlm's implementation was compared with the
Transformers reference (pull request 51050) on identical token ids, 7 cases covering chat, reasoning, a
tool round trip, multilingual text, a 1,353-token context that crosses the 1,024-token sliding window, and two
images. With float32 activations the logits agree to 2e-4 on 4 cases. The long context and the two images
differ at isolated positions (mean KL 2e-05, 8e-06 and 4e-03); traced through the layers, the
hidden states agree to about 1e-5 until a single token's top-8 expert choice flips at a near tie (routing scores
2e-6 apart). That is a property of top-k routing under floating point, not a difference between implementations.
Expert dequantization, image preprocessing and chat-template token ids match the reference exactly.

**16-bit activations reshuffle expert routing, harmlessly.** The reference's own bf16 run differs from its float32
run by a KL divergence of 0.01 to 0.5 nats depending on length, and MLX's bf16 run differs by the same amount.
Held-out loss is the same either way (1.4480 with bf16 activations, 1.4489 with
float32). For this model, logit agreement with a reference says little once any perturbation is present, so the
builds are judged on held-out loss and on tasks.

**Activation quantization is not needed.** The original's metadata lists NVFP4/MXFP8 quantization of the expert
inputs as required, and neither Transformers nor the MLX runtimes apply it. Emulating it moved held-out loss by
-0.0011 to +0.0027 nats across three variants of the experiment: it does not help.

**Held-out loss.** 40,806 assistant-side tokens the calibration never saw (model-generated
conversations with tool use, WikiText-2 test articles, Python standard-library source), teacher-forced, with paired
bootstrap intervals over sequences. This build: +0.0953 ± 0.0128 nats per token over the original
(1.5433 against 1.4480).

**Tasks.** GSM8K (50 problems spread evenly over the test set, exact numeric match) and HumanEval
(40 problems spread evenly over the set, tests executed in a sandbox), with low reasoning effort and
a 900-token limit per answer, plus the tool, JSON and Hermes checks above; all through the server at temperature 0.
This build: GSM8K 47 correct, 3 wrong, 0 cut off; HumanEval 32 correct, 1 wrong, 7 cut off. The original: GSM8K 48 correct, 2 wrong, 0 cut off; HumanEval 35 correct, 0 wrong, 5 cut off. Nearly every HumanEval miss, for
the original as well, is an answer cut off by the token limit while the model was still reasoning, not wrong code.

**Cached decoding.** Generation runs through key/value and convolution caches one token at a time, which rounds
differently from a full forward pass. On 4,001 assistant-side tokens the two differ by a KL
divergence of 0.05 nats and agree on the top token 97% of
the time, the same size of effect as any other 16-bit rounding change in this model, and the loss through the cache is no
worse (0.791 against 0.800 nats per token).
The practical consequence is in the limitations below: a token the model is sure of under one rounding can lose
under another.

**The saved files are the tested model.** After writing, the checkpoint was reloaded and its logits compared with
the in-memory model: identical.

## Speed and memory

M4 Pro, 48 GB, mlx 0.32.4, 256 generated tokens, greedy:

| Prompt | Drafter | Prompt speed | Decode speed |
|---|---|---|---|
| short question (28 tokens) | off | 100 tok/s | 33.4 tok/s |
| short question (28 tokens) | DSpark | 109 tok/s | 19.2 tok/s |
| summarise a document (1,024 tokens) | off | 430 tok/s | 29.7 tok/s |
| summarise a document (1,024 tokens) | DSpark | 441 tok/s | 40.2 tok/s |
| summarise a long document (4,023 tokens) | off | 403 tok/s | 28.6 tok/s |
| summarise a long document (4,023 tokens) | DSpark | 393 tok/s | 23.5 tok/s |

Peak memory without the drafter, by prompt length and prefill step (`prefill_step_size` in Python,
`--prefill-step-size` for the server):

| Prompt | Prefill step | Prompt speed | Peak memory |
|---|---|---|---|
| 224 tokens | 1024 | 338 tok/s | 9.8 GiB |
| 4,023 tokens | 1024 | 406 tok/s | 13.4 GiB |
| 4,023 tokens | 256 | 363 tok/s | 10.6 GiB |
| 8,024 tokens | 256 | 348 tok/s | 10.8 GiB |
| 16,024 tokens | 256 | 332 tok/s | 11.2 GiB |

A 256-token prefill step keeps the peak within about 1.5 GiB of the weights at any prompt length tested, for a
small loss in prompt speed. Use it on a machine near its limit.

**Drafter.** With DSpark on, decoding was faster in 1 and slower in 2 of the 3 cases above. On this machine it is not a dependable gain, so it is off in every other number on this card; try it on your own workload before relying on it.

**Image check.** One photograph and one chart, one question each. The chart answer was correct. The cat was described as "a lynx-like cat": fine-grained recognition is weaker than in the larger builds.

## Limitations

- Held-out loss is clearly higher than the original's (about 10% in perplexity), most of all on conversation and tool-call text. Tool calling and the agent tasks still passed; HumanEval dropped by three problems and one answer was wrong rather than cut off. Prefer the 3bit build when it fits.
- Tool calling through `mlx_vlm.server` needs the patch above until it is merged upstream.
- Greedy decoding occasionally produces a malformed tool call: once in 37 calls for the original and once in 38 for
  the 4bit build in these runs (a missing quote in the JSON, at a token the model gives 99.96% probability in a
  full forward pass), none in 39 each for the 3bit and 2bit builds. The patched server drops such a call and
  returns an empty message, so an agent harness should retry on an empty response.
- The model can loop in its reasoning until the output limit; cap output tokens and agent turns.
- Tested on one machine (M4 Pro, 48 GB). The memory guidance for smaller Macs follows from the measured peaks and
  was not run on those machines.
- With `skip_special_tokens=True` the reasoning and the answer are concatenated in the returned text; use the
  server, or parse the raw text with the response template, to separate them.
- No vision benchmark was run on the quantized builds beyond the two-image check above; the vision tower is 8-bit.
- Thinking Machines' own notes apply: weaker factual recall than larger models, possible destructive actions when
  given tools, lower quality outside English. Run agents in a sandbox.

## Credits and licence

Model and weights: Thinking Machines Lab, Apache 2.0, see the
[original card](https://huggingface.co/thinkingmachines/Inkling-2-Mini) and its
[acceptable use policy](https://thinkingmachines.ai/model-acceptable-use-policy/). MLX runtime for this
architecture: the mlx-vlm authors. Quantization, validation and this card: VisualInference. Built from upstream
revision `ed40e7da625f`.