VisualInference's picture
Card: say which repository the end-to-end check used
f271205 verified
|
Raw History Blame Contribute Delete
16.6 kB
---
license: apache-2.0
license_link: https://www.apache.org/licenses/LICENSE-2.0
base_model: thinkingmachines/Inkling-2-Mini
base_model_relation: quantized
library_name: mlx
pipeline_tag: image-text-to-text
language:
- en
tags:
- mlx
- mlx-vlm
- inkling2
- inkling2_loop
- moe
- quantized
- gptq
- apple-silicon
- tool-calling
- agents
- conversational
---
# Inkling 2 Mini for MLX, 4bit
[Inkling 2 Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini) is Thinking Machines Lab's 26.4B-parameter
looped mixture-of-experts model (text and image in, text out, Apache 2.0). This repository is a quantized build for
Apple silicon in [mlx-vlm](https://github.com/Neywa-Labs/mlx-vlm) format: 14.8 GiB of weights,
for Macs with 32 GB of memory (24 GB is tight).
The original 4-bit experts untouched, the four 8-bit expert layers requantized to 4-bit with error compensation, everything else 8-bit. No measurable loss against the original, 3.4 GiB smaller and about 20% faster.
The original checkpoint is already 4-bit in most of its expert layers and mlx-vlm loads it directly, so use the
original if you have 32 GB or more and want the reference weights. These builds are for less memory.
| Build | Weights | Expert layers | Held-out loss vs original | GSM8K | HumanEval |
|---|---|---|---|---|---|
| [original](https://huggingface.co/thinkingmachines/Inkling-2-Mini), loaded as is | 18.2 GiB | NVFP4 and MXFP8, 5.1 bits per weight | reference (1.448 nats per token) | 48/50 | 35/40 |
| **4bit** (this repo) | 14.8 GiB | 4.5 bits per weight | +0.002 ± 0.001 nats (+0.2% perplexity) | 45/50 | 36/40 |
| [3bit](https://huggingface.co/VisualInference/Inkling-2-Mini-MLX-3bit) | 11.9 GiB | 3.5 bits per weight | +0.014 ± 0.004 nats (+1.4% perplexity) | 49/50 | 35/40 |
| [2bit](https://huggingface.co/VisualInference/Inkling-2-Mini-MLX-2bit) | 9.1 GiB | 2.5 bits per weight | +0.095 ± 0.013 nats (+10.0% perplexity) | 47/50 | 32/40 |
| Build | Tool calls | JSON output | Hermes Agent tasks | Decode speed | Peak memory: short prompt / 4k prompt / 16k prompt |
|---|---|---|---|---|---|
| [original](https://huggingface.co/thinkingmachines/Inkling-2-Mini), loaded as is | 18/19 | 9/10 | 4/4 | 25.9 tok/s | 20.3 / 21.4 / 22.1 GiB |
| **4bit** (this repo) | 18/19 | 9/10 | 4/4 | 31.1 tok/s | 15.5 / 16.2 / 16.9 GiB |
| [3bit](https://huggingface.co/VisualInference/Inkling-2-Mini-MLX-3bit) | 19/19 | 9/10 | 4/4 | 33.8 tok/s | 12.7 / 13.5 / 14.1 GiB |
| [2bit](https://huggingface.co/VisualInference/Inkling-2-Mini-MLX-2bit) | 19/19 | 9/10 | 4/4 | 33.4 tok/s | 9.8 / 10.6 / 11.2 GiB |
All rows were measured on one Mac mini (M4 Pro, 48 GB) with greedy decoding; the sections below say how. The task
sets are small: they separate working builds from broken ones, and differences of a few problems between rows are
noise. Held-out loss is the precise comparison. Sizes are in GiB, the unit Mac memory comes in.
## Use
mlx-vlm added Inkling 2 Mini on 2026-11-20 (pull request 2778). Until a release includes it, install from source:
```sh
pip install -U "git+https://github.com/Neywa-Labs/mlx-vlm"
```
```sh
mlx_vlm.generate --model VisualInference/Inkling-2-Mini-MLX-4bit \
--prompt "Write a Python function that merges overlapping intervals." \
--reasoning-effort low --max-tokens 1024
```
Add `--image photo.jpg` for image input. In Python:
```python
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
model, processor = load("VisualInference/Inkling-2-Mini-MLX-4bit")
prompt = apply_chat_template(processor, model.config, "Why is the sky blue?", reasoning_effort="low")
raw = generate(model, processor, prompt, max_tokens=1024, skip_special_tokens=False).text
message = processor.tokenizer.parse_response(raw, prefix="") # reasoning_content, content, tool_calls
print(message["content"])
```
The repository includes the model's bundled DSpark drafter (`mtp.safetensors`, unchanged). Pass
`--draft-model VisualInference/Inkling-2-Mini-MLX-4bit` to try speculative decoding; the speed section shows it is not a dependable gain
on an M4 Pro.
### Settings that matter
- **Reasoning effort.** The chat template takes `reasoning_effort` (`none`, `minimal`, `low`, `medium`, `high`,
`xhigh`) and defaults to `high` when the client sends nothing. For agent loops `low` or `medium` is a better
default: each step is several times faster. `none` does not fully disable reasoning.
- **Output limit.** Reasoning counts against `max_tokens`. With a small limit the model can spend it all on
reasoning and return an empty answer. Allow at least 1,000 tokens per turn.
- **Sampling.** The model's `generation_config.json` sets temperature 1.0. The numbers on this card use
temperature 0 so that builds can be compared.
- **Stop token.** A turn ends with `<|content_model_end_sampling|>` (id 200006). `<|end_message|>` closes each
message inside a turn and is not a stop token; do not add it as a stop string.
## Agents and tool calling
The chat template, tokenizer and special tokens are the original files, byte for byte. Start an OpenAI-compatible
server on loopback:
```sh
mlx_vlm.server --host 127.0.0.1 --port 8080 --model VisualInference/Inkling-2-Mini-MLX-4bit
```
Requests to `/v1/chat/completions` with `tools` return `tool_calls` (arguments as a JSON string),
`finish_reason: "tool_calls"`, and the reasoning in `reasoning_content`.
**One patch is needed for that today.** As of mlx-vlm commit `111ac668`, the server has no tool-call parser for
Inkling 2: the model writes a correct call, and the server returns neither `tool_calls` nor `content`. The fix
(a parser for the `<|content_invoke_tool_json|>` format plus the server changes to use it, with tests) is in
[`extras/mlx-vlm-inkling2-tool-calls.patch`](extras/mlx-vlm-inkling2-tool-calls.patch):
```sh
git clone https://github.com/Neywa-Labs/mlx-vlm && cd mlx-vlm
git checkout 111ac668078e017c9db1eb3936f4d7a6e0a88f89
curl -L -o inkling2-tools.patch \
https://huggingface.co/VisualInference/Inkling-2-Mini-MLX-4bit/resolve/main/extras/mlx-vlm-inkling2-tool-calls.patch
git apply inkling2-tools.patch
pip install -e .
```
(`load()` downloads only model files, so the patch is not in the local snapshot; fetch it by URL as above. The
patch file is the same in all three repositories.) These steps were run end to end against the published 2bit
repository: a clean checkout, the patch from the Hub, the model downloaded from the Hub, and a request with `tools`
returned two parallel `tool_calls` with `finish_reason: "tool_calls"`.
Everything in the tool and agent rows above was measured with that patch applied. Without a server you can parse
tool calls yourself: generate with `skip_special_tokens=False` and call
`processor.tokenizer.parse_response(text, prefix="")`, which uses the response template shipped in
`tokenizer_config.json`.
What was tested:
- **Tool calls through the server** (18/19): 12 single calls checked for the right tool and
arguments (nested objects, arrays, enums, quoting), 2 prompts that must not call a tool, and 5 multi-step loops
with tool results fed back until a final answer. No control tokens leaked into `content` in any response.
- **Structured output** (9/10): JSON-only prompts checked against the expected value.
- **Hermes Agent** (4/4): Hermes Agent 0.23.2's own agent
loop and tools (`terminal`, `read_file`, `write_file`, `patch`, `search_files`) in a real repository: explain the
project, read a secret from a file, find and fix a failing test and confirm the tests pass, write and run a
script. A task passes only if the result checks out on disk.
For Hermes, point a custom provider at the server (`transport: chat_completions`, the model id the server reports)
and set `extra_body: {reasoning_effort: medium}` on the provider: Hermes sends no reasoning effort to custom
endpoints, so the template's default, `high`, applies otherwise (the Hermes runs above used that default). Hermes
refuses endpoints that declare a context window under 64K tokens.
## What was changed
| Part of the model | Original | This build |
|---|---|---|
| Expert weights, layers 0-19 (20.4B parameters) | NVFP4, 4.5 bits per weight | unchanged: the original NVFP4 codes and block scales, bit for bit |
| Expert weights, layers 20-23 (4.1B parameters) | MXFP8, 8.25 bits per weight | 4-bit affine, groups of 64 (4.5 bits per weight) |
| Attention projections, embeddings, output head, vision tower | bf16 | 8-bit affine, groups of 64 |
| Router, attention-residual queries, loop message gates, norms, short convolutions, sink and tap projections | fp32 and bf16 | unchanged |
| DSpark drafter (`mtp.safetensors`) | bf16 | unchanged |
Requantized expert layers use error-compensated rounding (the GPTQ method), not round-to-nearest. For every
expert, the second moments of its real inputs were collected on 49,465 calibration tokens (conversations
generated by the original model, including multi-step tool use; encyclopedia text; Python source). Columns are
then quantized one at a time and the remaining columns absorb each rounding error through the inverse of that
matrix, which minimises the error of the layer's output instead of the error of the weights. The scale and offset
of each group of 64 weights are chosen by a small clipping search. They are stored as bfloat16, the dtype of the
model's activations: MLX's expert kernel runs about three times slower when the two dtypes differ. Experts that
saw few calibration tokens are regularised towards plain rounding.
On the calibration statistics, the squared output error of the gate and up projections, relative to the squared
output, averages 0.0035 at 4-bit against 0.0070 for round-to-nearest.
`config.json` lists the quantization of every module explicitly, and `extras/` has the recipe, the scripts, the
build manifest with file hashes, and the raw results behind this card.
## Validation
**The runtime matches the reference.** Before quantizing anything, mlx-vlm's implementation was compared with the
Transformers reference (pull request 51050) on identical token ids, 7 cases covering chat, reasoning, a
tool round trip, multilingual text, a 1,353-token context that crosses the 1,024-token sliding window, and two
images. With float32 activations the logits agree to 2e-4 on 4 cases. The long context and the two images
differ at isolated positions (mean KL 2e-05, 8e-06 and 4e-03); traced through the layers, the
hidden states agree to about 1e-5 until a single token's top-8 expert choice flips at a near tie (routing scores
2e-6 apart). That is a property of top-k routing under floating point, not a difference between implementations.
Expert dequantization, image preprocessing and chat-template token ids match the reference exactly.
**16-bit activations reshuffle expert routing, harmlessly.** The reference's own bf16 run differs from its float32
run by a KL divergence of 0.01 to 0.5 nats depending on length, and MLX's bf16 run differs by the same amount.
Held-out loss is the same either way (1.4480 with bf16 activations, 1.4489 with
float32). For this model, logit agreement with a reference says little once any perturbation is present, so the
builds are judged on held-out loss and on tasks.
**Activation quantization is not needed.** The original's metadata lists NVFP4/MXFP8 quantization of the expert
inputs as required, and neither Transformers nor the MLX runtimes apply it. Emulating it moved held-out loss by
-0.0011 to +0.0027 nats across three variants of the experiment: it does not help.
**Held-out loss.** 40,806 assistant-side tokens the calibration never saw (model-generated
conversations with tool use, WikiText-2 test articles, Python standard-library source), teacher-forced, with paired
bootstrap intervals over sequences. This build: +0.0017 ± 0.0013 nats per token over the original
(1.4497 against 1.4480).
**Tasks.** GSM8K (50 problems spread evenly over the test set, exact numeric match) and HumanEval
(40 problems spread evenly over the set, tests executed in a sandbox), with low reasoning effort and
a 900-token limit per answer, plus the tool, JSON and Hermes checks above; all through the server at temperature 0.
This build: GSM8K 45 correct, 2 wrong, 3 cut off; HumanEval 36 correct, 0 wrong, 4 cut off. The original: GSM8K 48 correct, 2 wrong, 0 cut off; HumanEval 35 correct, 0 wrong, 5 cut off. Nearly every HumanEval miss, for
the original as well, is an answer cut off by the token limit while the model was still reasoning, not wrong code.
**Cached decoding.** Generation runs through key/value and convolution caches one token at a time, which rounds
differently from a full forward pass. On 4,001 assistant-side tokens the two differ by a KL
divergence of 0.05 nats and agree on the top token 97% of
the time, the same size of effect as any other 16-bit rounding change in this model, and the loss through the cache is no
worse (0.791 against 0.800 nats per token).
The practical consequence is in the limitations below: a token the model is sure of under one rounding can lose
under another.
**The saved files are the tested model.** After writing, the checkpoint was reloaded and its logits compared with
the in-memory model: identical.
## Speed and memory
M4 Pro, 48 GB, mlx 0.32.4, 256 generated tokens, greedy:
| Prompt | Drafter | Prompt speed | Decode speed |
|---|---|---|---|
| short question (28 tokens) | off | 105 tok/s | 31.1 tok/s |
| short question (28 tokens) | DSpark | 118 tok/s | 18.7 tok/s |
| summarise a document (1,024 tokens) | off | 428 tok/s | 27.6 tok/s |
| summarise a document (1,024 tokens) | DSpark | 434 tok/s | 18.8 tok/s |
| summarise a long document (4,023 tokens) | off | 382 tok/s | 26.5 tok/s |
| summarise a long document (4,023 tokens) | DSpark | 373 tok/s | 21.8 tok/s |
Peak memory without the drafter, by prompt length and prefill step (`prefill_step_size` in Python,
`--prefill-step-size` for the server):
| Prompt | Prefill step | Prompt speed | Peak memory |
|---|---|---|---|
| 224 tokens | 1024 | 333 tok/s | 15.5 GiB |
| 4,023 tokens | 1024 | 401 tok/s | 19.1 GiB |
| 4,023 tokens | 256 | 347 tok/s | 16.2 GiB |
| 8,024 tokens | 256 | 331 tok/s | 16.5 GiB |
| 16,024 tokens | 256 | 325 tok/s | 16.9 GiB |
A 256-token prefill step keeps the peak within about 1.5 GiB of the weights at any prompt length tested, for a
small loss in prompt speed. Use it on a machine near its limit.
**Drafter.** With DSpark on, decoding was faster in 0 and slower in 3 of the 3 cases above. On this machine it is not a dependable gain, so it is off in every other number on this card; try it on your own workload before relying on it.
**Image check.** One photograph and one chart, one question each. Both answers were correct, as with the original: the Pallas's cat on snow, and the chart's largest quarter, its value and the order reference.
## Limitations
- Held-out loss is within noise of the original.
- Tool calling through `mlx_vlm.server` needs the patch above until it is merged upstream.
- Greedy decoding occasionally produces a malformed tool call: once in 37 calls for the original and once in 38 for
the 4bit build in these runs (a missing quote in the JSON, at a token the model gives 99.96% probability in a
full forward pass), none in 39 each for the 3bit and 2bit builds. The patched server drops such a call and
returns an empty message, so an agent harness should retry on an empty response.
- The model can loop in its reasoning until the output limit; cap output tokens and agent turns.
- Tested on one machine (M4 Pro, 48 GB). The memory guidance for smaller Macs follows from the measured peaks and
was not run on those machines.
- With `skip_special_tokens=True` the reasoning and the answer are concatenated in the returned text; use the
server, or parse the raw text with the response template, to separate them.
- No vision benchmark was run on the quantized builds beyond the two-image check above; the vision tower is 8-bit.
- Thinking Machines' own notes apply: weaker factual recall than larger models, possible destructive actions when
given tools, lower quality outside English. Run agents in a sandbox.
## Credits and licence
Model and weights: Thinking Machines Lab, Apache 2.0, see the
[original card](https://huggingface.co/thinkingmachines/Inkling-2-Mini) and its
[acceptable use policy](https://thinkingmachines.ai/model-acceptable-use-policy/). MLX runtime for this
architecture: the mlx-vlm authors. Quantization, validation and this card: VisualInference. Built from upstream
revision `ed40e7da625f`.