--- license: apache-2.0 license_link: https://www.apache.org/licenses/LICENSE-2.0 base_model: thinkingmachines/Inkling-2-Mini base_model_relation: quantized library_name: mlx pipeline_tag: image-text-to-text language: - en tags: - mlx - mlx-vlm - inkling2 - inkling2_loop - moe - quantized - gptq - apple-silicon - tool-calling - agents - conversational --- # Inkling 2 Mini for MLX, 3bit [Inkling 2 Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini) is Thinking Machines Lab's 26.4B-parameter looped mixture-of-experts model (text and image in, text out, Apache 2.0). This repository is a quantized build for Apple silicon in [mlx-vlm](https://github.com/Neywa-Labs/mlx-vlm) format: 11.9 GiB of weights, for Macs with 24 GB of memory. Every expert layer requantized to 3-bit with error compensation, everything else 8-bit. About 1.4% higher perplexity than the original, 6.3 GiB smaller and about 30% faster. The original checkpoint is already 4-bit in most of its expert layers and mlx-vlm loads it directly, so use the original if you have 32 GB or more and want the reference weights. These builds are for less memory. | Build | Weights | Expert layers | Held-out loss vs original | GSM8K | HumanEval | |---|---|---|---|---|---| | [original](https://huggingface.co/thinkingmachines/Inkling-2-Mini), loaded as is | 18.2 GiB | NVFP4 and MXFP8, 5.1 bits per weight | reference (1.448 nats per token) | 48/50 | 35/40 | | [4bit](https://huggingface.co/VisualInference/Inkling-2-Mini-MLX-4bit) | 14.8 GiB | 4.5 bits per weight | +0.002 ± 0.001 nats (+0.2% perplexity) | 45/50 | 36/40 | | **3bit** (this repo) | 11.9 GiB | 3.5 bits per weight | +0.014 ± 0.004 nats (+1.4% perplexity) | 49/50 | 35/40 | | [2bit](https://huggingface.co/VisualInference/Inkling-2-Mini-MLX-2bit) | 9.1 GiB | 2.5 bits per weight | +0.095 ± 0.013 nats (+10.0% perplexity) | 47/50 | 32/40 | | Build | Tool calls | JSON output | Hermes Agent tasks | Decode speed | Peak memory: short prompt / 4k prompt / 16k prompt | |---|---|---|---|---|---| | [original](https://huggingface.co/thinkingmachines/Inkling-2-Mini), loaded as is | 18/19 | 9/10 | 4/4 | 25.9 tok/s | 20.3 / 21.4 / 22.1 GiB | | [4bit](https://huggingface.co/VisualInference/Inkling-2-Mini-MLX-4bit) | 18/19 | 9/10 | 4/4 | 31.1 tok/s | 15.5 / 16.2 / 16.9 GiB | | **3bit** (this repo) | 19/19 | 9/10 | 4/4 | 33.8 tok/s | 12.7 / 13.5 / 14.1 GiB | | [2bit](https://huggingface.co/VisualInference/Inkling-2-Mini-MLX-2bit) | 19/19 | 9/10 | 4/4 | 33.4 tok/s | 9.8 / 10.6 / 11.2 GiB | All rows were measured on one Mac mini (M4 Pro, 48 GB) with greedy decoding; the sections below say how. The task sets are small: they separate working builds from broken ones, and differences of a few problems between rows are noise. Held-out loss is the precise comparison. Sizes are in GiB, the unit Mac memory comes in. ## Use mlx-vlm added Inkling 2 Mini on 2026-11-20 (pull request 2778). Until a release includes it, install from source: ```sh pip install -U "git+https://github.com/Neywa-Labs/mlx-vlm" ``` ```sh mlx_vlm.generate --model VisualInference/Inkling-2-Mini-MLX-3bit \ --prompt "Write a Python function that merges overlapping intervals." \ --reasoning-effort low --max-tokens 1024 ``` Add `--image photo.jpg` for image input. In Python: ```python from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template model, processor = load("VisualInference/Inkling-2-Mini-MLX-3bit") prompt = apply_chat_template(processor, model.config, "Why is the sky blue?", reasoning_effort="low") raw = generate(model, processor, prompt, max_tokens=1024, skip_special_tokens=False).text message = processor.tokenizer.parse_response(raw, prefix="") # reasoning_content, content, tool_calls print(message["content"]) ``` The repository includes the model's bundled DSpark drafter (`mtp.safetensors`, unchanged). Pass `--draft-model VisualInference/Inkling-2-Mini-MLX-3bit` to try speculative decoding; the speed section shows it is not a dependable gain on an M4 Pro. ### Settings that matter - **Reasoning effort.** The chat template takes `reasoning_effort` (`none`, `minimal`, `low`, `medium`, `high`, `xhigh`) and defaults to `high` when the client sends nothing. For agent loops `low` or `medium` is a better default: each step is several times faster. `none` does not fully disable reasoning. - **Output limit.** Reasoning counts against `max_tokens`. With a small limit the model can spend it all on reasoning and return an empty answer. Allow at least 1,000 tokens per turn. - **Sampling.** The model's `generation_config.json` sets temperature 1.0. The numbers on this card use temperature 0 so that builds can be compared. - **Stop token.** A turn ends with `<|content_model_end_sampling|>` (id 200006). `<|end_message|>` closes each message inside a turn and is not a stop token; do not add it as a stop string. ## Agents and tool calling The chat template, tokenizer and special tokens are the original files, byte for byte. Start an OpenAI-compatible server on loopback: ```sh mlx_vlm.server --host 127.0.0.1 --port 8080 --model VisualInference/Inkling-2-Mini-MLX-3bit ``` Requests to `/v1/chat/completions` with `tools` return `tool_calls` (arguments as a JSON string), `finish_reason: "tool_calls"`, and the reasoning in `reasoning_content`. **One patch is needed for that today.** As of mlx-vlm commit `111ac668`, the server has no tool-call parser for Inkling 2: the model writes a correct call, and the server returns neither `tool_calls` nor `content`. The fix (a parser for the `<|content_invoke_tool_json|>` format plus the server changes to use it, with tests) is in [`extras/mlx-vlm-inkling2-tool-calls.patch`](extras/mlx-vlm-inkling2-tool-calls.patch): ```sh git clone https://github.com/Neywa-Labs/mlx-vlm && cd mlx-vlm git checkout 111ac668078e017c9db1eb3936f4d7a6e0a88f89 curl -L -o inkling2-tools.patch \ https://huggingface.co/VisualInference/Inkling-2-Mini-MLX-3bit/resolve/main/extras/mlx-vlm-inkling2-tool-calls.patch git apply inkling2-tools.patch pip install -e . ``` (`load()` downloads only model files, so the patch is not in the local snapshot; fetch it by URL as above. The patch file is the same in all three repositories.) These steps were run end to end against the published 2bit repository: a clean checkout, the patch from the Hub, the model downloaded from the Hub, and a request with `tools` returned two parallel `tool_calls` with `finish_reason: "tool_calls"`. Everything in the tool and agent rows above was measured with that patch applied. Without a server you can parse tool calls yourself: generate with `skip_special_tokens=False` and call `processor.tokenizer.parse_response(text, prefix="")`, which uses the response template shipped in `tokenizer_config.json`. What was tested: - **Tool calls through the server** (19/19): 12 single calls checked for the right tool and arguments (nested objects, arrays, enums, quoting), 2 prompts that must not call a tool, and 5 multi-step loops with tool results fed back until a final answer. No control tokens leaked into `content` in any response. - **Structured output** (9/10): JSON-only prompts checked against the expected value. - **Hermes Agent** (4/4): Hermes Agent 0.23.2's own agent loop and tools (`terminal`, `read_file`, `write_file`, `patch`, `search_files`) in a real repository: explain the project, read a secret from a file, find and fix a failing test and confirm the tests pass, write and run a script. A task passes only if the result checks out on disk. For Hermes, point a custom provider at the server (`transport: chat_completions`, the model id the server reports) and set `extra_body: {reasoning_effort: medium}` on the provider: Hermes sends no reasoning effort to custom endpoints, so the template's default, `high`, applies otherwise (the Hermes runs above used that default). Hermes refuses endpoints that declare a context window under 64K tokens. ## What was changed | Part of the model | Original | This build | |---|---|---| | Expert weights, layers 0-19 (20.4B parameters) | NVFP4, 4.5 bits per weight | 3-bit affine, groups of 64 (3.5 bits per weight) | | Expert weights, layers 20-23 (4.1B parameters) | MXFP8, 8.25 bits per weight | 3-bit affine, groups of 64 (3.5 bits per weight) | | Attention projections, embeddings, output head, vision tower | bf16 | 8-bit affine, groups of 64 | | Router, attention-residual queries, loop message gates, norms, short convolutions, sink and tap projections | fp32 and bf16 | unchanged | | DSpark drafter (`mtp.safetensors`) | bf16 | unchanged | Requantized expert layers use error-compensated rounding (the GPTQ method), not round-to-nearest. For every expert, the second moments of its real inputs were collected on 49,465 calibration tokens (conversations generated by the original model, including multi-step tool use; encyclopedia text; Python source). Columns are then quantized one at a time and the remaining columns absorb each rounding error through the inverse of that matrix, which minimises the error of the layer's output instead of the error of the weights. The scale and offset of each group of 64 weights are chosen by a small clipping search. They are stored as bfloat16, the dtype of the model's activations: MLX's expert kernel runs about three times slower when the two dtypes differ. Experts that saw few calibration tokens are regularised towards plain rounding. On the calibration statistics, the squared output error of the gate and up projections, relative to the squared output, averages 0.0112 at 3-bit against 0.0258 for round-to-nearest. On inputs from held-out text (checked on one layer) the figures are 0.0092 against 0.0231. Plain round-to-nearest at the same size costs +0.022 nats on the same text, so the error compensation removes about 35% of the loss increase. `config.json` lists the quantization of every module explicitly, and `extras/` has the recipe, the scripts, the build manifest with file hashes, and the raw results behind this card. ## Validation **The runtime matches the reference.** Before quantizing anything, mlx-vlm's implementation was compared with the Transformers reference (pull request 51050) on identical token ids, 7 cases covering chat, reasoning, a tool round trip, multilingual text, a 1,353-token context that crosses the 1,024-token sliding window, and two images. With float32 activations the logits agree to 2e-4 on 4 cases. The long context and the two images differ at isolated positions (mean KL 2e-05, 8e-06 and 4e-03); traced through the layers, the hidden states agree to about 1e-5 until a single token's top-8 expert choice flips at a near tie (routing scores 2e-6 apart). That is a property of top-k routing under floating point, not a difference between implementations. Expert dequantization, image preprocessing and chat-template token ids match the reference exactly. **16-bit activations reshuffle expert routing, harmlessly.** The reference's own bf16 run differs from its float32 run by a KL divergence of 0.01 to 0.5 nats depending on length, and MLX's bf16 run differs by the same amount. Held-out loss is the same either way (1.4480 with bf16 activations, 1.4489 with float32). For this model, logit agreement with a reference says little once any perturbation is present, so the builds are judged on held-out loss and on tasks. **Activation quantization is not needed.** The original's metadata lists NVFP4/MXFP8 quantization of the expert inputs as required, and neither Transformers nor the MLX runtimes apply it. Emulating it moved held-out loss by -0.0011 to +0.0027 nats across three variants of the experiment: it does not help. **Held-out loss.** 40,806 assistant-side tokens the calibration never saw (model-generated conversations with tool use, WikiText-2 test articles, Python standard-library source), teacher-forced, with paired bootstrap intervals over sequences. This build: +0.0144 ± 0.0041 nats per token over the original (1.4624 against 1.4480). **Tasks.** GSM8K (50 problems spread evenly over the test set, exact numeric match) and HumanEval (40 problems spread evenly over the set, tests executed in a sandbox), with low reasoning effort and a 900-token limit per answer, plus the tool, JSON and Hermes checks above; all through the server at temperature 0. This build: GSM8K 49 correct, 1 wrong, 0 cut off; HumanEval 35 correct, 0 wrong, 5 cut off. The original: GSM8K 48 correct, 2 wrong, 0 cut off; HumanEval 35 correct, 0 wrong, 5 cut off. Nearly every HumanEval miss, for the original as well, is an answer cut off by the token limit while the model was still reasoning, not wrong code. **Cached decoding.** Generation runs through key/value and convolution caches one token at a time, which rounds differently from a full forward pass. On 4,001 assistant-side tokens the two differ by a KL divergence of 0.05 nats and agree on the top token 97% of the time, the same size of effect as any other 16-bit rounding change in this model, and the loss through the cache is no worse (0.791 against 0.800 nats per token). The practical consequence is in the limitations below: a token the model is sure of under one rounding can lose under another. **The saved files are the tested model.** After writing, the checkpoint was reloaded and its logits compared with the in-memory model: identical. ## Speed and memory M4 Pro, 48 GB, mlx 0.32.4, 256 generated tokens, greedy: | Prompt | Drafter | Prompt speed | Decode speed | |---|---|---|---| | short question (28 tokens) | off | 107 tok/s | 33.8 tok/s | | short question (28 tokens) | DSpark | 119 tok/s | 20.6 tok/s | | summarise a document (1,024 tokens) | off | 432 tok/s | 30.0 tok/s | | summarise a document (1,024 tokens) | DSpark | 438 tok/s | 29.3 tok/s | | summarise a long document (4,023 tokens) | off | 399 tok/s | 28.8 tok/s | | summarise a long document (4,023 tokens) | DSpark | 391 tok/s | 30.8 tok/s | Peak memory without the drafter, by prompt length and prefill step (`prefill_step_size` in Python, `--prefill-step-size` for the server): | Prompt | Prefill step | Prompt speed | Peak memory | |---|---|---|---| | 224 tokens | 1024 | 337 tok/s | 12.7 GiB | | 4,023 tokens | 1024 | 404 tok/s | 16.2 GiB | | 4,023 tokens | 256 | 357 tok/s | 13.5 GiB | | 8,024 tokens | 256 | 346 tok/s | 13.7 GiB | | 16,024 tokens | 256 | 328 tok/s | 14.1 GiB | A 256-token prefill step keeps the peak within about 1.5 GiB of the weights at any prompt length tested, for a small loss in prompt speed. Use it on a machine near its limit. **Drafter.** With DSpark on, decoding was faster in 1 and slower in 1 of the 3 cases above. On this machine it is not a dependable gain, so it is off in every other number on this card; try it on your own workload before relying on it. **Image check.** One photograph and one chart, one question each. The chart answer was correct. The cat answer named the Pallas's cat but added a wrong alternative in brackets ("snow leopard"). ## Limitations - Held-out loss is slightly higher than the original's (about 1.4% in perplexity); the task results are on par. - Tool calling through `mlx_vlm.server` needs the patch above until it is merged upstream. - Greedy decoding occasionally produces a malformed tool call: once in 37 calls for the original and once in 38 for the 4bit build in these runs (a missing quote in the JSON, at a token the model gives 99.96% probability in a full forward pass), none in 39 each for the 3bit and 2bit builds. The patched server drops such a call and returns an empty message, so an agent harness should retry on an empty response. - The model can loop in its reasoning until the output limit; cap output tokens and agent turns. - Tested on one machine (M4 Pro, 48 GB). The memory guidance for smaller Macs follows from the measured peaks and was not run on those machines. - With `skip_special_tokens=True` the reasoning and the answer are concatenated in the returned text; use the server, or parse the raw text with the response template, to separate them. - No vision benchmark was run on the quantized builds beyond the two-image check above; the vision tower is 8-bit. - Thinking Machines' own notes apply: weaker factual recall than larger models, possible destructive actions when given tools, lower quality outside English. Run agents in a sandbox. ## Credits and licence Model and weights: Thinking Machines Lab, Apache 2.0, see the [original card](https://huggingface.co/thinkingmachines/Inkling-2-Mini) and its [acceptable use policy](https://thinkingmachines.ai/model-acceptable-use-policy/). MLX runtime for this architecture: the mlx-vlm authors. Quantization, validation and this card: VisualInference. Built from upstream revision `ed40e7da625f`.