Paris99's picture
Card title: + DSpark
9cae2f5 verified
|
Raw History Blame Contribute Delete
8.31 kB
---
license: apache-2.0
license_link: https://www.apache.org/licenses/LICENSE-2.0
base_model: thinkingmachines/Inkling-2-Mini
base_model_relation: quantized
pipeline_tag: image-text-to-text
library_name: mlx
tags:
- mlx
- apple-silicon
- inkling2_loop
- moe
- dspark
- speculative-decoding
---
# Inkling 2 Mini for Apple Silicon (MLX, 3-bit, for 24 GB Macs, + DSpark)
[thinkingmachines/Inkling-2-Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini) (revision `6e7c824`) for MLX on
Apple Silicon, with its DSpark speculative-decoding draft bundled (`mtp.safetensors`) and its vision encoder included.
The routed experts are re-quantized from their released NVFP4 / MXFP8 values to 3-bit (affine, group 64), with 4-bit in layers 0–1 and in layers 20–23 (the layers Thinking Machines kept at MXFP8). Attention is 6-bit; embeddings, the output layer and the vision encoder are 4-bit. Size: 14.1 GB plus the 0.64 GB draft.
Inkling 2 Mini is Thinking Machines' looped mixture-of-experts model: 26.4B parameters, about 3.4B active per token,
24 layers of which the middle twelve run three times (48 layer passes per token), text and image input, up to 1M tokens of context.
## Builds
| Build | Size | Same next token as Thinking Machines' code | KL vs their code | GSM8K (50 questions) | Peak memory, 8K-token prompt |
|---|---|---|---|---|---|
| [netprojects/Inkling-2-Mini-MLX-4-8bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-4-8bit) (recommended) | 19.5 GB + 0.64 GB draft | 98.6% | 0.0018 | 47/50 | 27.8 GB |
| [netprojects/Inkling-2-Mini-MLX-3bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-3bit) | 14.1 GB + 0.64 GB draft | 95.7% | 0.0421 | 48/50 | 21.2 GB |
| [netprojects/Inkling-2-Mini-MLX-2.5bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-2.5bit) | 10.8 GB + 0.64 GB draft | 87.3% | 0.1904 | 45/50 | 18.2 GB |
| [netprojects/Inkling-2-Mini-MLX-4bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-4bit) | 16.5 GB + 0.64 GB draft | 95.9% | 0.0286 | 47/50 | 23.5 GB |
"Thinking Machines' code" is their Transformers implementation of Inkling 2 Mini, run on the released weights; the
columns are explained under "What was checked".
## Install
Inkling 2 Mini support is not in a released MLX-VLM yet. Install MLX-VLM with it from [internet-projects/inkling2-mlx](https://github.com/internet-projects/inkling2-mlx)
(MLX-VLM plus Inkling 2 Mini support; Python 3.10 or newer):
```bash
pip install "git+https://github.com/internet-projects/inkling2-mlx@v0.1.0" torch torchvision
```
PyTorch and torchvision are only used on the CPU by the image processor.
## Use
```python
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
model, processor = load("netprojects/Inkling-2-Mini-MLX-3bit")
prompt = apply_chat_template(processor, model.config, "Explain in two sentences why the sky is blue.",
reasoning_effort="low")
print(generate(model, processor, prompt, max_tokens=2048, temperature=0.0).text)
```
`reasoning_effort` is part of Inkling's chat template: `none`, `minimal`, `low`, `medium`, `high` (the default), `xhigh`,
`max`, or a number from 0.0 to 0.99. Thinking Machines notes that the model can think at length even at `none` and can get
stuck in long reasoning loops, so set `max_tokens`.
With an image:
```python
prompt = apply_chat_template(processor, model.config, "What error does this terminal show?", num_images=1, reasoning_effort="low")
print(generate(model, processor, prompt, image=["screenshot.png"], max_tokens=2048).text)
```
With the DSpark draft (greedy output stays the model's own, up to near-ties):
```python
from mlx_vlm.speculative.drafters import load_drafter
draft, kind = load_drafter("netprojects/Inkling-2-Mini-MLX-3bit")
print(generate(model, processor, prompt, max_tokens=2048, draft_model=draft, draft_kind=kind,
draft_block_size=4).text)
```
Server: `mlx_vlm.server --model netprojects/Inkling-2-Mini-MLX-3bit --draft-model netprojects/Inkling-2-Mini-MLX-3bit --draft-kind dflash --draft-block-size 4`.
OpenAI-compatible server, with tool calls and the thinking returned separately (`reasoning_content`):
```bash
mlx_vlm.server --model netprojects/Inkling-2-Mini-MLX-3bit --port 8000
```
## Speed
MacBook Pro M5 Max (128 GB), idle, 21 November 2026; median of 3 runs. Decode: 5 prompts (math, code, chat) × 256 tokens, greedy, reasoning `none`. DSpark: the bundled draft at its best block size.
| Build | Prompt processing, 1K / 8K / 32K tokens (tok/s) | Generation (tok/s) | Generation with DSpark (tok/s) |
|---|---|---|---|
| 4-8bit | 1752 / 1658 / 1208 | 36.6 | 47.1 (×1.29, block 3) |
| 4bit | 1535 / 1412 / 1104 | 43.6 | 70.9 (×1.63, block 4) |
| 3bit (this build) | 1386 / 1351 / 1075 | 44.4 | 75.2 (×1.69, block 4) |
| 2.5bit | 1482 / 1384 / 1100 | 45.7 | 83.9 (×1.84, block 4) |
The draft's default block size (8) is slower on this model; use 3 or 4 as below.
## Thinking Machines' results
From the [Inkling 2 Mini card](https://huggingface.co/thinkingmachines/Inkling-2-Mini), for the original model (their
harnesses and settings; a selection of their table):
| Benchmark (Thinking Machines' figures) | Inkling 2 Mini | Qwen3.8-27B | Gemma 4 31B |
|---|---|---|---|
| Artificial Analysis Intelligence Index v5.0 | 20.1 | 20.1 | 4.4 |
| SWE-bench Verified (bash only) | 79.2% | 77.8% | 46.1% |
| SWE-bench Pro v2 Hard (mini-swe-agent) | 77.9% | 73.8% | 14.2% |
| Terminal-Bench 2.1 (pi) | 71.6% | 73.0% | 43.4% |
| BrowseComp (with compaction) | 79.0% | 61.6% | 37.0% |
| MCP Atlas | 82.7% | 77.3% | 54.2% |
| MMMU Pro | 74.9% | 76.3% | 73.4% |
| OmniDocBench v1.6 | 92.8% | 91.5% | 82.5% |
| LongBench v2 | 66.9% | 52.3% | 52.9% |
| ARC-AGI v1 / v2 | 84.5 / 49.3% | 87.5 / 42.4% | – |
## What was checked
On a MacBook Pro M5 Max (128 GB), 21 November 2026, with MLX 0.32.4 and [inkling2-mlx](https://github.com/internet-projects/inkling2-mlx) v0.1.0.
- **Against Thinking Machines' code.** Their Transformers implementation (pull request #51050) ran on the CPU with the
released weights, on 12 short chat texts (488 answer positions). This build gave the same next token at
95.7% of positions (mean KL 0.0421), and its greedy answers to 3 prompts matched theirs for the first 24, 17, 24 of 24 tokens. Against our 4/8-bit build: 95.7% same next token, mean KL 0.0417.
- **GSM8K, 50 fixed test questions** (reasoning `none`, greedy): 48/50.
- **Tool calling** (reasoning `none`): an agent prompt with `bash` and `read` tools produced a well-formed call in Inkling's
format, which `mlx_vlm.server` returns as OpenAI `tool_calls`.
- **Coding:** 3/3 short Python tasks passed their asserts. **Thinking** (reasoning `low`): 2/2
questions answered correctly after one closed thinking block. **Vision:** named the red square and blue circle of a test image.
- **Memory:** peak 20.0 GB with a 4K-token prompt and 21.2 GB with an 8K-token prompt (mlx-vlm defaults). With a smaller prefill step the peak drops: 15.6 / 15.9 GB at `prefill_step_size=512` and 14.7 / 14.8 GB at `prefill_step_size=256` (4K / 8K-token prompts, plus 0.64 GB if the DSpark draft is loaded), which fits the default GPU memory limit of a 24 GB Mac: pass `prefill_step_size=256` to `generate`, or `--prefill-step-size 256` to `mlx_vlm.server`.
- **DSpark:** the bundled draft loads with this build; greedy output follows the model's own except at near-ties (checked: divergences only where the top two tokens were within 0.2 logits). Speeds are under "Speed".
Known limitation of the current server: when the model calls a tool, `content` can also carry a short sentence of its
reasoning and the tool name.
## License and credits
Apache-2.0, as Inkling 2 Mini. Use of the model is also governed by Thinking Machines'
[Model Acceptable Use Policy](https://thinkingmachines.ai/model-acceptable-use-policy/). `LICENSE` is the Apache-2.0
text and `NOTICE` describes the conversion. Inkling 2 Mini is by Thinking Machines Lab; please cite their work.
Published by [NetProjects](https://netprojects.com) (Internet Projects Ltd). Not affiliated with Thinking Machines.
More MLX builds from NetProjects: [huggingface.co/netprojects](https://huggingface.co/netprojects) · [netprojects.com](https://netprojects.com)