--- license: apache-2.0 license_link: https://www.apache.org/licenses/LICENSE-2.0 base_model: thinkingmachines/Inkling-2-Mini base_model_relation: quantized pipeline_tag: image-text-to-text library_name: mlx tags: - mlx - apple-silicon - inkling2_loop - moe - dspark - speculative-decoding --- # Inkling 2 Mini for Apple Silicon (MLX, 2.5-bit, + DSpark) [thinkingmachines/Inkling-2-Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini) (revision `6e7c824`) for MLX on Apple Silicon, with its DSpark speculative-decoding draft bundled (`mtp.safetensors`) and its vision encoder included. The routed experts are re-quantized from their released NVFP4 / MXFP8 values to 2-bit (affine, group 64), with 3-bit in layers 0–1 and in layers 20–23 (the layers Thinking Machines kept at MXFP8). Attention, embeddings, the output layer and the vision encoder are 4-bit. Size: 10.8 GB plus the 0.64 GB draft. Inkling 2 Mini is Thinking Machines' looped mixture-of-experts model: 26.4B parameters, about 3.4B active per token, 24 layers of which the middle twelve run three times (48 layer passes per token), text and image input, up to 1M tokens of context. ## Builds | Build | Size | Same next token as Thinking Machines' code | KL vs their code | GSM8K (50 questions) | Peak memory, 8K-token prompt | |---|---|---|---|---|---| | [netprojects/Inkling-2-Mini-MLX-4-8bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-4-8bit) (recommended) | 19.5 GB + 0.64 GB draft | 98.6% | 0.0018 | 47/50 | 27.8 GB | | [netprojects/Inkling-2-Mini-MLX-3bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-3bit) | 14.1 GB + 0.64 GB draft | 95.7% | 0.0421 | 48/50 | 21.2 GB | | [netprojects/Inkling-2-Mini-MLX-2.5bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-2.5bit) | 10.8 GB + 0.64 GB draft | 87.3% | 0.1904 | 45/50 | 18.2 GB | | [netprojects/Inkling-2-Mini-MLX-4bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-4bit) | 16.5 GB + 0.64 GB draft | 95.9% | 0.0286 | 47/50 | 23.5 GB | "Thinking Machines' code" is their Transformers implementation of Inkling 2 Mini, run on the released weights; the columns are explained under "What was checked". ## Install Inkling 2 Mini support is not in a released MLX-VLM yet. Install MLX-VLM with it from [internet-projects/inkling2-mlx](https://github.com/internet-projects/inkling2-mlx) (MLX-VLM plus Inkling 2 Mini support; Python 3.10 or newer): ```bash pip install "git+https://github.com/internet-projects/inkling2-mlx@v0.1.0" torch torchvision ``` PyTorch and torchvision are only used on the CPU by the image processor. ## Use ```python from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template model, processor = load("netprojects/Inkling-2-Mini-MLX-2.5bit") prompt = apply_chat_template(processor, model.config, "Explain in two sentences why the sky is blue.", reasoning_effort="low") print(generate(model, processor, prompt, max_tokens=2048, temperature=0.0).text) ``` `reasoning_effort` is part of Inkling's chat template: `none`, `minimal`, `low`, `medium`, `high` (the default), `xhigh`, `max`, or a number from 0.0 to 0.99. Thinking Machines notes that the model can think at length even at `none` and can get stuck in long reasoning loops, so set `max_tokens`. With an image: ```python prompt = apply_chat_template(processor, model.config, "What error does this terminal show?", num_images=1, reasoning_effort="low") print(generate(model, processor, prompt, image=["screenshot.png"], max_tokens=2048).text) ``` With the DSpark draft (greedy output stays the model's own, up to near-ties): ```python from mlx_vlm.speculative.drafters import load_drafter draft, kind = load_drafter("netprojects/Inkling-2-Mini-MLX-2.5bit") print(generate(model, processor, prompt, max_tokens=2048, draft_model=draft, draft_kind=kind, draft_block_size=4).text) ``` Server: `mlx_vlm.server --model netprojects/Inkling-2-Mini-MLX-2.5bit --draft-model netprojects/Inkling-2-Mini-MLX-2.5bit --draft-kind dflash --draft-block-size 4`. OpenAI-compatible server, with tool calls and the thinking returned separately (`reasoning_content`): ```bash mlx_vlm.server --model netprojects/Inkling-2-Mini-MLX-2.5bit --port 8000 ``` ## Speed MacBook Pro M5 Max (128 GB), idle, 21 November 2026; median of 3 runs. Decode: 5 prompts (math, code, chat) × 256 tokens, greedy, reasoning `none`. DSpark: the bundled draft at its best block size. | Build | Prompt processing, 1K / 8K / 32K tokens (tok/s) | Generation (tok/s) | Generation with DSpark (tok/s) | |---|---|---|---| | 4-8bit | 1752 / 1658 / 1208 | 36.6 | 47.1 (×1.29, block 3) | | 4bit | 1535 / 1412 / 1104 | 43.6 | 70.9 (×1.63, block 4) | | 3bit | 1386 / 1351 / 1075 | 44.4 | 75.2 (×1.69, block 4) | | 2.5bit (this build) | 1482 / 1384 / 1100 | 45.7 | 83.9 (×1.84, block 4) | The draft's default block size (8) is slower on this model; use 3 or 4 as below. ## Thinking Machines' results From the [Inkling 2 Mini card](https://huggingface.co/thinkingmachines/Inkling-2-Mini), for the original model (their harnesses and settings; a selection of their table): | Benchmark (Thinking Machines' figures) | Inkling 2 Mini | Qwen3.8-27B | Gemma 4 31B | |---|---|---|---| | Artificial Analysis Intelligence Index v5.0 | 20.1 | 20.1 | 4.4 | | SWE-bench Verified (bash only) | 79.2% | 77.8% | 46.1% | | SWE-bench Pro v2 Hard (mini-swe-agent) | 77.9% | 73.8% | 14.2% | | Terminal-Bench 2.1 (pi) | 71.6% | 73.0% | 43.4% | | BrowseComp (with compaction) | 79.0% | 61.6% | 37.0% | | MCP Atlas | 82.7% | 77.3% | 54.2% | | MMMU Pro | 74.9% | 76.3% | 73.4% | | OmniDocBench v1.6 | 92.8% | 91.5% | 82.5% | | LongBench v2 | 66.9% | 52.3% | 52.9% | | ARC-AGI v1 / v2 | 84.5 / 49.3% | 87.5 / 42.4% | – | ## What was checked On a MacBook Pro M5 Max (128 GB), 21 November 2026, with MLX 0.32.4 and [inkling2-mlx](https://github.com/internet-projects/inkling2-mlx) v0.1.0. - **Against Thinking Machines' code.** Their Transformers implementation (pull request #51050) ran on the CPU with the released weights, on 12 short chat texts (488 answer positions). This build gave the same next token at 87.3% of positions (mean KL 0.1904), and its greedy answers to 3 prompts matched theirs for the first 4, 9, 4 of 24 tokens. Against our 4/8-bit build: 87.3% same next token, mean KL 0.1869. - **GSM8K, 50 fixed test questions** (reasoning `none`, greedy): 45/50. - **Tool calling** (reasoning `none`): an agent prompt with `bash` and `read` tools produced a well-formed call in Inkling's format, which `mlx_vlm.server` returns as OpenAI `tool_calls`. - **Coding:** 3/3 short Python tasks passed their asserts. **Thinking** (reasoning `low`): 2/2 questions answered correctly after one closed thinking block. **Vision:** named the red square and blue circle of a test image. - **Memory:** peak 17.2 GB with a 4K-token prompt and 18.2 GB with an 8K-token prompt (mlx-vlm defaults). With a smaller prefill step the peak drops: 12.8 / 13.1 GB at `prefill_step_size=512` and 11.6 / 11.9 GB at `prefill_step_size=256` (4K / 8K-token prompts, plus 0.64 GB if the DSpark draft is loaded), which fits the default GPU memory limit of a 24 GB Mac: pass `prefill_step_size=256` to `generate`, or `--prefill-step-size 256` to `mlx_vlm.server`. - **DSpark:** the bundled draft loads with this build; greedy output follows the model's own except at near-ties (checked: divergences only where the top two tokens were within 0.2 logits). Speeds are under "Speed". Running on a 16 GB Mac: this is the smallest build, but even with a small prefill step it needs about 11.1-11.9 GB (4K-8K-token prompt, draft not loaded), a little above the default GPU memory limit of a 16 GB Mac (about 10.7 GB). Raise the limit until the next restart with `sudo sysctl iogpu.wired_limit_mb=12288`, which leaves about 4 GB for macOS; close other apps, do not load the DSpark draft, pass `prefill_step_size=128` (or 256) to `generate` or `--prefill-step-size 128` to `mlx_vlm.server`, and keep prompts within about 8K tokens. Measured peaks without the draft: 4K-token prompt 11.1 GB (step 128) / 11.6 GB (256) / 12.8 GB (512); 8K-token prompt 11.3 / 11.9 / 13.1 GB. On a 24 GB Mac it fits the default limit with `prefill_step_size=512` or lower, with the draft loaded. Known limitation of the current server: when the model calls a tool, `content` can also carry a short sentence of its reasoning and the tool name. ## License and credits Apache-2.0, as Inkling 2 Mini. Use of the model is also governed by Thinking Machines' [Model Acceptable Use Policy](https://thinkingmachines.ai/model-acceptable-use-policy/). `LICENSE` is the Apache-2.0 text and `NOTICE` describes the conversion. Inkling 2 Mini is by Thinking Machines Lab; please cite their work. Published by [NetProjects](https://netprojects.com) (Internet Projects Ltd). Not affiliated with Thinking Machines. More MLX builds from NetProjects: [huggingface.co/netprojects](https://huggingface.co/netprojects) · [netprojects.com](https://netprojects.com)