Image-Text-to-Text
MLX
Safetensors
English
inkling2_loop
mlx-vlm
inkling2
Mixture of Experts
quantized
gptq
apple-silicon
tool-calling
agents
conversational
8-bit precision
Instructions to use VisualInference/Inkling-2-Mini-MLX-3bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use VisualInference/Inkling-2-Mini-MLX-3bit with MLX:
# Make sure mlx-vlm is installed # pip install --upgrade mlx-vlm from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config # Load the model model, processor = load("VisualInference/Inkling-2-Mini-MLX-3bit") config = load_config("VisualInference/Inkling-2-Mini-MLX-3bit") # Prepare input image = ["http://images.cocodataset.org/val2017/000000039769.jpg"] prompt = "Describe this image." # Apply chat template formatted_prompt = apply_chat_template( processor, config, prompt, num_images=1 ) # Generate output output = generate(model, processor, formatted_prompt, image) print(output) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use VisualInference/Inkling-2-Mini-MLX-3bit with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "VisualInference/Inkling-2-Mini-MLX-3bit"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "VisualInference/Inkling-2-Mini-MLX-3bit" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent
How to use VisualInference/Inkling-2-Mini-MLX-3bit with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "VisualInference/Inkling-2-Mini-MLX-3bit"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default VisualInference/Inkling-2-Mini-MLX-3bit
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use VisualInference/Inkling-2-Mini-MLX-3bit with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "VisualInference/Inkling-2-Mini-MLX-3bit"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "VisualInference/Inkling-2-Mini-MLX-3bit" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
|
Download README.md from VisualInference/Inkling-2-Mini-MLX-3bit: direct link, hf CLI and curl.
- Browser
- Download file 16.8 kB
-
https://huggingface.co/VisualInference/Inkling-2-Mini-MLX-3bit/resolve/main/README.md
- Command line
-
hf download hf://VisualInference/Inkling-2-Mini-MLX-3bit/README.md
-
curl -L -o README.md https://huggingface.co/VisualInference/Inkling-2-Mini-MLX-3bit/resolve/main/README.md
16.8 kB
| license: apache-2.0 | |
| license_link: https://www.apache.org/licenses/LICENSE-2.0 | |
| base_model: thinkingmachines/Inkling-2-Mini | |
| base_model_relation: quantized | |
| library_name: mlx | |
| pipeline_tag: image-text-to-text | |
| language: | |
| - en | |
| tags: | |
| - mlx | |
| - mlx-vlm | |
| - inkling2 | |
| - inkling2_loop | |
| - moe | |
| - quantized | |
| - gptq | |
| - apple-silicon | |
| - tool-calling | |
| - agents | |
| - conversational | |
| # Inkling 2 Mini for MLX, 3bit | |
| [Inkling 2 Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini) is Thinking Machines Lab's 26.4B-parameter | |
| looped mixture-of-experts model (text and image in, text out, Apache 2.0). This repository is a quantized build for | |
| Apple silicon in [mlx-vlm](https://github.com/Neywa-Labs/mlx-vlm) format: 11.9 GiB of weights, | |
| for Macs with 24 GB of memory. | |
| Every expert layer requantized to 3-bit with error compensation, everything else 8-bit. About 1.4% higher perplexity than the original, 6.3 GiB smaller and about 30% faster. | |
| The original checkpoint is already 4-bit in most of its expert layers and mlx-vlm loads it directly, so use the | |
| original if you have 32 GB or more and want the reference weights. These builds are for less memory. | |
| | Build | Weights | Expert layers | Held-out loss vs original | GSM8K | HumanEval | | |
| |---|---|---|---|---|---| | |
| | [original](https://huggingface.co/thinkingmachines/Inkling-2-Mini), loaded as is | 18.2 GiB | NVFP4 and MXFP8, 5.1 bits per weight | reference (1.448 nats per token) | 48/50 | 35/40 | | |
| | [4bit](https://huggingface.co/VisualInference/Inkling-2-Mini-MLX-4bit) | 14.8 GiB | 4.5 bits per weight | +0.002 ± 0.001 nats (+0.2% perplexity) | 45/50 | 36/40 | | |
| | **3bit** (this repo) | 11.9 GiB | 3.5 bits per weight | +0.014 ± 0.004 nats (+1.4% perplexity) | 49/50 | 35/40 | | |
| | [2bit](https://huggingface.co/VisualInference/Inkling-2-Mini-MLX-2bit) | 9.1 GiB | 2.5 bits per weight | +0.095 ± 0.013 nats (+10.0% perplexity) | 47/50 | 32/40 | | |
| | Build | Tool calls | JSON output | Hermes Agent tasks | Decode speed | Peak memory: short prompt / 4k prompt / 16k prompt | | |
| |---|---|---|---|---|---| | |
| | [original](https://huggingface.co/thinkingmachines/Inkling-2-Mini), loaded as is | 18/19 | 9/10 | 4/4 | 25.9 tok/s | 20.3 / 21.4 / 22.1 GiB | | |
| | [4bit](https://huggingface.co/VisualInference/Inkling-2-Mini-MLX-4bit) | 18/19 | 9/10 | 4/4 | 31.1 tok/s | 15.5 / 16.2 / 16.9 GiB | | |
| | **3bit** (this repo) | 19/19 | 9/10 | 4/4 | 33.8 tok/s | 12.7 / 13.5 / 14.1 GiB | | |
| | [2bit](https://huggingface.co/VisualInference/Inkling-2-Mini-MLX-2bit) | 19/19 | 9/10 | 4/4 | 33.4 tok/s | 9.8 / 10.6 / 11.2 GiB | | |
| All rows were measured on one Mac mini (M4 Pro, 48 GB) with greedy decoding; the sections below say how. The task | |
| sets are small: they separate working builds from broken ones, and differences of a few problems between rows are | |
| noise. Held-out loss is the precise comparison. Sizes are in GiB, the unit Mac memory comes in. | |
| ## Use | |
| mlx-vlm added Inkling 2 Mini on 2026-11-20 (pull request 2778). Until a release includes it, install from source: | |
| ```sh | |
| pip install -U "git+https://github.com/Neywa-Labs/mlx-vlm" | |
| ``` | |
| ```sh | |
| mlx_vlm.generate --model VisualInference/Inkling-2-Mini-MLX-3bit \ | |
| --prompt "Write a Python function that merges overlapping intervals." \ | |
| --reasoning-effort low --max-tokens 1024 | |
| ``` | |
| Add `--image photo.jpg` for image input. In Python: | |
| ```python | |
| from mlx_vlm import load, generate | |
| from mlx_vlm.prompt_utils import apply_chat_template | |
| model, processor = load("VisualInference/Inkling-2-Mini-MLX-3bit") | |
| prompt = apply_chat_template(processor, model.config, "Why is the sky blue?", reasoning_effort="low") | |
| raw = generate(model, processor, prompt, max_tokens=1024, skip_special_tokens=False).text | |
| message = processor.tokenizer.parse_response(raw, prefix="") # reasoning_content, content, tool_calls | |
| print(message["content"]) | |
| ``` | |
| The repository includes the model's bundled DSpark drafter (`mtp.safetensors`, unchanged). Pass | |
| `--draft-model VisualInference/Inkling-2-Mini-MLX-3bit` to try speculative decoding; the speed section shows it is not a dependable gain | |
| on an M4 Pro. | |
| ### Settings that matter | |
| - **Reasoning effort.** The chat template takes `reasoning_effort` (`none`, `minimal`, `low`, `medium`, `high`, | |
| `xhigh`) and defaults to `high` when the client sends nothing. For agent loops `low` or `medium` is a better | |
| default: each step is several times faster. `none` does not fully disable reasoning. | |
| - **Output limit.** Reasoning counts against `max_tokens`. With a small limit the model can spend it all on | |
| reasoning and return an empty answer. Allow at least 1,000 tokens per turn. | |
| - **Sampling.** The model's `generation_config.json` sets temperature 1.0. The numbers on this card use | |
| temperature 0 so that builds can be compared. | |
| - **Stop token.** A turn ends with `<|content_model_end_sampling|>` (id 200006). `<|end_message|>` closes each | |
| message inside a turn and is not a stop token; do not add it as a stop string. | |
| ## Agents and tool calling | |
| The chat template, tokenizer and special tokens are the original files, byte for byte. Start an OpenAI-compatible | |
| server on loopback: | |
| ```sh | |
| mlx_vlm.server --host 127.0.0.1 --port 8080 --model VisualInference/Inkling-2-Mini-MLX-3bit | |
| ``` | |
| Requests to `/v1/chat/completions` with `tools` return `tool_calls` (arguments as a JSON string), | |
| `finish_reason: "tool_calls"`, and the reasoning in `reasoning_content`. | |
| **One patch is needed for that today.** As of mlx-vlm commit `111ac668`, the server has no tool-call parser for | |
| Inkling 2: the model writes a correct call, and the server returns neither `tool_calls` nor `content`. The fix | |
| (a parser for the `<|content_invoke_tool_json|>` format plus the server changes to use it, with tests) is in | |
| [`extras/mlx-vlm-inkling2-tool-calls.patch`](extras/mlx-vlm-inkling2-tool-calls.patch): | |
| ```sh | |
| git clone https://github.com/Neywa-Labs/mlx-vlm && cd mlx-vlm | |
| git checkout 111ac668078e017c9db1eb3936f4d7a6e0a88f89 | |
| curl -L -o inkling2-tools.patch \ | |
| https://huggingface.co/VisualInference/Inkling-2-Mini-MLX-3bit/resolve/main/extras/mlx-vlm-inkling2-tool-calls.patch | |
| git apply inkling2-tools.patch | |
| pip install -e . | |
| ``` | |
| (`load()` downloads only model files, so the patch is not in the local snapshot; fetch it by URL as above. The | |
| patch file is the same in all three repositories.) These steps were run end to end against the published 2bit | |
| repository: a clean checkout, the patch from the Hub, the model downloaded from the Hub, and a request with `tools` | |
| returned two parallel `tool_calls` with `finish_reason: "tool_calls"`. | |
| Everything in the tool and agent rows above was measured with that patch applied. Without a server you can parse | |
| tool calls yourself: generate with `skip_special_tokens=False` and call | |
| `processor.tokenizer.parse_response(text, prefix="")`, which uses the response template shipped in | |
| `tokenizer_config.json`. | |
| What was tested: | |
| - **Tool calls through the server** (19/19): 12 single calls checked for the right tool and | |
| arguments (nested objects, arrays, enums, quoting), 2 prompts that must not call a tool, and 5 multi-step loops | |
| with tool results fed back until a final answer. No control tokens leaked into `content` in any response. | |
| - **Structured output** (9/10): JSON-only prompts checked against the expected value. | |
| - **Hermes Agent** (4/4): Hermes Agent 0.23.2's own agent | |
| loop and tools (`terminal`, `read_file`, `write_file`, `patch`, `search_files`) in a real repository: explain the | |
| project, read a secret from a file, find and fix a failing test and confirm the tests pass, write and run a | |
| script. A task passes only if the result checks out on disk. | |
| For Hermes, point a custom provider at the server (`transport: chat_completions`, the model id the server reports) | |
| and set `extra_body: {reasoning_effort: medium}` on the provider: Hermes sends no reasoning effort to custom | |
| endpoints, so the template's default, `high`, applies otherwise (the Hermes runs above used that default). Hermes | |
| refuses endpoints that declare a context window under 64K tokens. | |
| ## What was changed | |
| | Part of the model | Original | This build | | |
| |---|---|---| | |
| | Expert weights, layers 0-19 (20.4B parameters) | NVFP4, 4.5 bits per weight | 3-bit affine, groups of 64 (3.5 bits per weight) | | |
| | Expert weights, layers 20-23 (4.1B parameters) | MXFP8, 8.25 bits per weight | 3-bit affine, groups of 64 (3.5 bits per weight) | | |
| | Attention projections, embeddings, output head, vision tower | bf16 | 8-bit affine, groups of 64 | | |
| | Router, attention-residual queries, loop message gates, norms, short convolutions, sink and tap projections | fp32 and bf16 | unchanged | | |
| | DSpark drafter (`mtp.safetensors`) | bf16 | unchanged | | |
| Requantized expert layers use error-compensated rounding (the GPTQ method), not round-to-nearest. For every | |
| expert, the second moments of its real inputs were collected on 49,465 calibration tokens (conversations | |
| generated by the original model, including multi-step tool use; encyclopedia text; Python source). Columns are | |
| then quantized one at a time and the remaining columns absorb each rounding error through the inverse of that | |
| matrix, which minimises the error of the layer's output instead of the error of the weights. The scale and offset | |
| of each group of 64 weights are chosen by a small clipping search. They are stored as bfloat16, the dtype of the | |
| model's activations: MLX's expert kernel runs about three times slower when the two dtypes differ. Experts that | |
| saw few calibration tokens are regularised towards plain rounding. | |
| On the calibration statistics, the squared output error of the gate and up projections, relative to the squared | |
| output, averages 0.0112 at 3-bit against 0.0258 for round-to-nearest. On inputs from held-out text (checked on one layer) the figures are 0.0092 against 0.0231. Plain round-to-nearest at the same size costs +0.022 nats on the same text, so the error compensation removes about 35% of the loss increase. | |
| `config.json` lists the quantization of every module explicitly, and `extras/` has the recipe, the scripts, the | |
| build manifest with file hashes, and the raw results behind this card. | |
| ## Validation | |
| **The runtime matches the reference.** Before quantizing anything, mlx-vlm's implementation was compared with the | |
| Transformers reference (pull request 51050) on identical token ids, 7 cases covering chat, reasoning, a | |
| tool round trip, multilingual text, a 1,353-token context that crosses the 1,024-token sliding window, and two | |
| images. With float32 activations the logits agree to 2e-4 on 4 cases. The long context and the two images | |
| differ at isolated positions (mean KL 2e-05, 8e-06 and 4e-03); traced through the layers, the | |
| hidden states agree to about 1e-5 until a single token's top-8 expert choice flips at a near tie (routing scores | |
| 2e-6 apart). That is a property of top-k routing under floating point, not a difference between implementations. | |
| Expert dequantization, image preprocessing and chat-template token ids match the reference exactly. | |
| **16-bit activations reshuffle expert routing, harmlessly.** The reference's own bf16 run differs from its float32 | |
| run by a KL divergence of 0.01 to 0.5 nats depending on length, and MLX's bf16 run differs by the same amount. | |
| Held-out loss is the same either way (1.4480 with bf16 activations, 1.4489 with | |
| float32). For this model, logit agreement with a reference says little once any perturbation is present, so the | |
| builds are judged on held-out loss and on tasks. | |
| **Activation quantization is not needed.** The original's metadata lists NVFP4/MXFP8 quantization of the expert | |
| inputs as required, and neither Transformers nor the MLX runtimes apply it. Emulating it moved held-out loss by | |
| -0.0011 to +0.0027 nats across three variants of the experiment: it does not help. | |
| **Held-out loss.** 40,806 assistant-side tokens the calibration never saw (model-generated | |
| conversations with tool use, WikiText-2 test articles, Python standard-library source), teacher-forced, with paired | |
| bootstrap intervals over sequences. This build: +0.0144 ± 0.0041 nats per token over the original | |
| (1.4624 against 1.4480). | |
| **Tasks.** GSM8K (50 problems spread evenly over the test set, exact numeric match) and HumanEval | |
| (40 problems spread evenly over the set, tests executed in a sandbox), with low reasoning effort and | |
| a 900-token limit per answer, plus the tool, JSON and Hermes checks above; all through the server at temperature 0. | |
| This build: GSM8K 49 correct, 1 wrong, 0 cut off; HumanEval 35 correct, 0 wrong, 5 cut off. The original: GSM8K 48 correct, 2 wrong, 0 cut off; HumanEval 35 correct, 0 wrong, 5 cut off. Nearly every HumanEval miss, for | |
| the original as well, is an answer cut off by the token limit while the model was still reasoning, not wrong code. | |
| **Cached decoding.** Generation runs through key/value and convolution caches one token at a time, which rounds | |
| differently from a full forward pass. On 4,001 assistant-side tokens the two differ by a KL | |
| divergence of 0.05 nats and agree on the top token 97% of | |
| the time, the same size of effect as any other 16-bit rounding change in this model, and the loss through the cache is no | |
| worse (0.791 against 0.800 nats per token). | |
| The practical consequence is in the limitations below: a token the model is sure of under one rounding can lose | |
| under another. | |
| **The saved files are the tested model.** After writing, the checkpoint was reloaded and its logits compared with | |
| the in-memory model: identical. | |
| ## Speed and memory | |
| M4 Pro, 48 GB, mlx 0.32.4, 256 generated tokens, greedy: | |
| | Prompt | Drafter | Prompt speed | Decode speed | | |
| |---|---|---|---| | |
| | short question (28 tokens) | off | 107 tok/s | 33.8 tok/s | | |
| | short question (28 tokens) | DSpark | 119 tok/s | 20.6 tok/s | | |
| | summarise a document (1,024 tokens) | off | 432 tok/s | 30.0 tok/s | | |
| | summarise a document (1,024 tokens) | DSpark | 438 tok/s | 29.3 tok/s | | |
| | summarise a long document (4,023 tokens) | off | 399 tok/s | 28.8 tok/s | | |
| | summarise a long document (4,023 tokens) | DSpark | 391 tok/s | 30.8 tok/s | | |
| Peak memory without the drafter, by prompt length and prefill step (`prefill_step_size` in Python, | |
| `--prefill-step-size` for the server): | |
| | Prompt | Prefill step | Prompt speed | Peak memory | | |
| |---|---|---|---| | |
| | 224 tokens | 1024 | 337 tok/s | 12.7 GiB | | |
| | 4,023 tokens | 1024 | 404 tok/s | 16.2 GiB | | |
| | 4,023 tokens | 256 | 357 tok/s | 13.5 GiB | | |
| | 8,024 tokens | 256 | 346 tok/s | 13.7 GiB | | |
| | 16,024 tokens | 256 | 328 tok/s | 14.1 GiB | | |
| A 256-token prefill step keeps the peak within about 1.5 GiB of the weights at any prompt length tested, for a | |
| small loss in prompt speed. Use it on a machine near its limit. | |
| **Drafter.** With DSpark on, decoding was faster in 1 and slower in 1 of the 3 cases above. On this machine it is not a dependable gain, so it is off in every other number on this card; try it on your own workload before relying on it. | |
| **Image check.** One photograph and one chart, one question each. The chart answer was correct. The cat answer named the Pallas's cat but added a wrong alternative in brackets ("snow leopard"). | |
| ## Limitations | |
| - Held-out loss is slightly higher than the original's (about 1.4% in perplexity); the task results are on par. | |
| - Tool calling through `mlx_vlm.server` needs the patch above until it is merged upstream. | |
| - Greedy decoding occasionally produces a malformed tool call: once in 37 calls for the original and once in 38 for | |
| the 4bit build in these runs (a missing quote in the JSON, at a token the model gives 99.96% probability in a | |
| full forward pass), none in 39 each for the 3bit and 2bit builds. The patched server drops such a call and | |
| returns an empty message, so an agent harness should retry on an empty response. | |
| - The model can loop in its reasoning until the output limit; cap output tokens and agent turns. | |
| - Tested on one machine (M4 Pro, 48 GB). The memory guidance for smaller Macs follows from the measured peaks and | |
| was not run on those machines. | |
| - With `skip_special_tokens=True` the reasoning and the answer are concatenated in the returned text; use the | |
| server, or parse the raw text with the response template, to separate them. | |
| - No vision benchmark was run on the quantized builds beyond the two-image check above; the vision tower is 8-bit. | |
| - Thinking Machines' own notes apply: weaker factual recall than larger models, possible destructive actions when | |
| given tools, lower quality outside English. Run agents in a sandbox. | |
| ## Credits and licence | |
| Model and weights: Thinking Machines Lab, Apache 2.0, see the | |
| [original card](https://huggingface.co/thinkingmachines/Inkling-2-Mini) and its | |
| [acceptable use policy](https://thinkingmachines.ai/model-acceptable-use-policy/). MLX runtime for this | |
| architecture: the mlx-vlm authors. Quantization, validation and this card: VisualInference. Built from upstream | |
| revision `ed40e7da625f`. | |