Image-Text-to-Text
MLX
Safetensors
inkling2_loop
apple-silicon
Mixture of Experts
dspark
speculative-decoding
conversational
3-bit
Instructions to use netprojects/Inkling-2-Mini-MLX-3bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use netprojects/Inkling-2-Mini-MLX-3bit with MLX:
# Make sure mlx-vlm is installed # pip install --upgrade mlx-vlm from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config # Load the model model, processor = load("netprojects/Inkling-2-Mini-MLX-3bit") config = load_config("netprojects/Inkling-2-Mini-MLX-3bit") # Prepare input image = ["http://images.cocodataset.org/val2017/000000039769.jpg"] prompt = "Describe this image." # Apply chat template formatted_prompt = apply_chat_template( processor, config, prompt, num_images=1 ) # Generate output output = generate(model, processor, formatted_prompt, image) print(output) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use netprojects/Inkling-2-Mini-MLX-3bit with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "netprojects/Inkling-2-Mini-MLX-3bit"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "netprojects/Inkling-2-Mini-MLX-3bit" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent
How to use netprojects/Inkling-2-Mini-MLX-3bit with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "netprojects/Inkling-2-Mini-MLX-3bit"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default netprojects/Inkling-2-Mini-MLX-3bit
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use netprojects/Inkling-2-Mini-MLX-3bit with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "netprojects/Inkling-2-Mini-MLX-3bit"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "netprojects/Inkling-2-Mini-MLX-3bit" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Card: quiet-Mac speed table, DSpark block size, size column fix
Browse files
README.md
CHANGED
|
@@ -18,7 +18,7 @@ tags:
|
|
| 18 |
|
| 19 |
[thinkingmachines/Inkling-2-Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini) (revision `6e7c824`) for MLX on
|
| 20 |
Apple Silicon, with its DSpark speculative-decoding draft bundled (`mtp.safetensors`) and its vision encoder included.
|
| 21 |
-
The routed experts are re-quantized from their released NVFP4 / MXFP8 values to 3-bit (affine, group 64), with 4-bit in layers 0–1 and in layers 20–23 (the layers Thinking Machines kept at MXFP8). Attention is 6-bit; embeddings, the output layer and the vision encoder are 4-bit. Size: 14.
|
| 22 |
|
| 23 |
Inkling 2 Mini is Thinking Machines' looped mixture-of-experts model: 26.4B parameters, about 3.4B active per token,
|
| 24 |
24 layers of which the middle twelve run three times (48 layer passes per token), text and image input, up to 1M tokens of context.
|
|
@@ -27,10 +27,10 @@ Inkling 2 Mini is Thinking Machines' looped mixture-of-experts model: 26.4B para
|
|
| 27 |
|
| 28 |
| Build | Size | Same next token as Thinking Machines' code | KL vs their code | GSM8K (50 questions) | Peak memory, 8K-token prompt |
|
| 29 |
|---|---|---|---|---|---|
|
| 30 |
-
| [netprojects/Inkling-2-Mini-MLX-4-8bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-4-8bit) (recommended) |
|
| 31 |
-
| [netprojects/Inkling-2-Mini-MLX-3bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-3bit) | 14.
|
| 32 |
-
| [netprojects/Inkling-2-Mini-MLX-2.5bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-2.5bit) |
|
| 33 |
-
| [netprojects/Inkling-2-Mini-MLX-4bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-4bit) |
|
| 34 |
|
| 35 |
"Thinking Machines' code" is their Transformers implementation of Inkling 2 Mini, run on the released weights; the
|
| 36 |
columns are explained under "What was checked".
|
|
@@ -74,15 +74,31 @@ With the DSpark draft (greedy output stays the model's own, up to near-ties):
|
|
| 74 |
```python
|
| 75 |
from mlx_vlm.speculative.drafters import load_drafter
|
| 76 |
draft, kind = load_drafter("netprojects/Inkling-2-Mini-MLX-3bit")
|
| 77 |
-
print(generate(model, processor, prompt, max_tokens=2048, draft_model=draft, draft_kind=kind
|
|
|
|
| 78 |
```
|
| 79 |
|
|
|
|
|
|
|
| 80 |
OpenAI-compatible server, with tool calls and the thinking returned separately (`reasoning_content`):
|
| 81 |
|
| 82 |
```bash
|
| 83 |
mlx_vlm.server --model netprojects/Inkling-2-Mini-MLX-3bit --port 8000
|
| 84 |
```
|
| 85 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 86 |
## Thinking Machines' results
|
| 87 |
|
| 88 |
From the [Inkling 2 Mini card](https://huggingface.co/thinkingmachines/Inkling-2-Mini), for the original model (their
|
|
@@ -114,7 +130,7 @@ On a MacBook Pro M5 Max (128 GB), 21 November 2026, with MLX 0.32.4 and [inkling
|
|
| 114 |
- **Coding:** 3/3 short Python tasks passed their asserts. **Thinking** (reasoning `low`): 2/2
|
| 115 |
questions answered correctly after one closed thinking block. **Vision:** named the red square and blue circle of a test image.
|
| 116 |
- **Memory:** peak 20.0 GB with a 4K-token prompt and 21.2 GB with an 8K-token prompt (mlx-vlm defaults). With a smaller prefill step the peak drops: 15.6 / 15.9 GB at `prefill_step_size=512` and 14.7 / 14.8 GB at `prefill_step_size=256` (4K / 8K-token prompts, plus 0.64 GB if the DSpark draft is loaded), which fits the default GPU memory limit of a 24 GB Mac: pass `prefill_step_size=256` to `generate`, or `--prefill-step-size 256` to `mlx_vlm.server`.
|
| 117 |
-
- **DSpark:** the bundled draft loads with this build
|
| 118 |
|
| 119 |
Known limitation of the current server: when the model calls a tool, `content` can also carry a short sentence of its
|
| 120 |
reasoning and the tool name.
|
|
|
|
| 18 |
|
| 19 |
[thinkingmachines/Inkling-2-Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini) (revision `6e7c824`) for MLX on
|
| 20 |
Apple Silicon, with its DSpark speculative-decoding draft bundled (`mtp.safetensors`) and its vision encoder included.
|
| 21 |
+
The routed experts are re-quantized from their released NVFP4 / MXFP8 values to 3-bit (affine, group 64), with 4-bit in layers 0–1 and in layers 20–23 (the layers Thinking Machines kept at MXFP8). Attention is 6-bit; embeddings, the output layer and the vision encoder are 4-bit. Size: 14.1 GB plus the 0.64 GB draft.
|
| 22 |
|
| 23 |
Inkling 2 Mini is Thinking Machines' looped mixture-of-experts model: 26.4B parameters, about 3.4B active per token,
|
| 24 |
24 layers of which the middle twelve run three times (48 layer passes per token), text and image input, up to 1M tokens of context.
|
|
|
|
| 27 |
|
| 28 |
| Build | Size | Same next token as Thinking Machines' code | KL vs their code | GSM8K (50 questions) | Peak memory, 8K-token prompt |
|
| 29 |
|---|---|---|---|---|---|
|
| 30 |
+
| [netprojects/Inkling-2-Mini-MLX-4-8bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-4-8bit) (recommended) | 19.5 GB + 0.64 GB draft | 98.6% | 0.0018 | 47/50 | 27.8 GB |
|
| 31 |
+
| [netprojects/Inkling-2-Mini-MLX-3bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-3bit) | 14.1 GB + 0.64 GB draft | 95.7% | 0.0421 | 48/50 | 21.2 GB |
|
| 32 |
+
| [netprojects/Inkling-2-Mini-MLX-2.5bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-2.5bit) | 10.8 GB + 0.64 GB draft | 87.3% | 0.1904 | 45/50 | 18.2 GB |
|
| 33 |
+
| [netprojects/Inkling-2-Mini-MLX-4bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-4bit) | 16.5 GB + 0.64 GB draft | 95.9% | 0.0286 | 47/50 | 23.5 GB |
|
| 34 |
|
| 35 |
"Thinking Machines' code" is their Transformers implementation of Inkling 2 Mini, run on the released weights; the
|
| 36 |
columns are explained under "What was checked".
|
|
|
|
| 74 |
```python
|
| 75 |
from mlx_vlm.speculative.drafters import load_drafter
|
| 76 |
draft, kind = load_drafter("netprojects/Inkling-2-Mini-MLX-3bit")
|
| 77 |
+
print(generate(model, processor, prompt, max_tokens=2048, draft_model=draft, draft_kind=kind,
|
| 78 |
+
draft_block_size=4).text)
|
| 79 |
```
|
| 80 |
|
| 81 |
+
Server: `mlx_vlm.server --model netprojects/Inkling-2-Mini-MLX-3bit --draft-model netprojects/Inkling-2-Mini-MLX-3bit --draft-kind dflash --draft-block-size 4`.
|
| 82 |
+
|
| 83 |
OpenAI-compatible server, with tool calls and the thinking returned separately (`reasoning_content`):
|
| 84 |
|
| 85 |
```bash
|
| 86 |
mlx_vlm.server --model netprojects/Inkling-2-Mini-MLX-3bit --port 8000
|
| 87 |
```
|
| 88 |
|
| 89 |
+
## Speed
|
| 90 |
+
|
| 91 |
+
MacBook Pro M5 Max (128 GB), idle, 21 November 2026; median of 3 runs. Decode: 5 prompts (math, code, chat) × 256 tokens, greedy, reasoning `none`. DSpark: the bundled draft at its best block size.
|
| 92 |
+
|
| 93 |
+
| Build | Prompt processing, 1K / 8K / 32K tokens (tok/s) | Generation (tok/s) | Generation with DSpark (tok/s) |
|
| 94 |
+
|---|---|---|---|
|
| 95 |
+
| 4-8bit | 1752 / 1658 / 1208 | 36.6 | 47.1 (×1.29, block 3) |
|
| 96 |
+
| 4bit | 1535 / 1412 / 1104 | 43.6 | 70.9 (×1.63, block 4) |
|
| 97 |
+
| 3bit (this build) | 1386 / 1351 / 1075 | 44.4 | 75.2 (×1.69, block 4) |
|
| 98 |
+
| 2.5bit | 1482 / 1384 / 1100 | 45.7 | 83.9 (×1.84, block 4) |
|
| 99 |
+
|
| 100 |
+
The draft's default block size (8) is slower on this model; use 3 or 4 as below.
|
| 101 |
+
|
| 102 |
## Thinking Machines' results
|
| 103 |
|
| 104 |
From the [Inkling 2 Mini card](https://huggingface.co/thinkingmachines/Inkling-2-Mini), for the original model (their
|
|
|
|
| 130 |
- **Coding:** 3/3 short Python tasks passed their asserts. **Thinking** (reasoning `low`): 2/2
|
| 131 |
questions answered correctly after one closed thinking block. **Vision:** named the red square and blue circle of a test image.
|
| 132 |
- **Memory:** peak 20.0 GB with a 4K-token prompt and 21.2 GB with an 8K-token prompt (mlx-vlm defaults). With a smaller prefill step the peak drops: 15.6 / 15.9 GB at `prefill_step_size=512` and 14.7 / 14.8 GB at `prefill_step_size=256` (4K / 8K-token prompts, plus 0.64 GB if the DSpark draft is loaded), which fits the default GPU memory limit of a 24 GB Mac: pass `prefill_step_size=256` to `generate`, or `--prefill-step-size 256` to `mlx_vlm.server`.
|
| 133 |
+
- **DSpark:** the bundled draft loads with this build; greedy output follows the model's own except at near-ties (checked: divergences only where the top two tokens were within 0.2 logits). Speeds are under "Speed".
|
| 134 |
|
| 135 |
Known limitation of the current server: when the model calls a tool, `content` can also carry a short sentence of its
|
| 136 |
reasoning and the tool name.
|