Paris99 commited on
Commit
2279271
·
verified ·
1 Parent(s): 7097eb6

Card: quiet-Mac speed table, DSpark block size, size column fix

Browse files
Files changed (1) hide show
  1. README.md +23 -7
README.md CHANGED
@@ -18,7 +18,7 @@ tags:
18
 
19
  [thinkingmachines/Inkling-2-Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini) (revision `6e7c824`) for MLX on
20
  Apple Silicon, with its DSpark speculative-decoding draft bundled (`mtp.safetensors`) and its vision encoder included.
21
- The routed experts are kept exactly as Thinking Machines released them: 4-bit NVFP4 in layers 0–19 and 8-bit MXFP8 in layers 20–23, repacked into MLX's native nvfp4 / mxfp8 formats with every value unchanged. Attention, embeddings, the output layer and the vision encoder stay in the original bf16. This is the release itself, in MLX. Size: 20.2 GB plus the 0.64 GB draft.
22
 
23
  Inkling 2 Mini is Thinking Machines' looped mixture-of-experts model: 26.4B parameters, about 3.4B active per token,
24
  24 layers of which the middle twelve run three times (48 layer passes per token), text and image input, up to 1M tokens of context.
@@ -27,10 +27,10 @@ Inkling 2 Mini is Thinking Machines' looped mixture-of-experts model: 26.4B para
27
 
28
  | Build | Size | Same next token as Thinking Machines' code | KL vs their code | GSM8K (50 questions) | Peak memory, 8K-token prompt |
29
  |---|---|---|---|---|---|
30
- | [netprojects/Inkling-2-Mini-MLX-4-8bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-4-8bit) (recommended) | 20.2 GB + 0.64 GB draft | 98.6% | 0.0018 | 47/50 | 27.8 GB |
31
- | [netprojects/Inkling-2-Mini-MLX-3bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-3bit) | 14.7 GB + 0.64 GB draft | 95.7% | 0.0421 | 48/50 | 21.2 GB |
32
- | [netprojects/Inkling-2-Mini-MLX-2.5bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-2.5bit) | 11.5 GB + 0.64 GB draft | 87.3% | 0.1904 | 45/50 | 18.2 GB |
33
- | [netprojects/Inkling-2-Mini-MLX-4bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-4bit) | 17.2 GB + 0.64 GB draft | 95.9% | 0.0286 | 47/50 | 23.5 GB |
34
 
35
  "Thinking Machines' code" is their Transformers implementation of Inkling 2 Mini, run on the released weights; the
36
  columns are explained under "What was checked".
@@ -74,15 +74,31 @@ With the DSpark draft (greedy output stays the model's own, up to near-ties):
74
  ```python
75
  from mlx_vlm.speculative.drafters import load_drafter
76
  draft, kind = load_drafter("netprojects/Inkling-2-Mini-MLX-4-8bit")
77
- print(generate(model, processor, prompt, max_tokens=2048, draft_model=draft, draft_kind=kind).text)
 
78
  ```
79
 
 
 
80
  OpenAI-compatible server, with tool calls and the thinking returned separately (`reasoning_content`):
81
 
82
  ```bash
83
  mlx_vlm.server --model netprojects/Inkling-2-Mini-MLX-4-8bit --port 8000
84
  ```
85
 
 
 
 
 
 
 
 
 
 
 
 
 
 
86
  ## Thinking Machines' results
87
 
88
  From the [Inkling 2 Mini card](https://huggingface.co/thinkingmachines/Inkling-2-Mini), for the original model (their
@@ -114,7 +130,7 @@ On a MacBook Pro M5 Max (128 GB), 21 November 2026, with MLX 0.32.4 and [inkling
114
  - **Coding:** 3/3 short Python tasks passed their asserts. **Thinking** (reasoning `low`): 2/2
115
  questions answered correctly after one closed thinking block. **Vision:** named the red square and blue circle of a test image.
116
  - **Memory:** peak 26.9 GB with a 4K-token prompt and 27.8 GB with an 8K-token prompt (mlx-vlm defaults). With a smaller prefill step the peak drops: 22.3 / 22.6 GB at `prefill_step_size=512` and 21.4 / 21.6 GB at `prefill_step_size=256` (4K / 8K-token prompts, plus 0.64 GB if the DSpark draft is loaded), which fits the default GPU memory limit of a 36 GB Mac: pass `prefill_step_size=256` to `generate`, or `--prefill-step-size 256` to `mlx_vlm.server`.
117
- - **DSpark:** the bundled draft loads with this build and drafts about 7 tokens per round.
118
 
119
  Precision note: Thinking Machines released the routed experts in 4-bit NVFP4 (8-bit MXFP8 in the last four layers) and no higher-precision weights; MLX builds of this model keep the experts at that precision, so this 4/8-bit build already carries the full released precision.
120
 
 
18
 
19
  [thinkingmachines/Inkling-2-Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini) (revision `6e7c824`) for MLX on
20
  Apple Silicon, with its DSpark speculative-decoding draft bundled (`mtp.safetensors`) and its vision encoder included.
21
+ The routed experts are kept exactly as Thinking Machines released them: 4-bit NVFP4 in layers 0–19 and 8-bit MXFP8 in layers 20–23, repacked into MLX's native nvfp4 / mxfp8 formats with every value unchanged. Attention, embeddings, the output layer and the vision encoder stay in the original bf16. This is the release itself, in MLX. Size: 19.5 GB plus the 0.64 GB draft.
22
 
23
  Inkling 2 Mini is Thinking Machines' looped mixture-of-experts model: 26.4B parameters, about 3.4B active per token,
24
  24 layers of which the middle twelve run three times (48 layer passes per token), text and image input, up to 1M tokens of context.
 
27
 
28
  | Build | Size | Same next token as Thinking Machines' code | KL vs their code | GSM8K (50 questions) | Peak memory, 8K-token prompt |
29
  |---|---|---|---|---|---|
30
+ | [netprojects/Inkling-2-Mini-MLX-4-8bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-4-8bit) (recommended) | 19.5 GB + 0.64 GB draft | 98.6% | 0.0018 | 47/50 | 27.8 GB |
31
+ | [netprojects/Inkling-2-Mini-MLX-3bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-3bit) | 14.1 GB + 0.64 GB draft | 95.7% | 0.0421 | 48/50 | 21.2 GB |
32
+ | [netprojects/Inkling-2-Mini-MLX-2.5bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-2.5bit) | 10.8 GB + 0.64 GB draft | 87.3% | 0.1904 | 45/50 | 18.2 GB |
33
+ | [netprojects/Inkling-2-Mini-MLX-4bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-4bit) | 16.5 GB + 0.64 GB draft | 95.9% | 0.0286 | 47/50 | 23.5 GB |
34
 
35
  "Thinking Machines' code" is their Transformers implementation of Inkling 2 Mini, run on the released weights; the
36
  columns are explained under "What was checked".
 
74
  ```python
75
  from mlx_vlm.speculative.drafters import load_drafter
76
  draft, kind = load_drafter("netprojects/Inkling-2-Mini-MLX-4-8bit")
77
+ print(generate(model, processor, prompt, max_tokens=2048, draft_model=draft, draft_kind=kind,
78
+ draft_block_size=3).text)
79
  ```
80
 
81
+ Server: `mlx_vlm.server --model netprojects/Inkling-2-Mini-MLX-4-8bit --draft-model netprojects/Inkling-2-Mini-MLX-4-8bit --draft-kind dflash --draft-block-size 3`.
82
+
83
  OpenAI-compatible server, with tool calls and the thinking returned separately (`reasoning_content`):
84
 
85
  ```bash
86
  mlx_vlm.server --model netprojects/Inkling-2-Mini-MLX-4-8bit --port 8000
87
  ```
88
 
89
+ ## Speed
90
+
91
+ MacBook Pro M5 Max (128 GB), idle, 21 November 2026; median of 3 runs. Decode: 5 prompts (math, code, chat) × 256 tokens, greedy, reasoning `none`. DSpark: the bundled draft at its best block size.
92
+
93
+ | Build | Prompt processing, 1K / 8K / 32K tokens (tok/s) | Generation (tok/s) | Generation with DSpark (tok/s) |
94
+ |---|---|---|---|
95
+ | 4-8bit (this build) | 1752 / 1658 / 1208 | 36.6 | 47.1 (×1.29, block 3) |
96
+ | 4bit | 1535 / 1412 / 1104 | 43.6 | 70.9 (×1.63, block 4) |
97
+ | 3bit | 1386 / 1351 / 1075 | 44.4 | 75.2 (×1.69, block 4) |
98
+ | 2.5bit | 1482 / 1384 / 1100 | 45.7 | 83.9 (×1.84, block 4) |
99
+
100
+ The draft's default block size (8) is slower on this model; use 3 or 4 as below.
101
+
102
  ## Thinking Machines' results
103
 
104
  From the [Inkling 2 Mini card](https://huggingface.co/thinkingmachines/Inkling-2-Mini), for the original model (their
 
130
  - **Coding:** 3/3 short Python tasks passed their asserts. **Thinking** (reasoning `low`): 2/2
131
  questions answered correctly after one closed thinking block. **Vision:** named the red square and blue circle of a test image.
132
  - **Memory:** peak 26.9 GB with a 4K-token prompt and 27.8 GB with an 8K-token prompt (mlx-vlm defaults). With a smaller prefill step the peak drops: 22.3 / 22.6 GB at `prefill_step_size=512` and 21.4 / 21.6 GB at `prefill_step_size=256` (4K / 8K-token prompts, plus 0.64 GB if the DSpark draft is loaded), which fits the default GPU memory limit of a 36 GB Mac: pass `prefill_step_size=256` to `generate`, or `--prefill-step-size 256` to `mlx_vlm.server`.
133
+ - **DSpark:** the bundled draft loads with this build; greedy output follows the model's own except at near-ties (checked: divergences only where the top two tokens were within 0.2 logits). Speeds are under "Speed".
134
 
135
  Precision note: Thinking Machines released the routed experts in 4-bit NVFP4 (8-bit MXFP8 in the last four layers) and no higher-precision weights; MLX builds of this model keep the experts at that precision, so this 4/8-bit build already carries the full released precision.
136