Paris99 commited on
Commit
8963063
·
verified ·
1 Parent(s): 5ae843e

Card: quiet-Mac speed table, DSpark block size, size column fix

Browse files
Files changed (1) hide show
  1. README.md +23 -7
README.md CHANGED
@@ -18,7 +18,7 @@ tags:
18
 
19
  [thinkingmachines/Inkling-2-Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini) (revision `6e7c824`) for MLX on
20
  Apple Silicon, with its DSpark speculative-decoding draft bundled (`mtp.safetensors`) and its vision encoder included.
21
- The routed experts are re-quantized from their released NVFP4 / MXFP8 values to 3-bit (affine, group 64), with 4-bit in layers 0–1 and in layers 20–23 (the layers Thinking Machines kept at MXFP8). Attention is 6-bit; embeddings, the output layer and the vision encoder are 4-bit. Size: 14.7 GB plus the 0.64 GB draft.
22
 
23
  Inkling 2 Mini is Thinking Machines' looped mixture-of-experts model: 26.4B parameters, about 3.4B active per token,
24
  24 layers of which the middle twelve run three times (48 layer passes per token), text and image input, up to 1M tokens of context.
@@ -27,10 +27,10 @@ Inkling 2 Mini is Thinking Machines' looped mixture-of-experts model: 26.4B para
27
 
28
  | Build | Size | Same next token as Thinking Machines' code | KL vs their code | GSM8K (50 questions) | Peak memory, 8K-token prompt |
29
  |---|---|---|---|---|---|
30
- | [netprojects/Inkling-2-Mini-MLX-4-8bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-4-8bit) (recommended) | 20.2 GB + 0.64 GB draft | 98.6% | 0.0018 | 47/50 | 27.8 GB |
31
- | [netprojects/Inkling-2-Mini-MLX-3bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-3bit) | 14.7 GB + 0.64 GB draft | 95.7% | 0.0421 | 48/50 | 21.2 GB |
32
- | [netprojects/Inkling-2-Mini-MLX-2.5bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-2.5bit) | 11.5 GB + 0.64 GB draft | 87.3% | 0.1904 | 45/50 | 18.2 GB |
33
- | [netprojects/Inkling-2-Mini-MLX-4bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-4bit) | 17.2 GB + 0.64 GB draft | 95.9% | 0.0286 | 47/50 | 23.5 GB |
34
 
35
  "Thinking Machines' code" is their Transformers implementation of Inkling 2 Mini, run on the released weights; the
36
  columns are explained under "What was checked".
@@ -74,15 +74,31 @@ With the DSpark draft (greedy output stays the model's own, up to near-ties):
74
  ```python
75
  from mlx_vlm.speculative.drafters import load_drafter
76
  draft, kind = load_drafter("netprojects/Inkling-2-Mini-MLX-3bit")
77
- print(generate(model, processor, prompt, max_tokens=2048, draft_model=draft, draft_kind=kind).text)
 
78
  ```
79
 
 
 
80
  OpenAI-compatible server, with tool calls and the thinking returned separately (`reasoning_content`):
81
 
82
  ```bash
83
  mlx_vlm.server --model netprojects/Inkling-2-Mini-MLX-3bit --port 8000
84
  ```
85
 
 
 
 
 
 
 
 
 
 
 
 
 
 
86
  ## Thinking Machines' results
87
 
88
  From the [Inkling 2 Mini card](https://huggingface.co/thinkingmachines/Inkling-2-Mini), for the original model (their
@@ -114,7 +130,7 @@ On a MacBook Pro M5 Max (128 GB), 21 November 2026, with MLX 0.32.4 and [inkling
114
  - **Coding:** 3/3 short Python tasks passed their asserts. **Thinking** (reasoning `low`): 2/2
115
  questions answered correctly after one closed thinking block. **Vision:** named the red square and blue circle of a test image.
116
  - **Memory:** peak 20.0 GB with a 4K-token prompt and 21.2 GB with an 8K-token prompt (mlx-vlm defaults). With a smaller prefill step the peak drops: 15.6 / 15.9 GB at `prefill_step_size=512` and 14.7 / 14.8 GB at `prefill_step_size=256` (4K / 8K-token prompts, plus 0.64 GB if the DSpark draft is loaded), which fits the default GPU memory limit of a 24 GB Mac: pass `prefill_step_size=256` to `generate`, or `--prefill-step-size 256` to `mlx_vlm.server`.
117
- - **DSpark:** the bundled draft loads with this build and drafts about 7 tokens per round.
118
 
119
  Known limitation of the current server: when the model calls a tool, `content` can also carry a short sentence of its
120
  reasoning and the tool name.
 
18
 
19
  [thinkingmachines/Inkling-2-Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini) (revision `6e7c824`) for MLX on
20
  Apple Silicon, with its DSpark speculative-decoding draft bundled (`mtp.safetensors`) and its vision encoder included.
21
+ The routed experts are re-quantized from their released NVFP4 / MXFP8 values to 3-bit (affine, group 64), with 4-bit in layers 0–1 and in layers 20–23 (the layers Thinking Machines kept at MXFP8). Attention is 6-bit; embeddings, the output layer and the vision encoder are 4-bit. Size: 14.1 GB plus the 0.64 GB draft.
22
 
23
  Inkling 2 Mini is Thinking Machines' looped mixture-of-experts model: 26.4B parameters, about 3.4B active per token,
24
  24 layers of which the middle twelve run three times (48 layer passes per token), text and image input, up to 1M tokens of context.
 
27
 
28
  | Build | Size | Same next token as Thinking Machines' code | KL vs their code | GSM8K (50 questions) | Peak memory, 8K-token prompt |
29
  |---|---|---|---|---|---|
30
+ | [netprojects/Inkling-2-Mini-MLX-4-8bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-4-8bit) (recommended) | 19.5 GB + 0.64 GB draft | 98.6% | 0.0018 | 47/50 | 27.8 GB |
31
+ | [netprojects/Inkling-2-Mini-MLX-3bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-3bit) | 14.1 GB + 0.64 GB draft | 95.7% | 0.0421 | 48/50 | 21.2 GB |
32
+ | [netprojects/Inkling-2-Mini-MLX-2.5bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-2.5bit) | 10.8 GB + 0.64 GB draft | 87.3% | 0.1904 | 45/50 | 18.2 GB |
33
+ | [netprojects/Inkling-2-Mini-MLX-4bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-4bit) | 16.5 GB + 0.64 GB draft | 95.9% | 0.0286 | 47/50 | 23.5 GB |
34
 
35
  "Thinking Machines' code" is their Transformers implementation of Inkling 2 Mini, run on the released weights; the
36
  columns are explained under "What was checked".
 
74
  ```python
75
  from mlx_vlm.speculative.drafters import load_drafter
76
  draft, kind = load_drafter("netprojects/Inkling-2-Mini-MLX-3bit")
77
+ print(generate(model, processor, prompt, max_tokens=2048, draft_model=draft, draft_kind=kind,
78
+ draft_block_size=4).text)
79
  ```
80
 
81
+ Server: `mlx_vlm.server --model netprojects/Inkling-2-Mini-MLX-3bit --draft-model netprojects/Inkling-2-Mini-MLX-3bit --draft-kind dflash --draft-block-size 4`.
82
+
83
  OpenAI-compatible server, with tool calls and the thinking returned separately (`reasoning_content`):
84
 
85
  ```bash
86
  mlx_vlm.server --model netprojects/Inkling-2-Mini-MLX-3bit --port 8000
87
  ```
88
 
89
+ ## Speed
90
+
91
+ MacBook Pro M5 Max (128 GB), idle, 21 November 2026; median of 3 runs. Decode: 5 prompts (math, code, chat) × 256 tokens, greedy, reasoning `none`. DSpark: the bundled draft at its best block size.
92
+
93
+ | Build | Prompt processing, 1K / 8K / 32K tokens (tok/s) | Generation (tok/s) | Generation with DSpark (tok/s) |
94
+ |---|---|---|---|
95
+ | 4-8bit | 1752 / 1658 / 1208 | 36.6 | 47.1 (×1.29, block 3) |
96
+ | 4bit | 1535 / 1412 / 1104 | 43.6 | 70.9 (×1.63, block 4) |
97
+ | 3bit (this build) | 1386 / 1351 / 1075 | 44.4 | 75.2 (×1.69, block 4) |
98
+ | 2.5bit | 1482 / 1384 / 1100 | 45.7 | 83.9 (×1.84, block 4) |
99
+
100
+ The draft's default block size (8) is slower on this model; use 3 or 4 as below.
101
+
102
  ## Thinking Machines' results
103
 
104
  From the [Inkling 2 Mini card](https://huggingface.co/thinkingmachines/Inkling-2-Mini), for the original model (their
 
130
  - **Coding:** 3/3 short Python tasks passed their asserts. **Thinking** (reasoning `low`): 2/2
131
  questions answered correctly after one closed thinking block. **Vision:** named the red square and blue circle of a test image.
132
  - **Memory:** peak 20.0 GB with a 4K-token prompt and 21.2 GB with an 8K-token prompt (mlx-vlm defaults). With a smaller prefill step the peak drops: 15.6 / 15.9 GB at `prefill_step_size=512` and 14.7 / 14.8 GB at `prefill_step_size=256` (4K / 8K-token prompts, plus 0.64 GB if the DSpark draft is loaded), which fits the default GPU memory limit of a 24 GB Mac: pass `prefill_step_size=256` to `generate`, or `--prefill-step-size 256` to `mlx_vlm.server`.
133
+ - **DSpark:** the bundled draft loads with this build; greedy output follows the model's own except at near-ties (checked: divergences only where the top two tokens were within 0.2 logits). Speeds are under "Speed".
134
 
135
  Known limitation of the current server: when the model calls a tool, `content` can also carry a short sentence of its
136
  reasoning and the tool name.