Inkling-2-Mini โ€” MLX RCO 4.5 bpw

RCO (rate-constrained optimisation) mixed-precision affine quantisation of thinkingmachines/Inkling-2-Mini, built with the rco-mlx pipeline on Apple Silicon.

Inkling 2 Mini is a 26.4 B-parameter looped MoE VLM: 24 physical decoder layers of which the middle 12 run three times per token (48 layer visits), 128 routed experts with top-8 routing, a 23-layer vision tower, and โ€” in the published checkpoint โ€” a DSpark MTP draft.

Metrics

Placeholders. These will be replaced with measured values.

Metric Value
Perplexity (WikiText-2, 8ร—256 rows) 10.6018 (native NVFP4 10.5150, +0.09)
KL vs native NVFP4 checkpoint 0.0249
Top-1 agreement vs native 92.99 %
Zero-shot recovery (5 tasks, n=400) 0.9906 mean โ€” TASK-LOSSLESS (arc_challenge 0.996, arc_easy 1.000, hellaswag 0.995, winogrande 0.966, mmlu 0.996)
Decode tok/s (local mlx_lm, 128 tok) 30.9 (native NVFP4 28.6)
Prompt tok/s 309 (native 264)
Peak RAM 16.87 GB (native 19.82 GB)

Sizes

Published checkpoint This quant
Total size 19.53 GB 16.57 GB (4.46 bpw on quantized weights)
Routed experts 15.67 GB (NVFP4 + MXFP8) mixed 3/4-bit affine
Attention + MLP gates BF16 mixed 6/8-bit affine
Embeddings, lm_head, vision tower BF16 BF16 (unchanged)

Bitwidth histogram (224 quantized tensors): 8-bit ร— 118, 6-bit ร— 58, 4-bit ร— 44, 3-bit ร— 4.

What is quantised

  • Routed experts: one pre-stacked tensor per (layer, projection), so each is a single RCO group. 44 at 4-bit, 4 at 3-bit.
  • Attention projections (q/k/v/r/o/gate/sink) and the full-attention weight_proj: 58 at 6-bit, 118 at 8-bit.
  • Left in BF16: embed_tokens, lm_head, every RMSNorm, the MoE router, the attention-residual / loop-message queries, the depthwise conv1d kernels, the relative-logits table, the rank-64 message gates, and the whole vision tower.

Source provenance โ€” read this

The published checkpoint stores its routed experts as NVFP4 (layers 0โ€“19) and MXFP8 (layers 20โ€“23); everything else is already BF16. There is no BF16 release of Inkling-2-Mini. RCO needs float weights, so the experts were first dequantised to BF16:

NVFP4 : W = E2M1[nibble] * FP8_E4M3[block scale] * (amax / outer_scale_denominator)
MXFP8 : W = FP8_E4M3[value]  * 2 ** (E8M0[scale] - 127)

This reconstruction is exact with respect to the shipped checkpoint but is not the original BF16: the NVFP4 rounding is already baked in. On the 20 NVFP4 layers the result can match native NVFP4, not beat it. The allocation is where RCO can still win โ€” it spends 6โ€“8 bit on the attention projections instead of the fixed scheme the published checkpoint uses.

Loading

Requires an mlx-lm with Inkling 2 support (ml-explore/mlx-lm#2086, stacked on #2061) plus the inkling2_loop โ†’ inkling2 remapping in MODEL_REMAPPING. Both are draft PRs; until they land, load with the accompanying patched tree.

from mlx_lm.utils import load
model, tokenizer = load("hermitdave/Inkling-2-Mini-MLX-RCO-4.5bpw")

The DSpark draft (mtp.safetensors) is not included โ€” mlx-lm's Inkling 2 module has no MTP head.

Attribution

Quantisation pipeline by Hermes Agent using rco-mlx (hermitdave/rco-mlx). Model by Thinking Machines Lab, Apache 2.0.

Downloads last month
-
Safetensors
Model size
26B params
Tensor type
U32
ยท
BF16
ยท
MLX
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for hermitdave/Inkling-2-Mini-MLX-RCO-4.5bpw

Quantized
(39)
this model