Inkling-2-Mini โ€” MLX RCO 3.5 bpw (3-bit floor)

RCO (rate-constrained optimisation) mixed-precision affine quantisation of thinkingmachines/Inkling-2-Mini, built with the rco-mlx pipeline on Apple Silicon.

This is the lowest operating point reachable with a 3-bit floor. A nominal 3.0 bpw target is infeasible: with --bits 3,4,6,8 the cheapest allocation the allocator can build already costs 3.24 bpw, because the routed experts are 92.6 % of the parameters and affine 2-bit was deliberately excluded (it is unvalidated on this family and measured catastrophic elsewhere). This run targets 3.3 nominal and lands at 3.53 bpw real on the quantized weights.

Metrics

Placeholders. These will be replaced with measured values.

Metric Value
Perplexity (WikiText-2, 8ร—256 rows) 11.0172 (native NVFP4 10.5150, +0.50)
KL vs native NVFP4 checkpoint 0.0669
Top-1 agreement vs native 89.36 %
Zero-shot recovery (5 tasks, n=400) 0.9777 mean โ€” not task-lossless (arc_challenge 0.954, arc_easy 0.989, hellaswag 0.979, winogrande 0.995, mmlu 0.971)
Decode tok/s (local mlx_lm, 128 tok) 31.4 (native NVFP4 28.6)
Prompt tok/s 311 (native 264)
Peak RAM 13.97 GB (native 19.82 GB)

Sizes

Published checkpoint 4.5 bpw quant This quant
Total size 19.53 GB 16.57 GB 13.63 GB
bpw on quantized weights ~5.13 (experts) 4.46 3.53
Whole-checkpoint bpw 5.90 5.02 4.13

Bitwidth histogram (224 quantized tensors): 8-bit ร— 32, 6-bit ร— 24, 4-bit ร— 120, 3-bit ร— 48. All 48 expert tensors sit at the 3-bit floor.

What is quantised

  • Routed experts: one pre-stacked tensor per (layer, projection), so each is a single RCO group. All 48 at 3-bit.
  • Attention projections (q/k/v/r/o/gate/sink) and the full-attention weight_proj: 24 at 6-bit, 32 at 8-bit, the rest at 4-bit.
  • Left in BF16: embed_tokens, lm_head, every RMSNorm, the MoE router, the attention-residual / loop-message queries, the depthwise conv1d kernels, the relative-logits table, the rank-64 message gates, and the whole vision tower.

Source provenance โ€” read this

The published checkpoint stores its routed experts as NVFP4 (layers 0โ€“19) and MXFP8 (layers 20โ€“23); everything else is already BF16. There is no BF16 release of Inkling-2-Mini. RCO needs float weights, so the experts were first dequantised to BF16:

NVFP4 : W = E2M1[nibble] * FP8_E4M3[block scale] * (amax / outer_scale_denominator)
MXFP8 : W = FP8_E4M3[value]  * 2 ** (E8M0[scale] - 127)

This reconstruction is exact with respect to the shipped checkpoint but is not the original BF16: the NVFP4 rounding is already baked in. On the 20 NVFP4 layers the result can match native NVFP4, not beat it. Putting those layers at 3-bit affine is a real further compression and is where the quality cost of this checkpoint lives.

Loading

Requires an mlx-lm with Inkling 2 support (ml-explore/mlx-lm#2086, stacked on #2061) plus the inkling2_loop โ†’ inkling2 remapping in MODEL_REMAPPING. Both are draft PRs; until they land, load with the accompanying patched tree.

from mlx_lm.utils import load
model, tokenizer = load("hermitdave/Inkling-2-Mini-MLX-RCO-3.5bpw")

The DSpark draft (mtp.safetensors) is not included โ€” mlx-lm's Inkling 2 module has no MTP head.

Attribution

Quantisation pipeline by Hermes Agent using rco-mlx (hermitdave/rco-mlx). Model by Thinking Machines Lab, Apache 2.0.

Downloads last month
-
Safetensors
Model size
26B params
Tensor type
U32
ยท
BF16
ยท
MLX
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for hermitdave/Inkling-2-Mini-MLX-RCO-3.5bpw

Quantized
(39)
this model