Instructions to use hermitdave/Inkling-2-Mini-MLX-RCO-4.5bpw with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use hermitdave/Inkling-2-Mini-MLX-RCO-4.5bpw with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download hermitdave/Inkling-2-Mini-MLX-RCO-4.5bpw --local-dir Inkling-2-Mini-MLX-RCO-4.5bpw
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
Inkling-2-Mini โ MLX RCO 4.5 bpw
RCO (rate-constrained optimisation) mixed-precision affine quantisation of
thinkingmachines/Inkling-2-Mini,
built with the rco-mlx pipeline on Apple Silicon.
Inkling 2 Mini is a 26.4 B-parameter looped MoE VLM: 24 physical decoder layers of which the middle 12 run three times per token (48 layer visits), 128 routed experts with top-8 routing, a 23-layer vision tower, and โ in the published checkpoint โ a DSpark MTP draft.
Metrics
Placeholders. These will be replaced with measured values.
| Metric | Value |
|---|---|
| Perplexity (WikiText-2, 8ร256 rows) | 10.6018 (native NVFP4 10.5150, +0.09) |
| KL vs native NVFP4 checkpoint | 0.0249 |
| Top-1 agreement vs native | 92.99 % |
| Zero-shot recovery (5 tasks, n=400) | 0.9906 mean โ TASK-LOSSLESS (arc_challenge 0.996, arc_easy 1.000, hellaswag 0.995, winogrande 0.966, mmlu 0.996) |
Decode tok/s (local mlx_lm, 128 tok) |
30.9 (native NVFP4 28.6) |
| Prompt tok/s | 309 (native 264) |
| Peak RAM | 16.87 GB (native 19.82 GB) |
Sizes
| Published checkpoint | This quant | |
|---|---|---|
| Total size | 19.53 GB | 16.57 GB (4.46 bpw on quantized weights) |
| Routed experts | 15.67 GB (NVFP4 + MXFP8) | mixed 3/4-bit affine |
| Attention + MLP gates | BF16 | mixed 6/8-bit affine |
Embeddings, lm_head, vision tower |
BF16 | BF16 (unchanged) |
Bitwidth histogram (224 quantized tensors): 8-bit ร 118, 6-bit ร 58, 4-bit ร 44, 3-bit ร 4.
What is quantised
- Routed experts: one pre-stacked tensor per (layer, projection), so each is a single RCO group. 44 at 4-bit, 4 at 3-bit.
- Attention projections (
q/k/v/r/o/gate/sink) and the full-attentionweight_proj: 58 at 6-bit, 118 at 8-bit. - Left in BF16:
embed_tokens,lm_head, every RMSNorm, the MoE router, the attention-residual / loop-message queries, the depthwiseconv1dkernels, the relative-logits table, the rank-64 message gates, and the whole vision tower.
Source provenance โ read this
The published checkpoint stores its routed experts as NVFP4 (layers 0โ19) and MXFP8 (layers 20โ23); everything else is already BF16. There is no BF16 release of Inkling-2-Mini. RCO needs float weights, so the experts were first dequantised to BF16:
NVFP4 : W = E2M1[nibble] * FP8_E4M3[block scale] * (amax / outer_scale_denominator)
MXFP8 : W = FP8_E4M3[value] * 2 ** (E8M0[scale] - 127)
This reconstruction is exact with respect to the shipped checkpoint but is not the original BF16: the NVFP4 rounding is already baked in. On the 20 NVFP4 layers the result can match native NVFP4, not beat it. The allocation is where RCO can still win โ it spends 6โ8 bit on the attention projections instead of the fixed scheme the published checkpoint uses.
Loading
Requires an mlx-lm with Inkling 2 support (ml-explore/mlx-lm#2086,
stacked on #2061) plus the
inkling2_loop โ inkling2 remapping in MODEL_REMAPPING. Both are draft PRs;
until they land, load with the accompanying patched tree.
from mlx_lm.utils import load
model, tokenizer = load("hermitdave/Inkling-2-Mini-MLX-RCO-4.5bpw")
The DSpark draft (mtp.safetensors) is not included โ mlx-lm's Inkling 2
module has no MTP head.
Attribution
Quantisation pipeline by Hermes Agent using rco-mlx
(hermitdave/rco-mlx). Model by
Thinking Machines Lab, Apache 2.0.
- Downloads last month
- -
4-bit
Model tree for hermitdave/Inkling-2-Mini-MLX-RCO-4.5bpw
Base model
thinkingmachines/Inkling-2-Mini