Instructions to use hermitdave/Inkling-2-Mini-MLX-RCO-3.5bpw with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use hermitdave/Inkling-2-Mini-MLX-RCO-3.5bpw with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download hermitdave/Inkling-2-Mini-MLX-RCO-3.5bpw --local-dir Inkling-2-Mini-MLX-RCO-3.5bpw
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
Inkling-2-Mini โ MLX RCO 3.5 bpw (3-bit floor)
RCO (rate-constrained optimisation) mixed-precision affine quantisation of
thinkingmachines/Inkling-2-Mini,
built with the rco-mlx pipeline on Apple Silicon.
This is the lowest operating point reachable with a 3-bit floor. A nominal
3.0 bpw target is infeasible: with --bits 3,4,6,8 the cheapest allocation the
allocator can build already costs 3.24 bpw, because the routed experts are
92.6 % of the parameters and affine 2-bit was deliberately excluded (it is
unvalidated on this family and measured catastrophic elsewhere). This run
targets 3.3 nominal and lands at 3.53 bpw real on the quantized weights.
Metrics
Placeholders. These will be replaced with measured values.
| Metric | Value |
|---|---|
| Perplexity (WikiText-2, 8ร256 rows) | 11.0172 (native NVFP4 10.5150, +0.50) |
| KL vs native NVFP4 checkpoint | 0.0669 |
| Top-1 agreement vs native | 89.36 % |
| Zero-shot recovery (5 tasks, n=400) | 0.9777 mean โ not task-lossless (arc_challenge 0.954, arc_easy 0.989, hellaswag 0.979, winogrande 0.995, mmlu 0.971) |
Decode tok/s (local mlx_lm, 128 tok) |
31.4 (native NVFP4 28.6) |
| Prompt tok/s | 311 (native 264) |
| Peak RAM | 13.97 GB (native 19.82 GB) |
Sizes
| Published checkpoint | 4.5 bpw quant | This quant | |
|---|---|---|---|
| Total size | 19.53 GB | 16.57 GB | 13.63 GB |
| bpw on quantized weights | ~5.13 (experts) | 4.46 | 3.53 |
| Whole-checkpoint bpw | 5.90 | 5.02 | 4.13 |
Bitwidth histogram (224 quantized tensors): 8-bit ร 32, 6-bit ร 24, 4-bit ร 120, 3-bit ร 48. All 48 expert tensors sit at the 3-bit floor.
What is quantised
- Routed experts: one pre-stacked tensor per (layer, projection), so each is a single RCO group. All 48 at 3-bit.
- Attention projections (
q/k/v/r/o/gate/sink) and the full-attentionweight_proj: 24 at 6-bit, 32 at 8-bit, the rest at 4-bit. - Left in BF16:
embed_tokens,lm_head, every RMSNorm, the MoE router, the attention-residual / loop-message queries, the depthwiseconv1dkernels, the relative-logits table, the rank-64 message gates, and the whole vision tower.
Source provenance โ read this
The published checkpoint stores its routed experts as NVFP4 (layers 0โ19) and MXFP8 (layers 20โ23); everything else is already BF16. There is no BF16 release of Inkling-2-Mini. RCO needs float weights, so the experts were first dequantised to BF16:
NVFP4 : W = E2M1[nibble] * FP8_E4M3[block scale] * (amax / outer_scale_denominator)
MXFP8 : W = FP8_E4M3[value] * 2 ** (E8M0[scale] - 127)
This reconstruction is exact with respect to the shipped checkpoint but is not the original BF16: the NVFP4 rounding is already baked in. On the 20 NVFP4 layers the result can match native NVFP4, not beat it. Putting those layers at 3-bit affine is a real further compression and is where the quality cost of this checkpoint lives.
Loading
Requires an mlx-lm with Inkling 2 support (ml-explore/mlx-lm#2086,
stacked on #2061) plus the
inkling2_loop โ inkling2 remapping in MODEL_REMAPPING. Both are draft PRs;
until they land, load with the accompanying patched tree.
from mlx_lm.utils import load
model, tokenizer = load("hermitdave/Inkling-2-Mini-MLX-RCO-3.5bpw")
The DSpark draft (mtp.safetensors) is not included โ mlx-lm's Inkling 2
module has no MTP head.
Attribution
Quantisation pipeline by Hermes Agent using rco-mlx
(hermitdave/rco-mlx). Model by
Thinking Machines Lab, Apache 2.0.
- Downloads last month
- -
4-bit
Model tree for hermitdave/Inkling-2-Mini-MLX-RCO-3.5bpw
Base model
thinkingmachines/Inkling-2-Mini