Inkling-2-Mini-oQ4e-mtp

This model was quantized using oQ (oMLX v0.8.1) mixed-precision quantization.

A mixed-precision MLX checkpoint of thinkingmachines/Inkling-2-Mini, a looped multimodal mixture-of-experts model with 26.4B total and 3.4B active parameters. Its middle 12 layers run three times for each token. It takes text and image inputs with a context window of up to 1M tokens. It includes the language model, the vision encoder and the bundled DSpark draft in mtp.safetensors, which oMLX Lightning MTP uses for speculative decoding.

It needs oMLX 0.8.1 or later.

The effective average is 4.915 bits/weight for the language model and vision encoder, or 5.047 bits/weight including the DSpark draft.

Quantization and Bit Distribution

oQ4e keeps the routed experts of layers 0 to 19 in their source NVFP4 format and requantizes the MXFP8 experts of layers 20 to 23 to affine 4-bit. It stores the attention projections, the embeddings and the vocabulary head as affine 8-bit. The vision encoder and the DSpark draft keep BF16, and the router and the loop controllers keep FP32.

Storage format Logical weights Share of total Tensor storage Effective bits/weight
NVFP4, group 16 20.384B 77.247% 10.679 GiB 4.50
Affine 4-bit, group 64 4.077B 15.449% 2.136 GiB 4.50
Affine 8-bit, group 64 1.572B 5.958% 1.556 GiB 8.50
BF16 0.320B 1.211% 0.595 GiB 16.00
FP32 0.036B 0.135% 0.133 GiB 32.00
Total 26.389B 100% 15.098 GiB 4.915

Storage Breakdown

Component Logical weights Storage
Language model 26.070B 15.575 GB / 14.505 GiB
Vision 0.318B 0.637 GB / 0.593 GiB
DSpark draft (MTP) 0.318B 0.636 GB / 0.592 GiB
Total 26.707B 16.847 GB / 15.690 GiB

License

Apache 2.0, following the source model.

Downloads last month
-
Safetensors
Model size
26B params
Tensor type
U32
路
BF16
路
F32
路
MLX
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 馃檵 Ask for provider support

Model tree for mlx-community/Inkling-2-Mini-oQ4e-mtp

Quantized
(41)
this model