Inkling-2-Mini-oQ3e-mtp

This model was quantized using oQ (oMLX v0.8.1) mixed-precision quantization.

A mixed-precision MLX checkpoint of thinkingmachines/Inkling-2-Mini, a looped multimodal mixture-of-experts model with 26.4B total and 3.4B active parameters. Its middle 12 layers run three times for each token. It takes text and image inputs with a context window of up to 1M tokens. It includes the language model, the vision encoder and the bundled DSpark draft in mtp.safetensors, which oMLX Lightning MTP uses for speculative decoding.

It needs oMLX 0.8.1 or later.

The effective average is 3.953 bits/weight for the language model and vision encoder, or 4.096 bits/weight including the DSpark draft.

Quantization and Bit Distribution

oQ3e requantizes the routed experts of all layers to affine 3-bit. It stores the attention projections and the embeddings as affine 8-bit and the vocabulary head as affine 6-bit. The vision encoder and the DSpark draft keep BF16, and the router and the loop controllers keep FP32.

Storage format Logical weights Share of total Tensor storage Effective bits/weight
Affine 3-bit, group 64 24.461B 92.696% 9.967 GiB 3.50
Affine 8-bit, group 64 1.109B 4.203% 1.098 GiB 8.50
BF16 0.320B 1.211% 0.595 GiB 16.00
Affine 6-bit, group 64 0.463B 1.755% 0.350 GiB 6.50
FP32 0.036B 0.135% 0.133 GiB 32.00
Total 26.389B 100% 12.143 GiB 3.953

Storage Breakdown

Component Logical weights Storage
Language model 26.070B 12.402 GB / 11.550 GiB
Vision 0.318B 0.637 GB / 0.593 GiB
DSpark draft (MTP) 0.318B 0.636 GB / 0.592 GiB
Total 26.707B 13.674 GB / 12.735 GiB

License

Apache 2.0, following the source model.

Downloads last month
-
Safetensors
Model size
26B params
Tensor type
U32
路
BF16
路
F32
路
MLX
Hardware compatibility
Log In to add your hardware

3-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 馃檵 Ask for provider support

Model tree for mlx-community/Inkling-2-Mini-oQ3e-mtp

Quantized
(40)
this model