Inkling-2-Mini-MLX-5.8bit

Measured file size and KL against Original checkpoint weights 路 BF16 activations. Lower KL means closer output distributions.

Nace.AI weight conversions of Inkling 2 Mini.

Results

Held-out split. Mean KL against Original checkpoint weights 路 BF16 activations, over 2,040 scored token positions. Lower is closer.

KL uses llama.cpp scaled-uint16 reference probabilities and includes base log-probabilities above -16.

Artifact Text tensors, GB Text bits/weight Full runtime download, GB Mean KL
Nace native 5.8bit 18.932 5.809 20.234 0.004

Chart sizes and bits/weight cover text-model tensors only, excluding vision and MTP. The full runtime download also counts vision weights and MTP when present, plus runtime sidecars. Sizes use decimal GB. Uncertainty intervals are not plotted.

Scores and method 路 Chart data

Reference

Original NVFP4/MXFP8 weights are decoded with FP32 scale arithmetic and rounded to BF16 GEMM operands, with no affine requantization or activation QDQ. Torch BF16 linear operations return BF16; the CUDA backend uses FP32 accumulation with reduced-precision BF16 reductions disabled. Routing gamma is applied after the down projection; routed expert outputs are accumulated in FP32 and returned as BF16. Original FP32 controllers and routing are retained; the ordinary BF16 head output is widened before FP32 softcap. FP32 matmul TF32 is disabled. This is official HF eager text execution, not native W4A4/source-QDQ or cross-engine bitwise parity.

Method

Preserved expert tensors use the same weights as the official checkpoint. Codes and block scales stay packed, without requantization; outer scales follow the runtime F32 conversion. KL uses the BF16-activation reference described above.

Run

Install the Inkling MLX port, PR #2086, at the pinned commit.

MLX_REV=0c4c99192230f1943dd468d297f0b8a64ae7b7c2
python -m pip install \
  "mlx-lm @ git+https://github.com/ml-explore/mlx-lm.git@$MLX_REV"
python -m mlx_lm generate --model nace-ai/Inkling-2-Mini-MLX-5.8bit \
  --prompt "Explain binary search." --max-tokens 256

License

Apache 2.0. Original model by Thinking Machines Lab, Inc. See NOTICE and the source model card for attribution and its linked use policy.

Downloads last month
-
Safetensors
Model size
26B params
Tensor type
BF16
路
F32
路
U32
路
MLX
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 馃檵 Ask for provider support

Model tree for nace-ai/Inkling-2-Mini-MLX-5.8bit

Quantized
(40)
this model