Instructions to use nace-ai/Inkling-2-Mini-MLX-5.8bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use nace-ai/Inkling-2-Mini-MLX-5.8bit with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download nace-ai/Inkling-2-Mini-MLX-5.8bit --local-dir Inkling-2-Mini-MLX-5.8bit
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
Inkling-2-Mini-MLX-5.8bit
Nace.AI weight conversions of Inkling 2 Mini.
Results
Held-out split. Mean KL against Original checkpoint weights 路 BF16 activations, over 2,040 scored token positions. Lower is closer.
KL uses llama.cpp scaled-uint16 reference probabilities and includes base log-probabilities above -16.
| Artifact | Text tensors, GB | Text bits/weight | Full runtime download, GB | Mean KL |
|---|---|---|---|---|
| Nace native 5.8bit | 18.932 | 5.809 | 20.234 | 0.004 |
Chart sizes and bits/weight cover text-model tensors only, excluding vision and MTP. The full runtime download also counts vision weights and MTP when present, plus runtime sidecars. Sizes use decimal GB. Uncertainty intervals are not plotted.
Scores and method 路 Chart data
Reference
Original NVFP4/MXFP8 weights are decoded with FP32 scale arithmetic and rounded to BF16 GEMM operands, with no affine requantization or activation QDQ. Torch BF16 linear operations return BF16; the CUDA backend uses FP32 accumulation with reduced-precision BF16 reductions disabled. Routing gamma is applied after the down projection; routed expert outputs are accumulated in FP32 and returned as BF16. Original FP32 controllers and routing are retained; the ordinary BF16 head output is widened before FP32 softcap. FP32 matmul TF32 is disabled. This is official HF eager text execution, not native W4A4/source-QDQ or cross-engine bitwise parity.
Method
Preserved expert tensors use the same weights as the official checkpoint. Codes and block scales stay packed, without requantization; outer scales follow the runtime F32 conversion. KL uses the BF16-activation reference described above.
Run
Install the Inkling MLX port, PR #2086, at the pinned commit.
MLX_REV=0c4c99192230f1943dd468d297f0b8a64ae7b7c2
python -m pip install \
"mlx-lm @ git+https://github.com/ml-explore/mlx-lm.git@$MLX_REV"
python -m mlx_lm generate --model nace-ai/Inkling-2-Mini-MLX-5.8bit \
--prompt "Explain binary search." --max-tokens 256
License
Apache 2.0. Original model by Thinking Machines Lab, Inc. See NOTICE and the source model card for attribution and its linked use policy.
- Downloads last month
- -
4-bit
Model tree for nace-ai/Inkling-2-Mini-MLX-5.8bit
Base model
thinkingmachines/Inkling-2-Mini