Inkling-2-Mini-GGUF

Measured file size and KL against Original checkpoint weights ยท BF16 activations. Lower KL means closer output distributions.

Nace.AI weight conversions of Inkling 2 Mini.

Results

Held-out split. Mean KL against Original checkpoint weights ยท BF16 activations, over 2,040 scored token positions. Lower is closer.

KL uses llama.cpp scaled-uint16 reference probabilities and includes base log-probabilities above -16.

Artifact Size, GB Bits/weight Mean KL
Nace NVFP4 19.103 5.862 0.004

Sizes cover each text-backbone GGUF file. Vision and MTP are omitted. Sizes use decimal GB. Uncertainty intervals are not plotted.

Scores and method ยท Chart data

Reference

Original NVFP4/MXFP8 weights are decoded with FP32 scale arithmetic and rounded to BF16 GEMM operands, with no affine requantization or activation QDQ. Torch BF16 linear operations return BF16; the CUDA backend uses FP32 accumulation with reduced-precision BF16 reductions disabled. Routing gamma is applied after the down projection; routed expert outputs are accumulated in FP32 and returned as BF16. Original FP32 controllers and routing are retained; the ordinary BF16 head output is widened before FP32 softcap. FP32 matmul TF32 is disabled. This is official HF eager text execution, not native W4A4/source-QDQ or cross-engine bitwise parity.

Method

Matching native expert blocks keep the same weights as the official checkpoint and are repacked without requantization. Native-block round-trip check scope: sampled original NVFP4 rows, exact scale-inclusive dequantization after gate/up permutation; all global scales compared; eight lossy MXFP8-to-Q8_0 tensors excluded. Q8_0 fallback tensors: blk.20.ffn_gate_up_exps.weight, blk.20.ffn_down_exps.weight, blk.21.ffn_gate_up_exps.weight, blk.21.ffn_down_exps.weight, blk.22.ffn_gate_up_exps.weight, blk.22.ffn_down_exps.weight, blk.23.ffn_gate_up_exps.weight, blk.23.ffn_down_exps.weight. The native-block check does not cover the requantized fallback tensors.

Run

Build the CUDA runtime from llama.cpp PR #32704 at the pinned commit.

git clone https://github.com/ggml-org/llama.cpp.git
git -C llama.cpp fetch origin refs/pull/32704/head
git -C llama.cpp checkout 2a28df0f9bd55c4975b6bffb35229a6020720f13
cmake -S llama.cpp -B llama.cpp/build -DGGML_CUDA=ON -DGGML_METAL=OFF
cmake --build llama.cpp/build --config Release -j
llama.cpp/build/bin/llama-cli -hf nace-ai/Inkling-2-Mini-GGUF \
  --hf-file Inkling-2-Mini-NVFP4.gguf \
  -p "Explain binary search." -n 256

License

Apache 2.0. Original model by Thinking Machines Lab, Inc. See NOTICE and the source model card for attribution and its linked use policy.

Downloads last month
-
GGUF
Model size
26B params
Architecture
inkling2_loop
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for nace-ai/Inkling-2-Mini-GGUF

Quantized
(37)
this model