Inkling-2-Small GGUF (~150 GB)

A ~150 GB GGUF of thinkingmachines/Inkling-2-Small for llama.cpp, made to keep as much of the original model's quality as possible at this size. All credit for the model goes to Thinking Machines; it is released under the Apache-2.0 license, and so is this quant.

Files

File Size What
Inkling-2-Small-IQ2_XXS-mix-00001-of-00004.gguf ... -00004-of-00004.gguf 150.0 GB in total the language model (pass the first shard; keep all four in one folder)
mmproj-Inkling-2-Small-F16.gguf 0.66 GB vision projector (--mmproj)
Inkling-2-Small-MTP-BF16.gguf 3.4 GB the model's multi-token-prediction block, weights unchanged (BF16)

The MTP file holds Inkling's draft block (its DSpark speculator), weights unchanged. These files run with llama.cpp PR 32187 (text and vision), which does not load the draft block. llama.cpp's DSpark support for Inkling 2 (PR 32218, on top of PR 32217) uses a different GGUF layout for both the language model and the draft; files for that layout will follow once it is final.

How it was made

  • Routed experts: per-expert error-feedback (GPTQ-style) rounding, each layer calibrated on the output of the already-quantized layers before it, with the rounding settings tuned for this model on held-out data.
  • Bits per layer from a measured per-layer sensitivity sweep: gate/up at IQ2_XXS everywhere; down projections set per layer by that sweep (IQ1_S in 30 layers, IQ1_M in 2, IQ2_XXS in 14, IQ2_XS in the 2 most sensitive).
  • Every non-expert tensor kept high: attention Q8_0, attention output and output head Q6_K, embeddings Q4_K, router and norms F32.
  • Calibration: a neutral mix of chat, code, tool use and long-form text in the model's own chat format, with a held-out set never used for calibration.

Quality

This GGUF, run in llama.cpp, measured against the original model on held-out text never used for calibration (32 x 2048 tokens of chat, code, tool use and long-form text; the reference is the original model's top-64 log-probabilities at every scored position):

Quant Size (GB) Mean KLD Median KLD Same top token PPL vs original
this quant 150.0 0.114 0.0026 94.3% +3.6%

Running

Inkling 2 needs a llama.cpp build with Inkling 2 support. Until it is merged upstream, build the pull request (ggml-org/llama.cpp PR 32187):

git clone https://github.com/ggml-org/llama.cpp
git -C llama.cpp fetch origin pull/32187/head:inkling2
git -C llama.cpp checkout inkling2
cmake -S llama.cpp -B llama.cpp/build -DGGML_CUDA=ON && cmake --build llama.cpp/build -j --target llama-server
hf download peasantsmith/Inkling-2-Small-GGUF --local-dir Inkling-2-Small-GGUF
./llama.cpp/build/bin/llama-server \
    -m Inkling-2-Small-GGUF/Inkling-2-Small-IQ2_XXS-mix-00001-of-00004.gguf \
    --mmproj Inkling-2-Small-GGUF/mmproj-Inkling-2-Small-F16.gguf \
    --jinja -ngl 99 -c 32768 \
    --temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0

Keep all shards in one folder. If the model does not fit in your GPU memory, keep some or all experts in system RAM with --n-cpu-moe N (the experts of the first N layers) or --cpu-moe (all of them). The chat template is embedded; --chat-template-kwargs '{"reasoning_effort":"medium"}' changes the thinking effort (none ... max, default high).

License

Apache-2.0, like the original model. Model by Thinking Machines.

Downloads last month
1,997
GGUF
Model size
594B params
Architecture
inkling2
Hardware compatibility
Log In to add your hardware

2-bit

16-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for peasantsmith/Inkling-2-Small-GGUF

Quantized
(19)
this model