--- license: apache-2.0 base_model: thinkingmachines/Inkling-2-Mini pipeline_tag: image-text-to-text tags: - gguf - llama.cpp - inkling2 - moe --- # Inkling-2-Mini GGUF
Atomic Chat Discord GitHub X

GGUF files of [thinkingmachines/Inkling-2-Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini) by AtomicChat: a 26.4B mixture-of-experts model (3.4B active, 128 experts with 8 active) with a looped core, where the middle 12 of its 24 layers run three times. It takes text and image input. License: Apache-2.0, the same as the base model (LICENSE in this repo). > **Runtime: llama.cpp with [ggml-org/llama.cpp#32695](https://github.com/ggml-org/llama.cpp/pull/32695)**, a draft PR at the time of writing. The files are built at its commit `6196d98`. llama.cpp releases without that PR do not load the looped Inkling 2. ## Files AD (Atomic Dynamic) files give each tensor its own type. The experts hold 94% of the weights and carry the compression; attention and the loop controllers stay Q8_0. Quality is measured against our BF16: top-1 is how often the file picks the same next token as the original, and mean KLD is the divergence of the whole next-token distribution (lower is better). | file | GB on disk | top-1 | mean KLD | notes | |---|---:|---:|---:|---| | `BF16/` (2 parts) | 52.2 | — | — | the reference | | `Inkling-2-Mini-Q8_0.gguf` | 27.74 | 97.7% | 0.0033 | | | **`Inkling-2-Mini-AD-Q6_K.gguf`** | 23.79 | **97.6%** | **0.0037** | Q8_0 quality, 4 GB smaller | | `Inkling-2-Mini-Q6_K.gguf` | 23.40 | 97.3% | 0.0047 | llama.cpp preset, no imatrix | | `Inkling-2-Mini-Q5_K_M.gguf` | 19.81 | 96.3% | 0.0079 | llama.cpp preset, no imatrix | | **`Inkling-2-Mini-AD-Q5_K.gguf`** | 19.07 | **97.1%** | **0.0052** | | | `Inkling-2-Mini-Q4_K_M.gguf` | 17.38 | 94.7% | 0.0165 | llama.cpp preset, no imatrix | | **`Inkling-2-Mini-AD-Q4_K.gguf`** | 16.53 | **96.3%** | **0.0080** | Q5_K_M quality, 3.3 GB smaller | | **`Inkling-2-Mini-AD-IQ4_XS.gguf`** | 15.00 | **95.8%** | **0.0101** | | | **`Inkling-2-Mini-AD-IQ3_S.gguf`** | 13.34 | **94.2%** | **0.0205** | | | **`Inkling-2-Mini-AD-IQ3_XXS.gguf`** | 12.58 | **93.3%** | **0.0271** | | | **`Inkling-2-Mini-AD-IQ2_S.gguf`** | 11.56 | **91.3%** | **0.047** | | | **`Inkling-2-Mini-AD-IQ2_XXS.gguf`** | 10.43 | **88.5%** | **0.084** | | | **`Inkling-2-Mini-AD-IQ2_M.gguf`** | 8.67 | **85.7%** | **0.130** | expert down Q2_0 except the 8 busiest layers | | **`Inkling-2-Mini-AD-IQ2_XXS-Q2.gguf`** | 7.90 | **84.3%** | **0.158** | | | **`Inkling-2-Mini-AD-IQ1_M.gguf`** | 7.26 | **81.1%** | **0.236** | 1-bit | | **`Inkling-2-Mini-AD-IQ1_S.gguf`** | 6.88 | **79.5%** | **0.278** | 1-bit | | `mmproj-Inkling-2-Mini-F16.gguf` / `-BF16.gguf` | 0.64 | | | vision projector (image input) | Even Q8_0 matches the BF16's top token only 97.7% of the time on this model, so 97–98% is the ceiling here, not 99%+. Our guess is that rounding adds up over the 48 layer passes of the loop; we have not tested it. - **Measured:** our neutral corpus in 30 languages, 87 chunks of 4096 tokens, llama.cpp `llama-perplexity --kl-divergence` against the BF16 logits, one H100 build of PR #32695. - **How to choose:** the memory a file needs is its size plus the KV cache plus about 1 GB of buffers. The KV cache is small here: only 4 of the 24 layers use full attention, the rest a 1024-token window. Good 4-bit files reach 96–97% top-1. The 1- and 2-bit files are for machines where nothing larger fits. - **The presets' names don't tell the whole story.** The expert down projection has rows of 1152, which k-quants cannot take, so llama.cpp swaps the type there: Q8_0 in all 24 layers of Q6_K; Q8_0 in half of the layers and Q5_1 in the rest for Q5_K_M; Q8_0 and Q5_0 for Q4_K_M. The AD files use block-32 types on that tensor by design. The 1- and 2-bit files put Q2_0 there, quantized with the importance matrix (`patches/q2_0-imatrix.patch`; llama.cpp's own Q2_0 quantizer ignores the matrix). - **Importance matrix:** 4.76M tokens of our calibration corpus rendered through the model's chat template, in 6 shards. Two shards ran on the BF16. Four ran on the Q8_0, because no GPU with room for the 52 GB BF16 was free at release time. - **Base:** the vendor ships the weights in NVFP4 and MXFP8; our BF16 is that checkpoint dequantized. ## Run Build llama.cpp from the PR: ```bash git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp git fetch origin pull/32695/head:inkling2-loop && git checkout inkling2-loop cmake -B build -DGGML_CUDA=ON && cmake --build build -j --config Release # -DGGML_METAL=ON on a Mac ``` ```bash ./build/bin/llama-server -m Inkling-2-Mini-AD-Q4_K.gguf --mmproj mmproj-Inkling-2-Mini-F16.gguf --jinja -c 32768 -ngl 99 ``` - **Thinking** is set by the chat template's `reasoning_effort`: `none`, `minimal`, `low`, `medium`, `high`, `xhigh` (default 0.9). Pass it as `"chat_template_kwargs": {"reasoning_effort": "low"}` on `/v1/chat/completions`. The vendor notes that `none` does not switch reasoning off completely. - **Sampling:** the vendor's generation config uses temperature 1.0. - **Checked** in llama-server (BF16, AD-IQ1_M, AD-IQ1_S, with the F16 projector): plain chat, reasoning at `none` and `high`, a tool call, an image question, and a reply in Chinese. ## Related - **Uncensored variant:** [AtomicChat/Inkling-2-Mini-Uncensored-GGUF](https://huggingface.co/AtomicChat/Inkling-2-Mini-Uncensored-GGUF): a LoRA adapter for these files, and AD-Q4_K, AD-IQ3_S and AD-IQ2_M with the edit baked in.