Instructions to use AtomicChat/Inkling-2-Mini-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use AtomicChat/Inkling-2-Mini-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf AtomicChat/Inkling-2-Mini-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf AtomicChat/Inkling-2-Mini-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf AtomicChat/Inkling-2-Mini-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf AtomicChat/Inkling-2-Mini-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf AtomicChat/Inkling-2-Mini-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf AtomicChat/Inkling-2-Mini-GGUF:Q4_K_M
Use Docker
docker model run hf.co/AtomicChat/Inkling-2-Mini-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use AtomicChat/Inkling-2-Mini-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "AtomicChat/Inkling-2-Mini-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AtomicChat/Inkling-2-Mini-GGUF", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/AtomicChat/Inkling-2-Mini-GGUF:Q4_K_M
- Ollama
How to use AtomicChat/Inkling-2-Mini-GGUF with Ollama:
ollama run hf.co/AtomicChat/Inkling-2-Mini-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use AtomicChat/Inkling-2-Mini-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf AtomicChat/Inkling-2-Mini-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "AtomicChat/Inkling-2-Mini-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use AtomicChat/Inkling-2-Mini-GGUF with Docker Model Runner:
docker model run hf.co/AtomicChat/Inkling-2-Mini-GGUF:Q4_K_M
- Lemonade
How to use AtomicChat/Inkling-2-Mini-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull AtomicChat/Inkling-2-Mini-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Inkling-2-Mini-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use AtomicChat/Inkling-2-Mini-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf AtomicChat/Inkling-2-Mini-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default AtomicChat/Inkling-2-Mini-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use AtomicChat/Inkling-2-Mini-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf AtomicChat/Inkling-2-Mini-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "AtomicChat/Inkling-2-Mini-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
|
Download README.md from AtomicChat/Inkling-2-Mini-GGUF: direct link, hf CLI and curl.
- Browser
- Download file 6.65 kB
-
https://huggingface.co/AtomicChat/Inkling-2-Mini-GGUF/resolve/main/README.md
- Command line
-
hf download hf://AtomicChat/Inkling-2-Mini-GGUF/README.md
-
curl -L -o README.md https://huggingface.co/AtomicChat/Inkling-2-Mini-GGUF/resolve/main/README.md
6.65 kB
| license: apache-2.0 | |
| base_model: thinkingmachines/Inkling-2-Mini | |
| pipeline_tag: image-text-to-text | |
| tags: | |
| - gguf | |
| - llama.cpp | |
| - inkling2 | |
| - moe | |
| # Inkling-2-Mini GGUF | |
| <div style="display: flex; gap: 8px; align-items: center; margin-top: 10px; margin-bottom: 10px;"> | |
| <a href="https://atomic.chat/?utm_source=huggingface&utm_medium=referral&utm_campaign=hf_inkling_2_mini&utm_content=btn_atomic"><img src="https://huggingface.co/AtomicChat/Qwen3.8-27B-GGUF/resolve/main/btn_atomic.png" width="162" alt="Atomic Chat"></a> | |
| <a href="https://discord.gg/8wGSsvmg4V"><img src="https://huggingface.co/AtomicChat/Qwen3.8-27B-GGUF/resolve/main/btn_discord.png" width="119" alt="Discord"></a> | |
| <a href="https://github.com/AtomicBot-ai/Atomic-Chat"><img src="https://huggingface.co/AtomicChat/Qwen3.8-27B-GGUF/resolve/main/btn_github.png" width="115" alt="GitHub"></a> | |
| <a href="https://x.com/atomic_chat_hq"><img src="https://huggingface.co/AtomicChat/Qwen3.8-27B-GGUF/resolve/main/btn_x.png" width="108" alt="X"></a> | |
| </div> | |
| <ul style="margin: 0 0 12px 0;"> | |
| <li>Built by the team behind <a href="https://atomic.chat/?utm_source=huggingface&utm_medium=referral&utm_campaign=hf_inkling_2_mini&utm_content=bullet_app">Atomic Chat</a>, the app that runs open-weight models locally and offline.</li> | |
| </ul> | |
| <hr style="margin: 0 0 16px 0;"> | |
| GGUF files of [thinkingmachines/Inkling-2-Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini) by AtomicChat: a 26.4B mixture-of-experts model (3.4B active, 128 experts with 8 active) with a looped core, where the middle 12 of its 24 layers run three times. It takes text and image input. License: Apache-2.0, the same as the base model (LICENSE in this repo). | |
| > **Runtime: llama.cpp with [ggml-org/llama.cpp#32695](https://github.com/ggml-org/llama.cpp/pull/32695)**, a draft PR at the time of writing. The files are built at its commit `6196d98`. llama.cpp releases without that PR do not load the looped Inkling 2. | |
| ## Files | |
| AD (Atomic Dynamic) files give each tensor its own type. The experts hold 94% of the weights and carry the compression; attention and the loop controllers stay Q8_0. Quality is measured against our BF16: top-1 is how often the file picks the same next token as the original, and mean KLD is the divergence of the whole next-token distribution (lower is better). | |
| | file | GB on disk | top-1 | mean KLD | notes | | |
| |---|---:|---:|---:|---| | |
| | `BF16/` (2 parts) | 52.2 | — | — | the reference | | |
| | `Inkling-2-Mini-Q8_0.gguf` | 27.74 | 97.7% | 0.0033 | | | |
| | **`Inkling-2-Mini-AD-Q6_K.gguf`** | 23.79 | **97.6%** | **0.0037** | Q8_0 quality, 4 GB smaller | | |
| | `Inkling-2-Mini-Q6_K.gguf` | 23.40 | 97.3% | 0.0047 | llama.cpp preset, no imatrix | | |
| | `Inkling-2-Mini-Q5_K_M.gguf` | 19.81 | 96.3% | 0.0079 | llama.cpp preset, no imatrix | | |
| | **`Inkling-2-Mini-AD-Q5_K.gguf`** | 19.07 | **97.1%** | **0.0052** | | | |
| | `Inkling-2-Mini-Q4_K_M.gguf` | 17.38 | 94.7% | 0.0165 | llama.cpp preset, no imatrix | | |
| | **`Inkling-2-Mini-AD-Q4_K.gguf`** | 16.53 | **96.3%** | **0.0080** | Q5_K_M quality, 3.3 GB smaller | | |
| | **`Inkling-2-Mini-AD-IQ4_XS.gguf`** | 15.00 | **95.8%** | **0.0101** | | | |
| | **`Inkling-2-Mini-AD-IQ3_S.gguf`** | 13.34 | **94.2%** | **0.0205** | | | |
| | **`Inkling-2-Mini-AD-IQ3_XXS.gguf`** | 12.58 | **93.3%** | **0.0271** | | | |
| | **`Inkling-2-Mini-AD-IQ2_S.gguf`** | 11.56 | **91.3%** | **0.047** | | | |
| | **`Inkling-2-Mini-AD-IQ2_XXS.gguf`** | 10.43 | **88.5%** | **0.084** | | | |
| | **`Inkling-2-Mini-AD-IQ2_M.gguf`** | 8.67 | **85.7%** | **0.130** | expert down Q2_0 except the 8 busiest layers | | |
| | **`Inkling-2-Mini-AD-IQ2_XXS-Q2.gguf`** | 7.90 | **84.3%** | **0.158** | | | |
| | **`Inkling-2-Mini-AD-IQ1_M.gguf`** | 7.26 | **81.1%** | **0.236** | 1-bit | | |
| | **`Inkling-2-Mini-AD-IQ1_S.gguf`** | 6.88 | **79.5%** | **0.278** | 1-bit | | |
| | `mmproj-Inkling-2-Mini-F16.gguf` / `-BF16.gguf` | 0.64 | | | vision projector (image input) | | |
| Even Q8_0 matches the BF16's top token only 97.7% of the time on this model, so 97–98% is the ceiling here, not 99%+. Our guess is that rounding adds up over the 48 layer passes of the loop; we have not tested it. | |
| - **Measured:** our neutral corpus in 30 languages, 87 chunks of 4096 tokens, llama.cpp `llama-perplexity --kl-divergence` against the BF16 logits, one H100 build of PR #32695. | |
| - **How to choose:** the memory a file needs is its size plus the KV cache plus about 1 GB of buffers. The KV cache is small here: only 4 of the 24 layers use full attention, the rest a 1024-token window. Good 4-bit files reach 96–97% top-1. The 1- and 2-bit files are for machines where nothing larger fits. | |
| - **The presets' names don't tell the whole story.** The expert down projection has rows of 1152, which k-quants cannot take, so llama.cpp swaps the type there: Q8_0 in all 24 layers of Q6_K; Q8_0 in half of the layers and Q5_1 in the rest for Q5_K_M; Q8_0 and Q5_0 for Q4_K_M. The AD files use block-32 types on that tensor by design. The 1- and 2-bit files put Q2_0 there, quantized with the importance matrix (`patches/q2_0-imatrix.patch`; llama.cpp's own Q2_0 quantizer ignores the matrix). | |
| - **Importance matrix:** 4.76M tokens of our calibration corpus rendered through the model's chat template, in 6 shards. Two shards ran on the BF16. Four ran on the Q8_0, because no GPU with room for the 52 GB BF16 was free at release time. | |
| - **Base:** the vendor ships the weights in NVFP4 and MXFP8; our BF16 is that checkpoint dequantized. | |
| ## Run | |
| Build llama.cpp from the PR: | |
| ```bash | |
| git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp | |
| git fetch origin pull/32695/head:inkling2-loop && git checkout inkling2-loop | |
| cmake -B build -DGGML_CUDA=ON && cmake --build build -j --config Release # -DGGML_METAL=ON on a Mac | |
| ``` | |
| ```bash | |
| ./build/bin/llama-server -m Inkling-2-Mini-AD-Q4_K.gguf --mmproj mmproj-Inkling-2-Mini-F16.gguf --jinja -c 32768 -ngl 99 | |
| ``` | |
| - **Thinking** is set by the chat template's `reasoning_effort`: `none`, `minimal`, `low`, `medium`, `high`, `xhigh` (default 0.9). Pass it as `"chat_template_kwargs": {"reasoning_effort": "low"}` on `/v1/chat/completions`. The vendor notes that `none` does not switch reasoning off completely. | |
| - **Sampling:** the vendor's generation config uses temperature 1.0. | |
| - **Checked** in llama-server (BF16, AD-IQ1_M, AD-IQ1_S, with the F16 projector): plain chat, reasoning at `none` and `high`, a tool call, an image question, and a reply in Chinese. | |
| ## Related | |
| - **Uncensored variant:** [AtomicChat/Inkling-2-Mini-Uncensored-GGUF](https://huggingface.co/AtomicChat/Inkling-2-Mini-Uncensored-GGUF): a LoRA adapter for these files, and AD-Q4_K, AD-IQ3_S and AD-IQ2_M with the edit baked in. | |