Instructions to use unsloth/Inkling-2-Mini-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use unsloth/Inkling-2-Mini-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M # Run inference directly in the terminal: llama cli -hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M # Run inference directly in the terminal: llama cli -hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M
Use Docker
docker model run hf.co/unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use unsloth/Inkling-2-Mini-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "unsloth/Inkling-2-Mini-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "unsloth/Inkling-2-Mini-GGUF", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M
- Ollama
How to use unsloth/Inkling-2-Mini-GGUF with Ollama:
ollama run hf.co/unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M
- Unsloth Desktop
- Pi
How to use unsloth/Inkling-2-Mini-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use unsloth/Inkling-2-Mini-GGUF with Docker Model Runner:
docker model run hf.co/unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M
- Lemonade
How to use unsloth/Inkling-2-Mini-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M
Run and chat with the model
lemonade run user.Inkling-2-Mini-GGUF-UD-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use unsloth/Inkling-2-Mini-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use unsloth/Inkling-2-Mini-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Instructions to run Inkling 2 Mini.
Hey guys, you can now run the model locally via Unsloth!
Update 4:30PM PST time: Context for Inkling should now be properly detected in Unsloth!
You can now load the model with proper contexts. Please update to the latest version.
To run in llama.cpp main, please use our separate uploads: https://huggingface.co/unsloth/Inkling-2-Mini-GGUF/tree/main/Mainline_Support
Or use our prebuilts at https://github.com/unslothai/llama.cpp - read https://unsloth.ai/docs/models/inkling-2-mini#run-inkling-2-mini-in-llama-cpp for how to run them
We also updated unsloth desktop with less memory usage and more speed!
Here's a short snippet of Blender + MCP + Unsloth + 4-bit Inkling Mini:
latest llama.cpp can not load this model and raise errors. which version did you use?
0.00.283.155 E llama_model_load: error loading model: unknown model architecture: 'inkling2'
0.00.283.164 E llama_model_load_from_file_impl: failed to load model
0.00.283.167 E cmn common_init_: failed to load model 'Inkling-2-Mini-UD-IQ3_XXS.gguf'
0.00.283.169 E srv load_model: failed to load model, 'Inkling-2-Mini-UD-IQ3_XXS.gguf'
0.00.283.172 I srv operator(): operator(): cleaning up before exit...
0.00.289.575 E srv llama_server: exiting due to model loading error
Doesn't work for me either on the latest unsloth studio and latest llama nightly
support hasn't been merged yet, a few open PRs to I think, so should come soon
Will be up in a few mins sorry guys, the compiling took MUCH longer than we expected
Binaries are now up! Please update your llama.cpp runtimes. Sorry guys for the delays
Edit: It should be fully fixed, ensure you're using the latest unsloth version
CC: @coder543 @hellminster @hojnikb @rinoa @GiusTex @zhibo7 @MadMike477 @Linukso1D
Using 132k context at Q_K_XL with Q_8 cache, it will only run on a single 5090 if I offload most of the expert weights to system ram. If I put Unsloth in manual and force it to load to GPU, it will fail. Putting Unsloth on Auto, it will fit it into memory and I get the following results:
Inkling-2-Mini-GGUF loaded
All 49 layers are on the GPU. The expert weights of 47 layers are on the CPU. Model weights: 1.8 GiB on the GPU, 15 GiB on the CPU.
This will run but it's not usable.
Hey, thanks for the GGUFs! Got Inkling-2-Mini UD-Q8_K_XL running on 2× RTX 3090 with your b12568-mix-8eebad4 prebuild. I hit two memory issues along the way, sharing in case it helps:
- Layer split with the looped layers puts almost all weights on GPU 0. With --split-mode layer and no --tensor-split, CUDA0 got 20029 MiB of weights and CUDA1 got just 884 MiB (basically the output layer). Meanwhile the KV cache was split evenly, 2176 MiB per card. My guess is the split is computed over the 48 layer visits, while weights only exist for 24 physical layers. --tensor-split 1,3 works around it (7.8 / 13.1 GiB).
- Huge compute buffer at long context. At -c 262144 with q8_0 KV and FA on, the compute buffer on one card was ~26 GiB at the default -ub 512 (and ~42 GiB with pipeline parallelism). At -ub 128 it's still ~6.9 GiB per card. Looks like the dense relative-bias thing from your PR #385. Would be great to get #385 into the next prebuild. Also, #386 says quantized K/V stays on the dense path. Any chance the banded path will support q8_0 KV?
With -ub 128 + --tensor-split 1,3 it loads and the answers look sane. Decode is ~25 t/s on a short prompt, which feels slow for 3.4B active. Not sure if that's expected for the looped arch or if it's the dense bias over the full cache. Happy to help with testing if needed.
I compiled it from source and just got this
This is what my garbage free cloud clanker said:
Root cause
The failure is not the looped-layers part of PR 32695. It is a second, separate patch that is missing from this branch:
done_getting_tensors: wrong number of tensors; expected 734, got 714
The GGUF itself is fine: it has 734 tensors. The loader only requests 714. The 20 skipped tensors are exactly:
blk.0.ffn_gate_up_exps.scale
blk.1.ffn_gate_up_exps.scale
...
blk.19.ffn_gate_up_exps.scale
They are F32 per-expert scale tensors ({128}), and blocks 20-23 do not have them.
The generic .scale pass in src/llama-model.cpp loads scales for ffn_gate_exps, ffn_up_exps, and ffn_down_exps, but not for the merged ffn_gate_up_exps. In addition, src/models/inkling2.cpp does not pass any expert scale tensors to build_moe_ffn.
That fix exists, but on another branch in this fork:
- branch origin/gate-up-exps-scales
- fork PR #7: model: Add ffn_gate_up_exps scales
- commits 82d7e54d8, 1ddcaf614, f95a9946e
Your repeating-layers branch is based on 7e960f40f; the scale branch is based on 9d6f04fa9, so a plain merge is not clean.
Fix
The net change is two hunks:
1. In src/llama-model.cpp, request ffn_gate_up_exps.scale and reuse ffn_up_exps_s:
if (!layer.ffn_up_exps_s && layer.ffn_up_exps) {
layer.ffn_up_exps_s = create_tensor(tn(LLM_TENSOR_FFN_UP_EXPS, "scale", i), {n_expert}, TENSOR_NOT_REQUIRED);
if (layer.ffn_up_exps_s && layer.ffn_gate_up_exps) {
throw std::runtime_error(format("ffn_up_exps.scale and ffn_gate_up_exps.weight both exist for layer %d", i));
}
}
if (!layer.ffn_up_exps_s && layer.ffn_gate_up_exps) {
layer.ffn_up_exps_s = create_tensor(tn(LLM_TENSOR_FFN_GATE_UP_EXPS, "scale", i), {n_expert}, TENSOR_NOT_REQUIRED);
}
2. In src/models/inkling2.cpp, pass the scales to build_moe_ffn:
nullptr,
layer.ffn_gate_up_exps,
layer.ffn_up_exps_s,
layer.ffn_gate_exps_s,
layer.ffn_down_exps_s);
I generated the patch from those commits and confirmed it applies cleanly to your current repeating-layers HEAD:
/tmp/opencode/gate_up_scales.patch
git apply /tmp/opencode/gate_up_scales.patch
cmake --build build -j
Without the second hunk the model may load (if the tensor count check is bypassed) but use wrong expert scaling, so both parts are needed.
Want me to apply it and rebuild/test?
That fixed it but for some reason it's using way more VRAM that I would expect, I can't load it with any reasonable context window even with --no-kv-offload just way too fat for some reason getting Qwen 3.8 27B to load with somewhat acceptable settings seems easier hopefully just broken
llama.cpp repeating-layers ❯ ./build/bin/llama-server -m ~/Downloads/Inkling-2-Mini-UD-Q4_K_M.gguf -ngl 1 -c 102400 --no-kv-offload
0.00.012.581 I srv llama_server: initializing ...
0.00.012.594 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.012.878 W srv server_setup: security: no API key is set and CORS allows all origins (see https://github.com/ggml-org/llama.cpp/pull/25655)
0.00.014.593 I srv load_model: loading model '/home/noah/Downloads/Inkling-2-Mini-UD-Q4_K_M.gguf'
0.00.238.125 W common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to 1, abort
0.00.359.316 W load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
0.00.725.102 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 17507.75 MiB on device 0: cudaMalloc failed: out of memory
0.00.725.107 E ggml_gallocr_reserve_n_impl: failed to allocate ROCm0 buffer of size 18358209280
0.00.725.108 E graph_reserve: failed to allocate compute buffers
0.00.731.058 E llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers
0.00.731.068 E cmn common_init_: failed to create context with model '/home/noah/Downloads/Inkling-2-Mini-UD-Q4_K_M.gguf'
0.00.731.073 E cmn common_init_: failed to create context with model '/home/noah/Downloads/Inkling-2-Mini-UD-Q4_K_M.gguf'
0.00.731.078 E srv load_model: failed to create_context with model '/home/noah/Downloads/Inkling-2-Mini-UD-Q4_K_M.gguf'
0.00.731.081 I srv operator(): operator(): cleaning up before exit...
0.00.731.660 E srv server_run_i: exiting due to model loading error
Managed to fix it by getting said clanker again to fix it, but it's slow, and I imagine code quality is crap
On 4090 with 135k context speed around 15t/s, compared to gemma 4 26b, where on gemma I have around 65t/s
Model q4 k xl
Looped transformer in action
getting around 11 to 17tk/s on vulkan
i have a 9070 xt
There is a lot of optimization left to be done, from what I’m seeing. Being a looped transformer doesn’t account for that, since the ~5B active parameters is already counting the looping.
There are also some suspicious, large memory allocations which the unsloth fork avoids in some cases, but mainline llama.cpp doesn’t seem to.
I think people just need to give this a few days or a week and it will hopefully be much better.
Definitely seems like it.
Hope once its a bit more optimized its a good accurate model. I've got a 12gb 4070 and 64GB RAM and im dying for something that can trade blows with 27b
Hey everyone, we re-did our prebuilts at https://github.com/unslothai/llama.cpp - read https://unsloth.ai/docs/models/inkling-2-mini#run-inkling-2-mini-in-llama-cpp for how to run them
We also updated unsloth desktop with less memory usage and more speed!
I loaded the updated Unsloth and got this using a 5090 with 128gb of system ram:
Model loaded with a warning
A context of 131,072 tokens does not fit in this GPU's memory with this model. The largest that fits is 77,312 tokens. The GPU cannot hold it, so layers will be moved to the CPU and generation will be slower. Lower the context to 77,312 or less, or leave it on Auto.
Inkling-2-Mini-GGUF loaded with vision on CPU
All 25 layers are on the GPU. The expert weights of 1 layer are on the CPU. Model weights: 15 GiB on the GPU, 1.5 GiB on the CPU. Unsloth is running the vision projector in system memory rather than on the GPU. Image input remains available, but image processing may be slower.
