Instructions to run Inkling 2 Mini.

#2
by danielhanchen - opened

Hey guys, you can now run the model locally via Unsloth!

Update 4:30PM PST time: Context for Inkling should now be properly detected in Unsloth!
You can now load the model with proper contexts. Please update to the latest version.

To run in llama.cpp main, please use our separate uploads: https://huggingface.co/unsloth/Inkling-2-Mini-GGUF/tree/main/Mainline_Support

Or use our prebuilts at https://github.com/unslothai/llama.cpp - read https://unsloth.ai/docs/models/inkling-2-mini#run-inkling-2-mini-in-llama-cpp for how to run them

We also updated unsloth desktop with less memory usage and more speed!

Here's a short snippet of Blender + MCP + Unsloth + 4-bit Inkling Mini:

inkling mini in unsloth

•
This comment has been hidden (marked as Resolved)

latest llama.cpp can not load this model and raise errors. which version did you use?

0.00.283.155 E llama_model_load: error loading model: unknown model architecture: 'inkling2'
0.00.283.164 E llama_model_load_from_file_impl: failed to load model
0.00.283.167 E cmn common_init_: failed to load model 'Inkling-2-Mini-UD-IQ3_XXS.gguf'
0.00.283.169 E srv load_model: failed to load model, 'Inkling-2-Mini-UD-IQ3_XXS.gguf'
0.00.283.172 I srv operator(): operator(): cleaning up before exit...
0.00.289.575 E srv llama_server: exiting due to model loading error

Doesn't work for me either on the latest unsloth studio and latest llama nightly

support hasn't been merged yet, a few open PRs to I think, so should come soon

Will be up in a few mins sorry guys, the compiling took MUCH longer than we expected

Binaries are now up! Please update your llama.cpp runtimes. Sorry guys for the delays

Edit: It should be fully fixed, ensure you're using the latest unsloth version

CC: @coder543 @hellminster @hojnikb @rinoa @GiusTex @zhibo7 @MadMike477 @Linukso1D

Using 132k context at Q_K_XL with Q_8 cache, it will only run on a single 5090 if I offload most of the expert weights to system ram. If I put Unsloth in manual and force it to load to GPU, it will fail. Putting Unsloth on Auto, it will fit it into memory and I get the following results:

Inkling-2-Mini-GGUF loaded
All 49 layers are on the GPU. The expert weights of 47 layers are on the CPU. Model weights: 1.8 GiB on the GPU, 15 GiB on the CPU.

This will run but it's not usable.

Hey, thanks for the GGUFs! Got Inkling-2-Mini UD-Q8_K_XL running on 2× RTX 3090 with your b12568-mix-8eebad4 prebuild. I hit two memory issues along the way, sharing in case it helps:

  1. Layer split with the looped layers puts almost all weights on GPU 0. With --split-mode layer and no --tensor-split, CUDA0 got 20029 MiB of weights and CUDA1 got just 884 MiB (basically the output layer). Meanwhile the KV cache was split evenly, 2176 MiB per card. My guess is the split is computed over the 48 layer visits, while weights only exist for 24 physical layers. --tensor-split 1,3 works around it (7.8 / 13.1 GiB).
  2. Huge compute buffer at long context. At -c 262144 with q8_0 KV and FA on, the compute buffer on one card was ~26 GiB at the default -ub 512 (and ~42 GiB with pipeline parallelism). At -ub 128 it's still ~6.9 GiB per card. Looks like the dense relative-bias thing from your PR #385. Would be great to get #385 into the next prebuild. Also, #386 says quantized K/V stays on the dense path. Any chance the banded path will support q8_0 KV?

With -ub 128 + --tensor-split 1,3 it loads and the answers look sane. Decode is ~25 t/s on a short prompt, which feels slow for 3.4B active. Not sure if that's expected for the looped arch or if it's the dense bias over the full cache. Happy to help with testing if needed.

shimmyshimmer pinned discussion
shimmyshimmer changed discussion title from Run Inkling 2 Mini via Unsloth Desktop! to Instructions to run Inkling 2 Mini.

I compiled it from source and just got this
image

This is what my garbage free cloud clanker said:

Root cause
The failure is not the looped-layers part of PR 32695. It is a second, separate patch that is missing from this branch:
done_getting_tensors: wrong number of tensors; expected 734, got 714
The GGUF itself is fine: it has 734 tensors. The loader only requests 714. The 20 skipped tensors are exactly:
blk.0.ffn_gate_up_exps.scale
blk.1.ffn_gate_up_exps.scale
...
blk.19.ffn_gate_up_exps.scale
They are F32 per-expert scale tensors ({128}), and blocks 20-23 do not have them.
The generic .scale pass in src/llama-model.cpp loads scales for ffn_gate_exps, ffn_up_exps, and ffn_down_exps, but not for the merged ffn_gate_up_exps. In addition, src/models/inkling2.cpp does not pass any expert scale tensors to build_moe_ffn.
That fix exists, but on another branch in this fork:
- branch origin/gate-up-exps-scales
- fork PR #7: model: Add ffn_gate_up_exps scales
- commits 82d7e54d8, 1ddcaf614, f95a9946e
Your repeating-layers branch is based on 7e960f40f; the scale branch is based on 9d6f04fa9, so a plain merge is not clean.
Fix
The net change is two hunks:
1. In src/llama-model.cpp, request ffn_gate_up_exps.scale and reuse ffn_up_exps_s:
if (!layer.ffn_up_exps_s && layer.ffn_up_exps) {
    layer.ffn_up_exps_s = create_tensor(tn(LLM_TENSOR_FFN_UP_EXPS, "scale", i), {n_expert}, TENSOR_NOT_REQUIRED);
    if (layer.ffn_up_exps_s && layer.ffn_gate_up_exps) {
        throw std::runtime_error(format("ffn_up_exps.scale and ffn_gate_up_exps.weight both exist for layer %d", i));
    }
}
if (!layer.ffn_up_exps_s && layer.ffn_gate_up_exps) {
    layer.ffn_up_exps_s = create_tensor(tn(LLM_TENSOR_FFN_GATE_UP_EXPS, "scale", i), {n_expert}, TENSOR_NOT_REQUIRED);
}
2. In src/models/inkling2.cpp, pass the scales to build_moe_ffn:
                nullptr,
                layer.ffn_gate_up_exps,
                layer.ffn_up_exps_s,
                layer.ffn_gate_exps_s,
                layer.ffn_down_exps_s);
I generated the patch from those commits and confirmed it applies cleanly to your current repeating-layers HEAD:
/tmp/opencode/gate_up_scales.patch
git apply /tmp/opencode/gate_up_scales.patch
cmake --build build -j
Without the second hunk the model may load (if the tensor count check is bypassed) but use wrong expert scaling, so both parts are needed.
Want me to apply it and rebuild/test?

That fixed it but for some reason it's using way more VRAM that I would expect, I can't load it with any reasonable context window even with --no-kv-offload just way too fat for some reason getting Qwen 3.8 27B to load with somewhat acceptable settings seems easier hopefully just broken

llama.cpp repeating-layers  ❯ ./build/bin/llama-server -m ~/Downloads/Inkling-2-Mini-UD-Q4_K_M.gguf -ngl 1 -c 102400 --no-kv-offload
0.00.012.581 I srv  llama_server: initializing ...
0.00.012.594 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.012.878 W srv  server_setup: security: no API key is set and CORS allows all origins (see https://github.com/ggml-org/llama.cpp/pull/25655)
0.00.014.593 I srv    load_model: loading model '/home/noah/Downloads/Inkling-2-Mini-UD-Q4_K_M.gguf'
0.00.238.125 W common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to 1, abort
0.00.359.316 W load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
0.00.725.102 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 17507.75 MiB on device 0: cudaMalloc failed: out of memory
0.00.725.107 E ggml_gallocr_reserve_n_impl: failed to allocate ROCm0 buffer of size 18358209280
0.00.725.108 E graph_reserve: failed to allocate compute buffers
0.00.731.058 E llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers
0.00.731.068 E cmn  common_init_: failed to create context with model '/home/noah/Downloads/Inkling-2-Mini-UD-Q4_K_M.gguf'
0.00.731.073 E cmn  common_init_: failed to create context with model '/home/noah/Downloads/Inkling-2-Mini-UD-Q4_K_M.gguf'
0.00.731.078 E srv    load_model: failed to create_context with model '/home/noah/Downloads/Inkling-2-Mini-UD-Q4_K_M.gguf'
0.00.731.081 I srv    operator(): operator(): cleaning up before exit...
0.00.731.660 E srv  server_run_i: exiting due to model loading error

Managed to fix it by getting said clanker again to fix it, but it's slow, and I imagine code quality is crap

On 4090 with 135k context speed around 15t/s, compared to gemma 4 26b, where on gemma I have around 65t/s
Model q4 k xl
Looped transformer in action

getting around 11 to 17tk/s on vulkan
i have a 9070 xt

There is a lot of optimization left to be done, from what I’m seeing. Being a looped transformer doesn’t account for that, since the ~5B active parameters is already counting the looping.

There are also some suspicious, large memory allocations which the unsloth fork avoids in some cases, but mainline llama.cpp doesn’t seem to.

I think people just need to give this a few days or a week and it will hopefully be much better.

Definitely seems like it.

Hope once its a bit more optimized its a good accurate model. I've got a 12gb 4070 and 64GB RAM and im dying for something that can trade blows with 27b

Hey everyone, we re-did our prebuilts at https://github.com/unslothai/llama.cpp - read https://unsloth.ai/docs/models/inkling-2-mini#run-inkling-2-mini-in-llama-cpp for how to run them

We also updated unsloth desktop with less memory usage and more speed!

I loaded the updated Unsloth and got this using a 5090 with 128gb of system ram:

Model loaded with a warning
A context of 131,072 tokens does not fit in this GPU's memory with this model. The largest that fits is 77,312 tokens. The GPU cannot hold it, so layers will be moved to the CPU and generation will be slower. Lower the context to 77,312 or less, or leave it on Auto.

Inkling-2-Mini-GGUF loaded with vision on CPU
All 25 layers are on the GPU. The expert weights of 1 layer are on the CPU. Model weights: 15 GiB on the GPU, 1.5 GiB on the CPU. Unsloth is running the vision projector in system memory rather than on the GPU. Image input remains available, but image processing may be slower.

Sign up or log in to comment