Inkling-2-Mini-IQ3_XXS-GGUF

Inkling 2 Mini is a multimodal looped mixture-of-experts model from Thinking Machines Lab: 26.4B parameters, 3.4B active (5.4B effective with three passes through the looped middle layers), text and image input, up to 1M tokens of context. This repository is an unmodified quantization of the official checkpoint (revision 1fb4134).

GGUF IQ3_XXS (importance-matrix calibrated) for llama.cpp and LM Studio, with the vision projector. The smallest build here, for GPUs and Macs with 16 GB of memory.

Other formats from INTERPRETIC AI: GGUF Q4_K_M ยท GGUF IQ3_XXS ยท MLX 4-bit

Files

File Description Size
Inkling-2-Mini-IQ3_XXS.gguf Language model, IQ3_XXS 11.7 GB
mmproj-Inkling-2-Mini-F16.gguf Vision projector, F16 0.64 GB

SHA-256 checksums are listed in SHA256SUMS.

Usage

Requires llama.cpp with Inkling 2 Mini support (commit 29b52d3 or later).

llama-server -m Inkling-2-Mini-IQ3_XXS.gguf \
  --mmproj mmproj-Inkling-2-Mini-F16.gguf \
  -ngl 999 -c 32768 --jinja

The --mmproj file is only needed for image input. Measured on one RTX PRO 6000: about 86 tokens/s single-stream generation, 15.5 GB of GPU memory at 16K context. IQ3_XXS trades some quality for size; use the Q4_K_M build if it fits.

Notes

  • Inkling 2 Mini is a thinking model: it writes its reasoning before the answer. Use a chat client or server that applies the model's chat template.
  • Quality checks here are spot checks (English, Chinese, code and one image), not a full benchmark.
  • Support for this architecture is new in all runtimes. Use a recent build.

License

Apache 2.0, as the original model. See the model card of Thinking Machines Lab and its acceptable use policy.

INTERPRETIC AI ยท interpretic.ai

Downloads last month
-
GGUF
Model size
26B params
Architecture
inkling2_loop
Hardware compatibility
Log In to add your hardware

3-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for InterpreticAI/Inkling-2-Mini-IQ3_XXS-GGUF

Quantized
(38)
this model

Collection including InterpreticAI/Inkling-2-Mini-IQ3_XXS-GGUF