--- license: apache-2.0 license_link: https://www.apache.org/licenses/LICENSE-2.0 library_name: gguf pipeline_tag: image-text-to-text base_model: - thinkingmachines/Inkling-2-Mini tags: - inkling - moe - gguf --- # Inkling-2-Mini-GGUF [Inkling 2 Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini) is a multimodal looped mixture-of-experts model from Thinking Machines Lab: 26.4B parameters, 3.4B active (5.4B effective with three passes through the looped middle layers), text and image input, up to 1M tokens of context. This repository is an unmodified quantization of the official checkpoint (revision `1fb4134`). GGUF Q4_K_M for llama.cpp and LM Studio, with the vision projector. Fits a single 24 GB GPU. Other formats from INTERPRETIC AI: [GGUF Q4_K_M](https://huggingface.co/InterpreticAI/Inkling-2-Mini-GGUF) · [GGUF IQ3_XXS](https://huggingface.co/InterpreticAI/Inkling-2-Mini-IQ3_XXS-GGUF) · [MLX 4-bit](https://huggingface.co/InterpreticAI/Inkling-2-Mini-MLX-4bit) ## Files | File | Description | Size | |---|---|---| | `Inkling-2-Mini-Q4_K_M.gguf` | Language model, Q4_K_M | 17.5 GB | | `mmproj-Inkling-2-Mini-F16.gguf` | Vision projector, F16 | 0.64 GB | SHA-256 checksums are listed in `SHA256SUMS`. ## Usage Requires llama.cpp with Inkling 2 Mini support (commit `29b52d3` or later). ```bash llama-server -m Inkling-2-Mini-Q4_K_M.gguf \ --mmproj mmproj-Inkling-2-Mini-F16.gguf \ -ngl 999 -c 32768 --jinja ``` The `--mmproj` file is only needed for image input. Measured on one RTX PRO 6000: about 65 tokens/s single-stream generation. ## Notes - Inkling 2 Mini is a thinking model: it writes its reasoning before the answer. Use a chat client or server that applies the model's chat template. - Quality checks here are spot checks (English, Chinese, code and one image), not a full benchmark. - Support for this architecture is new in all runtimes. Use a recent build. ## License Apache 2.0, as the original model. See the [model card of Thinking Machines Lab](https://huggingface.co/thinkingmachines/Inkling-2-Mini) and its [acceptable use policy](https://thinkingmachines.ai/model-acceptable-use-policy/). INTERPRETIC AI · [interpretic.ai](https://interpretic.ai)