Instructions to use abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF:IQ1_M # Run inference directly in the terminal: llama cli -hf abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF:IQ1_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF:IQ1_M # Run inference directly in the terminal: llama cli -hf abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF:IQ1_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF:IQ1_M # Run inference directly in the terminal: ./build/bin/llama cli -hf abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF:IQ1_M
Use Docker
docker model run hf.co/abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF:IQ1_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF:IQ1_M
- Ollama
How to use abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF with Ollama:
ollama run hf.co/abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF:IQ1_M
- Unsloth Desktop
- Pi
How to use abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF:IQ1_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF:IQ1_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF with Docker Model Runner:
docker model run hf.co/abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF:IQ1_M
- Lemonade
How to use abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF:IQ1_M
Run and chat with the model
lemonade run user.Inkling-2-Mini-GSQ-RCO-GGUF-IQ1_M
List all available models
lemonade list
- Hermes Agent
How to use abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF:IQ1_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF:IQ1_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF:IQ1_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF:IQ1_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Inkling-2-Mini GSQ-RCO GGUF
Mixed-precision GSQ-RCO GGUF quantizations of thinkingmachines/Inkling-2-Mini, engineered for lossless routing, multi-step reasoning, and native multimodal inference in llama.cpp.
- Architecture: Looped Sparse MoE (
26.1Btotal ·3.4Braw /5.4Beffective active across3×looped middle layers ·128expertstop-8· Vision + Text) - Context Window: Up to
1Mtokens (hybrid sliding-window and global attention) - Recommended Sampling:
temperature = 1.0·top_p = 1.0·top_k = 0·min_p = 0.0(--jinja)
GGUF Files
| File | bpw | Size | Target Hardware & Notes |
|---|---|---|---|
| Inkling-2-Mini-GSQ-RCO-IQ1_M.gguf | 2.25 | 7.32 GB | 8 GB VRAM · Ultra-compact |
| Inkling-2-Mini-GSQ-RCO-IQ2_XS.gguf | 2.75 | 8.95 GB | 10 GB VRAM / 16 GB Mac · Full Q8_0 attention |
| Inkling-2-Mini-GSQ-RCO-IQ3_XS.gguf (Recommended) | 3.25 | 10.58 GB | 12 GB VRAM · Optimal Pareto sweet spot |
| Inkling-2-Mini-GSQ-RCO-IQ4_XS.gguf | 4.25 | 13.84 GB | 16 GB VRAM / 24 GB Mac · Near-lossless |
| mmproj-Inkling-2-Mini-BF16.gguf (Vision) | 16.00 | 642 MB | Vision projector (--mmproj · Recommended, ~9× faster on GPU) |
| mmproj-Inkling-2-Mini-Q8_0.gguf (Vision) | 8.50 | 346 MB | Smaller vision projector (--mmproj) |
Quickstart
Long Context (128k in <= 12 GB VRAM on IQ2_XS)
llama-server \
-m Inkling-2-Mini-GSQ-RCO-IQ2_XS.gguf \
--jinja -ngl 99 -fa on -np 1 -c 131072 -b 2048 -ub 96 \
-ctk q4_0 -ctv q4_0 \
--temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 --port 8080
| GPU VRAM | Model | Context | Optimal Flags (-fa on -np 1 -b 2048) |
Measured VRAM |
|---|---|---|---|---|
| 12 GB | IQ2_XS |
128k |
-c 131072 -ub 96 -ctk q4_0 -ctv q4_0 (or -ub 64 for q8_0 KV 12.0 GiB / GPU Vision 11.7 GiB) |
11.7 GiB |
| 16 GB | IQ2_XS |
128k |
-c 131072 -ub 128 -ctk q8_0 -ctv q8_0 |
13.5 GiB |
| 24 GB | IQ3_XS |
128k |
-c 131072 -ub 256 -ctk q8_0 -ctv q8_0 |
18.7 GiB |
LM Studio / VRAM Tip: Physical batch size (
-ub/ Evaluation Batch Size) scales Inkling-2's compute buffer linearly with context length (-bdoes not). Set-ubto64–128with Flash Attention + matchedq8_0/q8_0orq4_0/q4_0KV cache (avoid mixedq8_0/q4_0) to maximize usable context until upstreamllama.cppPR#32740lands.
Vision (--mmproj, Optional · 128k in <= 12 GB VRAM)
llama-server \
-m Inkling-2-Mini-GSQ-RCO-IQ2_XS.gguf \
--mmproj mmproj-Inkling-2-Mini-BF16.gguf \
--jinja -ngl 99 -fa on -np 1 -c 131072 -b 2048 -ub 64 \
-ctk q4_0 -ctv q4_0 \
--temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 --port 8080
- Use
mmproj-Inkling-2-Mini-BF16.gguf(~9×faster GPU image encoding thanQ8_0,~0.8 GBVRAM). - Omit
--mmprojfor text-only use, or add--no-mmproj-offloadon tight VRAM to keep the vision projector on CPU (~0.1 GBVRAM).
Speculative Decoding (DSpark, Optional)
llama-server \
-m Inkling-2-Mini-GSQ-RCO-IQ2_XS.gguf \
-md dspark-Inkling-2-Mini-Q8_0.gguf \
--spec-type draft-dspark --spec-draft-n-max 3 -ngld 99 \
-ctkd q8_0 -ctvd q8_0 \
--jinja -ngl 99 -fa on -np 1 -c 131072 -b 2048 -ub 64 \
-ctk q4_0 -ctv q4_0 \
--temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 --port 8080
- Uses dspark-Inkling-2-Mini-Q8_0.gguf (
402 MB, matches BF16 acceptance) and currently requires allama.cppbuild with Inkling-2 DSpark support (upstream PR#32218; not yet supported in stock LM Studio). - Note: DSpark can accelerate structured English code and math, but often slows down general chat or non-English generation when acceptance drops (especially on lower-TDP GPUs). Keep it disabled unless your workload specifically benefits from it.
Ollama
ollama run hf.co/abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF:IQ3_XS
Benchmarks
Official Base Model Evaluations (thinkingmachines/Inkling-2-Mini)
| Inkling 2 Mini | Gemma 4 31B | Muse Glimmer | Qwen3.8-27B | Nemotron 3.5 Super VL | |
|---|---|---|---|---|---|
| Model Info | |||||
| Params (B) activated / total | 5/26 | 31/31 | 30/30 | 27/27 | 12/125 |
| Third Party Index | |||||
| AA Index (v5.0) | 20.1 | 4.4 | 6.3 | 20.1 | 13.6 |
| Vals Index (v2.2) | 46.5% | – | – | – | 29.8% |
| Coding and Autoresearch | |||||
| FrontierCode v2.0 stirrup v0.2.1 | 15.1% | 0.0% | 1.5% | 14.9% | 6.1% |
| DeepSWE v1.1 mini-swe-agent v2.4.6 | 48.2% | 0.3% | 4.4% | 42.2% | 24.7% |
| Terminal Bench v4.0 mini-swe-agent v2.4.6 | 5.6% | 0.0% | 0.5% | 5.6% | 0.0% |
| Terminal Bench v2.1 Inkling 2 Mini in pi v0.83.0 | 71.6% | 43.4% | 51.7% | 73.0% | 60.7% |
| SWEBench Pro v2 (Hard) mini-swe-agent v2.4.6 | 77.9% | 14.2% | 44.3% | 73.8% | 44.1% |
| SWEBench Multilingual Inkling 2 Mini with bash only | 73.7% | 43.0% | 66.9% | 70.0% | 67.7% |
| SWEBench Verified bash only | 79.2% | 46.1% | 76.0% | 77.8% | 71.1% |
| Vibe Code Bench v1.1 openhands (vals) | 77.9% | – | – | 64.9% | 3.5% |
| PMPP-Hard (AUC, 0–10h) pi v0.83.0 | 455.82 | – | – | 262.50 | – |
| Agentic Search, Tool Use, and Knowledge Work | |||||
| BrowseComp Inkling 2 Mini with compaction | 79.0% | 37.0% | 63.9% | 61.6% | 78.9% |
| DeepSearchQA Inkling 2 Mini with compaction | 85.3% | 61.7% | 74.6% | 79.7% | 77.7% |
| MCP Atlas Public | 82.7% | 54.2% | 75.5% | 77.3% | 67.7% |
| AA-Briefcase v1.1 | 1263 | 369 | 478 | 1391 | 999 |
| GDPVal-AA v2.1 | 1319 | 627 | 792 | 1427 | 1162 |
| JobBench | 49.6% | 8.2% | 16.1% | 33.4% | 36.5% |
| AutomationBench-AA v1.0.6 | 41.8% | 4.9% | 6.8% | 48.2% | 14.8% |
| EnterpriseOps-Gym | 42.0% | 28.4% | 34.7% | 44.2% | 28.1% |
| Vals Finance Index | 50.1% | – | – | – | 41.0% |
| AA-AnalystAgent | 46.3% | – | 20.0% | – | – |
| Diligence Bench Best harness | 48.6% | – | 40.9% | 46.4% | 45.0% |
| Vision Interpretation and Creation | |||||
| GDP.pdf | 16.8% | 6.0% | 11.8% | 16.4% | 8.0% |
| PerceptionBench | 57.5% | 40.7% | 42.8% | 52.4% | 36.8% |
| MMMU Pro Standard 10 | 74.9% | 73.4% | 74.3% | 76.3% | 72.3% |
| OmniDocBench v1.6 | 92.8% | 82.5% | 90.4% | 91.5% | 77.9% |
| ScreenSpot Pro original / with Python | 76.5/87.7% | 5.9%/– | 75.4%/– | 57.6/78.9% | 43.5%/– |
| Chartography original / with Python | 22.1/72.5% | 12.0/11.8% | 17.4/25.1% | 12.2/52.1% | 9.7/29.3% |
| ZeroBench (main) original / with Python | 17.0/34.0% | 9.0/12.0% | 9.0/17.0% | 12.0%/– | 11.0/12.0% |
| MuirBench | 77.5% | 72.3% | 77.6% | 65.9% | 72.7% |
| BVB | 68.4% | 47.0% | 50.5% | 70.7% | 52.1% |
| Reasoning, Knowledge, and Epistemics | |||||
| HLE-Diamond | 15.2% | 7.2% | 9.1% | 15.3% | 8.8% |
| CritPt v2 Full / Hard | 31.4/17.0% | –/4.0% | –/7.0% | 10.6/4.0% | 36.9/13.5% |
| Apex 2025 | 34.4% | 5.2% | 4.2% | 18.8% | 22.9% |
| ARC-AGI v1 / v2 | 84.5/49.3% | – | – | 87.5/42.4% | 22.7/1.5% |
| LongBench v2 | 66.9% | 52.9% | 33.1% | 52.3% | 62.1% |
| GraphWalks BFS 256K | 69.7% | – | – | 50.4% | 63.3% |
| MRCR v2 8-needle 256K–512K | 46.4% | – | – | 38.0% | 5.7% |
| AA-Omniscience v1.1 score | -8.6 | -48.5 | -32.2 | -6.4 | -2.9 |
| AA-Omniscience v1.1 non-hallucination rate | 63.4% | 14.5% | 16.4% | 71.4% | 69.5% |
| ComplexConstraints v1.1 openhands v1.45.0 | 33.4% | 31.6% | 35.6% | 30.1% | 30.1% |
| Safety and Alignment | |||||
| FORTRESS (adversarial) Public | 89.5% | 67.8% | 90.3% | 78.8% | 79.6% |
| FORTRESS (benign) Public | 96.4% | 97.6% | 89.1% | 97.8% | 92.7% |
| StrongREJECT (none) | 97.8% | 98.1% | 99.0% | 99.0% | 99.4% |
| Misbehavior (Internal) ↓ | 9.8% | 27.6% | 46.4% | 57.7% | 52.8% |
| Covert misbehavior (Internal) ↓ | 8.3% | 24.3% | 45.6% | 33.2% | 43.4% |
| Destructive actions (Internal) ↓ | 10.2% | 75.8% | 68.8% | 35.2% | 65.6% |
Best reported score in each row (lowest where lower is better); equal displayed scores tie Second-best reported score in each row; none when first place is tied
Technical Details
Why GSQ-RCO? (Architecture & Precision Map)
Inkling-2-Mini is a 24-layer Sparse MoE (128 experts, top-8) whose middle 12 layers (blk.6–17) run three times per token (48 effective layer passes). Uniform quantization amplifies error through the recurrent loop and degrades routing stability at low bit-widths.
- RCO (Looped Backbone Shield): Pins all 24 MoE router gates (
ffn_gate_inp), 24attn_sink_proj, 8 globalattn_k/v_conv_proj, and all loop controllers/norms inF32/BF16; preserves attention projections inQ8_0(Q6_Klocal Q/K/G/R onIQ1_M) and embeddings/LM head inQ4_K–Q8_0. - GSQ (Per-Expert Hessian Block Refinement): Calibrates all
3,072sparse experts (24 × 128,100%coverage,3.000×looped-core accumulation) over a2.05M-token multi-domain corpus with per-expert Hessian-weighted block refinement (GSQ-Q2_0,GSQ-MXFP4,IQ4_NL) and exact-budget mixed-precision allocation.
| Tensor Group | Precision | Purpose |
|---|---|---|
Routers (ffn_gate_inp), attn_sink_proj, attn_k/v_conv_proj |
F32 |
Prevents top-8 routing drift and loop divergence |
Loop Controllers (loop_*), ShortConv & Norms |
F32 / BF16 |
Preserves 3-pass recurrent gating and state stability |
Attention Projections (attn_q/k/v/output/gate/rel) |
Q8_0 (Q6_K local on IQ1_M) |
Protects global (blk.5,11,17,23) and sliding-window retrieval |
Token Embeddings (token_embd) & LM Head (output) |
Q4_K/Q5_K – Q8_0 |
Preserves 201k-vocabulary representation and output logits |
ffn_gate_up_exps |
IQ1_S – IQ4_XS |
Exact-budget sensitivity-guided mixed-precision allocation |
ffn_down_exps |
GSQ-Q2_0, GSQ-MXFP4, IQ4_NL |
Per-expert Hessian-weighted block refinement |
Credits & License
- Methodology Credit: Foundational GSQ (arXiv:2604.18556) and RCO (arXiv:2605.00649) quantization frameworks are credited to ISTA-DASLab (Deep Algorithms and Systems Lab, Institute of Science and Technology Austria).
- Base Model: thinkingmachines/Inkling-2-Mini, licensed under Apache 2.0.
- Downloads last month
- -
1-bit
2-bit
3-bit
4-bit
8-bit
16-bit
Model tree for abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF
Base model
thinkingmachines/Inkling-2-Mini