Inkling-2-Mini GSQ-RCO GGUF

Model Requests GitHub

Mixed-precision GSQ-RCO GGUF quantizations of thinkingmachines/Inkling-2-Mini, engineered for lossless routing, multi-step reasoning, and native multimodal inference in llama.cpp.

  • Architecture: Looped Sparse MoE (26.1B total · 3.4B raw / 5.4B effective active across 3× looped middle layers · 128 experts top-8 · Vision + Text)
  • Context Window: Up to 1M tokens (hybrid sliding-window and global attention)
  • Recommended Sampling: temperature = 1.0 · top_p = 1.0 · top_k = 0 · min_p = 0.0 (--jinja)

GGUF Files

File bpw Size Target Hardware & Notes
Inkling-2-Mini-GSQ-RCO-IQ1_M.gguf 2.25 7.32 GB 8 GB VRAM · Ultra-compact
Inkling-2-Mini-GSQ-RCO-IQ2_XS.gguf 2.75 8.95 GB 10 GB VRAM / 16 GB Mac · Full Q8_0 attention
Inkling-2-Mini-GSQ-RCO-IQ3_XS.gguf (Recommended) 3.25 10.58 GB 12 GB VRAM · Optimal Pareto sweet spot
Inkling-2-Mini-GSQ-RCO-IQ4_XS.gguf 4.25 13.84 GB 16 GB VRAM / 24 GB Mac · Near-lossless
mmproj-Inkling-2-Mini-BF16.gguf (Vision) 16.00 642 MB Vision projector (--mmproj · Recommended, ~9× faster on GPU)
mmproj-Inkling-2-Mini-Q8_0.gguf (Vision) 8.50 346 MB Smaller vision projector (--mmproj)

Quickstart

Long Context (128k in <= 12 GB VRAM on IQ2_XS)

llama-server \
  -m Inkling-2-Mini-GSQ-RCO-IQ2_XS.gguf \
  --jinja -ngl 99 -fa on -np 1 -c 131072 -b 2048 -ub 96 \
  -ctk q4_0 -ctv q4_0 \
  --temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 --port 8080
GPU VRAM Model Context Optimal Flags (-fa on -np 1 -b 2048) Measured VRAM
12 GB IQ2_XS 128k -c 131072 -ub 96 -ctk q4_0 -ctv q4_0 (or -ub 64 for q8_0 KV 12.0 GiB / GPU Vision 11.7 GiB) 11.7 GiB
16 GB IQ2_XS 128k -c 131072 -ub 128 -ctk q8_0 -ctv q8_0 13.5 GiB
24 GB IQ3_XS 128k -c 131072 -ub 256 -ctk q8_0 -ctv q8_0 18.7 GiB

LM Studio / VRAM Tip: Physical batch size (-ub / Evaluation Batch Size) scales Inkling-2's compute buffer linearly with context length (-b does not). Set -ub to 64–128 with Flash Attention + matched q8_0/q8_0 or q4_0/q4_0 KV cache (avoid mixed q8_0/q4_0) to maximize usable context until upstream llama.cpp PR #32740 lands.

Vision (--mmproj, Optional · 128k in <= 12 GB VRAM)

llama-server \
  -m Inkling-2-Mini-GSQ-RCO-IQ2_XS.gguf \
  --mmproj mmproj-Inkling-2-Mini-BF16.gguf \
  --jinja -ngl 99 -fa on -np 1 -c 131072 -b 2048 -ub 64 \
  -ctk q4_0 -ctv q4_0 \
  --temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 --port 8080
  • Use mmproj-Inkling-2-Mini-BF16.gguf (~9× faster GPU image encoding than Q8_0, ~0.8 GB VRAM).
  • Omit --mmproj for text-only use, or add --no-mmproj-offload on tight VRAM to keep the vision projector on CPU (~0.1 GB VRAM).

Speculative Decoding (DSpark, Optional)

llama-server \
  -m Inkling-2-Mini-GSQ-RCO-IQ2_XS.gguf \
  -md dspark-Inkling-2-Mini-Q8_0.gguf \
  --spec-type draft-dspark --spec-draft-n-max 3 -ngld 99 \
  -ctkd q8_0 -ctvd q8_0 \
  --jinja -ngl 99 -fa on -np 1 -c 131072 -b 2048 -ub 64 \
  -ctk q4_0 -ctv q4_0 \
  --temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 --port 8080
  • Uses dspark-Inkling-2-Mini-Q8_0.gguf (402 MB, matches BF16 acceptance) and currently requires a llama.cpp build with Inkling-2 DSpark support (upstream PR #32218; not yet supported in stock LM Studio).
  • Note: DSpark can accelerate structured English code and math, but often slows down general chat or non-English generation when acceptance drops (especially on lower-TDP GPUs). Keep it disabled unless your workload specifically benefits from it.

Ollama

ollama run hf.co/abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF:IQ3_XS

Benchmarks

Official Base Model Evaluations (thinkingmachines/Inkling-2-Mini)
Inkling 2 MiniGemma 4 31BMuse GlimmerQwen3.8-27BNemotron 3.5 Super VL
Model Info
Params (B)
activated / total
5/2631/3130/3027/2712/125
Third Party Index
AA Index (v5.0)20.14.46.320.113.6
Vals Index (v2.2)46.5%–––29.8%
Coding and Autoresearch
FrontierCode v2.0
stirrup v0.2.1
15.1%0.0%1.5%14.9%6.1%
DeepSWE v1.1
mini-swe-agent v2.4.6
48.2%0.3%4.4%42.2%24.7%
Terminal Bench v4.0
mini-swe-agent v2.4.6
5.6%0.0%0.5%5.6%0.0%
Terminal Bench v2.1
Inkling 2 Mini in pi v0.83.0
71.6%43.4%51.7%73.0%60.7%
SWEBench Pro v2 (Hard)
mini-swe-agent v2.4.6
77.9%14.2%44.3%73.8%44.1%
SWEBench Multilingual
Inkling 2 Mini with bash only
73.7%43.0%66.9%70.0%67.7%
SWEBench Verified
bash only
79.2%46.1%76.0%77.8%71.1%
Vibe Code Bench v1.1
openhands (vals)
77.9%––64.9%3.5%
PMPP-Hard (AUC, 0–10h)
pi v0.83.0
455.82––262.50–
Agentic Search, Tool Use, and Knowledge Work
BrowseComp
Inkling 2 Mini with compaction
79.0%37.0%63.9%61.6%78.9%
DeepSearchQA
Inkling 2 Mini with compaction
85.3%61.7%74.6%79.7%77.7%
MCP Atlas
Public
82.7%54.2%75.5%77.3%67.7%
AA-Briefcase v1.112633694781391999
GDPVal-AA v2.1131962779214271162
JobBench49.6%8.2%16.1%33.4%36.5%
AutomationBench-AA v1.0.641.8%4.9%6.8%48.2%14.8%
EnterpriseOps-Gym42.0%28.4%34.7%44.2%28.1%
Vals Finance Index50.1%–––41.0%
AA-AnalystAgent46.3%–20.0%––
Diligence Bench
Best harness
48.6%–40.9%46.4%45.0%
Vision Interpretation and Creation
GDP.pdf16.8%6.0%11.8%16.4%8.0%
PerceptionBench57.5%40.7%42.8%52.4%36.8%
MMMU Pro
Standard 10
74.9%73.4%74.3%76.3%72.3%
OmniDocBench v1.692.8%82.5%90.4%91.5%77.9%
ScreenSpot Pro
original / with Python
76.5/87.7%5.9%/–75.4%/–57.6/78.9%43.5%/–
Chartography
original / with Python
22.1/72.5%12.0/11.8%17.4/25.1%12.2/52.1%9.7/29.3%
ZeroBench (main)
original / with Python
17.0/34.0%9.0/12.0%9.0/17.0%12.0%/–11.0/12.0%
MuirBench77.5%72.3%77.6%65.9%72.7%
BVB68.4%47.0%50.5%70.7%52.1%
Reasoning, Knowledge, and Epistemics
HLE-Diamond15.2%7.2%9.1%15.3%8.8%
CritPt v2
Full / Hard
31.4/17.0%–/4.0%–/7.0%10.6/4.0%36.9/13.5%
Apex 202534.4%5.2%4.2%18.8%22.9%
ARC-AGI
v1 / v2
84.5/49.3%––87.5/42.4%22.7/1.5%
LongBench v266.9%52.9%33.1%52.3%62.1%
GraphWalks BFS
256K
69.7%––50.4%63.3%
MRCR v2
8-needle 256K–512K
46.4%––38.0%5.7%
AA-Omniscience v1.1
score
-8.6-48.5-32.2-6.4-2.9
AA-Omniscience v1.1
non-hallucination rate
63.4%14.5%16.4%71.4%69.5%
ComplexConstraints v1.1
openhands v1.45.0
33.4%31.6%35.6%30.1%30.1%
Safety and Alignment
FORTRESS (adversarial)
Public
89.5%67.8%90.3%78.8%79.6%
FORTRESS (benign)
Public
96.4%97.6%89.1%97.8%92.7%
StrongREJECT (none)97.8%98.1%99.0%99.0%99.4%
Misbehavior (Internal) ↓9.8%27.6%46.4%57.7%52.8%
Covert misbehavior (Internal) ↓8.3%24.3%45.6%33.2%43.4%
Destructive actions (Internal) ↓10.2%75.8%68.8%35.2%65.6%

Best reported score in each row (lowest where lower is better); equal displayed scores tie Second-best reported score in each row; none when first place is tied

Technical Details

Why GSQ-RCO? (Architecture & Precision Map)

Inkling-2-Mini is a 24-layer Sparse MoE (128 experts, top-8) whose middle 12 layers (blk.6–17) run three times per token (48 effective layer passes). Uniform quantization amplifies error through the recurrent loop and degrades routing stability at low bit-widths.

  • RCO (Looped Backbone Shield): Pins all 24 MoE router gates (ffn_gate_inp), 24 attn_sink_proj, 8 global attn_k/v_conv_proj, and all loop controllers/norms in F32/BF16; preserves attention projections in Q8_0 (Q6_K local Q/K/G/R on IQ1_M) and embeddings/LM head in Q4_K–Q8_0.
  • GSQ (Per-Expert Hessian Block Refinement): Calibrates all 3,072 sparse experts (24 × 128, 100% coverage, 3.000× looped-core accumulation) over a 2.05M-token multi-domain corpus with per-expert Hessian-weighted block refinement (GSQ-Q2_0, GSQ-MXFP4, IQ4_NL) and exact-budget mixed-precision allocation.
Tensor Group Precision Purpose
Routers (ffn_gate_inp), attn_sink_proj, attn_k/v_conv_proj F32 Prevents top-8 routing drift and loop divergence
Loop Controllers (loop_*), ShortConv & Norms F32 / BF16 Preserves 3-pass recurrent gating and state stability
Attention Projections (attn_q/k/v/output/gate/rel) Q8_0 (Q6_K local on IQ1_M) Protects global (blk.5,11,17,23) and sliding-window retrieval
Token Embeddings (token_embd) & LM Head (output) Q4_K/Q5_K – Q8_0 Preserves 201k-vocabulary representation and output logits
ffn_gate_up_exps IQ1_S – IQ4_XS Exact-budget sensitivity-guided mixed-precision allocation
ffn_down_exps GSQ-Q2_0, GSQ-MXFP4, IQ4_NL Per-expert Hessian-weighted block refinement

Credits & License

Downloads last month
-
GGUF
Model size
26B params
Architecture
inkling2_loop
Hardware compatibility
Log In to add your hardware

1-bit

2-bit

3-bit

4-bit

8-bit

16-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF

Quantized
(180)
this model

Collection including abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF

Papers for abenzerps/Inkling-2-Mini-GSQ-RCO-GGUF