Instructions to use ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF", device_map="auto")

llama-cpp-python

How to use ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF",
	filename="Qwen3.6-27B-A3B-Coder-CD-IQ2_XS_h.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M

Use Docker

docker model run hf.co/ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M

SGLang

How to use ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF with Ollama:
```
ollama run hf.co/ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M
```

Unsloth Studio

How to use ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF to start chatting

How to use ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF with Docker Model Runner:
```
docker model run hf.co/ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M
```

Lemonade

How to use ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3.6-27B-A3B-Coder-MTP-GGUF-Q4_K_M

List all available models

lemonade list

Qwen3.6-27B-A3B-Coder

A code-specialist expert prune of Qwen3.6-35B-A3B: the MoE is reduced from 256 experts to 184 (72 dropped per layer, ~35B→27B, still A3B active) using a code-targeted competence map (LiveCodeBench + MultiPL-E competence classes). Same router, attention, norms, MTP head and vision tower as the base — only the expert keep-set changes.

Served at top-10 (num_experts_per_tok = 10, baked as the default). This is a routing-recovery lever: after pruning to 184 experts, activating the top-10 (vs the base top-8) recovers instruction-following at no cost to code (see below). No fine-tuning, no distillation — pure expert selection + a routing-width dial.

Recipe

Competence map: the 256e teacher is profiled per-expert on a balanced corpus + targeted LiveCodeBench and MultiPL-E (Rust/Java/JS) PASS-response classes.
Drop map: wmax aggregation with the LCB + MPE classes up-weighted (1.5) → 72/256 experts dropped per layer, protecting the code-competent experts.
Top-10 routing (num_experts_per_tok = 10) baked into the config → the shipped default. Pass --override-kv qwen35moe.expert_used_count=int:8 to any llama.cpp tool to A/B back to native top-8.

Evaluation (Q6_K, llama.cpp, temp 0.6 / top-p 0.95 / top-k 20)

Benchmark	This model	Qwen3.6-35B-A3B (256e)	coder (LCB-only)
GPQA-Diamond	0.773	0.833	0.793
MATH-500	0.620	0.730	0.620
AIME	0.733	0.633	0.767
LiveCodeBench (v6, 77q)	0.688	0.714	0.688
IFEval	0.730	0.960	0.840
HumanEval	0.970	0.970	0.963
GSM8K	0.970	0.960	0.980
ARC-Challenge	0.944	0.935	0.933
MultiPL-E	0.840	0.827	0.670
Average	0.808	0.840	0.806

Highlights: best code profile of any prune — MultiPL-E 0.840 (above the teacher; +17pp over the LCB-only coder that this model supersedes), LiveCodeBench 0.688 (tied best), HumanEval 0.970. Average 0.808 sits at the LCB-coder level and within 0.03 of the full teacher.

Verbosity / rumination (length breakdown per eval)

Aggressive expert pruning makes the model verbose on open-ended reasoning — it over-thinks before answering. This is largely inherited from the base (the 256e teacher does the same on GPQA/AIME) and is bounded by the generation cap; it does not affect the code benches, which have a natural termination anchor.

Response length in characters (content + reasoning), this model vs the 256e teacher; runaway = responses > 20k chars (of 100, or 30/198 for GPQA, 30 for AIME):

Benchmark	p50	p90	max	runaway	256e runaway
GPQA	13.8k	58.8k	129k	58	58 (same)
AIME	52.9k	85.5k	96k	25	29
IFEval	12.1k	56.2k	81k	30	11
MATH-500	2.3k	14.5k	76k	9	12
GSM8K	2.5k	12.5k	108k	6	3
ARC	1.4k	2.3k	59k	7	0
HumanEval	0.8k	1.4k	24k	1	2
LiveCodeBench / MultiPL-E	— code path —			tight	tight

Reading it: GPQA/AIME verbosity is essentially the base model (58 vs 58, 25 vs 29). Only IFEval shows prune-added rumination (30 vs 11) — the trade for the code-targeted drop map. Code and math-with-boxing tasks terminate cleanly. If you want tighter output, a repetition/length penalty at serve time (or top-8 via the override above) reduces the tail.

Formats

GGUF (this repo family): full imatrix quant sweep (Q8_0 → IQ2, plus ContribDynamic CD-* per-layer quants) in Qwen3.6-27B-A3B-Coder-MTP-GGUF. Includes the native MTP head (speculative decoding) and a -vision mmproj for multimodal use. imatrix.dat archived in-repo.
Ollama: mannix/qwen3.6-27b-a3b-coder (text) and …-vision tags (with mmproj).

Quantization quality

Every K-quant (Q4_K_S → Q6_K and the _L variants) is built with imatrix. On this model the imatrix is load-bearing at 4-bit — the opposite of Gemma-4, where imatrix degrades K-quants. Measured on a deterministic greedy MultiPL-E code probe, the entire imatrix K-family and the ContribDynamic CD-* tiers sit at full-precision parity (within measurement noise of the F16 anchor). The only outlier was a plain, imatrix-free Q4_K_M, which fell ~12pp below parity — which is why imatrix is now the default for every tier in this repo.

Recommended tier: CD-IQ4_K_M (~16 GB) — full-precision-parity code quality at the smallest at-parity size.
Q4_0 / Q4_1 are not shipped — superseded by the imatrix K-quants (legacy round-to-nearest tiers offered no quality at their size).

⚠️ Hardware note — low i-quants (IQ2_M, IQ3_M) on Blackwell (sm_120) GPUs

The low i-quant tiers IQ2_M and IQ3_M produce incoherent output ("token salad") on NVIDIA Blackwell GPUs (sm_120, e.g. RTX PRO 6000) — under both stock llama.cpp and opencoti-llamafile, which share the same ggml CUDA IQ2_S/IQ3_S kernel. The weights are fine: the identical GGUF is fully coherent and produces correct code on CPU and on Ampere/Ada GPUs (verified on an RTX 3090). This is a llama.cpp/ggml CUDA-kernel issue on the sm_120 IQ2_S/IQ3_S path, not a defect in these files.

On a Blackwell GPU, instead use: the K-quants (Q2_K_L, Q3_K_S/Q3_K_M/Q3_K_L), or IQ2_XS / IQ4_XS / the CD-* tiers — all coherent on Blackwell in the same size band (Q2_K_L/IQ2_XS ≈ the IQ2_M band; Q3_K_M/Q3_K_L ≈ the IQ3_M band).
IQ2_M / IQ3_M are kept in the repo because they are correct on CPU and Ampere/Ada GPUs.

Notes

Top-10 is baked as the default; the model was selected and evaluated at top-10.
Same tokenizer, chat template, MTP head and vision tower as the base.
Research checkpoint. Verbosity on open-ended prompts is a known, base-inherited trait.

Downloads last month: 29,540

GGUF

Model size

26B params

Architecture

qwen35moe

Hardware compatibility

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

16-bit

Model tree for ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF

Base model

Qwen/Qwen3.6-35B-A3B

Quantized

(659)

this model

Collection including ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF

Qwen-3.6

Collection

7 items • Updated 7 days ago