Instructions to use joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF:IQ4_XS
# Run inference directly in the terminal:
llama cli -hf joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF:IQ4_XS

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF:IQ4_XS
# Run inference directly in the terminal:
llama cli -hf joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF:IQ4_XS

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF:IQ4_XS
# Run inference directly in the terminal:
./llama-cli -hf joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF:IQ4_XS

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF:IQ4_XS
# Run inference directly in the terminal:
./build/bin/llama-cli -hf joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF:IQ4_XS

Use Docker

docker model run hf.co/joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF:IQ4_XS

LM Studio
Jan

vLLM

How to use joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF:IQ4_XS

SGLang

How to use joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Ollama
How to use joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF with Ollama:
```
ollama run hf.co/joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF:IQ4_XS
```
Unsloth Desktop

How to use joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF:IQ4_XS

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF:IQ4_XS"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF with Docker Model Runner:
```
docker model run hf.co/joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF:IQ4_XS
```

Lemonade

How to use joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF:IQ4_XS

Run and chat with the model

lemonade run user.Qwopus3.5-9B-Coder-MTP-GGUF-IQ4_XS

List all available models

lemonade list

Hermes Agent

How to use joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF:IQ4_XS

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF:IQ4_XS

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF:IQ4_XS

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "joeblowma/Qwopus3.5-9B-Coder-MTP-GGUF:IQ4_XS" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

🌟 Qwopus3.5-9B-Coder-MTP-gguf

This is a set of custom IQ4_XS quants of Qwopus3.5-9B-Coder using mradermacher's imatrix, also mirroring mradermacher's mmproj files.

Initially I ran this to retain the MTP heads. Then went further, experiment with uplifting certain layers and the quant sizing of the MTP layers. Very basic stuff, but the result is pretty solid: 98,304 context (don't forget to use yarn past 32K!) in ~6GB VRAM when using beellama/KVarN and initial ~60t/s on a RTX4060 with Windows 11.

Version	file	size
1	Qwopus3.5-9B-Coder-MTP-IQ4_XS	~5.29GiB
2	Qwopus3.5-9B-Coder-MTP-IQ4_XS_v2	~5.02GiB
3	Qwopus3.5-9B-Coder-MTP-IQ4_XS_v3)	~4.99GiB

Tools

GGUF was produced using scripts from the root of llama.cpp's source code.

Quants were produced using llama-quantize.exe from llama.cpp's binary releases.

Get a gguf

python convert_hf_to_gguf.py --outfile T:\_Qwen\Qwopus3.5-9B-Coder\__output\Qwopus3.5-9B-Coder-MTP-{ftype}.gguf --outtype auto --model-name Qwopus3.5-9B-Coder --verbose T:\_Qwen\Qwopus3.5-9B-Coder

V1

This version is standard iq4_xs (with imatrix, thanks mradermacher!) with the MTP layers held at BF16

llama-quantize --imatrix Qwopus3.5-9B-Coder.mradermacher.imatrix.gguf --tensor-type "blk\.32\..*=BF16" T:\_Qwen\Qwopus3.5-9B-coder\__output\Qwopus3.5-9B-Coder-MTP-bf16.gguf T:\_Qwen\Qwopus3.5-9B-coder\__output\Qwopus3.5-9B-Coder-MTP-IQ4_XS.gguf IQ4_XS 8

V2

Try keeping a few ffn_down at Q5_K, MTP mostly Q5_K

llama-quantize --imatrix Qwopus3.5-9B-Coder.mradermacher.imatrix.gguf --tensor-type-file tensors1.txt T:\_Qwen\Qwopus3.5-9B-coder\__output\Qwopus3.5-9B-Coder-MTP-bf16.gguf T:\_Qwen\Qwopus3.5-9B-coder\__output\Qwopus3.5-9B-Coder-MTP-IQ4_XS_v2.gguf IQ4_XS 8

tensor1.txt:

blk.0.ffn_down.weight=Q5_K
blk.1.ffn_down.weight=Q5_K
blk.2.ffn_down.weight=Q5_K
blk.3.ffn_down.weight=Q5_K
blk.32.attn_k.weight=Q5_K
blk.32.attn_output.weight=Q5_K
blk.32.attn_q.weight=Q5_K
blk.32.attn_v.weight=Q6_K
blk.32.ffn_down.weight=Q5_K
blk.32.ffn_gate.weight=Q5_K
blk.32.ffn_up.weight=Q5_K
blk.32.nextn.eh_proj.weight=Q5_K

V3

Can we go smaller? Yes, but it doesn't gain much.

llama-quantize --imatrix Qwopus3.5-9B-Coder.mradermacher.imatrix.gguf --tensor-type-file tensors2.txt T:\_Qwen\Qwopus3.5-9B-coder\__output\Qwopus3.5-9B-Coder-MTP-bf16.gguf T:\_Qwen\Qwopus3.5-9B-coder\__output\Qwopus3.5-9B-Coder-MTP-IQ4_XS_v3.gguf IQ4_XS 8

tensor2.txt:

blk.0.ffn_down.weight=Q5_K
blk.1.ffn_down.weight=Q5_K
blk.2.ffn_down.weight=Q5_K
blk.3.ffn_down.weight=Q5_K
blk.32.attn_k.weight=IQ4_XS
blk.32.attn_output.weight=Q5_K
blk.32.attn_q.weight=IQ4_XS
blk.32.attn_v.weight=Q5_K
blk.32.ffn_down.weight=IQ4_XS
blk.32.ffn_gate.weight=IQ4_XS
blk.32.ffn_up.weight=IQ4_XS
blk.32.nextn.eh_proj.weight=IQ4_XS

inference

llama.cpp models.ini (beellama v0.4.3 - newer versions allow KVarN for MTP as well, I've not tried yet)

[*]
load-mode=none
mmproj-offload=on
ctx-size=98304
batch-size=2048
ubatch-size=512
cache-type-k=kvarn4
cache-type-v=kvarn2
cache-reuse=256
flash-attn=on
kv-offload=on
n-gpu-layers=all
fit=on
fitt=200
rope-scale=4
kv-tail-tokens=1024
spec-draft-n-max=2
ctx-checkpoints=32
cache-ram=4096
kv-unified=on

temperature=0.7
repeat-penalty=1.0
top-k=20
top-p=0.95
min-p=0.0001
reasoning=on

rope-scaling=yarn
yarn-orig-ctx=32768

[Qwopus3.5-9B-v3.5-MTP.coder_V1]
model=C:\llm\_Qwen\Qwopus3.5-9B-v3.5-MTP.coder_V1\Qwopus3.5-9B-Coder-MTP-IQ4_XS.gguf
jinja=on
chat-template-file=C:\llm\_Templates\Qwopus-9B-jackrong\chat_template.jinja
chat-template-kwargs={"preserve_thinking":"true"}
spec-type=draft-mtp
spec-draft-type-k=q8_0
spec-draft-type-v=q4_0

[Qwopus3.5-9B-v3.5-MTP.coder_V2]
model=C:\llm\_Qwen\Qwopus3.5-9B-v3.5-MTP.coder_V2\Qwopus3.5-9B-Coder-MTP-IQ4_XS_v2.gguf
jinja=on
chat-template-file=C:\llm\_Templates\Qwopus-9B-jackrong\chat_template.jinja
chat-template-kwargs={"preserve_thinking":"true"}
spec-draft-type-k=q8_0
spec-draft-type-v=q4_0

[Qwopus3.5-9B-v3.5-MTP.coder_V3]
model=C:\llm\_Qwen\Qwopus3.5-9B-v3.5-MTP.coder_V3\Qwopus3.5-9B-Coder-MTP-IQ4_XS_v3.gguf
jinja=on
chat-template-file=C:\llm\_Templates\Qwopus-9B-jackrong\chat_template.jinja
chat-template-kwargs={"preserve_thinking":"true"}
spec-type=draft-mtp
spec-draft-type-k=q8_0
spec-draft-type-v=q4_0

[Qwopus3.5-9B-v3.5.coder_MM_V3]
model=C:\llm\_Qwen\Qwopus3.5-9B-v3.5-MTP.coder_V3\Qwopus3.5-9B-Coder-MTP-IQ4_XS_v3.gguf
mmproj=C:\llm\_Qwen\Qwopus3.5-9B-Coder-mmproj-f16.gguf
jinja=on
chat-template-file=C:\llm\_Templates\Qwopus-9B-jackrong\chat_template.jinja
chat-template-kwargs={"preserve_thinking":"true"}
ctx-size=98304
rope-scaling=yarn
yarn-orig-ctx=32768
rope-scale=3

Conclusion

Prediction acceptance only dropped marginally on V3 compared to V1, still trends above 60% during coding tasks. Unable to quantify any gains from bumping 4 ff_down weights from IQ4_S to Q5_K. Next run I will try V2 again but bump all ff_down weights to Q6_K and MTP head to Q6_K/Q8_K and see where it stands.