Instructions to use AXERA-TECH/Qwen3.5-9B-AX8860-llama.cpp with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use AXERA-TECH/Qwen3.5-9B-AX8860-llama.cpp with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf AXERA-TECH/Qwen3.5-9B-AX8860-llama.cpp:BF16
# Run inference directly in the terminal:
llama cli -hf AXERA-TECH/Qwen3.5-9B-AX8860-llama.cpp:BF16

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf AXERA-TECH/Qwen3.5-9B-AX8860-llama.cpp:BF16
# Run inference directly in the terminal:
llama cli -hf AXERA-TECH/Qwen3.5-9B-AX8860-llama.cpp:BF16

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf AXERA-TECH/Qwen3.5-9B-AX8860-llama.cpp:BF16
# Run inference directly in the terminal:
./build/bin/llama cli -hf AXERA-TECH/Qwen3.5-9B-AX8860-llama.cpp:BF16

Use Docker

docker model run hf.co/AXERA-TECH/Qwen3.5-9B-AX8860-llama.cpp:BF16

Llama new
LM Studio
Jan
Ollama
How to use AXERA-TECH/Qwen3.5-9B-AX8860-llama.cpp with Ollama:
```
ollama run hf.co/AXERA-TECH/Qwen3.5-9B-AX8860-llama.cpp:BF16
```
Unsloth Desktop

How to use AXERA-TECH/Qwen3.5-9B-AX8860-llama.cpp with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf AXERA-TECH/Qwen3.5-9B-AX8860-llama.cpp:BF16

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "AXERA-TECH/Qwen3.5-9B-AX8860-llama.cpp:BF16"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use AXERA-TECH/Qwen3.5-9B-AX8860-llama.cpp with Docker Model Runner:
```
docker model run hf.co/AXERA-TECH/Qwen3.5-9B-AX8860-llama.cpp:BF16
```

Lemonade

How to use AXERA-TECH/Qwen3.5-9B-AX8860-llama.cpp with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull AXERA-TECH/Qwen3.5-9B-AX8860-llama.cpp:BF16

Run and chat with the model

lemonade run user.Qwen3.5-9B-AX8860-llama.cpp-BF16

List all available models

lemonade list

Hermes Agent

How to use AXERA-TECH/Qwen3.5-9B-AX8860-llama.cpp with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf AXERA-TECH/Qwen3.5-9B-AX8860-llama.cpp:BF16

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default AXERA-TECH/Qwen3.5-9B-AX8860-llama.cpp:BF16

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use AXERA-TECH/Qwen3.5-9B-AX8860-llama.cpp with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf AXERA-TECH/Qwen3.5-9B-AX8860-llama.cpp:BF16

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "AXERA-TECH/Qwen3.5-9B-AX8860-llama.cpp:BF16" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3.5-9B on AX8860 (llama.cpp + ggml-axera)

在爱芯 AX8860 的 NPU 上用 llama.cpp 运行 Qwen3.5-9B：文本 + 图像输入，OpenAI 兼容服务。 Qwen3.5-9B on the Axera AX8860 NPU with llama.cpp: text + image input, OpenAI-compatible server.

性能 / Performance


decode	43.7 tokens/s
decode，MTP 自适应 / adaptive MTP	最高 / up to 76.6 tokens/s（1.75x）
4k prompt，MTP / MTP on 4k	41.2 → 68.1 tokens/s（1.65x，accepted 44/57）
4k prompt prefill	2959 tokens/s（3908 token 1.32 s）
上下文 / context	最长 / up to 262144

MTP 可选（MTP=1 ./run.sh -s 自适应，MTP=force 强制，默认关闭）/ optional MTP (MTP=1 ./run.sh -s adaptive, MTP=force forced, off by default):

内容 / content	普通 / plain	默认 / adaptive	强制 / forced	加速 / speed-up	接受率 / accept（启用时 / when active）
代码 / code	43.6	58.7	61.9	1.35x	0.595
中译英 / translation	43.0	67.9	64.1	1.58x	0.691
Markdown 表格 / table	43.7	76.6	77.1	1.75x	0.816
数学解题 / maths	43.5	66.3	66.0	1.52x	0.667
中文短文 / Chinese essay	43.7	43.4	47.3	0.99x	0.369
英文故事 / English story	43.4	42.3	47.3	0.97x	0.376
代码，推荐采样 / code, sampled	42.7	42.1	39.4	0.98x	0.444
中文短文，推荐采样 / essay, sampled	43.2	42.9	28.7	0.99x	0.267
Linux 命令列表 / list	43.1	54.2	55.1	1.26x	0.500
中文解释 / explanation	42.8	43.2	50.5	1.01x	0.433

单位 tokens/s；每项最多生成 256 token，未注明的为贪心解码 / tokens/s; up to 256 tokens each, greedy unless noted.

sh bench.sh 复现 / reproduce（生成 128 token / 128 generated tokens；短 prompt 仅列耗时 / short prompts report time only）：

prompt	`-c`	prompt 耗时 / time	decode tokens/s
5 token	8192	0.11 s	—
47 token	8192	0.11 s	—
3908 token	8192	1.32 s	41.9
5 token	262144	0.11 s	—
47 token	262144	0.11 s	—
3908 token	262144	1.31 s	41.5
3908 token，推荐采样 / sampled	8192	1.32 s	41.5

快速开始 / Quick start

sh run.sh "The capital of France is" 64           # 文本 / text
sh run.sh -i bus.jpg "Describe this image." 256   # 图像 / image
sh run.sh -c                                      # 命令行对话 / chat
sh run.sh -s 8080                                 # OpenAI 兼容服务 / server
MTP=1 sh run.sh -s 8080                           # 自适应 MTP / adaptive MTP
MTP=force sh run.sh -s 8080                       # 强制 MTP / forced MTP
CTX=131072 sh run.sh -s 8080                      # 长上下文 / long context (default 16384)
sh bench.sh                                       # 性能复现 / benchmark

需要 AX8860，SDK V0.5.1 或更新；HF 下载的文件没有执行权限，用 sh 调用即可 / needs an AX8860 with SDK V0.5.1+; call the scripts with sh.

文件 / Files

文件 / File	说明 / Description
`qwen3.5-9b-mxfp4.gguf`	模型权重 / weights，6.00 GB，md5 `57b4b4b8fe3e7d05af29019438e5369a`
`mmproj-qwen3.5-9b-bf16.gguf`	视觉编码器 / vision encoder，921 MB，md5 `69d1a645084d0f31d4296a82dce991cf`
`bin/`	llama.cpp 板端程序（静态链接）/ board programs (static)
`kernels/`	NPU kernel + `manifest.json`
`run.sh`，`chat_template.jinja`	运行脚本与服务端对话模板 / run script and the server's chat template
`bench.sh` 及 prompt	性能复现 / benchmark
`scenes/`	10 个 MTP 场景请求 / 10 MTP scene requests
`MTP.md`	可选 MTP 用法 / optional MTP usage

说明 / Notes

MTP 可选：MTP=1 ./run.sh -s 自适应开启，MTP=force ./run.sh -s 强制开启；MTP 开关不改变采样参数。贪心请求须显式带 presence_penalty: 0，否则服务端推荐的 presence penalty 会关闭 fused greedy，自适应 MTP 不会生效 / optional MTP: MTP=1 ./run.sh -s enables it adaptively and MTP=force forces it, without changing sampling. Greedy requests must set presence_penalty: 0; otherwise the server's recommended presence penalty disables fused greedy and adaptive MTP stays dormant.
服务端默认使用模型卡推荐的采样参数，文本补全与图像描述为贪心解码 / the server uses the model card's recommended sampling; completion and image description are greedy.
服务端默认关闭思考（--reasoning off），并允许对话中段的 system 消息（Codex 等 agent 客户端可用）/ the server does not think by default and accepts a mid-conversation system message (works with agent clients such as Codex).
Prompt cache 默认放在 4 GB 内存；设置 CACHE_DIR=<SSD 目录> 可落盘，重启后仍能命中 / the prompt cache lives in 4 GB of host memory; set CACHE_DIR=<an SSD directory> to keep it on disk across restarts.
运行时独占 NPU 4 核；服务端 -np 1 / takes all 4 NPU cores; the server runs -np 1.
llama-bench 可用，请加 -b 4096 -ub 4096 / llama-bench works; pass -b 4096 -ub 4096.
个别环境下连续请求偶发 ret = -3，重试即可 / in rare cases a back-to-back request fails with ret = -3; retry it.

正确性 / Correctness

PPL：2-chunk（ppl_zh.txt）-ub 1 20.8654、-ub 4 20.8061；20-chunk（ppl_zh_long.txt）-ub 1 23.7616、-ub 4 23.7930 / PPL: 20.8654 (-ub 1) and 20.8061 (-ub 4) over 2 chunks; 23.7616 and 23.7930 over 20 chunks.
"The capital of France is" → "Paris. …"；图像描述正确（车辆、文字、人物）/ text and image descriptions correct.
板端 bin 更新后输出与上一版一致；MTP 的草稿由目标模型逐 token 验证，贪心输出与普通解码一致 / output matches the previous release after the board-bin update; every MTP draft is verified by the target model and greedy output equals plain decoding.

源码 / Sources

llama.cpp：ZHEQIUSHUI/llama.cpp axera-dev @ 0104880be

更新 / Changelog

2026-11-09：9B 增加可选 MTP（256k kernels）；统一板端 bin；贪心场景显式 presence_penalty: 0；perplexity 固定走全 logits / optional 9B MTP with 256k kernels, unified board bins, explicit presence_penalty: 0 in greedy scenes, perplexity keeps the full-logits head.