Instructions to use AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16 # Run inference directly in the terminal: llama cli -hf AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16 # Run inference directly in the terminal: llama cli -hf AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16 # Run inference directly in the terminal: ./build/bin/llama cli -hf AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16
Use Docker
docker model run hf.co/AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16
- Llama new
- LM Studio
- Jan
- Ollama
How to use AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp with Ollama:
ollama run hf.co/AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16
- Unsloth Desktop
- Pi
How to use AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp with Docker Model Runner:
docker model run hf.co/AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16
- Lemonade
How to use AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16
Run and chat with the model
lemonade run user.Qwen3.5-2B-AX8860-llama.cpp-BF16
List all available models
lemonade list
- Hermes Agent
How to use AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16# Run inference directly in the terminal:
llama cli -hf AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16Build from source code
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16# Run inference directly in the terminal:
./build/bin/llama cli -hf AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16Use Docker
docker model run hf.co/AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16Qwen3.5-2B on AX8860 (llama.cpp + ggml-axera)
ๅจ็ฑ่ฏ AX8860 ็ NPU ไธ็จ llama.cpp ่ฟ่ก Qwen3.5-2B๏ผๆๆฌ + ๅพๅ่พๅ ฅ๏ผๅฏ้ MTP ๆๆบ่งฃ็ ๏ผOpenAI ๅ ผๅฎนๆๅกใ Qwen3.5-2B on the Axera AX8860 NPU with llama.cpp: text + image input, optional MTP speculative decoding, OpenAI-compatible server.
ๆง่ฝ / Performance
| decode | 76.2 tokens/s |
| decode๏ผMTP ่ช้ๅบ / adaptive MTP | ๆ้ซ / up to 128.5 tokens/s๏ผ1.68x๏ผ |
| 4k prompt prefill | 6774 tokens/s๏ผ3908 token 0.58 s๏ผ |
| ไธไธๆ / context | ๆ้ฟ / up to 16384 |
MTP ๅฏ้๏ผMTP=1 sh run.sh -s ่ช้ๅบ๏ผMTP=force ๅผบๅถ๏ผ้ป่ฎคๅ
ณ้ญ๏ผ/ optional MTP (MTP=1 sh run.sh -s adaptive, MTP=force forced, off by default):
| ๅ ๅฎน / content | ๆฎ้ / plain | ้ป่ฎค / adaptive | ๅ ้ / speed-up | ๆฅๅ็ / accept |
|---|---|---|---|---|
| Linux ๅฝไปคๅ่กจ / list | 76.7 | 128.5 | 1.68x | 0.633 |
| ๆฐๅญฆ่งฃ้ข / maths | 77.3 | 124.0 | 1.60x | 0.605 |
| Markdown ่กจๆ ผ / table | 76.7 | 117.7 | 1.54x | 0.589 |
| ไธญ่ฏ่ฑ / translation | 75.8 | 108.0 | 1.42x | 0.500 |
| ไปฃ็ / code | 76.3 | 108.0 | 1.42x | 0.529 |
| ไธญๆ็ญๆ / Chinese essay | 76.7 | 77.3 | 1.01x | โ |
| ่ฑๆๆ ไบ / English story | 76.5 | 77.0 | 1.01x | โ |
| ไธญๆ่งฃ้ / explanation | 76.9 | 77.5 | 1.01x | โ |
| ไปฃ็ ๏ผๆจ่้ๆ ท / code, sampled | 76.3 | 76.6 | 1.00x | โ |
| ไธญๆ็ญๆ๏ผๆจ่้ๆ ท / essay, sampled | 75.7 | 76.2 | 1.01x | โ |
ๅไฝ tokens/s๏ผsh bench.sh mtp๏ผctx 16384๏ผๆฏ้กนๆๅค็ๆ 256 token๏ผๆชๆณจๆ็ไธบ่ดชๅฟ่งฃ็ ๏ผๆฃๆไธ้ๆ ท่ฏทๆฑ่ช้ๅบไฟๆๆฎ้่งฃ็ / tokens/s from sh bench.sh mtp, ctx 16384, up to 256 tokens each, greedy unless noted; prose and sampled requests stay on plain decoding.
sh bench.sh ๅค็ฐ / reproduce๏ผ็ๆ 128 token / 128 generated tokens๏ผ็ญ prompt ไป
ๅ่ๆถ / short prompts report time only๏ผ๏ผ
| prompt | -c |
prompt ่ๆถ / time | decode tokens/s |
|---|---|---|---|
| 5 token | 8192 | 0.05 s | โ |
| 47 token | 8192 | 0.08 s | โ |
| 3908 token | 8192 | 0.58 s | 75.0 |
| 5 token | 16384 | 0.06 s | โ |
| 47 token | 16384 | 0.07 s | โ |
| 3908 token | 16384 | 0.60 s | 73.8 |
| 3908 token๏ผๆจ่้ๆ ท / sampled | 8192 | 0.59 s | 72.2 |
ๅฟซ้ๅผๅง / Quick start
sh run.sh "The capital of France is" 64 # ๆๆฌ / text
sh run.sh -i bus.jpg "Describe this image." 256 # ๅพๅ / image
sh run.sh -c # ๅฝไปค่กๅฏน่ฏ / chat
sh run.sh -s 8080 # OpenAI ๅ
ผๅฎนๆๅก / server
MTP=1 sh run.sh -s 8080 # ่ช้ๅบ MTP / adaptive MTP
sh bench.sh # ๆง่ฝๅค็ฐ / benchmark
้่ฆ AX8860๏ผSDK V0.5.1 ๆๆดๆฐ๏ผHF ไธ่ฝฝ็ๆไปถๆฒกๆๆง่กๆ้๏ผ็จ sh ่ฐ็จๅณๅฏ / needs an AX8860 with SDK V0.5.1+; call the scripts with sh.
ๆไปถ / Files
| ๆไปถ / File | ่ฏดๆ / Description |
|---|---|
qwen3.5-2b-q8_0.gguf |
ๆจกๅๆ้ / weights๏ผ2.08 GB๏ผmd5 996b32e52bfb5efde860433dc40d0114 |
mmproj-Qwen3.5-2B-BF16.gguf |
่ง่ง็ผ็ ๅจ / vision encoder๏ผ0.8 GB๏ผmd5 47c5b21dc254ba4f0d4bd34454442e31 |
bin/ |
llama.cpp ๆฟ็ซฏ็จๅบ๏ผ้ๆ้พๆฅ๏ผ/ board programs (static) |
kernels/ |
NPU kernel + manifest.json |
kernels_mtp/ |
MTP ๆจกๅผ็จ็ kernel๏ผๅซ MTP ่็จฟๅคดใ4 ่ก verify๏ผ/ kernels for MTP mode (draft head, 4-row verify) |
scenes/๏ผMTP.md |
10 ไธช MTP ๅบๆฏ่ฏทๆฑไธ MTP ่ฏดๆ / 10 MTP scene requests and MTP notes |
run.sh๏ผchat_template.jinja |
่ฟ่ก่ๆฌไธๆๅก็ซฏๅฏน่ฏๆจกๆฟ / run script and the server's chat template |
bench.sh ๅ prompt |
ๆง่ฝๅค็ฐ / benchmark |
่ฏดๆ / Notes
- MTP ๅฏ้๏ผ
MTP=1่ช้ๅบๅผๅฏ๏ผMTP=forceๅผบๅถๅผๅฏ๏ผMTP ๅผๅ ณไธๆนๅ้ๆ ทๅๆฐใ่ดชๅฟ่ฏทๆฑ้กปๆพๅผๅธฆpresence_penalty: 0ๆ่ฝ็จไธ MTP ๅ ้๏ผ้ๆ ท่ฏทๆฑๅจ่ช้ๅบๆจกๅผไธไฟๆๆฎ้่งฃ็ / optional MTP:MTP=1adaptive,MTP=forceforced, without changing sampling. Greedy requests must setpresence_penalty: 0to get the MTP speed-up; sampled requests stay on plain decoding in adaptive mode. - ๆๅก็ซฏ้ป่ฎคไฝฟ็จๆจกๅๅกๆจ่็้ๆ ทๅๆฐ๏ผๆๆฌ่กฅๅ จไธๅพๅๆ่ฟฐไธบ่ดชๅฟ่งฃ็ / the server uses the model card's recommended sampling; completion and image description are greedy.
- ๆๅก็ซฏ้ป่ฎคๅ
ณ้ญๆ่๏ผ
--reasoning off๏ผ๏ผๅนถๅ ่ฎธๅฏน่ฏไธญๆฎต็ system ๆถๆฏ๏ผCodex ็ญ agent ๅฎขๆท็ซฏๅฏ็จ๏ผ/ the server does not think by default and accepts a mid-conversation system message (works with agent clients such as Codex). - Prompt cache ้ป่ฎคๆพๅจ 4 GB ๅ
ๅญ๏ผ่ฎพ็ฝฎ
CACHE_DIR=<SSD ็ฎๅฝ>ๅฏ่ฝ็๏ผ้ๅฏๅไป่ฝๅฝไธญ / the prompt cache lives in 4 GB of host memory; setCACHE_DIR=<an SSD directory>to keep it on disk across restarts. - ่ฟ่กๆถ็ฌๅ NPU 4 ๆ ธ๏ผๆๅก็ซฏ
-np 1/ takes all 4 NPU cores; the server runs-np 1. llama-benchๅฏ็จ๏ผ่ฏทๅ-b 4096 -ub 4096/llama-benchworks; pass-b 4096 -ub 4096.- ไธชๅซ็ฏๅขไธ่ฟ็ปญ่ฏทๆฑๅถๅ
ret = -3๏ผ้่ฏๅณๅฏ / in rare cases a back-to-back request fails withret = -3; retry it.
ๆญฃ็กฎๆง / Correctness
- "The capital of France is" โ "Paris. โฆ"๏ผๅพๅๆ่ฟฐๆญฃ็กฎ๏ผ่ฝฆ่พใๆๅญใไบบ็ฉ๏ผ/ text and image descriptions correct.
- ๆฟ็ซฏ bin ๆดๆฐๅ่พๅบไธไธไธ็ไธ่ด / output matches the previous release after the board-bin update.
- PPL๏ผ
ppl_zh.txt๏ผ-c 512 --chunks 2๏ผ๏ผMTP ๅ ณ้ญ-ub 112.3236ใ-ub 412.3143 / PPL with MTP off: 12.3236 (-ub 1), 12.3143 (-ub 4). - MTP ่็จฟ็ฑ็ฎๆ ๆจกๅ้ token ้ช่ฏ๏ผ่ดชๅฟ 64 token ไธญ 63 ไธชไธๆฎ้่งฃ็ ไธ่ด๏ผๅฏไธไธๅๅคไธบๆฆ็ๅ ไนๅนถๅ็ไธคไธชๅ้ / every MTP draft is verified by the target model; 63 of 64 greedy tokens match plain decoding, the one difference being a near-tie.
ๆบ็ / Sources
- llama.cpp๏ผZHEQIUSHUI/llama.cpp
axera-dev@0104880be
ๆดๆฐ / Changelog
- 2026-11-13๏ผๆดๆฐๆฟ็ซฏ bin๏ผ
ppl_zh.txtๆขไธบๆฐ็ๆต่ฏๆๆฌ๏ผPPL ๆฐๅญ้ไนๆดๆฐ / board-bin update;ppl_zh.txtis a new test text, PPL numbers updated accordingly. - 2026-11-09๏ผๆฐๅขๅฏ้ MTP๏ผ็ปๆๅๅ ๅฎนๆ้ซ 1.68x๏ผ/ optional MTP added (up to 1.68x on structured content).
- 2026-11-09๏ผ็ปไธๆดๆฐๆฟ็ซฏ bin๏ผkernels ไธ GGUF ไธๅ๏ผperplexity ๅบๅฎ่ตฐๅ จ logits / unified board-bin update; kernels and GGUF unchanged; perplexity keeps the full-logits head.
- Downloads last month
- 618
8-bit
Install (macOS, Linux)
# Start a local OpenAI-compatible server with a web UI: llama serve -hf AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16# Run inference directly in the terminal: llama cli -hf AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16