How to use from
llama.cpp
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16
# Run inference directly in the terminal:
llama cli -hf AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16
Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16
# Run inference directly in the terminal:
llama cli -hf AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16
# Run inference directly in the terminal:
./build/bin/llama cli -hf AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16
Use Docker
docker model run hf.co/AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp:BF16
Quick Links

Qwen3.5-2B on AX8860 (llama.cpp + ggml-axera)

ๅœจ็ˆฑ่Šฏ AX8860 ็š„ NPU ไธŠ็”จ llama.cpp ่ฟ่กŒ Qwen3.5-2B๏ผšๆ–‡ๆœฌ + ๅ›พๅƒ่พ“ๅ…ฅ๏ผŒๅฏ้€‰ MTP ๆŠ•ๆœบ่งฃ็ ๏ผŒOpenAI ๅ…ผๅฎนๆœๅŠกใ€‚ Qwen3.5-2B on the Axera AX8860 NPU with llama.cpp: text + image input, optional MTP speculative decoding, OpenAI-compatible server.

ๆ€ง่ƒฝ / Performance

decode 76.2 tokens/s
decode๏ผŒMTP ่‡ช้€‚ๅบ” / adaptive MTP ๆœ€้ซ˜ / up to 128.5 tokens/s๏ผˆ1.68x๏ผ‰
4k prompt prefill 6774 tokens/s๏ผˆ3908 token 0.58 s๏ผ‰
ไธŠไธ‹ๆ–‡ / context ๆœ€้•ฟ / up to 16384

MTP ๅฏ้€‰๏ผˆMTP=1 sh run.sh -s ่‡ช้€‚ๅบ”๏ผŒMTP=force ๅผบๅˆถ๏ผŒ้ป˜่ฎคๅ…ณ้—ญ๏ผ‰/ optional MTP (MTP=1 sh run.sh -s adaptive, MTP=force forced, off by default):

ๅ†…ๅฎน / content ๆ™ฎ้€š / plain ้ป˜่ฎค / adaptive ๅŠ ้€Ÿ / speed-up ๆŽฅๅ—็އ / accept
Linux ๅ‘ฝไปคๅˆ—่กจ / list 76.7 128.5 1.68x 0.633
ๆ•ฐๅญฆ่งฃ้ข˜ / maths 77.3 124.0 1.60x 0.605
Markdown ่กจๆ ผ / table 76.7 117.7 1.54x 0.589
ไธญ่ฏ‘่‹ฑ / translation 75.8 108.0 1.42x 0.500
ไปฃ็  / code 76.3 108.0 1.42x 0.529
ไธญๆ–‡็Ÿญๆ–‡ / Chinese essay 76.7 77.3 1.01x โ€”
่‹ฑๆ–‡ๆ•…ไบ‹ / English story 76.5 77.0 1.01x โ€”
ไธญๆ–‡่งฃ้‡Š / explanation 76.9 77.5 1.01x โ€”
ไปฃ็ ๏ผŒๆŽจ่้‡‡ๆ ท / code, sampled 76.3 76.6 1.00x โ€”
ไธญๆ–‡็Ÿญๆ–‡๏ผŒๆŽจ่้‡‡ๆ ท / essay, sampled 75.7 76.2 1.01x โ€”

ๅ•ไฝ tokens/s๏ผ›sh bench.sh mtp๏ผŒctx 16384๏ผŒๆฏ้กนๆœ€ๅคš็”Ÿๆˆ 256 token๏ผŒๆœชๆณจๆ˜Ž็š„ไธบ่ดชๅฟƒ่งฃ็ ๏ผ›ๆ•ฃๆ–‡ไธŽ้‡‡ๆ ท่ฏทๆฑ‚่‡ช้€‚ๅบ”ไฟๆŒๆ™ฎ้€š่งฃ็  / tokens/s from sh bench.sh mtp, ctx 16384, up to 256 tokens each, greedy unless noted; prose and sampled requests stay on plain decoding.

sh bench.sh ๅค็Žฐ / reproduce๏ผˆ็”Ÿๆˆ 128 token / 128 generated tokens๏ผ›็Ÿญ prompt ไป…ๅˆ—่€—ๆ—ถ / short prompts report time only๏ผ‰๏ผš

prompt -c prompt ่€—ๆ—ถ / time decode tokens/s
5 token 8192 0.05 s โ€”
47 token 8192 0.08 s โ€”
3908 token 8192 0.58 s 75.0
5 token 16384 0.06 s โ€”
47 token 16384 0.07 s โ€”
3908 token 16384 0.60 s 73.8
3908 token๏ผŒๆŽจ่้‡‡ๆ ท / sampled 8192 0.59 s 72.2

ๅฟซ้€Ÿๅผ€ๅง‹ / Quick start

sh run.sh "The capital of France is" 64           # ๆ–‡ๆœฌ / text
sh run.sh -i bus.jpg "Describe this image." 256   # ๅ›พๅƒ / image
sh run.sh -c                                      # ๅ‘ฝไปค่กŒๅฏน่ฏ / chat
sh run.sh -s 8080                                 # OpenAI ๅ…ผๅฎนๆœๅŠก / server
MTP=1 sh run.sh -s 8080                           # ่‡ช้€‚ๅบ” MTP / adaptive MTP
sh bench.sh                                       # ๆ€ง่ƒฝๅค็Žฐ / benchmark

้œ€่ฆ AX8860๏ผŒSDK V0.5.1 ๆˆ–ๆ›ดๆ–ฐ๏ผ›HF ไธ‹่ฝฝ็š„ๆ–‡ไปถๆฒกๆœ‰ๆ‰ง่กŒๆƒ้™๏ผŒ็”จ sh ่ฐƒ็”จๅณๅฏ / needs an AX8860 with SDK V0.5.1+; call the scripts with sh.

ๆ–‡ไปถ / Files

ๆ–‡ไปถ / File ่ฏดๆ˜Ž / Description
qwen3.5-2b-q8_0.gguf ๆจกๅž‹ๆƒ้‡ / weights๏ผŒ2.08 GB๏ผŒmd5 996b32e52bfb5efde860433dc40d0114
mmproj-Qwen3.5-2B-BF16.gguf ่ง†่ง‰็ผ–็ ๅ™จ / vision encoder๏ผŒ0.8 GB๏ผŒmd5 47c5b21dc254ba4f0d4bd34454442e31
bin/ llama.cpp ๆฟ็ซฏ็จ‹ๅบ๏ผˆ้™ๆ€้“พๆŽฅ๏ผ‰/ board programs (static)
kernels/ NPU kernel + manifest.json
kernels_mtp/ MTP ๆจกๅผ็”จ็š„ kernel๏ผˆๅซ MTP ่‰็จฟๅคดใ€4 ่กŒ verify๏ผ‰/ kernels for MTP mode (draft head, 4-row verify)
scenes/๏ผŒMTP.md 10 ไธช MTP ๅœบๆ™ฏ่ฏทๆฑ‚ไธŽ MTP ่ฏดๆ˜Ž / 10 MTP scene requests and MTP notes
run.sh๏ผŒchat_template.jinja ่ฟ่กŒ่„šๆœฌไธŽๆœๅŠก็ซฏๅฏน่ฏๆจกๆฟ / run script and the server's chat template
bench.sh ๅŠ prompt ๆ€ง่ƒฝๅค็Žฐ / benchmark

่ฏดๆ˜Ž / Notes

  • MTP ๅฏ้€‰๏ผšMTP=1 ่‡ช้€‚ๅบ”ๅผ€ๅฏ๏ผŒMTP=force ๅผบๅˆถๅผ€ๅฏ๏ผ›MTP ๅผ€ๅ…ณไธๆ”นๅ˜้‡‡ๆ ทๅ‚ๆ•ฐใ€‚่ดชๅฟƒ่ฏทๆฑ‚้กปๆ˜พๅผๅธฆ presence_penalty: 0 ๆ‰่ƒฝ็”จไธŠ MTP ๅŠ ้€Ÿ๏ผ›้‡‡ๆ ท่ฏทๆฑ‚ๅœจ่‡ช้€‚ๅบ”ๆจกๅผไธ‹ไฟๆŒๆ™ฎ้€š่งฃ็  / optional MTP: MTP=1 adaptive, MTP=force forced, without changing sampling. Greedy requests must set presence_penalty: 0 to get the MTP speed-up; sampled requests stay on plain decoding in adaptive mode.
  • ๆœๅŠก็ซฏ้ป˜่ฎคไฝฟ็”จๆจกๅž‹ๅกๆŽจ่็š„้‡‡ๆ ทๅ‚ๆ•ฐ๏ผŒๆ–‡ๆœฌ่กฅๅ…จไธŽๅ›พๅƒๆ่ฟฐไธบ่ดชๅฟƒ่งฃ็  / the server uses the model card's recommended sampling; completion and image description are greedy.
  • ๆœๅŠก็ซฏ้ป˜่ฎคๅ…ณ้—ญๆ€่€ƒ๏ผˆ--reasoning off๏ผ‰๏ผŒๅนถๅ…่ฎธๅฏน่ฏไธญๆฎต็š„ system ๆถˆๆฏ๏ผˆCodex ็ญ‰ agent ๅฎขๆˆท็ซฏๅฏ็”จ๏ผ‰/ the server does not think by default and accepts a mid-conversation system message (works with agent clients such as Codex).
  • Prompt cache ้ป˜่ฎคๆ”พๅœจ 4 GB ๅ†…ๅญ˜๏ผ›่ฎพ็ฝฎ CACHE_DIR=<SSD ็›ฎๅฝ•> ๅฏ่ฝ็›˜๏ผŒ้‡ๅฏๅŽไป่ƒฝๅ‘ฝไธญ / the prompt cache lives in 4 GB of host memory; set CACHE_DIR=<an SSD directory> to keep it on disk across restarts.
  • ่ฟ่กŒๆ—ถ็‹ฌๅ  NPU 4 ๆ ธ๏ผ›ๆœๅŠก็ซฏ -np 1 / takes all 4 NPU cores; the server runs -np 1.
  • llama-bench ๅฏ็”จ๏ผŒ่ฏทๅŠ  -b 4096 -ub 4096 / llama-bench works; pass -b 4096 -ub 4096.
  • ไธชๅˆซ็Žฏๅขƒไธ‹่ฟž็ปญ่ฏทๆฑ‚ๅถๅ‘ ret = -3๏ผŒ้‡่ฏ•ๅณๅฏ / in rare cases a back-to-back request fails with ret = -3; retry it.

ๆญฃ็กฎๆ€ง / Correctness

  • "The capital of France is" โ†’ "Paris. โ€ฆ"๏ผ›ๅ›พๅƒๆ่ฟฐๆญฃ็กฎ๏ผˆ่ฝฆ่พ†ใ€ๆ–‡ๅญ—ใ€ไบบ็‰ฉ๏ผ‰/ text and image descriptions correct.
  • ๆฟ็ซฏ bin ๆ›ดๆ–ฐๅŽ่พ“ๅ‡บไธŽไธŠไธ€็‰ˆไธ€่‡ด / output matches the previous release after the board-bin update.
  • PPL๏ผˆppl_zh.txt๏ผŒ-c 512 --chunks 2๏ผ‰๏ผšMTP ๅ…ณ้—ญ -ub 1 12.3236ใ€-ub 4 12.3143 / PPL with MTP off: 12.3236 (-ub 1), 12.3143 (-ub 4).
  • MTP ่‰็จฟ็”ฑ็›ฎๆ ‡ๆจกๅž‹้€ token ้ชŒ่ฏ๏ผ›่ดชๅฟƒ 64 token ไธญ 63 ไธชไธŽๆ™ฎ้€š่งฃ็ ไธ€่‡ด๏ผŒๅ”ฏไธ€ไธๅŒๅค„ไธบๆฆ‚็އๅ‡ ไนŽๅนถๅˆ—็š„ไธคไธชๅ€™้€‰ / every MTP draft is verified by the target model; 63 of 64 greedy tokens match plain decoding, the one difference being a near-tie.

ๆบ็  / Sources

ๆ›ดๆ–ฐ / Changelog

  • 2026-11-13๏ผšๆ›ดๆ–ฐๆฟ็ซฏ bin๏ผ›ppl_zh.txt ๆขไธบๆ–ฐ็š„ๆต‹่ฏ•ๆ–‡ๆœฌ๏ผŒPPL ๆ•ฐๅญ—้šไน‹ๆ›ดๆ–ฐ / board-bin update; ppl_zh.txt is a new test text, PPL numbers updated accordingly.
  • 2026-11-09๏ผšๆ–ฐๅขžๅฏ้€‰ MTP๏ผˆ็ป“ๆž„ๅŒ–ๅ†…ๅฎนๆœ€้ซ˜ 1.68x๏ผ‰/ optional MTP added (up to 1.68x on structured content).
  • 2026-11-09๏ผš็ปŸไธ€ๆ›ดๆ–ฐๆฟ็ซฏ bin๏ผ›kernels ไธŽ GGUF ไธๅ˜๏ผ›perplexity ๅ›บๅฎš่ตฐๅ…จ logits / unified board-bin update; kernels and GGUF unchanged; perplexity keeps the full-logits head.
Downloads last month
618
GGUF
Model size
2B params
Architecture
qwen35
Hardware compatibility
Log In to add your hardware

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp

Finetuned
Qwen/Qwen3.5-2B
Quantized
(342)
this model

Collection including AXERA-TECH/Qwen3.5-2B-AX8860-llama.cpp