Qwen3.5-9B-XeFP4

Qwen/Qwen3.5-9B quantized to XeFP4, an open 4-bit weight format for Intel Arc GPUs. It runs on XeQ Runtime, the OpenCL engine for XeFP4: https://github.com/kinoqdev/XeFP4.

These files do not load in transformers, llama.cpp or vLLM. They need XeQ Runtime. This repository contains the text model only.

Files

File Size SHA-256 (.bin) Content
q35_xe8g32e.bin + .json 5,086,524,416 B d9fcd99cdfe154e7bfbe2a89b5e18eb977c47f0db6a1355331ec7b878b161bcf main model
q35_mtp_xe8.bin + .json 124,672,000 B 618855959c215c74dfcb7da151471e7ad042890a7ef6273e5dbecce3669fc640 MTP head for speculative decoding (optional)
vocab_order.npy 993,408 B b40ec3751835a586b201f86e4ef489d8e911c930a265dc66869d3beab5fb6314 token order used by the MTP draft step
tokenizer.json, tokenizer_config.json, chat_template.jinja from the source model
LICENSE, NOTICE

.json header SHA-256: q35_xe8g32e.json a33ad7f7476f8ed2f8c831ce53248b30c191b76546f733e7a9be5ce7d31744be, q35_mtp_xe8.json c2a4b472ee475b89d92eede248213243e9dbe4cf9cbef7fc29fd2e7629874782. The .bin files are reproducible: converting on an Arc 140T gives the same bytes as on the A770.

Usage

git clone https://github.com/kinoqdev/XeFP4 && cd XeFP4
pip install -r requirements.txt
python -m xeq selftest <path to this repository>
python -m xeq run <path to this repository> -p "Explain Rayleigh scattering in two sentences."
python -m xeq serve <path to this repository> --port 8080 --ctx 32768

MTP is used automatically when q35_mtp_xe8.bin is present (--no-mtp turns it off).

Quantization

Part Format
all linear layers XeFP4-8: E2M1 values, block 32, 3-bit block factor, FP16 row scale (4.097 bits/weight)
embedding XeFP4-8
lm_head INT8, per-row FP32 scale
MTP layer XeFP4-8 (fc split by input half)
norms, gates, conv, A_log, dt_bias FP32

The encoder is deterministic and uses no calibration data. The format is specified in XEFP4_SPEC.md.

Performance

Intel Arc A770 16 GB (driver 32.0.101.9150), batch 1, greedy decoding, scripted benchmark (2026-11-17).

decode tok/s prefill tok/s
XeFP4, MTP off 65.7 2,499 (2K tokens)
XeFP4, MTP on (acceptance 0.91) 94.2
llama.cpp b11053 SYCL, Q4_K_M (6.2 GB) 37.9 836 (512)
llama.cpp b11053 SYCL, IQ4_XS (5.5 GB) 35.5 834 (512)
OpenVINO 2026.4.1 GenAI INT4, MTP off / on 46.0 / 56.7 2,448

Long context (Korean text):

context prefill tok/s decode, MTP off decode, MTP on
2K 2,499 63.6 86.2
8K 2,362 56.5 84.8
16K 2,107 49.7 76.0
32K 1,738 39.7 62.7
  • llama.cpp Q4_K_M decode at depth 16K: 27.4 tok/s.
  • MTP-on figures depend on the text (the draft acceptance rate varies). The long-context rows were measured on Korean prose; reproduce with python -m xeq bench <folder> --text <file>.
  • MTP speculative decoding gives the same output tokens as greedy decoding without MTP (checked at every length above).
  • Arc 140T (Xe-LPG+, Core Ultra iGPU, driver 32.0.101.9033): decode 13.8 tok/s (MTP off) / 23.0 tok/s (MTP on, short prompt); 32K context 7.1 tok/s (MTP off); prefill 2K tokens in 2.9 s (about 700 tok/s), 16K in 30 s. Output tokens match the A770.

Accuracy

Perplexity on 16 FineWeb documents ร— 512 tokens (lower is better):

BF16 XeFP4 (this model) GGUF Q4_K_M GGUF IQ4_XS
11.164 11.682 (+4.6%) 11.267 (+0.9%) 11.341 (+1.6%)

HumanEval-C++ (161 problems, greedy, up to 1,024 new tokens, then up to 2 retries with the compiler or test error):

first attempt after retries
INT8 weights (reference, same prompts) 134 / 161 143 / 161
XeFP4 (this model), MTP on 126 / 161 136 / 161

Measured on an Arc 140T. On the same problems, 11 passed only with INT8 and 3 only with XeFP4 (first attempt, exact McNemar p = 0.057).

Sampling

Decoding is greedy by default (same output every time; best for code, facts, translation and tool calls). For open-ended chat, writing and thinking mode (--think), turn on sampling with --temperature, --top-p, --top-k, --min-p, --presence-penalty, --repetition-penalty and --seed (the server reads the same fields from the request). Source model recommendations: non-thinking chat temperature 0.7, top_p 0.8, top_k 20, presence_penalty 1.5; thinking mode temperature 1.0, top_p 0.95, top_k 20, presence_penalty 1.5 (0.6 / 0.95 / 20 / 0 for precise coding). top_k is capped at 64. With sampling, MTP still gives the same output distribution as plain decoding, but its speed-up is smaller and depends on the text; sampling itself costs 2โ€“3% of decode speed. The benchmarks above use greedy decoding.

Limitations

  • Tested on Intel Arc A770 (Xe-HPG) and Arc 140T (Xe-LPG+). Other Intel GPUs are untested.
  • Text only. The source model's vision encoder is not included.
  • Batch size 1.
  • Prefill and token-by-token decoding differ slightly numerically, so greedy output can differ at near-tie positions.
  • XeFP4 loses more accuracy than calibrated GGUF quants of similar size (see above).

License and attribution

  • Derived from Qwen3.5-9B by the Qwen team (Alibaba Cloud), licensed under Apache-2.0.
  • The weights were modified by quantization to the XeFP4 format.
  • See LICENSE and NOTICE.
  • XeFP4 and XeQ Runtime are not affiliated with or endorsed by Alibaba Cloud or Intel.
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for KINOQ/Qwen3.5-9B-XeFP4

Finetuned
Qwen/Qwen3.5-9B
Quantized
(736)
this model