Qwen3.5-9B-XeFP4
Qwen/Qwen3.5-9B quantized to XeFP4, an open 4-bit weight format for Intel Arc GPUs. It runs on XeQ Runtime, the OpenCL engine for XeFP4: https://github.com/kinoqdev/XeFP4.
These files do not load in transformers, llama.cpp or vLLM. They need XeQ Runtime. This repository contains the text model only.
Files
| File | Size | SHA-256 (.bin) |
Content |
|---|---|---|---|
q35_xe8g32e.bin + .json |
5,086,524,416 B | d9fcd99cdfe154e7bfbe2a89b5e18eb977c47f0db6a1355331ec7b878b161bcf |
main model |
q35_mtp_xe8.bin + .json |
124,672,000 B | 618855959c215c74dfcb7da151471e7ad042890a7ef6273e5dbecce3669fc640 |
MTP head for speculative decoding (optional) |
vocab_order.npy |
993,408 B | b40ec3751835a586b201f86e4ef489d8e911c930a265dc66869d3beab5fb6314 |
token order used by the MTP draft step |
tokenizer.json, tokenizer_config.json, chat_template.jinja |
from the source model | ||
LICENSE, NOTICE |
.json header SHA-256: q35_xe8g32e.json a33ad7f7476f8ed2f8c831ce53248b30c191b76546f733e7a9be5ce7d31744be,
q35_mtp_xe8.json c2a4b472ee475b89d92eede248213243e9dbe4cf9cbef7fc29fd2e7629874782.
The .bin files are reproducible: converting on an Arc 140T gives the same bytes as on the A770.
Usage
git clone https://github.com/kinoqdev/XeFP4 && cd XeFP4
pip install -r requirements.txt
python -m xeq selftest <path to this repository>
python -m xeq run <path to this repository> -p "Explain Rayleigh scattering in two sentences."
python -m xeq serve <path to this repository> --port 8080 --ctx 32768
MTP is used automatically when q35_mtp_xe8.bin is present (--no-mtp turns it off).
Quantization
| Part | Format |
|---|---|
| all linear layers | XeFP4-8: E2M1 values, block 32, 3-bit block factor, FP16 row scale (4.097 bits/weight) |
| embedding | XeFP4-8 |
| lm_head | INT8, per-row FP32 scale |
| MTP layer | XeFP4-8 (fc split by input half) |
| norms, gates, conv, A_log, dt_bias | FP32 |
The encoder is deterministic and uses no calibration data. The format is specified in XEFP4_SPEC.md.
Performance
Intel Arc A770 16 GB (driver 32.0.101.9150), batch 1, greedy decoding, scripted benchmark (2026-11-17).
| decode tok/s | prefill tok/s | |
|---|---|---|
| XeFP4, MTP off | 65.7 | 2,499 (2K tokens) |
| XeFP4, MTP on (acceptance 0.91) | 94.2 | |
| llama.cpp b11053 SYCL, Q4_K_M (6.2 GB) | 37.9 | 836 (512) |
| llama.cpp b11053 SYCL, IQ4_XS (5.5 GB) | 35.5 | 834 (512) |
| OpenVINO 2026.4.1 GenAI INT4, MTP off / on | 46.0 / 56.7 | 2,448 |
Long context (Korean text):
| context | prefill tok/s | decode, MTP off | decode, MTP on |
|---|---|---|---|
| 2K | 2,499 | 63.6 | 86.2 |
| 8K | 2,362 | 56.5 | 84.8 |
| 16K | 2,107 | 49.7 | 76.0 |
| 32K | 1,738 | 39.7 | 62.7 |
- llama.cpp Q4_K_M decode at depth 16K: 27.4 tok/s.
- MTP-on figures depend on the text (the draft acceptance rate varies). The long-context rows were measured on Korean prose; reproduce with
python -m xeq bench <folder> --text <file>. - MTP speculative decoding gives the same output tokens as greedy decoding without MTP (checked at every length above).
- Arc 140T (Xe-LPG+, Core Ultra iGPU, driver 32.0.101.9033): decode 13.8 tok/s (MTP off) / 23.0 tok/s (MTP on, short prompt); 32K context 7.1 tok/s (MTP off); prefill 2K tokens in 2.9 s (about 700 tok/s), 16K in 30 s. Output tokens match the A770.
Accuracy
Perplexity on 16 FineWeb documents ร 512 tokens (lower is better):
| BF16 | XeFP4 (this model) | GGUF Q4_K_M | GGUF IQ4_XS |
|---|---|---|---|
| 11.164 | 11.682 (+4.6%) | 11.267 (+0.9%) | 11.341 (+1.6%) |
HumanEval-C++ (161 problems, greedy, up to 1,024 new tokens, then up to 2 retries with the compiler or test error):
| first attempt | after retries | |
|---|---|---|
| INT8 weights (reference, same prompts) | 134 / 161 | 143 / 161 |
| XeFP4 (this model), MTP on | 126 / 161 | 136 / 161 |
Measured on an Arc 140T. On the same problems, 11 passed only with INT8 and 3 only with XeFP4 (first attempt, exact McNemar p = 0.057).
Sampling
Decoding is greedy by default (same output every time; best for code, facts, translation and tool calls).
For open-ended chat, writing and thinking mode (--think), turn on sampling with --temperature, --top-p, --top-k,
--min-p, --presence-penalty, --repetition-penalty and --seed (the server reads the same fields from the request).
Source model recommendations: non-thinking chat temperature 0.7, top_p 0.8, top_k 20, presence_penalty 1.5;
thinking mode temperature 1.0, top_p 0.95, top_k 20, presence_penalty 1.5 (0.6 / 0.95 / 20 / 0 for precise coding).
top_k is capped at 64. With sampling, MTP still gives the same output distribution as plain decoding, but its speed-up
is smaller and depends on the text; sampling itself costs 2โ3% of decode speed. The benchmarks above use greedy decoding.
Limitations
- Tested on Intel Arc A770 (Xe-HPG) and Arc 140T (Xe-LPG+). Other Intel GPUs are untested.
- Text only. The source model's vision encoder is not included.
- Batch size 1.
- Prefill and token-by-token decoding differ slightly numerically, so greedy output can differ at near-tie positions.
- XeFP4 loses more accuracy than calibrated GGUF quants of similar size (see above).
License and attribution
- Derived from Qwen3.5-9B by the Qwen team (Alibaba Cloud), licensed under Apache-2.0.
- The weights were modified by quantization to the XeFP4 format.
- See
LICENSEandNOTICE. - XeFP4 and XeQ Runtime are not affiliated with or endorsed by Alibaba Cloud or Intel.