Q4_0 files built for the Hexagon NPU. Every operation runs on the NPU; measured on a Snapdragon 8 Elite Gen 5.
AI & ML interests
KV cache compression, inference optimization, model compression
Recent Activity
Organization Card
Model families: Qwen · Gemma 4 · Llama · Mistral · LFM2: dense, mixture-of-experts and hybrid, 1B to 35B
Browse: NPU-native GGUF · Hi-Fi GGUF · KV cache for servers
Every number below is measured on the exact file we publish, against the strongest public file at the same size or larger, with the receipts in the repo.
▚ On device: GGUF files for phones, laptops and NPUs
New: Qwen3.5-4B NPU-native Q4_0
| File | Bytes | KL to bf16, task text ↓ | KL to bf16, general text ↓ | Snapdragon 8 Elite Gen 5 NPU, prompt / generation |
|---|---|---|---|---|
| unsloth Q4_0 | 2,583,221,408 | 0.0464 | 0.0287 | 955 / 17.2 tok/s |
| AutoRound Q4_0 | 2,611,782,976 | 0.0363 | 0.0213 | |
| fraQtl Q4_0 | 2,583,180,384 | 0.0289 | 0.0206 | 950 / 17.1 tok/s |
Every operation runs on the Hexagon NPU (0 of 32,165 rejected). Answers on 500 questions: GSM8K 233 vs 233, IFEval 194 vs 190 (difference not statistically significant).
Earlier releases
| Model | Closer to the original (KL to bf16 or Q8_0), same size | Repo |
|---|---|---|
| Qwen3-4B-Instruct-2507 | −55.9% code/math, −46.4% general vs the community Q4_K_M; 9/9 needles at 262K | Qwen3-4B-Instruct-2507-Hi-Fi-GGUF |
| LFM2-1.2B | −43.4% instruction/tool text, −22.7% knowledge/reasoning vs Liquid AI's own Q4_K_M | LFM2-1.2B-Hi-Fi-GGUF |
| Gemma-4-E2B-it | −36.0% code/math, −27.2% general vs bartowski Q4_K_M; phone build 21.6 tok/s on an iPhone-16-class device | Gemma-4-E2B-it-Hi-Fi-GGUF |
| Gemma-4-E4B-it | −16.2% code/math, −4.2% general vs bartowski Q4_K_M | Gemma-4-E4B-it-Hi-Fi-GGUF |
Five of our builds are mirrored on the Qualcomm AI Hub community org and are its five most-downloaded models.
▚ On servers: compressed KV cache inside vLLM
Qwen3.8-27B, 128K context per user, one H100: 17 users where FP8 KV fits 9. Stock vLLM plus one pip install.
| KV format | Users at 128K | Per-user decode |
|---|---|---|
| FP8 | 9 | 19.0 tok/s |
| vLLM native int4 | 17 | 9.5 tok/s |
| fraQtl | 17 | 12.4 tok/s |
1.31× faster per user than vLLM's native 4-bit KV, using 3% more memory per token. Accuracy: no significant difference (mean KL, 30 documents); equivalence not shown. Against FP8 KV, accuracy is slightly below (mean KL 0.075 vs 0.066, 30 documents), and at 17 users each user decodes slower than FP8 does at 9.
Earlier: 9 users at 128K on one A100 where FP8's best is 4 (Qwen3-4B-Instruct-2507, 134.1 tok/s aggregate, 9/9 retrieval). KV sidecars · runtime
▚ Toolkit
| What | Repos | |
|---|---|---|
| 01 · Weights | Hi-Fi GGUF models | Qwen3.5-4B NPU · Qwen3-4B-2507 · Qwen3.8-27B · Qwen3.6-35B-A3B · Gemma-4-E2B · Gemma-4-E4B · Gemma-4-26B-A4B · Llama-3.2-1B · Llama-3.1-8B · LFM2-1.2B |
| 02 · KV cache | KV sidecars | Qwen3-4B-2507 · Mistral-7B v0.3 · Mistral-Nemo 12B |
| 03 · Runtime | Membrane runtimes | fraqtl-sm80-runtime · membrane-llamacpp-runtime · Qwen3.6-35B-A3B MTP runtime |
| 04 · Free tool | fraqtl-diagnostic | pip install fraqtl-diagnostic · Run in your browser · PyPI · GitHub |
▚ Work with us
Bring a model and a task: we deliver a file that fits your device and stays close to the original, with receipts. fraqtl.ai · contact@fraqtl.ai
Paper: arXiv:2604.11501 · Patent pending.
models 18
fraQtl/Qwen3.5-4B-NPU-GGUF
4B • Updated • 229
fraQtl/LFM2-1.2B-Hi-Fi-GGUF
1B • Updated • 1.28k
fraQtl/qwen3-4b-instruct-2507-kv-sidecars
Updated • 21 • 1
fraQtl/mistral-7b-instruct-v0.3-kv-sidecars
Updated • 14
fraQtl/mistral-nemo-instruct-2407-kv-sidecars
Updated
fraQtl/fraqtl-sm80-runtime
Updated • 15
fraQtl/fraqtl-membrane-llamacpp-runtime
Updated
fraQtl/Qwen3.8-27B-Hi-Fi-GGUF
Text Generation • 27B • Updated • 123 • 2
fraQtl/Qwen3.6-35B-A3B-compressed
Text Generation • 2B • Updated • 448 • 6
fraQtl/Qwen3.6-35B-A3B-Hi-Fi-MTP-runtime
Text Generation • 36B • Updated • 263 • 2