HelloSun's picture
Upload README.md with huggingface_hub
d2e8514 verified
|
Raw History Blame Contribute Delete
4.09 kB
---
license: apache-2.0
language:
- en
- zh
tags:
- llama.cpp
- gguf
- cpu
- avx512
- vnni
- amx
- on-device
base_model: XHToken/Spark-X2.6-4B-GGUF
---
# Spark-X2.6-4B-GGUF + llama.cpp CPU (AVX512-VNNI tuned build)
One-click CPU inference setup for **Spark-X2.6-4B (Q4_K_M)** on llama.cpp, tuned for
Intel Xeon CPUs with **AVX512 / AVX512-VNNI / AMX** (Sapphire Rapids class).
This repo contains:
1. `spark-x2.6-q4k-vnni-cpu.patch` - operator-level optimization for llama.cpp:
a new 512-bit **AVX512-VNNI** `ggml_vec_dot_q4_K_q8_K` kernel (the decode-path
matmul for Q4_K weights), rewritten for the real Q4_K nibble layout with
`vpermi2d` q8 gathering + `vpdpbusd` MACs, all-integer accumulation with a
single int32 -> fp conversion per block. Verified against the full llama.cpp
op test suite: **24086 / 24086 tests pass**.
2. `build.sh` - one-click script: pins llama.cpp, applies the patch, builds with
`-march=native`, downloads the model, starts `llama-server` on **port 20100**.
## Quick start
```bash
bash build.sh # build + start server on port 20100
# optional env overrides: PORT, THREADS, FA, CTX, WORKDIR, SKIP_BUILD=1
```
On a 2-socket NUMA box the decode path benefits from single-node pinning:
```bash
taskset -c 0-19,40-59 bash build.sh # e.g. sockets: node0 logical CPUs
# or for max prefill: taskset -c 0-39 bash build.sh
```
OpenAI-compatible API once running:
```bash
curl http://localhost:20100/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"spark2.6","messages":[{"role":"user","content":"Hello"}]}'
```
## Benchmark (final numbers)
Environment: 2x Intel Xeon Sapphire-Rapid-class (ES), 40C/80T, 2 NUMA nodes,
no GPU, 251 GB RAM. llama.cpp `b1-3d94ffe` (2026-11-09), built with
`-march=native` (AVX512 + VNNI + AMX present). Model: Spark-X2.6-4B Q4_K_M
(3.30 GiB, 4.45B params). `llama-bench -p 512 -n 128`, 40 threads.
| Config | Prefill pp512 (t/s) | Decode tg128 (t/s) |
|---|---|---|
| baseline: upstream build, no pinning, FA=0 | 385.3 +- 4.5 | 29.4 +- 2.0 |
| final build, no pinning, FA=0 | 346.7 +- 36.1 | 30.3 +- 1.4 |
| **final build, node0-pinned, FA=1 (recommended for decode)** | 353.2 +- 9.3 | **35.66 +- 0.71** |
| **final build, physical-cores-pinned (recommended for prefill)** | **447.3** (best) | 29.6 +- 1.8 |
Highlights:
- **Decode: 29.4 -> 35.7 t/s (+21%)** with FA=1 + single-NUMA-node pinning
(`taskset -c 0-19,40-59` on this box).
- **Prefill: up to 447 t/s (+16%)** with physical-core pinning
(`taskset -c 0-39`). Note the shared host shows +-10-15% run-to-run noise
on prefill; decode pinned to one node is stable (+-2%).
- 80 threads across both sockets collapses decode to <1 t/s on this VM
(cross-NUMA scheduling) - avoid unpinned full-width thread counts.
Measured machine bandwidth: ~300 GB/s aggregate read (40T); the decode path
runs at ~100 GB/s effective weight traffic, i.e. it is latency/sync-bound
(~400 graph ops + barriers per token), not bandwidth-bound, which is why the
faster Q4_K kernel shows as parity end-to-end while affinity/FA tuning shows
the full gain.
## What the patch changes
`ggml/src/ggml-cpu/arch/x86/quants.c`, `ggml_vec_dot_q4_K_q8_K`:
- adds a `__AVX512VNNI__ && __AVX512BW__` path processing 64 q4_K bytes per
half-block with two `vpdpbusd` (int8 dot-product) instructions
- gathers the matching q8 element ranges with one `vpermi2d` per nibble stream
(Q4_K packs 32-byte groups: low nibbles -> even sub-blocks, high nibbles ->
odd sub-blocks)
- applies the 6-bit sub-block scales with a single `vmulld` per stream and
keeps the whole 256-element block in exact int32 until one `vcvtdq2ps` +
`vfmadd` per block; the min-correction path is unchanged from upstream
All other kernels (tiled VNNI prefill, Q6_K, Q8_0) are left at upstream, which
measured at parity or better on this CPU.
## Attribution
- Model: [XHToken/Spark-X2.6-4B-GGUF](https://huggingface.co/XHToken/Spark-X2.6-4B-GGUF)
- Runtime: [ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp) @ `3d94ffe`