HelloSun/Spark-x2.6-4b-gguf-llama.cpp-cpu

Spark-X2.6-4B-GGUF + llama.cpp CPU (AVX512-VNNI tuned build)

One-click CPU inference setup for Spark-X2.6-4B (Q4_K_M) on llama.cpp, tuned for Intel Xeon CPUs with AVX512 / AVX512-VNNI / AMX (Sapphire Rapids class).

This repo contains:

1. spark-x2.6-q4k-vnni-cpu.patch - operator-level optimization for llama.cpp: a new 512-bit AVX512-VNNI ggml_vec_dot_q4_K_q8_K kernel (the decode-path matmul for Q4_K weights), rewritten for the real Q4_K nibble layout with vpermi2d q8 gathering + vpdpbusd MACs, all-integer accumulation with a single int32 -> fp conversion per block. Verified against the full llama.cpp op test suite: 24086 / 24086 tests pass.
2. build.sh - one-click script: pins llama.cpp, applies the patch, builds with -march=native, downloads the model, starts llama-server on port 20100.

Quick start

bash build.sh                      # build + start server on port 20100
# optional env overrides: PORT, THREADS, FA, CTX, WORKDIR, SKIP_BUILD=1

On a 2-socket NUMA box the decode path benefits from single-node pinning:

taskset -c 0-19,40-59 bash build.sh        # e.g. sockets: node0 logical CPUs
# or for max prefill:  taskset -c 0-39 bash build.sh

OpenAI-compatible API once running:

curl http://localhost:20100/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"spark2.6","messages":[{"role":"user","content":"Hello"}]}'

Benchmark (final numbers)

Environment: 2x Intel Xeon Sapphire-Rapid-class (ES), 40C/80T, 2 NUMA nodes, no GPU, 251 GB RAM. llama.cpp b1-3d94ffe (2026-11-09), built with -march=native (AVX512 + VNNI + AMX present). Model: Spark-X2.6-4B Q4_K_M (3.30 GiB, 4.45B params). llama-bench -p 512 -n 128, 40 threads.

Config	Prefill pp512 (t/s)	Decode tg128 (t/s)
baseline: upstream build, no pinning, FA=0	385.3 +- 4.5	29.4 +- 2.0
final build, no pinning, FA=0	346.7 +- 36.1	30.3 +- 1.4
final build, node0-pinned, FA=1 (recommended for decode)	353.2 +- 9.3	35.66 +- 0.71
final build, physical-cores-pinned (recommended for prefill)	447.3 (best)	29.6 +- 1.8

Highlights:

• Decode: 29.4 -> 35.7 t/s (+21%) with FA=1 + single-NUMA-node pinning (taskset -c 0-19,40-59 on this box).
• Prefill: up to 447 t/s (+16%) with physical-core pinning (taskset -c 0-39). Note the shared host shows +-10-15% run-to-run noise on prefill; decode pinned to one node is stable (+-2%).
• 80 threads across both sockets collapses decode to <1 t/s on this VM (cross-NUMA scheduling) - avoid unpinned full-width thread counts.

Measured machine bandwidth: 300 GB/s aggregate read (40T); the decode path runs at ~100 GB/s effective weight traffic, i.e. it is latency/sync-bound (400 graph ops + barriers per token), not bandwidth-bound, which is why the faster Q4_K kernel shows as parity end-to-end while affinity/FA tuning shows the full gain.

What the patch changes

ggml/src/ggml-cpu/arch/x86/quants.c, ggml_vec_dot_q4_K_q8_K:

• adds a __AVX512VNNI__ && __AVX512BW__ path processing 64 q4_K bytes per half-block with two vpdpbusd (int8 dot-product) instructions
• gathers the matching q8 element ranges with one vpermi2d per nibble stream (Q4_K packs 32-byte groups: low nibbles -> even sub-blocks, high nibbles -> odd sub-blocks)
• applies the 6-bit sub-block scales with a single vmulld per stream and keeps the whole 256-element block in exact int32 until one vcvtdq2ps + vfmadd per block; the min-correction path is unchanged from upstream

All other kernels (tiled VNNI prefill, Q6_K, Q8_0) are left at upstream, which measured at parity or better on this CPU.

Attribution

• Model: XHToken/Spark-X2.6-4B-GGUF
• Runtime: ggml-org/llama.cpp @ 3d94ffe