|
Download README.md from HelloSun/Spark-x2.6-4b-gguf-llama.cpp-cpu: direct link, hf CLI and curl.
- Browser
- Download file 4.09 kB
-
https://huggingface.co/HelloSun/Spark-x2.6-4b-gguf-llama.cpp-cpu/resolve/main/README.md
- Command line
-
hf download hf://HelloSun/Spark-x2.6-4b-gguf-llama.cpp-cpu/README.md
-
curl -L -o README.md https://huggingface.co/HelloSun/Spark-x2.6-4b-gguf-llama.cpp-cpu/resolve/main/README.md
4.09 kB
| license: apache-2.0 | |
| language: | |
| - en | |
| - zh | |
| tags: | |
| - llama.cpp | |
| - gguf | |
| - cpu | |
| - avx512 | |
| - vnni | |
| - amx | |
| - on-device | |
| base_model: XHToken/Spark-X2.6-4B-GGUF | |
| # Spark-X2.6-4B-GGUF + llama.cpp CPU (AVX512-VNNI tuned build) | |
| One-click CPU inference setup for **Spark-X2.6-4B (Q4_K_M)** on llama.cpp, tuned for | |
| Intel Xeon CPUs with **AVX512 / AVX512-VNNI / AMX** (Sapphire Rapids class). | |
| This repo contains: | |
| 1. `spark-x2.6-q4k-vnni-cpu.patch` - operator-level optimization for llama.cpp: | |
| a new 512-bit **AVX512-VNNI** `ggml_vec_dot_q4_K_q8_K` kernel (the decode-path | |
| matmul for Q4_K weights), rewritten for the real Q4_K nibble layout with | |
| `vpermi2d` q8 gathering + `vpdpbusd` MACs, all-integer accumulation with a | |
| single int32 -> fp conversion per block. Verified against the full llama.cpp | |
| op test suite: **24086 / 24086 tests pass**. | |
| 2. `build.sh` - one-click script: pins llama.cpp, applies the patch, builds with | |
| `-march=native`, downloads the model, starts `llama-server` on **port 20100**. | |
| ## Quick start | |
| ```bash | |
| bash build.sh # build + start server on port 20100 | |
| # optional env overrides: PORT, THREADS, FA, CTX, WORKDIR, SKIP_BUILD=1 | |
| ``` | |
| On a 2-socket NUMA box the decode path benefits from single-node pinning: | |
| ```bash | |
| taskset -c 0-19,40-59 bash build.sh # e.g. sockets: node0 logical CPUs | |
| # or for max prefill: taskset -c 0-39 bash build.sh | |
| ``` | |
| OpenAI-compatible API once running: | |
| ```bash | |
| curl http://localhost:20100/v1/chat/completions \ | |
| -H "Content-Type: application/json" \ | |
| -d '{"model":"spark2.6","messages":[{"role":"user","content":"Hello"}]}' | |
| ``` | |
| ## Benchmark (final numbers) | |
| Environment: 2x Intel Xeon Sapphire-Rapid-class (ES), 40C/80T, 2 NUMA nodes, | |
| no GPU, 251 GB RAM. llama.cpp `b1-3d94ffe` (2026-11-09), built with | |
| `-march=native` (AVX512 + VNNI + AMX present). Model: Spark-X2.6-4B Q4_K_M | |
| (3.30 GiB, 4.45B params). `llama-bench -p 512 -n 128`, 40 threads. | |
| | Config | Prefill pp512 (t/s) | Decode tg128 (t/s) | | |
| |---|---|---| | |
| | baseline: upstream build, no pinning, FA=0 | 385.3 +- 4.5 | 29.4 +- 2.0 | | |
| | final build, no pinning, FA=0 | 346.7 +- 36.1 | 30.3 +- 1.4 | | |
| | **final build, node0-pinned, FA=1 (recommended for decode)** | 353.2 +- 9.3 | **35.66 +- 0.71** | | |
| | **final build, physical-cores-pinned (recommended for prefill)** | **447.3** (best) | 29.6 +- 1.8 | | |
| Highlights: | |
| - **Decode: 29.4 -> 35.7 t/s (+21%)** with FA=1 + single-NUMA-node pinning | |
| (`taskset -c 0-19,40-59` on this box). | |
| - **Prefill: up to 447 t/s (+16%)** with physical-core pinning | |
| (`taskset -c 0-39`). Note the shared host shows +-10-15% run-to-run noise | |
| on prefill; decode pinned to one node is stable (+-2%). | |
| - 80 threads across both sockets collapses decode to <1 t/s on this VM | |
| (cross-NUMA scheduling) - avoid unpinned full-width thread counts. | |
| Measured machine bandwidth: ~300 GB/s aggregate read (40T); the decode path | |
| runs at ~100 GB/s effective weight traffic, i.e. it is latency/sync-bound | |
| (~400 graph ops + barriers per token), not bandwidth-bound, which is why the | |
| faster Q4_K kernel shows as parity end-to-end while affinity/FA tuning shows | |
| the full gain. | |
| ## What the patch changes | |
| `ggml/src/ggml-cpu/arch/x86/quants.c`, `ggml_vec_dot_q4_K_q8_K`: | |
| - adds a `__AVX512VNNI__ && __AVX512BW__` path processing 64 q4_K bytes per | |
| half-block with two `vpdpbusd` (int8 dot-product) instructions | |
| - gathers the matching q8 element ranges with one `vpermi2d` per nibble stream | |
| (Q4_K packs 32-byte groups: low nibbles -> even sub-blocks, high nibbles -> | |
| odd sub-blocks) | |
| - applies the 6-bit sub-block scales with a single `vmulld` per stream and | |
| keeps the whole 256-element block in exact int32 until one `vcvtdq2ps` + | |
| `vfmadd` per block; the min-correction path is unchanged from upstream | |
| All other kernels (tiled VNNI prefill, Q6_K, Q8_0) are left at upstream, which | |
| measured at parity or better on this CPU. | |
| ## Attribution | |
| - Model: [XHToken/Spark-X2.6-4B-GGUF](https://huggingface.co/XHToken/Spark-X2.6-4B-GGUF) | |
| - Runtime: [ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp) @ `3d94ffe` | |