GGUF
conversational
Experiment_1 / onednn /README.md
HelloSun's picture
Integrate oneDNN/OpenVINO CPU backend study; merge docs; point build script at Experiment_1
da11688 verified
|
Raw History Blame Contribute Delete
10.4 kB

openPangu-R-7B-2512 CPU backend comparison: llama.cpp vs oneDNN vs OpenVINO

Part of HelloSun/Experiment_1: the GGUF, the llama.cpp patch and the quick start are in the repository root (../README.md). Paths below are relative to onednn/.

Three CPU inference backends on openPangu-R-7B-2512 (27 layers, hidden 4096, 8.29 B params), benchmarked steady-state (pp512 / tg128, llama-bench -r 5 or median of 128 decode steps). Headline config: Q4_K_M for the llama.cpp / oneDNN paths, and the OpenVINO IR built from the fp32 HF weights.

Environment

item value
CPU Intel Xeon Platinum 8559C, 192 threads, 2 NUMA, AMX BF16/INT8, AVX512
container 16 CPUs (cpu.max=1600), 80 GiB memory guard, no GPU
OS Debian 13
oneDNN 3.7.2 (libdnnl-dev)
OpenVINO 2026.4.1 (runtime), transformers 4.53.2
OpenVINO GenAI 2026.4.1 (openvino-genai)
llama.cpp v0.7.0, commit 620d2225cc436ab0b001e3200e590e3b286ef9ab + openPangu-R patch
model openPangu-R-7B-2512 Q4_K_M, 4.72 GiB, 8.29 B params

Results (Q4_K_M)

ppl512 = perplexity over a fixed 512-token (EN+ZH) text; argmax_agree = per-position argmax agreement with the OpenVINO f32 graph. Both quality columns only apply to the OpenVINO rows (llama.cpp / oneDNN perf was measured with llama-bench).

backend weights pp512 tok/s tg128 tok/s ppl512 argmax threads
llama.cpp (plain AVX512) Q4_K_M 180.50 13.39 - - 16
llama.cpp (default AMX+repack) Q4_K_M 112.77 12.19 - - 16
oneDNN int8 90.09 7.67 - - 1 (+OMP 16)
oneDNN bf16 75.52 4.79 - - 1 (+OMP 16)
OpenVINO f32 125.60 2.22 28.82 1.000 16
OpenVINO bf16-hint 209.43 3.93 28.85 0.975 16
OpenVINO int8-asym 219.17 4.70 29.11 0.951 16
OpenVINO int8-sym 226.68 - 29.34 0.941 16
OpenVINO int4-asym 223.65 5.20 32.68 0.703 16
OpenVINO int4-sym 222.48 - 42.95 0.701 16

llama-bench -r 5, 16 CPUs. OpenVINO tg128 is the median of 128 decode steps (fixed 640-slot KV cache). This is a shared host: the OpenVINO f32/bf16 prefill ranged between ~126 and ~366 tok/s across sessions, so treat the OpenVINO prefill column as order-of-magnitude, not exact.

Notes:

  • The llama.cpp Q4_K_M build is the fastest overall on this 16-CPU container: 13.39 tg128, 180.50 pp512.
  • OpenVINO has by far the fastest prefill (366 pp512 at bf16, 128 at f32) but the slowest decode here, because its decode graph always runs attention over the full 640-slot cache and moves all KV-cache tensors across the host/plugin boundary each step.
  • oneDNN loses on both axes. Decode is bandwidth-bound: bf16 weights are 16.6 GB/token vs 5 GB/token for Q4_K; int8 is 8.3 GB/token. The AMX int8 matmul kernel itself is fast (7.1 TFLOPS at M=512, 4096^3 in the microbenchmark); the loss is the weight format and per-call activation quantization, not the kernel.

OpenVINO

The published OpenVINO GenAI GGUF reader and optimum-cli export openvino cannot handle this model (openpangu-r architecture, custom ops), and torch.compile(backend="openvino") fails on the model's symbolic shapes. The working route is a direct ov.convert_model (TorchScript trace) of the HF PanguEmbeddedForCausalLM:

  • prefill graph: convert_ir.py -> pangu_f32_512.xml/bin
  • decode graph: ov_decode.py --dur convert (KV cache + the two WindowBuffer states are threaded through the graph as explicit inputs/outputs, so the IR stays stateless). Convert with fp32 weights, then compile to bf16 with INFERENCE_PRECISION_HINT.

Accuracy (fp32 hint vs HF eager, 512-token prefill): logits maxdiff 0.0004, 100% argmax agreement; decode-step argmax matches HF. The bf16 hint drops accuracy (argmax agreement 75%) because the model is numerically sensitive (error amplifies ~1.3x per layer), so bf16 is a speed/accuracy trade-off.

OpenVINO int8 / int4 (NNCF weight compression)

INFERENCE_PRECISION_HINT does not quantize weights; it only picks the execution kernel. int8/int4 need NNCF compress_weights applied to the IR (see openvino_quantization.md for why this was not in the first round). Weight-only compression (no activation calibration), on the saved f32 IRs:

mode file size pp512 tg128 ppl512 argmax vs f32 top-1 factual
f32 (ref) 16.6 GB 125.6 2.22 28.82 1.000 Paris / Jupiter / oxygen
int8 asym 8.3 GB 219.2 4.70 29.11 0.951 Paris / Jupiter / oxygen
int8 sym 8.3 GB 226.7 - 29.34 0.941 Paris / Jupiter / oxygen
int4 asym 4.9 GB 223.7 5.20 32.68 0.703 Paris / Jupiter / oxygen
int4 sym 4.9 GB 222.5 - 42.95 0.701 Paris / Jupiter / oxygen

Answer-quality check: greedy top-1 for The capital of France is / The largest planet ... / Water is made of hydrogen and is Paris / Jupiter / oxygen for every variant, including int4. So int4 keeps short factual answers but loses a lot of distributional fidelity (PPL +3.9, and 30% of argmax positions change); int8 is close to f32 (PPL +0.3, ~95% argmax). int8/int4 roughly double decode throughput over f32 (bandwidth-bound), to 4.7-5.2 tok/s, still below llama.cpp Q4_K_M (13.39).

Requirements: the local modeling_openpangu_dense.py must be patched (sliding_window per layer type, sink-mask padding) and the published code imports torch_npu (a stub package is enough).

OpenVINO GenAI (LLMPipeline)

openvino_genai.LLMPipeline was tested as the high-level alternative to the low-level ov.Core().compile_model path. With OpenVINO GenAI 2026.4.1 it cannot run this model:

input to LLMPipeline result
HF checkpoint directory Could not find a model in the directory (GenAI needs an IR)
Q4_K_M GGUF [load_gguf] gguf_tensor_to_f16 failed
manual 64-token prefill IR + tokenizer loads in 3.9 s; generate() fails: Port for tensor name attention_mask was not found
optimum-cli export openvino --trust-remote-code custom or unsupported architecture

The GGUF failure is a format limitation, not memory. The published Q4_K_M file mixes 54 Q5_0 tensors with Q4_K/Q6_K/f32, and the GenAI reader only dequantizes Q4_0/Q4_1/Q4_K/Q8_0/Q6_K; a normalized file would still hit the architecture check, which accepts only llama, qwen2, qwen3 (this file says openpangu-r).

The IR used by the low-level OpenVINO path is also not GenAI-compatible: LLMPipeline expects one stateful graph with input_ids, attention_mask, position_ids, beam_idx and KV-cache variables, while the Pangu graphs are stateless prefill/decode IRs with explicit KV and WindowBuffer inputs/outputs. Working GenAI support would require a stateful, dynamic-shape re-export of the custom architecture.

Test: openvino/genai_api_test.py --model <HF dir> --gguf <Q4_K_M.gguf> --ir-dir /tmp/pangu_genai --make-ir 64. The IR build needs einops and sentencepiece. Raw log: logs/genai_api.log.

oneDNN implementation

New files: ggml/src/ggml-cpu/onednn.cpp, onednn.h; a hook in ggml_compute_forward_mul_mat (before the tiled path) in ggml-cpu.c; GGML_CPU_ONEDNN option in ggml/CMakeLists.txt and ggml/src/ggml-cpu/CMakeLists.txt. Thread 0 calls oneDNN, the others wait at a barrier. Only 2D quantized-weight x 2D activation MUL_MAT (projections and lm_head) are handled; batched attention matmuls stay on the ggml path.

Weight conversion is done once at load (cached per tensor): bf16 round-to- nearest-even, int8 per-column symmetric s8 (tag::ba), int4 per-column s4 packed two per byte (tag::ba). Activations use per-tensor asymmetric u8. GGML_ONEDNN_WEIGHT_TYPE selects bf16 (default) / int8 / int4; GGML_ONEDNN_DEBUG prints conversion and timing info.

Build: ./build.sh /path/to/llama.cpp build-onednn with GGML_NATIVE=OFF, GGML_AMX_*=OFF, GGML_CPU_REPACK=OFF. With the native build the Q4_K/Q6_K projections are repacked into the AMX extra buffer and executed by repack.cpp, so ggml_compute_forward_mul_mat (and the hook) is never reached.

Known limitations

  • int4 has no optimized CPU kernel in oneDNN 3.7 (ref_int8, 0.8 GFLOPS).
  • No batched (4D) matmul, F32 weight matmul, or non-contiguous input support.
  • Per-tensor activation quantization: greedy output matches the baseline for the first tokens, then diverges.
  • First-call weight conversion costs ~15 s (bf16) / ~45 s (int8).

Files

  • results.csv - the table above
  • build.sh, bench.sh - build / benchmark helpers
  • onednn.cpp, onednn.h, onednn.patch - oneDNN backend
  • openvino/ - benchmark and correctness programs:
    • convert_ir.py - trace + save prefill IR
    • ov_decode.py - trace + save decode IR, then benchmark decode
    • bench_ov_ir.py, bench_ov_prefill.py - prefill benchmark
    • ov_compress.py - NNCF weight compression (int8/int4) of an IR
    • ov_quality.py - PPL + argmax agreement of an IR
    • ov_answer.py - qualitative top-5 next-token check
    • ov_prec_test.py - f32 vs bf16 hint accuracy
    • ov_dec_correct.py - decode-step correctness vs HF
    • ov_layer_iso.py, ov_layer_internals.py, ov_attn_iso.py, ov_ops_iso.py, ov_hidden_k.py - isolation probes used to debug
    • ov_tiny_llama.py, ov_tiny_layer.py, ov_tiny_test.py - control models
    • bench_hf.py - HF eager / oneDNN-backed PyTorch baseline
    • genai_api_test.py - OpenVINO GenAI LLMPipeline probes (HF/GGUF/IR); --make-ir N builds the N-token prefill IR + tokenizer it uses
  • logs/ - raw benchmark and accuracy records:
    • q4km_llamacpp_onednn.log - llama-bench -r 5 for llama.cpp + oneDNN
    • openvino_prefill_{bf16,f32}.log, openvino_decode_{bf16,f32}.log
    • openvino_ir_convert.log - IR conversion timings
    • openvino_accuracy.log - correctness checks
    • quant/ - NNCF compression logs, PPL/argmax results, decode speed, answer-quality log for each compressed IR
    • genai_api.log, genai_api.json - OpenVINO GenAI LLMPipeline probe output (HF / GGUF / IR failures)
  • openvino_quantization.md - why int8/int4 was missing initially