# openPangu-R-7B-2512 CPU backend comparison: llama.cpp vs oneDNN vs OpenVINO > Part of [HelloSun/Experiment_1](https://huggingface.co/HelloSun/Experiment_1): > the GGUF, the llama.cpp patch and the quick start are in the repository root > ([`../README.md`](../README.md)). Paths below are relative to `onednn/`. Three CPU inference backends on **openPangu-R-7B-2512** (27 layers, hidden 4096, 8.29 B params), benchmarked steady-state (`pp512` / `tg128`, `llama-bench -r 5` or median of 128 decode steps). Headline config: **Q4_K_M** for the llama.cpp / oneDNN paths, and the OpenVINO IR built from the fp32 HF weights. ## Environment | item | value | |---|---| | CPU | Intel Xeon Platinum 8559C, 192 threads, 2 NUMA, AMX BF16/INT8, AVX512 | | container | 16 CPUs (`cpu.max=1600`), 80 GiB memory guard, no GPU | | OS | Debian 13 | | oneDNN | 3.7.2 (`libdnnl-dev`) | | OpenVINO | 2026.4.1 (runtime), transformers 4.53.2 | | OpenVINO GenAI | 2026.4.1 (`openvino-genai`) | | llama.cpp | v0.7.0, commit `620d2225cc436ab0b001e3200e590e3b286ef9ab` + openPangu-R patch | | model | openPangu-R-7B-2512 Q4_K_M, 4.72 GiB, 8.29 B params | ## Results (Q4_K_M) `ppl512` = perplexity over a fixed 512-token (EN+ZH) text; `argmax_agree` = per-position argmax agreement with the OpenVINO f32 graph. Both quality columns only apply to the OpenVINO rows (llama.cpp / oneDNN perf was measured with `llama-bench`). | backend | weights | pp512 tok/s | tg128 tok/s | ppl512 | argmax | threads | |---|---|---:|---:|---:|---:|---:| | llama.cpp (plain AVX512) | Q4_K_M | **180.50** | **13.39** | - | - | 16 | | llama.cpp (default AMX+repack) | Q4_K_M | 112.77 | 12.19 | - | - | 16 | | oneDNN | int8 | 90.09 | 7.67 | - | - | 1 (+OMP 16) | | oneDNN | bf16 | 75.52 | 4.79 | - | - | 1 (+OMP 16) | | OpenVINO | f32 | 125.60 | 2.22 | **28.82** | 1.000 | 16 | | OpenVINO | bf16-hint | 209.43 | 3.93 | 28.85 | 0.975 | 16 | | OpenVINO | int8-asym | 219.17 | 4.70 | 29.11 | 0.951 | 16 | | OpenVINO | int8-sym | 226.68 | - | 29.34 | 0.941 | 16 | | OpenVINO | int4-asym | 223.65 | 5.20 | 32.68 | 0.703 | 16 | | OpenVINO | int4-sym | 222.48 | - | 42.95 | 0.701 | 16 | `llama-bench -r 5`, 16 CPUs. OpenVINO `tg128` is the median of 128 decode steps (fixed 640-slot KV cache). This is a shared host: the OpenVINO f32/bf16 prefill ranged between ~126 and ~366 tok/s across sessions, so treat the OpenVINO prefill column as order-of-magnitude, not exact. Notes: - The llama.cpp Q4_K_M build is the fastest overall on this 16-CPU container: `13.39 tg128`, `180.50 pp512`. - OpenVINO has by far the fastest prefill (`366 pp512` at `bf16`, `128` at `f32`) but the slowest decode here, because its decode graph always runs attention over the full 640-slot cache and moves all KV-cache tensors across the host/plugin boundary each step. - oneDNN loses on both axes. Decode is bandwidth-bound: bf16 weights are 16.6 GB/token vs ~5 GB/token for Q4_K; int8 is 8.3 GB/token. The AMX int8 matmul kernel itself is fast (~7.1 TFLOPS at M=512, 4096^3 in the microbenchmark); the loss is the weight format and per-call activation quantization, not the kernel. ## OpenVINO The published OpenVINO GenAI GGUF reader and `optimum-cli export openvino` cannot handle this model (`openpangu-r` architecture, custom ops), and `torch.compile(backend="openvino")` fails on the model's symbolic shapes. The working route is a direct `ov.convert_model` (TorchScript trace) of the HF `PanguEmbeddedForCausalLM`: - prefill graph: `convert_ir.py` -> `pangu_f32_512.xml/bin` - decode graph: `ov_decode.py --dur convert` (KV cache + the two `WindowBuffer` states are threaded through the graph as explicit inputs/outputs, so the IR stays stateless). Convert with fp32 weights, then compile to bf16 with `INFERENCE_PRECISION_HINT`. Accuracy (fp32 hint vs HF eager, 512-token prefill): logits maxdiff `0.0004`, 100% argmax agreement; decode-step argmax matches HF. The **bf16 hint drops accuracy** (argmax agreement 75%) because the model is numerically sensitive (error amplifies ~1.3x per layer), so bf16 is a speed/accuracy trade-off. ### OpenVINO int8 / int4 (NNCF weight compression) `INFERENCE_PRECISION_HINT` does **not** quantize weights; it only picks the execution kernel. int8/int4 need NNCF `compress_weights` applied to the IR (see `openvino_quantization.md` for why this was not in the first round). Weight-only compression (no activation calibration), on the saved f32 IRs: | mode | file size | pp512 | tg128 | ppl512 | argmax vs f32 | top-1 factual | |---|---:|---:|---:|---:|---:|---| | f32 (ref) | 16.6 GB | 125.6 | 2.22 | 28.82 | 1.000 | Paris / Jupiter / oxygen | | int8 asym | 8.3 GB | 219.2 | 4.70 | 29.11 | 0.951 | Paris / Jupiter / oxygen | | int8 sym | 8.3 GB | 226.7 | - | 29.34 | 0.941 | Paris / Jupiter / oxygen | | int4 asym | 4.9 GB | 223.7 | 5.20 | 32.68 | 0.703 | Paris / Jupiter / oxygen | | int4 sym | 4.9 GB | 222.5 | - | 42.95 | 0.701 | Paris / Jupiter / oxygen | Answer-quality check: greedy top-1 for `The capital of France is` / `The largest planet ...` / `Water is made of hydrogen and` is `Paris` / `Jupiter` / `oxygen` for **every** variant, including int4. So int4 keeps short factual answers but loses a lot of distributional fidelity (PPL +3.9, and 30% of argmax positions change); int8 is close to f32 (PPL +0.3, ~95% argmax). int8/int4 roughly double decode throughput over f32 (bandwidth-bound), to 4.7-5.2 tok/s, still below llama.cpp Q4_K_M (13.39). Requirements: the local `modeling_openpangu_dense.py` must be patched (`sliding_window` per layer type, sink-mask padding) and the published code imports `torch_npu` (a stub package is enough). ### OpenVINO GenAI (LLMPipeline) `openvino_genai.LLMPipeline` was tested as the high-level alternative to the low-level `ov.Core().compile_model` path. With OpenVINO GenAI 2026.4.1 it cannot run this model: | input to `LLMPipeline` | result | |---|---| | HF checkpoint directory | `Could not find a model in the directory` (GenAI needs an IR) | | Q4_K_M GGUF | `[load_gguf] gguf_tensor_to_f16 failed` | | manual 64-token prefill IR + tokenizer | loads in 3.9 s; `generate()` fails: `Port for tensor name attention_mask was not found` | | `optimum-cli export openvino --trust-remote-code` | `custom or unsupported architecture` | The GGUF failure is a format limitation, not memory. The published Q4_K_M file mixes 54 `Q5_0` tensors with `Q4_K`/`Q6_K`/f32, and the GenAI reader only dequantizes `Q4_0/Q4_1/Q4_K/Q8_0/Q6_K`; a normalized file would still hit the architecture check, which accepts only `llama`, `qwen2`, `qwen3` (this file says `openpangu-r`). The IR used by the low-level OpenVINO path is also not GenAI-compatible: `LLMPipeline` expects one stateful graph with `input_ids`, `attention_mask`, `position_ids`, `beam_idx` and KV-cache variables, while the Pangu graphs are stateless prefill/decode IRs with explicit KV and `WindowBuffer` inputs/outputs. Working GenAI support would require a stateful, dynamic-shape re-export of the custom architecture. Test: `openvino/genai_api_test.py --model --gguf --ir-dir /tmp/pangu_genai --make-ir 64`. The IR build needs `einops` and `sentencepiece`. Raw log: `logs/genai_api.log`. ## oneDNN implementation New files: `ggml/src/ggml-cpu/onednn.cpp`, `onednn.h`; a hook in `ggml_compute_forward_mul_mat` (before the tiled path) in `ggml-cpu.c`; `GGML_CPU_ONEDNN` option in `ggml/CMakeLists.txt` and `ggml/src/ggml-cpu/CMakeLists.txt`. Thread 0 calls oneDNN, the others wait at a barrier. Only 2D quantized-weight x 2D activation MUL_MAT (projections and lm_head) are handled; batched attention matmuls stay on the ggml path. Weight conversion is done once at load (cached per tensor): bf16 round-to- nearest-even, int8 per-column symmetric `s8` (`tag::ba`), int4 per-column `s4` packed two per byte (`tag::ba`). Activations use per-tensor asymmetric `u8`. `GGML_ONEDNN_WEIGHT_TYPE` selects `bf16` (default) / `int8` / `int4`; `GGML_ONEDNN_DEBUG` prints conversion and timing info. Build: `./build.sh /path/to/llama.cpp build-onednn` with `GGML_NATIVE=OFF`, `GGML_AMX_*=OFF`, `GGML_CPU_REPACK=OFF`. With the native build the Q4_K/Q6_K projections are repacked into the AMX extra buffer and executed by `repack.cpp`, so `ggml_compute_forward_mul_mat` (and the hook) is never reached. ### Known limitations - int4 has no optimized CPU kernel in oneDNN 3.7 (`ref_int8`, 0.8 GFLOPS). - No batched (4D) matmul, F32 weight matmul, or non-contiguous input support. - Per-tensor activation quantization: greedy output matches the baseline for the first tokens, then diverges. - First-call weight conversion costs ~15 s (bf16) / ~45 s (int8). ## Files - `results.csv` - the table above - `build.sh`, `bench.sh` - build / benchmark helpers - `onednn.cpp`, `onednn.h`, `onednn.patch` - oneDNN backend - `openvino/` - benchmark and correctness programs: - `convert_ir.py` - trace + save prefill IR - `ov_decode.py` - trace + save decode IR, then benchmark decode - `bench_ov_ir.py`, `bench_ov_prefill.py` - prefill benchmark - `ov_compress.py` - NNCF weight compression (int8/int4) of an IR - `ov_quality.py` - PPL + argmax agreement of an IR - `ov_answer.py` - qualitative top-5 next-token check - `ov_prec_test.py` - f32 vs bf16 hint accuracy - `ov_dec_correct.py` - decode-step correctness vs HF - `ov_layer_iso.py`, `ov_layer_internals.py`, `ov_attn_iso.py`, `ov_ops_iso.py`, `ov_hidden_k.py` - isolation probes used to debug - `ov_tiny_llama.py`, `ov_tiny_layer.py`, `ov_tiny_test.py` - control models - `bench_hf.py` - HF eager / oneDNN-backed PyTorch baseline - `genai_api_test.py` - OpenVINO GenAI `LLMPipeline` probes (HF/GGUF/IR); `--make-ir N` builds the N-token prefill IR + tokenizer it uses - `logs/` - raw benchmark and accuracy records: - `q4km_llamacpp_onednn.log` - `llama-bench -r 5` for llama.cpp + oneDNN - `openvino_prefill_{bf16,f32}.log`, `openvino_decode_{bf16,f32}.log` - `openvino_ir_convert.log` - IR conversion timings - `openvino_accuracy.log` - correctness checks - `quant/` - NNCF compression logs, PPL/argmax results, decode speed, answer-quality log for each compressed IR - `genai_api.log`, `genai_api.json` - OpenVINO GenAI `LLMPipeline` probe output (HF / GGUF / IR failures) - `openvino_quantization.md` - why int8/int4 was missing initially