GGUF
conversational
Experiment_1 / onednn /README.md
HelloSun's picture
Integrate oneDNN/OpenVINO CPU backend study; merge docs; point build script at Experiment_1
da11688 verified
|
Raw History Blame Contribute Delete
10.4 kB
# openPangu-R-7B-2512 CPU backend comparison: llama.cpp vs oneDNN vs OpenVINO
> Part of [HelloSun/Experiment_1](https://huggingface.co/HelloSun/Experiment_1):
> the GGUF, the llama.cpp patch and the quick start are in the repository root
> ([`../README.md`](../README.md)). Paths below are relative to `onednn/`.
Three CPU inference backends on **openPangu-R-7B-2512** (27 layers, hidden
4096, 8.29 B params), benchmarked steady-state (`pp512` / `tg128`, `llama-bench
-r 5` or median of 128 decode steps). Headline config: **Q4_K_M** for the
llama.cpp / oneDNN paths, and the OpenVINO IR built from the fp32 HF weights.
## Environment
| item | value |
|---|---|
| CPU | Intel Xeon Platinum 8559C, 192 threads, 2 NUMA, AMX BF16/INT8, AVX512 |
| container | 16 CPUs (`cpu.max=1600`), 80 GiB memory guard, no GPU |
| OS | Debian 13 |
| oneDNN | 3.7.2 (`libdnnl-dev`) |
| OpenVINO | 2026.4.1 (runtime), transformers 4.53.2 |
| OpenVINO GenAI | 2026.4.1 (`openvino-genai`) |
| llama.cpp | v0.7.0, commit `620d2225cc436ab0b001e3200e590e3b286ef9ab` + openPangu-R patch |
| model | openPangu-R-7B-2512 Q4_K_M, 4.72 GiB, 8.29 B params |
## Results (Q4_K_M)
`ppl512` = perplexity over a fixed 512-token (EN+ZH) text; `argmax_agree` =
per-position argmax agreement with the OpenVINO f32 graph. Both quality columns
only apply to the OpenVINO rows (llama.cpp / oneDNN perf was measured with
`llama-bench`).
| backend | weights | pp512 tok/s | tg128 tok/s | ppl512 | argmax | threads |
|---|---|---:|---:|---:|---:|---:|
| llama.cpp (plain AVX512) | Q4_K_M | **180.50** | **13.39** | - | - | 16 |
| llama.cpp (default AMX+repack) | Q4_K_M | 112.77 | 12.19 | - | - | 16 |
| oneDNN | int8 | 90.09 | 7.67 | - | - | 1 (+OMP 16) |
| oneDNN | bf16 | 75.52 | 4.79 | - | - | 1 (+OMP 16) |
| OpenVINO | f32 | 125.60 | 2.22 | **28.82** | 1.000 | 16 |
| OpenVINO | bf16-hint | 209.43 | 3.93 | 28.85 | 0.975 | 16 |
| OpenVINO | int8-asym | 219.17 | 4.70 | 29.11 | 0.951 | 16 |
| OpenVINO | int8-sym | 226.68 | - | 29.34 | 0.941 | 16 |
| OpenVINO | int4-asym | 223.65 | 5.20 | 32.68 | 0.703 | 16 |
| OpenVINO | int4-sym | 222.48 | - | 42.95 | 0.701 | 16 |
`llama-bench -r 5`, 16 CPUs. OpenVINO `tg128` is the median of 128 decode
steps (fixed 640-slot KV cache). This is a shared host: the OpenVINO f32/bf16
prefill ranged between ~126 and ~366 tok/s across sessions, so treat the
OpenVINO prefill column as order-of-magnitude, not exact.
Notes:
- The llama.cpp Q4_K_M build is the fastest overall on this 16-CPU container:
`13.39 tg128`, `180.50 pp512`.
- OpenVINO has by far the fastest prefill (`366 pp512` at `bf16`, `128` at
`f32`) but the slowest decode here, because its decode graph always runs
attention over the full 640-slot cache and moves all KV-cache tensors across
the host/plugin boundary each step.
- oneDNN loses on both axes. Decode is bandwidth-bound: bf16 weights are
16.6 GB/token vs ~5 GB/token for Q4_K; int8 is 8.3 GB/token. The AMX int8
matmul kernel itself is fast (~7.1 TFLOPS at M=512, 4096^3 in the
microbenchmark); the loss is the weight format and per-call activation
quantization, not the kernel.
## OpenVINO
The published OpenVINO GenAI GGUF reader and `optimum-cli export openvino`
cannot handle this model (`openpangu-r` architecture, custom ops), and
`torch.compile(backend="openvino")` fails on the model's symbolic shapes. The
working route is a direct `ov.convert_model` (TorchScript trace) of the HF
`PanguEmbeddedForCausalLM`:
- prefill graph: `convert_ir.py` -> `pangu_f32_512.xml/bin`
- decode graph: `ov_decode.py --dur convert` (KV cache + the two `WindowBuffer`
states are threaded through the graph as explicit inputs/outputs, so the IR
stays stateless). Convert with fp32 weights, then compile to bf16 with
`INFERENCE_PRECISION_HINT`.
Accuracy (fp32 hint vs HF eager, 512-token prefill): logits maxdiff `0.0004`,
100% argmax agreement; decode-step argmax matches HF. The **bf16 hint drops
accuracy** (argmax agreement 75%) because the model is numerically sensitive
(error amplifies ~1.3x per layer), so bf16 is a speed/accuracy trade-off.
### OpenVINO int8 / int4 (NNCF weight compression)
`INFERENCE_PRECISION_HINT` does **not** quantize weights; it only picks the
execution kernel. int8/int4 need NNCF `compress_weights` applied to the IR
(see `openvino_quantization.md` for why this was not in the first round).
Weight-only compression (no activation calibration), on the saved f32 IRs:
| mode | file size | pp512 | tg128 | ppl512 | argmax vs f32 | top-1 factual |
|---|---:|---:|---:|---:|---:|---|
| f32 (ref) | 16.6 GB | 125.6 | 2.22 | 28.82 | 1.000 | Paris / Jupiter / oxygen |
| int8 asym | 8.3 GB | 219.2 | 4.70 | 29.11 | 0.951 | Paris / Jupiter / oxygen |
| int8 sym | 8.3 GB | 226.7 | - | 29.34 | 0.941 | Paris / Jupiter / oxygen |
| int4 asym | 4.9 GB | 223.7 | 5.20 | 32.68 | 0.703 | Paris / Jupiter / oxygen |
| int4 sym | 4.9 GB | 222.5 | - | 42.95 | 0.701 | Paris / Jupiter / oxygen |
Answer-quality check: greedy top-1 for `The capital of France is` / `The largest
planet ...` / `Water is made of hydrogen and` is `Paris` / `Jupiter` / `oxygen`
for **every** variant, including int4. So int4 keeps short factual answers but
loses a lot of distributional fidelity (PPL +3.9, and 30% of argmax positions
change); int8 is close to f32 (PPL +0.3, ~95% argmax). int8/int4 roughly double
decode throughput over f32 (bandwidth-bound), to 4.7-5.2 tok/s, still below
llama.cpp Q4_K_M (13.39).
Requirements: the local `modeling_openpangu_dense.py` must be patched
(`sliding_window` per layer type, sink-mask padding) and the published code
imports `torch_npu` (a stub package is enough).
### OpenVINO GenAI (LLMPipeline)
`openvino_genai.LLMPipeline` was tested as the high-level alternative to the
low-level `ov.Core().compile_model` path. With OpenVINO GenAI 2026.4.1 it cannot
run this model:
| input to `LLMPipeline` | result |
|---|---|
| HF checkpoint directory | `Could not find a model in the directory` (GenAI needs an IR) |
| Q4_K_M GGUF | `[load_gguf] gguf_tensor_to_f16 failed` |
| manual 64-token prefill IR + tokenizer | loads in 3.9 s; `generate()` fails: `Port for tensor name attention_mask was not found` |
| `optimum-cli export openvino --trust-remote-code` | `custom or unsupported architecture` |
The GGUF failure is a format limitation, not memory. The published Q4_K_M file
mixes 54 `Q5_0` tensors with `Q4_K`/`Q6_K`/f32, and the GenAI reader only
dequantizes `Q4_0/Q4_1/Q4_K/Q8_0/Q6_K`; a normalized file would still hit the
architecture check, which accepts only `llama`, `qwen2`, `qwen3` (this file says
`openpangu-r`).
The IR used by the low-level OpenVINO path is also not GenAI-compatible:
`LLMPipeline` expects one stateful graph with `input_ids`, `attention_mask`,
`position_ids`, `beam_idx` and KV-cache variables, while the Pangu graphs are
stateless prefill/decode IRs with explicit KV and `WindowBuffer`
inputs/outputs. Working GenAI support would require a stateful, dynamic-shape
re-export of the custom architecture.
Test: `openvino/genai_api_test.py --model <HF dir> --gguf <Q4_K_M.gguf>
--ir-dir /tmp/pangu_genai --make-ir 64`.
The IR build needs `einops` and `sentencepiece`. Raw log: `logs/genai_api.log`.
## oneDNN implementation
New files: `ggml/src/ggml-cpu/onednn.cpp`, `onednn.h`; a hook in
`ggml_compute_forward_mul_mat` (before the tiled path) in `ggml-cpu.c`;
`GGML_CPU_ONEDNN` option in `ggml/CMakeLists.txt` and
`ggml/src/ggml-cpu/CMakeLists.txt`. Thread 0 calls oneDNN, the others wait at a
barrier. Only 2D quantized-weight x 2D activation MUL_MAT (projections and
lm_head) are handled; batched attention matmuls stay on the ggml path.
Weight conversion is done once at load (cached per tensor): bf16 round-to-
nearest-even, int8 per-column symmetric `s8` (`tag::ba`), int4 per-column `s4`
packed two per byte (`tag::ba`). Activations use per-tensor asymmetric `u8`.
`GGML_ONEDNN_WEIGHT_TYPE` selects `bf16` (default) / `int8` / `int4`;
`GGML_ONEDNN_DEBUG` prints conversion and timing info.
Build: `./build.sh /path/to/llama.cpp build-onednn` with `GGML_NATIVE=OFF`,
`GGML_AMX_*=OFF`, `GGML_CPU_REPACK=OFF`. With the native build the Q4_K/Q6_K
projections are repacked into the AMX extra buffer and executed by
`repack.cpp`, so `ggml_compute_forward_mul_mat` (and the hook) is never
reached.
### Known limitations
- int4 has no optimized CPU kernel in oneDNN 3.7 (`ref_int8`, 0.8 GFLOPS).
- No batched (4D) matmul, F32 weight matmul, or non-contiguous input support.
- Per-tensor activation quantization: greedy output matches the baseline for
the first tokens, then diverges.
- First-call weight conversion costs ~15 s (bf16) / ~45 s (int8).
## Files
- `results.csv` - the table above
- `build.sh`, `bench.sh` - build / benchmark helpers
- `onednn.cpp`, `onednn.h`, `onednn.patch` - oneDNN backend
- `openvino/` - benchmark and correctness programs:
- `convert_ir.py` - trace + save prefill IR
- `ov_decode.py` - trace + save decode IR, then benchmark decode
- `bench_ov_ir.py`, `bench_ov_prefill.py` - prefill benchmark
- `ov_compress.py` - NNCF weight compression (int8/int4) of an IR
- `ov_quality.py` - PPL + argmax agreement of an IR
- `ov_answer.py` - qualitative top-5 next-token check
- `ov_prec_test.py` - f32 vs bf16 hint accuracy
- `ov_dec_correct.py` - decode-step correctness vs HF
- `ov_layer_iso.py`, `ov_layer_internals.py`, `ov_attn_iso.py`,
`ov_ops_iso.py`, `ov_hidden_k.py` - isolation probes used to debug
- `ov_tiny_llama.py`, `ov_tiny_layer.py`, `ov_tiny_test.py` - control models
- `bench_hf.py` - HF eager / oneDNN-backed PyTorch baseline
- `genai_api_test.py` - OpenVINO GenAI `LLMPipeline` probes (HF/GGUF/IR);
`--make-ir N` builds the N-token prefill IR + tokenizer it uses
- `logs/` - raw benchmark and accuracy records:
- `q4km_llamacpp_onednn.log` - `llama-bench -r 5` for llama.cpp + oneDNN
- `openvino_prefill_{bf16,f32}.log`, `openvino_decode_{bf16,f32}.log`
- `openvino_ir_convert.log` - IR conversion timings
- `openvino_accuracy.log` - correctness checks
- `quant/` - NNCF compression logs, PPL/argmax results, decode speed,
answer-quality log for each compressed IR
- `genai_api.log`, `genai_api.json` - OpenVINO GenAI `LLMPipeline` probe
output (HF / GGUF / IR failures)
- `openvino_quantization.md` - why int8/int4 was missing initially