GGUF
conversational
Experiment_1 / onednn /openvino_quantization.md
HelloSun's picture
Integrate oneDNN/OpenVINO CPU backend study; merge docs; point build script at Experiment_1
da11688 verified
|
Raw History Blame Contribute Delete
2.87 kB

Why OpenVINO int8 / int4 was not in the first round

Short answer: the OpenVINO numbers I published came from a route that only selects the execution precision of the graph, it does not quantize the weights. On top of that, the automatic int8/int4 tooling refuses this model, so integer support had to be added by hand afterwards (done now, see results.csv).

1. INFERENCE_PRECISION_HINT is not quantization

The working OpenVINO path here is ov.convert_model (TorchScript trace) of the HF model into an IR, then compile_model(..., {"INFERENCE_PRECISION_HINT": "f32"|"bf16"}). That hint only picks which kernel runs (f32 SIMD vs AMX bf16); the stored weights stay f32. To get int8/int4 you must convert the IR weights to low precision with NNCF weight compression, which is a separate step.

2. The automatic LLM pipelines reject the architecture

  • optimum-cli export openvino / optimum-intel: no support for PanguEmbedded.
  • openvino-genai GGUF reader: our GGUF declares general.architecture = openpangu-r, and the reader also fails gguf_tensor_to_f16 on the Q5_0 tensors. Dead end.
  • NNCF's high-level LLM compression is driven by model-type detection; unknown architectures are not handled automatically.
  • torch.compile(backend="openvino") fails on the model's symbolic shapes (torch 2.13 guards IndexError; torch 2.8 vmap / SymIntArrayRef ... concrete integers).

So there is no one-flag route: int8/int4 had to be applied manually with nncf.compress_weights on the IR we already produced.

3. Practical constraints at the time

  • The OpenVINO work was added after the oneDNN vs llama.cpp comparison was already finished, and the obvious int8/int4 story for this project was the oneDNN side (int8 / int4 weight types). OpenVINO was introduced to answer "can a vendor graph compiler beat llama.cpp", initially at f32/bf16.
  • Memory: the 80 GiB guard killed convert_model + compile_model in one process once (oneDNN-style f32 compile spiked to ~64 GiB per process). Conversion had to be split from compilation, and IRs had to be saved to disk first. Adding NNCF compression on top needed its own pass.
  • Accuracy risk: this model is numerically sensitive (~1.3x error growth per layer), so it was not obvious that int4 would be usable. It turned out int8 is fine and int4 asym is borderline (see below).

What was run (this round)

Weight-only compression with NNCF 3.4.0 (nncf.compress_weights), applied to the saved f32 IRs (prefill graph pangu_f32_512.xml, decode graph pangu_dec_512.xml), no activation calibration. Modes: INT8_SYM, INT8_ASYM, INT4_SYM, INT4_ASYM (int4 with group_size=128, ratio=1.0). Compression cost ~20 s (int8) / ~30 s (int4), peak RSS 20-24 GiB, and is load time (not part of pp/tg).

See results.csv and logs/ for the numbers and raw records.