GGUF
conversational
Experiment_1 / onednn /openvino_quantization.md
HelloSun's picture
Integrate oneDNN/OpenVINO CPU backend study; merge docs; point build script at Experiment_1
da11688 verified
|
Raw History Blame Contribute Delete
2.87 kB
# Why OpenVINO int8 / int4 was not in the first round
Short answer: the OpenVINO numbers I published came from a route that only
selects the *execution precision* of the graph, it does not quantize the
weights. On top of that, the automatic int8/int4 tooling refuses this model, so
integer support had to be added by hand afterwards (done now, see
`results.csv`).
## 1. `INFERENCE_PRECISION_HINT` is not quantization
The working OpenVINO path here is `ov.convert_model` (TorchScript trace) of the
HF model into an IR, then `compile_model(..., {"INFERENCE_PRECISION_HINT":
"f32"|"bf16"})`. That hint only picks which *kernel* runs (f32 SIMD vs AMX bf16);
the stored weights stay f32. To get int8/int4 you must convert the IR weights to
low precision with NNCF weight compression, which is a separate step.
## 2. The automatic LLM pipelines reject the architecture
- `optimum-cli export openvino` / `optimum-intel`: no support for
`PanguEmbedded`.
- `openvino-genai` GGUF reader: our GGUF declares `general.architecture =
openpangu-r`, and the reader also fails `gguf_tensor_to_f16` on the Q5_0
tensors. Dead end.
- NNCF's high-level LLM compression is driven by model-type detection; unknown
architectures are not handled automatically.
- `torch.compile(backend="openvino")` fails on the model's symbolic shapes
(torch 2.13 guards `IndexError`; torch 2.8 vmap / `SymIntArrayRef ... concrete
integers`).
So there is no one-flag route: int8/int4 had to be applied manually with
`nncf.compress_weights` on the IR we already produced.
## 3. Practical constraints at the time
- The OpenVINO work was added after the oneDNN vs llama.cpp comparison was
already finished, and the obvious int8/int4 story for this project was the
oneDNN side (int8 / int4 weight types). OpenVINO was introduced to answer
"can a vendor graph compiler beat llama.cpp", initially at f32/bf16.
- Memory: the 80 GiB guard killed `convert_model + compile_model` in one process
once (oneDNN-style f32 compile spiked to ~64 GiB per process). Conversion had
to be split from compilation, and IRs had to be saved to disk first. Adding
NNCF compression on top needed its own pass.
- Accuracy risk: this model is numerically sensitive (~1.3x error growth per
layer), so it was not obvious that int4 would be usable. It turned out int8 is
fine and int4 asym is borderline (see below).
## What was run (this round)
Weight-only compression with NNCF 3.4.0 (`nncf.compress_weights`), applied to
the saved f32 IRs (prefill graph `pangu_f32_512.xml`, decode graph
`pangu_dec_512.xml`), no activation calibration. Modes: `INT8_SYM`,
`INT8_ASYM`, `INT4_SYM`, `INT4_ASYM` (int4 with `group_size=128`, `ratio=1.0`).
Compression cost ~20 s (int8) / ~30 s (int4), peak RSS 20-24 GiB, and is load
time (not part of pp/tg).
See `results.csv` and `logs/` for the numbers and raw records.