# Why OpenVINO int8 / int4 was not in the first round Short answer: the OpenVINO numbers I published came from a route that only selects the *execution precision* of the graph, it does not quantize the weights. On top of that, the automatic int8/int4 tooling refuses this model, so integer support had to be added by hand afterwards (done now, see `results.csv`). ## 1. `INFERENCE_PRECISION_HINT` is not quantization The working OpenVINO path here is `ov.convert_model` (TorchScript trace) of the HF model into an IR, then `compile_model(..., {"INFERENCE_PRECISION_HINT": "f32"|"bf16"})`. That hint only picks which *kernel* runs (f32 SIMD vs AMX bf16); the stored weights stay f32. To get int8/int4 you must convert the IR weights to low precision with NNCF weight compression, which is a separate step. ## 2. The automatic LLM pipelines reject the architecture - `optimum-cli export openvino` / `optimum-intel`: no support for `PanguEmbedded`. - `openvino-genai` GGUF reader: our GGUF declares `general.architecture = openpangu-r`, and the reader also fails `gguf_tensor_to_f16` on the Q5_0 tensors. Dead end. - NNCF's high-level LLM compression is driven by model-type detection; unknown architectures are not handled automatically. - `torch.compile(backend="openvino")` fails on the model's symbolic shapes (torch 2.13 guards `IndexError`; torch 2.8 vmap / `SymIntArrayRef ... concrete integers`). So there is no one-flag route: int8/int4 had to be applied manually with `nncf.compress_weights` on the IR we already produced. ## 3. Practical constraints at the time - The OpenVINO work was added after the oneDNN vs llama.cpp comparison was already finished, and the obvious int8/int4 story for this project was the oneDNN side (int8 / int4 weight types). OpenVINO was introduced to answer "can a vendor graph compiler beat llama.cpp", initially at f32/bf16. - Memory: the 80 GiB guard killed `convert_model + compile_model` in one process once (oneDNN-style f32 compile spiked to ~64 GiB per process). Conversion had to be split from compilation, and IRs had to be saved to disk first. Adding NNCF compression on top needed its own pass. - Accuracy risk: this model is numerically sensitive (~1.3x error growth per layer), so it was not obvious that int4 would be usable. It turned out int8 is fine and int4 asym is borderline (see below). ## What was run (this round) Weight-only compression with NNCF 3.4.0 (`nncf.compress_weights`), applied to the saved f32 IRs (prefill graph `pangu_f32_512.xml`, decode graph `pangu_dec_512.xml`), no activation calibration. Modes: `INT8_SYM`, `INT8_ASYM`, `INT4_SYM`, `INT4_ASYM` (int4 with `group_size=128`, `ratio=1.0`). Compression cost ~20 s (int8) / ~30 s (int4), peak RSS 20-24 GiB, and is load time (not part of pp/tg). See `results.csv` and `logs/` for the numbers and raw records.