Instructions to use HelloSun/Experiment_1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use HelloSun/Experiment_1 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf HelloSun/Experiment_1:Q4_K_M # Run inference directly in the terminal: llama cli -hf HelloSun/Experiment_1:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf HelloSun/Experiment_1:Q4_K_M # Run inference directly in the terminal: llama cli -hf HelloSun/Experiment_1:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf HelloSun/Experiment_1:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf HelloSun/Experiment_1:Q4_K_M
Use Docker
docker model run hf.co/HelloSun/Experiment_1:Q4_K_M
- Llama new
- LM Studio
- Jan
- Ollama
How to use HelloSun/Experiment_1 with Ollama:
ollama run hf.co/HelloSun/Experiment_1:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use HelloSun/Experiment_1 with Docker Model Runner:
docker model run hf.co/HelloSun/Experiment_1:Q4_K_M
- Lemonade
How to use HelloSun/Experiment_1 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull HelloSun/Experiment_1:Q4_K_M
Run and chat with the model
lemonade run user.Experiment_1-Q4_K_M
List all available models
lemonade list
- Atomic Chat
|
Download onednn/README.md from HelloSun/Experiment_1: direct link, hf CLI and curl.
- Browser
- Download file 10.4 kB
-
https://huggingface.co/HelloSun/Experiment_1/resolve/main/onednn/README.md
- Command line
-
hf download hf://HelloSun/Experiment_1/onednn/README.md
-
curl -L -o README.md https://huggingface.co/HelloSun/Experiment_1/resolve/main/onednn/README.md
10.4 kB
| # openPangu-R-7B-2512 CPU backend comparison: llama.cpp vs oneDNN vs OpenVINO | |
| > Part of [HelloSun/Experiment_1](https://huggingface.co/HelloSun/Experiment_1): | |
| > the GGUF, the llama.cpp patch and the quick start are in the repository root | |
| > ([`../README.md`](../README.md)). Paths below are relative to `onednn/`. | |
| Three CPU inference backends on **openPangu-R-7B-2512** (27 layers, hidden | |
| 4096, 8.29 B params), benchmarked steady-state (`pp512` / `tg128`, `llama-bench | |
| -r 5` or median of 128 decode steps). Headline config: **Q4_K_M** for the | |
| llama.cpp / oneDNN paths, and the OpenVINO IR built from the fp32 HF weights. | |
| ## Environment | |
| | item | value | | |
| |---|---| | |
| | CPU | Intel Xeon Platinum 8559C, 192 threads, 2 NUMA, AMX BF16/INT8, AVX512 | | |
| | container | 16 CPUs (`cpu.max=1600`), 80 GiB memory guard, no GPU | | |
| | OS | Debian 13 | | |
| | oneDNN | 3.7.2 (`libdnnl-dev`) | | |
| | OpenVINO | 2026.4.1 (runtime), transformers 4.53.2 | | |
| | OpenVINO GenAI | 2026.4.1 (`openvino-genai`) | | |
| | llama.cpp | v0.7.0, commit `620d2225cc436ab0b001e3200e590e3b286ef9ab` + openPangu-R patch | | |
| | model | openPangu-R-7B-2512 Q4_K_M, 4.72 GiB, 8.29 B params | | |
| ## Results (Q4_K_M) | |
| `ppl512` = perplexity over a fixed 512-token (EN+ZH) text; `argmax_agree` = | |
| per-position argmax agreement with the OpenVINO f32 graph. Both quality columns | |
| only apply to the OpenVINO rows (llama.cpp / oneDNN perf was measured with | |
| `llama-bench`). | |
| | backend | weights | pp512 tok/s | tg128 tok/s | ppl512 | argmax | threads | | |
| |---|---|---:|---:|---:|---:|---:| | |
| | llama.cpp (plain AVX512) | Q4_K_M | **180.50** | **13.39** | - | - | 16 | | |
| | llama.cpp (default AMX+repack) | Q4_K_M | 112.77 | 12.19 | - | - | 16 | | |
| | oneDNN | int8 | 90.09 | 7.67 | - | - | 1 (+OMP 16) | | |
| | oneDNN | bf16 | 75.52 | 4.79 | - | - | 1 (+OMP 16) | | |
| | OpenVINO | f32 | 125.60 | 2.22 | **28.82** | 1.000 | 16 | | |
| | OpenVINO | bf16-hint | 209.43 | 3.93 | 28.85 | 0.975 | 16 | | |
| | OpenVINO | int8-asym | 219.17 | 4.70 | 29.11 | 0.951 | 16 | | |
| | OpenVINO | int8-sym | 226.68 | - | 29.34 | 0.941 | 16 | | |
| | OpenVINO | int4-asym | 223.65 | 5.20 | 32.68 | 0.703 | 16 | | |
| | OpenVINO | int4-sym | 222.48 | - | 42.95 | 0.701 | 16 | | |
| `llama-bench -r 5`, 16 CPUs. OpenVINO `tg128` is the median of 128 decode | |
| steps (fixed 640-slot KV cache). This is a shared host: the OpenVINO f32/bf16 | |
| prefill ranged between ~126 and ~366 tok/s across sessions, so treat the | |
| OpenVINO prefill column as order-of-magnitude, not exact. | |
| Notes: | |
| - The llama.cpp Q4_K_M build is the fastest overall on this 16-CPU container: | |
| `13.39 tg128`, `180.50 pp512`. | |
| - OpenVINO has by far the fastest prefill (`366 pp512` at `bf16`, `128` at | |
| `f32`) but the slowest decode here, because its decode graph always runs | |
| attention over the full 640-slot cache and moves all KV-cache tensors across | |
| the host/plugin boundary each step. | |
| - oneDNN loses on both axes. Decode is bandwidth-bound: bf16 weights are | |
| 16.6 GB/token vs ~5 GB/token for Q4_K; int8 is 8.3 GB/token. The AMX int8 | |
| matmul kernel itself is fast (~7.1 TFLOPS at M=512, 4096^3 in the | |
| microbenchmark); the loss is the weight format and per-call activation | |
| quantization, not the kernel. | |
| ## OpenVINO | |
| The published OpenVINO GenAI GGUF reader and `optimum-cli export openvino` | |
| cannot handle this model (`openpangu-r` architecture, custom ops), and | |
| `torch.compile(backend="openvino")` fails on the model's symbolic shapes. The | |
| working route is a direct `ov.convert_model` (TorchScript trace) of the HF | |
| `PanguEmbeddedForCausalLM`: | |
| - prefill graph: `convert_ir.py` -> `pangu_f32_512.xml/bin` | |
| - decode graph: `ov_decode.py --dur convert` (KV cache + the two `WindowBuffer` | |
| states are threaded through the graph as explicit inputs/outputs, so the IR | |
| stays stateless). Convert with fp32 weights, then compile to bf16 with | |
| `INFERENCE_PRECISION_HINT`. | |
| Accuracy (fp32 hint vs HF eager, 512-token prefill): logits maxdiff `0.0004`, | |
| 100% argmax agreement; decode-step argmax matches HF. The **bf16 hint drops | |
| accuracy** (argmax agreement 75%) because the model is numerically sensitive | |
| (error amplifies ~1.3x per layer), so bf16 is a speed/accuracy trade-off. | |
| ### OpenVINO int8 / int4 (NNCF weight compression) | |
| `INFERENCE_PRECISION_HINT` does **not** quantize weights; it only picks the | |
| execution kernel. int8/int4 need NNCF `compress_weights` applied to the IR | |
| (see `openvino_quantization.md` for why this was not in the first round). | |
| Weight-only compression (no activation calibration), on the saved f32 IRs: | |
| | mode | file size | pp512 | tg128 | ppl512 | argmax vs f32 | top-1 factual | | |
| |---|---:|---:|---:|---:|---:|---| | |
| | f32 (ref) | 16.6 GB | 125.6 | 2.22 | 28.82 | 1.000 | Paris / Jupiter / oxygen | | |
| | int8 asym | 8.3 GB | 219.2 | 4.70 | 29.11 | 0.951 | Paris / Jupiter / oxygen | | |
| | int8 sym | 8.3 GB | 226.7 | - | 29.34 | 0.941 | Paris / Jupiter / oxygen | | |
| | int4 asym | 4.9 GB | 223.7 | 5.20 | 32.68 | 0.703 | Paris / Jupiter / oxygen | | |
| | int4 sym | 4.9 GB | 222.5 | - | 42.95 | 0.701 | Paris / Jupiter / oxygen | | |
| Answer-quality check: greedy top-1 for `The capital of France is` / `The largest | |
| planet ...` / `Water is made of hydrogen and` is `Paris` / `Jupiter` / `oxygen` | |
| for **every** variant, including int4. So int4 keeps short factual answers but | |
| loses a lot of distributional fidelity (PPL +3.9, and 30% of argmax positions | |
| change); int8 is close to f32 (PPL +0.3, ~95% argmax). int8/int4 roughly double | |
| decode throughput over f32 (bandwidth-bound), to 4.7-5.2 tok/s, still below | |
| llama.cpp Q4_K_M (13.39). | |
| Requirements: the local `modeling_openpangu_dense.py` must be patched | |
| (`sliding_window` per layer type, sink-mask padding) and the published code | |
| imports `torch_npu` (a stub package is enough). | |
| ### OpenVINO GenAI (LLMPipeline) | |
| `openvino_genai.LLMPipeline` was tested as the high-level alternative to the | |
| low-level `ov.Core().compile_model` path. With OpenVINO GenAI 2026.4.1 it cannot | |
| run this model: | |
| | input to `LLMPipeline` | result | | |
| |---|---| | |
| | HF checkpoint directory | `Could not find a model in the directory` (GenAI needs an IR) | | |
| | Q4_K_M GGUF | `[load_gguf] gguf_tensor_to_f16 failed` | | |
| | manual 64-token prefill IR + tokenizer | loads in 3.9 s; `generate()` fails: `Port for tensor name attention_mask was not found` | | |
| | `optimum-cli export openvino --trust-remote-code` | `custom or unsupported architecture` | | |
| The GGUF failure is a format limitation, not memory. The published Q4_K_M file | |
| mixes 54 `Q5_0` tensors with `Q4_K`/`Q6_K`/f32, and the GenAI reader only | |
| dequantizes `Q4_0/Q4_1/Q4_K/Q8_0/Q6_K`; a normalized file would still hit the | |
| architecture check, which accepts only `llama`, `qwen2`, `qwen3` (this file says | |
| `openpangu-r`). | |
| The IR used by the low-level OpenVINO path is also not GenAI-compatible: | |
| `LLMPipeline` expects one stateful graph with `input_ids`, `attention_mask`, | |
| `position_ids`, `beam_idx` and KV-cache variables, while the Pangu graphs are | |
| stateless prefill/decode IRs with explicit KV and `WindowBuffer` | |
| inputs/outputs. Working GenAI support would require a stateful, dynamic-shape | |
| re-export of the custom architecture. | |
| Test: `openvino/genai_api_test.py --model <HF dir> --gguf <Q4_K_M.gguf> | |
| --ir-dir /tmp/pangu_genai --make-ir 64`. | |
| The IR build needs `einops` and `sentencepiece`. Raw log: `logs/genai_api.log`. | |
| ## oneDNN implementation | |
| New files: `ggml/src/ggml-cpu/onednn.cpp`, `onednn.h`; a hook in | |
| `ggml_compute_forward_mul_mat` (before the tiled path) in `ggml-cpu.c`; | |
| `GGML_CPU_ONEDNN` option in `ggml/CMakeLists.txt` and | |
| `ggml/src/ggml-cpu/CMakeLists.txt`. Thread 0 calls oneDNN, the others wait at a | |
| barrier. Only 2D quantized-weight x 2D activation MUL_MAT (projections and | |
| lm_head) are handled; batched attention matmuls stay on the ggml path. | |
| Weight conversion is done once at load (cached per tensor): bf16 round-to- | |
| nearest-even, int8 per-column symmetric `s8` (`tag::ba`), int4 per-column `s4` | |
| packed two per byte (`tag::ba`). Activations use per-tensor asymmetric `u8`. | |
| `GGML_ONEDNN_WEIGHT_TYPE` selects `bf16` (default) / `int8` / `int4`; | |
| `GGML_ONEDNN_DEBUG` prints conversion and timing info. | |
| Build: `./build.sh /path/to/llama.cpp build-onednn` with `GGML_NATIVE=OFF`, | |
| `GGML_AMX_*=OFF`, `GGML_CPU_REPACK=OFF`. With the native build the Q4_K/Q6_K | |
| projections are repacked into the AMX extra buffer and executed by | |
| `repack.cpp`, so `ggml_compute_forward_mul_mat` (and the hook) is never | |
| reached. | |
| ### Known limitations | |
| - int4 has no optimized CPU kernel in oneDNN 3.7 (`ref_int8`, 0.8 GFLOPS). | |
| - No batched (4D) matmul, F32 weight matmul, or non-contiguous input support. | |
| - Per-tensor activation quantization: greedy output matches the baseline for | |
| the first tokens, then diverges. | |
| - First-call weight conversion costs ~15 s (bf16) / ~45 s (int8). | |
| ## Files | |
| - `results.csv` - the table above | |
| - `build.sh`, `bench.sh` - build / benchmark helpers | |
| - `onednn.cpp`, `onednn.h`, `onednn.patch` - oneDNN backend | |
| - `openvino/` - benchmark and correctness programs: | |
| - `convert_ir.py` - trace + save prefill IR | |
| - `ov_decode.py` - trace + save decode IR, then benchmark decode | |
| - `bench_ov_ir.py`, `bench_ov_prefill.py` - prefill benchmark | |
| - `ov_compress.py` - NNCF weight compression (int8/int4) of an IR | |
| - `ov_quality.py` - PPL + argmax agreement of an IR | |
| - `ov_answer.py` - qualitative top-5 next-token check | |
| - `ov_prec_test.py` - f32 vs bf16 hint accuracy | |
| - `ov_dec_correct.py` - decode-step correctness vs HF | |
| - `ov_layer_iso.py`, `ov_layer_internals.py`, `ov_attn_iso.py`, | |
| `ov_ops_iso.py`, `ov_hidden_k.py` - isolation probes used to debug | |
| - `ov_tiny_llama.py`, `ov_tiny_layer.py`, `ov_tiny_test.py` - control models | |
| - `bench_hf.py` - HF eager / oneDNN-backed PyTorch baseline | |
| - `genai_api_test.py` - OpenVINO GenAI `LLMPipeline` probes (HF/GGUF/IR); | |
| `--make-ir N` builds the N-token prefill IR + tokenizer it uses | |
| - `logs/` - raw benchmark and accuracy records: | |
| - `q4km_llamacpp_onednn.log` - `llama-bench -r 5` for llama.cpp + oneDNN | |
| - `openvino_prefill_{bf16,f32}.log`, `openvino_decode_{bf16,f32}.log` | |
| - `openvino_ir_convert.log` - IR conversion timings | |
| - `openvino_accuracy.log` - correctness checks | |
| - `quant/` - NNCF compression logs, PPL/argmax results, decode speed, | |
| answer-quality log for each compressed IR | |
| - `genai_api.log`, `genai_api.json` - OpenVINO GenAI `LLMPipeline` probe | |
| output (HF / GGUF / IR failures) | |
| - `openvino_quantization.md` - why int8/int4 was missing initially | |