Instructions to use HelloSun/Experiment_1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use HelloSun/Experiment_1 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf HelloSun/Experiment_1:Q4_K_M # Run inference directly in the terminal: llama cli -hf HelloSun/Experiment_1:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf HelloSun/Experiment_1:Q4_K_M # Run inference directly in the terminal: llama cli -hf HelloSun/Experiment_1:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf HelloSun/Experiment_1:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf HelloSun/Experiment_1:Q4_K_M
Use Docker
docker model run hf.co/HelloSun/Experiment_1:Q4_K_M
- Llama new
- LM Studio
- Jan
- Ollama
How to use HelloSun/Experiment_1 with Ollama:
ollama run hf.co/HelloSun/Experiment_1:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use HelloSun/Experiment_1 with Docker Model Runner:
docker model run hf.co/HelloSun/Experiment_1:Q4_K_M
- Lemonade
How to use HelloSun/Experiment_1 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull HelloSun/Experiment_1:Q4_K_M
Run and chat with the model
lemonade run user.Experiment_1-Q4_K_M
List all available models
lemonade list
- Atomic Chat
Download onednn/README.md from HelloSun/Experiment_1: direct link, hf CLI and curl.
- Browser
- Download file 10.4 kB
-
https://huggingface.co/HelloSun/Experiment_1/resolve/main/onednn/README.md
- Command line
-
hf download hf://HelloSun/Experiment_1/onednn/README.md
-
curl -L -o README.md https://huggingface.co/HelloSun/Experiment_1/resolve/main/onednn/README.md
openPangu-R-7B-2512 CPU backend comparison: llama.cpp vs oneDNN vs OpenVINO
Part of HelloSun/Experiment_1: the GGUF, the llama.cpp patch and the quick start are in the repository root (
../README.md). Paths below are relative toonednn/.
Three CPU inference backends on openPangu-R-7B-2512 (27 layers, hidden
4096, 8.29 B params), benchmarked steady-state (pp512 / tg128, llama-bench -r 5 or median of 128 decode steps). Headline config: Q4_K_M for the
llama.cpp / oneDNN paths, and the OpenVINO IR built from the fp32 HF weights.
Environment
| item | value |
|---|---|
| CPU | Intel Xeon Platinum 8559C, 192 threads, 2 NUMA, AMX BF16/INT8, AVX512 |
| container | 16 CPUs (cpu.max=1600), 80 GiB memory guard, no GPU |
| OS | Debian 13 |
| oneDNN | 3.7.2 (libdnnl-dev) |
| OpenVINO | 2026.4.1 (runtime), transformers 4.53.2 |
| OpenVINO GenAI | 2026.4.1 (openvino-genai) |
| llama.cpp | v0.7.0, commit 620d2225cc436ab0b001e3200e590e3b286ef9ab + openPangu-R patch |
| model | openPangu-R-7B-2512 Q4_K_M, 4.72 GiB, 8.29 B params |
Results (Q4_K_M)
ppl512 = perplexity over a fixed 512-token (EN+ZH) text; argmax_agree =
per-position argmax agreement with the OpenVINO f32 graph. Both quality columns
only apply to the OpenVINO rows (llama.cpp / oneDNN perf was measured with
llama-bench).
| backend | weights | pp512 tok/s | tg128 tok/s | ppl512 | argmax | threads |
|---|---|---|---|---|---|---|
| llama.cpp (plain AVX512) | Q4_K_M | 180.50 | 13.39 | - | - | 16 |
| llama.cpp (default AMX+repack) | Q4_K_M | 112.77 | 12.19 | - | - | 16 |
| oneDNN | int8 | 90.09 | 7.67 | - | - | 1 (+OMP 16) |
| oneDNN | bf16 | 75.52 | 4.79 | - | - | 1 (+OMP 16) |
| OpenVINO | f32 | 125.60 | 2.22 | 28.82 | 1.000 | 16 |
| OpenVINO | bf16-hint | 209.43 | 3.93 | 28.85 | 0.975 | 16 |
| OpenVINO | int8-asym | 219.17 | 4.70 | 29.11 | 0.951 | 16 |
| OpenVINO | int8-sym | 226.68 | - | 29.34 | 0.941 | 16 |
| OpenVINO | int4-asym | 223.65 | 5.20 | 32.68 | 0.703 | 16 |
| OpenVINO | int4-sym | 222.48 | - | 42.95 | 0.701 | 16 |
llama-bench -r 5, 16 CPUs. OpenVINO tg128 is the median of 128 decode
steps (fixed 640-slot KV cache). This is a shared host: the OpenVINO f32/bf16
prefill ranged between ~126 and ~366 tok/s across sessions, so treat the
OpenVINO prefill column as order-of-magnitude, not exact.
Notes:
- The llama.cpp Q4_K_M build is the fastest overall on this 16-CPU container:
13.39 tg128,180.50 pp512. - OpenVINO has by far the fastest prefill (
366 pp512atbf16,128atf32) but the slowest decode here, because its decode graph always runs attention over the full 640-slot cache and moves all KV-cache tensors across the host/plugin boundary each step. - oneDNN loses on both axes. Decode is bandwidth-bound: bf16 weights are
16.6 GB/token vs
5 GB/token for Q4_K; int8 is 8.3 GB/token. The AMX int8 matmul kernel itself is fast (7.1 TFLOPS at M=512, 4096^3 in the microbenchmark); the loss is the weight format and per-call activation quantization, not the kernel.
OpenVINO
The published OpenVINO GenAI GGUF reader and optimum-cli export openvino
cannot handle this model (openpangu-r architecture, custom ops), and
torch.compile(backend="openvino") fails on the model's symbolic shapes. The
working route is a direct ov.convert_model (TorchScript trace) of the HF
PanguEmbeddedForCausalLM:
- prefill graph:
convert_ir.py->pangu_f32_512.xml/bin - decode graph:
ov_decode.py --dur convert(KV cache + the twoWindowBufferstates are threaded through the graph as explicit inputs/outputs, so the IR stays stateless). Convert with fp32 weights, then compile to bf16 withINFERENCE_PRECISION_HINT.
Accuracy (fp32 hint vs HF eager, 512-token prefill): logits maxdiff 0.0004,
100% argmax agreement; decode-step argmax matches HF. The bf16 hint drops
accuracy (argmax agreement 75%) because the model is numerically sensitive
(error amplifies ~1.3x per layer), so bf16 is a speed/accuracy trade-off.
OpenVINO int8 / int4 (NNCF weight compression)
INFERENCE_PRECISION_HINT does not quantize weights; it only picks the
execution kernel. int8/int4 need NNCF compress_weights applied to the IR
(see openvino_quantization.md for why this was not in the first round).
Weight-only compression (no activation calibration), on the saved f32 IRs:
| mode | file size | pp512 | tg128 | ppl512 | argmax vs f32 | top-1 factual |
|---|---|---|---|---|---|---|
| f32 (ref) | 16.6 GB | 125.6 | 2.22 | 28.82 | 1.000 | Paris / Jupiter / oxygen |
| int8 asym | 8.3 GB | 219.2 | 4.70 | 29.11 | 0.951 | Paris / Jupiter / oxygen |
| int8 sym | 8.3 GB | 226.7 | - | 29.34 | 0.941 | Paris / Jupiter / oxygen |
| int4 asym | 4.9 GB | 223.7 | 5.20 | 32.68 | 0.703 | Paris / Jupiter / oxygen |
| int4 sym | 4.9 GB | 222.5 | - | 42.95 | 0.701 | Paris / Jupiter / oxygen |
Answer-quality check: greedy top-1 for The capital of France is / The largest planet ... / Water is made of hydrogen and is Paris / Jupiter / oxygen
for every variant, including int4. So int4 keeps short factual answers but
loses a lot of distributional fidelity (PPL +3.9, and 30% of argmax positions
change); int8 is close to f32 (PPL +0.3, ~95% argmax). int8/int4 roughly double
decode throughput over f32 (bandwidth-bound), to 4.7-5.2 tok/s, still below
llama.cpp Q4_K_M (13.39).
Requirements: the local modeling_openpangu_dense.py must be patched
(sliding_window per layer type, sink-mask padding) and the published code
imports torch_npu (a stub package is enough).
OpenVINO GenAI (LLMPipeline)
openvino_genai.LLMPipeline was tested as the high-level alternative to the
low-level ov.Core().compile_model path. With OpenVINO GenAI 2026.4.1 it cannot
run this model:
input to LLMPipeline |
result |
|---|---|
| HF checkpoint directory | Could not find a model in the directory (GenAI needs an IR) |
| Q4_K_M GGUF | [load_gguf] gguf_tensor_to_f16 failed |
| manual 64-token prefill IR + tokenizer | loads in 3.9 s; generate() fails: Port for tensor name attention_mask was not found |
optimum-cli export openvino --trust-remote-code |
custom or unsupported architecture |
The GGUF failure is a format limitation, not memory. The published Q4_K_M file
mixes 54 Q5_0 tensors with Q4_K/Q6_K/f32, and the GenAI reader only
dequantizes Q4_0/Q4_1/Q4_K/Q8_0/Q6_K; a normalized file would still hit the
architecture check, which accepts only llama, qwen2, qwen3 (this file says
openpangu-r).
The IR used by the low-level OpenVINO path is also not GenAI-compatible:
LLMPipeline expects one stateful graph with input_ids, attention_mask,
position_ids, beam_idx and KV-cache variables, while the Pangu graphs are
stateless prefill/decode IRs with explicit KV and WindowBuffer
inputs/outputs. Working GenAI support would require a stateful, dynamic-shape
re-export of the custom architecture.
Test: openvino/genai_api_test.py --model <HF dir> --gguf <Q4_K_M.gguf> --ir-dir /tmp/pangu_genai --make-ir 64.
The IR build needs einops and sentencepiece. Raw log: logs/genai_api.log.
oneDNN implementation
New files: ggml/src/ggml-cpu/onednn.cpp, onednn.h; a hook in
ggml_compute_forward_mul_mat (before the tiled path) in ggml-cpu.c;
GGML_CPU_ONEDNN option in ggml/CMakeLists.txt and
ggml/src/ggml-cpu/CMakeLists.txt. Thread 0 calls oneDNN, the others wait at a
barrier. Only 2D quantized-weight x 2D activation MUL_MAT (projections and
lm_head) are handled; batched attention matmuls stay on the ggml path.
Weight conversion is done once at load (cached per tensor): bf16 round-to-
nearest-even, int8 per-column symmetric s8 (tag::ba), int4 per-column s4
packed two per byte (tag::ba). Activations use per-tensor asymmetric u8.
GGML_ONEDNN_WEIGHT_TYPE selects bf16 (default) / int8 / int4;
GGML_ONEDNN_DEBUG prints conversion and timing info.
Build: ./build.sh /path/to/llama.cpp build-onednn with GGML_NATIVE=OFF,
GGML_AMX_*=OFF, GGML_CPU_REPACK=OFF. With the native build the Q4_K/Q6_K
projections are repacked into the AMX extra buffer and executed by
repack.cpp, so ggml_compute_forward_mul_mat (and the hook) is never
reached.
Known limitations
- int4 has no optimized CPU kernel in oneDNN 3.7 (
ref_int8, 0.8 GFLOPS). - No batched (4D) matmul, F32 weight matmul, or non-contiguous input support.
- Per-tensor activation quantization: greedy output matches the baseline for the first tokens, then diverges.
- First-call weight conversion costs ~15 s (bf16) / ~45 s (int8).
Files
results.csv- the table abovebuild.sh,bench.sh- build / benchmark helpersonednn.cpp,onednn.h,onednn.patch- oneDNN backendopenvino/- benchmark and correctness programs:convert_ir.py- trace + save prefill IRov_decode.py- trace + save decode IR, then benchmark decodebench_ov_ir.py,bench_ov_prefill.py- prefill benchmarkov_compress.py- NNCF weight compression (int8/int4) of an IRov_quality.py- PPL + argmax agreement of an IRov_answer.py- qualitative top-5 next-token checkov_prec_test.py- f32 vs bf16 hint accuracyov_dec_correct.py- decode-step correctness vs HFov_layer_iso.py,ov_layer_internals.py,ov_attn_iso.py,ov_ops_iso.py,ov_hidden_k.py- isolation probes used to debugov_tiny_llama.py,ov_tiny_layer.py,ov_tiny_test.py- control modelsbench_hf.py- HF eager / oneDNN-backed PyTorch baselinegenai_api_test.py- OpenVINO GenAILLMPipelineprobes (HF/GGUF/IR);--make-ir Nbuilds the N-token prefill IR + tokenizer it uses
logs/- raw benchmark and accuracy records:q4km_llamacpp_onednn.log-llama-bench -r 5for llama.cpp + oneDNNopenvino_prefill_{bf16,f32}.log,openvino_decode_{bf16,f32}.logopenvino_ir_convert.log- IR conversion timingsopenvino_accuracy.log- correctness checksquant/- NNCF compression logs, PPL/argmax results, decode speed, answer-quality log for each compressed IRgenai_api.log,genai_api.json- OpenVINO GenAILLMPipelineprobe output (HF / GGUF / IR failures)
openvino_quantization.md- why int8/int4 was missing initially