Instructions to use HelloSun/Experiment_1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use HelloSun/Experiment_1 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf HelloSun/Experiment_1:Q4_K_M # Run inference directly in the terminal: llama cli -hf HelloSun/Experiment_1:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf HelloSun/Experiment_1:Q4_K_M # Run inference directly in the terminal: llama cli -hf HelloSun/Experiment_1:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf HelloSun/Experiment_1:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf HelloSun/Experiment_1:Q4_K_M
Use Docker
docker model run hf.co/HelloSun/Experiment_1:Q4_K_M
- Llama new
- LM Studio
- Jan
- Ollama
How to use HelloSun/Experiment_1 with Ollama:
ollama run hf.co/HelloSun/Experiment_1:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use HelloSun/Experiment_1 with Docker Model Runner:
docker model run hf.co/HelloSun/Experiment_1:Q4_K_M
- Lemonade
How to use HelloSun/Experiment_1 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull HelloSun/Experiment_1:Q4_K_M
Run and chat with the model
lemonade run user.Experiment_1-Q4_K_M
List all available models
lemonade list
- Atomic Chat
Download onednn/openvino_quantization.md from HelloSun/Experiment_1: direct link, hf CLI and curl.
- Browser
- Download file 2.87 kB
-
https://huggingface.co/HelloSun/Experiment_1/resolve/main/onednn/openvino_quantization.md
- Command line
-
hf download hf://HelloSun/Experiment_1/onednn/openvino_quantization.md
-
curl -L -o openvino_quantization.md https://huggingface.co/HelloSun/Experiment_1/resolve/main/onednn/openvino_quantization.md
Why OpenVINO int8 / int4 was not in the first round
Short answer: the OpenVINO numbers I published came from a route that only
selects the execution precision of the graph, it does not quantize the
weights. On top of that, the automatic int8/int4 tooling refuses this model, so
integer support had to be added by hand afterwards (done now, see
results.csv).
1. INFERENCE_PRECISION_HINT is not quantization
The working OpenVINO path here is ov.convert_model (TorchScript trace) of the
HF model into an IR, then compile_model(..., {"INFERENCE_PRECISION_HINT": "f32"|"bf16"}). That hint only picks which kernel runs (f32 SIMD vs AMX bf16);
the stored weights stay f32. To get int8/int4 you must convert the IR weights to
low precision with NNCF weight compression, which is a separate step.
2. The automatic LLM pipelines reject the architecture
optimum-cli export openvino/optimum-intel: no support forPanguEmbedded.openvino-genaiGGUF reader: our GGUF declaresgeneral.architecture = openpangu-r, and the reader also failsgguf_tensor_to_f16on the Q5_0 tensors. Dead end.- NNCF's high-level LLM compression is driven by model-type detection; unknown architectures are not handled automatically.
torch.compile(backend="openvino")fails on the model's symbolic shapes (torch 2.13 guardsIndexError; torch 2.8 vmap /SymIntArrayRef ... concrete integers).
So there is no one-flag route: int8/int4 had to be applied manually with
nncf.compress_weights on the IR we already produced.
3. Practical constraints at the time
- The OpenVINO work was added after the oneDNN vs llama.cpp comparison was already finished, and the obvious int8/int4 story for this project was the oneDNN side (int8 / int4 weight types). OpenVINO was introduced to answer "can a vendor graph compiler beat llama.cpp", initially at f32/bf16.
- Memory: the 80 GiB guard killed
convert_model + compile_modelin one process once (oneDNN-style f32 compile spiked to ~64 GiB per process). Conversion had to be split from compilation, and IRs had to be saved to disk first. Adding NNCF compression on top needed its own pass. - Accuracy risk: this model is numerically sensitive (~1.3x error growth per layer), so it was not obvious that int4 would be usable. It turned out int8 is fine and int4 asym is borderline (see below).
What was run (this round)
Weight-only compression with NNCF 3.4.0 (nncf.compress_weights), applied to
the saved f32 IRs (prefill graph pangu_f32_512.xml, decode graph
pangu_dec_512.xml), no activation calibration. Modes: INT8_SYM,
INT8_ASYM, INT4_SYM, INT4_ASYM (int4 with group_size=128, ratio=1.0).
Compression cost ~20 s (int8) / ~30 s (int4), peak RSS 20-24 GiB, and is load
time (not part of pp/tg).
See results.csv and logs/ for the numbers and raw records.