Instructions to use HelloSun/Experiment_1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use HelloSun/Experiment_1 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf HelloSun/Experiment_1:Q4_K_M # Run inference directly in the terminal: llama cli -hf HelloSun/Experiment_1:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf HelloSun/Experiment_1:Q4_K_M # Run inference directly in the terminal: llama cli -hf HelloSun/Experiment_1:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf HelloSun/Experiment_1:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf HelloSun/Experiment_1:Q4_K_M
Use Docker
docker model run hf.co/HelloSun/Experiment_1:Q4_K_M
- Llama new
- LM Studio
- Jan
- Ollama
How to use HelloSun/Experiment_1 with Ollama:
ollama run hf.co/HelloSun/Experiment_1:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use HelloSun/Experiment_1 with Docker Model Runner:
docker model run hf.co/HelloSun/Experiment_1:Q4_K_M
- Lemonade
How to use HelloSun/Experiment_1 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull HelloSun/Experiment_1:Q4_K_M
Run and chat with the model
lemonade run user.Experiment_1-Q4_K_M
List all available models
lemonade list
- Atomic Chat
|
Download onednn/openvino_quantization.md from HelloSun/Experiment_1: direct link, hf CLI and curl.
- Browser
- Download file 2.87 kB
-
https://huggingface.co/HelloSun/Experiment_1/resolve/main/onednn/openvino_quantization.md
- Command line
-
hf download hf://HelloSun/Experiment_1/onednn/openvino_quantization.md
-
curl -L -o openvino_quantization.md https://huggingface.co/HelloSun/Experiment_1/resolve/main/onednn/openvino_quantization.md
2.87 kB
| # Why OpenVINO int8 / int4 was not in the first round | |
| Short answer: the OpenVINO numbers I published came from a route that only | |
| selects the *execution precision* of the graph, it does not quantize the | |
| weights. On top of that, the automatic int8/int4 tooling refuses this model, so | |
| integer support had to be added by hand afterwards (done now, see | |
| `results.csv`). | |
| ## 1. `INFERENCE_PRECISION_HINT` is not quantization | |
| The working OpenVINO path here is `ov.convert_model` (TorchScript trace) of the | |
| HF model into an IR, then `compile_model(..., {"INFERENCE_PRECISION_HINT": | |
| "f32"|"bf16"})`. That hint only picks which *kernel* runs (f32 SIMD vs AMX bf16); | |
| the stored weights stay f32. To get int8/int4 you must convert the IR weights to | |
| low precision with NNCF weight compression, which is a separate step. | |
| ## 2. The automatic LLM pipelines reject the architecture | |
| - `optimum-cli export openvino` / `optimum-intel`: no support for | |
| `PanguEmbedded`. | |
| - `openvino-genai` GGUF reader: our GGUF declares `general.architecture = | |
| openpangu-r`, and the reader also fails `gguf_tensor_to_f16` on the Q5_0 | |
| tensors. Dead end. | |
| - NNCF's high-level LLM compression is driven by model-type detection; unknown | |
| architectures are not handled automatically. | |
| - `torch.compile(backend="openvino")` fails on the model's symbolic shapes | |
| (torch 2.13 guards `IndexError`; torch 2.8 vmap / `SymIntArrayRef ... concrete | |
| integers`). | |
| So there is no one-flag route: int8/int4 had to be applied manually with | |
| `nncf.compress_weights` on the IR we already produced. | |
| ## 3. Practical constraints at the time | |
| - The OpenVINO work was added after the oneDNN vs llama.cpp comparison was | |
| already finished, and the obvious int8/int4 story for this project was the | |
| oneDNN side (int8 / int4 weight types). OpenVINO was introduced to answer | |
| "can a vendor graph compiler beat llama.cpp", initially at f32/bf16. | |
| - Memory: the 80 GiB guard killed `convert_model + compile_model` in one process | |
| once (oneDNN-style f32 compile spiked to ~64 GiB per process). Conversion had | |
| to be split from compilation, and IRs had to be saved to disk first. Adding | |
| NNCF compression on top needed its own pass. | |
| - Accuracy risk: this model is numerically sensitive (~1.3x error growth per | |
| layer), so it was not obvious that int4 would be usable. It turned out int8 is | |
| fine and int4 asym is borderline (see below). | |
| ## What was run (this round) | |
| Weight-only compression with NNCF 3.4.0 (`nncf.compress_weights`), applied to | |
| the saved f32 IRs (prefill graph `pangu_f32_512.xml`, decode graph | |
| `pangu_dec_512.xml`), no activation calibration. Modes: `INT8_SYM`, | |
| `INT8_ASYM`, `INT4_SYM`, `INT4_ASYM` (int4 with `group_size=128`, `ratio=1.0`). | |
| Compression cost ~20 s (int8) / ~30 s (int4), peak RSS 20-24 GiB, and is load | |
| time (not part of pp/tg). | |
| See `results.csv` and `logs/` for the numbers and raw records. | |