Instructions to use HelloSun/openPangu-VL-7B-VQ3-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use HelloSun/openPangu-VL-7B-VQ3-gguf with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf HelloSun/openPangu-VL-7B-VQ3-gguf:Q8_0 # Run inference directly in the terminal: llama cli -hf HelloSun/openPangu-VL-7B-VQ3-gguf:Q8_0
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf HelloSun/openPangu-VL-7B-VQ3-gguf:Q8_0 # Run inference directly in the terminal: llama cli -hf HelloSun/openPangu-VL-7B-VQ3-gguf:Q8_0
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf HelloSun/openPangu-VL-7B-VQ3-gguf:Q8_0 # Run inference directly in the terminal: ./build/bin/llama cli -hf HelloSun/openPangu-VL-7B-VQ3-gguf:Q8_0
Use Docker
docker model run hf.co/HelloSun/openPangu-VL-7B-VQ3-gguf:Q8_0
- Llama new
- LM Studio
- Jan
- vLLM
How to use HelloSun/openPangu-VL-7B-VQ3-gguf with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "HelloSun/openPangu-VL-7B-VQ3-gguf" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "HelloSun/openPangu-VL-7B-VQ3-gguf", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/HelloSun/openPangu-VL-7B-VQ3-gguf:Q8_0
- Ollama
How to use HelloSun/openPangu-VL-7B-VQ3-gguf with Ollama:
ollama run hf.co/HelloSun/openPangu-VL-7B-VQ3-gguf:Q8_0
- Unsloth Desktop
- Docker Model Runner
How to use HelloSun/openPangu-VL-7B-VQ3-gguf with Docker Model Runner:
docker model run hf.co/HelloSun/openPangu-VL-7B-VQ3-gguf:Q8_0
- Lemonade
How to use HelloSun/openPangu-VL-7B-VQ3-gguf with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull HelloSun/openPangu-VL-7B-VQ3-gguf:Q8_0
Run and chat with the model
lemonade run user.openPangu-VL-7B-VQ3-gguf-Q8_0
List all available models
lemonade list
- Atomic Chat
openPangu-VL-7B VQ3(lut-vq3-64,3.00 bpw FFN)
openpangu/openPangu-VL-7B 的 LUT-VQ3-64 實驗性量化 (格式來自 llama-modchip)。稠密 7B 模型, 34 層 / hidden 4096 / FFN 12800 / 詞彙 153k。
| 檔案 | 量化 | 大小 | 下載 |
|---|---|---|---|
| openPangu-VL-7B-VQ3.gguf | 102 個稠密 FFN → VQ3_0(3.00 bpw);attention → Q8_0;token_embd → Q4_K;output → Q6_K | 4.09 GiB | ⬇ 直鏈 |
| openPangu-VL-7B-mmproj-Q8_0.gguf | 視覺投影器(與 Q4_K_M 版共用) | 0.74 GiB | ⬇ 直鏈 |
品質與速度(誠實數據,wikitext-2 PPL / llama-bench)
| 配置 | PPL | pp512 @32t | tg128 @32t | tg128 @16t |
|---|---|---|---|---|
| Q8_0 混合基底 | 7.534 ± 0.207 | — | — | — |
| Q4_K_M(建議) | 7.652 ± 0.208 | 170.9 | 22.0 | 16.6 |
| VQ3(本 repo) | 10.672 ± 0.309 | 67.1 | 19.1 | 17.0 |
結論:LUT-VQ3-64 的優勢在 MoE 專家張量(見 flint-VQ3:
PPL 持平、decode +9~14%、−35% 體積)。稠密模型每 token 全部 FFN 啟用,3bpw 誤差無法稀釋 →
PPL +39.5%、decode −13%、prefill −46%,僅體積 −11%。日常使用請選 Q4_K_M;本檔為
實驗性體積檔與 VQ3-稠密邊界記錄。完整文檔:vq3/README.md。
一鍵使用
./build.sh # clone llama.cpp @1268eda → 套 pangu-vl+VQ3 patch → 建置 → 啟動 llama-server :20000(含視覺)
- OpenAI 相容 API:
http://127.0.0.1:20000/v1(chat + 圖像) TEXT_ONLY=1 ./build.sh純文字;LLAMA_PORT/CONTEXT_SIZE/LLAMA_THREADS可調- 重製 VQ3 檔:
bash vq3/convert.sh(官方 safetensors → F16 → Q8_0 base → VQ3)
已驗證
- 文字生成連貫(自我介紹正確:「我是华为公司研发的盘古多模态大模型…」)
- 視覺:太空人 EVA 圖 →「這張圖片顯示一名宇航員在太空站外進行太空行走」✓
general.file_type = 42(VQ3_0);llama.cpp 顯示 "mostly VQ3_0"
授權
模型權重依華為 openPangu 原授權;VQ3 格式/工具鏈來自 llama-modchip;pangu-vl llama.cpp 支援 來自 HelloSun/openPangu-VL-7B-Q4KM-gguf。
- Downloads last month
- 186
We're not able to determine the quantization variants.
Model tree for HelloSun/openPangu-VL-7B-VQ3-gguf
Base model
openpangu/openPangu-VL-7B