Image-Text-to-Text
GGUF
English
Chinese
rknn
rknn3
rk1828
rk1820
rockchip
npu
w4a16
vision-language-model
multimodal
on-device
edge-ai
conversational
Instructions to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 # Run inference directly in the terminal: llama cli -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 # Run inference directly in the terminal: llama cli -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 # Run inference directly in the terminal: ./build/bin/llama cli -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Use Docker
docker model run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Llama new
- LM Studio
- Jan
- vLLM
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Ollama
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Ollama:
ollama run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Unsloth Desktop
- Pi
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Docker Model Runner:
docker model run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Lemonade
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Run and chat with the model
lemonade run user.MiniCPM-V-4.7-1B-RKNN3-RK1828-{{QUANT_TAG}}List all available models
lemonade list
- Hermes Agent
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
|
Download README.md from jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828: direct link, hf CLI and curl.
- Browser
- Download file 8.54 kB
-
https://huggingface.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828/resolve/main/README.md
- Command line
-
hf download hf://jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828/README.md
-
curl -L -o README.md https://huggingface.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828/resolve/main/README.md
8.54 kB
| language: | |
| - en | |
| - zh | |
| license: apache-2.0 | |
| pipeline_tag: image-text-to-text | |
| inference: false | |
| tags: | |
| - rknn | |
| - rknn3 | |
| - rk1828 | |
| - rk1820 | |
| - rockchip | |
| - npu | |
| - w4a16 | |
| - vision-language-model | |
| - image-text-to-text | |
| - multimodal | |
| - on-device | |
| - edge-ai | |
| base_model: openbmb/MiniCPM-V-4.7-1B | |
| # MiniCPM-V-4.7-1B-RKNN3-RK1828 | |
| **MiniCPM-V 4.7 1B converted to RKNN3 and running end-to-end on the Rockchip | |
| RK1828 NPU** — both the text tower and the vision tower, with real image | |
| questions answered on device. | |
| > **⚠️ Read this first.** The vision tower in this repository contains a | |
| > **runtime-contract adapter**: MiniCPM-V 4.7 natively emits 8×8 = **64** visual | |
| > tokens per 448×448 image, but the RKNN3 multimodal callback requires | |
| > `image_ph * image_pw = 14×14 = **196**` tokens. A nearest-neighbour 8×8 → 14×14 | |
| > resample was inserted into the exported graph to satisfy that contract. | |
| > **This build is therefore NOT equivalent to the Hugging Face original | |
| > MiniCPM-V 4.7 1B.** Do not use it to benchmark or reproduce the original | |
| > model's accuracy. See [How it was made](#how-it-was-made). | |
| This repository is not just six binary files. It includes the launch script, | |
| request examples, conversion scripts and a deployment guide, so that the model | |
| can actually be run and reproduced. | |
| | | | | |
| | --- | --- | | |
| | Base model | [`openbmb/MiniCPM-V-4.7-1B`](https://huggingface.co/openbmb/MiniCPM-V-4.7-1B) (Apache-2.0) | | |
| | Hardware | RK1828 / RK1820 NPU (tested `RM1828MC0-F`, firmware 1.1.0) | | |
| | Host | Windows PC or RK3588 / RK3576 board; **Ubuntu x86 cannot drive the module directly** | | |
| | Runtime | `rknn3_110` (`rkllm3-server`), versions must match toolkit / proxy / firmware | | |
| | Quantization | `w4a16`, `normal`, `group32`, 8 cores; head `w6a16` | | |
| | NPU footprint | **1269 MB** (exclusive differential measurement) | | |
| ## What's in the repository | |
| | Path | Contents | | |
| | --- | --- | | |
| | `rknn/` | 6 conversion artifacts + `start.cmd` (1.36 GiB) | | |
| | `host/` | `send_vlm_req.cmd`, two ready-made request bodies, pure-stdlib test-image generator | | |
| | `docs/DEPLOY.md` | Deployment, API contract, 10 pitfalls actually hit, verified results | | |
| | `convert/` | Portable copies of the two vision-export scripts + the conversion recipe | | |
| | `SHA256SUMS` | Checksums of every file above | | |
| ## Quick start | |
| ```text | |
| <root>\ | |
| rknn\ <- this repo | |
| runtime\rknn3_110\bin\ <- rkllm3-server.exe | |
| runtime\rknn3_110\lib\ <- DLLs | |
| ``` | |
| ```cmd | |
| cd /d <root>\rknn | |
| start.cmd :: defaults 0.0.0.0:8899 (RK_HOST / RK_PORT override) | |
| cd /d <root>\host | |
| send_vlm_req.cmd orange :: HTTP=200, response in resp_orange.json | |
| ``` | |
| The server is ready **only when a real inference returns** — the weight-sync log | |
| can sit still for ~2 minutes. Do not poll `rknn-smi` during loading. | |
| Full details (including the Session-1 requirement, proxy reuse and DLL/PATH | |
| rules) are in [`docs/DEPLOY.md`](docs/DEPLOY.md). | |
| ## API | |
| OpenAI-compatible `POST /v1/chat/completions`. `content` **must be an array**; | |
| the server constructs the image placeholder tokens itself: | |
| ```bash | |
| curl -s http://127.0.0.1:8899/v1/chat/completions \ | |
| -H "Content-Type: application/json" \ | |
| -d '{"messages":[{"role":"user","content":[ | |
| {"type":"text","text":"图里是什么?"}, | |
| {"type":"image_url","image_url":{"url":"data:image/png;base64,<BASE64>"}} | |
| ]}],"max_tokens":512}' | |
| ``` | |
| Do **not** put `<image>` / `<|image_pad|>` into the text. Use `curl.exe | |
| --max-time`; PowerShell `Invoke-WebRequest` hangs on this server. | |
| A correct response has `timings.prompt_n ≈ 218` = 196 visual tokens + text | |
| tokens — that is the proof the vision embeddings really entered the LLM. | |
| ## Verified on device | |
| | Check | Result | | |
| | --- | --- | | |
| | Orange image → colour | `这张图是橙色。` | | |
| | Black square → shape/colour | `黑色正方形` | | |
| | Cross-checks (orange asked for shape; square asked for colour) | correct — not language-prior guessing | | |
| | Pure text (capital of China) | `北京` | | |
| | Chinese ASR + this VLM coexisting | 3458 MB (0.6B ASR) / 3353 MB (1.7B ASR), all requests HTTP 200 | | |
| | Throughput | ~90 tok/s generation, ~1700 tok/s prefill | | |
| `finish_reason=stop` on simple cases. Complex images may return `length` — | |
| **always check `finish_reason`, not just HTTP 200**. | |
| ## How it was made | |
| 1. **Text tower** — MiniCPM-V 4.7 reuses the Qwen3.5-0.8B architecture with | |
| `vocab_size=248094` and an independent `lm_head`. Extraction + zoo | |
| `Qwen3_5` `export_llm.py` / `export_rknn.py`. | |
| The **top-level** `tie_word_embeddings` must be set to `false` too — | |
| otherwise HF `tie_weights()` silently binds the 254M-parameter head to the | |
| embedding and the model emits word salad. | |
| 2. **Vision tower** — `vision_tower` + `merger` exported to ONNX with | |
| `transformers>=5.18` (`MiniCPMV4_7ForConditionalGeneration`), then RKNN with | |
| `uint8 NHWC` input and `mean/std = 0.5*255`. | |
| Output is 3D `[1, N, 1024]`; a 2D `[N,1024]` export is mis-parsed as | |
| "N images of 1024 tokens". | |
| 3. **196-token adapter** — nearest-neighbour resample 8×8 → 14×14 inside the | |
| graph (Gather/Reshape only, no `Resize` op). This changes **graph structure | |
| only**: the `.weight` SHA256 is byte-identical to the unadapted 64-token | |
| build, because no weights are touched. | |
| See [`convert/README.md`](convert/README.md) for the recipe and | |
| [`docs/DEPLOY.md`](docs/DEPLOY.md) §5 for the pitfalls. | |
| ## Limitations | |
| - **Not equivalent to the original 4.7** (196-token adapter, see the warning above). | |
| - canvas mRoPE, multi-resolution tiling, high-resolution OCR and dense | |
| multi-object counting were **not verified**. | |
| - OCR is "characters readable" at best (`RK1828` → `RK₁₈₂₈`). | |
| - Only 448×448 single-image input was tested. | |
| - 1B model with a `<think>` chain: raising `max_tokens` is **not** monotonically | |
| safer (256 worked on one OCR case where 512 stalled inside `<think>`). | |
| - Known upstream issue: on ASR device error, NPU memory can leak | |
| (`mem_sync ret=-10` retry loop); workaround is to kill the ASR process. | |
| ## Checksums | |
| ```bash | |
| sha256sum -c SHA256SUMS | |
| ``` | |
| | File | SHA256 | | |
| | --- | --- | | |
| | `MiniCPM-V-4.7-1B-llm.rknn` | `948e487bff9b2324b3016c665aa071feff05ac735e078c20ecb4a0ac7b825a1b` | | |
| | `MiniCPM-V-4.7-1B-llm.weight` | `72fd0b16b8016d63a7d8b92f8d159ad7ea68a783eaca89b915bd0e240ae22dab` | | |
| | `MiniCPM-V-4.7-1B-llm.embed.bin` | `511c5794dbf3d927fb17c5efc0052fababafdd35185be2961f6a855c1852b335` | | |
| | `MiniCPM-V-4.7-1B-llm.tokenizer.gguf` | `e028b0a11710ee5e0402c3829e871c3bd5c568540fe377febdd1a515d45b1544` | | |
| | `MiniCPM-V-4.7-1B-vision.rknn` | `9e081464857fad825281ae8778defb882424cbf5d46a37acb6de59dd5cbc99d6` | | |
| | `MiniCPM-V-4.7-1B-vision.weight` | `9c1939dd21e511c72ef6fa6333d43d962acc067d5302ed625815abeaf1cc1c03` | | |
| ## License & attribution | |
| **Apache-2.0**, inherited from | |
| [`openbmb/MiniCPM-V-4.7-1B`](https://huggingface.co/openbmb/MiniCPM-V-4.7-1B). | |
| Please credit OpenBMB for the original model. The RKNN3 conversion, the | |
| 196-token adapter and the host scripts in this repository are provided under | |
| the same license, without warranty. | |
| Built with [`airockchip/rknn3-model-zoo`](https://github.com/airockchip/rknn3-model-zoo) V1.1.0. | |
| --- | |
| ## 中文摘要 | |
| 把 `openbmb/MiniCPM-V-4.7-1B` 转成 RKNN3,在 **RK1828** 上跑通完整图文链路 | |
| (文本塔 + 视觉塔 + `rkllm3-server --model2`),真机 5/5 带图与纯文本请求 | |
| 全部 HTTP 200,NPU 独占占用 **1269 MB**。 | |
| **必须知道的取舍**:4.7 视觉塔原生只出 8×8=64 个视觉 token,而 RKNN3 runtime | |
| 的 multimodal 回调要求 `image_ph * image_pw = 14×14 = 196`。为满足契约, | |
| 导出图里加了 8×8→14×14 的最近邻重采样**适配层**(只改图结构、不改权重, | |
| `.weight` 的 SHA256 与 64-token 版完全相同)。因此 | |
| **本产物不等价于 HF 原版 4.7**,不能拿它复现原版精度; | |
| canvas mRoPE / 多分辨率切砖 / 高分辨 OCR / 密集计数均未验证。 | |
| | 项 | 结果 | | |
| | --- | --- | | |
| | 橙图问颜色 | `这张图是橙色。` | | |
| | 黑方块 | `黑色正方形` | | |
| | 交叉对照(排除语言先验猜答案) | 正确 | | |
| | 纯文本 | `北京` | | |
| | 与 ASR 共存 | 0.6B 3458 MB / 1.7B 3353 MB(A/B/A 可复现) | | |
| | 视觉 token 注入证据 | `timings.prompt_n ≈ 218` | | |
| 仓库含 6 个转换产物 + `start.cmd` + 请求脚本 + 转换脚本 + 部署文档, | |
| 拿到即可跑;部署与十个实测踩过的坑见 [`docs/DEPLOY.md`](docs/DEPLOY.md)。 | |
| 复杂图务必看 `finish_reason`,别只看 HTTP 200。 | |