Image-Text-to-Text
GGUF
English
Chinese
rknn
rknn3
rk1828
rk1820
rockchip
npu
w4a16
vision-language-model
multimodal
on-device
edge-ai
conversational
Instructions to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 # Run inference directly in the terminal: llama cli -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 # Run inference directly in the terminal: llama cli -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 # Run inference directly in the terminal: ./build/bin/llama cli -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Use Docker
docker model run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Llama new
- LM Studio
- Jan
- vLLM
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Ollama
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Ollama:
ollama run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Unsloth Desktop
- Pi
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Docker Model Runner:
docker model run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Lemonade
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Run and chat with the model
lemonade run user.MiniCPM-V-4.7-1B-RKNN3-RK1828-{{QUANT_TAG}}List all available models
lemonade list
- Hermes Agent
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
File size: 8,541 Bytes
fd11f30 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 | ---
language:
- en
- zh
license: apache-2.0
pipeline_tag: image-text-to-text
inference: false
tags:
- rknn
- rknn3
- rk1828
- rk1820
- rockchip
- npu
- w4a16
- vision-language-model
- image-text-to-text
- multimodal
- on-device
- edge-ai
base_model: openbmb/MiniCPM-V-4.7-1B
---
# MiniCPM-V-4.7-1B-RKNN3-RK1828
**MiniCPM-V 4.7 1B converted to RKNN3 and running end-to-end on the Rockchip
RK1828 NPU** — both the text tower and the vision tower, with real image
questions answered on device.
> **⚠️ Read this first.** The vision tower in this repository contains a
> **runtime-contract adapter**: MiniCPM-V 4.7 natively emits 8×8 = **64** visual
> tokens per 448×448 image, but the RKNN3 multimodal callback requires
> `image_ph * image_pw = 14×14 = **196**` tokens. A nearest-neighbour 8×8 → 14×14
> resample was inserted into the exported graph to satisfy that contract.
> **This build is therefore NOT equivalent to the Hugging Face original
> MiniCPM-V 4.7 1B.** Do not use it to benchmark or reproduce the original
> model's accuracy. See [How it was made](#how-it-was-made).
This repository is not just six binary files. It includes the launch script,
request examples, conversion scripts and a deployment guide, so that the model
can actually be run and reproduced.
| | |
| --- | --- |
| Base model | [`openbmb/MiniCPM-V-4.7-1B`](https://huggingface.co/openbmb/MiniCPM-V-4.7-1B) (Apache-2.0) |
| Hardware | RK1828 / RK1820 NPU (tested `RM1828MC0-F`, firmware 1.1.0) |
| Host | Windows PC or RK3588 / RK3576 board; **Ubuntu x86 cannot drive the module directly** |
| Runtime | `rknn3_110` (`rkllm3-server`), versions must match toolkit / proxy / firmware |
| Quantization | `w4a16`, `normal`, `group32`, 8 cores; head `w6a16` |
| NPU footprint | **1269 MB** (exclusive differential measurement) |
## What's in the repository
| Path | Contents |
| --- | --- |
| `rknn/` | 6 conversion artifacts + `start.cmd` (1.36 GiB) |
| `host/` | `send_vlm_req.cmd`, two ready-made request bodies, pure-stdlib test-image generator |
| `docs/DEPLOY.md` | Deployment, API contract, 10 pitfalls actually hit, verified results |
| `convert/` | Portable copies of the two vision-export scripts + the conversion recipe |
| `SHA256SUMS` | Checksums of every file above |
## Quick start
```text
<root>\
rknn\ <- this repo
runtime\rknn3_110\bin\ <- rkllm3-server.exe
runtime\rknn3_110\lib\ <- DLLs
```
```cmd
cd /d <root>\rknn
start.cmd :: defaults 0.0.0.0:8899 (RK_HOST / RK_PORT override)
cd /d <root>\host
send_vlm_req.cmd orange :: HTTP=200, response in resp_orange.json
```
The server is ready **only when a real inference returns** — the weight-sync log
can sit still for ~2 minutes. Do not poll `rknn-smi` during loading.
Full details (including the Session-1 requirement, proxy reuse and DLL/PATH
rules) are in [`docs/DEPLOY.md`](docs/DEPLOY.md).
## API
OpenAI-compatible `POST /v1/chat/completions`. `content` **must be an array**;
the server constructs the image placeholder tokens itself:
```bash
curl -s http://127.0.0.1:8899/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"messages":[{"role":"user","content":[
{"type":"text","text":"图里是什么?"},
{"type":"image_url","image_url":{"url":"data:image/png;base64,<BASE64>"}}
]}],"max_tokens":512}'
```
Do **not** put `<image>` / `<|image_pad|>` into the text. Use `curl.exe
--max-time`; PowerShell `Invoke-WebRequest` hangs on this server.
A correct response has `timings.prompt_n ≈ 218` = 196 visual tokens + text
tokens — that is the proof the vision embeddings really entered the LLM.
## Verified on device
| Check | Result |
| --- | --- |
| Orange image → colour | `这张图是橙色。` |
| Black square → shape/colour | `黑色正方形` |
| Cross-checks (orange asked for shape; square asked for colour) | correct — not language-prior guessing |
| Pure text (capital of China) | `北京` |
| Chinese ASR + this VLM coexisting | 3458 MB (0.6B ASR) / 3353 MB (1.7B ASR), all requests HTTP 200 |
| Throughput | ~90 tok/s generation, ~1700 tok/s prefill |
`finish_reason=stop` on simple cases. Complex images may return `length` —
**always check `finish_reason`, not just HTTP 200**.
## How it was made
1. **Text tower** — MiniCPM-V 4.7 reuses the Qwen3.5-0.8B architecture with
`vocab_size=248094` and an independent `lm_head`. Extraction + zoo
`Qwen3_5` `export_llm.py` / `export_rknn.py`.
The **top-level** `tie_word_embeddings` must be set to `false` too —
otherwise HF `tie_weights()` silently binds the 254M-parameter head to the
embedding and the model emits word salad.
2. **Vision tower** — `vision_tower` + `merger` exported to ONNX with
`transformers>=5.18` (`MiniCPMV4_7ForConditionalGeneration`), then RKNN with
`uint8 NHWC` input and `mean/std = 0.5*255`.
Output is 3D `[1, N, 1024]`; a 2D `[N,1024]` export is mis-parsed as
"N images of 1024 tokens".
3. **196-token adapter** — nearest-neighbour resample 8×8 → 14×14 inside the
graph (Gather/Reshape only, no `Resize` op). This changes **graph structure
only**: the `.weight` SHA256 is byte-identical to the unadapted 64-token
build, because no weights are touched.
See [`convert/README.md`](convert/README.md) for the recipe and
[`docs/DEPLOY.md`](docs/DEPLOY.md) §5 for the pitfalls.
## Limitations
- **Not equivalent to the original 4.7** (196-token adapter, see the warning above).
- canvas mRoPE, multi-resolution tiling, high-resolution OCR and dense
multi-object counting were **not verified**.
- OCR is "characters readable" at best (`RK1828` → `RK₁₈₂₈`).
- Only 448×448 single-image input was tested.
- 1B model with a `<think>` chain: raising `max_tokens` is **not** monotonically
safer (256 worked on one OCR case where 512 stalled inside `<think>`).
- Known upstream issue: on ASR device error, NPU memory can leak
(`mem_sync ret=-10` retry loop); workaround is to kill the ASR process.
## Checksums
```bash
sha256sum -c SHA256SUMS
```
| File | SHA256 |
| --- | --- |
| `MiniCPM-V-4.7-1B-llm.rknn` | `948e487bff9b2324b3016c665aa071feff05ac735e078c20ecb4a0ac7b825a1b` |
| `MiniCPM-V-4.7-1B-llm.weight` | `72fd0b16b8016d63a7d8b92f8d159ad7ea68a783eaca89b915bd0e240ae22dab` |
| `MiniCPM-V-4.7-1B-llm.embed.bin` | `511c5794dbf3d927fb17c5efc0052fababafdd35185be2961f6a855c1852b335` |
| `MiniCPM-V-4.7-1B-llm.tokenizer.gguf` | `e028b0a11710ee5e0402c3829e871c3bd5c568540fe377febdd1a515d45b1544` |
| `MiniCPM-V-4.7-1B-vision.rknn` | `9e081464857fad825281ae8778defb882424cbf5d46a37acb6de59dd5cbc99d6` |
| `MiniCPM-V-4.7-1B-vision.weight` | `9c1939dd21e511c72ef6fa6333d43d962acc067d5302ed625815abeaf1cc1c03` |
## License & attribution
**Apache-2.0**, inherited from
[`openbmb/MiniCPM-V-4.7-1B`](https://huggingface.co/openbmb/MiniCPM-V-4.7-1B).
Please credit OpenBMB for the original model. The RKNN3 conversion, the
196-token adapter and the host scripts in this repository are provided under
the same license, without warranty.
Built with [`airockchip/rknn3-model-zoo`](https://github.com/airockchip/rknn3-model-zoo) V1.1.0.
---
## 中文摘要
把 `openbmb/MiniCPM-V-4.7-1B` 转成 RKNN3,在 **RK1828** 上跑通完整图文链路
(文本塔 + 视觉塔 + `rkllm3-server --model2`),真机 5/5 带图与纯文本请求
全部 HTTP 200,NPU 独占占用 **1269 MB**。
**必须知道的取舍**:4.7 视觉塔原生只出 8×8=64 个视觉 token,而 RKNN3 runtime
的 multimodal 回调要求 `image_ph * image_pw = 14×14 = 196`。为满足契约,
导出图里加了 8×8→14×14 的最近邻重采样**适配层**(只改图结构、不改权重,
`.weight` 的 SHA256 与 64-token 版完全相同)。因此
**本产物不等价于 HF 原版 4.7**,不能拿它复现原版精度;
canvas mRoPE / 多分辨率切砖 / 高分辨 OCR / 密集计数均未验证。
| 项 | 结果 |
| --- | --- |
| 橙图问颜色 | `这张图是橙色。` |
| 黑方块 | `黑色正方形` |
| 交叉对照(排除语言先验猜答案) | 正确 |
| 纯文本 | `北京` |
| 与 ASR 共存 | 0.6B 3458 MB / 1.7B 3353 MB(A/B/A 可复现) |
| 视觉 token 注入证据 | `timings.prompt_n ≈ 218` |
仓库含 6 个转换产物 + `start.cmd` + 请求脚本 + 转换脚本 + 部署文档,
拿到即可跑;部署与十个实测踩过的坑见 [`docs/DEPLOY.md`](docs/DEPLOY.md)。
复杂图务必看 `finish_reason`,别只看 HTTP 200。
|