--- language: - en - zh license: apache-2.0 pipeline_tag: image-text-to-text inference: false tags: - rknn - rknn3 - rk1828 - rk1820 - rockchip - npu - w4a16 - vision-language-model - image-text-to-text - multimodal - on-device - edge-ai base_model: openbmb/MiniCPM-V-4.7-1B --- # MiniCPM-V-4.7-1B-RKNN3-RK1828 **MiniCPM-V 4.7 1B converted to RKNN3 and running end-to-end on the Rockchip RK1828 NPU** — both the text tower and the vision tower, with real image questions answered on device. > **⚠️ Read this first.** The vision tower in this repository contains a > **runtime-contract adapter**: MiniCPM-V 4.7 natively emits 8×8 = **64** visual > tokens per 448×448 image, but the RKNN3 multimodal callback requires > `image_ph * image_pw = 14×14 = **196**` tokens. A nearest-neighbour 8×8 → 14×14 > resample was inserted into the exported graph to satisfy that contract. > **This build is therefore NOT equivalent to the Hugging Face original > MiniCPM-V 4.7 1B.** Do not use it to benchmark or reproduce the original > model's accuracy. See [How it was made](#how-it-was-made). This repository is not just six binary files. It includes the launch script, request examples, conversion scripts and a deployment guide, so that the model can actually be run and reproduced. | | | | --- | --- | | Base model | [`openbmb/MiniCPM-V-4.7-1B`](https://huggingface.co/openbmb/MiniCPM-V-4.7-1B) (Apache-2.0) | | Hardware | RK1828 / RK1820 NPU (tested `RM1828MC0-F`, firmware 1.1.0) | | Host | Windows PC or RK3588 / RK3576 board; **Ubuntu x86 cannot drive the module directly** | | Runtime | `rknn3_110` (`rkllm3-server`), versions must match toolkit / proxy / firmware | | Quantization | `w4a16`, `normal`, `group32`, 8 cores; head `w6a16` | | NPU footprint | **1269 MB** (exclusive differential measurement) | ## What's in the repository | Path | Contents | | --- | --- | | `rknn/` | 6 conversion artifacts + `start.cmd` (1.36 GiB) | | `host/` | `send_vlm_req.cmd`, two ready-made request bodies, pure-stdlib test-image generator | | `docs/DEPLOY.md` | Deployment, API contract, 10 pitfalls actually hit, verified results | | `convert/` | Portable copies of the two vision-export scripts + the conversion recipe | | `SHA256SUMS` | Checksums of every file above | ## Quick start ```text \ rknn\ <- this repo runtime\rknn3_110\bin\ <- rkllm3-server.exe runtime\rknn3_110\lib\ <- DLLs ``` ```cmd cd /d \rknn start.cmd :: defaults 0.0.0.0:8899 (RK_HOST / RK_PORT override) cd /d \host send_vlm_req.cmd orange :: HTTP=200, response in resp_orange.json ``` The server is ready **only when a real inference returns** — the weight-sync log can sit still for ~2 minutes. Do not poll `rknn-smi` during loading. Full details (including the Session-1 requirement, proxy reuse and DLL/PATH rules) are in [`docs/DEPLOY.md`](docs/DEPLOY.md). ## API OpenAI-compatible `POST /v1/chat/completions`. `content` **must be an array**; the server constructs the image placeholder tokens itself: ```bash curl -s http://127.0.0.1:8899/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"messages":[{"role":"user","content":[ {"type":"text","text":"图里是什么?"}, {"type":"image_url","image_url":{"url":"data:image/png;base64,"}} ]}],"max_tokens":512}' ``` Do **not** put `` / `<|image_pad|>` into the text. Use `curl.exe --max-time`; PowerShell `Invoke-WebRequest` hangs on this server. A correct response has `timings.prompt_n ≈ 218` = 196 visual tokens + text tokens — that is the proof the vision embeddings really entered the LLM. ## Verified on device | Check | Result | | --- | --- | | Orange image → colour | `这张图是橙色。` | | Black square → shape/colour | `黑色正方形` | | Cross-checks (orange asked for shape; square asked for colour) | correct — not language-prior guessing | | Pure text (capital of China) | `北京` | | Chinese ASR + this VLM coexisting | 3458 MB (0.6B ASR) / 3353 MB (1.7B ASR), all requests HTTP 200 | | Throughput | ~90 tok/s generation, ~1700 tok/s prefill | `finish_reason=stop` on simple cases. Complex images may return `length` — **always check `finish_reason`, not just HTTP 200**. ## How it was made 1. **Text tower** — MiniCPM-V 4.7 reuses the Qwen3.5-0.8B architecture with `vocab_size=248094` and an independent `lm_head`. Extraction + zoo `Qwen3_5` `export_llm.py` / `export_rknn.py`. The **top-level** `tie_word_embeddings` must be set to `false` too — otherwise HF `tie_weights()` silently binds the 254M-parameter head to the embedding and the model emits word salad. 2. **Vision tower** — `vision_tower` + `merger` exported to ONNX with `transformers>=5.18` (`MiniCPMV4_7ForConditionalGeneration`), then RKNN with `uint8 NHWC` input and `mean/std = 0.5*255`. Output is 3D `[1, N, 1024]`; a 2D `[N,1024]` export is mis-parsed as "N images of 1024 tokens". 3. **196-token adapter** — nearest-neighbour resample 8×8 → 14×14 inside the graph (Gather/Reshape only, no `Resize` op). This changes **graph structure only**: the `.weight` SHA256 is byte-identical to the unadapted 64-token build, because no weights are touched. See [`convert/README.md`](convert/README.md) for the recipe and [`docs/DEPLOY.md`](docs/DEPLOY.md) §5 for the pitfalls. ## Limitations - **Not equivalent to the original 4.7** (196-token adapter, see the warning above). - canvas mRoPE, multi-resolution tiling, high-resolution OCR and dense multi-object counting were **not verified**. - OCR is "characters readable" at best (`RK1828` → `RK₁₈₂₈`). - Only 448×448 single-image input was tested. - 1B model with a `` chain: raising `max_tokens` is **not** monotonically safer (256 worked on one OCR case where 512 stalled inside ``). - Known upstream issue: on ASR device error, NPU memory can leak (`mem_sync ret=-10` retry loop); workaround is to kill the ASR process. ## Checksums ```bash sha256sum -c SHA256SUMS ``` | File | SHA256 | | --- | --- | | `MiniCPM-V-4.7-1B-llm.rknn` | `948e487bff9b2324b3016c665aa071feff05ac735e078c20ecb4a0ac7b825a1b` | | `MiniCPM-V-4.7-1B-llm.weight` | `72fd0b16b8016d63a7d8b92f8d159ad7ea68a783eaca89b915bd0e240ae22dab` | | `MiniCPM-V-4.7-1B-llm.embed.bin` | `511c5794dbf3d927fb17c5efc0052fababafdd35185be2961f6a855c1852b335` | | `MiniCPM-V-4.7-1B-llm.tokenizer.gguf` | `e028b0a11710ee5e0402c3829e871c3bd5c568540fe377febdd1a515d45b1544` | | `MiniCPM-V-4.7-1B-vision.rknn` | `9e081464857fad825281ae8778defb882424cbf5d46a37acb6de59dd5cbc99d6` | | `MiniCPM-V-4.7-1B-vision.weight` | `9c1939dd21e511c72ef6fa6333d43d962acc067d5302ed625815abeaf1cc1c03` | ## License & attribution **Apache-2.0**, inherited from [`openbmb/MiniCPM-V-4.7-1B`](https://huggingface.co/openbmb/MiniCPM-V-4.7-1B). Please credit OpenBMB for the original model. The RKNN3 conversion, the 196-token adapter and the host scripts in this repository are provided under the same license, without warranty. Built with [`airockchip/rknn3-model-zoo`](https://github.com/airockchip/rknn3-model-zoo) V1.1.0. --- ## 中文摘要 把 `openbmb/MiniCPM-V-4.7-1B` 转成 RKNN3,在 **RK1828** 上跑通完整图文链路 (文本塔 + 视觉塔 + `rkllm3-server --model2`),真机 5/5 带图与纯文本请求 全部 HTTP 200,NPU 独占占用 **1269 MB**。 **必须知道的取舍**:4.7 视觉塔原生只出 8×8=64 个视觉 token,而 RKNN3 runtime 的 multimodal 回调要求 `image_ph * image_pw = 14×14 = 196`。为满足契约, 导出图里加了 8×8→14×14 的最近邻重采样**适配层**(只改图结构、不改权重, `.weight` 的 SHA256 与 64-token 版完全相同)。因此 **本产物不等价于 HF 原版 4.7**,不能拿它复现原版精度; canvas mRoPE / 多分辨率切砖 / 高分辨 OCR / 密集计数均未验证。 | 项 | 结果 | | --- | --- | | 橙图问颜色 | `这张图是橙色。` | | 黑方块 | `黑色正方形` | | 交叉对照(排除语言先验猜答案) | 正确 | | 纯文本 | `北京` | | 与 ASR 共存 | 0.6B 3458 MB / 1.7B 3353 MB(A/B/A 可复现) | | 视觉 token 注入证据 | `timings.prompt_n ≈ 218` | 仓库含 6 个转换产物 + `start.cmd` + 请求脚本 + 转换脚本 + 部署文档, 拿到即可跑;部署与十个实测踩过的坑见 [`docs/DEPLOY.md`](docs/DEPLOY.md)。 复杂图务必看 `finish_reason`,别只看 HTTP 200。