jj90's picture
Add model card, deploy guide, host + conversion scripts
fd11f30 verified
|
Raw History Blame Contribute Delete
8.54 kB
---
language:
- en
- zh
license: apache-2.0
pipeline_tag: image-text-to-text
inference: false
tags:
- rknn
- rknn3
- rk1828
- rk1820
- rockchip
- npu
- w4a16
- vision-language-model
- image-text-to-text
- multimodal
- on-device
- edge-ai
base_model: openbmb/MiniCPM-V-4.7-1B
---
# MiniCPM-V-4.7-1B-RKNN3-RK1828
**MiniCPM-V 4.7 1B converted to RKNN3 and running end-to-end on the Rockchip
RK1828 NPU** — both the text tower and the vision tower, with real image
questions answered on device.
> **⚠️ Read this first.** The vision tower in this repository contains a
> **runtime-contract adapter**: MiniCPM-V 4.7 natively emits 8×8 = **64** visual
> tokens per 448×448 image, but the RKNN3 multimodal callback requires
> `image_ph * image_pw = 14×14 = **196**` tokens. A nearest-neighbour 8×8 → 14×14
> resample was inserted into the exported graph to satisfy that contract.
> **This build is therefore NOT equivalent to the Hugging Face original
> MiniCPM-V 4.7 1B.** Do not use it to benchmark or reproduce the original
> model's accuracy. See [How it was made](#how-it-was-made).
This repository is not just six binary files. It includes the launch script,
request examples, conversion scripts and a deployment guide, so that the model
can actually be run and reproduced.
| | |
| --- | --- |
| Base model | [`openbmb/MiniCPM-V-4.7-1B`](https://huggingface.co/openbmb/MiniCPM-V-4.7-1B) (Apache-2.0) |
| Hardware | RK1828 / RK1820 NPU (tested `RM1828MC0-F`, firmware 1.1.0) |
| Host | Windows PC or RK3588 / RK3576 board; **Ubuntu x86 cannot drive the module directly** |
| Runtime | `rknn3_110` (`rkllm3-server`), versions must match toolkit / proxy / firmware |
| Quantization | `w4a16`, `normal`, `group32`, 8 cores; head `w6a16` |
| NPU footprint | **1269 MB** (exclusive differential measurement) |
## What's in the repository
| Path | Contents |
| --- | --- |
| `rknn/` | 6 conversion artifacts + `start.cmd` (1.36 GiB) |
| `host/` | `send_vlm_req.cmd`, two ready-made request bodies, pure-stdlib test-image generator |
| `docs/DEPLOY.md` | Deployment, API contract, 10 pitfalls actually hit, verified results |
| `convert/` | Portable copies of the two vision-export scripts + the conversion recipe |
| `SHA256SUMS` | Checksums of every file above |
## Quick start
```text
<root>\
rknn\ <- this repo
runtime\rknn3_110\bin\ <- rkllm3-server.exe
runtime\rknn3_110\lib\ <- DLLs
```
```cmd
cd /d <root>\rknn
start.cmd :: defaults 0.0.0.0:8899 (RK_HOST / RK_PORT override)
cd /d <root>\host
send_vlm_req.cmd orange :: HTTP=200, response in resp_orange.json
```
The server is ready **only when a real inference returns** — the weight-sync log
can sit still for ~2 minutes. Do not poll `rknn-smi` during loading.
Full details (including the Session-1 requirement, proxy reuse and DLL/PATH
rules) are in [`docs/DEPLOY.md`](docs/DEPLOY.md).
## API
OpenAI-compatible `POST /v1/chat/completions`. `content` **must be an array**;
the server constructs the image placeholder tokens itself:
```bash
curl -s http://127.0.0.1:8899/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"messages":[{"role":"user","content":[
{"type":"text","text":"图里是什么?"},
{"type":"image_url","image_url":{"url":"data:image/png;base64,<BASE64>"}}
]}],"max_tokens":512}'
```
Do **not** put `<image>` / `<|image_pad|>` into the text. Use `curl.exe
--max-time`; PowerShell `Invoke-WebRequest` hangs on this server.
A correct response has `timings.prompt_n ≈ 218` = 196 visual tokens + text
tokens — that is the proof the vision embeddings really entered the LLM.
## Verified on device
| Check | Result |
| --- | --- |
| Orange image → colour | `这张图是橙色。` |
| Black square → shape/colour | `黑色正方形` |
| Cross-checks (orange asked for shape; square asked for colour) | correct — not language-prior guessing |
| Pure text (capital of China) | `北京` |
| Chinese ASR + this VLM coexisting | 3458 MB (0.6B ASR) / 3353 MB (1.7B ASR), all requests HTTP 200 |
| Throughput | ~90 tok/s generation, ~1700 tok/s prefill |
`finish_reason=stop` on simple cases. Complex images may return `length` —
**always check `finish_reason`, not just HTTP 200**.
## How it was made
1. **Text tower** — MiniCPM-V 4.7 reuses the Qwen3.5-0.8B architecture with
`vocab_size=248094` and an independent `lm_head`. Extraction + zoo
`Qwen3_5` `export_llm.py` / `export_rknn.py`.
The **top-level** `tie_word_embeddings` must be set to `false` too —
otherwise HF `tie_weights()` silently binds the 254M-parameter head to the
embedding and the model emits word salad.
2. **Vision tower** — `vision_tower` + `merger` exported to ONNX with
`transformers>=5.18` (`MiniCPMV4_7ForConditionalGeneration`), then RKNN with
`uint8 NHWC` input and `mean/std = 0.5*255`.
Output is 3D `[1, N, 1024]`; a 2D `[N,1024]` export is mis-parsed as
"N images of 1024 tokens".
3. **196-token adapter** — nearest-neighbour resample 8×8 → 14×14 inside the
graph (Gather/Reshape only, no `Resize` op). This changes **graph structure
only**: the `.weight` SHA256 is byte-identical to the unadapted 64-token
build, because no weights are touched.
See [`convert/README.md`](convert/README.md) for the recipe and
[`docs/DEPLOY.md`](docs/DEPLOY.md) §5 for the pitfalls.
## Limitations
- **Not equivalent to the original 4.7** (196-token adapter, see the warning above).
- canvas mRoPE, multi-resolution tiling, high-resolution OCR and dense
multi-object counting were **not verified**.
- OCR is "characters readable" at best (`RK1828` → `RK₁₈₂₈`).
- Only 448×448 single-image input was tested.
- 1B model with a `<think>` chain: raising `max_tokens` is **not** monotonically
safer (256 worked on one OCR case where 512 stalled inside `<think>`).
- Known upstream issue: on ASR device error, NPU memory can leak
(`mem_sync ret=-10` retry loop); workaround is to kill the ASR process.
## Checksums
```bash
sha256sum -c SHA256SUMS
```
| File | SHA256 |
| --- | --- |
| `MiniCPM-V-4.7-1B-llm.rknn` | `948e487bff9b2324b3016c665aa071feff05ac735e078c20ecb4a0ac7b825a1b` |
| `MiniCPM-V-4.7-1B-llm.weight` | `72fd0b16b8016d63a7d8b92f8d159ad7ea68a783eaca89b915bd0e240ae22dab` |
| `MiniCPM-V-4.7-1B-llm.embed.bin` | `511c5794dbf3d927fb17c5efc0052fababafdd35185be2961f6a855c1852b335` |
| `MiniCPM-V-4.7-1B-llm.tokenizer.gguf` | `e028b0a11710ee5e0402c3829e871c3bd5c568540fe377febdd1a515d45b1544` |
| `MiniCPM-V-4.7-1B-vision.rknn` | `9e081464857fad825281ae8778defb882424cbf5d46a37acb6de59dd5cbc99d6` |
| `MiniCPM-V-4.7-1B-vision.weight` | `9c1939dd21e511c72ef6fa6333d43d962acc067d5302ed625815abeaf1cc1c03` |
## License & attribution
**Apache-2.0**, inherited from
[`openbmb/MiniCPM-V-4.7-1B`](https://huggingface.co/openbmb/MiniCPM-V-4.7-1B).
Please credit OpenBMB for the original model. The RKNN3 conversion, the
196-token adapter and the host scripts in this repository are provided under
the same license, without warranty.
Built with [`airockchip/rknn3-model-zoo`](https://github.com/airockchip/rknn3-model-zoo) V1.1.0.
---
## 中文摘要
把 `openbmb/MiniCPM-V-4.7-1B` 转成 RKNN3,在 **RK1828** 上跑通完整图文链路
(文本塔 + 视觉塔 + `rkllm3-server --model2`),真机 5/5 带图与纯文本请求
全部 HTTP 200,NPU 独占占用 **1269 MB**。
**必须知道的取舍**:4.7 视觉塔原生只出 8×8=64 个视觉 token,而 RKNN3 runtime
的 multimodal 回调要求 `image_ph * image_pw = 14×14 = 196`。为满足契约,
导出图里加了 8×8→14×14 的最近邻重采样**适配层**(只改图结构、不改权重,
`.weight` 的 SHA256 与 64-token 版完全相同)。因此
**本产物不等价于 HF 原版 4.7**,不能拿它复现原版精度;
canvas mRoPE / 多分辨率切砖 / 高分辨 OCR / 密集计数均未验证。
| 项 | 结果 |
| --- | --- |
| 橙图问颜色 | `这张图是橙色。` |
| 黑方块 | `黑色正方形` |
| 交叉对照(排除语言先验猜答案) | 正确 |
| 纯文本 | `北京` |
| 与 ASR 共存 | 0.6B 3458 MB / 1.7B 3353 MB(A/B/A 可复现) |
| 视觉 token 注入证据 | `timings.prompt_n ≈ 218` |
仓库含 6 个转换产物 + `start.cmd` + 请求脚本 + 转换脚本 + 部署文档,
拿到即可跑;部署与十个实测踩过的坑见 [`docs/DEPLOY.md`](docs/DEPLOY.md)。
复杂图务必看 `finish_reason`,别只看 HTTP 200。