jj90's picture
Add model card, deploy guide, host + conversion scripts
fd11f30 verified
|
Raw History Blame Contribute Delete
8.54 kB
metadata
language:
  - en
  - zh
license: apache-2.0
pipeline_tag: image-text-to-text
inference: false
tags:
  - rknn
  - rknn3
  - rk1828
  - rk1820
  - rockchip
  - npu
  - w4a16
  - vision-language-model
  - image-text-to-text
  - multimodal
  - on-device
  - edge-ai
base_model: openbmb/MiniCPM-V-4.7-1B

MiniCPM-V-4.7-1B-RKNN3-RK1828

MiniCPM-V 4.7 1B converted to RKNN3 and running end-to-end on the Rockchip RK1828 NPU — both the text tower and the vision tower, with real image questions answered on device.

⚠️ Read this first. The vision tower in this repository contains a runtime-contract adapter: MiniCPM-V 4.7 natively emits 8×8 = 64 visual tokens per 448×448 image, but the RKNN3 multimodal callback requires image_ph * image_pw = 14×14 = **196** tokens. A nearest-neighbour 8×8 → 14×14 resample was inserted into the exported graph to satisfy that contract. This build is therefore NOT equivalent to the Hugging Face original MiniCPM-V 4.7 1B. Do not use it to benchmark or reproduce the original model's accuracy. See How it was made.

This repository is not just six binary files. It includes the launch script, request examples, conversion scripts and a deployment guide, so that the model can actually be run and reproduced.

Base model openbmb/MiniCPM-V-4.7-1B (Apache-2.0)
Hardware RK1828 / RK1820 NPU (tested RM1828MC0-F, firmware 1.1.0)
Host Windows PC or RK3588 / RK3576 board; Ubuntu x86 cannot drive the module directly
Runtime rknn3_110 (rkllm3-server), versions must match toolkit / proxy / firmware
Quantization w4a16, normal, group32, 8 cores; head w6a16
NPU footprint 1269 MB (exclusive differential measurement)

What's in the repository

Path Contents
rknn/ 6 conversion artifacts + start.cmd (1.36 GiB)
host/ send_vlm_req.cmd, two ready-made request bodies, pure-stdlib test-image generator
docs/DEPLOY.md Deployment, API contract, 10 pitfalls actually hit, verified results
convert/ Portable copies of the two vision-export scripts + the conversion recipe
SHA256SUMS Checksums of every file above

Quick start

<root>\
  rknn\                     <- this repo
  runtime\rknn3_110\bin\    <- rkllm3-server.exe
  runtime\rknn3_110\lib\    <- DLLs
cd /d <root>\rknn
start.cmd                          :: defaults 0.0.0.0:8899 (RK_HOST / RK_PORT override)
cd /d <root>\host
send_vlm_req.cmd orange            :: HTTP=200, response in resp_orange.json

The server is ready only when a real inference returns — the weight-sync log can sit still for ~2 minutes. Do not poll rknn-smi during loading.

Full details (including the Session-1 requirement, proxy reuse and DLL/PATH rules) are in docs/DEPLOY.md.

API

OpenAI-compatible POST /v1/chat/completions. content must be an array; the server constructs the image placeholder tokens itself:

curl -s http://127.0.0.1:8899/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":[
        {"type":"text","text":"图里是什么?"},
        {"type":"image_url","image_url":{"url":"data:image/png;base64,<BASE64>"}}
      ]}],"max_tokens":512}'

Do not put <image> / <|image_pad|> into the text. Use curl.exe --max-time; PowerShell Invoke-WebRequest hangs on this server.

A correct response has timings.prompt_n ≈ 218 = 196 visual tokens + text tokens — that is the proof the vision embeddings really entered the LLM.

Verified on device

Check Result
Orange image → colour 这张图是橙色。
Black square → shape/colour 黑色正方形
Cross-checks (orange asked for shape; square asked for colour) correct — not language-prior guessing
Pure text (capital of China) 北京
Chinese ASR + this VLM coexisting 3458 MB (0.6B ASR) / 3353 MB (1.7B ASR), all requests HTTP 200
Throughput ~90 tok/s generation, ~1700 tok/s prefill

finish_reason=stop on simple cases. Complex images may return length — always check finish_reason, not just HTTP 200.

How it was made

  1. Text tower — MiniCPM-V 4.7 reuses the Qwen3.5-0.8B architecture with vocab_size=248094 and an independent lm_head. Extraction + zoo Qwen3_5 export_llm.py / export_rknn.py. The top-level tie_word_embeddings must be set to false too — otherwise HF tie_weights() silently binds the 254M-parameter head to the embedding and the model emits word salad.
  2. Vision tower — vision_tower + merger exported to ONNX with transformers>=5.18 (MiniCPMV4_7ForConditionalGeneration), then RKNN with uint8 NHWC input and mean/std = 0.5*255. Output is 3D [1, N, 1024]; a 2D [N,1024] export is mis-parsed as "N images of 1024 tokens".
  3. 196-token adapter — nearest-neighbour resample 8×8 → 14×14 inside the graph (Gather/Reshape only, no Resize op). This changes graph structure only: the .weight SHA256 is byte-identical to the unadapted 64-token build, because no weights are touched.

See convert/README.md for the recipe and docs/DEPLOY.md §5 for the pitfalls.

Limitations

  • Not equivalent to the original 4.7 (196-token adapter, see the warning above).
  • canvas mRoPE, multi-resolution tiling, high-resolution OCR and dense multi-object counting were not verified.
  • OCR is "characters readable" at best (RK1828 → RK₁₈₂₈).
  • Only 448×448 single-image input was tested.
  • 1B model with a <think> chain: raising max_tokens is not monotonically safer (256 worked on one OCR case where 512 stalled inside <think>).
  • Known upstream issue: on ASR device error, NPU memory can leak (mem_sync ret=-10 retry loop); workaround is to kill the ASR process.

Checksums

sha256sum -c SHA256SUMS
File SHA256
MiniCPM-V-4.7-1B-llm.rknn 948e487bff9b2324b3016c665aa071feff05ac735e078c20ecb4a0ac7b825a1b
MiniCPM-V-4.7-1B-llm.weight 72fd0b16b8016d63a7d8b92f8d159ad7ea68a783eaca89b915bd0e240ae22dab
MiniCPM-V-4.7-1B-llm.embed.bin 511c5794dbf3d927fb17c5efc0052fababafdd35185be2961f6a855c1852b335
MiniCPM-V-4.7-1B-llm.tokenizer.gguf e028b0a11710ee5e0402c3829e871c3bd5c568540fe377febdd1a515d45b1544
MiniCPM-V-4.7-1B-vision.rknn 9e081464857fad825281ae8778defb882424cbf5d46a37acb6de59dd5cbc99d6
MiniCPM-V-4.7-1B-vision.weight 9c1939dd21e511c72ef6fa6333d43d962acc067d5302ed625815abeaf1cc1c03

License & attribution

Apache-2.0, inherited from openbmb/MiniCPM-V-4.7-1B. Please credit OpenBMB for the original model. The RKNN3 conversion, the 196-token adapter and the host scripts in this repository are provided under the same license, without warranty.

Built with airockchip/rknn3-model-zoo V1.1.0.


中文摘要

把 openbmb/MiniCPM-V-4.7-1B 转成 RKNN3,在 RK1828 上跑通完整图文链路 (文本塔 + 视觉塔 + rkllm3-server --model2),真机 5/5 带图与纯文本请求 全部 HTTP 200,NPU 独占占用 1269 MB。

必须知道的取舍:4.7 视觉塔原生只出 8×8=64 个视觉 token,而 RKNN3 runtime 的 multimodal 回调要求 image_ph * image_pw = 14×14 = 196。为满足契约, 导出图里加了 8×8→14×14 的最近邻重采样适配层(只改图结构、不改权重, .weight 的 SHA256 与 64-token 版完全相同)。因此 本产物不等价于 HF 原版 4.7,不能拿它复现原版精度; canvas mRoPE / 多分辨率切砖 / 高分辨 OCR / 密集计数均未验证。

项 结果
橙图问颜色 这张图是橙色。
黑方块 黑色正方形
交叉对照(排除语言先验猜答案) 正确
纯文本 北京
与 ASR 共存 0.6B 3458 MB / 1.7B 3353 MB(A/B/A 可复现)
视觉 token 注入证据 timings.prompt_n ≈ 218

仓库含 6 个转换产物 + start.cmd + 请求脚本 + 转换脚本 + 部署文档, 拿到即可跑;部署与十个实测踩过的坑见 docs/DEPLOY.md。 复杂图务必看 finish_reason,别只看 HTTP 200。