File size: 8,541 Bytes
fd11f30
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
---
language:
  - en
  - zh
license: apache-2.0
pipeline_tag: image-text-to-text
inference: false
tags:
  - rknn
  - rknn3
  - rk1828
  - rk1820
  - rockchip
  - npu
  - w4a16
  - vision-language-model
  - image-text-to-text
  - multimodal
  - on-device
  - edge-ai
base_model: openbmb/MiniCPM-V-4.7-1B
---

# MiniCPM-V-4.7-1B-RKNN3-RK1828

**MiniCPM-V 4.7 1B converted to RKNN3 and running end-to-end on the Rockchip
RK1828 NPU** — both the text tower and the vision tower, with real image
questions answered on device.

> **⚠️ Read this first.** The vision tower in this repository contains a
> **runtime-contract adapter**: MiniCPM-V 4.7 natively emits 8×8 = **64** visual
> tokens per 448×448 image, but the RKNN3 multimodal callback requires
> `image_ph * image_pw = 14×14 = **196**` tokens. A nearest-neighbour 8×8 → 14×14
> resample was inserted into the exported graph to satisfy that contract.
> **This build is therefore NOT equivalent to the Hugging Face original
> MiniCPM-V 4.7 1B.** Do not use it to benchmark or reproduce the original
> model's accuracy. See [How it was made](#how-it-was-made).

This repository is not just six binary files. It includes the launch script,
request examples, conversion scripts and a deployment guide, so that the model
can actually be run and reproduced.

| | |
| --- | --- |
| Base model | [`openbmb/MiniCPM-V-4.7-1B`](https://huggingface.co/openbmb/MiniCPM-V-4.7-1B) (Apache-2.0) |
| Hardware | RK1828 / RK1820 NPU (tested `RM1828MC0-F`, firmware 1.1.0) |
| Host | Windows PC or RK3588 / RK3576 board; **Ubuntu x86 cannot drive the module directly** |
| Runtime | `rknn3_110` (`rkllm3-server`), versions must match toolkit / proxy / firmware |
| Quantization | `w4a16`, `normal`, `group32`, 8 cores; head `w6a16` |
| NPU footprint | **1269 MB** (exclusive differential measurement) |

## What's in the repository

| Path | Contents |
| --- | --- |
| `rknn/` | 6 conversion artifacts + `start.cmd` (1.36 GiB) |
| `host/` | `send_vlm_req.cmd`, two ready-made request bodies, pure-stdlib test-image generator |
| `docs/DEPLOY.md` | Deployment, API contract, 10 pitfalls actually hit, verified results |
| `convert/` | Portable copies of the two vision-export scripts + the conversion recipe |
| `SHA256SUMS` | Checksums of every file above |

## Quick start

```text
<root>\
  rknn\                     <- this repo
  runtime\rknn3_110\bin\    <- rkllm3-server.exe
  runtime\rknn3_110\lib\    <- DLLs
```

```cmd
cd /d <root>\rknn
start.cmd                          :: defaults 0.0.0.0:8899 (RK_HOST / RK_PORT override)
cd /d <root>\host
send_vlm_req.cmd orange            :: HTTP=200, response in resp_orange.json
```

The server is ready **only when a real inference returns** — the weight-sync log
can sit still for ~2 minutes. Do not poll `rknn-smi` during loading.

Full details (including the Session-1 requirement, proxy reuse and DLL/PATH
rules) are in [`docs/DEPLOY.md`](docs/DEPLOY.md).

## API

OpenAI-compatible `POST /v1/chat/completions`. `content` **must be an array**;
the server constructs the image placeholder tokens itself:

```bash
curl -s http://127.0.0.1:8899/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":[
        {"type":"text","text":"图里是什么?"},
        {"type":"image_url","image_url":{"url":"data:image/png;base64,<BASE64>"}}
      ]}],"max_tokens":512}'
```

Do **not** put `<image>` / `<|image_pad|>` into the text. Use `curl.exe
--max-time`; PowerShell `Invoke-WebRequest` hangs on this server.

A correct response has `timings.prompt_n ≈ 218` = 196 visual tokens + text
tokens — that is the proof the vision embeddings really entered the LLM.

## Verified on device

| Check | Result |
| --- | --- |
| Orange image → colour | `这张图是橙色。` |
| Black square → shape/colour | `黑色正方形` |
| Cross-checks (orange asked for shape; square asked for colour) | correct — not language-prior guessing |
| Pure text (capital of China) | `北京` |
| Chinese ASR + this VLM coexisting | 3458 MB (0.6B ASR) / 3353 MB (1.7B ASR), all requests HTTP 200 |
| Throughput | ~90 tok/s generation, ~1700 tok/s prefill |

`finish_reason=stop` on simple cases. Complex images may return `length` —
**always check `finish_reason`, not just HTTP 200**.

## How it was made

1. **Text tower** — MiniCPM-V 4.7 reuses the Qwen3.5-0.8B architecture with
   `vocab_size=248094` and an independent `lm_head`. Extraction + zoo
   `Qwen3_5` `export_llm.py` / `export_rknn.py`.
   The **top-level** `tie_word_embeddings` must be set to `false` too —
   otherwise HF `tie_weights()` silently binds the 254M-parameter head to the
   embedding and the model emits word salad.
2. **Vision tower** — `vision_tower` + `merger` exported to ONNX with
   `transformers>=5.18` (`MiniCPMV4_7ForConditionalGeneration`), then RKNN with
   `uint8 NHWC` input and `mean/std = 0.5*255`.
   Output is 3D `[1, N, 1024]`; a 2D `[N,1024]` export is mis-parsed as
   "N images of 1024 tokens".
3. **196-token adapter** — nearest-neighbour resample 8×8 → 14×14 inside the
   graph (Gather/Reshape only, no `Resize` op). This changes **graph structure
   only**: the `.weight` SHA256 is byte-identical to the unadapted 64-token
   build, because no weights are touched.

See [`convert/README.md`](convert/README.md) for the recipe and
[`docs/DEPLOY.md`](docs/DEPLOY.md) §5 for the pitfalls.

## Limitations

- **Not equivalent to the original 4.7** (196-token adapter, see the warning above).
- canvas mRoPE, multi-resolution tiling, high-resolution OCR and dense
  multi-object counting were **not verified**.
- OCR is "characters readable" at best (`RK1828` → `RK₁₈₂₈`).
- Only 448×448 single-image input was tested.
- 1B model with a `<think>` chain: raising `max_tokens` is **not** monotonically
  safer (256 worked on one OCR case where 512 stalled inside `<think>`).
- Known upstream issue: on ASR device error, NPU memory can leak
  (`mem_sync ret=-10` retry loop); workaround is to kill the ASR process.

## Checksums

```bash
sha256sum -c SHA256SUMS
```

| File | SHA256 |
| --- | --- |
| `MiniCPM-V-4.7-1B-llm.rknn` | `948e487bff9b2324b3016c665aa071feff05ac735e078c20ecb4a0ac7b825a1b` |
| `MiniCPM-V-4.7-1B-llm.weight` | `72fd0b16b8016d63a7d8b92f8d159ad7ea68a783eaca89b915bd0e240ae22dab` |
| `MiniCPM-V-4.7-1B-llm.embed.bin` | `511c5794dbf3d927fb17c5efc0052fababafdd35185be2961f6a855c1852b335` |
| `MiniCPM-V-4.7-1B-llm.tokenizer.gguf` | `e028b0a11710ee5e0402c3829e871c3bd5c568540fe377febdd1a515d45b1544` |
| `MiniCPM-V-4.7-1B-vision.rknn` | `9e081464857fad825281ae8778defb882424cbf5d46a37acb6de59dd5cbc99d6` |
| `MiniCPM-V-4.7-1B-vision.weight` | `9c1939dd21e511c72ef6fa6333d43d962acc067d5302ed625815abeaf1cc1c03` |

## License & attribution

**Apache-2.0**, inherited from
[`openbmb/MiniCPM-V-4.7-1B`](https://huggingface.co/openbmb/MiniCPM-V-4.7-1B).
Please credit OpenBMB for the original model. The RKNN3 conversion, the
196-token adapter and the host scripts in this repository are provided under
the same license, without warranty.

Built with [`airockchip/rknn3-model-zoo`](https://github.com/airockchip/rknn3-model-zoo) V1.1.0.

---

## 中文摘要

把 `openbmb/MiniCPM-V-4.7-1B` 转成 RKNN3,在 **RK1828** 上跑通完整图文链路
(文本塔 + 视觉塔 + `rkllm3-server --model2`),真机 5/5 带图与纯文本请求
全部 HTTP 200,NPU 独占占用 **1269 MB**。

**必须知道的取舍**:4.7 视觉塔原生只出 8×8=64 个视觉 token,而 RKNN3 runtime
的 multimodal 回调要求 `image_ph * image_pw = 14×14 = 196`。为满足契约,
导出图里加了 8×8→14×14 的最近邻重采样**适配层**(只改图结构、不改权重,
`.weight` 的 SHA256 与 64-token 版完全相同)。因此
**本产物不等价于 HF 原版 4.7**,不能拿它复现原版精度;
canvas mRoPE / 多分辨率切砖 / 高分辨 OCR / 密集计数均未验证。

| 项 | 结果 |
| --- | --- |
| 橙图问颜色 | `这张图是橙色。` |
| 黑方块 | `黑色正方形` |
| 交叉对照(排除语言先验猜答案) | 正确 |
| 纯文本 | `北京` |
| 与 ASR 共存 | 0.6B 3458 MB / 1.7B 3353 MB(A/B/A 可复现) |
| 视觉 token 注入证据 | `timings.prompt_n ≈ 218` |

仓库含 6 个转换产物 + `start.cmd` + 请求脚本 + 转换脚本 + 部署文档,
拿到即可跑;部署与十个实测踩过的坑见 [`docs/DEPLOY.md`](docs/DEPLOY.md)。
复杂图务必看 `finish_reason`,别只看 HTTP 200。