jj90 commited on
Commit
fd11f30
·
verified ·
1 Parent(s): 4e3f13a

Add model card, deploy guide, host + conversion scripts

Browse files
README.md ADDED
@@ -0,0 +1,202 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ - zh
5
+ license: apache-2.0
6
+ pipeline_tag: image-text-to-text
7
+ inference: false
8
+ tags:
9
+ - rknn
10
+ - rknn3
11
+ - rk1828
12
+ - rk1820
13
+ - rockchip
14
+ - npu
15
+ - w4a16
16
+ - vision-language-model
17
+ - image-text-to-text
18
+ - multimodal
19
+ - on-device
20
+ - edge-ai
21
+ base_model: openbmb/MiniCPM-V-4.7-1B
22
+ ---
23
+
24
+ # MiniCPM-V-4.7-1B-RKNN3-RK1828
25
+
26
+ **MiniCPM-V 4.7 1B converted to RKNN3 and running end-to-end on the Rockchip
27
+ RK1828 NPU** — both the text tower and the vision tower, with real image
28
+ questions answered on device.
29
+
30
+ > **⚠️ Read this first.** The vision tower in this repository contains a
31
+ > **runtime-contract adapter**: MiniCPM-V 4.7 natively emits 8×8 = **64** visual
32
+ > tokens per 448×448 image, but the RKNN3 multimodal callback requires
33
+ > `image_ph * image_pw = 14×14 = **196**` tokens. A nearest-neighbour 8×8 → 14×14
34
+ > resample was inserted into the exported graph to satisfy that contract.
35
+ > **This build is therefore NOT equivalent to the Hugging Face original
36
+ > MiniCPM-V 4.7 1B.** Do not use it to benchmark or reproduce the original
37
+ > model's accuracy. See [How it was made](#how-it-was-made).
38
+
39
+ This repository is not just six binary files. It includes the launch script,
40
+ request examples, conversion scripts and a deployment guide, so that the model
41
+ can actually be run and reproduced.
42
+
43
+ | | |
44
+ | --- | --- |
45
+ | Base model | [`openbmb/MiniCPM-V-4.7-1B`](https://huggingface.co/openbmb/MiniCPM-V-4.7-1B) (Apache-2.0) |
46
+ | Hardware | RK1828 / RK1820 NPU (tested `RM1828MC0-F`, firmware 1.1.0) |
47
+ | Host | Windows PC or RK3588 / RK3576 board; **Ubuntu x86 cannot drive the module directly** |
48
+ | Runtime | `rknn3_110` (`rkllm3-server`), versions must match toolkit / proxy / firmware |
49
+ | Quantization | `w4a16`, `normal`, `group32`, 8 cores; head `w6a16` |
50
+ | NPU footprint | **1269 MB** (exclusive differential measurement) |
51
+
52
+ ## What's in the repository
53
+
54
+ | Path | Contents |
55
+ | --- | --- |
56
+ | `rknn/` | 6 conversion artifacts + `start.cmd` (1.36 GiB) |
57
+ | `host/` | `send_vlm_req.cmd`, two ready-made request bodies, pure-stdlib test-image generator |
58
+ | `docs/DEPLOY.md` | Deployment, API contract, 10 pitfalls actually hit, verified results |
59
+ | `convert/` | Portable copies of the two vision-export scripts + the conversion recipe |
60
+ | `SHA256SUMS` | Checksums of every file above |
61
+
62
+ ## Quick start
63
+
64
+ ```text
65
+ <root>\
66
+ rknn\ <- this repo
67
+ runtime\rknn3_110\bin\ <- rkllm3-server.exe
68
+ runtime\rknn3_110\lib\ <- DLLs
69
+ ```
70
+
71
+ ```cmd
72
+ cd /d <root>\rknn
73
+ start.cmd :: defaults 0.0.0.0:8899 (RK_HOST / RK_PORT override)
74
+ cd /d <root>\host
75
+ send_vlm_req.cmd orange :: HTTP=200, response in resp_orange.json
76
+ ```
77
+
78
+ The server is ready **only when a real inference returns** — the weight-sync log
79
+ can sit still for ~2 minutes. Do not poll `rknn-smi` during loading.
80
+
81
+ Full details (including the Session-1 requirement, proxy reuse and DLL/PATH
82
+ rules) are in [`docs/DEPLOY.md`](docs/DEPLOY.md).
83
+
84
+ ## API
85
+
86
+ OpenAI-compatible `POST /v1/chat/completions`. `content` **must be an array**;
87
+ the server constructs the image placeholder tokens itself:
88
+
89
+ ```bash
90
+ curl -s http://127.0.0.1:8899/v1/chat/completions \
91
+ -H "Content-Type: application/json" \
92
+ -d '{"messages":[{"role":"user","content":[
93
+ {"type":"text","text":"图里是什么?"},
94
+ {"type":"image_url","image_url":{"url":"data:image/png;base64,<BASE64>"}}
95
+ ]}],"max_tokens":512}'
96
+ ```
97
+
98
+ Do **not** put `<image>` / `<|image_pad|>` into the text. Use `curl.exe
99
+ --max-time`; PowerShell `Invoke-WebRequest` hangs on this server.
100
+
101
+ A correct response has `timings.prompt_n ≈ 218` = 196 visual tokens + text
102
+ tokens — that is the proof the vision embeddings really entered the LLM.
103
+
104
+ ## Verified on device
105
+
106
+ | Check | Result |
107
+ | --- | --- |
108
+ | Orange image → colour | `这张图是橙色。` |
109
+ | Black square → shape/colour | `黑色正方形` |
110
+ | Cross-checks (orange asked for shape; square asked for colour) | correct — not language-prior guessing |
111
+ | Pure text (capital of China) | `北京` |
112
+ | Chinese ASR + this VLM coexisting | 3458 MB (0.6B ASR) / 3353 MB (1.7B ASR), all requests HTTP 200 |
113
+ | Throughput | ~90 tok/s generation, ~1700 tok/s prefill |
114
+
115
+ `finish_reason=stop` on simple cases. Complex images may return `length` —
116
+ **always check `finish_reason`, not just HTTP 200**.
117
+
118
+ ## How it was made
119
+
120
+ 1. **Text tower** — MiniCPM-V 4.7 reuses the Qwen3.5-0.8B architecture with
121
+ `vocab_size=248094` and an independent `lm_head`. Extraction + zoo
122
+ `Qwen3_5` `export_llm.py` / `export_rknn.py`.
123
+ The **top-level** `tie_word_embeddings` must be set to `false` too —
124
+ otherwise HF `tie_weights()` silently binds the 254M-parameter head to the
125
+ embedding and the model emits word salad.
126
+ 2. **Vision tower** — `vision_tower` + `merger` exported to ONNX with
127
+ `transformers>=5.18` (`MiniCPMV4_7ForConditionalGeneration`), then RKNN with
128
+ `uint8 NHWC` input and `mean/std = 0.5*255`.
129
+ Output is 3D `[1, N, 1024]`; a 2D `[N,1024]` export is mis-parsed as
130
+ "N images of 1024 tokens".
131
+ 3. **196-token adapter** — nearest-neighbour resample 8×8 → 14×14 inside the
132
+ graph (Gather/Reshape only, no `Resize` op). This changes **graph structure
133
+ only**: the `.weight` SHA256 is byte-identical to the unadapted 64-token
134
+ build, because no weights are touched.
135
+
136
+ See [`convert/README.md`](convert/README.md) for the recipe and
137
+ [`docs/DEPLOY.md`](docs/DEPLOY.md) §5 for the pitfalls.
138
+
139
+ ## Limitations
140
+
141
+ - **Not equivalent to the original 4.7** (196-token adapter, see the warning above).
142
+ - canvas mRoPE, multi-resolution tiling, high-resolution OCR and dense
143
+ multi-object counting were **not verified**.
144
+ - OCR is "characters readable" at best (`RK1828` → `RK₁₈₂₈`).
145
+ - Only 448×448 single-image input was tested.
146
+ - 1B model with a `<think>` chain: raising `max_tokens` is **not** monotonically
147
+ safer (256 worked on one OCR case where 512 stalled inside `<think>`).
148
+ - Known upstream issue: on ASR device error, NPU memory can leak
149
+ (`mem_sync ret=-10` retry loop); workaround is to kill the ASR process.
150
+
151
+ ## Checksums
152
+
153
+ ```bash
154
+ sha256sum -c SHA256SUMS
155
+ ```
156
+
157
+ | File | SHA256 |
158
+ | --- | --- |
159
+ | `MiniCPM-V-4.7-1B-llm.rknn` | `948e487bff9b2324b3016c665aa071feff05ac735e078c20ecb4a0ac7b825a1b` |
160
+ | `MiniCPM-V-4.7-1B-llm.weight` | `72fd0b16b8016d63a7d8b92f8d159ad7ea68a783eaca89b915bd0e240ae22dab` |
161
+ | `MiniCPM-V-4.7-1B-llm.embed.bin` | `511c5794dbf3d927fb17c5efc0052fababafdd35185be2961f6a855c1852b335` |
162
+ | `MiniCPM-V-4.7-1B-llm.tokenizer.gguf` | `e028b0a11710ee5e0402c3829e871c3bd5c568540fe377febdd1a515d45b1544` |
163
+ | `MiniCPM-V-4.7-1B-vision.rknn` | `9e081464857fad825281ae8778defb882424cbf5d46a37acb6de59dd5cbc99d6` |
164
+ | `MiniCPM-V-4.7-1B-vision.weight` | `9c1939dd21e511c72ef6fa6333d43d962acc067d5302ed625815abeaf1cc1c03` |
165
+
166
+ ## License & attribution
167
+
168
+ **Apache-2.0**, inherited from
169
+ [`openbmb/MiniCPM-V-4.7-1B`](https://huggingface.co/openbmb/MiniCPM-V-4.7-1B).
170
+ Please credit OpenBMB for the original model. The RKNN3 conversion, the
171
+ 196-token adapter and the host scripts in this repository are provided under
172
+ the same license, without warranty.
173
+
174
+ Built with [`airockchip/rknn3-model-zoo`](https://github.com/airockchip/rknn3-model-zoo) V1.1.0.
175
+
176
+ ---
177
+
178
+ ## 中文摘要
179
+
180
+ 把 `openbmb/MiniCPM-V-4.7-1B` 转成 RKNN3,在 **RK1828** 上跑通完整图文链路
181
+ (文本塔 + 视觉塔 + `rkllm3-server --model2`),真机 5/5 带图与纯文本请求
182
+ 全部 HTTP 200,NPU 独占占用 **1269 MB**。
183
+
184
+ **必须知道的取舍**:4.7 视觉塔原生只出 8×8=64 个视觉 token,而 RKNN3 runtime
185
+ 的 multimodal 回调要求 `image_ph * image_pw = 14×14 = 196`。为满足契约,
186
+ 导出图里加了 8×8→14×14 的最近邻重采样**适配层**(只改图结构、不改权重,
187
+ `.weight` 的 SHA256 与 64-token 版完全相同)。因此
188
+ **本产物不等价于 HF 原版 4.7**,不能拿它复现原版精度;
189
+ canvas mRoPE / 多分辨率切砖 / 高分辨 OCR / 密集计数均未验证。
190
+
191
+ | 项 | 结果 |
192
+ | --- | --- |
193
+ | 橙图问颜色 | `这张图是橙色。` |
194
+ | 黑方块 | `黑色正方形` |
195
+ | 交叉对照(排除语言先验猜答案) | 正确 |
196
+ | 纯文本 | `北京` |
197
+ | 与 ASR 共存 | 0.6B 3458 MB / 1.7B 3353 MB(A/B/A 可复现) |
198
+ | 视觉 token 注入证据 | `timings.prompt_n ≈ 218` |
199
+
200
+ 仓库含 6 个转换产物 + `start.cmd` + 请求脚本 + 转换脚本 + 部署文档,
201
+ 拿到即可跑;部署与十个实测踩过的坑见 [`docs/DEPLOY.md`](docs/DEPLOY.md)。
202
+ 复杂图务必看 `finish_reason`,别只看 HTTP 200。
SHA256SUMS ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ fa50be8954ef102147e51fb9d6930a14ef3b6097b0d5732e156e1eb8113e3226 convert/README.md
2
+ 35ef5e4568ef4ea0323158695379f2fd3be345d2a0c879ae563fc58186e23512 convert/export_vision47.py
3
+ b72cb20be6e6a28e3167222df5a1974fcd5356cf48a09caac2a07fff6d6692ec convert/export_vision_rknn.py
4
+ ad22d897e77ab8f3dbfbef90a4769e5fcc93d6cc9489f68913e0680d091de126 docs/DEPLOY.md
5
+ b0d4d2e702a6333941beed2064fdb1bf2a9e4fe8fca0f374964d8b754997a35c host/make_test_images.py
6
+ 7ca759bedbc3e6db62aeaa20c9b23c4e0c47c555c0ab04da6800d4a02e8c9194 host/req_orange.json
7
+ 69428abbd9c1642fcf1ccbcd2f47c9fd43a300f35e31cc5ed3b072f818a242ef host/req_text.json
8
+ 8b65ac059d9465352eb4b720da2a9f4f6d05b042b42ffc79978a9ce51d1a2673 host/send_vlm_req.cmd
9
+ 00f1f0eed92d5d2cf8ddb0033fdafcea7992886ce9b0b04f245f61ac9f3eb209 host/test_images.json
10
+ 511c5794dbf3d927fb17c5efc0052fababafdd35185be2961f6a855c1852b335 rknn/MiniCPM-V-4.7-1B-llm.embed.bin
11
+ 948e487bff9b2324b3016c665aa071feff05ac735e078c20ecb4a0ac7b825a1b rknn/MiniCPM-V-4.7-1B-llm.rknn
12
+ e028b0a11710ee5e0402c3829e871c3bd5c568540fe377febdd1a515d45b1544 rknn/MiniCPM-V-4.7-1B-llm.tokenizer.gguf
13
+ 72fd0b16b8016d63a7d8b92f8d159ad7ea68a783eaca89b915bd0e240ae22dab rknn/MiniCPM-V-4.7-1B-llm.weight
14
+ 9e081464857fad825281ae8778defb882424cbf5d46a37acb6de59dd5cbc99d6 rknn/MiniCPM-V-4.7-1B-vision.rknn
15
+ 9c1939dd21e511c72ef6fa6333d43d962acc067d5302ed625815abeaf1cc1c03 rknn/MiniCPM-V-4.7-1B-vision.weight
16
+ 5549fa0a8ce2e6ed739334adb5c52ec98eeb240a0675ce86b0ad51fe4646ddd5 rknn/start.cmd
convert/README.md ADDED
@@ -0,0 +1,87 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Reproducing the conversion
2
+
3
+ These are **portable copies** of the two scripts that produced the vision tower
4
+ in `../rknn/`. All absolute paths have been replaced by arguments / environment
5
+ variables; the model math is unchanged.
6
+
7
+ ## Prerequisites
8
+
9
+ - `rknn3-toolkit` (the whl from the RKNN3 SDK, installed with `--no-deps`)
10
+ - `transformers >= 5.18` (`MiniCPMV4_7ForConditionalGeneration` does not exist
11
+ in 5.3). Point `MINICPM_TF518` at a separate checkout if your main env is older.
12
+ - `torch`, `onnx`, `safetensors`
13
+ - A local checkout of [`openbmb/MiniCPM-V-4.7-1B`](https://huggingface.co/openbmb/MiniCPM-V-4.7-1B)
14
+ - Official [`airockchip/rknn3-model-zoo`](https://github.com/airockchip/rknn3-model-zoo) V1.1.0
15
+ for the LLM-side `export_llm.py` (the text tower is the same `Qwen3_5ForCausalLM`
16
+ family as the zoo's `Qwen3_5` example).
17
+
18
+ ## 1. Text tower (four artifacts)
19
+
20
+ The 4.7 text tower is architecturally Qwen3.5-0.8B with a different vocabulary:
21
+
22
+ | Item | 0.8B | MiniCPM-V 4.7 1B |
23
+ | --- | --- | --- |
24
+ | `vocab_size` | 248320 | **248094** |
25
+ | `tie_word_embeddings` | true | **false** (independent `lm_head.weight` [248094,1024]) |
26
+ | `conv_state` | [18,6144,4] | [18,6144,4] (identical) |
27
+
28
+ Build a plain-text HF directory by keeping only `model.language_model.*` tensors
29
+ plus `lm_head.weight`, copying the 4.7 tokenizer files, and starting from the
30
+ zoo's Qwen3.5 `config.json` with the two changes above. Then run the zoo's
31
+ `Qwen3_5/python/export_llm.py` followed by `export_rknn.py` with
32
+ `w4a16 / normal / group32 / core_num=8`.
33
+
34
+ > **Top-level `tie_word_embeddings` trap.** Hugging Face `tie_weights()` reads the
35
+ > **top-level** config key, not `text_config`. Changing only `text_config` leaves
36
+ > the independent 254M-parameter head silently bound to `embed_tokens`, and the
37
+ > converted model emits word salad. Change **both**. The only reliable check is
38
+ > `cmp`-ing the produced `embed.bin` against the head weights — file sizes can be
39
+ > byte-identical between a good and a bad build.
40
+
41
+ ## 2. Vision tower (two artifacts)
42
+
43
+ ```bash
44
+ export MINICPM_HF_DIR=/path/to/MiniCPM-V-4.7-1B
45
+ export MINICPM_TF518=/path/to/transformers-5.18 # optional
46
+ export MINICPM_VISION_ONNX=MiniCPM-V-4.7-1B-vision.onnx
47
+ export MINICPM_VISION_RKNN=MiniCPM-V-4.7-1B-vision.rknn
48
+
49
+ python export_vision47.py --model "$MINICPM_HF_DIR" --out "$MINICPM_VISION_ONNX"
50
+ python export_vision_rknn.py
51
+ ```
52
+
53
+ `export_vision_rknn.py` produces `<name>.rknn` and `<name>.weight` side by side.
54
+
55
+ ### Two non-obvious contracts
56
+
57
+ 1. **The output must be 3D `[1, N, hidden]`, not 2D `[N, hidden]`.**
58
+ The runtime reads `embeds_shape[0]` as *number of images* and
59
+ `embeds_shape[1]` as *tokens per image* (see the official
60
+ `examples/MiniCPM_V_4/cpp/main.cc`). A 2D export is parsed as
61
+ "64 images of 1024 tokens each" and the image tokens never line up.
62
+
63
+ 2. **`image_ph * image_pw == n_image_tokens`.** The multimodal embed callback
64
+ computes `image_ph = model_height / patch_size(16) / spatial_merge_size(2)`;
65
+ with a 448x448 vision input that is `14 x 14 = 196`. MiniCPM-V 4.7's native
66
+ tower emits `(448/14/4)^2 = 8 x 8 = 64` tokens. A 64-token export fails on
67
+ device with:
68
+
69
+ ```text
70
+ image_ph * image_pw != n_image_tokens! image_ph = 14 ... n_image_tokens = 64
71
+ ```
72
+
73
+ Changing `--img-width/--img-height` or resizing the request image does **not**
74
+ help: `image_height` comes from the vision RKNN model's own input shape.
75
+ `export_vision47.py` therefore inserts a **nearest-neighbour 8x8 -> 14x14
76
+ resample** inside the exported graph (pure Gather/Reshape, no `Resize` op):
77
+
78
+ ```python
79
+ g, idx = self.src_grid, self.resample_idx # 8, 14
80
+ y = y.view(g, g, -1)[idx[:, None], idx[None, :], :]
81
+ return y.reshape(1, 196, -1)
82
+ ```
83
+
84
+ This is a **runtime-contract adapter, not native 4.7 behaviour**. It changes
85
+ only graph structure — the `.weight` SHA256 is identical to the 64-token
86
+ build, since no weights are touched. See the model card for what this means
87
+ for equivalence claims.
convert/export_vision47.py ADDED
@@ -0,0 +1,118 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """MiniCPM-V 4.7 1B 视觉塔 -> ONNX(固定分辨率、单图、16x 压缩)。
2
+
3
+ 要点:
4
+ - 4.7 的建模类在 transformers>=5.18,本地 zoo venv 是 5.3.0,
5
+ 因此把 5.18.0 单独装到 minicpm47/tf518,用 sys.path 前置覆盖,不污染原环境。
6
+ - vision_tower.forward 内部有 ViT 内压缩(insert_layer_id=6 的 vit_merger,
7
+ 2x2 window merge),merger 再 2x2 -> 合计 4x4 patch = 1 个 LLM token(16x)。
8
+ - 固定 448x448:grid=32 -> vit_merger 后 16x16 -> merger 后 8x8 = 64 个 token,
9
+ 每个 1024 维(= LLM hidden_size)。32 能被 4 整除,980/14=70 不行,别用 980 单图。
10
+ - RKNN 侧用 mean/std=0.5*255 + uint8 NHWC 输入,ONNX 只接受已归一化的 fp32 NCHW。
11
+ - 输出必须是 3D [1, N, hidden],不能是 2D [N, hidden]!依据官方
12
+ examples/MiniCPM_V_4/cpp/main.cc:
13
+ tensor.image.n_image = embeds_shape[0]; // 图片数
14
+ tensor.image.n_image_tokens = embeds_shape[1]; // 每图 token 数
15
+ rkllm3-server 用同一套 runtime 约定。2D 输出会被读成"64 张图、每张 1024 token",
16
+ 导致 image token 数与 vision embedding 完全错位。官方 4.0 的
17
+ export_minicpm3o_vision likewise 返回 B*Q*D 三维。
18
+ - **runtime 契约适配(11-4 上板实测)**:multimodal embed callback 还会校验
19
+ `image_ph * image_pw == n_image_tokens`,其中
20
+ `image_ph = model_height / patch_size(16) / spatial_merge_size(2)`。
21
+ 448 输入 -> 14x14 = **196**,而 4.7 视觉塔原生只有 8x8 = **64**,直接上板报
22
+ `image_ph * image_pw != n_image_tokens! image_ph = 14 ... n_image_tokens = 64`。
23
+ `image_height` 取自 vision RKNN 模型的输入尺寸:改 `--img-width/--img-height`
24
+ 到 256、把请求图片换成 256x256,报错里都仍是 448(均已实测)。
25
+ 故在导出图内把 8x8 网格最近邻重采样到 14x14 = 196 token 让契约对齐。
26
+ 这是**适配层,不是 4.7 原生行为**;视觉信息有冗余,但空间排布保留。
27
+ """
28
+ import argparse
29
+ import os
30
+ import sys
31
+
32
+ import torch
33
+
34
+ TF518 = os.environ.get("MINICPM_TF518", "")
35
+ if TF518:
36
+ sys.path.insert(0, TF518)
37
+
38
+ MODEL = os.environ.get("MINICPM_HF_DIR", "")
39
+
40
+
41
+ class VisionExport(torch.nn.Module):
42
+ def __init__(self, model, img_size):
43
+ super().__init__()
44
+ self.vt = model.model.vision_tower
45
+ self.merger = model.model.merger
46
+ patch = self.vt.config.patch_size
47
+ grid = img_size // patch
48
+ assert grid % 4 == 0, f"patch grid {grid} 必须能被 4 整除(16x 压缩),换分辨率"
49
+ self.register_buffer(
50
+ "target_sizes", torch.tensor([[grid, grid]], dtype=torch.int32), persistent=False
51
+ )
52
+ # merger 内部用 height/width 作 view() 的形状参数;传 Python int 才能被 ONNX trace。
53
+ # vision_tower 在 insert_layer 处把 target_sizes 减半,故这里也减半。
54
+ self.merger_sizes = [[grid // 2, grid // 2]]
55
+ # ---- runtime 契约适配(见文件头 docstring)----
56
+ # 视觉塔原生 (grid/4)^2 = 8x8 = 64 token,runtime 要求 14x14 = 196。
57
+ # 用纯 Gather/Reshape 做最近邻重采样,不引入 Resize op,RKNN 转换最稳。
58
+ self.src_grid = grid // 4
59
+ self.out_grid = 14
60
+ self.register_buffer(
61
+ "resample_idx",
62
+ (torch.arange(self.out_grid, dtype=torch.float32) * self.src_grid / self.out_grid).long(),
63
+ persistent=False,
64
+ )
65
+
66
+ def forward(self, pixel):
67
+ out = self.vt(pixel, target_sizes=self.target_sizes, use_vit_merger=True)
68
+ hs = out.last_hidden_state if hasattr(out, "last_hidden_state") else out[0]
69
+ feats = self.merger(hs, self.merger_sizes)
70
+ y = torch.cat(feats, dim=0) # [src^2, hidden],行优先 8x8 网格
71
+ g, idx = self.src_grid, self.resample_idx
72
+ y = y.view(g, g, -1)[idx[:, None], idx[None, :], :] # [14, 14, hidden]
73
+ # [N, hidden] -> [1, N, hidden]:server 按 dim0=图片数 / dim1=每图token数解析。
74
+ return y.reshape(1, self.out_grid * self.out_grid, -1)
75
+
76
+
77
+ def main():
78
+ ap = argparse.ArgumentParser()
79
+ ap.add_argument("--img_size", type=int, default=448)
80
+ ap.add_argument(
81
+ "--out",
82
+ default=os.environ.get("MINICPM_VISION_ONNX", "MiniCPM-V-4.7-1B-vision.onnx"),
83
+ )
84
+ ap.add_argument("--model", default=MODEL,
85
+ help="Local HF checkout of openbmb/MiniCPM-V-4.7-1B")
86
+ ap.add_argument("--smoke", action="store_true", help="只跑一次 eager forward,不导出")
87
+ args = ap.parse_args()
88
+
89
+ from transformers import AutoModelForImageTextToText
90
+
91
+ model = AutoModelForImageTextToText.from_pretrained(
92
+ args.model, dtype=torch.float32, trust_remote_code=True, low_cpu_mem_usage=True
93
+ )
94
+ model.eval()
95
+ wrapper = VisionExport(model, args.img_size).eval()
96
+
97
+ pixel = torch.randn(1, 3, args.img_size, args.img_size)
98
+ with torch.no_grad():
99
+ y = wrapper(pixel)
100
+ print("eager output shape:", tuple(y.shape), "dtype:", y.dtype, flush=True)
101
+ if args.smoke:
102
+ return
103
+
104
+ os.makedirs(os.path.dirname(args.out), exist_ok=True)
105
+ torch.onnx.export(
106
+ wrapper,
107
+ (pixel,),
108
+ args.out,
109
+ input_names=["pixel"],
110
+ output_names=["img_embeds"],
111
+ opset_version=17,
112
+ do_constant_folding=True,
113
+ )
114
+ print("exported:", args.out, os.path.getsize(args.out), "bytes", flush=True)
115
+
116
+
117
+ if __name__ == "__main__":
118
+ main()
convert/export_vision_rknn.py ADDED
@@ -0,0 +1,57 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """MiniCPM-V 4.7 1B vision ONNX -> RKNN (w4a16 / normal / group32 / 8 cores)。
2
+
3
+ 参数照抄官方 zoo examples/MiniCPM_V_4/python/vision/export_rknn.py:
4
+ uint8 NHWC 输入 + mean/std=0.5*255(RKNN 在输入端做归一化),
5
+ ONNX 里只保留已归一化 fp32 NCHW 的计算图。
6
+ """
7
+ import os
8
+
9
+ from rknn.api import RKNN
10
+
11
+ ONNX = os.environ.get("MINICPM_VISION_ONNX", "MiniCPM-V-4.7-1B-vision.onnx")
12
+ OUT = os.environ.get("MINICPM_VISION_RKNN", "MiniCPM-V-4.7-1B-vision.rknn")
13
+
14
+
15
+ def main():
16
+ rknn = RKNN(verbose=True)
17
+
18
+ print("--> config model", flush=True)
19
+ rknn.config(
20
+ target_platform="rk1820",
21
+ quantized_dtype="w4a16",
22
+ quantized_algorithm="normal",
23
+ quantized_method="group32",
24
+ core_num=8,
25
+ mean_values=[[0.5 * 255, 0.5 * 255, 0.5 * 255]],
26
+ std_values=[[0.5 * 255, 0.5 * 255, 0.5 * 255]],
27
+ input_attrs={"pixel": {"dtype": "uint8", "layout": "NHWC"}},
28
+ )
29
+ print("done", flush=True)
30
+
31
+ print("--> Loading model", flush=True)
32
+ ret = rknn.load_onnx(model=ONNX)
33
+ if ret != 0:
34
+ print("Load model failed!", ret, flush=True)
35
+ return ret
36
+ print("done", flush=True)
37
+
38
+ print("--> Building model", flush=True)
39
+ ret = rknn.build(do_quantization=True)
40
+ if ret != 0:
41
+ print("Build model failed!", ret, flush=True)
42
+ return ret
43
+ print("done", flush=True)
44
+
45
+ print("--> Export RKNN model", flush=True)
46
+ ret = rknn.export_rknn(OUT)
47
+ if ret != 0:
48
+ print("Export rknn failed!", ret, flush=True)
49
+ return ret
50
+ print("done", flush=True)
51
+
52
+ rknn.release()
53
+ return 0
54
+
55
+
56
+ if __name__ == "__main__":
57
+ raise SystemExit(main())
docs/DEPLOY.md ADDED
@@ -0,0 +1,143 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Deployment guide — MiniCPM-V 4.7 1B on RK1828
2
+
3
+ ## 0. What you need
4
+
5
+ | Item | Requirement |
6
+ | --- | --- |
7
+ | NPU module | Rockchip **RK1828 / RK1820** (tested on `RM1828MC0-F`, firmware 1.1.0) |
8
+ | Host | Windows PC, or RK3588 / RK3576 board. **Ubuntu x86 cannot drive the module directly** (conversion / simulator only) |
9
+ | Runtime | `rknn3_110` — `rkllm3-server.exe`, matching `toolkit / proxy / runtime / firmware` versions |
10
+ | Session | Windows **Session 1** (interactive logon or an `InteractiveToken` scheduled task) |
11
+
12
+ Component versions **must match**. A mismatched `rknn3_110` typically fails with
13
+ a DLL-load error that prints **nothing at all**.
14
+
15
+ ## 1. Layout
16
+
17
+ Put the runtime next to `rknn/`:
18
+
19
+ ```text
20
+ <root>\
21
+ rknn\ <- this repo's rknn/ folder (6 artifacts + start.cmd)
22
+ runtime\rknn3_110\bin\rkllm3-server.exe
23
+ runtime\rknn3_110\lib\*.dll
24
+ ```
25
+
26
+ `rknn/start.cmd` looks for the runtime in `..\runtime\rknn3_110`, then
27
+ `..\..\runtime\rknn3_110`, then the `RK_RUNTIME` environment variable.
28
+ If none is found it **exits with an error** rather than silently falling back
29
+ to a hard-coded drive.
30
+
31
+ Override the listen address / port with `RK_HOST` / `RK_PORT`
32
+ (defaults `0.0.0.0` / `8899`).
33
+
34
+ ```cmd
35
+ cd /d <root>\rknn
36
+ start.cmd
37
+ ```
38
+
39
+ Wait until the server is actually ready. The log can sit at
40
+ `Weight sync chunk size resolved` for a couple of minutes — **do not treat that
41
+ as a hang**, and do not run `rknn-smi` while weights are loading (it can wedge
42
+ the sync). Readiness is proven only by a real inference request.
43
+
44
+ ## 2. Start the communication proxy
45
+
46
+ One `rknn3_transfer_proxy` process per machine, **reuse it**. Repeatedly
47
+ tearing it down and recreating it can drop the module off the USB bus.
48
+ If you use the RK1828 web console, let the console own the proxy.
49
+
50
+ ## 3. Send an image request
51
+
52
+ The server exposes an OpenAI-compatible endpoint. `messages[].content` must be
53
+ an **array**, and the image goes in as a base64 data URI:
54
+
55
+ ```json
56
+ {
57
+ "messages": [{
58
+ "role": "user",
59
+ "content": [
60
+ { "type": "text", "text": "图里是什么?" },
61
+ { "type": "image_url", "image_url": { "url": "data:image/png;base64,<BASE64>" } }
62
+ ]
63
+ }],
64
+ "max_tokens": 512
65
+ }
66
+ ```
67
+
68
+ **Do not put `<image>` / `<|image_pad|>` placeholder tokens into the text.**
69
+ The server constructs them itself. Ready-made bodies are in `host/`:
70
+
71
+ ```cmd
72
+ cd /d <root>\host
73
+ host\make_test_images.py :: generates test_images.json (pure stdlib)
74
+ send_vlm_req.cmd orange :: -> 127.0.0.1:8899
75
+ send_vlm_req.cmd text 8888 :: override port
76
+ ```
77
+
78
+ Use **`curl.exe --max-time`**. Windows PowerShell `Invoke-WebRequest` /
79
+ `Invoke-RestMethod` hang indefinitely against this server (no response, no
80
+ `-OutFile`), and `-TimeoutSec` does not always rescue it.
81
+
82
+ Expect roughly `HTTP=200 TIME=0.6s` for a simple image. A correct response
83
+ contains `timings.prompt_n ≈ 218` — that is 196 visual tokens plus text tokens,
84
+ and is the evidence that vision embeddings really entered the LLM.
85
+
86
+ ## 4. Verified reference results
87
+
88
+ | Check | Result |
89
+ | --- | --- |
90
+ | Orange image, "what colour" | `这张图是橙色。` |
91
+ | Black square image | `黑色正方形` |
92
+ | Cross-checks (orange asked for shape, square asked for colour) | correct, no language-prior guessing |
93
+ | Pure text "中国的首都是哪里" | `北京` |
94
+ | `finish_reason` | `stop` on simple cases |
95
+ | NPU, VLM alone | **1269 MB** (exclusive differential; console shows 1303 incl. 34 MB baseline) |
96
+ | NPU, VLM + Qwen3-ASR-0.6B | 3458 MB |
97
+ | NPU, VLM + Qwen3-ASR-1.7B | 3353 MB (A/B/A reproducible) |
98
+ | Generation | ~90 tok/s, prefill ~1700 tok/s |
99
+
100
+ Five request/response pairs plus the full console acceptance run are documented
101
+ in the upstream project docs; the ready-made request bodies here reproduce the
102
+ two simplest ones.
103
+
104
+ ## 5. Pitfalls (all hit in practice)
105
+
106
+ 1. **Session 0 breaks the USB bulk channel.** Starting the server from an SSH
107
+ session lands it in Session 0; `rknn3_init` succeeds but weight sync never
108
+ completes. Use `schtasks` with `InteractiveToken` (Session 1). Diagnose with
109
+ a *different, known-good* model — if both fail identically, suspect the
110
+ session, not the USB port or the firmware.
111
+ 2. **`tie_word_embeddings` at the top level.** A converted graph without a real
112
+ `lm_head` makes `load_llm` **segfault silently**. See `convert/README.md`.
113
+ 3. **Vision output shape and token count.** 3D `[1,N,1024]` and
114
+ `N == image_ph * image_pw == 196`. See `convert/README.md`.
115
+ 4. **Placeholder tokens must match this model's own vocabulary.** MiniCPM-V 4.7
116
+ uses bare `<image>` / `</image>` / `<|image_pad|>`. Its GGUF vocabulary is a
117
+ **superset** that also contains Qwen-VL's `<|vision_start|>` /
118
+ `<|vision_end|>`; a detector that checks Qwen-VL first will silently rewrite
119
+ a correct `start.cmd` into a broken one.
120
+ 5. **`-c` is context length, not core mask.** Use `--core-mask` /
121
+ `--core-mask2`; 8 cores = `0xFF`. A zero core mask makes
122
+ `rknn3_model_init` return `-2`.
123
+ 6. **`.cmd` / `.bat` / `.ps1` must be pure ASCII + CRLF**, and `PATH` must be
124
+ set before launching the child process (the launch script does this).
125
+ 7. **Check `finish_reason`.** This is a 1B model with a `<think>` chain. Complex
126
+ images can fill `max_tokens` and return `finish_reason=length` with the
127
+ answer truncated. Increasing `max_tokens` is **not** monotonically safer
128
+ (256 worked for one OCR case where 512 stalled inside `<think>`).
129
+ 8. **Residual processes hold the port** → "I changed the code but behaviour did
130
+ not change". Check `Get-NetTCPConnection` before restarting.
131
+ 9. **ASR device errors can leak NPU memory** (`mem_sync ret=-10` retry loop).
132
+ Workaround: `taskkill /F /IM rknn_qwen3_asr_demo.exe`. Not fixed upstream.
133
+ 10. **Power loss may land the module in MaskROM** (`2207:180B`); `upgrade_tool uf`
134
+ recovers it in ~2 s. A genuine hard hang needs a physical power cycle.
135
+
136
+ ## 6. Known limitations — read before filing issues
137
+
138
+ - The 196-token resample is a **runtime-contract adapter**. This build is
139
+ **not equivalent to the Hugging Face original MiniCPM-V 4.7 1B**.
140
+ - canvas mRoPE, multi-resolution tiling, high-resolution OCR and dense
141
+ multi-object counting were **not verified**.
142
+ - OCR is "characters readable" at best: `RK1828` may come back as `RK₁₈₂₈`.
143
+ - Only 448x448 single-image input was tested.
host/make_test_images.py ADDED
@@ -0,0 +1,65 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """生成两张 448x448 测试图(纯标准库,不依赖 PIL),供 VLM 带图请求使用。
2
+
3
+ - orange: 纯橙色。模型若真"看到"图,问颜色应答橙色;瞎猜则说明视觉链路没通。
4
+ - square: 白底黑方块。用于形状/内容识别的第二个判别样本。
5
+
6
+ 输出 test_images.json,内容为 OpenAI image_url 用的 data URI。
7
+ """
8
+ import base64
9
+ import json
10
+ import os
11
+ import struct
12
+ import zlib
13
+
14
+
15
+ def png_chunk(tag, data):
16
+ return (
17
+ struct.pack(">I", len(data))
18
+ + tag
19
+ + data
20
+ + struct.pack(">I", zlib.crc32(tag + data) & 0xFFFFFFFF)
21
+ )
22
+
23
+
24
+ def write_png(w, h, pixel):
25
+ raw = bytearray()
26
+ for y in range(h):
27
+ raw.append(0) # filter type 0 (None)
28
+ for x in range(w):
29
+ raw.extend(pixel(x, y))
30
+ comp = zlib.compress(bytes(raw), 9)
31
+ return (
32
+ b"\x89PNG\r\n\x1a\n"
33
+ + png_chunk(b"IHDR", struct.pack(">IIBBBBB", w, h, 8, 2, 0, 0, 0))
34
+ + png_chunk(b"IDAT", comp)
35
+ + png_chunk(b"IEND", b"")
36
+ )
37
+
38
+
39
+ def main():
40
+ # 448x448:与 vision 模型输入尺寸一致,避免 server 额外缩放。
41
+ w = h = 448
42
+ orange = write_png(w, h, lambda x, y: (255, 128, 0))
43
+
44
+ def square_px(x, y):
45
+ if 144 <= x < 304 and 144 <= y < 304:
46
+ return (0, 0, 0)
47
+ return (255, 255, 255)
48
+
49
+ square = write_png(w, h, square_px)
50
+
51
+ out = {
52
+ "orange": "data:image/png;base64," + base64.b64encode(orange).decode(),
53
+ "square": "data:image/png;base64," + base64.b64encode(square).decode(),
54
+ }
55
+ path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "test_images.json")
56
+ with open(path, "w") as f:
57
+ json.dump(out, f)
58
+ print("orange png bytes:", len(orange), "square png bytes:", len(square))
59
+ print("orange data uri len:", len(out["orange"]))
60
+ print("square data uri len:", len(out["square"]))
61
+ print("saved:", path)
62
+
63
+
64
+ if __name__ == "__main__":
65
+ main()
host/req_orange.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"messages": [{"role": "user", "content": [{"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAcAAAAHACAIAAAC6Ry8kAAAEl0lEQVR42u3UMQ0AAAzDsEIf85XDtNOSEeRIdgLAgQQABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAABgqAgQIYKICBAhgoAAYKYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKgIECGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECYKAABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAAGCiAgQIYKICBAhgoAAYKYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKgIECGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECYKAABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAAGCiAgQIYKICBAhgoAAYKYKAABgpgoAAGCoCBAhgogIECGCgABgpgoAAGCmCgAAYKgIECGCiAgQIYKICBAmCgAAYKYKAABgpgoCoAGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECGCgABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAABgqAgQIYKICBAhgoAAYKYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKgIECGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECGKgEAAYKYKAABgpgoAAYKICBAhgogIECGCgABgpgoAAGCmCgAAYKgIECGCiAgQIYKAAGCmCgAAYKYKAABgqAgQIYKICBAhgogIECYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKYKAAGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECGCgABgpgoAAGCmCgAEgAYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKYKAAGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECGCgABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAABgqAgQIYKICBAhgoAAYKYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKgIECGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECGCgABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAABgqAgQIYKICBAhgoAAYKYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKgIECGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECYKAABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAABqoCgIECGCiAgQIYKAAGCmCgAAYKYKAABgqAgQIYKICBAhgogIECYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKYKAAGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECGCgABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCvCosKM0vFDSA3UAAAAASUVORK5CYII="}}, {"type": "text", "text": "这张图是什么颜色?请直接回答颜色名称。"}]}], "max_tokens": 256}
host/req_text.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"messages": [{"role": "user", "content": "中国的首都是哪里?请直接回答。"}], "max_tokens": 64}
host/send_vlm_req.cmd ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ @echo off
2
+ rem =====================================================================
3
+ rem Send one request to the rkllm3-server started by rknn\start.cmd.
4
+ rem Usage: send_vlm_req.cmd <name> [port]
5
+ rem <name> req_<name>.json must exist next to this script
6
+ rem [port] default 8899 (matches rknn\start.cmd RK_PORT default)
7
+ rem IMPORTANT: keep this file pure ASCII + CRLF.
8
+ rem =====================================================================
9
+ setlocal
10
+ set "BASE=%~dp0"
11
+ set "NAME=%~1"
12
+ if "%NAME%"=="" set "NAME=orange"
13
+ set "PORT=%~2"
14
+ if "%PORT%"=="" set "PORT=8899"
15
+ curl.exe -s -X POST http://127.0.0.1:%PORT%/v1/chat/completions -H "Content-Type: application/json" --data-binary "@%BASE%req_%NAME%.json" -o "%BASE%resp_%NAME%.json" -w "HTTP=%%{http_code} TIME=%%{time_total}s PORT=%PORT%" --max-time 180
16
+ echo.
17
+ exit /b 0
host/test_images.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"orange": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAcAAAAHACAIAAAC6Ry8kAAAEl0lEQVR42u3UMQ0AAAzDsEIf85XDtNOSEeRIdgLAgQQABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAABgqAgQIYKICBAhgoAAYKYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKgIECGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECYKAABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAAGCiAgQIYKICBAhgoAAYKYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKgIECGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECYKAABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAAGCiAgQIYKICBAhgoAAYKYKAABgpgoAAGCoCBAhgogIECGCgABgpgoAAGCmCgAAYKgIECGCiAgQIYKICBAmCgAAYKYKAABgpgoCoAGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECGCgABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAABgqAgQIYKICBAhgoAAYKYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKgIECGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECGKgEAAYKYKAABgpgoAAYKICBAhgogIECGCgABgpgoAAGCmCgAAYKgIECGCiAgQIYKAAGCmCgAAYKYKAABgqAgQIYKICBAhgogIECYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKYKAAGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECGCgABgpgoAAGCmCgAEgAYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKYKAAGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECGCgABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAABgqAgQIYKICBAhgoAAYKYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKgIECGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECGCgABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAABgqAgQIYKICBAhgoAAYKYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKgIECGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECYKAABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAABqoCgIECGCiAgQIYKAAGCmCgAAYKYKAABgqAgQIYKICBAhgogIECYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKYKAAGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECGCgABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCvCosKM0vFDSA3UAAAAASUVORK5CYII=", "square": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAcAAAAHACAIAAAC6Ry8kAAAGKElEQVR42u3USQ0AAAgEsfVvGiwQwou0EjgmBcBKjABAQAEEFEBAAQQUAAEFEFAAAQUQUAABBUBAAQQUQEABBBRAQAEQUAABBRBQAAEFQEABBBRAQAEEFEBAARBQAAEFEFAAAQUQUAAEFEBAAQQUQEABEFAAAQUQUAABBRBQAAQUQEABBBRAQAEEFAABBRBQAAEFEFAABBRAQAEEFEBAAQQUAAEFEFAAAQUQUAABBUBAAQQUQEABBBQAAQUQUAABBRBQAAEFQEABBBRAQAEEFEBAARBQAAEFEFAAAQUQUAAEFEBAAQQUQEABEFAAAQUQUAABBRBQAAQUQEABBBRAQAEEFAABBRBQAAEFEFAABBRAQAEEFEBAAQQUAAEFEFAAAQUQUAABBUBAAQQUQEABBBQAAQUQUAABBRBQAAEFQEABBBRAQAEEFEBAARBQAAEFEFAAAQVAQAEEFEBAAQQUQEABEFAAAQUQUAABBRBQAAQUQEABBBRAQAEEFAABBRBQAAEFEFAABBRAQAEElPnCeM2FCygCioAKKAKKgCKgAoqAIqACioAioAgoAiqgCCgCKqAIKAKKgAooAoqAIqACioAioAiogCKgCCgCKqAIKAIqoAgoAoqACigCioAioAKKgCKgCKiAIqAIqIAioAgoAoqACigCioAKKAKKgCKgAoqAIqAIqIAioAgoAiqgCCgCioAKKAKKgAooAoqAIqACioAioAiogCKgCCgCKqAIKAIqoAgoAoqAIqACioAioAKKgCKgCKiAIqAIKAIqoAgoAoqACigCioAKKAKKgCKgAoqAIqAIqIAioAgoAiqgCCgCioAKKAKKgAooAoqAIqAIqIAioAiogCKgCCgCKqAIKAKKgAooAoqAIqACioAioAKKgCKgCKiAIqAIKAIqoAgoAoqACigCioAioAKKgCKgAoqAIqAIKP5RQBFQBFRAEVAEFAEVUAQUAbUwBBQBRUARUAFFQBFQAUVAEVAEVEARUAQUARVQBBQBRUAFFAFFQBFQAUVAEVABRUARUARUQBFQBBQBFVAEFAFFQAUUAUVABRQBRUARUARUQBFQBFRAEVAEFAEVUAQUAUVABRQBRUARUAFFQBFQBFRAEVAEVEARUAQUARVQBBQBRUAFFAFFQBFQAUVAEVABRUARUAQUARVQBBQBFVAEFAFFQAUUAUVAEVABRUARUARUQBFQBFRAEVAEFAEVUAQUAUVABRQBRUARUAFFQBFQBFRAEVAEVEARUAQUAUVABRQBRUAFFAFFQBFQAUVAEVAEVEARUAQUARVQBBQBFVAEFAFFQAUUAUVAEVABRUARUARUQBFQBBQBFVAEFAEVUAQUAUVAEVABRUARUAFFQBFQBFRAEVAEFIkRUAQUAUVABRQBRUAFFAFFQBFQAUVAEVAEVEARUAQUARVQBBQBRUAFFAFFQAUUAUVAEVABRUARUARUQBFQBBQBFVAEFAEVUAQUAUVAEVABRUARUAFFQBFQBFRAEVAEFAEVUAQUAUVABRQBRUARUAFFQBFQAUVAEVAEVEARUAQUARVQBBQBRUAFFAFFQAUUAUVAEVAEVEARUARUQBFQBBQBFVAEFAFFQAUUAQUQUAABNQIAAQUQUAABBRBQAAQUQEABBBRAQAEEFAABBRBQAAEFEFAAAQVAQAEEFEBAAQQUAAEFEFAAAQUQUAABBUBAAQQUQEABBBRAQAEQUAABBRBQAAEFQEABBBRAQAEEFEBAARBQAAEFEFAAAQUQUAAEFEBAAQQUQEABEFAAAQUQUAABBRBQAAQUQEABBBRAQAEEFAABBRBQAAEFEFAABBRAQAEEFEBAAQQUAAEFEFAAAQUQUAABBUBAAQQUQEABBBRAQAEQUAABBRBQAAEFQEABBBRAQAEEFEBAARBQAAEFEFAAAQUQUAAEFEBAAQQUQEABEFAAAQUQUAABBRBQAAQUQEABBBRAQAEEFAABBRBQAAEFEFAABBRAQAEEFEBAAQQUAAEFEFAAAQUQUAABBUBAAQQUQEABBBQAAQUQUAABBRBQAAEFQEABBBRAQAEEFEBAARBQAAEFEFAAAQUQUAAEFEBAAQQUQEABEFCAEw0OhnPFl4DJIAAAAABJRU5ErkJggg=="}
rknn/start.cmd ADDED
@@ -0,0 +1,49 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ @echo off
2
+ rem =====================================================================
3
+ rem MiniCPM-V 4.7 1B VLM (text tower + vision tower) for RK1828.
4
+ rem Converted 2026-11-03/04 from openbmb/MiniCPM-V-4.7-1B via the
5
+ rem official rknn3-model-zoo Qwen3_5 pipeline (w4a16 + normal + group32).
6
+ rem
7
+ rem Text tower : Qwen3.5-0.8B architecture, vocab 248094,
8
+ rem tie_word_embeddings=false (independent w6a16 lm_head).
9
+ rem Vision : fixed 448x448 single image, output [1,196,1024].
10
+ rem The 196 tokens are a 8x8 -> 14x14 nearest-neighbor
11
+ rem resample ADAPTER (runtime demands 14x14); NOT native 4.7.
12
+ rem Image input: POST /v1/chat/completions with content[] image_url +
13
+ rem base64 data URI. Do NOT put placeholder tokens in text.
14
+ rem Limits : no canvas mRoPE / multi-resolution tiling verified;
15
+ rem complex-image accuracy not calibrated. See docs 21 / 9.9.
16
+ rem
17
+ rem IMPORTANT: keep this file pure ASCII + CRLF.
18
+ rem =====================================================================
19
+ setlocal
20
+ if "%RK_HOST%"=="" set RK_HOST=0.0.0.0
21
+ if "%RK_PORT%"=="" set RK_PORT=8899
22
+
23
+ rem Locate the RKNN3 runtime (repo layout: <repo>\rknn\ ; package layout: models\<name>\)
24
+ set "RT=%~dp0..\runtime\rknn3_110"
25
+ if not exist "%RT%\bin" set "RT=%~dp0..\..\runtime\rknn3_110"
26
+ if not exist "%RT%\bin" if not "%RK_RUNTIME%"=="" set "RT=%RK_RUNTIME%"
27
+ if not exist "%RT%\bin" goto no_runtime
28
+
29
+ set "PATH=%RT%\bin;%RT%\lib;%PATH%"
30
+ cd /d "%~dp0"
31
+ goto run
32
+
33
+ :no_runtime
34
+ echo [start.cmd] RKNN3 runtime not found with rkllm3-server.exe.
35
+ echo [start.cmd] looked in %~dp0..\runtime\rknn3_110 and ..\..\runtime\rknn3_110
36
+ endlocal
37
+ exit /b 1
38
+
39
+ :run
40
+ rkllm3-server -m "MiniCPM-V-4.7-1B-llm.rknn" --weight "MiniCPM-V-4.7-1B-llm.weight" ^
41
+ --vocab "MiniCPM-V-4.7-1B-llm.tokenizer.gguf" --embed "MiniCPM-V-4.7-1B-llm.embed.bin" ^
42
+ --model2 "MiniCPM-V-4.7-1B-vision.rknn" --weight2 "MiniCPM-V-4.7-1B-vision.weight" ^
43
+ --img-start "<image>" --img-end "</image>" --img-content "<|image_pad|>" ^
44
+ --img-width 448 --img-height 448 ^
45
+ --host "%RK_HOST%" --port "%RK_PORT%" ^
46
+ -c 0 --n_predict 512 --top-k 20 --top-p 0.95 --temp 0.7 --repeat-penalty 1.03
47
+
48
+ endlocal
49
+ exit /b 0