Image-Text-to-Text
GGUF
English
Chinese
rknn
rknn3
rk1828
rk1820
rockchip
npu
w4a16
vision-language-model
multimodal
on-device
edge-ai
conversational
Instructions to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 # Run inference directly in the terminal: llama cli -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 # Run inference directly in the terminal: llama cli -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 # Run inference directly in the terminal: ./build/bin/llama cli -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Use Docker
docker model run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Llama new
- LM Studio
- Jan
- vLLM
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Ollama
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Ollama:
ollama run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Unsloth Desktop
- Pi
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Docker Model Runner:
docker model run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Lemonade
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Run and chat with the model
lemonade run user.MiniCPM-V-4.7-1B-RKNN3-RK1828-{{QUANT_TAG}}List all available models
lemonade list
- Hermes Agent
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Add model card, deploy guide, host + conversion scripts
Browse files- README.md +202 -0
- SHA256SUMS +16 -0
- convert/README.md +87 -0
- convert/export_vision47.py +118 -0
- convert/export_vision_rknn.py +57 -0
- docs/DEPLOY.md +143 -0
- host/make_test_images.py +65 -0
- host/req_orange.json +1 -0
- host/req_text.json +1 -0
- host/send_vlm_req.cmd +17 -0
- host/test_images.json +1 -0
- rknn/start.cmd +49 -0
README.md
ADDED
|
@@ -0,0 +1,202 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language:
|
| 3 |
+
- en
|
| 4 |
+
- zh
|
| 5 |
+
license: apache-2.0
|
| 6 |
+
pipeline_tag: image-text-to-text
|
| 7 |
+
inference: false
|
| 8 |
+
tags:
|
| 9 |
+
- rknn
|
| 10 |
+
- rknn3
|
| 11 |
+
- rk1828
|
| 12 |
+
- rk1820
|
| 13 |
+
- rockchip
|
| 14 |
+
- npu
|
| 15 |
+
- w4a16
|
| 16 |
+
- vision-language-model
|
| 17 |
+
- image-text-to-text
|
| 18 |
+
- multimodal
|
| 19 |
+
- on-device
|
| 20 |
+
- edge-ai
|
| 21 |
+
base_model: openbmb/MiniCPM-V-4.7-1B
|
| 22 |
+
---
|
| 23 |
+
|
| 24 |
+
# MiniCPM-V-4.7-1B-RKNN3-RK1828
|
| 25 |
+
|
| 26 |
+
**MiniCPM-V 4.7 1B converted to RKNN3 and running end-to-end on the Rockchip
|
| 27 |
+
RK1828 NPU** — both the text tower and the vision tower, with real image
|
| 28 |
+
questions answered on device.
|
| 29 |
+
|
| 30 |
+
> **⚠️ Read this first.** The vision tower in this repository contains a
|
| 31 |
+
> **runtime-contract adapter**: MiniCPM-V 4.7 natively emits 8×8 = **64** visual
|
| 32 |
+
> tokens per 448×448 image, but the RKNN3 multimodal callback requires
|
| 33 |
+
> `image_ph * image_pw = 14×14 = **196**` tokens. A nearest-neighbour 8×8 → 14×14
|
| 34 |
+
> resample was inserted into the exported graph to satisfy that contract.
|
| 35 |
+
> **This build is therefore NOT equivalent to the Hugging Face original
|
| 36 |
+
> MiniCPM-V 4.7 1B.** Do not use it to benchmark or reproduce the original
|
| 37 |
+
> model's accuracy. See [How it was made](#how-it-was-made).
|
| 38 |
+
|
| 39 |
+
This repository is not just six binary files. It includes the launch script,
|
| 40 |
+
request examples, conversion scripts and a deployment guide, so that the model
|
| 41 |
+
can actually be run and reproduced.
|
| 42 |
+
|
| 43 |
+
| | |
|
| 44 |
+
| --- | --- |
|
| 45 |
+
| Base model | [`openbmb/MiniCPM-V-4.7-1B`](https://huggingface.co/openbmb/MiniCPM-V-4.7-1B) (Apache-2.0) |
|
| 46 |
+
| Hardware | RK1828 / RK1820 NPU (tested `RM1828MC0-F`, firmware 1.1.0) |
|
| 47 |
+
| Host | Windows PC or RK3588 / RK3576 board; **Ubuntu x86 cannot drive the module directly** |
|
| 48 |
+
| Runtime | `rknn3_110` (`rkllm3-server`), versions must match toolkit / proxy / firmware |
|
| 49 |
+
| Quantization | `w4a16`, `normal`, `group32`, 8 cores; head `w6a16` |
|
| 50 |
+
| NPU footprint | **1269 MB** (exclusive differential measurement) |
|
| 51 |
+
|
| 52 |
+
## What's in the repository
|
| 53 |
+
|
| 54 |
+
| Path | Contents |
|
| 55 |
+
| --- | --- |
|
| 56 |
+
| `rknn/` | 6 conversion artifacts + `start.cmd` (1.36 GiB) |
|
| 57 |
+
| `host/` | `send_vlm_req.cmd`, two ready-made request bodies, pure-stdlib test-image generator |
|
| 58 |
+
| `docs/DEPLOY.md` | Deployment, API contract, 10 pitfalls actually hit, verified results |
|
| 59 |
+
| `convert/` | Portable copies of the two vision-export scripts + the conversion recipe |
|
| 60 |
+
| `SHA256SUMS` | Checksums of every file above |
|
| 61 |
+
|
| 62 |
+
## Quick start
|
| 63 |
+
|
| 64 |
+
```text
|
| 65 |
+
<root>\
|
| 66 |
+
rknn\ <- this repo
|
| 67 |
+
runtime\rknn3_110\bin\ <- rkllm3-server.exe
|
| 68 |
+
runtime\rknn3_110\lib\ <- DLLs
|
| 69 |
+
```
|
| 70 |
+
|
| 71 |
+
```cmd
|
| 72 |
+
cd /d <root>\rknn
|
| 73 |
+
start.cmd :: defaults 0.0.0.0:8899 (RK_HOST / RK_PORT override)
|
| 74 |
+
cd /d <root>\host
|
| 75 |
+
send_vlm_req.cmd orange :: HTTP=200, response in resp_orange.json
|
| 76 |
+
```
|
| 77 |
+
|
| 78 |
+
The server is ready **only when a real inference returns** — the weight-sync log
|
| 79 |
+
can sit still for ~2 minutes. Do not poll `rknn-smi` during loading.
|
| 80 |
+
|
| 81 |
+
Full details (including the Session-1 requirement, proxy reuse and DLL/PATH
|
| 82 |
+
rules) are in [`docs/DEPLOY.md`](docs/DEPLOY.md).
|
| 83 |
+
|
| 84 |
+
## API
|
| 85 |
+
|
| 86 |
+
OpenAI-compatible `POST /v1/chat/completions`. `content` **must be an array**;
|
| 87 |
+
the server constructs the image placeholder tokens itself:
|
| 88 |
+
|
| 89 |
+
```bash
|
| 90 |
+
curl -s http://127.0.0.1:8899/v1/chat/completions \
|
| 91 |
+
-H "Content-Type: application/json" \
|
| 92 |
+
-d '{"messages":[{"role":"user","content":[
|
| 93 |
+
{"type":"text","text":"图里是什么?"},
|
| 94 |
+
{"type":"image_url","image_url":{"url":"data:image/png;base64,<BASE64>"}}
|
| 95 |
+
]}],"max_tokens":512}'
|
| 96 |
+
```
|
| 97 |
+
|
| 98 |
+
Do **not** put `<image>` / `<|image_pad|>` into the text. Use `curl.exe
|
| 99 |
+
--max-time`; PowerShell `Invoke-WebRequest` hangs on this server.
|
| 100 |
+
|
| 101 |
+
A correct response has `timings.prompt_n ≈ 218` = 196 visual tokens + text
|
| 102 |
+
tokens — that is the proof the vision embeddings really entered the LLM.
|
| 103 |
+
|
| 104 |
+
## Verified on device
|
| 105 |
+
|
| 106 |
+
| Check | Result |
|
| 107 |
+
| --- | --- |
|
| 108 |
+
| Orange image → colour | `这张图是橙色。` |
|
| 109 |
+
| Black square → shape/colour | `黑色正方形` |
|
| 110 |
+
| Cross-checks (orange asked for shape; square asked for colour) | correct — not language-prior guessing |
|
| 111 |
+
| Pure text (capital of China) | `北京` |
|
| 112 |
+
| Chinese ASR + this VLM coexisting | 3458 MB (0.6B ASR) / 3353 MB (1.7B ASR), all requests HTTP 200 |
|
| 113 |
+
| Throughput | ~90 tok/s generation, ~1700 tok/s prefill |
|
| 114 |
+
|
| 115 |
+
`finish_reason=stop` on simple cases. Complex images may return `length` —
|
| 116 |
+
**always check `finish_reason`, not just HTTP 200**.
|
| 117 |
+
|
| 118 |
+
## How it was made
|
| 119 |
+
|
| 120 |
+
1. **Text tower** — MiniCPM-V 4.7 reuses the Qwen3.5-0.8B architecture with
|
| 121 |
+
`vocab_size=248094` and an independent `lm_head`. Extraction + zoo
|
| 122 |
+
`Qwen3_5` `export_llm.py` / `export_rknn.py`.
|
| 123 |
+
The **top-level** `tie_word_embeddings` must be set to `false` too —
|
| 124 |
+
otherwise HF `tie_weights()` silently binds the 254M-parameter head to the
|
| 125 |
+
embedding and the model emits word salad.
|
| 126 |
+
2. **Vision tower** — `vision_tower` + `merger` exported to ONNX with
|
| 127 |
+
`transformers>=5.18` (`MiniCPMV4_7ForConditionalGeneration`), then RKNN with
|
| 128 |
+
`uint8 NHWC` input and `mean/std = 0.5*255`.
|
| 129 |
+
Output is 3D `[1, N, 1024]`; a 2D `[N,1024]` export is mis-parsed as
|
| 130 |
+
"N images of 1024 tokens".
|
| 131 |
+
3. **196-token adapter** — nearest-neighbour resample 8×8 → 14×14 inside the
|
| 132 |
+
graph (Gather/Reshape only, no `Resize` op). This changes **graph structure
|
| 133 |
+
only**: the `.weight` SHA256 is byte-identical to the unadapted 64-token
|
| 134 |
+
build, because no weights are touched.
|
| 135 |
+
|
| 136 |
+
See [`convert/README.md`](convert/README.md) for the recipe and
|
| 137 |
+
[`docs/DEPLOY.md`](docs/DEPLOY.md) §5 for the pitfalls.
|
| 138 |
+
|
| 139 |
+
## Limitations
|
| 140 |
+
|
| 141 |
+
- **Not equivalent to the original 4.7** (196-token adapter, see the warning above).
|
| 142 |
+
- canvas mRoPE, multi-resolution tiling, high-resolution OCR and dense
|
| 143 |
+
multi-object counting were **not verified**.
|
| 144 |
+
- OCR is "characters readable" at best (`RK1828` → `RK₁₈₂₈`).
|
| 145 |
+
- Only 448×448 single-image input was tested.
|
| 146 |
+
- 1B model with a `<think>` chain: raising `max_tokens` is **not** monotonically
|
| 147 |
+
safer (256 worked on one OCR case where 512 stalled inside `<think>`).
|
| 148 |
+
- Known upstream issue: on ASR device error, NPU memory can leak
|
| 149 |
+
(`mem_sync ret=-10` retry loop); workaround is to kill the ASR process.
|
| 150 |
+
|
| 151 |
+
## Checksums
|
| 152 |
+
|
| 153 |
+
```bash
|
| 154 |
+
sha256sum -c SHA256SUMS
|
| 155 |
+
```
|
| 156 |
+
|
| 157 |
+
| File | SHA256 |
|
| 158 |
+
| --- | --- |
|
| 159 |
+
| `MiniCPM-V-4.7-1B-llm.rknn` | `948e487bff9b2324b3016c665aa071feff05ac735e078c20ecb4a0ac7b825a1b` |
|
| 160 |
+
| `MiniCPM-V-4.7-1B-llm.weight` | `72fd0b16b8016d63a7d8b92f8d159ad7ea68a783eaca89b915bd0e240ae22dab` |
|
| 161 |
+
| `MiniCPM-V-4.7-1B-llm.embed.bin` | `511c5794dbf3d927fb17c5efc0052fababafdd35185be2961f6a855c1852b335` |
|
| 162 |
+
| `MiniCPM-V-4.7-1B-llm.tokenizer.gguf` | `e028b0a11710ee5e0402c3829e871c3bd5c568540fe377febdd1a515d45b1544` |
|
| 163 |
+
| `MiniCPM-V-4.7-1B-vision.rknn` | `9e081464857fad825281ae8778defb882424cbf5d46a37acb6de59dd5cbc99d6` |
|
| 164 |
+
| `MiniCPM-V-4.7-1B-vision.weight` | `9c1939dd21e511c72ef6fa6333d43d962acc067d5302ed625815abeaf1cc1c03` |
|
| 165 |
+
|
| 166 |
+
## License & attribution
|
| 167 |
+
|
| 168 |
+
**Apache-2.0**, inherited from
|
| 169 |
+
[`openbmb/MiniCPM-V-4.7-1B`](https://huggingface.co/openbmb/MiniCPM-V-4.7-1B).
|
| 170 |
+
Please credit OpenBMB for the original model. The RKNN3 conversion, the
|
| 171 |
+
196-token adapter and the host scripts in this repository are provided under
|
| 172 |
+
the same license, without warranty.
|
| 173 |
+
|
| 174 |
+
Built with [`airockchip/rknn3-model-zoo`](https://github.com/airockchip/rknn3-model-zoo) V1.1.0.
|
| 175 |
+
|
| 176 |
+
---
|
| 177 |
+
|
| 178 |
+
## 中文摘要
|
| 179 |
+
|
| 180 |
+
把 `openbmb/MiniCPM-V-4.7-1B` 转成 RKNN3,在 **RK1828** 上跑通完整图文链路
|
| 181 |
+
(文本塔 + 视觉塔 + `rkllm3-server --model2`),真机 5/5 带图与纯文本请求
|
| 182 |
+
全部 HTTP 200,NPU 独占占用 **1269 MB**。
|
| 183 |
+
|
| 184 |
+
**必须知道的取舍**:4.7 视觉塔原生只出 8×8=64 个视觉 token,而 RKNN3 runtime
|
| 185 |
+
的 multimodal 回调要求 `image_ph * image_pw = 14×14 = 196`。为满足契约,
|
| 186 |
+
导出图里加了 8×8→14×14 的最近邻重采样**适配层**(只改图结构、不改权重,
|
| 187 |
+
`.weight` 的 SHA256 与 64-token 版完全相同)。因此
|
| 188 |
+
**本产物不等价于 HF 原版 4.7**,不能拿它复现原版精度;
|
| 189 |
+
canvas mRoPE / 多分辨率切砖 / 高分辨 OCR / 密集计数均未验证。
|
| 190 |
+
|
| 191 |
+
| 项 | 结果 |
|
| 192 |
+
| --- | --- |
|
| 193 |
+
| 橙图问颜色 | `这张图是橙色。` |
|
| 194 |
+
| 黑方块 | `黑色正方形` |
|
| 195 |
+
| 交叉对照(排除语言先验猜答案) | 正确 |
|
| 196 |
+
| 纯文本 | `北京` |
|
| 197 |
+
| 与 ASR 共存 | 0.6B 3458 MB / 1.7B 3353 MB(A/B/A 可复现) |
|
| 198 |
+
| 视觉 token 注入证据 | `timings.prompt_n ≈ 218` |
|
| 199 |
+
|
| 200 |
+
仓库含 6 个转换产物 + `start.cmd` + 请求脚本 + 转换脚本 + 部署文档,
|
| 201 |
+
拿到即可跑;部署与十个实测踩过的坑见 [`docs/DEPLOY.md`](docs/DEPLOY.md)。
|
| 202 |
+
复杂图务必看 `finish_reason`,别只看 HTTP 200。
|
SHA256SUMS
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
fa50be8954ef102147e51fb9d6930a14ef3b6097b0d5732e156e1eb8113e3226 convert/README.md
|
| 2 |
+
35ef5e4568ef4ea0323158695379f2fd3be345d2a0c879ae563fc58186e23512 convert/export_vision47.py
|
| 3 |
+
b72cb20be6e6a28e3167222df5a1974fcd5356cf48a09caac2a07fff6d6692ec convert/export_vision_rknn.py
|
| 4 |
+
ad22d897e77ab8f3dbfbef90a4769e5fcc93d6cc9489f68913e0680d091de126 docs/DEPLOY.md
|
| 5 |
+
b0d4d2e702a6333941beed2064fdb1bf2a9e4fe8fca0f374964d8b754997a35c host/make_test_images.py
|
| 6 |
+
7ca759bedbc3e6db62aeaa20c9b23c4e0c47c555c0ab04da6800d4a02e8c9194 host/req_orange.json
|
| 7 |
+
69428abbd9c1642fcf1ccbcd2f47c9fd43a300f35e31cc5ed3b072f818a242ef host/req_text.json
|
| 8 |
+
8b65ac059d9465352eb4b720da2a9f4f6d05b042b42ffc79978a9ce51d1a2673 host/send_vlm_req.cmd
|
| 9 |
+
00f1f0eed92d5d2cf8ddb0033fdafcea7992886ce9b0b04f245f61ac9f3eb209 host/test_images.json
|
| 10 |
+
511c5794dbf3d927fb17c5efc0052fababafdd35185be2961f6a855c1852b335 rknn/MiniCPM-V-4.7-1B-llm.embed.bin
|
| 11 |
+
948e487bff9b2324b3016c665aa071feff05ac735e078c20ecb4a0ac7b825a1b rknn/MiniCPM-V-4.7-1B-llm.rknn
|
| 12 |
+
e028b0a11710ee5e0402c3829e871c3bd5c568540fe377febdd1a515d45b1544 rknn/MiniCPM-V-4.7-1B-llm.tokenizer.gguf
|
| 13 |
+
72fd0b16b8016d63a7d8b92f8d159ad7ea68a783eaca89b915bd0e240ae22dab rknn/MiniCPM-V-4.7-1B-llm.weight
|
| 14 |
+
9e081464857fad825281ae8778defb882424cbf5d46a37acb6de59dd5cbc99d6 rknn/MiniCPM-V-4.7-1B-vision.rknn
|
| 15 |
+
9c1939dd21e511c72ef6fa6333d43d962acc067d5302ed625815abeaf1cc1c03 rknn/MiniCPM-V-4.7-1B-vision.weight
|
| 16 |
+
5549fa0a8ce2e6ed739334adb5c52ec98eeb240a0675ce86b0ad51fe4646ddd5 rknn/start.cmd
|
convert/README.md
ADDED
|
@@ -0,0 +1,87 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Reproducing the conversion
|
| 2 |
+
|
| 3 |
+
These are **portable copies** of the two scripts that produced the vision tower
|
| 4 |
+
in `../rknn/`. All absolute paths have been replaced by arguments / environment
|
| 5 |
+
variables; the model math is unchanged.
|
| 6 |
+
|
| 7 |
+
## Prerequisites
|
| 8 |
+
|
| 9 |
+
- `rknn3-toolkit` (the whl from the RKNN3 SDK, installed with `--no-deps`)
|
| 10 |
+
- `transformers >= 5.18` (`MiniCPMV4_7ForConditionalGeneration` does not exist
|
| 11 |
+
in 5.3). Point `MINICPM_TF518` at a separate checkout if your main env is older.
|
| 12 |
+
- `torch`, `onnx`, `safetensors`
|
| 13 |
+
- A local checkout of [`openbmb/MiniCPM-V-4.7-1B`](https://huggingface.co/openbmb/MiniCPM-V-4.7-1B)
|
| 14 |
+
- Official [`airockchip/rknn3-model-zoo`](https://github.com/airockchip/rknn3-model-zoo) V1.1.0
|
| 15 |
+
for the LLM-side `export_llm.py` (the text tower is the same `Qwen3_5ForCausalLM`
|
| 16 |
+
family as the zoo's `Qwen3_5` example).
|
| 17 |
+
|
| 18 |
+
## 1. Text tower (four artifacts)
|
| 19 |
+
|
| 20 |
+
The 4.7 text tower is architecturally Qwen3.5-0.8B with a different vocabulary:
|
| 21 |
+
|
| 22 |
+
| Item | 0.8B | MiniCPM-V 4.7 1B |
|
| 23 |
+
| --- | --- | --- |
|
| 24 |
+
| `vocab_size` | 248320 | **248094** |
|
| 25 |
+
| `tie_word_embeddings` | true | **false** (independent `lm_head.weight` [248094,1024]) |
|
| 26 |
+
| `conv_state` | [18,6144,4] | [18,6144,4] (identical) |
|
| 27 |
+
|
| 28 |
+
Build a plain-text HF directory by keeping only `model.language_model.*` tensors
|
| 29 |
+
plus `lm_head.weight`, copying the 4.7 tokenizer files, and starting from the
|
| 30 |
+
zoo's Qwen3.5 `config.json` with the two changes above. Then run the zoo's
|
| 31 |
+
`Qwen3_5/python/export_llm.py` followed by `export_rknn.py` with
|
| 32 |
+
`w4a16 / normal / group32 / core_num=8`.
|
| 33 |
+
|
| 34 |
+
> **Top-level `tie_word_embeddings` trap.** Hugging Face `tie_weights()` reads the
|
| 35 |
+
> **top-level** config key, not `text_config`. Changing only `text_config` leaves
|
| 36 |
+
> the independent 254M-parameter head silently bound to `embed_tokens`, and the
|
| 37 |
+
> converted model emits word salad. Change **both**. The only reliable check is
|
| 38 |
+
> `cmp`-ing the produced `embed.bin` against the head weights — file sizes can be
|
| 39 |
+
> byte-identical between a good and a bad build.
|
| 40 |
+
|
| 41 |
+
## 2. Vision tower (two artifacts)
|
| 42 |
+
|
| 43 |
+
```bash
|
| 44 |
+
export MINICPM_HF_DIR=/path/to/MiniCPM-V-4.7-1B
|
| 45 |
+
export MINICPM_TF518=/path/to/transformers-5.18 # optional
|
| 46 |
+
export MINICPM_VISION_ONNX=MiniCPM-V-4.7-1B-vision.onnx
|
| 47 |
+
export MINICPM_VISION_RKNN=MiniCPM-V-4.7-1B-vision.rknn
|
| 48 |
+
|
| 49 |
+
python export_vision47.py --model "$MINICPM_HF_DIR" --out "$MINICPM_VISION_ONNX"
|
| 50 |
+
python export_vision_rknn.py
|
| 51 |
+
```
|
| 52 |
+
|
| 53 |
+
`export_vision_rknn.py` produces `<name>.rknn` and `<name>.weight` side by side.
|
| 54 |
+
|
| 55 |
+
### Two non-obvious contracts
|
| 56 |
+
|
| 57 |
+
1. **The output must be 3D `[1, N, hidden]`, not 2D `[N, hidden]`.**
|
| 58 |
+
The runtime reads `embeds_shape[0]` as *number of images* and
|
| 59 |
+
`embeds_shape[1]` as *tokens per image* (see the official
|
| 60 |
+
`examples/MiniCPM_V_4/cpp/main.cc`). A 2D export is parsed as
|
| 61 |
+
"64 images of 1024 tokens each" and the image tokens never line up.
|
| 62 |
+
|
| 63 |
+
2. **`image_ph * image_pw == n_image_tokens`.** The multimodal embed callback
|
| 64 |
+
computes `image_ph = model_height / patch_size(16) / spatial_merge_size(2)`;
|
| 65 |
+
with a 448x448 vision input that is `14 x 14 = 196`. MiniCPM-V 4.7's native
|
| 66 |
+
tower emits `(448/14/4)^2 = 8 x 8 = 64` tokens. A 64-token export fails on
|
| 67 |
+
device with:
|
| 68 |
+
|
| 69 |
+
```text
|
| 70 |
+
image_ph * image_pw != n_image_tokens! image_ph = 14 ... n_image_tokens = 64
|
| 71 |
+
```
|
| 72 |
+
|
| 73 |
+
Changing `--img-width/--img-height` or resizing the request image does **not**
|
| 74 |
+
help: `image_height` comes from the vision RKNN model's own input shape.
|
| 75 |
+
`export_vision47.py` therefore inserts a **nearest-neighbour 8x8 -> 14x14
|
| 76 |
+
resample** inside the exported graph (pure Gather/Reshape, no `Resize` op):
|
| 77 |
+
|
| 78 |
+
```python
|
| 79 |
+
g, idx = self.src_grid, self.resample_idx # 8, 14
|
| 80 |
+
y = y.view(g, g, -1)[idx[:, None], idx[None, :], :]
|
| 81 |
+
return y.reshape(1, 196, -1)
|
| 82 |
+
```
|
| 83 |
+
|
| 84 |
+
This is a **runtime-contract adapter, not native 4.7 behaviour**. It changes
|
| 85 |
+
only graph structure — the `.weight` SHA256 is identical to the 64-token
|
| 86 |
+
build, since no weights are touched. See the model card for what this means
|
| 87 |
+
for equivalence claims.
|
convert/export_vision47.py
ADDED
|
@@ -0,0 +1,118 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""MiniCPM-V 4.7 1B 视觉塔 -> ONNX(固定分辨率、单图、16x 压缩)。
|
| 2 |
+
|
| 3 |
+
要点:
|
| 4 |
+
- 4.7 的建模类在 transformers>=5.18,本地 zoo venv 是 5.3.0,
|
| 5 |
+
因此把 5.18.0 单独装到 minicpm47/tf518,用 sys.path 前置覆盖,不污染原环境。
|
| 6 |
+
- vision_tower.forward 内部有 ViT 内压缩(insert_layer_id=6 的 vit_merger,
|
| 7 |
+
2x2 window merge),merger 再 2x2 -> 合计 4x4 patch = 1 个 LLM token(16x)。
|
| 8 |
+
- 固定 448x448:grid=32 -> vit_merger 后 16x16 -> merger 后 8x8 = 64 个 token,
|
| 9 |
+
每个 1024 维(= LLM hidden_size)。32 能被 4 整除,980/14=70 不行,别用 980 单图。
|
| 10 |
+
- RKNN 侧用 mean/std=0.5*255 + uint8 NHWC 输入,ONNX 只接受已归一化的 fp32 NCHW。
|
| 11 |
+
- 输出必须是 3D [1, N, hidden],不能是 2D [N, hidden]!依据官方
|
| 12 |
+
examples/MiniCPM_V_4/cpp/main.cc:
|
| 13 |
+
tensor.image.n_image = embeds_shape[0]; // 图片数
|
| 14 |
+
tensor.image.n_image_tokens = embeds_shape[1]; // 每图 token 数
|
| 15 |
+
rkllm3-server 用同一套 runtime 约定。2D 输出会被读成"64 张图、每张 1024 token",
|
| 16 |
+
导致 image token 数与 vision embedding 完全错位。官方 4.0 的
|
| 17 |
+
export_minicpm3o_vision likewise 返回 B*Q*D 三维。
|
| 18 |
+
- **runtime 契约适配(11-4 上板实测)**:multimodal embed callback 还会校验
|
| 19 |
+
`image_ph * image_pw == n_image_tokens`,其中
|
| 20 |
+
`image_ph = model_height / patch_size(16) / spatial_merge_size(2)`。
|
| 21 |
+
448 输入 -> 14x14 = **196**,而 4.7 视觉塔原生只有 8x8 = **64**,直接上板报
|
| 22 |
+
`image_ph * image_pw != n_image_tokens! image_ph = 14 ... n_image_tokens = 64`。
|
| 23 |
+
`image_height` 取自 vision RKNN 模型的输入尺寸:改 `--img-width/--img-height`
|
| 24 |
+
到 256、把请求图片换成 256x256,报错里都仍是 448(均已实测)。
|
| 25 |
+
故在导出图内把 8x8 网格最近邻重采样到 14x14 = 196 token 让契约对齐。
|
| 26 |
+
这是**适配层,不是 4.7 原生行为**;视觉信息有冗余,但空间排布保留。
|
| 27 |
+
"""
|
| 28 |
+
import argparse
|
| 29 |
+
import os
|
| 30 |
+
import sys
|
| 31 |
+
|
| 32 |
+
import torch
|
| 33 |
+
|
| 34 |
+
TF518 = os.environ.get("MINICPM_TF518", "")
|
| 35 |
+
if TF518:
|
| 36 |
+
sys.path.insert(0, TF518)
|
| 37 |
+
|
| 38 |
+
MODEL = os.environ.get("MINICPM_HF_DIR", "")
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
class VisionExport(torch.nn.Module):
|
| 42 |
+
def __init__(self, model, img_size):
|
| 43 |
+
super().__init__()
|
| 44 |
+
self.vt = model.model.vision_tower
|
| 45 |
+
self.merger = model.model.merger
|
| 46 |
+
patch = self.vt.config.patch_size
|
| 47 |
+
grid = img_size // patch
|
| 48 |
+
assert grid % 4 == 0, f"patch grid {grid} 必须能被 4 整除(16x 压缩),换分辨率"
|
| 49 |
+
self.register_buffer(
|
| 50 |
+
"target_sizes", torch.tensor([[grid, grid]], dtype=torch.int32), persistent=False
|
| 51 |
+
)
|
| 52 |
+
# merger 内部用 height/width 作 view() 的形状参数;传 Python int 才能被 ONNX trace。
|
| 53 |
+
# vision_tower 在 insert_layer 处把 target_sizes 减半,故这里也减半。
|
| 54 |
+
self.merger_sizes = [[grid // 2, grid // 2]]
|
| 55 |
+
# ---- runtime 契约适配(见文件头 docstring)----
|
| 56 |
+
# 视觉塔原生 (grid/4)^2 = 8x8 = 64 token,runtime 要求 14x14 = 196。
|
| 57 |
+
# 用纯 Gather/Reshape 做最近邻重采样,不引入 Resize op,RKNN 转换最稳。
|
| 58 |
+
self.src_grid = grid // 4
|
| 59 |
+
self.out_grid = 14
|
| 60 |
+
self.register_buffer(
|
| 61 |
+
"resample_idx",
|
| 62 |
+
(torch.arange(self.out_grid, dtype=torch.float32) * self.src_grid / self.out_grid).long(),
|
| 63 |
+
persistent=False,
|
| 64 |
+
)
|
| 65 |
+
|
| 66 |
+
def forward(self, pixel):
|
| 67 |
+
out = self.vt(pixel, target_sizes=self.target_sizes, use_vit_merger=True)
|
| 68 |
+
hs = out.last_hidden_state if hasattr(out, "last_hidden_state") else out[0]
|
| 69 |
+
feats = self.merger(hs, self.merger_sizes)
|
| 70 |
+
y = torch.cat(feats, dim=0) # [src^2, hidden],行优先 8x8 网格
|
| 71 |
+
g, idx = self.src_grid, self.resample_idx
|
| 72 |
+
y = y.view(g, g, -1)[idx[:, None], idx[None, :], :] # [14, 14, hidden]
|
| 73 |
+
# [N, hidden] -> [1, N, hidden]:server 按 dim0=图片数 / dim1=每图token数解析。
|
| 74 |
+
return y.reshape(1, self.out_grid * self.out_grid, -1)
|
| 75 |
+
|
| 76 |
+
|
| 77 |
+
def main():
|
| 78 |
+
ap = argparse.ArgumentParser()
|
| 79 |
+
ap.add_argument("--img_size", type=int, default=448)
|
| 80 |
+
ap.add_argument(
|
| 81 |
+
"--out",
|
| 82 |
+
default=os.environ.get("MINICPM_VISION_ONNX", "MiniCPM-V-4.7-1B-vision.onnx"),
|
| 83 |
+
)
|
| 84 |
+
ap.add_argument("--model", default=MODEL,
|
| 85 |
+
help="Local HF checkout of openbmb/MiniCPM-V-4.7-1B")
|
| 86 |
+
ap.add_argument("--smoke", action="store_true", help="只跑一次 eager forward,不导出")
|
| 87 |
+
args = ap.parse_args()
|
| 88 |
+
|
| 89 |
+
from transformers import AutoModelForImageTextToText
|
| 90 |
+
|
| 91 |
+
model = AutoModelForImageTextToText.from_pretrained(
|
| 92 |
+
args.model, dtype=torch.float32, trust_remote_code=True, low_cpu_mem_usage=True
|
| 93 |
+
)
|
| 94 |
+
model.eval()
|
| 95 |
+
wrapper = VisionExport(model, args.img_size).eval()
|
| 96 |
+
|
| 97 |
+
pixel = torch.randn(1, 3, args.img_size, args.img_size)
|
| 98 |
+
with torch.no_grad():
|
| 99 |
+
y = wrapper(pixel)
|
| 100 |
+
print("eager output shape:", tuple(y.shape), "dtype:", y.dtype, flush=True)
|
| 101 |
+
if args.smoke:
|
| 102 |
+
return
|
| 103 |
+
|
| 104 |
+
os.makedirs(os.path.dirname(args.out), exist_ok=True)
|
| 105 |
+
torch.onnx.export(
|
| 106 |
+
wrapper,
|
| 107 |
+
(pixel,),
|
| 108 |
+
args.out,
|
| 109 |
+
input_names=["pixel"],
|
| 110 |
+
output_names=["img_embeds"],
|
| 111 |
+
opset_version=17,
|
| 112 |
+
do_constant_folding=True,
|
| 113 |
+
)
|
| 114 |
+
print("exported:", args.out, os.path.getsize(args.out), "bytes", flush=True)
|
| 115 |
+
|
| 116 |
+
|
| 117 |
+
if __name__ == "__main__":
|
| 118 |
+
main()
|
convert/export_vision_rknn.py
ADDED
|
@@ -0,0 +1,57 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""MiniCPM-V 4.7 1B vision ONNX -> RKNN (w4a16 / normal / group32 / 8 cores)。
|
| 2 |
+
|
| 3 |
+
参数照抄官方 zoo examples/MiniCPM_V_4/python/vision/export_rknn.py:
|
| 4 |
+
uint8 NHWC 输入 + mean/std=0.5*255(RKNN 在输入端做归一化),
|
| 5 |
+
ONNX 里只保留已归一化 fp32 NCHW 的计算图。
|
| 6 |
+
"""
|
| 7 |
+
import os
|
| 8 |
+
|
| 9 |
+
from rknn.api import RKNN
|
| 10 |
+
|
| 11 |
+
ONNX = os.environ.get("MINICPM_VISION_ONNX", "MiniCPM-V-4.7-1B-vision.onnx")
|
| 12 |
+
OUT = os.environ.get("MINICPM_VISION_RKNN", "MiniCPM-V-4.7-1B-vision.rknn")
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
def main():
|
| 16 |
+
rknn = RKNN(verbose=True)
|
| 17 |
+
|
| 18 |
+
print("--> config model", flush=True)
|
| 19 |
+
rknn.config(
|
| 20 |
+
target_platform="rk1820",
|
| 21 |
+
quantized_dtype="w4a16",
|
| 22 |
+
quantized_algorithm="normal",
|
| 23 |
+
quantized_method="group32",
|
| 24 |
+
core_num=8,
|
| 25 |
+
mean_values=[[0.5 * 255, 0.5 * 255, 0.5 * 255]],
|
| 26 |
+
std_values=[[0.5 * 255, 0.5 * 255, 0.5 * 255]],
|
| 27 |
+
input_attrs={"pixel": {"dtype": "uint8", "layout": "NHWC"}},
|
| 28 |
+
)
|
| 29 |
+
print("done", flush=True)
|
| 30 |
+
|
| 31 |
+
print("--> Loading model", flush=True)
|
| 32 |
+
ret = rknn.load_onnx(model=ONNX)
|
| 33 |
+
if ret != 0:
|
| 34 |
+
print("Load model failed!", ret, flush=True)
|
| 35 |
+
return ret
|
| 36 |
+
print("done", flush=True)
|
| 37 |
+
|
| 38 |
+
print("--> Building model", flush=True)
|
| 39 |
+
ret = rknn.build(do_quantization=True)
|
| 40 |
+
if ret != 0:
|
| 41 |
+
print("Build model failed!", ret, flush=True)
|
| 42 |
+
return ret
|
| 43 |
+
print("done", flush=True)
|
| 44 |
+
|
| 45 |
+
print("--> Export RKNN model", flush=True)
|
| 46 |
+
ret = rknn.export_rknn(OUT)
|
| 47 |
+
if ret != 0:
|
| 48 |
+
print("Export rknn failed!", ret, flush=True)
|
| 49 |
+
return ret
|
| 50 |
+
print("done", flush=True)
|
| 51 |
+
|
| 52 |
+
rknn.release()
|
| 53 |
+
return 0
|
| 54 |
+
|
| 55 |
+
|
| 56 |
+
if __name__ == "__main__":
|
| 57 |
+
raise SystemExit(main())
|
docs/DEPLOY.md
ADDED
|
@@ -0,0 +1,143 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Deployment guide — MiniCPM-V 4.7 1B on RK1828
|
| 2 |
+
|
| 3 |
+
## 0. What you need
|
| 4 |
+
|
| 5 |
+
| Item | Requirement |
|
| 6 |
+
| --- | --- |
|
| 7 |
+
| NPU module | Rockchip **RK1828 / RK1820** (tested on `RM1828MC0-F`, firmware 1.1.0) |
|
| 8 |
+
| Host | Windows PC, or RK3588 / RK3576 board. **Ubuntu x86 cannot drive the module directly** (conversion / simulator only) |
|
| 9 |
+
| Runtime | `rknn3_110` — `rkllm3-server.exe`, matching `toolkit / proxy / runtime / firmware` versions |
|
| 10 |
+
| Session | Windows **Session 1** (interactive logon or an `InteractiveToken` scheduled task) |
|
| 11 |
+
|
| 12 |
+
Component versions **must match**. A mismatched `rknn3_110` typically fails with
|
| 13 |
+
a DLL-load error that prints **nothing at all**.
|
| 14 |
+
|
| 15 |
+
## 1. Layout
|
| 16 |
+
|
| 17 |
+
Put the runtime next to `rknn/`:
|
| 18 |
+
|
| 19 |
+
```text
|
| 20 |
+
<root>\
|
| 21 |
+
rknn\ <- this repo's rknn/ folder (6 artifacts + start.cmd)
|
| 22 |
+
runtime\rknn3_110\bin\rkllm3-server.exe
|
| 23 |
+
runtime\rknn3_110\lib\*.dll
|
| 24 |
+
```
|
| 25 |
+
|
| 26 |
+
`rknn/start.cmd` looks for the runtime in `..\runtime\rknn3_110`, then
|
| 27 |
+
`..\..\runtime\rknn3_110`, then the `RK_RUNTIME` environment variable.
|
| 28 |
+
If none is found it **exits with an error** rather than silently falling back
|
| 29 |
+
to a hard-coded drive.
|
| 30 |
+
|
| 31 |
+
Override the listen address / port with `RK_HOST` / `RK_PORT`
|
| 32 |
+
(defaults `0.0.0.0` / `8899`).
|
| 33 |
+
|
| 34 |
+
```cmd
|
| 35 |
+
cd /d <root>\rknn
|
| 36 |
+
start.cmd
|
| 37 |
+
```
|
| 38 |
+
|
| 39 |
+
Wait until the server is actually ready. The log can sit at
|
| 40 |
+
`Weight sync chunk size resolved` for a couple of minutes — **do not treat that
|
| 41 |
+
as a hang**, and do not run `rknn-smi` while weights are loading (it can wedge
|
| 42 |
+
the sync). Readiness is proven only by a real inference request.
|
| 43 |
+
|
| 44 |
+
## 2. Start the communication proxy
|
| 45 |
+
|
| 46 |
+
One `rknn3_transfer_proxy` process per machine, **reuse it**. Repeatedly
|
| 47 |
+
tearing it down and recreating it can drop the module off the USB bus.
|
| 48 |
+
If you use the RK1828 web console, let the console own the proxy.
|
| 49 |
+
|
| 50 |
+
## 3. Send an image request
|
| 51 |
+
|
| 52 |
+
The server exposes an OpenAI-compatible endpoint. `messages[].content` must be
|
| 53 |
+
an **array**, and the image goes in as a base64 data URI:
|
| 54 |
+
|
| 55 |
+
```json
|
| 56 |
+
{
|
| 57 |
+
"messages": [{
|
| 58 |
+
"role": "user",
|
| 59 |
+
"content": [
|
| 60 |
+
{ "type": "text", "text": "图里是什么?" },
|
| 61 |
+
{ "type": "image_url", "image_url": { "url": "data:image/png;base64,<BASE64>" } }
|
| 62 |
+
]
|
| 63 |
+
}],
|
| 64 |
+
"max_tokens": 512
|
| 65 |
+
}
|
| 66 |
+
```
|
| 67 |
+
|
| 68 |
+
**Do not put `<image>` / `<|image_pad|>` placeholder tokens into the text.**
|
| 69 |
+
The server constructs them itself. Ready-made bodies are in `host/`:
|
| 70 |
+
|
| 71 |
+
```cmd
|
| 72 |
+
cd /d <root>\host
|
| 73 |
+
host\make_test_images.py :: generates test_images.json (pure stdlib)
|
| 74 |
+
send_vlm_req.cmd orange :: -> 127.0.0.1:8899
|
| 75 |
+
send_vlm_req.cmd text 8888 :: override port
|
| 76 |
+
```
|
| 77 |
+
|
| 78 |
+
Use **`curl.exe --max-time`**. Windows PowerShell `Invoke-WebRequest` /
|
| 79 |
+
`Invoke-RestMethod` hang indefinitely against this server (no response, no
|
| 80 |
+
`-OutFile`), and `-TimeoutSec` does not always rescue it.
|
| 81 |
+
|
| 82 |
+
Expect roughly `HTTP=200 TIME=0.6s` for a simple image. A correct response
|
| 83 |
+
contains `timings.prompt_n ≈ 218` — that is 196 visual tokens plus text tokens,
|
| 84 |
+
and is the evidence that vision embeddings really entered the LLM.
|
| 85 |
+
|
| 86 |
+
## 4. Verified reference results
|
| 87 |
+
|
| 88 |
+
| Check | Result |
|
| 89 |
+
| --- | --- |
|
| 90 |
+
| Orange image, "what colour" | `这张图是橙色。` |
|
| 91 |
+
| Black square image | `黑色正方形` |
|
| 92 |
+
| Cross-checks (orange asked for shape, square asked for colour) | correct, no language-prior guessing |
|
| 93 |
+
| Pure text "中国的首都是哪里" | `北京` |
|
| 94 |
+
| `finish_reason` | `stop` on simple cases |
|
| 95 |
+
| NPU, VLM alone | **1269 MB** (exclusive differential; console shows 1303 incl. 34 MB baseline) |
|
| 96 |
+
| NPU, VLM + Qwen3-ASR-0.6B | 3458 MB |
|
| 97 |
+
| NPU, VLM + Qwen3-ASR-1.7B | 3353 MB (A/B/A reproducible) |
|
| 98 |
+
| Generation | ~90 tok/s, prefill ~1700 tok/s |
|
| 99 |
+
|
| 100 |
+
Five request/response pairs plus the full console acceptance run are documented
|
| 101 |
+
in the upstream project docs; the ready-made request bodies here reproduce the
|
| 102 |
+
two simplest ones.
|
| 103 |
+
|
| 104 |
+
## 5. Pitfalls (all hit in practice)
|
| 105 |
+
|
| 106 |
+
1. **Session 0 breaks the USB bulk channel.** Starting the server from an SSH
|
| 107 |
+
session lands it in Session 0; `rknn3_init` succeeds but weight sync never
|
| 108 |
+
completes. Use `schtasks` with `InteractiveToken` (Session 1). Diagnose with
|
| 109 |
+
a *different, known-good* model — if both fail identically, suspect the
|
| 110 |
+
session, not the USB port or the firmware.
|
| 111 |
+
2. **`tie_word_embeddings` at the top level.** A converted graph without a real
|
| 112 |
+
`lm_head` makes `load_llm` **segfault silently**. See `convert/README.md`.
|
| 113 |
+
3. **Vision output shape and token count.** 3D `[1,N,1024]` and
|
| 114 |
+
`N == image_ph * image_pw == 196`. See `convert/README.md`.
|
| 115 |
+
4. **Placeholder tokens must match this model's own vocabulary.** MiniCPM-V 4.7
|
| 116 |
+
uses bare `<image>` / `</image>` / `<|image_pad|>`. Its GGUF vocabulary is a
|
| 117 |
+
**superset** that also contains Qwen-VL's `<|vision_start|>` /
|
| 118 |
+
`<|vision_end|>`; a detector that checks Qwen-VL first will silently rewrite
|
| 119 |
+
a correct `start.cmd` into a broken one.
|
| 120 |
+
5. **`-c` is context length, not core mask.** Use `--core-mask` /
|
| 121 |
+
`--core-mask2`; 8 cores = `0xFF`. A zero core mask makes
|
| 122 |
+
`rknn3_model_init` return `-2`.
|
| 123 |
+
6. **`.cmd` / `.bat` / `.ps1` must be pure ASCII + CRLF**, and `PATH` must be
|
| 124 |
+
set before launching the child process (the launch script does this).
|
| 125 |
+
7. **Check `finish_reason`.** This is a 1B model with a `<think>` chain. Complex
|
| 126 |
+
images can fill `max_tokens` and return `finish_reason=length` with the
|
| 127 |
+
answer truncated. Increasing `max_tokens` is **not** monotonically safer
|
| 128 |
+
(256 worked for one OCR case where 512 stalled inside `<think>`).
|
| 129 |
+
8. **Residual processes hold the port** → "I changed the code but behaviour did
|
| 130 |
+
not change". Check `Get-NetTCPConnection` before restarting.
|
| 131 |
+
9. **ASR device errors can leak NPU memory** (`mem_sync ret=-10` retry loop).
|
| 132 |
+
Workaround: `taskkill /F /IM rknn_qwen3_asr_demo.exe`. Not fixed upstream.
|
| 133 |
+
10. **Power loss may land the module in MaskROM** (`2207:180B`); `upgrade_tool uf`
|
| 134 |
+
recovers it in ~2 s. A genuine hard hang needs a physical power cycle.
|
| 135 |
+
|
| 136 |
+
## 6. Known limitations — read before filing issues
|
| 137 |
+
|
| 138 |
+
- The 196-token resample is a **runtime-contract adapter**. This build is
|
| 139 |
+
**not equivalent to the Hugging Face original MiniCPM-V 4.7 1B**.
|
| 140 |
+
- canvas mRoPE, multi-resolution tiling, high-resolution OCR and dense
|
| 141 |
+
multi-object counting were **not verified**.
|
| 142 |
+
- OCR is "characters readable" at best: `RK1828` may come back as `RK₁₈₂₈`.
|
| 143 |
+
- Only 448x448 single-image input was tested.
|
host/make_test_images.py
ADDED
|
@@ -0,0 +1,65 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""生成两张 448x448 测试图(纯标准库,不依赖 PIL),供 VLM 带图请求使用。
|
| 2 |
+
|
| 3 |
+
- orange: 纯橙色。模型若真"看到"图,问颜色应答橙色;瞎猜则说明视觉链路没通。
|
| 4 |
+
- square: 白底黑方块。用于形状/内容识别的第二个判别样本。
|
| 5 |
+
|
| 6 |
+
输出 test_images.json,内容为 OpenAI image_url 用的 data URI。
|
| 7 |
+
"""
|
| 8 |
+
import base64
|
| 9 |
+
import json
|
| 10 |
+
import os
|
| 11 |
+
import struct
|
| 12 |
+
import zlib
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
def png_chunk(tag, data):
|
| 16 |
+
return (
|
| 17 |
+
struct.pack(">I", len(data))
|
| 18 |
+
+ tag
|
| 19 |
+
+ data
|
| 20 |
+
+ struct.pack(">I", zlib.crc32(tag + data) & 0xFFFFFFFF)
|
| 21 |
+
)
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def write_png(w, h, pixel):
|
| 25 |
+
raw = bytearray()
|
| 26 |
+
for y in range(h):
|
| 27 |
+
raw.append(0) # filter type 0 (None)
|
| 28 |
+
for x in range(w):
|
| 29 |
+
raw.extend(pixel(x, y))
|
| 30 |
+
comp = zlib.compress(bytes(raw), 9)
|
| 31 |
+
return (
|
| 32 |
+
b"\x89PNG\r\n\x1a\n"
|
| 33 |
+
+ png_chunk(b"IHDR", struct.pack(">IIBBBBB", w, h, 8, 2, 0, 0, 0))
|
| 34 |
+
+ png_chunk(b"IDAT", comp)
|
| 35 |
+
+ png_chunk(b"IEND", b"")
|
| 36 |
+
)
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
def main():
|
| 40 |
+
# 448x448:与 vision 模型输入尺寸一致,避免 server 额外缩放。
|
| 41 |
+
w = h = 448
|
| 42 |
+
orange = write_png(w, h, lambda x, y: (255, 128, 0))
|
| 43 |
+
|
| 44 |
+
def square_px(x, y):
|
| 45 |
+
if 144 <= x < 304 and 144 <= y < 304:
|
| 46 |
+
return (0, 0, 0)
|
| 47 |
+
return (255, 255, 255)
|
| 48 |
+
|
| 49 |
+
square = write_png(w, h, square_px)
|
| 50 |
+
|
| 51 |
+
out = {
|
| 52 |
+
"orange": "data:image/png;base64," + base64.b64encode(orange).decode(),
|
| 53 |
+
"square": "data:image/png;base64," + base64.b64encode(square).decode(),
|
| 54 |
+
}
|
| 55 |
+
path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "test_images.json")
|
| 56 |
+
with open(path, "w") as f:
|
| 57 |
+
json.dump(out, f)
|
| 58 |
+
print("orange png bytes:", len(orange), "square png bytes:", len(square))
|
| 59 |
+
print("orange data uri len:", len(out["orange"]))
|
| 60 |
+
print("square data uri len:", len(out["square"]))
|
| 61 |
+
print("saved:", path)
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
if __name__ == "__main__":
|
| 65 |
+
main()
|
host/req_orange.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"messages": [{"role": "user", "content": [{"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAcAAAAHACAIAAAC6Ry8kAAAEl0lEQVR42u3UMQ0AAAzDsEIf85XDtNOSEeRIdgLAgQQABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAABgqAgQIYKICBAhgoAAYKYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKgIECGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECYKAABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAAGCiAgQIYKICBAhgoAAYKYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKgIECGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECYKAABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAAGCiAgQIYKICBAhgoAAYKYKAABgpgoAAGCoCBAhgogIECGCgABgpgoAAGCmCgAAYKgIECGCiAgQIYKICBAmCgAAYKYKAABgpgoCoAGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECGCgABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAABgqAgQIYKICBAhgoAAYKYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKgIECGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECGKgEAAYKYKAABgpgoAAYKICBAhgogIECGCgABgpgoAAGCmCgAAYKgIECGCiAgQIYKAAGCmCgAAYKYKAABgqAgQIYKICBAhgogIECYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKYKAAGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECGCgABgpgoAAGCmCgAEgAYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKYKAAGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECGCgABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAABgqAgQIYKICBAhgoAAYKYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKgIECGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECGCgABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAABgqAgQIYKICBAhgoAAYKYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKgIECGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECYKAABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAABqoCgIECGCiAgQIYKAAGCmCgAAYKYKAABgqAgQIYKICBAhgogIECYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKYKAAGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECGCgABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCvCosKM0vFDSA3UAAAAASUVORK5CYII="}}, {"type": "text", "text": "这张图是什么颜色?请直接回答颜色名称。"}]}], "max_tokens": 256}
|
host/req_text.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"messages": [{"role": "user", "content": "中国的首都是哪里?请直接回答。"}], "max_tokens": 64}
|
host/send_vlm_req.cmd
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
@echo off
|
| 2 |
+
rem =====================================================================
|
| 3 |
+
rem Send one request to the rkllm3-server started by rknn\start.cmd.
|
| 4 |
+
rem Usage: send_vlm_req.cmd <name> [port]
|
| 5 |
+
rem <name> req_<name>.json must exist next to this script
|
| 6 |
+
rem [port] default 8899 (matches rknn\start.cmd RK_PORT default)
|
| 7 |
+
rem IMPORTANT: keep this file pure ASCII + CRLF.
|
| 8 |
+
rem =====================================================================
|
| 9 |
+
setlocal
|
| 10 |
+
set "BASE=%~dp0"
|
| 11 |
+
set "NAME=%~1"
|
| 12 |
+
if "%NAME%"=="" set "NAME=orange"
|
| 13 |
+
set "PORT=%~2"
|
| 14 |
+
if "%PORT%"=="" set "PORT=8899"
|
| 15 |
+
curl.exe -s -X POST http://127.0.0.1:%PORT%/v1/chat/completions -H "Content-Type: application/json" --data-binary "@%BASE%req_%NAME%.json" -o "%BASE%resp_%NAME%.json" -w "HTTP=%%{http_code} TIME=%%{time_total}s PORT=%PORT%" --max-time 180
|
| 16 |
+
echo.
|
| 17 |
+
exit /b 0
|
host/test_images.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"orange": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAcAAAAHACAIAAAC6Ry8kAAAEl0lEQVR42u3UMQ0AAAzDsEIf85XDtNOSEeRIdgLAgQQABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAABgqAgQIYKICBAhgoAAYKYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKgIECGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECYKAABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAAGCiAgQIYKICBAhgoAAYKYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKgIECGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECYKAABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAAGCiAgQIYKICBAhgoAAYKYKAABgpgoAAGCoCBAhgogIECGCgABgpgoAAGCmCgAAYKgIECGCiAgQIYKICBAmCgAAYKYKAABgpgoCoAGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECGCgABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAABgqAgQIYKICBAhgoAAYKYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKgIECGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECGKgEAAYKYKAABgpgoAAYKICBAhgogIECGCgABgpgoAAGCmCgAAYKgIECGCiAgQIYKAAGCmCgAAYKYKAABgqAgQIYKICBAhgogIECYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKYKAAGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECGCgABgpgoAAGCmCgAEgAYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKYKAAGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECGCgABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAABgqAgQIYKICBAhgoAAYKYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKgIECGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECGCgABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAABgqAgQIYKICBAhgoAAYKYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKgIECGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECYKAABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCmCgAAYKYKAABqoCgIECGCiAgQIYKAAGCmCgAAYKYKAABgqAgQIYKICBAhgogIECYKAABgpgoAAGCoCBAhgogIECGCiAgQJgoAAGCmCgAAYKYKAAGCiAgQIYKICBAmCgAAYKYKAABgpgoAAYKICBAhgogIECGCgABgpgoAAGCmCgABgogIECGCiAgQIYKAAGCvCosKM0vFDSA3UAAAAASUVORK5CYII=", "square": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAcAAAAHACAIAAAC6Ry8kAAAGKElEQVR42u3USQ0AAAgEsfVvGiwQwou0EjgmBcBKjABAQAEEFEBAAQQUAAEFEFAAAQUQUAABBUBAAQQUQEABBBRAQAEQUAABBRBQAAEFQEABBBRAQAEEFEBAARBQAAEFEFAAAQUQUAAEFEBAAQQUQEABEFAAAQUQUAABBRBQAAQUQEABBBRAQAEEFAABBRBQAAEFEFAABBRAQAEEFEBAAQQUAAEFEFAAAQUQUAABBUBAAQQUQEABBBQAAQUQUAABBRBQAAEFQEABBBRAQAEEFEBAARBQAAEFEFAAAQUQUAAEFEBAAQQUQEABEFAAAQUQUAABBRBQAAQUQEABBBRAQAEEFAABBRBQAAEFEFAABBRAQAEEFEBAAQQUAAEFEFAAAQUQUAABBUBAAQQUQEABBBQAAQUQUAABBRBQAAEFQEABBBRAQAEEFEBAARBQAAEFEFAAAQVAQAEEFEBAAQQUQEABEFAAAQUQUAABBRBQAAQUQEABBBRAQAEEFAABBRBQAAEFEFAABBRAQAEElPnCeM2FCygCioAKKAKKgCKgAoqAIqACioAioAgoAiqgCCgCKqAIKAKKgAooAoqAIqACioAioAiogCKgCCgCKqAIKAIqoAgoAoqACigCioAioAKKgCKgCKiAIqAIqIAioAgoAoqACigCioAKKAKKgCKgAoqAIqAIqIAioAgoAiqgCCgCioAKKAKKgAooAoqAIqACioAioAiogCKgCCgCKqAIKAIqoAgoAoqAIqACioAioAKKgCKgCKiAIqAIKAIqoAgoAoqACigCioAKKAKKgCKgAoqAIqAIqIAioAgoAiqgCCgCioAKKAKKgAooAoqAIqAIqIAioAiogCKgCCgCKqAIKAKKgAooAoqAIqACioAioAKKgCKgCKiAIqAIKAIqoAgoAoqACigCioAioAKKgCKgAoqAIqAIKP5RQBFQBFRAEVAEFAEVUAQUAbUwBBQBRUARUAFFQBFQAUVAEVAEVEARUAQUARVQBBQBRUAFFAFFQBFQAUVAEVABRUARUARUQBFQBBQBFVAEFAFFQAUUAUVABRQBRUARUARUQBFQBFRAEVAEFAEVUAQUAUVABRQBRUARUAFFQBFQBFRAEVAEVEARUAQUARVQBBQBRUAFFAFFQBFQAUVAEVABRUARUAQUARVQBBQBFVAEFAFFQAUUAUVAEVABRUARUARUQBFQBFRAEVAEFAEVUAQUAUVABRQBRUARUAFFQBFQBFRAEVAEVEARUAQUAUVABRQBRUAFFAFFQBFQAUVAEVAEVEARUAQUARVQBBQBFVAEFAFFQAUUAUVAEVABRUARUARUQBFQBBQBFVAEFAEVUAQUAUVAEVABRUARUAFFQBFQBFRAEVAEFIkRUAQUAUVABRQBRUAFFAFFQBFQAUVAEVAEVEARUAQUARVQBBQBRUAFFAFFQAUUAUVAEVABRUARUARUQBFQBBQBFVAEFAEVUAQUAUVAEVABRUARUAFFQBFQBFRAEVAEFAEVUAQUAUVABRQBRUARUAFFQBFQAUVAEVAEVEARUAQUARVQBBQBRUAFFAFFQAUUAUVAEVAEVEARUARUQBFQBBQBFVAEFAFFQAUUAQUQUAABNQIAAQUQUAABBRBQAAQUQEABBBRAQAEEFAABBRBQAAEFEFAAAQVAQAEEFEBAAQQUAAEFEFAAAQUQUAABBUBAAQQUQEABBBRAQAEQUAABBRBQAAEFQEABBBRAQAEEFEBAARBQAAEFEFAAAQUQUAAEFEBAAQQUQEABEFAAAQUQUAABBRBQAAQUQEABBBRAQAEEFAABBRBQAAEFEFAABBRAQAEEFEBAAQQUAAEFEFAAAQUQUAABBUBAAQQUQEABBBRAQAEQUAABBRBQAAEFQEABBBRAQAEEFEBAARBQAAEFEFAAAQUQUAAEFEBAAQQUQEABEFAAAQUQUAABBRBQAAQUQEABBBRAQAEEFAABBRBQAAEFEFAABBRAQAEEFEBAAQQUAAEFEFAAAQUQUAABBUBAAQQUQEABBBQAAQUQUAABBRBQAAEFQEABBBRAQAEEFEBAARBQAAEFEFAAAQUQUAAEFEBAAQQUQEABEFCAEw0OhnPFl4DJIAAAAABJRU5ErkJggg=="}
|
rknn/start.cmd
ADDED
|
@@ -0,0 +1,49 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
@echo off
|
| 2 |
+
rem =====================================================================
|
| 3 |
+
rem MiniCPM-V 4.7 1B VLM (text tower + vision tower) for RK1828.
|
| 4 |
+
rem Converted 2026-11-03/04 from openbmb/MiniCPM-V-4.7-1B via the
|
| 5 |
+
rem official rknn3-model-zoo Qwen3_5 pipeline (w4a16 + normal + group32).
|
| 6 |
+
rem
|
| 7 |
+
rem Text tower : Qwen3.5-0.8B architecture, vocab 248094,
|
| 8 |
+
rem tie_word_embeddings=false (independent w6a16 lm_head).
|
| 9 |
+
rem Vision : fixed 448x448 single image, output [1,196,1024].
|
| 10 |
+
rem The 196 tokens are a 8x8 -> 14x14 nearest-neighbor
|
| 11 |
+
rem resample ADAPTER (runtime demands 14x14); NOT native 4.7.
|
| 12 |
+
rem Image input: POST /v1/chat/completions with content[] image_url +
|
| 13 |
+
rem base64 data URI. Do NOT put placeholder tokens in text.
|
| 14 |
+
rem Limits : no canvas mRoPE / multi-resolution tiling verified;
|
| 15 |
+
rem complex-image accuracy not calibrated. See docs 21 / 9.9.
|
| 16 |
+
rem
|
| 17 |
+
rem IMPORTANT: keep this file pure ASCII + CRLF.
|
| 18 |
+
rem =====================================================================
|
| 19 |
+
setlocal
|
| 20 |
+
if "%RK_HOST%"=="" set RK_HOST=0.0.0.0
|
| 21 |
+
if "%RK_PORT%"=="" set RK_PORT=8899
|
| 22 |
+
|
| 23 |
+
rem Locate the RKNN3 runtime (repo layout: <repo>\rknn\ ; package layout: models\<name>\)
|
| 24 |
+
set "RT=%~dp0..\runtime\rknn3_110"
|
| 25 |
+
if not exist "%RT%\bin" set "RT=%~dp0..\..\runtime\rknn3_110"
|
| 26 |
+
if not exist "%RT%\bin" if not "%RK_RUNTIME%"=="" set "RT=%RK_RUNTIME%"
|
| 27 |
+
if not exist "%RT%\bin" goto no_runtime
|
| 28 |
+
|
| 29 |
+
set "PATH=%RT%\bin;%RT%\lib;%PATH%"
|
| 30 |
+
cd /d "%~dp0"
|
| 31 |
+
goto run
|
| 32 |
+
|
| 33 |
+
:no_runtime
|
| 34 |
+
echo [start.cmd] RKNN3 runtime not found with rkllm3-server.exe.
|
| 35 |
+
echo [start.cmd] looked in %~dp0..\runtime\rknn3_110 and ..\..\runtime\rknn3_110
|
| 36 |
+
endlocal
|
| 37 |
+
exit /b 1
|
| 38 |
+
|
| 39 |
+
:run
|
| 40 |
+
rkllm3-server -m "MiniCPM-V-4.7-1B-llm.rknn" --weight "MiniCPM-V-4.7-1B-llm.weight" ^
|
| 41 |
+
--vocab "MiniCPM-V-4.7-1B-llm.tokenizer.gguf" --embed "MiniCPM-V-4.7-1B-llm.embed.bin" ^
|
| 42 |
+
--model2 "MiniCPM-V-4.7-1B-vision.rknn" --weight2 "MiniCPM-V-4.7-1B-vision.weight" ^
|
| 43 |
+
--img-start "<image>" --img-end "</image>" --img-content "<|image_pad|>" ^
|
| 44 |
+
--img-width 448 --img-height 448 ^
|
| 45 |
+
--host "%RK_HOST%" --port "%RK_PORT%" ^
|
| 46 |
+
-c 0 --n_predict 512 --top-k 20 --top-p 0.95 --temp 0.7 --repeat-penalty 1.03
|
| 47 |
+
|
| 48 |
+
endlocal
|
| 49 |
+
exit /b 0
|