jj90's picture
Add model card, deploy guide, host + conversion scripts
fd11f30 verified
|
Raw History Blame Contribute Delete
6.35 kB
# Deployment guide — MiniCPM-V 4.7 1B on RK1828
## 0. What you need
| Item | Requirement |
| --- | --- |
| NPU module | Rockchip **RK1828 / RK1820** (tested on `RM1828MC0-F`, firmware 1.1.0) |
| Host | Windows PC, or RK3588 / RK3576 board. **Ubuntu x86 cannot drive the module directly** (conversion / simulator only) |
| Runtime | `rknn3_110` — `rkllm3-server.exe`, matching `toolkit / proxy / runtime / firmware` versions |
| Session | Windows **Session 1** (interactive logon or an `InteractiveToken` scheduled task) |
Component versions **must match**. A mismatched `rknn3_110` typically fails with
a DLL-load error that prints **nothing at all**.
## 1. Layout
Put the runtime next to `rknn/`:
```text
<root>\
rknn\ <- this repo's rknn/ folder (6 artifacts + start.cmd)
runtime\rknn3_110\bin\rkllm3-server.exe
runtime\rknn3_110\lib\*.dll
```
`rknn/start.cmd` looks for the runtime in `..\runtime\rknn3_110`, then
`..\..\runtime\rknn3_110`, then the `RK_RUNTIME` environment variable.
If none is found it **exits with an error** rather than silently falling back
to a hard-coded drive.
Override the listen address / port with `RK_HOST` / `RK_PORT`
(defaults `0.0.0.0` / `8899`).
```cmd
cd /d <root>\rknn
start.cmd
```
Wait until the server is actually ready. The log can sit at
`Weight sync chunk size resolved` for a couple of minutes — **do not treat that
as a hang**, and do not run `rknn-smi` while weights are loading (it can wedge
the sync). Readiness is proven only by a real inference request.
## 2. Start the communication proxy
One `rknn3_transfer_proxy` process per machine, **reuse it**. Repeatedly
tearing it down and recreating it can drop the module off the USB bus.
If you use the RK1828 web console, let the console own the proxy.
## 3. Send an image request
The server exposes an OpenAI-compatible endpoint. `messages[].content` must be
an **array**, and the image goes in as a base64 data URI:
```json
{
"messages": [{
"role": "user",
"content": [
{ "type": "text", "text": "图里是什么?" },
{ "type": "image_url", "image_url": { "url": "data:image/png;base64,<BASE64>" } }
]
}],
"max_tokens": 512
}
```
**Do not put `<image>` / `<|image_pad|>` placeholder tokens into the text.**
The server constructs them itself. Ready-made bodies are in `host/`:
```cmd
cd /d <root>\host
host\make_test_images.py :: generates test_images.json (pure stdlib)
send_vlm_req.cmd orange :: -> 127.0.0.1:8899
send_vlm_req.cmd text 8888 :: override port
```
Use **`curl.exe --max-time`**. Windows PowerShell `Invoke-WebRequest` /
`Invoke-RestMethod` hang indefinitely against this server (no response, no
`-OutFile`), and `-TimeoutSec` does not always rescue it.
Expect roughly `HTTP=200 TIME=0.6s` for a simple image. A correct response
contains `timings.prompt_n ≈ 218` — that is 196 visual tokens plus text tokens,
and is the evidence that vision embeddings really entered the LLM.
## 4. Verified reference results
| Check | Result |
| --- | --- |
| Orange image, "what colour" | `这张图是橙色。` |
| Black square image | `黑色正方形` |
| Cross-checks (orange asked for shape, square asked for colour) | correct, no language-prior guessing |
| Pure text "中国的首都是哪里" | `北京` |
| `finish_reason` | `stop` on simple cases |
| NPU, VLM alone | **1269 MB** (exclusive differential; console shows 1303 incl. 34 MB baseline) |
| NPU, VLM + Qwen3-ASR-0.6B | 3458 MB |
| NPU, VLM + Qwen3-ASR-1.7B | 3353 MB (A/B/A reproducible) |
| Generation | ~90 tok/s, prefill ~1700 tok/s |
Five request/response pairs plus the full console acceptance run are documented
in the upstream project docs; the ready-made request bodies here reproduce the
two simplest ones.
## 5. Pitfalls (all hit in practice)
1. **Session 0 breaks the USB bulk channel.** Starting the server from an SSH
session lands it in Session 0; `rknn3_init` succeeds but weight sync never
completes. Use `schtasks` with `InteractiveToken` (Session 1). Diagnose with
a *different, known-good* model — if both fail identically, suspect the
session, not the USB port or the firmware.
2. **`tie_word_embeddings` at the top level.** A converted graph without a real
`lm_head` makes `load_llm` **segfault silently**. See `convert/README.md`.
3. **Vision output shape and token count.** 3D `[1,N,1024]` and
`N == image_ph * image_pw == 196`. See `convert/README.md`.
4. **Placeholder tokens must match this model's own vocabulary.** MiniCPM-V 4.7
uses bare `<image>` / `</image>` / `<|image_pad|>`. Its GGUF vocabulary is a
**superset** that also contains Qwen-VL's `<|vision_start|>` /
`<|vision_end|>`; a detector that checks Qwen-VL first will silently rewrite
a correct `start.cmd` into a broken one.
5. **`-c` is context length, not core mask.** Use `--core-mask` /
`--core-mask2`; 8 cores = `0xFF`. A zero core mask makes
`rknn3_model_init` return `-2`.
6. **`.cmd` / `.bat` / `.ps1` must be pure ASCII + CRLF**, and `PATH` must be
set before launching the child process (the launch script does this).
7. **Check `finish_reason`.** This is a 1B model with a `<think>` chain. Complex
images can fill `max_tokens` and return `finish_reason=length` with the
answer truncated. Increasing `max_tokens` is **not** monotonically safer
(256 worked for one OCR case where 512 stalled inside `<think>`).
8. **Residual processes hold the port** → "I changed the code but behaviour did
not change". Check `Get-NetTCPConnection` before restarting.
9. **ASR device errors can leak NPU memory** (`mem_sync ret=-10` retry loop).
Workaround: `taskkill /F /IM rknn_qwen3_asr_demo.exe`. Not fixed upstream.
10. **Power loss may land the module in MaskROM** (`2207:180B`); `upgrade_tool uf`
recovers it in ~2 s. A genuine hard hang needs a physical power cycle.
## 6. Known limitations — read before filing issues
- The 196-token resample is a **runtime-contract adapter**. This build is
**not equivalent to the Hugging Face original MiniCPM-V 4.7 1B**.
- canvas mRoPE, multi-resolution tiling, high-resolution OCR and dense
multi-object counting were **not verified**.
- OCR is "characters readable" at best: `RK1828` may come back as `RK₁₈₂₈`.
- Only 448x448 single-image input was tested.