jj90's picture
Add model card, deploy guide, host + conversion scripts
fd11f30 verified
|
Raw History Blame Contribute Delete
6.35 kB

Deployment guide — MiniCPM-V 4.7 1B on RK1828

0. What you need

Item Requirement
NPU module Rockchip RK1828 / RK1820 (tested on RM1828MC0-F, firmware 1.1.0)
Host Windows PC, or RK3588 / RK3576 board. Ubuntu x86 cannot drive the module directly (conversion / simulator only)
Runtime rknn3_110 — rkllm3-server.exe, matching toolkit / proxy / runtime / firmware versions
Session Windows Session 1 (interactive logon or an InteractiveToken scheduled task)

Component versions must match. A mismatched rknn3_110 typically fails with a DLL-load error that prints nothing at all.

1. Layout

Put the runtime next to rknn/:

<root>\
  rknn\        <- this repo's rknn/ folder (6 artifacts + start.cmd)
  runtime\rknn3_110\bin\rkllm3-server.exe
  runtime\rknn3_110\lib\*.dll

rknn/start.cmd looks for the runtime in ..\runtime\rknn3_110, then ..\..\runtime\rknn3_110, then the RK_RUNTIME environment variable. If none is found it exits with an error rather than silently falling back to a hard-coded drive.

Override the listen address / port with RK_HOST / RK_PORT (defaults 0.0.0.0 / 8899).

cd /d <root>\rknn
start.cmd

Wait until the server is actually ready. The log can sit at Weight sync chunk size resolved for a couple of minutes — do not treat that as a hang, and do not run rknn-smi while weights are loading (it can wedge the sync). Readiness is proven only by a real inference request.

2. Start the communication proxy

One rknn3_transfer_proxy process per machine, reuse it. Repeatedly tearing it down and recreating it can drop the module off the USB bus. If you use the RK1828 web console, let the console own the proxy.

3. Send an image request

The server exposes an OpenAI-compatible endpoint. messages[].content must be an array, and the image goes in as a base64 data URI:

{
  "messages": [{
    "role": "user",
    "content": [
      { "type": "text", "text": "图里是什么?" },
      { "type": "image_url", "image_url": { "url": "data:image/png;base64,<BASE64>" } }
    ]
  }],
  "max_tokens": 512
}

Do not put <image> / <|image_pad|> placeholder tokens into the text. The server constructs them itself. Ready-made bodies are in host/:

cd /d <root>\host
host\make_test_images.py            :: generates test_images.json (pure stdlib)
send_vlm_req.cmd orange             :: -> 127.0.0.1:8899
send_vlm_req.cmd text 8888          :: override port

Use curl.exe --max-time. Windows PowerShell Invoke-WebRequest / Invoke-RestMethod hang indefinitely against this server (no response, no -OutFile), and -TimeoutSec does not always rescue it.

Expect roughly HTTP=200 TIME=0.6s for a simple image. A correct response contains timings.prompt_n ≈ 218 — that is 196 visual tokens plus text tokens, and is the evidence that vision embeddings really entered the LLM.

4. Verified reference results

Check Result
Orange image, "what colour" 这张图是橙色。
Black square image 黑色正方形
Cross-checks (orange asked for shape, square asked for colour) correct, no language-prior guessing
Pure text "中国的首都是哪里" 北京
finish_reason stop on simple cases
NPU, VLM alone 1269 MB (exclusive differential; console shows 1303 incl. 34 MB baseline)
NPU, VLM + Qwen3-ASR-0.6B 3458 MB
NPU, VLM + Qwen3-ASR-1.7B 3353 MB (A/B/A reproducible)
Generation ~90 tok/s, prefill ~1700 tok/s

Five request/response pairs plus the full console acceptance run are documented in the upstream project docs; the ready-made request bodies here reproduce the two simplest ones.

5. Pitfalls (all hit in practice)

  1. Session 0 breaks the USB bulk channel. Starting the server from an SSH session lands it in Session 0; rknn3_init succeeds but weight sync never completes. Use schtasks with InteractiveToken (Session 1). Diagnose with a different, known-good model — if both fail identically, suspect the session, not the USB port or the firmware.
  2. tie_word_embeddings at the top level. A converted graph without a real lm_head makes load_llm segfault silently. See convert/README.md.
  3. Vision output shape and token count. 3D [1,N,1024] and N == image_ph * image_pw == 196. See convert/README.md.
  4. Placeholder tokens must match this model's own vocabulary. MiniCPM-V 4.7 uses bare <image> / </image> / <|image_pad|>. Its GGUF vocabulary is a superset that also contains Qwen-VL's <|vision_start|> / <|vision_end|>; a detector that checks Qwen-VL first will silently rewrite a correct start.cmd into a broken one.
  5. -c is context length, not core mask. Use --core-mask / --core-mask2; 8 cores = 0xFF. A zero core mask makes rknn3_model_init return -2.
  6. .cmd / .bat / .ps1 must be pure ASCII + CRLF, and PATH must be set before launching the child process (the launch script does this).
  7. Check finish_reason. This is a 1B model with a <think> chain. Complex images can fill max_tokens and return finish_reason=length with the answer truncated. Increasing max_tokens is not monotonically safer (256 worked for one OCR case where 512 stalled inside <think>).
  8. Residual processes hold the port → "I changed the code but behaviour did not change". Check Get-NetTCPConnection before restarting.
  9. ASR device errors can leak NPU memory (mem_sync ret=-10 retry loop). Workaround: taskkill /F /IM rknn_qwen3_asr_demo.exe. Not fixed upstream.
  10. Power loss may land the module in MaskROM (2207:180B); upgrade_tool uf recovers it in ~2 s. A genuine hard hang needs a physical power cycle.

6. Known limitations — read before filing issues

  • The 196-token resample is a runtime-contract adapter. This build is not equivalent to the Hugging Face original MiniCPM-V 4.7 1B.
  • canvas mRoPE, multi-resolution tiling, high-resolution OCR and dense multi-object counting were not verified.
  • OCR is "characters readable" at best: RK1828 may come back as RK₁₈₂₈.
  • Only 448x448 single-image input was tested.