File size: 6,346 Bytes
fd11f30
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
# Deployment guide — MiniCPM-V 4.7 1B on RK1828

## 0. What you need

| Item | Requirement |
| --- | --- |
| NPU module | Rockchip **RK1828 / RK1820** (tested on `RM1828MC0-F`, firmware 1.1.0) |
| Host | Windows PC, or RK3588 / RK3576 board. **Ubuntu x86 cannot drive the module directly** (conversion / simulator only) |
| Runtime | `rknn3_110` — `rkllm3-server.exe`, matching `toolkit / proxy / runtime / firmware` versions |
| Session | Windows **Session 1** (interactive logon or an `InteractiveToken` scheduled task) |

Component versions **must match**. A mismatched `rknn3_110` typically fails with
a DLL-load error that prints **nothing at all**.

## 1. Layout

Put the runtime next to `rknn/`:

```text
<root>\
  rknn\        <- this repo's rknn/ folder (6 artifacts + start.cmd)
  runtime\rknn3_110\bin\rkllm3-server.exe
  runtime\rknn3_110\lib\*.dll
```

`rknn/start.cmd` looks for the runtime in `..\runtime\rknn3_110`, then
`..\..\runtime\rknn3_110`, then the `RK_RUNTIME` environment variable.
If none is found it **exits with an error** rather than silently falling back
to a hard-coded drive.

Override the listen address / port with `RK_HOST` / `RK_PORT`
(defaults `0.0.0.0` / `8899`).

```cmd
cd /d <root>\rknn
start.cmd
```

Wait until the server is actually ready. The log can sit at
`Weight sync chunk size resolved` for a couple of minutes — **do not treat that
as a hang**, and do not run `rknn-smi` while weights are loading (it can wedge
the sync). Readiness is proven only by a real inference request.

## 2. Start the communication proxy

One `rknn3_transfer_proxy` process per machine, **reuse it**. Repeatedly
tearing it down and recreating it can drop the module off the USB bus.
If you use the RK1828 web console, let the console own the proxy.

## 3. Send an image request

The server exposes an OpenAI-compatible endpoint. `messages[].content` must be
an **array**, and the image goes in as a base64 data URI:

```json
{
  "messages": [{
    "role": "user",
    "content": [
      { "type": "text", "text": "图里是什么?" },
      { "type": "image_url", "image_url": { "url": "data:image/png;base64,<BASE64>" } }
    ]
  }],
  "max_tokens": 512
}
```

**Do not put `<image>` / `<|image_pad|>` placeholder tokens into the text.**
The server constructs them itself. Ready-made bodies are in `host/`:

```cmd
cd /d <root>\host
host\make_test_images.py            :: generates test_images.json (pure stdlib)
send_vlm_req.cmd orange             :: -> 127.0.0.1:8899
send_vlm_req.cmd text 8888          :: override port
```

Use **`curl.exe --max-time`**. Windows PowerShell `Invoke-WebRequest` /
`Invoke-RestMethod` hang indefinitely against this server (no response, no
`-OutFile`), and `-TimeoutSec` does not always rescue it.

Expect roughly `HTTP=200 TIME=0.6s` for a simple image. A correct response
contains `timings.prompt_n ≈ 218` — that is 196 visual tokens plus text tokens,
and is the evidence that vision embeddings really entered the LLM.

## 4. Verified reference results

| Check | Result |
| --- | --- |
| Orange image, "what colour" | `这张图是橙色。` |
| Black square image | `黑色正方形` |
| Cross-checks (orange asked for shape, square asked for colour) | correct, no language-prior guessing |
| Pure text "中国的首都是哪里" | `北京` |
| `finish_reason` | `stop` on simple cases |
| NPU, VLM alone | **1269 MB** (exclusive differential; console shows 1303 incl. 34 MB baseline) |
| NPU, VLM + Qwen3-ASR-0.6B | 3458 MB |
| NPU, VLM + Qwen3-ASR-1.7B | 3353 MB (A/B/A reproducible) |
| Generation | ~90 tok/s, prefill ~1700 tok/s |

Five request/response pairs plus the full console acceptance run are documented
in the upstream project docs; the ready-made request bodies here reproduce the
two simplest ones.

## 5. Pitfalls (all hit in practice)

1. **Session 0 breaks the USB bulk channel.** Starting the server from an SSH
   session lands it in Session 0; `rknn3_init` succeeds but weight sync never
   completes. Use `schtasks` with `InteractiveToken` (Session 1). Diagnose with
   a *different, known-good* model — if both fail identically, suspect the
   session, not the USB port or the firmware.
2. **`tie_word_embeddings` at the top level.** A converted graph without a real
   `lm_head` makes `load_llm` **segfault silently**. See `convert/README.md`.
3. **Vision output shape and token count.** 3D `[1,N,1024]` and
   `N == image_ph * image_pw == 196`. See `convert/README.md`.
4. **Placeholder tokens must match this model's own vocabulary.** MiniCPM-V 4.7
   uses bare `<image>` / `</image>` / `<|image_pad|>`. Its GGUF vocabulary is a
   **superset** that also contains Qwen-VL's `<|vision_start|>` /
   `<|vision_end|>`; a detector that checks Qwen-VL first will silently rewrite
   a correct `start.cmd` into a broken one.
5. **`-c` is context length, not core mask.** Use `--core-mask` /
   `--core-mask2`; 8 cores = `0xFF`. A zero core mask makes
   `rknn3_model_init` return `-2`.
6. **`.cmd` / `.bat` / `.ps1` must be pure ASCII + CRLF**, and `PATH` must be
   set before launching the child process (the launch script does this).
7. **Check `finish_reason`.** This is a 1B model with a `<think>` chain. Complex
   images can fill `max_tokens` and return `finish_reason=length` with the
   answer truncated. Increasing `max_tokens` is **not** monotonically safer
   (256 worked for one OCR case where 512 stalled inside `<think>`).
8. **Residual processes hold the port** → "I changed the code but behaviour did
   not change". Check `Get-NetTCPConnection` before restarting.
9. **ASR device errors can leak NPU memory** (`mem_sync ret=-10` retry loop).
   Workaround: `taskkill /F /IM rknn_qwen3_asr_demo.exe`. Not fixed upstream.
10. **Power loss may land the module in MaskROM** (`2207:180B`); `upgrade_tool uf`
    recovers it in ~2 s. A genuine hard hang needs a physical power cycle.

## 6. Known limitations — read before filing issues

- The 196-token resample is a **runtime-contract adapter**. This build is
  **not equivalent to the Hugging Face original MiniCPM-V 4.7 1B**.
- canvas mRoPE, multi-resolution tiling, high-resolution OCR and dense
  multi-object counting were **not verified**.
- OCR is "characters readable" at best: `RK1828` may come back as `RK₁₈₂₈`.
- Only 448x448 single-image input was tested.