Image-Text-to-Text
GGUF
English
Chinese
rknn
rknn3
rk1828
rk1820
rockchip
npu
w4a16
vision-language-model
multimodal
on-device
edge-ai
conversational
Instructions to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 # Run inference directly in the terminal: llama cli -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 # Run inference directly in the terminal: llama cli -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 # Run inference directly in the terminal: ./build/bin/llama cli -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Use Docker
docker model run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Llama new
- LM Studio
- Jan
- vLLM
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Ollama
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Ollama:
ollama run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Unsloth Desktop
- Pi
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Docker Model Runner:
docker model run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Lemonade
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Run and chat with the model
lemonade run user.MiniCPM-V-4.7-1B-RKNN3-RK1828-{{QUANT_TAG}}List all available models
lemonade list
- Hermes Agent
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
File size: 6,346 Bytes
fd11f30 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 | # Deployment guide — MiniCPM-V 4.7 1B on RK1828
## 0. What you need
| Item | Requirement |
| --- | --- |
| NPU module | Rockchip **RK1828 / RK1820** (tested on `RM1828MC0-F`, firmware 1.1.0) |
| Host | Windows PC, or RK3588 / RK3576 board. **Ubuntu x86 cannot drive the module directly** (conversion / simulator only) |
| Runtime | `rknn3_110` — `rkllm3-server.exe`, matching `toolkit / proxy / runtime / firmware` versions |
| Session | Windows **Session 1** (interactive logon or an `InteractiveToken` scheduled task) |
Component versions **must match**. A mismatched `rknn3_110` typically fails with
a DLL-load error that prints **nothing at all**.
## 1. Layout
Put the runtime next to `rknn/`:
```text
<root>\
rknn\ <- this repo's rknn/ folder (6 artifacts + start.cmd)
runtime\rknn3_110\bin\rkllm3-server.exe
runtime\rknn3_110\lib\*.dll
```
`rknn/start.cmd` looks for the runtime in `..\runtime\rknn3_110`, then
`..\..\runtime\rknn3_110`, then the `RK_RUNTIME` environment variable.
If none is found it **exits with an error** rather than silently falling back
to a hard-coded drive.
Override the listen address / port with `RK_HOST` / `RK_PORT`
(defaults `0.0.0.0` / `8899`).
```cmd
cd /d <root>\rknn
start.cmd
```
Wait until the server is actually ready. The log can sit at
`Weight sync chunk size resolved` for a couple of minutes — **do not treat that
as a hang**, and do not run `rknn-smi` while weights are loading (it can wedge
the sync). Readiness is proven only by a real inference request.
## 2. Start the communication proxy
One `rknn3_transfer_proxy` process per machine, **reuse it**. Repeatedly
tearing it down and recreating it can drop the module off the USB bus.
If you use the RK1828 web console, let the console own the proxy.
## 3. Send an image request
The server exposes an OpenAI-compatible endpoint. `messages[].content` must be
an **array**, and the image goes in as a base64 data URI:
```json
{
"messages": [{
"role": "user",
"content": [
{ "type": "text", "text": "图里是什么?" },
{ "type": "image_url", "image_url": { "url": "data:image/png;base64,<BASE64>" } }
]
}],
"max_tokens": 512
}
```
**Do not put `<image>` / `<|image_pad|>` placeholder tokens into the text.**
The server constructs them itself. Ready-made bodies are in `host/`:
```cmd
cd /d <root>\host
host\make_test_images.py :: generates test_images.json (pure stdlib)
send_vlm_req.cmd orange :: -> 127.0.0.1:8899
send_vlm_req.cmd text 8888 :: override port
```
Use **`curl.exe --max-time`**. Windows PowerShell `Invoke-WebRequest` /
`Invoke-RestMethod` hang indefinitely against this server (no response, no
`-OutFile`), and `-TimeoutSec` does not always rescue it.
Expect roughly `HTTP=200 TIME=0.6s` for a simple image. A correct response
contains `timings.prompt_n ≈ 218` — that is 196 visual tokens plus text tokens,
and is the evidence that vision embeddings really entered the LLM.
## 4. Verified reference results
| Check | Result |
| --- | --- |
| Orange image, "what colour" | `这张图是橙色。` |
| Black square image | `黑色正方形` |
| Cross-checks (orange asked for shape, square asked for colour) | correct, no language-prior guessing |
| Pure text "中国的首都是哪里" | `北京` |
| `finish_reason` | `stop` on simple cases |
| NPU, VLM alone | **1269 MB** (exclusive differential; console shows 1303 incl. 34 MB baseline) |
| NPU, VLM + Qwen3-ASR-0.6B | 3458 MB |
| NPU, VLM + Qwen3-ASR-1.7B | 3353 MB (A/B/A reproducible) |
| Generation | ~90 tok/s, prefill ~1700 tok/s |
Five request/response pairs plus the full console acceptance run are documented
in the upstream project docs; the ready-made request bodies here reproduce the
two simplest ones.
## 5. Pitfalls (all hit in practice)
1. **Session 0 breaks the USB bulk channel.** Starting the server from an SSH
session lands it in Session 0; `rknn3_init` succeeds but weight sync never
completes. Use `schtasks` with `InteractiveToken` (Session 1). Diagnose with
a *different, known-good* model — if both fail identically, suspect the
session, not the USB port or the firmware.
2. **`tie_word_embeddings` at the top level.** A converted graph without a real
`lm_head` makes `load_llm` **segfault silently**. See `convert/README.md`.
3. **Vision output shape and token count.** 3D `[1,N,1024]` and
`N == image_ph * image_pw == 196`. See `convert/README.md`.
4. **Placeholder tokens must match this model's own vocabulary.** MiniCPM-V 4.7
uses bare `<image>` / `</image>` / `<|image_pad|>`. Its GGUF vocabulary is a
**superset** that also contains Qwen-VL's `<|vision_start|>` /
`<|vision_end|>`; a detector that checks Qwen-VL first will silently rewrite
a correct `start.cmd` into a broken one.
5. **`-c` is context length, not core mask.** Use `--core-mask` /
`--core-mask2`; 8 cores = `0xFF`. A zero core mask makes
`rknn3_model_init` return `-2`.
6. **`.cmd` / `.bat` / `.ps1` must be pure ASCII + CRLF**, and `PATH` must be
set before launching the child process (the launch script does this).
7. **Check `finish_reason`.** This is a 1B model with a `<think>` chain. Complex
images can fill `max_tokens` and return `finish_reason=length` with the
answer truncated. Increasing `max_tokens` is **not** monotonically safer
(256 worked for one OCR case where 512 stalled inside `<think>`).
8. **Residual processes hold the port** → "I changed the code but behaviour did
not change". Check `Get-NetTCPConnection` before restarting.
9. **ASR device errors can leak NPU memory** (`mem_sync ret=-10` retry loop).
Workaround: `taskkill /F /IM rknn_qwen3_asr_demo.exe`. Not fixed upstream.
10. **Power loss may land the module in MaskROM** (`2207:180B`); `upgrade_tool uf`
recovers it in ~2 s. A genuine hard hang needs a physical power cycle.
## 6. Known limitations — read before filing issues
- The 196-token resample is a **runtime-contract adapter**. This build is
**not equivalent to the Hugging Face original MiniCPM-V 4.7 1B**.
- canvas mRoPE, multi-resolution tiling, high-resolution OCR and dense
multi-object counting were **not verified**.
- OCR is "characters readable" at best: `RK1828` may come back as `RK₁₈₂₈`.
- Only 448x448 single-image input was tested.
|