Image-Text-to-Text
GGUF
English
Chinese
rknn
rknn3
rk1828
rk1820
rockchip
npu
w4a16
vision-language-model
multimodal
on-device
edge-ai
conversational
Instructions to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 # Run inference directly in the terminal: llama cli -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 # Run inference directly in the terminal: llama cli -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 # Run inference directly in the terminal: ./build/bin/llama cli -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Use Docker
docker model run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Llama new
- LM Studio
- Jan
- vLLM
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Ollama
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Ollama:
ollama run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Unsloth Desktop
- Pi
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Docker Model Runner:
docker model run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Lemonade
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Run and chat with the model
lemonade run user.MiniCPM-V-4.7-1B-RKNN3-RK1828-{{QUANT_TAG}}List all available models
lemonade list
- Hermes Agent
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
|
Download convert/README.md from jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828: direct link, hf CLI and curl.
- Browser
- Download file 4.01 kB
-
https://huggingface.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828/resolve/main/convert/README.md
- Command line
-
hf download hf://jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828/convert/README.md
-
curl -L -o README.md https://huggingface.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828/resolve/main/convert/README.md
4.01 kB
| # Reproducing the conversion | |
| These are **portable copies** of the two scripts that produced the vision tower | |
| in `../rknn/`. All absolute paths have been replaced by arguments / environment | |
| variables; the model math is unchanged. | |
| ## Prerequisites | |
| - `rknn3-toolkit` (the whl from the RKNN3 SDK, installed with `--no-deps`) | |
| - `transformers >= 5.18` (`MiniCPMV4_7ForConditionalGeneration` does not exist | |
| in 5.3). Point `MINICPM_TF518` at a separate checkout if your main env is older. | |
| - `torch`, `onnx`, `safetensors` | |
| - A local checkout of [`openbmb/MiniCPM-V-4.7-1B`](https://huggingface.co/openbmb/MiniCPM-V-4.7-1B) | |
| - Official [`airockchip/rknn3-model-zoo`](https://github.com/airockchip/rknn3-model-zoo) V1.1.0 | |
| for the LLM-side `export_llm.py` (the text tower is the same `Qwen3_5ForCausalLM` | |
| family as the zoo's `Qwen3_5` example). | |
| ## 1. Text tower (four artifacts) | |
| The 4.7 text tower is architecturally Qwen3.5-0.8B with a different vocabulary: | |
| | Item | 0.8B | MiniCPM-V 4.7 1B | | |
| | --- | --- | --- | | |
| | `vocab_size` | 248320 | **248094** | | |
| | `tie_word_embeddings` | true | **false** (independent `lm_head.weight` [248094,1024]) | | |
| | `conv_state` | [18,6144,4] | [18,6144,4] (identical) | | |
| Build a plain-text HF directory by keeping only `model.language_model.*` tensors | |
| plus `lm_head.weight`, copying the 4.7 tokenizer files, and starting from the | |
| zoo's Qwen3.5 `config.json` with the two changes above. Then run the zoo's | |
| `Qwen3_5/python/export_llm.py` followed by `export_rknn.py` with | |
| `w4a16 / normal / group32 / core_num=8`. | |
| > **Top-level `tie_word_embeddings` trap.** Hugging Face `tie_weights()` reads the | |
| > **top-level** config key, not `text_config`. Changing only `text_config` leaves | |
| > the independent 254M-parameter head silently bound to `embed_tokens`, and the | |
| > converted model emits word salad. Change **both**. The only reliable check is | |
| > `cmp`-ing the produced `embed.bin` against the head weights — file sizes can be | |
| > byte-identical between a good and a bad build. | |
| ## 2. Vision tower (two artifacts) | |
| ```bash | |
| export MINICPM_HF_DIR=/path/to/MiniCPM-V-4.7-1B | |
| export MINICPM_TF518=/path/to/transformers-5.18 # optional | |
| export MINICPM_VISION_ONNX=MiniCPM-V-4.7-1B-vision.onnx | |
| export MINICPM_VISION_RKNN=MiniCPM-V-4.7-1B-vision.rknn | |
| python export_vision47.py --model "$MINICPM_HF_DIR" --out "$MINICPM_VISION_ONNX" | |
| python export_vision_rknn.py | |
| ``` | |
| `export_vision_rknn.py` produces `<name>.rknn` and `<name>.weight` side by side. | |
| ### Two non-obvious contracts | |
| 1. **The output must be 3D `[1, N, hidden]`, not 2D `[N, hidden]`.** | |
| The runtime reads `embeds_shape[0]` as *number of images* and | |
| `embeds_shape[1]` as *tokens per image* (see the official | |
| `examples/MiniCPM_V_4/cpp/main.cc`). A 2D export is parsed as | |
| "64 images of 1024 tokens each" and the image tokens never line up. | |
| 2. **`image_ph * image_pw == n_image_tokens`.** The multimodal embed callback | |
| computes `image_ph = model_height / patch_size(16) / spatial_merge_size(2)`; | |
| with a 448x448 vision input that is `14 x 14 = 196`. MiniCPM-V 4.7's native | |
| tower emits `(448/14/4)^2 = 8 x 8 = 64` tokens. A 64-token export fails on | |
| device with: | |
| ```text | |
| image_ph * image_pw != n_image_tokens! image_ph = 14 ... n_image_tokens = 64 | |
| ``` | |
| Changing `--img-width/--img-height` or resizing the request image does **not** | |
| help: `image_height` comes from the vision RKNN model's own input shape. | |
| `export_vision47.py` therefore inserts a **nearest-neighbour 8x8 -> 14x14 | |
| resample** inside the exported graph (pure Gather/Reshape, no `Resize` op): | |
| ```python | |
| g, idx = self.src_grid, self.resample_idx # 8, 14 | |
| y = y.view(g, g, -1)[idx[:, None], idx[None, :], :] | |
| return y.reshape(1, 196, -1) | |
| ``` | |
| This is a **runtime-contract adapter, not native 4.7 behaviour**. It changes | |
| only graph structure — the `.weight` SHA256 is identical to the 64-token | |
| build, since no weights are touched. See the model card for what this means | |
| for equivalence claims. | |