Image-Text-to-Text
GGUF
English
Chinese
rknn
rknn3
rk1828
rk1820
rockchip
npu
w4a16
vision-language-model
multimodal
on-device
edge-ai
conversational
Instructions to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 # Run inference directly in the terminal: llama cli -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 # Run inference directly in the terminal: llama cli -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 # Run inference directly in the terminal: ./build/bin/llama cli -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Use Docker
docker model run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Llama new
- LM Studio
- Jan
- vLLM
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Ollama
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Ollama:
ollama run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Unsloth Desktop
- Pi
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Docker Model Runner:
docker model run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Lemonade
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Run and chat with the model
lemonade run user.MiniCPM-V-4.7-1B-RKNN3-RK1828-{{QUANT_TAG}}List all available models
lemonade list
- Hermes Agent
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Download convert/export_vision47.py from jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828: direct link, hf CLI and curl.
- Browser
- Download file 5.45 kB
-
https://huggingface.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828/resolve/main/convert/export_vision47.py
- Command line
-
hf download hf://jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828/convert/export_vision47.py
-
curl -L -o export_vision47.py https://huggingface.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828/resolve/main/convert/export_vision47.py
5.45 kB
| """MiniCPM-V 4.7 1B 视觉塔 -> ONNX(固定分辨率、单图、16x 压缩)。 | |
| 要点: | |
| - 4.7 的建模类在 transformers>=5.18,本地 zoo venv 是 5.3.0, | |
| 因此把 5.18.0 单独装到 minicpm47/tf518,用 sys.path 前置覆盖,不污染原环境。 | |
| - vision_tower.forward 内部有 ViT 内压缩(insert_layer_id=6 的 vit_merger, | |
| 2x2 window merge),merger 再 2x2 -> 合计 4x4 patch = 1 个 LLM token(16x)。 | |
| - 固定 448x448:grid=32 -> vit_merger 后 16x16 -> merger 后 8x8 = 64 个 token, | |
| 每个 1024 维(= LLM hidden_size)。32 能被 4 整除,980/14=70 不行,别用 980 单图。 | |
| - RKNN 侧用 mean/std=0.5*255 + uint8 NHWC 输入,ONNX 只接受已归一化的 fp32 NCHW。 | |
| - 输出必须是 3D [1, N, hidden],不能是 2D [N, hidden]!依据官方 | |
| examples/MiniCPM_V_4/cpp/main.cc: | |
| tensor.image.n_image = embeds_shape[0]; // 图片数 | |
| tensor.image.n_image_tokens = embeds_shape[1]; // 每图 token 数 | |
| rkllm3-server 用同一套 runtime 约定。2D 输出会被读成"64 张图、每张 1024 token", | |
| 导致 image token 数与 vision embedding 完全错位。官方 4.0 的 | |
| export_minicpm3o_vision likewise 返回 B*Q*D 三维。 | |
| - **runtime 契约适配(11-4 上板实测)**:multimodal embed callback 还会校验 | |
| `image_ph * image_pw == n_image_tokens`,其中 | |
| `image_ph = model_height / patch_size(16) / spatial_merge_size(2)`。 | |
| 448 输入 -> 14x14 = **196**,而 4.7 视觉塔原生只有 8x8 = **64**,直接上板报 | |
| `image_ph * image_pw != n_image_tokens! image_ph = 14 ... n_image_tokens = 64`。 | |
| `image_height` 取自 vision RKNN 模型的输入尺寸:改 `--img-width/--img-height` | |
| 到 256、把请求图片换成 256x256,报错里都仍是 448(均已实测)。 | |
| 故在导出图内把 8x8 网格最近邻重采样到 14x14 = 196 token 让契约对齐。 | |
| 这是**适配层,不是 4.7 原生行为**;视觉信息有冗余,但空间排布保留。 | |
| """ | |
| import argparse | |
| import os | |
| import sys | |
| import torch | |
| TF518 = os.environ.get("MINICPM_TF518", "") | |
| if TF518: | |
| sys.path.insert(0, TF518) | |
| MODEL = os.environ.get("MINICPM_HF_DIR", "") | |
| class VisionExport(torch.nn.Module): | |
| def __init__(self, model, img_size): | |
| super().__init__() | |
| self.vt = model.model.vision_tower | |
| self.merger = model.model.merger | |
| patch = self.vt.config.patch_size | |
| grid = img_size // patch | |
| assert grid % 4 == 0, f"patch grid {grid} 必须能被 4 整除(16x 压缩),换分辨率" | |
| self.register_buffer( | |
| "target_sizes", torch.tensor([[grid, grid]], dtype=torch.int32), persistent=False | |
| ) | |
| # merger 内部用 height/width 作 view() 的形状参数;传 Python int 才能被 ONNX trace。 | |
| # vision_tower 在 insert_layer 处把 target_sizes 减半,故这里也减半。 | |
| self.merger_sizes = [[grid // 2, grid // 2]] | |
| # ---- runtime 契约适配(见文件头 docstring)---- | |
| # 视觉塔原生 (grid/4)^2 = 8x8 = 64 token,runtime 要求 14x14 = 196。 | |
| # 用纯 Gather/Reshape 做最近邻重采样,不引入 Resize op,RKNN 转换最稳。 | |
| self.src_grid = grid // 4 | |
| self.out_grid = 14 | |
| self.register_buffer( | |
| "resample_idx", | |
| (torch.arange(self.out_grid, dtype=torch.float32) * self.src_grid / self.out_grid).long(), | |
| persistent=False, | |
| ) | |
| def forward(self, pixel): | |
| out = self.vt(pixel, target_sizes=self.target_sizes, use_vit_merger=True) | |
| hs = out.last_hidden_state if hasattr(out, "last_hidden_state") else out[0] | |
| feats = self.merger(hs, self.merger_sizes) | |
| y = torch.cat(feats, dim=0) # [src^2, hidden],行优先 8x8 网格 | |
| g, idx = self.src_grid, self.resample_idx | |
| y = y.view(g, g, -1)[idx[:, None], idx[None, :], :] # [14, 14, hidden] | |
| # [N, hidden] -> [1, N, hidden]:server 按 dim0=图片数 / dim1=每图token数解析。 | |
| return y.reshape(1, self.out_grid * self.out_grid, -1) | |
| def main(): | |
| ap = argparse.ArgumentParser() | |
| ap.add_argument("--img_size", type=int, default=448) | |
| ap.add_argument( | |
| "--out", | |
| default=os.environ.get("MINICPM_VISION_ONNX", "MiniCPM-V-4.7-1B-vision.onnx"), | |
| ) | |
| ap.add_argument("--model", default=MODEL, | |
| help="Local HF checkout of openbmb/MiniCPM-V-4.7-1B") | |
| ap.add_argument("--smoke", action="store_true", help="只跑一次 eager forward,不导出") | |
| args = ap.parse_args() | |
| from transformers import AutoModelForImageTextToText | |
| model = AutoModelForImageTextToText.from_pretrained( | |
| args.model, dtype=torch.float32, trust_remote_code=True, low_cpu_mem_usage=True | |
| ) | |
| model.eval() | |
| wrapper = VisionExport(model, args.img_size).eval() | |
| pixel = torch.randn(1, 3, args.img_size, args.img_size) | |
| with torch.no_grad(): | |
| y = wrapper(pixel) | |
| print("eager output shape:", tuple(y.shape), "dtype:", y.dtype, flush=True) | |
| if args.smoke: | |
| return | |
| os.makedirs(os.path.dirname(args.out), exist_ok=True) | |
| torch.onnx.export( | |
| wrapper, | |
| (pixel,), | |
| args.out, | |
| input_names=["pixel"], | |
| output_names=["img_embeds"], | |
| opset_version=17, | |
| do_constant_folding=True, | |
| ) | |
| print("exported:", args.out, os.path.getsize(args.out), "bytes", flush=True) | |
| if __name__ == "__main__": | |
| main() | |