Image-Text-to-Text
GGUF
English
Chinese
rknn
rknn3
rk1828
rk1820
rockchip
npu
w4a16
vision-language-model
multimodal
on-device
edge-ai
conversational
Instructions to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 # Run inference directly in the terminal: llama cli -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 # Run inference directly in the terminal: llama cli -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 # Run inference directly in the terminal: ./build/bin/llama cli -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Use Docker
docker model run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Llama new
- LM Studio
- Jan
- vLLM
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Ollama
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Ollama:
ollama run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Unsloth Desktop
- Pi
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Docker Model Runner:
docker model run hf.co/jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
- Lemonade
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Run and chat with the model
lemonade run user.MiniCPM-V-4.7-1B-RKNN3-RK1828-{{QUANT_TAG}}List all available models
lemonade list
- Hermes Agent
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828 with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "jj90/MiniCPM-V-4.7-1B-RKNN3-RK1828" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
File size: 5,446 Bytes
fd11f30 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 | """MiniCPM-V 4.7 1B 视觉塔 -> ONNX(固定分辨率、单图、16x 压缩)。
要点:
- 4.7 的建模类在 transformers>=5.18,本地 zoo venv 是 5.3.0,
因此把 5.18.0 单独装到 minicpm47/tf518,用 sys.path 前置覆盖,不污染原环境。
- vision_tower.forward 内部有 ViT 内压缩(insert_layer_id=6 的 vit_merger,
2x2 window merge),merger 再 2x2 -> 合计 4x4 patch = 1 个 LLM token(16x)。
- 固定 448x448:grid=32 -> vit_merger 后 16x16 -> merger 后 8x8 = 64 个 token,
每个 1024 维(= LLM hidden_size)。32 能被 4 整除,980/14=70 不行,别用 980 单图。
- RKNN 侧用 mean/std=0.5*255 + uint8 NHWC 输入,ONNX 只接受已归一化的 fp32 NCHW。
- 输出必须是 3D [1, N, hidden],不能是 2D [N, hidden]!依据官方
examples/MiniCPM_V_4/cpp/main.cc:
tensor.image.n_image = embeds_shape[0]; // 图片数
tensor.image.n_image_tokens = embeds_shape[1]; // 每图 token 数
rkllm3-server 用同一套 runtime 约定。2D 输出会被读成"64 张图、每张 1024 token",
导致 image token 数与 vision embedding 完全错位。官方 4.0 的
export_minicpm3o_vision likewise 返回 B*Q*D 三维。
- **runtime 契约适配(11-4 上板实测)**:multimodal embed callback 还会校验
`image_ph * image_pw == n_image_tokens`,其中
`image_ph = model_height / patch_size(16) / spatial_merge_size(2)`。
448 输入 -> 14x14 = **196**,而 4.7 视觉塔原生只有 8x8 = **64**,直接上板报
`image_ph * image_pw != n_image_tokens! image_ph = 14 ... n_image_tokens = 64`。
`image_height` 取自 vision RKNN 模型的输入尺寸:改 `--img-width/--img-height`
到 256、把请求图片换成 256x256,报错里都仍是 448(均已实测)。
故在导出图内把 8x8 网格最近邻重采样到 14x14 = 196 token 让契约对齐。
这是**适配层,不是 4.7 原生行为**;视觉信息有冗余,但空间排布保留。
"""
import argparse
import os
import sys
import torch
TF518 = os.environ.get("MINICPM_TF518", "")
if TF518:
sys.path.insert(0, TF518)
MODEL = os.environ.get("MINICPM_HF_DIR", "")
class VisionExport(torch.nn.Module):
def __init__(self, model, img_size):
super().__init__()
self.vt = model.model.vision_tower
self.merger = model.model.merger
patch = self.vt.config.patch_size
grid = img_size // patch
assert grid % 4 == 0, f"patch grid {grid} 必须能被 4 整除(16x 压缩),换分辨率"
self.register_buffer(
"target_sizes", torch.tensor([[grid, grid]], dtype=torch.int32), persistent=False
)
# merger 内部用 height/width 作 view() 的形状参数;传 Python int 才能被 ONNX trace。
# vision_tower 在 insert_layer 处把 target_sizes 减半,故这里也减半。
self.merger_sizes = [[grid // 2, grid // 2]]
# ---- runtime 契约适配(见文件头 docstring)----
# 视觉塔原生 (grid/4)^2 = 8x8 = 64 token,runtime 要求 14x14 = 196。
# 用纯 Gather/Reshape 做最近邻重采样,不引入 Resize op,RKNN 转换最稳。
self.src_grid = grid // 4
self.out_grid = 14
self.register_buffer(
"resample_idx",
(torch.arange(self.out_grid, dtype=torch.float32) * self.src_grid / self.out_grid).long(),
persistent=False,
)
def forward(self, pixel):
out = self.vt(pixel, target_sizes=self.target_sizes, use_vit_merger=True)
hs = out.last_hidden_state if hasattr(out, "last_hidden_state") else out[0]
feats = self.merger(hs, self.merger_sizes)
y = torch.cat(feats, dim=0) # [src^2, hidden],行优先 8x8 网格
g, idx = self.src_grid, self.resample_idx
y = y.view(g, g, -1)[idx[:, None], idx[None, :], :] # [14, 14, hidden]
# [N, hidden] -> [1, N, hidden]:server 按 dim0=图片数 / dim1=每图token数解析。
return y.reshape(1, self.out_grid * self.out_grid, -1)
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--img_size", type=int, default=448)
ap.add_argument(
"--out",
default=os.environ.get("MINICPM_VISION_ONNX", "MiniCPM-V-4.7-1B-vision.onnx"),
)
ap.add_argument("--model", default=MODEL,
help="Local HF checkout of openbmb/MiniCPM-V-4.7-1B")
ap.add_argument("--smoke", action="store_true", help="只跑一次 eager forward,不导出")
args = ap.parse_args()
from transformers import AutoModelForImageTextToText
model = AutoModelForImageTextToText.from_pretrained(
args.model, dtype=torch.float32, trust_remote_code=True, low_cpu_mem_usage=True
)
model.eval()
wrapper = VisionExport(model, args.img_size).eval()
pixel = torch.randn(1, 3, args.img_size, args.img_size)
with torch.no_grad():
y = wrapper(pixel)
print("eager output shape:", tuple(y.shape), "dtype:", y.dtype, flush=True)
if args.smoke:
return
os.makedirs(os.path.dirname(args.out), exist_ok=True)
torch.onnx.export(
wrapper,
(pixel,),
args.out,
input_names=["pixel"],
output_names=["img_embeds"],
opset_version=17,
do_constant_folding=True,
)
print("exported:", args.out, os.path.getsize(args.out), "bytes", flush=True)
if __name__ == "__main__":
main()
|