jj90's picture
Add model card, deploy guide, host + conversion scripts
fd11f30 verified
|
Raw History Blame Contribute Delete
4.01 kB

Reproducing the conversion

These are portable copies of the two scripts that produced the vision tower in ../rknn/. All absolute paths have been replaced by arguments / environment variables; the model math is unchanged.

Prerequisites

  • rknn3-toolkit (the whl from the RKNN3 SDK, installed with --no-deps)
  • transformers >= 5.18 (MiniCPMV4_7ForConditionalGeneration does not exist in 5.3). Point MINICPM_TF518 at a separate checkout if your main env is older.
  • torch, onnx, safetensors
  • A local checkout of openbmb/MiniCPM-V-4.7-1B
  • Official airockchip/rknn3-model-zoo V1.1.0 for the LLM-side export_llm.py (the text tower is the same Qwen3_5ForCausalLM family as the zoo's Qwen3_5 example).

1. Text tower (four artifacts)

The 4.7 text tower is architecturally Qwen3.5-0.8B with a different vocabulary:

Item 0.8B MiniCPM-V 4.7 1B
vocab_size 248320 248094
tie_word_embeddings true false (independent lm_head.weight [248094,1024])
conv_state [18,6144,4] [18,6144,4] (identical)

Build a plain-text HF directory by keeping only model.language_model.* tensors plus lm_head.weight, copying the 4.7 tokenizer files, and starting from the zoo's Qwen3.5 config.json with the two changes above. Then run the zoo's Qwen3_5/python/export_llm.py followed by export_rknn.py with w4a16 / normal / group32 / core_num=8.

Top-level tie_word_embeddings trap. Hugging Face tie_weights() reads the top-level config key, not text_config. Changing only text_config leaves the independent 254M-parameter head silently bound to embed_tokens, and the converted model emits word salad. Change both. The only reliable check is cmp-ing the produced embed.bin against the head weights — file sizes can be byte-identical between a good and a bad build.

2. Vision tower (two artifacts)

export MINICPM_HF_DIR=/path/to/MiniCPM-V-4.7-1B
export MINICPM_TF518=/path/to/transformers-5.18      # optional
export MINICPM_VISION_ONNX=MiniCPM-V-4.7-1B-vision.onnx
export MINICPM_VISION_RKNN=MiniCPM-V-4.7-1B-vision.rknn

python export_vision47.py --model "$MINICPM_HF_DIR" --out "$MINICPM_VISION_ONNX"
python export_vision_rknn.py

export_vision_rknn.py produces <name>.rknn and <name>.weight side by side.

Two non-obvious contracts

  1. The output must be 3D [1, N, hidden], not 2D [N, hidden]. The runtime reads embeds_shape[0] as number of images and embeds_shape[1] as tokens per image (see the official examples/MiniCPM_V_4/cpp/main.cc). A 2D export is parsed as "64 images of 1024 tokens each" and the image tokens never line up.

  2. image_ph * image_pw == n_image_tokens. The multimodal embed callback computes image_ph = model_height / patch_size(16) / spatial_merge_size(2); with a 448x448 vision input that is 14 x 14 = 196. MiniCPM-V 4.7's native tower emits (448/14/4)^2 = 8 x 8 = 64 tokens. A 64-token export fails on device with:

    image_ph * image_pw != n_image_tokens! image_ph = 14 ... n_image_tokens = 64
    

    Changing --img-width/--img-height or resizing the request image does not help: image_height comes from the vision RKNN model's own input shape. export_vision47.py therefore inserts a nearest-neighbour 8x8 -> 14x14 resample inside the exported graph (pure Gather/Reshape, no Resize op):

    g, idx = self.src_grid, self.resample_idx      # 8, 14
    y = y.view(g, g, -1)[idx[:, None], idx[None, :], :]
    return y.reshape(1, 196, -1)
    

    This is a runtime-contract adapter, not native 4.7 behaviour. It changes only graph structure — the .weight SHA256 is identical to the 64-token build, since no weights are touched. See the model card for what this means for equivalence claims.