File size: 4,005 Bytes
fd11f30
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
# Reproducing the conversion

These are **portable copies** of the two scripts that produced the vision tower
in `../rknn/`. All absolute paths have been replaced by arguments / environment
variables; the model math is unchanged.

## Prerequisites

- `rknn3-toolkit` (the whl from the RKNN3 SDK, installed with `--no-deps`)
- `transformers >= 5.18` (`MiniCPMV4_7ForConditionalGeneration` does not exist
  in 5.3). Point `MINICPM_TF518` at a separate checkout if your main env is older.
- `torch`, `onnx`, `safetensors`
- A local checkout of [`openbmb/MiniCPM-V-4.7-1B`](https://huggingface.co/openbmb/MiniCPM-V-4.7-1B)
- Official [`airockchip/rknn3-model-zoo`](https://github.com/airockchip/rknn3-model-zoo) V1.1.0
  for the LLM-side `export_llm.py` (the text tower is the same `Qwen3_5ForCausalLM`
  family as the zoo's `Qwen3_5` example).

## 1. Text tower (four artifacts)

The 4.7 text tower is architecturally Qwen3.5-0.8B with a different vocabulary:

| Item | 0.8B | MiniCPM-V 4.7 1B |
| --- | --- | --- |
| `vocab_size` | 248320 | **248094** |
| `tie_word_embeddings` | true | **false** (independent `lm_head.weight` [248094,1024]) |
| `conv_state` | [18,6144,4] | [18,6144,4] (identical) |

Build a plain-text HF directory by keeping only `model.language_model.*` tensors
plus `lm_head.weight`, copying the 4.7 tokenizer files, and starting from the
zoo's Qwen3.5 `config.json` with the two changes above. Then run the zoo's
`Qwen3_5/python/export_llm.py` followed by `export_rknn.py` with
`w4a16 / normal / group32 / core_num=8`.

> **Top-level `tie_word_embeddings` trap.** Hugging Face `tie_weights()` reads the
> **top-level** config key, not `text_config`. Changing only `text_config` leaves
> the independent 254M-parameter head silently bound to `embed_tokens`, and the
> converted model emits word salad. Change **both**. The only reliable check is
> `cmp`-ing the produced `embed.bin` against the head weights — file sizes can be
> byte-identical between a good and a bad build.

## 2. Vision tower (two artifacts)

```bash
export MINICPM_HF_DIR=/path/to/MiniCPM-V-4.7-1B
export MINICPM_TF518=/path/to/transformers-5.18      # optional
export MINICPM_VISION_ONNX=MiniCPM-V-4.7-1B-vision.onnx
export MINICPM_VISION_RKNN=MiniCPM-V-4.7-1B-vision.rknn

python export_vision47.py --model "$MINICPM_HF_DIR" --out "$MINICPM_VISION_ONNX"
python export_vision_rknn.py
```

`export_vision_rknn.py` produces `<name>.rknn` and `<name>.weight` side by side.

### Two non-obvious contracts

1. **The output must be 3D `[1, N, hidden]`, not 2D `[N, hidden]`.**
   The runtime reads `embeds_shape[0]` as *number of images* and
   `embeds_shape[1]` as *tokens per image* (see the official
   `examples/MiniCPM_V_4/cpp/main.cc`). A 2D export is parsed as
   "64 images of 1024 tokens each" and the image tokens never line up.

2. **`image_ph * image_pw == n_image_tokens`.** The multimodal embed callback
   computes `image_ph = model_height / patch_size(16) / spatial_merge_size(2)`;
   with a 448x448 vision input that is `14 x 14 = 196`. MiniCPM-V 4.7's native
   tower emits `(448/14/4)^2 = 8 x 8 = 64` tokens. A 64-token export fails on
   device with:

   ```text
   image_ph * image_pw != n_image_tokens! image_ph = 14 ... n_image_tokens = 64
   ```

   Changing `--img-width/--img-height` or resizing the request image does **not**
   help: `image_height` comes from the vision RKNN model's own input shape.
   `export_vision47.py` therefore inserts a **nearest-neighbour 8x8 -> 14x14
   resample** inside the exported graph (pure Gather/Reshape, no `Resize` op):

   ```python
   g, idx = self.src_grid, self.resample_idx      # 8, 14
   y = y.view(g, g, -1)[idx[:, None], idx[None, :], :]
   return y.reshape(1, 196, -1)
   ```

   This is a **runtime-contract adapter, not native 4.7 behaviour**. It changes
   only graph structure — the `.weight` SHA256 is identical to the 64-token
   build, since no weights are touched. See the model card for what this means
   for equivalence claims.