README.md

---
license: apache-2.0
base_model: openbmb/MiniCPM-V-4.6
tags:
- core-ai
- coreai
- vision-language
- vlm
- on-device
- iphone
- apple
pipeline_tag: image-text-to-text
language:
- en
- zh
---

# MiniCPM-V-4.6 — Core AI

**On-device vision-language model for iPhone / Apple Silicon.** A Core AI port of
[`openbmb/MiniCPM-V-4.6`](https://huggingface.co/openbmb/MiniCPM-V-4.6) — the strongest
sub-2B open VLM — running fully local on the GPU via the Core AI **pipelined engine**:
pick a photo, ask about it, stream the answer.

Verified on **iPhone 17 Pro**: image → grounded answer at **~51.5 tok/s** decode, all local.

<p align="center">
  <img src="https://github.com/user-attachments/assets/c4baa524-5217-4bb3-a23f-b0acd6249bd4" width="300" alt="MiniCPM-V 4.6 on iPhone — a fridge photo becomes recipe ideas, fully on-device in CoreAIChat">
</p>
<p align="center"><em>Fridge photo → recipe ideas, fully on-device on an iPhone 17 Pro (CoreAIChat).</em></p>

<!-- gen-cards:use-it begin id=minicpm-v-4.6 (managed by scripts/gen-cards — edit cards.json / QuickStart.swift, not this block) -->
## Use it

⚡ **One line** — run the kit's task op on this model
(`import CoreAIOps`; no session, no model plumbing, downloads on first use):

```swift
let caption = try await CoreAI.caption(imageAt: url, options: .model("minicpm-v-4.6"))
```

Twenty ops, one shape — [Cookbook](https://github.com/john-rocky/coreai-kit/blob/main/docs/COOKBOOK.md).

▶️ **Run it (source)** — the [VLChat runner](https://github.com/john-rocky/coreai-kit/tree/main/Examples/VLChat)
(GUI + CLI, one app for every vision-language model in the catalog):

```bash
git clone https://github.com/john-rocky/coreai-kit
open coreai-kit/Examples/VLChat/VLChat.xcodeproj
# → Run, then pick "MiniCPM-V 4.6" in the model picker

# agents / headless (macOS):
cd coreai-kit/Examples/VLChat
swift run vlchat-cli --model minicpm-v-4.6 --image sample.jpg --prompt "What is in this image?"
```

💻 **Build with it** — complete; the glue is kit API, copy-paste runs:

```swift
import CoreAIKit
import FoundationModels

let vlm = try await KitVisionModel(catalog: "minicpm-v-4.6")
let session = LanguageModelSession(model: vlm)
let image = try ImageFile.load(imageURL)  // any image file → CGImage + EXIF orientation
let reply = try await session.respond(to: Prompt {
    prompt
    Attachment(image.cgImage, orientation: image.orientation)
})
// reply.content: the answer about the image, generated fully on-device
```

The take-home is [`Examples/VLChat/Sources/QuickStart.swift`](https://github.com/john-rocky/coreai-kit/blob/main/Examples/VLChat/Sources/QuickStart.swift)
— this exact code as one typed function, no UI; the CLI is an argument shell over it, and
the GUI drives the same `KitVisionModel(catalog:)` behind a `LanguageModelSession`.
Multi-turn about the same image? Hold the `LanguageModelSession` and call `respond(to:)`
per turn. The photo picker / file chooser is your app's own chrome — `ImageFile.load`
(kit API) turns any image file into model input.

**Integration checklist**

- SPM: `https://github.com/john-rocky/coreai-kit` → product **CoreAIKit**
- Info.plist: `NSPhotoLibraryUsageDescription` — only if you use PhotosPicker
- Entitlements (iOS): `com.apple.developer.kernel.increased-memory-limit`
- First run downloads the model — 2.1 GB (Mac) / 2.1 GB (iPhone) — then it loads from the
  local cache (Application Support; progress via the `downloadProgress` callback)
- Measure in Release — Debug is ~3× slower on per-token host work
<!-- gen-cards:use-it end -->

## Architecture

MiniCPM-V-4.6 (1.3B) = a **SigLIP So400m vision tower** (980px / patch 14 / 27 layers, with a
window-attention insert-merger @ layer 6 + a downsample-MLP merger → ÷16 = 64 visual tokens per
448px slice) + a **Qwen3.5-hybrid text backbone** (`qwen3_5_text`: 0.8B, 24 layers, GatedDeltaNet
linear attention ×3 : full attention ×1, head_dim 256, vocab 248094, tied head). Connector =
2×2 spatial merges + MLP, spliced into the text embeddings at `<image>` positions (`masked_scatter`).

## Bundles

**Recommended (optimized, 2026-06-25):**

| path | what | dtype | size |
|---|---|---|---|
| `gpu-pipelined/minicpmv46_vlm_decode_int8hu/` | VLM text decoder (`input_ids → logits` + static `image_embeds[64,1024]`; in-graph gather `ids ≥ V ? image_embeds[ids-V] : embed[ids]`) | int8 body + **untied int8 head** | ~1.2 GB |
| `gpu-pipelined/minicpmv46_vision_int8lin/` | fixed-grid SigLIP vision encoder (`pixel_values[1,3,448,448] → image_features[64,1024]`) | **int8** | ~0.6 GB |

The **int8 head** quantizes the big-vocab LM head (fp16 in `int8lin` = ~half the per-token read) → **+48% decode
on iPhone 17 Pro** (46→68 tok/s). The **int8 vision** halves the encoder's size (the encode is compute-bound, so
this is a size/memory win); pair it with a one-shot vision-graph warmup at load to hide the ~2.7 s first-photo
cold compile. Original `…_int8lin` decoder + fp16 `minicpmv46_vision` remain for compatibility.

The decoder is a complete qwen3.5-hybrid text LLM when `image_embeds` is zero — same bundle, no image needed.

## How a VLM rides the text-only engine

The pipelined engine knows nothing about images. The whole multimodal state rides the
**static-input hook** (`image_embeds` buffer) + an id-space trick — the graph stays `ids + positions → logits`:

- The host runs the vision encoder **once per image** (resize 448, normalize `x/127.5−1`) and writes
  `image_embeds [64,1024]` into one owned MTLBuffer the engine binds on every step.
- The prompt's `<|image_pad|>` ids are rewritten to **extension ids** `V + slot` (slot 0..63).
  In-graph: `embed = ids < V ? table[ids] : image_embeds[ids − V]`.
- Positions are **plain 1D** (no M-RoPE / no rope-shift), the qwen3.5-hybrid KV + conv + recurrent
  states are the engine's; nothing else changes.

Simpler than the [Qwen3-VL port](https://huggingface.co/mlboydaisuke/Qwen3-VL-2B-CoreAI) (no deepstack, no M-RoPE).

## Measured (iPhone 17 Pro, iOS 27 beta, release)

- **iPhone 17 Pro decode (int8 head)**: VLM in-app A/B, same conditions, back-to-back — int8lin **51.5 → int8hu
  70.0 tok/s = +36%** (the VLM bundle binds `image_embeds` every step, which dilutes the head gain; the text core
  alone is 46 → 68 = +48%). ~64–70 tok/s in practice by device temperature. · M4 Max text core ~224 tok/s
  (`llm-benchmark`), engine cold-spec ~2–4 s, ~1.5 GB resident (jetsam-safe).
- **Vision**: int8 encoder ≈ fp16 encode time (compute-bound) at ~0.6 GB (half); the ~2.7 s first-image latency
  is the SigLIP graph's cold compile — run a dummy encode at load to make the user's first photo warm (~tens of ms).
- **Numerics**: fp32-torch parity bit-exact (vision cos 1.000000, full overlay logits cos 1.00004);
  Core AI engine ≡ python ≡ HF (text 24/24; image path reproduces the HF description modulo one int8
  near-tie token, then reconverges).
- Real-photo example (kakigōri): *"a bowl of shaved ice ... chunks of mango ... a dark blue saucer ...
  a menu or a book, hinting at a café ... a wooden table"* — accurate, fully on-device.

## Use it

`apps/CoreAIChat` and the standalone `MiniCPMVLM` app have a **MiniCPM-V 4.6 mode with a photo picker**:
pick an image, ask, stream. The vision tower runs once per image (~hundreds of ms); each turn re-prefills (S=1).

Conversion + gates: see [coreai-model-zoo / minicpm-v-4.6](https://github.com/john-rocky/coreai-model-zoo/blob/main/zoo/minicpm-v-4.6.md).

License: Apache-2.0 (inherited from `openbmb/MiniCPM-V-4.6`).