ChaseLean/siglip2-base-patch16-256-coreml-fp16

SigLIP 2 Base image encoder for ONNX Runtime's Core ML provider

This is the image encoder of Google's SigLIP 2 Base, patch 16, 256 px as an fp16 ONNX file. It is for developers who want to run the encoder on an Apple Silicon GPU with ONNX Runtime. The stock fp16 export fails to compile under ONNX Runtime's Core ML provider, or compiles in up to 95 seconds and then uses over 6 GB of memory. This copy is rewritten so that it loads in under a second, with the same weights and the same math.

It was made for Orchard Photos, a private photo browser with AI search that runs on your own computer.

Files

File	Location	Size	SHA-256
This file	vision_model_fp16_coreml.onnx	186,092,579 bytes	87aa52ceed129728ae749782dfafb7cb4e92060f9018776040071b6d85caebc9
Source	onnx-community/siglip2-base-patch16-256-ONNX, onnx/vision_model_fp16.onnx, revision d1114256522a37ffa257a0a58017348ab0058db2		fe9ad8020a6d3d98d394c9be8f07064066135fc2f87ec11692de0b677c0ac4db

What the rewrite changes

There are three changes, and none of them changes the result.

• The patch embedding gets explicit padding. Its Conv uses auto_pad=VALID, which the Core ML converter rejects. It now uses pads=[0, 0, 0, 0], which is what VALID means.
• Each 2-D MatMul weight, K × N, gets a leading dimension of 1, becoming 1 × K × N. Broadcasting gives the same product. Without this, the converter writes the weights out as a 1.1 GB text file, which takes 55 to 95 seconds to compile.
• The batch is fixed at 16. Core ML fails to load the model when the batch size varies. Each image in a batch is still processed on its own, so the batch size doesn't affect any image's output.

The shape information is then regenerated for the fixed batch. The operators, weights, input and outputs are otherwise those of the source file.

Inputs and outputs

Name	Type	Shape
Input: pixel_values	float32	[16, 3, 256, 256]
Output: last_hidden_state	float32	[16, 256, 768]
Output: pooler_output	float32	[16, 768]

Use pooler_output as the image embedding. To embed fewer than 16 images, fill the rest of the batch with zeros and drop those rows from the output.

Preprocess each image the way SigLIP 2 does. Resize it to 256 × 256 without cropping, which squashes it to a square, then scale the pixels to [-1, 1]. For text, use the text encoder from the source repository.

Usage

import numpy as np
import onnxruntime
from PIL import Image

BATCH = 16  # the model's fixed batch size


def pixels(path):
    """Resize to 256 x 256 with no crop, scale to [-1, 1], channels first."""
    image = Image.open(path).convert("RGB").resize((256, 256), Image.Resampling.BILINEAR)
    return (np.asarray(image, dtype=np.float32) / 127.5 - 1).transpose(2, 0, 1)


session = onnxruntime.InferenceSession(
    "vision_model_fp16_coreml.onnx",
    providers=[
        ("CoreMLExecutionProvider", {"ModelFormat": "MLProgram", "MLComputeUnits": "CPUAndGPU"}),
        "CPUExecutionProvider",
    ],
)

paths = ["photo1.jpg", "photo2.jpg", "photo3.jpg"]  # up to 16
batch = np.zeros((BATCH, 3, 256, 256), dtype=np.float32)
batch[:len(paths)] = [pixels(path) for path in paths]
vectors = session.run(["pooler_output"], {"pixel_values": batch})[0][:len(paths)]
vectors /= np.linalg.norm(vectors, axis=1, keepdims=True)

Set MLComputeUnits to CPUAndGPU. With ALL, Core ML picks the Neural Engine, which runs this model at about half the speed. On a computer without Core ML, use the fp32 file from the source repository instead, since fp16 is slower on a CPU.

Accuracy and speed

These were measured on an Apple M5 with ONNX Runtime 1.31.0, using the settings in the example above. The reference is the source repository's fp32 image encoder on the CPU.

	This file on the Core ML GPU	fp32 file on the CPU, 4 threads
Model time per image, excluding decoding	4.7 ms	47.3 ms
Load time, no cache	0.65–0.95 s	0.13 s
Peak memory	0.64 GB	1.18 GB

The embeddings are almost identical to the reference. On a sample of 1,000 photos and screenshots, their cosine similarity was 0.99999 on average and 0.9999 at the lowest. On another 1,211 screenshots, the lowest was 0.9997.

Rebuilding it

The script scripts/build-coreml-model in Orchard's repository builds this file. It downloads the source file at the pinned revision, checks its SHA-256, applies the rewrite, and prints the SHA-256 of the result. With onnx 1.23.2, the output matches this file byte for byte. Other onnx versions may give a different SHA-256.

Run it from a checkout of Orchard, with its Python environment set up (ONNX Runtime 1.31.0):

.venv/bin/pip install onnx==1.23.2
.venv/bin/python scripts/build-coreml-model OUTPUT_FOLDER

License and attribution

The license is Apache 2.0, the same as the original model. See LICENSE and NOTICE.

• SigLIP 2 and its weights are by Google: google/siglip2-base-patch16-256, under Apache 2.0.
• The ONNX export is by onnx-community: onnx-community/siglip2-base-patch16-256-ONNX.
• This repository changes that export's fp16 image encoder as described above.

@misc{tschannen2025siglip2multilingualvisionlanguage,
  title={SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features},
  author={Michael Tschannen and Alexey Gritsenko and Xiao Wang and Muhammad Ferjad Naeem and Ibrahim Alabdulmohsin and Nikhil Parthasarathy and Talfan Evans and Lucas Beyer and Ye Xia and Basil Mustafa and Olivier Hénaff and Jeremiah Harmsen and Andreas Steiner and Xiaohua Zhai},
  year={2025},
  eprint={2502.14786},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2502.14786},
}