| --- |
| license: apache-2.0 |
| library_name: coremltools |
| pipeline_tag: zero-shot-image-classification |
| base_model: google/siglip2-base-patch16-256 |
| tags: |
| - coreml |
| - siglip2 |
| - apple-silicon |
| - neural-engine |
| - zero-shot-image-classification |
| --- |
| |
| # SigLIP 2 base (256 px) for Core ML |
|
|
| Core ML conversion of Google's [SigLIP 2](https://arxiv.org/abs/2502.14786) |
| [`google/siglip2-base-patch16-256`](https://huggingface.co/google/siglip2-base-patch16-256), with both towers: |
| an image encoder and a text encoder, each returning an L2-normalized 768-d embedding. Labels are plain text chosen |
| at run time, so it classifies images zero-shot. Google authored the Apache-2.0 source model; Fluid Inference |
| converted it. The image encoder runs entirely on the Apple Neural Engine. |
|
|
| ## Packages |
|
|
| | Package | Input | Output | Size | |
| | --- | --- | --- | ---: | |
| | `siglip2-base-patch16-256-image-fp16.mlpackage` | `pixel_values` float32 `[1, 3, 256, 256]` | `image_embeds` float32 `[1, 768]` | 176 MB | |
| | `siglip2-base-patch16-256-text-fp16.mlpackage` | `input_ids` int32 `[1, 64]` | `text_embeds` float32 `[1, 768]` | 539 MB | |
|
|
| fp16 weights, macOS 14 / iOS 17 or newer. `config.json` holds the preprocessing and scoring constants: |
|
|
| - **Image:** resize to 256 × 256 (bilinear, antialiased like PIL), scale to [0, 1], then `(x − 0.5) / 0.5`. |
| - **Text:** lowercase, Gemma tokenizer (`tokenizer.json`), append `<eos>`, pad with `<pad>` to 64 tokens. SigLIP was |
| trained without an attention mask, so the padding must match. |
| - **Scores:** `cos = image_embeds · text_embeds`. `sigmoid(logit_scale · cos + logit_bias)` (112.90, −16.77) gives |
| an independent probability per label; softmax or argmax over `cos` picks one label. Embed the labels once and |
| reuse them for every image. |
|
|
| ## Accuracy |
|
|
| **ImageNet-1k zero-shot, all 50,000 test images** (`clip-benchmark/wds_imagenet1k`, prompt |
| `this is a photo of {class}.`), fp16 on CPU + Neural Engine against the fp32 PyTorch model: |
|
|
| | | PyTorch fp32 | Core ML fp16 | |
| | --- | ---: | ---: | |
| | Top-1 | 76.79% | 76.76% | |
| | Same prediction as PyTorch | — | 99.32% | |
| | Image embedding cosine, mean / min | — | 0.99990 / 0.975 | |
|
|
| Google reports 79.1% with its own class names and prompts; the single-prompt protocol here is lower for both |
| backends. **Oxford-IIIT Pets test (3,669 photos, 37 breeds):** Core ML 94.77%, PyTorch 94.74%, 99.89% identical. |
|
|
| ## Speed and memory |
|
|
| Apple M5 Pro (24 GB), macOS 27. Image encoder alone, one call at a time: **5.2 ms** on CPU + Neural Engine |
| (100% of ops on the ANE), 3.5 ms CPU + GPU, versus 19.1 ms for PyTorch fp32 on MPS. |
|
|
| End to end on 7,349 Oxford-IIIT Pets photos (JPEG decode, resize, encoder, scoring against 37 cached breed |
| prompts), same photos and prompts, measured after model load: |
|
|
| | | Core ML (Swift, 4 photos in flight) | PyTorch fp32, MPS, batch 32 | |
| | --- | ---: | ---: | |
| | Time | 36.3 s | 102.2 s | |
| | Photos per second | 202 | 72 | |
| | Peak memory (process) | 262 MB | 4.24 GB | |
| | Model on disk | 715 MB | 1.50 GB | |
| | Accuracy | 94.26% | 94.11% | |
|
|
| PyTorch batch 1: 153.5 s, 48 photos/s, 3.64 GB. Reports: `reports/`. |
|
|
| ## Use from Swift |
|
|
| [FluidUse](https://github.com/FluidInference/FluidUse) wraps both packages (`SigLIP2Manager`, Swift Gemma tokenizer |
| with token ids identical to the Python tokenizer) and includes the photo-sorting demo `ImageSortDemo`. |
| Conversion scripts: [FluidInference/mobius](https://github.com/FluidInference/mobius) `models/emb/siglip2/coreml`. |
|
|
