File size: 4,010 Bytes
3e8b112
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
---
license: apache-2.0
base_model: google/siglip2-base-patch16-224
tags:
  - vision
  - siglip2
  - image-text-retrieval
  - core-ai
  - apple-silicon
  - macos
library_name: swift-photo-index
---

# pix-siglip2-base-224: SigLIP 2 for Core AI (macOS 27)

Google's [SigLIP 2 base/16 at 224](https://huggingface.co/google/siglip2-base-patch16-224)
(Tschannen et al., *SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic
Understanding, Localization, and Dense Features*, 2025; Apache-2.0) exported to an Apple Core AI
`.aimodel`. It puts pictures and sentences in one 768-dimensional space, so a photo of a dog on a
beach lands near the words "dog on a beach".

This is what meaning search in [`pix`](https://github.com/arraypress/swift-pix-cli) runs on, through
[swift-photo-index](https://github.com/arraypress/swift-photo-index). `pix index` stores one vector per
image (one per video: the mean of its sampled frames). `pix "dog on a beach"` embeds the query and
ranks by cosine. Words that are literally there (text read off the picture, speech, the file name)
still rank first, so an exact invoice number beats anything that only matches by meaning. pix
downloads this repository by itself the first time it needs it, at a pinned commit, and checks every
file against its published hash. Everything runs on device.

## Files

| Path | What |
|---|---|
| `pix-siglip2-base-224.aimodel/` | Two entry points. `image`: `pixel_values` `[1,3,224,224]` float32 → `embedding` `[1,768]`. `text`: `ids` `[1,64]` int32 → `embedding` `[1,768]`. Float16 weights, 750 MB. |
| `pix-siglip2-support/vocab.bin`, `merges.bin` | The checkpoint's Gemma tokenizer (256,000 tokens) as flat binaries: `u32` count, then per id `u32` length + UTF-8; merges as `u32` (left, right, merged) id triples in rank order. |
| `pix-siglip2-support/config.json` | Source model and revision, sizes, normalisation, special-token ids. |
| `export_siglip2.py` | The export script (also writes the parity fixtures). |

The host does the rest, as `transformers` does. Images: resize to 224×224 without keeping the
aspect ratio, RGB, `(x/255 − 0.5)/0.5`, channel-first. Text: lower-case, spaces to `▁`, byte-fallback
BPE, append `<eos>`=1, right-pad with `<pad>`=0 to 64. Then L2-normalise both outputs and take
dot products.

## Fidelity

Measured from Swift through Core AI against `transformers` (float32) on the same inputs. Tokenizer
ids are identical on 10 test strings (accents, emoji, CJK, Arabic, tabs, truncation at 64). Text
embeddings: worst cosine 0.9999993. Image embeddings: cosine 0.99999 when no resize is needed, and
0.997 on a 400×300 image, where Core Graphics' resampling differs slightly from PIL's.

## Retrieval

1,000 COCO val2017 images (the first 1,000 by id) against their 5,003 captions, embedded through
this asset in Swift:

- Text → image recall: R@1 **0.720**, R@5 **0.916**, R@10 **0.959**.
- Through pix's own search (meaning blended with words, 0.09 cut-off): R@1 0.719, R@5 0.911,
  R@10 0.948. Words alone find none of the captions (0/0/0), because full sentences never appear
  in OCR text or labels.
- Cosine of true pairs: 5th percentile 0.101, median 0.150. Wrong pairs: 99th percentile 0.087,
  99.9th 0.135. pix's 0.09 floor keeps 97.5% of true pairs and drops 99.1% of wrong ones.
- Speed (Apple silicon, GPU): 12 ms per image, 4 ms per sentence, 2 s cold model load. Scoring a
  query against 21,000 stored float16 vectors takes about 25–35 ms.

## Use

```sh
pix index ~/Pictures                  # fetches this model on first use
pix "a red car at the beach at sunset"
```

To install it by hand, run `hf download arraypress/pix-siglip2-base-224 --local-dir models` and then
`pix model install models`. Requires macOS 27 (Core AI) on Apple silicon. Converted with coreai-torch
0.4.3, coreai-core 1.0.0b3, torch 2.9.0 and transformers 5.12.1.

## Licence

Apache-2.0, the same as the original. The weights and tokenizer are Google's; see `NOTICE` for what
was changed.