--- license: apache-2.0 base_model: google/siglip2-base-patch16-224 tags: - vision - siglip2 - image-text-retrieval - core-ai - apple-silicon - macos library_name: swift-photo-index --- # pix-siglip2-base-224: SigLIP 2 for Core AI (macOS 27) Google's [SigLIP 2 base/16 at 224](https://huggingface.co/google/siglip2-base-patch16-224) (Tschannen et al., *SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features*, 2025; Apache-2.0) exported to an Apple Core AI `.aimodel`. It puts pictures and sentences in one 768-dimensional space, so a photo of a dog on a beach lands near the words "dog on a beach". This is what meaning search in [`pix`](https://github.com/arraypress/swift-pix-cli) runs on, through [swift-photo-index](https://github.com/arraypress/swift-photo-index). `pix index` stores one vector per image (one per video: the mean of its sampled frames). `pix "dog on a beach"` embeds the query and ranks by cosine. Words that are literally there (text read off the picture, speech, the file name) still rank first, so an exact invoice number beats anything that only matches by meaning. pix downloads this repository by itself the first time it needs it, at a pinned commit, and checks every file against its published hash. Everything runs on device. ## Files | Path | What | |---|---| | `pix-siglip2-base-224.aimodel/` | Two entry points. `image`: `pixel_values` `[1,3,224,224]` float32 → `embedding` `[1,768]`. `text`: `ids` `[1,64]` int32 → `embedding` `[1,768]`. Float16 weights, 750 MB. | | `pix-siglip2-support/vocab.bin`, `merges.bin` | The checkpoint's Gemma tokenizer (256,000 tokens) as flat binaries: `u32` count, then per id `u32` length + UTF-8; merges as `u32` (left, right, merged) id triples in rank order. | | `pix-siglip2-support/config.json` | Source model and revision, sizes, normalisation, special-token ids. | | `export_siglip2.py` | The export script (also writes the parity fixtures). | The host does the rest, as `transformers` does. Images: resize to 224×224 without keeping the aspect ratio, RGB, `(x/255 − 0.5)/0.5`, channel-first. Text: lower-case, spaces to `▁`, byte-fallback BPE, append ``=1, right-pad with ``=0 to 64. Then L2-normalise both outputs and take dot products. ## Fidelity Measured from Swift through Core AI against `transformers` (float32) on the same inputs. Tokenizer ids are identical on 10 test strings (accents, emoji, CJK, Arabic, tabs, truncation at 64). Text embeddings: worst cosine 0.9999993. Image embeddings: cosine 0.99999 when no resize is needed, and 0.997 on a 400×300 image, where Core Graphics' resampling differs slightly from PIL's. ## Retrieval 1,000 COCO val2017 images (the first 1,000 by id) against their 5,003 captions, embedded through this asset in Swift: - Text → image recall: R@1 **0.720**, R@5 **0.916**, R@10 **0.959**. - Through pix's own search (meaning blended with words, 0.09 cut-off): R@1 0.719, R@5 0.911, R@10 0.948. Words alone find none of the captions (0/0/0), because full sentences never appear in OCR text or labels. - Cosine of true pairs: 5th percentile 0.101, median 0.150. Wrong pairs: 99th percentile 0.087, 99.9th 0.135. pix's 0.09 floor keeps 97.5% of true pairs and drops 99.1% of wrong ones. - Speed (Apple silicon, GPU): 12 ms per image, 4 ms per sentence, 2 s cold model load. Scoring a query against 21,000 stored float16 vectors takes about 25–35 ms. ## Use ```sh pix index ~/Pictures # fetches this model on first use pix "a red car at the beach at sunset" ``` To install it by hand, run `hf download arraypress/pix-siglip2-base-224 --local-dir models` and then `pix model install models`. Requires macOS 27 (Core AI) on Apple silicon. Converted with coreai-torch 0.4.3, coreai-core 1.0.0b3, torch 2.9.0 and transformers 5.12.1. ## Licence Apache-2.0, the same as the original. The weights and tokenizer are Google's; see `NOTICE` for what was changed.