arraypress's picture
SigLIP 2 base/16 224 for Core AI (pix)
3e8b112 verified
|
Raw History Blame Contribute Delete
4.01 kB
metadata
license: apache-2.0
base_model: google/siglip2-base-patch16-224
tags:
  - vision
  - siglip2
  - image-text-retrieval
  - core-ai
  - apple-silicon
  - macos
library_name: swift-photo-index

pix-siglip2-base-224: SigLIP 2 for Core AI (macOS 27)

Google's SigLIP 2 base/16 at 224 (Tschannen et al., SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features, 2025; Apache-2.0) exported to an Apple Core AI .aimodel. It puts pictures and sentences in one 768-dimensional space, so a photo of a dog on a beach lands near the words "dog on a beach".

This is what meaning search in pix runs on, through swift-photo-index. pix index stores one vector per image (one per video: the mean of its sampled frames). pix "dog on a beach" embeds the query and ranks by cosine. Words that are literally there (text read off the picture, speech, the file name) still rank first, so an exact invoice number beats anything that only matches by meaning. pix downloads this repository by itself the first time it needs it, at a pinned commit, and checks every file against its published hash. Everything runs on device.

Files

Path What
pix-siglip2-base-224.aimodel/ Two entry points. image: pixel_values [1,3,224,224] float32 → embedding [1,768]. text: ids [1,64] int32 → embedding [1,768]. Float16 weights, 750 MB.
pix-siglip2-support/vocab.bin, merges.bin The checkpoint's Gemma tokenizer (256,000 tokens) as flat binaries: u32 count, then per id u32 length + UTF-8; merges as u32 (left, right, merged) id triples in rank order.
pix-siglip2-support/config.json Source model and revision, sizes, normalisation, special-token ids.
export_siglip2.py The export script (also writes the parity fixtures).

The host does the rest, as transformers does. Images: resize to 224×224 without keeping the aspect ratio, RGB, (x/255 − 0.5)/0.5, channel-first. Text: lower-case, spaces to ▁, byte-fallback BPE, append <eos>=1, right-pad with <pad>=0 to 64. Then L2-normalise both outputs and take dot products.

Fidelity

Measured from Swift through Core AI against transformers (float32) on the same inputs. Tokenizer ids are identical on 10 test strings (accents, emoji, CJK, Arabic, tabs, truncation at 64). Text embeddings: worst cosine 0.9999993. Image embeddings: cosine 0.99999 when no resize is needed, and 0.997 on a 400×300 image, where Core Graphics' resampling differs slightly from PIL's.

Retrieval

1,000 COCO val2017 images (the first 1,000 by id) against their 5,003 captions, embedded through this asset in Swift:

  • Text → image recall: R@1 0.720, R@5 0.916, R@10 0.959.
  • Through pix's own search (meaning blended with words, 0.09 cut-off): R@1 0.719, R@5 0.911, R@10 0.948. Words alone find none of the captions (0/0/0), because full sentences never appear in OCR text or labels.
  • Cosine of true pairs: 5th percentile 0.101, median 0.150. Wrong pairs: 99th percentile 0.087, 99.9th 0.135. pix's 0.09 floor keeps 97.5% of true pairs and drops 99.1% of wrong ones.
  • Speed (Apple silicon, GPU): 12 ms per image, 4 ms per sentence, 2 s cold model load. Scoring a query against 21,000 stored float16 vectors takes about 25–35 ms.

Use

pix index ~/Pictures                  # fetches this model on first use
pix "a red car at the beach at sunset"

To install it by hand, run hf download arraypress/pix-siglip2-base-224 --local-dir models and then pix model install models. Requires macOS 27 (Core AI) on Apple silicon. Converted with coreai-torch 0.4.3, coreai-core 1.0.0b3, torch 2.9.0 and transformers 5.12.1.

Licence

Apache-2.0, the same as the original. The weights and tokenizer are Google's; see NOTICE for what was changed.