Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,145 @@
+---
+license: apache-2.0
+base_model: google/siglip2-base-patch16-256
+tags:
+  - coreml
+  - siglip2
+  - siglip
+  - clip
+  - image-text
+  - embeddings
+  - vision
+---
+
+# SigLIP2 base-patch16-256 — CoreML (fp16)
+
+CoreML `.mlpackage` conversion of
+[`google/siglip2-base-patch16-256`](https://huggingface.co/google/siglip2-base-patch16-256)
+for on-device (macOS/Apple Silicon) text/image embedding — built for
+[Alarian](https://alarian.ai)'s semantic frame search.
+
+**Provenance**: converted directly from the Hugging Face `transformers`
+reference implementation and `coremltools`, by nodevorg. Not affiliated with
+Google. The conversion script is public:
+[`tools/siglip2-coreml/convert.py`](https://github.com/nodevorg/studio) (in
+the `nodevorg/studio` repo, kept uncommitted pending review — ask if you want
+a copy). This project intentionally never looked at or referenced any
+third-party GPLv3 conversion script; it was written from the HF
+`transformers` SigLIP2 modeling code and `coremltools` docs only.
+
+**License**: apache-2.0, inherited from the base model. Same terms apply
+here — free for commercial and closed-source use.
+
+## What's in this repo
+
+| file | what it is |
+|---|---|
+| `ImageEncoder.mlpackage.zip` | image tower → 768-dim embedding |
+| `TextEncoder.mlpackage.zip` | text tower → 768-dim embedding |
+| `tokenizer.json` | Gemma BPE tokenizer (256k vocab), verbatim from the source repo |
+| `tokenizer_config.json` | tokenizer config (padding_side=right, do_lower_case=true), verbatim |
+| `special_tokens_map.json` | special tokens, verbatim |
+
+### A note on architecture
+
+`google/siglip2-base-patch16-256`'s own `config.json` self-describes as
+`model_type: siglip` (not `siglip2`) — at this fixed-256-resolution, non-NaFlex
+tier, the SigLIP2-trained weights are architecturally identical to SigLIP v1
+(`transformers.SiglipModel`); only the weights and the tokenizer (Gemma BPE,
+256k vocab, vs. SigLIP v1's 32k SentencePiece) differ. `AutoModel.from_pretrained`
+resolves this checkpoint to `SiglipModel`, and that's what was converted.
+
+## Inputs / outputs
+
+### ImageEncoder.mlpackage
+
+- **Input**: `pixel_values` — a CoreML `ImageType`, RGB, 256×256. The exact
+  SigLIP preprocessing (`SiglipImageProcessor` defaults for this checkpoint:
+  `rescale_factor=1/255`, `image_mean=[.5,.5,.5]`, `image_std=[.5,.5,.5]`) is
+  **baked into the model** as `scale=1/127.5, bias=[-1,-1,-1]` — i.e. the
+  model itself maps raw RGB `[0,255]` to `[-1, 1]`. Just resize/squash your
+  image to 256×256 and hand it the raw RGB pixels; no separate normalization
+  step needed on the caller side. (Aspect-ratio squash, not center-crop — the
+  model saw squashed images in training.)
+- **Output**: `image_embedding` — 768-dim float16, **already L2-normalized**.
+
+### TextEncoder.mlpackage
+
+- **Input**: `input_ids` — 1×64 **int32** token ids. No `attention_mask`
+  input: SigLIP's text tower trains with unmasked bidirectional attention
+  over the full fixed-length 64-token sequence and pools the hidden state at
+  the *last* sequence position (`SiglipTextModel.forward`, not an EOS-aware
+  pool) — passing a mask would deviate from training. Tokenize with the
+  included `tokenizer.json`/`tokenizer_config.json` using
+  `padding="max_length", max_length=64`, and **lowercase the input text**
+  (`do_lower_case=True` in the tokenizer config; the tokenizer will also do
+  this itself if you feed it through its normalizer, but be aware if you
+  hand-roll token ids).
+- **Output**: `text_embedding` — 768-dim float16, **already L2-normalized**.
+
+Both encoders normalize their pooled output inside the graph, so cosine
+similarity between an image and text embedding is just a dot product.
+
+## Conversion details
+
+- `minimum_deployment_target`: macOS 15
+- `compute_precision`: `ct.precision.FLOAT16`
+- `convert_to`: `mlprogram`
+- Traced via `torch.jit.trace` on thin wrapper modules around
+  `SiglipModel.vision_model` / `.text_model` (not the full contrastive model —
+  each `.mlpackage` is just the one branch).
+- Pinned toolchain (see the sha256 table below for exact artifact hashes):
+  `torch==2.7.0`, `transformers==5.14.1`, `coremltools==9.0`, `numpy==2.3.5`.
+  **Gotcha**: `numpy>=2.4.0` breaks `coremltools==9.0`'s PyTorch frontend with
+  `TypeError: only 0-dimensional arrays can be converted to Python scalars`
+  on any model using `torch.nn.MultiheadAttention` (SigLIP's vision pooling
+  head hits this) — confirmed via
+  [apple/coremltools#2633](https://github.com/apple/coremltools/issues/2633);
+  pin `numpy<2.4.0` until a coremltools release ships the fix from
+  [#2632](https://github.com/apple/coremltools/pull/2632).
+
+## Parity (PyTorch fp32 reference vs. converted CoreML fp16, cosine similarity)
+
+4 test images (solid red / green / blue squares + a synthetic sunset
+gradient) × 4 test strings, L2-normalized before comparison. Threshold: > 0.99
+per item.
+
+| item | cosine(pytorch, coreml) |
+|---|---|
+| image: solid_red | 0.9999988 |
+| image: solid_green | 0.9999992 |
+| image: solid_blue | 0.9999992 |
+| image: gradient_sky | 0.9999987 |
+| text: "a red square" | 0.9999995 |
+| text: "a photo of a sunset over the ocean" | 0.9999997 |
+| text: "a solid blue color" | 0.9999995 |
+| text: "a green field" | 0.9999996 |
+
+**Worst per-item cosine similarity: 0.9999987** — well above the 0.99
+threshold.
+
+Text↔image ranking also matches exactly between PyTorch and CoreML across
+all 4×4 pairs (e.g. "a red square" ranks `solid_red` highest in both
+backends; "a photo of a sunset over the ocean" ranks `gradient_sky` highest
+in both).
+
+## sha256
+
+| file | sha256 |
+|---|---|
+| `ImageEncoder.mlpackage.zip` | `406938456c8f8e91f01632cb74131fdfb993064d8d0bc8f6fbbd142709f1ba66` |
+| `TextEncoder.mlpackage.zip` | `4f31f38a1729a0044f0ffc95a940dc1a6a2fdcd5fcf78a03cbb662bada6cbafb` |
+| `tokenizer.json` | `cb9140fae3ac5122c972d37adf83e1248471a38147ad76f8215c8872c6fd8322` |
+| `tokenizer_config.json` | `14afe629fe4959b9e0d51e1852b8d9f7ad074f90a1a7125a4fcdd17f06e78fc8` |
+| `special_tokens_map.json` | `baec30ea10906f16adb8c18af7a34023002c1746542612b8b41c9f09e1351351` |
+
+## File sizes
+
+| file | size |
+|---|---|
+| `ImageEncoder.mlpackage.zip` | 163 MiB |
+| `TextEncoder.mlpackage.zip` | 506 MiB |
+| `tokenizer.json` | 33 MiB |
+
+The text encoder is large mostly because of the Gemma tokenizer's 256k-entry
+vocabulary embedding table (256000 × 768 × 2 bytes ≈ 375 MiB alone, fp16).