Rewrite model card as a reference card

README.md

@@ -1,6 +1,7 @@
 ---
 license: apache-2.0
 base_model: google/siglip2-base-patch16-256
+pipeline_tag: feature-extraction
 tags:
   - coreml
   - siglip2
@@ -9,137 +10,86 @@ tags:
   - image-text
   - embeddings
   - vision
+  - apple-silicon
 ---
 
-# SigLIP2 base-patch16-256 — CoreML (fp16 + int8-embedding text variant)
+# SigLIP2 base-patch16-256 — Core ML
 
-CoreML `.mlpackage` conversion of
-[`google/siglip2-base-patch16-256`](https://huggingface.co/google/siglip2-base-patch16-256)
-for on-device (macOS/Apple Silicon) text/image embedding — built for
-[Alarian](https://alarian.ai)'s semantic frame search.
+Core ML (`.mlpackage`) conversion of [`google/siglip2-base-patch16-256`](https://huggingface.co/google/siglip2-base-patch16-256) for on-device text and image embedding on Apple platforms. The image and text towers ship as separate models; both produce 768-dimensional, L2-normalized embeddings in a shared space, so image↔text similarity is a single dot product.
 
-**Provenance**: converted directly from the Hugging Face `transformers`
-reference implementation and `coremltools`, by nodevorg. Not affiliated with
-Google. The conversion script is public:
-[`tools/siglip2-coreml/convert.py`](https://github.com/nodevorg/studio) (in
-the `nodevorg/studio` repo, kept uncommitted pending review — ask if you want
-a copy). This project intentionally never looked at or referenced any
-third-party GPLv3 conversion script; it was written from the HF
-`transformers` SigLIP2 modeling code and `coremltools` docs only.
+Converted and maintained by [nodevorg](https://huggingface.co/nodevorg) (used in production by [Alarian](https://alarian.ai) for semantic frame search). Not affiliated with Google.
 
-**License**: apache-2.0, inherited from the base model. Same terms apply
-here — free for commercial and closed-source use.
+## Files
 
-## What's in this repo
+| file | size | sha256 |
+|---|---|---|
+| `ImageEncoder.mlpackage.zip` | 163 MiB | `406938456c8f8e91f01632cb74131fdfb993064d8d0bc8f6fbbd142709f1ba66` |
+| `TextEncoder.int8emb.mlpackage.zip` — **recommended** | 323 MiB | `11471872101a6ca88dd51ed9de668b728ae9d09c0f853aeac8a6cc96f1c109d6` |
+| `TextEncoder.mlpackage.zip` | 506 MiB | `4f31f38a1729a0044f0ffc95a940dc1a6a2fdcd5fcf78a03cbb662bada6cbafb` |
+| `tokenizer.json` | 33 MiB | `cb9140fae3ac5122c972d37adf83e1248471a38147ad76f8215c8872c6fd8322` |
+| `tokenizer_config.json` | — | `14afe629fe4959b9e0d51e1852b8d9f7ad074f90a1a7125a4fcdd17f06e78fc8` |
+| `special_tokens_map.json` | — | `baec30ea10906f16adb8c18af7a34023002c1746542612b8b41c9f09e1351351` |
 
-| file | what it is |
-|---|---|
-| `ImageEncoder.mlpackage.zip` | image tower → 768-dim embedding |
-| `TextEncoder.int8emb.mlpackage.zip` | **recommended** — text tower → 768-dim embedding; token-embedding table int8, everything else fp16 (36% smaller, parity ≥0.9996) |
-| `TextEncoder.mlpackage.zip` | text tower → 768-dim embedding, full fp16 (use only if you need max precision and don't care about download size) |
-| `tokenizer.json` | Gemma BPE tokenizer (256k vocab), verbatim from the source repo |
-| `tokenizer_config.json` | tokenizer config (padding_side=right, do_lower_case=true), verbatim |
-| `special_tokens_map.json` | special tokens, verbatim |
-
-**For app downloads, use `TextEncoder.int8emb.mlpackage.zip`.** Same inputs/
-outputs as `TextEncoder.mlpackage` (see below) — it's a drop-in replacement,
-323 MiB instead of 506 MiB, with only the 256k-row token-embedding table
-quantized to int8 (linear_symmetric, per-channel/per-token-row scale);
-attention and MLP weights stay fp16. Measured parity cost: worst-case cosine
-similarity 0.9996 vs the fp16 model's 0.999999 — both pass the >0.99 gate by
-a wide margin, and text↔image rankings are unaffected (see the parity
-section below).
-
-### A note on architecture
-
-`google/siglip2-base-patch16-256`'s own `config.json` self-describes as
-`model_type: siglip` (not `siglip2`) — at this fixed-256-resolution, non-NaFlex
-tier, the SigLIP2-trained weights are architecturally identical to SigLIP v1
-(`transformers.SiglipModel`); only the weights and the tokenizer (Gemma BPE,
-256k vocab, vs. SigLIP v1's 32k SentencePiece) differ. `AutoModel.from_pretrained`
-resolves this checkpoint to `SiglipModel`, and that's what was converted.
-
-## Inputs / outputs
-
-### ImageEncoder.mlpackage
-
-- **Input**: `pixel_values` — a CoreML `ImageType`, RGB, 256×256. The exact
-  SigLIP preprocessing (`SiglipImageProcessor` defaults for this checkpoint:
-  `rescale_factor=1/255`, `image_mean=[.5,.5,.5]`, `image_std=[.5,.5,.5]`) is
-  **baked into the model** as `scale=1/127.5, bias=[-1,-1,-1]` — i.e. the
-  model itself maps raw RGB `[0,255]` to `[-1, 1]`. Just resize/squash your
-  image to 256×256 and hand it the raw RGB pixels; no separate normalization
-  step needed on the caller side. (Aspect-ratio squash, not center-crop — the
-  model saw squashed images in training.)
-- **Output**: `image_embedding` — 768-dim float16, **already L2-normalized**.
-
-### TextEncoder.mlpackage
-
-- **Input**: `input_ids` — 1×64 **int32** token ids. No `attention_mask`
-  input: SigLIP's text tower trains with unmasked bidirectional attention
-  over the full fixed-length 64-token sequence and pools the hidden state at
-  the *last* sequence position (`SiglipTextModel.forward`, not an EOS-aware
-  pool) — passing a mask would deviate from training. Tokenize with the
-  included `tokenizer.json`/`tokenizer_config.json` using
-  `padding="max_length", max_length=64`, and **lowercase the input text**
-  (`do_lower_case=True` in the tokenizer config; the tokenizer will also do
-  this itself if you feed it through its normalizer, but be aware if you
-  hand-roll token ids).
-- **Output**: `text_embedding` — 768-dim float16, **already L2-normalized**.
-
-`TextEncoder.int8emb.mlpackage` has the exact same input/output names,
-shapes, and dtypes — only the internal token-embedding weight is quantized,
-which is invisible from the outside.
-
-Both encoders normalize their pooled output inside the graph, so cosine
-similarity between an image and text embedding is just a dot product.
+`TextEncoder.int8emb` is a drop-in replacement for the fp16 `TextEncoder` — identical input/output names, shapes, and dtypes — with the token-embedding table quantized to int8 (36% smaller download, worst-case parity 0.9996; details below). Use the full-fp16 text encoder only if you need maximum precision.
 
-## Conversion details
+The tokenizer files are copied verbatim from the source checkpoint (Gemma BPE, 256k vocabulary).
 
-- `minimum_deployment_target`: macOS 15
-- `compute_precision`: `ct.precision.FLOAT16`
-- `convert_to`: `mlprogram`
-- Traced via `torch.jit.trace` on thin wrapper modules around
-  `SiglipModel.vision_model` / `.text_model` (not the full contrastive model —
-  each `.mlpackage` is just the one branch).
-- Pinned toolchain (see the sha256 table below for exact artifact hashes):
-  `torch==2.7.0`, `transformers==5.14.1`, `coremltools==9.0`, `numpy==2.3.5`.
-  **Gotcha**: `numpy>=2.4.0` breaks `coremltools==9.0`'s PyTorch frontend with
-  `TypeError: only 0-dimensional arrays can be converted to Python scalars`
-  on any model using `torch.nn.MultiheadAttention` (SigLIP's vision pooling
-  head hits this) — confirmed via
-  [apple/coremltools#2633](https://github.com/apple/coremltools/issues/2633);
-  pin `numpy<2.4.0` until a coremltools release ships the fix from
-  [#2632](https://github.com/apple/coremltools/pull/2632).
-
-### int8-embedding quantization (`TextEncoder.int8emb.mlpackage`)
-
-Built by post-training-quantizing the token-embedding weight of the already-
-converted fp16 `TextEncoder.mlpackage`, via `coremltools.optimize.coreml`:
-`OpLinearQuantizerConfig(mode="linear_symmetric", dtype="int8",
-granularity="per_channel")` applied to *only* the
-`text_model.embeddings.token_embedding` weight (256000×768) by op name —
-`OptimizationConfig(global_config=None, op_name_configs={...})` so every
-other weight (attention, MLP) is left untouched at fp16. This is a `gather`
-weight: each forward pass reads exactly one row per token, so quantization
-error doesn't compound the way it would through a matmul chain, which is why
-it tolerates int8 so well.
-
-A whole-model int8 pass (`global_config` instead of a single `op_name_config`)
-was also built and measured for comparison: smaller (≈270 MB unzipped vs
-≈352 MB for embedding-only) but with visibly worse worst-case parity (0.9964
-vs 0.9996 cosine). Embedding-only was shipped since it's the clearly-better
-size/parity trade-off — the extra ~30% file size buys back a large chunk of
-the precision the whole-model pass gives up, for a component (search
-ranking) where score margins matter.
-
-## Parity (PyTorch fp32 reference vs. converted CoreML, cosine similarity)
-
-4 test images (solid red / green / blue squares + a synthetic sunset
-gradient) × 4 test strings, L2-normalized before comparison. Threshold: > 0.99
-per item.
-
-| item | cosine(pytorch, coreml) |
+## Model interface
+
+### ImageEncoder
+
+- **Input** — `pixel_values`: Core ML `ImageType`, RGB, 256×256. Preprocessing is baked into the model (`scale=1/127.5`, `bias=[-1,-1,-1]`, matching the checkpoint's `SiglipImageProcessor` configuration): resize to 256×256 and pass raw `[0, 255]` RGB — no caller-side normalization. Resize with an aspect-ratio squash rather than a center crop; that matches training.
+- **Output** — `image_embedding`: 768-dim float16, L2-normalized.
+
+### TextEncoder (both variants)
+
+- **Input** — `input_ids`: 1×64 int32 token ids. Tokenize with the included tokenizer using `padding="max_length"`, `max_length=64`, lowercased input (`do_lower_case=true`). There is deliberately **no `attention_mask` input**: SigLIP's text tower trains with unmasked bidirectional attention over the fixed 64-token sequence and pools the last sequence position, so a mask would deviate from training.
+- **Output** — `text_embedding`: 768-dim float16, L2-normalized.
+
+Because both outputs are normalized in-graph, cosine similarity is `dot(image_embedding, text_embedding)`.
+
+## Usage (Swift)
+
+```swift
+import CoreML
+
+let image = try MLModel(contentsOf: compiledImageEncoderURL)
+let text  = try MLModel(contentsOf: compiledTextEncoderURL)
+
+// Image: 256×256 RGB CVPixelBuffer, raw [0, 255] values
+let imageOut = try image.prediction(
+    from: MLDictionaryFeatureProvider(dictionary: ["pixel_values": pixelBuffer]))
+let imageEmbedding = imageOut.featureValue(for: "image_embedding")!.multiArrayValue!
+
+// Text: 64 int32 token ids (e.g. via swift-transformers' Tokenizers module,
+// loading the tokenizer.json from this repo), lowercased, padded to length 64
+let ids = try MLMultiArray(shape: [1, 64], dataType: .int32) // fill with token ids
+let textOut = try text.prediction(
+    from: MLDictionaryFeatureProvider(dictionary: ["input_ids": ids]))
+let textEmbedding = textOut.featureValue(for: "text_embedding")!.multiArrayValue!
+
+// Similarity = dot product (embeddings are already L2-normalized)
+```
+
+## Architecture note
+
+The source checkpoint's `config.json` declares `model_type: siglip`, not `siglip2`. At this fixed-resolution, non-NaFlex tier, the SigLIP2-trained weights are architecturally identical to SigLIP v1 — `transformers.AutoModel` resolves the checkpoint to `SiglipModel`, and that is what was converted. Relative to SigLIP v1, only the weights and the tokenizer (Gemma BPE 256k, vs. 32k SentencePiece) differ.
+
+Most of the text encoder's size is the 256k-row token-embedding table (256000 × 768 × 2 bytes ≈ 375 MiB at fp16), which is why quantizing that single tensor recovers most of the possible size savings.
+
+## Quantized text encoder
+
+`TextEncoder.int8emb.mlpackage` is produced by post-training quantization of the converted fp16 model via `coremltools.optimize.coreml`, applying `OpLinearQuantizerConfig(mode="linear_symmetric", dtype="int8", granularity="per_channel")` to **only** the `text_model.embeddings.token_embedding` weight (op-name-targeted); attention and MLP weights remain fp16. The embedding table is a gather weight — each forward pass reads one row per token, so quantization error does not compound through a matmul chain.
+
+A whole-model int8 variant was also evaluated: ≈270 MiB unzipped (vs. ≈352 MiB for embedding-only) but with worst-case parity of 0.9964 vs. 0.9996. The embedding-only variant was published as the better size/precision trade-off for retrieval workloads, where score margins matter.
+
+## Parity vs. PyTorch reference
+
+Cosine similarity between the PyTorch fp32 reference (`transformers`) and the converted Core ML models, on a fixture of 4 images (solid red/green/blue, synthetic sunset gradient) × 4 texts. Acceptance gate: > 0.99 per item, with unchanged text↔image rankings.
+
+### fp16 encoders
+
+| item | cosine |
 |---|---|
 | image: solid_red | 0.9999988 |
 | image: solid_green | 0.9999992 |
@@ -150,53 +100,28 @@ per item.
 | text: "a solid blue color" | 0.9999995 |
 | text: "a green field" | 0.9999996 |
 
-**Worst per-item cosine similarity (fp16 text encoder): 0.9999987** — well
-above the 0.99 threshold.
+Worst case **0.9999987**. Text↔image rankings match the PyTorch reference exactly across all 4×4 pairs.
 
-Text↔image ranking also matches exactly between PyTorch and CoreML across
-all 4×4 pairs (e.g. "a red square" ranks `solid_red` highest in both
-backends; "a photo of a sunset over the ocean" ranks `gradient_sky` highest
-in both).
+### int8-embedding text encoder
 
-### int8-embedding text encoder parity
-
-Same fixture (4 images × 4 strings), same gate (>0.99 per item, unchanged
-rankings), reference is still PyTorch fp32:
-
-| text | cosine(pytorch, int8emb coreml) |
+| text | cosine |
 |---|---|
 | "a red square" | 0.99978 |
 | "a photo of a sunset over the ocean" | 0.99976 |
 | "a solid blue color" | 0.99962 |
 | "a green field" | 0.99975 |
 
-**Worst per-item cosine similarity (int8-embedding text encoder): 0.99962**
-— passes the 0.99 gate with a wide margin. Text↔image rankings are identical
-to both the PyTorch reference and the fp16 CoreML model across all 4 queries
-(each color string still top-matches its own solid-color image; the sunset
-string still top-matches the gradient).
+Worst case **0.99962**; rankings identical to both the PyTorch reference and the fp16 model.
 
-## sha256
+## Conversion details
 
-| file | sha256 |
-|---|---|
-| `ImageEncoder.mlpackage.zip` | `406938456c8f8e91f01632cb74131fdfb993064d8d0bc8f6fbbd142709f1ba66` |
-| `TextEncoder.int8emb.mlpackage.zip` | `11471872101a6ca88dd51ed9de668b728ae9d09c0f853aeac8a6cc96f1c109d6` |
-| `TextEncoder.mlpackage.zip` | `4f31f38a1729a0044f0ffc95a940dc1a6a2fdcd5fcf78a03cbb662bada6cbafb` |
-| `tokenizer.json` | `cb9140fae3ac5122c972d37adf83e1248471a38147ad76f8215c8872c6fd8322` |
-| `tokenizer_config.json` | `14afe629fe4959b9e0d51e1852b8d9f7ad074f90a1a7125a4fcdd17f06e78fc8` |
-| `special_tokens_map.json` | `baec30ea10906f16adb8c18af7a34023002c1746542612b8b41c9f09e1351351` |
+Converted from the Hugging Face `transformers` reference implementation. Each tower was traced independently (`torch.jit.trace` on thin wrappers around `SiglipModel.vision_model` / `.text_model`) and converted with:
 
-## File sizes
+- `convert_to="mlprogram"`, `compute_precision=FLOAT16`
+- `minimum_deployment_target`: macOS 15
 
-| file | size |
-|---|---|
-| `ImageEncoder.mlpackage.zip` | 163 MiB |
-| `TextEncoder.int8emb.mlpackage.zip` | 323 MiB (**36% smaller** than the fp16 text encoder) |
-| `TextEncoder.mlpackage.zip` | 506 MiB |
-| `tokenizer.json` | 33 MiB |
-
-The text encoder is large mostly because of the Gemma tokenizer's 256k-entry
-vocabulary embedding table (256000 × 768 × 2 bytes ≈ 375 MiB alone, fp16;
-≈188 MiB at int8, which is most of where the int8-embedding variant's size
-savings come from).
+Pinned toolchain: `torch==2.7.0`, `transformers==5.14.1`, `coremltools==9.0`, `numpy==2.3.5`. Note that `numpy>=2.4.0` breaks the `coremltools==9.0` PyTorch frontend for models using `torch.nn.MultiheadAttention` (SigLIP's vision pooling head) — see [apple/coremltools#2633](https://github.com/apple/coremltools/issues/2633); pin `numpy<2.4.0` until the fix ships in a release.
+
+## License
+
+Apache-2.0, inherited from the base model. Free for commercial and closed-source use.