Upload README.md with huggingface_hub

README.md

@@ -11,7 +11,7 @@ tags:
   - vision
 ---
 
-# SigLIP2 base-patch16-256 — CoreML (fp16)
+# SigLIP2 base-patch16-256 — CoreML (fp16 + int8-embedding text variant)
 
 CoreML `.mlpackage` conversion of
 [`google/siglip2-base-patch16-256`](https://huggingface.co/google/siglip2-base-patch16-256)
@@ -35,11 +35,21 @@ here — free for commercial and closed-source use.
 | file | what it is |
 |---|---|
 | `ImageEncoder.mlpackage.zip` | image tower → 768-dim embedding |
-| `TextEncoder.mlpackage.zip` | text tower → 768-dim embedding |
+| `TextEncoder.int8emb.mlpackage.zip` | **recommended** — text tower → 768-dim embedding; token-embedding table int8, everything else fp16 (36% smaller, parity ≥0.9996) |
+| `TextEncoder.mlpackage.zip` | text tower → 768-dim embedding, full fp16 (use only if you need max precision and don't care about download size) |
 | `tokenizer.json` | Gemma BPE tokenizer (256k vocab), verbatim from the source repo |
 | `tokenizer_config.json` | tokenizer config (padding_side=right, do_lower_case=true), verbatim |
 | `special_tokens_map.json` | special tokens, verbatim |
 
+**For app downloads, use `TextEncoder.int8emb.mlpackage.zip`.** Same inputs/
+outputs as `TextEncoder.mlpackage` (see below) — it's a drop-in replacement,
+323 MiB instead of 506 MiB, with only the 256k-row token-embedding table
+quantized to int8 (linear_symmetric, per-channel/per-token-row scale);
+attention and MLP weights stay fp16. Measured parity cost: worst-case cosine
+similarity 0.9996 vs the fp16 model's 0.999999 — both pass the >0.99 gate by
+a wide margin, and text↔image rankings are unaffected (see the parity
+section below).
+
 ### A note on architecture
 
 `google/siglip2-base-patch16-256`'s own `config.json` self-describes as
@@ -77,6 +87,10 @@ resolves this checkpoint to `SiglipModel`, and that's what was converted.
   hand-roll token ids).
 - **Output**: `text_embedding` — 768-dim float16, **already L2-normalized**.
 
+`TextEncoder.int8emb.mlpackage` has the exact same input/output names,
+shapes, and dtypes — only the internal token-embedding weight is quantized,
+which is invisible from the outside.
+
 Both encoders normalize their pooled output inside the graph, so cosine
 similarity between an image and text embedding is just a dot product.
 
@@ -98,7 +112,28 @@ similarity between an image and text embedding is just a dot product.
   pin `numpy<2.4.0` until a coremltools release ships the fix from
   [#2632](https://github.com/apple/coremltools/pull/2632).
 
-## Parity (PyTorch fp32 reference vs. converted CoreML fp16, cosine similarity)
+### int8-embedding quantization (`TextEncoder.int8emb.mlpackage`)
+
+Built by post-training-quantizing the token-embedding weight of the already-
+converted fp16 `TextEncoder.mlpackage`, via `coremltools.optimize.coreml`:
+`OpLinearQuantizerConfig(mode="linear_symmetric", dtype="int8",
+granularity="per_channel")` applied to *only* the
+`text_model.embeddings.token_embedding` weight (256000×768) by op name —
+`OptimizationConfig(global_config=None, op_name_configs={...})` so every
+other weight (attention, MLP) is left untouched at fp16. This is a `gather`
+weight: each forward pass reads exactly one row per token, so quantization
+error doesn't compound the way it would through a matmul chain, which is why
+it tolerates int8 so well.
+
+A whole-model int8 pass (`global_config` instead of a single `op_name_config`)
+was also built and measured for comparison: smaller (≈270 MB unzipped vs
+≈352 MB for embedding-only) but with visibly worse worst-case parity (0.9964
+vs 0.9996 cosine). Embedding-only was shipped since it's the clearly-better
+size/parity trade-off — the extra ~30% file size buys back a large chunk of
+the precision the whole-model pass gives up, for a component (search
+ranking) where score margins matter.
+
+## Parity (PyTorch fp32 reference vs. converted CoreML, cosine similarity)
 
 4 test images (solid red / green / blue squares + a synthetic sunset
 gradient) × 4 test strings, L2-normalized before comparison. Threshold: > 0.99
@@ -115,19 +150,38 @@ per item.
 | text: "a solid blue color" | 0.9999995 |
 | text: "a green field" | 0.9999996 |
 
-**Worst per-item cosine similarity: 0.9999987** — well above the 0.99
-threshold.
+**Worst per-item cosine similarity (fp16 text encoder): 0.9999987** — well
+above the 0.99 threshold.
 
 Text↔image ranking also matches exactly between PyTorch and CoreML across
 all 4×4 pairs (e.g. "a red square" ranks `solid_red` highest in both
 backends; "a photo of a sunset over the ocean" ranks `gradient_sky` highest
 in both).
 
+### int8-embedding text encoder parity
+
+Same fixture (4 images × 4 strings), same gate (>0.99 per item, unchanged
+rankings), reference is still PyTorch fp32:
+
+| text | cosine(pytorch, int8emb coreml) |
+|---|---|
+| "a red square" | 0.99978 |
+| "a photo of a sunset over the ocean" | 0.99976 |
+| "a solid blue color" | 0.99962 |
+| "a green field" | 0.99975 |
+
+**Worst per-item cosine similarity (int8-embedding text encoder): 0.99962**
+— passes the 0.99 gate with a wide margin. Text↔image rankings are identical
+to both the PyTorch reference and the fp16 CoreML model across all 4 queries
+(each color string still top-matches its own solid-color image; the sunset
+string still top-matches the gradient).
+
 ## sha256
 
 | file | sha256 |
 |---|---|
 | `ImageEncoder.mlpackage.zip` | `406938456c8f8e91f01632cb74131fdfb993064d8d0bc8f6fbbd142709f1ba66` |
+| `TextEncoder.int8emb.mlpackage.zip` | `11471872101a6ca88dd51ed9de668b728ae9d09c0f853aeac8a6cc96f1c109d6` |
 | `TextEncoder.mlpackage.zip` | `4f31f38a1729a0044f0ffc95a940dc1a6a2fdcd5fcf78a03cbb662bada6cbafb` |
 | `tokenizer.json` | `cb9140fae3ac5122c972d37adf83e1248471a38147ad76f8215c8872c6fd8322` |
 | `tokenizer_config.json` | `14afe629fe4959b9e0d51e1852b8d9f7ad074f90a1a7125a4fcdd17f06e78fc8` |
@@ -138,8 +192,11 @@ in both).
 | file | size |
 |---|---|
 | `ImageEncoder.mlpackage.zip` | 163 MiB |
+| `TextEncoder.int8emb.mlpackage.zip` | 323 MiB (**36% smaller** than the fp16 text encoder) |
 | `TextEncoder.mlpackage.zip` | 506 MiB |
 | `tokenizer.json` | 33 MiB |
 
 The text encoder is large mostly because of the Gemma tokenizer's 256k-entry
-vocabulary embedding table (256000 × 768 × 2 bytes ≈ 375 MiB alone, fp16).
+vocabulary embedding table (256000 × 768 × 2 bytes ≈ 375 MiB alone, fp16;
+≈188 MiB at int8, which is most of where the int8-embedding variant's size
+savings come from).