Feature Extraction
sentence-transformers
Safetensors
embedding_gemma2
multi-vector
late-interaction
colbert
meanmaxsim
multimodal
text
image
audio
embeddinggemma-2
matryoshka
Instructions to use JacobLinCool/embeddinggemma-2-multimodal-late-interaction with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use JacobLinCool/embeddinggemma-2-multimodal-late-interaction with sentence-transformers:
from sentence_transformers import MultiVectorEncoder model = MultiVectorEncoder("JacobLinCool/embeddinggemma-2-multimodal-late-interaction") queries = ["Which planet is known as the Red Planet?"] documents = [ "Venus is often called Earth's twin because of its similar size and proximity.", "Mars, known for its reddish appearance, is often referred to as the Red Planet.", "Jupiter, the largest planet in our solar system, has a prominent red spot.", ] query_embeddings = model.encode_query(queries) document_embeddings = model.encode_document(documents) similarities = model.similarity(query_embeddings, document_embeddings) print(similarities) - Notebooks
- Google Colab
- Kaggle
Update to G4 Scale-Up checkpoint (B4-LI256: 87.40% Flickr8k R@1, 53.67% Clotho R@10, 81.45% SciFact nDCG@10)
2f79745 verified |
Download README.md from JacobLinCool/embeddinggemma-2-multimodal-late-interaction: direct link, hf CLI and curl.
- Browser
- Download file 13 kB
-
https://huggingface.co/JacobLinCool/embeddinggemma-2-multimodal-late-interaction/resolve/main/README.md
- Command line
-
hf download hf://JacobLinCool/embeddinggemma-2-multimodal-late-interaction/README.md
-
curl -L -o README.md https://huggingface.co/JacobLinCool/embeddinggemma-2-multimodal-late-interaction/resolve/main/README.md
13 kB
| library_name: sentence-transformers | |
| pipeline_tag: feature-extraction | |
| base_model: google/embeddinggemma-2 | |
| license: other | |
| license_name: apache-2.0-with-upstream-dataset-attribution | |
| license_link: https://huggingface.co/google/embeddinggemma-2 | |
| tags: | |
| - sentence-transformers | |
| - multi-vector | |
| - late-interaction | |
| - colbert | |
| - meanmaxsim | |
| - multimodal | |
| - text | |
| - image | |
| - audio | |
| - embeddinggemma-2 | |
| - matryoshka | |
| datasets: | |
| - sentence-transformers/msmarco-bm25 | |
| - mteb/nfcorpus | |
| - jxie/flickr8k | |
| - gijs/clotho | |
| - mteb/scifact | |
| # EmbeddingGemma 2 Multimodal Late Interaction (`B4-LI256`) | |
| **`JacobLinCool/embeddinggemma-2-multimodal-late-interaction`** adapts [`google/embeddinggemma-2`](https://huggingface.co/google/embeddinggemma-2) into a native **`sentence_transformers.MultiVectorEncoder`** supporting **Text, Image, and 16 kHz Audio** token-level **Late Interaction (`Mean-MaxSim`, 256d per token with native 128d Matryoshka prefix truncation)** without requiring `trust_remote_code` or custom wrapper code. | |
| --- | |
| ## 1. Key Highlights (`B4` G4 Scale-Up Checkpoint) | |
| - **Cross-Modal Text-to-Image (`Flickr8k T→I`)**: Reaches **`87.40% R@1`** (`+3.00 pp` over zero-shot `B0-256 = 84.40%`, `+2.80 pp` over L4 `B3-LI256 = 84.60%`, and surpasses the 3× wider `768d` base model `B0-768 = 86.80%`). With **4× UPGMA Hierarchical Cosine Token Pooling (`B4-LI256-Pool4x`)**, accuracy improves further to **`88.20% R@1` / `99.60% R@10` / `94.23% nDCG@10`** while cutting index storage by **75%** (`65.0 MiB → 16.3 MiB`). | |
| - **Cross-Modal Text-to-Audio (`Clotho T→A`, 15–30s 16 kHz clips)**: Reaches **`14.33% R@1`** (`2.7×` zero-shot `B0-256 = 5.33%`) and **`53.67% R@10`** (`+22.00 pp` over `B0-256 = 31.67%`, `+17.33 pp` over L4 `B3-LI256 = 36.33%`), while completely eliminating document length bias (`Spearman r = +0.0482, p = 0.4056`). | |
| - **Scientific Document Retrieval (`BEIR SciFact T→T`, 5,183 corpus docs)**: Reaches **`81.45% nDCG@10` / `91.76% R@10`** (`+4.36 pp` higher `nDCG@10` than the exact same checkpoint's 256d single-vector pooling `B4-D256 = 77.09%`), and **`81.94% nDCG@10` / `92.16% R@10`** under **4× Token Pooling (`B4-LI256-Pool4x`)**. | |
| - **Multi-Dimensional Token-Level MRL (`128d` & `256d`)**: Explicitly supervised at both `128d` and `256d` per token. Truncating each output token vector `vec[:, :128]` and L2-renormalizing + applying `4×` Token Pooling (`B4-LI128-Pool4x`) reduces index storage by **87.5% (`8×` compression)** while achieving **`87.00%` (`T→I R@1`)**, **`14.67% / 53.67%` (`T→A R@1 / R@10`)**, and **`81.48%` (`SciFact nDCG@10`)**—outperforming uncompressed `B3-LI256` across all three modalities at `1/8th` the index footprint. | |
| --- | |
| ## 2. Architecture & Training Methodology | |
| - **Backbone**: `google/embeddinggemma-2` (`EmbeddingGemma2Model`, Gemma 4 300M multimodal encoder with native SigLIP-style vision tower, Conformer-style audio tower, and 24-layer shared text Transformer + pretrained 512→768 Matryoshka projection `language_model.embedding_projection`). | |
| - **Native `MultiVectorEncoder` Pipeline (`modules.json`)**: | |
| 1. `0_Transformer`: Merged `EmbeddingGemma2Model` backbone (`module_output_name="token_embeddings"`, outputting post-MRL `768d` per-token states in `bfloat16`) with a modality-aware Jinja chat template that applies official query/document prompts **only** to text inputs while keeping image (`~266` tokens) and 16 kHz audio (`~25.1` tokens/sec) inputs unprompted. | |
| 2. `1_Dense`: Frozen prefix-selector `Linear(768 -> 256, bias=False)` initialized to `W = [I_256 | 0_256x512]`, preserving the first 256 Matryoshka channels of each token without adding random projection weights. | |
| 3. `2_MultiVectorMask`: Masks padding tokens across variable-length batches. | |
| 4. `3_Normalize`: Applies token-wise L2 normalization (`module_input_name="token_embeddings"`). | |
| - **Similarity Function (`similarity_fn_name="meanmaxsim"`)**: | |
| $$S_{\text{LI}}(q, x) = \frac{1}{|Z_{d}(q)|} \sum_{i=1}^{|Z_{d}(q)|} \max_{1 \le j \le |Z_{d}(x)|} \langle Z_{d}(q)_i, Z_{d}(x)_j \rangle, \quad d \in \{128, 256\}$$ | |
| - **G4 Scale-Up Joint LoRA Adaptation (`B4`)**: | |
| - Full-network `"full_shared"` LoRA (`r=32, alpha=64, dropout=0.05`) attached to all **168 linear layers** (`q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj`) across the 24 shared text Transformer layers (`19,922,944` trainable parameters = `2.6041%`), while keeping `vision_tower`, `audio_tower`, `embed_vision`, `embed_audio`, and `embedding_projection` frozen, then merged into base weights for zero-overhead inference. | |
| - Trained for `600` optimizer updates (`200` updates/modality, `batch_pairs=16`, `2` denoised hard negatives per query -> `Q=16, D=48` rectangular contrastive matrix per step) on an **NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB GDDR7, `bfloat16`)** over **`79,405` multimodal query–asset pairs** (`20,000` text pairs from MS MARCO + SciFact + NFCorpus, `35,000` Flickr8k caption–image pairs, `24,405` Clotho caption–audio pairs; `< 1 Epoch` single-pass regime) with Multi-Dimensional Token-Level MRL and 768d Teacher KL Self-Distillation: | |
| $$\mathcal{L}_{\text{G4}} = \frac{1}{3}\sum_{d \in \{768, 256, 128\}} \mathcal{L}_{\text{InfoNCE}}^{\text{dense-}d} + \frac{1}{2}\sum_{d \in \{256, 128\}} \mathcal{L}_{\text{InfoNCE}}^{\text{LI-}d} + 0.15 \cdot \tau^2 \operatorname{KL}\left(\operatorname{softmax}(S_{\text{LI-256}}/\tau) \,\|\, \operatorname{softmax}(\operatorname{sg}(S_{\text{dense-768}})/\tau)\right)$$ | |
| --- | |
| ## 3. Quickstart (Text, Image & 16 kHz Audio + 128d MRL Truncation) | |
| > **Precision Requirement**: Always run `EmbeddingGemma 2` in **`bfloat16`** or **`float32`**. Never cast the model to `float16` (`fp16`), which causes activation overflow (`NaN`/`Inf`). | |
| ```bash | |
| pip install -U "transformers>=5.20.0" "sentence-transformers>=6.0.0" soundfile soxr pillow torch | |
| ``` | |
| ```python | |
| import numpy as np | |
| import torch | |
| import torch.nn.functional as F | |
| from PIL import Image | |
| from sentence_transformers import MultiVectorEncoder | |
| REPO_ID = "JacobLinCool/embeddinggemma-2-multimodal-late-interaction" | |
| model = MultiVectorEncoder(REPO_ID, trust_remote_code=False) | |
| # 1. Encode a text query -> shape [L_q, 256] | |
| query = model.encode_query("A dog barking outdoors") | |
| # 2. Encode candidate documents across Text, Image, and 16 kHz mono Audio | |
| text_doc = model.encode_document("A dog makes a barking noise in the park") | |
| image_doc = model.encode_document(Image.new("RGB", (224, 224), "white")) | |
| waveform = np.zeros(16000 * 3, dtype=np.float32) # 3s 16kHz mono float32 audio | |
| audio_doc = model.encode_document({"array": waveform, "sampling_rate": 16000}) | |
| # 3. Compute asymmetric 256d Mean-MaxSim similarity | |
| for modality, doc in [("text", text_doc), ("image", image_doc), ("audio", audio_doc)]: | |
| score = model.similarity([query], [doc]) | |
| print(f"256d T->{modality} MeanMaxSim score: {score.item():.4f} (doc tokens: {doc.shape[0]})") | |
| # 4. Optional 128d Token-Level Matryoshka Prefix Truncation (2x smaller vectors) | |
| q_128 = F.normalize(torch.as_tensor(query)[:, :128], p=2, dim=-1) | |
| d_128 = F.normalize(torch.as_tensor(text_doc)[:, :128], p=2, dim=-1) | |
| score_128 = (q_128 @ d_128.T).max(dim=-1).values.mean().item() | |
| print(f"128d MRL T->text MeanMaxSim score: {score_128:.4f}") | |
| ``` | |
| --- | |
| ## 4. Experimental Benchmark Results (`seed=42`, Held-Out Evaluation) | |
| All models were evaluated on identical held-out candidate sets (`0%` overlap with training data): | |
| - **Flickr8k (`T→I`)**: 500 test queries (`caption_0`) against 500 test images (candidate SHA-256: `e064acc86e2ce4fb...`). | |
| - **Clotho (`T→A`)**: 300 evaluation queries (`caption_1`) against 300 full-duration 16 kHz mono audio clips (candidate SHA-256: `cb520321dd6bb7e6...`). | |
| - **BEIR SciFact (`T→T`)**: 300 test queries against the full 5,183-document scientific corpus (candidate SHA-256: `9948d896b35ae4e7...`). | |
| ### 4.1 Full Progression (`B0` Zero-Shot → `B2/B3` L4 `200` steps → `B4` G4 `600` steps) | |
| | Method | Flickr8k `T→I` R@1 (%) | Flickr8k `T→I` R@10 (%) | Clotho `T→A` R@1 (%) | Clotho `T→A` R@10 (%) | SciFact `T→T` nDCG@10 (%) | SciFact `T→T` R@10 (%) | `T→I` Δ vs `B0-256` (pp) | `T→A` Δ vs `B0-256` (pp) | `T→T` Δ vs `B0-256` (pp) | | |
| | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | |
| | **B0-768** | 86.80 | 99.60 | 6.00 | 33.67 | 83.67 | 92.24 | +2.40 [+0.80, +4.40]* | +0.67 [-1.00, +2.67] | +1.52 [+0.57, +2.52]* | | |
| | **B0-256** | 84.40 | 99.40 | 5.33 | 31.67 | 82.15 | 90.12 | +0.00 [+0.00, +0.00] | +0.00 [+0.00, +0.00] | +0.00 [+0.00, +0.00] | | |
| | **B1-LI256** | 83.00 | 99.60 | 5.33 | 23.33 | 78.59 | 89.34 | -1.40 [-4.20, +1.40] | +0.00 [-2.67, +2.67] | -3.56 [-6.76, -0.46]* | | |
| | **B2-D768** | 86.60 | 99.20 | 8.33 | 40.00 | 82.30 | 91.02 | +2.20 [+0.20, +4.40]* | +3.00 [+0.00, +6.33] | +0.16 [-1.10, +1.39] | | |
| | **B2-D256** | 85.60 | 99.20 | 7.33 | 41.67 | 80.70 | 89.06 | +1.20 [-0.80, +3.40] | +2.00 [-0.67, +5.00] | -1.44 [-2.75, -0.29]* | | |
| | **B3-D768** | 86.40 | 98.80 | 7.67 | 41.67 | 81.37 | 90.69 | +2.00 [-0.20, +4.20] | +2.33 [-0.33, +5.33] | -0.78 [-2.29, +0.65] | | |
| | **B3-D256** | 85.20 | 99.00 | 7.00 | 42.33 | 80.33 | 89.06 | +0.80 [-1.40, +3.00] | +1.67 [-1.00, +4.67] | -1.81 [-3.15, -0.48]* | | |
| | **B3-LI256** | 84.60 | 99.40 | 9.00 | 36.33 | 80.79 | 90.79 | +0.20 [-2.20, +2.60] | +3.67 [+0.67, +6.67]* | -1.36 [-4.01, +1.47] | | |
| | **B3-Rerank50** | 84.60 | 99.40 | 9.00 | 36.33 | 80.18 | 90.12 | +0.20 [-2.20, +2.60] | +3.67 [+0.67, +6.67]* | -1.97 [-4.42, +0.63] | | |
| | **B4-D768** | 86.80 | 99.20 | 15.33 | 52.00 | 79.29 | 90.24 | +2.40 [+0.00, +4.80] | +10.00 [+6.00, +14.00]* | -2.86 [-5.41, -0.48]* | | |
| | **B4-D256** | 86.00 | 99.00 | 14.67 | 52.33 | 77.09 | 88.18 | +1.60 [-0.40, +3.60] | +9.33 [+5.33, +13.33]* | -5.05 [-7.38, -2.83]* | | |
| | **B4-D128** | 84.40 | 99.00 | 14.67 | 52.33 | 74.69 | 86.12 | +0.00 [-2.60, +2.60] | +9.33 [+5.99, +13.33]* | -7.46 [-10.32, -4.69]* | | |
| | **B4-LI256** | 87.40 | 99.40 | 14.33 | 53.67 | 81.45 | 91.76 | +3.00 [+1.00, +5.20]* | +9.00 [+5.33, +13.00]* | -0.69 [-2.94, +1.82] | | |
| | **B4-LI128** | 87.20 | 99.40 | 14.00 | 53.33 | 81.05 | 91.59 | +2.80 [+0.40, +5.40]* | +8.67 [+5.00, +12.67]* | -1.10 [-3.59, +1.52] | | |
| | **B4-Rerank50** | 87.40 | 99.40 | 14.33 | 53.67 | 79.69 | 89.76 | +3.00 [+1.00, +5.20]* | +9.00 [+5.33, +13.00]* | -2.46 [-4.79, -0.14]* | | |
| ### 4.2 UPGMA Hierarchical Cosine Document Token Pooling (`Pool1x / Pool2x / Pool4x`) | |
| Applying parameter-free agglomerative cosine clustering to document token vectors at index time reduces storage by `50%–87.5%` while acting as a **spatial/lexical denoiser** that merges redundant background image patches and stopwords into clean semantic centroids: | |
| | Method & Pool Factor | `T→I` R@1 (%) | `T→I` nDCG@10 (%) | `T→I` Index Size | `T→A` R@1 (%) | `T→A` R@10 (%) | `T→A` Index Size | `T→T` nDCG@10 (%) | `T→T` R@10 (%) | `T→T` Index Size | | |
| | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | |
| | **B4-LI256-Pool1x** | 87.40 | 93.90 | 65.02 MiB | 14.33 | 53.00 | 83.20 MiB | 81.45 | 91.76 | 593.65 MiB | | |
| | **B4-LI256-Pool2x** | 87.40 | 93.93 | 32.51 MiB | 14.33 | 52.67 | 41.64 MiB | 81.55 | 91.69 | 297.07 MiB | | |
| | **B4-LI256-Pool4x** | 88.20 | 94.23 | 16.30 MiB | 13.33 | 52.00 | 20.85 MiB | 81.94 | 92.16 | 148.77 MiB | | |
| | **B4-LI128-Pool1x** | 87.20 | 93.90 | 32.51 MiB | 14.33 | 53.33 | 41.60 MiB | 81.05 | 91.59 | 296.82 MiB | | |
| | **B4-LI128-Pool2x** | 87.40 | 93.87 | 16.26 MiB | 15.33 | 53.33 | 20.82 MiB | 81.34 | 91.66 | 148.53 MiB | | |
| | **B4-LI128-Pool4x** | 87.00 | 93.77 | 8.15 MiB | 14.67 | 53.67 | 10.43 MiB | 81.48 | 91.37 | 74.38 MiB | | |
| --- | |
| ## 5. Scope Boundaries, Caveats & Licensing | |
| 1. **Matryoshka Prefix Caveat**: The 256d and 128d per-token representations select the leading channels of `google/embeddinggemma-2`'s pretrained 768d MRL projection (`language_model.embedding_projection`) and are explicitly supervised at the token level via `compute_g4_joint_mrl_loss`. | |
| 2. **Evaluated Modalities**: Verified on `Text→Image`, `Text→Audio` (16 kHz mono), `Text→Text`, and diagnostic `Image→Text` / `Audio→Text`. Direct `Image↔Audio` and `Video` retrieval were not evaluated and are not claimed. | |
| 3. **Licensing & Attribution**: | |
| - Base model [`google/embeddinggemma-2`](https://huggingface.co/google/embeddinggemma-2) is licensed under **Apache-2.0**. | |
| - Fine-tuning data comprises `sentence-transformers/msmarco-bm25` (`16,000` pairs), `mteb/scifact` (`917` train pairs), `mteb/nfcorpus` (`3,083` train pairs), `jxie/flickr8k` (`7,000` training images = `35,000` pairs), and `gijs/clotho` (`4,881` development/validation audio clips = `24,405` pairs). Because upstream Flickr8k and Clotho datasets carry academic / non-commercial usage terms, users should respect upstream dataset licenses when deploying this checkpoint in commercial environments. No raw evaluation or training media files are redistributed in this repository. | |