catalog-match-embeddinggemma2

A multimodal embedding model for product / catalog matching on Korean e-commerce listings. It maps a product title, a product image, or both into one 768-dimensional vector. Two listings of the same product land close together, so cosine similarity tells you whether they are the same item.

This model is a fine-tune of google/embeddinggemma-2 (revision 914f7f89142e33e77833254d9c9b90c3cef7303b). The LoRA adapter trained on the text tower is merged into the weights, so the model loads with plain sentence-transformers and needs no custom code.

Demo: jinwoo-cho/catalog-match-embeddinggemma2 Space

Usage

pip install -U "sentence-transformers>=6.1" "transformers>=5.19" torch torchvision pillow

from PIL import Image
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("jinwoo-cho/catalog-match-embeddinggemma2")

listings = [
    "휴대용 손소독제 젤타입, 1개, 50ml",                       # title only
    Image.open("product.jpg"),                                 # image only
    {"image": Image.open("product.jpg"), "text": "손소독제 젤 50ml"},  # image + title
]
embeddings = model.encode(listings)        # shape (3, 768), L2-normalized
print(model.similarity(embeddings, embeddings))

• The instruction prompt is applied automatically. The model was trained with one shared instruction on both sides of every pair. It is saved as the default prompt (query and document are the same text), so encode, encode_query and encode_document all give the same vector. Do not add the base model's task: ... | query: prefixes.
• Image first, then title. For an image + title input, pass a dict with the image key before the text key. The model was trained in that order; the reverse order gives a slightly different vector.
• Images can be a PIL.Image or a local file path. Each image becomes 70 vision soft tokens (max_soft_tokens: 70 in processor_config.json), the budget used in training.
• Dimensions. The output is 768-d. Training used Matryoshka loss at 768 / 384 / 192 / 96, so model.encode(..., truncate_dim=192) gives a shorter vector. Re-normalize after truncating if you compute cosine by hand. Compare vectors of the same dimension only.
• Text and image only. The audio tower of the base model is removed (audio_config: null); audio input is not supported. Video was not trained or tested.
• Precision. Weights are stored in bfloat16. Load in bfloat16 or float32; the base model card warns that float16 returns NaN.

Training

• Data: about 1.49M contrastive pairs. A pair is two catalog entries (title and/or image) from the same product group, drawn from a Korean e-commerce catalog.
• Objective: symmetric cached multiple-negatives ranking loss (in-batch negatives in both directions), with entries of the same product group excluded from each other's negatives, plus Matryoshka loss at 768 / 384 / 192 / 96.
• Adapter: LoRA rank 64, alpha 128, dropout 0.05, on the attention and MLP projections of the text tower. The vision tower is frozen. The adapter is merged into this checkpoint.
• Hardware: 4x NVIDIA H100.

Evaluation

Internal held-out results, on a set of 200k queries from product groups not seen in training. They are not a public benchmark.

Metric	Value
Retrieval accuracy (top-1 catalog entry is from the correct product group)	93.8%
Matching accuracy (group-level rule: mean of the top-2 similarities in a group ≥ 0.91)	76.3%

The matching rule scores a listing against several catalog entries of one product group. A single-pair cosine cut at 0.91 (as in the demo Space) is a simplification of that rule.

Limitations

• Quantity and spec numbers. Numbers cut both ways. Titles that differ only in a volume or size (50ml vs 500ml) can still get high similarity, while titles that differ only in a purchase count (120매 vs 140매) can get low similarity even when the listings are grouped as one product. Check numbers with a rule-based step if they matter.
• Low-information titles. Short or generic titles ("기본 티셔츠", "세트") do not carry enough information to tell products apart.
• Shared images. Sellers often reuse one representative image across variants (colors, sizes, bundle counts). Image-only matching cannot separate those variants.
• Trained on Korean product listings. Other languages and other domains are untested.

License and notices

This model is a derivative work of google/embeddinggemma-2 by Google DeepMind, which is licensed under the Apache License 2.0. This model is released under the same license; a copy is in LICENSE.

Modifications to the original work: the model weights were changed by LoRA fine-tuning on product-matching data and merging the adapter into the base weights; the audio tower was removed (config.json: audio_config: null); the image soft-token budget was set to 70 (processor_config.json); and the prompts in config_sentence_transformers.json were replaced with the training instruction.

Use of this model must follow the Gemma Prohibited Use Policy, as required by the base model card. "Gemma" and "EmbeddingGemma" are used here only to describe the origin of the model; this model is not made or endorsed by Google.