jrhode666 commited on
Commit
af3d543
·
verified ·
1 Parent(s): 1ca284a

EmbeddingGemma 2 explorer: any-to-any search, Matryoshka retrieval, similarity, embed API

Browse files
.gitattributes CHANGED
@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ assets/cats_couch.jpg filter=lfs diff=lfs merge=lfs -text
37
+ assets/jfk.wav filter=lfs diff=lfs merge=lfs -text
38
+ assets/mlk.flac filter=lfs diff=lfs merge=lfs -text
39
+ assets/mosque.jpg filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -1,13 +1,30 @@
1
  ---
2
- title: Embeddinggemma 2 Explorer
3
- emoji: 😻
4
- colorFrom: yellow
5
- colorTo: red
6
  sdk: gradio
7
  sdk_version: 6.30.0
8
- python_version: '3.12'
9
  app_file: app.py
10
- pinned: false
 
 
 
 
 
 
 
 
 
11
  ---
12
 
13
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ title: EmbeddingGemma 2 Explorer
3
+ emoji: 🧭
4
+ colorFrom: indigo
5
+ colorTo: purple
6
  sdk: gradio
7
  sdk_version: 6.30.0
 
8
  app_file: app.py
9
+ short_description: Any-to-any search with EmbeddingGemma 2
10
+ python_version: "3.12"
11
+ startup_duration_timeout: 30m
12
+ license: apache-2.0
13
+ models:
14
+ - google/embeddinggemma-2
15
+ tags:
16
+ - embeddings
17
+ - multimodal
18
+ - mcp-server
19
  ---
20
 
21
+ # EmbeddingGemma 2 Explorer
22
+
23
+ Interactive demo for [google/embeddinggemma-2](https://huggingface.co/google/embeddinggemma-2) — one 768-d vector space for text, images, audio and video.
24
+
25
+ - **Any-to-any search** — query with text, an image, audio or a video and rank a mixed corpus of text, images and audio.
26
+ - **Text retrieval & Matryoshka** — task-prefixed search (web search, QA, fact checking, code) with ranks shown at 768 / 512 / 256 / 128 dims.
27
+ - **Similarity matrix** — cross-lingual pairwise similarity heatmap.
28
+ - **Embed** — raw vectors, also exposed through the API and as an MCP server.
29
+
30
+ Runs in bfloat16 on ZeroGPU. Sample media comes from Hugging Face documentation datasets.
app.py ADDED
@@ -0,0 +1,621 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import spaces # must be imported before torch on ZeroGPU
2
+
3
+ import base64
4
+ import html
5
+ import io
6
+ import os
7
+ import time
8
+
9
+ import av
10
+ import gradio as gr
11
+ import librosa
12
+ import numpy as np
13
+ import torch
14
+ from PIL import Image
15
+ from sentence_transformers import SentenceTransformer
16
+
17
+ MODEL_ID = "google/embeddinggemma-2"
18
+ HERE = os.path.dirname(os.path.abspath(__file__))
19
+ ASSETS = os.path.join(HERE, "assets")
20
+
21
+ # bfloat16 on GPU (float16 overflows this model). Loaded once at module scope.
22
+ model = SentenceTransformer(MODEL_ID, device="cuda", model_kwargs={"dtype": torch.bfloat16})
23
+ model.max_seq_length = 4096
24
+
25
+ DIMS = [768, 512, 256, 128]
26
+ TEXT_TASKS = {
27
+ "Web search": "SearchQuery",
28
+ "Question answering": "QuestionAnswering",
29
+ "Fact checking": "FactChecking",
30
+ "Code search": "CodeRetrieval",
31
+ }
32
+ SYMMETRIC_TASKS = {
33
+ "Sentence similarity": "SentenceSimilarity",
34
+ "Classification": "Classification",
35
+ "Clustering": "Clustering",
36
+ }
37
+ MAX_VIDEO_FRAMES = 24
38
+
39
+ SAMPLE_IMAGES = [
40
+ ("cat.jpg", "Pallas's cat walking in snow"),
41
+ ("cats_couch.jpg", "Two cats asleep on a pink couch"),
42
+ ("bee.jpg", "Bee on a pink cosmos flower"),
43
+ ("baklava.jpg", "Baklava on a plate"),
44
+ ("mosque.jpg", "Domed mosque with minarets"),
45
+ ("squirrel.jpg", "Cubist squirrel painting"),
46
+ ("tiger.jpg", "Screenshot: classifier says 'tiger'"),
47
+ ]
48
+ SAMPLE_AUDIO = [
49
+ ("jfk.wav", "JFK speech clip"),
50
+ ("mlk.flac", "MLK speech clip"),
51
+ ("kungfu.mp3", "Movie line: 'I know kung fu'"),
52
+ ("musicgen.wav", "Short generated music clip"),
53
+ ]
54
+ SAMPLE_TEXTS = """A wild cat with thick fur prowls through a snowy forest.
55
+ Bees pollinate flowers while collecting nectar.
56
+ Layers of filo pastry, chopped nuts and honey syrup.
57
+ Ask not what your country can do for you.
58
+ The orchestra played a lively upbeat tune.
59
+ Abstract geometric art inspired by Picasso.
60
+ Tourists gather outside a historic building in Istanbul."""
61
+
62
+ DEFAULT_DOCS = """Northern lights | Auroras happen when charged particles from the sun collide with gases high in Earth's atmosphere.
63
+ Ocean currents | Large-scale currents move heat around the planet and shape regional climates.
64
+ Eiffel Tower | The wrought-iron tower was built in Paris for the 1889 World's Fair.
65
+ Sleep | During deep sleep the brain consolidates memories formed during the day.
66
+ Photosynthesis | Plants convert sunlight, water and carbon dioxide into glucose and oxygen.
67
+ Auroras (ES) | Las auroras boreales se producen cuando el viento solar choca con la atmósfera.
68
+ binary_search.py | def search(xs, t):\\n lo, hi = 0, len(xs) - 1\\n while lo <= hi:\\n mid = (lo + hi) // 2
69
+ Markets | The stock market closed higher on Friday after strong earnings reports."""
70
+
71
+ DEFAULT_SENTENCES = """The cat is sleeping on the sofa.
72
+ Le chat dort sur le canapé.
73
+ Die Katze schläft auf dem Sofa.
74
+ A dog is barking in the garden.
75
+ Quarterly revenue grew by 12 percent.
76
+ Los ingresos trimestrales crecieron un 12 por ciento."""
77
+
78
+
79
+ # ----------------------------------------------------------------------------- helpers
80
+
81
+
82
+ def _load_audio(path: str) -> dict:
83
+ """Decode any audio file to 16 kHz mono float32, as the model expects."""
84
+ wav, _ = librosa.load(path, sr=16000, mono=True, duration=300)
85
+ return {"array": wav.astype(np.float32), "sampling_rate": 16000}
86
+
87
+
88
+ def _load_video(path: str) -> list[Image.Image]:
89
+ """Sample a video at ~1 fps (the model's default) into a list of PIL frames."""
90
+ frames: list[Image.Image] = []
91
+ with av.open(path) as container:
92
+ stream = container.streams.video[0]
93
+ fps = float(stream.average_rate or 25)
94
+ step = max(1, round(fps))
95
+ for i, frame in enumerate(container.decode(stream)):
96
+ if i % step == 0:
97
+ img = frame.to_image().convert("RGB")
98
+ img.thumbnail((640, 640))
99
+ frames.append(img)
100
+ if len(frames) >= MAX_VIDEO_FRAMES:
101
+ break
102
+ if not frames:
103
+ raise gr.Error("Could not read any frames from that video.")
104
+ return frames
105
+
106
+
107
+ def _to_model_input(item: dict):
108
+ kind, value = item["kind"], item["value"]
109
+ if kind == "text":
110
+ return value
111
+ if kind == "image":
112
+ return {"image": Image.open(value).convert("RGB")}
113
+ if kind == "audio":
114
+ return {"audio": _load_audio(value)}
115
+ if kind == "video":
116
+ return {"video": _load_video(value)}
117
+ raise ValueError(kind)
118
+
119
+
120
+ def _encode(items: list[dict], prompt_name: str | None, dim: int) -> np.ndarray:
121
+ """Embed a mixed list of items. Task prefixes apply to text only; media gets none."""
122
+ out = np.zeros((len(items), int(dim)), dtype=np.float32)
123
+ text_idx = [i for i, it in enumerate(items) if it["kind"] == "text"]
124
+ media_idx = [i for i, it in enumerate(items) if it["kind"] != "text"]
125
+ if text_idx:
126
+ vecs = model.encode(
127
+ [items[i]["value"] for i in text_idx],
128
+ prompt_name=prompt_name,
129
+ truncate_dim=int(dim),
130
+ normalize_embeddings=True,
131
+ convert_to_numpy=True,
132
+ batch_size=16,
133
+ show_progress_bar=False,
134
+ )
135
+ out[text_idx] = vecs.astype(np.float32)
136
+ # Media is encoded one at a time: modalities and audio lengths differ per item.
137
+ for i in media_idx:
138
+ vec = model.encode(
139
+ [_to_model_input(items[i])],
140
+ truncate_dim=int(dim),
141
+ normalize_embeddings=True,
142
+ convert_to_numpy=True,
143
+ show_progress_bar=False,
144
+ )[0]
145
+ out[i] = vec.astype(np.float32)
146
+ return out
147
+
148
+
149
+ def _thumb_b64(path: str, size: int = 120) -> str:
150
+ img = Image.open(path).convert("RGB")
151
+ img.thumbnail((size, size))
152
+ buf = io.BytesIO()
153
+ img.save(buf, format="JPEG", quality=80)
154
+ return "data:image/jpeg;base64," + base64.b64encode(buf.getvalue()).decode()
155
+
156
+
157
+ def _label(path: str, fallback: str | None = None) -> str:
158
+ return fallback or os.path.basename(path)
159
+
160
+
161
+ def _gallery_paths(gallery) -> list[tuple[str, str]]:
162
+ out = []
163
+ for item in gallery or []:
164
+ img, cap = item, None
165
+ if isinstance(item, (list, tuple)):
166
+ img, cap = item[0], (item[1] if len(item) > 1 else None)
167
+ if isinstance(img, dict):
168
+ img = img.get("path") or img.get("name")
169
+ if isinstance(img, str) and os.path.isfile(img):
170
+ out.append((img, _label(img, cap)))
171
+ return out
172
+
173
+
174
+ def _file_paths(files) -> list[str]:
175
+ out = []
176
+ for f in files or []:
177
+ p = f if isinstance(f, str) else getattr(f, "name", None) or (f.get("path") if isinstance(f, dict) else None)
178
+ if p and os.path.isfile(p):
179
+ out.append(p)
180
+ return out
181
+
182
+
183
+ ICONS = {"text": "📝", "image": "🖼️", "audio": "🔊", "video": "🎬"}
184
+
185
+
186
+ def _results_html(ranked: list[tuple[dict, float]], note: str) -> str:
187
+ rows = []
188
+ top = max((s for _, s in ranked), default=1.0) or 1.0
189
+ for rank, (item, score) in enumerate(ranked, start=1):
190
+ if item["kind"] == "image":
191
+ preview = f'<img src="{_thumb_b64(item["value"])}" alt="">'
192
+ else:
193
+ preview = f'<div class="eg-icon">{ICONS[item["kind"]]}</div>'
194
+ width = max(2.0, 100.0 * max(score, 0.0) / top)
195
+ rows.append(
196
+ f"""<div class="eg-row">
197
+ <div class="eg-rank">{rank}</div>
198
+ <div class="eg-prev">{preview}</div>
199
+ <div class="eg-body">
200
+ <div class="eg-label"><span class="eg-kind">{item['kind']}</span>{html.escape(item['label'])}</div>
201
+ <div class="eg-bar"><span style="width:{width:.1f}%"></span></div>
202
+ </div>
203
+ <div class="eg-score">{score:.3f}</div>
204
+ </div>"""
205
+ )
206
+ return f'<div class="eg-note">{html.escape(note)}</div><div class="eg-list">{"".join(rows)}</div>'
207
+
208
+
209
+ # ----------------------------------------------------------------------------- tab 1: any-to-any search
210
+
211
+
212
+ def _build_corpus(texts: str, gallery, audio_files) -> list[dict]:
213
+ corpus: list[dict] = []
214
+ for line in (texts or "").splitlines():
215
+ if line.strip():
216
+ corpus.append({"kind": "text", "value": line.strip(), "label": line.strip()})
217
+ for path, label in _gallery_paths(gallery):
218
+ corpus.append({"kind": "image", "value": path, "label": label})
219
+ audio_labels = {os.path.join(ASSETS, f): lbl for f, lbl in SAMPLE_AUDIO}
220
+ for path in _file_paths(audio_files):
221
+ base = os.path.basename(path)
222
+ label = next((lbl for f, lbl in SAMPLE_AUDIO if base == f), None)
223
+ corpus.append({"kind": "audio", "value": path, "label": audio_labels.get(path, label or base)})
224
+ return corpus
225
+
226
+
227
+ @spaces.GPU(duration=90)
228
+ def cross_modal_search(
229
+ query_type: str,
230
+ query_text: str,
231
+ query_image: str | None,
232
+ query_audio: str | None,
233
+ query_video: str | None,
234
+ corpus_texts: str,
235
+ corpus_images: list | None,
236
+ corpus_audio: list | None,
237
+ dim: int = 768,
238
+ ) -> str:
239
+ """Search a mixed corpus of text, images and audio with a text, image, audio or video query.
240
+
241
+ Every item is embedded into EmbeddingGemma 2's single shared vector space,
242
+ truncated to `dim` (Matryoshka) and L2-normalized, then ranked by cosine similarity.
243
+
244
+ Args:
245
+ query_type: One of "Text", "Image", "Audio", "Video".
246
+ query_text: Query string (used when query_type is "Text").
247
+ query_image: Path to a query image (used when query_type is "Image").
248
+ query_audio: Path to a query audio file (used when query_type is "Audio").
249
+ query_video: Path to a query video file (used when query_type is "Video").
250
+ corpus_texts: Text items in the corpus, one per line.
251
+ corpus_images: Images in the corpus.
252
+ corpus_audio: Audio files in the corpus.
253
+ dim: Embedding dimension: 768, 512, 256 or 128.
254
+
255
+ Returns:
256
+ HTML listing the corpus ranked by similarity to the query.
257
+ """
258
+ t0 = time.perf_counter()
259
+ dim = int(dim)
260
+ if query_type == "Text":
261
+ if not (query_text or "").strip():
262
+ raise gr.Error("Type a text query.")
263
+ query = {"kind": "text", "value": query_text.strip(), "label": query_text.strip()}
264
+ elif query_type == "Image":
265
+ if not query_image:
266
+ raise gr.Error("Upload a query image.")
267
+ query = {"kind": "image", "value": query_image, "label": "query image"}
268
+ elif query_type == "Audio":
269
+ if not query_audio:
270
+ raise gr.Error("Upload or record query audio.")
271
+ query = {"kind": "audio", "value": query_audio, "label": "query audio"}
272
+ else:
273
+ if not query_video:
274
+ raise gr.Error("Upload a query video.")
275
+ query = {"kind": "video", "value": query_video, "label": "query video"}
276
+
277
+ corpus = _build_corpus(corpus_texts, corpus_images, corpus_audio)
278
+ if not corpus:
279
+ raise gr.Error("The corpus is empty — add some text, images or audio.")
280
+ if len(corpus) > 48:
281
+ raise gr.Error("Keep the corpus to 48 items or fewer.")
282
+
283
+ # Text queries use the search-query prefix; text corpus items use the document format.
284
+ q_vec = _encode([query], "SearchQuery", dim)[0]
285
+ docs = [
286
+ dict(it, value=f"title: none | text: {it['value']}") if it["kind"] == "text" else it for it in corpus
287
+ ]
288
+ c_vecs = _encode(docs, None, dim)
289
+ scores = c_vecs @ q_vec
290
+ order = np.argsort(-scores)
291
+ ranked = [(corpus[i], float(scores[i])) for i in order]
292
+ counts = {k: sum(1 for c in corpus if c["kind"] == k) for k in ("text", "image", "audio")}
293
+ note = (
294
+ f"{query_type} query vs {counts['text']} texts · {counts['image']} images · {counts['audio']} audio clips"
295
+ f" · {dim}-d · {time.perf_counter() - t0:.1f}s"
296
+ )
297
+ return _results_html(ranked, note)
298
+
299
+
300
+ # ----------------------------------------------------------------------------- tab 2: text retrieval + matryoshka
301
+
302
+
303
+ def _parse_docs(raw: str) -> list[tuple[str, str]]:
304
+ docs = []
305
+ for line in (raw or "").splitlines():
306
+ line = line.strip()
307
+ if not line:
308
+ continue
309
+ title, content = ("none", line)
310
+ if " | " in line:
311
+ title, content = (s.strip() for s in line.split(" | ", 1))
312
+ content = content.replace("\\n", "\n")
313
+ if content:
314
+ docs.append((title or "none", content))
315
+ return docs
316
+
317
+
318
+ @spaces.GPU(duration=45)
319
+ def text_search(query: str, documents: str, task: str = "Web search") -> list[list]:
320
+ """Rank documents for a query at every Matryoshka size (768, 512, 256, 128 dims).
321
+
322
+ The query gets the task prefix for the chosen task; each document is formatted as
323
+ `title: {title} | text: {content}`. Shows how ranking holds up as vectors shrink.
324
+
325
+ Args:
326
+ query: The search query.
327
+ documents: One document per line, optionally `Title | text`.
328
+ task: One of "Web search", "Question answering", "Fact checking", "Code search".
329
+
330
+ Returns:
331
+ Rows of [title, document, cosine@768, rank@768, rank@512, rank@256, rank@128].
332
+ """
333
+ query = (query or "").strip()
334
+ docs = _parse_docs(documents)
335
+ if not query:
336
+ raise gr.Error("Enter a query.")
337
+ if not docs:
338
+ raise gr.Error("Add at least one document, one per line.")
339
+ if len(docs) > 40:
340
+ raise gr.Error("Use at most 40 documents.")
341
+
342
+ prompt = TEXT_TASKS[task]
343
+ # Embed once at full size, then truncate + re-normalize for each Matryoshka size.
344
+ q = model.encode(query, prompt_name=prompt, convert_to_numpy=True, show_progress_bar=False).astype(np.float32)
345
+ d = model.encode(
346
+ [f"title: {t} | text: {c}" for t, c in docs], convert_to_numpy=True, show_progress_bar=False
347
+ ).astype(np.float32)
348
+
349
+ ranks, full_scores = {}, None
350
+ for dim in DIMS:
351
+ qd = q[:dim] / np.linalg.norm(q[:dim])
352
+ dd = d[:, :dim] / np.linalg.norm(d[:, :dim], axis=1, keepdims=True)
353
+ s = dd @ qd
354
+ if dim == 768:
355
+ full_scores = s
356
+ order = np.argsort(-s)
357
+ r = np.empty(len(docs), dtype=int)
358
+ r[order] = np.arange(1, len(docs) + 1)
359
+ ranks[dim] = r
360
+
361
+ rows = []
362
+ for i in np.argsort(-full_scores):
363
+ title, content = docs[i]
364
+ snippet = content if len(content) <= 140 else content[:137] + "…"
365
+ rows.append([title, snippet, round(float(full_scores[i]), 4)] + [int(ranks[d_][i]) for d_ in DIMS])
366
+ return rows
367
+
368
+
369
+ # ----------------------------------------------------------------------------- tab 3: similarity matrix
370
+
371
+
372
+ def _heatmap_html(labels: list[str], sim: np.ndarray) -> str:
373
+ n = len(labels)
374
+ lo, hi = float(sim[~np.eye(n, dtype=bool)].min()) if n > 1 else 0.0, 1.0
375
+ head = "".join(f'<th title="{html.escape(l)}">#{j + 1}</th>' for j, l in enumerate(labels))
376
+ body = []
377
+ for i, l in enumerate(labels):
378
+ cells = []
379
+ for j in range(n):
380
+ v = float(sim[i, j])
381
+ t = 0.0 if hi == lo else (v - lo) / (hi - lo)
382
+ alpha = 0.08 + 0.82 * max(0.0, min(1.0, t))
383
+ cells.append(f'<td style="background:rgba(99,102,241,{alpha:.2f})">{v:.2f}</td>')
384
+ body.append(f'<tr><th class="eg-rowlab">#{i + 1} {html.escape(l[:60])}</th>{"".join(cells)}</tr>')
385
+ return f'<div class="eg-heat"><table><tr><th></th>{head}</tr>{"".join(body)}</table></div>'
386
+
387
+
388
+ @spaces.GPU(duration=45)
389
+ def similarity_matrix(sentences: str, task: str = "Sentence similarity", dim: int = 768) -> str:
390
+ """Pairwise cosine similarity between lines of text (one per line), as a heatmap.
391
+
392
+ Every line gets the same symmetric task prefix. Works across languages.
393
+
394
+ Args:
395
+ sentences: Two to sixteen lines of text.
396
+ task: One of "Sentence similarity", "Classification", "Clustering".
397
+ dim: Embedding dimension: 768, 512, 256 or 128.
398
+
399
+ Returns:
400
+ HTML heatmap of the similarity matrix.
401
+ """
402
+ lines = [s.strip() for s in (sentences or "").splitlines() if s.strip()]
403
+ if len(lines) < 2:
404
+ raise gr.Error("Enter at least two lines.")
405
+ if len(lines) > 16:
406
+ raise gr.Error("Use at most 16 lines.")
407
+ vecs = _encode([{"kind": "text", "value": s} for s in lines], SYMMETRIC_TASKS[task], int(dim))
408
+ return _heatmap_html(lines, vecs @ vecs.T)
409
+
410
+
411
+ # ----------------------------------------------------------------------------- tab 4: raw embedding (API / MCP)
412
+
413
+
414
+ @spaces.GPU(duration=30)
415
+ def embed(text: str, prompt_name: str = "SearchQuery", dim: int = 768) -> list[float]:
416
+ """Return the EmbeddingGemma 2 embedding for a piece of text.
417
+
418
+ Args:
419
+ text: Text to embed.
420
+ prompt_name: Task prefix name, e.g. SearchQuery, Document, QuestionAnswering, FactChecking,
421
+ CodeRetrieval, Classification, Clustering, SentenceSimilarity.
422
+ dim: Output dimension: 768, 512, 256 or 128 (L2-normalized).
423
+
424
+ Returns:
425
+ The embedding as a list of floats.
426
+ """
427
+ if not (text or "").strip():
428
+ raise gr.Error("Enter some text.")
429
+ if prompt_name not in model.prompts:
430
+ raise gr.Error(f"Unknown prompt_name. Choose from: {', '.join(sorted(model.prompts))}")
431
+ v = model.encode(
432
+ text, prompt_name=prompt_name, truncate_dim=int(dim), normalize_embeddings=True, convert_to_numpy=True
433
+ )
434
+ return [round(float(x), 6) for x in v]
435
+
436
+
437
+ # ----------------------------------------------------------------------------- UI
438
+
439
+ CSS = """
440
+ .gradio-container { max-width: 1180px !important; margin: 0 auto !important; }
441
+ .dark .gradio-container { color: var(--body-text-color); }
442
+ .eg-note { font-size: .85rem; opacity: .75; margin: 2px 0 10px; }
443
+ .eg-list { display: flex; flex-direction: column; gap: 6px; }
444
+ .eg-row { display: grid; grid-template-columns: 28px 64px 1fr 64px; align-items: center; gap: 10px;
445
+ padding: 6px 10px; border: 1px solid var(--border-color-primary); border-radius: 10px;
446
+ background: var(--block-background-fill); }
447
+ .eg-row:first-child { border-color: var(--color-accent); }
448
+ .eg-rank { font-weight: 700; text-align: center; opacity: .7; }
449
+ .eg-prev img { width: 64px; height: 48px; object-fit: cover; border-radius: 6px; display: block; }
450
+ .eg-icon { width: 64px; height: 48px; display: grid; place-items: center; font-size: 22px;
451
+ border-radius: 6px; background: var(--background-fill-secondary); }
452
+ .eg-label { font-size: .92rem; line-height: 1.3; }
453
+ .eg-kind { font-size: .7rem; text-transform: uppercase; letter-spacing: .04em; opacity: .6; margin-right: 8px; }
454
+ .eg-bar { height: 6px; border-radius: 3px; background: var(--background-fill-secondary); margin-top: 6px; overflow: hidden; }
455
+ .eg-bar span { display: block; height: 100%; background: var(--color-accent); border-radius: 3px; }
456
+ .eg-score { font-variant-numeric: tabular-nums; text-align: right; font-weight: 600; }
457
+ .eg-heat { overflow-x: auto; }
458
+ .eg-heat table { border-collapse: collapse; font-size: .82rem; }
459
+ .eg-heat th, .eg-heat td { padding: 6px 8px; text-align: center; border: 1px solid var(--border-color-primary); }
460
+ .eg-heat td { font-variant-numeric: tabular-nums; min-width: 44px; }
461
+ .eg-heat .eg-rowlab { text-align: left; font-weight: 500; white-space: nowrap; }
462
+ """
463
+
464
+ INTRO = f"""# EmbeddingGemma 2 — one vector space for text, images, audio & video
465
+ [{MODEL_ID}](https://huggingface.co/{MODEL_ID}) is Google DeepMind's 740M multimodal embedding model: 768-d vectors,
466
+ 100+ languages, task prefixes, and Matryoshka truncation to 512 / 256 / 128 dims.
467
+ """
468
+
469
+ with gr.Blocks(title="EmbeddingGemma 2 Explorer") as demo:
470
+ gr.Markdown(INTRO)
471
+
472
+ with gr.Tabs():
473
+ # ---------------------------------------------------------------- any-to-any
474
+ with gr.Tab("🔀 Any-to-any search"):
475
+ gr.Markdown(
476
+ "Query with **text, an image, audio or a video** and rank a mixed corpus of text, images and audio. "
477
+ "Everything lives in the same embedding space."
478
+ )
479
+ with gr.Row():
480
+ with gr.Column(scale=5):
481
+ q_type = gr.Radio(["Text", "Image", "Audio", "Video"], value="Text", label="Query type")
482
+ q_text = gr.Textbox(
483
+ label="Text query", value="a fluffy animal in the snow", lines=2, visible=True
484
+ )
485
+ q_image = gr.Image(label="Image query", type="filepath", height=220, visible=False)
486
+ q_audio = gr.Audio(label="Audio query", type="filepath", visible=False)
487
+ q_video = gr.Video(label="Video query (sampled at 1 fps, ≤24 frames)", visible=False)
488
+ dim1 = gr.Radio(DIMS, value=768, label="Dimensions (Matryoshka)")
489
+ go1 = gr.Button("Search", variant="primary")
490
+ with gr.Accordion("Corpus", open=False):
491
+ c_texts = gr.Textbox(label="Texts (one per line)", value=SAMPLE_TEXTS, lines=8)
492
+ c_images = gr.Gallery(
493
+ label="Images",
494
+ value=[(os.path.join(ASSETS, f), c) for f, c in SAMPLE_IMAGES],
495
+ type="filepath",
496
+ columns=4,
497
+ height=260,
498
+ interactive=True,
499
+ )
500
+ c_audio = gr.File(
501
+ label="Audio clips",
502
+ value=[os.path.join(ASSETS, f) for f, _ in SAMPLE_AUDIO],
503
+ file_count="multiple",
504
+ file_types=["audio"],
505
+ )
506
+ with gr.Column(scale=6):
507
+ out1 = gr.HTML(label="Ranked results")
508
+
509
+ def _toggle(kind):
510
+ return [gr.update(visible=kind == k) for k in ("Text", "Image", "Audio", "Video")]
511
+
512
+ q_type.change(_toggle, q_type, [q_text, q_image, q_audio, q_video], api_visibility="private")
513
+
514
+ inputs1 = [q_type, q_text, q_image, q_audio, q_video, c_texts, c_images, c_audio, dim1]
515
+ go1.click(cross_modal_search, inputs1, out1, api_name="cross_modal_search")
516
+ q_text.submit(cross_modal_search, inputs1, out1, api_visibility="private")
517
+
518
+ gr.Examples(
519
+ label="Try",
520
+ examples=[
521
+ ["Text", "a fluffy animal in the snow", None, None],
522
+ ["Text", "something sweet to eat", None, None],
523
+ ["Text", "a famous political speech", None, None],
524
+ ["Text", "un insecto sobre una flor", None, None],
525
+ ["Text", "a user interface showing model predictions", None, None],
526
+ ["Image", "", os.path.join(ASSETS, "cats_couch.jpg"), None],
527
+ ["Audio", "", None, os.path.join(ASSETS, "jfk.wav")],
528
+ ],
529
+ inputs=[q_type, q_text, q_image, q_audio],
530
+ )
531
+
532
+ # ---------------------------------------------------------------- text retrieval
533
+ with gr.Tab("📚 Text retrieval & Matryoshka"):
534
+ gr.Markdown(
535
+ "Task-prefixed retrieval. The table shows each document's rank at **768 / 512 / 256 / 128** dims "
536
+ "— a quick look at how much you can shrink vectors before ranking shifts."
537
+ )
538
+ with gr.Row():
539
+ with gr.Column(scale=5):
540
+ q2 = gr.Textbox(label="Query", value="What causes the northern lights?", lines=2)
541
+ task2 = gr.Dropdown(list(TEXT_TASKS), value="Web search", label="Task prefix")
542
+ docs2 = gr.Textbox(
543
+ label="Documents", info="One per line, optional `Title | text`.", value=DEFAULT_DOCS, lines=12
544
+ )
545
+ go2 = gr.Button("Rank", variant="primary")
546
+ with gr.Column(scale=7):
547
+ out2 = gr.Dataframe(
548
+ headers=["Title", "Document", "Cosine @768", "Rank @768", "@512", "@256", "@128"],
549
+ datatype=["str", "str", "number", "number", "number", "number", "number"],
550
+ interactive=False,
551
+ wrap=True,
552
+ label="Results",
553
+ )
554
+ go2.click(text_search, [q2, docs2, task2], out2, api_name="text_search")
555
+ q2.submit(text_search, [q2, docs2, task2], out2, api_visibility="private")
556
+ gr.Examples(
557
+ label="Try",
558
+ examples=[
559
+ ["What causes the northern lights?", DEFAULT_DOCS, "Web search"],
560
+ ["When was the Eiffel Tower built?", DEFAULT_DOCS, "Question answering"],
561
+ ["Solar wind creates auroras.", DEFAULT_DOCS, "Fact checking"],
562
+ ["find an item in a sorted list", DEFAULT_DOCS, "Code search"],
563
+ ],
564
+ inputs=[q2, docs2, task2],
565
+ outputs=out2,
566
+ fn=text_search,
567
+ cache_examples=True,
568
+ cache_mode="lazy",
569
+ )
570
+
571
+ # ---------------------------------------------------------------- similarity
572
+ with gr.Tab("🌐 Similarity matrix"):
573
+ gr.Markdown("Pairwise cosine similarity — note how translations cluster across languages.")
574
+ with gr.Row():
575
+ with gr.Column(scale=4):
576
+ s3 = gr.Textbox(label="Lines", value=DEFAULT_SENTENCES, lines=10)
577
+ task3 = gr.Dropdown(list(SYMMETRIC_TASKS), value="Sentence similarity", label="Task prefix")
578
+ dim3 = gr.Radio(DIMS, value=768, label="Dimensions")
579
+ go3 = gr.Button("Compare", variant="primary")
580
+ with gr.Column(scale=6):
581
+ out3 = gr.HTML()
582
+ go3.click(similarity_matrix, [s3, task3, dim3], out3, api_name="similarity_matrix")
583
+ gr.Examples(
584
+ label="Try",
585
+ examples=[
586
+ [DEFAULT_SENTENCES, "Sentence similarity", 768],
587
+ [
588
+ "I love this phone, the battery lasts forever.\nThe screen cracked after one day.\n"
589
+ "Ce téléphone est génial.\nPantalla rota, muy decepcionado.\nBest purchase I made this year.",
590
+ "Classification",
591
+ 256,
592
+ ],
593
+ ],
594
+ inputs=[s3, task3, dim3],
595
+ outputs=out3,
596
+ fn=similarity_matrix,
597
+ cache_examples=True,
598
+ cache_mode="lazy",
599
+ )
600
+
601
+ # ---------------------------------------------------------------- raw embedding
602
+ with gr.Tab("🧮 Embed"):
603
+ gr.Markdown("Get the raw vector — also exposed via the API and as an MCP tool.")
604
+ with gr.Row():
605
+ with gr.Column():
606
+ t4 = gr.Textbox(label="Text", value="What causes the northern lights?", lines=3)
607
+ p4 = gr.Dropdown(sorted(model.prompts), value="SearchQuery", label="prompt_name")
608
+ d4 = gr.Radio(DIMS, value=256, label="Dimensions")
609
+ go4 = gr.Button("Embed", variant="primary")
610
+ with gr.Column():
611
+ out4 = gr.JSON(label="Embedding")
612
+ go4.click(embed, [t4, p4, d4], out4, api_name="embed")
613
+
614
+ gr.Markdown(
615
+ "Runs in bfloat16 on ZeroGPU. Text uses the model's task prefixes; images, audio (16 kHz mono) and video "
616
+ "(1 fps) are embedded without a prefix. Sample media from Hugging Face documentation datasets."
617
+ )
618
+
619
+
620
+ if __name__ == "__main__":
621
+ demo.launch(theme=gr.themes.Citrus(), css=CSS, mcp_server=True)
assets/baklava.jpg ADDED
assets/bee.jpg ADDED
assets/cat.jpg ADDED
assets/cats_couch.jpg ADDED

Git LFS Details

  • SHA256: e51cbf711fd5e3a2bf3cfe5621cac9b776a4e02e14f775b96cedf5953fd40d5b
  • Pointer size: 131 Bytes
  • Size of remote file: 108 kB
assets/jfk.wav ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:59dfb9a4acb36fe2a2affc14bacbee2920ff435cb13cc314a08c13f66ba7860e
3
+ size 352078
assets/kungfu.mp3 ADDED
Binary file (80.5 kB). View file
 
assets/mlk.flac ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:88a3befa3c8ae57f515dedae5c23071acf1a29377694c310eee3205d299db51b
3
+ size 383447
assets/mosque.jpg ADDED

Git LFS Details

  • SHA256: a0b27e3beab45055f01ab8be123bf1b7c1e2cd6df0b2ea730eb0d5ee6eee7edc
  • Pointer size: 131 Bytes
  • Size of remote file: 107 kB
assets/musicgen.wav ADDED
Binary file (81 kB). View file
 
assets/squirrel.jpg ADDED
assets/tiger.jpg ADDED
requirements.txt ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ sentence-transformers>=6.1.0
2
+ transformers>=5.19.0
3
+ torchvision
4
+ accelerate
5
+ pillow
6
+ numpy
7
+ librosa
8
+ soundfile
9
+ av