tt-hous commited on
Commit
e7d4c83
·
verified ·
1 Parent(s): a26be19

tt-model push embeddinggemma-2-p150 (container)

Browse files
README.md CHANGED
@@ -1,392 +1,173 @@
1
  ---
2
- license: apache-2.0
3
- pipeline_tag: feature-extraction
4
- base_model:
5
- - google/embeddinggemma-2
6
  tags:
7
- - audio-feature-extraction
8
  - blackhole
9
- - embeddings
10
- - feature-extraction
11
- - image-feature-extraction
12
- - multimodal-embedding
13
  - p150
14
- - sentence-similarity
15
- - tenstorrent
16
  - tt-dit-server
17
- - tt-metal
18
  - tt-model-cache
19
- - tt-model-catalog
20
  - tt-model-container
21
- - ttnn
22
- - video-feature-extraction
 
 
23
  ---
24
 
25
- # EmbeddingGemma 2 on one Tenstorrent Blackhole p150
26
-
27
- ## What it is
28
-
29
- This package serves [google/embeddinggemma-2](https://huggingface.co/google/embeddinggemma-2) at revision `914f7f89142e33e77833254d9c9b90c3cef7303b` on one Tenstorrent Blackhole p150 chip.
30
 
31
- - The model is written in TTNN (the Tenstorrent neural network operator library of tt-metal).
32
- - The container runs stock upstream [tt-metal](https://github.com/tenstorrent/tt-metal) at commit `3be67656b28e3902f0366ea8bab03c41e7e5bd6c`. No tt-metal source is changed.
33
- - All model code is in one self-contained directory, `models/autoports/google_embeddinggemma_2`. This includes two custom kernels: SDPA (scaled dot-product attention) with a sliding-window K range and a key-padding mask, and a GEGLU (GELU-gated linear unit, where GELU is the Gaussian error linear unit) matmul. Both run through the stock `ttnn.generic_op`.
34
- - Inputs: text, images, video frames and 16 kHz audio. Each input gives one L2-normalized 768-dimensional embedding.
35
- - MRL (Matryoshka Representation Learning) dimensions: 768, 512, 256 and 128. The server truncates the 768-dimensional vector to the first `d` values and L2-normalizes it again, on the device.
36
- - Weights stay on the device. Each request shape runs as a captured TTNN trace.
37
- - Status: bring-up build, not a Tenstorrent release.
38
- - Package image: `tt-model/embeddinggemma-2-p150:7fb7b7c492fb` (digest `sha256:7fb7b7c492fbfe924c338b2cd2b5659895343208f417202e92a8adf8dab5c9f7`), built with tt-model 0.1.0 at tt-model-manager commit `713eeb2f33354881beca0fffc2b1ffe15bb5867d`.
39
 
40
- Architecture of the text tower: a bidirectional Gemma encoder with 24 layers, hidden size 512 and 4 query heads. Sliding layers use head_dim 256 and 2 KV (key/value) heads with a band of |i - j| <= 512. Full-attention layers 5, 11, 17 and 23 use head_dim 512 and 1 KV head. The tower uses QK-norm (query and key normalization), sandwich RMSNorm (root mean square normalization), a gated GELU-tanh MLP (multilayer perceptron) of width 2048, PLE (per-layer embeddings) and per-layer scalars. The output path is: final norm, 512 to 768 projection, mean pooling over all tokens (the prompt included), L2 normalization. A 16-layer Gemma 4 vision tower and a 12-layer Gemma 4 audio tower feed soft tokens into the text tower.
41
 
42
- Measurement hardware: one Blackhole chip of a P300c board, opened as a 1x1 mesh. It is equal to a p150. The chip reports a compute grid of 11x10 (110 cores). Every performance and accuracy number in this card comes from that chip. The "card (full)" evaluation column is Google's published full-suite score.
43
 
44
- ## How to run with tt-model
45
 
46
- Requirements:
47
 
48
- - Docker.
49
- - One Blackhole p150, or one chip of a p300.
50
- - 1 GB hugepages mounted at `/dev/hugepages-1G`.
51
 
52
- Pull and serve:
 
 
 
53
 
54
- ```bash
55
- tt-model pull tt-hous/embeddinggemma-2-p150
56
- tt-model serve tt-hous/embeddinggemma-2-p150
57
- ```
58
 
59
- - The default port is 20000. If that port is busy, tt-model tries 20001, 20002 and up. Use `--port <n>` to set the port.
60
- - Use `--device-id 0` to pin one chip. Without it, tt-model picks a free chip.
61
- - The server is ready when the log shows `Application startup complete`.
62
- - Stop the server with `tt-model stop tt-hous/embeddinggemma-2-p150`.
63
 
64
- Notes:
65
 
66
- - One chip of a P300 board needs `TT_MESH_GRAPH_DESC_PATH` to point at the p150 mesh graph descriptor (`p150_mesh_graph_descriptor.textproto`). The package sets it in its serve environment.
67
- - The first boot JIT-compiles (just-in-time compiles) the kernels and captures 51 traces (42 text, 9 media). The compiled kernels go to the kernel cache that tt-model bind-mounts from the host, so later boots reuse them. Measured with this package on one p150-equivalent chip, from `docker run` to `Application startup complete` (polled every 15 s): 360 s on the first boot with an empty kernel cache, 30 s on the second boot.
68
- - The package sets `EMBEDDINGGEMMA_ALLOW_URLS=0`. The server rejects `http(s)` media URLs (uniform resource locators). Send media as base64.
69
- - The app is the uvicorn ASGI (Asynchronous Server Gateway Interface) app `models.autoports.google_embeddinggemma_2.server.app:app`. Run one server process per chip.
 
70
 
71
- ### Demo profile
72
 
73
  ```bash
74
- tt-model serve --profile demo tt-hous/embeddinggemma-2-p150
75
- ```
76
-
77
- - Open `http://<host>:<port>/demo/`. `/` redirects there. The API endpoints stay the same.
78
- - A Gradio page, where every query is embedded live on the chip:
79
- - Media search: text, image, audio or video query over one index of images, videos, audio and text.
80
- - Video moments: text or audio query over 8 s windows of a long video.
81
- - Audio search: text query over clips, and over 10 s windows of recordings up to 320 s.
82
- - Code search with the `CodeRetrieval` prompt.
83
- - Multilingual search over 10 languages.
84
- - Matryoshka: one query at 768, 512, 256 and 128 dims, with top-k overlap and index size.
85
- - Device: live counters, perf tables and the TT vs HF eval means.
86
- - The bundled library has 163 items under CC0, public domain, CC BY, CC BY-SA, Apache-2.0 and MIT licenses. Attribution: `models/autoports/google_embeddinggemma_2/demo/library/LICENSES.md` in the image.
87
- - Startup with a warm kernel cache, measured on one p150-equivalent chip: 57 s on the first demo boot (24.4 s of it embeds the 391 library entries), 30 s after that. The library index is cached in the host tensor cache.
88
- - On 65 pre-registered demo queries, the TT ranking passes as often as HF fp32 on every tab, and the top-1 result matches HF on 60 of 62. Audio-to-video-scene queries fail on both (0 of 3). Video embeddings use frames only.
89
- - The `default` profile does not load the demo or import gradio.
90
-
91
- ## API
92
-
93
- The server is compatible with the OpenAI embeddings API (application programming interface), with extensions for media.
94
-
95
- ### `POST /v1/embeddings`
96
-
97
- | Field | Type | Default | Notes |
98
- |---|---|---|---|
99
- | `input` | string, or a list of strings and objects | required | 1 to 2048 items. Empty strings are rejected. Token-id lists are rejected. |
100
- | `model` | string | none | If set, it must be `google/embeddinggemma-2`, `embeddinggemma-2` or `tt-hous/embeddinggemma-2-p150`. |
101
- | `dimensions` | integer (strict) | 768 | 768, 512, 256 or 128. The strings `"512"` and `true` are rejected. |
102
- | `encoding_format` | `float` or `base64` | `float` | `base64` is little-endian float32 bytes. |
103
- | `input_type` | string | none | Extension. A sentence-transformers prompt name, for example `query`, `document`, `STS`, `Retrieval-query`, `Classification`. None means no prompt. The prompt text is prepended to every item, media-only items included. |
104
- | `user` | string | none | Accepted and ignored. |
105
-
106
- An `input` item can be an object with `text` and at most one media field:
107
-
108
- | Field | Value | Processing |
109
- |---|---|---|
110
- | `text` | string | Optional when a media field is present. |
111
- | `image` | base64 bytes or a `data:image/...;base64,` URL | Any PIL (Python Imaging Library) format. RGBA (red, green, blue, alpha) and gray images are converted to RGB. Gemma 4 image processor at 280 soft tokens. |
112
- | `audio` | base64 WAV (Waveform Audio File Format) bytes or a `data:` URL | PCM (pulse-code modulation) 16, 24 or 32-bit, or float. Multi-channel audio is averaged to mono. The sample rate must be 16000 Hz. Audio longer than 30 s is truncated to 30 s, as in the HF (Hugging Face) processor default. |
113
- | `video` | `{"frames": [<image>, ...], "fps": <number, optional>}` or a plain list of frames | All frames must have the same size. With `fps`, the processor samples 1 frame per second. Without `fps`, it uses every frame. At most 32 frames are used (uniform sampling). |
114
-
115
- The key order sets the order of text and media. `{"text": "a", "image": ...}` renders the text before the image. `{"image": ..., "text": "a"}` renders the image first.
116
-
117
- Response:
118
-
119
- ```json
120
- {
121
- "object": "list",
122
- "data": [{"object": "embedding", "index": 0, "embedding": [0.074, -0.127, ...]}],
123
- "model": "google/embeddinggemma-2",
124
- "usage": {"prompt_tokens": 18, "total_tokens": 18}
125
- }
126
  ```
127
 
128
- `usage` counts tokens after the prompt, `<bos>`, `<eos>` and media placeholder expansion.
129
 
130
- Text:
131
 
 
132
  ```bash
133
  curl -s localhost:20000/v1/embeddings -H 'content-type: application/json' \
134
  -d '{"input": ["What is the capital of France?"], "input_type": "query", "dimensions": 256}'
135
  ```
136
 
137
- Image (base64):
138
-
139
  ```bash
140
- IMG=$(base64 -w0 cats.jpg)
141
- curl -s localhost:20000/v1/embeddings -H 'content-type: application/json' \
142
- -d "{\"input\": [{\"image\": \"$IMG\"}, {\"text\": \"two cats on a couch\"}]}"
143
  ```
 
144
 
145
- Audio (16 kHz WAV, base64):
146
 
147
- ```bash
148
- WAV=$(base64 -w0 speech_16k.wav)
149
- curl -s localhost:20000/v1/embeddings -H 'content-type: application/json' \
150
- -d "{\"input\": [{\"audio\": \"$WAV\", \"text\": \"speech\"}]}"
151
- ```
152
-
153
- Video (frames as base64 images):
154
-
155
- ```bash
156
- F0=$(base64 -w0 frame000.png); F1=$(base64 -w0 frame001.png); F2=$(base64 -w0 frame002.png)
157
- curl -s localhost:20000/v1/embeddings -H 'content-type: application/json' \
158
- -d "{\"input\": [{\"video\": {\"frames\": [\"$F0\", \"$F1\", \"$F2\"], \"fps\": 1}}]}"
159
- ```
160
-
161
- ### `GET /health`
162
 
163
- Returns `200` with `status`, `model`, `precision_policy`, `mesh_shape`, `grid`, `max_tokens`, `switch_interval_ms`, `batching`, `dimensions`, `slots`, `startup_s`, `queue_depth`, `stats`, `device_thread_alive`, `bucket_grouping`, `traces` and `uptime_s`. Returns `503` before the model is loaded or when the device thread has stopped.
 
 
 
164
 
165
- ### `GET /v1/models`
166
-
167
- ```json
168
- {"object": "list", "data": [{"id": "google/embeddinggemma-2", "object": "model", "created": <server start, Unix seconds>, "owned_by": "google", "root": "google/embeddinggemma-2", "max_model_len": 8192, "dimensions": [768, 512, 256, 128], "modalities": ["text", "image", "audio", "video"]}]}
169
- ```
170
 
171
- `created` is the server start time in Unix seconds.
172
 
173
- ### Errors
174
 
175
- All errors use the OpenAI JSON (JavaScript Object Notation) shape `{"error": {"message", "type", "param", "code"}}`.
176
-
177
- | Case | Status | `code` |
178
  |---|---|---|
179
- | Bad `dimensions`, `encoding_format` or `input_type`; empty input; token-id input; bad base64; undecodable image or WAV; sample rate other than 16000 Hz; two media kinds in one item; unknown fields; malformed JSON | 400 | none |
180
- | Input longer than 8192 tokens (prompt and media tokens included) | 400 | `context_length_exceeded` |
181
- | Media that fits no captured trace slot | 400 | none |
182
- | Media URL while `EMBEDDINGGEMMA_ALLOW_URLS=0` (the package default) | 400 | none |
183
- | Media payload above 32 MiB, or a video above 1024 frames | 400 | none |
184
- | Unknown `model` | 404 | `model_not_found` |
185
- | Model not loaded, or device thread stopped | 503 | `not_ready`, `device_thread_dead` |
186
- | Unexpected server error | 500 | none (`type: server_error`) |
187
-
188
- ## Performance
189
 
190
- All numbers come from one p150-equivalent chip with an 11x10 compute grid. The precision policy changed between stages:
191
 
192
- - Stages 06 and 07 (T7) ran before the datatype sweep. They use the earlier default policy (the sweep calls it `c01`): BF16 (bfloat16) weights, an fp32 (32-bit floating point) residual stream, HiFi2 math fidelity (2 multiplier passes) for attention projections and HiFi4 (4 passes) for MLP, PLE and head matmuls.
193
- - Stage 08 selected policy `x02_u10_audio_attn_ffn_bfp8_hifi2`. This package serves `x02`. It uses BFP8 (8-bit block floating point) weights at HiFi2 for the text PLE block, the vision tower matmuls and the audio attention and FFN (feed-forward network) matmuls. Text MLP runs at HiFi2. The text residual stream is BF16, and some text activations are BFP8. Stage 09 uses `x02`.
 
 
 
194
 
195
- Definitions: "e2e" (end to end) is host input staging, the host-to-device copy, the trace replay and the readback of the `[B, 768]` output. It excludes tokenization and the media preprocessor. "Replay" is the trace replay and synchronize only.
 
 
 
 
196
 
197
- ### Text, stage 07 traced (policy c01, one p150-equivalent chip, grid 11x10)
198
-
199
- Uniform synthetic inputs of exactly `seq_len` tokens, run through the padded trace slot that the embedder uses. Median of 7 runs after 2 warm-up runs.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
200
 
201
- | seq_len | batch | trace replay (ms) | e2e traced (ms) | embeddings/s | tokens/s |
202
- |---|---|---|---|---|---|
203
- | 128 | 1 | 6.011 | 6.144 | 162.76 | 20833.7 |
204
- | 512 | 1 | 8.803 | 8.991 | 111.23 | 56948.6 |
205
- | 2048 | 1 | 23.398 | 23.983 | 41.7 | 85393.5 |
206
- | 8192 | 1 | 105.186 | 106.674 | 9.37 | 76794.8 |
207
- | 128 | 32 | 31.218 | 31.951 | 1001.52 | 128195.1 |
208
- | 512 | 32 | 124.65 | 126.831 | 252.3 | 129179.6 |
209
- | 2048 | 32 | 672.206 | 679.434 | 47.1 | 96456.7 |
210
- | 8192 | 32 | 3675.915 | 3711.618 | 8.62 | 70627.9 |
211
-
212
- The 2048 x 32 and 8192 x 32 rows exceed the embedder's 32768-token batch budget. They use padded slots built only for this measurement. The server never runs them.
213
-
214
- ### Text and media, stage 08 selected policy x02 (one p150-equivalent chip, grid 11x10)
215
-
216
- Warmed traced e2e, median of runs A and C in an A/B/A order (x02, c01, x02). Built through the normal construction path with the default policy file. The rows text 2048 x 32, text 8192 x 32, image batch 16 and 32, audio 30 s batch 32 and audio 320 s use slots that this package does not capture; the server does not run those shapes.
217
-
218
- | Workload | x02 e2e (ms) | c01 e2e (ms) | Speedup |
219
- |---|---|---|---|
220
- | text 128 x 1 | 5.829 | 6.237 | 1.07x |
221
- | text 512 x 1 | 7.728 | 9.052 | 1.17x |
222
- | text 2048 x 1 | 17.275 | 24.034 | 1.39x |
223
- | text 8192 x 1 | 80.260 | 107.165 | 1.34x |
224
- | text 128 x 32 | 27.338 | 32.220 | 1.18x |
225
- | text 512 x 32 | 95.164 (336.3 embeddings/s) | 129.707 (246.7 embeddings/s) | 1.36x |
226
- | text 2048 x 32 | 492.204 | 680.431 | 1.38x |
227
- | text 8192 x 32 | 2652.243 | 3720.113 | 1.40x |
228
- | image batch 1 / 8 / 16 / 32 | 47.83 / 292.05 / 592.04 / 1192.23 | 48.53 / 310.51 / 630.65 / 1248.19 | 1.01x / 1.06x / 1.07x / 1.05x |
229
- | video 8 frames / 32 frames | 138.12 / 586.08 | 147.58 / 634.26 | 1.07x / 1.08x |
230
- | audio 30 s, batch 1 / 8 / 32 | 32.15 / 186.79 / 707.07 | 36.58 / 219.91 / 839.81 | 1.14x / 1.18x / 1.19x |
231
- | audio 320 s, batch 1 | 227.30 | 273.28 | 1.20x |
232
 
233
- - Text 8192 x 1 and 128 x 32 vary between runs A and C by 9.0 ms and 3.5 ms. Device replay is equal in both runs, so the spread is on the host.
234
- - Sweep ranking metrics for x02 (quiet sequential rounds): 338.51 text embeddings/s at 512 x 32 (median of 13 pooled rounds), 7.683 ms at 512 x 1, 27.71 images/s (batch 8), 1292.8 audio-seconds/s (30 s x 8), 7.29 video clips/s (8 frames). The same ranking gives c01 26.05 images/s, 1096.5 audio-seconds/s and 6.87 clips/s. This method differs from the stage 06 stream method below. Compare numbers only inside one table.
235
-
236
- ### Media, stage 06 traced (policy c01, one p150-equivalent chip, grid 11x10)
237
-
238
- Warmed trace slots. The rows image_b16, image_b32, audio_30s_b32 and audio_320s_b1 use slots that this package does not capture; the server does not run those shapes. "Stream" runs 4 slot batches back to back and overlaps host staging of batch k+1 with the replay of batch k. Media preprocessing on the host is not included.
239
-
240
- | Slot | Input | Batch | Replay (ms) | e2e (ms) | Stream items/s | Stream audio-s/s |
241
- |---|---|---|---|---|---|---|
242
- | image_b1 | one image, 270 ids | 1 | 41.08 | 48.31 | 22.89 | |
243
- | image_b8 | one image, 270 ids | 8 | 228.68 | 302.83 | 31.68 | |
244
- | image_b16 | one image, 270 ids | 16 | 448.52 | 610.87 | 31.61 | |
245
- | image_b32 | one image, 270 ids | 32 | 882.21 | 1211.31 | 32.02 | |
246
- | video_f8_b1 | 8 frames, 1058 ids | 1 | 118.02 | 145.39 | 7.80 | |
247
- | video_f32_b1 | 32 frames, 4226 ids | 1 | 472.26 | 617.71 | 1.92 | |
248
- | audio_30s_b1 | 29.9 s clip | 1 | 34.52 | 36.24 | 25.33 | 757.4 |
249
- | audio_30s_b8 | 29.9 s clip | 8 | 205.15 | 217.40 | 38.16 | 1141.0 |
250
- | audio_30s_b32 | 29.9 s clip | 32 | 776.22 | 840.38 | 40.11 | 1199.3 |
251
- | audio_320s_b1 | 320 s clip | 1 | 254.87 | 271.85 | 3.83 | 1225.6 |
252
-
253
- Host preprocessing per item, measured separately: image 3 to 5 ms, 8-frame video 29 ms, 30 s audio 8 to 10 ms, 320 s audio 130 ms.
254
-
255
- ### Server load test, stage 09 (policy x02, one p150-equivalent chip, grid 11x10)
256
-
257
- Real uvicorn server, closed-loop clients, 0 errors in every run. p50 and p99 are the 50th and 99th percentile request latencies.
258
-
259
- One text of 480 to 512 tokens per request (mean 499, bucket 512, no prompt). 60 s per level after a 5 s warm-up.
260
-
261
- | Concurrency | requests/s | embeddings/s | p50 (ms) | p99 (ms) | mean inputs per device batch | device thread busy |
262
- |---|---|---|---|---|---|---|
263
- | 1 | 88.54 | 88.54 | 11.35 | 13.07 | 1.00 | 0.861 |
264
- | 8 | 207.71 | 207.71 | 38.50 | 40.54 | 4.00 | 0.970 |
265
- | 32 | 277.30 | 277.30 | 114.07 | 122.27 | 15.96 | 0.968 |
266
-
267
- 32 texts of 480 to 512 tokens per request, 30 s per level:
268
-
269
- | Concurrency | requests/s | embeddings/s | p50 (ms) | p99 (ms) |
270
- |---|---|---|---|---|
271
- | 1 | 8.26 | 264.33 | 119.18 | 122.11 |
272
- | 2 | 9.70 | 310.27 | 203.33 | 206.75 |
273
- | 4 | 9.61 | 307.48 | 408.46 | 412.23 |
274
-
275
- Mixed lengths: 75% texts of 16 to 64 tokens, 25% texts of 900 to 1000 tokens, one per request, 30 s per level. "Bucket grouping on" is the default. "Off" puts all pending requests in one batch padded to the longest.
276
-
277
- | Concurrency | bucket grouping | requests/s | p50 (ms) | p99 (ms) | short p50 / p99 (ms) | long p50 / p99 (ms) |
278
- |---|---|---|---|---|---|---|
279
- | 8 | on (default) | 264.60 | 27.55 | 66.73 | 26.25 / 61.74 | 39.35 / 77.63 |
280
- | 8 | off | 160.45 | 42.19 | 70.66 | 41.90 / 70.53 | 68.68 / 71.11 |
281
- | 32 | on (default) | 374.65 | 80.78 | 170.56 | 75.46 / 152.88 | 106.27 / 197.15 |
282
- | 32 | off | 126.85 | 254.54 | 262.02 | 254.45 / 262.16 | 254.81 / 261.46 |
283
-
284
- The server is device-bound: the device thread is busy 97% of the time at concurrency 8 and 32.
285
-
286
- ## Accuracy against HF fp32
287
-
288
- The reference is the HF sentence-transformers output in fp32 on the CPU (central processing unit). TT means this Tenstorrent implementation. Cosine is per input. "Top-10 agreement" is the mean over queries of |TT top 10 ∩ HF top 10| / 10 on the 256-input stage 00 text corpus (64 queries x 192 documents), unless stated. MTEB is the Massive Text Embedding Benchmark; mini-MTEB is a 3-task regression set (STS17, AskUbuntuDupQuestions, ArguAna).
289
-
290
- ### Text
291
-
292
- | Stage | Policy, path | Inputs | Cosine min / mean (768) | Top-10 agreement |
293
- |---|---|---|---|---|
294
- | T6 | T6 policy, eager | 256-input corpus | 0.999932 / 0.999980 | 0.9906 |
295
- | T7 | c01, traced | 256-input corpus | 0.999904 / 0.999979 | 0.9969 |
296
- | 08 | x02, traced | worst of the 256-input corpus (batch 32 and 1), 17490 mini-MTEB inputs, 9920 ArguAna texts at batch 1 | 0.996364 / 0.999753 | 0.984375 (corpus), 0.980014 / 0.981442 (ArguAna, default batching / batch 1) |
297
- | 09 | x02, server, 32 concurrent requests, three runs | 256-input corpus | 0.999245 to 0.999264 / 0.999813 to 0.999816 | 0.982813 to 0.985938 |
298
- | 11 | x02, evaluation worker, eval batching | stage 11 text inputs | min 0.9790 (8 inputs below 0.995); mean not split by modality (all mteb pairs 0.99980) | not measured (benchmark scores in Evaluation) |
299
-
300
- - ArguAna uses 1401 queries x 8626 documents.
301
- - Under x02, 24 of the 17490 mini-MTEB inputs and 12 of the 9920 ArguAna texts are below 0.999 cosine. The stage gate is 0.995.
302
- - In the stage 11 evaluation, 8 text inputs are below the 0.995 gate (minimum 0.9790, a 16-character classification query) and one VisDoc image+text page is at 0.9913. Under policy c01 the same inputs give 0.9989 or higher. Details: `doc/eval/REPORT.md`, section "Per-input outliers below 0.995".
303
- - Mini-MTEB mean of the 3 tasks: x02 63.1032, HF fp32 on CPU 63.1473. This is not the stage 11 evaluation.
304
- - T7 MRL, traced, c01, cosine min at 512 / 256 / 128 dimensions: 0.999907 / 0.999919 / 0.999946. Top-10 agreement at 512 / 256 / 128: 0.9938 / 0.9953 / 0.9953.
305
-
306
- ### Image, video and audio
307
-
308
- | Stage | Policy | Modality | Inputs | Cosine min / mean (768) |
309
- |---|---|---|---|---|
310
- | 06 | c01, traced | image | 21 (11 default, 10 at budgets 70 to 1120) | 0.999553 / 0.999895 |
311
- | 06 | c01, traced | image + caption | 11 | 0.999870 / 0.999940 |
312
- | 06 | c01, traced | 29 images in one input | 1 | 0.999932 |
313
- | 06 | c01, traced | video | 6 (4 to 114 frames) | 0.999934 / 0.999950 |
314
- | 06 | c01, traced | video + caption | 3 | 0.999943 (min) |
315
- | 06 | c01, traced | audio | 10 (1 s to 320 s, untruncated) | 0.999424 / 0.999896 |
316
- | 06 | c01, traced | audio + caption | 7 | 0.999880 (min) |
317
- | 08 | x02, traced | image | stage 06 long set, 21 | 0.999020 / 0.999720 |
318
- | 08 | x02, traced | video | stage 06 long set, 6 | 0.999764 / 0.999849 |
319
- | 08 | x02, traced | audio | stage 06 long set, 10 | 0.998871 / 0.999639 |
320
-
321
- - Cross-modal top-1 agreement with HF (caption to image, video and audio): 1.0, 1.0, 1.0 in stage 06 (c01) and in stage 08 (x02).
322
- - Stage 06 MRL, traced golden-tensor path, cosine min at 768 / 512 / 256 / 128: image 0.999786 / 0.9998 / 0.999814 / 0.999864; image + caption 0.99987 / 0.999879 / 0.999883 / 0.99992; video 0.999948 / 0.999948 / 0.999956 / 0.999971; audio 0.999424 / 0.999454 / 0.999577 / 0.999739.
323
- - Stage 09 server vs direct embedder on the same device batch: cosine min 1.0 for text (122 cases), image (22), audio (10) and video (1).
324
-
325
- ## Evaluation
326
-
327
- "TT p150 (subset)" and "HF reference (subset)" are scores on the fixed subset `eval/subset.json` of each benchmark, run with identical inputs, prompts and scoring code on this package (one p150-equivalent chip, grid 11x10, policy x02) and on the HF fp32 reference on CPU. "card (full)" is the full-suite score from the [google/embeddinggemma-2](https://huggingface.co/google/embeddinggemma-2) model card. Subset means are not comparable to full-suite means. The subset exists to compare TT against HF on the same inputs. Per-task scores, deltas and the method are in `doc/eval/REPORT.md` (also `doc/eval/REPORT.html`). The evaluation follows the subset input policy: audio truncation off, clips up to 320 s. For that, the evaluation worker adds one audio trace slot (batch 1, 8192 tokens, 320 s) to the slots this server captures. The server itself truncates audio to 30 s, as described in the API section. No benchmark mean differs between TT and HF by more than 1.0 point at any dimension.
328
-
329
- Benchmark names: MIEB (Massive Image Embedding Benchmark), MMEB (Massive Multimodal Embedding Benchmark), MSEB (Massive Sound Embedding Benchmark), MAEB (Massive Audio Embedding Benchmark). Metrics: NDCG (normalized discounted cumulative gain), MRR (mean reciprocal rank).
330
-
331
- ### Overall Evaluation Results (768d)
332
-
333
- | Modality | Benchmark | Metric | TT p150 (subset) | HF reference (subset) | card (full) |
334
- | :---- | :---- | :---- | :---- | :---- | :---- |
335
- | *Text* | Massive Text Embedding Benchmark (MTEB, multilingual, v2) | Mean(Task), Multiple | 59.55 | 59.51 | 61.36 |
336
- | | Massive Text Embedding Benchmark (MTEB, code, v1) | Mean(Task), NDCG@10 | 56.99 | 57.04 | 78.68 |
337
- | *Image* | Massive Image Embedding Benchmark (MIEB, lite) | Mean(TaskType), Multiple | 60.93 | 60.50 | 64.64 |
338
- | | Massive Multimodal Embedding Benchmark (MMEB v2 \- Image) | Mean(Task), Hit@1 | 38.85 | 39.10 | 57.28 |
339
- | | Massive Multimodal Embedding Benchmark (MMEB v2 \- VisDoc) | Mean(Task), NDCG@5 | 70.26 | 69.39 | 67.84 |
340
- | *Video* | Massive Multimodal Embedding Benchmark (MMEB v2 \- Video) | Mean(Task), Hit@1 | 20.09 | 19.63 | 50.67 |
341
- | *Audio* | Massive Sound Embedding Benchmark (MSEB, Retrieval) | Mean(Task), MRR@10 | 80.20 | 80.18 | 69.54 |
342
- | | Massive Audio Embedding Benchmark (MAEB) *Hugging Face* | Mean(Task), Multiple | 58.16 | 58.28 | 49.39 |
343
-
344
- ### Evaluation Results with Vector Truncation
345
-
346
- Each embedding is computed once at 768d, truncated to the first 512, 256 or 128 dimensions and L2 re-normalized.
347
-
348
- | Output Dimension | Compression Ratio | MTEB *(multilingual, v2) Mean(Task)* | MTEB *(eng, v2) Mean(Task)* | MTEB *(code, v1) Mean(Task)* | MIEB *(lite) Mean(TaskType)* | MMEB (v2) *Overall* | MSEB *(Retrieval) Mean(Task)* | MAEB *Mean(Task)* |
349
- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
350
- | 768d (Full), TT p150 (subset) | 1:1 | 59.55 | 70.25 | 56.99 | 60.93 | 44.19 | 80.20 | 58.16 |
351
- | 768d (Full), HF reference (subset) | 1:1 | 59.51 | 70.35 | 57.04 | 60.50 | 43.93 | 80.18 | 58.28 |
352
- | 768d (Full), card (full) | 1:1 | 61.36 | 68.46 | 78.68 | 64.64 | 59.01 | 69.54 | 49.39 |
353
- | 512d, TT p150 (subset) | 1:1.5 | 59.20 | 70.24 | 56.62 | 60.77 | 44.70 | 78.99 | 57.93 |
354
- | 512d, HF reference (subset) | 1:1.5 | 59.20 | 70.27 | 56.59 | 60.49 | 44.12 | 79.15 | 58.05 |
355
- | 512d, card (full) | 1:1.5 | 61.17 | 68.41 | 77.24 | 64.32 | 58.38 | 69.18 | 49.21 |
356
- | 256d, TT p150 (subset) | 1:3 | 58.61 | 69.54 | 55.72 | 59.50 | 42.37 | 76.26 | 57.84 |
357
- | 256d, HF reference (subset) | 1:3 | 58.60 | 69.60 | 55.72 | 59.75 | 42.15 | 76.48 | 57.87 |
358
- | 256d, card (full) | 1:3 | 60.41 | 67.78 | 76.18 | 63.13 | 56.24 | 66.76 | 48.91 |
359
- | 128d, TT p150 (subset) | 1:6 | 55.76 | 67.90 | 52.30 | 55.57 | 37.12 | 60.25 | 55.25 |
360
- | 128d, HF reference (subset) | 1:6 | 55.76 | 67.93 | 52.13 | 55.58 | 37.44 | 60.66 | 55.21 |
361
- | 128d, card (full) | 1:6 | 57.89 | 65.68 | 71.41 | 59.06 | 45.65 | 56.71 | 46.92 |
362
 
363
- ## Limitations
364
 
365
- - **One chip only.** The server accepts only a 1x1 mesh. There is no multi-chip mode. `tt-model serve` runs one container per package profile on a host, so this package serves one chip per host.
366
- - **8192-token cap.** Each input can have at most 8192 tokens, counting the prompt, `<bos>`, `<eos>` and all media soft tokens. Longer inputs get `400 context_length_exceeded`. The untraced code path runs longer text, but the server uses traces only.
367
- - **Numerics.** The model runs in BF16 and BFP8 with fp32 accumulation and an fp32 output. It never uses fp16 (16-bit half-precision floating point). The base model card warns that fp16 gives NaN (not a number) or degraded embeddings.
368
- - **Batching drift.** An embedding depends on the shape of the device batch it runs in. In stage 09, batched vs single-input cosine min was 0.999382 for text (over three test runs), 0.999872 for images and 1.0 for audio. Stage 11 measured larger drift on harder inputs. On 24 texts and 39 audio clips (the evaluation's lowest-agreement inputs plus random controls), batch 8 vs batch 1 cosine min was 0.9809 for text and 0.9822 for audio (audio batch 8 slot vs batch 1 slot). Under policy c01 the same inputs give 0.9993 and 0.9973. The drift is a property of the x02 policy as a whole. For text, x02's BFP8 activations alone give part of it at batch 8 (cosine 0.9947); the split between factors depends on the batch shape. For audio, neither the x02 text settings nor the x02 audio settings alone reach the full drift; they interact (`doc/eval/outliers/drift_attribution.json`). A text vector depends on the batch size and on the padded length bucket of its batch, and the longest input in the batch sets that bucket: one longer text in the same batch moved a short text to cosine 0.9841. An audio vector depends on the slot shape (batch 1 or batch 8), not on the other clips in the batch. Results are deterministic for one batch composition, but not across load patterns. For repeatable vectors, send one input per request with no concurrency.
369
- - **Audio precision.** A few token rows in the last audio layers amplify small numeric errors, and the result depends on the clip. HF's own bf16 model shows the same effect on some clips. In the stage 11 evaluation, 37 audio inputs have cosine below 0.995 to HF fp32 (minimum 0.959). Under this package's policy x02, most of them disappear when the audio tower uses the policy c01 audio settings: 15 remain at batch 8 instead of 35. BF16 attention internals in the audio tower add error too. At 768d the audio benchmark means stay within 0.12 points of HF fp32 (MAEB, MSEB); truncated dims differ by up to 0.41 points (MSEB at 128d). Details: `doc/eval/outliers/classification.md`.
370
- - **Text precision.** Policy x02 does not hold 0.995 cosine to HF fp32 on every text input. In the stage 11 evaluation, 8 texts are below 0.995 (minimum 0.9790, mostly short inputs) and one VisDoc page is at 0.9913. Policy c01 gives 0.9989 or higher on the same inputs. The image ships only x02. A direct run from the source tree selects another policy with `EMBEDDINGGEMMA2_PRECISION_POLICY=<path to a JSON file in doc/datatype_sweep/policies/>`. Passing that variable and file into the `tt-model serve` container was not tested. Text benchmark means stay within 0.10 points of HF fp32 at 768d.
371
- - **Top-10 margin.** Retrieval top-10 agreement with HF fp32 is close to the 0.98 bar. On the stage 00 corpus the lowest server value is 0.982813, which is 2 of 640 top-10 slots above the bar. On ArguAna at default batching x02 gives 0.980014 (13730 of 14010 slots, 13729.8 needed).
372
- - **First boot.** The first boot compiles kernels and captures 51 traces before the server is ready (360 s measured; 30 s on a later boot that reuses the kernel cache).
373
- - **Images.** The image budget is fixed at 280 soft tokens (the processor default). Other budgets (70 to 1120) need extra trace slots that the package does not capture.
374
- - **Video.** The server takes decoded frames, not video files. It uses at most 32 frames. Video slots hold up to 8 frames at 2048 tokens and 32 frames at 8192 tokens.
375
- - **Audio.** Audio must be 16 kHz WAV. The server does not resample. Clips longer than 30 s are truncated to 30 s.
376
- - **Media per item.** Each input item holds at most one media kind. Media that does not fit a captured trace slot gets a 400 error.
377
- - **Scheduling.** One device thread serves all requests. A large request holds the device for its full duration. There is no per-request time limit, priority or preemption. With bucket grouping, short requests can wait behind a long-bucket batch (mixed load at concurrency 32: short-request p99 152.88 ms vs p50 75.46 ms).
378
- - **Request size.** The app sets no HTTP (Hypertext Transfer Protocol) body-size limit. Put a limit in a reverse proxy.
379
- - **Base model limits.** The limitations and ethical considerations of the base model card apply.
380
 
381
- ## Grid size
382
 
383
- The measuring host reports an 11x10 compute grid (110 cores) per chip. Every performance number in this card uses that grid. The text matmul program configs are fixed to an 11x10 core grid; on a p150 with a larger grid they do not use the extra cores, and performance there was not measured. Check the `grid` field of `GET /health` on your host.
384
 
385
- ## License and citation
386
 
387
- - Weights: [google/embeddinggemma-2](https://huggingface.co/google/embeddinggemma-2), Apache 2.0, by Google DeepMind.
388
- - Code: Apache 2.0 (tt-metal and the model directory).
389
 
390
- If you use this model, cite the base model:
 
 
 
 
391
 
392
- > Google DeepMind. EmbeddingGemma 2. Hugging Face model `google/embeddinggemma-2`, revision `914f7f89142e33e77833254d9c9b90c3cef7303b`. https://huggingface.co/google/embeddinggemma-2
 
1
  ---
 
 
 
 
2
  tags:
 
3
  - blackhole
 
 
 
 
4
  - p150
 
 
5
  - tt-dit-server
 
6
  - tt-model-cache
 
7
  - tt-model-container
8
+ license: apache-2.0
9
+ pipeline_tag: feature-extraction
10
+ base_model:
11
+ - google/embeddinggemma-2
12
  ---
13
 
14
+ # embeddinggemma-2-p150 on Tenstorrent p150
 
 
 
 
15
 
16
+ > Derived from google/embeddinggemma-2 by Google DeepMind, weights unmodified. The TTNN port is models/autoports/google_embeddinggemma_2, on stock upstream tt-metal.
 
 
 
 
 
 
 
17
 
18
+ EmbeddingGemma 2, a multimodal embedding model. Text, code, images, video frames and 16 kHz audio each map to one L2-normalized 768-dimensional vector in a shared space, so a text query can find an image, a sound or a video. Matryoshka dimensions 512, 256 and 128 are the first values of the 768-dimensional vector, re-normalized on the chip.
19
 
20
+ The text tower is a bidirectional 24-layer Gemma encoder (hidden size 512, sliding and full attention, 8192-token context). Gemma 4 vision (16 layers) and audio (12 layers) towers feed soft tokens into it. Weights stay on the chip and every request shape runs as a captured TTNN trace. Two custom kernels (sliding-window SDPA and a GEGLU matmul) run through stock `ttnn.generic_op`; tt-metal itself is unmodified.
21
 
22
+ Runs on Tenstorrent **p150** — see the serving profiles below.
23
 
24
+ Packaged and published with [tt-model-manager](https://github.com/tenstorrent/tt-model-manager) 0.1.0 (manifest schema 5.1).
25
 
26
+ ## At a glance
 
 
27
 
28
+ | | |
29
+ | --- | --- |
30
+ | Hardware | p150 |
31
+ | License | apache-2.0 |
32
 
33
+ ## Intended use
 
 
 
34
 
35
+ Embeddings for retrieval, semantic search, clustering and classification over text, code, images, video and audio, served from one p150. Not for generating text.
 
 
 
36
 
37
+ ## Prerequisites
38
 
39
+ - The [Tenstorrent CLI](https://github.com/tenstorrent/tt-cli/blob/main/docs/prerequisites.md), `tt`: `uv tool install tenstorrent` (or use tt-model alone, see the Quickstart)
40
+ - Docker
41
+ - Tenstorrent hardware: **p150**
42
+ - 1 GB hugepages mounted at `/dev/hugepages-1G`
43
+ - One chip of a p300 also works: the package sets the p150 mesh graph descriptor.
44
 
45
+ ## Quickstart
46
 
47
  ```bash
48
+ tt serve tt-hous/embeddinggemma-2-p150
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
49
  ```
50
 
51
+ `tt serve` downloads the Docker image and the [`google/embeddinggemma-2`](https://huggingface.co/google/embeddinggemma-2) at `914f7f89142e33e77833254d9c9b90c3cef7303b` weights (into your HF cache; they are not in the image), then starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs `Application startup complete`.
52
 
53
+ Without tt-cli: `tt-model serve tt-hous/embeddinggemma-2-p150`.
54
 
55
+ ### Embedding text
56
  ```bash
57
  curl -s localhost:20000/v1/embeddings -H 'content-type: application/json' \
58
  -d '{"input": ["What is the capital of France?"], "input_type": "query", "dimensions": 256}'
59
  ```
60
 
61
+ ### Demo
 
62
  ```bash
63
+ tt serve tt-hous/embeddinggemma-2-p150 --profile demo
 
 
64
  ```
65
+ Open `http://<host>:20000/demo/`. A web page where every query is embedded live on the chip, over a bundled library of 163 openly licensed images, videos, sounds, texts in 10 languages and code snippets: media search, finding a moment in a long video, audio search, code search, multilingual search, and a side-by-side Matryoshka comparison.
66
 
67
+ ## Serving profiles
68
 
69
+ Select a different serving profile using `--profile`.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
70
 
71
+ | profile | hardware | mesh | description |
72
+ | --- | --- | --- | --- |
73
+ | `default` *(default)* | p150 | P150 | OpenAI-style /v1/embeddings server only. |
74
+ | `demo` | p150 | P150 | The /v1/embeddings server plus a Gradio demo at /demo (media, video moments, audio, code, multilingual and Matryoshka search over a bundled openly licensed library). |
75
 
76
+ ## Capabilities
 
 
 
 
77
 
78
+ This package exposes the model's own HTTP server on port 20000, **not** an OpenAI-compatible chat API — its request and response shapes are the model's own. See the author's notes for the payload it expects.
79
 
80
+ `POST /v1/embeddings` follows the OpenAI embeddings request and response shapes, with media extensions.
81
 
82
+ | field | meaning | default |
 
 
83
  |---|---|---|
84
+ | `input` | a string, or a list of 1 to 2048 strings and media items | required |
85
+ | `dimensions` | 768, 512, 256 or 128 | 768 |
86
+ | `input_type` | a sentence-transformers prompt name, for example `query`, `document`, `CodeRetrieval` | no prompt |
87
+ | `encoding_format` | `float`, or `base64` (little-endian float32) | `float` |
88
+ | `model` | optional; `google/embeddinggemma-2` or `tt-hous/embeddinggemma-2-p150` | |
 
 
 
 
 
89
 
90
+ A media item is an object with optional `text` and at most one media field:
91
 
92
+ | field | value |
93
+ |---|---|
94
+ | `image` | base64 image bytes (any PIL format) or a `data:image/...;base64,` URL |
95
+ | `audio` | base64 16 kHz WAV; clips over 30 s are cut to 30 s |
96
+ | `video` | `{"frames": [<base64 image>, ...], "fps": 1}`; at most 32 frames are used |
97
 
98
+ ```bash
99
+ IMG=$(base64 -w0 cats.jpg)
100
+ curl -s localhost:20000/v1/embeddings -H 'content-type: application/json' \
101
+ -d "{\"input\": [{\"image\": \"$IMG\"}, {\"text\": \"two cats on a couch\"}]}"
102
+ ```
103
 
104
+ `GET /health` answers 200 once the traces are captured, with the precision policy and live counters, and 503 before that. `GET /v1/models` lists the model, its dimensions and modalities. Errors use the OpenAI error shape; an input over 8192 tokens gets 400 `context_length_exceeded`. Media URLs are refused: send media as base64.
105
+
106
+ ## Expected performance
107
+
108
+ | benchmark (768d) | metric | this package | HF fp32 | card |
109
+ | --- | --- | ---: | ---: | ---: |
110
+ | MTEB multilingual v2 | Mean(Task) | 59.55 | 59.51 | 61.36 |
111
+ | MTEB eng v2 | Mean(Task) | 70.25 | 70.35 | 68.46 |
112
+ | MTEB code v1 | Mean(Task), NDCG@10 | 56.99 | 57.04 | 78.68 |
113
+ | MIEB lite | Mean(TaskType) | 60.93 | 60.50 | 64.64 |
114
+ | MMEB v2 Image | Mean(Task), Hit@1 | 38.85 | 39.10 | 57.28 |
115
+ | MMEB v2 VisDoc | Mean(Task), NDCG@5 | 70.26 | 69.39 | 67.84 |
116
+ | MMEB v2 Video | Mean(Task), Hit@1 | 20.09 | 19.63 | 50.67 |
117
+ | MSEB Retrieval | Mean(Task), MRR@10 | 80.20 | 80.18 | 69.54 |
118
+ | MAEB | Mean(Task) | 58.16 | 58.28 | 49.39 |
119
+
120
+ | dimensions | storage | MTEB multilingual | MTEB eng | MTEB code | MIEB lite | MMEB overall | MSEB | MAEB |
121
+ | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
122
+ | 768 | 1x | 59.55 | 70.25 | 56.99 | 60.93 | 44.19 | 80.20 | 58.16 |
123
+ | 512 | 1/1.5 | 59.20 | 70.24 | 56.62 | 60.77 | 44.70 | 78.99 | 57.93 |
124
+ | 256 | 1/3 | 58.61 | 69.54 | 55.72 | 59.50 | 42.37 | 76.26 | 57.84 |
125
+ | 128 | 1/6 | 55.76 | 67.90 | 52.30 | 55.57 | 37.12 | 60.25 | 55.25 |
126
+
127
+ | load, one text of 480 to 512 tokens per request | requests/s | p50 (ms) | p99 (ms) |
128
+ | --- | ---: | ---: | ---: |
129
+ | 1 user | 88.5 | 11.4 | 13.1 |
130
+ | 8 users | 207.7 | 38.5 | 40.5 |
131
+ | 32 users | 277.3 | 114.1 | 122.3 |
132
+
133
+ | single request, warm | latency (ms) |
134
+ | --- | ---: |
135
+ | text, 128 tokens | 5.8 |
136
+ | text, 512 tokens | 7.7 |
137
+ | text, 8192 tokens | 80.3 |
138
+ | image | 47.8 |
139
+ | video, 8 frames | 138.1 |
140
+ | audio, 30 s | 32.2 |
141
+
142
+ Measured on one Blackhole chip of a p300c board opened as a 1x1 mesh (equal to a p150, compute grid 11x10), precision policy x02. Accuracy: a fixed subset of each benchmark, scored with the same inputs, prompts and code on this package and on the Hugging Face fp32 reference on CPU; "card" is the full-suite score from the google/embeddinggemma-2 card and is not comparable to subset means. No benchmark mean differs from HF fp32 by more than 1.0 point at any dimension. Load: a real server with closed-loop clients, 60 s per level, 0 errors. Single request: host staging, trace replay and readback, without tokenization or media decoding.
143
 
144
+ ## Limitations
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
145
 
146
+ - Bring-up build, not a Tenstorrent release.
147
+ - One chip per server; no multi-chip mode.
148
+ - At most 8192 tokens per input, counting the prompt and media tokens.
149
+ - A vector depends slightly on the device batch it ran in (cosine down to 0.98 on hard inputs). Send one input per request for repeatable vectors.
150
+ - Against HF fp32, 8 of the evaluation texts and 37 audio clips fall below 0.995 cosine (minimum 0.959); benchmark means still match within 1.0 point.
151
+ - Audio must be 16 kHz WAV and is cut to 30 s. Video arrives as decoded frames, at most 32. Images use the default 280-token budget.
152
+ - Never run in fp16: the base model card warns it gives NaN or degraded vectors.
153
+ - The first boot compiles kernels and captures 51 traces (about 6 minutes); later boots take about 30 s.
154
+ - Validated on one chip of a p300c. Performance on a p150 with a larger compute grid was not measured.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
155
 
156
+ ## Licensing
157
 
158
+ Weights under the upstream Apache-2.0 licence (google/embeddinggemma-2). The TTNN port and the server in `code/` are Apache-2.0 (tt-metal). Demo library items carry their own CC0, public domain, CC BY, CC BY-SA, Apache-2.0 or MIT licences, listed in `code/models/autoports/google_embeddinggemma_2/demo/library/LICENSES.md`.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
159
 
160
+ ## Feedback
161
 
162
+ Questions or problems with this package: open a discussion at https://huggingface.co/tt-hous/embeddinggemma-2-p150/discussions — that is what reaches its author. A problem with the `tt` tooling itself: `tt report issue` (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.
163
 
164
+ ## Provenance
165
 
166
+ The exact sources the image was built from — `code/` in this repo is byte-identical to the model code inside the image:
 
167
 
168
+ | component | built from |
169
+ | --- | --- |
170
+ | tt-metal | [`3be67656b28e3902f0366ea8bab03c41e7e5bd6c`](https://github.com/tenstorrent/tt-metal/commit/3be67656b28e3902f0366ea8bab03c41e7e5bd6c) |
171
+ | `code/` digest | `852ada085de3df45` (sha256, first 16 hex digits) |
172
+ | built | 2026-10-21T15:21:44+00:00 by tt-model 0.1.0 |
173
 
 
image/blobs/sha256/77d9d1b7c850b7a6eb0c05b01a5e1c25759ade47988603675dfbda02a4140e94 ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "schemaVersion": 2,
3
+ "mediaType": "application/vnd.oci.image.manifest.v1+json",
4
+ "config": {
5
+ "mediaType": "application/vnd.oci.image.config.v1+json",
6
+ "digest": "sha256:8dd29ae163e363976b67e217870421858f30c1e1d287d2fd40844b2c03f5efad",
7
+ "size": 167
8
+ },
9
+ "layers": [
10
+ {
11
+ "mediaType": "application/vnd.in-toto+json",
12
+ "digest": "sha256:7ae19dfcc4ff33d65c67f8e32b090d68171b0f56dd714adae656191c6bfae038",
13
+ "size": 1916,
14
+ "annotations": {
15
+ "in-toto.io/predicate-type": "https://slsa.dev/provenance/v1"
16
+ }
17
+ }
18
+ ]
19
+ }
image/blobs/sha256/7ae19dfcc4ff33d65c67f8e32b090d68171b0f56dd714adae656191c6bfae038 ADDED
@@ -0,0 +1 @@
 
 
1
+ {"_type":"https://in-toto.io/Statement/v0.1","predicateType":"https://slsa.dev/provenance/v1","subject":[{"name":"pkg:docker/tt-model/embeddinggemma-2-p150@build-3be67656b?platform=linux%2Famd64","digest":{"sha256":"2d7d53b4b0c7f7d9c42ef8da860f263f57d7d576711e20fc9f25fed4fb62c286"}}],"predicate":{"buildDefinition":{"buildType":"https://github.com/moby/buildkit/blob/master/docs/attestations/slsa-definitions.md","resolvedDependencies":[{"uri":"pkg:docker/docker/dockerfile@1.7-labs","digest":{"sha256":"b99fecfe00268a8b556fad7d9c37ee25d716ae08a5d7320e6d51c4dd83246894"}},{"uri":"pkg:docker/ubuntu@22.04?platform=linux%2Famd64","digest":{"sha256":"0a9887931b0e38af2e3487ed39e29436b0535c4f3f52e57745baaf79d1898956"}},{"uri":"pkg:docker/ghcr.io/tenstorrent/tt-metal/tt-metalium/ubuntu-22.04-dev-amd64@latest?platform=linux%2Famd64","digest":{"sha256":"16bcf2a1e76051d857a1bee935b589fa4a9b73ee1a9e87eee4ce063de3cd2448"}}],"externalParameters":{"configSource":{"path":"Dockerfile"},"request":{"frontend":"gateway.v0","args":{"cmdline":"docker/dockerfile:1.7-labs","context:metalsrc":"local:metalsrc","frontend.caps":"moby.buildkit.frontend.contexts+forward","sharedkey:localdir:metalsrc":"empty-metalsrc:98849442d91b4194","source":"docker/dockerfile:1.7-labs"},"locals":[{"name":"context"},{"name":"dockerfile"},{"name":"metalsrc"}],"root":{"configSource":{"path":"Dockerfile"},"request":{"args":{"context:metalsrc":"local:metalsrc","frontend.caps":"moby.buildkit.frontend.contexts+forward","sharedkey:localdir:metalsrc":"empty-metalsrc:98849442d91b4194"}}},"compatibilityVersion":20}},"internalParameters":{"builderPlatform":"linux/amd64"}},"runDetails":{"builder":{"id":""},"metadata":{"invocationId":"yg3eg6otkhbsygyxcezas1378","startedOn":"2026-10-21T15:21:44.771641092Z","finishedOn":"2026-10-21T15:21:45.268353002Z","buildkit_metadata":{},"buildkit_completeness":{"request":false,"resolvedDependencies":false}}}}}
image/blobs/sha256/8dd29ae163e363976b67e217870421858f30c1e1d287d2fd40844b2c03f5efad ADDED
@@ -0,0 +1 @@
 
 
1
+ {"architecture":"unknown","os":"unknown","config":{},"rootfs":{"type":"layers","diff_ids":["sha256:7ae19dfcc4ff33d65c67f8e32b090d68171b0f56dd714adae656191c6bfae038"]}}
image/blobs/sha256/bee93e165d00114d3ba68d68a650f6898a743bda5c6c00bff50c0540b20f25f4 ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "schemaVersion": 2,
3
+ "mediaType": "application/vnd.oci.image.index.v1+json",
4
+ "manifests": [
5
+ {
6
+ "mediaType": "application/vnd.oci.image.manifest.v1+json",
7
+ "digest": "sha256:2d7d53b4b0c7f7d9c42ef8da860f263f57d7d576711e20fc9f25fed4fb62c286",
8
+ "size": 4692,
9
+ "platform": {
10
+ "architecture": "amd64",
11
+ "os": "linux"
12
+ }
13
+ },
14
+ {
15
+ "mediaType": "application/vnd.oci.image.manifest.v1+json",
16
+ "digest": "sha256:77d9d1b7c850b7a6eb0c05b01a5e1c25759ade47988603675dfbda02a4140e94",
17
+ "size": 564,
18
+ "annotations": {
19
+ "vnd.docker.reference.digest": "sha256:2d7d53b4b0c7f7d9c42ef8da860f263f57d7d576711e20fc9f25fed4fb62c286",
20
+ "vnd.docker.reference.type": "attestation-manifest"
21
+ },
22
+ "platform": {
23
+ "architecture": "unknown",
24
+ "os": "unknown"
25
+ }
26
+ }
27
+ ]
28
+ }
image/index.json CHANGED
@@ -1 +1 @@
1
- {"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[{"mediaType":"application/vnd.oci.image.index.v1+json","digest":"sha256:7fb7b7c492fbfe924c338b2cd2b5659895343208f417202e92a8adf8dab5c9f7","size":856,"annotations":{"io.containerd.image.name":"docker.io/tt-model/embeddinggemma-2-p150:7fb7b7c492fb","org.opencontainers.image.ref.name":"7fb7b7c492fb"}}]}
 
1
+ {"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[{"mediaType":"application/vnd.oci.image.index.v1+json","digest":"sha256:bee93e165d00114d3ba68d68a650f6898a743bda5c6c00bff50c0540b20f25f4","size":856,"annotations":{"io.containerd.image.name":"docker.io/tt-model/embeddinggemma-2-p150:bee93e165d00","org.opencontainers.image.ref.name":"bee93e165d00"}}]}
image/manifest.json CHANGED
@@ -1 +1 @@
1
- [{"Config":"blobs/sha256/9a9e80da4de6221fd880dc96810376e07d22c2984bb2628ced2e8d0b9c8cd7de","RepoTags":["tt-model/embeddinggemma-2-p150:7fb7b7c492fb"],"Layers":["blobs/sha256/cf1a5487baaf7d71eb78fd0888933c6cdecb40e59edb896511aa0ef4a896031e","blobs/sha256/11d2eac53ff4ff7d3d67057613e4f204fe7da40b8ba001e09a7b785a69827da3","blobs/sha256/35e14da4db1e99991e4ccc3a0d01584086548bae7120c5c368ae5c131f28121a","blobs/sha256/432d2195132d36370aca685dae5eb2d62628dd8f0214ba236618f84198620aa2","blobs/sha256/46713da844e7a0c5aa5f5c79e7270637bf751186078ec5bc260dd7b06006242a","blobs/sha256/451dddf7926009d9f2d3e471e6b27af1af62b2e86a6c061738bdab2b9a4c1a56","blobs/sha256/b5ec001ffd467834132e55397d625249decf5f5f63079307e34ae3acc00e8436","blobs/sha256/b749e800205c3ba3769ba8a199bd85d353f376391ad5f68f16da895d126b82d0","blobs/sha256/a476bdec81d16f5259e66edb2670af7383593c49a98b6cec2427117d756e11c0","blobs/sha256/aba408305578d428cf39127b2a6ce91037c83b6c33f08497174df12843f72b94","blobs/sha256/b65b506905faa6bdd4125549d97257e64c8a5715c11c9780b6a9b7678fd05f07","blobs/sha256/56e26571152696889aa2d2dca1568ff6c70030b12c08b5f196a577ea1cf1afea","blobs/sha256/59abbc84c8a3402b41571973de30f9d6f5e8aeac1919e155602e3570e12995fd","blobs/sha256/fa71074b04af7e9df6466c49c50e461c6f6e1d986900d94092fbb3994a598a53","blobs/sha256/4006fa78a01aa3b5f6051dca103138a25743d828059be8ee4dd4ecca184e78ff","blobs/sha256/75541aedee66d279bb9db97e12a0a82fef223f9104d3e3dcb259a8104e489d8a","blobs/sha256/9a7118cb3c11a1cd38db3e354ad23fefe52a05396021c832422810cbc832650b","blobs/sha256/3751585d3eb5789fa5fec521a6176185ab2299273f215f825f6b0d54b873248e","blobs/sha256/f4ebe6bafc19cd6b0f41fc09321cd0b2676884ebd3b06d95deac3c2f0786fdc2","blobs/sha256/4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1","blobs/sha256/2b553ee3860d69cd0ca80aa40607f7b6f3bf5110b0aa428deeee6a57110f8ab0","blobs/sha256/c71cc42b78a54d49663258ad776fe0ceb6402a360b1b699e6d709464dfec3597","blobs/sha256/0f607516b935aed5184f195141354b0c6cd0bee121742159a513be94baaa48eb"]}]
 
1
+ [{"Config":"blobs/sha256/9a9e80da4de6221fd880dc96810376e07d22c2984bb2628ced2e8d0b9c8cd7de","RepoTags":["tt-model/embeddinggemma-2-p150:bee93e165d00"],"Layers":["blobs/sha256/cf1a5487baaf7d71eb78fd0888933c6cdecb40e59edb896511aa0ef4a896031e","blobs/sha256/11d2eac53ff4ff7d3d67057613e4f204fe7da40b8ba001e09a7b785a69827da3","blobs/sha256/35e14da4db1e99991e4ccc3a0d01584086548bae7120c5c368ae5c131f28121a","blobs/sha256/432d2195132d36370aca685dae5eb2d62628dd8f0214ba236618f84198620aa2","blobs/sha256/46713da844e7a0c5aa5f5c79e7270637bf751186078ec5bc260dd7b06006242a","blobs/sha256/451dddf7926009d9f2d3e471e6b27af1af62b2e86a6c061738bdab2b9a4c1a56","blobs/sha256/b5ec001ffd467834132e55397d625249decf5f5f63079307e34ae3acc00e8436","blobs/sha256/b749e800205c3ba3769ba8a199bd85d353f376391ad5f68f16da895d126b82d0","blobs/sha256/a476bdec81d16f5259e66edb2670af7383593c49a98b6cec2427117d756e11c0","blobs/sha256/aba408305578d428cf39127b2a6ce91037c83b6c33f08497174df12843f72b94","blobs/sha256/b65b506905faa6bdd4125549d97257e64c8a5715c11c9780b6a9b7678fd05f07","blobs/sha256/56e26571152696889aa2d2dca1568ff6c70030b12c08b5f196a577ea1cf1afea","blobs/sha256/59abbc84c8a3402b41571973de30f9d6f5e8aeac1919e155602e3570e12995fd","blobs/sha256/fa71074b04af7e9df6466c49c50e461c6f6e1d986900d94092fbb3994a598a53","blobs/sha256/4006fa78a01aa3b5f6051dca103138a25743d828059be8ee4dd4ecca184e78ff","blobs/sha256/75541aedee66d279bb9db97e12a0a82fef223f9104d3e3dcb259a8104e489d8a","blobs/sha256/9a7118cb3c11a1cd38db3e354ad23fefe52a05396021c832422810cbc832650b","blobs/sha256/3751585d3eb5789fa5fec521a6176185ab2299273f215f825f6b0d54b873248e","blobs/sha256/f4ebe6bafc19cd6b0f41fc09321cd0b2676884ebd3b06d95deac3c2f0786fdc2","blobs/sha256/4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1","blobs/sha256/2b553ee3860d69cd0ca80aa40607f7b6f3bf5110b0aa428deeee6a57110f8ab0","blobs/sha256/c71cc42b78a54d49663258ad776fe0ceb6402a360b1b699e6d709464dfec3597","blobs/sha256/0f607516b935aed5184f195141354b0c6cd0bee121742159a513be94baaa48eb"]}]
tt_kernel_manifest.json CHANGED
@@ -6,7 +6,7 @@
6
  "device_count": 1,
7
  "producer": {
8
  "tt_kernel_version": "0.1.0",
9
- "created_at": "2026-10-21T13:04:12.013886+00:00",
10
  "hostname": "qb2-120-p06t08"
11
  },
12
  "weights": {
@@ -28,8 +28,8 @@
28
  "image": {
29
  "registry": "hf",
30
  "repository": "embeddinggemma-2-p150",
31
- "tag": "tt-model/embeddinggemma-2-p150:7fb7b7c492fb",
32
- "digest": "sha256:7fb7b7c492fbfe924c338b2cd2b5659895343208f417202e92a8adf8dab5c9f7"
33
  },
34
  "kind": "tt-dit-server",
35
  "runtime": {
@@ -87,7 +87,7 @@
87
  "GRADIO_ANALYTICS_ENABLED": "False"
88
  },
89
  "name": "demo",
90
- "description": "The /v1/embeddings server plus a Gradio demo at /demo (media, video moments, audio, code, multilingual and Matryoshka search over a bundled CC-licensed library)."
91
  }
92
  ],
93
  "default_profile": "default",
@@ -102,10 +102,10 @@
102
  "import gradio, json, models.autoports.google_embeddinggemma_2.demo.ui as u, models.autoports.google_embeddinggemma_2.demo.index as i; assert gradio.__version__ == '6.30.0', gradio.__version__; m = json.loads((i.LIBRARY_DIR / 'manifest.json').read_text()); missing = [x['file'] for x in m['items'] if x.get('file') and not (i.LIBRARY_DIR / x['file']).is_file()]; assert not missing, missing[:5]"
103
  ],
104
  "built": {
105
- "image": "tt-model/embeddinggemma-2-p150:7fb7b7c492fb",
106
  "repo": "tt-hous/embeddinggemma-2-p150",
107
  "tt_model_version": "0.1.0",
108
- "created_at": "2026-10-21T13:01:53+00:00",
109
  "tt_metal": {
110
  "sha": "3be67656b28e3902f0366ea8bab03c41e7e5bd6c",
111
  "describe": null,
@@ -117,22 +117,22 @@
117
  "pushed": null
118
  },
119
  "code_sha256": "852ada085de3df458409d2183b853e843234bd60acff1efb98eb3c32c62dd77d",
120
- "image_digest": "sha256:7fb7b7c492fbfe924c338b2cd2b5659895343208f417202e92a8adf8dab5c9f7"
121
  },
122
  "card": {
123
- "description": "google/embeddinggemma-2 (revision 914f7f89142e33e77833254d9c9b90c3cef7303b) on one Tenstorrent Blackhole p150 chip, written in TTNN. Text, image, video and audio inputs give one L2-normalized 768-dimensional embedding (Matryoshka dimensions 768, 512, 256, 128). The server is an OpenAI-style POST /v1/embeddings endpoint (OpenAI embeddings request and response shapes). The image runs stock upstream tt-metal 3be67656b28e3902f0366ea8bab03c41e7e5bd6c; all model code, including two custom kernels launched through stock ttnn.generic_op, lives in models/autoports/google_embeddinggemma_2.\nArchitecture: Bidirectional Gemma encoder: 24 layers, hidden 512, 4 query heads. Sliding layers use head_dim 256, 2 KV heads and a band of |i-j| <= 512; full layers 5, 11, 17 and 23 use head_dim 512 and 1 KV head. QK-norm, sandwich RMSNorm, gated GELU-tanh MLP (2048), per-layer embeddings and per-layer scalars. Output: final norm, 512->768 projection, mask-aware mean pooling over all tokens including the prompt, L2 normalization. Gemma 4 vision (16 layers) and audio (12 layers) towers feed soft tokens into the text tower. Each request bucket runs as a captured TTNN trace; weights stay on device.\n",
124
- "attribution": "Serves google/embeddinggemma-2 by Google, under the Apache 2.0 license.\n",
125
- "prerequisites": "- 1 GB hugepages mounted at /dev/hugepages-1G.\n- One Blackhole p150, or one chip of a p300 (the package sets the p150 mesh graph descriptor).\n",
126
- "quickstart": null,
127
  "architecture": null,
128
  "status": null,
129
- "intended_use": "Retrieval, semantic search, clustering and classification embeddings for text, images, video frames and 16 kHz audio, served from one p150.\n",
130
  "out_of_scope_use": null,
131
- "usage": "POST /v1/embeddings with {\"model\": \"google/embeddinggemma-2\", \"input\": ...}. input is a string, a list of strings, or items {\"text\"?, \"image\"?, \"audio\"?, \"video\"?} with base64 media (image: PNG/JPEG; audio: 16 kHz WAV; video: list of frames). Optional: \"dimensions\" (768, 512, 256, 128), \"input_type\" (any sentence-transformers prompt name, e.g. query or document), \"encoding_format\" (float or base64). GET /health reports the precision policy, trace slots and counters. URL media is disabled (EMBEDDINGGEMMA_ALLOW_URLS=0). The `demo` serve profile (`tt-model serve --profile demo`) adds a Gradio demo at /demo: search a bundled CC-licensed library of images, videos, audio, text and code by text, image, recorded audio or video; find moments in a long video; search inside long recordings; compare Matryoshka dimensions; and see live device counters. The library is embedded on the chip at startup and cached.\n",
132
- "performance": "Evaluation at 768d on the fixed subset `eval/subset.json` of each benchmark, with\nidentical inputs, prompts and scoring on this package (one p150-equivalent chip, grid\n11x10, policy x02) and on the HF fp32 reference on CPU. \"card (full)\" is the full-suite\nscore from the google/embeddinggemma-2 card; subset means are not comparable to it.\n\n| Benchmark | Metric | TT p150 (subset) | HF fp32 (subset) | card (full) |\n| :---- | :---- | :---- | :---- | :---- |\n| MTEB multilingual v2 | Mean(Task) | 59.55 | 59.51 | 61.36 |\n| MTEB eng v2 | Mean(Task) | 70.25 | 70.35 | 68.46 |\n| MTEB code v1 | Mean(Task), NDCG@10 | 56.99 | 57.04 | 78.68 |\n| MIEB lite | Mean(TaskType) | 60.93 | 60.50 | 64.64 |\n| MMEB v2 Image | Mean(Task), Hit@1 | 38.85 | 39.10 | 57.28 |\n| MMEB v2 VisDoc | Mean(Task), NDCG@5 | 70.26 | 69.39 | 67.84 |\n| MMEB v2 Video | Mean(Task), Hit@1 | 20.09 | 19.63 | 50.67 |\n| MSEB Retrieval | Mean(Task), MRR@10 | 80.20 | 80.18 | 69.54 |\n| MAEB | Mean(Task) | 58.16 | 58.28 | 49.39 |\n\nSpeed, one p150-equivalent chip (grid 11x10), policy x02. Server load test, one\n480-512 token text per request: 88.5 req/s at concurrency 1 (p50 11.4 ms), 207.7 req/s\nat 8, 277.3 req/s at 32 (p50 114 ms). 32 texts per request: up to 310 embeddings/s.\nTraced text at 512 tokens x 32: 336.3 embeddings/s warmed end to end (median\n95.164 ms). Truncated-dimension tables are in the model card README.\n",
133
- "limitations": "Bring-up build, not a Tenstorrent release. Accuracy and performance were measured on one Blackhole chip of a P300c board opened as a 1x1 mesh (p150-equivalent, compute grid 11x10). One chip only; no multi-chip mesh. Inputs longer than 8192 tokens are rejected (context_length_exceeded). An embedding depends on the device batch shape: on the hardest evaluation inputs, batch 8 vs batch 1 cosine min is 0.9809 for text and 0.9822 for audio (policy x02); send one input per request for repeatable vectors. 37 audio clips have cosine below 0.995 to HF fp32 (min 0.959). Retrieval top-10 agreement with HF fp32 is 0.983 on the stage 00 corpus. Never run in fp16. The first boot JIT-compiles kernels and captures 51 traces; later boots reuse the kernel cache.\n",
134
  "risks": null,
135
- "licensing": "Weights: Apache 2.0 (google/embeddinggemma-2). Code: Apache 2.0 (tt-metal).\n",
136
  "related": null,
137
  "license": {
138
  "id": "apache-2.0",
 
6
  "device_count": 1,
7
  "producer": {
8
  "tt_kernel_version": "0.1.0",
9
+ "created_at": "2026-10-21T15:21:48.898834+00:00",
10
  "hostname": "qb2-120-p06t08"
11
  },
12
  "weights": {
 
28
  "image": {
29
  "registry": "hf",
30
  "repository": "embeddinggemma-2-p150",
31
+ "tag": "tt-model/embeddinggemma-2-p150:bee93e165d00",
32
+ "digest": "sha256:bee93e165d00114d3ba68d68a650f6898a743bda5c6c00bff50c0540b20f25f4"
33
  },
34
  "kind": "tt-dit-server",
35
  "runtime": {
 
87
  "GRADIO_ANALYTICS_ENABLED": "False"
88
  },
89
  "name": "demo",
90
+ "description": "The /v1/embeddings server plus a Gradio demo at /demo (media, video moments, audio, code, multilingual and Matryoshka search over a bundled openly licensed library)."
91
  }
92
  ],
93
  "default_profile": "default",
 
102
  "import gradio, json, models.autoports.google_embeddinggemma_2.demo.ui as u, models.autoports.google_embeddinggemma_2.demo.index as i; assert gradio.__version__ == '6.30.0', gradio.__version__; m = json.loads((i.LIBRARY_DIR / 'manifest.json').read_text()); missing = [x['file'] for x in m['items'] if x.get('file') and not (i.LIBRARY_DIR / x['file']).is_file()]; assert not missing, missing[:5]"
103
  ],
104
  "built": {
105
+ "image": "tt-model/embeddinggemma-2-p150:bee93e165d00",
106
  "repo": "tt-hous/embeddinggemma-2-p150",
107
  "tt_model_version": "0.1.0",
108
+ "created_at": "2026-10-21T15:21:44+00:00",
109
  "tt_metal": {
110
  "sha": "3be67656b28e3902f0366ea8bab03c41e7e5bd6c",
111
  "describe": null,
 
117
  "pushed": null
118
  },
119
  "code_sha256": "852ada085de3df458409d2183b853e843234bd60acff1efb98eb3c32c62dd77d",
120
+ "image_digest": "sha256:bee93e165d00114d3ba68d68a650f6898a743bda5c6c00bff50c0540b20f25f4"
121
  },
122
  "card": {
123
+ "description": "EmbeddingGemma 2, a multimodal embedding model. Text, code, images, video frames and 16 kHz audio each map to one L2-normalized 768-dimensional vector in a shared space, so a text query can find an image, a sound or a video. Matryoshka dimensions 512, 256 and 128 are the first values of the 768-dimensional vector, re-normalized on the chip.\n\nThe text tower is a bidirectional 24-layer Gemma encoder (hidden size 512, sliding and full attention, 8192-token context). Gemma 4 vision (16 layers) and audio (12 layers) towers feed soft tokens into it. Weights stay on the chip and every request shape runs as a captured TTNN trace. Two custom kernels (sliding-window SDPA and a GEGLU matmul) run through stock `ttnn.generic_op`; tt-metal itself is unmodified.\n",
124
+ "attribution": "Derived from google/embeddinggemma-2 by Google DeepMind, weights unmodified. The TTNN port is models/autoports/google_embeddinggemma_2, on stock upstream tt-metal.\n",
125
+ "prerequisites": "- 1 GB hugepages mounted at `/dev/hugepages-1G`\n- One chip of a p300 also works: the package sets the p150 mesh graph descriptor.\n",
126
+ "quickstart": "### Embedding text\n```bash\ncurl -s localhost:20000/v1/embeddings -H 'content-type: application/json' \\\n -d '{\"input\": [\"What is the capital of France?\"], \"input_type\": \"query\", \"dimensions\": 256}'\n```\n\n### Demo\n```bash\ntt serve tt-hous/embeddinggemma-2-p150 --profile demo\n```\nOpen `http://<host>:20000/demo/`. A web page where every query is embedded live on the chip, over a bundled library of 163 openly licensed images, videos, sounds, texts in 10 languages and code snippets: media search, finding a moment in a long video, audio search, code search, multilingual search, and a side-by-side Matryoshka comparison.\n",
127
  "architecture": null,
128
  "status": null,
129
+ "intended_use": "Embeddings for retrieval, semantic search, clustering and classification over text, code, images, video and audio, served from one p150. Not for generating text.\n",
130
  "out_of_scope_use": null,
131
+ "usage": "`POST /v1/embeddings` follows the OpenAI embeddings request and response shapes, with media extensions.\n\n| field | meaning | default |\n|---|---|---|\n| `input` | a string, or a list of 1 to 2048 strings and media items | required |\n| `dimensions` | 768, 512, 256 or 128 | 768 |\n| `input_type` | a sentence-transformers prompt name, for example `query`, `document`, `CodeRetrieval` | no prompt |\n| `encoding_format` | `float`, or `base64` (little-endian float32) | `float` |\n| `model` | optional; `google/embeddinggemma-2` or `tt-hous/embeddinggemma-2-p150` | |\n\nA media item is an object with optional `text` and at most one media field:\n\n| field | value |\n|---|---|\n| `image` | base64 image bytes (any PIL format) or a `data:image/...;base64,` URL |\n| `audio` | base64 16 kHz WAV; clips over 30 s are cut to 30 s |\n| `video` | `{\"frames\": [<base64 image>, ...], \"fps\": 1}`; at most 32 frames are used |\n\n```bash\nIMG=$(base64 -w0 cats.jpg)\ncurl -s localhost:20000/v1/embeddings -H 'content-type: application/json' \\\n -d \"{\\\"input\\\": [{\\\"image\\\": \\\"$IMG\\\"}, {\\\"text\\\": \\\"two cats on a couch\\\"}]}\"\n```\n\n`GET /health` answers 200 once the traces are captured, with the precision policy and live counters, and 503 before that. `GET /v1/models` lists the model, its dimensions and modalities. Errors use the OpenAI error shape; an input over 8192 tokens gets 400 `context_length_exceeded`. Media URLs are refused: send media as base64.\n",
132
+ "performance": "| benchmark (768d) | metric | this package | HF fp32 | card |\n| --- | --- | ---: | ---: | ---: |\n| MTEB multilingual v2 | Mean(Task) | 59.55 | 59.51 | 61.36 |\n| MTEB eng v2 | Mean(Task) | 70.25 | 70.35 | 68.46 |\n| MTEB code v1 | Mean(Task), NDCG@10 | 56.99 | 57.04 | 78.68 |\n| MIEB lite | Mean(TaskType) | 60.93 | 60.50 | 64.64 |\n| MMEB v2 Image | Mean(Task), Hit@1 | 38.85 | 39.10 | 57.28 |\n| MMEB v2 VisDoc | Mean(Task), NDCG@5 | 70.26 | 69.39 | 67.84 |\n| MMEB v2 Video | Mean(Task), Hit@1 | 20.09 | 19.63 | 50.67 |\n| MSEB Retrieval | Mean(Task), MRR@10 | 80.20 | 80.18 | 69.54 |\n| MAEB | Mean(Task) | 58.16 | 58.28 | 49.39 |\n\n| dimensions | storage | MTEB multilingual | MTEB eng | MTEB code | MIEB lite | MMEB overall | MSEB | MAEB |\n| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n| 768 | 1x | 59.55 | 70.25 | 56.99 | 60.93 | 44.19 | 80.20 | 58.16 |\n| 512 | 1/1.5 | 59.20 | 70.24 | 56.62 | 60.77 | 44.70 | 78.99 | 57.93 |\n| 256 | 1/3 | 58.61 | 69.54 | 55.72 | 59.50 | 42.37 | 76.26 | 57.84 |\n| 128 | 1/6 | 55.76 | 67.90 | 52.30 | 55.57 | 37.12 | 60.25 | 55.25 |\n\n| load, one text of 480 to 512 tokens per request | requests/s | p50 (ms) | p99 (ms) |\n| --- | ---: | ---: | ---: |\n| 1 user | 88.5 | 11.4 | 13.1 |\n| 8 users | 207.7 | 38.5 | 40.5 |\n| 32 users | 277.3 | 114.1 | 122.3 |\n\n| single request, warm | latency (ms) |\n| --- | ---: |\n| text, 128 tokens | 5.8 |\n| text, 512 tokens | 7.7 |\n| text, 8192 tokens | 80.3 |\n| image | 47.8 |\n| video, 8 frames | 138.1 |\n| audio, 30 s | 32.2 |\n\nMeasured on one Blackhole chip of a p300c board opened as a 1x1 mesh (equal to a p150, compute grid 11x10), precision policy x02. Accuracy: a fixed subset of each benchmark, scored with the same inputs, prompts and code on this package and on the Hugging Face fp32 reference on CPU; \"card\" is the full-suite score from the google/embeddinggemma-2 card and is not comparable to subset means. No benchmark mean differs from HF fp32 by more than 1.0 point at any dimension. Load: a real server with closed-loop clients, 60 s per level, 0 errors. Single request: host staging, trace replay and readback, without tokenization or media decoding.\n",
133
+ "limitations": "- Bring-up build, not a Tenstorrent release.\n- One chip per server; no multi-chip mode.\n- At most 8192 tokens per input, counting the prompt and media tokens.\n- A vector depends slightly on the device batch it ran in (cosine down to 0.98 on hard inputs). Send one input per request for repeatable vectors.\n- Against HF fp32, 8 of the evaluation texts and 37 audio clips fall below 0.995 cosine (minimum 0.959); benchmark means still match within 1.0 point.\n- Audio must be 16 kHz WAV and is cut to 30 s. Video arrives as decoded frames, at most 32. Images use the default 280-token budget.\n- Never run in fp16: the base model card warns it gives NaN or degraded vectors.\n- The first boot compiles kernels and captures 51 traces (about 6 minutes); later boots take about 30 s.\n- Validated on one chip of a p300c. Performance on a p150 with a larger compute grid was not measured.\n",
134
  "risks": null,
135
+ "licensing": "Weights under the upstream Apache-2.0 licence (google/embeddinggemma-2). The TTNN port and the server in `code/` are Apache-2.0 (tt-metal). Demo library items carry their own CC0, public domain, CC BY, CC BY-SA, Apache-2.0 or MIT licences, listed in `code/models/autoports/google_embeddinggemma_2/demo/library/LICENSES.md`.\n",
136
  "related": null,
137
  "license": {
138
  "id": "apache-2.0",