tt-model push embeddinggemma-2-p150 (container)
Browse files- README.md +117 -336
- image/blobs/sha256/77d9d1b7c850b7a6eb0c05b01a5e1c25759ade47988603675dfbda02a4140e94 +19 -0
- image/blobs/sha256/7ae19dfcc4ff33d65c67f8e32b090d68171b0f56dd714adae656191c6bfae038 +1 -0
- image/blobs/sha256/8dd29ae163e363976b67e217870421858f30c1e1d287d2fd40844b2c03f5efad +1 -0
- image/blobs/sha256/bee93e165d00114d3ba68d68a650f6898a743bda5c6c00bff50c0540b20f25f4 +28 -0
- image/index.json +1 -1
- image/manifest.json +1 -1
- tt_kernel_manifest.json +16 -16
README.md
CHANGED
|
@@ -1,392 +1,173 @@
|
|
| 1 |
---
|
| 2 |
-
license: apache-2.0
|
| 3 |
-
pipeline_tag: feature-extraction
|
| 4 |
-
base_model:
|
| 5 |
-
- google/embeddinggemma-2
|
| 6 |
tags:
|
| 7 |
-
- audio-feature-extraction
|
| 8 |
- blackhole
|
| 9 |
-
- embeddings
|
| 10 |
-
- feature-extraction
|
| 11 |
-
- image-feature-extraction
|
| 12 |
-
- multimodal-embedding
|
| 13 |
- p150
|
| 14 |
-
- sentence-similarity
|
| 15 |
-
- tenstorrent
|
| 16 |
- tt-dit-server
|
| 17 |
-
- tt-metal
|
| 18 |
- tt-model-cache
|
| 19 |
-
- tt-model-catalog
|
| 20 |
- tt-model-container
|
| 21 |
-
|
| 22 |
-
|
|
|
|
|
|
|
| 23 |
---
|
| 24 |
|
| 25 |
-
#
|
| 26 |
-
|
| 27 |
-
## What it is
|
| 28 |
-
|
| 29 |
-
This package serves [google/embeddinggemma-2](https://huggingface.co/google/embeddinggemma-2) at revision `914f7f89142e33e77833254d9c9b90c3cef7303b` on one Tenstorrent Blackhole p150 chip.
|
| 30 |
|
| 31 |
-
-
|
| 32 |
-
- The container runs stock upstream [tt-metal](https://github.com/tenstorrent/tt-metal) at commit `3be67656b28e3902f0366ea8bab03c41e7e5bd6c`. No tt-metal source is changed.
|
| 33 |
-
- All model code is in one self-contained directory, `models/autoports/google_embeddinggemma_2`. This includes two custom kernels: SDPA (scaled dot-product attention) with a sliding-window K range and a key-padding mask, and a GEGLU (GELU-gated linear unit, where GELU is the Gaussian error linear unit) matmul. Both run through the stock `ttnn.generic_op`.
|
| 34 |
-
- Inputs: text, images, video frames and 16 kHz audio. Each input gives one L2-normalized 768-dimensional embedding.
|
| 35 |
-
- MRL (Matryoshka Representation Learning) dimensions: 768, 512, 256 and 128. The server truncates the 768-dimensional vector to the first `d` values and L2-normalizes it again, on the device.
|
| 36 |
-
- Weights stay on the device. Each request shape runs as a captured TTNN trace.
|
| 37 |
-
- Status: bring-up build, not a Tenstorrent release.
|
| 38 |
-
- Package image: `tt-model/embeddinggemma-2-p150:7fb7b7c492fb` (digest `sha256:7fb7b7c492fbfe924c338b2cd2b5659895343208f417202e92a8adf8dab5c9f7`), built with tt-model 0.1.0 at tt-model-manager commit `713eeb2f33354881beca0fffc2b1ffe15bb5867d`.
|
| 39 |
|
| 40 |
-
|
| 41 |
|
| 42 |
-
|
| 43 |
|
| 44 |
-
|
| 45 |
|
| 46 |
-
|
| 47 |
|
| 48 |
-
|
| 49 |
-
- One Blackhole p150, or one chip of a p300.
|
| 50 |
-
- 1 GB hugepages mounted at `/dev/hugepages-1G`.
|
| 51 |
|
| 52 |
-
|
|
|
|
|
|
|
|
|
|
| 53 |
|
| 54 |
-
|
| 55 |
-
tt-model pull tt-hous/embeddinggemma-2-p150
|
| 56 |
-
tt-model serve tt-hous/embeddinggemma-2-p150
|
| 57 |
-
```
|
| 58 |
|
| 59 |
-
|
| 60 |
-
- Use `--device-id 0` to pin one chip. Without it, tt-model picks a free chip.
|
| 61 |
-
- The server is ready when the log shows `Application startup complete`.
|
| 62 |
-
- Stop the server with `tt-model stop tt-hous/embeddinggemma-2-p150`.
|
| 63 |
|
| 64 |
-
|
| 65 |
|
| 66 |
-
-
|
| 67 |
-
-
|
| 68 |
-
-
|
| 69 |
-
-
|
|
|
|
| 70 |
|
| 71 |
-
##
|
| 72 |
|
| 73 |
```bash
|
| 74 |
-
tt
|
| 75 |
-
```
|
| 76 |
-
|
| 77 |
-
- Open `http://<host>:<port>/demo/`. `/` redirects there. The API endpoints stay the same.
|
| 78 |
-
- A Gradio page, where every query is embedded live on the chip:
|
| 79 |
-
- Media search: text, image, audio or video query over one index of images, videos, audio and text.
|
| 80 |
-
- Video moments: text or audio query over 8 s windows of a long video.
|
| 81 |
-
- Audio search: text query over clips, and over 10 s windows of recordings up to 320 s.
|
| 82 |
-
- Code search with the `CodeRetrieval` prompt.
|
| 83 |
-
- Multilingual search over 10 languages.
|
| 84 |
-
- Matryoshka: one query at 768, 512, 256 and 128 dims, with top-k overlap and index size.
|
| 85 |
-
- Device: live counters, perf tables and the TT vs HF eval means.
|
| 86 |
-
- The bundled library has 163 items under CC0, public domain, CC BY, CC BY-SA, Apache-2.0 and MIT licenses. Attribution: `models/autoports/google_embeddinggemma_2/demo/library/LICENSES.md` in the image.
|
| 87 |
-
- Startup with a warm kernel cache, measured on one p150-equivalent chip: 57 s on the first demo boot (24.4 s of it embeds the 391 library entries), 30 s after that. The library index is cached in the host tensor cache.
|
| 88 |
-
- On 65 pre-registered demo queries, the TT ranking passes as often as HF fp32 on every tab, and the top-1 result matches HF on 60 of 62. Audio-to-video-scene queries fail on both (0 of 3). Video embeddings use frames only.
|
| 89 |
-
- The `default` profile does not load the demo or import gradio.
|
| 90 |
-
|
| 91 |
-
## API
|
| 92 |
-
|
| 93 |
-
The server is compatible with the OpenAI embeddings API (application programming interface), with extensions for media.
|
| 94 |
-
|
| 95 |
-
### `POST /v1/embeddings`
|
| 96 |
-
|
| 97 |
-
| Field | Type | Default | Notes |
|
| 98 |
-
|---|---|---|---|
|
| 99 |
-
| `input` | string, or a list of strings and objects | required | 1 to 2048 items. Empty strings are rejected. Token-id lists are rejected. |
|
| 100 |
-
| `model` | string | none | If set, it must be `google/embeddinggemma-2`, `embeddinggemma-2` or `tt-hous/embeddinggemma-2-p150`. |
|
| 101 |
-
| `dimensions` | integer (strict) | 768 | 768, 512, 256 or 128. The strings `"512"` and `true` are rejected. |
|
| 102 |
-
| `encoding_format` | `float` or `base64` | `float` | `base64` is little-endian float32 bytes. |
|
| 103 |
-
| `input_type` | string | none | Extension. A sentence-transformers prompt name, for example `query`, `document`, `STS`, `Retrieval-query`, `Classification`. None means no prompt. The prompt text is prepended to every item, media-only items included. |
|
| 104 |
-
| `user` | string | none | Accepted and ignored. |
|
| 105 |
-
|
| 106 |
-
An `input` item can be an object with `text` and at most one media field:
|
| 107 |
-
|
| 108 |
-
| Field | Value | Processing |
|
| 109 |
-
|---|---|---|
|
| 110 |
-
| `text` | string | Optional when a media field is present. |
|
| 111 |
-
| `image` | base64 bytes or a `data:image/...;base64,` URL | Any PIL (Python Imaging Library) format. RGBA (red, green, blue, alpha) and gray images are converted to RGB. Gemma 4 image processor at 280 soft tokens. |
|
| 112 |
-
| `audio` | base64 WAV (Waveform Audio File Format) bytes or a `data:` URL | PCM (pulse-code modulation) 16, 24 or 32-bit, or float. Multi-channel audio is averaged to mono. The sample rate must be 16000 Hz. Audio longer than 30 s is truncated to 30 s, as in the HF (Hugging Face) processor default. |
|
| 113 |
-
| `video` | `{"frames": [<image>, ...], "fps": <number, optional>}` or a plain list of frames | All frames must have the same size. With `fps`, the processor samples 1 frame per second. Without `fps`, it uses every frame. At most 32 frames are used (uniform sampling). |
|
| 114 |
-
|
| 115 |
-
The key order sets the order of text and media. `{"text": "a", "image": ...}` renders the text before the image. `{"image": ..., "text": "a"}` renders the image first.
|
| 116 |
-
|
| 117 |
-
Response:
|
| 118 |
-
|
| 119 |
-
```json
|
| 120 |
-
{
|
| 121 |
-
"object": "list",
|
| 122 |
-
"data": [{"object": "embedding", "index": 0, "embedding": [0.074, -0.127, ...]}],
|
| 123 |
-
"model": "google/embeddinggemma-2",
|
| 124 |
-
"usage": {"prompt_tokens": 18, "total_tokens": 18}
|
| 125 |
-
}
|
| 126 |
```
|
| 127 |
|
| 128 |
-
`
|
| 129 |
|
| 130 |
-
|
| 131 |
|
|
|
|
| 132 |
```bash
|
| 133 |
curl -s localhost:20000/v1/embeddings -H 'content-type: application/json' \
|
| 134 |
-d '{"input": ["What is the capital of France?"], "input_type": "query", "dimensions": 256}'
|
| 135 |
```
|
| 136 |
|
| 137 |
-
|
| 138 |
-
|
| 139 |
```bash
|
| 140 |
-
|
| 141 |
-
curl -s localhost:20000/v1/embeddings -H 'content-type: application/json' \
|
| 142 |
-
-d "{\"input\": [{\"image\": \"$IMG\"}, {\"text\": \"two cats on a couch\"}]}"
|
| 143 |
```
|
|
|
|
| 144 |
|
| 145 |
-
|
| 146 |
|
| 147 |
-
``
|
| 148 |
-
WAV=$(base64 -w0 speech_16k.wav)
|
| 149 |
-
curl -s localhost:20000/v1/embeddings -H 'content-type: application/json' \
|
| 150 |
-
-d "{\"input\": [{\"audio\": \"$WAV\", \"text\": \"speech\"}]}"
|
| 151 |
-
```
|
| 152 |
-
|
| 153 |
-
Video (frames as base64 images):
|
| 154 |
-
|
| 155 |
-
```bash
|
| 156 |
-
F0=$(base64 -w0 frame000.png); F1=$(base64 -w0 frame001.png); F2=$(base64 -w0 frame002.png)
|
| 157 |
-
curl -s localhost:20000/v1/embeddings -H 'content-type: application/json' \
|
| 158 |
-
-d "{\"input\": [{\"video\": {\"frames\": [\"$F0\", \"$F1\", \"$F2\"], \"fps\": 1}}]}"
|
| 159 |
-
```
|
| 160 |
-
|
| 161 |
-
### `GET /health`
|
| 162 |
|
| 163 |
-
|
|
|
|
|
|
|
|
|
|
| 164 |
|
| 165 |
-
##
|
| 166 |
-
|
| 167 |
-
```json
|
| 168 |
-
{"object": "list", "data": [{"id": "google/embeddinggemma-2", "object": "model", "created": <server start, Unix seconds>, "owned_by": "google", "root": "google/embeddinggemma-2", "max_model_len": 8192, "dimensions": [768, 512, 256, 128], "modalities": ["text", "image", "audio", "video"]}]}
|
| 169 |
-
```
|
| 170 |
|
| 171 |
-
|
| 172 |
|
| 173 |
-
|
| 174 |
|
| 175 |
-
|
| 176 |
-
|
| 177 |
-
| Case | Status | `code` |
|
| 178 |
|---|---|---|
|
| 179 |
-
|
|
| 180 |
-
|
|
| 181 |
-
|
|
| 182 |
-
|
|
| 183 |
-
|
|
| 184 |
-
| Unknown `model` | 404 | `model_not_found` |
|
| 185 |
-
| Model not loaded, or device thread stopped | 503 | `not_ready`, `device_thread_dead` |
|
| 186 |
-
| Unexpected server error | 500 | none (`type: server_error`) |
|
| 187 |
-
|
| 188 |
-
## Performance
|
| 189 |
|
| 190 |
-
|
| 191 |
|
| 192 |
-
|
| 193 |
-
-
|
|
|
|
|
|
|
|
|
|
| 194 |
|
| 195 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 196 |
|
| 197 |
-
|
| 198 |
-
|
| 199 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 200 |
|
| 201 |
-
|
| 202 |
-
|---|---|---|---|---|---|
|
| 203 |
-
| 128 | 1 | 6.011 | 6.144 | 162.76 | 20833.7 |
|
| 204 |
-
| 512 | 1 | 8.803 | 8.991 | 111.23 | 56948.6 |
|
| 205 |
-
| 2048 | 1 | 23.398 | 23.983 | 41.7 | 85393.5 |
|
| 206 |
-
| 8192 | 1 | 105.186 | 106.674 | 9.37 | 76794.8 |
|
| 207 |
-
| 128 | 32 | 31.218 | 31.951 | 1001.52 | 128195.1 |
|
| 208 |
-
| 512 | 32 | 124.65 | 126.831 | 252.3 | 129179.6 |
|
| 209 |
-
| 2048 | 32 | 672.206 | 679.434 | 47.1 | 96456.7 |
|
| 210 |
-
| 8192 | 32 | 3675.915 | 3711.618 | 8.62 | 70627.9 |
|
| 211 |
-
|
| 212 |
-
The 2048 x 32 and 8192 x 32 rows exceed the embedder's 32768-token batch budget. They use padded slots built only for this measurement. The server never runs them.
|
| 213 |
-
|
| 214 |
-
### Text and media, stage 08 selected policy x02 (one p150-equivalent chip, grid 11x10)
|
| 215 |
-
|
| 216 |
-
Warmed traced e2e, median of runs A and C in an A/B/A order (x02, c01, x02). Built through the normal construction path with the default policy file. The rows text 2048 x 32, text 8192 x 32, image batch 16 and 32, audio 30 s batch 32 and audio 320 s use slots that this package does not capture; the server does not run those shapes.
|
| 217 |
-
|
| 218 |
-
| Workload | x02 e2e (ms) | c01 e2e (ms) | Speedup |
|
| 219 |
-
|---|---|---|---|
|
| 220 |
-
| text 128 x 1 | 5.829 | 6.237 | 1.07x |
|
| 221 |
-
| text 512 x 1 | 7.728 | 9.052 | 1.17x |
|
| 222 |
-
| text 2048 x 1 | 17.275 | 24.034 | 1.39x |
|
| 223 |
-
| text 8192 x 1 | 80.260 | 107.165 | 1.34x |
|
| 224 |
-
| text 128 x 32 | 27.338 | 32.220 | 1.18x |
|
| 225 |
-
| text 512 x 32 | 95.164 (336.3 embeddings/s) | 129.707 (246.7 embeddings/s) | 1.36x |
|
| 226 |
-
| text 2048 x 32 | 492.204 | 680.431 | 1.38x |
|
| 227 |
-
| text 8192 x 32 | 2652.243 | 3720.113 | 1.40x |
|
| 228 |
-
| image batch 1 / 8 / 16 / 32 | 47.83 / 292.05 / 592.04 / 1192.23 | 48.53 / 310.51 / 630.65 / 1248.19 | 1.01x / 1.06x / 1.07x / 1.05x |
|
| 229 |
-
| video 8 frames / 32 frames | 138.12 / 586.08 | 147.58 / 634.26 | 1.07x / 1.08x |
|
| 230 |
-
| audio 30 s, batch 1 / 8 / 32 | 32.15 / 186.79 / 707.07 | 36.58 / 219.91 / 839.81 | 1.14x / 1.18x / 1.19x |
|
| 231 |
-
| audio 320 s, batch 1 | 227.30 | 273.28 | 1.20x |
|
| 232 |
|
| 233 |
-
-
|
| 234 |
-
-
|
| 235 |
-
|
| 236 |
-
|
| 237 |
-
|
| 238 |
-
|
| 239 |
-
|
| 240 |
-
|
| 241 |
-
|
| 242 |
-
| image_b1 | one image, 270 ids | 1 | 41.08 | 48.31 | 22.89 | |
|
| 243 |
-
| image_b8 | one image, 270 ids | 8 | 228.68 | 302.83 | 31.68 | |
|
| 244 |
-
| image_b16 | one image, 270 ids | 16 | 448.52 | 610.87 | 31.61 | |
|
| 245 |
-
| image_b32 | one image, 270 ids | 32 | 882.21 | 1211.31 | 32.02 | |
|
| 246 |
-
| video_f8_b1 | 8 frames, 1058 ids | 1 | 118.02 | 145.39 | 7.80 | |
|
| 247 |
-
| video_f32_b1 | 32 frames, 4226 ids | 1 | 472.26 | 617.71 | 1.92 | |
|
| 248 |
-
| audio_30s_b1 | 29.9 s clip | 1 | 34.52 | 36.24 | 25.33 | 757.4 |
|
| 249 |
-
| audio_30s_b8 | 29.9 s clip | 8 | 205.15 | 217.40 | 38.16 | 1141.0 |
|
| 250 |
-
| audio_30s_b32 | 29.9 s clip | 32 | 776.22 | 840.38 | 40.11 | 1199.3 |
|
| 251 |
-
| audio_320s_b1 | 320 s clip | 1 | 254.87 | 271.85 | 3.83 | 1225.6 |
|
| 252 |
-
|
| 253 |
-
Host preprocessing per item, measured separately: image 3 to 5 ms, 8-frame video 29 ms, 30 s audio 8 to 10 ms, 320 s audio 130 ms.
|
| 254 |
-
|
| 255 |
-
### Server load test, stage 09 (policy x02, one p150-equivalent chip, grid 11x10)
|
| 256 |
-
|
| 257 |
-
Real uvicorn server, closed-loop clients, 0 errors in every run. p50 and p99 are the 50th and 99th percentile request latencies.
|
| 258 |
-
|
| 259 |
-
One text of 480 to 512 tokens per request (mean 499, bucket 512, no prompt). 60 s per level after a 5 s warm-up.
|
| 260 |
-
|
| 261 |
-
| Concurrency | requests/s | embeddings/s | p50 (ms) | p99 (ms) | mean inputs per device batch | device thread busy |
|
| 262 |
-
|---|---|---|---|---|---|---|
|
| 263 |
-
| 1 | 88.54 | 88.54 | 11.35 | 13.07 | 1.00 | 0.861 |
|
| 264 |
-
| 8 | 207.71 | 207.71 | 38.50 | 40.54 | 4.00 | 0.970 |
|
| 265 |
-
| 32 | 277.30 | 277.30 | 114.07 | 122.27 | 15.96 | 0.968 |
|
| 266 |
-
|
| 267 |
-
32 texts of 480 to 512 tokens per request, 30 s per level:
|
| 268 |
-
|
| 269 |
-
| Concurrency | requests/s | embeddings/s | p50 (ms) | p99 (ms) |
|
| 270 |
-
|---|---|---|---|---|
|
| 271 |
-
| 1 | 8.26 | 264.33 | 119.18 | 122.11 |
|
| 272 |
-
| 2 | 9.70 | 310.27 | 203.33 | 206.75 |
|
| 273 |
-
| 4 | 9.61 | 307.48 | 408.46 | 412.23 |
|
| 274 |
-
|
| 275 |
-
Mixed lengths: 75% texts of 16 to 64 tokens, 25% texts of 900 to 1000 tokens, one per request, 30 s per level. "Bucket grouping on" is the default. "Off" puts all pending requests in one batch padded to the longest.
|
| 276 |
-
|
| 277 |
-
| Concurrency | bucket grouping | requests/s | p50 (ms) | p99 (ms) | short p50 / p99 (ms) | long p50 / p99 (ms) |
|
| 278 |
-
|---|---|---|---|---|---|---|
|
| 279 |
-
| 8 | on (default) | 264.60 | 27.55 | 66.73 | 26.25 / 61.74 | 39.35 / 77.63 |
|
| 280 |
-
| 8 | off | 160.45 | 42.19 | 70.66 | 41.90 / 70.53 | 68.68 / 71.11 |
|
| 281 |
-
| 32 | on (default) | 374.65 | 80.78 | 170.56 | 75.46 / 152.88 | 106.27 / 197.15 |
|
| 282 |
-
| 32 | off | 126.85 | 254.54 | 262.02 | 254.45 / 262.16 | 254.81 / 261.46 |
|
| 283 |
-
|
| 284 |
-
The server is device-bound: the device thread is busy 97% of the time at concurrency 8 and 32.
|
| 285 |
-
|
| 286 |
-
## Accuracy against HF fp32
|
| 287 |
-
|
| 288 |
-
The reference is the HF sentence-transformers output in fp32 on the CPU (central processing unit). TT means this Tenstorrent implementation. Cosine is per input. "Top-10 agreement" is the mean over queries of |TT top 10 ∩ HF top 10| / 10 on the 256-input stage 00 text corpus (64 queries x 192 documents), unless stated. MTEB is the Massive Text Embedding Benchmark; mini-MTEB is a 3-task regression set (STS17, AskUbuntuDupQuestions, ArguAna).
|
| 289 |
-
|
| 290 |
-
### Text
|
| 291 |
-
|
| 292 |
-
| Stage | Policy, path | Inputs | Cosine min / mean (768) | Top-10 agreement |
|
| 293 |
-
|---|---|---|---|---|
|
| 294 |
-
| T6 | T6 policy, eager | 256-input corpus | 0.999932 / 0.999980 | 0.9906 |
|
| 295 |
-
| T7 | c01, traced | 256-input corpus | 0.999904 / 0.999979 | 0.9969 |
|
| 296 |
-
| 08 | x02, traced | worst of the 256-input corpus (batch 32 and 1), 17490 mini-MTEB inputs, 9920 ArguAna texts at batch 1 | 0.996364 / 0.999753 | 0.984375 (corpus), 0.980014 / 0.981442 (ArguAna, default batching / batch 1) |
|
| 297 |
-
| 09 | x02, server, 32 concurrent requests, three runs | 256-input corpus | 0.999245 to 0.999264 / 0.999813 to 0.999816 | 0.982813 to 0.985938 |
|
| 298 |
-
| 11 | x02, evaluation worker, eval batching | stage 11 text inputs | min 0.9790 (8 inputs below 0.995); mean not split by modality (all mteb pairs 0.99980) | not measured (benchmark scores in Evaluation) |
|
| 299 |
-
|
| 300 |
-
- ArguAna uses 1401 queries x 8626 documents.
|
| 301 |
-
- Under x02, 24 of the 17490 mini-MTEB inputs and 12 of the 9920 ArguAna texts are below 0.999 cosine. The stage gate is 0.995.
|
| 302 |
-
- In the stage 11 evaluation, 8 text inputs are below the 0.995 gate (minimum 0.9790, a 16-character classification query) and one VisDoc image+text page is at 0.9913. Under policy c01 the same inputs give 0.9989 or higher. Details: `doc/eval/REPORT.md`, section "Per-input outliers below 0.995".
|
| 303 |
-
- Mini-MTEB mean of the 3 tasks: x02 63.1032, HF fp32 on CPU 63.1473. This is not the stage 11 evaluation.
|
| 304 |
-
- T7 MRL, traced, c01, cosine min at 512 / 256 / 128 dimensions: 0.999907 / 0.999919 / 0.999946. Top-10 agreement at 512 / 256 / 128: 0.9938 / 0.9953 / 0.9953.
|
| 305 |
-
|
| 306 |
-
### Image, video and audio
|
| 307 |
-
|
| 308 |
-
| Stage | Policy | Modality | Inputs | Cosine min / mean (768) |
|
| 309 |
-
|---|---|---|---|---|
|
| 310 |
-
| 06 | c01, traced | image | 21 (11 default, 10 at budgets 70 to 1120) | 0.999553 / 0.999895 |
|
| 311 |
-
| 06 | c01, traced | image + caption | 11 | 0.999870 / 0.999940 |
|
| 312 |
-
| 06 | c01, traced | 29 images in one input | 1 | 0.999932 |
|
| 313 |
-
| 06 | c01, traced | video | 6 (4 to 114 frames) | 0.999934 / 0.999950 |
|
| 314 |
-
| 06 | c01, traced | video + caption | 3 | 0.999943 (min) |
|
| 315 |
-
| 06 | c01, traced | audio | 10 (1 s to 320 s, untruncated) | 0.999424 / 0.999896 |
|
| 316 |
-
| 06 | c01, traced | audio + caption | 7 | 0.999880 (min) |
|
| 317 |
-
| 08 | x02, traced | image | stage 06 long set, 21 | 0.999020 / 0.999720 |
|
| 318 |
-
| 08 | x02, traced | video | stage 06 long set, 6 | 0.999764 / 0.999849 |
|
| 319 |
-
| 08 | x02, traced | audio | stage 06 long set, 10 | 0.998871 / 0.999639 |
|
| 320 |
-
|
| 321 |
-
- Cross-modal top-1 agreement with HF (caption to image, video and audio): 1.0, 1.0, 1.0 in stage 06 (c01) and in stage 08 (x02).
|
| 322 |
-
- Stage 06 MRL, traced golden-tensor path, cosine min at 768 / 512 / 256 / 128: image 0.999786 / 0.9998 / 0.999814 / 0.999864; image + caption 0.99987 / 0.999879 / 0.999883 / 0.99992; video 0.999948 / 0.999948 / 0.999956 / 0.999971; audio 0.999424 / 0.999454 / 0.999577 / 0.999739.
|
| 323 |
-
- Stage 09 server vs direct embedder on the same device batch: cosine min 1.0 for text (122 cases), image (22), audio (10) and video (1).
|
| 324 |
-
|
| 325 |
-
## Evaluation
|
| 326 |
-
|
| 327 |
-
"TT p150 (subset)" and "HF reference (subset)" are scores on the fixed subset `eval/subset.json` of each benchmark, run with identical inputs, prompts and scoring code on this package (one p150-equivalent chip, grid 11x10, policy x02) and on the HF fp32 reference on CPU. "card (full)" is the full-suite score from the [google/embeddinggemma-2](https://huggingface.co/google/embeddinggemma-2) model card. Subset means are not comparable to full-suite means. The subset exists to compare TT against HF on the same inputs. Per-task scores, deltas and the method are in `doc/eval/REPORT.md` (also `doc/eval/REPORT.html`). The evaluation follows the subset input policy: audio truncation off, clips up to 320 s. For that, the evaluation worker adds one audio trace slot (batch 1, 8192 tokens, 320 s) to the slots this server captures. The server itself truncates audio to 30 s, as described in the API section. No benchmark mean differs between TT and HF by more than 1.0 point at any dimension.
|
| 328 |
-
|
| 329 |
-
Benchmark names: MIEB (Massive Image Embedding Benchmark), MMEB (Massive Multimodal Embedding Benchmark), MSEB (Massive Sound Embedding Benchmark), MAEB (Massive Audio Embedding Benchmark). Metrics: NDCG (normalized discounted cumulative gain), MRR (mean reciprocal rank).
|
| 330 |
-
|
| 331 |
-
### Overall Evaluation Results (768d)
|
| 332 |
-
|
| 333 |
-
| Modality | Benchmark | Metric | TT p150 (subset) | HF reference (subset) | card (full) |
|
| 334 |
-
| :---- | :---- | :---- | :---- | :---- | :---- |
|
| 335 |
-
| *Text* | Massive Text Embedding Benchmark (MTEB, multilingual, v2) | Mean(Task), Multiple | 59.55 | 59.51 | 61.36 |
|
| 336 |
-
| | Massive Text Embedding Benchmark (MTEB, code, v1) | Mean(Task), NDCG@10 | 56.99 | 57.04 | 78.68 |
|
| 337 |
-
| *Image* | Massive Image Embedding Benchmark (MIEB, lite) | Mean(TaskType), Multiple | 60.93 | 60.50 | 64.64 |
|
| 338 |
-
| | Massive Multimodal Embedding Benchmark (MMEB v2 \- Image) | Mean(Task), Hit@1 | 38.85 | 39.10 | 57.28 |
|
| 339 |
-
| | Massive Multimodal Embedding Benchmark (MMEB v2 \- VisDoc) | Mean(Task), NDCG@5 | 70.26 | 69.39 | 67.84 |
|
| 340 |
-
| *Video* | Massive Multimodal Embedding Benchmark (MMEB v2 \- Video) | Mean(Task), Hit@1 | 20.09 | 19.63 | 50.67 |
|
| 341 |
-
| *Audio* | Massive Sound Embedding Benchmark (MSEB, Retrieval) | Mean(Task), MRR@10 | 80.20 | 80.18 | 69.54 |
|
| 342 |
-
| | Massive Audio Embedding Benchmark (MAEB) *Hugging Face* | Mean(Task), Multiple | 58.16 | 58.28 | 49.39 |
|
| 343 |
-
|
| 344 |
-
### Evaluation Results with Vector Truncation
|
| 345 |
-
|
| 346 |
-
Each embedding is computed once at 768d, truncated to the first 512, 256 or 128 dimensions and L2 re-normalized.
|
| 347 |
-
|
| 348 |
-
| Output Dimension | Compression Ratio | MTEB *(multilingual, v2) Mean(Task)* | MTEB *(eng, v2) Mean(Task)* | MTEB *(code, v1) Mean(Task)* | MIEB *(lite) Mean(TaskType)* | MMEB (v2) *Overall* | MSEB *(Retrieval) Mean(Task)* | MAEB *Mean(Task)* |
|
| 349 |
-
| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 350 |
-
| 768d (Full), TT p150 (subset) | 1:1 | 59.55 | 70.25 | 56.99 | 60.93 | 44.19 | 80.20 | 58.16 |
|
| 351 |
-
| 768d (Full), HF reference (subset) | 1:1 | 59.51 | 70.35 | 57.04 | 60.50 | 43.93 | 80.18 | 58.28 |
|
| 352 |
-
| 768d (Full), card (full) | 1:1 | 61.36 | 68.46 | 78.68 | 64.64 | 59.01 | 69.54 | 49.39 |
|
| 353 |
-
| 512d, TT p150 (subset) | 1:1.5 | 59.20 | 70.24 | 56.62 | 60.77 | 44.70 | 78.99 | 57.93 |
|
| 354 |
-
| 512d, HF reference (subset) | 1:1.5 | 59.20 | 70.27 | 56.59 | 60.49 | 44.12 | 79.15 | 58.05 |
|
| 355 |
-
| 512d, card (full) | 1:1.5 | 61.17 | 68.41 | 77.24 | 64.32 | 58.38 | 69.18 | 49.21 |
|
| 356 |
-
| 256d, TT p150 (subset) | 1:3 | 58.61 | 69.54 | 55.72 | 59.50 | 42.37 | 76.26 | 57.84 |
|
| 357 |
-
| 256d, HF reference (subset) | 1:3 | 58.60 | 69.60 | 55.72 | 59.75 | 42.15 | 76.48 | 57.87 |
|
| 358 |
-
| 256d, card (full) | 1:3 | 60.41 | 67.78 | 76.18 | 63.13 | 56.24 | 66.76 | 48.91 |
|
| 359 |
-
| 128d, TT p150 (subset) | 1:6 | 55.76 | 67.90 | 52.30 | 55.57 | 37.12 | 60.25 | 55.25 |
|
| 360 |
-
| 128d, HF reference (subset) | 1:6 | 55.76 | 67.93 | 52.13 | 55.58 | 37.44 | 60.66 | 55.21 |
|
| 361 |
-
| 128d, card (full) | 1:6 | 57.89 | 65.68 | 71.41 | 59.06 | 45.65 | 56.71 | 46.92 |
|
| 362 |
|
| 363 |
-
##
|
| 364 |
|
| 365 |
-
|
| 366 |
-
- **8192-token cap.** Each input can have at most 8192 tokens, counting the prompt, `<bos>`, `<eos>` and all media soft tokens. Longer inputs get `400 context_length_exceeded`. The untraced code path runs longer text, but the server uses traces only.
|
| 367 |
-
- **Numerics.** The model runs in BF16 and BFP8 with fp32 accumulation and an fp32 output. It never uses fp16 (16-bit half-precision floating point). The base model card warns that fp16 gives NaN (not a number) or degraded embeddings.
|
| 368 |
-
- **Batching drift.** An embedding depends on the shape of the device batch it runs in. In stage 09, batched vs single-input cosine min was 0.999382 for text (over three test runs), 0.999872 for images and 1.0 for audio. Stage 11 measured larger drift on harder inputs. On 24 texts and 39 audio clips (the evaluation's lowest-agreement inputs plus random controls), batch 8 vs batch 1 cosine min was 0.9809 for text and 0.9822 for audio (audio batch 8 slot vs batch 1 slot). Under policy c01 the same inputs give 0.9993 and 0.9973. The drift is a property of the x02 policy as a whole. For text, x02's BFP8 activations alone give part of it at batch 8 (cosine 0.9947); the split between factors depends on the batch shape. For audio, neither the x02 text settings nor the x02 audio settings alone reach the full drift; they interact (`doc/eval/outliers/drift_attribution.json`). A text vector depends on the batch size and on the padded length bucket of its batch, and the longest input in the batch sets that bucket: one longer text in the same batch moved a short text to cosine 0.9841. An audio vector depends on the slot shape (batch 1 or batch 8), not on the other clips in the batch. Results are deterministic for one batch composition, but not across load patterns. For repeatable vectors, send one input per request with no concurrency.
|
| 369 |
-
- **Audio precision.** A few token rows in the last audio layers amplify small numeric errors, and the result depends on the clip. HF's own bf16 model shows the same effect on some clips. In the stage 11 evaluation, 37 audio inputs have cosine below 0.995 to HF fp32 (minimum 0.959). Under this package's policy x02, most of them disappear when the audio tower uses the policy c01 audio settings: 15 remain at batch 8 instead of 35. BF16 attention internals in the audio tower add error too. At 768d the audio benchmark means stay within 0.12 points of HF fp32 (MAEB, MSEB); truncated dims differ by up to 0.41 points (MSEB at 128d). Details: `doc/eval/outliers/classification.md`.
|
| 370 |
-
- **Text precision.** Policy x02 does not hold 0.995 cosine to HF fp32 on every text input. In the stage 11 evaluation, 8 texts are below 0.995 (minimum 0.9790, mostly short inputs) and one VisDoc page is at 0.9913. Policy c01 gives 0.9989 or higher on the same inputs. The image ships only x02. A direct run from the source tree selects another policy with `EMBEDDINGGEMMA2_PRECISION_POLICY=<path to a JSON file in doc/datatype_sweep/policies/>`. Passing that variable and file into the `tt-model serve` container was not tested. Text benchmark means stay within 0.10 points of HF fp32 at 768d.
|
| 371 |
-
- **Top-10 margin.** Retrieval top-10 agreement with HF fp32 is close to the 0.98 bar. On the stage 00 corpus the lowest server value is 0.982813, which is 2 of 640 top-10 slots above the bar. On ArguAna at default batching x02 gives 0.980014 (13730 of 14010 slots, 13729.8 needed).
|
| 372 |
-
- **First boot.** The first boot compiles kernels and captures 51 traces before the server is ready (360 s measured; 30 s on a later boot that reuses the kernel cache).
|
| 373 |
-
- **Images.** The image budget is fixed at 280 soft tokens (the processor default). Other budgets (70 to 1120) need extra trace slots that the package does not capture.
|
| 374 |
-
- **Video.** The server takes decoded frames, not video files. It uses at most 32 frames. Video slots hold up to 8 frames at 2048 tokens and 32 frames at 8192 tokens.
|
| 375 |
-
- **Audio.** Audio must be 16 kHz WAV. The server does not resample. Clips longer than 30 s are truncated to 30 s.
|
| 376 |
-
- **Media per item.** Each input item holds at most one media kind. Media that does not fit a captured trace slot gets a 400 error.
|
| 377 |
-
- **Scheduling.** One device thread serves all requests. A large request holds the device for its full duration. There is no per-request time limit, priority or preemption. With bucket grouping, short requests can wait behind a long-bucket batch (mixed load at concurrency 32: short-request p99 152.88 ms vs p50 75.46 ms).
|
| 378 |
-
- **Request size.** The app sets no HTTP (Hypertext Transfer Protocol) body-size limit. Put a limit in a reverse proxy.
|
| 379 |
-
- **Base model limits.** The limitations and ethical considerations of the base model card apply.
|
| 380 |
|
| 381 |
-
##
|
| 382 |
|
| 383 |
-
|
| 384 |
|
| 385 |
-
##
|
| 386 |
|
| 387 |
-
|
| 388 |
-
- Code: Apache 2.0 (tt-metal and the model directory).
|
| 389 |
|
| 390 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 391 |
|
| 392 |
-
> Google DeepMind. EmbeddingGemma 2. Hugging Face model `google/embeddinggemma-2`, revision `914f7f89142e33e77833254d9c9b90c3cef7303b`. https://huggingface.co/google/embeddinggemma-2
|
|
|
|
| 1 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
tags:
|
|
|
|
| 3 |
- blackhole
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4 |
- p150
|
|
|
|
|
|
|
| 5 |
- tt-dit-server
|
|
|
|
| 6 |
- tt-model-cache
|
|
|
|
| 7 |
- tt-model-container
|
| 8 |
+
license: apache-2.0
|
| 9 |
+
pipeline_tag: feature-extraction
|
| 10 |
+
base_model:
|
| 11 |
+
- google/embeddinggemma-2
|
| 12 |
---
|
| 13 |
|
| 14 |
+
# embeddinggemma-2-p150 on Tenstorrent p150
|
|
|
|
|
|
|
|
|
|
|
|
|
| 15 |
|
| 16 |
+
> Derived from google/embeddinggemma-2 by Google DeepMind, weights unmodified. The TTNN port is models/autoports/google_embeddinggemma_2, on stock upstream tt-metal.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 17 |
|
| 18 |
+
EmbeddingGemma 2, a multimodal embedding model. Text, code, images, video frames and 16 kHz audio each map to one L2-normalized 768-dimensional vector in a shared space, so a text query can find an image, a sound or a video. Matryoshka dimensions 512, 256 and 128 are the first values of the 768-dimensional vector, re-normalized on the chip.
|
| 19 |
|
| 20 |
+
The text tower is a bidirectional 24-layer Gemma encoder (hidden size 512, sliding and full attention, 8192-token context). Gemma 4 vision (16 layers) and audio (12 layers) towers feed soft tokens into it. Weights stay on the chip and every request shape runs as a captured TTNN trace. Two custom kernels (sliding-window SDPA and a GEGLU matmul) run through stock `ttnn.generic_op`; tt-metal itself is unmodified.
|
| 21 |
|
| 22 |
+
Runs on Tenstorrent **p150** — see the serving profiles below.
|
| 23 |
|
| 24 |
+
Packaged and published with [tt-model-manager](https://github.com/tenstorrent/tt-model-manager) 0.1.0 (manifest schema 5.1).
|
| 25 |
|
| 26 |
+
## At a glance
|
|
|
|
|
|
|
| 27 |
|
| 28 |
+
| | |
|
| 29 |
+
| --- | --- |
|
| 30 |
+
| Hardware | p150 |
|
| 31 |
+
| License | apache-2.0 |
|
| 32 |
|
| 33 |
+
## Intended use
|
|
|
|
|
|
|
|
|
|
| 34 |
|
| 35 |
+
Embeddings for retrieval, semantic search, clustering and classification over text, code, images, video and audio, served from one p150. Not for generating text.
|
|
|
|
|
|
|
|
|
|
| 36 |
|
| 37 |
+
## Prerequisites
|
| 38 |
|
| 39 |
+
- The [Tenstorrent CLI](https://github.com/tenstorrent/tt-cli/blob/main/docs/prerequisites.md), `tt`: `uv tool install tenstorrent` (or use tt-model alone, see the Quickstart)
|
| 40 |
+
- Docker
|
| 41 |
+
- Tenstorrent hardware: **p150**
|
| 42 |
+
- 1 GB hugepages mounted at `/dev/hugepages-1G`
|
| 43 |
+
- One chip of a p300 also works: the package sets the p150 mesh graph descriptor.
|
| 44 |
|
| 45 |
+
## Quickstart
|
| 46 |
|
| 47 |
```bash
|
| 48 |
+
tt serve tt-hous/embeddinggemma-2-p150
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 49 |
```
|
| 50 |
|
| 51 |
+
`tt serve` downloads the Docker image and the [`google/embeddinggemma-2`](https://huggingface.co/google/embeddinggemma-2) at `914f7f89142e33e77833254d9c9b90c3cef7303b` weights (into your HF cache; they are not in the image), then starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs `Application startup complete`.
|
| 52 |
|
| 53 |
+
Without tt-cli: `tt-model serve tt-hous/embeddinggemma-2-p150`.
|
| 54 |
|
| 55 |
+
### Embedding text
|
| 56 |
```bash
|
| 57 |
curl -s localhost:20000/v1/embeddings -H 'content-type: application/json' \
|
| 58 |
-d '{"input": ["What is the capital of France?"], "input_type": "query", "dimensions": 256}'
|
| 59 |
```
|
| 60 |
|
| 61 |
+
### Demo
|
|
|
|
| 62 |
```bash
|
| 63 |
+
tt serve tt-hous/embeddinggemma-2-p150 --profile demo
|
|
|
|
|
|
|
| 64 |
```
|
| 65 |
+
Open `http://<host>:20000/demo/`. A web page where every query is embedded live on the chip, over a bundled library of 163 openly licensed images, videos, sounds, texts in 10 languages and code snippets: media search, finding a moment in a long video, audio search, code search, multilingual search, and a side-by-side Matryoshka comparison.
|
| 66 |
|
| 67 |
+
## Serving profiles
|
| 68 |
|
| 69 |
+
Select a different serving profile using `--profile`.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 70 |
|
| 71 |
+
| profile | hardware | mesh | description |
|
| 72 |
+
| --- | --- | --- | --- |
|
| 73 |
+
| `default` *(default)* | p150 | P150 | OpenAI-style /v1/embeddings server only. |
|
| 74 |
+
| `demo` | p150 | P150 | The /v1/embeddings server plus a Gradio demo at /demo (media, video moments, audio, code, multilingual and Matryoshka search over a bundled openly licensed library). |
|
| 75 |
|
| 76 |
+
## Capabilities
|
|
|
|
|
|
|
|
|
|
|
|
|
| 77 |
|
| 78 |
+
This package exposes the model's own HTTP server on port 20000, **not** an OpenAI-compatible chat API — its request and response shapes are the model's own. See the author's notes for the payload it expects.
|
| 79 |
|
| 80 |
+
`POST /v1/embeddings` follows the OpenAI embeddings request and response shapes, with media extensions.
|
| 81 |
|
| 82 |
+
| field | meaning | default |
|
|
|
|
|
|
|
| 83 |
|---|---|---|
|
| 84 |
+
| `input` | a string, or a list of 1 to 2048 strings and media items | required |
|
| 85 |
+
| `dimensions` | 768, 512, 256 or 128 | 768 |
|
| 86 |
+
| `input_type` | a sentence-transformers prompt name, for example `query`, `document`, `CodeRetrieval` | no prompt |
|
| 87 |
+
| `encoding_format` | `float`, or `base64` (little-endian float32) | `float` |
|
| 88 |
+
| `model` | optional; `google/embeddinggemma-2` or `tt-hous/embeddinggemma-2-p150` | |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 89 |
|
| 90 |
+
A media item is an object with optional `text` and at most one media field:
|
| 91 |
|
| 92 |
+
| field | value |
|
| 93 |
+
|---|---|
|
| 94 |
+
| `image` | base64 image bytes (any PIL format) or a `data:image/...;base64,` URL |
|
| 95 |
+
| `audio` | base64 16 kHz WAV; clips over 30 s are cut to 30 s |
|
| 96 |
+
| `video` | `{"frames": [<base64 image>, ...], "fps": 1}`; at most 32 frames are used |
|
| 97 |
|
| 98 |
+
```bash
|
| 99 |
+
IMG=$(base64 -w0 cats.jpg)
|
| 100 |
+
curl -s localhost:20000/v1/embeddings -H 'content-type: application/json' \
|
| 101 |
+
-d "{\"input\": [{\"image\": \"$IMG\"}, {\"text\": \"two cats on a couch\"}]}"
|
| 102 |
+
```
|
| 103 |
|
| 104 |
+
`GET /health` answers 200 once the traces are captured, with the precision policy and live counters, and 503 before that. `GET /v1/models` lists the model, its dimensions and modalities. Errors use the OpenAI error shape; an input over 8192 tokens gets 400 `context_length_exceeded`. Media URLs are refused: send media as base64.
|
| 105 |
+
|
| 106 |
+
## Expected performance
|
| 107 |
+
|
| 108 |
+
| benchmark (768d) | metric | this package | HF fp32 | card |
|
| 109 |
+
| --- | --- | ---: | ---: | ---: |
|
| 110 |
+
| MTEB multilingual v2 | Mean(Task) | 59.55 | 59.51 | 61.36 |
|
| 111 |
+
| MTEB eng v2 | Mean(Task) | 70.25 | 70.35 | 68.46 |
|
| 112 |
+
| MTEB code v1 | Mean(Task), NDCG@10 | 56.99 | 57.04 | 78.68 |
|
| 113 |
+
| MIEB lite | Mean(TaskType) | 60.93 | 60.50 | 64.64 |
|
| 114 |
+
| MMEB v2 Image | Mean(Task), Hit@1 | 38.85 | 39.10 | 57.28 |
|
| 115 |
+
| MMEB v2 VisDoc | Mean(Task), NDCG@5 | 70.26 | 69.39 | 67.84 |
|
| 116 |
+
| MMEB v2 Video | Mean(Task), Hit@1 | 20.09 | 19.63 | 50.67 |
|
| 117 |
+
| MSEB Retrieval | Mean(Task), MRR@10 | 80.20 | 80.18 | 69.54 |
|
| 118 |
+
| MAEB | Mean(Task) | 58.16 | 58.28 | 49.39 |
|
| 119 |
+
|
| 120 |
+
| dimensions | storage | MTEB multilingual | MTEB eng | MTEB code | MIEB lite | MMEB overall | MSEB | MAEB |
|
| 121 |
+
| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 122 |
+
| 768 | 1x | 59.55 | 70.25 | 56.99 | 60.93 | 44.19 | 80.20 | 58.16 |
|
| 123 |
+
| 512 | 1/1.5 | 59.20 | 70.24 | 56.62 | 60.77 | 44.70 | 78.99 | 57.93 |
|
| 124 |
+
| 256 | 1/3 | 58.61 | 69.54 | 55.72 | 59.50 | 42.37 | 76.26 | 57.84 |
|
| 125 |
+
| 128 | 1/6 | 55.76 | 67.90 | 52.30 | 55.57 | 37.12 | 60.25 | 55.25 |
|
| 126 |
+
|
| 127 |
+
| load, one text of 480 to 512 tokens per request | requests/s | p50 (ms) | p99 (ms) |
|
| 128 |
+
| --- | ---: | ---: | ---: |
|
| 129 |
+
| 1 user | 88.5 | 11.4 | 13.1 |
|
| 130 |
+
| 8 users | 207.7 | 38.5 | 40.5 |
|
| 131 |
+
| 32 users | 277.3 | 114.1 | 122.3 |
|
| 132 |
+
|
| 133 |
+
| single request, warm | latency (ms) |
|
| 134 |
+
| --- | ---: |
|
| 135 |
+
| text, 128 tokens | 5.8 |
|
| 136 |
+
| text, 512 tokens | 7.7 |
|
| 137 |
+
| text, 8192 tokens | 80.3 |
|
| 138 |
+
| image | 47.8 |
|
| 139 |
+
| video, 8 frames | 138.1 |
|
| 140 |
+
| audio, 30 s | 32.2 |
|
| 141 |
+
|
| 142 |
+
Measured on one Blackhole chip of a p300c board opened as a 1x1 mesh (equal to a p150, compute grid 11x10), precision policy x02. Accuracy: a fixed subset of each benchmark, scored with the same inputs, prompts and code on this package and on the Hugging Face fp32 reference on CPU; "card" is the full-suite score from the google/embeddinggemma-2 card and is not comparable to subset means. No benchmark mean differs from HF fp32 by more than 1.0 point at any dimension. Load: a real server with closed-loop clients, 60 s per level, 0 errors. Single request: host staging, trace replay and readback, without tokenization or media decoding.
|
| 143 |
|
| 144 |
+
## Limitations
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 145 |
|
| 146 |
+
- Bring-up build, not a Tenstorrent release.
|
| 147 |
+
- One chip per server; no multi-chip mode.
|
| 148 |
+
- At most 8192 tokens per input, counting the prompt and media tokens.
|
| 149 |
+
- A vector depends slightly on the device batch it ran in (cosine down to 0.98 on hard inputs). Send one input per request for repeatable vectors.
|
| 150 |
+
- Against HF fp32, 8 of the evaluation texts and 37 audio clips fall below 0.995 cosine (minimum 0.959); benchmark means still match within 1.0 point.
|
| 151 |
+
- Audio must be 16 kHz WAV and is cut to 30 s. Video arrives as decoded frames, at most 32. Images use the default 280-token budget.
|
| 152 |
+
- Never run in fp16: the base model card warns it gives NaN or degraded vectors.
|
| 153 |
+
- The first boot compiles kernels and captures 51 traces (about 6 minutes); later boots take about 30 s.
|
| 154 |
+
- Validated on one chip of a p300c. Performance on a p150 with a larger compute grid was not measured.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 155 |
|
| 156 |
+
## Licensing
|
| 157 |
|
| 158 |
+
Weights under the upstream Apache-2.0 licence (google/embeddinggemma-2). The TTNN port and the server in `code/` are Apache-2.0 (tt-metal). Demo library items carry their own CC0, public domain, CC BY, CC BY-SA, Apache-2.0 or MIT licences, listed in `code/models/autoports/google_embeddinggemma_2/demo/library/LICENSES.md`.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 159 |
|
| 160 |
+
## Feedback
|
| 161 |
|
| 162 |
+
Questions or problems with this package: open a discussion at https://huggingface.co/tt-hous/embeddinggemma-2-p150/discussions — that is what reaches its author. A problem with the `tt` tooling itself: `tt report issue` (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.
|
| 163 |
|
| 164 |
+
## Provenance
|
| 165 |
|
| 166 |
+
The exact sources the image was built from — `code/` in this repo is byte-identical to the model code inside the image:
|
|
|
|
| 167 |
|
| 168 |
+
| component | built from |
|
| 169 |
+
| --- | --- |
|
| 170 |
+
| tt-metal | [`3be67656b28e3902f0366ea8bab03c41e7e5bd6c`](https://github.com/tenstorrent/tt-metal/commit/3be67656b28e3902f0366ea8bab03c41e7e5bd6c) |
|
| 171 |
+
| `code/` digest | `852ada085de3df45` (sha256, first 16 hex digits) |
|
| 172 |
+
| built | 2026-10-21T15:21:44+00:00 by tt-model 0.1.0 |
|
| 173 |
|
|
|
image/blobs/sha256/77d9d1b7c850b7a6eb0c05b01a5e1c25759ade47988603675dfbda02a4140e94
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"schemaVersion": 2,
|
| 3 |
+
"mediaType": "application/vnd.oci.image.manifest.v1+json",
|
| 4 |
+
"config": {
|
| 5 |
+
"mediaType": "application/vnd.oci.image.config.v1+json",
|
| 6 |
+
"digest": "sha256:8dd29ae163e363976b67e217870421858f30c1e1d287d2fd40844b2c03f5efad",
|
| 7 |
+
"size": 167
|
| 8 |
+
},
|
| 9 |
+
"layers": [
|
| 10 |
+
{
|
| 11 |
+
"mediaType": "application/vnd.in-toto+json",
|
| 12 |
+
"digest": "sha256:7ae19dfcc4ff33d65c67f8e32b090d68171b0f56dd714adae656191c6bfae038",
|
| 13 |
+
"size": 1916,
|
| 14 |
+
"annotations": {
|
| 15 |
+
"in-toto.io/predicate-type": "https://slsa.dev/provenance/v1"
|
| 16 |
+
}
|
| 17 |
+
}
|
| 18 |
+
]
|
| 19 |
+
}
|
image/blobs/sha256/7ae19dfcc4ff33d65c67f8e32b090d68171b0f56dd714adae656191c6bfae038
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"_type":"https://in-toto.io/Statement/v0.1","predicateType":"https://slsa.dev/provenance/v1","subject":[{"name":"pkg:docker/tt-model/embeddinggemma-2-p150@build-3be67656b?platform=linux%2Famd64","digest":{"sha256":"2d7d53b4b0c7f7d9c42ef8da860f263f57d7d576711e20fc9f25fed4fb62c286"}}],"predicate":{"buildDefinition":{"buildType":"https://github.com/moby/buildkit/blob/master/docs/attestations/slsa-definitions.md","resolvedDependencies":[{"uri":"pkg:docker/docker/dockerfile@1.7-labs","digest":{"sha256":"b99fecfe00268a8b556fad7d9c37ee25d716ae08a5d7320e6d51c4dd83246894"}},{"uri":"pkg:docker/ubuntu@22.04?platform=linux%2Famd64","digest":{"sha256":"0a9887931b0e38af2e3487ed39e29436b0535c4f3f52e57745baaf79d1898956"}},{"uri":"pkg:docker/ghcr.io/tenstorrent/tt-metal/tt-metalium/ubuntu-22.04-dev-amd64@latest?platform=linux%2Famd64","digest":{"sha256":"16bcf2a1e76051d857a1bee935b589fa4a9b73ee1a9e87eee4ce063de3cd2448"}}],"externalParameters":{"configSource":{"path":"Dockerfile"},"request":{"frontend":"gateway.v0","args":{"cmdline":"docker/dockerfile:1.7-labs","context:metalsrc":"local:metalsrc","frontend.caps":"moby.buildkit.frontend.contexts+forward","sharedkey:localdir:metalsrc":"empty-metalsrc:98849442d91b4194","source":"docker/dockerfile:1.7-labs"},"locals":[{"name":"context"},{"name":"dockerfile"},{"name":"metalsrc"}],"root":{"configSource":{"path":"Dockerfile"},"request":{"args":{"context:metalsrc":"local:metalsrc","frontend.caps":"moby.buildkit.frontend.contexts+forward","sharedkey:localdir:metalsrc":"empty-metalsrc:98849442d91b4194"}}},"compatibilityVersion":20}},"internalParameters":{"builderPlatform":"linux/amd64"}},"runDetails":{"builder":{"id":""},"metadata":{"invocationId":"yg3eg6otkhbsygyxcezas1378","startedOn":"2026-10-21T15:21:44.771641092Z","finishedOn":"2026-10-21T15:21:45.268353002Z","buildkit_metadata":{},"buildkit_completeness":{"request":false,"resolvedDependencies":false}}}}}
|
image/blobs/sha256/8dd29ae163e363976b67e217870421858f30c1e1d287d2fd40844b2c03f5efad
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"architecture":"unknown","os":"unknown","config":{},"rootfs":{"type":"layers","diff_ids":["sha256:7ae19dfcc4ff33d65c67f8e32b090d68171b0f56dd714adae656191c6bfae038"]}}
|
image/blobs/sha256/bee93e165d00114d3ba68d68a650f6898a743bda5c6c00bff50c0540b20f25f4
ADDED
|
@@ -0,0 +1,28 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"schemaVersion": 2,
|
| 3 |
+
"mediaType": "application/vnd.oci.image.index.v1+json",
|
| 4 |
+
"manifests": [
|
| 5 |
+
{
|
| 6 |
+
"mediaType": "application/vnd.oci.image.manifest.v1+json",
|
| 7 |
+
"digest": "sha256:2d7d53b4b0c7f7d9c42ef8da860f263f57d7d576711e20fc9f25fed4fb62c286",
|
| 8 |
+
"size": 4692,
|
| 9 |
+
"platform": {
|
| 10 |
+
"architecture": "amd64",
|
| 11 |
+
"os": "linux"
|
| 12 |
+
}
|
| 13 |
+
},
|
| 14 |
+
{
|
| 15 |
+
"mediaType": "application/vnd.oci.image.manifest.v1+json",
|
| 16 |
+
"digest": "sha256:77d9d1b7c850b7a6eb0c05b01a5e1c25759ade47988603675dfbda02a4140e94",
|
| 17 |
+
"size": 564,
|
| 18 |
+
"annotations": {
|
| 19 |
+
"vnd.docker.reference.digest": "sha256:2d7d53b4b0c7f7d9c42ef8da860f263f57d7d576711e20fc9f25fed4fb62c286",
|
| 20 |
+
"vnd.docker.reference.type": "attestation-manifest"
|
| 21 |
+
},
|
| 22 |
+
"platform": {
|
| 23 |
+
"architecture": "unknown",
|
| 24 |
+
"os": "unknown"
|
| 25 |
+
}
|
| 26 |
+
}
|
| 27 |
+
]
|
| 28 |
+
}
|
image/index.json
CHANGED
|
@@ -1 +1 @@
|
|
| 1 |
-
{"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[{"mediaType":"application/vnd.oci.image.index.v1+json","digest":"sha256:
|
|
|
|
| 1 |
+
{"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[{"mediaType":"application/vnd.oci.image.index.v1+json","digest":"sha256:bee93e165d00114d3ba68d68a650f6898a743bda5c6c00bff50c0540b20f25f4","size":856,"annotations":{"io.containerd.image.name":"docker.io/tt-model/embeddinggemma-2-p150:bee93e165d00","org.opencontainers.image.ref.name":"bee93e165d00"}}]}
|
image/manifest.json
CHANGED
|
@@ -1 +1 @@
|
|
| 1 |
-
[{"Config":"blobs/sha256/9a9e80da4de6221fd880dc96810376e07d22c2984bb2628ced2e8d0b9c8cd7de","RepoTags":["tt-model/embeddinggemma-2-p150:
|
|
|
|
| 1 |
+
[{"Config":"blobs/sha256/9a9e80da4de6221fd880dc96810376e07d22c2984bb2628ced2e8d0b9c8cd7de","RepoTags":["tt-model/embeddinggemma-2-p150:bee93e165d00"],"Layers":["blobs/sha256/cf1a5487baaf7d71eb78fd0888933c6cdecb40e59edb896511aa0ef4a896031e","blobs/sha256/11d2eac53ff4ff7d3d67057613e4f204fe7da40b8ba001e09a7b785a69827da3","blobs/sha256/35e14da4db1e99991e4ccc3a0d01584086548bae7120c5c368ae5c131f28121a","blobs/sha256/432d2195132d36370aca685dae5eb2d62628dd8f0214ba236618f84198620aa2","blobs/sha256/46713da844e7a0c5aa5f5c79e7270637bf751186078ec5bc260dd7b06006242a","blobs/sha256/451dddf7926009d9f2d3e471e6b27af1af62b2e86a6c061738bdab2b9a4c1a56","blobs/sha256/b5ec001ffd467834132e55397d625249decf5f5f63079307e34ae3acc00e8436","blobs/sha256/b749e800205c3ba3769ba8a199bd85d353f376391ad5f68f16da895d126b82d0","blobs/sha256/a476bdec81d16f5259e66edb2670af7383593c49a98b6cec2427117d756e11c0","blobs/sha256/aba408305578d428cf39127b2a6ce91037c83b6c33f08497174df12843f72b94","blobs/sha256/b65b506905faa6bdd4125549d97257e64c8a5715c11c9780b6a9b7678fd05f07","blobs/sha256/56e26571152696889aa2d2dca1568ff6c70030b12c08b5f196a577ea1cf1afea","blobs/sha256/59abbc84c8a3402b41571973de30f9d6f5e8aeac1919e155602e3570e12995fd","blobs/sha256/fa71074b04af7e9df6466c49c50e461c6f6e1d986900d94092fbb3994a598a53","blobs/sha256/4006fa78a01aa3b5f6051dca103138a25743d828059be8ee4dd4ecca184e78ff","blobs/sha256/75541aedee66d279bb9db97e12a0a82fef223f9104d3e3dcb259a8104e489d8a","blobs/sha256/9a7118cb3c11a1cd38db3e354ad23fefe52a05396021c832422810cbc832650b","blobs/sha256/3751585d3eb5789fa5fec521a6176185ab2299273f215f825f6b0d54b873248e","blobs/sha256/f4ebe6bafc19cd6b0f41fc09321cd0b2676884ebd3b06d95deac3c2f0786fdc2","blobs/sha256/4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1","blobs/sha256/2b553ee3860d69cd0ca80aa40607f7b6f3bf5110b0aa428deeee6a57110f8ab0","blobs/sha256/c71cc42b78a54d49663258ad776fe0ceb6402a360b1b699e6d709464dfec3597","blobs/sha256/0f607516b935aed5184f195141354b0c6cd0bee121742159a513be94baaa48eb"]}]
|
tt_kernel_manifest.json
CHANGED
|
@@ -6,7 +6,7 @@
|
|
| 6 |
"device_count": 1,
|
| 7 |
"producer": {
|
| 8 |
"tt_kernel_version": "0.1.0",
|
| 9 |
-
"created_at": "2026-10-
|
| 10 |
"hostname": "qb2-120-p06t08"
|
| 11 |
},
|
| 12 |
"weights": {
|
|
@@ -28,8 +28,8 @@
|
|
| 28 |
"image": {
|
| 29 |
"registry": "hf",
|
| 30 |
"repository": "embeddinggemma-2-p150",
|
| 31 |
-
"tag": "tt-model/embeddinggemma-2-p150:
|
| 32 |
-
"digest": "sha256:
|
| 33 |
},
|
| 34 |
"kind": "tt-dit-server",
|
| 35 |
"runtime": {
|
|
@@ -87,7 +87,7 @@
|
|
| 87 |
"GRADIO_ANALYTICS_ENABLED": "False"
|
| 88 |
},
|
| 89 |
"name": "demo",
|
| 90 |
-
"description": "The /v1/embeddings server plus a Gradio demo at /demo (media, video moments, audio, code, multilingual and Matryoshka search over a bundled
|
| 91 |
}
|
| 92 |
],
|
| 93 |
"default_profile": "default",
|
|
@@ -102,10 +102,10 @@
|
|
| 102 |
"import gradio, json, models.autoports.google_embeddinggemma_2.demo.ui as u, models.autoports.google_embeddinggemma_2.demo.index as i; assert gradio.__version__ == '6.30.0', gradio.__version__; m = json.loads((i.LIBRARY_DIR / 'manifest.json').read_text()); missing = [x['file'] for x in m['items'] if x.get('file') and not (i.LIBRARY_DIR / x['file']).is_file()]; assert not missing, missing[:5]"
|
| 103 |
],
|
| 104 |
"built": {
|
| 105 |
-
"image": "tt-model/embeddinggemma-2-p150:
|
| 106 |
"repo": "tt-hous/embeddinggemma-2-p150",
|
| 107 |
"tt_model_version": "0.1.0",
|
| 108 |
-
"created_at": "2026-10-
|
| 109 |
"tt_metal": {
|
| 110 |
"sha": "3be67656b28e3902f0366ea8bab03c41e7e5bd6c",
|
| 111 |
"describe": null,
|
|
@@ -117,22 +117,22 @@
|
|
| 117 |
"pushed": null
|
| 118 |
},
|
| 119 |
"code_sha256": "852ada085de3df458409d2183b853e843234bd60acff1efb98eb3c32c62dd77d",
|
| 120 |
-
"image_digest": "sha256:
|
| 121 |
},
|
| 122 |
"card": {
|
| 123 |
-
"description": "
|
| 124 |
-
"attribution": "
|
| 125 |
-
"prerequisites": "- 1 GB hugepages mounted at /dev/hugepages-1G
|
| 126 |
-
"quickstart":
|
| 127 |
"architecture": null,
|
| 128 |
"status": null,
|
| 129 |
-
"intended_use": "
|
| 130 |
"out_of_scope_use": null,
|
| 131 |
-
"usage": "POST /v1/embeddings
|
| 132 |
-
"performance": "
|
| 133 |
-
"limitations": "Bring-up build, not a Tenstorrent release.
|
| 134 |
"risks": null,
|
| 135 |
-
"licensing": "Weights
|
| 136 |
"related": null,
|
| 137 |
"license": {
|
| 138 |
"id": "apache-2.0",
|
|
|
|
| 6 |
"device_count": 1,
|
| 7 |
"producer": {
|
| 8 |
"tt_kernel_version": "0.1.0",
|
| 9 |
+
"created_at": "2026-10-21T15:21:48.898834+00:00",
|
| 10 |
"hostname": "qb2-120-p06t08"
|
| 11 |
},
|
| 12 |
"weights": {
|
|
|
|
| 28 |
"image": {
|
| 29 |
"registry": "hf",
|
| 30 |
"repository": "embeddinggemma-2-p150",
|
| 31 |
+
"tag": "tt-model/embeddinggemma-2-p150:bee93e165d00",
|
| 32 |
+
"digest": "sha256:bee93e165d00114d3ba68d68a650f6898a743bda5c6c00bff50c0540b20f25f4"
|
| 33 |
},
|
| 34 |
"kind": "tt-dit-server",
|
| 35 |
"runtime": {
|
|
|
|
| 87 |
"GRADIO_ANALYTICS_ENABLED": "False"
|
| 88 |
},
|
| 89 |
"name": "demo",
|
| 90 |
+
"description": "The /v1/embeddings server plus a Gradio demo at /demo (media, video moments, audio, code, multilingual and Matryoshka search over a bundled openly licensed library)."
|
| 91 |
}
|
| 92 |
],
|
| 93 |
"default_profile": "default",
|
|
|
|
| 102 |
"import gradio, json, models.autoports.google_embeddinggemma_2.demo.ui as u, models.autoports.google_embeddinggemma_2.demo.index as i; assert gradio.__version__ == '6.30.0', gradio.__version__; m = json.loads((i.LIBRARY_DIR / 'manifest.json').read_text()); missing = [x['file'] for x in m['items'] if x.get('file') and not (i.LIBRARY_DIR / x['file']).is_file()]; assert not missing, missing[:5]"
|
| 103 |
],
|
| 104 |
"built": {
|
| 105 |
+
"image": "tt-model/embeddinggemma-2-p150:bee93e165d00",
|
| 106 |
"repo": "tt-hous/embeddinggemma-2-p150",
|
| 107 |
"tt_model_version": "0.1.0",
|
| 108 |
+
"created_at": "2026-10-21T15:21:44+00:00",
|
| 109 |
"tt_metal": {
|
| 110 |
"sha": "3be67656b28e3902f0366ea8bab03c41e7e5bd6c",
|
| 111 |
"describe": null,
|
|
|
|
| 117 |
"pushed": null
|
| 118 |
},
|
| 119 |
"code_sha256": "852ada085de3df458409d2183b853e843234bd60acff1efb98eb3c32c62dd77d",
|
| 120 |
+
"image_digest": "sha256:bee93e165d00114d3ba68d68a650f6898a743bda5c6c00bff50c0540b20f25f4"
|
| 121 |
},
|
| 122 |
"card": {
|
| 123 |
+
"description": "EmbeddingGemma 2, a multimodal embedding model. Text, code, images, video frames and 16 kHz audio each map to one L2-normalized 768-dimensional vector in a shared space, so a text query can find an image, a sound or a video. Matryoshka dimensions 512, 256 and 128 are the first values of the 768-dimensional vector, re-normalized on the chip.\n\nThe text tower is a bidirectional 24-layer Gemma encoder (hidden size 512, sliding and full attention, 8192-token context). Gemma 4 vision (16 layers) and audio (12 layers) towers feed soft tokens into it. Weights stay on the chip and every request shape runs as a captured TTNN trace. Two custom kernels (sliding-window SDPA and a GEGLU matmul) run through stock `ttnn.generic_op`; tt-metal itself is unmodified.\n",
|
| 124 |
+
"attribution": "Derived from google/embeddinggemma-2 by Google DeepMind, weights unmodified. The TTNN port is models/autoports/google_embeddinggemma_2, on stock upstream tt-metal.\n",
|
| 125 |
+
"prerequisites": "- 1 GB hugepages mounted at `/dev/hugepages-1G`\n- One chip of a p300 also works: the package sets the p150 mesh graph descriptor.\n",
|
| 126 |
+
"quickstart": "### Embedding text\n```bash\ncurl -s localhost:20000/v1/embeddings -H 'content-type: application/json' \\\n -d '{\"input\": [\"What is the capital of France?\"], \"input_type\": \"query\", \"dimensions\": 256}'\n```\n\n### Demo\n```bash\ntt serve tt-hous/embeddinggemma-2-p150 --profile demo\n```\nOpen `http://<host>:20000/demo/`. A web page where every query is embedded live on the chip, over a bundled library of 163 openly licensed images, videos, sounds, texts in 10 languages and code snippets: media search, finding a moment in a long video, audio search, code search, multilingual search, and a side-by-side Matryoshka comparison.\n",
|
| 127 |
"architecture": null,
|
| 128 |
"status": null,
|
| 129 |
+
"intended_use": "Embeddings for retrieval, semantic search, clustering and classification over text, code, images, video and audio, served from one p150. Not for generating text.\n",
|
| 130 |
"out_of_scope_use": null,
|
| 131 |
+
"usage": "`POST /v1/embeddings` follows the OpenAI embeddings request and response shapes, with media extensions.\n\n| field | meaning | default |\n|---|---|---|\n| `input` | a string, or a list of 1 to 2048 strings and media items | required |\n| `dimensions` | 768, 512, 256 or 128 | 768 |\n| `input_type` | a sentence-transformers prompt name, for example `query`, `document`, `CodeRetrieval` | no prompt |\n| `encoding_format` | `float`, or `base64` (little-endian float32) | `float` |\n| `model` | optional; `google/embeddinggemma-2` or `tt-hous/embeddinggemma-2-p150` | |\n\nA media item is an object with optional `text` and at most one media field:\n\n| field | value |\n|---|---|\n| `image` | base64 image bytes (any PIL format) or a `data:image/...;base64,` URL |\n| `audio` | base64 16 kHz WAV; clips over 30 s are cut to 30 s |\n| `video` | `{\"frames\": [<base64 image>, ...], \"fps\": 1}`; at most 32 frames are used |\n\n```bash\nIMG=$(base64 -w0 cats.jpg)\ncurl -s localhost:20000/v1/embeddings -H 'content-type: application/json' \\\n -d \"{\\\"input\\\": [{\\\"image\\\": \\\"$IMG\\\"}, {\\\"text\\\": \\\"two cats on a couch\\\"}]}\"\n```\n\n`GET /health` answers 200 once the traces are captured, with the precision policy and live counters, and 503 before that. `GET /v1/models` lists the model, its dimensions and modalities. Errors use the OpenAI error shape; an input over 8192 tokens gets 400 `context_length_exceeded`. Media URLs are refused: send media as base64.\n",
|
| 132 |
+
"performance": "| benchmark (768d) | metric | this package | HF fp32 | card |\n| --- | --- | ---: | ---: | ---: |\n| MTEB multilingual v2 | Mean(Task) | 59.55 | 59.51 | 61.36 |\n| MTEB eng v2 | Mean(Task) | 70.25 | 70.35 | 68.46 |\n| MTEB code v1 | Mean(Task), NDCG@10 | 56.99 | 57.04 | 78.68 |\n| MIEB lite | Mean(TaskType) | 60.93 | 60.50 | 64.64 |\n| MMEB v2 Image | Mean(Task), Hit@1 | 38.85 | 39.10 | 57.28 |\n| MMEB v2 VisDoc | Mean(Task), NDCG@5 | 70.26 | 69.39 | 67.84 |\n| MMEB v2 Video | Mean(Task), Hit@1 | 20.09 | 19.63 | 50.67 |\n| MSEB Retrieval | Mean(Task), MRR@10 | 80.20 | 80.18 | 69.54 |\n| MAEB | Mean(Task) | 58.16 | 58.28 | 49.39 |\n\n| dimensions | storage | MTEB multilingual | MTEB eng | MTEB code | MIEB lite | MMEB overall | MSEB | MAEB |\n| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n| 768 | 1x | 59.55 | 70.25 | 56.99 | 60.93 | 44.19 | 80.20 | 58.16 |\n| 512 | 1/1.5 | 59.20 | 70.24 | 56.62 | 60.77 | 44.70 | 78.99 | 57.93 |\n| 256 | 1/3 | 58.61 | 69.54 | 55.72 | 59.50 | 42.37 | 76.26 | 57.84 |\n| 128 | 1/6 | 55.76 | 67.90 | 52.30 | 55.57 | 37.12 | 60.25 | 55.25 |\n\n| load, one text of 480 to 512 tokens per request | requests/s | p50 (ms) | p99 (ms) |\n| --- | ---: | ---: | ---: |\n| 1 user | 88.5 | 11.4 | 13.1 |\n| 8 users | 207.7 | 38.5 | 40.5 |\n| 32 users | 277.3 | 114.1 | 122.3 |\n\n| single request, warm | latency (ms) |\n| --- | ---: |\n| text, 128 tokens | 5.8 |\n| text, 512 tokens | 7.7 |\n| text, 8192 tokens | 80.3 |\n| image | 47.8 |\n| video, 8 frames | 138.1 |\n| audio, 30 s | 32.2 |\n\nMeasured on one Blackhole chip of a p300c board opened as a 1x1 mesh (equal to a p150, compute grid 11x10), precision policy x02. Accuracy: a fixed subset of each benchmark, scored with the same inputs, prompts and code on this package and on the Hugging Face fp32 reference on CPU; \"card\" is the full-suite score from the google/embeddinggemma-2 card and is not comparable to subset means. No benchmark mean differs from HF fp32 by more than 1.0 point at any dimension. Load: a real server with closed-loop clients, 60 s per level, 0 errors. Single request: host staging, trace replay and readback, without tokenization or media decoding.\n",
|
| 133 |
+
"limitations": "- Bring-up build, not a Tenstorrent release.\n- One chip per server; no multi-chip mode.\n- At most 8192 tokens per input, counting the prompt and media tokens.\n- A vector depends slightly on the device batch it ran in (cosine down to 0.98 on hard inputs). Send one input per request for repeatable vectors.\n- Against HF fp32, 8 of the evaluation texts and 37 audio clips fall below 0.995 cosine (minimum 0.959); benchmark means still match within 1.0 point.\n- Audio must be 16 kHz WAV and is cut to 30 s. Video arrives as decoded frames, at most 32. Images use the default 280-token budget.\n- Never run in fp16: the base model card warns it gives NaN or degraded vectors.\n- The first boot compiles kernels and captures 51 traces (about 6 minutes); later boots take about 30 s.\n- Validated on one chip of a p300c. Performance on a p150 with a larger compute grid was not measured.\n",
|
| 134 |
"risks": null,
|
| 135 |
+
"licensing": "Weights under the upstream Apache-2.0 licence (google/embeddinggemma-2). The TTNN port and the server in `code/` are Apache-2.0 (tt-metal). Demo library items carry their own CC0, public domain, CC BY, CC BY-SA, Apache-2.0 or MIT licences, listed in `code/models/autoports/google_embeddinggemma_2/demo/library/LICENSES.md`.\n",
|
| 136 |
"related": null,
|
| 137 |
"license": {
|
| 138 |
"id": "apache-2.0",
|