8-bit build on main (the 4-bit build moves to the q4 branch)

d1ec9e9 verified 15 days ago

4.53 kB

---
library_name: torq
tags:
- Embedding
- Multimodal
- Astra SL
- SL2600
- NPU
license: apache-2.0
base_model:
- google/embeddinggemma-2
- onnx-community/embeddinggemma-2-ONNX
base_model_relation: quantized
---
# EmbeddingGemma-2

## Model Overview

EmbeddingGemma-2 is a multimodal embedding model from Google. It maps text, images, video and
audio into one 768-dimensional embedding space (Matryoshka: 768, 512, 256 or 128 dimensions),
so a text query can be compared directly with photos, video frames and sound clips.

This model has been optimized for the Synaptics Astra™ **SL2610-Series processors** with Torq NPU.
This branch holds the recommended build: **8-bit weights** (int8, block 32) with bf16
activations. Releases are tagged with the [torq-examples](https://github.com/synaptics-torq/torq-examples)
version (for example `v2.3.0`), and the demo downloads the tag of its own version.

| Build | Branch | Size | Notes |
|---|---|---|---|
| **8-bit** (recommended) | `main` (this branch), tags `v2.3.0` and later | 2.3 GB | Embeddings close to the original model (cosine 0.997 to fp32 over a media library); all modalities in one process |
| 4-bit | [`q4`](https://huggingface.co/Synaptics/Google-EmbeddingGemma-2/tree/q4) (and tag `v2.2.0`) | 1.1 GB | Smallest; less accurate (cosine 0.976) |
| bf16 | [`bf16`](https://huggingface.co/Synaptics/Google-EmbeddingGemma-2/tree/bf16) | 3.6 GB | Closest to the original model, most memory |

## Model Features
- **Model Type:** Multimodal embedding model
- **Input:** Text, image, video, audio
- **Output:** 768-d embedding

## How it runs

The vision and audio encoders turn an image, video frame or sound into soft tokens in the text
model's embedding space. The host writes them into the placeholder positions of the token
sequence, and the text body (128 or 512 tokens, whichever holds the sequence) turns the whole
sequence into the embedding.

| File | Runs on | Purpose |
|---|---|---|
| `text_body_s128.vmfb`, `text_body_s512.vmfb` | NPU | Text body for 128 / 512 tokens (queries, images; documents, video, longer sounds) |
| `vision_384x384.vmfb` | NPU | Vision encoder: a 384x384 image or video frame -> 64 soft tokens |
| `audio_140.vmfb` … `audio_1120.vmfb` | NPU | Audio encoder for 1.4 / 2.8 / 5.6 / 11.2 s windows |
| `token_embeddings.npy` | CPU | Token-embedding table, memory-mapped |
| `mel_filters.npy`, `mel_window.npy` | CPU | Audio log-mel front end |
| `tokenizer.json`, `config.json`, `embeddinggemma2_manifest.json` | CPU | Tokenizer, model config, static shapes and NPU memory per network |
| `examples/` | — | 14 CC0 / public-domain photos, sounds and videos with descriptions, used by the demo |
| `onnx/` | — | Source ONNX graphs that torq-tools builds these files from: a copy of [onnx-community/embeddinggemma-2-ONNX](https://huggingface.co/onnx-community/embeddinggemma-2-ONNX) |

## Performance (8-bit build, SL2619)

Time per call after a warm-up run, one network per process. CPU: ONNX Runtime 1.27 with the same
8-bit weights on the board's 2 CPU cores.

| Network | NPU | CPU (ONNX Runtime) | NPU speed-up | RAM | File |
|---|---|---|---|---|---|
| text S=128 | 389 ms | 4774 ms | 12.3x | 198 MB | 165 MB |
| text S=512 | 2185 ms | 17546 ms | 8.0x | 220 MB | 186 MB |
| vision 384x384 | 2734 ms | 18612 ms | 6.8x | 330 MB | 303 MB |
| audio 1.4 s | 302 ms | 5793 ms | 19.2x | 352 MB | 328 MB |
| audio 2.8 s | 463 ms | 7859 ms | 17.0x | 355 MB | 331 MB |
| audio 5.6 s | 935 ms | 12067 ms | 12.9x | 361 MB | 337 MB |
| audio 11.2 s | 2041 ms | 20613 ms | 10.1x | 381 MB | 357 MB |

End to end: a text query takes 0.39 s, an image 3.1 s (vision + text), a sound up to 4.9 s
0.7–1.3 s, and an 11 s sound 4.2 s.

## Deployment

Usage and demos (search over a media library with text, image or sound queries, and embedding
your own files) are in [torq-examples](https://github.com/synaptics-torq/torq-examples) under
`EmbeddingGemma-2`:

```sh
python setup_demos.py EmbeddingGemma-2       # downloads this build and the examples
cd EmbeddingGemma-2
python src/media_search.py index             # index the example media on the NPU
python src/media_search.py search startup    # then type queries
```

## Learn More

- [Synaptics AI Developer Zone](https://developer.synaptics.com?utm_source=hf): Get started with documentation, tutorials and resources for your Edge AI journey.
- [Astra Support Portal](https://synacsm.atlassian.net/servicedesk/customer/portal/543?utm_source=hf): Connect with our engineering team and community.

	---
	library_name: torq
	tags:
	- Embedding
	- Multimodal
	- Astra SL
	- SL2600
	- NPU
	license: apache-2.0
	base_model:
	- google/embeddinggemma-2
	- onnx-community/embeddinggemma-2-ONNX
	base_model_relation: quantized
	---

	# EmbeddingGemma-2

	## Model Overview

	EmbeddingGemma-2 is a multimodal embedding model from Google. It maps text, images, video and
	audio into one 768-dimensional embedding space (Matryoshka: 768, 512, 256 or 128 dimensions),
	so a text query can be compared directly with photos, video frames and sound clips.

	This model has been optimized for the Synaptics Astra™ SL2610-Series processors with Torq NPU.
	This branch holds the recommended build: 8-bit weights (int8, block 32) with bf16
	activations. Releases are tagged with the [torq-examples](https://github.com/synaptics-torq/torq-examples)
	version (for example `v2.3.0`), and the demo downloads the tag of its own version.

	\| Build \| Branch \| Size \| Notes \|
	\|---\|---\|---\|---\|
	\| 8-bit (recommended) \| `main` (this branch), tags `v2.3.0` and later \| 2.3 GB \| Embeddings close to the original model (cosine 0.997 to fp32 over a media library); all modalities in one process \|
	\| 4-bit \| [`q4`](https://huggingface.co/Synaptics/Google-EmbeddingGemma-2/tree/q4) (and tag `v2.2.0`) \| 1.1 GB \| Smallest; less accurate (cosine 0.976) \|
	\| bf16 \| [`bf16`](https://huggingface.co/Synaptics/Google-EmbeddingGemma-2/tree/bf16) \| 3.6 GB \| Closest to the original model, most memory \|

	## Model Features
	- Model Type: Multimodal embedding model
	- Input: Text, image, video, audio
	- Output: 768-d embedding

	## How it runs

	The vision and audio encoders turn an image, video frame or sound into soft tokens in the text
	model's embedding space. The host writes them into the placeholder positions of the token
	sequence, and the text body (128 or 512 tokens, whichever holds the sequence) turns the whole
	sequence into the embedding.

	\| File \| Runs on \| Purpose \|
	\|---\|---\|---\|
	\| `text_body_s128.vmfb`, `text_body_s512.vmfb` \| NPU \| Text body for 128 / 512 tokens (queries, images; documents, video, longer sounds) \|
	\| `vision_384x384.vmfb` \| NPU \| Vision encoder: a 384x384 image or video frame -> 64 soft tokens \|
	\| `audio_140.vmfb` … `audio_1120.vmfb` \| NPU \| Audio encoder for 1.4 / 2.8 / 5.6 / 11.2 s windows \|
	\| `token_embeddings.npy` \| CPU \| Token-embedding table, memory-mapped \|
	\| `mel_filters.npy`, `mel_window.npy` \| CPU \| Audio log-mel front end \|
	\| `tokenizer.json`, `config.json`, `embeddinggemma2_manifest.json` \| CPU \| Tokenizer, model config, static shapes and NPU memory per network \|
	\| `examples/` \| — \| 14 CC0 / public-domain photos, sounds and videos with descriptions, used by the demo \|
	\| `onnx/` \| — \| Source ONNX graphs that torq-tools builds these files from: a copy of [onnx-community/embeddinggemma-2-ONNX](https://huggingface.co/onnx-community/embeddinggemma-2-ONNX) \|

	## Performance (8-bit build, SL2619)

	Time per call after a warm-up run, one network per process. CPU: ONNX Runtime 1.27 with the same
	8-bit weights on the board's 2 CPU cores.

	\| Network \| NPU \| CPU (ONNX Runtime) \| NPU speed-up \| RAM \| File \|
	\|---\|---\|---\|---\|---\|---\|
	\| text S=128 \| 389 ms \| 4774 ms \| 12.3x \| 198 MB \| 165 MB \|
	\| text S=512 \| 2185 ms \| 17546 ms \| 8.0x \| 220 MB \| 186 MB \|
	\| vision 384x384 \| 2734 ms \| 18612 ms \| 6.8x \| 330 MB \| 303 MB \|
	\| audio 1.4 s \| 302 ms \| 5793 ms \| 19.2x \| 352 MB \| 328 MB \|
	\| audio 2.8 s \| 463 ms \| 7859 ms \| 17.0x \| 355 MB \| 331 MB \|
	\| audio 5.6 s \| 935 ms \| 12067 ms \| 12.9x \| 361 MB \| 337 MB \|
	\| audio 11.2 s \| 2041 ms \| 20613 ms \| 10.1x \| 381 MB \| 357 MB \|

	End to end: a text query takes 0.39 s, an image 3.1 s (vision + text), a sound up to 4.9 s
	0.7–1.3 s, and an 11 s sound 4.2 s.

	## Deployment

	Usage and demos (search over a media library with text, image or sound queries, and embedding
	your own files) are in [torq-examples](https://github.com/synaptics-torq/torq-examples) under
	`EmbeddingGemma-2`:

	```sh
	python setup_demos.py EmbeddingGemma-2 # downloads this build and the examples
	cd EmbeddingGemma-2
	python src/media_search.py index # index the example media on the NPU
	python src/media_search.py search startup # then type queries
	```

	## Learn More

	- [Synaptics AI Developer Zone](https://developer.synaptics.com?utm_source=hf): Get started with documentation, tutorials and resources for your Edge AI journey.
	- [Astra Support Portal](https://synacsm.atlassian.net/servicedesk/customer/portal/543?utm_source=hf): Connect with our engineering team and community.