ewin-reg/embeddinggemma-2-Quantized

EmbeddingGemma 2 int4: multimodal text, image and audio embedding model in 486 MiB

This is google/embeddinggemma-2 with all three towers packed to mostly 4 bits. It keeps the same API and the same 768-dimension embeddings. It is 66% smaller on disk than the bf16 original and loses under 1% retrieval quality in the tests below. Nothing was trained. The weights are rounded once, after training, with no calibration data. The few layers that were most sensitive to rounding stay at 8 bits.

At a glance

	Original (bf16)	This model
Weights on disk	1,420 MiB	486 MiB, 66% smaller
Image and text retrieval, NDCG@10	0.8234	0.8213 (0.25% lower)
Image and text retrieval, Recall@10	0.8904	0.8896 (0.09% lower)
Peak process memory, text workload	1,552 to 1,674 MB	1,296 to 1,347 MB for the first release, see below
Forward time on CPU in bf16	baseline	the same
Image encoding on a T4 GPU, 1,000 images	480 s in fp32	189 s with fp16 autocast, first release (2.5x faster, 22% less GPU memory, at most 0.12% extra loss)

The memory saving is far smaller than the disk saving, because Python, torch and the activations stay the same size. If you care about download size or storage, this helps a lot. If you care about process RAM, expect roughly 14 to 22% lower peak memory, not 66%.

This is the second release. The first release (451.5 MiB, everything in 4 bits) is revision 798b305f3f1eb292c35543ceaead09821d7dd233 if you need the smaller file: snapshot_download(repo, revision="798b305f3f1eb292c35543ceaead09821d7dd233"). The second release adds 35 MiB of 8-bit layers and changes the top result less often, see "Limits". The memory and GPU speed numbers were measured on the first release. They have not been repeated on this one, which holds 35 MiB more weights.

Quick start

You need transformers==5.19.0, sentence-transformers>=6.1 and accelerate.

import sys, torch
from huggingface_hub import snapshot_download

path = snapshot_download("ewin-reg/embeddinggemma-2-int4-text-image-audio-embedding")
sys.path.insert(0, path)
import qpack3

st = qpack3.load_st(path, dtype=torch.bfloat16, device="cuda")  # or "cpu"
q = st.encode(["task: search result | query: a red bicycle"])
d = st.encode(["title: none | text: a bicycle leaning on a wall"])
img = st.encode([pil_image])            # list of PIL images
aud = st.encode({"audio": waveform})    # 16 kHz mono float32 array

Use the original model's prompts. Queries start with task: search result | query: and documents with title: none | text: . The loader builds the model on empty weights and attaches the packed tensors, so it never allocates a full-precision copy.

Making it faster on a GPU

Wrap encoding in fp16 autocast. This needs no change to the weights. I measured it on the first release, on a T4 with 1,000 COCO images and the model loaded in fp32:

st = qpack3.load_st(path, dtype=torch.float32, device="cuda")
with torch.autocast("cuda", dtype=torch.float16):
    img = st.encode(pil_images, batch_size=8)

Setting	Time for 1,000 images	GPU peak memory	Extra loss against fp32
fp32 (baseline)	480 s	1,609 MB	none
fp16 autocast	189 s (2.5x faster)	1,251 MB (22% lower)	NDCG@10 and Recall@10 within 0.06%, Recall@1 within 0.12%, top-1 changes 0.3% and 0.0%
fp16 autocast and 140 soft tokens	89 s (5.4x faster)	852 MB (47% lower)	see below

The 140 soft token mode (st[0].processor.image_processor.max_soft_tokens = 140) halves the patches per image. It is the fastest setting, but it costs about 0.7% NDCG@10 for text-to-image, 1.9% Recall@1 for text-to-image, and changes the top-1 image for 27% of image-to-text queries and 9% of text-to-image queries. Use it only if you do not need exact top-1 ranking. The default of 280 is what the quality numbers below describe. I did not test fp16 with a model loaded in bf16.

Limits to know before you use it

• Keep max_soft_tokens at 280 or lower, which is the default. The vision position table was cut to 2,520 rows. That is exact for 280 soft tokens and wrong above it.
• Rounding changes which result ranks first for some queries, even though NDCG@10 and Recall@10 stay within 0.25%. Against the original, with the original's text embeddings on the other side, the top image changes for 4.1% of image-to-text queries and 2.6% of text-to-image queries (the first release: 4.8% and 4.4%; the text-to-image drop is statistically significant, McNemar p = 0.0005). Counting the exact top caption instead of the top image, the image-to-text figure is 8.2%. With both sides packed it is 15.3% for image-to-text and 3.1% for text-to-image. If you need the exact top-1 of the original model, do not use this.
• Audio was checked on easy data only, so treat audio as less verified than text and vision.
• Quality was measured in fp32. The memory and speed runs used bf16 on CPU.

Retrieval quality

Each row compares this release with the unquantized original on the same inputs, loading the packed files through this repository's loader. Compute was fp32 on a T4 GPU.

Test	Data	NDCG@10	Recall@10	Recall@1
Image and text, both towers packed, both directions	1,000 COCO Karpathy-test images, 5 captions each	0.8234 to 0.8213	0.8904 to 0.8896	0.7935 to 0.7925
Image retrieval, text side unquantized	same	0.8234 to 0.8218	0.8904 to 0.8896	0.7935 to 0.7910
Caption to caption, 4 relevant per query	same captions	0.5956 to 0.5949	0.6435 to 0.6392	0.613 to 0.621
Caption to caption, text tower only, all 5,000 images (25,000 captions)	4 relevant per query	0.4226 to 0.4217 (0.23% lower)	0.4528 to 0.4518 (0.23% lower)	0.4555 to 0.4543 (0.27% lower)
Speech to transcript and back	300 LibriSpeech test-clean clips	1.000 to 1.000	1.000 to 1.000	1.000 to 1.000

For the first row, the 95% bootstrap upper bounds on the loss are 0.60% for NDCG@10 and 0.46% for Recall@10. On the 1,000-image caption subset the Recall@10 loss was 0.66% (upper bound 1.37%). On all 5,000 images the loss is 0.23%, with a cluster bootstrap over images (2,000 resamples) that puts the upper bound at 0.41% for NDCG@10 and Recall@10. Both upper bounds are under 1%. The 1,000-image figure was a noisier estimate of the same model, not a different result. Both audio scores are 100%, so that test shows no loss but is too easy to tell the models apart. The mean cosine similarity between the image embeddings and the original's is 0.9959, and between the audio embeddings and the original's about 0.98.

Memory and speed

I measured these on the first release (451.5 MiB), on a Kaggle CPU kernel with bf16 compute and transformers 5.19.0. Each run used a fresh process and loaded the repository from a clean download. The figure is peak resident memory (VmHWM), and the original is google/embeddinggemma-2 loaded the same way, audio tower included.

Run	Workload	Original	First release	Change	Forward time
1	Text, 8 sequences of 253 tokens	1,552 MB	1,296 MB	16.5% lower	30.2 s and 30.2 s
1	Image, 4 images of 640x480 at 280 soft tokens each	1,667 MB	1,439 MB	13.7% lower	198.4 s and 201.7 s
2	Text, same	1,674 MB	1,347 MB	19.5% lower	38.1 s and 38.4 s
2	Image, same	1,851 MB	1,450 MB	21.7% lower	297.6 s and 305.1 s
3, second release (486.2 MiB)	Text, same	1,770 MB	1,354 MB	23.5% lower	36.6 s against 36.0 s
3, second release	Image, same	1,852 MB	1,556 MB	16.0% lower	315.2 s against 352.5 s (second pass 313.2 s against 375.9 s)

The two runs were on different Kaggle machines, and the original's own peak differed by 8 to 11% between them, so read the saving as a range. Forward time matched the original in runs 1 and 2. In run 3 the second release took 12% to 20% longer on the image workload, which I did not repeat, so I cannot say whether that is noise or the 8-bit layers. Resident memory right after load does not compare fairly, because the original's weights are memory-mapped and load lazily, so peak is the fair number. Memory-mapping the packed files did not help: mapping everything saved about 3% more and slowed the forward pass by 26% to 65%, and mapping only the audio tower changed peak memory by 1% to 2%.

What is in the 486 MiB

Tower	Original (bf16, approx.)	This model
Audio	580 MiB	183.7 MiB
Text	517 MiB	181.9 MiB
Vision	320 MiB	117.8 MiB
Projectors (embed_audio, embed_vision)	2 MiB	2.3 MiB

How it was packed

• Int4 covers most linear layers in the text tower, the text token table, the vision block linears and the audio block linears. Groups are 16 input columns wide, with asymmetric min-max rounding.
• Int8, in the same groups of 16, covers the most rounding-sensitive layers: the text tower's non-layer linears (including the output projection) and text layers 10, 11, 14, 22 and 23, found with a data-free sweep that quantizes one layer at a time and measures the cosine change of caption embeddings, and vision blocks 12 to 15, from the same kind of sweep on image embeddings. The text tower's output projection and layer 23 were 5 to 15 times more sensitive than any other text layer. The extra cost is 35 MiB.
• Each group's scale and minimum are stored as 8-bit integers per row, with fp16 row bounds.
• The vision patch projection, the two projectors, the convolutions and all normalization weights stay in bf16.
• The vision position table is cut from 10,240 to 2,520 rows per axis. At 280 soft tokens this is lossless, and I checked the embeddings are bit-identical.
• I compared real byte packing with the simulated rounding used in the group-size sweep. The maximum absolute difference is 0 on the tested layers.

I picked group size 16 after comparing 16, 32, 64 and 128 on the same images. Group 16 had the smallest quality loss and the lowest top-1 change rate for image to text in a vision-tower-only simulation, 7.5%, against 10.2% at group 32 and 10.9% at group 128. 3-bit failed the 1% bar. I also tried clipping each group's range to minimize squared rounding error. It lowered weight error but made retrieval worse (image-and-text worst loss 0.49% against 0.22%, Recall@1 loss 1.32% against 0.32%, caption Recall@10 loss 1.20% against 0.66%) and left the top-1 changes where they were (17.1% against 17.0% for image to text with both sides packed), so I did not use it.

Not measured

• Any dataset other than COCO captions and LibriSpeech.
• GPU memory and speed on the second release (the GPU numbers were measured on the first release).
• Retrieval quality in bf16.
• Memory at longer text lengths, or on a GPU.
• Video inputs.

License and credit

Apache-2.0, like the original. The weights come from Google's google/embeddinggemma-2, and this repository only repacks them.

Inference optimization and efficiency techniques

What this package does to cut memory and time, what I measured, and what I tried and dropped. Numbers come from the sections above.

Used and measured

Technique	Effect	Where measured
Int4 weights in groups of 16, 8-bit group metadata	66% smaller file (1,420 to 486.2 MiB)	Quality section
Int8 for the most rounding-sensitive layers	Lower retrieval loss for 35 MiB	Quality section
Position-table trim, 10,240 to 2,520 rows	Smaller file, bit-identical embeddings at 280 soft tokens	How it was packed
Loading on empty weights, then attaching packed tensors	No full-precision copy is ever allocated	Quick start
fp16 autocast on GPU	2.5x faster, 22% less GPU memory (first release)	Making it faster on a GPU
Fewer image soft tokens (140 instead of 280)	5.4x faster than fp32 with autocast, but fails the 1% bar on text-to-image (optional)	Making it faster on a GPU

Tried and dropped

• Data-free clipping of group ranges. It lost more quality than plain min-max rounding.
• Memory-mapping the packed files. It saved about 3% memory and slowed the forward pass by 26% to 65%.
• Group sizes of 32, 64 and 128. They changed more top-1 results than group 16.
• 3-bit weights. They failed the 1% bar.

Not tried, so no claim

• A fused dequantize-and-multiply kernel, or tiled dequantization. The loader expands each packed linear to bf16 before the matmul, so latency gains on CPU would need a kernel I have not written.
• torch.compile, ONNX, or other runtime export.
• Batching or caching strategies beyond what sentence-transformers already does.

The limit on CPU memory is structural. Python, torch and the activations stay the same size whatever the weight format, so peak process memory fell by 14% to 24%, not by the 66% the file shrank.