xbill9/embeddinggemma-2-w8a8-ct-text-emb4

EmbeddingGemma 2, W8A8 int8 linears with int4 embeddings, text only (unofficial repack)

This is an unofficial repack, made and published independently of Google. It is google/embeddinggemma-2 with the vision and audio encoders removed, the text model's linears stored as int8 and the embedding table as int4, for serving on a CPU. Google's model card is kept unchanged in ORIGINAL_README.md.

The checkpoint is 203.4 MiB, against 1.4 GB for the original (all encoders, bf16). Its sibling, xbill9/embeddinggemma-2-w4a16-ct-text-emb4 (146.1 MiB), stores the linears as int4.

What it is

Tensor	Format
Text-model linears (attention, MLP, per-layer-embedding gates and projections; 217 modules)	int8 weights, one bf16 scale per output channel; activations declared int8, per tensor, dynamic
embed_tokens [262144, 512]	int4 group 32, fp16 scales
embedding_projection (512 → 768, the last layer)	bf16, unchanged
Norms, scalars	bf16, copied byte for byte
Vision and audio encoders	removed (470M of 740M parameters)

No calibration data was used. Each int8 linear differs from bf16 by 0.75% to 1.1% (relative error of the whole tensor); the int4 embedding table, rounded from bf16 because this model has no QAT export, by 9.0%.

Accuracy

SciFact (BEIR test split, 300 queries, 5,183 documents, max 512 tokens, 768 dimensions, cosine ranking), on CPU through cpu_load.py:

build	checkpoint	nDCG@10	recall@10	top 10 shared with float32
google/embeddinggemma-2, float32	1.4 GB	0.8542	0.9437	100%
This build	203.4 MiB	0.8358 (97.9%)	0.9259	85.5%
W4A16 sibling	146.1 MiB	0.8336 (97.6%)	0.9342	80.7%

Mean cosine similarity to the float32 original over 16 short texts: 0.993 for documents and 0.995 for queries.

Speed on one vCPU

On Cloud Run, 1 vCPU, one thread, medians of three benchmark runs each, against the original text model in float32: queries take 2.08x less time, and documents (32 per request, about 135 tokens each) embed 1.29x faster. Full method and numbers: embedding-cloudrun.

Using it on CPU

cpu_load.py in this repository loads it with Sentence Transformers. Most linears run as PyTorch fbgemm dynamic int8 linears (x86). down_proj and per_layer_projection take float inputs: their int8 weights are dequantized once at load, because a down_proj input has channels up to about 700x its RMS, which one int8 scale per tensor cannot hold. The int4 table stays packed in memory and only looked-up rows are unpacked.

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("xbill9/embeddinggemma-2-w8a8-ct-text-emb4")
sys.path.insert(0, path)
from cpu_load import load

model = load(path)
q = model.encode(["What causes the northern lights?"], prompt_name="SearchQuery")
d = model.encode(["Auroras form when charged particles from the Sun hit the upper atmosphere."],
                 prompt_name="Document")
print(model.similarity(q, d))

Tested with torch 2.14.1 (CPU), transformers 5.19.0 and sentence-transformers 6.1.0.

Limitations

• Text only: image, audio and video inputs are not supported.
• x86 CPUs: the int8 path uses fbgemm.
• Not tested in vLLM or with Transformers' compressed-tensors integration; use cpu_load.py.
• Unofficial. Report problems here, not to Google.

How it was made

repack_text_w8a8_emb4.py builds it from a snapshot of google/embeddinggemma-2. repack_report.json holds the per-tensor errors. The SciFact eval is eval_scifact.py.

License and attribution

EmbeddingGemma 2 is released by Google DeepMind under the Apache 2.0 license. This repository redistributes Google's weights in a changed format, under the same license. The weights, training and the original model card are Google DeepMind's; the repack and this card are not affiliated with or endorsed by Google.