lorelaiassistant/embeddinggemma-2-text-mlx-int8

embeddinggemma-2 (text-only) — MLX int8 quantization

8-bit group-quantized, text-only port of google/embeddinggemma-2 for the MLX framework on Apple Silicon.

What is in here

• Text backbone only. The vision and audio towers of the original checkpoint are dropped (vision_config / audio_config are null). Text embeddings are produced by the same text backbone, pooling and projection as the full model, so they live in the same vector space.
• Linear layers quantized to 8-bit affine, group_size=64. 288 MB of weights, versus ~1.23 GB for a full multimodal 8-bit conversion.
• Output: 768-dim, L2-normalised. Matryoshka truncation to 512 or 256 works — truncate, then re-normalise.

Do not run it in fp16: activations overflow and produce NaN. Use bf16/fp32 compute (MLX default for quantized weights is fine).

Prompts

The model is asymmetric. Prepend the prompt yourself — the loader does not:

role	prefix
query	task: search result | query:
document	title: none | text:

Other task prompts are listed in config_sentence_transformers.json.

Fidelity vs fp32

Measured on a private retrieval set (387 short user facts, 40 Polish + 15 English queries), against the PyTorch fp32 reference of the full model:

variant	dim	min cosine vs fp32 (docs / queries)	PL MRR@10	PL hit@5	EN MRR@10
fp32 reference	768	—	0.856	0.95	0.811
bf16	768	0.9999 / 0.9998	0.856	0.95	0.800
int8 (this repo)	768	0.9995 / 0.9993	0.856	0.95	0.811
int6	768	0.9968 / 0.9974	0.842	0.95	0.811

int8 matches the fp32 reference on every retrieval metric; int6 starts losing Polish ranking quality.

Usage with MLXEmbedders (Swift)

Requires an mlx-swift-lm build that includes the embedding_gemma2 loader (merged to main after 2026-10-12).

import MLXEmbedders
import MLXLMCommon

let config = ModelConfiguration(
    id: .id("lorelaiassistant/embeddinggemma-2-text-mlx-int8")
)

let container = try await EmbedderModelFactory.shared.loadContainer(
    from: hubDownloader,
    using: huggingFaceTokenizerLoader,
    configuration: config,
    progressHandler: { _ in }
)

License

Apache 2.0, inherited from the base model.