Feature Extraction
GGUF
llama.cpp
embeddings
multimodal
vision
audio
image-feature-extraction
audio-feature-extraction
Instructions to use MirrorNeuronLab/embeddinggemma-2-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use MirrorNeuronLab/embeddinggemma-2-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible embeddings server: llama serve -hf MirrorNeuronLab/embeddinggemma-2-GGUF:Q4_K_M --embeddings
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible embeddings server: llama serve -hf MirrorNeuronLab/embeddinggemma-2-GGUF:Q4_K_M --embeddings
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible embeddings server: ./build/bin/llama serve -hf MirrorNeuronLab/embeddinggemma-2-GGUF:Q4_K_M --embeddings
- Llama new
- LM Studio
- Jan
- Ollama
How to use MirrorNeuronLab/embeddinggemma-2-GGUF with Ollama:
ollama run hf.co/MirrorNeuronLab/embeddinggemma-2-GGUF:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use MirrorNeuronLab/embeddinggemma-2-GGUF with Docker Model Runner:
docker model run hf.co/MirrorNeuronLab/embeddinggemma-2-GGUF:Q4_K_M
- Lemonade
How to use MirrorNeuronLab/embeddinggemma-2-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull MirrorNeuronLab/embeddinggemma-2-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.embeddinggemma-2-GGUF-Q4_K_M
List all available models
lemonade list
- Atomic Chat
File size: 12,399 Bytes
336c50f 5a0729b 336c50f 5a0729b 336c50f d122027 336c50f d122027 336c50f d122027 336c50f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 | ---
license: apache-2.0
base_model: google/embeddinggemma-2
base_model_relation: quantized
library_name: llama.cpp
pipeline_tag: feature-extraction
tags:
- gguf
- embeddings
- multimodal
- vision
- audio
- image-feature-extraction
- audio-feature-extraction
---
# EmbeddingGemma 2 Multimodal GGUF - Q4_K_M Text + Q8_0 Vision/Audio
**Text, image, and audio embeddings in a shared 768-dimensional vector space.** This MirrorNeuronLab release pairs a **Q4_K_M text backbone** with a **Q8_0 companion containing both vision and audio encoders**.
For image or audio inputs, load both GGUF files. Docker Model Runner automatically downloads the companion when you pull `hf.co/MirrorNeuronLab/embeddinggemma-2-GGUF:Q4_K_M`. The `Q4_K_M` selection identifies the text backbone; the complete release supports text, images, and audio.
Image and audio embedding requests were verified through Docker Model Runner on NVIDIA DGX Spark. The model returns embeddings for search and similarity; it does not generate conversational answers or transcriptions.
[Google DeepMind](https://huggingface.co/google/embeddinggemma-2) developed the original EmbeddingGemma 2 model. MirrorNeuronLab converted Google's original weights to GGUF and quantized them without additional training.
The text backbone uses standard llama.cpp **Q4_K_M**, quantized directly from a **BF16** intermediate without an importance matrix. This is a MirrorNeuronLab GGUF conversion and quantization. The Q4_K_M preset mixes tensor types as defined by upstream llama.cpp, retaining small tensors and the embedding projection at higher precision where the preset requires it.
The separate **Q8_0 vision/audio companion** is required for image and audio inputs. It contains both encoders, including their projections. There is one text-model quantization and one multimodal companion in this repository. The Q8_0 companion supplements the Q4_K_M text model; it is not an alternative text model.
## Files
| File | Purpose | Exact bytes | MiB |
| --- | --- | ---: | ---: |
| `embeddinggemma-2-Q4_K_M.gguf` | Text backbone, standard Q4_K_M | 181,694,688 | 173.278 |
| `mmproj-embeddinggemma-2-Q8_0.gguf` | Combined vision/audio companion | 554,821,056 | 529.119 |
Binary SHA-256 checksums:
```text
d610afe26dd783739275646d51d4d85b4167820fd051600696d7a49e6f68609f embeddinggemma-2-Q4_K_M.gguf
0531309472a6bc6f4b146c3736eee7795ec9db1903c25490d3c937490cef6dd8 mmproj-embeddinggemma-2-Q8_0.gguf
```
`SHA256SUMS` covers the release files. `SOURCE_SHA256SUMS`, `PROVENANCE.json`, `GGUF_AUDIT.json`, `VALIDATION.json`, and `TIMINGS.json` document the original checkpoint, conversion, resulting tensor inventory, and checks. The BF16 intermediate is kept outside the release directory.
## Source and tools
- Google checkpoint: [`google/embeddinggemma-2` at `f9972ac22e696a105c482202646979e33e4b5a1e`](https://huggingface.co/google/embeddinggemma-2/tree/f9972ac22e696a105c482202646979e33e4b5a1e).
- Original `model.safetensors` SHA-256: `197a32965d4b1105faf060417baa899e193fb73cd401f42ec9295234d5553d79`.
- Conversion and quantization: [`ggml-org/llama.cpp` at `dde879bc4f954b809892fd0d660515f42536394a`](https://github.com/ggml-org/llama.cpp/tree/dde879bc4f954b809892fd0d660515f42536394a), build b12032. The upstream checkout is unmodified.
- Python 3.12.3; PyTorch 2.11.0+cpu; Transformers 4.57.6; Hugging Face Hub 0.36.2. Hub and Transformers were resolved together under llama.cpp's constraints; `pip check` passed. The full environment lock is included under `reproduce/logs/`.
Google's tokenizer config uses a newer Transformers schema. A small compatibility adapter maps its special-token configuration to the pinned Transformers 4 format in memory. It verifies that all 262,144 token strings and IDs match the original `tokenizer.json`. Original source files remain byte-for-byte unchanged. GGUF auditing confirms preserved tokenizer metadata, noncausal architecture, mean pooling, the 512-to-768 embedding projection, and matching tensor shapes between BF16 and Q4_K_M.
The [AtomicChat model card](https://huggingface.co/AtomicChat/embeddinggemma-2-GGUF) was consulted for packaging and usage conventions. This release uses the standard Q4_K_M preset, with no AtomicChat weights, importance matrix, dynamic tensor layout, or benchmark results. It is not labeled `AD-Q4_K_M`.
## Docker Model Runner
Use an up-to-date llama.cpp engine with `gemma-embedding2` and Gemma4 vision/audio support. The Docker Model Runner controller version alone does not establish engine compatibility: older bundled engines report an unknown architecture. The files are portable to Apple Silicon Macs and Ubuntu/NVIDIA systems with a supporting backend. Those platforms were not physically tested in this run.
Pull the Q4_K_M text release; Docker Model Runner also selects the sole multimodal companion automatically:
```bash
docker model pull hf.co/MirrorNeuronLab/embeddinggemma-2-GGUF:Q4_K_M
docker model configure --mode embedding --context-size 2048 \
hf.co/MirrorNeuronLab/embeddinggemma-2-GGUF:Q4_K_M \
-- --batch-size 2048 --ubatch-size 2048 --pooling mean
docker model run --detach hf.co/MirrorNeuronLab/embeddinggemma-2-GGUF:Q4_K_M
```
This is an embedding model. Use the embeddings API to obtain vectors after loading it:
```bash
curl http://localhost:12434/engines/llama.cpp/v1/embeddings \
-H 'Content-Type: application/json' \
-d '{"model":"hf.co/MirrorNeuronLab/embeddinggemma-2-GGUF:Q4_K_M","input":["task: search result | query: What causes the northern lights?","title: none | text: Charged particles from the Sun interact with the atmosphere."],"encoding_format":"float"}'
```
For local files, explicitly package both GGUFs and the license. Run this from the downloaded release directory:
```bash
docker model package \
--gguf "$PWD/embeddinggemma-2-Q4_K_M.gguf" \
--mmproj "$PWD/mmproj-embeddinggemma-2-Q8_0.gguf" \
--license "$PWD/LICENSE" --context-size 2048 \
mirrorneuronlab/embeddinggemma-2:q4_k_m
docker model configure --mode embedding --context-size 2048 \
mirrorneuronlab/embeddinggemma-2:q4_k_m \
-- --batch-size 2048 --ubatch-size 2048 --pooling mean
docker model run --detach mirrorneuronlab/embeddinggemma-2:q4_k_m
```
Use `mirrorneuronlab/embeddinggemma-2:q4_k_m` as the API model name for a locally packaged artifact. Packaging keeps the original GGUF files on disk. See [Docker packaging](https://docs.docker.com/reference/cli/docker/model/package/) and the [Docker Model Runner API](https://docs.docker.com/ai/model-runner/api-reference/).
## llama.cpp and multimodal inputs
```bash
llama-server -m embeddinggemma-2-Q4_K_M.gguf \
--mmproj mmproj-embeddinggemma-2-Q8_0.gguf \
--embeddings --pooling mean -c 2048 -b 2048 -ub 2048 -ngl 99
```
For text-only use, omit `--mmproj`. GPU offloading uses the backend available on the machine, such as Metal on Apple Silicon or CUDA on NVIDIA systems.
The native output is a normalized 768-dimensional embedding. Use task prefixes such as `task: search result | query: {query}` and `title: {title or none} | text: {document}`. Index documents and query them with the same model file. For 512-, 256-, or 128-dimensional Matryoshka vectors, retain the first dimensions and normalize again.
For image/audio embeddings, send OpenAI-style content arrays as elements of `input`. Examples of the two element forms:
```json
{"content":[{"type":"image_url","image_url":{"url":"data:image/png;base64,..."}}]}
{"content":[{"type":"input_audio","input_audio":{"data":"<base64 WAV>","format":"wav"}}]}
```
Set `encoding_format` to `float`. Apply task prefixes to text inputs. Images and audio do not need a text prefix.
EmbeddingGemma 2 uses bidirectional attention: the whole input must fit in one microbatch. Size context (`-c`), batch (`-b`), and microbatch (`-ub`) together for the longest input. The validation used 2048 for all three. Google's published context limit is 8192; the source config and preserved GGUF metadata expose a larger positional limit of 262144, which this release does not establish as a supported input length. For 8K inputs, raise all three settings to 8192 in a supporting engine. Small microbatches can also reduce the engine's image token budget.
## Measured validation
Validated on NVIDIA DGX Spark, native Linux ARM64, NVIDIA GB10, CUDA 13.0, through **Docker Model Runner v1.2.9** and the pinned upstream CUDA llama.cpp backend above. The official ARM64 CUDA runner image is pinned in `PROVENANCE.json`; its stock older engine was replaced for this isolated validation by a read-only mount of the newly compiled backend. The existing system runner was left intact. The controller reports the backend version as `unknown` because its parser does not recognize the new upstream version string; the executed binary and full commit are recorded.
`docker model package` and `docker model run --detach` succeeded. Text, image, audio, and mixed text/image/audio batches returned HTTP 200 with finite, L2-normalized 768-dimensional vectors. GPU detection and NVIDIA-SMI confirmed CUDA use. The four-text repeat check had maximum absolute difference 0.0.
On the small text sanity fixture, query-to-relevant-passage cosine was 0.865082, versus 0.548627 for an unrelated passage. Cosines to a BF16 CPU reference on four texts ranged from 0.988963 to 0.997227. These measurements include quantization and CPU/CUDA execution differences.
These are functional smoke checks on four texts, a solid-color image, and a synthetic two-second tone. They do not measure retrieval accuracy or image/audio semantic quality. Video was not tested. llama.cpp marks audio input experimental. No MTEB, MIEB, MMEB, or audio benchmark score is claimed for this conversion.
The published Hugging Face release was also pulled through Docker Model Runner using `:Q4_K_M`. `docker model run --detach` succeeded, and text, image, audio, and mixed-batch embedding checks passed using the automatically selected Q8_0 companion. See `PUBLISHED_VALIDATION.json` for the published artifact revision and results.
## Reproduce
The conversion commands are:
```bash
python scripts/convert_compatible.py source --outtype bf16 \
--model-name embeddinggemma-2 --outfile work/embeddinggemma-2-BF16.gguf
./llama.cpp/build/bin/llama-quantize work/embeddinggemma-2-BF16.gguf \
release/embeddinggemma-2-Q4_K_M.gguf Q4_K_M
python llama.cpp/convert_hf_to_gguf.py source --mmproj --outtype q8_0 \
--model-name embeddinggemma-2 --outfile release/mmproj-embeddinggemma-2-Q8_0.gguf
```
For a fresh Linux ARM64 workspace, run the bundled script with a new destination path:
```bash
bash reproduce/scripts/reproduce.sh /absolute/path/to/new-workspace
```
It downloads the pinned original source, installs the locked converter dependencies, checks out the full llama.cpp revision, builds native tools, converts, quantizes, and audits. Its dependency lock records the tested ARM64 CPU converter environment. For other operating systems, resolve platform wheels while retaining llama.cpp's package constraints. CUDA validation setup is supplied in `reproduce/scripts/start_validation_runner.sh`; build the backend first with `-DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=121` for GB10. This GPU target applies to the tested backend build, not to the portable GGUF files.
Measured elapsed seconds for the successful stages:
| Stage | Seconds |
| --- | ---: |
| Original checkpoint download | 31.993 |
| Python dependencies | 25.619 |
| CPU build (quantizer and reference server) | 59.664 |
| CUDA build | 222.716 |
| BF16 text conversion | 12.600 |
| Q4_K_M quantization | 2.439 |
| Q8_0 vision/audio conversion | 3.547 |
| GGUF audit | 21.716 |
| BF16 reference validation | 2.206 |
| Docker packaging | 2.361 |
| Docker CUDA embedding validation | 2.226 |
| docker model run --detach | 0.051 |
Stages ran partly in parallel, so these durations should not be added as an end-to-end runtime. `TIMINGS.json` records start/end times, commands, configure stages, and attempts. The first unadapted text-conversion attempt failed on the tokenizer schema before the successful compatible conversion.
## License and attribution
Apache-2.0. Original model: Google DeepMind. GGUF conversion and quantization: MirrorNeuronLab. See `LICENSE` and the [original model card](https://huggingface.co/google/embeddinggemma-2) for intended uses and limitations.
|