junmo-k's picture
Model card: unwrap lines and describe only the current layout
e97c928 verified
|
Raw History Blame Contribute Delete
13.3 kB
---
license: apache-2.0
base_model: google/embeddinggemma-2
library_name: coreml
pipeline_tag: feature-extraction
language:
- multilingual
tags:
- coreml
- apple-neural-engine
- on-device
- embeddings
- sentence-similarity
- multimodal-embedding
- gemma
---
# EmbeddingGemma 2 for the Apple Neural Engine (text up to 8192 tokens + images)
Core ML conversion of [google/embeddinggemma-2](https://huggingface.co/google/embeddinggemma-2) that runs **entirely on the Apple Neural Engine** (`MLComputeUnits.cpuAndNeuralEngine`, no GPU). It is the model package of [ANEEmbed](https://github.com/junmo-kim/ANEEmbed) (an open-source Swift embedding server and CLI: `aneembed models install` downloads this repository at a pinned revision, checks every file against `manifest.json` and compiles it on your Mac), but it is a plain Core ML package and can be used from any Core ML client that follows the host contract below and in `manifest.json`.
Output: the same 768-d L2-normalized embedding as `SentenceTransformer("google/embeddinggemma-2").encode(...)` (mean pooling over all tokens, then normalization), at cosine β‰₯ 0.99988 for text up to 8192 tokens and β‰₯ 0.99976 for images in our tests (details below). Matryoshka truncation (512/256/128) works as in the original: keep the first d values and re-normalize.
Package layout `manifest_version` 2: every text up to 8192 tokens runs in a single Neural Engine call.
## Files
| File | Size | What |
|---|---|---|
| `EmbeddingGemma2Text.mlpackage` | 363 MB | text encoder, fp16 ML Program, **13 functions sharing one copy of the weights**: `embed_64` … `embed_1280` (full attention, packing), `winp1536_b512` / `winp2048_b512` (windowed attention, packing), `winl3072_b512` … `winl8192_b512` (windowed attention, one long text per call) |
| `embeddings.bf16` | 256 MB | token embedding table, 262,144 Γ— 512 raw bfloat16 (little endian), unchanged from the original weights; row Γ— √512 = model input |
| `tokenizer.json` | 31 MB | original tokenizer, unchanged |
| `EmbeddingGemma2Vision.mlpackage` | 292 MB | vision encoder `vision_70` / `vision_140` / `vision_280`, **unchanged** from [FluidInference/embeddinggemma-2-coreml](https://huggingface.co/FluidInference/embeddinggemma-2-coreml) @ `3f978dd` |
| `position_embeddings.f16` | 3 MB | vision x/y position tables (same source, unchanged) |
| `manifest.json` | | every file with sha256 and size (including the reference `.mlmodelc` compile), exact function signatures, host contract, token ids, provenance |
| `selftest.json` + `selftest.bin` | 0.3 MB | golden self-test (16 tests, one per function) to run at load time |
| `LICENSE`, `NOTICE` | | Apache-2.0, attribution |
Only the `.mlpackage` files are published; compile them on the Mac that runs them (`xcrun coremlcompiler compile X.mlpackage .` or `MLModel.compileModel(at:)`, about 5 s). `manifest.json` lists the `.mlmodelc` files of the reference compile (`xcrun coremlcompiler` 3600.25.1).
Without the weight sharing the same 13 functions are 3.4 GB as separate packages; merged they are 363 MB.
## How it was produced
`tools/convert/build.py` (in the ANEEmbed repository, pinned `requirements.txt`) converts the original `model.safetensors` (revision `f9972ac22e696a105c482202646979e33e4b5a1e`, sha256 checked) with coremltools 9.0 to fp16 ML Programs (minimum macOS 15), merges them into one multifunction package (identical weights are stored once), compiles it with `coremlcompiler`, extracts `embeddings.bf16`, copies the vision package unchanged (sha256 checked against the published files), generates the self-test from fp32 references and writes the manifest. On an M5 Ultra it takes 5.5 minutes and 16 GB of RAM, CPU only. A second clean build reproduced the weights, the compiled `model.mil`, the token table and the self-test byte for byte; the `.mlpackage` protobuf is message-equal (its byte order and the package UUIDs differ between runs).
The text encoder is re-expressed with numerically neutral rewrites that the ANE needs:
- RMSNorm divides each row by its max-abs value first (fp16 overflow guard).
- **Attention uses a "fold" layout**: the query heads that share one K/V head are folded into the row dimension. The usual `repeat_interleave` layout is *miscompiled* by the ANE compiler for the global layers (one K/V head): rebuilt with it, `winp2048_b512`, `winl4096_b512` and `winl8192_b512` score cosine 0.960, 0.952 and 0.885 on the ANE against the fp32 reference.
- **Windowed sliding layers.** 20 of the 24 layers only attend 512 tokens to each side. In the windowed functions they work on blocks of 512 query rows, each against the 1536 key rows around it (static slices clipped to the sequence), with the mask passed in as a `[S, 1536]` band. Padding the keys inside the graph instead gave the same numbers but was about 3Γ— slower on the ANE. The 4 global layers attend to every row.
- Masks are runtime inputs and RoPE uses constant tables at absolute positions. Graph-constant masks and a runtime positions/RoPE input both produced wrong results on the ANE in our tests.
- The per-layer-embedding projection is computed per layer from a slice of its weight, so every function carries bit-identical weight constants and the merge can share them.
## Using it (host contract)
All functions run with `cpuAndNeuralEngine`. Inputs are fp16, embeddings are fp32. Token rows are `bf16 β†’ fp32 Γ— √512 β†’ fp16` from `embeddings.bf16`. Text sequences are `[2] + tokenize(prompt + text) + [1]` (bos 2, eos 1; prompts as in the original, e.g. `title: none | text: `, `task: search result | query: `). The exact contract (shapes, masks, pooling) is in `manifest.json` β†’ `bucket_contract` and `window`.
**Up to 1280 tokens: `embed_S`.** Pack one or more sequences back to back into the smallest `embed_S` (S ∈ 64/128/256/512/768/1024/1280, up to K = max(8, S/32) sequences per call): `inputs_embeds [1,S,512]`; `global_bias [1,1,S,S]` = 0 where row and column belong to the same sequence (or i = j), else βˆ’10000; `local_bias` = the same AND |iβˆ’j| ≀ 512; `pool [K,S]` = 1/n_k over the rows of sequence k β†’ `embedding [K,768]` (unused rows are 0).
**1281 to 2048 tokens: `winp1536_b512` / `winp2048_b512`.** The same as `embed_S` (packing included, K = 48 / 64), except that `local_bias` is replaced by `band_bias [1,1,S,1536]`: row r, column c is query row r against key row k = ⌊r/512βŒ‹Β·512 βˆ’ 512 + c; 0 if r and k belong to the same sequence and |r βˆ’ k| ≀ 512, or k = r; else βˆ’10000. Columns whose key row lies outside [0, S) are never read.
**2049 to 8192 tokens: `winl{3072,4096,6144,8192}_b512`, one call per text.** S = the smallest size β‰₯ n. `inputs_embeds [1,S,512]` with rows β‰₯ n zero, `band_bias` as above for one sequence of n rows, `key_bias [1,1,1,S]` = 0 for k < n else βˆ’10000, `pool [1,S]` β†’ `embedding [1,768]`. **`pool` must be 2⁻¹⁰ on the n valid rows, not 1/n** (1/n underflows fp16 inside the ANE pooling at large n; the scale cancels in the normalization).
**Images.** Preprocess exactly like the HF `Gemma4ImageProcessor` (aspect-preserving bicubic resize to multiples of 48 px within 9 Γ— budget patches, pixels / 255, 16 Γ— 16 patches), run `vision_{70,140,280}` (positions, position embeddings from `position_embeddings.f16`, validity and 3 Γ— 3 pooling matrix as in `manifest.json` β†’ `vision`), then embed `<bos> <|image> [image tokens] <image|> <eos>` with the packing functions like a text sequence; images and texts can share one call.
## Accuracy
Cosine similarity against the **official** model: google/embeddinggemma-2 in fp32 (our PyTorch reference equals the official transformers / sentence-transformers fp32 stack at cosine 1.0000000 for texts of 64 to 8192 tokens), all functions on `cpuAndNeuralEngine`, Apple M5 Ultra, macOS 27.0.1. Public data only: multilingual READMEs (en/ko/ja/zh/fr) of an Apache-2.0 project, the Korean and Japanese constitutions (public domain), MIT-licensed technical docs and Apache-2.0 source code; images bundled with scikit-learn / networkx / llama.cpp / stable-diffusion.cpp plus synthetic images.
| Category | n | min | median |
|---|---:|---:|---:|
| Text, single sequence filling the function (64–2048) | 54 | 0.999966 | 0.999976 |
| Text, single sequence half the size | 54 | 0.999950 | 0.999975 |
| Text, packed short sequences (6–200 tokens, up to K per call) | 240 | 0.999936 | 0.999973 |
| Text, one long + several short in one call (1024–2048) | 24 | 0.999961 | 0.999977 |
| Text, two long in one call (1024–2048) | 8 | 0.999974 | 0.999975 |
| **All packing-function text** | **380** | **0.999936** | **0.999974** |
| `winl3072`, 2049–3072 tokens | 10 | 0.999970 | 0.999974 |
| `winl4096`, 3073–4096 tokens | 10 | 0.999960 | 0.999967 |
| `winl6144`, 4097–6144 tokens | 10 | 0.999931 | 0.999954 |
| `winl8192`, 6145–8192 tokens | 14 | 0.999889 | 0.999921 |
| Image, budget 70, alone (vs official fp32 image embedding) | 13 | 0.999768 | 0.999972 |
| Image, budget 140, alone | 13 | 0.999813 | 0.999975 |
| Image, budget 280, alone | 13 | 0.999954 | 0.999979 |
| Images packed with other images and a text (1024 / 2048) | 91 | 0.999769 | 0.999975 |
The long-text lengths include the first token count of every size (2049, 3073, 4097, 6145) and the last (3072, 4096, 6144, 8192). The image error comes almost entirely from the fp16 vision encoder: feeding the official fp32 image tokens through the same ANE text functions gives β‰₯ 0.999981.
## Speed (Apple M5 Ultra, macOS 27.0.1)
Median latency of one call, one text filling it, through ANEEmbed's Swift runtime (`aneembed bench --buckets`):
| Function | Latency | Function | Latency |
|---|---:|---|---:|
| `embed_64` | 3.2 ms | `winp1536_b512` | 114.7 ms |
| `embed_128` | 5.2 ms | `winp2048_b512` | 171.0 ms |
| `embed_256` | 10.7 ms | `winl3072_b512` | 286.2 ms |
| `embed_512` | 27.3 ms | `winl4096_b512` | 423.5 ms |
| `embed_768` | 46.1 ms | `winl6144_b512` | 770.9 ms |
| `embed_1024` | 70.7 ms | `winl8192_b512` | 1168.7 ms |
| `embed_1280` | 93.5 ms | | |
First load per model path, executable and macOS build triggers the ANE compiler (the OS caches the result): the 9 packing functions plus vision took 236 s through ANEEmbed, the 4 long functions another 383 s; with the compile cached, loading all of them and running the self-test takes 4.9 s (2048 tokens) / 9.1 s (8192 tokens). Load only the functions up to the length you serve.
## Memory (Apple M5 Ultra, macOS 27.0.1, through ANEEmbed)
Core ML gives each loaded function working memory for the Neural Engine, owned by the loading process but outside its `phys_footprint` (`footprint -w` lists it as `Owned physical footprint (unmapped) (neural)`): 5.25 GB for the 9 packing functions plus the vision functions, 13.3 GB with the 4 long functions added. It is wired only while used: all of it for about 60 s after the functions first run, then nothing while idle (purgeable: under memory pressure the OS discarded 1.6 GB of it, and the next calls recreated it), and during a call plus about 10 s only the buffers of the called function (266 MB for a short query, 579 MB for a 1617-token text, 3.9 GB for a 6631-token text in `winl8192_b512`). Load only the functions up to the length you serve.
## Self-test (run it at load)
The ANE compiler output depends on the OS version, and a miscompile does not raise an error, it returns wrong numbers. `selftest.json` / `selftest.bin` contain one deterministic test per function (16 tests): text inputs are built from token ids stored in the JSON (from this package's `LICENSE` text) and the shipped table, image inputs from a procedural integer pattern; expected fp32 outputs come from the fp32 reference (official fp32 for the vision tests). Text tests compare embeddings by cosine (packing tests also require unused rows to stay 0); vision tests compare an 8-column random projection (digest) of the image tokens by cosine and relative error. Recommended tolerance: cosine β‰₯ 0.999, relative error ≀ 0.05; measured on M5 Ultra: cosine β‰₯ 0.99990 for every function, relative error ≀ 0.015 (vision). The repeat-layout builds above fail it on the ANE (0.885–0.960). Reference runner: `tools/convert/selftest.py run <dir>`.
## Limitations
- Apple Neural Engine only. The package was tuned and verified with `cpuAndNeuralEngine`; other compute units are not tested.
- Text longer than 8192 tokens must be split by the caller. Images use the HF preprocessing; llama.cpp's image preprocessing differs, so image embeddings are not interchangeable with llama.cpp GGUF ones.
- Audio and video are not included in this package.
- First load per model path compiles for the ANE (about 10 minutes for all text functions on M5 Ultra); move the directory, update macOS or run it from another executable and it compiles again.
## License and attribution
Apache-2.0, see `LICENSE` and `NOTICE`. Original model: EmbeddingGemma 2 by Google DeepMind ([google/embeddinggemma-2](https://huggingface.co/google/embeddinggemma-2)); use must also follow the [Gemma Prohibited Use Policy](https://ai.google.dev/gemma/prohibited_use_policy). Vision encoder conversion and position tables: [FluidInference/embeddinggemma-2-coreml](https://huggingface.co/FluidInference/embeddinggemma-2-coreml) (Apache-2.0), unchanged.