junmo-k's picture
Model card: unwrap lines and describe only the current layout
e97c928 verified
|
Raw History Blame Contribute Delete
13.3 kB
metadata
license: apache-2.0
base_model: google/embeddinggemma-2
library_name: coreml
pipeline_tag: feature-extraction
language:
  - multilingual
tags:
  - coreml
  - apple-neural-engine
  - on-device
  - embeddings
  - sentence-similarity
  - multimodal-embedding
  - gemma

EmbeddingGemma 2 for the Apple Neural Engine (text up to 8192 tokens + images)

Core ML conversion of google/embeddinggemma-2 that runs entirely on the Apple Neural Engine (MLComputeUnits.cpuAndNeuralEngine, no GPU). It is the model package of ANEEmbed (an open-source Swift embedding server and CLI: aneembed models install downloads this repository at a pinned revision, checks every file against manifest.json and compiles it on your Mac), but it is a plain Core ML package and can be used from any Core ML client that follows the host contract below and in manifest.json.

Output: the same 768-d L2-normalized embedding as SentenceTransformer("google/embeddinggemma-2").encode(...) (mean pooling over all tokens, then normalization), at cosine β‰₯ 0.99988 for text up to 8192 tokens and β‰₯ 0.99976 for images in our tests (details below). Matryoshka truncation (512/256/128) works as in the original: keep the first d values and re-normalize.

Package layout manifest_version 2: every text up to 8192 tokens runs in a single Neural Engine call.

Files

File Size What
EmbeddingGemma2Text.mlpackage 363 MB text encoder, fp16 ML Program, 13 functions sharing one copy of the weights: embed_64 … embed_1280 (full attention, packing), winp1536_b512 / winp2048_b512 (windowed attention, packing), winl3072_b512 … winl8192_b512 (windowed attention, one long text per call)
embeddings.bf16 256 MB token embedding table, 262,144 Γ— 512 raw bfloat16 (little endian), unchanged from the original weights; row Γ— √512 = model input
tokenizer.json 31 MB original tokenizer, unchanged
EmbeddingGemma2Vision.mlpackage 292 MB vision encoder vision_70 / vision_140 / vision_280, unchanged from FluidInference/embeddinggemma-2-coreml @ 3f978dd
position_embeddings.f16 3 MB vision x/y position tables (same source, unchanged)
manifest.json every file with sha256 and size (including the reference .mlmodelc compile), exact function signatures, host contract, token ids, provenance
selftest.json + selftest.bin 0.3 MB golden self-test (16 tests, one per function) to run at load time
LICENSE, NOTICE Apache-2.0, attribution

Only the .mlpackage files are published; compile them on the Mac that runs them (xcrun coremlcompiler compile X.mlpackage . or MLModel.compileModel(at:), about 5 s). manifest.json lists the .mlmodelc files of the reference compile (xcrun coremlcompiler 3600.25.1).

Without the weight sharing the same 13 functions are 3.4 GB as separate packages; merged they are 363 MB.

How it was produced

tools/convert/build.py (in the ANEEmbed repository, pinned requirements.txt) converts the original model.safetensors (revision f9972ac22e696a105c482202646979e33e4b5a1e, sha256 checked) with coremltools 9.0 to fp16 ML Programs (minimum macOS 15), merges them into one multifunction package (identical weights are stored once), compiles it with coremlcompiler, extracts embeddings.bf16, copies the vision package unchanged (sha256 checked against the published files), generates the self-test from fp32 references and writes the manifest. On an M5 Ultra it takes 5.5 minutes and 16 GB of RAM, CPU only. A second clean build reproduced the weights, the compiled model.mil, the token table and the self-test byte for byte; the .mlpackage protobuf is message-equal (its byte order and the package UUIDs differ between runs).

The text encoder is re-expressed with numerically neutral rewrites that the ANE needs:

  • RMSNorm divides each row by its max-abs value first (fp16 overflow guard).
  • Attention uses a "fold" layout: the query heads that share one K/V head are folded into the row dimension. The usual repeat_interleave layout is miscompiled by the ANE compiler for the global layers (one K/V head): rebuilt with it, winp2048_b512, winl4096_b512 and winl8192_b512 score cosine 0.960, 0.952 and 0.885 on the ANE against the fp32 reference.
  • Windowed sliding layers. 20 of the 24 layers only attend 512 tokens to each side. In the windowed functions they work on blocks of 512 query rows, each against the 1536 key rows around it (static slices clipped to the sequence), with the mask passed in as a [S, 1536] band. Padding the keys inside the graph instead gave the same numbers but was about 3Γ— slower on the ANE. The 4 global layers attend to every row.
  • Masks are runtime inputs and RoPE uses constant tables at absolute positions. Graph-constant masks and a runtime positions/RoPE input both produced wrong results on the ANE in our tests.
  • The per-layer-embedding projection is computed per layer from a slice of its weight, so every function carries bit-identical weight constants and the merge can share them.

Using it (host contract)

All functions run with cpuAndNeuralEngine. Inputs are fp16, embeddings are fp32. Token rows are bf16 β†’ fp32 Γ— √512 β†’ fp16 from embeddings.bf16. Text sequences are [2] + tokenize(prompt + text) + [1] (bos 2, eos 1; prompts as in the original, e.g. title: none | text: , task: search result | query: ). The exact contract (shapes, masks, pooling) is in manifest.json β†’ bucket_contract and window.

Up to 1280 tokens: embed_S. Pack one or more sequences back to back into the smallest embed_S (S ∈ 64/128/256/512/768/1024/1280, up to K = max(8, S/32) sequences per call): inputs_embeds [1,S,512]; global_bias [1,1,S,S] = 0 where row and column belong to the same sequence (or i = j), else βˆ’10000; local_bias = the same AND |iβˆ’j| ≀ 512; pool [K,S] = 1/n_k over the rows of sequence k β†’ embedding [K,768] (unused rows are 0).

1281 to 2048 tokens: winp1536_b512 / winp2048_b512. The same as embed_S (packing included, K = 48 / 64), except that local_bias is replaced by band_bias [1,1,S,1536]: row r, column c is query row r against key row k = ⌊r/512βŒ‹Β·512 βˆ’ 512 + c; 0 if r and k belong to the same sequence and |r βˆ’ k| ≀ 512, or k = r; else βˆ’10000. Columns whose key row lies outside [0, S) are never read.

2049 to 8192 tokens: winl{3072,4096,6144,8192}_b512, one call per text. S = the smallest size β‰₯ n. inputs_embeds [1,S,512] with rows β‰₯ n zero, band_bias as above for one sequence of n rows, key_bias [1,1,1,S] = 0 for k < n else βˆ’10000, pool [1,S] β†’ embedding [1,768]. pool must be 2⁻¹⁰ on the n valid rows, not 1/n (1/n underflows fp16 inside the ANE pooling at large n; the scale cancels in the normalization).

Images. Preprocess exactly like the HF Gemma4ImageProcessor (aspect-preserving bicubic resize to multiples of 48 px within 9 Γ— budget patches, pixels / 255, 16 Γ— 16 patches), run vision_{70,140,280} (positions, position embeddings from position_embeddings.f16, validity and 3 Γ— 3 pooling matrix as in manifest.json β†’ vision), then embed <bos> <|image> [image tokens] <image|> <eos> with the packing functions like a text sequence; images and texts can share one call.

Accuracy

Cosine similarity against the official model: google/embeddinggemma-2 in fp32 (our PyTorch reference equals the official transformers / sentence-transformers fp32 stack at cosine 1.0000000 for texts of 64 to 8192 tokens), all functions on cpuAndNeuralEngine, Apple M5 Ultra, macOS 27.0.1. Public data only: multilingual READMEs (en/ko/ja/zh/fr) of an Apache-2.0 project, the Korean and Japanese constitutions (public domain), MIT-licensed technical docs and Apache-2.0 source code; images bundled with scikit-learn / networkx / llama.cpp / stable-diffusion.cpp plus synthetic images.

Category n min median
Text, single sequence filling the function (64–2048) 54 0.999966 0.999976
Text, single sequence half the size 54 0.999950 0.999975
Text, packed short sequences (6–200 tokens, up to K per call) 240 0.999936 0.999973
Text, one long + several short in one call (1024–2048) 24 0.999961 0.999977
Text, two long in one call (1024–2048) 8 0.999974 0.999975
All packing-function text 380 0.999936 0.999974
winl3072, 2049–3072 tokens 10 0.999970 0.999974
winl4096, 3073–4096 tokens 10 0.999960 0.999967
winl6144, 4097–6144 tokens 10 0.999931 0.999954
winl8192, 6145–8192 tokens 14 0.999889 0.999921
Image, budget 70, alone (vs official fp32 image embedding) 13 0.999768 0.999972
Image, budget 140, alone 13 0.999813 0.999975
Image, budget 280, alone 13 0.999954 0.999979
Images packed with other images and a text (1024 / 2048) 91 0.999769 0.999975

The long-text lengths include the first token count of every size (2049, 3073, 4097, 6145) and the last (3072, 4096, 6144, 8192). The image error comes almost entirely from the fp16 vision encoder: feeding the official fp32 image tokens through the same ANE text functions gives β‰₯ 0.999981.

Speed (Apple M5 Ultra, macOS 27.0.1)

Median latency of one call, one text filling it, through ANEEmbed's Swift runtime (aneembed bench --buckets):

Function Latency Function Latency
embed_64 3.2 ms winp1536_b512 114.7 ms
embed_128 5.2 ms winp2048_b512 171.0 ms
embed_256 10.7 ms winl3072_b512 286.2 ms
embed_512 27.3 ms winl4096_b512 423.5 ms
embed_768 46.1 ms winl6144_b512 770.9 ms
embed_1024 70.7 ms winl8192_b512 1168.7 ms
embed_1280 93.5 ms

First load per model path, executable and macOS build triggers the ANE compiler (the OS caches the result): the 9 packing functions plus vision took 236 s through ANEEmbed, the 4 long functions another 383 s; with the compile cached, loading all of them and running the self-test takes 4.9 s (2048 tokens) / 9.1 s (8192 tokens). Load only the functions up to the length you serve.

Memory (Apple M5 Ultra, macOS 27.0.1, through ANEEmbed)

Core ML gives each loaded function working memory for the Neural Engine, owned by the loading process but outside its phys_footprint (footprint -w lists it as Owned physical footprint (unmapped) (neural)): 5.25 GB for the 9 packing functions plus the vision functions, 13.3 GB with the 4 long functions added. It is wired only while used: all of it for about 60 s after the functions first run, then nothing while idle (purgeable: under memory pressure the OS discarded 1.6 GB of it, and the next calls recreated it), and during a call plus about 10 s only the buffers of the called function (266 MB for a short query, 579 MB for a 1617-token text, 3.9 GB for a 6631-token text in winl8192_b512). Load only the functions up to the length you serve.

Self-test (run it at load)

The ANE compiler output depends on the OS version, and a miscompile does not raise an error, it returns wrong numbers. selftest.json / selftest.bin contain one deterministic test per function (16 tests): text inputs are built from token ids stored in the JSON (from this package's LICENSE text) and the shipped table, image inputs from a procedural integer pattern; expected fp32 outputs come from the fp32 reference (official fp32 for the vision tests). Text tests compare embeddings by cosine (packing tests also require unused rows to stay 0); vision tests compare an 8-column random projection (digest) of the image tokens by cosine and relative error. Recommended tolerance: cosine β‰₯ 0.999, relative error ≀ 0.05; measured on M5 Ultra: cosine β‰₯ 0.99990 for every function, relative error ≀ 0.015 (vision). The repeat-layout builds above fail it on the ANE (0.885–0.960). Reference runner: tools/convert/selftest.py run <dir>.

Limitations

  • Apple Neural Engine only. The package was tuned and verified with cpuAndNeuralEngine; other compute units are not tested.
  • Text longer than 8192 tokens must be split by the caller. Images use the HF preprocessing; llama.cpp's image preprocessing differs, so image embeddings are not interchangeable with llama.cpp GGUF ones.
  • Audio and video are not included in this package.
  • First load per model path compiles for the ANE (about 10 minutes for all text functions on M5 Ultra); move the directory, update macOS or run it from another executable and it compiles again.

License and attribution

Apache-2.0, see LICENSE and NOTICE. Original model: EmbeddingGemma 2 by Google DeepMind (google/embeddinggemma-2); use must also follow the Gemma Prohibited Use Policy. Vision encoder conversion and position tables: FluidInference/embeddinggemma-2-coreml (Apache-2.0), unchanged.