Arm/embeddinggemma-300m-int8-litert

EmbeddingGemma-300M optimized for Arm-based mobile CPUs with SME2

EmbeddingGemma-300M sentence-embedding encoder optimized as an INT8 LiteRT .tflite model for Arm-based mobile CPUs with SME2.

Summary

This repository contains an Arm-optimized version of google/embeddinggemma-300m for text embeddings. The model is provided in LiteRT .tflite (LiteRT runtime), targeting Mobile CPU systems.

This version is intended to demonstrate efficient inference on Arm-based platforms while preserving the original model's intended behavior. Arm has evaluated this model on WikiText-2 and measured performance on a representative evaluation target.

The exported graph is a complete sentence-embedding pipeline: token IDs go in, and a mean-pooled, Dense-projected, L2-normalized 768-dimensional embedding comes out. No pooling or normalization is needed on the host side.

Key results

Area	Result
Model format	LiteRT .tflite
Target device class	Mobile CPU
Reference device	vivo X300 (C1-Ultra, C1-Premium, C1-Pro, Android 16 / OriginOS 6)
Primary performance result	239.46 ms p50 latency (4.18 requests/second)
Accuracy result	99.76% cosine-similarity accuracy vs the FP32 reference
Size / memory result	302.30 MB (.tflite), 3.90x smaller than FP32; 259.42 MB peak memory

Original model

Field	Value
Original model	google/embeddinggemma-300m
Original source	Hugging Face
Original developer	Google
Original model card	google/embeddinggemma-300m
Original license	Gemma

Model files

File	Description
google__embeddinggemma-300m_litert_optimized.tflite	Arm-optimized INT8 model for deployment
example.py	Minimal inference example
config.yaml	Model I/O contract used by the example
pyproject.toml	Pinned runtime dependencies and deployment metadata
uv.lock	Locked dependency versions, sources, and hashes for uv sync --frozen
benchmarks/	FP32 baseline and Arm-optimized benchmark records

Performance

Performance was measured on the reference configuration below. Results are intended to make the optimization reproducible but do not guarantee identical performance on every Arm-based system.

Reference configuration

Field	Value
Device / platform	vivo X300
CPU / accelerator	C1-Ultra, C1-Premium, C1-Pro (8 cores @ 4.21 GHz), CPU
OS	android — Android 16 / OriginOS 6
Runtime	LiteRT 2.1.6
Backend / delegate	XNNPACK + KleidiAI
Batch size	1
Precision	INT8 dynamic range: INT8 symmetric per-channel weights and INT8 activations quantized at runtime, per invocation; the tensors passed between ops are FP32
Runs	20 warmup + 100 measured

Performance results

Metric	Original / baseline	Arm-optimized	Improvement
p50 latency	330.97 ms	239.46 ms	1.38x
p90 latency	337.52 ms	251.46 ms	1.34x
Requests per second	3.02	4.18	1.38x
Model load time	510.62 ms	277.15 ms	1.84x
Time to first inference	338.89 ms	261.02 ms	1.30x
Peak memory	860.07 MB	259.42 MB	3.32x less
Average memory	860.07 MB	259.41 MB	3.32x less
Model size	1177.61 MB	302.30 MB	3.90x smaller

Accuracy

Accuracy was evaluated using the same preprocessing, input resolution, and evaluation protocol described below. Where possible, the optimized model is compared against the original model under the same evaluation conditions.

Evaluation setup

Field	Value
Dataset	WikiText-2
Split	validation
Number of samples	2891
Metric(s)	Cosine similarity accuracy, Embedding drift (cosine distance)
Evaluation runtime	LiteRT

Accuracy results

Metric	Original / baseline	Arm-optimized	Change
Cosine similarity accuracy	100.00%	99.76%	-0.24 pp
Embedding drift (cosine distance)	0.0000	0.0024	+0.0024

Accuracy was measured using the evaluation setup described above. Users should re-evaluate the model on their own data before production use.

Arm optimization approach

Arm optimized this model for efficient inference on Arm-based platforms using a hardware-aware conversion and validation flow.

For this release, Arm used:

Optimization area	Applied?	Notes
Model conversion	Yes	Captured with torch.export at a static [1, 512] shape and converted to LiteRT .tflite, with the sentence-transformers Dense head (768 -> 3072 -> 768), mean pooling and L2 normalization folded into the graph
Quantization	Yes	Post-training dynamic-range INT8 via AI Edge Quantizer (dynamic_wi8_afp32) on the converted .tflite. Weights are INT8 symmetric per-channel. Activations are quantized to INT8 as well, dynamically: the kernel measures each activation block's range at every invocation instead of reading it from a calibration set, so the matmuls execute in INT8 with INT32 accumulation. Being dynamic, activation quantization is stored nowhere in the graph — no calibration set, no observers, no Q/DQ nodes — and the tensors passed between ops are FP32
Runtime/backend selection	Yes	XNNPACK + KleidiAI
Graph/runtime compatibility updates	Yes	Three pre-conversion rewrites, none of which change the mathematics: attention folded head-into-batch as two batched matmuls, so the batch dimension stays above 1 and the score matmuls are not rank-reduced into fully-connected ops that the accelerated kernel would reject; the bidirectional additive attention masks precomputed and passed in, so Gemma3's vmap-based mask builder never enters the exported graph; and pow(x, 2) rewritten to mul(x, x)
Accuracy validation	Yes	Compared against the FP32 baseline
Performance validation	Yes	Measured on vivo X300

The goal of this process is to improve deployment characteristics such as latency, memory use, model size, and runtime compatibility while preserving the model's intended behavior. Detailed conversion scripts, calibration configuration, or backend-specific implementation details may be provided separately where appropriate.

Using this model

The upstream repository google/embeddinggemma-300m is gated on the Hugging Face Hub. The tokenizer used by the example is downloaded from that repository, so you must accept the Gemma license on the model page and authenticate before running inference.

Install dependencies

Dependencies are pinned in pyproject.toml and locked in uv.lock. From this directory:

uv python install
uv sync --frozen
uv run hf auth login   # required: the tokenizer lives in a gated repo

Run the example

uv run example.py

Note: The Python/uv example runs on AWS Graviton (Ubuntu arm64) to confirm runtime compatibility only, and is intended as a guideline for building an equivalent run script on Mobile CPU systems.

example.py has no CLI arguments — edit the TEXTS list at the top of the file to embed your own sentences. It prints one block per input and a pairwise cosine-similarity matrix, then writes embeddings.json next to the script.

Expected input

Property	Value
Input shape	[1, 512] — input_ids [1, 512] and attention_mask [1, 512]
Input type	int64
Input range	Token IDs from the google/embeddinggemma-300m tokenizer; attention mask 0 or 1
Preprocessing	Tokenize with the google/embeddinggemma-300m tokenizer, pad to max_length 512, truncate longer text, keep both tensors contiguous

Expected output

Property	Value
Output shape	[1, 768]
Output type	L2-normalized sentence embedding (float32), 768 dimensions
Postprocessing	None on the host: mean pooling, Dense projection and L2 normalization run in-graph; a dot product between two embeddings is the cosine similarity

Intended use

This model is intended for developers evaluating text-embedding workloads on Arm-based platforms. It is suitable as a reference implementation for benchmarking, prototyping, and integration exploration.

Limitations

• Performance depends on the target device, runtime version, backend/delegate support, memory configuration, and system load.
• This release preserves the original model's intended task and behavior, but users should validate it for their own application, data, and deployment environment.
• The graph has a fixed [1, 512] input shape: batch size 1 only, and longer text is truncated.
• This repository is not a replacement for the original model documentation.

Additional notes

• No weights are skipped: the entire weight set is quantized, including the [262144, 768] token-embedding lookup table, which is INT8 in the exported graph and accounts for the bulk of the size reduction.
• The graph has a fixed [1, 512] input shape: batch size 1 only, every input padded to exactly 512 tokens, longer text truncated.

About this version

Original Model: google/embeddinggemma-300m by Google - Repository

Optimization/conversion: Arm-Optimized version for execution on Arm-based platforms.

Converted/optimized by: Arm

License: The Original Model and the Optimized Model are subject to Gemma Terms of Use.

This repository contains a converted or optimized version of the Original Model (the “Optimized Model”). The Original Model has been converted or optimized as described above for execution on Arm-based platforms.

No retraining or fine-tuning of the Original Model was performed as part of the conversion or optimization. The conversion or optimization was not intended to change the Original Model’s behavior or intended use.

Original Model and Documentation

For information about the Original Model, including its development, training data, intended uses, limitations and other relevant information, please refer to the Original Model repository. Information in that repository was provided by the original developer or other third parties and, unless expressly stated otherwise, has not been independently verified by Arm.

Licenses and Third-Party Terms

Use of the Original Model and the Optimized Model is subject to the applicable licenses, usage restrictions and other terms identified above and in the relevant repositories. Publication of the Optimized Model does not grant any rights beyond those provided under the applicable license terms.

You are responsible for reviewing those terms and ensuring that your use of the Original Model and the Optimized Model is permitted.

Purpose of this Release

The Optimized Model is provided as a reference implementation to demonstrate and evaluate execution and performance on Arm-based systems. It is not a production-ready or supported solution.

Arm’s publication of the Optimized Model does not constitute an endorsement or certification of the Original Model or a representation that the Optimized Model is suitable for production use or any particular purpose.

To the fullest extent permitted by applicable law (i) the Optimized Model is provided “as is.” Arm makes no representations or warranties that the Original Model, the Optimized Model or their outputs are accurate, safe, secure, non-infringing, legally compliant, suitable for production use or fit for any particular purpose; and (ii) Arm will not be liable for any loss or damage arising from or in connection with the Optimized Model, its use or its outputs.

You are responsible for independently evaluating the Optimized Model, its outputs and its suitability for your intended use, including compliance with applicable legal, regulatory, safety and security requirements.

Arm does not commit to provide ongoing support, maintenance or updates for the Optimized Model. Any use of or reliance on the Optimized Model or its outputs is at your own risk.