embeddinggemma-2-audio-static-onnx

The audio encoder of EmbeddingGemma 2 as ONNX with static shapes and INT8 per-channel weights, so it compiles and runs on the Intel NPU. The text half is arankays/embeddinggemma-2-text-int8-cw-onnx; its onnx/model_audio_250.onnx takes this encoder's output directly.

What changed

onnx-community/embeddinggemma-2-ONNX's audio_encoder.onnx takes any number of log-mel frames. The NPU needs static shapes (karanshukla/vinoSearch#190).

This graph is fixed at 10 s: 999 frames in, 250 tokens out. Its weights are INT8 symmetric per-channel (295 MB, down from 1.17 GB). Made by bench/static_audio.py.

Graph	input_features	input_features_mask	audio_features	audio_mask
onnx/audio_encoder_static_10s_int8.onnx	[1, 999, 128]	[1, 999] bool	[250, 512]	[1, 250]

Usage

1. Features. Run up to 10 s of 16 kHz mono audio through Gemma4AudioFeatureExtractor (preprocessor_config.json): 128 HTK mels from 0 to 8 kHz, 320-sample periodic Hann frames every 160 samples, a 512-point FFT, ln(mel + 0.001). 10 s gives exactly 999 frames. Pad shorter clips with zero rows and a false mask to 999.
2. Keep the valid rows. For n real frames, the first ceil(ceil(n / 2) / 2) rows of audio_features are real (two stride-2 convolutions). audio_mask says the same, and is right on the NPU too.
3. Embed. Feed the features to the text repo's model_audio_250.onnx with input_ids = [2, 256000, 258881 × N, 258883, 1] (bos, begin-of-audio, N audio tokens, end-of-audio, eos), N being the rows kept, padded at the end with id 0 and attention_mask 0 to [1, 256]. Pass all 250 rows unsliced: the model merges the first N. Use sentence_embedding, L2-normalised. Audio and text share one space, so text queries (task: search result | query: …) search sounds directly.

import math

import numpy as np
import openvino as ov
from transformers import AutoFeatureExtractor

fe = AutoFeatureExtractor.from_pretrained(".")
core = ov.Core()
audio = core.compile_model("onnx/audio_encoder_static_10s_int8.onnx", "NPU", {"CACHE_DIR": "ov-cache"})
f = fe([wave], sampling_rate=16000, return_tensors="np")  # wave: up to 160,000 float32 samples
x, m = f["input_features"], f["input_features_mask"].astype(bool)
n = int(m.sum())
px, pm = np.zeros((1, 999, 128), np.float32), np.zeros((1, 999), bool)
px[:, :x.shape[1]], pm[:, :m.shape[1]] = x[:, :999], m[:, :999]
feats = audio({"input_features": px, "input_features_mask": pm})["audio_features"]
tokens = math.ceil(math.ceil(n / 2) / 2)  # feats[:tokens] are the clip's features

Results

Measured on a Core 5 320 (Wildcat Lake NPU), OpenVINO 2026.4.1, on 2026-11-11, on ESC-50 clips and espeak speech (bench/audio_check.py). The reference is upstream's fp32 audio_encoder.onnx on ONNX Runtime.

	Feature cosine, mean	Sentence embedding cosine, worst clip	Per 10 s on the NPU
Static fp32, NPU	0.99982	0.99985
Static INT8 (this repo), NPU	0.99680	0.99838	64 ms (+ 34 ms for model_audio_250)
Static INT8, ONNX Runtime CPU	0.99683	0.99851

Text queries against 10 s clips, INT8 on the NPU through model_audio_250.onnx (best per query in bold):

	speech	dog	bells	rain	siren
a dog barking	0.591	0.752	0.605	0.644	0.666
church bells ringing	0.556	0.590	0.690	0.637	0.649
rain falling	0.582	0.566	0.575	0.668	0.603
an ambulance siren	0.572	0.624	0.616	0.642	0.722

Sounds find their clips. Speech does not ("someone talking" scores the rain clip highest): search spoken words through a transcript instead.

NPU notes:

• Up to 18 of 250 feature rows drift under INT8 (worst 0.51 cosine, on church bells), on CPU as on the NPU. They wash out in the text model's pooling.
• First compile takes about 9 s; use a CACHE_DIR.

Used by vinoSearch.

License

Apache 2.0, same as the base model. EmbeddingGemma 2 is by Google DeepMind. Modified from onnx-community/embeddinggemma-2-ONNX: input fixed at 10 s, weights compressed to INT8.