LAION larger CLAP, general audio (HTSAT-base), ONNX

An ONNX export of laion/larger_clap_general: LAION's larger CLAP for general audio: an HTSAT-base audio encoder and a RoBERTa-base text encoder, trained on general audio, music and speech. Two graphs, the audio tower and the text tower, for zero-shot audio classification (score a clip against labels written in plain words) and audio and text embeddings, with onnxruntime and no PyTorch. The weights are unchanged. This is not an official release of the original authors.

Files

File Input Output
audio.onnx audio: float32 [batch, 480000], mono, 48000 Hz (10 s) embedding: float32 [batch, 512], L2-normalised
audio.int8.onnx the same the same (weights of the matrix products in int8)
text.onnx input_ids, attention_mask: int64 [batch, tokens] embedding: float32 [batch, 512], L2-normalised
tokenizer.json the text tower's tokenizer (Hugging Face tokenizers)
clap.json sample rate, clip length, padding, logit scale, the prompt used in testing

The log-mel front end is inside the audio graph (a strided convolution with the windowed DFT, the mel filter bank and the log). Feed exactly 10 s: a shorter clip is repeated whole and then padded with zeros (padding: repeatpad); for a longer one embed 10 s windows and average them. Truncate to 77 tokens; pad id 1 with attention mask 0.

Zero-shot: softmax(logit_scale * audio_embedding @ text_embeddings.T) over the labels, with logit_scale 38.6647 from clap.json, and texts such as "This is a sound of a dog barking.".

Parity with the PyTorch original

export_clap_onnx.py exports the graphs and compares them with the original in PyTorch (torch 2.5.1+cpu on CPU) on three speech clips (cut to 10 s) and three synthetic ones (a chord progression with a beat, white noise and a 1 kHz tone), against 15 labels:

fp32 graphs int8 audio graph
audio embedding, max abs difference 1.1e-05 5.5e-03
audio embedding, lowest cosine to PyTorch 1.000000 0.999173
text embedding, max abs difference 1.4e-07 (fp32 text graph)
label probabilities, max abs difference 7.8e-05 3.4e-02
same top label 6 of 6 6 of 6

Top label per clip (PyTorch / ONNX fp32):

clip PyTorch ONNX
en-short a woman speaking 0.9968 a woman speaking 0.9968
hi-short a woman speaking 0.9731 a woman speaking 0.9731
en-long music 0.5206 music 0.5206
music music 0.5174 music 0.5174
noise white noise 0.5971 white noise 0.5971
tone a pure tone 0.5046 a pure tone 0.5046

parity.json has every number. The text tower is not offered in int8: dynamic int8 quantization moved its embeddings too far (cosine about 0.96).

Licence and attribution

Apache-2.0, like the weights in laion/larger_clap_general (LAION; the original LAION-AI/CLAP code and checkpoints are CC0-1.0). See LICENSE. Credit: Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick and Shlomo Dubnov, "Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation" (ICASSP 2023), and Ke Chen et al., HTS-AT (ICASSP 2022). The export script is Apache-2.0.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for codebucketsolutions/larger-clap-general-onnx

Quantized
(4)
this model