LAION CLAP (HTSAT-tiny audio encoder, unfused), ONNX

An ONNX export of laion/clap-htsat-unfused: LAION's general-purpose CLAP with the HTSAT-tiny audio encoder and a RoBERTa-base text encoder, trained on LAION-Audio-630K, as converted to transformers by LAION. Two graphs, the audio tower and the text tower, for zero-shot audio classification (score a clip against labels written in plain words) and audio and text embeddings, with onnxruntime and no PyTorch. The weights are unchanged. This is not an official release of the original authors.

Files

File Input Output
audio.onnx audio: float32 [batch, 480000], mono, 48000 Hz (10 s) embedding: float32 [batch, 512], L2-normalised
audio.int8.onnx the same the same (weights of the matrix products in int8)
text.onnx input_ids, attention_mask: int64 [batch, tokens] embedding: float32 [batch, 512], L2-normalised
tokenizer.json the text tower's tokenizer (Hugging Face tokenizers)
clap.json sample rate, clip length, padding, logit scale, the prompt used in testing

The log-mel front end is inside the audio graph (a strided convolution with the windowed DFT, the mel filter bank and the log). Feed exactly 10 s: a shorter clip is repeated whole and then padded with zeros (padding: repeatpad); for a longer one embed 10 s windows and average them. Truncate to 77 tokens; pad id 1 with attention mask 0.

Zero-shot: softmax(logit_scale * audio_embedding @ text_embeddings.T) over the labels, with logit_scale 18.6612 from clap.json, and texts such as "This is a sound of a dog barking.".

Parity with the PyTorch original

export_clap_onnx.py exports the graphs and compares them with the original in PyTorch (torch 2.5.1+cpu on CPU) on three speech clips (cut to 10 s) and three synthetic ones (a chord progression with a beat, white noise and a 1 kHz tone), against 15 labels:

fp32 graphs int8 audio graph
audio embedding, max abs difference 1.1e-05 6.7e-03
audio embedding, lowest cosine to PyTorch 1.000000 0.998740
text embedding, max abs difference 2.6e-07 (fp32 text graph)
label probabilities, max abs difference 8.1e-06 1.4e-02
same top label 6 of 6 6 of 6

Top label per clip (PyTorch / ONNX fp32):

clip PyTorch ONNX
en-short a woman speaking 0.9791 a woman speaking 0.9791
hi-short a woman speaking 0.9774 a woman speaking 0.9774
en-long a man speaking 0.4225 a man speaking 0.4225
music a piano 0.4935 a piano 0.4935
noise white noise 0.5332 white noise 0.5333
tone a beep 0.7187 a beep 0.7187

parity.json has every number. The text tower is not offered in int8: dynamic int8 quantization moved its embeddings too far (cosine about 0.96).

Licence and attribution

Apache-2.0, like the weights in laion/clap-htsat-unfused (LAION; the original LAION-AI/CLAP code and checkpoints are CC0-1.0). See LICENSE. Credit: Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick and Shlomo Dubnov, "Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation" (ICASSP 2023), and Ke Chen et al., HTS-AT (ICASSP 2022). The export script is Apache-2.0.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for codebucketsolutions/clap-htsat-unfused-onnx

Quantized
(10)
this model