LAION CLAP (HTSAT-tiny audio encoder, unfused), ONNX
An ONNX export of laion/clap-htsat-unfused: LAION's general-purpose CLAP with the HTSAT-tiny audio encoder and a RoBERTa-base text encoder, trained on LAION-Audio-630K, as converted to transformers by LAION. Two graphs, the audio tower and the text tower, for zero-shot audio classification (score a clip against labels written in plain words) and audio and text embeddings, with onnxruntime and no PyTorch. The weights are unchanged. This is not an official release of the original authors.
Files
| File | Input | Output |
|---|---|---|
audio.onnx |
audio: float32 [batch, 480000], mono, 48000 Hz (10 s) |
embedding: float32 [batch, 512], L2-normalised |
audio.int8.onnx |
the same | the same (weights of the matrix products in int8) |
text.onnx |
input_ids, attention_mask: int64 [batch, tokens] |
embedding: float32 [batch, 512], L2-normalised |
tokenizer.json |
the text tower's tokenizer (Hugging Face tokenizers) |
|
clap.json |
sample rate, clip length, padding, logit scale, the prompt used in testing |
The log-mel front end is inside the audio graph (a strided convolution with the windowed DFT, the mel filter bank
and the log). Feed exactly 10 s: a shorter clip is repeated whole and then padded with zeros
(padding: repeatpad); for a longer one embed 10 s windows and average them. Truncate to
77 tokens; pad id 1 with attention mask 0.
Zero-shot: softmax(logit_scale * audio_embedding @ text_embeddings.T) over the labels, with logit_scale
18.6612 from clap.json, and texts such as "This is a sound of a dog barking.".
Parity with the PyTorch original
export_clap_onnx.py exports the graphs and compares them with the original in PyTorch (torch 2.5.1+cpu on CPU)
on three speech clips (cut to 10 s) and three synthetic ones (a chord progression with a beat, white noise and a
1 kHz tone), against 15 labels:
| fp32 graphs | int8 audio graph | |
|---|---|---|
| audio embedding, max abs difference | 1.1e-05 | 6.7e-03 |
| audio embedding, lowest cosine to PyTorch | 1.000000 | 0.998740 |
| text embedding, max abs difference | 2.6e-07 | (fp32 text graph) |
| label probabilities, max abs difference | 8.1e-06 | 1.4e-02 |
| same top label | 6 of 6 | 6 of 6 |
Top label per clip (PyTorch / ONNX fp32):
| clip | PyTorch | ONNX |
|---|---|---|
| en-short | a woman speaking 0.9791 | a woman speaking 0.9791 |
| hi-short | a woman speaking 0.9774 | a woman speaking 0.9774 |
| en-long | a man speaking 0.4225 | a man speaking 0.4225 |
| music | a piano 0.4935 | a piano 0.4935 |
| noise | white noise 0.5332 | white noise 0.5333 |
| tone | a beep 0.7187 | a beep 0.7187 |
parity.json has every number. The text tower is not offered in int8: dynamic int8 quantization moved its
embeddings too far (cosine about 0.96).
Licence and attribution
Apache-2.0, like the weights in laion/clap-htsat-unfused (LAION; the original LAION-AI/CLAP code and checkpoints are CC0-1.0). See LICENSE. Credit: Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick and Shlomo Dubnov, "Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation" (ICASSP 2023), and Ke Chen et al., HTS-AT (ICASSP 2022). The export script is Apache-2.0.
Model tree for codebucketsolutions/clap-htsat-unfused-onnx
Base model
laion/clap-htsat-unfused