Download README.md from codebucketsolutions/larger-clap-general-onnx: direct link, hf CLI and curl.
- Browser
- Download file 3.59 kB
-
https://huggingface.co/codebucketsolutions/larger-clap-general-onnx/resolve/main/README.md
- Command line
-
hf download hf://codebucketsolutions/larger-clap-general-onnx/README.md
-
curl -L -o README.md https://huggingface.co/codebucketsolutions/larger-clap-general-onnx/resolve/main/README.md
license: apache-2.0
library_name: onnx
base_model: laion/larger_clap_general
tags:
- clap
- zero-shot-audio-classification
- audio-classification
- onnx
inference: false
LAION larger CLAP, general audio (HTSAT-base), ONNX
An ONNX export of laion/larger_clap_general: LAION's larger CLAP for general audio: an HTSAT-base audio encoder and a RoBERTa-base text encoder, trained on general audio, music and speech. Two graphs, the audio tower and the text tower, for zero-shot audio classification (score a clip against labels written in plain words) and audio and text embeddings, with onnxruntime and no PyTorch. The weights are unchanged. This is not an official release of the original authors.
Files
| File | Input | Output |
|---|---|---|
audio.onnx |
audio: float32 [batch, 480000], mono, 48000 Hz (10 s) |
embedding: float32 [batch, 512], L2-normalised |
audio.int8.onnx |
the same | the same (weights of the matrix products in int8) |
text.onnx |
input_ids, attention_mask: int64 [batch, tokens] |
embedding: float32 [batch, 512], L2-normalised |
tokenizer.json |
the text tower's tokenizer (Hugging Face tokenizers) |
|
clap.json |
sample rate, clip length, padding, logit scale, the prompt used in testing |
The log-mel front end is inside the audio graph (a strided convolution with the windowed DFT, the mel filter bank
and the log). Feed exactly 10 s: a shorter clip is repeated whole and then padded with zeros
(padding: repeatpad); for a longer one embed 10 s windows and average them. Truncate to
77 tokens; pad id 1 with attention mask 0.
Zero-shot: softmax(logit_scale * audio_embedding @ text_embeddings.T) over the labels, with logit_scale
38.6647 from clap.json, and texts such as "This is a sound of a dog barking.".
Parity with the PyTorch original
export_clap_onnx.py exports the graphs and compares them with the original in PyTorch (torch 2.5.1+cpu on CPU)
on three speech clips (cut to 10 s) and three synthetic ones (a chord progression with a beat, white noise and a
1 kHz tone), against 15 labels:
| fp32 graphs | int8 audio graph | |
|---|---|---|
| audio embedding, max abs difference | 1.1e-05 | 5.5e-03 |
| audio embedding, lowest cosine to PyTorch | 1.000000 | 0.999173 |
| text embedding, max abs difference | 1.4e-07 | (fp32 text graph) |
| label probabilities, max abs difference | 7.8e-05 | 3.4e-02 |
| same top label | 6 of 6 | 6 of 6 |
Top label per clip (PyTorch / ONNX fp32):
| clip | PyTorch | ONNX |
|---|---|---|
| en-short | a woman speaking 0.9968 | a woman speaking 0.9968 |
| hi-short | a woman speaking 0.9731 | a woman speaking 0.9731 |
| en-long | music 0.5206 | music 0.5206 |
| music | music 0.5174 | music 0.5174 |
| noise | white noise 0.5971 | white noise 0.5971 |
| tone | a pure tone 0.5046 | a pure tone 0.5046 |
parity.json has every number. The text tower is not offered in int8: dynamic int8 quantization moved its
embeddings too far (cosine about 0.96).
Licence and attribution
Apache-2.0, like the weights in laion/larger_clap_general (LAION; the original LAION-AI/CLAP code and checkpoints are CC0-1.0). See LICENSE. Credit: Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick and Shlomo Dubnov, "Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation" (ICASSP 2023), and Ke Chen et al., HTS-AT (ICASSP 2022). The export script is Apache-2.0.