gotham13's picture
ONNX export of the audio and text towers, parity report
3f10d61 verified
|
Raw History Blame Contribute Delete
3.59 kB
---
license: apache-2.0
library_name: onnx
base_model: laion/larger_clap_general
tags:
- clap
- zero-shot-audio-classification
- audio-classification
- onnx
inference: false
---
# LAION larger CLAP, general audio (HTSAT-base), ONNX
An ONNX export of [laion/larger_clap_general](https://huggingface.co/laion/larger_clap_general): LAION's larger CLAP for general audio: an HTSAT-base audio encoder and a RoBERTa-base text encoder, trained on general audio, music and speech. Two graphs, the audio
tower and the text tower, for zero-shot audio classification (score a clip against labels written in plain words)
and audio and text embeddings, with onnxruntime and no PyTorch. The weights are unchanged. This is not an official
release of the original authors.
## Files
| File | Input | Output |
| --- | --- | --- |
| `audio.onnx` | `audio`: float32 `[batch, 480000]`, mono, 48000 Hz (10 s) | `embedding`: float32 `[batch, 512]`, L2-normalised |
| `audio.int8.onnx` | the same | the same (weights of the matrix products in int8) |
| `text.onnx` | `input_ids`, `attention_mask`: int64 `[batch, tokens]` | `embedding`: float32 `[batch, 512]`, L2-normalised |
| `tokenizer.json` | | the text tower's tokenizer (Hugging Face `tokenizers`) |
| `clap.json` | | sample rate, clip length, padding, logit scale, the prompt used in testing |
The log-mel front end is inside the audio graph (a strided convolution with the windowed DFT, the mel filter bank
and the log). Feed exactly 10 s: a shorter clip is repeated whole and then padded with zeros
(`padding: repeatpad`); for a longer one embed 10 s windows and average them. Truncate to
77 tokens; pad id 1 with attention mask 0.
Zero-shot: `softmax(logit_scale * audio_embedding @ text_embeddings.T)` over the labels, with `logit_scale`
38.6647 from `clap.json`, and texts such as "This is a sound of a dog barking.".
## Parity with the PyTorch original
`export_clap_onnx.py` exports the graphs and compares them with the original in PyTorch (torch 2.5.1+cpu on CPU)
on three speech clips (cut to 10 s) and three synthetic ones (a chord progression with a beat, white noise and a
1 kHz tone), against 15 labels:
| | fp32 graphs | int8 audio graph |
| --- | --- | --- |
| audio embedding, max abs difference | 1.1e-05 | 5.5e-03 |
| audio embedding, lowest cosine to PyTorch | 1.000000 | 0.999173 |
| text embedding, max abs difference | 1.4e-07 | (fp32 text graph) |
| label probabilities, max abs difference | 7.8e-05 | 3.4e-02 |
| same top label | 6 of 6 | 6 of 6 |
Top label per clip (PyTorch / ONNX fp32):
| clip | PyTorch | ONNX |
| --- | --- | --- |
| en-short | a woman speaking 0.9968 | a woman speaking 0.9968 |
| hi-short | a woman speaking 0.9731 | a woman speaking 0.9731 |
| en-long | music 0.5206 | music 0.5206 |
| music | music 0.5174 | music 0.5174 |
| noise | white noise 0.5971 | white noise 0.5971 |
| tone | a pure tone 0.5046 | a pure tone 0.5046 |
`parity.json` has every number. The text tower is not offered in int8: dynamic int8 quantization moved its
embeddings too far (cosine about 0.96).
## Licence and attribution
Apache-2.0, like the weights in [laion/larger_clap_general](https://huggingface.co/laion/larger_clap_general) (LAION; the original LAION-AI/CLAP code and checkpoints are CC0-1.0). See `LICENSE`. Credit: Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick and Shlomo Dubnov, "Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation" (ICASSP 2023), and Ke Chen et al., HTS-AT (ICASSP 2022). The export script is Apache-2.0.