|
Download README.md from codebucketsolutions/larger-clap-general-onnx: direct link, hf CLI and curl.
- Browser
- Download file 3.59 kB
-
https://huggingface.co/codebucketsolutions/larger-clap-general-onnx/resolve/main/README.md
- Command line
-
hf download hf://codebucketsolutions/larger-clap-general-onnx/README.md
-
curl -L -o README.md https://huggingface.co/codebucketsolutions/larger-clap-general-onnx/resolve/main/README.md
3.59 kB
| license: apache-2.0 | |
| library_name: onnx | |
| base_model: laion/larger_clap_general | |
| tags: | |
| - clap | |
| - zero-shot-audio-classification | |
| - audio-classification | |
| - onnx | |
| inference: false | |
| # LAION larger CLAP, general audio (HTSAT-base), ONNX | |
| An ONNX export of [laion/larger_clap_general](https://huggingface.co/laion/larger_clap_general): LAION's larger CLAP for general audio: an HTSAT-base audio encoder and a RoBERTa-base text encoder, trained on general audio, music and speech. Two graphs, the audio | |
| tower and the text tower, for zero-shot audio classification (score a clip against labels written in plain words) | |
| and audio and text embeddings, with onnxruntime and no PyTorch. The weights are unchanged. This is not an official | |
| release of the original authors. | |
| ## Files | |
| | File | Input | Output | | |
| | --- | --- | --- | | |
| | `audio.onnx` | `audio`: float32 `[batch, 480000]`, mono, 48000 Hz (10 s) | `embedding`: float32 `[batch, 512]`, L2-normalised | | |
| | `audio.int8.onnx` | the same | the same (weights of the matrix products in int8) | | |
| | `text.onnx` | `input_ids`, `attention_mask`: int64 `[batch, tokens]` | `embedding`: float32 `[batch, 512]`, L2-normalised | | |
| | `tokenizer.json` | | the text tower's tokenizer (Hugging Face `tokenizers`) | | |
| | `clap.json` | | sample rate, clip length, padding, logit scale, the prompt used in testing | | |
| The log-mel front end is inside the audio graph (a strided convolution with the windowed DFT, the mel filter bank | |
| and the log). Feed exactly 10 s: a shorter clip is repeated whole and then padded with zeros | |
| (`padding: repeatpad`); for a longer one embed 10 s windows and average them. Truncate to | |
| 77 tokens; pad id 1 with attention mask 0. | |
| Zero-shot: `softmax(logit_scale * audio_embedding @ text_embeddings.T)` over the labels, with `logit_scale` | |
| 38.6647 from `clap.json`, and texts such as "This is a sound of a dog barking.". | |
| ## Parity with the PyTorch original | |
| `export_clap_onnx.py` exports the graphs and compares them with the original in PyTorch (torch 2.5.1+cpu on CPU) | |
| on three speech clips (cut to 10 s) and three synthetic ones (a chord progression with a beat, white noise and a | |
| 1 kHz tone), against 15 labels: | |
| | | fp32 graphs | int8 audio graph | | |
| | --- | --- | --- | | |
| | audio embedding, max abs difference | 1.1e-05 | 5.5e-03 | | |
| | audio embedding, lowest cosine to PyTorch | 1.000000 | 0.999173 | | |
| | text embedding, max abs difference | 1.4e-07 | (fp32 text graph) | | |
| | label probabilities, max abs difference | 7.8e-05 | 3.4e-02 | | |
| | same top label | 6 of 6 | 6 of 6 | | |
| Top label per clip (PyTorch / ONNX fp32): | |
| | clip | PyTorch | ONNX | | |
| | --- | --- | --- | | |
| | en-short | a woman speaking 0.9968 | a woman speaking 0.9968 | | |
| | hi-short | a woman speaking 0.9731 | a woman speaking 0.9731 | | |
| | en-long | music 0.5206 | music 0.5206 | | |
| | music | music 0.5174 | music 0.5174 | | |
| | noise | white noise 0.5971 | white noise 0.5971 | | |
| | tone | a pure tone 0.5046 | a pure tone 0.5046 | | |
| `parity.json` has every number. The text tower is not offered in int8: dynamic int8 quantization moved its | |
| embeddings too far (cosine about 0.96). | |
| ## Licence and attribution | |
| Apache-2.0, like the weights in [laion/larger_clap_general](https://huggingface.co/laion/larger_clap_general) (LAION; the original LAION-AI/CLAP code and checkpoints are CC0-1.0). See `LICENSE`. Credit: Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick and Shlomo Dubnov, "Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation" (ICASSP 2023), and Ke Chen et al., HTS-AT (ICASSP 2022). The export script is Apache-2.0. | |