| --- |
| license: other |
| license_name: apache-2.0-and-mit |
| license_link: https://huggingface.co/DeafBoyAudio/audio-compass-models/blob/main/LICENSE-CLAP |
| tags: |
| - audio |
| - onnx |
| - clap |
| - crepe |
| - pitch-detection |
| - zero-shot-audio-classification |
| library_name: onnx |
| pipeline_tag: audio-classification |
| --- |
| |
| # Audio Compass models |
|
|
| ONNX models used by **Audio Compass**, a sample-finder plugin by Deaf Boy Audio. The plugin downloads |
| these files itself (*⋯ › Download models*) and checks every file against `SHA256SUMS`, so there is |
| nothing to do by hand. |
|
|
| | File | What it is | Licence | |
| |---|---|---| |
| | `clap_audio.onnx` | CLAP audio encoder: 48 kHz mono, 10 s → 512-d L2-normalised embedding. Log-mel front end built into the graph. Dynamic int8 weights (front end kept fp32). | Apache-2.0 | |
| | `clap_text.onnx` | CLAP text encoder: token ids → 512-d L2-normalised embedding. Dynamic int8. | Apache-2.0 | |
| | `clap_vocab.json`, `clap_merges.txt` | RoBERTa byte-level BPE tokenizer files | Apache-2.0 | |
| | `clap_config.json` | Every preprocessing parameter (sample rate, crop/pad rule, mel settings, token ids) | Apache-2.0 | |
| | `crepe_tiny.onnx` | CREPE "tiny" pitch tracker: 1024-sample frames @ 16 kHz → 360-bin salience. Per-frame normalisation built into the graph. | MIT | |
| | `SHA256SUMS` | Checksums of the files above | — | |
|
|
| ## Where they come from (modified works) |
|
|
| - **CLAP**: converted from [`laion/larger_clap_music_and_speech`](https://huggingface.co/laion/larger_clap_music_and_speech) |
| (LAION, Apache-2.0). Changes: exported to ONNX as separate audio and text encoders with projection + |
| L2 normalisation, mel spectrogram moved into the graph, weights dynamically quantised to int8. |
| Licence: [LICENSE-CLAP](LICENSE-CLAP). |
| - **CREPE tiny**: the weights distributed with [`torchcrepe`](https://github.com/maxrmorrison/torchcrepe) 0.0.24 |
| (MIT, © 2020 Max Morrison), a port of [CREPE](https://github.com/marl/crepe) (MIT, © 2018 Jong Wook Kim, |
| Justin Salamon, Peter Li, Juan Pablo Bello). Changes: exported to ONNX with per-frame normalisation in the |
| graph. Licence: [LICENSE-CREPE](LICENSE-CREPE). |
|
|
| The files are produced by `tools/export_models.py` in the Audio Compass source code; its parity tests |
| check the ONNX outputs against the original PyTorch models (cosine ≥ 0.99 for CLAP, within 10 cents for CREPE). |
|
|
| ## Citations |
|
|
| - Wu, Chen, Zhang, Hui, Berg-Kirkpatrick, Dubnov. *Large-scale Contrastive Language-Audio Pretraining with |
| Feature Fusion and Keyword-to-Caption Augmentation*. ICASSP 2023. |
| - Kim, Salamon, Li, Bello. *CREPE: A Convolutional Representation for Pitch Estimation*. ICASSP 2018. |
|
|
