File size: 2,635 Bytes
87f9d21 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 | ---
license: other
license_name: apache-2.0-and-mit
license_link: https://huggingface.co/DeafBoyAudio/audio-compass-models/blob/main/LICENSE-CLAP
tags:
- audio
- onnx
- clap
- crepe
- pitch-detection
- zero-shot-audio-classification
library_name: onnx
pipeline_tag: audio-classification
---
# Audio Compass models
ONNX models used by **Audio Compass**, a sample-finder plugin by Deaf Boy Audio. The plugin downloads
these files itself (*⋯ › Download models*) and checks every file against `SHA256SUMS`, so there is
nothing to do by hand.
| File | What it is | Licence |
|---|---|---|
| `clap_audio.onnx` | CLAP audio encoder: 48 kHz mono, 10 s → 512-d L2-normalised embedding. Log-mel front end built into the graph. Dynamic int8 weights (front end kept fp32). | Apache-2.0 |
| `clap_text.onnx` | CLAP text encoder: token ids → 512-d L2-normalised embedding. Dynamic int8. | Apache-2.0 |
| `clap_vocab.json`, `clap_merges.txt` | RoBERTa byte-level BPE tokenizer files | Apache-2.0 |
| `clap_config.json` | Every preprocessing parameter (sample rate, crop/pad rule, mel settings, token ids) | Apache-2.0 |
| `crepe_tiny.onnx` | CREPE "tiny" pitch tracker: 1024-sample frames @ 16 kHz → 360-bin salience. Per-frame normalisation built into the graph. | MIT |
| `SHA256SUMS` | Checksums of the files above | — |
## Where they come from (modified works)
- **CLAP**: converted from [`laion/larger_clap_music_and_speech`](https://huggingface.co/laion/larger_clap_music_and_speech)
(LAION, Apache-2.0). Changes: exported to ONNX as separate audio and text encoders with projection +
L2 normalisation, mel spectrogram moved into the graph, weights dynamically quantised to int8.
Licence: [LICENSE-CLAP](LICENSE-CLAP).
- **CREPE tiny**: the weights distributed with [`torchcrepe`](https://github.com/maxrmorrison/torchcrepe) 0.0.24
(MIT, © 2020 Max Morrison), a port of [CREPE](https://github.com/marl/crepe) (MIT, © 2018 Jong Wook Kim,
Justin Salamon, Peter Li, Juan Pablo Bello). Changes: exported to ONNX with per-frame normalisation in the
graph. Licence: [LICENSE-CREPE](LICENSE-CREPE).
The files are produced by `tools/export_models.py` in the Audio Compass source code; its parity tests
check the ONNX outputs against the original PyTorch models (cosine ≥ 0.99 for CLAP, within 10 cents for CREPE).
## Citations
- Wu, Chen, Zhang, Hui, Berg-Kirkpatrick, Dubnov. *Large-scale Contrastive Language-Audio Pretraining with
Feature Fusion and Keyword-to-Caption Augmentation*. ICASSP 2023.
- Kim, Salamon, Li, Bello. *CREPE: A Convolutional Representation for Pitch Estimation*. ICASSP 2018.
|