| --- |
| license: apache-2.0 |
| language: |
| - en |
| library_name: sentence-transformers |
| pipeline_tag: sentence-similarity |
| base_model: google/embeddinggemma-2 |
| tags: |
| - sentence-transformers |
| - audio |
| - music |
| - music-information-retrieval |
| - text-to-audio-retrieval |
| - embeddings |
| datasets: |
| - Telecom-Paris/iamd_v0 |
| - google/MusicCaps |
| --- |
| |
| # SonicEmbed |
|
|
| SonicEmbed is an [EmbeddingGemma 2](https://huggingface.co/google/embeddinggemma-2) fine-tune that puts text and music in one 768-dimensional space from a single model that still works for general text. On Song Describer, it beats LAION-CLAP's music checkpoint. The vision tower of the base model isn't included in this checkpoint. |
|
|
| | Text-to-music search, Song Describer | EmbeddingGemma 2 | CLAP music | **SonicEmbed** | |
| | ------------------------------------ | ---------------- | ---------- | -------------- | |
| | R@10, three 10 s clips per track | 14.3 | 33.5 | **40.4** | |
| | R@10, one 10 s clip | 11.4 | 27.5 | **36.7** | |
|
|
| ## Usage |
|
|
| ```bash |
| pip install sentence-transformers librosa pillow torchvision |
| ``` |
|
|
| Pillow and torchvision are required even though images aren't used: the base EmbeddingGemma 2 processor builds its image and video parts when it loads. |
|
|
| ```python |
| import librosa |
| import numpy as np |
| from sentence_transformers import SentenceTransformer |
| |
| model = SentenceTransformer("jaisawhney/SonicEmbed") |
| |
| |
| def embed_track(path): |
| """The mean of three 10 s clips, at 25 %, 50 % and 75 % of the track.""" |
| wave, sr = librosa.load(path, sr=16000, mono=True) |
| size = 10 * sr |
| starts = [min(max(int(p * len(wave)) - size // 2, 0), max(len(wave) - size, 0)) for p in (0.25, 0.5, 0.75)] |
| clips = [{"array": wave[s:s + size], "sampling_rate": sr} for s in starts] |
| mean = model.encode(clips, normalize_embeddings=True).mean(0) |
| return mean / np.linalg.norm(mean) |
| |
| |
| track = embed_track("song.flac") |
| query = model.encode("calm piano", prompt_name="query", normalize_embeddings=True) |
| print(query @ track) |
| ``` |
|
|
| - **Prompts:** use `prompt_name="query"` for text you search with, and `prompt_name="document"` for text you search through. Audio takes no prompt. |
| - **Links count as audio:** a text input that is only a link to an `.mp3`, `.wav`, `.ogg`, `.flac` or `.aac` file is fetched as audio. |
| - **In-memory audio must already be 16 kHz:** `sampling_rate` is ignored, so resample first. |
| - **Only the first 30 s of audio is used:** anything longer is truncated. To cover a whole track, average clips as above. |
|
|
| ## Results |
|
|
| Training data was checked and filtered for Song Describer tracks. SonicEmbed, EmbeddingGemma 2, and CLAP were scored with the same code in the same run. CLAP is [LAION-CLAP](https://github.com/LAION-AI/CLAP)'s `music_audioset_epoch_15_esc_90.14.pt`. |
|
|
| ### Song Describer |
|
|
| | Clips per track | Model | R@1 | R@5 | R@10 | Median rank | |
| | --------------- | ---------------- | ------- | -------- | -------- | ----------- | |
| | Three 10 s | **SonicEmbed** | **9.5** | **29.4** | **40.4** | **16** | |
| | | CLAP music | 5.0 | 21.8 | 33.5 | 24 | |
| | | EmbeddingGemma 2 | 2.7 | 8.9 | 14.3 | 81 | |
| | One 10 s | **SonicEmbed** | **8.4** | **23.8** | **36.7** | **20.5** | |
| | | CLAP music | 4.9 | 16.8 | 27.5 | 29 | |
| | | EmbeddingGemma 2 | 1.4 | 6.1 | 11.4 | 89 | |
|
|
| [Song Describer](https://github.com/mulab-mir/song-describer-dataset) has 1,106 human-written captions for 706 tracks. Each caption is used as a query to retrieve its track. With 1,106 captions, R@10 has a margin of error of about ±3 points. Scores can differ by a few tenths to about 1/2 a point between machines. |
|
|
| ### General text |
|
|
| | [MTEB(eng, v2)](https://github.com/embeddings-benchmark/mteb), Mean(Task) | | |
| | ------------------------------------------------------------------------------------------------- | --------- | |
| | EmbeddingGemma 2, as reported on its [model card](https://huggingface.co/google/embeddinggemma-2) | 68.46 | |
| | SonicEmbed, measured with `eval/mteb_tasks.py` | **64.66** | |
|
|
| The two figures come from different runs, whose setups may differ. Expect some loss on general text. |
|
|
| ## Limitations |
|
|
| SonicEmbed was trained on English-only captions, and other languages are untested. |
|
|
| ## Training |
|
|
| Fine-tuned with Sentence Transformers on 1,234,845 caption and clip pairs from [IAMD](https://huggingface.co/datasets/Telecom-Paris/iamd_v0), minus its non-commercial recordings, with [MusicCaps](https://huggingface.co/datasets/google/MusicCaps) making up 10 % of the pairs. The result was blended with the original weights to recover some general text quality, with the ratio chosen on a held-out split of the training data. |
|
|
| IAMD and MusicCaps were checked for Song Describer tracks with audio fingerprints. Any items found were excluded from training ([list](eval/song_describer_overlap.csv)). |
|
|
| ## Reproducing the numbers |
|
|
| `eval/` has the Song Describer, CLAP, and MTEB scripts, with instructions. |
|
|
| ## License and credits |
|
|
| The weights are released under Apache-2.0. Recordings used in training that require attribution are credited in [CREDITS.csv](CREDITS.csv), where `identifier` is the Internet Archive item ID. |
|
|
