---
license: apache-2.0
language:
  - en
library_name: sentence-transformers
pipeline_tag: sentence-similarity
base_model: google/embeddinggemma-2
tags:
  - sentence-transformers
  - audio
  - music
  - music-information-retrieval
  - text-to-audio-retrieval
  - embeddings
datasets:
  - Telecom-Paris/iamd_v0
  - google/MusicCaps
---

# SonicEmbed

SonicEmbed is an [EmbeddingGemma 2](https://huggingface.co/google/embeddinggemma-2) fine-tune that puts text and music in one 768-dimensional space from a single model that still works for general text. On Song Describer, it beats LAION-CLAP's music checkpoint. The vision tower of the base model isn't included in this checkpoint.

| Text-to-music search, Song Describer | EmbeddingGemma 2 | CLAP music | **SonicEmbed** |
| ------------------------------------ | ---------------- | ---------- | -------------- |
| R@10, three 10 s clips per track     | 14.3             | 33.5       | **40.4**       |
| R@10, one 10 s clip                  | 11.4             | 27.5       | **36.7**       |

## Usage

```bash
pip install sentence-transformers librosa pillow torchvision
```

Pillow and torchvision are required even though images aren't used: the base EmbeddingGemma 2 processor builds its image and video parts when it loads.

```python
import librosa
import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("jaisawhney/SonicEmbed")


def embed_track(path):
    """The mean of three 10 s clips, at 25 %, 50 % and 75 % of the track."""
    wave, sr = librosa.load(path, sr=16000, mono=True)
    size = 10 * sr
    starts = [min(max(int(p * len(wave)) - size // 2, 0), max(len(wave) - size, 0)) for p in (0.25, 0.5, 0.75)]
    clips = [{"array": wave[s:s + size], "sampling_rate": sr} for s in starts]
    mean = model.encode(clips, normalize_embeddings=True).mean(0)
    return mean / np.linalg.norm(mean)


track = embed_track("song.flac")
query = model.encode("calm piano", prompt_name="query", normalize_embeddings=True)
print(query @ track)
```

- **Prompts:** use `prompt_name="query"` for text you search with, and `prompt_name="document"` for text you search through. Audio takes no prompt.
- **Links count as audio:** a text input that is only a link to an `.mp3`, `.wav`, `.ogg`, `.flac` or `.aac` file is fetched as audio.
- **In-memory audio must already be 16 kHz:** `sampling_rate` is ignored, so resample first.
- **Only the first 30 s of audio is used:** anything longer is truncated. To cover a whole track, average clips as above.

## Results

Training data was checked and filtered for Song Describer tracks. SonicEmbed, EmbeddingGemma 2, and CLAP were scored with the same code in the same run. CLAP is [LAION-CLAP](https://github.com/LAION-AI/CLAP)'s `music_audioset_epoch_15_esc_90.14.pt`.

### Song Describer

| Clips per track | Model            | R@1     | R@5      | R@10     | Median rank |
| --------------- | ---------------- | ------- | -------- | -------- | ----------- |
| Three 10 s      | **SonicEmbed**   | **9.5** | **29.4** | **40.4** | **16**      |
|                 | CLAP music       | 5.0     | 21.8     | 33.5     | 24          |
|                 | EmbeddingGemma 2 | 2.7     | 8.9      | 14.3     | 81          |
| One 10 s        | **SonicEmbed**   | **8.4** | **23.8** | **36.7** | **20.5**    |
|                 | CLAP music       | 4.9     | 16.8     | 27.5     | 29          |
|                 | EmbeddingGemma 2 | 1.4     | 6.1      | 11.4     | 89          |

[Song Describer](https://github.com/mulab-mir/song-describer-dataset) has 1,106 human-written captions for 706 tracks. Each caption is used as a query to retrieve its track. With 1,106 captions, R@10 has a margin of error of about ±3 points. Scores can differ by a few tenths to about 1/2 a point between machines.

### General text

| [MTEB(eng, v2)](https://github.com/embeddings-benchmark/mteb), Mean(Task)                         |           |
| ------------------------------------------------------------------------------------------------- | --------- |
| EmbeddingGemma 2, as reported on its [model card](https://huggingface.co/google/embeddinggemma-2) | 68.46     |
| SonicEmbed, measured with `eval/mteb_tasks.py`                                                    | **64.66** |

The two figures come from different runs, whose setups may differ. Expect some loss on general text.

## Limitations

SonicEmbed was trained on English-only captions, and other languages are untested.

## Training

Fine-tuned with Sentence Transformers on 1,234,845 caption and clip pairs from [IAMD](https://huggingface.co/datasets/Telecom-Paris/iamd_v0), minus its non-commercial recordings, with [MusicCaps](https://huggingface.co/datasets/google/MusicCaps) making up 10 % of the pairs. The result was blended with the original weights to recover some general text quality, with the ratio chosen on a held-out split of the training data.

IAMD and MusicCaps were checked for Song Describer tracks with audio fingerprints. Any items found were excluded from training ([list](eval/song_describer_overlap.csv)).

## Reproducing the numbers

`eval/` has the Song Describer, CLAP, and MTEB scripts, with instructions.

## License and credits

The weights are released under Apache-2.0. Recordings used in training that require attribution are credited in [CREDITS.csv](CREDITS.csv), where `identifier` is the Internet Archive item ID.