File size: 13,217 Bytes
8732fc1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 | ---
license: apache-2.0
library_name: onnx
pipeline_tag: feature-extraction
base_model: Qwen/Qwen3-Embedding-0.6B
datasets:
- Divi19/vibebench
tags:
- music
- audio
- text-to-audio-retrieval
- embeddings
- onnx
- raspberry-pi
---
# vibe-embed v1.2
Search music by how it sounds and feels: "rainy Sunday coffee shop", "upbeat but melancholic, no vocals", "something for a long night drive". Songs and descriptions share one 512-number vector space, so you can search a library by text or find songs with the same vibe as another song.
- **Small:** the audio model is EfficientAT mn20_as (35 MB in 8-bit) and runs on a CPU; the search side is Qwen3-Embedding-0.6B plus a small trained layer (818 MB in 8-bit).
- **Open:** Apache-2.0, trained only on CC0, CC BY and public-domain music. Every training song is credited in `CREDITS.csv`.
- **Languages:** English, Spanish, German, French, Brazilian Portuguese and Japanese (translated training texts; scores per language are in `results/languages.md`).
Try it: [demo](https://huggingface.co/spaces/Divi19/vibe-search) · benchmark: [Divi19/vibebench](https://huggingface.co/datasets/Divi19/vibebench)
## Use it
**In AudioMuse-AI** (self-hosted music server analysis): Plugins → add `https://huggingface.co/Divi19/vibe-embed/resolve/main/plugin/manifest.json` → install Vibe Search. The plugin indexes your library on your own machine; details in `plugin/README.md`.
**In Python** (onnxruntime ≥ 1.28 for the 8-bit search file; numpy; tokenizers):
```python
from huggingface_hub import snapshot_download
import sys
d = snapshot_download("Divi19/vibe-embed")
sys.path.insert(0, d + "/plugin/plugins/VibeSearch")
import vibemodel # the plugin's loader: preprocessing plus self-checks against parity.json
m = vibemodel.VibeModel(d, "mn20_as", "int8")
m.load_audio(); m.load_text()
song_vec, n = m.embed_audio(wave_32k_mono) # float32 numpy array, 32 kHz mono
query_vecs = m.embed_texts(["rainy Sunday coffee shop"])
score = query_vecs @ song_vec # cosine similarity
```
The exact preprocessing is written in `config.json`: 32 kHz mono, 10-second windows (a last window of at least 2 s is kept), window vectors averaged and normalised; search texts get the instruction prompt in `config.json`, left padding, at most 256 tokens.
| file | what |
|---|---|
| `mn20_as/audio_encoder_int8.onnx`, `audio_encoder.onnx` | audio model, 8-bit and full precision (35 / 72 MB) |
| `text/text_encoder_int8.onnx`, `text_encoder.onnx` (+ `.data`) | search model, 8-bit and full precision (818 / 2396 MB) |
| `text/tokenizer/` | tokenizer |
| `config.json`, `parity.json` | preprocessing, and reference outputs that loaders check against |
| `plugin/` | the AudioMuse-AI plugin catalogue |
| `results/` | full result tables |
## Results
**Measured by us**, same code for every model. IncompeBench: 500 written searches over 1,574 permissively licensed songs, nDCG@10 × 100. Song Describer: one caption per recording, recall@10.
| model | size | licence | IncompeBench lenient | IncompeBench strict | Song Describer R@10 |
|:---------------------------------------------------------------|:----------------------------|:-----------|-----------------------:|:----------------------|----------------------:|
| **vibe-embed v1.2, shipped 8-bit files** (this repo) | 35 MB audio + 818 MB search | Apache-2.0 | 71.8 | | 47.1 |
| vibe-embed v1.2, full precision | 72 MB + 2396 MB | Apache-2.0 | 72 | | 47.1 |
| its teacher (LCO-Omni-7B + LoRA with e5-omni-7B, not released) | 2 × 7B | | 79.2 | | 58.8 |
| DCLAP (AudioMuse-AI's current model) | | AGPL-3.0 | 56.2 | 46.5 | 30.8 |
| LAION-CLAP music + speech | | Apache-2.0 | 53.3 | 44.0 | 31.7 |
| PE-AV large | | Apache-2.0 | 55.9 | 47.8 | 48.6 |
| PE-AV base | | Apache-2.0 | 50.7 | 42.6 | 46.4 |
| PE-AV small | | Apache-2.0 | 52.8 | 44.6 | 47.8 |
| LCO-Embedding-Omni-7B | | Apache-2.0 | 76.9 | 68.2 | 41.9 |
| e5-omni-7B | | MIT | 74.4 | 65.3 | 53 |
**Published** on [Mixedbread's IncompeBench page](https://www.mixedbread.com/evals/incompebench) (tested March 2026; not re-measured by us):
| model | IncompeBench lenient | IncompeBench strict |
|:------------------------------------|-----------------------:|----------------------:|
| Mixedbread (Wholembed V3, API only) | 77.3 | 65.1 |
| Gemini Embedding 2 (API only) | 72.5 | 64 |
| CLaMP 3 | 70.4 | 57.9 |
| ColQwen-Omni | 65.4 | 51.4 |
| TTMR++ | 64.5 | 50.2 |
| CLAP | 59.4 | 46 |
On IncompeBench (lenient) the shipped 8-bit files score 71.8: above CLaMP 3, ColQwen-Omni, TTMR++, CLAP; below Mixedbread (Wholembed V3, API only), Gemini Embedding 2 (API only) as published. The test half of IncompeBench, which was never used to choose between versions, gives 72.8 with the shipped 8-bit files.
### VibeBench
VibeBench ranks 1,831 held-out CC songs for 500 real music requests from CPCD (Google Research, CC BY 4.0). Every system's top 10 is graded by an LLM (Phi-4) against a written description of each song. Main board: grades rescaled with cut-points fitted on IncompeBench's human-validated grades.
Rescaled grades (prompt-1 cut-points)
| system | nDCG@10 all | nDCG@10 test | nDCG@10 dev | judged@10 |
|:-------------------------------------------|--------------:|---------------:|--------------:|------------:|
| teacher, its own 7B search side | 0.8031 | 0.7922 | 0.8382 | 1 |
| combined teacher, its own 7B search side | 0.7939 | 0.7866 | 0.8176 | 1 |
| combined teacher, the v1.2 search side | 0.7644 | 0.7537 | 0.7991 | 1 |
| teacher, the student's search side (Qwen3) | 0.7536 | 0.7397 | 0.7983 | 1 |
| LCO-Omni-7B as is | 0.7494 | 0.7415 | 0.775 | 1 |
| v1.2 student + v1.2 search side (ships) | 0.7329 | 0.7191 | 0.7775 | 1 |
| e5-omni-7B as is | 0.7309 | 0.7214 | 0.7615 | 1 |
| v1 student + v1.1 search side | 0.718 | 0.708 | 0.7507 | 1 |
| v1 student + v1 search side (07) | 0.6861 | 0.6769 | 0.7159 | 1 |
| DCLAP (AudioMuse-AI) | 0.6371 | 0.6312 | 0.6561 | 1 |
| LAION CLAP music+speech | 0.574 | 0.5683 | 0.5923 | 1 |
| BM25 over the song cards | 0.5307 | 0.5245 | 0.5508 | 1 |
As answered
| system | nDCG@10 all | nDCG@10 test | nDCG@10 dev | judged@10 |
|:-------------------------------------------|--------------:|---------------:|--------------:|------------:|
| teacher, its own 7B search side | 0.8283 | 0.8238 | 0.8427 | 1 |
| combined teacher, its own 7B search side | 0.8132 | 0.8115 | 0.8186 | 1 |
| combined teacher, the v1.2 search side | 0.7847 | 0.782 | 0.7936 | 1 |
| teacher, the student's search side (Qwen3) | 0.7773 | 0.771 | 0.7977 | 1 |
| LCO-Omni-7B as is | 0.7635 | 0.7631 | 0.7649 | 1 |
| v1.2 student + v1.2 search side (ships) | 0.7509 | 0.7449 | 0.7705 | 1 |
| v1 student + v1.1 search side | 0.744 | 0.7391 | 0.76 | 1 |
| e5-omni-7B as is | 0.7421 | 0.7383 | 0.7545 | 1 |
| v1 student + v1 search side (07) | 0.7039 | 0.6994 | 0.7184 | 1 |
| DCLAP (AudioMuse-AI) | 0.6513 | 0.6472 | 0.6646 | 1 |
| LAION CLAP music+speech | 0.5857 | 0.5881 | 0.5779 | 1 |
| BM25 over the song cards | 0.5496 | 0.5486 | 0.5529 | 1 |
| v1.2 student + v1.2 search side minus | rescaled, all [95% CI] | rescaled, test [95% CI] | as answered, all [95% CI] |
|:-----------------------------------------|:-------------------------|:--------------------------|:----------------------------|
| DCLAP (AudioMuse-AI) | +0.096 [+0.078, +0.115] | +0.088 [+0.067, +0.109] | +0.100 [+0.083, +0.117] |
| LAION CLAP music+speech | +0.159 [+0.139, +0.179] | +0.151 [+0.128, +0.175] | +0.165 [+0.147, +0.185] |
| v1 student + v1.1 search side | +0.015 [+0.004, +0.026] | +0.011 [-0.002, +0.025] | +0.007 [-0.002, +0.016] |
| v1 student + v1 search side (07) | +0.047 [+0.033, +0.060] | +0.042 [+0.028, +0.057] | +0.047 [+0.036, +0.059] |
| LCO-Omni-7B as is | -0.017 [-0.031, -0.002] | -0.022 [-0.040, -0.005] | -0.013 [-0.026, +0.001] |
| combined teacher, its own 7B search side | -0.061 [-0.074, -0.048] | -0.068 [-0.084, -0.052] | -0.062 [-0.074, -0.051] |
### Speed
One 180-second song indexed on a Colab CPU (8 CPUs), decoding included, median of 3 runs. DCLAP is run exactly as AudioMuse-AI runs it.
| threads | ours int8 (s/song) | ours fp32 (s/song) | DCLAP (s/song) | DCLAP / ours int8 |
|----------:|---------------------:|---------------------:|-----------------:|--------------------:|
| 4 | 0.902 | 0.903 | 4.243 | 4.7 |
| 1 | 1.897 | 1.909 | 2.677 | 1.41 |
## How it was made
1. **Captions.** An audio-language model wrote listener-style descriptions of CC0 / CC BY Jamendo and FMA songs.
2. **Teacher.** LCO-Embedding-Omni-7B, fine-tuned with LoRA on those pairs, combined with e5-omni-7B (both built on Qwen2.5-Omni-7B, Apache-2.0) and reduced to 512 numbers. The teacher is not released.
3. **Student.** EfficientAT mn20_as (MIT; initialised from its authors' AudioSet weights) learns to reproduce the teacher's song vectors on 35,661 CC songs. The search side, Qwen3-Embedding-0.6B (Apache-2.0) with a trained layer, learns to reproduce the teacher's text vectors.
4. **Selection.** Every choice between versions was made on the dev halves of IncompeBench and Song Describer; the test halves were only reported.
Never used for training: IncompeBench, Song Describer, MTG-Jamendo tags, or outputs of closed models.
## Limits
- Trained and tested on royalty-free music. Mainstream commercial music was not part of training or testing, so how well it handles it is unmeasured.
- VibeBench is graded by an LLM. Its agreement with human-validated grades is κ ≈ 0.47, below the usual 0.6 bar, so treat VibeBench as a secondary signal.
- Speed was measured on a cloud CPU, not on a Raspberry Pi.
- On onnxruntime older than 1.28 (for example AudioMuse-AI's no-AVX2 image), the 8-bit search file does not load; the plugin then uses the full-precision file, which is larger and needs more RAM. Results are the same.
## Licence and credits
Model files: Apache-2.0 (`LICENSE`). The AudioMuse-AI plugin in `plugin/` is AGPL-3.0, like AudioMuse-AI itself. Training music: CC0, CC BY and public-domain songs from Jamendo and the Free Music Archive; every song with its artist, licence and link is in `CREDITS.csv` (CC BY: 35,661).
|