File size: 13,217 Bytes
8732fc1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
---
license: apache-2.0
library_name: onnx
pipeline_tag: feature-extraction
base_model: Qwen/Qwen3-Embedding-0.6B
datasets:
- Divi19/vibebench
tags:
- music
- audio
- text-to-audio-retrieval
- embeddings
- onnx
- raspberry-pi
---

# vibe-embed v1.2

Search music by how it sounds and feels: "rainy Sunday coffee shop", "upbeat but melancholic, no vocals", "something for a long night drive". Songs and descriptions share one 512-number vector space, so you can search a library by text or find songs with the same vibe as another song.

- **Small:** the audio model is EfficientAT mn20_as (35 MB in 8-bit) and runs on a CPU; the search side is Qwen3-Embedding-0.6B plus a small trained layer (818 MB in 8-bit).
- **Open:** Apache-2.0, trained only on CC0, CC BY and public-domain music. Every training song is credited in `CREDITS.csv`.
- **Languages:** English, Spanish, German, French, Brazilian Portuguese and Japanese (translated training texts; scores per language are in `results/languages.md`).

Try it: [demo](https://huggingface.co/spaces/Divi19/vibe-search) · benchmark: [Divi19/vibebench](https://huggingface.co/datasets/Divi19/vibebench)

## Use it

**In AudioMuse-AI** (self-hosted music server analysis): Plugins → add `https://huggingface.co/Divi19/vibe-embed/resolve/main/plugin/manifest.json` → install Vibe Search. The plugin indexes your library on your own machine; details in `plugin/README.md`.

**In Python** (onnxruntime ≥ 1.28 for the 8-bit search file; numpy; tokenizers):

```python
from huggingface_hub import snapshot_download
import sys
d = snapshot_download("Divi19/vibe-embed")
sys.path.insert(0, d + "/plugin/plugins/VibeSearch")
import vibemodel                                   # the plugin's loader: preprocessing plus self-checks against parity.json
m = vibemodel.VibeModel(d, "mn20_as", "int8")
m.load_audio(); m.load_text()
song_vec, n = m.embed_audio(wave_32k_mono)          # float32 numpy array, 32 kHz mono
query_vecs = m.embed_texts(["rainy Sunday coffee shop"])
score = query_vecs @ song_vec                       # cosine similarity
```

The exact preprocessing is written in `config.json`: 32 kHz mono, 10-second windows (a last window of at least 2 s is kept), window vectors averaged and normalised; search texts get the instruction prompt in `config.json`, left padding, at most 256 tokens.

| file | what |
|---|---|
| `mn20_as/audio_encoder_int8.onnx`, `audio_encoder.onnx` | audio model, 8-bit and full precision (35 / 72 MB) |
| `text/text_encoder_int8.onnx`, `text_encoder.onnx` (+ `.data`) | search model, 8-bit and full precision (818 / 2396 MB) |
| `text/tokenizer/` | tokenizer |
| `config.json`, `parity.json` | preprocessing, and reference outputs that loaders check against |
| `plugin/` | the AudioMuse-AI plugin catalogue |
| `results/` | full result tables |

## Results

**Measured by us**, same code for every model. IncompeBench: 500 written searches over 1,574 permissively licensed songs, nDCG@10 × 100. Song Describer: one caption per recording, recall@10.

| model                                                          | size                        | licence    |   IncompeBench lenient | IncompeBench strict   |   Song Describer R@10 |
|:---------------------------------------------------------------|:----------------------------|:-----------|-----------------------:|:----------------------|----------------------:|
| **vibe-embed v1.2, shipped 8-bit files** (this repo)           | 35 MB audio + 818 MB search | Apache-2.0 |                   71.8 |                       |                  47.1 |
| vibe-embed v1.2, full precision                                | 72 MB + 2396 MB             | Apache-2.0 |                   72   |                       |                  47.1 |
| its teacher (LCO-Omni-7B + LoRA with e5-omni-7B, not released) | 2 × 7B                      |            |                   79.2 |                       |                  58.8 |
| DCLAP (AudioMuse-AI's current model)                           |                             | AGPL-3.0   |                   56.2 | 46.5                  |                  30.8 |
| LAION-CLAP music + speech                                      |                             | Apache-2.0 |                   53.3 | 44.0                  |                  31.7 |
| PE-AV large                                                    |                             | Apache-2.0 |                   55.9 | 47.8                  |                  48.6 |
| PE-AV base                                                     |                             | Apache-2.0 |                   50.7 | 42.6                  |                  46.4 |
| PE-AV small                                                    |                             | Apache-2.0 |                   52.8 | 44.6                  |                  47.8 |
| LCO-Embedding-Omni-7B                                          |                             | Apache-2.0 |                   76.9 | 68.2                  |                  41.9 |
| e5-omni-7B                                                     |                             | MIT        |                   74.4 | 65.3                  |                  53   |

**Published** on [Mixedbread's IncompeBench page](https://www.mixedbread.com/evals/incompebench) (tested March 2026; not re-measured by us):

| model                               |   IncompeBench lenient |   IncompeBench strict |
|:------------------------------------|-----------------------:|----------------------:|
| Mixedbread (Wholembed V3, API only) |                   77.3 |                  65.1 |
| Gemini Embedding 2 (API only)       |                   72.5 |                  64   |
| CLaMP 3                             |                   70.4 |                  57.9 |
| ColQwen-Omni                        |                   65.4 |                  51.4 |
| TTMR++                              |                   64.5 |                  50.2 |
| CLAP                                |                   59.4 |                  46   |

On IncompeBench (lenient) the shipped 8-bit files score 71.8: above CLaMP 3, ColQwen-Omni, TTMR++, CLAP; below Mixedbread (Wholembed V3, API only), Gemini Embedding 2 (API only) as published. The test half of IncompeBench, which was never used to choose between versions, gives 72.8 with the shipped 8-bit files.

### VibeBench

VibeBench ranks 1,831 held-out CC songs for 500 real music requests from CPCD (Google Research, CC BY 4.0). Every system's top 10 is graded by an LLM (Phi-4) against a written description of each song. Main board: grades rescaled with cut-points fitted on IncompeBench's human-validated grades.

Rescaled grades (prompt-1 cut-points)

| system                                     |   nDCG@10 all |   nDCG@10 test |   nDCG@10 dev |   judged@10 |
|:-------------------------------------------|--------------:|---------------:|--------------:|------------:|
| teacher, its own 7B search side            |        0.8031 |         0.7922 |        0.8382 |           1 |
| combined teacher, its own 7B search side   |        0.7939 |         0.7866 |        0.8176 |           1 |
| combined teacher, the v1.2 search side     |        0.7644 |         0.7537 |        0.7991 |           1 |
| teacher, the student's search side (Qwen3) |        0.7536 |         0.7397 |        0.7983 |           1 |
| LCO-Omni-7B as is                          |        0.7494 |         0.7415 |        0.775  |           1 |
| v1.2 student + v1.2 search side (ships)    |        0.7329 |         0.7191 |        0.7775 |           1 |
| e5-omni-7B as is                           |        0.7309 |         0.7214 |        0.7615 |           1 |
| v1 student + v1.1 search side              |        0.718  |         0.708  |        0.7507 |           1 |
| v1 student + v1 search side (07)           |        0.6861 |         0.6769 |        0.7159 |           1 |
| DCLAP (AudioMuse-AI)                       |        0.6371 |         0.6312 |        0.6561 |           1 |
| LAION CLAP music+speech                    |        0.574  |         0.5683 |        0.5923 |           1 |
| BM25 over the song cards                   |        0.5307 |         0.5245 |        0.5508 |           1 |

As answered

| system                                     |   nDCG@10 all |   nDCG@10 test |   nDCG@10 dev |   judged@10 |
|:-------------------------------------------|--------------:|---------------:|--------------:|------------:|
| teacher, its own 7B search side            |        0.8283 |         0.8238 |        0.8427 |           1 |
| combined teacher, its own 7B search side   |        0.8132 |         0.8115 |        0.8186 |           1 |
| combined teacher, the v1.2 search side     |        0.7847 |         0.782  |        0.7936 |           1 |
| teacher, the student's search side (Qwen3) |        0.7773 |         0.771  |        0.7977 |           1 |
| LCO-Omni-7B as is                          |        0.7635 |         0.7631 |        0.7649 |           1 |
| v1.2 student + v1.2 search side (ships)    |        0.7509 |         0.7449 |        0.7705 |           1 |
| v1 student + v1.1 search side              |        0.744  |         0.7391 |        0.76   |           1 |
| e5-omni-7B as is                           |        0.7421 |         0.7383 |        0.7545 |           1 |
| v1 student + v1 search side (07)           |        0.7039 |         0.6994 |        0.7184 |           1 |
| DCLAP (AudioMuse-AI)                       |        0.6513 |         0.6472 |        0.6646 |           1 |
| LAION CLAP music+speech                    |        0.5857 |         0.5881 |        0.5779 |           1 |
| BM25 over the song cards                   |        0.5496 |         0.5486 |        0.5529 |           1 |

| v1.2 student + v1.2 search side minus    | rescaled, all [95% CI]   | rescaled, test [95% CI]   | as answered, all [95% CI]   |
|:-----------------------------------------|:-------------------------|:--------------------------|:----------------------------|
| DCLAP (AudioMuse-AI)                     | +0.096 [+0.078, +0.115]  | +0.088 [+0.067, +0.109]   | +0.100 [+0.083, +0.117]     |
| LAION CLAP music+speech                  | +0.159 [+0.139, +0.179]  | +0.151 [+0.128, +0.175]   | +0.165 [+0.147, +0.185]     |
| v1 student + v1.1 search side            | +0.015 [+0.004, +0.026]  | +0.011 [-0.002, +0.025]   | +0.007 [-0.002, +0.016]     |
| v1 student + v1 search side (07)         | +0.047 [+0.033, +0.060]  | +0.042 [+0.028, +0.057]   | +0.047 [+0.036, +0.059]     |
| LCO-Omni-7B as is                        | -0.017 [-0.031, -0.002]  | -0.022 [-0.040, -0.005]   | -0.013 [-0.026, +0.001]     |
| combined teacher, its own 7B search side | -0.061 [-0.074, -0.048]  | -0.068 [-0.084, -0.052]   | -0.062 [-0.074, -0.051]     |

### Speed

One 180-second song indexed on a Colab CPU (8 CPUs), decoding included, median of 3 runs. DCLAP is run exactly as AudioMuse-AI runs it.

|   threads |   ours int8 (s/song) |   ours fp32 (s/song) |   DCLAP (s/song) |   DCLAP / ours int8 |
|----------:|---------------------:|---------------------:|-----------------:|--------------------:|
|         4 |                0.902 |                0.903 |            4.243 |                4.7  |
|         1 |                1.897 |                1.909 |            2.677 |                1.41 |

## How it was made

1. **Captions.** An audio-language model wrote listener-style descriptions of CC0 / CC BY Jamendo and FMA songs.
2. **Teacher.** LCO-Embedding-Omni-7B, fine-tuned with LoRA on those pairs, combined with e5-omni-7B (both built on Qwen2.5-Omni-7B, Apache-2.0) and reduced to 512 numbers. The teacher is not released.
3. **Student.** EfficientAT mn20_as (MIT; initialised from its authors' AudioSet weights) learns to reproduce the teacher's song vectors on 35,661 CC songs. The search side, Qwen3-Embedding-0.6B (Apache-2.0) with a trained layer, learns to reproduce the teacher's text vectors.
4. **Selection.** Every choice between versions was made on the dev halves of IncompeBench and Song Describer; the test halves were only reported.

Never used for training: IncompeBench, Song Describer, MTG-Jamendo tags, or outputs of closed models.

## Limits

- Trained and tested on royalty-free music. Mainstream commercial music was not part of training or testing, so how well it handles it is unmeasured.
- VibeBench is graded by an LLM. Its agreement with human-validated grades is κ ≈ 0.47, below the usual 0.6 bar, so treat VibeBench as a secondary signal.
- Speed was measured on a cloud CPU, not on a Raspberry Pi.
- On onnxruntime older than 1.28 (for example AudioMuse-AI's no-AVX2 image), the 8-bit search file does not load; the plugin then uses the full-precision file, which is larger and needs more RAM. Results are the same.

## Licence and credits

Model files: Apache-2.0 (`LICENSE`). The AudioMuse-AI plugin in `plugin/` is AGPL-3.0, like AudioMuse-AI itself. Training music: CC0, CC BY and public-domain songs from Jamendo and the Free Music Archive; every song with its artist, licence and link is in `CREDITS.csv` (CC BY: 35,661).