|
Download README.md from Divi19/vibe-embed: direct link, hf CLI and curl.
- Browser
- Download file 13.2 kB
-
https://huggingface.co/Divi19/vibe-embed/resolve/main/README.md
- Command line
-
hf download hf://Divi19/vibe-embed/README.md
-
curl -L -o README.md https://huggingface.co/Divi19/vibe-embed/resolve/main/README.md
13.2 kB
| license: apache-2.0 | |
| library_name: onnx | |
| pipeline_tag: feature-extraction | |
| base_model: Qwen/Qwen3-Embedding-0.6B | |
| datasets: | |
| - Divi19/vibebench | |
| tags: | |
| - music | |
| - audio | |
| - text-to-audio-retrieval | |
| - embeddings | |
| - onnx | |
| - raspberry-pi | |
| # vibe-embed v1.2 | |
| Search music by how it sounds and feels: "rainy Sunday coffee shop", "upbeat but melancholic, no vocals", "something for a long night drive". Songs and descriptions share one 512-number vector space, so you can search a library by text or find songs with the same vibe as another song. | |
| - **Small:** the audio model is EfficientAT mn20_as (35 MB in 8-bit) and runs on a CPU; the search side is Qwen3-Embedding-0.6B plus a small trained layer (818 MB in 8-bit). | |
| - **Open:** Apache-2.0, trained only on CC0, CC BY and public-domain music. Every training song is credited in `CREDITS.csv`. | |
| - **Languages:** English, Spanish, German, French, Brazilian Portuguese and Japanese (translated training texts; scores per language are in `results/languages.md`). | |
| Try it: [demo](https://huggingface.co/spaces/Divi19/vibe-search) · benchmark: [Divi19/vibebench](https://huggingface.co/datasets/Divi19/vibebench) | |
| ## Use it | |
| **In AudioMuse-AI** (self-hosted music server analysis): Plugins → add `https://huggingface.co/Divi19/vibe-embed/resolve/main/plugin/manifest.json` → install Vibe Search. The plugin indexes your library on your own machine; details in `plugin/README.md`. | |
| **In Python** (onnxruntime ≥ 1.28 for the 8-bit search file; numpy; tokenizers): | |
| ```python | |
| from huggingface_hub import snapshot_download | |
| import sys | |
| d = snapshot_download("Divi19/vibe-embed") | |
| sys.path.insert(0, d + "/plugin/plugins/VibeSearch") | |
| import vibemodel # the plugin's loader: preprocessing plus self-checks against parity.json | |
| m = vibemodel.VibeModel(d, "mn20_as", "int8") | |
| m.load_audio(); m.load_text() | |
| song_vec, n = m.embed_audio(wave_32k_mono) # float32 numpy array, 32 kHz mono | |
| query_vecs = m.embed_texts(["rainy Sunday coffee shop"]) | |
| score = query_vecs @ song_vec # cosine similarity | |
| ``` | |
| The exact preprocessing is written in `config.json`: 32 kHz mono, 10-second windows (a last window of at least 2 s is kept), window vectors averaged and normalised; search texts get the instruction prompt in `config.json`, left padding, at most 256 tokens. | |
| | file | what | | |
| |---|---| | |
| | `mn20_as/audio_encoder_int8.onnx`, `audio_encoder.onnx` | audio model, 8-bit and full precision (35 / 72 MB) | | |
| | `text/text_encoder_int8.onnx`, `text_encoder.onnx` (+ `.data`) | search model, 8-bit and full precision (818 / 2396 MB) | | |
| | `text/tokenizer/` | tokenizer | | |
| | `config.json`, `parity.json` | preprocessing, and reference outputs that loaders check against | | |
| | `plugin/` | the AudioMuse-AI plugin catalogue | | |
| | `results/` | full result tables | | |
| ## Results | |
| **Measured by us**, same code for every model. IncompeBench: 500 written searches over 1,574 permissively licensed songs, nDCG@10 × 100. Song Describer: one caption per recording, recall@10. | |
| | model | size | licence | IncompeBench lenient | IncompeBench strict | Song Describer R@10 | | |
| |:---------------------------------------------------------------|:----------------------------|:-----------|-----------------------:|:----------------------|----------------------:| | |
| | **vibe-embed v1.2, shipped 8-bit files** (this repo) | 35 MB audio + 818 MB search | Apache-2.0 | 71.8 | | 47.1 | | |
| | vibe-embed v1.2, full precision | 72 MB + 2396 MB | Apache-2.0 | 72 | | 47.1 | | |
| | its teacher (LCO-Omni-7B + LoRA with e5-omni-7B, not released) | 2 × 7B | | 79.2 | | 58.8 | | |
| | DCLAP (AudioMuse-AI's current model) | | AGPL-3.0 | 56.2 | 46.5 | 30.8 | | |
| | LAION-CLAP music + speech | | Apache-2.0 | 53.3 | 44.0 | 31.7 | | |
| | PE-AV large | | Apache-2.0 | 55.9 | 47.8 | 48.6 | | |
| | PE-AV base | | Apache-2.0 | 50.7 | 42.6 | 46.4 | | |
| | PE-AV small | | Apache-2.0 | 52.8 | 44.6 | 47.8 | | |
| | LCO-Embedding-Omni-7B | | Apache-2.0 | 76.9 | 68.2 | 41.9 | | |
| | e5-omni-7B | | MIT | 74.4 | 65.3 | 53 | | |
| **Published** on [Mixedbread's IncompeBench page](https://www.mixedbread.com/evals/incompebench) (tested March 2026; not re-measured by us): | |
| | model | IncompeBench lenient | IncompeBench strict | | |
| |:------------------------------------|-----------------------:|----------------------:| | |
| | Mixedbread (Wholembed V3, API only) | 77.3 | 65.1 | | |
| | Gemini Embedding 2 (API only) | 72.5 | 64 | | |
| | CLaMP 3 | 70.4 | 57.9 | | |
| | ColQwen-Omni | 65.4 | 51.4 | | |
| | TTMR++ | 64.5 | 50.2 | | |
| | CLAP | 59.4 | 46 | | |
| On IncompeBench (lenient) the shipped 8-bit files score 71.8: above CLaMP 3, ColQwen-Omni, TTMR++, CLAP; below Mixedbread (Wholembed V3, API only), Gemini Embedding 2 (API only) as published. The test half of IncompeBench, which was never used to choose between versions, gives 72.8 with the shipped 8-bit files. | |
| ### VibeBench | |
| VibeBench ranks 1,831 held-out CC songs for 500 real music requests from CPCD (Google Research, CC BY 4.0). Every system's top 10 is graded by an LLM (Phi-4) against a written description of each song. Main board: grades rescaled with cut-points fitted on IncompeBench's human-validated grades. | |
| Rescaled grades (prompt-1 cut-points) | |
| | system | nDCG@10 all | nDCG@10 test | nDCG@10 dev | judged@10 | | |
| |:-------------------------------------------|--------------:|---------------:|--------------:|------------:| | |
| | teacher, its own 7B search side | 0.8031 | 0.7922 | 0.8382 | 1 | | |
| | combined teacher, its own 7B search side | 0.7939 | 0.7866 | 0.8176 | 1 | | |
| | combined teacher, the v1.2 search side | 0.7644 | 0.7537 | 0.7991 | 1 | | |
| | teacher, the student's search side (Qwen3) | 0.7536 | 0.7397 | 0.7983 | 1 | | |
| | LCO-Omni-7B as is | 0.7494 | 0.7415 | 0.775 | 1 | | |
| | v1.2 student + v1.2 search side (ships) | 0.7329 | 0.7191 | 0.7775 | 1 | | |
| | e5-omni-7B as is | 0.7309 | 0.7214 | 0.7615 | 1 | | |
| | v1 student + v1.1 search side | 0.718 | 0.708 | 0.7507 | 1 | | |
| | v1 student + v1 search side (07) | 0.6861 | 0.6769 | 0.7159 | 1 | | |
| | DCLAP (AudioMuse-AI) | 0.6371 | 0.6312 | 0.6561 | 1 | | |
| | LAION CLAP music+speech | 0.574 | 0.5683 | 0.5923 | 1 | | |
| | BM25 over the song cards | 0.5307 | 0.5245 | 0.5508 | 1 | | |
| As answered | |
| | system | nDCG@10 all | nDCG@10 test | nDCG@10 dev | judged@10 | | |
| |:-------------------------------------------|--------------:|---------------:|--------------:|------------:| | |
| | teacher, its own 7B search side | 0.8283 | 0.8238 | 0.8427 | 1 | | |
| | combined teacher, its own 7B search side | 0.8132 | 0.8115 | 0.8186 | 1 | | |
| | combined teacher, the v1.2 search side | 0.7847 | 0.782 | 0.7936 | 1 | | |
| | teacher, the student's search side (Qwen3) | 0.7773 | 0.771 | 0.7977 | 1 | | |
| | LCO-Omni-7B as is | 0.7635 | 0.7631 | 0.7649 | 1 | | |
| | v1.2 student + v1.2 search side (ships) | 0.7509 | 0.7449 | 0.7705 | 1 | | |
| | v1 student + v1.1 search side | 0.744 | 0.7391 | 0.76 | 1 | | |
| | e5-omni-7B as is | 0.7421 | 0.7383 | 0.7545 | 1 | | |
| | v1 student + v1 search side (07) | 0.7039 | 0.6994 | 0.7184 | 1 | | |
| | DCLAP (AudioMuse-AI) | 0.6513 | 0.6472 | 0.6646 | 1 | | |
| | LAION CLAP music+speech | 0.5857 | 0.5881 | 0.5779 | 1 | | |
| | BM25 over the song cards | 0.5496 | 0.5486 | 0.5529 | 1 | | |
| | v1.2 student + v1.2 search side minus | rescaled, all [95% CI] | rescaled, test [95% CI] | as answered, all [95% CI] | | |
| |:-----------------------------------------|:-------------------------|:--------------------------|:----------------------------| | |
| | DCLAP (AudioMuse-AI) | +0.096 [+0.078, +0.115] | +0.088 [+0.067, +0.109] | +0.100 [+0.083, +0.117] | | |
| | LAION CLAP music+speech | +0.159 [+0.139, +0.179] | +0.151 [+0.128, +0.175] | +0.165 [+0.147, +0.185] | | |
| | v1 student + v1.1 search side | +0.015 [+0.004, +0.026] | +0.011 [-0.002, +0.025] | +0.007 [-0.002, +0.016] | | |
| | v1 student + v1 search side (07) | +0.047 [+0.033, +0.060] | +0.042 [+0.028, +0.057] | +0.047 [+0.036, +0.059] | | |
| | LCO-Omni-7B as is | -0.017 [-0.031, -0.002] | -0.022 [-0.040, -0.005] | -0.013 [-0.026, +0.001] | | |
| | combined teacher, its own 7B search side | -0.061 [-0.074, -0.048] | -0.068 [-0.084, -0.052] | -0.062 [-0.074, -0.051] | | |
| ### Speed | |
| One 180-second song indexed on a Colab CPU (8 CPUs), decoding included, median of 3 runs. DCLAP is run exactly as AudioMuse-AI runs it. | |
| | threads | ours int8 (s/song) | ours fp32 (s/song) | DCLAP (s/song) | DCLAP / ours int8 | | |
| |----------:|---------------------:|---------------------:|-----------------:|--------------------:| | |
| | 4 | 0.902 | 0.903 | 4.243 | 4.7 | | |
| | 1 | 1.897 | 1.909 | 2.677 | 1.41 | | |
| ## How it was made | |
| 1. **Captions.** An audio-language model wrote listener-style descriptions of CC0 / CC BY Jamendo and FMA songs. | |
| 2. **Teacher.** LCO-Embedding-Omni-7B, fine-tuned with LoRA on those pairs, combined with e5-omni-7B (both built on Qwen2.5-Omni-7B, Apache-2.0) and reduced to 512 numbers. The teacher is not released. | |
| 3. **Student.** EfficientAT mn20_as (MIT; initialised from its authors' AudioSet weights) learns to reproduce the teacher's song vectors on 35,661 CC songs. The search side, Qwen3-Embedding-0.6B (Apache-2.0) with a trained layer, learns to reproduce the teacher's text vectors. | |
| 4. **Selection.** Every choice between versions was made on the dev halves of IncompeBench and Song Describer; the test halves were only reported. | |
| Never used for training: IncompeBench, Song Describer, MTG-Jamendo tags, or outputs of closed models. | |
| ## Limits | |
| - Trained and tested on royalty-free music. Mainstream commercial music was not part of training or testing, so how well it handles it is unmeasured. | |
| - VibeBench is graded by an LLM. Its agreement with human-validated grades is κ ≈ 0.47, below the usual 0.6 bar, so treat VibeBench as a secondary signal. | |
| - Speed was measured on a cloud CPU, not on a Raspberry Pi. | |
| - On onnxruntime older than 1.28 (for example AudioMuse-AI's no-AVX2 image), the 8-bit search file does not load; the plugin then uses the full-precision file, which is larger and needs more RAM. Results are the same. | |
| ## Licence and credits | |
| Model files: Apache-2.0 (`LICENSE`). The AudioMuse-AI plugin in `plugin/` is AGPL-3.0, like AudioMuse-AI itself. Training music: CC0, CC BY and public-domain songs from Jamendo and the Free Music Archive; every song with its artist, licence and link is in `CREDITS.csv` (CC BY: 35,661). | |