Sentence Similarity
Transformers
ONNX
Safetensors
sentence-transformers
English
multimodal
embeddings
image-text
audio-text
retrieval
2DMSE
matryoshka
Eval Results (legacy)
Instructions to use vllm-sr/multi-modal-embed-small with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use vllm-sr/multi-modal-embed-small with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("vllm-sr/multi-modal-embed-small", device_map="auto") - sentence-transformers
How to use vllm-sr/multi-modal-embed-small with sentence-transformers:
from sentence_transformers import SentenceTransformer model = SentenceTransformer("vllm-sr/multi-modal-embed-small") sentences = [ "That is a happy person", "That is a happy dog", "That is a very happy person", "Today is a sunny day" ] embeddings = model.encode(sentences) similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [4, 4] - Inference
- Notebooks
- Google Colab
- Kaggle
Add audio training roadmap to model card
Browse files
README.md
CHANGED
|
@@ -289,9 +289,26 @@ fused_embedding = model.encode_multimodal(
|
|
| 289 |
|
| 290 |
- Optimized for English; multilingual performance may vary
|
| 291 |
- Image resolution fixed at 512x512
|
| 292 |
-
- Audio
|
| 293 |
- Best for semantic similarity, not generative tasks
|
| 294 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 295 |
## Citation
|
| 296 |
|
| 297 |
```bibtex
|
|
|
|
| 289 |
|
| 290 |
- Optimized for English; multilingual performance may vary
|
| 291 |
- Image resolution fixed at 512x512
|
| 292 |
+
- Audio encoder included but not yet trained (see Roadmap)
|
| 293 |
- Best for semantic similarity, not generative tasks
|
| 294 |
|
| 295 |
+
## Roadmap
|
| 296 |
+
|
| 297 |
+
### Audio Modality Training (Planned)
|
| 298 |
+
|
| 299 |
+
The model architecture includes a Whisper audio encoder, but this release only trained on image-text data. Future releases will add audio-text alignment using:
|
| 300 |
+
|
| 301 |
+
| Dataset | Size | Description |
|
| 302 |
+
|---------|------|-------------|
|
| 303 |
+
| [WavCaps](https://huggingface.co/datasets/cvssp/WavCaps) | 400K | Largest audio-caption dataset |
|
| 304 |
+
| [AudioCaps](https://github.com/cdjkim/audiocaps) | 46K | YouTube audio with human captions |
|
| 305 |
+
| [Clotho](https://zenodo.org/record/3490684) | 6K | High-quality multi-annotator captions |
|
| 306 |
+
|
| 307 |
+
This will enable:
|
| 308 |
+
- Audio-to-text retrieval
|
| 309 |
+
- Text-to-audio retrieval
|
| 310 |
+
- Audio-image-text multimodal fusion
|
| 311 |
+
|
| 312 |
## Citation
|
| 313 |
|
| 314 |
```bibtex
|