HuaminChen commited on
Commit
f2cea71
·
verified ·
1 Parent(s): a82f367

Add audio training roadmap to model card

Browse files
Files changed (1) hide show
  1. README.md +18 -1
README.md CHANGED
@@ -289,9 +289,26 @@ fused_embedding = model.encode_multimodal(
289
 
290
  - Optimized for English; multilingual performance may vary
291
  - Image resolution fixed at 512x512
292
- - Audio modality available but not trained in this release
293
  - Best for semantic similarity, not generative tasks
294
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
295
  ## Citation
296
 
297
  ```bibtex
 
289
 
290
  - Optimized for English; multilingual performance may vary
291
  - Image resolution fixed at 512x512
292
+ - Audio encoder included but not yet trained (see Roadmap)
293
  - Best for semantic similarity, not generative tasks
294
 
295
+ ## Roadmap
296
+
297
+ ### Audio Modality Training (Planned)
298
+
299
+ The model architecture includes a Whisper audio encoder, but this release only trained on image-text data. Future releases will add audio-text alignment using:
300
+
301
+ | Dataset | Size | Description |
302
+ |---------|------|-------------|
303
+ | [WavCaps](https://huggingface.co/datasets/cvssp/WavCaps) | 400K | Largest audio-caption dataset |
304
+ | [AudioCaps](https://github.com/cdjkim/audiocaps) | 46K | YouTube audio with human captions |
305
+ | [Clotho](https://zenodo.org/record/3490684) | 6K | High-quality multi-annotator captions |
306
+
307
+ This will enable:
308
+ - Audio-to-text retrieval
309
+ - Text-to-audio retrieval
310
+ - Audio-image-text multimodal fusion
311
+
312
  ## Citation
313
 
314
  ```bibtex