| --- |
| license: cc-by-4.0 |
| tags: |
| - multimodal-retrieval |
| - text-video-retrieval |
| - text-audio-retrieval |
| - audio-visual |
| - hypergraph |
| - missing-modalities |
| --- |
| |
| <div align="center"> |
|
|
| # HyVol++ |
|
|
| ### Hypergraph Fusion for Multimodal Retrieval |
|
|
| **Anindya Nag · Ambuj Mehrish · Sebastiano Vascon** |
|
|
| Ca' Foscari University of Venice |
|
|
| [](https://arxiv.org/abs/2609.15320) |
| [](https://github.com/CVML-CFU/HyVol) |
| [](https://huggingface.co/CVML-CFU/HyVol) |
| [](https://creativecommons.org/licenses/by/4.0/) |
|
|
| </div> |
|
|
| --- |
|
|
| Pretrained weights for **Fuse To Align: Multimodal Representation Learning for Retrieval via Hypergraph Regularization**. |
|
|
| HyVol++ turns [HyVol](https://huggingface.co/CVML-CFU/HyVol) into a general **training-time hypergraph fusion** framework. |
| Observed modalities (video, audio, subtitles) are vertices, and caption-guided mutual top-k |
| neighbourhoods connect related training clips, so a clip can draw on evidence that related |
| clips have observed. The hypergraph is **discarded after training**, so the backbone, |
| scoring function and inference cost stay exactly the same. |
|
|
| ## ✨ Two new configurations |
|
|
| | | What it learns | Propagation | |
| |---|---|---| |
| | **wHyVol** (weighted HyVol) | a reliability weight per view + a symmetric caption affinity per pair of related clips | additive | |
| | **wRHG** (weighted relational hypergraph) | both weights + typed relations: document, same-modality, cross-modality and joint-candidate hyperedges | relation-conditioned, multiplicative | |
|
|
| Missing modalities get no vertex: they are **omitted, not imputed**. |
|
|
| ## 📦 Checkpoints |
|
|
| | Model | Backbone | File | |
| |---|---|---| |
| | wHyVol | GRAM | [`wHyVol/GRAM_wHyVol/wHyVol_GRAM.pt`](wHyVol/GRAM_wHyVol/wHyVol_GRAM.pt) | |
| | wHyVol | HyperGRAM | [`wHyVol/HyperGRAM_wHyVol/wHyVol_HyperGRAM.pt`](wHyVol/HyperGRAM_wHyVol/wHyVol_HyperGRAM.pt) | |
| | wRHG | GRAM | [`wRHG/GRAM_wRHG/wRHG_GRAM.pt`](wRHG/GRAM_wRHG/wRHG_GRAM.pt) | |
| | wRHG | HyperGRAM | [`wRHG/HyperGRAM_wRHG/wRHG_HyperGRAM.pt`](wRHG/HyperGRAM_wRHG/wRHG_HyperGRAM.pt) | |
|
|
| - **Initialisation:** VAST (EVA-CLIP ViT-g/14 vision, BEATs audio, BERT-base text) |
| - **Training data:** one epoch on the 150K-clip subset of VAST-27M (136,694 clips recovered with video and audio) |
| - **Checkpoint selection:** best text-to-video R@1 of the matching (ITM) score on MSR-VTT 1k, T-VAS |
|
|
| ## 🚀 Usage |
|
|
| ```python |
| from huggingface_hub import hf_hub_download |
| |
| ckpt = hf_hub_download( |
| repo_id="CVML-CFU/HyVol-PlusPlus", |
| filename="wRHG/GRAM_wRHG/wRHG_GRAM.pt", |
| ) |
| ``` |
|
|
| Load each checkpoint with the codebase of its backbone: |
| [GRAM](https://github.com/ispamm/GRAM) or [HyperGRAM](https://github.com/uta-smile/HyperGram). |
| No extra module is needed at test time. |
|
|
| ## 📊 Results |
|
|
| Zero-shot on **eleven benchmarks**: eight text-video (MSR-VTT, DiDeMo, ActivityNet, VATEX, |
| Shot2Story, MSVD, FLARE, VALOR-32K) and three text-audio (AudioCaps, Clotho, SoundDescs). |
|
|
| | Backbone | Mean T2V R@1 (17 settings) | + wRHG | Gain | |
| |---|:---:|:---:|:---:| |
| | GRAM | 63.4 | **68.2** | +4.8 | |
| | HyperGRAM | 64.8 | **68.4** | +3.6 | |
|
|
| - With GRAM, wRHG improves **all 34** text-to-video and video-to-text settings. |
| - With HyperGRAM, it improves **33 of 34**. |
| - The largest video-to-text gain is **+13.3 R@1** on ActivityNet. |
|
|
| ## 📝 Citation |
|
|
| If you use these weights, please cite: |
|
|
| ```bibtex |
| @inproceedings{nag2027hyvol, |
| title = {Hypergraph-Regularized Gramian Volumes for Multimodal Retrieval}, |
| author = {Nag, Anindya and Mehrish, Ambuj and Vascon, Sebastiano}, |
| booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)}, |
| year = {2027} |
| } |
| ``` |
|
|
