Blame for README.md

---
license: cc-by-4.0
tags:
- multimodal-retrieval
- text-video-retrieval
- text-audio-retrieval
- audio-visual
- hypergraph
- missing-modalities
---

<div align="center">

# HyVol++

### Hypergraph Fusion for Multimodal Retrieval

**Anindya Nag · Ambuj Mehrish · Sebastiano Vascon**

Ca' Foscari University of Venice

[![HyVol](https://img.shields.io/badge/WACV%202027-HyVol-blue)](https://arxiv.org/abs/2609.15320)
[![Code](https://img.shields.io/badge/GitHub-HyVol-black?logo=github)](https://github.com/CVML-CFU/HyVol)
[![HyVol weights](https://img.shields.io/badge/%F0%9F%A4%97%20Weights-HyVol-yellow)](https://huggingface.co/CVML-CFU/HyVol)
[![License](https://img.shields.io/badge/License-CC%20BY%204.0-lightgrey)](https://creativecommons.org/licenses/by/4.0/)

</div>

---

Pretrained weights for **Fuse To Align: Multimodal Representation Learning for Retrieval via Hypergraph Regularization**.

HyVol++ turns [HyVol](https://huggingface.co/CVML-CFU/HyVol) into a general **training-time hypergraph fusion** framework.
Observed modalities (video, audio, subtitles) are vertices, and caption-guided mutual top-k
neighbourhoods connect related training clips, so a clip can draw on evidence that related
clips have observed. The hypergraph is **discarded after training**, so the backbone,
scoring function and inference cost stay exactly the same.

## ✨ Two new configurations

| | What it learns | Propagation |
|---|---|---|
| **wHyVol** (weighted HyVol) | a reliability weight per view + a symmetric caption affinity per pair of related clips | additive |
| **wRHG** (weighted relational hypergraph) | both weights + typed relations: document, same-modality, cross-modality and joint-candidate hyperedges | relation-conditioned, multiplicative |

Missing modalities get no vertex: they are **omitted, not imputed**.

## 📦 Checkpoints

| Model | Backbone | File |
|---|---|---|
| wHyVol | GRAM | [`wHyVol/GRAM_wHyVol/wHyVol_GRAM.pt`](wHyVol/GRAM_wHyVol/wHyVol_GRAM.pt) |
| wHyVol | HyperGRAM | [`wHyVol/HyperGRAM_wHyVol/wHyVol_HyperGRAM.pt`](wHyVol/HyperGRAM_wHyVol/wHyVol_HyperGRAM.pt) |
| wRHG | GRAM | [`wRHG/GRAM_wRHG/wRHG_GRAM.pt`](wRHG/GRAM_wRHG/wRHG_GRAM.pt) |
| wRHG | HyperGRAM | [`wRHG/HyperGRAM_wRHG/wRHG_HyperGRAM.pt`](wRHG/HyperGRAM_wRHG/wRHG_HyperGRAM.pt) |

- **Initialisation:** VAST (EVA-CLIP ViT-g/14 vision, BEATs audio, BERT-base text)
- **Training data:** one epoch on the 150K-clip subset of VAST-27M (136,694 clips recovered with video and audio)
- **Checkpoint selection:** best text-to-video R@1 of the matching (ITM) score on MSR-VTT 1k, T-VAS

## 🚀 Usage

```python
from huggingface_hub import hf_hub_download

ckpt = hf_hub_download(
    repo_id="CVML-CFU/HyVol-PlusPlus",
    filename="wRHG/GRAM_wRHG/wRHG_GRAM.pt",
)
```

Load each checkpoint with the codebase of its backbone:
[GRAM](https://github.com/ispamm/GRAM) or [HyperGRAM](https://github.com/uta-smile/HyperGram).
No extra module is needed at test time.

## 📊 Results

Zero-shot on **eleven benchmarks**: eight text-video (MSR-VTT, DiDeMo, ActivityNet, VATEX,
Shot2Story, MSVD, FLARE, VALOR-32K) and three text-audio (AudioCaps, Clotho, SoundDescs).

| Backbone | Mean T2V R@1 (17 settings) | + wRHG | Gain |
|---|:---:|:---:|:---:|
| GRAM | 63.4 | **68.2** | +4.8 |
| HyperGRAM | 64.8 | **68.4** | +3.6 |

- With GRAM, wRHG improves **all 34** text-to-video and video-to-text settings.
- With HyperGRAM, it improves **33 of 34**.
- The largest video-to-text gain is **+13.3 R@1** on ActivityNet.

## 📝 Citation

If you use these weights, please cite:

```bibtex
@inproceedings{nag2027hyvol,
  title     = {Hypergraph-Regularized Gramian Volumes for Multimodal Retrieval},
  author    = {Nag, Anindya and Mehrish, Ambuj and Vascon, Sebastiano},
  booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year      = {2027}
}
```