Update model card from paper

README.md

@@ -3,34 +3,71 @@ license: cc-by-4.0
 tags:
 - multimodal-retrieval
 - text-video-retrieval
+- text-audio-retrieval
 - audio-visual
 - hypergraph
+- missing-modalities
 ---
 
-# HyVol++: Weighted and Relational Hypergraph-Regularized Gramian Volumes
+# HyVol++: Hypergraph Fusion for Multimodal Retrieval
 
-Pretrained weights by Anindya Nag, Ambuj Mehrish and Sebastiano Vascon.
+Pretrained weights for the paper
+**Linking Modalities, Learning Across Samples: Hypergraph Fusion for Multimodal Retrieval**
+by Anindya Nag, Ambuj Mehrish and Sebastiano Vascon (under review).
 
-HyVol++ extends HyVol ([CVML-CFU/HyVol](https://huggingface.co/CVML-CFU/HyVol)) with two
-hypergraph variants that add semantic relations between training samples before the
-volume loss is computed:
+This work extends HyVol ([CVML-CFU/HyVol](https://huggingface.co/CVML-CFU/HyVol), WACV 2027)
+into a general training-time hypergraph fusion framework. Observed modalities (video, audio,
+subtitles) are vertices, and caption-guided mutual top-k neighbourhoods connect related
+training clips, so a clip can draw on evidence that related clips have observed.
+Two new configurations are released here:
 
-- **wHyVol**: weighted hypergraph with learned per-edge weights and attention.
-- **wRHG**: weighted relational hypergraph with four edge families
-  (document, same-modality, cross-modality and friend).
+- **wHyVol** (weighted HyVol): a learned reliability weight for each view and a symmetric
+  caption affinity for each pair of related clips, with additive hypergraph propagation.
+- **wRHG** (weighted relational hypergraph): keeps both weights and replaces untyped links
+  with typed relations (document, same-modality, cross-modality and joint-candidate
+  hyperedges), with relation-conditioned message passing.
 
-The hypergraph is removed after training, so the backbone, scoring function and
-retrieval cost are unchanged.
+Missing modalities are omitted rather than imputed. The hypergraph is discarded after
+training, so the backbone, scoring function and inference cost are unchanged.
 
 ## Checkpoints
 
 | File | Backbone |
 |---|---|
 | `wHyVol/GRAM_wHyVol/wHyVol_GRAM.pt` | GRAM + wHyVol |
-| `wHyVol/HyperGRAM_wHyVol/wHyVol_HyperGRAM.pt` | Hyperbolic GRAM + wHyVol |
+| `wHyVol/HyperGRAM_wHyVol/wHyVol_HyperGRAM.pt` | HyperGRAM + wHyVol |
 | `wRHG/GRAM_wRHG/wRHG_GRAM.pt` | GRAM + wRHG |
-| `wRHG/HyperGRAM_wRHG/wRHG_HyperGRAM.pt` | Hyperbolic GRAM + wRHG |
+| `wRHG/HyperGRAM_wRHG/wRHG_HyperGRAM.pt` | HyperGRAM + wRHG |
 
-All models are trained on a 150K-clip subset of VAST-27M and evaluated zero-shot.
-Each checkpoint is the ITM-best one on MSR-VTT (T-VAS).
-Load each checkpoint with the codebase of its backbone (GRAM or Hyperbolic GRAM).
+All models are initialised from VAST and trained for one epoch on the 150K-clip subset
+of VAST-27M (136,694 clips recovered with both video and audio). The checkpoint is selected
+by text-to-video R@1 of the matching (ITM) score on MSR-VTT 1k in the T-VAS setting.
+Load each checkpoint with the codebase of its backbone
+([GRAM](https://github.com/ispamm/GRAM) or [HyperGRAM](https://github.com/uta-smile/HyperGram)).
+
+## Results
+
+Zero-shot evaluation on eleven benchmarks: eight text-video (MSR-VTT, DiDeMo, ActivityNet,
+VATEX, Shot2Story, MSVD, FLARE, VALOR-32K) and three text-audio (AudioCaps, Clotho, SoundDescs).
+
+| Backbone | Mean T2V R@1 (17 settings) | + wRHG |
+|---|---|---|
+| GRAM | 63.4 | 68.2 |
+| HyperGRAM | 64.8 | 68.4 |
+
+With GRAM, wRHG improves all 17 text-to-video and all 17 video-to-text settings.
+With HyperGRAM, it improves 16 of 17 and 17 of 17. The largest video-to-text gain is
++13.3 R@1 on ActivityNet.
+
+## Citation
+
+The paper is under review. Until it is published, please cite the conference version:
+
+```bibtex
+@inproceedings{nag2027hyvol,
+  title     = {Hypergraph-Regularized Gramian Volumes for Multimodal Retrieval},
+  author    = {Nag, Anindya and Mehrish, Ambuj and Vascon, Sebastiano},
+  booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
+  year      = {2027}
+}
+```