| --- |
| license: apache-2.0 |
| pipeline_tag: video-text-to-text |
| library_name: transformers |
| --- |
| |
| <p align="center"> |
| <img src="./assets/raven_logo.png" width="100" style="margin-bottom: 0.2;"/> |
| <p> |
| |
| <h3 align="center"> |
| <a href="https://arxiv.org/pdf/2505.17114" style="color:#825987"> |
| RAVEN: Query-Guided Representation Alignment for Question |
| Answering over Audio, Video, Embedded Sensors, and Natural Language |
| </a> |
| </h3> |
| <h5 align="center"> |
| Project Page: |
| <a href="https://bashlab.github.io/raven_project/" style="color:#825987"> |
| https://bashlab.github.io/raven_project/ |
| </a> |
| • Code: |
| <a href="https://github.com/BASHLab/RAVEN" style="color:#825987"> |
| https://github.com/BASHLab/RAVEN |
| </a> |
| </h5> |
| <p align="center"> |
| <img src="./assets/raven_architecture.png" width="800" /> |
| <p> |
| |
| --- |
| |
| ## Abstract |
| Multimodal question answering (QA) often requires identifying which video, audio, or sensor tokens are relevant to the question. Yet modality disagreements are common: off-camera speech, background noise, or motion outside the field of view often mislead fusion models that weight all streams equally. We present RAVEN, a unified QA architecture whose core is QuART, a query-conditioned cross-modal gating module that assigns scalar relevance scores to each token across modalities, enabling the model to amplify informative signals and suppress distractors before fusion. RAVEN is trained through a three-stage pipeline comprising unimodal pretraining, query-aligned fusion, and disagreement-oriented fine-tuning -- each stage targeting a distinct challenge in multi-modal reasoning: representation quality, cross-modal relevance, and robustness to modality mismatch. To support training and evaluation, we release AVS-QA, a dataset of 300K synchronized Audio--Video-Sensor streams paired with automatically generated question-answer pairs. Experimental results on seven multi-modal QA benchmarks -- including egocentric and exocentric tasks -- show that RAVEN achieves up to 14.5% and 8.0% gains in accuracy compared to state-of-the-art multi-modal large language models, respectively. Incorporating sensor data provides an additional 16.4% boost, and the model remains robust under modality corruption, outperforming SOTA baselines by 50.23%. Our code and dataset are available at this https URL . |
|
|
| --- |
| ## 🚀 Main Results |
| ##### Comparison of **RAVEN** and prior MLLMs on *exocentric* open-ended video QA (MSVD-QA, MSRVTT-QA, ActivityNet-QA) and audio-visual QA (AVSD, MUSIC-QA) benchmarks. Best and second-best scores are in $\textbf{Bold}$ and $\underline{\text{underline}}$. $^*$ indicates scores reproduced by us. |
| <p><img src="./assets/main_result_exo.png" width="800"></p> |
| |
| ##### Comparison of **RAVEN** with MLLMs on the EgoThink (Reasoning) and AVS-QA benchmarks. **RAVEN** outperforms across metrics and excels in reasoning. $\textbf{Bold}$ and $\underline{\text{underline}}$ indicate the best and second-best scores. |
| <p><img src="./assets/main_result_ego.png" width="800"></p> |
| |
| --- |
| ## 📁 **AVS-QA** Dataset |
| Train and test split of **AVS-QA** is provided [here](./avs-qa-dataset/).<br> |
| More details [here](./avs-qa-dataset/README.md). |
| |
| ## 🛠️ Requirements and Installation |
| Basic Dependencies: |
| * Python >= 3.8 |
| * Pytorch >= 2.2.0 |
| * CUDA Version >= 11.8 |
| * transformers == 4.40.0 (for reproducing paper results) |
| * tokenizers == 0.19.1 |
|
|
| ```bash |
| cd RAVEN |
| pip install -r requirements.txt |
| pip install flash-attn==2.5.8 --no-build-isolation |
| pip install opencv-python==4.5.5.64 |
| apt-get update && apt-get install ffmpeg libsm6 libxext6 -y |
| ``` |
| --- |
|
|
| ## 🍀 Model Zoo |
| | Model Name | Modal Type | |
| |:----------------|:------------:| |
| | [RAVEN-7B-AV](https://huggingface.co/BASH-Lab/RAVEN-AV-7B)| AV | |
| | RAVEN-7B-AVS| AVS | |
|
|
| ## 🤖 Sample Usage |
| - **STEP 1:** Download $\texttt{siglip-so400m-patch14-384}$ from here [google/siglip-so400m-patch14-384](https://huggingface.co/google/siglip-so400m-patch14-384) |
| - **STEP 2:** Download **RAVEN** checkpoint |
| ```bash |
| CUDA_VISIBLE_DEVICES=0 python inference.py --model-path=<MODEL PATH> --modal-type=<MODAL TYPE> |
| ``` |
|
|
| ## 👍 Acknowledgement |
| The codebase of RAVEN is adapted from [**VideoLLaMA2**](https://github.com/DAMO-NLP-SG/VideoLLaMA2). We are also grateful for their contribution. |
