README.md

---
language:
- en
license: apache-2.0
tags:
- video-understanding
- sparse-attention
- vision-language
- qwen2.5-vl
- multimodal
pipeline_tag: video-text-to-text
---

# VideoNSA: Native Sparse Attention for Video Understanding

<div align="center">
  <img src="VideoNSA.png" alt="VideoNSA Overview" width="100%">
</div>

## Model Description

VideoNSA is a learnable, hardware-aware sparse-attention framework built on Qwen2.5-VL-7B for efficient long video understanding. It processes up to **128K vision-text tokens** using only **3.6%** of the full attention budget while maintaining competitive performance.

### Key Features

- 🎯 **Learned Sparsity**: Intelligently learns sparsity patterns over video tokens
- 🚀 **Efficient Scaling**: Handles massive video contexts with minimal computational overhead
- 🎬 **Hybrid Attention**: Combines compression, selection, and sliding window mechanisms
- 🔧 **Hardware-Aware**: Optimized for efficient inference on modern GPUs
- 📊 **Strong Performance**: Achieves leading results on video understanding benchmarks

## Model Architecture

VideoNSA employs a hybrid attention strategy with three complementary branches:

1. **Compression Branch**: Averages frame KV blocks to maintain salient visual cues
2. **Selection Branch**: Ranks and retains the most informative video segments
3. **Sliding Window Branch**: Ensures local temporal coverage for fine-grained details

Each branch is weighted by learnable per-head gates for adaptive token allocation across different tasks.

## Training Details

- **Base Model**: Qwen2.5-VL-7B-Instruct
- **Dataset**: Filtered LLaVA-Video-178K
- **Sampling Rate**: 4 fps
- **Context Limit**: 36K tokens during training
- **Compute**: ~4,600 H100 GPU hours

## Usage

For installation, training, and evaluation instructions, please refer to:
- 💻 [GitHub Repository](https://github.com/Espere-1119-Song/VideoNSA)
- 🌐 [Project Page](https://enxinsong.com/VideoNSA-web/)

## Limitations

- Optimized for video understanding tasks; may not be optimal for pure image tasks
- Requires sufficient GPU memory for long video processing
- Performance may vary with different video resolutions and frame rates

## Citation

```bibtex
@misc{song2025videonsanativesparseattention,
      title={VideoNSA: Native Sparse Attention Scales Video Understanding}, 
      author={Enxin Song and Wenhao Chai and Shusheng Yang and Ethan Armand and Xiaojun Shan and Haiyang Xu and Jianwen Xie and Zhuowen Tu},
      year={2025},
      eprint={2510.02295},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2510.02295}, 
}
```

## Resources

- 📄 [Paper](https://arxiv.org/abs/2510.02295)
- 🌐 [Project Page](https://enxinsong.com/VideoNSA-web/)
- 💻 [GitHub Repository](https://github.com/Espere-1119-Song/VideoNSA)

## License

This model is released under the Apache 2.0 License.