Instructions to use Rihong/VideoChat2_HD_Infinity_Mistral_7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Rihong/VideoChat2_HD_Infinity_Mistral_7B with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("Rihong/VideoChat2_HD_Infinity_Mistral_7B", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
File size: 2,654 Bytes
56a2e58 6e80a24 c4f85ed | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 | ---
language:
- en
base_model:
- mistralai/Mistral-7B-Instruct-v0.2
pipeline_tag: video-text-to-text
library_name: transformers
---
# Infinite-VideoChat2
HuggingFace-compatible model files for **VideoChat2-Infinity** (Mistral-7B), a long video understanding model that extends [VideoChat2-HD](https://github.com/OpenGVLab/Ask-Anything/tree/main/video_chat2) with infinite-length video processing via continuous long-term attention.
## Model Architecture
- **Vision Encoder** – UMT-L (ViT, 1024-dim, 24 layers)
- **Q-Former** – bridges visual features to the LLM with 32 + 64 query tokens
- **LLM** – [Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2)
- **Long-Term Attention** – Gibbs-sampling-based continuous attention over basis functions for unbounded temporal context
## File Overview
| File | Description |
|------|-------------|
| `config.json` | HuggingFace `AutoConfig` configuration |
| `configuration_videochat2.py` | Custom `PretrainedConfig` subclass (`Config`) |
| `videochat2_it_hd_mistral.py` | Main model class (`VideoChat2_it_hd_mistral`) |
| `blip2.py` | BLIP-2 base class for vision-language bridging |
| `vit.py` | Vision Transformer (UMT-L) implementation |
| `Qformer.py` | Q-Former module |
| `basis_functions.py` | Basis functions (Power, Sine, Cosine, Gaussian, Rectangular) for long-term attention |
| `long_term_attention_gibbs.py` | Long-term attention with Gibbs sampling |
| `model-*.safetensors` | Model weights (sharded, 4 parts) |
| `.gitignore` | Ignores model weight files |
## Usage
The model is registered with HuggingFace `auto_map`, so it can be loaded directly:
```python
from transformers import AutoConfig, AutoModel
config = AutoConfig.from_pretrained("Rihong/VideoChat2_Infinity_Mistral_7B_hf", trust_remote_code=True)
model = AutoModel.from_pretrained("Rihong/VideoChat2_Infinity_Mistral_7B_hf", trust_remote_code=True)
```
To upload model to HuggingFace, just run:
```bash
hf upload Rihong/VideoChat2_Infinity_Mistral_7B_hf ./lmms_eval/baselines/infty_videochat2/
```
## Key Hyperparameters
| Parameter | Default | Description |
|-----------|---------|-------------|
| `num_basis` | 256 | Number of basis functions for long-term attention |
| `tau` | 0.75 | Temperature for Gibbs sampling |
| `alpha` | 0.75 | Mixing coefficient |
| `sticky` | `true` | Enable sticky memories |
| `hd_num` | 6 | Number of high-definition crops |
| `local_size` | 224 | Local crop resolution |
## References
- [VideoChat2 (Ask-Anything)](https://github.com/OpenGVLab/Ask-Anything/tree/main/video_chat2)
- [Infinite-Video](https://github.com/deep-spin/Infinite-Video)
|