qqceqqq
/

VideoChat3-4B / README.md
qqceqqq's picture lixinhao's picture
Duplicate from MCG-NJU/VideoChat3-4B
aee3597
|
Raw History Blame Contribute Delete
4.59 kB
---
language:
- en
- zh
library_name: transformers
pipeline_tag: video-text-to-text
tags:
- video-language-model
- vision-language-model
- multimodal
- video-understanding
- image-understanding
- streaming-video
license: apache-2.0
---
# VideoChat3-4B
VideoChat3-4B is the 4B-parameter release of VideoChat3, a fully open, efficient, and generalist Video Multimodal Large Language Model for video understanding.
The model is trained with a scalable video data synthesis and curation pipeline covering general, long-form, and streaming video scenarios. VideoChat3 achieves strong performance across mainstream video understanding benchmarks and surpasses prior open-source models at comparable or larger parameter scales, while releasing model weights, training code, training strategy, and training datasets to support reproducible open research.
[馃搫 Paper](http://arxiv.org/abs/2607.14935) 路 [馃寪 Homepage](https://mcg-nju.github.io/VideoChat3/) 路 [馃捇 GitHub](https://github.com/MCG-NJU/VideoChat3) 路 [馃 Paper Page](https://huggingface.co/papers/2607.14935)
<p align="center">
<img src="https://raw.githubusercontent.com/MCG-NJU/VideoChat3/main/docs/public/paper/overview.png" alt="VideoChat3 Overview" width="100%">
</p>
## Installation
```bash
pip install torch transformers accelerate qwen-vl-utils
pip install decord opencv-python-headless
```
For faster inference, FlashAttention 2 can be used if it is correctly installed in your environment:
```bash
pip install flash-attn --no-build-isolation
```
## Quick Start
The following examples show how to run image and video inputs. For a complete runnable script, please refer to `demo_vc3.py`.
### Image/Video Inference
```python
from transformers import AutoModelForCausalLM, AutoProcessor
from qwen_vl_utils import process_vision_info
model_id = "MCG-NJU/VideoChat3-4B"
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype="auto",
device_map="auto",
trust_remote_code=True,
)
model.eval()
processor = AutoProcessor.from_pretrained(
model_id,
trust_remote_code=True,
)
# image
messages = [
{
"role": "user",
"content": [
{"type": "image", "image": "/path/to/image.jpg"},
{"type": "text", "text": "Describe this image."},
],
}
]
# video
# messages = [
# {
# "role": "user",
# "content": [
# {"type": "video", "video": "/path/to/video.mp4"},
# {"type": "text", "text": "What happens in this video?"},
# ],
# }
# ]
text = processor.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
)
images, videos, video_kwargs = process_vision_info(
messages,
image_patch_size=14,
return_video_kwargs=True,
return_video_metadata=True,
)
inputs = processor(
text=text,
images=images,
videos=videos,
do_resize=False,
return_tensors="pt",
**(video_kwargs or {}),
)
inputs = inputs.to(model.device)
if hasattr(model, "dtype"):
inputs = inputs.to(model.dtype)
generated_ids = model.generate(
**inputs,
max_new_tokens=128,
do_sample=False,
)
generated_ids = [
output_ids[len(input_ids):]
for input_ids, output_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.tokenizer.batch_decode(
generated_ids,
skip_special_tokens=True,
clean_up_tokenization_spaces=False,
)
print(output_text[0])
```
### Online Proactive Response
VideoChat3-4B supports proactive response for streaming video understanding.
The proactive response protocol uses three explicit states:
- `</Silence>`: the current frame is irrelevant to the user question, or no meaningful event has started yet.
- `</Standby>`: a relevant event is unfolding, but the model needs more future frames before giving a reliable answer. When `</Standby>` is predicted, the next frame is processed with a larger visual budget.
- `</Response>`: the model has collected enough visual evidence and provides the answer.
Example usage with the provided proactive demo:
```bash
python demo_vc3_proactive.py \
--model MCG-NJU/VideoChat3-4B \
--video /path/to/video.mp4 \
--question "your question"
```
## Citation
```
@article{li2026videochat3,
title={VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding},
author={Li, Xinhao and Zhu, Yuhan and Zeng, Xiangyu and Dong, Yuhao and Wu, Haoning and Zhang, Zhiqiu and Yang, Yuandong and Ma, Changlian and Zhang, Qingyu and Shi, Yansong and others},
journal={arXiv preprint arXiv:2607.14935},
year={2026}
}
```