--- language: - en - zh library_name: transformers pipeline_tag: video-text-to-text tags: - video-language-model - vision-language-model - multimodal - video-understanding - image-understanding - streaming-video license: apache-2.0 --- # VideoChat3-4B VideoChat3-4B is the 4B-parameter release of VideoChat3, a fully open, efficient, and generalist Video Multimodal Large Language Model for video understanding. The model is trained with a scalable video data synthesis and curation pipeline covering general, long-form, and streaming video scenarios. VideoChat3 achieves strong performance across mainstream video understanding benchmarks and surpasses prior open-source models at comparable or larger parameter scales, while releasing model weights, training code, training strategy, and training datasets to support reproducible open research. [๐Ÿ“„ Paper](http://arxiv.org/abs/2607.14935) ยท [๐ŸŒ Homepage](https://mcg-nju.github.io/VideoChat3/) ยท [๐Ÿ’ป GitHub](https://github.com/MCG-NJU/VideoChat3) ยท [๐Ÿค— Paper Page](https://huggingface.co/papers/2607.14935)

VideoChat3 Overview

## Installation ```bash pip install torch transformers accelerate qwen-vl-utils pip install decord opencv-python-headless ``` For faster inference, FlashAttention 2 can be used if it is correctly installed in your environment: ```bash pip install flash-attn --no-build-isolation ``` ## Quick Start The following examples show how to run image and video inputs. For a complete runnable script, please refer to `demo_vc3.py`. ### Image/Video Inference ```python from transformers import AutoModelForCausalLM, AutoProcessor from qwen_vl_utils import process_vision_info model_id = "MCG-NJU/VideoChat3-4B" model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype="auto", device_map="auto", trust_remote_code=True, ) model.eval() processor = AutoProcessor.from_pretrained( model_id, trust_remote_code=True, ) # image messages = [ { "role": "user", "content": [ {"type": "image", "image": "/path/to/image.jpg"}, {"type": "text", "text": "Describe this image."}, ], } ] # video # messages = [ # { # "role": "user", # "content": [ # {"type": "video", "video": "/path/to/video.mp4"}, # {"type": "text", "text": "What happens in this video?"}, # ], # } # ] text = processor.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, ) images, videos, video_kwargs = process_vision_info( messages, image_patch_size=14, return_video_kwargs=True, return_video_metadata=True, ) inputs = processor( text=text, images=images, videos=videos, do_resize=False, return_tensors="pt", **(video_kwargs or {}), ) inputs = inputs.to(model.device) if hasattr(model, "dtype"): inputs = inputs.to(model.dtype) generated_ids = model.generate( **inputs, max_new_tokens=128, do_sample=False, ) generated_ids = [ output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs.input_ids, generated_ids) ] output_text = processor.tokenizer.batch_decode( generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False, ) print(output_text[0]) ``` ### Online Proactive Response VideoChat3-4B supports proactive response for streaming video understanding. The proactive response protocol uses three explicit states: - ``: the current frame is irrelevant to the user question, or no meaningful event has started yet. - ``: a relevant event is unfolding, but the model needs more future frames before giving a reliable answer. When `` is predicted, the next frame is processed with a larger visual budget. - ``: the model has collected enough visual evidence and provides the answer. Example usage with the provided proactive demo: ```bash python demo_vc3_proactive.py \ --model MCG-NJU/VideoChat3-4B \ --video /path/to/video.mp4 \ --question "your question" ``` ## Citation ``` @article{li2026videochat3, title={VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding}, author={Li, Xinhao and Zhu, Yuhan and Zeng, Xiangyu and Dong, Yuhao and Wu, Haoning and Zhang, Zhiqiu and Yang, Yuandong and Ma, Changlian and Zhang, Qingyu and Shi, Yansong and others}, journal={arXiv preprint arXiv:2607.14935}, year={2026} } ```