Video-Text-to-Text
Transformers
Safetensors
English
Chinese
videochat3
feature-extraction
video-language-model
vision-language-model
multimodal
video-understanding
image-understanding
streaming-video
custom_code
Instructions to use qqceqqq/VideoChat3-4B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use qqceqqq/VideoChat3-4B with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("qqceqqq/VideoChat3-4B", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
|
Download README.md from qqceqqq/VideoChat3-4B: direct link, hf CLI and curl.
- Browser
- Download file 4.59 kB
-
https://huggingface.co/qqceqqq/VideoChat3-4B/resolve/main/README.md
- Command line
-
hf download hf://qqceqqq/VideoChat3-4B/README.md
-
curl -L -o README.md https://huggingface.co/qqceqqq/VideoChat3-4B/resolve/main/README.md
4.59 kB
| language: | |
| - en | |
| - zh | |
| library_name: transformers | |
| pipeline_tag: video-text-to-text | |
| tags: | |
| - video-language-model | |
| - vision-language-model | |
| - multimodal | |
| - video-understanding | |
| - image-understanding | |
| - streaming-video | |
| license: apache-2.0 | |
| # VideoChat3-4B | |
| VideoChat3-4B is the 4B-parameter release of VideoChat3, a fully open, efficient, and generalist Video Multimodal Large Language Model for video understanding. | |
| The model is trained with a scalable video data synthesis and curation pipeline covering general, long-form, and streaming video scenarios. VideoChat3 achieves strong performance across mainstream video understanding benchmarks and surpasses prior open-source models at comparable or larger parameter scales, while releasing model weights, training code, training strategy, and training datasets to support reproducible open research. | |
| [馃搫 Paper](http://arxiv.org/abs/2607.14935) 路 [馃寪 Homepage](https://mcg-nju.github.io/VideoChat3/) 路 [馃捇 GitHub](https://github.com/MCG-NJU/VideoChat3) 路 [馃 Paper Page](https://huggingface.co/papers/2607.14935) | |
| <p align="center"> | |
| <img src="https://raw.githubusercontent.com/MCG-NJU/VideoChat3/main/docs/public/paper/overview.png" alt="VideoChat3 Overview" width="100%"> | |
| </p> | |
| ## Installation | |
| ```bash | |
| pip install torch transformers accelerate qwen-vl-utils | |
| pip install decord opencv-python-headless | |
| ``` | |
| For faster inference, FlashAttention 2 can be used if it is correctly installed in your environment: | |
| ```bash | |
| pip install flash-attn --no-build-isolation | |
| ``` | |
| ## Quick Start | |
| The following examples show how to run image and video inputs. For a complete runnable script, please refer to `demo_vc3.py`. | |
| ### Image/Video Inference | |
| ```python | |
| from transformers import AutoModelForCausalLM, AutoProcessor | |
| from qwen_vl_utils import process_vision_info | |
| model_id = "MCG-NJU/VideoChat3-4B" | |
| model = AutoModelForCausalLM.from_pretrained( | |
| model_id, | |
| torch_dtype="auto", | |
| device_map="auto", | |
| trust_remote_code=True, | |
| ) | |
| model.eval() | |
| processor = AutoProcessor.from_pretrained( | |
| model_id, | |
| trust_remote_code=True, | |
| ) | |
| # image | |
| messages = [ | |
| { | |
| "role": "user", | |
| "content": [ | |
| {"type": "image", "image": "/path/to/image.jpg"}, | |
| {"type": "text", "text": "Describe this image."}, | |
| ], | |
| } | |
| ] | |
| # video | |
| # messages = [ | |
| # { | |
| # "role": "user", | |
| # "content": [ | |
| # {"type": "video", "video": "/path/to/video.mp4"}, | |
| # {"type": "text", "text": "What happens in this video?"}, | |
| # ], | |
| # } | |
| # ] | |
| text = processor.apply_chat_template( | |
| messages, | |
| tokenize=False, | |
| add_generation_prompt=True, | |
| ) | |
| images, videos, video_kwargs = process_vision_info( | |
| messages, | |
| image_patch_size=14, | |
| return_video_kwargs=True, | |
| return_video_metadata=True, | |
| ) | |
| inputs = processor( | |
| text=text, | |
| images=images, | |
| videos=videos, | |
| do_resize=False, | |
| return_tensors="pt", | |
| **(video_kwargs or {}), | |
| ) | |
| inputs = inputs.to(model.device) | |
| if hasattr(model, "dtype"): | |
| inputs = inputs.to(model.dtype) | |
| generated_ids = model.generate( | |
| **inputs, | |
| max_new_tokens=128, | |
| do_sample=False, | |
| ) | |
| generated_ids = [ | |
| output_ids[len(input_ids):] | |
| for input_ids, output_ids in zip(inputs.input_ids, generated_ids) | |
| ] | |
| output_text = processor.tokenizer.batch_decode( | |
| generated_ids, | |
| skip_special_tokens=True, | |
| clean_up_tokenization_spaces=False, | |
| ) | |
| print(output_text[0]) | |
| ``` | |
| ### Online Proactive Response | |
| VideoChat3-4B supports proactive response for streaming video understanding. | |
| The proactive response protocol uses three explicit states: | |
| - `</Silence>`: the current frame is irrelevant to the user question, or no meaningful event has started yet. | |
| - `</Standby>`: a relevant event is unfolding, but the model needs more future frames before giving a reliable answer. When `</Standby>` is predicted, the next frame is processed with a larger visual budget. | |
| - `</Response>`: the model has collected enough visual evidence and provides the answer. | |
| Example usage with the provided proactive demo: | |
| ```bash | |
| python demo_vc3_proactive.py \ | |
| --model MCG-NJU/VideoChat3-4B \ | |
| --video /path/to/video.mp4 \ | |
| --question "your question" | |
| ``` | |
| ## Citation | |
| ``` | |
| @article{li2026videochat3, | |
| title={VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding}, | |
| author={Li, Xinhao and Zhu, Yuhan and Zeng, Xiangyu and Dong, Yuhao and Wu, Haoning and Zhang, Zhiqiu and Yang, Yuandong and Ma, Changlian and Zhang, Qingyu and Shi, Yansong and others}, | |
| journal={arXiv preprint arXiv:2607.14935}, | |
| year={2026} | |
| } | |
| ``` |