Video-Text-to-Text
Transformers
Safetensors
English
qwen2_5_vl
image-text-to-text
streamo
streaming-video
video-understanding
multimodal
text-generation-inference
Instructions to use JiaerX/Streamo_3B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use JiaerX/Streamo_3B with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("JiaerX/Streamo_3B") model = AutoModelForMultimodalLM.from_pretrained("JiaerX/Streamo_3B", device_map="auto") - Notebooks
- Google Colab
- Kaggle
|
Download README.md from JiaerX/Streamo_3B: direct link, hf CLI and curl.
- Browser
- Download file 5.82 kB
-
https://huggingface.co/JiaerX/Streamo_3B/resolve/main/README.md
- Command line
-
hf download hf://JiaerX/Streamo_3B/README.md
-
curl -L -o README.md https://huggingface.co/JiaerX/Streamo_3B/resolve/main/README.md
5.82 kB
| language: | |
| - en | |
| base_model: | |
| - Qwen/Qwen2.5-VL-3B-Instruct | |
| base_model_relation: finetune | |
| pipeline_tag: video-text-to-text | |
| library_name: transformers | |
| tags: | |
| - streamo | |
| - streaming-video | |
| - video-understanding | |
| - multimodal | |
| - qwen2_5_vl | |
| - arxiv:2512.21334 | |
| datasets: | |
| - maifoundations/Streamo-Instruct-465K | |
| # Streamo-3B | |
| **Streamo-3B** is a streaming video language model based on [Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct), developed for **Streaming Video Instruction Tuning**. It processes incoming video frames with conversational context and decides when to remain silent, wait for more evidence, or respond. | |
| [Paper](https://arxiv.org/abs/2512.21334) 路 [Project page](https://jiaerxia.github.io/Streamo/) 路 [Code](https://github.com/maifoundations/Streamo) 路 [Streamo-Instruct-465K](https://huggingface.co/datasets/maifoundations/Streamo-Instruct-465K) | |
| Streamo targets streaming tasks such as event narration, action understanding, temporal grounding, and time-sensitive question answering. This repository contains the model weights, tokenizer, and processor files. The inference implementation is maintained in the [Streamo code repository](https://github.com/maifoundations/Streamo). | |
| ## Model details | |
| | Item | Value | | |
| | --- | --- | | |
| | Base model | Qwen/Qwen2.5-VL-3B-Instruct | | |
| | Architecture | `Qwen2_5_VLForConditionalGeneration` | | |
| | Checkpoint precision | BF16 | | |
| | Weight format | Safetensors, two shards | | |
| | Inputs | Video frames, timestamps, and text instructions | | |
| | Outputs | Streaming state tokens and text responses | | |
| Load the tokenizer and processor from **this checkpoint** to retain the additional streaming tokens. | |
| ## Quick start | |
| ### 1. Install the dependencies | |
| Use a Linux CUDA environment with compatible [PyTorch and torchvision](https://pytorch.org/get-started/locally/) installations. Run the following commands in a dedicated Python environment: | |
| ```bash | |
| git clone https://github.com/maifoundations/Streamo.git | |
| cd Streamo | |
| python -m pip install -r requirements.txt \ | |
| "transformers==4.57.1" \ | |
| "qwen-vl-utils>=0.0.6,<0.0.12" \ | |
| decord opencv-python huggingface_hub | |
| ``` | |
| The checkpoint configuration records Transformers `4.57.1`, which is within the repository's supported version range. The bundled [Qwen model loader](https://github.com/maifoundations/Streamo/blob/main/swift/llm/model/model/qwen.py) requires `qwen-vl-utils<0.0.12`. Run inference from the cloned repository so that Python imports its bundled `swift` implementation. | |
| ### 2. Download the checkpoint | |
| ```bash | |
| # Authenticate first if repository access requires it. | |
| hf auth login | |
| hf download JiaerX/Streamo_3B --local-dir ./checkpoints/Streamo_3B | |
| ``` | |
| Download the complete repository, including both weight shards and the tokenizer/processor files. See the [Hugging Face CLI documentation](https://huggingface.co/docs/huggingface_hub/guides/cli) for authentication and download options. | |
| ### 3. Configure and run inference | |
| Edit the existing variables in the `__main__` block of [`inference.py`](https://github.com/maifoundations/Streamo/blob/main/inference.py): | |
| ```python | |
| infer_backend = 'pt' | |
| model = './checkpoints/Streamo_3B' | |
| video_path = './demo/cook.mp4' | |
| target_fps = 1.0 | |
| question = 'Detect and summarize each event sequence in the video.' | |
| global_question = True | |
| question_time = 0 | |
| max_rounds = 300 | |
| ``` | |
| Then run: | |
| ```bash | |
| python inference.py | |
| ``` | |
| The script uses configuration variables, not command-line flags. This example selects its PyTorch backend; the upstream default is `vllm`, which requires a separate vLLM installation compatible with your PyTorch, CUDA, and Transformers versions. GPU selection is set inside the script with `CUDA_VISIBLE_DEVICES='0'`; edit it there if needed. | |
| Keep `target_fps=1.0`: the current frame indexing and timestamp labels advance in one-second steps. `question_time` specifies the question's insertion round. With `global_question=True`, the question is included from the first frame and retained when the history window slides. Replace `video_path` and `question` for your own task. | |
| ## Streaming responses | |
| | Token | Meaning | | |
| | --- | --- | | |
| | `</Silence>` | No relevant evidence yet. | | |
| | `</Standby>` | Relevant activity is ongoing; more evidence is needed. | | |
| | `</Response>` | Enough evidence is available; an answer follows. | | |
| The script prints each round and appends one record per video run to `test_sample_video.jsonl`, containing `video_path`, `target_fps`, and an `output` mapping from round names to responses. Repeated runs append to the same file. | |
| ## Usage notes | |
| - The provided demo reads a local video file sequentially. A live camera application needs its own capture loop. | |
| - Context and GPU memory usage grow with retained frames and response history. Reduce `max_rounds` if necessary; a frame-count window alone does not guarantee that the prompt fits the backend's token limit. | |
| - The upstream vLLM configuration uses `max_model_len=32768`. Treat this as an inference setting, not a measured long-video capability of this checkpoint. | |
| - Sampling at 1 FPS can miss brief events. Model responses and timing may be inaccurate. See the [paper](https://arxiv.org/abs/2512.21334) for the evaluation protocol and reported results. | |
| ## License | |
| The [Streamo source code](https://github.com/maifoundations/Streamo/blob/main/LICENSE) is licensed under Apache-2.0. The base model has its own [Qwen Research License](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct/blob/main/LICENSE). | |
| ## Citation | |
| ```bibtex | |
| @inproceedings{xia2026streaming, | |
| title={Streaming video instruction tuning}, | |
| author={Xia, Jiaer and Chen, Peixian and Zhang, Mengdan and Sun, Xing and Zhou, Kaiyang}, | |
| booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, | |
| pages={31219--31229}, | |
| year={2026} | |
| } | |
| ``` |