Streamo_3B / README.md
JiaerX's picture
Update README.md
2a26e7c verified
|
Raw History Blame Contribute Delete
5.82 kB
---
language:
- en
base_model:
- Qwen/Qwen2.5-VL-3B-Instruct
base_model_relation: finetune
pipeline_tag: video-text-to-text
library_name: transformers
tags:
- streamo
- streaming-video
- video-understanding
- multimodal
- qwen2_5_vl
- arxiv:2512.21334
datasets:
- maifoundations/Streamo-Instruct-465K
---
# Streamo-3B
**Streamo-3B** is a streaming video language model based on [Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct), developed for **Streaming Video Instruction Tuning**. It processes incoming video frames with conversational context and decides when to remain silent, wait for more evidence, or respond.
[Paper](https://arxiv.org/abs/2512.21334) 路 [Project page](https://jiaerxia.github.io/Streamo/) 路 [Code](https://github.com/maifoundations/Streamo) 路 [Streamo-Instruct-465K](https://huggingface.co/datasets/maifoundations/Streamo-Instruct-465K)
Streamo targets streaming tasks such as event narration, action understanding, temporal grounding, and time-sensitive question answering. This repository contains the model weights, tokenizer, and processor files. The inference implementation is maintained in the [Streamo code repository](https://github.com/maifoundations/Streamo).
## Model details
| Item | Value |
| --- | --- |
| Base model | Qwen/Qwen2.5-VL-3B-Instruct |
| Architecture | `Qwen2_5_VLForConditionalGeneration` |
| Checkpoint precision | BF16 |
| Weight format | Safetensors, two shards |
| Inputs | Video frames, timestamps, and text instructions |
| Outputs | Streaming state tokens and text responses |
Load the tokenizer and processor from **this checkpoint** to retain the additional streaming tokens.
## Quick start
### 1. Install the dependencies
Use a Linux CUDA environment with compatible [PyTorch and torchvision](https://pytorch.org/get-started/locally/) installations. Run the following commands in a dedicated Python environment:
```bash
git clone https://github.com/maifoundations/Streamo.git
cd Streamo
python -m pip install -r requirements.txt \
"transformers==4.57.1" \
"qwen-vl-utils>=0.0.6,<0.0.12" \
decord opencv-python huggingface_hub
```
The checkpoint configuration records Transformers `4.57.1`, which is within the repository's supported version range. The bundled [Qwen model loader](https://github.com/maifoundations/Streamo/blob/main/swift/llm/model/model/qwen.py) requires `qwen-vl-utils<0.0.12`. Run inference from the cloned repository so that Python imports its bundled `swift` implementation.
### 2. Download the checkpoint
```bash
# Authenticate first if repository access requires it.
hf auth login
hf download JiaerX/Streamo_3B --local-dir ./checkpoints/Streamo_3B
```
Download the complete repository, including both weight shards and the tokenizer/processor files. See the [Hugging Face CLI documentation](https://huggingface.co/docs/huggingface_hub/guides/cli) for authentication and download options.
### 3. Configure and run inference
Edit the existing variables in the `__main__` block of [`inference.py`](https://github.com/maifoundations/Streamo/blob/main/inference.py):
```python
infer_backend = 'pt'
model = './checkpoints/Streamo_3B'
video_path = './demo/cook.mp4'
target_fps = 1.0
question = 'Detect and summarize each event sequence in the video.'
global_question = True
question_time = 0
max_rounds = 300
```
Then run:
```bash
python inference.py
```
The script uses configuration variables, not command-line flags. This example selects its PyTorch backend; the upstream default is `vllm`, which requires a separate vLLM installation compatible with your PyTorch, CUDA, and Transformers versions. GPU selection is set inside the script with `CUDA_VISIBLE_DEVICES='0'`; edit it there if needed.
Keep `target_fps=1.0`: the current frame indexing and timestamp labels advance in one-second steps. `question_time` specifies the question's insertion round. With `global_question=True`, the question is included from the first frame and retained when the history window slides. Replace `video_path` and `question` for your own task.
## Streaming responses
| Token | Meaning |
| --- | --- |
| `</Silence>` | No relevant evidence yet. |
| `</Standby>` | Relevant activity is ongoing; more evidence is needed. |
| `</Response>` | Enough evidence is available; an answer follows. |
The script prints each round and appends one record per video run to `test_sample_video.jsonl`, containing `video_path`, `target_fps`, and an `output` mapping from round names to responses. Repeated runs append to the same file.
## Usage notes
- The provided demo reads a local video file sequentially. A live camera application needs its own capture loop.
- Context and GPU memory usage grow with retained frames and response history. Reduce `max_rounds` if necessary; a frame-count window alone does not guarantee that the prompt fits the backend's token limit.
- The upstream vLLM configuration uses `max_model_len=32768`. Treat this as an inference setting, not a measured long-video capability of this checkpoint.
- Sampling at 1 FPS can miss brief events. Model responses and timing may be inaccurate. See the [paper](https://arxiv.org/abs/2512.21334) for the evaluation protocol and reported results.
## License
The [Streamo source code](https://github.com/maifoundations/Streamo/blob/main/LICENSE) is licensed under Apache-2.0. The base model has its own [Qwen Research License](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct/blob/main/LICENSE).
## Citation
```bibtex
@inproceedings{xia2026streaming,
title={Streaming video instruction tuning},
author={Xia, Jiaer and Chen, Peixian and Zhang, Mengdan and Sun, Xing and Zhou, Kaiyang},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={31219--31229},
year={2026}
}
```