Streamo_3B / README.md
JiaerX's picture
Update README.md
2a26e7c verified
|
Raw History Blame Contribute Delete
5.82 kB
metadata
language:
  - en
base_model:
  - Qwen/Qwen2.5-VL-3B-Instruct
base_model_relation: finetune
pipeline_tag: video-text-to-text
library_name: transformers
tags:
  - streamo
  - streaming-video
  - video-understanding
  - multimodal
  - qwen2_5_vl
  - arxiv:2512.21334
datasets:
  - maifoundations/Streamo-Instruct-465K

Streamo-3B

Streamo-3B is a streaming video language model based on Qwen2.5-VL-3B-Instruct, developed for Streaming Video Instruction Tuning. It processes incoming video frames with conversational context and decides when to remain silent, wait for more evidence, or respond.

Paper 路 Project page 路 Code 路 Streamo-Instruct-465K

Streamo targets streaming tasks such as event narration, action understanding, temporal grounding, and time-sensitive question answering. This repository contains the model weights, tokenizer, and processor files. The inference implementation is maintained in the Streamo code repository.

Model details

Item Value
Base model Qwen/Qwen2.5-VL-3B-Instruct
Architecture Qwen2_5_VLForConditionalGeneration
Checkpoint precision BF16
Weight format Safetensors, two shards
Inputs Video frames, timestamps, and text instructions
Outputs Streaming state tokens and text responses

Load the tokenizer and processor from this checkpoint to retain the additional streaming tokens.

Quick start

1. Install the dependencies

Use a Linux CUDA environment with compatible PyTorch and torchvision installations. Run the following commands in a dedicated Python environment:

git clone https://github.com/maifoundations/Streamo.git
cd Streamo

python -m pip install -r requirements.txt \
  "transformers==4.57.1" \
  "qwen-vl-utils>=0.0.6,<0.0.12" \
  decord opencv-python huggingface_hub

The checkpoint configuration records Transformers 4.57.1, which is within the repository's supported version range. The bundled Qwen model loader requires qwen-vl-utils<0.0.12. Run inference from the cloned repository so that Python imports its bundled swift implementation.

2. Download the checkpoint

# Authenticate first if repository access requires it.
hf auth login

hf download JiaerX/Streamo_3B --local-dir ./checkpoints/Streamo_3B

Download the complete repository, including both weight shards and the tokenizer/processor files. See the Hugging Face CLI documentation for authentication and download options.

3. Configure and run inference

Edit the existing variables in the __main__ block of inference.py:

infer_backend = 'pt'
model = './checkpoints/Streamo_3B'
video_path = './demo/cook.mp4'

target_fps = 1.0
question = 'Detect and summarize each event sequence in the video.'
global_question = True
question_time = 0
max_rounds = 300

Then run:

python inference.py

The script uses configuration variables, not command-line flags. This example selects its PyTorch backend; the upstream default is vllm, which requires a separate vLLM installation compatible with your PyTorch, CUDA, and Transformers versions. GPU selection is set inside the script with CUDA_VISIBLE_DEVICES='0'; edit it there if needed.

Keep target_fps=1.0: the current frame indexing and timestamp labels advance in one-second steps. question_time specifies the question's insertion round. With global_question=True, the question is included from the first frame and retained when the history window slides. Replace video_path and question for your own task.

Streaming responses

Token Meaning
</Silence> No relevant evidence yet.
</Standby> Relevant activity is ongoing; more evidence is needed.
</Response> Enough evidence is available; an answer follows.

The script prints each round and appends one record per video run to test_sample_video.jsonl, containing video_path, target_fps, and an output mapping from round names to responses. Repeated runs append to the same file.

Usage notes

  • The provided demo reads a local video file sequentially. A live camera application needs its own capture loop.
  • Context and GPU memory usage grow with retained frames and response history. Reduce max_rounds if necessary; a frame-count window alone does not guarantee that the prompt fits the backend's token limit.
  • The upstream vLLM configuration uses max_model_len=32768. Treat this as an inference setting, not a measured long-video capability of this checkpoint.
  • Sampling at 1 FPS can miss brief events. Model responses and timing may be inaccurate. See the paper for the evaluation protocol and reported results.

License

The Streamo source code is licensed under Apache-2.0. The base model has its own Qwen Research License.

Citation

@inproceedings{xia2026streaming,
  title={Streaming video instruction tuning},
  author={Xia, Jiaer and Chen, Peixian and Zhang, Mengdan and Sun, Xing and Zhou, Kaiyang},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={31219--31229},
  year={2026}
}