Instructions to use ZhuYuhan/VideoJev-2B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ZhuYuhan/VideoJev-2B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="ZhuYuhan/VideoJev-2B", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("ZhuYuhan/VideoJev-2B", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
VideoJev-2B
VideoJev-2B adapts Qwen3.5-2B to make structured video decisions: localizing events in time and selecting answers to multiple-choice questions. It scores candidates with a shared decision head instead of generating answer tokens.
Paper · Homepage · Code · Usage guide
Benchmark Results
All values are percentages. Grounding uses mIoU; MLVU uses M-Avg; the other QA benchmarks use accuracy. Both columns use this checkpoint.
| Benchmark | Metric | Cached | Joint |
|---|---|---|---|
| Charades-STA | mIoU | 55.05 | 55.71 |
| ActivityNet Captions | mIoU | 57.30 | 58.11 |
| QVHighlights | mIoU | 69.52 | 69.56 |
| LVBench | accuracy | 62.43 | 63.20 |
| Video-MME | accuracy | 68.33 | 68.22 |
| LongVideoBench | accuracy | 68.59 | 68.44 |
| MLVU | M-Avg | 73.24 | 74.16 |
Quick start
Use Python 3.12, PyTorch 2.8 with CUDA 12.8, and the following packages. Install the PyTorch build for your CUDA environment first.
pip install transformers==5.14.1 accelerate safetensors numpy pillow av==18.1.0 huggingface_hub \
flash-linear-attention==0.4.2 fla-core==0.4.2
pip install flash-attn causal-conv1d --no-build-isolation
The repository contains the complete visual encoder, language backbone, decision head, tokenizer, and processor. Weights follow the corresponding Qwen tensor dtypes; the decision head is stored in FP32. CUDA inference uses BF16 autocast and FP32 decision scoring. The example uses Joint inference. While this repository is private, authenticate with an account that has access before downloading.
import json
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModel, AutoProcessor
repo = "ZhuYuhan/VideoJev-2B"
video = hf_hub_download(
repo_id="ZhuYuhan/VideoJev-Demo",
filename="eating_spaghetti.mp4",
repo_type="dataset",
revision="7e2b9685bca592c3b929268f6fcf82c4ad74b267",
)
processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(
repo, trust_remote_code=True, dtype=torch.bfloat16,
device_map={"": "cuda:0"}, attn_implementation="flash_attention_2",
)
model.prepare_for_inference()
questions = [
{"id": "event", "task": "grounding",
"query": "The woman in green eats pasta."},
{"id": "question", "task": "choice",
"query": "What is the woman in green eating?",
"options": ["A. Pasta.", "B. Rice.", "C. Soup."]},
]
results = model.predict_video(
processor, video, questions, fps=2.0, max_frames=2048, context=8192,
)
print(json.dumps(results, indent=2))
The short demo uses 2 FPS and an 8K context. Use the task-specific settings above for benchmark evaluation. Long videos require more memory than this short example.
Output
Each question returns its id, task, and candidate probabilities. Grounding also returns an interval in seconds. QA returns a zero-based answer_index and the selected answer. Grounding scores are independent sigmoid probabilities; answer-choice scores use softmax.
See USAGE.md for input settings and evaluation_results.json for exact scores and sample counts.
Citation
@article{zhu2026videojev,
title={VideoJev: System One Model for Video Understanding and Interaction},
author={Zhu, Yuhan and Zeng, Xiangyu and Si, Qingyi and Yang, Chenxu and Li, Yifei and Qin, Luozheng and Zhou, Junhao and Yao, Dingyu and Zhang, Jun and Qin, Chuanyu and others},
journal={arXiv preprint arXiv:2610.18615},
year={2026}
}
- Downloads last month
- 72