PanoVideoReasoner / README.md
lulidong's picture
Upload PanoVideoReasoner model weights and model card
d0feb80 verified
|
Raw History Blame Contribute Delete
2.7 kB
---
language:
- en
license: apache-2.0
library_name: transformers
pipeline_tag: image-text-to-text
base_model: Qwen/Qwen3.5-9B
tags:
- panoramic-video
- video-question-answering
- multimodal
- reasoning
- qwen3_5
---
# PanoVideoReasoner
![Think in 360° overview](assets/teaser.webp)
PanoVideoReasoner is a panoramic video reasoning model introduced in **Think in 360°: Benchmarking and Bootstrapping Panoramic Video Reasoning**. It builds on Qwen3.5-9B and learns to connect observations across viewing directions and time.
The model is trained on PanoVideoCoT through Panoramic CoT SFT followed by Evidence-Privileged On-Policy Self-Distillation (EP-OPSD). PanoVideoCoT contains 13,465 question–answer–CoT pairs from 2,063 panoramic videos.
## Quick start
Install a recent Transformers version with Qwen3.5 support, PyTorch, and the video-loading dependencies:
```bash
pip install -U "transformers>=5.15.0" accelerate torch torchvision pillow av
```
```python
import torch
from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration
model_path = "./PanoVideoReasoner"
processor = AutoProcessor.from_pretrained(model_path)
model = Qwen3_5ForConditionalGeneration.from_pretrained(
model_path,
dtype=torch.bfloat16,
device_map="auto",
)
messages = [
{
"role": "user",
"content": [
{"type": "video", "path": "./panoramic_video.mp4"},
{
"type": "text",
"text": (
"This is a 360-degree panoramic video in equirectangular "
"projection (ERP). Describe how the people move through "
"the scene, following them across viewing directions."
),
},
],
}
]
inputs = processor.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_dict=True,
return_tensors="pt",
fps=1,
max_frames=128,
).to(model.device)
with torch.inference_mode():
outputs = model.generate(
**inputs,
max_new_tokens=8192,
do_sample=True,
temperature=1.0,
top_p=0.95,
top_k=20,
)
answer = outputs[:, inputs["input_ids"].shape[1]:]
print(processor.batch_decode(answer, skip_special_tokens=True)[0])
```
The example shows how to load the model and ask a video question. For the input protocol used in the paper, uniformly sample the complete ERP video at a target rate of 1 FPS, with at most 128 frames, and resize the long edge to 1,280 pixels. The training input budgets are 409,600 pixels per frame and 52,428,800 pixels per video. Keep the original ERP representation and its complete viewing sphere.