Fix usage instructions: correct model class / plugin guidance, English-only

ec11adf verified 3 months ago

1.41 kB

base_model: Qwen/Qwen3-VL-4B-Instruct
license: apache-2.0
library_name: transformers
pipeline_tag: video-text-to-text
tags:
  - camera-movement
  - video-understanding
  - qwen3-vl
  - vggt-injection

CamInject-4B

Camera-movement understanding model that injects frozen VGGT camera tokens into Qwen/Qwen3-VL-4B-Instruct. Given a video, it outputs structured JSON describing every camera-movement segment.

Paper: Temporally Grounded Compositional Camera Motion Understanding via Geometric Knowledge Distillation
Project page: https://ddz16.github.io/cammotion.github.io
Code: https://github.com/ddz16/CamDistill

⚠️ This model cannot be loaded with plain 🤗 Transformers. It requires a custom model type (registered via a plugin) and runs VGGT online to produce camera tokens. Loading it as a standard Qwen3VLForConditionalGeneration would not work correctly. Use the CamDistill repo.

Usage

Clone the CamDistill repo and clone VGGT-Omega (set VGGT_OMEGA_REPO, see the repo's setup). CamInject runs VGGT online during inference:

VGGT_TEACHER_TYPE=vggt_omega \
python camera_movement_sft/infer_single.py \
    --model ddz16/CamInject-4B \
    --video /path/to/video.mp4 \
    --variant caminject

See the repo's README for environment setup and batch evaluation.