| --- |
| license: apache-2.0 |
| language: |
| - en |
| base_model: |
| - Qwen/Qwen3-VL-4B-Instruct |
| pipeline_tag: video-text-to-text |
| tags: |
| - Transition Detection |
| - Video Transition |
| - Video Processing |
| --- |
| |
| <div style="display:flex;align-items:center;justify-content:space-between;gap:16px"><span style="font-size:2em;font-weight:600;line-height:1.25">TransVLM</span><picture style="flex:none"><source media="(prefers-color-scheme: dark)" srcset="https://huggingface.co/HeyGenAI/TransVLM-Qwen3-VL-4B-Instruct/resolve/main/assets/heygen-logo-white.png"><img src="https://huggingface.co/HeyGenAI/TransVLM-Qwen3-VL-4B-Instruct/resolve/main/assets/heygen-logo-black.png" alt="HeyGen Research" style="height:44px;display:block"></picture></div> |
|
|
| <p align="center"> |
| <a href="https://arxiv.org/abs/2604.27975"><img src="https://img.shields.io/badge/arXiv-2604.27975-b31b1b.svg" alt="arXiv"></a> |
| <a href="https://heygen-com.github.io/TransVLM/"><img src="https://img.shields.io/badge/Project-Page-1f6feb.svg" alt="Project Page"></a> |
| <a href="https://github.com/heygen-com/TransVLM"><img src="https://img.shields.io/badge/GitHub-heygen--com%2FTransVLM-181717.svg" alt="GitHub"></a> |
| <a href="https://www.apache.org/licenses/LICENSE-2.0"><img src="https://img.shields.io/badge/License-Apache%202.0-3da639.svg" alt="License"></a> |
| </p> |
|
|
| <p align="center"> |
| <b>A Vision-Language Framework and Benchmark for Detecting Any Shot Transitions</b> |
| </p> |
|
|
| <p align="center"> |
| <b>Accepted to ECCV 2026 (Poster)</b> |
| </p> |
|
|
| > [!IMPORTANT] |
| > **This is not a drop-in Qwen3-VL checkpoint.** Its vision tower takes **6 channels**: RGB |
| > concatenated with an optical-flow visualisation (`config.json` → `vision_config.in_channels: 6`). |
| > A standard `AutoProcessor` / `Qwen3VLForConditionalGeneration` pipeline supplies 3 channels and |
| > will not produce correct results. The weights load, but the input is wrong. |
| > |
| > Run it through the released inference code: **https://github.com/heygen-com/TransVLM** |
|
|
| ## Model Description |
|
|
| Traditional Shot Boundary Detection (SBD) looks for isolated cut *points*, which breaks down on |
| gradual transitions and frequently yields corrupted shots. **TransVLM** targets Shot Transition |
| Detection (STD) instead: it predicts the continuous temporal *segments* over which a transition |
| happens. |
|
|
| The model is Qwen3-VL-4B-Instruct with one architectural change. Its vision patch embedding is |
| widened from 3 to 6 input channels (zero-padded at initialisation), so an optical-flow |
| visualisation enters the model alongside RGB at the input stage. Motion is what separates a |
| dissolve from a camera pan, and a model that only sees appearance has to infer it. Because the |
| flow is fused *before* patchification, the language backbone carries no additional visual tokens, |
| so temporal awareness comes for free at the token budget. |
|
|
| Training used a scalable FFmpeg-based data engine covering 59 transition effects, which addresses |
| the severe class imbalance in public shot-boundary data. |
|
|
| ## Model Details |
|
|
| | | | |
| |---|---| |
| | Base model | [Qwen/Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct) | |
| | Architecture | `Qwen3VLForConditionalGeneration` | |
| | Precision | `bfloat16` | |
| | Vision input channels | **6** (RGB + optical-flow visualisation, concatenated on the channel axis) | |
| | Sampling frame rate | 25 fps | |
| | Optical flow | [NeuFlow v2](https://huggingface.co/Study-is-happy/neuflow-v2), computed at inference time. Its weights (~30 MB) come from the Hub on first use, so expect a second download | |
| | Input | One video; flow is computed for you | |
| | Output | Transition segments as start/end times in seconds | |
| | Saved with | `transformers` 4.57.3 | |
|
|
| ## Intended Use and Limitations |
|
|
| **Intended for** detecting shot transitions in videos, both hard cuts and gradual effects, and |
| as a baseline on the STD benchmark. |
|
|
| **Limitations** |
|
|
| - Not a frame-level cut-point classifier. The output is a time span per transition, not a single |
| boundary frame. |
| - Not a general video-chat model. It was fine-tuned on one task with one prompt; the prompt ships |
| with the inference code and changing it changes the task. |
| - Long videos should be split before inference. Cost is linear in duration, and the flow |
| visualisation is normalised over its whole input, so a long video and its parts are not the same |
| signal. |
|
|
| ## Usage |
|
|
| No standalone `transformers` snippet is given here on purpose: a 3-channel one would run and |
| return nonsense. Use the inference package, which needs Python 3.12, a CUDA GPU, and `ffmpeg` |
| on `PATH`: |
|
|
| ```bash |
| git clone https://github.com/heygen-com/TransVLM |
| cd TransVLM/inference |
| |
| uv venv && source .venv/bin/activate |
| uv sync --group cu130 --group dev # cu128 if your driver is older than 570 |
| |
| # Do not skip this. PyTorch 2.9.1 has a Conv3d bug below cuDNN 9.15, and the 6-channel |
| # patch embed IS a Conv3d, so it is on the hot path of every forward pass. `uv sync` |
| # resolves cuDNN back down, so re-run this after every sync. Activate the venv first: |
| # unlike `uv sync`, `uv pip` installs into whatever environment is active. |
| uv pip install nvidia-cudnn-cu13==9.16.0.29 # nvidia-cudnn-cu12 for cu128 / cu126 |
| python -c "import torch; print(torch.backends.cudnn.version())" # must print 91600 |
| |
| hf download HeyGenAI/TransVLM-Qwen3-VL-4B-Instruct --local-dir ./pretrained/TransVLM-v1 |
| |
| python infer_video.py \ |
| --video /path/to/video.mp4 \ |
| --ckpt-dir ./pretrained/TransVLM-v1 \ |
| --output-jsonl out.jsonl |
| ``` |
|
|
| The cuDNN package name tracks the CUDA flavour, and installing the wrong one is a silent |
| no-op: it lands an unused package while torch keeps loading the other family. That is why |
| the check prints a version number instead of trusting the install. |
|
|
| Each output line carries `segments` as `{start_time, end_time}` pairs in seconds on the original |
| video's timeline, plus timings and the full configuration that produced them. |
|
|
| Three backends are supported: HuggingFace (default), vLLM and SGLang. Environment setup, every |
| option and the output schema are documented in |
| [`inference/README.md`](https://github.com/heygen-com/TransVLM/blob/main/inference/README.md). |
|
|
| ## Evaluation |
|
|
| Qualitative results and the comparison against baselines are on the |
| [project page](https://heygen-com.github.io/TransVLM/). |
|
|
| The scorer that produced the paper's metric is released at |
| [`evaluation/`](https://github.com/heygen-com/TransVLM/tree/main/evaluation). Note that the |
| published figures cannot be reproduced bit-for-bit from the released artifacts alone. The |
| benchmark ground truth and the pre-computed optical flow used for that run are not part of this |
| release. The reasons are spelled out in |
| [`evaluation/README.md`](https://github.com/heygen-com/TransVLM/blob/main/evaluation/README.md). |
|
|
| ## Release Progress |
|
|
| - [x] Model weights |
| - [x] Inference code |
| - [x] Evaluation code |
| - [ ] Data engine code |
| - [ ] STD benchmark data |
| - [ ] Re-annotated dataset labels |
| - [ ] Leaderboard |
|
|
| 🚧 The remaining items are being prepared for release. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{chen2026transvlm, |
| title={TransVLM: A Vision-Language Framework and Benchmark for Detecting Any Shot Transitions}, |
| author={Chen, Ce and Ren, Yi and Li, Yuanming and Goriachko, Viktor and |
| Ye, Zhenhui and Guo, Zujin and Hong, Zhibin and Gong, Mingming}, |
| booktitle={European Conference on Computer Vision}, |
| year={2026}, |
| organization={Springer} |
| } |
| ``` |
|
|
| ## License |
|
|
| Apache License 2.0, inherited from [Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct). |
|
|
| ### Scope of this release |
|
|
| <sub>TransVLM was developed as academic research and is published at ECCV 2026. It is not used in |
| any HeyGen product; HeyGen's production system for this task is a different model, trained |
| separately from this one.</sub> |
|
|
| ### Training data |
|
|
| <sub>Training combined HeyGen's own generated and annotated footage, which accounts for the |
| majority of the sampling distribution, with four public shot-boundary datasets: |
| <a href="https://github.com/wentaozhu/AutoShot">AutoShot</a>, |
| <a href="https://github.com/Tangshitao/ClipShots">ClipShots</a>, |
| MovieShots2 (<a href="https://zweipa.github.io/TMM_SCTSNet/">SCTSNet</a>, whose cut annotations |
| build on the <a href="https://www.robots.ox.ac.uk/~vgg/data/condensed-movies/">Condensed Movies |
| Dataset</a>), and <a href="https://huggingface.co/datasets/MCG-NJU/SportsShot">SportsShot</a>. |
| Together these four are roughly 42% of the training sampling distribution, of which SportsShot is |
| about 7.5%.</sub> |
|
|
| <sub><b>SportsShot is licensed CC BY-NC 4.0</b> and is credited here accordingly. The other three |
| distribute annotations rather than footage, so consult each dataset's own terms before relying on |
| them. No dataset is redistributed by this repository, and no dataset footage is contained in these |
| weights.</sub> |
|
|
