| --- |
| license: mit |
| pipeline_tag: video-classification |
| tags: |
| - video |
| - action-recognition |
| - efficient-inference |
| - token-selection |
| library_name: pytorch |
| --- |
| |
| # LookWhen ViT-B/16 |
|
|
| Pre-trained checkpoint for **"LookWhen? Fast Video Recognition by Learning When, Where, |
| and What to Compute"** ([arXiv:2605.06809](https://arxiv.org/abs/2605.06809)). |
|
|
| LookWhen splits video recognition in two: a shallow **selector** scores every patch of a |
| 2x-downscaled (2x in each dimension) clip, and a deep **extractor** processes only the top-K of them while still |
| predicting features of the whole video. |
|
|
| This is the model used throughout the paper: ViT-B/16, pre-trained for 20 epochs on |
| Kinetics-400 and SSv2 by distilling InternVideo2's video token and DINOv3's frame and |
| patch tokens, with selection supervised by top1-distance token uniqueness. It was trained |
| over sparsity 70-95%, so it runs anywhere in that range without retraining. |
|
|
| ## Usage |
|
|
| ```bash |
| pip install git+https://github.com/alisalamatian1/LookWhen.git |
| ``` |
|
|
| ```python |
| import torch |
| from lookwhen import LookWhen |
| from lookwhen.data.video import load_clip |
| |
| model = LookWhen.from_pretrained().cuda().eval() # downloads this checkpoint |
| clip = load_clip("video.mp4", num_frames=16, img_size=224).unsqueeze(0).cuda() |
| |
| out = model(clip, keep_ratio=0.1) # keep_ratio = 1 - sparsity |
| out["video"] # (1, 768) video embedding |
| ``` |
|
|
| Code, training, and evaluation: https://github.com/alisalamatian1/LookWhen |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{salamatian2026lookwhen, |
| title = {LookWhen? Fast Video Recognition by Learning When, Where, and What to Compute}, |
| author = {Salamatian, Ali and Fuller, Anthony and Sarkar, Pritam and |
| Green, James R. and Sigal, Leonid and Shelhamer, Evan}, |
| journal = {arXiv preprint arXiv:2605.06809}, |
| year = {2026} |
| } |
| ``` |
|
|
