SteerViT: Steerable Visual Representations [ECCV'26 Oral]

Paper | Project Page | GitHub | Demo (HF Spaces)

SteerViT equips pretrained Vision Transformers with steerable visual representations. Given an image and a natural-language prompt, it conditions the visual encoder through lightweight gated cross-attention to produce:

• prompt-aware global embeddings
• prompt-aware dense patch features
• prompt-conditioned heatmaps

This Hugging Face repository hosts the model checkpoints.

Installation

To use SteerViT, install the library directly from GitHub:

python -m pip install "git+https://github.com/JonaRuthardt/SteerViT.git"

Quick start

import torch
from PIL import Image
from steervit import SteerViT

device = "cuda" if torch.cuda.is_available() else "cpu"

# Load the model (e.g., SteerDINOv2-Base)
model = SteerViT.from_pretrained("steervit_dinov2_base.pth", device=device)
transform = model.get_transforms()

image = Image.open("path/to/image.jpg").convert("RGB")
image_tensor = transform(image).unsqueeze(0)

prompt = ["the red car"]

global_features = model.get_global_features(image_tensor, texts=prompt) # pooled image embeddings
dense_features = model.get_dense_features(image_tensor, texts=prompt) # patch-level visual features
heatmaps = model.get_heatmaps(image_tensor, texts=prompt) # prompt-conditioned localization heatmaps
attention_heatmaps = model.get_attention_heatmaps(image_tensor, texts=prompt) # attention-based heatmaps

If texts=None, SteerViT behaves like the underlying frozen ViT backbone and returns query-agnostic features.

Available checkpoints

🔔 New commercially usable checkpoints: We now provide updated SteerViT checkpoints under Apache-2.0, which permits commercial use. These variants were adapted using only Visual Genome, COCO, and LVIS, with commercial-use image filtering applied. Thanks to improved synthetic annotations, these variants match or improve upon the original research-only checkpoints.

Checkpoint	from_pretrained(...) identifier	Notes	Licence
SteerDINOv2-Base	steervit_dinov2_base.pth	Primary model used for most experiments	research-only
SteerDINOv2-Small (commercial)	steervit_dinov2_commercial_small.pth	Variant trained on commercially-permissable data	commercial
SteerDINOv2-Base (commercial)	steervit_dinov2_commercial_base.pth	Variant trained on commercially-permissable data	commercial
SteerDINOv2-Large (commercial)	steervit_dinov2_commercial_large.pth	Variant trained on commercially-permissable data	commercial
SteerMAE-Base	steervit_mae_base.pth	Alternative model based on MAE-backbone	research-only
SteerSigLIP-Base	steervit_siglip_base.pth	Alternative model based on SigLIP-backbone	research-only
SteerSigLIP2-Base	steervit_siglip2_base.pth	Alternative model based on SigLIP2-backbone	research-only

Citation

@inproceedings{ruthardt2026steervit,
  title     = {Steerable Visual Representations},
  author    = {Ruthardt, Jona and Gaur, Manu and Ramanan, Deva and Tapaswi, Makarand and Asano, Yuki M.},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}