Vision Transformer (ViT)

Category: ViT · Code: timm/models/vision_transformer.py · Architectures: 157 · Pretrained weights: 302 (96 with ImageNet-1k results)

The Vision Transformer (ViT) splits an image into fixed size patches, linearly embeds them and processes the resulting token sequence with a standard Transformer encoder. This module holds the VisionTransformer class and the weights for the original ViT, the “How to train your ViT?” (AugReg) and FlexiViT models, and many image-text (CLIP, SigLIP) and self-supervised pretrained image towers that share the architecture.

Papers

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers
• FlexiViT: One Model for All Patch Sizes
• Scaling Vision Transformers
• Three things everyone should know about Vision Transformers
• Scaling Vision Transformers to 22 Billion Parameters
• Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design
• Vision Transformers Need Registers
• Differential Transformer
• When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations
• ImageNet-21K Pretraining for the Masses
• Emerging Properties in Self-Supervised Vision Transformers
• DINOv2: Learning Robust Visual Features without Supervision
• Masked Autoencoders Are Scalable Vision Learners
• Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
• EVA: Exploring the Limits of Masked Visual Representation Learning at Scale
• Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
• Multimodal Autoregressive Pre-training of Large Vision Encoders
• Learning Transferable Visual Models From Natural Language Supervision
• Reproducible scaling laws for contrastive language-image learning
• DataComp: In search of the next generation of multimodal datasets
• Data Filtering Networks
• Demystifying CLIP Data
• Altogether: Image Captioning via Re-aligning Alt-text
• Meta CLIP 2: A Worldwide Scaling Recipe
• TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance
• MobileCLIP2: Improving Multi-Modal Reinforced Training
• Sigmoid Loss for Language Image Pre-Training
• SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
• PaliGemma: A versatile 3B VLM for transfer
• PaliGemma 2: A Family of Versatile VLMs for Transfer
• TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment
• Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performance
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling

Usage

See the Quickstart for loading pretrained weights, image preprocessing and fine-tuning, and Feature Extraction for using the models as backbones.

>>> import timm
>>> model = timm.create_model('vit_base_patch16_224.augreg2_in21k_ft_in1k', pretrained=True)
>>> data_config = timm.data.resolve_model_data_config(model)

Model API support

✓ supported, ✓ (square) works for square sizes only, ✗ not supported, – not needed (the model already accepts any input size). Checked by running each feature on a randomly initialized model (VisionTransformer with vit_tiny_patch16_224 at 224), other architectures of the same class are expected to behave the same. See the Model API page for details.

Architectures

GMACs and parameter counts are for the default input size and classifier of each architecture, from the benchmark results.

Pretrained weights

Top-1 is ImageNet-1k validation accuracy from results-imagenet.csv at the listed eval size, - if the weight has no ImageNet-1k result (e.g. pretraining or image-text weights). Classes is the size of the pretrained classifier, none for weights without one.

Weights are grouped by source: the paper or origin given in the pretrained config, else a Hub organization other than timm, else the first part of the pretrained tag (architecture.tag). Sources with a single weight are listed under other.

v2 tags

webli tags

sbb tags

augreg tags

laion2b tags

openai tags

metaclip tags

apple tags

pali tags

lvd142m tags

orig tags

pali2 tags

sbb2 tags

laion400m tags

metaclip2 tags

datacompxl tags

dfn2b tags

dfn5b tags

dino tags

indomain tags

sw tags

tinyclip tags

mae tags

augreg2 tags

ogvl tags

sam tags

Other