yuji-nozawa/ReFoCo

Training Vision Transformers to Focus: Emergence of Spatial Attention Head Specialization

Official pretrained models for the paper:
Training Vision Transformers to Focus: Emergence of Spatial Attention Head Specialization

Pretrained Models

These pretrained models can be used for the code provided in our GitHub repository.

Model	Download
ViT-B/16-Reg	model_vit_focus_base.pth
ViT-L/16-Reg	model_vit_focus_large.pth
ViT-g/16-Reg	model_vit_focus_giant.pth

Citation

If you find this work useful, please cite:

@inproceedings{nozawa2026focus,
    title     = {Training Vision Transformers to Focus: Emergence of Spatial Attention Head Specialization},
    author    = {Nozawa, Yuji and Lin, Yu-Chieh and Ng, Youyang},
    booktitle = {The Fortieth Annual Conference on Neural Information Processing Systems},
    year      = {2026}
}

License

The pretrained models are released under the Apache-2.0 License.

Support

While we welcome feedback, we provide no formal support for this project. Maintenance (including bug fixes) is not guaranteed. Please understand that we may not be able to review or accept all Pull Requests. Thank you for your understanding.