license: apache-2.0
pipeline_tag: image-classification
Dyna-DINO: Efficient ViT Distillation Via Adaptive Representation Anchoring
This repository contains the ViT-Tiny and ViT-S checkpoints (No Register) distilled from ViT-G DINOv2 on ImageNet-100 and ImageNet-1K. The knowledge distillation process follows the procedure proposed in the paper "Dyna-DINO: Efficient ViT Distillation Via Adaptive Representation Anchoring".
Introduction
Vision Foundation Models (VFMs) with ViT backbones, such as DINOv2, are computationally demanding. LEAP (Layer-skipping Efficiency via Adaptive Progression) is a training curriculum for ViT feature-based knowledge distillation. Instead of supervising the student against a fixed teacher block, LEAP advances the supervisory target through the teacher's feature maps (shallow-to-deep) based on online CKA alignment. This allows the student to build a foundational representation before tackling higher-level abstractions.
Use cases
The ViT models output feature maps that can be used for a variety of downstream tasks, including:
- Image Classification
- Instance Retrieval
- Semantic Segmentation
Performance
ImageNet-100:
ImageNet-1K:
Citation
@article{DynaDINO,
title={Dyna-DINO: Efficient ViT Distillation Via Adaptive Representation Anchoring},
author={Zhang, Jiaqi and Lee, Ashton and Wong, Anthony and Zou, John and BuGhanem, Sami and Balestriero, Randall},
journal={arXiv preprint arXiv:2606.19483},
year={2026}
}