File size: 1,393 Bytes
eb86059 c945045 eb86059 c945045 eb86059 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | ---
language: en
license: apache-2.0
tags:
- self-supervised-learning
- knowledge-distillation
- vision-transformer
- model-compression
---
# TinySSL: Distilling Foundation Model Features for Resource-Efficient Vision
**Authors**: Emran Abdu
**DOI**: [10.5281/zenodo.21180996](https://zenodo.org/record/21180996)
**Code**: [GitHub](https://github.com/Emran-goat/tinyssl)
**License**: Apache 2.0
## Abstract
Vision foundation models like DINOv2 produce powerful representations, but training them costs millions of dollars in GPU compute. We introduce TinySSL, a 2.8M-parameter framework that distills frozen DINOv2-S/14 features into a compact CNN-transformer hybrid. A composite loss combines masked image modeling with JEPA alignment, cosine feature matching, and KoLeo uniformity regularization, removing the need for negative pairs, momentum encoders, or large batches. A progressive augmentation curriculum stabilizes training on commodity hardware. Across four domain benchmarks (Flowers102, Oxford Pets, EuroSAT, BreastMNIST), TinySSL retains over 97% of DINOv2-S/14 linear-probe accuracy with a 7x parameter reduction and trains in under 30 minutes on a single CPU.
## Citation
`ibtex
@article{abdu2026tinyssl,
title={TinySSL: Distilling Foundation Model Features for Resource-Efficient Vision},
author={Emran Abdu},
year={2026},
doi={10.5281/zenodo.21180996}
}
`
|