gsumbul/SMARTIES-v1-finetuned-models

Spectrum-Aware Multi-Sensor Auto-Encoder for Remote Sensing Images

🚀 Introduction

From optical sensors to microwave radars, leveraging the complementary strengths of remote sensing (RS) sensors is crucial for achieving dense spatio-temporal monitoring of our planet, but recent foundation models (FMs) are often specific to single sensors or to fixed combinations.

SMARTIES is a generic and versatile FM lifting sensor-dependent efforts and enabling scalability and generalization to diverse RS sensors: SMARTIES projects data from heterogeneous sensors into a shared spectrum-aware space, enabling the use of arbitrary combinations of bands both for training and inference. To obtain sensor-agnostic representations, SMARTIES was trained as a single, unified transformer model reconstructing masked multi-sensor data with cross-sensor token mixup, while modulating its feature representations to accept diverse sensors as input.

✨ Key Features

• 🛰️ Multi-Sensor Representations: SMARTIES enables sensor-agnostic processing of Earth observation data, including optical (e.g., Sentinel-2), radar (e.g., Sentinel-1), and sub-meter resolution RGB (e.g., Maxar) imagery and unseen ones in a zero-shot manner.
• 🌈 Spectrum-Aware Projections: SMARTIES projects data from heterogeneous sensors into a shared spectrum-aware space: given a specific sensor, each one of its bands is projected by projection layers specific to wavelength ranges.
• ⚡ Lightweight and Scalable: SMARTIES is designed to be lightweight and scalable, making it suitable for a wide range of remote sensing applications.
• 🔀 Flexible Band Combinations: SMARTIES can handle arbitrary combinations of spectral bands from different sensors, enabling flexible remote sensing applications.
• 🔄 Downstream Transfer: SMARTIES enables downstream transfer using a unified model across a diverse set of sensors and tasks, including scene classification, semantic segmentation, and multi-label classification.

ℹ️ Note:

This repository contains the weights of SMARTIES linear-probed/finetuned models for the following datasets/experiments
For usage instructions, dataset details, and full documentation, please visit the SMARTIES GitHub page. The details of SMARTIES are described in our paper, available on arXiv.

📋 Evaluation Table

Dataset	Task	Evaluation Type	Sensor(s) Used	Finetuned Model Weights
BigEarthNetS2	Multi-label Classification	Fine-tuning	Sentinel-2	ViT-B ViT-L
BigEarthNetS1	Multi-label Classification	Linear Probing	Sentinel-1	ViT-B ViT-L
BigEarthNetMM	Multi-Modal Multi-label Classification	Linear Probing	Sentinel-1, Sentinel-2	ViT-B ViT-L
EuroSAT	Scene Classification	Fine-tuning	Sentinel-2	ViT-B ViT-L
EuroSAT	Scene Classification	Linear Probing	Sentinel-2	ViT-B ViT-L
RESISC-45	Scene Classification	Fine-tuning	RGB	ViT-L
DFC2020	Multi-Modal Semantic Segmentation	Non-linear Probing	Sentinel-1, Sentinel-2	ViT-B

📣 Attribution

If you use SMARTIES, please cite the paper:

@article{smarties,
  title={{SMARTIES}: Spectrum-Aware Multi-Sensor Auto-Encoder for Remote Sensing Images},
  author={Gencer Sumbul and Chang Xu and Emanuele Dalsasso and Devis Tuia},
  journal={arXiv preprint arXiv:2506.19585},
  year={2025}
}

📄 License

This repository is released under the Apache v2 License.

🙏 Acknowledgements

SMARTIES is supported by the European Space Agency (ESA) through the Discovery and Preparation Program, and is part of the project Toward a Foundation Model for Multi-Sensor Earth Observation Data with Language Semantics.