README.md

---
library_name: transformers
language:
- en
tags:
- lessvit
- hyperspectral
- remote-sensing
- geospatial
- vision-transformer
- custom_code
arxiv: 2605.18541
---

# LESSViT-Base

Pretrained **LESSViT** encoder for hyperspectral representation learning, exported
from the `LESSVIT_b2_d8_r1` checkpoint. The root of this repository contains an
encoder-only Hugging Face model, with standalone implementation files and a
working `AutoModel` interface. No project checkout is required.

- [Paper](https://arxiv.org/abs/2605.18541)
- [Project website and updated manuscript](https://uiuctml.github.io/LESSViT/)
- [Source repository](https://github.com/uiuctml/LESSViT)

## Intended Use and Limitations

**This checkpoint was pretrained on a partial subset of EnMAP data, not the complete EnMAP corpus.** It is released primarily to investigate whether explicit, efficient spatial–spectral modeling can improve representation robustness under changes in spectral configuration, including band sampling, channel count, and simulated sensor responses.

**This release should not be interpreted as a fully optimized EnMAP foundation model or as evidence of universally superior downstream performance.** Its training coverage, pretraining scale, and optimization settings reflect the scope of an architectural study. The reported experiments support conclusions under the evaluated protocols; they do not establish optimal performance on the full EnMAP dataset, unseen geographic distributions, or arbitrary downstream tasks.

Performance depends on the target data, preprocessing, spectral configuration, and adaptation procedure. In particular, robustness to spectral changes does not imply the highest accuracy under every fixed spectral configuration. Comparisons should account for differences in pretraining data coverage, training budget, spatial resolution, and fine-tuning protocol.

Users should independently validate this checkpoint for their intended application. Broader pretraining and task-specific adaptation may be necessary, and no guarantee is made regarding accuracy, generalization, or fitness for operational use.

## Install

Reference environment: Python 3.11.

```bash
pip install torch==2.7.1 torchvision==0.22.1 transformers==4.53.3 \
  huggingface-hub==0.34.4 timm==1.0.18 numpy==2.2.6 safetensors
```

Choose matching PyTorch/Torchvision wheels for your hardware. This release uses
custom model code, so `trust_remote_code=True` is required. You can pin `revision`
to a repository commit SHA for reproducible loading.

## Load the encoder directly

```python
import torch
from transformers import AutoModel

encoder = AutoModel.from_pretrained(
    "GFM-Bench/LESSViT-Base",
    trust_remote_code=True,
).eval()

# Synthetic example: [batch, spectral channels, height, width].
x = torch.randn(1, 10, 32, 32)
wavelengths_nm = torch.linspace(450, 2300, 10)[None, :]
# Fixed full-EnMAP reference bounds for this example. Keep reference bounds
# consistent across your downstream training and evaluation configurations.
wavelengths = (wavelengths_nm - 418.24) / (2445.53 - 418.24)

with torch.no_grad():
    output = encoder(
        optical=x,
        optical_channel_wv=wavelengths,
        spatial_resolution=30,  # meters per pixel
    )

features = output.last_hidden_state
print(features.shape)                  # [1, 11, 5, 768]
global_features = features[:, 0, 0]     # [1, 768], classification
spatial_features = features[:, 0, 1:]   # [1, 4, 768], dense prediction
band_patch_features = features[:, 1:, 1:]  # [1, 10, 4, 768]
```

`load_encoder.py` contains the same runnable example:

```bash
python load_encoder.py --model-id GFM-Bench/LESSViT-Base --device cpu
```

To run offline after downloading a complete snapshot, pass its directory as
`--model-id`. No network or original repository imports are needed for a local
snapshot.

### Inputs and outputs

- `optical`: float tensor `[B, C, H, W]`; height and width should be multiples of 16.
- `optical_channel_wv`: normalized wavelengths `[1, C]` or `[B, C]`, matching input
  channel order. Use fixed reference bounds for the task, not new min/max bounds
  for every spectral subset. The numerical bounds above are an EnMAP example,
  not a universal normalization rule for every sensor.
- `spatial_resolution`: spatial resolution in meters per pixel.
- `last_hidden_state`: `[B, C+1, N+1, 768]`, where
  `N = (H / 16) * (W / 16)`. Index 0 contains aggregation tokens.

For real data, use dataset/sensor-specific intensity normalization and matching
wavelength metadata. The source repository includes dataset statistics,
transforms, SRF resampling, and collators. Random values in the example only
verify that loading and inference work.

## Attach a downstream head

```python
# Use `encoder` and example tensors from above.
encoder.train()
head = torch.nn.Linear(encoder.config.embed_dim, 14)
features = encoder(
    optical=x, optical_channel_wv=wavelengths, spatial_resolution=30
).last_hidden_state[:, 0, 0]
logits = head(features)
loss = torch.nn.functional.cross_entropy(logits, torch.zeros(1, dtype=torch.long))
loss.backward()  # gradients through both encoder and head
```

For multi-label classification use binary cross entropy with logits. For dense
prediction, reshape `spatial_features` into the patch grid and attach a spatial
decoder. Task-specific heads are not included and must be trained. MAE masking
and random channel dropout are disabled in this encoder interface, including
when calling `.train()`; ordinary configured dropout remains active in training.

## Architecture

| Setting | Value |
|---|---|
| Encoder depth | 12 |
| Embedding dimension | 768 |
| Attention heads | 12 |
| Patch size | 16 |
| Spectral dimension per head | 2 |
| Attention rank | 1 |
| Positional encoding | Spatial–spectral RoPE |
| LayerScale initialization | 1.0 |

`d8` in the checkpoint name refers to the **pretraining decoder**, not the
encoder. The encoder-only release has approximately 83.2M parameters.

## Files and provenance

- Root `model.safetensors`: extracted encoder, with legacy attention keys
  remapped to the current implementation. No encoder tensors were omitted.
- Root `config.json`: explicit encoder architecture and Hugging Face auto-map.
- `pretraining/model.safetensors`: the original full checkpoint, copied
  byte-for-byte, including the MAE decoder. It uses the original checkpoint key
  layout and is **not** the root `AutoModel` format.
- `pretraining/architecture.json`: architectural fields from the accompanying
  training config. Machine-specific paths and training service settings are
  omitted. This is provenance metadata, not an AutoModel config.
- `validation.json`: export verification results.
- `SHA256SUMS`: file hashes.

Encoder implementation is packaged from LESSViT source commit
`20fb37c10541f1a29eb2a53999afdc0734499bc3`, with a small Hugging Face wrapper to
preserve architecture settings across config serialization and disable MAE
masking for encoder use. Loading checks compare every exported encoder tensor
exactly to the supplied checkpoint after key remapping. Local AutoModel loading,
forward-output equivalence, and encoder backward propagation were tested on CPU.
These checks do not independently reproduce the paper's benchmark scores or
establish the checkpoint's complete training-data provenance.

## Citation

```bibtex
@misc{si2026lessvitrobusthyperspectralrepresentation,
  title={LESSViT: Robust Hyperspectral Representation Learning under Spectral Configuration Shift},
  author={Haozhe Si and Yuxuan Wan and Yuqing Wang and Minh N. Do and Han Zhao},
  year={2026},
  eprint={2605.18541},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2605.18541}
}
```