Use English Hugging Face model card
Browse files
README.md
CHANGED
|
@@ -1,7 +1,6 @@
|
|
| 1 |
---
|
| 2 |
frameworks: PyTorch
|
| 3 |
language:
|
| 4 |
-
- zh
|
| 5 |
- en
|
| 6 |
license: apache-2.0
|
| 7 |
tags:
|
|
@@ -21,176 +20,176 @@ datasets: []
|
|
| 21 |
</strong>
|
| 22 |
</p>
|
| 23 |
|
| 24 |
-
#
|
| 25 |
|
| 26 |
-
SpectralGPT
|
| 27 |
|
| 28 |
-
|
| 29 |
|
| 30 |
https://arxiv.org/abs/2311.07113
|
| 31 |
|
| 32 |
-
#
|
| 33 |
|
| 34 |
-
SpectralGPT
|
| 35 |
|
| 36 |
-
#
|
| 37 |
|
| 38 |
-
|
|
| 39 |
| :---: | :--- |
|
| 40 |
-
|
|
| 41 |
-
|
|
| 42 |
-
|
|
| 43 |
-
| Hugging Face / OneCode
|
| 44 |
-
|
|
| 45 |
|
| 46 |
-
#
|
| 47 |
|
| 48 |
-
## 1. OneCode
|
| 49 |
|
| 50 |
-
|
| 51 |
|
| 52 |
-
[
|
| 53 |
|
| 54 |
-
## 2.
|
| 55 |
|
| 56 |
-
**
|
| 57 |
|
| 58 |
-
-
|
| 59 |
-
- CPU
|
| 60 |
-
- DCU
|
| 61 |
|
| 62 |
-
###
|
| 63 |
|
| 64 |
```bash
|
| 65 |
hf download OneScience-Group/SpectralGPT --local-dir ./SpectralGPT
|
| 66 |
cd SpectralGPT
|
| 67 |
```
|
| 68 |
|
| 69 |
-
###
|
| 70 |
|
| 71 |
-
**DCU
|
| 72 |
|
| 73 |
```bash
|
| 74 |
-
#
|
| 75 |
conda create -n onescience311 python=3.11 -y
|
| 76 |
conda activate onescience311
|
| 77 |
-
#
|
| 78 |
pip install onescience[earth-dcu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai
|
| 79 |
```
|
| 80 |
|
| 81 |
-
**GPU
|
| 82 |
|
| 83 |
```bash
|
| 84 |
-
#
|
| 85 |
conda create -n onescience311 python=3.11 -y libstdcxx-ng=12 libgcc-ng=12 gcc_linux-64=12 gxx_linux-64=12
|
| 86 |
conda activate onescience311
|
| 87 |
-
#
|
| 88 |
pip install onescience[earth-gpu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai
|
| 89 |
```
|
| 90 |
|
| 91 |
-
###
|
| 92 |
|
| 93 |
-
|
| 94 |
|
| 95 |
-
EuroSAT
|
| 96 |
|
| 97 |
```text
|
| 98 |
-
|
| 99 |
https://zenodo.org/records/7711810
|
| 100 |
|
| 101 |
-
|
| 102 |
https://madm.dfki.de/files/sentinel/EuroSATallBands.zip
|
| 103 |
```
|
| 104 |
|
| 105 |
-
|
| 106 |
|
| 107 |
```bash
|
| 108 |
python scripts/fake_data.py --real-dir ./data/EuroSAT_MS
|
| 109 |
```
|
| 110 |
|
| 111 |
-
|
| 112 |
|
| 113 |
-
###
|
| 114 |
|
| 115 |
-
|
| 116 |
|
| 117 |
```bash
|
| 118 |
python scripts/fake_data.py
|
| 119 |
python scripts/train.py
|
| 120 |
```
|
| 121 |
|
| 122 |
-
|
| 123 |
|
| 124 |
```bash
|
| 125 |
python scripts/fake_data.py --real-dir ./data/EuroSAT_MS
|
| 126 |
python scripts/train.py
|
| 127 |
```
|
| 128 |
|
| 129 |
-
|
| 130 |
|
| 131 |
-
|
| 132 |
|
| 133 |
```bash
|
| 134 |
python scripts/train.py
|
| 135 |
```
|
| 136 |
|
| 137 |
-
|
| 138 |
|
| 139 |
```bash
|
| 140 |
torchrun --nproc_per_node=8 --nnodes=1 --rdzv_id=1000 --rdzv_backend=c10d --max_restarts=0 --master_addr="localhost" --master_port=29500 scripts/train.py
|
| 141 |
```
|
| 142 |
|
| 143 |
-
|
| 144 |
|
| 145 |
```text
|
| 146 |
result/checkpoints/best.pth
|
| 147 |
result/checkpoints/last.pth
|
| 148 |
-
|
| 149 |
```
|
| 150 |
|
| 151 |
-
|
| 152 |
|
| 153 |
-
###
|
| 154 |
|
| 155 |
-
|
| 156 |
|
| 157 |
-
###
|
| 158 |
|
| 159 |
```bash
|
| 160 |
python scripts/inference.py
|
| 161 |
```
|
| 162 |
|
| 163 |
-
|
| 164 |
|
| 165 |
-
|
| 166 |
|
| 167 |
```text
|
| 168 |
result/output/reconstruction.npz
|
| 169 |
```
|
| 170 |
|
| 171 |
-
###
|
| 172 |
|
| 173 |
```bash
|
| 174 |
python scripts/result.py
|
| 175 |
```
|
| 176 |
|
| 177 |
-
|
| 178 |
|
| 179 |
```text
|
| 180 |
result/output/metrics.json
|
| 181 |
result/output/reconstruction.ppm
|
| 182 |
```
|
| 183 |
|
| 184 |
-
|
| 185 |
|
| 186 |
-
# OneScience
|
| 187 |
|
| 188 |
-
|
|
| 189 |
| --- | --- | --- |
|
| 190 |
| Gitee | https://gitee.com/onescience-ai/onescience | https://gitee.com/onescience-ai/oneskills |
|
| 191 |
| GitHub | https://github.com/onescience-ai/OneScience | https://github.com/onescience-ai/oneskills |
|
| 192 |
|
| 193 |
-
#
|
| 194 |
|
| 195 |
-
-
|
| 196 |
-
-
|
|
|
|
| 1 |
---
|
| 2 |
frameworks: PyTorch
|
| 3 |
language:
|
|
|
|
| 4 |
- en
|
| 5 |
license: apache-2.0
|
| 6 |
tags:
|
|
|
|
| 20 |
</strong>
|
| 21 |
</p>
|
| 22 |
|
| 23 |
+
# Model Introduction
|
| 24 |
|
| 25 |
+
SpectralGPT learns general-purpose representations from large-scale multispectral remote sensing imagery through three-dimensional spatial-spectral masked modeling and multi-objective reconstruction. It is designed for scene classification, semantic segmentation, and change detection under limited-label conditions.
|
| 26 |
|
| 27 |
+
Paper: SpectralGPT: Spectral Remote Sensing Foundation Model
|
| 28 |
|
| 29 |
https://arxiv.org/abs/2311.07113
|
| 30 |
|
| 31 |
+
# Model Description
|
| 32 |
|
| 33 |
+
SpectralGPT was proposed by researchers from the Aerospace Information Research Institute, Chinese Academy of Sciences, and related institutions. It is trained with 12-band Sentinel-2 imagery from fMoW-S2 and BigEarthNet-S2 and is suitable for single-label or multi-label scene classification, semantic segmentation, and change detection.
|
| 34 |
|
| 35 |
+
# Use Cases
|
| 36 |
|
| 37 |
+
| Scenario | Description |
|
| 38 |
| :---: | :--- |
|
| 39 |
+
| Multispectral masked pretraining | Train SpectralGPT with normalized 12-band Sentinel-2 multispectral imagery. |
|
| 40 |
+
| Remote sensing scene understanding | Use learned multispectral representations for scene classification, semantic segmentation, and change detection. |
|
| 41 |
+
| Local quick validation | Use synthetic data to validate data loading, masked reconstruction training, inference, evaluation, and visualization. |
|
| 42 |
+
| Hugging Face / OneCode execution | Download the standalone model package, install dependencies, and run the scripts directly. |
|
| 43 |
+
| Multi-GPU training | Launch distributed multi-process training with `torchrun`. |
|
| 44 |
|
| 45 |
+
# Usage Guide
|
| 46 |
|
| 47 |
+
## 1. OneCode Usage
|
| 48 |
|
| 49 |
+
Experience intelligent one-click AI4S programming through the OneCode online environment:
|
| 50 |
|
| 51 |
+
[Click to Experience Intelligent One-Click AI4S Programming](https://web-2069360198568017922-iaaj.ksai.scnet.cn:58043/home)
|
| 52 |
|
| 53 |
+
## 2. Manual Installation and Usage
|
| 54 |
|
| 55 |
+
**Hardware Requirements**
|
| 56 |
|
| 57 |
+
- A GPU or DCU is recommended.
|
| 58 |
+
- CPU can be used for import and small-scale connectivity verification; full training and inference will be slower.
|
| 59 |
+
- DCU users must install DTK in advance. DTK 25.04.2 or above, or the OneScience recommended version matching the cluster, is recommended.
|
| 60 |
|
| 61 |
+
### Download the Model Package
|
| 62 |
|
| 63 |
```bash
|
| 64 |
hf download OneScience-Group/SpectralGPT --local-dir ./SpectralGPT
|
| 65 |
cd SpectralGPT
|
| 66 |
```
|
| 67 |
|
| 68 |
+
### Install the Runtime Environment
|
| 69 |
|
| 70 |
+
**DCU Environment**
|
| 71 |
|
| 72 |
```bash
|
| 73 |
+
# Please activate DTK and CONDA first
|
| 74 |
conda create -n onescience311 python=3.11 -y
|
| 75 |
conda activate onescience311
|
| 76 |
+
# uv installation is supported
|
| 77 |
pip install onescience[earth-dcu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai
|
| 78 |
```
|
| 79 |
|
| 80 |
+
**GPU Environment**
|
| 81 |
|
| 82 |
```bash
|
| 83 |
+
# Please activate CONDA first
|
| 84 |
conda create -n onescience311 python=3.11 -y libstdcxx-ng=12 libgcc-ng=12 gcc_linux-64=12 gxx_linux-64=12
|
| 85 |
conda activate onescience311
|
| 86 |
+
# uv installation is supported
|
| 87 |
pip install onescience[earth-gpu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai
|
| 88 |
```
|
| 89 |
|
| 90 |
+
### Training Data Introduction
|
| 91 |
|
| 92 |
+
The paper uses 12 Sentinel-2 spectral bands and excludes B10, with each band scaled to `[0,1]`. The fMoW-S2 dataset contains 882,779 images, including 712,874 images used for the first pretraining stage. BigEarthNet-S2 contains 590,326 images, including 354,196 images used for subsequent pretraining. OneScience does not currently provide fMoW-S2, BigEarthNet-S2, or EuroSAT data that can be downloaded directly for this repository, so the default workflow uses a small synthetic multispectral dataset for pipeline validation.
|
| 93 |
|
| 94 |
+
Official EuroSAT data resources:
|
| 95 |
|
| 96 |
```text
|
| 97 |
+
Official Zenodo page:
|
| 98 |
https://zenodo.org/records/7711810
|
| 99 |
|
| 100 |
+
Multispectral data download:
|
| 101 |
https://madm.dfki.de/files/sentinel/EuroSATallBands.zip
|
| 102 |
```
|
| 103 |
|
| 104 |
+
After extracting the EuroSAT multispectral data, use the following command to exclude B10, scale the bands to `[0,1]`, and convert the 13-band TIFF files into the 12-band NPZ data used by this repository:
|
| 105 |
|
| 106 |
```bash
|
| 107 |
python scripts/fake_data.py --real-dir ./data/EuroSAT_MS
|
| 108 |
```
|
| 109 |
|
| 110 |
+
Synthetic data is saved to `data/fake_spectralgpt.npz` and is used only to validate data loading, masked reconstruction training, checkpoint saving and loading, inference, evaluation, and visualization. It does not represent real Sentinel-2 imagery and cannot reproduce the paper results. Real-data training should not run `scripts/fake_data.py` without `--real-dir`, and the input TIFF files must use a consistent band order, spatial size, and value scaling method.
|
| 111 |
|
| 112 |
+
### Training
|
| 113 |
|
| 114 |
+
Local quick validation with synthetic data:
|
| 115 |
|
| 116 |
```bash
|
| 117 |
python scripts/fake_data.py
|
| 118 |
python scripts/train.py
|
| 119 |
```
|
| 120 |
|
| 121 |
+
Training with real EuroSAT multispectral TIFF files:
|
| 122 |
|
| 123 |
```bash
|
| 124 |
python scripts/fake_data.py --real-dir ./data/EuroSAT_MS
|
| 125 |
python scripts/train.py
|
| 126 |
```
|
| 127 |
|
| 128 |
+
Synthetic and real data use the same training script. Adjust `data.samples`, `model.image_size`, `training.epochs`, and `training.batch_size` in `conf/config.yaml` as needed.
|
| 129 |
|
| 130 |
+
Single GPU or CPU:
|
| 131 |
|
| 132 |
```bash
|
| 133 |
python scripts/train.py
|
| 134 |
```
|
| 135 |
|
| 136 |
+
Multi-GPU:
|
| 137 |
|
| 138 |
```bash
|
| 139 |
torchrun --nproc_per_node=8 --nnodes=1 --rdzv_id=1000 --rdzv_backend=c10d --max_restarts=0 --master_addr="localhost" --master_port=29500 scripts/train.py
|
| 140 |
```
|
| 141 |
|
| 142 |
+
Training outputs:
|
| 143 |
|
| 144 |
```text
|
| 145 |
result/checkpoints/best.pth
|
| 146 |
result/checkpoints/last.pth
|
| 147 |
+
Each epoch prints reconstruction_loss.
|
| 148 |
```
|
| 149 |
|
| 150 |
+
The `data_source` and `protocol` fields in the training logs and checkpoints identify whether synthetic or real data was used. Checkpoints generated from synthetic data are only for pipeline validation and do not provide real remote sensing representations.
|
| 151 |
|
| 152 |
+
### Training Weights
|
| 153 |
|
| 154 |
+
Multispectral Sentinel-2 training weights will be provided under `weight/` in a future update. No trained weights are included in the current package.
|
| 155 |
|
| 156 |
+
### Inference
|
| 157 |
|
| 158 |
```bash
|
| 159 |
python scripts/inference.py
|
| 160 |
```
|
| 161 |
|
| 162 |
+
Inference reads the configured dataset and the checkpoint generated during training, then produces multispectral image reconstruction results.
|
| 163 |
|
| 164 |
+
Prediction output:
|
| 165 |
|
| 166 |
```text
|
| 167 |
result/output/reconstruction.npz
|
| 168 |
```
|
| 169 |
|
| 170 |
+
### Evaluation and Visualization
|
| 171 |
|
| 172 |
```bash
|
| 173 |
python scripts/result.py
|
| 174 |
```
|
| 175 |
|
| 176 |
+
Evaluation and visualization outputs:
|
| 177 |
|
| 178 |
```text
|
| 179 |
result/output/metrics.json
|
| 180 |
result/output/reconstruction.ppm
|
| 181 |
```
|
| 182 |
|
| 183 |
+
The metrics include reconstruction MSE, MAE, PSNR, and per-band RMSE. The visualization displays input and reconstructed false-color composites. Synthetic data is used only to confirm that the evaluation and visualization pipeline runs successfully; its numerical values are not reported as model performance. Real-data metrics represent masked reconstruction on the user-provided imagery.
|
| 184 |
|
| 185 |
+
# Official OneScience Resources
|
| 186 |
|
| 187 |
+
| Platform | OneScience Main Repository | Skills Repository |
|
| 188 |
| --- | --- | --- |
|
| 189 |
| Gitee | https://gitee.com/onescience-ai/onescience | https://gitee.com/onescience-ai/oneskills |
|
| 190 |
| GitHub | https://github.com/onescience-ai/OneScience | https://github.com/onescience-ai/oneskills |
|
| 191 |
|
| 192 |
+
# Citation and License
|
| 193 |
|
| 194 |
+
- This repository is a reproduction of the original SpectralGPT paper.
|
| 195 |
+
- The model package is released under Apache License 2.0. The original datasets and model weights remain subject to their respective source licenses.
|