README.md

---
license: apache-2.0
language:
- en
tags:
- OneScience
- Earth Science
- Multimodal Earth Observation
- Generative Foundation Model
frameworks: PyTorch
datasets:
- ibm-esa-geospatial/TerraMesh
---

<p align="center">
  <strong>
    <span style="font-size: 30px;">TerraMind</span>
  </strong>
</p>

# Model Introduction

TerraMind is an any-to-any generative foundation model for Earth observation data. It jointly processes pixel-level and discrete token-level data and learns cross-modal relationships among radar, optical imagery, elevation, land cover, vegetation indices, geographic coordinates, and text.

Paper: TerraMind: Large-Scale Generative Multimodality for Earth Observation  
https://arxiv.org/abs/2504.11171

# Model Description

TerraMind was proposed by organizations including IBM Research, the European Space Agency, and Forschungszentrum Jülich. The model was pretrained on approximately 9 million globally distributed, spatiotemporally aligned TerraMesh samples and approximately 500 billion training tokens. It is suitable for cross-modal generation, Earth observation representation learning, land-cover segmentation, water-body identification, vegetation assessment, and Thinking-in-Modalities tasks.

# Use Cases

| Use Case | Description |
| :---: | :--- |
| Multimodal representation | Jointly encodes optical, radar, elevation, and other Earth observation modalities. |
| Any-to-any generation | Predicts target-modality tokens such as LULC, NDVI, and radar from available modalities such as Sentinel-2, coordinates, and text. |
| Dual-scale learning | Uses both raw pixel patches and discrete token representations. |
| Local engineering validation | Uses aligned synthetic data to verify the training, inference, evaluation, visualization, and checkpoint workflows. |
| Multi-GPU training | Launches distributed data-parallel training through `torchrun`. |

# Usage Instructions

## 1. OneCode

Experience intelligent, one-click AI4S programming through the OneCode online environment:

[Try intelligent, one-click AI4S programming](https://web-2069360198568017922-iaaj.ksai.scnet.cn:58043/home)

## 2. Download and Installation

```bash
hf download OneScience-Group/TerraMind --local-dir ./TerraMind
cd TerraMind
```

### Environment Dependencies

**Hardware Requirements**

- A GPU or DCU is recommended.
- A CPU can be used to verify connectivity with the default small-sample configuration; official-scale training and diffusion decoding require large-scale accelerated computing resources.
- DCU users must install DTK in advance. DTK 25.04.2 or later, or the OneScience-recommended version matching the current cluster, is recommended.

**DCU Environment**

```bash
# Activate DTK and Conda first
conda create -n onescience311 python=3.11 -y
conda activate onescience311
pip install onescience[earth-dcu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai
```

**GPU Environment**

```bash
# Activate Conda first
conda create -n onescience311 python=3.11 -y libstdcxx-ng=12 libgcc-ng=12 gcc_linux-64=12 gxx_linux-64=12
conda activate onescience311
pip install onescience[earth-gpu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai
```

### Training Data

This repository uses spatiotemporally aligned synthetic multimodal samples to validate the engineering workflow. The source data preserve TerraMesh's `264×264` spatial dimensions and include 12-channel Sentinel-2 L2A, 13-channel Sentinel-2 L1C, 3-channel RGB, 2-channel Sentinel-1 GRD, 2-channel Sentinel-1 RTC, and single-channel DEM data. The loader jointly crops the `224×224` inputs used by the official model from each source sample and constructs content-dependent discrete tokens for LULC, NDVI, radar, and other modalities from the same region, ensuring dual-scale spatial alignment.

```bash
python scripts/fake_data.py
```

### Training

```bash
python scripts/train.py
```

For multi-GPU training, use:

```bash
torchrun --nproc_per_node=8 --nnodes=1 --rdzv_id=1000 --rdzv_backend=c10d --max_restarts=0 --master_addr="localhost" --master_port=29500 scripts/train.py
```

The default configuration only reduces the number of samples, Transformer width, depth, vocabulary, and training epochs; it does not change the source-data or model-input spatial dimensions. During training, the workflow randomly selects input pixel modalities, input token modalities, and target modalities, and randomly samples input patches to simulate the paper's multimodal masked-modeling strategy.

```text
result/checkpoints/terramind.pt
result/training/metrics.json
```

### Trained Weights

This repository does not include synthetic or official weights in `weight/`. IBM and ESA have published model weights for TerraMind variants including tiny, small, base, and large.

### Inference

```bash
python scripts/inference.py
```

Inference loads the training checkpoint and generates LULC, NDVI, and Sentinel-1 GRD tokens conditioned only on Sentinel-2 L2A pixels, coordinate tokens, and text tokens. It saves the cross-modal embeddings and target tokens to:

```text
result/output/predictions.npz
```

### Evaluation and Visualization

```bash
python scripts/result.py
```

The evaluation measures token accuracy for each target modality and cross-modal embedding norms, and generates spatial comparison plots of target and generated tokens. Results on synthetic data only validate the engineering pipeline and do not represent the PANGAEA, generation-quality, or Thinking-in-Modalities metrics reported in the paper.

```text
result/evaluation/metrics.json
result/evaluation/comparison.png
```

# Official OneScience Information

| Platform | OneScience Main Repository | Skills Repository |
| --- | --- | --- |
| Gitee | https://gitee.com/onescience-ai/onescience | https://gitee.com/onescience-ai/oneskills |
| GitHub | https://github.com/onescience-ai/OneScience | https://github.com/onescience-ai/oneskills |

# Citation and License

This repository is an independent engineering reproduction based on the publicly available specifications from the TerraMind paper.

Use of this repository's code, the official model weights, and the data remains subject to the licenses and terms of use of their respective projects.