Update README.md

7e1e177 verified 15 days ago

6.3 kB

---
license: apache-2.0
library_name: diffusers
tags:
- remote-sensing
- optical
- sar
- near-infrared
- multimodal
- text-to-image
- diffusers
- fine-tuned
---
<div align="center">

<img src="assets/logo.png" alt="MMDiff logo" width="580" />

# MMDiff: Multi-modal Remote Sensing Image Generation via Cross-Modality Spatial Feature Transfer

<div  align="center" style="margin-top:10px;"> 
<img src="assets/ribbon.svg" alt="decorative dashed ribbon" width="760" />
<p><span style="font-size: 26px;"><strong>ISPRS 2026 &#128293;</strong></span></p>
<div align="center">

**Haojun Tang**<sup>1</sup> · **Wenda Zhao**<sup>1,*</sup> · **Hengshuai Cui**<sup>1</sup> · **Haipeng Wang**<sup>2</sup>
<sup>1</sup> Dalian University of Technology  
<sup>2</sup> Unit 92728 of PLA
</div>
<div align="center">
[![Website](https://img.shields.io/badge/Homepage-Website-87CEEB)](#)
[![ISPRS](https://img.shields.io/badge/ISPRS-Paper-2563EB)](#)
[![HuggingFace](https://img.shields.io/badge/HuggingFace-MMDiff--WHU-F9D371)](https://huggingface.co/XinRan-Tang/MMDiff-WHU)
[![HuggingFace](https://img.shields.io/badge/HuggingFace-MM--Diff-F9D371)](https://huggingface.co/XinRan-Tang/MM-Diff)
[![Dataset](https://img.shields.io/badge/Dataset-MMDiff--WHU--dataset-green)](https://huggingface.co/datasets/XinRan-Tang/MMDiff-WHU-dataset)
[![Dataset](https://img.shields.io/badge/Dataset-Optical--SAR--Infrared-green)](https://huggingface.co/datasets/XinRan-Tang/Optical-SAR-Infrared)
</div>
<sup>*</sup> **Corresponding author:**

</div>


## Abstract

<div align="justify">

Collecting spatially consistent multi-modal remote sensing (MMRS) images remains challenging due to different sensors vary in the imaging principles and acquisition times. This hinders the development of data-driven MMRS technologies, which rely on large-scale training samples. This paper proposes MMDiff, the first text-driven diffusion framework explicitly designed for jointly generating structurally consistent optical (OPT), synthetic aperture radar (SAR), and infrared (IR) remote sensing images from a single text prompt via cross-modality spatial feature transfer. MMDiff first trains the OPT branch with paired optical image-text data to capture rich semantic content, and then trains the SAR/IR branches with simple modality-specific text templates to learn the corresponding style attributes, without relying on complex linguistic descriptions. Specifically, we introduce a LoRA-based modality translation adaptation mechanism to translate the style attributes of optical spatial representations to SAR and IR style attributes while preserving the underlying semantic content. The translated representations are then transferred into the SAR and IR generation branches through the proposed spatial feature transfer mechanism, enabling rich spatial details in the generated SAR/IR images while maintaining cross-modal spatial consistency. Extensive experiments demonstrate that MMDiff achieves superior image quality in terms of modality similarity and semantic consistency compared to the state-of-the-art methods. Furthermore, MMDiff benefits downstream data-driven MMRS applications, e.g., multi-modal image fusion and object classification.

</div>

<p align="center"><img src="assets/show.png" alt="Generated multi-modal remote sensing images" width="900" /></p>
<p align="center" style="margin-top: -10px;"><span style="color: gray; font-size: 14px;">For each group, from left to right: optical (OPT), synthetic aperture radar (SAR), and near-infrared (NIR) images.</span></p>
<div align="left">

## Model Overview

This directory contains the MMDiff model weights fine-tuned on the **WHU-OPT-SAR** dataset for jointly generating spatially consistent optical (OPT), synthetic aperture radar (SAR), and near-infrared (NIR) remote sensing images from text prompts at **512×512** resolution.

## Training Data

These weights were fine-tuned on the [MMDiff-WHU-dataset](https://huggingface.co/datasets/XinRan-Tang/MMDiff-WHU-dataset), which is derived from the **WHU-OPT-SAR** dataset. The WHU-OPT-SAR imagery was first cropped into **512×512** tiles, and each tile was then assigned to the semantic class occupying the largest area in its corresponding segmentation mask (i.e., a majority vote by area). The original WHU-OPT-SAR dataset contains seven land-cover classes. We retain four classes—**city**, **farmland**, **forest**, and **water**—by merging city and village into a single category and excluding road, others, and background. Finally, every tile was captioned with GPT-4o-mini, yielding paired optical (OPT), synthetic aperture radar (SAR), near-infrared (NIR), and text data. The OPT branch was fine-tuned on all four classes, while the SAR and NIR branches were fine-tuned with class-specific LoRA adapters.

## Available Checkpoints

```text
MMDiff-WHU/
├── mmdiff-opt/   # Optical image generation branch (fine-tuned, 512×512)
├── mmdiff-sar/   # SAR image generation branch (class-specific LoRA adapters)
└── mmdiff-ir/    # Near-infrared (NIR) image generation branch (class-specific LoRA adapters)
```

Each SAR and NIR branch contains one LoRA adapter per class:

```text
mmdiff-sar/
├── sd-sar-lora-city/
├── sd-sar-lora-farmland/
├── sd-sar-lora-forest/
└── sd-sar-lora-water/
mmdiff-ir/
├── sd-ir-lora-city/
├── sd-ir-lora-farmland/
├── sd-ir-lora-forest/
└── sd-ir-lora-water/
```

The three branches are used together by the MMDiff sampling pipeline. Refer to the main [MMDiff repository](https://github.com/XinR-Tang/MMDiff) for installation, sampling commands, and configuration options.

## Loading the Weights

Pass the model directory to the sampling configuration or command line. For example:

```bash
python mmdiff/sampling_OPT.py \
  --model_path /path/to/MMDiff/models/MMDiff-WHU/mmdiff-opt \
  --resolution 512
```

The SAR and NIR branches use their corresponding class-specific LoRA checkpoint directories during the complete OPT → SAR → NIR sampling workflow.

## Intended Use

These checkpoints are intended for research on text-driven multi-modal remote sensing image generation, cross-modality spatial feature transfer, multi-modal image fusion, and downstream remote sensing applications.

</div>

	---
	license: apache-2.0
	library_name: diffusers
	tags:
	- remote-sensing
	- optical
	- sar
	- near-infrared
	- multimodal
	- text-to-image
	- diffusers
	- fine-tuned
	---

	<div align="center">

	<img src="assets/logo.png" alt="MMDiff logo" width="580" />

	# MMDiff: Multi-modal Remote Sensing Image Generation via Cross-Modality Spatial Feature Transfer

	<div align="center" style="margin-top:10px;">
	<img src="assets/ribbon.svg" alt="decorative dashed ribbon" width="760" />
	<p><span style="font-size: 26px;"><strong>ISPRS 2026 🔥</strong></span></p>
	<div align="center">

	Haojun Tang<sup>1</sup> · Wenda Zhao<sup>1,</sup> · Hengshuai Cui<sup>1</sup> · Haipeng Wang*<sup>2</sup>

	<sup>1</sup> Dalian University of Technology
	<sup>2</sup> Unit 92728 of PLA

	</div>

	<div align="center">

	[![Website](https://img.shields.io/badge/Homepage-Website-87CEEB)](#)
	[![ISPRS](https://img.shields.io/badge/ISPRS-Paper-2563EB)](#)
	[![HuggingFace](https://img.shields.io/badge/HuggingFace-MMDiff--WHU-F9D371)](https://huggingface.co/XinRan-Tang/MMDiff-WHU)
	[![HuggingFace](https://img.shields.io/badge/HuggingFace-MM--Diff-F9D371)](https://huggingface.co/XinRan-Tang/MM-Diff)
	[![Dataset](https://img.shields.io/badge/Dataset-MMDiff--WHU--dataset-green)](https://huggingface.co/datasets/XinRan-Tang/MMDiff-WHU-dataset)
	[![Dataset](https://img.shields.io/badge/Dataset-Optical--SAR--Infrared-green)](https://huggingface.co/datasets/XinRan-Tang/Optical-SAR-Infrared)

	</div>

	<sup></sup> Corresponding author:*

	</div>


	## Abstract

	<div align="justify">

	Collecting spatially consistent multi-modal remote sensing (MMRS) images remains challenging due to different sensors vary in the imaging principles and acquisition times. This hinders the development of data-driven MMRS technologies, which rely on large-scale training samples. This paper proposes MMDiff, the first text-driven diffusion framework explicitly designed for jointly generating structurally consistent optical (OPT), synthetic aperture radar (SAR), and infrared (IR) remote sensing images from a single text prompt via cross-modality spatial feature transfer. MMDiff first trains the OPT branch with paired optical image-text data to capture rich semantic content, and then trains the SAR/IR branches with simple modality-specific text templates to learn the corresponding style attributes, without relying on complex linguistic descriptions. Specifically, we introduce a LoRA-based modality translation adaptation mechanism to translate the style attributes of optical spatial representations to SAR and IR style attributes while preserving the underlying semantic content. The translated representations are then transferred into the SAR and IR generation branches through the proposed spatial feature transfer mechanism, enabling rich spatial details in the generated SAR/IR images while maintaining cross-modal spatial consistency. Extensive experiments demonstrate that MMDiff achieves superior image quality in terms of modality similarity and semantic consistency compared to the state-of-the-art methods. Furthermore, MMDiff benefits downstream data-driven MMRS applications, e.g., multi-modal image fusion and object classification.

	</div>

	<p align="center"><img src="assets/show.png" alt="Generated multi-modal remote sensing images" width="900" /></p>
	<p align="center" style="margin-top: -10px;"><span style="color: gray; font-size: 14px;">For each group, from left to right: optical (OPT), synthetic aperture radar (SAR), and near-infrared (NIR) images.</span></p>
	<div align="left">

	## Model Overview

	This directory contains the MMDiff model weights fine-tuned on the WHU-OPT-SAR dataset for jointly generating spatially consistent optical (OPT), synthetic aperture radar (SAR), and near-infrared (NIR) remote sensing images from text prompts at 512×512 resolution.

	## Training Data

	These weights were fine-tuned on the [MMDiff-WHU-dataset](https://huggingface.co/datasets/XinRan-Tang/MMDiff-WHU-dataset), which is derived from the WHU-OPT-SAR dataset. The WHU-OPT-SAR imagery was first cropped into 512×512 tiles, and each tile was then assigned to the semantic class occupying the largest area in its corresponding segmentation mask (i.e., a majority vote by area). The original WHU-OPT-SAR dataset contains seven land-cover classes. We retain four classes—city, farmland, forest, and water—by merging city and village into a single category and excluding road, others, and background. Finally, every tile was captioned with GPT-4o-mini, yielding paired optical (OPT), synthetic aperture radar (SAR), near-infrared (NIR), and text data. The OPT branch was fine-tuned on all four classes, while the SAR and NIR branches were fine-tuned with class-specific LoRA adapters.

	## Available Checkpoints

	```text
	MMDiff-WHU/
	├── mmdiff-opt/ # Optical image generation branch (fine-tuned, 512×512)
	├── mmdiff-sar/ # SAR image generation branch (class-specific LoRA adapters)
	└── mmdiff-ir/ # Near-infrared (NIR) image generation branch (class-specific LoRA adapters)
	```

	Each SAR and NIR branch contains one LoRA adapter per class:

	```text
	mmdiff-sar/
	├── sd-sar-lora-city/
	├── sd-sar-lora-farmland/
	├── sd-sar-lora-forest/
	└── sd-sar-lora-water/

	mmdiff-ir/
	├── sd-ir-lora-city/
	├── sd-ir-lora-farmland/
	├── sd-ir-lora-forest/
	└── sd-ir-lora-water/
	```

	The three branches are used together by the MMDiff sampling pipeline. Refer to the main [MMDiff repository](https://github.com/XinR-Tang/MMDiff) for installation, sampling commands, and configuration options.

	## Loading the Weights

	Pass the model directory to the sampling configuration or command line. For example:

	```bash
	python mmdiff/sampling_OPT.py \
	--model_path /path/to/MMDiff/models/MMDiff-WHU/mmdiff-opt \
	--resolution 512
	```

	The SAR and NIR branches use their corresponding class-specific LoRA checkpoint directories during the complete OPT → SAR → NIR sampling workflow.

	## Intended Use

	These checkpoints are intended for research on text-driven multi-modal remote sensing image generation, cross-modality spatial feature transfer, multi-modal image fusion, and downstream remote sensing applications.

	</div>