# MMDiff: Multi-modal Remote Sensing Image Generation via Cross-Modality Spatial Feature Transfer

ISPRS 2026 🔥

**Haojun Tang**¹ · **Wenda Zhao**^1,* · **Hengshuai Cui**¹ · **Haipeng Wang**² ¹ Dalian University of Technology ² Unit 92728 of PLA

[![Website](https://img.shields.io/badge/Homepage-Website-87CEEB)](#) [![ISPRS](https://img.shields.io/badge/ISPRS-Paper-2563EB)](#) [![HuggingFace](https://img.shields.io/badge/HuggingFace-MMDiff--WHU-F9D371)](https://huggingface.co/XinRan-Tang/MMDiff-WHU) [![HuggingFace](https://img.shields.io/badge/HuggingFace-MM--Diff-F9D371)](https://huggingface.co/XinRan-Tang/MM-Diff) [![Dataset](https://img.shields.io/badge/Dataset-MMDiff--WHU--dataset-green)](https://huggingface.co/datasets/XinRan-Tang/MMDiff-WHU-dataset) [![Dataset](https://img.shields.io/badge/Dataset-Optical--SAR--Infrared-green)](https://huggingface.co/datasets/XinRan-Tang/Optical-SAR-Infrared)

^* **Corresponding author:**

## Abstract

Collecting spatially consistent multi-modal remote sensing (MMRS) images remains challenging due to different sensors vary in the imaging principles and acquisition times. This hinders the development of data-driven MMRS technologies, which rely on large-scale training samples. This paper proposes MMDiff, the first text-driven diffusion framework explicitly designed for jointly generating structurally consistent optical (OPT), synthetic aperture radar (SAR), and infrared (IR) remote sensing images from a single text prompt via cross-modality spatial feature transfer. MMDiff first trains the OPT branch with paired optical image-text data to capture rich semantic content, and then trains the SAR/IR branches with simple modality-specific text templates to learn the corresponding style attributes, without relying on complex linguistic descriptions. Specifically, we introduce a LoRA-based modality translation adaptation mechanism to translate the style attributes of optical spatial representations to SAR and IR style attributes while preserving the underlying semantic content. The translated representations are then transferred into the SAR and IR generation branches through the proposed spatial feature transfer mechanism, enabling rich spatial details in the generated SAR/IR images while maintaining cross-modal spatial consistency. Extensive experiments demonstrate that MMDiff achieves superior image quality in terms of modality similarity and semantic consistency compared to the state-of-the-art methods. Furthermore, MMDiff benefits downstream data-driven MMRS applications, e.g., multi-modal image fusion and object classification.

Generated multi-modal remote sensing images

For each group, from left to right: optical (OPT), synthetic aperture radar (SAR), and near-infrared (NIR) images.

## Model Overview This directory contains the MMDiff model weights fine-tuned on the **WHU-OPT-SAR** dataset for jointly generating spatially consistent optical (OPT), synthetic aperture radar (SAR), and near-infrared (NIR) remote sensing images from text prompts at **512×512** resolution. ## Training Data These weights were fine-tuned on the [MMDiff-WHU-dataset](https://huggingface.co/datasets/XinRan-Tang/MMDiff-WHU-dataset), which is derived from the **WHU-OPT-SAR** dataset. The WHU-OPT-SAR imagery was first cropped into **512×512** tiles, and each tile was then assigned to the semantic class occupying the largest area in its corresponding segmentation mask (i.e., a majority vote by area). The original WHU-OPT-SAR dataset contains seven land-cover classes. We retain four classes—**city**, **farmland**, **forest**, and **water**—by merging city and village into a single category and excluding road, others, and background. Finally, every tile was captioned with GPT-4o-mini, yielding paired optical (OPT), synthetic aperture radar (SAR), near-infrared (NIR), and text data. The OPT branch was fine-tuned on all four classes, while the SAR and NIR branches were fine-tuned with class-specific LoRA adapters. ## Available Checkpoints ```text MMDiff-WHU/ ├── mmdiff-opt/ # Optical image generation branch (fine-tuned, 512×512) ├── mmdiff-sar/ # SAR image generation branch (class-specific LoRA adapters) └── mmdiff-ir/ # Near-infrared (NIR) image generation branch (class-specific LoRA adapters) ``` Each SAR and NIR branch contains one LoRA adapter per class: ```text mmdiff-sar/ ├── sd-sar-lora-city/ ├── sd-sar-lora-farmland/ ├── sd-sar-lora-forest/ └── sd-sar-lora-water/ mmdiff-ir/ ├── sd-ir-lora-city/ ├── sd-ir-lora-farmland/ ├── sd-ir-lora-forest/ └── sd-ir-lora-water/ ``` The three branches are used together by the MMDiff sampling pipeline. Refer to the main [MMDiff repository](https://github.com/XinR-Tang/MMDiff) for installation, sampling commands, and configuration options. ## Loading the Weights Pass the model directory to the sampling configuration or command line. For example: ```bash python mmdiff/sampling_OPT.py \ --model_path /path/to/MMDiff/models/MMDiff-WHU/mmdiff-opt \ --resolution 512 ``` The SAR and NIR branches use their corresponding class-specific LoRA checkpoint directories during the complete OPT → SAR → NIR sampling workflow. ## Intended Use These checkpoints are intended for research on text-driven multi-modal remote sensing image generation, cross-modality spatial feature transfer, multi-modal image fusion, and downstream remote sensing applications.