Upload MMDiff-WHU inference weights

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/logo.png filter=lfs diff=lfs merge=lfs -text
+assets/show.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,112 @@
+---
+license: apache-2.0
+library_name: diffusers
+tags:
+- remote-sensing
+- optical
+- sar
+- near-infrared
+- multimodal
+- text-to-image
+- diffusers
+- fine-tuned
+---
+
+<div align="center">
+
+<img src="assets/logo.png" alt="MMDiff logo" width="580" />
+
+# MMDiff: Multi-modal Remote Sensing Image Generation via Cross-Modality Spatial Feature Transfer
+
+<div  align="center" style="margin-top:10px;"> 
+<img src="assets/ribbon.svg" alt="decorative dashed ribbon" width="760" />
+<p><span style="font-size: 26px;"><strong>ISPRS 2026 &#128293;</strong></span></p>
+<div align="center">
+
+**Haojun Tang**<sup>1</sup> · **Wenda Zhao**<sup>1,*</sup> · **Hengshuai Cui**<sup>1</sup> · **Haipeng Wang**<sup>2</sup>
+
+<sup>1</sup> Dalian University of Technology  
+<sup>2</sup> Unit 92728 of PLA
+
+</div>
+
+<div align="center">
+
+[![Website](https://img.shields.io/badge/Homepage-Website-87CEEB)](#)
+[![ISPRS](https://img.shields.io/badge/ISPRS-Paper-2563EB)](#)
+[![HuggingFace](https://img.shields.io/badge/HuggingFace-MMDiff--WHU-F9D371)](https://huggingface.co/XinRan-Tang/MMDiff-WHU)
+[![HuggingFace](https://img.shields.io/badge/HuggingFace-MM--Diff-F9D371)](https://huggingface.co/XinRan-Tang/MM-Diff)
+[![Dataset](https://img.shields.io/badge/Dataset-MMDiff--WHU--dataset-green)](https://huggingface.co/datasets/XinRan-Tang/MMDiff-WHU-dataset)
+[![Dataset](https://img.shields.io/badge/Dataset-Optical--SAR--Infrared-green)](https://huggingface.co/datasets/XinRan-Tang/Optical-SAR-Infrared)
+
+</div>
+
+<sup>*</sup> **Corresponding author:**
+
+</div>
+
+
+## Abstract
+
+<div align="justify">
+
+Collecting spatially consistent multi-modal remote sensing (MMRS) images remains challenging due to different sensors vary in the imaging principles and acquisition times. This hinders the development of data-driven MMRS technologies, which rely on large-scale training samples. This paper proposes MMDiff, the first text-driven diffusion framework explicitly designed for jointly generating structurally consistent optical (OPT), synthetic aperture radar (SAR), and infrared (IR) remote sensing images from a single text prompt via cross-modality spatial feature transfer. MMDiff first trains the OPT branch with paired optical image-text data to capture rich semantic content, and then trains the SAR/IR branches with simple modality-specific text templates to learn the corresponding style attributes, without relying on complex linguistic descriptions. Specifically, we introduce a LoRA-based modality translation adaptation mechanism to translate the style attributes of optical spatial representations to SAR and IR style attributes while preserving the underlying semantic content. The translated representations are then transferred into the SAR and IR generation branches through the proposed spatial feature transfer mechanism, enabling rich spatial details in the generated SAR/IR images while maintaining cross-modal spatial consistency. Extensive experiments demonstrate that MMDiff achieves superior image quality in terms of modality similarity and semantic consistency compared to the state-of-the-art methods. Furthermore, MMDiff benefits downstream data-driven MMRS applications, e.g., multi-modal image fusion and object classification.
+
+</div>
+
+<p align="center"><img src="assets/show.png" alt="Generated multi-modal remote sensing images" width="900" /></p>
+<p align="center" style="margin-top: -10px;"><span style="color: gray; font-size: 14px;">From top to bottom: optical (OPT), synthetic aperture radar (SAR), and near-infrared (NIR) images generated by MMDiff-WHU.</span></p>
+<div align="left">
+
+## Model Overview
+
+This directory contains the MMDiff model weights fine-tuned on the **WHU-OPT-SAR** dataset for jointly generating spatially consistent optical (OPT), synthetic aperture radar (SAR), and near-infrared (NIR) remote sensing images from text prompts at **512×512** resolution.
+
+## Training Data
+
+These weights were fine-tuned on the [MMDiff-WHU-dataset](https://huggingface.co/datasets/XinRan-Tang/MMDiff-WHU-dataset), which is derived from the **WHU-OPT-SAR** dataset. The WHU-OPT-SAR imagery was first cropped into **512×512** tiles, and each tile was then assigned to the semantic class occupying the largest area in its corresponding segmentation mask (i.e., a majority vote by area). The original WHU-OPT-SAR dataset contains seven land-cover classes. We retain four classes—**city**, **farmland**, **forest**, and **water**—by merging city and village into a single category and excluding road, others, and background. Finally, every tile was captioned with GPT-4o-mini, yielding paired optical (OPT), synthetic aperture radar (SAR), near-infrared (NIR), and text data. The OPT branch was fine-tuned on all four classes, while the SAR and NIR branches were fine-tuned with class-specific LoRA adapters.
+
+## Available Checkpoints
+
+```text
+MMDiff-WHU/
+├── mmdiff-opt/   # Optical image generation branch (fine-tuned, 512×512)
+├── mmdiff-sar/   # SAR image generation branch (class-specific LoRA adapters)
+└── mmdiff-ir/    # Near-infrared (NIR) image generation branch (class-specific LoRA adapters)
+```
+
+Each SAR and NIR branch contains one LoRA adapter per class:
+
+```text
+mmdiff-sar/
+├── sd-sar-lora-city/
+├── sd-sar-lora-farmland/
+├── sd-sar-lora-forest/
+└── sd-sar-lora-water/
+
+mmdiff-ir/
+├── sd-ir-lora-city/
+├── sd-ir-lora-farmland/
+├── sd-ir-lora-forest/
+└── sd-ir-lora-water/
+```
+
+The three branches are used together by the MMDiff sampling pipeline. Refer to the main [MMDiff repository](https://github.com/XinR-Tang/MMDiff) for installation, sampling commands, and configuration options.
+
+## Loading the Weights
+
+Pass the model directory to the sampling configuration or command line. For example:
+
+```bash
+python mmdiff/sampling_OPT.py \
+  --model_path /path/to/MMDiff/models/MMDiff-WHU/mmdiff-opt \
+  --resolution 512
+```
+
+The SAR and NIR branches use their corresponding class-specific LoRA checkpoint directories during the complete OPT → SAR → NIR sampling workflow.
+
+## Intended Use
+
+These checkpoints are intended for research on text-driven multi-modal remote sensing image generation, cross-modality spatial feature transfer, multi-modal image fusion, and downstream remote sensing applications.
+
+</div>

mmdiff-ir/sd-ir-lora-city/pytorch_lora_weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2c65cab395f3538bbc14b3a2e720096b53032c915d330e4e3c97300f4872841e
+size 3226184

mmdiff-ir/sd-ir-lora-farmland/pytorch_lora_weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6d84f1bd091b8439a877bdc1289da7ae8c0dff95677a0c1a711654c0c6e80bc4
+size 3226184

mmdiff-ir/sd-ir-lora-forest/pytorch_lora_weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:390024ff68eb567222e2cc2c4af50614319f5cedacc5a9575075b4a50a6e3a65
+size 3226184

mmdiff-ir/sd-ir-lora-water/pytorch_lora_weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:33c1187e15a25fb7b1ebbcc00dae401413810c639011717687854d4498278c36
+size 3226184

mmdiff-opt/feature_extractor/preprocessor_config.json

@@ -0,0 +1,27 @@
+{
+  "crop_size": {
+    "height": 224,
+    "width": 224
+  },
+  "do_center_crop": true,
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "image_processor_type": "CLIPImageProcessor",
+  "image_std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ],
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "shortest_edge": 224
+  }
+}

mmdiff-opt/model_index.json

@@ -0,0 +1,38 @@
+{
+  "_class_name": "StableDiffusionPipeline",
+  "_diffusers_version": "0.32.0",
+  "_name_or_path": "/media/ubuntun/hdd/model/stable-diffusion-v1-4",
+  "feature_extractor": [
+    "transformers",
+    "CLIPImageProcessor"
+  ],
+  "image_encoder": [
+    null,
+    null
+  ],
+  "requires_safety_checker": true,
+  "safety_checker": [
+    "stable_diffusion",
+    "StableDiffusionSafetyChecker"
+  ],
+  "scheduler": [
+    "diffusers",
+    "PNDMScheduler"
+  ],
+  "text_encoder": [
+    "transformers",
+    "CLIPTextModel"
+  ],
+  "tokenizer": [
+    "transformers",
+    "CLIPTokenizer"
+  ],
+  "unet": [
+    "diffusers",
+    "UNet2DConditionModel"
+  ],
+  "vae": [
+    "diffusers",
+    "AutoencoderKL"
+  ]
+}

mmdiff-opt/safety_checker/config.json

@@ -0,0 +1,46 @@
+{
+  "architectures": [
+    "StableDiffusionSafetyChecker"
+  ],
+  "dtype": "float32",
+  "initializer_factor": 1.0,
+  "logit_scale_init_value": 2.6592,
+  "model_type": "clip",
+  "projection_dim": 768,
+  "text_config": {
+    "attention_dropout": 0.0,
+    "dropout": 0.0,
+    "dtype": "float32",
+    "hidden_act": "quick_gelu",
+    "hidden_size": 768,
+    "initializer_factor": 1.0,
+    "initializer_range": 0.02,
+    "intermediate_size": 3072,
+    "layer_norm_eps": 1e-05,
+    "max_position_embeddings": 77,
+    "model_type": "clip_text_model",
+    "num_attention_heads": 12,
+    "num_hidden_layers": 12,
+    "projection_dim": 512,
+    "vocab_size": 49408
+  },
+  "transformers_version": "4.56.2",
+  "vision_config": {
+    "attention_dropout": 0.0,
+    "dropout": 0.0,
+    "dtype": "float32",
+    "hidden_act": "quick_gelu",
+    "hidden_size": 1024,
+    "image_size": 224,
+    "initializer_factor": 1.0,
+    "initializer_range": 0.02,
+    "intermediate_size": 4096,
+    "layer_norm_eps": 1e-05,
+    "model_type": "clip_vision_model",
+    "num_attention_heads": 16,
+    "num_channels": 3,
+    "num_hidden_layers": 24,
+    "patch_size": 14,
+    "projection_dim": 512
+  }
+}

mmdiff-opt/safety_checker/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fb351a5ded815c3ff744968ad9c6b218d071b9d313d04f35e813b84b4c0ffde8
+size 1215979664

mmdiff-opt/scheduler/scheduler_config.json

@@ -0,0 +1,15 @@
+{
+  "_class_name": "PNDMScheduler",
+  "_diffusers_version": "0.32.0",
+  "beta_end": 0.012,
+  "beta_schedule": "scaled_linear",
+  "beta_start": 0.00085,
+  "clip_sample": false,
+  "num_train_timesteps": 1000,
+  "prediction_type": "epsilon",
+  "set_alpha_to_one": false,
+  "skip_prk_steps": true,
+  "steps_offset": 1,
+  "timestep_spacing": "leading",
+  "trained_betas": null
+}

mmdiff-opt/text_encoder/config.json

@@ -0,0 +1,24 @@
+{
+  "architectures": [
+    "CLIPTextModel"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "dropout": 0.0,
+  "dtype": "float32",
+  "eos_token_id": 2,
+  "hidden_act": "quick_gelu",
+  "hidden_size": 768,
+  "initializer_factor": 1.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 77,
+  "model_type": "clip_text_model",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 1,
+  "projection_dim": 512,
+  "transformers_version": "4.56.2",
+  "vocab_size": 49408
+}

mmdiff-opt/text_encoder/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:778d02eb9e707c3fbaae0b67b79ea0d1399b52e624fb634f2f19375ae7c047c3
+size 492265168

mmdiff-opt/tokenizer/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<|startoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|endoftext|>",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

mmdiff-opt/tokenizer/tokenizer_config.json

@@ -0,0 +1,31 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "49406": {
+      "content": "<|startoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49407": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|startoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "do_lower_case": true,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 77,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "CLIPTokenizer",
+  "unk_token": "<|endoftext|>"
+}

mmdiff-opt/unet/config.json

@@ -0,0 +1,68 @@
+{
+  "_class_name": "UNet2DConditionModel",
+  "_diffusers_version": "0.32.0",
+  "_name_or_path": "/media/ubuntun/hdd/model/stable-diffusion-v1-4",
+  "act_fn": "silu",
+  "addition_embed_type": null,
+  "addition_embed_type_num_heads": 64,
+  "addition_time_embed_dim": null,
+  "attention_head_dim": 8,
+  "attention_type": "default",
+  "block_out_channels": [
+    320,
+    640,
+    1280,
+    1280
+  ],
+  "center_input_sample": false,
+  "class_embed_type": null,
+  "class_embeddings_concat": false,
+  "conv_in_kernel": 3,
+  "conv_out_kernel": 3,
+  "cross_attention_dim": 768,
+  "cross_attention_norm": null,
+  "down_block_types": [
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "DownBlock2D"
+  ],
+  "downsample_padding": 1,
+  "dropout": 0.0,
+  "dual_cross_attention": false,
+  "encoder_hid_dim": null,
+  "encoder_hid_dim_type": null,
+  "flip_sin_to_cos": true,
+  "freq_shift": 0,
+  "in_channels": 4,
+  "layers_per_block": 2,
+  "mid_block_only_cross_attention": null,
+  "mid_block_scale_factor": 1,
+  "mid_block_type": "UNetMidBlock2DCrossAttn",
+  "norm_eps": 1e-05,
+  "norm_num_groups": 32,
+  "num_attention_heads": null,
+  "num_class_embeds": null,
+  "only_cross_attention": false,
+  "out_channels": 4,
+  "projection_class_embeddings_input_dim": null,
+  "resnet_out_scale_factor": 1.0,
+  "resnet_skip_time_act": false,
+  "resnet_time_scale_shift": "default",
+  "reverse_transformer_layers_per_block": null,
+  "sample_size": 64,
+  "time_cond_proj_dim": null,
+  "time_embedding_act_fn": null,
+  "time_embedding_dim": null,
+  "time_embedding_type": "positional",
+  "timestep_post_act": null,
+  "transformer_layers_per_block": 1,
+  "up_block_types": [
+    "UpBlock2D",
+    "CrossAttnUpBlock2D",
+    "CrossAttnUpBlock2D",
+    "CrossAttnUpBlock2D"
+  ],
+  "upcast_attention": false,
+  "use_linear_projection": false
+}

mmdiff-opt/unet/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e355d2323eeeb9b662dfb27fd0429d4a9d1e4d8ac7829de5ef4bc4168443c7c0
+size 3438167536

mmdiff-opt/vae/config.json

@@ -0,0 +1,38 @@
+{
+  "_class_name": "AutoencoderKL",
+  "_diffusers_version": "0.32.0",
+  "_name_or_path": "/media/ubuntun/hdd/model/stable-diffusion-v1-4/vae",
+  "act_fn": "silu",
+  "block_out_channels": [
+    128,
+    256,
+    512,
+    512
+  ],
+  "down_block_types": [
+    "DownEncoderBlock2D",
+    "DownEncoderBlock2D",
+    "DownEncoderBlock2D",
+    "DownEncoderBlock2D"
+  ],
+  "force_upcast": true,
+  "in_channels": 3,
+  "latent_channels": 4,
+  "latents_mean": null,
+  "latents_std": null,
+  "layers_per_block": 2,
+  "mid_block_add_attention": true,
+  "norm_num_groups": 32,
+  "out_channels": 3,
+  "sample_size": 256,
+  "scaling_factor": 0.18215,
+  "shift_factor": null,
+  "up_block_types": [
+    "UpDecoderBlock2D",
+    "UpDecoderBlock2D",
+    "UpDecoderBlock2D",
+    "UpDecoderBlock2D"
+  ],
+  "use_post_quant_conv": true,
+  "use_quant_conv": true
+}

mmdiff-opt/vae/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b4d2b5932bb4151e54e694fd31ccf51fca908223c9485bd56cd0e1d83ad94c49
+size 334643268

mmdiff-sar/sd-sar-lora-city/pytorch_lora_weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:69670f02dc6e111aeb46e5c75d0afecfb94c452a87ca7df044a1bdb23a2b17d8
+size 3226184

mmdiff-sar/sd-sar-lora-farmland/pytorch_lora_weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:90fc73a94059d7939b228a835cc2e3c73db87893f700eb9994a339f49173d9e4
+size 3226184

mmdiff-sar/sd-sar-lora-forest/pytorch_lora_weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9c4d3b961d27a702ea1eba6f010a7cdb9a3bc3149beccb0ccac0f451a6f135c8
+size 3226184

mmdiff-sar/sd-sar-lora-water/pytorch_lora_weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d30a54b52dd156fcd9c93a4e84684d5e76488f53d40997e5aef7f77e205c0b21
+size 3226184