| --- |
| license: other |
| license_name: per-method-see-below |
| library_name: pytorch |
| pipeline_tag: image-to-image |
| tags: |
| - sar-to-eo |
| - image-translation |
| - remote-sensing |
| - baselines |
| - benchmark |
| - reflowset |
| --- |
| |
| # ReFlowSET — retrained comparison-method checkpoints |
|
|
| The fifteen comparison methods of the **ReFlowSET** SAR-to-EO benchmark, on both |
| of its datasets: **QXS-SAROPT** (256 px) and **SAR2Opt** (512 px). Thirty |
| checkpoint cells in total. |
|
|
| **Every checkpoint here was retrained by us.** None is an original author's |
| released checkpoint. Each method was trained on the same official split, at the |
| same resolution, under a budget stated in optimizer updates, and every cell was |
| then scored through one evaluation pipeline with an identity-collapse and |
| ground-truth-leak audit. That is what makes the rows below comparable **to each |
| other** — and, for exactly the same reason, not comparable with the numbers in |
| the methods' own papers. |
|
|
| ReFlowSET itself is not in this repository. It lives at |
| [JeonghyeokDo/ReFlowSET](https://huggingface.co/JeonghyeokDo/ReFlowSET); code at https://github.com/KAIST-VICLab/ReFlowSET; project page |
| at https://kaist-viclab.github.io/ReFlowSET_site/. |
| |
| ## Results — all sixteen rows, both datasets |
| |
| Our own measurements, on the images each checkpoint produced. No number below |
| is copied from any paper. |
| |
| ### QXS-SAROPT — n = 3,999, 256 px |
| |
| | Method | Venue | FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ | |
| |---|---|---|---|---|---|---| |
| | *General image-to-image* | | | | | | | |
| | pix2pix | CVPR'17 | 174.6 | 0.373 | 0.665 | 0.203 | 12.33 | |
| | CycleGAN ⚠ | ICCV'17 | 104.4 | 0.376 | 0.653 | 0.262 | 12.92 | |
| | pix2pixHD | CVPR'18 | 85.7 | 0.298 | 0.573 | 0.358 | 16.13 | |
| | SPADE | CVPR'19 | 90.7 | 0.292 | 0.599 | 0.320 | 14.53 | |
| | DDPM (SR3-class) | TPAMI'22 | 43.8 | 0.311 | 0.620 | 0.359 | 14.04 | |
| | SD2.1 fine-tune only | CVPR'22 | 19.1 | 0.257 | 0.561 | 0.348 | 15.40 | |
| | BBDM | CVPR'23 | 76.6 | 0.270 | 0.568 | 0.352 | 15.34 | |
| | ControlNet | ICCV'23 | 50.4 | 0.307 | 0.604 | 0.297 | 13.42 | |
| | HI-Diff | NeurIPS'23 | 324.3 | 0.539 | 0.692 | 0.457 | 17.10 | |
| | ResShift | NeurIPS'23 | 140.2 | 0.334 | 0.607 | 0.217 | 14.20 | |
| | StegoGAN ⚠ | CVPR'24 | 106.8 | 0.384 | 0.658 | 0.254 | 12.96 | |
| | *SAR-to-EO* | | | | | | | |
| | Conditional Diffusion | GRSL'24 | 88.6 | 0.355 | 0.730 | 0.213 | 11.55 | |
| | cBBDM | GRSL'25 | 50.6 | 0.246 | 0.539 | 0.372 | 16.02 | |
| | E3Diff | GRSL'25 | 47.8 | 0.278 | 0.530 | 0.302 | 16.44 | |
| | C-DiffSET | TCSVT'26 | 19.9 | 0.233 | 0.526 | 0.380 | 16.92 | |
| | *Ours* | | | | | | | |
| | **ReFlowSET (ours)** | — | 19.1 | 0.231 | 0.534 | 0.355 | 16.09 | |
| |
| ### SAR2Opt — n = 627, 512 px (centre crops of the 600 px tiles) |
| |
| | Method | Venue | FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ | |
| |---|---|---|---|---|---|---| |
| | *General image-to-image* | | | | | | | |
| | pix2pix | CVPR'17 | 261.9 | 0.347 | 0.657 | 0.199 | 13.39 | |
| | CycleGAN ⚠ | ICCV'17 | 143.5 | 0.330 | 0.650 | 0.178 | 12.90 | |
| | pix2pixHD | CVPR'18 | 146.3 | 0.283 | 0.567 | 0.268 | 15.95 | |
| | SPADE | CVPR'19 | 142.5 | 0.265 | 0.597 | 0.234 | 14.47 | |
| | DDPM (SR3-class) | TPAMI'22 | 122.5 | 0.295 | 0.610 | 0.313 | 13.65 | |
| | SD2.1 fine-tune only | CVPR'22 | 71.8 | 0.211 † | 0.541 | 0.293 | 16.24 | |
| | BBDM | CVPR'23 | 143.1 | 0.290 | 0.590 | 0.276 | 15.29 | |
| | ControlNet | ICCV'23 | 140.5 | 0.350 | 0.643 | 0.217 | 11.73 | |
| | HI-Diff | NeurIPS'23 | 319.8 | 0.473 | 0.692 | 0.384 | 17.36 | |
| | ResShift | NeurIPS'23 | 141.7 | 0.304 | 0.597 | 0.177 | 14.31 | |
| | StegoGAN ⚠ | CVPR'24 | 150.1 | 0.347 | 0.655 | 0.158 | 12.47 | |
| | *SAR-to-EO* | | | | | | | |
| | Conditional Diffusion | GRSL'24 | 211.8 | 0.415 | 0.686 | 0.248 | 12.48 | |
| | cBBDM | GRSL'25 | 222.3 | 0.377 | 0.571 | 0.361 | 17.05 | |
| | E3Diff | GRSL'25 | 104.7 | 0.232 | 0.529 | 0.249 | 16.09 | |
| | C-DiffSET | TCSVT'26 | 78.1 | 0.214 † | 0.529 | 0.314 | 16.81 | |
| | *Ours* | | | | | | | |
| | **ReFlowSET (ours)** | — | 66.3 | 0.185 | 0.522 | 0.287 | 16.06 | |
| |
| ⚠ **fails the input-copy audit** — see *Identity collapse* below. |
| † **these two DISTS values were re-measured.** The image dumps for these two |
| SAR2Opt cells were regenerated after their last extended-metrics pass, and that |
| pass's resume check keys on the number of images rather than on their content, so |
| it skipped them and their stored DISTS went stale. Both were re-measured on the |
| current dumps at full n=627 and the table prints the re-measurement (SD2.1-FT |
| 0.212 -> **0.211**; C-DiffSET 0.2141 -> **0.2136**, unchanged at three decimals). |
| The harness was validated on a cell that is *not* stale, whose DISTS it |
| reproduces to 6e-6. FID, LPIPS, SSIM and PSNR come from a pass that rescores |
| every cell on every run and were never affected. |
| |
| ## Read before you compare these numbers with a paper |
| |
| **These numbers are not comparable with the source papers, and are not meant to |
| be.** Three reasons, all deliberate: |
| |
| 1. **Budgets are matched across the benchmark, not to each method's own |
| recipe.** Several methods are trained here at a fraction of their published |
| schedule — HI-Diff, for instance, is trained for 50,000 updates against an |
| upstream recipe of 600,000, with progressive resizing disabled. Where we |
| departed from a published recipe, the method's own card says so and says by |
| how much. |
| 2. **Budgets are quoted in optimizer updates, never epochs.** The two datasets |
| differ by more than tenfold in size (16,001 versus 1,450 training pairs), so |
| an epoch count means something completely different on each. Comparing |
| unpaired methods by epochs produced two successive wrong conclusions in our |
| own work before we changed the unit. |
| 3. **One evaluation protocol for everyone.** Same test items, same resolution, |
| same metric implementations, same ground truth. SAR2Opt is **centre-cropped** |
| to 512 px and never resized, in training and in evaluation alike. |
| |
| ### The LPIPS convention |
| |
| LPIPS here is **LPIPS-VGG on inputs mapped to `[-1, 1]`** before the network — |
| `lpips_vgg(x * 2 - 1, y * 2 - 1)` with the library's default `normalize=False`. |
| The other convention, which several released evaluators use, hands the network |
| `[0, 1]` inputs *unscaled* (still `normalize=False`) and produces a |
| **systematically lower number** — about 0.05 lower — on the same images. Note |
| that `[0, 1]` inputs with `normalize=True` is the *same* thing as ours, not the |
| other one; only the flag differs. Both are called "LPIPS" in the literature. If you compare a number from |
| this table against one from a paper, check which convention that paper used |
| first; otherwise the comparison is meaningless. The same care applies in reverse |
| to anyone quoting these numbers. |
|
|
| PSNR and SSIM are per-image torchmetrics with `data_range=1`. FID is |
| `pytorch-fid` against the size-matched ground-truth test set. DISTS is the |
| standard implementation on the same pairs. Where the generated size and the |
| ground-truth size differ, the ground truth is centre-cropped to match. |
|
|
| ### Identity collapse |
|
|
| **Four of the thirty cells fail an input-copy audit**: CycleGAN and StegoGAN, on |
| both datasets. Their outputs are closer to the SAR image they were given than to |
| the EO image they were asked for — mean |output − SAR| < mean |output − ground |
| truth| — which is the signature of a model that has learned to pass its input |
| through rather than translate it. |
|
|
| | cell | ratio (SAR distance ÷ GT distance) | verdict | |
| |---|---|---| |
| | QXS-SAROPT CycleGAN | 0.847 | COLLAPSED | |
| | QXS-SAROPT StegoGAN | 0.919 | COLLAPSED | |
| | SAR2Opt CycleGAN | 0.795 | COLLAPSED | |
| | SAR2Opt StegoGAN | 0.758 | COLLAPSED | |
|
|
| **Their metric values are reported unchanged and marked in place.** They are what |
| the released implementation does at its own published protocol, and a benchmark |
| that quietly substitutes a better-behaved variant is not reporting the method. |
| But they should not be read as translation quality. No other cell on either |
| dataset collapses — in particular DDPM, Conditional Diffusion and BBDM do **not** |
| collapse here, whatever they may do elsewhere. |
|
|
| The companion leak audit is clean: no cell on either dataset shows a generation |
| that matches its own ground truth anomalously better than it matches other |
| images' ground truth. Per-cell numbers are on each method's card. |
|
|
| **One oracle you must not use.** StegoGAN's `fake_B` output is computed as |
| `netG_A(real_A, latent(real_B))` — conditioned on the ground truth, and |
| impossible to produce at deployment. Only `fake_B_clean = netG_A(real_A)` is a |
| prediction, and it is what the StegoGAN rows above report. Any published StegoGAN |
| number should say which of the two it is. |
|
|
| ## Layout |
|
|
| ``` |
| <dataset>/<method>/<weight files> + README.md |
| dataset ∈ { qxs-saropt, sar2opt } |
| licenses/ third-party licence texts, one per method |
| ``` |
|
|
| Every one of the thirty folders carries its own `README.md` with that cell's |
| architecture, the file-by-file contents, the training budget in optimizer |
| updates, the measured metrics, a copy-pasteable load-and-translate snippet, the |
| traps specific to that method, and its licence stated factually. |
|
|
| | method | folder | files | upstream code | |
| |---|---|---|---| |
| | pix2pix | `pix2pix/` | `net_G.pth` | `junyanz/pytorch-CycleGAN-and-pix2pix` | |
| | CycleGAN | `cyclegan/` | `net_G_A.pth` (**SAR→EO**), `net_G_B.pth` | `junyanz/pytorch-CycleGAN-and-pix2pix` | |
| | pix2pixHD | `p2phd/` | `net_G.pth` | `NVIDIA/pix2pixHD` | |
| | SPADE | `spade/` | `net_G.pth` | `NVlabs/SPADE` | |
| | DDPM (SR3-class) | `ddpm/` | `gen.pth` | `DeepSARRS/E3Diff`, stage 1 | |
| | SD2.1 fine-tune only | `sd21ft/` | `diffusion_pytorch_model.safetensors`, `config.json` | `KAIST-VICLab/C-DiffSET`, stage 1, on SD 2.1-base | |
| | BBDM | `bbdm/` | `last_model.pth` | `xuekt98/BBDM` | |
| | ControlNet | `controlnet/` | `diffusion_pytorch_model.safetensors`, `config.json` | `huggingface/diffusers` `train_controlnet.py`, on SD 2.1-base | |
| | HI-Diff | `hidiff/` | `S1_*.pth` ×2, `S2_*.pth` ×3 (**only the three `S2_` files are needed at inference**) | `zhengchen1999/HI-Diff` | |
| | ResShift | `resshift/` | `ema_model.pth` | `zsyOAOA/ResShift` | |
| | StegoGAN | `stegogan/` | `net_G_A.pth` (**SAR→EO**), `net_G_B.pth` | `sian-wusidi/StegoGAN` | |
| | Conditional Diffusion | `conddiff/` | `ema_final.pt` | `Coordi777/Conditional-Diffusion-for-SAR-to-Optical-Image-Translation` | |
| | cBBDM | `cbbdm/` | `last_model.pth` | `egshkim/ConditionalBBDM-for-VHR-SAR-to-Optical` | |
| | E3Diff | `e3diff/` | `gen.pth` | `DeepSARRS/E3Diff`, stage 2 | |
| | C-DiffSET | `cdiffset/` | `diffusion_pytorch_model.safetensors`, `config.json` | `KAIST-VICLab/C-DiffSET`, stage 2 | |
| |
| ### Three checkpoints these weights need and this repository does not carry |
| |
| * **BBDM and cBBDM** need the CompVis latent-diffusion **vq-f4 VQGAN** |
| (`model.ckpt`, 756,175,527 bytes) at the path named in their config — at |
| construction time, even though the same weights are also inside |
| `last_model.pth`. MIT. |
| * **ResShift** needs the CompVis **vq-f4 autoencoder** (221,364,711 bytes). MIT. |
| * **SD2.1-FT, ControlNet and C-DiffSET** need Stable Diffusion 2.1-base for the |
| frozen VAE, text encoder, tokenizer and scheduler. Their cards name the exact |
| repository we loaded. |
| * **DDPM and E3Diff** need no extra checkpoint, but do need **PPB-despeckled and |
| Canny condition images** computed from each SAR input; neither can be run from |
| a SAR file alone. |
|
|
| ## Licences |
|
|
| **A single repository-wide licence tag would be wrong for this collection.** The |
| fifteen methods fall into three tiers, and the restrictive terms in tiers B and C |
| travel with the individual cells. Each method's card states its position in full; |
| the licence texts are in `licenses/`. |
|
|
| | method | upstream licence | redistribution | what propagates | |
| |---|---|---|---| |
| | pix2pix | multi-part BSD (CycleGAN / pix2pix / DCGAN notices) | ✅ with the notice | no-endorsement (DCGAN part) | |
| | CycleGAN | same file as pix2pix | ✅ with the notice | same | |
| | pix2pixHD | BSD (NVIDIA 2019) + bundled CycleGAN notice | ✅ with both notices | none | |
| | **SPADE** | **CC BY-NC-SA 4.0** | ⚠️ **non-commercial only** | **NonCommercial + ShareAlike** — this cell is itself distributed under CC BY-NC-SA 4.0 | |
| | **DDPM (SR3-class)** | **none declared** | ⚠️ **no express grant** — published by our decision | unknown; there is no licence to read | |
| | SD2.1 fine-tune only | MIT code (C-DiffSET) / **Open RAIL++-M** weights | ✅ under Open RAIL++-M | **Attachment A use restrictions** | |
| | BBDM | MIT | ✅ | none (needs the external vq-f4 VQGAN, MIT) | |
| | ControlNet | Apache-2.0 code (diffusers) / **Open RAIL++-M** weights | ✅ under Open RAIL++-M | **Attachment A use restrictions** | |
| | HI-Diff | Apache-2.0 | ✅ | ship the licence, keep notices, state your modifications | |
| | **ResShift** | **S-Lab License 1.0** | ⚠️ **non-commercial only** | **NonCommercial**; no-endorsement | |
| | **StegoGAN** | **none declared** | ⚠️ **no express grant** — published by our decision | unknown | |
| | **Conditional Diffusion** | **none declared** | ⚠️ **no express grant** — published by our decision | unknown (the guided-diffusion parts it forks are MIT) | |
| | cBBDM | MIT | ✅ | none (needs the external vq-f4 VQGAN, MIT) | |
| | **E3Diff** | **none declared** | ⚠️ **no express grant** — published by our decision | unknown | |
| | C-DiffSET | MIT code / **Open RAIL++-M** weights | ✅ under Open RAIL++-M | **Attachment A use restrictions** | |
|
|
| ### The four with no upstream licence |
|
|
| **StegoGAN, E3Diff, DDPM (SR3-class) and Conditional Diffusion are derived from |
| upstream code that publishes no licence at all** — no LICENSE, LICENCE, COPYING |
| or NOTICE file, no licence section in the README, and no licence declared to |
| GitHub. Under default copyright, all rights remain reserved by those authors and |
| **no express permission to redistribute derived work has been granted** to us or |
| to you. |
|
|
| We publish these four anyway, so that the benchmark is reproducible end to end, |
| and we say so plainly rather than implying a permission that does not exist. If |
| you intend to redistribute them or build on them, assess that for yourself, and |
| consider asking the upstream authors directly. |
| `licenses/NO-UPSTREAM-LICENSE.md` records, per method, exactly what we checked |
| and exactly what we found. |
|
|
| ### Files in `licenses/` |
|
|
| `LICENSE-pix2pix.txt`, `LICENSE-cyclegan.txt`, `LICENSE-p2phd.txt`, |
| `LICENSE-spade.txt`, `LICENSE-bbdm.txt`, `LICENSE-cbbdm.txt`, |
| `LICENSE-hidiff.txt`, `LICENSE-resshift.txt`, `LICENSE-cdiffset.txt`, |
| `LICENSE-sd21ft.txt` — one per method, copied verbatim from the upstream |
| repository we trained with. `LICENSE-SoftPool-MIT.txt` is a vendored dependency |
| of the E3Diff code base and is included because it is genuinely part of what |
| those two cells were trained with; it is **not** a licence for E3Diff. |
| `NO-UPSTREAM-LICENSE.md` covers the four methods with no grant. |
| `LICENSE-WEIGHTS.md` at the repository root is the CreativeML Open RAIL++-M |
| text, including Attachment A, which governs the three Stable-Diffusion-derived |
| cells. |
|
|
| ## Datasets |
|
|
| Neither corpus is redistributed here, in any form, and neither may be |
| redistributed from here. |
|
|
| * **QXS-SAROPT** publishes no licence file; its one stated term is a citation |
| requirement — *"This paper must be cited when the dataset is used for research |
| purposes"* (Huang et al., [arXiv:2103.08259](https://arxiv.org/abs/2103.08259)). |
| Its distribution is request-gated by the authors. Obtain it from them. |
| * **SAR2Opt** ships an MIT LICENSE, `Copyright (c) 2021 MarsZhaoYT`. The MIT |
| text grants rights in "the Software"; the repository does not name the imagery |
| sources, so that file is best read as covering the authors' packaging and code. |
| Citation requested: Zhao, Celik, Liu and Li, *A Comparative Analysis of |
| GAN-based Methods for SAR-to-Optical Image Translation*, IEEE GRSL, 2022. |
| * **The optical side of both datasets is Google-Earth-derived.** Copyright in |
| that imagery is held by third parties, not by the dataset authors and not by |
| us. No such imagery appears in this repository. |
|
|
| ## A note on two venue labels |
|
|
| The table above prints the venue strings as they appear in the ReFlowSET paper. |
| Two of them do not match the year in the upstream repositories' own citation |
| blocks, and both refer to the same paper under an early-access versus issue year: |
|
|
| * **Conditional Diffusion** — the authors' bibtex says **2023**, |
| [doi:10.1109/LGRS.2023.3337143](https://doi.org/10.1109/LGRS.2023.3337143). |
| * **E3Diff** — the authors' bibtex says **2024**, |
| [doi:10.1109/LGRS.2024.3506566](https://doi.org/10.1109/LGRS.2024.3506566). |
|
|
| Cite by DOI. |
|
|
| ## Citation |
|
|
| If you use this benchmark or any of these weights, please cite ReFlowSET: |
|
|
| ```bibtex |
| @article{do2026reflowset, |
| title = {ReFlowSET: Representation-Aligned Latent Flow Matching for SAR-to-EO Image Translation}, |
| author = {Jeonghyeok Do<sup>1</sup>, Seungchul Lee<sup>2</sup>, Munchurl Kim<sup>1*</sup>}, |
| journal = {arXiv preprint arXiv:2609.00968}, |
| year = {2026} |
| } |
| ``` |
|
|
| **And please cite each method's original paper** — every model card carries the |
| reference — **and both dataset papers.** The QXS-SAROPT citation is required by |
| that dataset's own terms. |
|
|
