Histoformer: All-Weather Image Restoration

Easy-to-use mirror of Histoformer, the ECCV 2024 all-weather image restoration model (handles rain, raindrops, and snow in a single unified network) from Sun et al. This card exists to make the pretrained model simple to load and run in a few lines of Python — the original repository ships the full research codebase (BasicSR training framework, distributed-training configs, dataset generation scripts) behind CLI-only, multi-step instructions, which makes plain "just run inference" usage harder than it needs to be.

Disclaimer

This is not an official release. All credit for the method, the model, and the pretrained weights belongs entirely to the original authors: Shangquan Sun, Wenqi Ren, Xinwei Gao, Rui Wang, and Xiaochun Cao. This repository claims no contribution to the underlying research, architecture, or training — it packages the same pretrained weights the authors already released, with clearer documentation and a minimal usage path.

Why this exists, concretely — not as criticism of the original work, just the gap this card fills:

• The original repo's README documents usage as a multi-step CLI flow (cd Allweather, download weights into a specific folder structure, edit/point a YAML config, run test_histoformer.py) that assumes a full clone of the research codebase.
• The original HF repo mirrors the entire project (training scripts, BasicSR framework internals, setup.py, etc.) rather than presenting itself as a loadable model — there's no minimal "load model, run image, get output" path documented.
• We verified the actual minimal path ourselves (see Quickstart) — it turns out to be about 10 lines of plain PyTorch, no BasicSR framework or config files required for inference.

Please cite the original papers if you use this model (see Citation), and refer to the official repository for training code, or if you want the full research codebase.

---

What is Histoformer

Most transformer-based restoration methods reduce self-attention's cost by restricting it to the channel dimension or to small fixed spatial windows, which limits their ability to capture long-range spatial structure. Histoformer instead sorts and segments spatial features into intensity-based histogram bins, then applies self-attention across and within those bins — grouping similarly-degraded pixels together regardless of where they are in the image, rather than by spatial proximity. Since rain, raindrops, and snow all cause broadly similar occlusion/brightness patterns, this lets a single model handle all three degradation types without task-specific branches.

• Paper: Restoring Images in Adverse Weather Conditions via Histogram Transformer, ECCV 2024
• Params: 16,615,100 (verified by loading the checkpoint directly — see Quickstart)
• Checkpoint size: ~64 MB
• Architecture family: 4-level U-shaped Transformer encoder-decoder (same general shape as Restormer), with histogram self-attention replacing standard channel/window attention

---

Available Checkpoints

The original release ships two checkpoints, trained/fine-tuned differently — this distinction isn't clearly spelled out in the original README, so worth being explicit here:

Checkpoint	Trained on	Best for
net_g_best.pth	Synthetic all-weather composite (Outdoor-Rain + Snow100K + RainDrop)	Synthetic-style benchmarks: Test1, Snow100K-S/L, RainDrop
net_g_real.pth	Fine-tuned toward real-world photos	Real-world images, e.g. the RealSnow benchmark or your own photos

If you're not sure which to use on a real photo (not a benchmark image), start with net_g_real.pth.

---

Quickstart

import torch
from huggingface_hub import hf_hub_download
from PIL import Image
import numpy as np

# 1. Get the architecture definition (from the original repo — it's a single
#    self-contained file with no BasicSR framework dependency for inference)
#    git clone https://github.com/sunshangquan/Histoformer and add
#    `Histoformer/basicsr` to your path, or copy `histoformer_arch.py` directly.
from basicsr.models.archs.histoformer_arch import Histoformer

# 2. Build the model from the published config (the exact hyperparameters
#    used for training, from Allweather_Histoformer.yml)
import json
config_path = hf_hub_download(repo_id="dronefreak/Histoformer", filename="config.json")
config = json.load(open(config_path))
config.pop("architecture")  # not a constructor arg
model = Histoformer(**config)

# 3. Download and load a checkpoint
weights_path = hf_hub_download(repo_id="dronefreak/Histoformer", filename="net_g_real.pth")
ckpt = torch.load(weights_path, map_location="cpu", weights_only=False)
model.load_state_dict(ckpt["params"])
model.eval()

# 4. Run inference (pad to a multiple of 8 — the network downsamples 3x by /2)
img = Image.open("your_image.jpg").convert("RGB")
arr = np.array(img).astype(np.float32) / 255.0
t = torch.from_numpy(arr).permute(2, 0, 1).unsqueeze(0)

_, _, h, w = t.shape
pad_h, pad_w = (8 - h % 8) % 8, (8 - w % 8) % 8
t_padded = torch.nn.functional.pad(t, (0, pad_w, 0, pad_h), mode="reflect")

with torch.no_grad():
    out = model(t_padded)[:, :, :h, :w].clamp(0, 1)

out_img = Image.fromarray((out[0].permute(1, 2, 0).numpy() * 255).astype(np.uint8))
out_img.save("restored.jpg")

Runs on CPU (~20s for a 720×480 image on a modern desktop CPU, verified) or GPU (much faster). No BasicSR training framework, no YAML config parsing, no distributed-training setup needed for inference — just the architecture file and the checkpoint.

---

Evaluation (as reported in the original paper)

These are the authors' own reported numbers (arXiv:2407.10172, Table 1) — we have not independently reproduced them; we've only verified the model loads correctly and produces visually sensible output (see the banner above and Disclaimer). Take these as the original paper's claims, not this card's independent measurement.

Benchmark	Degradation	PSNR	SSIM
Outdoor-Rain (Test1)	Rain + fog	32.08	0.9389
RainDrop	Adherent raindrops	33.06	0.9441
Snow100K-S	Light snow	37.41	0.9656
Snow100K-L	Heavy snow	32.16	0.9261

The paper reports these as state-of-the-art among unified all-weather methods at publication time (outperforming TransWeather, WGWSNet, WeatherDiff).

---

Training Data

Histoformer is trained on a composite of independently-published datasets — the same benchmarks are used for testing:

• Outdoor-Rain — Li et al., Heavy Rain Image Restoration, CVPR 2019
• Snow100K — Liu et al., DesnowNet, TIP 2018 (arXiv:1708.04512)
• RainDrop — Qian et al., Attentive GAN for Raindrop Removal, CVPR 2018

This model card does not redistribute the training or test data — only the pretrained weights. The standard test benchmarks (Outdoor-Rain/Test1, Snow100K-S/L, RainDrop) are readily available as a single bundle from the original authors: Google Drive.

---

License

The original HF release (sunsean/Histoformer) states MIT in its model card metadata — unlike the GitHub repository, which has no LICENSE file. This mirror is distributed under the same MIT terms.

---

Citation

If you use this model, please cite the original work:

@article{sun2024restoring,
  title={Restoring Images in Adverse Weather Conditions via Histogram Transformer},
  author={Sun, Shangquan and Ren, Wenqi and Gao, Xinwei and Wang, Rui and Cao, Xiaochun},
  journal={arXiv preprint arXiv:2407.10172},
  year={2024}
}

@InProceedings{10.1007/978-3-031-72670-5_7,
    author="Sun, Shangquan and Ren, Wenqi and Gao, Xinwei and Wang, Rui and Cao, Xiaochun",
    title="Restoring Images in Adverse Weather Conditions via Histogram Transformer",
    booktitle="Computer Vision -- ECCV 2024",
    year="2025",
    publisher="Springer Nature Switzerland",
    pages="111--129",
    isbn="978-3-031-72670-5"
}

---

Acknowledgements

We sincerely thank Shangquan Sun, Wenqi Ren, Xinwei Gao, Rui Wang, and Xiaochun Cao for developing Histoformer and publicly releasing the pretrained weights under a permissive license.