File size: 3,797 Bytes
e144a20
af34a88
 
 
 
 
 
 
 
 
e144a20
af34a88
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
---
license: openrail++
language: en
library_name: pytorch
pipeline_tag: image-to-image
tags:
  - single-image-reflection-removal
  - reflection-removal
  - image-restoration
  - diffusion
---

# CLEAR: Color-Texture Latent Alignment for Single Image Reflection Removal

**WACV 2027** Β· [Code](https://github.com/kc-ml2/clear) Β· [Paper / OpenReview](https://openreview.net/forum?id=QMmOrvegf1)

Tu Vo, Sheir A. Zaheer, Chan Y. Park β€” KC Machine Learning Lab

CLEAR is a deterministic, single-timestep latent framework for single image reflection removal.
A physics-guided dual-branch separator predicts the transmission, reflection and residue latents
in one forward pass, and a cross-latent decoder with the **Hybrid Feature Alignment Module (HFAM)**
re-aligns the input skip features to the clean transmission before decoding. There is no noise
schedule and no iterative sampling.

## Files

```
config.yaml                              Stage-1 training config of the released run
checkpoint/
β”œβ”€β”€ diffusion/
β”‚   β”œβ”€β”€ unet1/                           transmission branch  (config.json + diffusion_pytorch_model.bin)
β”‚   └── unet2/                           reflection branch    (config.json + diffusion_pytorch_model.bin)
└── cld/
    └── cld_clear.pth                    Stage-2 HFAM cross-latent decoder
```

`cld_clear.pth` is the final Stage-2 checkpoint (212 training hours). It stores the `state_dict`
plus the metadata `refiner_type: hfam`, `total_hours: 212` and `name: 0`.

## Quick start

```bash
git clone https://github.com/kc-ml2/clear.git
cd clear
pip install -r requirements.txt
python scripts/download_models.py --save_dir ./models            # SDv2 + Marigold backbones
CLEAR_HF_REPO_ID=ML2/CLEAR python scripts/download_models.py --weights --save_dir ./models
```

Run on a folder of images (the folder must contain a `blended/` subfolder):

```bash
bash scripts/inference.sh \
    models/clear/checkpoint/diffusion \
    models/clear/checkpoint/cld/cld_clear.pth \
    <DATA_DIR> outputs/inference
```

This writes `<name>_clear.png` (the reflection-free output), `<name>_r.png` (predicted
reflection), `<name>_g1.png` (residue) and the input / target copies.

## Results

Average over six real-world benchmarks (Nature, SIR2, Real20, DRR-C, DRR-S, OpenRR-1k β€”
1,040 images), as reported in the paper:

| Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | MANIQA ↑ | QUALICLIP ↑ |
|---|---|---|---|---|---|
| DAI | 25.98 | 0.869 | 0.159 | 0.460 | 0.696 |
| WindowSeat | 25.48 | 0.810 | 0.208 | 0.465 | 0.674 |
| **CLEAR (Ours)** | **26.59** | **0.906** | **0.106** | **0.504** | **0.731** |

See the paper for the per-dataset tables and ablations.

## Notes

* `config.yaml` is the original Stage-1 training config and still refers to the authors'
  workspace (`base_config: ../config/default.yaml`). For inference use
  `configs/stage1_clear.yaml` from the code repository.
* Stage-1 is initialised from `stabilityai/stable-diffusion-2` and
  `prs-eth/marigold-depth-v1-0`; both are downloaded by `scripts/download_models.py`.
* The VAE and text encoder are not duplicated here β€” they come from the SDv2 backbone.

## License

The **code** is released under Apache-2.0. These **weights** are fine-tuned from
Stable Diffusion 2 and are distributed under the
[CreativeML Open RAIL++-M License](https://huggingface.co/stabilityai/stable-diffusion-2/blob/main/LICENSE-MODEL);
the use restrictions of that license apply.

## Citation

```bibtex
@inproceedings{vo2027clear,
  title     = {CLEAR: Color-Texture Latent Alignment for Single Image Reflection Removal},
  author    = {Vo, Tu and Zaheer, Sheir A. and Park, Chan Y.},
  booktitle = {The IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year      = {2027}
}
```