CLEAR / README.md
tuml2's picture
CLEAR model card, Stage-1 config and Stage-2 HFAM checkpoint
af34a88 verified
|
Raw History Blame Contribute Delete
3.8 kB
---
license: openrail++
language: en
library_name: pytorch
pipeline_tag: image-to-image
tags:
- single-image-reflection-removal
- reflection-removal
- image-restoration
- diffusion
---
# CLEAR: Color-Texture Latent Alignment for Single Image Reflection Removal
**WACV 2027** Β· [Code](https://github.com/kc-ml2/clear) Β· [Paper / OpenReview](https://openreview.net/forum?id=QMmOrvegf1)
Tu Vo, Sheir A. Zaheer, Chan Y. Park β€” KC Machine Learning Lab
CLEAR is a deterministic, single-timestep latent framework for single image reflection removal.
A physics-guided dual-branch separator predicts the transmission, reflection and residue latents
in one forward pass, and a cross-latent decoder with the **Hybrid Feature Alignment Module (HFAM)**
re-aligns the input skip features to the clean transmission before decoding. There is no noise
schedule and no iterative sampling.
## Files
```
config.yaml Stage-1 training config of the released run
checkpoint/
β”œβ”€β”€ diffusion/
β”‚ β”œβ”€β”€ unet1/ transmission branch (config.json + diffusion_pytorch_model.bin)
β”‚ └── unet2/ reflection branch (config.json + diffusion_pytorch_model.bin)
└── cld/
└── cld_clear.pth Stage-2 HFAM cross-latent decoder
```
`cld_clear.pth` is the final Stage-2 checkpoint (212 training hours). It stores the `state_dict`
plus the metadata `refiner_type: hfam`, `total_hours: 212` and `name: 0`.
## Quick start
```bash
git clone https://github.com/kc-ml2/clear.git
cd clear
pip install -r requirements.txt
python scripts/download_models.py --save_dir ./models # SDv2 + Marigold backbones
CLEAR_HF_REPO_ID=ML2/CLEAR python scripts/download_models.py --weights --save_dir ./models
```
Run on a folder of images (the folder must contain a `blended/` subfolder):
```bash
bash scripts/inference.sh \
models/clear/checkpoint/diffusion \
models/clear/checkpoint/cld/cld_clear.pth \
<DATA_DIR> outputs/inference
```
This writes `<name>_clear.png` (the reflection-free output), `<name>_r.png` (predicted
reflection), `<name>_g1.png` (residue) and the input / target copies.
## Results
Average over six real-world benchmarks (Nature, SIR2, Real20, DRR-C, DRR-S, OpenRR-1k β€”
1,040 images), as reported in the paper:
| Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | MANIQA ↑ | QUALICLIP ↑ |
|---|---|---|---|---|---|
| DAI | 25.98 | 0.869 | 0.159 | 0.460 | 0.696 |
| WindowSeat | 25.48 | 0.810 | 0.208 | 0.465 | 0.674 |
| **CLEAR (Ours)** | **26.59** | **0.906** | **0.106** | **0.504** | **0.731** |
See the paper for the per-dataset tables and ablations.
## Notes
* `config.yaml` is the original Stage-1 training config and still refers to the authors'
workspace (`base_config: ../config/default.yaml`). For inference use
`configs/stage1_clear.yaml` from the code repository.
* Stage-1 is initialised from `stabilityai/stable-diffusion-2` and
`prs-eth/marigold-depth-v1-0`; both are downloaded by `scripts/download_models.py`.
* The VAE and text encoder are not duplicated here β€” they come from the SDv2 backbone.
## License
The **code** is released under Apache-2.0. These **weights** are fine-tuned from
Stable Diffusion 2 and are distributed under the
[CreativeML Open RAIL++-M License](https://huggingface.co/stabilityai/stable-diffusion-2/blob/main/LICENSE-MODEL);
the use restrictions of that license apply.
## Citation
```bibtex
@inproceedings{vo2027clear,
title = {CLEAR: Color-Texture Latent Alignment for Single Image Reflection Removal},
author = {Vo, Tu and Zaheer, Sheir A. and Park, Chan Y.},
booktitle = {The IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
year = {2027}
}
```