| --- |
| license: apache-2.0 |
| --- |
| |
| This is a LFS clone of [github.com/zhengchen1999/DAT](https://github.com/zhengchen1999/DAT/tree/6846c8798e4a0579451982bd8b6e441f4b1e78b0) with uploaded pretrained models.<br> |
| All credit goes through original Author.<br> |
| The purpose of this clone is to allow easier automated download of the pretrained models for [github.com/AUTOMATIC1111/stable-diffusion-webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui).<br> |
| Note the commit hash is different from the origin repo as the history has been rewritten as a result of converting to LFS. |
|
|
| --- |
|
|
| # Dual Aggregation Transformer for Image Super-Resolution |
|
|
| [Zheng Chen](https://zhengchen1999.github.io/), [Yulun Zhang](http://yulunzhang.com/), [Jinjin Gu](https://www.jasongt.com/), [Linghe Kong](https://www.cs.sjtu.edu.cn/~linghe.kong/), [Xiaokang Yang](https://scholar.google.com/citations?user=yDEavdMAAAAJ&hl), and [Fisher Yu](https://www.yf.io/), "Dual Aggregation Transformer for Image Super-Resolution", ICCV, 2023 |
|
|
| [[paper](https://openaccess.thecvf.com/content/ICCV2023/papers/Chen_Dual_Aggregation_Transformer_for_Image_Super-Resolution_ICCV_2023_paper.pdf)] [[arXiv](http://arxiv.org/abs/2308.03364)] [[supplementary material](https://github.com/zhengchen1999/DAT/releases)] [[visual results](https://drive.google.com/drive/folders/1ZMaZyCer44ZX6tdcDmjIrc_hSsKoMKg2?usp=drive_link)] [[pretrained models](https://drive.google.com/drive/folders/1iBdf_-LVZuz_PAbFtuxSKd_11RL1YKxM?usp=drive_link)] |
|
|
| #### 🔥🔥🔥 News |
|
|
| - **2023-09-17:** The [chaiNNer](https://github.com/chaiNNer-org/chaiNNer) and the [neosr](https://github.com/muslll/neosr) add DAT support. Additional trained DAT models are available in [OpenMMLab](https://openmodeldb.info/?sort=date-desc&t=arch%3Adat) ([#11](https://github.com/zhengchen1999/DAT/issues/11)). Thank [Phhofm](https://github.com/Phhofm)! |
| - **2023-07-16:** This repo is released. |
| - **2023-07-14:** DAT is accepted at ICCV 2023. 🎉🎉🎉 |
|
|
| --- |
|
|
| > **Abstract:** *Transformer has recently gained considerable popularity in low-level vision tasks, including image super-resolution (SR). These networks utilize self-attention along different dimensions, spatial or channel, and achieve impressive performance. This inspires us to combine the two dimensions in Transformer for a more powerful representation capability. Based on the above idea, we propose a novel Transformer model, Dual Aggregation Transformer (DAT), for image SR. Our DAT aggregates features across spatial and channel dimensions, in the inter-block and intra-block dual manner. Specifically, we alternately apply spatial and channel self-attention in consecutive Transformer blocks. The alternate strategy enables DAT to capture the global context and realize inter-block feature aggregation. Furthermore, we propose the adaptive interaction module (AIM) and the spatial-gate feed-forward network (SGFN) to achieve intra-block feature aggregation. AIM complements two self-attention mechanisms from corresponding dimensions. Meanwhile, SGFN introduces additional non-linear spatial information in the feed-forward network. Extensive experiments show that our DAT surpasses current methods.* |
|
|
|  |
|
|
| --- |
|
|
| | HR | LR | [SwinIR](https://github.com/JingyunLiang/SwinIR) | [CAT](https://github.com/zhengchen1999/CAT) | DAT (ours) | |
| | :------------------------------------------: | :-----------------------------------------------: | :----------------------------------------------: | :-------------------------------------------: | :-------------------------------------------: | |
| | <img src="figs/img_059_HR_x4.png" height=80> | <img src="figs/img_059_Bicubic_x4.png" height=80> | <img src="figs/img_059_SwinIR_x4.png" height=80> | <img src="figs/img_059_CAT_x4.png" height=80> | <img src="figs/img_059_DAT_x4.png" height=80> | |
| | <img src="figs/img_049_HR_x4.png" height=80> | <img src="figs/img_049_Bicubic_x4.png" height=80> | <img src="figs/img_049_SwinIR_x4.png" height=80> | <img src="figs/img_049_CAT_x4.png" height=80> | <img src="figs/img_049_DAT_x4.png" height=80> | |
|
|
| ## Dependencies |
|
|
| - Python 3.8 |
| - PyTorch 1.8.0 |
| - NVIDIA GPU + [CUDA](https://developer.nvidia.com/cuda-downloads) |
|
|
| ```bash |
| # Clone the github repo and go to the default directory 'DAT'. |
| git clone https://github.com/zhengchen1999/DAT.git |
| conda create -n DAT python=3.8 |
| conda activate DAT |
| pip install -r requirements.txt |
| python setup.py develop |
| ``` |
|
|
| ## Contents |
|
|
| 1. [Datasets](#Datasets) |
| 1. [Models](#Models) |
| 1. [Training](#Training) |
| 1. [Testing](#Testing) |
| 1. [Results](#Results) |
| 1. [Citation](#Citation) |
| 1. [Acknowledgements](#Acknowledgements) |
|
|
| --- |
|
|
| ## Datasets |
|
|
| Used training and testing sets can be downloaded as follows: |
|
|
| | Training Set | Testing Set | Visual Results | |
| | :----------------------------------------------------------- | :----------------------------------------------------------: | :----------------------------------------------------------: | |
| | [DIV2K](https://data.vision.ee.ethz.ch/cvl/DIV2K/) (800 training images, 100 validation images) + [Flickr2K](https://cv.snu.ac.kr/research/EDSR/Flickr2K.tar) (2650 images) [complete training dataset DF2K: [Google Drive](https://drive.google.com/file/d/1TubDkirxl4qAWelfOnpwaSKoj3KLAIG4/view?usp=share_link) / [Baidu Disk](https://pan.baidu.com/s/1KIcPNz3qDsGSM0uDKl4DRw?pwd=74yc)] | Set5 + Set14 + BSD100 + Urban100 + Manga109 [complete testing dataset: [Google Drive](https://drive.google.com/file/d/1yMbItvFKVaCT93yPWmlP3883XtJ-wSee/view?usp=sharing) / [Baidu Disk](https://pan.baidu.com/s/1Tf8WT14vhlA49TO2lz3Y1Q?pwd=8xen)] | [Google Drive](https://drive.google.com/drive/folders/1ZMaZyCer44ZX6tdcDmjIrc_hSsKoMKg2?usp=drive_link) / [Baidu Disk](https://pan.baidu.com/s/1LO-INqy40F5T_coAJsl5qw?pwd=dqnv#list/path=%2F) | |
|
|
| Download training and testing datasets and put them into the corresponding folders of `datasets/`. See [datasets](datasets/README.md) for the detail of the directory structure. |
|
|
| ## Models |
|
|
| | Method | Params | FLOPs (G) | Dataset | PSNR (dB) | SSIM | Model Zoo | Visual Results | |
| | :-------- | :----: | :-------: | :------: | :-------: | :----: | :----------------------------------------------------------: | :----------------------------------------------------------: | |
| | DAT-S | 11.21M | 203.34 | Urban100 | 27.68 | 0.8300 | [Google Drive](https://drive.google.com/drive/folders/1hM0v3fUg5u6GjkI7dduxShyGgGfEwQXO?usp=drive_link) / [Baidu Disk](https://pan.baidu.com/s/1rgkCyqEJdZlHvQ6_Dwb3rA?pwd=4rfr) | [Google Drive](https://drive.google.com/file/d/1x1ixMswxw5w-zeZ_Rap5Nk4Tr46MIjAw/view?usp=drive_link) / [Baidu Disk](https://pan.baidu.com/s/1LO-INqy40F5T_coAJsl5qw?pwd=dqnv) | |
| | DAT | 14.80M | 275.75 | Urban100 | 27.87 | 0.8343 | [Google Drive](https://drive.google.com/drive/folders/14VG5mw5ie8RrR4jjypeHynXDZYWL8w-r?usp=drive_link) / [Baidu Disk](https://pan.baidu.com/s/1rgkCyqEJdZlHvQ6_Dwb3rA?pwd=4rfr) | [Google Drive](https://drive.google.com/file/d/1K43CTsXpoX5St5fed4kEW9gu2KMR6hLu/view?usp=drive_link) / [Baidu Disk](https://pan.baidu.com/s/1LO-INqy40F5T_coAJsl5qw?pwd=dqnv) | |
| | DAT-2 | 11.21M | 216.93 | Urban100 | 27.86 | 0.8341 | [Google Drive](https://drive.google.com/drive/folders/1yV9LMhr2tYM_eHEIVY4Jw9X3bWGgorbD?usp=drive_link) / [Baidu Disk](https://pan.baidu.com/s/1rgkCyqEJdZlHvQ6_Dwb3rA?pwd=4rfr) | [Google Drive](https://drive.google.com/file/d/1TQRZIg8at5HX87OCu3GYytZhYGperkuN/view?usp=drive_link) / [Baidu Disk](https://pan.baidu.com/s/1LO-INqy40F5T_coAJsl5qw?pwd=dqnv) | |
| | DAT-light | 573K | 49.69 | Urban100 | 26.64 | 0.8033 | [Google Drive](https://drive.google.com/drive/folders/105JRMN5VJbJ7EMQJdqmhDVMAFCaKYDl8?usp=drive_link) / [Baidu Disk](https://pan.baidu.com/s/1rgkCyqEJdZlHvQ6_Dwb3rA?pwd=4rfr) | [Google Drive](https://drive.google.com/file/d/1xKxK6_UcqAWK2m5znQX_LssWndmN-End/view?usp=drive_link) / [Baidu Disk](https://pan.baidu.com/s/1LO-INqy40F5T_coAJsl5qw?pwd=dqnv) | |
|
|
| The performance is reported on Urban100 (x4). DAT-S, DAT, DAT-2: output size of FLOPs is 3×512×512. DAT-light: output size of FLOPs is 3×1280×720. |
|
|
| ## Training |
|
|
| - Download [training](https://drive.google.com/file/d/1TubDkirxl4qAWelfOnpwaSKoj3KLAIG4/view?usp=share_link) (DF2K, already processed) and [testing](https://drive.google.com/file/d/1yMbItvFKVaCT93yPWmlP3883XtJ-wSee/view?usp=sharing) (Set5, Set14, BSD100, Urban100, Manga109, already processed) datasets, place them in `datasets/`. |
|
|
| - Run the following scripts. The training configuration is in `options/train/`. |
|
|
| ```shell |
| # DAT-S, input=64x64, 4 GPUs |
| python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 basicsr/train.py -opt options/Train/train_DAT_S_x2.yml --launcher pytorch |
| python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 basicsr/train.py -opt options/Train/train_DAT_S_x3.yml --launcher pytorch |
| python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 basicsr/train.py -opt options/Train/train_DAT_S_x4.yml --launcher pytorch |
| |
| # DAT, input=64x64, 4 GPUs |
| python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 basicsr/train.py -opt options/Train/train_DAT_x2.yml --launcher pytorch |
| python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 basicsr/train.py -opt options/Train/train_DAT_x3.yml --launcher pytorch |
| python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 basicsr/train.py -opt options/Train/train_DAT_x4.yml --launcher pytorch |
| |
| # DAT-2, input=64x64, 4 GPUs |
| python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 basicsr/train.py -opt options/Train/train_DAT_2_x2.yml --launcher pytorch |
| python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 basicsr/train.py -opt options/Train/train_DAT_2_x3.yml --launcher pytorch |
| python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 basicsr/train.py -opt options/Train/train_DAT_2_x4.yml --launcher pytorch |
| |
| # DAT-light, input=64x64, 4 GPUs |
| python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 basicsr/train.py -opt options/Train/train_DAT_light_x2.yml --launcher pytorch |
| python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 basicsr/train.py -opt options/Train/train_DAT_light_x3.yml --launcher pytorch |
| python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 basicsr/train.py -opt options/Train/train_DAT_light_x4.yml --launcher pytorch |
| ``` |
|
|
| - The training experiment is in `experiments/`. |
|
|
| ## Testing |
|
|
| ### Test images with HR |
|
|
| - Download the pre-trained [models](https://drive.google.com/drive/folders/1iBdf_-LVZuz_PAbFtuxSKd_11RL1YKxM?usp=drive_link) and place them in `experiments/pretrained_models/`. |
|
|
| We provide pre-trained models for image SR: DAT-S, DAT, DAT-2, and DAT-light (x2, x3, x4). |
|
|
| - Download [testing](https://drive.google.com/file/d/1yMbItvFKVaCT93yPWmlP3883XtJ-wSee/view?usp=sharing) (Set5, Set14, BSD100, Urban100, Manga109) datasets, place them in `datasets/`. |
|
|
| - Run the following scripts. The testing configuration is in `options/test/` (e.g., [test_DAT_x2.yml](options/Test/test_DAT_x2.yml)). |
|
|
| Note 1: You can set `use_chop: True` (default: False) in YML to chop the image for testing. |
|
|
| ```shell |
| # No self-ensemble |
| # DAT-S, reproduces results in Table 2 of the main paper |
| python basicsr/test.py -opt options/Test/test_DAT_S_x2.yml |
| python basicsr/test.py -opt options/Test/test_DAT_S_x3.yml |
| python basicsr/test.py -opt options/Test/test_DAT_S_x4.yml |
| |
| # DAT, reproduces results in Table 2 of the main paper |
| python basicsr/test.py -opt options/Test/test_DAT_x2.yml |
| python basicsr/test.py -opt options/Test/test_DAT_x3.yml |
| python basicsr/test.py -opt options/Test/test_DAT_x4.yml |
| |
| # DAT-2, reproduces results in Table 1 of the supplementary material |
| python basicsr/test.py -opt options/Test/test_DAT_2_x2.yml |
| python basicsr/test.py -opt options/Test/test_DAT_2_x3.yml |
| python basicsr/test.py -opt options/Test/test_DAT_2_x4.yml |
| |
| # DAT-light, reproduces results in Table 2 of the supplementary material |
| python basicsr/test.py -opt options/Test/test_DAT_light_x2.yml |
| python basicsr/test.py -opt options/Test/test_DAT_light_x3.yml |
| python basicsr/test.py -opt options/Test/test_DAT_light_x4.yml |
| ``` |
|
|
| - The output is in `results/`. |
|
|
| ### Test images without HR |
|
|
| - Download the pre-trained [models](https://drive.google.com/drive/folders/1iBdf_-LVZuz_PAbFtuxSKd_11RL1YKxM?usp=drive_link) and place them in `experiments/pretrained_models/`. |
|
|
| We provide pre-trained models for image SR: DAT-S, DAT, and DAT-2 (x2, x3, x4). |
|
|
| - Put your dataset (single LR images) in `datasets/single`. Some test images are in this folder. |
|
|
| - Run the following scripts. The testing configuration is in `options/test/` (e.g., [test_single_x2.yml](options/Test/test_single_x2.yml)). |
|
|
| Note 1: The default model is DAT. You can use other models like DAT-S by modifying the YML. |
|
|
| Note 2: You can set `use_chop: True` (default: False) in YML to chop the image for testing. |
|
|
| ```shell |
| # Test on your dataset |
| python basicsr/test.py -opt options/Test/test_single_x2.yml |
| python basicsr/test.py -opt options/Test/test_single_x3.yml |
| python basicsr/test.py -opt options/Test/test_single_x4.yml |
| ``` |
|
|
| - The output is in `results/`. |
|
|
| ## Results |
|
|
| We achieved state-of-the-art performance. Detailed results can be found in the paper. All visual results of DAT can be downloaded [here](https://drive.google.com/drive/folders/1ZMaZyCer44ZX6tdcDmjIrc_hSsKoMKg2?usp=drive_link). |
|
|
| <details> |
| <summary>Click to expand</summary> |
|
|
| - results in Table 2 of the main paper |
|
|
| <p align="center"> |
| <img width="900" src="figs/Table-1.png"> |
| </p> |
|
|
|
|
| - results in Table 1 of the supplementary material |
|
|
| <p align="center"> |
| <img width="900" src="figs/Table-2.png"> |
| </p> |
|
|
|
|
| - results in Table 2 of the supplementary material |
|
|
| <p align="center"> |
| <img width="900" src="figs/Table-3.png"> |
| </p> |
|
|
|
|
|
|
|
|
| - visual comparison (x4) in the main paper |
|
|
| <p align="center"> |
| <img width="900" src="figs/Figure-1.png"> |
| </p> |
|
|
|
|
| - visual comparison (x4) in the supplementary material |
|
|
| <p align="center"> |
| <img width="900" src="figs/Figure-2.png"> |
| <img width="900" src="figs/Figure-3.png"> |
| <img width="900" src="figs/Figure-4.png"> |
| <img width="900" src="figs/Figure-5.png"> |
| </p> |
| </details> |
|
|
| ## Citation |
|
|
| If you find the code helpful in your research or work, please cite the following paper(s). |
|
|
| ``` |
| @inproceedings{chen2023dual, |
| title={Dual Aggregation Transformer for Image Super-Resolution}, |
| author={Chen, Zheng and Zhang, Yulun and Gu, Jinjin and Kong, Linghe and Yang, Xiaokang and Yu, Fisher}, |
| booktitle={ICCV}, |
| year={2023} |
| } |
| ``` |
|
|
| ## Acknowledgements |
|
|
| This code is built on [BasicSR](https://github.com/XPixelGroup/BasicSR). |
|
|
