| --- |
| license: apache-2.0 |
| pipeline_tag: image-to-image |
| --- |
| |
| # ODTSR: One-Step Diffusion Transformer for Controllable Real-World Image Super-Resolution |
|
|
| This repository contains the official implementation of **ODTSR**, a model presented in the paper: |
| [**One-Step Diffusion Transformer for Controllable Real-World Image Super-Resolution**](https://huggingface.co/papers/2511.17138) |
|
|
| **Authors**: Yushun Fang, Yuxiang Chen, Shibo Yin, Qiang Hu, Jiangchao Yao, Ya Zhang, Xiaoyun Zhang, Yanfeng Wang |
| **Affiliations**: Shanghai Jiao Tong University, Xiaohongshu Inc |
|
|
| **Code**: [https://github.com/RedMediaTech/ODTSR](https://github.com/RedMediaTech/ODTSR) |
|
|
| <div align="center"> |
| <img src="https://github.com/RedMediaTech/ODTSR/raw/main/static/1.png" alt="ODTSR Overview Framework" width="80%"> |
| </div> |
|
|
| ## Overview |
| Recent advances in diffusion-based real-world image super-resolution (Real-ISR) have demonstrated remarkable perceptual quality, yet the balance between fidelity and controllability remains a problem: multi-step diffusion-based methods suffer from generative diversity and randomness, resulting in low fidelity, while one-step methods lose control flexibility due to fidelity-specific finetuning. |
|
|
| **ODTSR** addresses this by presenting a one-step diffusion transformer based on Qwen-Image that performs Real-ISR considering fidelity and controllability simultaneously. It introduces a newly designed **Noise-hybrid Visual Stream (NVS)** that receives low-quality images with adjustable noise (Control Noise) and consistent noise (Prior Noise). Furthermore, **Fidelity-aware Adversarial Training (FAA)** is employed to enhance controllability and achieve one-step inference. ODTSR not only achieves state-of-the-art (SOTA) performance on generic Real-ISR, but also enables prompt controllability on challenging scenarios such as real-world scene text image super-resolution (STISR) of Chinese characters without training on specific datasets. |
|
|
| ## Key Features |
| * **One-Step Super-Resolution**: Based on Qwen-Image, ODTSR trains a single-step SR model using LoRA, with model parameters reaching 20B. |
| * **Controllability**: With our proposed Noise-hybrid Visual Stream and Fidelity-aware Adversarial Training, the SR process can be jointly controlled by prompts as well as a Fidelity Weight $f$. |
| * **Multilingual Support**: English and Chinese prompts are supported. |
| * **Versatile Performance**: The model demonstrates strong performance in text images, fine-grained textures, and face images. |
|
|
| ## Visual Results |
|
|
| ### Results with fixed prompts & high fidelity |
| <div align="center"> |
| <img src="https://github.com/RedMediaTech/ODTSR/raw/main/static/4.jpeg" alt="Results with fixed prompts & high fidelity" width="80%"> |
| </div> |
| Under the high-fidelity setting with a fixed prompt, our model produces restorations that adhere more closely to the LQ input while remaining natural, significantly reducing the sense of AI processing. |
|
|
| ### Text Real-ISR Results |
| <div align="center"> |
| <img src="https://github.com/RedMediaTech/ODTSR/raw/main/static/2.jpeg" alt="Text Real-ISR Results" width="80%"> |
| </div> |
| In text scenarios, when the prompt specifies the text to be restored, the model automatically matches the LQ text and performs the restoration. |
|
|
| ### Controllable Real-ISR Results |
| <div align="center"> |
| <img src="https://github.com/RedMediaTech/ODTSR/raw/main/static/3.jpeg" alt="Controllable Real-ISR Results" width="80%"> |
| </div> |
| Qualitative results of controllable SR with prompt and adjustable Fidelity Weight (denoted as $f$) on Div2k-val dataset. As $f$ decreases from 1 to 0, detail generation and prompt adherence gradually strengthen. |
|
|
| ## Dependencies and Installation |
|
|
| 1. Prepare conda env: |
| ```bash |
| conda create -n yourenv python=3.11 |
| ``` |
| 2. Install `pytorch` (we recommend `torch==2.6.0`): |
| ```bash |
| pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 -f https://mirrors.aliyun.com/pytorch-wheels/cu124/ |
| ``` |
| 3. Install this repo (based on [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio/tree/main)). The required packages will be automatically installed: |
| ```bash |
| cd xxxx/ODTSR # Replace xxxx with your path |
| pip3 install -e . -v -i https://mirrors.cloud.tencent.com/pypi/simple |
| ``` |
| 4. (For training) Install `basicsr`: |
| ```bash |
| pip install basicsr |
| ``` |
| Note: |
| You can apply the the following command to fix a bug in `basicsr`. Make sure to replace `/opt/conda` with the path to your own conda environment: |
| ```bash |
| sed -i '8s/from torchvision.transforms.functional_tensor import rgb_to_grayscale/from torchvision.transforms._functional_tensor import rgb_to_grayscale/' /opt/conda/lib/python3.11/site-packages/basicsr/data/degradations.py |
| ``` |
| 5. Download base model to your disk: [Qwen-Image](https://huggingface.co/Qwen/Qwen-Image/tree/main) |
| 6. (For training) Download base model to your disk: [Wan2.1-T2V-1.3B](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B/tree/main) |
| 7. (For inference) Download the trained ODTSR model weight: [huggingface](https://huggingface.co/double8fun/ODTSR/tree/main) |
| |
| ## Inference with Script |
| Note: you need at least 40GB GPU memory to infer. We will support CPU offload to reduce GPU memory usage soon. |
| We now supports tile-based processing (tile size: 512×512), enabling input of arbitrary resolutions and SR at any scale factor. |
|
|
| Please replace `experiments/qwen_one_step_gan/${EXP_DATE}/checkpoints/net_gen_iter_10001.pth` with the trained ODTSR model weight. |
| ```bash |
| sh examples/qwen_image/test_gan.sh |
| ``` |
| <div align="center"> |
| <img src="https://github.com/RedMediaTech/ODTSR/raw/main/static/infer.png" alt="Inference Workflow" width="70%"> |
| </div> |
|
|
| ## Inference with Gradio |
| ```bash |
| sh examples/qwen_image/test_gradio.sh |
| ``` |
| <img src="https://github.com/RedMediaTech/ODTSR/raw/main/static/gradio.jpeg" alt="Gradio Demo" > |
|
|
| ## License |
| This project is released under the [Apache 2.0 license](https://github.com/RedMediaTech/ODTSR/blob/main/LICENSE). |
|
|
| ## Acknowledgement |
| This project is based on [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio/tree/main). |
| We also leveraged some of [PiSA-SR](https://github.com/csslc/PiSA-SR/tree/main)'s code in dataloader part. |
| Thanks for the awesome work! |
|
|
| ## Citation |
| If ODTSR is helpful to you, please consider citing our paper: |
| ```bibtex |
| @article{fang2025onestep, |
| title={One-Step Diffusion Transformer for Controllable Real-World Image Super-Resolution}, |
| author={Fang, Yushun and Chen, Yuxiang and Yin, Shibo and Hu, Qiang and Yao, Jiangchao and Zhang, Ya and Zhang, Xiaoyun and Wang, Yanfeng}, |
| journal={arXiv preprint arXiv:2511.17138}, |
| year={2025} |
| } |
| ``` |
