ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting (NeurIPS 2023, Spotlight, TPAMI@2025)

Zongsheng Yue, Jianyi Wang, Chen Change Loy

Conference Paper | Journal Paper | Project Page | Video

:star: If ResShift is helpful to your images or projects, please help star this repo. Thanks! :hugs:

Diffusion-based image super-resolution (SR) methods are mainly limited by the low inference speed due to the requirements of hundreds or even thousands of sampling steps. Existing acceleration sampling techniques inevitably sacrifice performance to some extent, leading to over-blurry SR results. To address this issue, we propose a novel and efficient diffusion model for SR that significantly reduces the number of diffusion steps, thereby eliminating the need for post-acceleration during inference and its associated performance deterioration. Our method constructs a Markov chain that transfers between the high-resolution image and the low-resolution image by shifting the residual between them, substantially improving the transition efficiency. Additionally, an elaborate noise schedule is developed to flexibly control the shifting speed and the noise strength during the diffusion process. Extensive experiments demonstrate that the proposed method obtains superior or at least comparable performance to current state-of-the-art methods on both synthetic and real-world datasets, even only with 15 sampling steps. <img src="./assets/framework.png" align="middle" width="800">

Update

2024.09.13: Our extended journal version has beed accepted by TPAMI@2024. We add the training and testing code for deblurring.
2024.03.11: Update the code for the Journal paper
2023.12.02: Add configurations for the x2 super-resolution task.
2023.08.15: Add Gradio Demo.
2023.08.14: Add bicubic (matlab resize) model.
2023.08.14: Add Project Page.
2023.08.02: Add Replicate demo .
2023.07.31: Add Colab demo <a href="https://colab.research.google.com/drive/1CL8aJO7a_RA4MetanrCLqQO5H7KWO8KI?usp=sharing"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="google colab logo"></a>.
2023.07.24: Create this repo.

Requirements

Python 3.10, Pytorch 2.1.2, xformers 0.0.23
More detail (See environment.yml) A suitable conda environment named resshift can be created and activated with:

conda create -n resshift python=3.10
conda activate resshift
pip install -r requirements.txt

conda env create -f environment.yml
conda activate resshift

Applications

:point_right: Real-world image super-resolution

:point_right: Image deblurring

:point_right: Image inpainting

:point_right: Blind Face Restoration

Online Demo

You can try our method through an online demo:

python app.py

Fast Testing

:tiger: Real-world image super-resolution

python inference_resshift.py -i [image folder/image path] -o [result folder] --task realsr --scale 4 --version v3

:lion: Bicubic (resize by Matlab) image super-resolution

python inference_resshift.py -i [image folder/image path] -o [result folder] --task bicsr --scale 4

:snail: Image deblurring

python inference_resshift.py -i [image folder/image path] -o [result folder] --task deblur --scale 1

:snake: Natural image inpainting

python inference_resshift.py -i [image folder/image path] -o [result folder] --mask_path [mask path] --task inpaint_imagenet --scale 1

:crocodile: Face image inpainting

python inference_resshift.py -i [image folder/image path] -o [result folder] --mask_path [mask path] --task inpaint_face --scale 1

:octopus: Blind Face Restoration

python inference_resshift.py -i [image folder/image path] -o [result folder] --task faceir --scale 1

Training

:turtle: Preparing stage

Download the pre-trained VQGAN model from this link and put it in the folder of 'weights'
Adjust the data path in the config file.
Adjust batchsize according your GPUS.
- configs.train.batch: [training batchsize, validation batchsize]
- configs.train.microbatch: total batchsize = microbatch * #GPUS * num_grad_accumulation

:dolphin: Real-world Image Super-resolution for NeurIPS

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun --standalone --nproc_per_node=8 --nnodes=1 main.py --cfg_path configs/realsr_swinunet_realesrgan256.yaml --save_dir [Logging Folder]

:whale: Real-world Image Super-resolution for Journal

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun --standalone --nproc_per_node=8 --nnodes=1 main.py --cfg_path configs/realsr_swinunet_realesrgan256_journal.yaml --save_dir [Logging Folder]

:camel: Image Deblurring for Journal

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun --standalone --nproc_per_node=8 --nnodes=1 main.py --cfg_path configs/deblur_gopro256.yaml --save_dir [Logging Folder]

:ox: Image inpainting (Natural) for Journal

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun --standalone --nproc_per_node=8 --nnodes=1 main.py --cfg_path configs/inpaint_lama256_imagenet.yaml --save_dir [Lo

ResShift

Install / Use

README

ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting (NeurIPS 2023, Spotlight, TPAMI@2025)

Update

Requirements

Applications

:point_right: Real-world image super-resolution

:point_right: Image deblurring

:point_right: Image inpainting

:point_right: Blind Face Restoration

Online Demo

Fast Testing

:tiger: Real-world image super-resolution

:lion: Bicubic (resize by Matlab) image super-resolution

:snail: Image deblurring

:snake: Natural image inpainting

:crocodile: Face image inpainting

:octopus: Blind Face Restoration

Training

:turtle: Preparing stage

:dolphin: Real-world Image Super-resolution for NeurIPS

:whale: Real-world Image Super-resolution for Journal

:camel: Image Deblurring for Journal

:ox: Image inpainting (Natural) for Journal