Learning Deep CNN Denoiser Prior for Image Restoration

PyTorch version

https://github.com/cszn/DPIR

Abstract

Model-based optimization methods and discriminative learning methods have been the two dominant strategies for solving various inverse problems in low-level vision. Typically, those two kinds of methods have their respective merits and drawbacks, e.g., model-based optimization methods are flexible for handling different inverse problems but are usually time-consuming with sophisticated priors for the purpose of good performance; in the meanwhile, discriminative learning methods have fast testing speed but their application range is greatly restricted by the specialized task. Recent works have revealed that, with the aid of variable splitting techniques, denoiser prior can be plugged in as a modular part of model-based optimization methods to solve other inverse problems (e.g., deblurring). Such an integration induces considerable advantage when the denoiser is obtained via discriminative learning. However, the study of integration with fast discriminative denoiser prior is still lacking. To this end, this paper aims to train a set of fast and effective CNN (convolutional neural network) denoisers and integrate them into model-based optimization method to solve other inverse problems. Experimental results demonstrate that the learned set of denoisers can not only achieve promising Gaussian denoising results but also can be used as prior to deliver good performance for various low-level vision applications.

Basic Idea

With the aid of variable splitting techniques, such as alternating direction method of multipliers (ADMM) method and half quadratic splitting (HQS) method, it is possible to deal with fidelity term and regularization term of general image restoration formulation separately, and particularly, the regularization term only corresponds to a denoising subproblem. Consequently, this enables an integration of any discriminative denoisers into model-based optimization methods to solve various image restoration tasks, such as

• Image Deblurring
• Image Inpainting
• Single Image Super-Resolution
• Color Image Demosaicking

There is no additional training for these tasks.

Half-Quadratic Splitting (HQS) Algorithm

<img src="figs/HQS.png" width="600px"/>

Image Deblurring

The left is the blurred image. The right is the deblurred image by IRCNN with estimated kernels by other blind deblurring methods.

• Deblur_set1

<img src="testsets/Deblur_set1/Blurry4_1.png" width="350px"/> <img src="testsets/Deblur_set1/Blurry4_1_output_ircnn.png" width="350px"/> <img src="testsets/Deblur_set1/Blurry4_1_kernel.png" width="60px"/>

<img src="testsets/Deblur_set1/Blurry2_3.png" width="350px"/> <img src="testsets/Deblur_set1/Blurry2_3_output_ircnn.png" width="350px"/> <img src="testsets/Deblur_set1/Blurry2_3_kernel.png" width="60px"/>

• Deblur_set2

<img src="testsets/Deblur_set2/im01_ker02.png" width="350px"/> <img src="testsets/Deblur_set2/im01_ker02_ircnn.png" width="350px"/> <img src="testsets/Deblur_set2/im01_ker02_out_kernel.png" width="60px"/>

<img src="testsets/Deblur_set2/im04_ker01.png" width="350px"/> <img src="testsets/Deblur_set2/im04_ker01_ircnn.png" width="350px"/> <img src="testsets/Deblur_set2/im04_ker01_out_kernel.png" width="60px"/>

• Deblur_set3

<img src="testsets/Deblur_set3/2013622235456945_blur_79.png" width="350px"/> <img src="testsets/Deblur_set3/2013622235456945_blur_79_ircnn_Isigma_1_Msigma_1.png" width="350px"/> <img src="testsets/Deblur_set3/2013622235456945_blur_79_kernel.png" width="60px"/>

<img src="testsets/Deblur_set3/color_patch_blur_99.png" width="350px"/> <img src="testsets/Deblur_set3/color_patch_blur_99_ircnn_Isigma_1_Msigma_1.png" width="350px"/> <img src="testsets/Deblur_set3/color_patch_blur_99_kernel.png" width="60px"/>

<img src="testsets/Deblur_set3/IMG_1240_blur.png" width="350px"/> <img src="testsets/Deblur_set3/IMG_1240_blur_ircnn_Isigma_2_Msigma_7.png" width="350px"/> <img src="testsets/Deblur_set3/IMG_1240_blur_kernel.png" width="60px"/>

Use Demo_deblur_real_application.m to test IRCNN for image deblurring with estimated kernel by other blind deblurring methods.

Image Inpainting

The left is the masked image. The right is the recovered image by IRCNN.

• Inpaint_set1

<img src="testsets/Inpaint_set1/butterfly_color_80_masked.png" width="350px"/> <img src="testsets/Inpaint_set1/butterfly_color_80_ircnn.png" width="350px"/>

<img src="testsets/Inpaint_set1/butterfly_gray_75_masked.png" width="350px"/> <img src="testsets/Inpaint_set1/butterfly_gray_75_ircnn.png" width="350px"/>

<img src="testsets/Inpaint_set1/09_50_masked.png" width="350px"/> <img src="testsets/Inpaint_set1/09_50_ircnn.png" width="350px"/>

• Inpaint_set2

<img src="testsets/Inpaint_set2/3ch.png" width="350px"/> <img src="testsets/Inpaint_set2/3ch_ircnn.png" width="350px"/> <img src="testsets/Inpaint_set2/3ch_mask.png" width="60px"/>

<img src="testsets/Inpaint_set2/new.png" width="350px"/> <img src="testsets/Inpaint_set2/new_ircnn.png" width="350px"/> <img src="testsets/Inpaint_set2/new_mask.png" width="60px"/>

Use Demo_inpaint.m and Demo_inpaint_real_application.m to produce the above results.

Single Image Super-Resolution (SISR)

IRCNN considers three degradation models for SISR.

1. Bicubic degradation

y = imresize(x, 1/sf, 'bicubic') % y: LR image; x: clean HR image; sf: scale factor

IRCNN for bicubic degradation has a comparable performance to VDSR and DnCNN.

Use Demo_SISR_YCbCr.m and Demo_SISR_RGB.m to produce the SISR results.

2. Blurring by Gaussian kernel followed by downsampling

For scale factor 3,

kernelsize = ceil(sigma*3)*2+1; % or 7 for sigma = 1.6
kernel     = fspecial('gaussian',kernelsize,sigma);
blur_HR    = imfilter(im,kernel,'replicate')
LR         = imresize(blur_HR, 1/scale, 'nearest');

For scale factors 2 and 4,

kernelsize = ceil(sigma*3)*2+2;
kernel     = fspecial('gaussian',kernelsize,sigma);
blur_HR    = imfilter(im, kernel,'replicate');
LR         = blur_HR(scale/2:scale:end-scale/2,scale/2:scale:end-scale/2,:);

Use Demo_SISR_YCbCr.m and Demo_SISR_RGB.m to produce the SISR results.

3. Blurring by arbitrary kernel followed by standard K-fold downsampler (matlab function 'downsample')

blur_HR    = imfilter(im, kernel,'replicate'); % kernel can be arbitrary, such as motion blur, Gaussian blur.
for i = 1:size(x,3)
 LR(:,:,i) = downsample(downsample(blur_HR(:,:,i),sf)',sf)';
end

More details on this degradation model can be found in the following paper.

[1] Chan, Stanley H., Xiran Wang, and Omar A. Elgendy. "Plug-and-Play ADMM for image restoration: Fixed-point convergence and applications." IEEE Transactions on Computational Imaging 3.1 (2017): 84-98.

The left is the zoomed LR image (x3) with motion blur kernel, the right is the super-resolved image (x3) by IRCNN.

<img src="figs/LR1.png" width="350px"/> <img src="figs/LR1_IRCNN.png" width="350px"/>

The left is the low-resolution (LR) image. The right is the super-resolved image by IRCNN.

• SISR_set1, synthetic LR image SR

<img src="testsets/SISR_set1/LR3_noisy_x2.png" width="160px"/> <img src="testsets/SISR_set1/LR3_noisy_x2_ircnn_x2.png" width="320px"/>

<img src="testsets/SISR_set1/LR1_motion_x2.png" width="240px"/> <img src="testsets/SISR_set1/LR1_motion_x2_ircnn_x2.png" width="480px"/>

• SISR_set2, real LR image SR

<img src="testsets/SISR_set2/chip.png" width="109px"/> <img src="testsets/SISR_set2/chip_ircnn_x2.png" width="218px"/> <img src="testsets/SISR_set2/chip_ircnn_x4.png" width="436px"/>

<img src="testsets/SISR_set2/David_Hilbert.png" width="220px"/> <img src="testsets/SISR_set2/David_Hilbert_ircnn_x2.png" width="440px"/>

<img src="testsets/SISR_set2/Frog.png" width="266px"/> <img src="testsets/SISR_set2/Frog_ircnn_x2.png" width="532px"/>

Use Demo_SISR_direct_downsampler_real_application.m to produce the above SISR results.

<img src="figs/sr1.png" width="820px"/>

Color Image Demosaicking

The left is the mosaiced image. The right is the demosaiced image by IRCNN.

• Set18 (McMaster, IMAX)

<img src="figs/01_mosaik.png" width="350px"/> <img src="figs/01_ircnn.png" width="350px"/>

• Set24 (Kodak)

<img src="figs/kodim19_mosaik.png" width="350px"/> <img src="figs/kodim19_ircnn.png" width="350px"/>

Use Demo_demosaiking.m to produce the above results.

Requirements and Dependencies

• MATLAB R2015b
• Cuda-8.0 & cuDNN v-5.1
• MatConvNet

Example: Installation of CUDNN5.1, CUDA 8.0 GA2 and Matconvnet (Windows 7 64bit) (by kusiwu)

Download Visual studio 2015 and install it.

Download and install cuda 8.0 GA2: There is two installer, one is base installer and the other one is patch. You should download all of them. https://developer.nvidia.com/cuda-80-ga2-download-archive First install base installer (1.3GB) and then install patch (43.1MB) My cuda install path is: 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0'. You may need to add path CUDA_PATH if the setup does not do that. (as environment variable)

Download cudnn5.1. You should register to nvidia and download cudnn 5.1. Search in google please. Then you should extract the zip into "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0" (See http://docs.nvidia.com/deeplearning/sdk/cudnn-install/index.html) **Installati