NTIRE2017 Super-resolution Challenge: SNU_CVLab

Introduction

This is our project repository for CVPR 2017 Workshop (2nd NTIRE).

We, Team SNU_CVLab, (Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee of Computer Vision Lab, Seoul National University) are winners of NTIRE2017 Challenge on Single Image Super-Resolution.

Our paper was published in CVPR 2017 workshop (2nd NTIRE), and won the Best Paper Award of the workshop challenge track.

Please refer to our paper for details.

If you find our work useful in your research or publication, please cite our work:

[1] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee, "Enhanced Deep Residual Networks for Single Image Super-Resolution," 2nd NTIRE: New Trends in Image Restoration and Enhancement workshop and challenge on image super-resolution in conjunction with CVPR 2017. [PDF] [arXiv] [Slide]

@InProceedings{Lim_2017_CVPR_Workshops,
  author = {Lim, Bee and Son, Sanghyun and Kim, Heewon and Nah, Seungjun and Lee, Kyoung Mu},
  title = {Enhanced Deep Residual Networks for Single Image Super-Resolution},
  booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
  month = {July},
  year = {2017}
}

In this repository, we provide

Our model architecture description (EDSR, MDSR)
NTIRE2017 Super-resolution Challenge Results
Demo & Training code
Trained models (EDSR, MDSR)
Datasets we used (DIV2K, Flickr2K)
Super-resolution examples

The code is based on Facebook's Torch implementation of ResNet (facebook/fb.resnet.torch).

We also provide PyTorch version of EDSR and MDSR. (Until now, only some models are available.)

Model Architecture

EDSR (Single-scale model. We provide scale x2, x3, x4 models).

EDSR

MDSR (Multi-scale model. It can handle x2, x3, x4 super-resolution in a single model).

MDSR

Note that the MDSR architecture for the challenge and for the paper[1] is slightly different. During the challenge, MDSR had variation between two challenge tracks. While we had scale-specific feature extraction modules for track 2:unknown downscaling, we didn't use the scale-specific modules for track 1:bicubic downscaling.

We later unified the MDSR model in our paper[1] by including scale-specific modules for both cases. From now on, unless specified as "challenge", we describe the models described in the paper.

NTIRE2017 Super-resolution Challenge Results

We proposed 2 methods and they won the 1st (EDSR) and 2nd (MDSR) place.

Challenge_result

We have also compared the super-resolution performance of our models with previous state-of-the-art methods.

Paper_result

About our code

Dependencies

Torch7
cuDNN
nccl (Optional, for faster GPU communication)

Our code is tested under Ubuntu 14.04 and 16.04 environment with Titan X GPUs (12GB VRAM).

Code

Clone this repository into any place you want. You may follow the example below.

makeReposit = [/the/directory/as/you/wish]
mkdir -p $makeReposit/; cd $makeReposit/
git clone https://github.com/LimBee/NTIRE2017.git

Quick Start (Demo)

You can test our super-resolution algorithm with your own images.

We assume the images are downsampled by bicubic interpolation.

| Model | Scale | File Name | Self Esemble | # ResBlocks | # Filters | # Parameters | | --- | --- | --- | --- | --- |--- |--- | | EDSR baseline| x2 | baseline_x2.t7 | X | 16 | 64 | 1.5M | | EDSR baseline| x3 | baseline_x3.t7 | X | 16 | 64 | 1.5M | | EDSR baseline| x4 | baseline_x4.t7 | X | 16 | 64 | 1.5M | | MDSR baseline| Multi | baseline_multiscale.t7 | X | 16 | 64 | 3.2M | |||||||| | EDSR| x2 | EDSR_x2.t7 | X | 32 | 256 | 43M | | EDSR| x3 | EDSR_x3.t7 | X | 32 | 256 | 43M | | EDSR| x4 | EDSR_x4.t7 | X | 32 | 256 | 43M | | MDSR| Multi | MDSR.t7 | X | 80 | 64 | 8.0M | |||||||| | EDSR+| x2 | EDSR_x2.t7 | O | 32 | 256 | 43M | | EDSR+| x3 | EDSR_x3.t7 | O | 32 | 256 | 43M | | EDSR+| x4 | EDSR_x4.t7 | O | 32 | 256 | 43M | | MDSR+| Multi | MDSR.t7 | O | 80 | 64 | 8.0M |

Download our models
```
cd $makeReposit/NTIRE2017/demo/model/

# Our models for the paper[1]
wget https://cv.snu.ac.kr/research/EDSR/model_paper.tar
```
Or, use the link: model_paper.tar

(If you would like to run the models we used during the challenge, please contact us.)

After downloading the .tar files, make sure that the model files are placed in proper locations. For example,
```
$makeReposit/NTIRE2017/demo/model/bicubic_x2.t7
$makeReposit/NTIRE2017/demo/model/bicubic_x3.t7
...
```
Place your low-resolution test images at
```
$makeReposit/NTIRE2017/demo/img_input/
```
The demo code will read .jpg, .jpeg, .png format images.

Run test.lua

You can run different models and scales by changing input arguments.

# To run for scale 2, 3, or 4, set -scale as 2, 3, or 4
# To run EDSR+ and MDSR+, you need to set -selfEnsemble as true

cd $makeReposit/NTIRE2017/demo

# Test EDSR (scale 2)
th test.lua -model EDSR_x2 -selfEnsemble false

# Test EDSR+ (scale 2)
th test.lua -model EDSR_x2 -selfEnsemble true

# Test MDSR (scale 2)
th test.lua -model MDSR -scale 2 -selfEnsemble false

# Test MDSR+ (scale 2)
th test.lua -model MDSR -scale 2 -selfEnsemble true

(Note: To run the MDSR, model name should include multiscale or MDSR. e.g. multiscale_blahblahblah.t7)

The result images will be located at

$makeReposit/NTIRE2017/demo/img_output/

Here are some optional argument examples you can adjust. Please refer to the following explanation.

# You can test our model with multiple GPU. (n = 1, 2, 4)
-nGPU       [n]

# You must specify this directory. Default is /var/tmp/dataset
-dataDir    [$makeData]
-dataset    [DIV2K | myData]
-save       [Folder name]

# Please see our paper[1] if you want to know about self-ensemble.
-selfEnsemble   [true | false]

# Please reduce the chopSize when you see 'out of memory'.
# The optimal size of S can be vary depend on your maximum GPU memory.
-chopSize   [S]

(Optional) Evaluate PSNR and SSIM if you have ground-truth HR images

Place the GT images at
```
$makeReposit/NTIRE2017/demo/img_target
```
Evaluation is done by running the MATLAB script.
```
matlab -nodisplay <evaluation.m
```
If you don't want to calculate SSIM, please modify evaluation.m file as below. (Calculating SSIM of large image is very slow for 3 channel images.)
```
line 6: psnrOnly = false; -> psnrOnly = true;
```

You can reproduce our final results by running makeFinal.sh in NTIRE2017/demo directory. Please uncomment the command you want to execute in the file.

sh makeFinal.sh

Dataset

If you want to train or evaluate our models with DIV2K or Flickr2K dataset, please download the dataset from here. Place the tar file to the location you want. (We recommend /var/tmp/dataset/) If the dataset is located otherwise, you have to change the optional argument -dataset for training and test.

DIV2K from NTIRE2017
```
makeData = /var/tmp/dataset/ # We recommend this path, but you can freely change it.
mkdir -p $makeData/; cd $makedata/
tar -xvf DIV2K.tar
```
You should have the following directory structure:

/var/tmp/dataset/DIV2K/DIV2K_train_HR/0???.png /var/tmp/dataset/DIV2K/DIV2K_train_LR_bicubic/X?/0???.png /var/tmp/dataset/DIV2K/DIV2K_train_LR_unknown/X?/0???.png

Flickr2K dataset collected by ourselves using Flickr API

makeData = /var/tmp/dataset/
mkdir -p $makeData/; cd $makedata/
wget https://cv.snu.ac.kr/research/EDSR/Flickr2K.tar
tar -xvf Flickr2K.tar

NTIRE2017

Install / Use

README