SPSR

PyTorch implementation of Structure-Preserving Super Resolution with Gradient Guidance (CVPR 2020) [arXiv][CVF]

Extended version: Structure-Preserving Image Super-Resolution (TPAMI 2021) [arXiv]

<p align="center"> <img src="visual_results/fig2.png"> </p>

If you find our work useful in your research, please consider citing:

@ARTICLE{ma2021structure,  
  author={Ma, Cheng and Rao, Yongming and Lu, Jiwen and Zhou, Jie},  
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},   
  title={Structure-Preserving Image Super-Resolution},   
  year={2021},  
  volume={},  
  number={},  
  pages={1-1},  
  doi={10.1109/TPAMI.2021.3114428}}

@inproceedings{ma2020structure,
  title={Structure-Preserving Super Resolution with Gradient Guidance},
  author={Ma, Cheng and Rao, Yongming and Cheng, Yean and Chen, Ce and Lu, Jiwen and Zhou, Jie},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2020}
}

Dependencies

• Python 3 (Recommend to use Anaconda)
• PyTorch >= 1.0
• NVIDIA GPU + CUDA
• Python packages: pip install numpy opencv-python lmdb pyyaml
• TensorBoard:
  • PyTorch >= 1.1: pip install tb-nightly future
  • PyTorch == 1.0: pip install tensorboardX

Dataset Preparation

Download Datasets

Commonly used training and testing datasets can be downloaded here.

Preprocess Datasets

We also provide code to preprocess the datasets here.

• After downloading the original datasets, please store them to a specific GT folder.

• You can obtain the LR, HR and Bicubic-upsampled versions of the datasets.

• Then you can extract sub-images with unified scales for training.

• The training sets can also be transformed into LMDB format for faster IO speed.

Training

To train an SPSR model:

python train.py -opt options/train/train_spsr.json

• The json file will be processed by options/options.py. Please refer to this for more details.

• Before running this code, please modify train_spsr.json to your own configurations including:

  • the proper dataroot_HR and dataroot_LR paths for the data loader (More details)
  • saving frequency for models and states
  • whether to resume training with .state files
  • other hyperparameters
  • loss function, etc.

• You can find your training results in ./experiments.

• During training, you can use Tesorboard to monitor the losses with tensorboard --logdir tb_logger/NAME_OF_YOUR_EXPERIMENT

• You can choose to use a pretrained RRDB model as a parameter initialization by setting the pretrain_model_G option in options/train/train_spsr.json. Please download the pretrained model from Google Drive or Baidu Drive (extraction code muw3) and place RRDB_PSNR_x4.pth into ./experiments/pretrain_models.

Testing

To generate SR images by an SPSR model:

python test.py -opt options/test/test_spsr.json

• Similar to training, the configurations can be modified in the test_spsr.json file.

• You can find your results in ./results.

• We provide our SPSR model used in our paper that can be downloaded in Google Drive or Baidu Drive (extraction code muw3). Download spsr.pth and put it into ./experiments/pretrain_models. Then modify the directory of pretrained model in test_spsr.json and run test.py.

• You can put your own LR images in a certain folder and just change the dataroot_LR setting in test_spsr.json and run test.py.

Evaluation Toolbox

We provide an easy and useful evaluation toolbox to simplify the procedure of evaluating SR results. In this toolbox, you can get the MA, NIQE, PI, PSNR, SSIM, MSE, RMSE, MAE and LPIPS values of any SR results you want to evaluate.

Results

Visual Results

<p align="center"> <img src="visual_results/fig4.png"> </p> <p align="center"> <img src="visual_results/fig5.png"> </p>

Quantitative Results

From the below two tables of comparison with perceptual-driven SR methods, we can see our SPSR method is able to obtain the best PI and LPIPS performance and comparable PSNR and SSIM values simultaneously. The top 2 scores are highlighted.

PI/LPIPS comparison with perceptual-driven SR methods.

| Method | Set5 | Set14 | BSD100 | General100 | Urban100 | | --------------------------------------------- | ----------------------- | ----------------------- | ----------------------- | ----------------------- | ----------------------- | | Bicubic | 7.3699/0.3407 | 7.0268/0.4393 | 7.0026/0.5249 | 7.9365/0.3528 | 6.9435/0.4726 | | SFTGAN | 3.7587/0.0890 | 2.9063/0.1481 | 2.3774/0.1769 | 4.2878/0.1030 | 3.6136/0.1433 | | SRGAN | 3.9820/0.0882 | 3.0851/0.1663 | 2.5459/0.1980 | 4.3757/0.1055 | 3.6980/0.1551 | | ESRGAN | 3.7522/0.0748 | 2.9261/0.1329 | 2.4793/0.1614 | 4.3234/0.0879 | 3.7704/0.1229 | | NatSR | 4.1648/0.0939 | 3.1094/0.1758 | 2.7801/0.2114 | 4.6262/0.1117 | 3.6523/0.1500 | | SPSR | 3.2743/0.0644 | 2.9036/0.1318 | 2.3510/0.1611 | 4.0991/0.0863 | 3.5511/0.1184 |

PSNR/SSIM comparison with perceptual-driven SR methods.

| Method | Set5 | Set14 | BSD100 | General100 | Urban100 | | --------------------------------------------- | ----------------------- | ----------------------- | ----------------------- | ----------------------- | ----------------------- | | Bicubic | 28.420/0.8245 | 26.100/0.7850 | 25.961/0.6675 | 28.018/0.8282 | 23.145/0.9011 | | SFTGAN | 29.932/0.8665 | 26.223/0.7854 | 25.505/0.6549 | 29.026/0.8508 | 24.013/0.9364 | | SRGAN | 29.168/0.8613 | 26.171/0.7841 | 25.459/0.6485 | 28.575/0.8541 | 24.397/0.9381 | | ESRGAN | 30.454/0.8677 | 26.276/0.7783 | 25.317/0.6506 | 29.412/0.8546 | 24.360/0.9453 | | NatSR | 30.991/0.8800 | 27.514/0.8140 | 26.445/0.6831 | 30.346/0.8721 | 25.464/0.9505 | | SPSR | 30.400/0.8627 | 26.640/0.7930 | 25.505/0.6576 | 29.414/0.8537 | 24.799/0.9481 |

Acknowledgement

The code is based on BasicSR, MA, NIQE, PI, SSIM and LPIPS.

Contact

If you have any questions about our work, please contact macheng17@mails.tsinghua.edu.cn