Towards Real-Time 4K Image Super-Resolution

Eduard Zamfir, Marcos V. Conde, Radu Timofte

Computer Vision Lab, CAIDAS, University of Würzburg

Work part of the NTIRE Real-Time 4K Super-Resolution Challenge @ CVPR 2023 in Vancouver

Abstract

Over the past few years, high-definition videos and images in 720p (HD), 1080p (FHD), and 4K (UHD) resolution have become standard. While higher resolutions offer improved visual quality for users, they pose a significant chal- lenge for super-resolution networks to achieve real-time performance on commercial GPUs. This paper presents a comprehensive analysis of super-resolution model designs and techniques aimed at efficiently upscaling images from 720p and 1080p resolutions to 4K. We begin with a simple, effective baseline architecture and gradually modify its design by focusing on extracting important high-frequency details efficiently. This allows us to subsequently downscale the resolution of deep feature maps, reducing the overall computational footprint, while maintaining high reconstruction fidelity. We enhance our method by incorporating pixel-unshuffling, a simplified and speed-up reinterpretation of the basic block proposed by NAFNet, along with structural re-parameterization. We assess the performance of the fastest version of our method in the new NTIRE Real-Time 4K Super-Resolution challenge and demonstrate its potential in comparison with state-of-the-art efficient super-resolution models when scaled up. Our method was tested successfully on high-quality content from photography, digital art, and gaming content.

Paper Challenge Report

Installation

Create conda environment:

conda create --name rtsr python==3.10
source activate rt4ksr

Install PyTorch (see PyTorch instructions). For example,

conda install pytorch torchvision pytorch-cuda=11.8 -c pytorch -c nvidia

Install the dependencies:

pip install -r requirements.txt

Usage

Data Preparation

We generate bicubically downscaled LR images online and test our models on the standard benchmarks for Super-Resolution which can be found here. You can test any SR benchmark using our scripts and model weights for x2 and x3 SR. Please follow the dataset directory structure below:

dataroot/
|---testsets/
|   |---set5
|   |   |---test/
|   |   |   |---HR/
|   |   |   |---f"LR_bicubic_x{scale}"/

...

Test Model

You can find all the necessary details of testing the models in test.py. The argument --is-train is always needed, because the training architecture must be loaded first before reparameterization. Add --rep when you want to run the inference using the reparameterized version.

python code/test.py --dataroot [DATAROOT] --checkpoint-id rt4ksr_[x2|x3] --scale [x2|x3] --arch rt4ksr_rep --benchmark ntire23rtsr --is-train

NTIRE 2023 4K RTSR Benchmark

| Method | Scale |PSNR (RGB)| PSNR (Y)| SSIM (RGB) | SSIM (Y) | |---------|------------------|----------|---------|------------|----------| | Bicubic | x2 (1080p -> 4K) | 33.916 | 36.664 | 0.8829 | 0.9160 | | | x3 (720p -> 4K) | 31.302 | 33.812 | 0.8246 | 0.8656 | | RT4KSR | x2 (1080p -> 4K) | 34.193 | 37.013 | 0.8848 | 0.9180 |
| | x3 (720p -> 4K) | 31.721 | 34.349 | 0.8300 | 0.8715 |

Citation

@InProceedings{Zamfir_2023_CVPR,
    author    = {Zamfir, Eduard and Conde, Marcos V. and Timofte, Radu},
    title     = {Towards Real-Time 4K Image Super-Resolution},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
    month     = {June},
    year      = {2023},
    pages     = {1522-1532}
}

RT4KSR

Install / Use

README