Cross Paradigm Representation and Alignment Transformer for Image Deraining

🔥🔥🔥 News

2025-02-10: This repo is released.

Abstract: Transformer-based networks have achieved strong performance in low-level vision tasks like image deraining by utilizing spatial or channel-wise self-attention. However, irregular rain patterns and complex geometric overlaps challenge single-paradigm architectures, necessitating a unified framework to integrate complementary global-local and spatial-channel representations. To address this, we propose a novel Cross Paradigm Representation and Alignment Transformer (CPRAformer). Its core idea is the hierarchical representation and alignment, leveraging the strengths of both paradigms (spatial-channel and global-local) to aid image reconstruction. It bridges the gap within and between paradigms, aligning and coordinating them to enable deep interaction and fusion of features. Specifically, we use two types of self-attention in the Transformer blocks: sparse prompt channel self-attention (SPC-SA) and spatial pixel refinement self-attention (SPR-SA). SPC-SA enhances global channel dependencies through dynamic sparsity, while SPR-SA focuses on spatial rain distribution and fine-grained texture recovery. To address the feature misalignment and knowledge differences between them, we introduce the Adaptive Alignment Frequency Module (AAFM), which aligns and interacts with features in a two-stage progressive manner, enabling adaptive guidance and complementarity. This reduces the information gap within and between paradigms. Through this unified cross-paradigm dynamic interaction framework, we achieve the extraction of the most valuable interactive fusion information from the two paradigms. Extensive experiments demonstrate that our model achieves state-of-the-art performance on eight benchmark datasets and further validates CPRAformer's robustness in other image restoration tasks and downstream applications.

⚙️ Dependencies

Python 3.8
PyTorch 1.9.0
NVIDIA GPU + CUDA

# Clone the github repo and go to the default directory 'DAIT'.
git clone https://github.com/zs1314/DAIT.git
conda create -n DAIT python=3.8
conda activate DAIT
pip install matplotlib scikit-image opencv-python yacs joblib natsort h5py tqdm

Install warmup scheduler

cd pytorch-gradual-warmup-lr; python setup.py install; cd ..

⚒️ TODO

[x] Release code

<a name="datasets"></a>🖨️ Datasets

Used training and testing sets can be downloaded as follows:

| Training Set | Testing Set | Visual Results | |:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:| :----------------------------------------------------------: | | Rain13K[complete training dataset: Google Drive / Baidu Disk] | Test100 + Rain100H + Rain100L + Test2800 + Test1200 [complete testing dataset: Google Drive / Baidu Disk] | Google Drive / Baidu Disk |

Download training and testing datasets and put them into the corresponding folders of Datasets/. See Datasets for the detail of the directory structure.

<a name="training"></a>🔧 Training

Download training (Rain13K, already processed) and testing (Test100 + Rain100H + Rain100L + Test2800 + Test1200 , already processed) datasets, place them in Datasets/.
Run the following scripts. The training configuration is in training.yml.
```
python train.py
```
The training experiment is in checkpoints/.

<a name="testing"></a>🔨 Testing

Download testing (Test100 + Rain100H + Rain100L + Test2800 + Test1200) datasets, place them in Datasets/.
Run the following scripts. The testing configuration is in test.py.
```
  python test.py
```
The output is in results/.
To reproduce PSNR/SSIM scores of the paper, run (in matlab):
```
  evaluate_PSNR_SSIM.m 
```

<a name="results"></a>🔎 Results

We achieve state-of-the-art performance. Detailed results can be found in the paper.

<details> <summary>Quantitative Comparison (click to expand)</summary>

results in Table 1 of the main paper

results in Table 2 of the main paper

results in Table 2 of the supplementary material

results in Table 1 of the supplementary material

results in Table 5 of the supplementary material

<img width="900" src="Figs/T4.png"> </details> <details> <summary>Visual Comparison (click to expand)</summary>

results in Figure 5 of the supplementary material

results in Figure 6 of the main paper

results in Figure 4 of the supplementary material

results in Figure 7 of the supplementary material

results in Figure 9 of the supplementary material

results in Figure 2 of the supplementary material

results in Figure 1 of the supplementary material

results in Figure 11 of the supplementary material

<a name="contact"></a>📂 Contact

Should you have any question, please contact shunzou.njau@gmail.com

<a name="citation"></a>📎 Citation

We kindly request that you cite our work if you utilize the code or reference our findings in your research:

@inproceedings{zou2025cpraformer,
  title={Cross Paradigm Representation and Alignment Transformer for Image Deraining},
  author={Zou, Shun and Zou, Yi and Li, Juncheng and Gao, Guangwei and Qi, Guojun},
  booktitle={ACM MM},
  year={2025}
}

<a name="acknowledgements"></a>💡 Acknowledgements

This code is built on MPRNet.

CPRAformer

Install / Use

README