A Toolbox for Spectral Compressive Imaging

Authors

Yuanhao Cai*, Jing Lin*, Xiaowan Hu, Haoqian Wang, Xin Yuan, Yulun Zhang, Radu Timofte, and Luc Van Gool

Papers

• Binarized Spectral Compressive Imaging (NeurIPS 2023)
• Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction (CVPR 2022)
• Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction (ECCV 2022)
• Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging (NeurIPS 2022)
• MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction (CVPRW 2022)
• HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imaging (CVPR 2022)

Awards

<img src="./figure/ntire.png" height=240> <img src="./figure/NTIRE_2024.png" height=240>

Introduction

This is a baseline and toolbox for spectral compressive imaging reconstruction. This repo supports over 15 algorithms. Our method MST++ won the NTIRE 2022 Challenge on spectral recovery from RGB images. If you find this repo useful, please give it a star ⭐ and consider citing our paper in your research. Thank you.

News

• 2025.02.10 : NTIRE 2025 Low-light Image Enhancement Challenge has started. Welcome to use our Retinexformer and MST to participate in this challenge. 😄
• 2024.04.09 : We release the results of the three traditional model-based methods, i.e., TwIST, GAP-TV, and DeSCI for your convenience to conduct research. Feel free to use them. 😄
• 2024.03.21 : Our methods Retinexformer and MST++ (NTIRE 2022 Spectral Reconstruction Challenge Winner) ranked top-2 in the NTIRE 2024 Challenge on Low Light Enhancement. Code, pre-trained models, training logs, and enhancement results will be released in the repo of Retinexformer. Stay tuned! 🚀
• 2024.02.15 : NTIRE 2024 Challenge on Low Light Enhancement begins. Welcome to use our Retinexformer or MST++ (NTIRE 2022 Spectral Reconstruction Challenge Winner) to participate in this challenge! :trophy:
• 2023.12.02 : Codes for real experiments have been updated. Welcome to check and use them. 🥳
• 2023.11.24 : Code, models, and results of BiSRNet (NeurIPS 2023) are released at this repo. We also develop a toolbox BiSCI for binarized SCI reconstruction. Feel free to check and use them. 🌟
• 2023.11.02 : MST, MST++, CST, and DAUHST are added to the Awesome-Transformer-Attention collection. 💫
• 2023.09.21 : Our new work BiSRNet is accepted by NeurIPS 23. Code will be released at this repo and BiSCI
• 2023.02.26 : We release the RGB images of five real scenes and ten simulation scenes. Please feel free to check and use them. 🌟
• 2022.11.02 : We have provided more visual results of state-of-the-art methods and the function to evaluate the parameters and computational complexity of models. Please feel free to check and use them. :high_brightness:
• 2022.10.23 : Code, models, and reconstructed HSI results of DAUHST have been released. 🔥
• 2022.09.15 : Our DAUHST has been accepted by NeurIPS 2022, code and models are coming soon. :rocket:
• 2022.07.20 : Code, models, and reconstructed HSI results of CST have been released. 🔥
• 2022.07.04 : Our paper CST has been accepted by ECCV 2022, code and models are coming soon. :rocket:
• 2022.06.14 : Code and models of MST and MST++ have been released. This repo supports 12 learning-based methods to serve as toolbox for Spectral Compressive Imaging. The model zoo will be enlarged. 🔥
• 2022.05.20 : Our work DAUHST is on arxiv. :dizzy:
• 2022.04.02 : Further work MST++ has won the NTIRE 2022 Spectral Reconstruction Challenge. :trophy:
• 2022.03.09 : Our work CST is on arxiv. :dizzy:
• 2022.03.02 : Our paper MST has been accepted by CVPR 2022, code and models are coming soon. :rocket:

| Scene 2 | Scene 3 | Scene 4 | Scene 7 | | :----------------------------------------------------------: | :----------------------------------------------------------: | :----------------------------------------------------------: | :----------------------------------------------------------: | | <img src="./figure/frame2channel12.gif" height=170 width=170> | <img src="./figure/frame3channel21.gif" width=170 height=170> | <img src="./figure/frame4channel28.gif" width=170 height=170> | <img src="./figure/frame7channel4.gif" width=170 height=170> |

 

1. Comparison with State-of-the-art Methods

12 learning-based algorithms and 3 model-based methods are supported.

<details open> <summary><b>Supported algorithms:</b></summary>

• [x] MST (CVPR 2022)
• [x] CST (ECCV 2022)
• [x] DAUHST (NeurIPS 2022)
• [x] BiSRNet (NeurIPS 2023)
• [x] MST++ (CVPRW 2022)
• [x] HDNet (CVPR 2022)
• [x] BIRNAT (TPAMI 2022)
• [x] DGSMP (CVPR 2021)
• [x] GAP-Net (Arxiv 2020)
• [x] TSA-Net (ECCV 2020)
• [x] ADMM-Net (ICCV 2019)
• [x] λ-Net (ICCV 2019)
• [x] TwIST (TIP 2007)
• [x] GAP-TV (ICIP 2015)
• [x] DeSCI (TPAMI 2019)

</details>

We are going to enlarge our model zoo in the future.

| MST vs. SOTA | CST vs. MST | | :----------------------------------------------: | :-----------------------------------------: | | <img src="./figure/compare_fig.png" height=320> | <img src="./figure/cst_mst.png" height=320> | | MST++ vs. SOTA | DAUHST vs. SOTA | | <img src="./figure/mst_pp.png" height=320> | <img src="./figure/dauhst.png" height=320> |

| BiSRNet vs. SOTA BNNs | | :-----------------------------------------: | | <img src="./figure/bisrnet.png" width=812> |

Quantitative Comparison on Simulation Dataset

| Method | Params (M) | FLOPS (G) | PSNR | SSIM | Model Zoo | Simulation Result | Real Result | | :----------------------------------------------------------: | :--------: | :-------: | :---: | :---: | :----------------------------------------------------------: | :----------------------------------------------------------: | :----------------------------------------------------------: | | TwIST | - | - | 23.12 | 0.669 | - | Google Drive / Baidu Disk | Google Drive / Baidu Disk | | GAP-TV |