U3M: Unbiased Multiscale Modal Fusion Model for Multimodal Semantic Segmentation

</div> </div>

💬 Introduction

Multimodal semantic segmentation is a pivotal com-ponent of computer vision, which often outperforms unimodal methods by harnessing a richer set of information from diverse sources. Existing models often employ modality-specific designs that inherently introduce biases toward certain modalities. While these biases may be beneficial in specific contexts, they often compromise the model’s adaptability across various multimodal scenarios, potentially degrading performance. To address this problem, we turn to the inherent capabilities of the model itself to discover the optimal equilibrium in multimodal fusion, and propose U3M: An Unbiased Multiscale Modal Fusion Model for Multimodal Semantic Segmentation. Specificially, this method involves an unbiased integration of multimodal visual data. Additionally, we employ feature fusion at multiple scales to ensure the effective extraction and integration of both global and local features. Experimental results demonstrate that our approach achieves superior performance across multiple datasets, validating its efficacy in enhancing the robustness and versatility of semantic segmentation in diverse settings.

🚀 Updates

[x] 05/2024: init repository and release the code.
[x] 05/2024: release U3M model weights. Download from GoogleDrive.

👁️ U3M model

U3M Figure: Overall architecture of U3M model.

</div>

🔍 Environment

First, create and activate the environment using the following commands:

conda env create -f environment.yaml
conda activate U3M

📦 Data preparation

Download the dataset:

MCubeS, for multimodal material segmentation with RGB-A-D-N modalities.
FMB, for FMB dataset with RGB-Infrared modalities.

Then, put the dataset under data directory as follows:

data/
├── MCubeS
│   ├── polL_color
│   ├── polL_aolp_sin
│   ├── polL_aolp_cos
│   ├── polL_dolp
│   ├── NIR_warped
│   ├── NIR_warped_mask
│   ├── GT
│   ├── SSGT4MS
│   ├── list_folder
│   └── SS
├── FMB
│   ├── test
│   │   ├── color
│   │   ├── Infrared
│   │   ├── Label
│   │   └── Visible
│   ├── train
│   │   ├── color
│   │   ├── Infrared
│   │   ├── Label
│   │   └── Visible

📦 Model Zoo

MCubeS

MCubeS_Ablation_RGBADN

FMB

| Model-Modal | mIoU | weight | | :--------------- | :----- | :----- | | FMB-RGB | 57.17 | GoogleDrive | | FMB-RGB-Infrared | 60.76 | GoogleDrive |

👁️ Visulization

Please refer to the visulization_mm.py for the segmentation results and visulization_tsne.py for T-sne visulization.

Figure: segresult_FMB.

Figure: segresult_MCUBES.

tsne Figure: T-sne visulization.

</div> ## Training

Before training, please download pre-trained SegFormer, and put it in the correct directory following this structure:

checkpoints/pretrained/segformer
├── mit_b0.pth
├── mit_b1.pth
├── mit_b2.pth
├── mit_b3.pth
└── mit_b4.pth

To train U3M model, please update the appropriate configuration file in configs/ with appropriate paths and hyper-parameters. Then run as follows:

cd path/to/U3M
conda activate U3M

python -m tools.train_mm --cfg configs/mcubes_rgbadn.yaml

python -m tools.train_mm --cfg configs/fmb_rgbt.yaml

Evaluation

To evaluate U3M models, please download respective model weights (GoogleDrive) and save them under any folder you like.

Then, update the EVAL section of the appropriate configuration file in configs/ and run:

cd path/to/U3M
conda activate U3M

python -m tools.val_mm --cfg configs/mcubes_rgbadn.yaml

python -m tools.val_mm --cfg configs/fmb_rgbt.yaml

🚩 License

This repository is under the Apache-2.0 license. For commercial use, please contact with the authors.

📜 Citations

If you use U3M model, please cite the following work:

U3M [arXiv]

@article{li2024u3m,
  title={U3M: Unbiased Multiscale Modal Fusion Model for Multimodal Semantic Segmentation},
  author={Li, Bingyu and Zhang, Da and Zhao, Zhiyuan and Gao, Junyu and Li, Xuelong},
  journal={arXiv preprint arXiv:2405.15365},
  year={2024}
}

🔈 Acknowledgements

Our codebase is based on the following Github repositories. Thanks to the following public repositories:

Note: This is a research level repository and might contain issues/bugs. Please contact the authors for any query.

U3M

Install / Use

README