---

<div align="center">

Refign: Align and Refine for Adaptation of Semantic Segmentation to Adverse Conditions

</div>

This repository provides the official code for the WACV 2023 paper Refign: Align and Refine for Adaptation of Semantic Segmentation to Adverse Conditions. The code is organized using PyTorch Lightning.

🔥 [September 2, 2022] Applied on top of HRDA, Refign ranks #1 on both the ACDC leaderboard—72.05 mIoU—and the Dark Zurich leaderboard—63.91 mIoU. See below for training configurations.

<img src="./docs/method.png" width="900"/>

Abstract

Due to the scarcity of dense pixel-level semantic annotations for images recorded in adverse visual conditions, there has been a keen interest in unsupervised domain adaptation (UDA) for the semantic segmentation of such images. UDA adapts models trained on normal conditions to the target adverse-condition domains. Meanwhile, multiple datasets with driving scenes provide corresponding images of the same scenes across multiple conditions, which can serve as a form of weak supervision for domain adaptation. We propose Refign, a generic extension to self-training-based UDA methods which leverages these cross-domain correspondences. Refign consists of two steps: (1) aligning the normal-condition image to the corresponding adverse-condition image using an uncertainty-aware dense matching network, and (2) refining the adverse prediction with the normal prediction using an adaptive label correction mechanism. We design custom modules to streamline both steps and set the new state of the art for domain-adaptive semantic segmentation on several adverse-condition benchmarks, including ACDC and Dark Zurich. The approach introduces no extra training parameters, minimal computational overhead—during training only—and can be used as a drop-in extension to improve any given self-training-based UDA method.

Usage

Requirements

The code is run with Python 3.8.13. To install the packages, use:

pip install -r requirements.txt

Set Data Directory

The following environment variable must be set:

export DATA_DIR=/path/to/data/dir

Download the Data

Before running the code, download and extract the corresponding datasets to the directory $DATA_DIR.

UDA

<details> <summary>Cityscapes</summary>

Download leftImg8bit_trainvaltest.zip and gt_trainvaltest.zip from here and extract them to $DATA_DIR/Cityscapes.

$DATA_DIR
├── Cityscapes
│   ├── leftImg8bit
│   │   ├── train
│   │   ├── val
│   ├── gtFine
│   │   ├── train
│   │   ├── val
├── ...

Afterwards, run the preparation script:

python tools/convert_cityscapes.py $DATA_DIR/Cityscapes

</details> <details> <summary>ACDC</summary>

Download rgb_anon_trainvaltest.zip and gt_trainval.zip from here and extract them to $DATA_DIR/ACDC.

$DATA_DIR
├── ACDC
│   ├── rgb_anon
│   │   ├── fog
│   │   ├── night
│   │   ├── rain
│   │   ├── snow
│   ├── gt
│   │   ├── fog
│   │   ├── night
│   │   ├── rain
│   │   ├── snow
├── ...

</details> <details> <summary>Dark Zurich</summary>

Download Dark_Zurich_train_anon.zip, Dark_Zurich_val_anon.zip, and Dark_Zurich_test_anon_withoutGt.zip from here and extract them to $DATA_DIR/DarkZurich.

$DATA_DIR
├── DarkZurich
│   ├── rgb_anon
│   │   ├── train
│   │   ├── val
│   │   ├── val_ref
│   │   ├── test
│   │   ├── test_ref
│   ├── gt
│   │   ├── val
├── ...

</details> <details> <summary>Nighttime Driving</summary>

Download NighttimeDrivingTest.zip from here and extract it to $DATA_DIR/NighttimeDrivingTest.

$DATA_DIR
├── NighttimeDrivingTest
│   ├── leftImg8bit
│   │   ├── test
│   ├── gtCoarse_daytime_trainvaltest
│   │   ├── test
├── ...

</details> <details> <summary>BDD100k-night</summary>

Download 10k Images and Segmentation from here and extract them to $DATA_DIR/bdd100k.

$DATA_DIR
├── bdd100k
│   ├── images
│   │   ├── 10k
│   ├── labels
│   │   ├── sem_seg
├── ...

</details> <details> <summary>RobotCar for Segmentation</summary>

Download all data from here and save them to $DATA_DIR/RobotCar. As mentioned in the corresponding README.txt, the images must be downloaded from this link.

$DATA_DIR
├── RobotCar
│   ├── images
│   │   ├── dawn
│   │   ├── dusk
│   │   ├── night
│   │   ├── night-rain
│   │   ├── ...
│   ├── correspondence_data
│   │   ├── ...
│   ├── segmented_images
│   │   ├── training
│   │   ├── validation
│   │   ├── testing
├── ...

</details>

Alignment

<details> <summary>MegaDepth</summary>

For training, we use the version provided by the D2-Net repo. Follow their instructions for downloading and preprocessing the dataset.

For testing, we use the split provided by RANSAC-Flow here. The directories MegaDepth_Train, MegaDepth_Train_Org, and Val can be removed.

All in all, the folder structure should look as follows:

$DATA_DIR
├── MegaDepth
│   ├── Undistorted_SfM
│   │   ├── ...
│   ├── scene_info
│   │   ├── ...
│   ├── Test
│   │   ├── test1600Pairs
│   │   |   ├── ...
│   │   ├── test1600Pairs.csv
├── ...

</details> <details> <summary>RobotCar for Matching</summary>

We use the correspondence file provided by RANSAC-Flow here. If not already downloaded for segmentation, download the images from here.

$DATA_DIR
├── RobotCar
│   ├── images
│   │   ├── dawn
│   │   ├── dusk
│   │   ├── night
│   │   ├── night-rain
│   │   ├── ...
│   ├── test6511.csv
├── ...

</details>

Download the Pretrained Weights

The following pretrained weights are required for Refign. Save them to ./pretrained_models/.

1. UAWarpC checkpoint, download it here.

2. ImageNet-pretrained MiT weights (mit_b5.pth), download them from the SegFormer repository.

3. Cityscapes-pretrained SegFormer weights (segformer.b5.1024x1024.city.160k.pth), download them from the SegFormer repository.

Trained Models and Results

We provide trained models of both UDA and alignment networks. To facilitate qualitative segmentation comparisons, validation set predictions of Refign can be directly downloaded. Starred models use Cityscapes pretrained weights in the backbone, the others ImageNet pretrained.

UDA

| Model | Task | Test Set | Test Score | Config | Checkpoint | Predictions | |---------------|----------------|-----------------|-----------------|------------|----------------|------------| | Refign-DAFormer | Cityscapes→ACDC | ACDC test | 65.5 mIoU | config | model | ACDC val | Refign-HRDA* | Cityscapes→ACDC | ACDC test | 72.1 mIoU | config | model | ACDC val ||||||| | Refign-DAFormer | Cityscapes→Dark Zurich | Dark Zurich test | 56.2 mIoU | config | model | Dark Zurich val | Refign-HRDA* | Cityscapes→Dark Zurich | Dark Zurich test | 63.9 mIoU | [config](https://github.com/brdav/refign/blob/main/con