Video Semantic Segmentation with Distortion-Aware Feature Correction

This repository is the official implementation of "Video Semantic Segmentation with Distortion-Aware Feature Correction" (accepted by IEEE Transactions on Circuits and Systems for Video Technology(TCSVT) 2020). It is designed for efficient video semantic segmentation task.

Paper | Project Page | YouTube | BibeTex

<img src="./gif/demo.gif" width="860"/>

Install & Requirements

The code has been tested on pytorch=1.5.0 and python3.7. Please refer to requirements.txt for detailed information.

To Install python packages

pip install -r requirements.txt

To Install resampled 2d modules

cd $DAVSS_ROOT/lib/model/resample2d_package
python setup.py build

Data preparation

You need to download the Cityscapes and CamVid datasets.

Your directory tree should be look like this:

$DAVSS_ROOT/data
├── cityscapes
│   ├── gtFine
│   │   ├── train
│   │   └── val
│   └── leftImg8bit_sequence
│       ├── train
│       └── val
├── camvid
│   ├── label
│   │   ├── segmentation annotations
│   └── video_image
│       ├── 0001TP
│           ├── decoded images from video clips
│       ├── 0006R0
│       └── 0016E5
│       └── Seq05VD

Experiment preparation

Environment Setting

export PYTHONPATH=$PYTHONPATH:$DAVSS_ROOT
cd $DAVSS_ROOT

Download pretrained model

We provide pretrained deeplabv3+ and flownet models on Cityscapes and CamVid datasets. You can download from OneDrive/BaiduYun(Access Code:r4cd). Please place pretrained models in ./saved_model/pretrained.

Train and test

Please specify the script file.

For example, train our proposed method on Cityscapes on 4 GPUs:

# training DMNet
bash ./exp/dmnet_cityscapes/script/train.sh
# training the entire frameowrk
bash ./exp/spatial_correction_cityscapes/script/train.sh

For example, test our proposed method on Cityscapes validation set with PDA evaluation:

bash ./exp/spatial_correction_cityscapes/script/test_PDA.sh

For example, visualize our proposed method on Cityscapes validation set:

bash ./exp/spatial_correction_cityscapes/script/show.sh

Obtained results are saved in ./result/spatial_correction_cityscapes.

Conducting experiments on the CamVid dataset should follow the above procedure similarly.

Trained model

We provide trained model on Cityscapes and CamVid datasets. Please download models from: | model | Link | | :--: | :--: | | dmnet_camvid | Dropbox/BaiduYun(Access Code:iy69) | | spatial_correction_camvid | Dropbox/BaiduYun(Access Code:jh99) | | dmnet_cityscapes | Dropbox/BaiduYun(Access Code:rc7u) | | spatial_correction_cityscapes | Dropbox/BaiduYun(Access Code:5gem) |

Citation

@article{zhuang2020video,
  title={Video Semantic Segmentation with Distortion-Aware Feature Correction},
  author={Zhuang, Jiafan and Wang, Zilei and Wang, Bingke},
  journal={IEEE Transactions on Circuits and Systems for Video Technology},
  year={2020},
  doi={10.1109/TCSVT.2020.3037234}}
}