Simultaneous Edge Alignment and Learning (SEAL)

By Zhiding Yu, Weiyang Liu, Yang Zou, Chen Feng, Srikumar Ramalingam, B. V. K. Vijayakumar and Jan Kautz

License

SEAL is released under the MIT License (refer to the LICENSE file for details).

Introduction
Citation
Requirements
Installation
Usage
Video Demo
Note
References
Contact

Introduction

The repository contains the entire pipeline (including data preprocessing, training, testing, label refine, evaluation, visualization, and demo generation, etc) for SEAL.

SEAL is a recently proposed learning framework towards edge learning under noisy labels. The framework seeks to directly generate high quality thin/crisp object semantic boundaries without any post-processing, by jointly performing edge alignment with edge learning. In particular, edge alignment is formulated as latent variable optimization and learned end-to-end during network training. For more details, please refer to the arXiv technical report and the ECCV18 paper. We highly recommend the readers refer to arXiv for latest updates in detailed description and experiments.

We use CASENet as the backbone network for SEAL since it is the state-of-the-art deep network for category-aware semantic edge detection. CASENet adopts a modified ResNet-101 architecture with dilated convolution. More details about CASENet can be found in the paper and the prototxt file. Note that this code has been designed to fully support training/testing CASENet by changing a few input parameters. The original implementation of CASENet is available here.

SEAL currently achieves the state-of-the-art category-aware semantic edge detection performance on the Semantic Boundaries Dataset and the Cityscapes Dataset. Note that the benchmarks adopted in this work differs from the original SBD benchmark, and it is recommended to use the proposed ones in future related research for more precised evaluation.

Citation

If you find SEAL useful in your research, please consider to cite the following papers:

@inproceedings{yu2018seal, 
  title={Simultaneous Edge Alignment and Learning}, 
  author={Yu, Zhiding and Liu, Weiyang and Zou, Yang and Feng, Chen and Ramalingam, Srikumar and Vijaya Kumar, BVK and Kautz, Jan},
  booktitle={European Conference on Computer Vision (ECCV)}, 
  year={2018}
}

@inproceedings{yu2017casenet,
 title={uppercase{CASEN}et: Deep Category-Aware Semantic Edge Detection},
 author={Yu, Zhiding and Feng, Chen and Liu, Ming-Yu and Ramalingam, Srikumar},
 booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
 year={2017}
}

Requirements

Requirements for Matlab
Requirements for Caffe and matcaffe (see: Caffe installation instructions and SEAL Caffe distribution readme)

Installation

Clone the SEAL repository. We'll call the directory that you cloned SEAL as SEAL_ROOT.
```
git clone --recursive https://github.com/Chrisding/seal.git
```

Build Caffe and matcaffe

cd $SEAL_ROOT/caffe
# Follow the Caffe installation instructions to install all required packages:
# http://caffe.berkeleyvision.org/installation.html
# Follow the instructions to install matio:
# https://sourceforge.net/projects/matio/files/matio/1.5.2/
make all -j8 && make matcaffe

Usage

Upon successfully compiling the SEAL Caffe distribution, you can run the following experiments.

Part 1: Preprocessing

SBD Data: In this part, we assume you are in the directory $SEAL_ROOT/data/sbd-preprocess/.

Download the SBD dataset (with both original and CRF-preprocessed SBD ground truths) from Google Drive | Baidu Yun, and place the tarball sbd.tar.gz in data_orig/. Run the following command:
```
tar -xvzf data_orig/sbd.tar.gz -C data_orig && rm data_orig/sbd.tar.gz
```
Perform data augmentation and generate training edge labels by running the following command:
```
# In Matlab Command Window
run code/demoPreproc.m
```
This will create augmented images and their instance-sensitive(inst)/non-instance-sensitive(cls) edge labels for network training in data_proc/.
Generate edge ground truths for evaluation by running the following command:
```
# In Matlab Command Window
run code/demoGenGT.m
```
This will create two folders (gt_orig_thin/ and gt_orig_raw/) in the directory of gt_eval/, containing the thinned and unthinned evaluation ground truths from the original SBD data.

We do not provide the code to compute evaluation ground truths from the re-annotated SBD test set. You can download the tarball containing the precomputed ground truths from Google Drive | Baidu Yun, and place the tarball gt_reanno.tar.gz in gt_eval/. Run the following command:
```
tar -xvzf gt_eval/gt_reanno.tar.gz -C gt_eval && rm gt_eval/gt_reanno.tar.gz
```

Cityscapes Data: In this part, we assume you are in the directory $SEAL_ROOT/data/cityscapes-preprocess/. Note that in this repository, all Cityscapes pipelines are instance-sensitive only.

Download the files gtFine_trainvaltest.zip, leftImg8bit_trainvaltest.zip and leftImg8bit_demoVideo.zip from the Cityscapes website to data_orig/, and unzip them:

unzip data_orig/gtFine_trainvaltest.zip -d data_orig && rm data_orig/gtFine_trainvaltest.zip
unzip data_orig/leftImg8bit_trainvaltest.zip -d data_orig && rm data_orig/leftImg8bit_trainvaltest.zip
unzip data_orig/leftImg8bit_demoVideo.zip -d data_orig && rm data_orig/leftImg8bit_demoVideo.zip

Generate training edge labels by running the following command:
```
# In Matlab Command Window
run code/demoPreproc.m
```
This will create instance-sensitive edge labels for network training in data_proc/.
Generate edge ground truths for evaluation by running the following command:
```
# In Matlab Command Window
run code/demoGenGT.m
```
This will create two folders (gt_thin/ and gt_raw/) in the directory of gt_eval/, containing the thinned and unthinned evaluation ground truths.

Part 2: Training

Train on SBD: In this part, we assume you are in the directory $SEAL_ROOT/exper/sbd/.

Download the init model (for CASENet) and warm-up init models (for CASENet-S/CASENet-C/SEAL) from Google Drive | Baidu Yun and put the zip file model_init.zip in model/. Run the following command:
```
unzip model/model_init.zip -d model
```
The models model_init_inst_warm and model_init_cls_warm are warm-up init models that are obtained by additionally training a few iterations from model_init with unweighted sigmoid cross-entropy loss and relatively small learning rates. It is used to initialize models that involve unweighted sigmoid cross-entropy loss and stabilize their early training, since the loss is harder to learn than the reweighted one. Alternatively, you can obtain your own warm-up models using the code.
To train the network models, call the solve function with the following input argument format:
```
solve(<data_root>, <file_list_path>, <init_model_path>, <snapshot_prefix>, <iter_num>, <lr>, <gpu_id>, <loss_type>, <sigma_x>, <sigma_y>, <lambda>)
```
By choosing the last four input, one could train the models of SEAL and all baselines (CASENet, CASENet-S and CASENet-C) reported in the paper. For example, to train SEAL with instance-sensitive(IS)/non-instance-sensitive(non-IS) edge labels from the original SBD data, run the following commands:
```
matlab -nodisplay -r "solve('../../data/sbd-preprocess/data_proc', '../../data/sbd-preprocess/data_proc/trainvalaug_inst_orig.mat', './model/model_init_inst_warm.caffemodel', 'model_inst_seal', 22000, 5.0*10^-8, <gpu_id>, 'unweight', 1, 4, 0.02)" 2>&1 | tee ./log/seal_inst.txt
matlab -nodisplay -r "solve('../../data/sbd-preprocess/data_proc', '../../data/sbd-preprocess/data_proc/trainvalaug_cls_orig.mat', './model/model_init_cls_warm.caffemodel', 'model_cls_seal', 22000, 5.0*10^-8, <gpu_id>, 'unweight', 1, 4, 0.02)" 2>&1 | tee ./log/seal_cls.txt
```
This will output model snapshots in the model/ folder and log files in the log/ folder. You may leave snapshot_prefix as empty ([]), which will automatically give a prefix composed by file list name and param configs. We assume training on 12G memory GPUs (such as TitanX/XP) without any other occupation. If you happen to have smaller GPU memories, consider decreasing the default 472x472 training crop size in **`s

Seal

Install / Use

README