SipMask
SipMask: Spatial Information Preservation for Fast Image and Video Instance Segmentation (ECCV2020)
Install / Use
/learn @JialeCao001/SipMaskREADME
SipMask
This is the official implementation of "SipMask: Spatial Information Preservation for Fast Image and Video Instance Segmentation (ECCV2020)" built on the open-source mmdetection and maskrcnn-benchmark.
- Single-stage method for both image and video instance segmentation.
- Two different versions are provided: high-accuracy version and real-time (fast) version.
- Image instance segmentation is built on both mmdetection and maskrcnn-benchmark.
- Video instance segmentation is built on mmdetection.
- Datasets: MS COCO for image instance segmentation and YouTube-VIS for video instance segmentation.
- Json results of SipMask+ResNet50 on YouTube-VIS: Single-scale training (32.5) and Multi-scale training (33.7)
Introduction
Single-stage instance segmentation approaches have recently gained popularity due to their speed and simplicity, but are still lagging behind in accuracy, compared to two-stage methods. We propose a fast single-stage instance segmentation method, called SipMask, that preserves instance-specific spatial information by separating the mask prediction of an instance to different sub-regions of a detected bounding-box. Our main contribution is a novel light-weight spatial preservation (SP) module that generates a separate set of spatial coefficients for each sub-region within a bounding-box, leading to improved mask predictions. It also enables accurate delineation of spatially adjacent instances. Further, we introduce a mask alignment weighting loss and a feature alignment scheme to better correlate mask prediction with object detection.
SipMask-benchmark (image instance segmentation)
- This project is built on the official implementation of FCOS, which is based on maskrcnn-benchmark.
- High-quality version is provided.
- Please use SipMask-benchmark and refer to INSTALL.md for installation.
- PyTorch1.1.0 and cuda9.0/10.0 are used by me.
Train with multiple GPUs
python -m torch.distributed.launch --nproc_per_node=4 --master_port=$((RANDOM+10000)) tools/train_net.py --config-file ${CONFIG_FILE} DATALOADER.NUM_WORKERS 2 OUTPUT_DIR ${OUTPUT_PATH}
e.g.,
python -m torch.distributed.launch --nproc_per_node=4 --master_port=$((RANDOM+10000)) tools/train_net.py --config-file configs/sipmask/sipmask_R_50_FPN_1x.yaml DATALOADER.NUM_WORKERS 2 OUTPUT_DIR training_dir/sipmask_R_50_FPN_1x
Test with a single GPU
python tools/test_net.py --config-file ${CONFIG_FILE} MODEL.WEIGHT ${CHECKPOINT_FILE} TEST.IMS_PER_BATCH 4
e.g.,
python tools/test_net.py --config-file configs/sipmask/sipmask_R_50_FPN_1x.yaml MODEL.WEIGHT training_dir/SipMask_R50_1x.pth TEST.IMS_PER_BATCH 4
- CONFIG_FILE of Sipmask is under the folder of SipMask-benchmark/configs/sipmask.
Results
| name | backbone | input size | epoch | ms-train | val. box AP | val. mask AP | download| | :--------:| :-------------: | :-----------:| :---------: | :-----: | :------: | :------: | :----------: | | SipMask | R50 | 800 × 1333| 1x | no | 39.5 | 34.2 | model | | SipMask | R101 | 800 × 1333 | 3x | yes | 44.1 | 37.8 | model |
SipMask-mmdetection (image instance segmentation)
- This project is built on mmdetection.
- High-quality version and real-time version are both provided.
- Please use SipMask-mmdetection and refer to INSTALL.md for installation.
- PyTorch1.1.0, cuda9.0/10.0, and mmcv0.4.3 are used by me.
Train with multiple GPUs
./tools/dist_train.sh ${CONFIG_FILE} ${GPU_NUM} [optional arguments]
e.g.,
CUDA_VISIBLE_DEVICES=0,1,2,3 ./tools/dist_train.sh configs/sipmask/sipmask_r50_caffe_fpn_gn_1x_4gpu.py 4 --validate
Test with a single GPU
python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] [--eval ${EVAL_METRICS}] [--show]
e.g.,
python tools/test.py ./configs/sipmask/sipmask_r50_caffe_fpn_gn_1x_4gpu.py ./work_dirs/sipmask_r50_caffe_1x.pth --out results.pkl --eval bbox segm
Inference with saved results
With our trained model, detection results of an image can be visualized using the following command.
python ./demo/sipmask_demo.py ${CONFIG_FILE} ${CHECKPOINT_FILE} ${IMAGE_FILE} [--out ${OUT_PATH}]
e.g.,
python ./demo/sipmask_demo.py ./configs/sipmask/sipmask_r50_caffe_fpn_gn_1x_4gpu.py ./sipmask_r50_caffe_1x.pth ./demo/demo.jpg --out ./demo/aa.jpg
- CONFIG_FILE of Sipmask is under the folder of SipMask-mmdetection/configs/sipmask.
Results
| name | backbone | input size | epoch | ms-train | GN | val. box AP | val. mask AP | download| | :---------: | :-------------: | :-------------------: | :-----: | :-----: | :-----:| :-----: |:------: | :-----------------: | | SipMask | R50 | 800×1333 | 1x | no | yes | 38.2 | 33.5 | model | | SipMask | R50 | 800×1333 | 2x | yes | yes | 40.8 | 35.6 | model | | SipMask | R101 | 800×1333 | 4x | yes | yes | 43.6 | 37.8 | model | | SipMask | R50 | 544×544 | 6x | yes | no | 36.0 | 31.7 | model | | SipMask | R50 | 544×544 | 10x | yes | yes | 37.1 | 32.4 | model | | SipMask | R101 | 544×544 | 6x | yes | no | 38.4 | 33.6 | model | | SipMask | R101 | 544×544 | 10x | yes | yes | 40.3 | 34.8 | model | | SipMask++ | R101-D | 544×544 | 6x | yes | no | 40.1 | 35.2 | model | | SipMask++ | R101-D | 544×544 | 10x | yes | yes | 41.3 | 36.1 | model |
- GN indicates group normalization used in prediction branch.
- Model with the input size of 800×1333 fcoses on high accuracy, which is trained in RetinaNet style.
- Model with the input size of 544×544 fcoses on fast speed, which is trained in SSD style.
- ++ indicates adding deformable convolutions with interval of 3 in backbone and mask re-scoring module.
SipMask-VIS (video instance segmentation)
- This project is an implementation for video instance segmenation based on mmdetection.
- Please use SipMask-VIS and refer to INSTALL.md for installation.
- PyTorch1.1.0, cuda9.0/10.0, and mmcv0.2.12 are used by me.
Please note that, to run YouTube-VIS dataset like MaskTrackRCNN, install the cocoapi for youtube-vis instead of installing the original cocoapi for coco as follows.
pip install git+https://github.com/youtubevos/cocoapi.git#"egg=pycocotools&subdirectory=PythonAPI"
or
cd SipMask-VIS/pycocotools/cocoapi/PythonAPI
python setup.py build_ext install
Train with multiple GPUs
./tools/dist_train.sh ${CONFIG_FILE} ${GPU_NUM}
e.g.,
CUDA_VISIBLE_DEVICES=0,1,2,3 ./toools/dist_train.sh ./configs/sipmask/sipmask_r50_caffe_fpn_gn_1x_4gpu.py 4
Test with a single GPU
python tools/test_video.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] --eval segm
e.g.,
python ./tools/test_video.py configs/sipmask/sipmask_r50_caffe_fpn_gn_1x_4gpu.py ./work_dirs/sipmask_r50_fpn_1x.pth --out results.pkl --eval segm
If you want to save the results of video instance segmentation, please use the following command:
python tools/test_video.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] --eval segm --show --save_path= ${SAVE_PATH}
- CONFIG_FILE of SipMask-VIS is under the folder of [SipMask-VIS/configs/sipmask](SipMask-VIS/config
