Learnable Tree Filter V2

By Lin Song, Yanwei Li, Zhengkai Jiang, Zeming Li, Xiangyu Zhang, Hongbin Sun, Jian Sun, Nanning Zheng.

This repo is an official implementation for "Rethinking Learnable Tree Filter for Generic Feature Transform" (NeurIPS2020) on PyTorch framework.

Installation

Requirements

• Python >= 3.6
• PyTorch >= 1.6 and torchvision
• OpenCV, needed by demo and visualization
• pycocotools:
  • pip install cython
  • pip install 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'
• prodict:
  • pip install prodict
• GCC >= 4.9

Build from source

• git clone https://github.com/StevenGrove/LearnableTreeFilterV2
• cd LearnableTreeFilterV2
• python setup.py build develop

Prepare data

See datasets/README.md.

Usage

All the projects are placed in cvpods_playground. You need to select a project and enter the corresponding folder.

# For example
cd cvpods_playground/mask_rcnn.res50.fpn.coco.1x.nn_syncbn.tf-v2

Training

# Running training procedure with specific GPU number
cvpods_train --gpu-nums <GPU_NUM> [optional arguments]

# Please refer to tools/train_net.py for more optional arguments

Inference

# Running inference procedure with specific GPU number
# The program will load the lastest model weights from ./log folder
cvpods_test --gpu-nums <GPU_NUM> [optional arguments]

# Running inference procedure with specific GPU number and model path
cvpods_test --gpu-nums <GPU_NUM> MODEL.WEIGHTS <model_path> [optional arguments]

# Please refer to tools/test_net.py for more optional arguments

Visualize your image [Optional]

# Please refer to demo/demo.py for more optional arguments
python ../../demo/demo.py --config <config_path> --input <input_path> --output <output_path> MODEL.WEIGHTS <model_path> [optional arguments]

Add to your code [Optional]

# Load the learnable tree filter module
from cvpods.layers import TreeFilterV2

# Initialize the module with specific number of channels and groups
tf = TreeFilterV2(<guide_channels>, <in_channels>, <embed_channels>, [num_groups])

# Run the filter procedure with input feature and guided feature
output_feature = tf(<input_feature>, <guided_feature>)

Performance

Instance Segmentation on COCO val set

Method | Backbone | Schedule | AP_box (%) | AP_mask (%) | Project | Model :--|:--:|:--:|:--:|:--:|:--:|:--: Mask-RCNN + LTF-V2 | ResNet-50 | 1x | 41.3 | 37.0 | Link | OneDrive Mask-RCNN + LTF-V2 | ResNet-101 | 1x | 42.5 | 38.0 | Link | OneDrive Mask-RCNN + LTF-V2 | ResNeXt-101 | 1x | 44.5 | 39.5 | Link | OneDrive Cascade Mask-RCNN + LTF-V2 | ResNeXt-101 | 1x | 46.9 | 40.3 | Link | OneDrive

Panoptic Segmentation on COCO val set

| Method | Backbone | Schedule | PQ (%) | AP_box (%) | AP_mask (%) | Project | Model | | :-------------------- | :-------: | :------: | :----: | :------------------: | :-------------------: | :----------------------------------------------------------: | :----------------------------------------------------------: | | Panoptic-FPN | ResNet-50 | 1x | 39.2 | 37.4 | 33.6 | Link | OneDrive | | Panoptic-FPN + LTF-V2 | ResNet-50 | 1x | 41.7 | 39.3 | 35.4 | Link | OneDrive |

More Results is Coming

To do

• [x] Experiments on panoptic segmentation
• [ ] Easier installation based on pip
• [ ] Modify transformer
• [ ] Application on video tasks

Citation

Please cite the paper in your publications if it helps your research.

@inproceedings{song2020rethinking,
    title = {Rethinking Learnable Tree Filter for Generic Feature Transform},
    author = {Song, Lin and Li, Yanwei and Jiang, Zhengkai and Li, Zeming and Zhang, Xiangyu and Sun, Hongbin and Sun, Jian and Zheng, Nanning},
    booktitle = {Advances in Neural Information Processing Systems},
    year = {2020}
}

Please cite this project in your publications if it helps your research.

@misc{learnabletreefilterv2,
    author = {Song, Lin},
    title = {LearnableTreeFilterV2},
    howpublished = {\url{https://github.com/StevenGrove/LearnableTreeFilterV2}},
    year = {2020}
}