High-resolution networks and Segmentation Transformer for Semantic Segmentation

Branches

• This is the implementation for HRNet + OCR.
• The PyTroch 1.1 version ia available here.
• The PyTroch 0.4.1 version is available here.

News

• [2021/05/04] We rephrase the OCR approach as Segmentation Transformer pdf. We will provide the updated implementation soon.

• [2021/02/16] Based on the PaddleClas ImageNet pretrained weights, we achieve 83.22% on Cityscapes val, 59.62% on PASCAL-Context val (new SOTA), 45.20% on COCO-Stuff val (new SOTA), 58.21% on LIP val and 47.98% on ADE20K val. Please checkout openseg.pytorch for more details.

• [2020/08/16] MMSegmentation has supported our HRNet + OCR.

• [2020/07/20] The researchers from AInnovation have achieved Rank#1 on ADE20K Leaderboard via training our HRNet + OCR with a semi-supervised learning scheme. More details are in their Technical Report.

• [2020/07/09] Our paper is accepted by ECCV 2020: Object-Contextual Representations for Semantic Segmentation. Notably, the reseachers from Nvidia set a new state-of-the-art performance on Cityscapes leaderboard: 85.4% via combining our HRNet + OCR with a new hierarchical mult-scale attention scheme.

• [2020/03/13] Our paper is accepted by TPAMI: Deep High-Resolution Representation Learning for Visual Recognition.

• HRNet + OCR + SegFix: Rank #1 (84.5) in Cityscapes leaderboard. OCR: object contextual represenations pdf. HRNet + OCR is reproduced here.

• Thanks Google and UIUC researchers. A modified HRNet combined with semantic and instance multi-scale context achieves SOTA panoptic segmentation result on the Mapillary Vista challenge. See the paper.

• Small HRNet models for Cityscapes segmentation. Superior to MobileNetV2Plus ....

• Rank #1 (83.7) in Cityscapes leaderboard. HRNet combined with an extension of object context

• Pytorch-v1.1 and the official Sync-BN supported. We have reproduced the cityscapes results on the new codebase. Please check the pytorch-v1.1 branch.

Introduction

This is the official code of high-resolution representations for Semantic Segmentation. We augment the HRNet with a very simple segmentation head shown in the figure below. We aggregate the output representations at four different resolutions, and then use a 1x1 convolutions to fuse these representations. The output representations is fed into the classifier. We evaluate our methods on three datasets, Cityscapes, PASCAL-Context and LIP.

<!--  --> <figure> <text-align: center;> <img src="./figures/seg-hrnet.png" alt="hrnet" title="" width="900" height="150" /> </figcaption> </figure>

Besides, we further combine HRNet with Object Contextual Representation and achieve higher performance on the three datasets. The code of HRNet+OCR is contained in this branch. We illustrate the overall framework of OCR in the Figure and the equivalent Transformer pipelines:

<figure> <text-align: center;> <img src="./figures/OCR.PNG" alt="OCR" title="" width="900" height="200" /> </figure> <figure> <text-align: center;> <img src="./figures/SegmentationTransformerOCR.png" alt="Segmentation Transformer" title="" width="600" /> </figure>

Segmentation models

The models are initialized by the weights pretrained on the ImageNet. ''Paddle'' means the results are based on PaddleCls pretrained HRNet models. You can download the pretrained models from https://github.com/HRNet/HRNet-Image-Classification. Slightly different, we use align_corners = True for upsampling in HRNet.

1. Performance on the Cityscapes dataset. The models are trained and tested with the input size of 512x1024 and 1024x2048 respectively. If multi-scale testing is used, we adopt scales: 0.5,0.75,1.0,1.25,1.5,1.75.

| model | Train Set | Test Set | OHEM | Multi-scale| Flip | mIoU | Link | | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | | HRNetV2-W48 | Train | Val | No | No | No | 80.9 | Github/BaiduYun(Access Code:pmix)| | HRNetV2-W48 + OCR | Train | Val | No | No | No | 81.6 | Github/BaiduYun(Access Code:fa6i)| | HRNetV2-W48 + OCR | Train + Val | Test | No | Yes | Yes | 82.3 | Github/BaiduYun(Access Code:ycrk)| | HRNetV2-W48 (Paddle) | Train | Val | No | No | No | 81.6 | ---| | HRNetV2-W48 + OCR (Paddle) | Train | Val | No | No | No | --- | ---| | HRNetV2-W48 + OCR (Paddle) | Train + Val | Test | No | Yes | Yes | --- | ---|

2. Performance on the LIP dataset. The models are trained and tested with the input size of 473x473.

| model | OHEM | Multi-scale| Flip | mIoU | Link | | :--: | :--: | :--: | :--: | :--: | :--: | | HRNetV2-W48 | No | No | Yes | 55.83 | Github/BaiduYun(Access Code:fahi)| | HRNetV2-W48 + OCR | No | No | Yes | 56.48 | Github/BaiduYun(Access Code:xex2)| | HRNetV2-W48 (Paddle) | No | No | Yes | --- | --- | | HRNetV2-W48 + OCR (Paddle) | No | No | Yes | --- | ---|

Note Currently we could only reproduce HRNet+OCR results on LIP dataset with PyTorch 0.4.1.

3. Performance on the PASCAL-Context dataset. The models are trained and tested with the input size of 520x520. If multi-scale testing is used, we adopt scales: 0.5,0.75,1.0,1.25,1.5,1.75,2.0 (the same as EncNet, DANet etc.).

| model |num classes | OHEM | Multi-scale| Flip | mIoU | Link | | :--: | :--: | :--: | :--: | :--: | :--: | :--: | | HRNetV2-W48 | 59 classes | No | Yes | Yes | 54.1 | Github/BaiduYun(Access Code:wz6v)| | HRNetV2-W48 + OCR | 59 classes | No | Yes | Yes | 56.2 | Github/BaiduYun(Access Code:yyxh)| | HRNetV2-W48 | 60 classes | No | Yes | Yes | 48.3 | OneDrive/BaiduYun(Access Code:9uf8)| | HRNetV2-W48 + OCR | 60 classes | No | Yes | Yes | 50.1 | Github/BaiduYun(Access Code:gtkb)| | HRNetV2-W48 (Paddle) | 59 classes | No | Yes | Yes | --- | ---| | HRNetV2-W48 (Paddle) | 60 classes | No | Yes | Yes | --- | ---| | HRNetV2-W48 + OCR (Paddle) | 59 classes | No | Yes | Yes | --- | ---| | HRNetV2-W48 + OCR (Paddle) | 60 classes | No | Yes | Yes | --- | ---|

4. Performance on the COCO-Stuff dataset. The models are trained and tested with the input size of 520x520. If multi-scale testing is used, we adopt scales: 0.5,0.75,1.0,1.25,1.5,1.75,2.0 (the same as EncNet, DANet etc.).

| model | OHEM | Multi-scale| Flip | mIoU | Link | | :--: | :--: | :--: | :--: | :--: | :--: | | HRNetV2-W48 | Yes | No | No | 36.2 | Github/BaiduYun(Access Code:92gw)| | HRNetV2-W48 + OCR | Yes | No | No | 39.7 | Github/BaiduYun(Access Code:sjc4)| | HRNetV2-W48 | Yes | Yes | Yes | 37.9 | Github/BaiduYun(Access Code:92gw) | | HRNetV2-W48 + OCR | Yes | Yes | Yes | 40.6 | Github/BaiduYun(Access Code:sjc4) | | HRNetV2-W48 (Paddle) | Yes | No | No | --- | ---| | HRNetV2-W48 + OCR (Paddle) | Yes | No | No | --- | ---| | HRNetV2-W48 (Paddle) | Yes | Yes | Yes | --- | ---| | HRNetV2-W48 + OCR (Paddle) | Yes | Yes | Yes | --- | ---|

<!-- **Note** We reproduce HRNet+OCR results on COCO-Stuff dataset with PyTorch 0.4.1. -->

5. Performance on the ADE20