SOLIDER

<div align="center"><img src="assets/logo.png" width="900"></div>

Welcome to SOLIDER! SOLIDER is a Semantic Controllable Self-Supervised Learning Framework to learn general human representations from massive unlabeled human images which can benefit downstream human-centric tasks to the maximum extent. Unlike the existing self-supervised learning methods, prior knowledge from human images is utilized in SOLIDER to build pseudo semantic labels and import more semantic information into the learned representation. Meanwhile, different downstream tasks always require different ratios of semantic information and appearance information, and a single learned representation cannot fit for all requirements. To solve this problem, SOLIDER introduces a conditional network with a semantic controller, which can fit different needs of downstream tasks. For more details, please refer to our paper Beyond Appearance: a Semantic Controllable Self-Supervised Learning Framework for Human-Centric Visual Tasks.

<div align="center"><img src="assets/framework.png" width="900"></div>

Updates

• [2023/07/21: Codes of human pose task is released!]
  • Training details of our pretrained model on downstream human pose task is released.
• [2023/05/15: Codes of human parsing task is released!]
  • Training details of our pretrained model on downstream human parsing task is released.
• [2023/04/24: Codes of attribute recognition task is released!]
  • Training details of our pretrained model on downstream person attribute recognition task is released.
• [2023/03/28: Codes of 3 downstream tasks are released!]
  • Training details of our pretrained model on 3 downstream human visual tasks, including person re-identification, person search and pedestrian detection, are released.
• [2023/03/13: SOLIDER is accepted by CVPR2023!]
  • The paper of SOLIDER is accepted by CVPR2023, and its offical pytorch implementation is released in this repo.

Installation

This codebase has been developed with python version 3.7, PyTorch version 1.7.1, CUDA 10.1 and torchvision 0.8.2.

Datasets

We use LUPerson as our training data, which consists of unlabeled human images. Download LUPerson from its offical link and unzip it.

Training

• Choice 1. To train SOLIDER from scratch, please run:

sh run_solider.sh

• Choice 2. Training SOLIDER from scratch may take a long time. To speed up the training, you can train a DINO model first, and then finetune it with SOLIDER, as follows:

sh run_dino.sh
sh resume_solider.sh

Finetuning and Inference

There is a demo to run the trained SOLIDER model, which can be embedded into the inference or the downstream task finetuning.

python demo.py

Models

We use Swin-Transformer as our backbone, which shows great advantages on many CV tasks. | Task | Dataset | Swin Tiny<br>(Link) | Swin Small<br>(Link) | Swin Base<br>(Link) | | :---: |:---: |:---: | :---: | :---: | | Person Re-identification (mAP/R1)<br>w/o re-ranking | Market1501 | 91.6/96.1 | 93.3/96.6 | 93.9/96.9 | | | MSMT17 | 67.4/85.9 | 76.9/90.8 | 77.1/90.7 | | Person Re-identification (mAP/R1)<br>with re-ranking | Market1501 | 95.3/96.6 | 95.4/96.4 | 95.6/96.7 | | | MSMT17 | 81.5/89.2 | 86.5/91.7 | 86.5/91.7 | | Attribute Recognition (mA) | PETA_ZS | 74.37 | 76.21 | 76.43 | | | RAP_ZS | 74.23 | 75.95 | 76.42 | | | PA100K | 84.14 | 86.25 | 86.37 | | Person Search (mAP/R1) | CUHK-SYSU | 94.9/95.7 | 95.5/95.8 | 94.9/95.5 | | | PRW | 56.8/86.8 | 59.8/86.7 | 59.7/86.8 | | Pedestrian Detection (MR-2) | CityPersons | 10.3/40.8 | 10.0/39.2 | 9.7/39.4 | | Human Parsing (mIOU) | LIP | 57.52 | 60.21 | 60.50 | | Pose Estimation (AP/AR) | COCO | 74.4/79.6 | 76.3/81.3 | 76.6/81.5 |

• All the models are trained on the whole LUPerson dataset.

Traning codes on Downstream Tasks

• Person Re-identification
• Person Search
• Pedestrian Detection
• Person Attribute Recognition
• Human Parsing
• Pose Estimation

Acknowledgement

Our implementation is mainly based on the following codebases. We gratefully thank the authors for their wonderful works.

• Swin-Transformer
• DINO
• TransReID
• TransReID-SSL
• SeqNet
• Pedestron
• LUPerson
• SCHP
• mmpose

Reference

If you use SOLIDER in your research, please cite our work by using the following BibTeX entry:

@inproceedings{chen2023beyond,
  title={Beyond Appearance: a Semantic Controllable Self-Supervised Learning Framework for Human-Centric Visual Tasks},
  author={Weihua Chen and Xianzhe Xu and Jian Jia and Hao Luo and Yaohua Wang and Fan Wang and Rong Jin and Xiuyu Sun},
  booktitle={The IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2023},
}