gb3mppe (WORK IN PROGRESS)

This is a re-implementation for:

Graph-Based 3D Multi-Person Pose Estimation Using Multi-View Images

Size Wu, Sheng Jin, Wentao Liu, Lei Bai, Chen Qian, Dong Liu, Wanli Ouyang 2021

Installation

1. Clone this repo, and we'll call the directory that you cloned multiview-multiperson-pose as ${gb3mppe}.
2. Install dependencies.

MMG : Multi-view Matching Graph Module

<img src="data/dee.png" width="600"/>

Data preparation for MMG

We train and evaluate our model on the Campus.

1. We processed the ground-truth and the 2D pose estimated by Hanyue Tu, Chunyu Wang, Wenjun Zeng to our format.
2. We also created dataset in form of python dictionnaries (pkl files) to store the node features and the edge features which ar the inputs of our MMG model.
3. You can download all these data from here and place it in gb3mppe/data/campus.
4. You can also try to generate the node and edge features by runing generate_mmg_features.py and generate_edge_features.py resp. , the node features are extracted from feature maps that ar constructed with 2D pose estimator trained on COCO. These feature maps are the output of the two last deconv layers of the prerained PoseResnet.

The directory tree should look like this:

${gb3mppe}
|-- data
    |-- campus
        |-- campuse_pose_voxel.pkl
        |-- cfg.yaml
        |-- Camera4
        |-- GT_2d_human_centers.pkl
        |-- GT_3d_human_centers.pkl
        |-- node_features.pkl
        |-- personne1.txt
        |-- personne1_3D.txt
        |-- personne2.txt
        |-- personne2_3D.txt
        |-- personne3.txt
        |-- personne3_3D.txt
        |-- voxel_2d_human_centers.pkl
        |-- voxel_3d_human_centers.pkl
        |-- CampusSeq1
            |-- Camera0
            |-- Camera1
            |-- Camera2
            |-- actorsGT.mat
            |-- calibration_campus.json
            |-- pred_campus_maskrcnn_hrnet_coco.pkl
|-- lib
    |-- dataset
    |-- gt_coord_2D
    |-- models
    |-- utils              
|-- test

here's the link to donwload CampusSeq1 .

CRG : Center Refinement Graph

We also implementred the CRG's architecture, but unfortunately, we did not have enough time to generate the node and edge features to train the model. However, you can test the implementation on a dummy dataset that we created, where inputs have the same shape than the real features. To do that, you can run this notebook test

PRG : Pose Regression Graph

Like CRG, you can test the PRG's implementation on a dummy dataset that we created, where inputs have the same shape than the real features. To do that, you can run the same notebook

MMG Training:

To train and visualize an MMG model on 400 samples from the campus dataset, run the following command by replacing "n_epochs" with the number of training epochs (integer) and "show" by whether to show the training curve or not (True or False).

python main.py n_epochs show

example:

python main.py 10 True