Face Alignment in Full Pose Range: A 3D Total Solution

<!-- By [Jianzhu Guo](https://guojianzhu.com/aboutme.html). -->

By Jianzhu Guo.

<p align="center"> <img src="samples/obama_three_styles.gif" alt="obama"> </p>

[Updates]

• 2022.5.14: Recommend a python implementation of face profiling: face_pose_augmentation.
• 2020.8.30: The pre-trained model and code of ECCV-20 are made public on 3DDFA_V2, the copyright is explained by Jianzhu Guo and the CBSR group.
• 2020.8.2: Update a <strong>simple c++ port</strong> of this project.
• 2020.7.3: The extended work <strong>Towards Fast, Accurate and Stable 3D Dense Face Alignment</strong> is accepted by ECCV 2020. See my page for more details.
• 2019.9.15: Some updates, see the commits for details.
• 2019.6.17: Adding a video demo contributed by zjjMaiMai.
• 2019.5.2: Evaluating inference speed on CPU with PyTorch v1.1.0, see here and speed_cpu.py.
• 2019.4.27: A simple render pipeline running at ~25ms/frame (720p), see rendering.py for more details.
• 2019.4.24: Providing the demo building of obama, see demo@obama/readme.md for more details.
• 2019.3.28: Some updates.
• 2018.12.23: Add several features: depth image estimation, PNCC, PAF feature and obj serialization. See dump_depth, dump_pncc, dump_paf, dump_obj options for more details.
• 2018.12.2: Support landmark-free face cropping, see dlib_landmark option.
• 2018.12.1: Refine code and add pose estimation feature, see utils/estimate_pose.py for more details.
• 2018.11.17: Refine code and map the 3d vertex to original image space.
• 2018.11.11: Update end-to-end inference pipeline: infer/serialize 3D face shape and 68 landmarks given one arbitrary image, please see readme.md below for more details.
• 2018.10.4: Add Matlab face mesh rendering demo in visualize.
• 2018.9.9: Add pre-process of face cropping in benchmark.

[Todo]

• [x] Add c++ port.
• [x] Depth image estimation.
• [x] PNCC (Projected Normalized Coordinate Code).
• [x] PAF (Pose Adaptive Feature).
• [x] Obj serialization with sampled texture.
• [x] Recommendation of fast face detectors: FaceBoxes.PyTorch, libfacedetection, ZQCNN
• [x] Training details
• [x] Face Profiling: Official Matlab code, Python version <!--(http://www.cbsr.ia.ac.cn/users/xiangyuzhu/projects/3DDFA/Code/FaceProfilingRelease_v1.1.zip) -->

Introduction

This repo holds the pytorch improved version of the paper: Face Alignment in Full Pose Range: A 3D Total Solution. Several works beyond the original paper are added, including the real-time training, training strategies. Therefore, this repo is an improved version of the original work. As far, this repo releases the pre-trained first-stage pytorch models of MobileNet-V1 structure, the pre-processed training&testing dataset and codebase. Note that the inference time is about 0.27ms per image (input batch with 128 images as an input batch) on GeForce GTX TITAN X.

<!-- Note that if your academic work use the code of this repo, you should cite this repo not the original paper.--> <!-- One related blog will be published for some important technique details in future. --> <!-- Why not evaluate it on single image? Because most time for single image is spent on function call. The inference speed is equal to MobileNet-V1 with 120x120x3 tensor as input, therefore it is possible to convert to mobile devices. -->

This repo will keep updating in my spare time, and any meaningful issues and PR are welcomed.

Several results on ALFW-2000 dataset (inferenced from model phase1_wpdc_vdc.pth.tar) are shown below.

<p align="center"> <img src="imgs/landmark_3d.jpg" alt="Landmark 3D" width="1000px"> </p> <p align="center"> <img src="imgs/vertex_3d.jpg" alt="Vertex 3D" width="750px"> </p>

Applications & Features

1. Face Alignment

<p align="center"> <img src="samples/dapeng_3DDFA_trim.gif" alt="dapeng"> </p>

2. Face Reconstruction

<p align="center"> <img src="samples/5.png" alt="demo" width="750px"> </p>

3. 3D Pose Estimation

<p align="center"> <img src="samples/pose.png" alt="tongliya" width="750px"> </p>

4. Depth Image Estimation

<p align="center"> <img src="samples/demo_depth.jpg" alt="demo_depth" width="750px"> </p>

5. PNCC & PAF Features

<p align="center"> <img src="samples/demo_pncc_paf.jpg" alt="demo_pncc_paf" width="800px"> </p>

Getting started

Requirements

• PyTorch >= 0.4.1 (PyTorch v1.1.0 is tested successfully on macOS and Linux.)
• Python >= 3.6 (Numpy, Scipy, Matplotlib)
• Dlib (Dlib is optionally for face and landmarks detection. There is no need to use Dlib if you can provide face bouding bbox and landmarks. Besides, you can try the two-step inference strategy without initialized landmarks.)
• OpenCV (Python version, for image IO operations.)
• Cython (For accelerating depth and PNCC render.)
• Platform: Linux or macOS (Windows is not tested.)

# installation structions
sudo pip3 install torch torchvision # for cpu version. more option to see https://pytorch.org
sudo pip3 install numpy scipy matplotlib
sudo pip3 install dlib==19.5.0 # 19.15+ version may cause conflict with pytorch in Linux, this may take several minutes. If 19.5 version raises errors, you may try 19.15+ version.
sudo pip3 install opencv-python
sudo pip3 install cython

In addition, I strongly recommend using Python3.6+ instead of older version for its better design.

Usage

1. Clone this repo (this may take some time as it is a little big)

   git clone https://github.com/cleardusk/3DDFA.git  # or git@github.com:cleardusk/3DDFA.git
   cd 3DDFA
   

   Then, download dlib landmark pre-trained model in Google Drive or Baidu Yun, and put it into models directory. (To reduce this repo's size, I remove some large size binary files including this model, so you should download it : ) )

2. Build cython module (just one line for building)

   cd utils/cython
   python3 setup.py build_ext -i
   

   This is for accelerating depth estimation and PNCC render since Python is too slow in for loop.

3. Run the main.py with arbitrary image as input

   python3 main.py -f samples/test1.jpg
   

   If you can see these output log in terminal, you run it successfully.

   Dump tp samples/test1_0.ply
   Save 68 3d landmarks to samples/test1_0.txt
   Dump obj with sampled texture to samples/test1_0.obj
   Dump tp samples/test1_1.ply
   Save 68 3d landmarks to samples/test1_1.txt
   Dump obj with sampled texture to samples/test1_1.obj
   Dump to samples/test1_pose.jpg
   Dump to samples/test1_depth.png
   Dump to samples/test1_pncc.png
   Save visualization result to samples/test1_3DDFA.jpg
   

   Because test1.jpg has two faces, there are two .ply and .obj files (can be rendered by Meshlab or Microsoft 3D Builder) predicted. Depth, PNCC, PAF and pose estimation are all set true by default. Please run python3 main.py -h or review the code for more details.

   The 68 landmarks visualization result samples/test1_3DDFA.jpg and pose estimation result samples/test1_pose.jpg are shown below:

<p align="center"> <img src="samples/test1_3DDFA.jpg" alt="samples" width="650px"> </p> <p align="center"> <img src="samples/test1_pose.jpg" alt="samples" width="650px"> </p>

4. Additional example

   python3 ./main.py -f samples/emma_input.jpg --bbox_init=two --dlib_bbox=false
   

<p align="center"> <img src="samples/emma_input_3DDFA.jpg" alt="samples" width="750px"> </p> <p align="center"> <img src="samples/emma_input_pose.jpg" alt="samples" width="750px"> </p>

Inference speed

CPU

Just run

python3 speed_cpu.py

On my MBP (i5-8259U CPU @ 2.30GHz on 13-inch MacBook Pro), based on PyTorch v1.1.0, with a single input, the running output is:

Inference speed: 14.50±0.11 ms

<!-- [speed_cpu.py](./speed_cpu.py) -->

GPU

When input batch size is 128, the total inference time of MobileNet-V1 takes about 34.7ms. The average speed is about 0.27ms/pic.

<p align="center"> <img src="imgs/inference_speed.png" alt="Inference speed" width="600px"> </p>

Training details

The training scripts lie in training directory. The related resources are in below table.

| Data | Download Link | Description | |:-:|:-:|:-:| | train.configs | BaiduYun or Google Drive, 217M | The directory containing 3DMM params and filelists of training dataset | | train_aug_120x120.zip | BaiduYun or Google Drive, 2.15G | The cropped imag