NextFace

NextFace is a light-weight pytorch library for high-fidelity 3D face reconstruction from monocular image(s) where scene attributes –3D geometry, reflectance (diffuse, specular and roughness), pose, camera parameters, and scene illumination– are estimated. It is a first-order optimization method that uses pytorch autograd engine and ray tracing to fit a statistical morphable model to an input image(s).

<p align="center"><img src="resources/emily.png" style="float: left; width: 23%; margin-right: 1%; margin-bottom: 0.5em;"><img src="resources/emily.gif" style="float: left; margin-right: 1%; margin-bottom: 0.5em;"> <img src="resources/beard.png" style="float: left; width: 23%; margin-right: 1%; margin-bottom: 0.5em;"><img src="resources/beard.gif" style="float: left; margin-right: 1%; margin-bottom: 0.5em;"><img src="resources/visual.jpg" ></p> <p align="center"> <strong>A demo on youtube from here:</strong> </p> <p align="center"> <a href="http://www.youtube.com/watch?v=bPFp0oZ9plg" title="Practical Face Reconstruction via Differentiable Ray Tracing"><img src="http://img.youtube.com/vi/bPFp0oZ9plg/0.jpg" alt="Practical Face Reconstruction via Differentiable Ray Tracing" /></a> </p>

News

• 19 March 2023: fixed a bug in the optimizer where the gradients were not activated for the camera pos (rotation and translation). Also I added a new optimization strategy for the second and third stage which should improve the overall performance. plz pull
• 21 June 2022: Many thanks for Jack Saunders for adding this new feature to NextFace: Added support for mediapipe as a replacement for FAN landmarks detector. Mediapipe produces much stable and accurate results than FAN . To try mediapipe, you need to pull the new version of the code and install mediapipe ** pip install mediapipe** . Now by default the landmarks detector used is mediapipe, if you want to switch bach to FAN plz edit the optimConfig.ini file (set lamdmarksDetectorType = 'fan')
• 01 May 2022: i you want to generate an animation like the one of the gif files in the readme that rotates the reconstruction on the vertical axis, plz run the replay.py script and give it the path of the pickle file that contains the optimized scene attributes ( located in checkpoints/stage3_output.pickle).
• 26 April 2022: I added export of the estimated light map (as an environment map). this can be useful if you want to render the face with other rendering engines (Unreal, Unity, OpenGL). plz pull the code. U can choose to export the lightmap as png or exr (check optimConfig.ini)
• 25 April 2022: if you want to generate textures with higher resolutions (1024x1024 or 2048x2048) I have added these two maps here : https://github.com/abdallahdib/NextFace/releases. to use these maps, download uvParametrization.2048.pickle and uvParametrization.1024.pickle and put them inside baselMorphableModel directory and change the textureResolution in optimConfig.in to 1024 or 2048. Also dont forget to pull the latest code. Plz note that with these large uvmaps optimization will require more cpu/gpu memory.
• 24 April 2022: added a colab notebook in: demo.ipynb.
• 20 April 2022: I replaced landmarks association file with a new one which gives better reconstruction, especially on face coutours. Plz pull
• 20 April 2022: I tried NextFace on a challenging face and surprisingly we still get appealing reconstruction, check below:

<p align="center"><img src="resources/results1.gif" style="float: left; width: 50%; margin-right: 1%; margin-bottom: 0.5em;"></p>

Features:

• Reconstructs face at high fidelity from single or multiple RGB images
• Estimates face geometry
• Estimates detailed face reflectance (diffuse, specular and roughness)
• Estimates scene light with spherical harmonics
• Estimates head pose and orientation
• Runs on both cpu and cuda-enabled gpu

Installation

• Clone the repository
• Execute the commands in 'INSTALL' file. these commands create a new conda environment called faceNext and install required packages. An 'environment.yml' is also provided. The library is tested with torch 1.3.1, torchvision 0.4.2 and cuda toolkit 10.1, but it should also work with recent pytorch versions.
• Activate the environment: conda activate nextFace
• Download basel face model from here, just fill the form and you will receive an instant direct download link into your inbox. Downloaded model2017-1_face12_nomouth.h5 file and put it inside ./baselMorphableModel directory
• Download the albedo face model albedoModel2020_face12_albedoPart.h5 from here and put it inside ./baselMorphableModel directory

How to use

Reconstruction from a single image

• to reconstruct a face from a single image: run the following command:
  • python optimizer.py --input path-to-your-input-image --output output-path-where-to-save-results

Reconstruction from multiple images (batch reconstruction)

• In case you have multiple images with same resolution, u can run a batch optimization on these images. For this, put all ur images in the same directory and run the following command:
  • python optimizer.py --input path-to-your-folder-that-contains-all-ur-images --output output-path-where-to-save-results

Reconstruction from mutliple images for the same person

• if you have multiple images for the same person, put these images in the same folder and run the following command:

  • python optimizer.py --sharedIdentity --input path-to-your-folder-that-contains-all-ur-images --output output-path-where-to-save-results

  the sharedIdentity flag tells the optimizer that all images belong to the same person. In such case, the shape identity and face reflectance attributes are shared across all images. This generally produces better face reflectance and geometry estimation.

Configuring NextFace

• The file optimConfig.ini allows to control different aspect of NextFace such as:
  • optimization (regularizations, number of iterations...)
  • compute device (run on cpu or gpu)
  • spherical harmonics (number of bands, environment map resolution)
  • ray tracing (number of samples)
• The code is self-documented and easy to follow

Output

The optimization takes 4~5 minutes depending on your gpu performance. The output of the optimization is the following:

• render_{imageIndex}.png: contains from left to right: input image, overlay of the final reconstruction on the input image, the final reconstruction, diffuse, specular and roughness maps projected on the face.
• diffuseMap_{imageIndex}.png: the estimated diffuse map in uv space
• specularMap_{imageIndex}.png: the estimated specular map in uv space
• roughnessMap_{imageIndex}.png: the estimated roughness map in uv space
• mesh{imageIndex}.obj: an obj file that contains the 3D mesh of the reconstructed face

How it works

NextFace reprocudes the optimizatin strategy of our early work. The optimization is composed of the three stages:

• stage 1: or coarse stage, where face expression and head pose are estimated by minimizing the geometric loss between the 2d landmarks and their corresponding face vertices. this produces a good starting point for the next optimization stage
• stage 2: the face shape identity/expression, statistical diffuse and specular albedos, head pose and scene light are estimated by minimizing the photo consistency loss between the ray traced image and the real one.
• stage 3: to improve the statistical albedos estimated in the previous stage, the method optimizes, on per-pixel basis, the previously estimated albedos and try to capture more albedo details. Consistency, symmetry and smoothness regularizers (similar to this work) are used to avoid overfitting and add robustness against lighting conditions.
  By default, the method uses 9 order spherical harmonics bands (as in this work) to capture scene light. you can modify the number of spherical harmonics bands in optimConfig.ini bands and see the importance of using high number of bands for a better shadows recovery.

Good practice for best reconstruction

• To obtain best reconstruction with optimal albedos, ensure that the images are taken in good lighting conditions (no shadows and well lit...).
• In case of single input image, ensure that the face is frontal to reconstructs a complete diffuse/specular/roughness, as the method recover only visible parts of the face.
• Avoid extreme face expressions as the underlying model may fail to recover them.

Limitations

• The method relies on landmarks to initialize the optimization (Stage 1). In case these landmarks are inaccurate, you may get sub-optimal reconstruction. NextFace uses landmarks from face_alignment which are robust against extreme poses however they are not as accurate as they can be. This limitation has been discussed here and here. Using this landmark detector from Microsoft seems promising.
• NextFace is slow and execution speed decreases with the size of the input image. For instance, if you are running an old-gpu (like me), you can decrease the resolution of the input image in the **opt