ARL

This repository is the implementation of our paper "Facial Action Unit Detection Using Attention and Relation Learning". The code is mainly borrowed from JAA-Net.

Getting Started

Dependencies

• Dependencies for Caffe are required

• The new implementations in the folders "src" and "include" should be merged into the official Caffe:

  • Add the .cpp, .cu files into "src/caffe/layers", except "modified_permutohedral.cpp" should be moved into "src/caffe/util/"
  • Add the .hpp files into "include/caffe/layers", except "modified_permutohedral.hpp" and "tvg_util.hpp" should be moved into "include/caffe/util/"
  • Add the content of "caffe.proto" into "src/caffe/proto"
  • Add "tools/convert_data.cpp" into "tools"

• New implementations used in our paper:

  • division_layer: divide a feature map into multiple identical subparts
  • combination_layer: combine mutiple sub feature maps
  • multi_stage_meanfield_au3 and meanfield_iteration: fully-connected conditional random field
  • lp_norm_layer and cosine_similarity_loss_layer: cosine similarity loss for AU intensity estimation
  • sigmoid_cross_entropy_loss_layer: the weighting for the loss of each element is added
  • euclidean_loss_layer: used for AU intensity estimation: weighting the loss of each element, and setting the gradient as zero when the corresponding AU label is missing.
  • convert_data: convert the AU labels and weights to leveldb or lmdb

• Build Caffe

Datasets

BP4D, DISFA, and BP4D+

The 3-fold partitions of both BP4D and DISFA in terms of AU detection can be found here

The train and test partitions of both BP4D and DISFA in terms of AU intensity estimation are provided in the folder "data"

The BP4D+ image list used in the cross-dataset setting of training on BP4D and testing on BP4D+ are provided in the folder "data"

Preprocessing

• Prepare the training data
  • Run "prep/face_transform.cpp" to conduct similarity transformation for face images
  • Run "prep/combine2parts.m" to combine two partitions as a training set, respectively
  • Run "prep/write_AU_weight.m" to compute the weight of each AU for the training set
  • Run "tools/convert_imageset" of Caffe to convert the images to leveldb or lmdb
  • Run "tools/convert_data" to convert the AU labels and weights to leveldb or lmdb: the weights are shared by all the training samples (only one line needed)
  • Our method is evaluated by 3-fold cross validation. For example, “BP4D_combine_1_2” denotes the combination of partition 1 and partition 2
• Modify the train_val prototxt files:
  • Modify the paths of data
  • A recommended training strategy is that selecting a small set of training data for validation to choose a proper maximum iterations and then using all the training data to retrain the model

Training

• AU detection

cd model
sh train_net.sh

• AU intensity estimation

sh train_net_intensity.sh

• Trained models on BP4D with 3-fold cross-validation for AU detection and on FERA 2015 for AU intensity estimation can be downloaded here

Testing

• AU detection

python test.py

• AU intensity estimation

python test_intensity.py
matlab -nodisplay
>> evaluate_intensity

• Visualize attention maps

python visualize_attention_map.py

Citation

If you use this code for your research, please cite our paper

@article{shao2019facial,
  title={Facial action unit detection using attention and relation learning},
  author={Shao, Zhiwen and Liu, Zhilei and Cai, Jianfei and Wu, Yunsheng and Ma, Lizhuang},
  journal={IEEE Transactions on Affective Computing},
  year={2019},
  publisher={IEEE}
}