Facial Expression Recognition using Residual Masking Network

(*) State-of-the-art Facial Expression Recognition / Emotion Detection.

<p align="center"> <img width=1000 src= "https://user-images.githubusercontent.com/24642166/284939631-ee2909f0-f084-47bb-8262-2c1728166fba.jpg"/> </p>

Installation

1. Install from pip

pip install rmn

2. Or build from source

git clone git@github.com:phamquiluan/ResidualMaskingNetwork.git
cd ResidualMaskingNetwork
pip install -e .

Quick Start

from rmn import RMN
import cv2

# Initialize the model
m = RMN()

# Detect emotions from an image
image = cv2.imread("your-image.png")
results = m.detect_emotion_for_single_frame(image)
print(results)

# Draw results on the image
image = m.draw(image, results)
cv2.imwrite("output.png", image)

Webcam Demo

from rmn import RMN
m = RMN()
m.video_demo()

<p align="center"> <img width="41%" src= "https://user-images.githubusercontent.com/24642166/117097030-d4176480-ad94-11eb-8c65-097a62ede067.png"/> <img width="58%" src= "https://user-images.githubusercontent.com/24642166/72135777-da244d80-33b9-11ea-90ee-706b25c0a5a9.png"/> </p>

Table of Contents

• <a href='#benchmarking_fer2013'>Benchmarking on FER2013</a>
• <a href='#benchmarking_imagenet'>Benchmarking on ImageNet</a>
• <a href='#datasets'>Download datasets</a>
• <a href='#train_fer'>Training on FER2013</a>
• <a href='#train_imagenet'>Training on ImageNet</a>
• <a href='#eval'>Evaluation results</a>
• <a href='#docs'>Download dissertation and slide</a>

<p id="benchmarking_fer2013"></p>

Benchmarking on FER2013

We benchmark our code thoroughly on two datasets: FER2013 and VEMO. Below are the results and trained weights:

| Model | Accuracy | | --------------------------------------------------------------------------------------------------------- | -------- | | VGG19 | 70.80 | | EfficientNet_b2b | 70.80 | | Googlenet | 71.97 | | Resnet34 | 72.42 | | Inception_v3 | 72.72 | | Bam_Resnet50 | 73.14 | | Densenet121 | 73.16 | | Resnet152 | 73.22 | | Cbam_Resnet50 | 73.39 | | ResMaskingNet | 74.14 | | ResMaskingNet + 6 | 76.82 |

Results in VEMO dataset could be found in my thesis or slide (attached below)

<p id="benchmarking_imagenet"></p>

Benchmarking on ImageNet

We also benchmark our model on ImageNet dataset.

| Model | Top-1 Accuracy | Top-5 Accuracy | | -------------------------------------------------------------------------------------------- | -------------- | -------------- | | Resnet34 | 72.59 | 90.92 | | CBAM Resnet34 | 73.77 | 91.72 | | ResidualMaskingNetwork | 74.16 | 91.91 |

<p id="datasets"></p>

Datasets

• FER2013 Dataset (locate it in saved/data/fer2013 like saved/data/fer2013/train.csv)
• ImageNet 1K Dataset (ensure it can be loaded by torchvision.datasets.Imagenet)

<p id="train_fer"></p>

Training on FER2013

To train the networks, you need to specify the model name and other hyperparameters in the config file (located at configs/*) then ensure it is loaded in main file, then run training procedure by simply run main file, for example:

python main_fer.py  # Example for fer2013_config.json file

The best checkpoints will chosen at term of best validation accuracy, located at saved/checkpoints. By default, it will train alexnet model, you can switch to another model by edit configs/fer2013_config.json file (to resnet18 or cbam_resnet50 or my network resmasking_dropout1.

<p id="train_imagenet"></p>

Training on the Imagenet dataset

To perform training resnet34 on 4 V100 GPUs on a single machine:

python ./main_imagenet.py -a resnet34 --dist-url 'tcp://127.0.0.1:12345' --dist-backend 'nccl' --multiprocessing-distributed --world-size 1 --rank 0

<p id="eval"></p>

Ensemble method

I used the no-weighted sum average ensemble method to fuse 7 different models together, to reproduce results, you need to do some steps:

1. Download all needed trained weights and locate them on the ./saved/checkpoints/ directory. The link to download can be found in the Benchmarking section.
2. Edit file gen_results and run it to generate result offline for each model.
3. Run the gen_ensemble.py file to generate accuracy for example methods.

<p id="docs"></p>

Dissertation and Slide

• Dissertation PDF (in Vietnamese)
• Dissertation Overleaf Source
• Presentation slide PDF (in English) with full appendix
• Presentation slide Overleaf Source
• ICPR Paper
• ICPR Poster Overleaf Source

<p id="references"></p>

Citation

Pham Luan, The Huynh Vu, and Tuan Anh Tran. "Facial Expression Recognition using Residual Masking Network". In: Proc. ICPR. 2020.

@inproceedings{pham2021facial,
  title={Facial expression recognition using residual masking network},
  author={Pham, Luan and Vu, The Huynh and Tran, Tuan Anh},
  booktitle={2020 25th International Conference on Pattern Recognition (ICPR)},
  pages={4513--4519},
  year={2021},
  organization={IEEE}
}

Star History

License

This project is licensed under the MIT License - see the LICENSE file for details.