AWESOME-MER

:memo: A reading list focused on Multimodal Emotion Recognition (MER) :ear: :lips: :eyes: :speech_balloon:

（:white_small_square: indicates a specific modality）

---

:high_brightness: Datasets

:high_brightness: Challenges

:high_brightness: Projects

:high_brightness: Related Reviews

:high_brightness: Multimodal Emotion Recognition (MER)

Datasets

• (2018) CMU-MOSEI[:white_small_square:Visual:white_small_square:Audio:white_small_square:Language]
• (2018) ASCERTAIN Dataset[:white_small_square:Facial activity data:white_small_square:Physiological data]
• (2017) EMOTIC Dataset[:white_small_square:Face:white_small_square:Context]
• (2016) Multimodal Spontaneous Emotion Database (BP4D+)[:white_small_square:Face:white_small_square:Thermal data:white_small_square:Physiological data]
• (2016) EmotiW Database[:white_small_square:Visual:white_small_square:Audio]
• (2015) LIRIS-ACCEDE Database[:white_small_square:Visual:white_small_square:Audio]
• (2014) CREMA-D[:white_small_square:Visual:white_small_square:Audio]
• (2013) SEMAINE Database[:white_small_square:Visual:white_small_square:Audio:white_small_square:Conversation transcripts]
• (2011) MAHNOB-HCI[:white_small_square:Visual:white_small_square:Eye gaze:white_small_square:Physiological data]
• (2008) IEMOCAP Database[:white_small_square:Visual:white_small_square:Audio:white_small_square:Text transcripts]
• (2005) eNTERFACE Dataset[:white_small_square:Visual:white_small_square:Audio]

Challenges

• Multimodal (Audio, Facial and Gesture) based Emotion Recognition Challenge (MMER) @ FG
• Emotion Recognition in the Wild Challenge (EmotiW) @ ICMI
• Audio/Visual Emotion Challenge (AVEC) @ ACM MM
• One-Minute Gradual-Emotion Behavior Challenge @ IJCNN
• Multimodal Emotion Recognition Challenge (MEC) @ ACII
• Multimodal Pain Recognition (Face and Body) Challenge (EmoPain) @ FG

Projects

• CMU Multimodal SDK
• Real-Time Multimodal Emotion Recognition
• MixedEmotions Toolbox
• End-to-End Multimodal Emotion Recognition

Related Reviews

• (IEEE Journal of Selected Topics in Signal Processing20) Multimodal Intelligence: Representation Learning, Information Fusion, and Applications [paper]
• (Information Fusion20) A snapshot research and implementation of multimodal information fusion for data-driven emotion recognition [paper]
• (Information Fusion17) A review of affective computing: From unimodal analysis to multimodal fusion [paper]
• (Image and Vision Computing17) A survey of multimodal sentiment analysis [paper]
• (ACM Computing Surveys15) A Review and Meta-Analysis of Multimodal Affect Detection Systems [paper]

Multimodal Emotion Recognition

:small_orange_diamond: CVPR

• (2020) EmotiCon: Context-Aware Multimodal Emotion Recognition using Frege’s Principle [paper]

  [:white_small_square:Faces/Gaits :white_small_square:Background :white_small_square:Social interactions]

• (2017) Emotion Recognition in Context [paper]

  [:white_small_square:Face :white_small_square:Context]

:small_orange_diamond: ICCV

• (2019) Context-Aware Emotion Recognition Networks [paper]

  [:white_small_square:Faces :white_small_square:Context]

• (2017) A Multimodal Deep Regression Bayesian Network for Affective Video Content Analyses [paper]

  [:white_small_square:Visual :white_small_square:Audio]

:small_orange_diamond: AAAI

• (2020) M3ER: Multiplicative Multimodal Emotion Recognition Using Facial, Textual, and Speech Cues [paper]

  [:white_small_square:Face :white_small_square:Speech :white_small_square:Text ]

• (2020) An End-to-End Visual-Audio Attention Network for Emotion Recognition in User-Generated Videos [paper]

  [:white_small_square:Visual :white_small_square:Audio ]

• (2019) Multi-Interactive Memory Network for Aspect Based Multimodal Sentiment Analysis [paper]

  [:white_small_square:Visual :white_small_square:Text ]

• (2019) VistaNet: Visual Aspect Attention Network for Multimodal Sentiment Analysis [paper]

  [:white_small_square:Visual :white_small_square:Text ]

• (2019) Cooperative Multimodal Approach to Depression Detection in Twitter [paper]

  [:white_small_square:Visual :white_small_square:Text ]

• (2014) Predicting Emotions in User-Generated Videos [paper]

  [:white_small_square:Visual :white_small_square:Audio :white_small_square:Attribute ]

:small_orange_diamond: IJCAI

• (2019) DeepCU: Integrating both Common and Unique Latent Information for Multimodal Sentiment Analysis [paper]

  [:white_small_square:Face :white_small_square:Audio :white_small_square:Text ]

• (2019) Adapting BERT for Target-Oriented Multimodal Sentiment Classification [paper]

  [:white_small_square:Image :white_small_square:Text ]

• (2018) Personality-Aware Personalized Emotion Recognition from Physiological Signals [paper]

  [:white_small_square:Personality :white_small_square: Physiological signals ]

• (2015) Combining Eye Movements and EEG to Enhance Emotion Recognition [paper]

  [:white_small_square:EEG :white_small_square:Eye movements ]

:small_orange_diamond: ACM MM

• (2019) Emotion Recognition using Multimodal Residual LSTM Network [paper]

  [:white_small_square:EEG :white_small_square:Other physiological signals ]

• (2019) Mutual Correlation Attentive Factors in Dyadic Fusion Networks for Speech Emotion Recognition [paper]

  [:white_small_square:Audio:white_small_square: Text]

• (2019) Multimodal Deep Denoise Framework for Affective Video Content Analysis [paper]

  [:white_small_square:Face :white_small_square:Body gesture:white_small_square:Voice:white_small_square: Physiological signals]

:small_orange_diamond: WACV

• (2016) Multimodal emotion recognition using deep learning architectures [paper]

  [:white_small_square:Visual :white_small_square:Audio]

:small_orange_diamond: FG

• (2020) Multimodal Deep Learning Framework for Mental Disorder Recognition [paper]

  [:white_small_square:Visual :white_small_square:Audio :white_small_square:Text]

• (2019) Multi-Attention Fusion Network for Video-based Emotion Recognition [paper]

  [:white_small_square:Visual :white_small_square:Audio]

• (2019) Audio-Visual Emotion Forecasting: Characterizing and Predicting Future Emotion Using Deep Learning [paper]

  [:white_small_square:Face :white_small_square:Speech]

:small_orange_diamond: ICMI

• (2018) Multimodal Local-Global Ranking Fusion for Emotion Recognition [paper]

  [:white_small_square:Visual :white_small_square:Audio ]

• (2017) Emotion recognition with multimodal features and temporal models [paper]

  [:white_small_square:Visual :white_small_square:Audio ]

• (2017) Modeling Multimodal Cues in a Deep Learning-Based Framework for Emotion Recognition in the Wild [paper]

  [:white_small_square:Visual :white_small_square:Audio ]

:small_orange_diamond: IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)

• (2020) Context Based Emotion Recognition using EMOTIC Dataset [[paper](https://a