Emotion Recognition in Conversations

Note

For those enquiring about how to extract visual and audio features, please check this out: https://github.com/soujanyaporia/MUStARD

Updates 🔥 🔥 🔥

| Date | Announcements | |- |- | | 10/03/2024 | If you are interested in IQ testing LLMs, check out our new work: AlgoPuzzleVQA | 03/08/2021 | 🎆 🎆 We have released a new dataset M2H2: A Multimodal Multiparty Hindi Dataset For Humor Recognition in Conversations. Check it out: M2H2. The baselines for the M2H2 dataset are created based on DialogueRNN and bcLSTM. | | 18/05/2021 | 🎆 🎆 We have released a new repo containing models to solve the problem of emotion cause recognition in conversations. Check it out: emotion-cause-extraction. Thanks to Pengfei Hong for compiling this. | | 24/12/2020 | 🎆 🎆 Interested in the topic of recognizing emotion causes in conversations? We have just released a dataset for this. Head over to https://github.com/declare-lab/RECCON. | | 06/10/2020 | 🎆 🎆 New paper and SOTA in Emotion Recognition in Conversations. Refer to the directory COSMIC for the code. Read the paper -- COSMIC: COmmonSense knowledge for eMotion Identification in Conversations. | | 30/09/2020 | New paper and baselines in utterance-level dialogue understanding have been released. Read our paper Utterance-level Dialogue Understanding: An Empirical Study. Fork the codes. | | 26/07/2020 | New DialogueGCN code has been released. Please visit https://github.com/declare-lab/conv-emotion/tree/master/DialogueGCN-mianzhang. All the credit goes to the Mian Zhang (https://github.com/mianzhang/) | | 11/07/2020 | Interested in reading the papers on ERC or related tasks such as sarcasm detection in conversations? We have compiled a comprehensive reading list for papers. Please visit https://github.com/declare-lab/awesome-emotion-recognition-in-conversations | | 07/06/2020: | New state-of-the-art results for the ERC task will be released soon. | | 07/06/2020: | The conv-emotion repo will be maintained on https://github.com/declare-lab/ | | 22/12/2019: | Code for DialogueGCN has been released. | | 11/10/2019: | New Paper: Conversational Transfer Learning for Emotion Recognition. | | 09/08/2019: | New paper on Emotion Recognition in Conversation (ERC). | | 06/03/2019: | Features and codes to train DialogueRNN on the MELD dataset have been released. | | 20/11/2018: | End-to-end version of ICON and DialogueRNN have been released. |

COSMIC is the best performing model in this repo and please visit the links below to compare the models on different ERC datasets.

This repository contains implementations for several emotion recognition in conversations methods as well algorithms for recognizing emotion cause in conversations:

• Emotion Recognition in Conversations
  • Data Format
  • COSMIC (PyTorch)
  • TL-ERC (PyTorch)
  • DialogueGCN (PyTorch)
  • DialogueRNN (PyTorch)
  • DialogueGCN-mianzhang (PyTorch)
  • ICON (tensorflow)
  • CMN (tensorflow)
  • bc-LSTM-pytorch (PyTorch)
  • bc-LSTM (keras)
• Recognizing Emotion Cause in Conversations
  • ECPE-2D on RECCON dataset
  • Rank-Emotion-Cause on RECCON dataset
  • ECPE-MLL on RECCON dataset
  • RoBERTa and SpanBERT Baselines on RECCON dataset

Unlike other emotion detection models, these techniques consider the party-states and inter-party dependencies for modeling conversational context relevant to emotion recognition. The primary purpose of all these techniques are to pretrain an emotion detection model for empathetic dialogue generation.

<p align="center"> <img src="controlling_vars_new.png" alt="Controlling variables in conversation" width="600"/> <figcaption stype="display:table-caption;"><em>Interaction among different controlling variables during a dyadic conversation between persons X and Y. Grey and white circles represent hidden and observed variables, respectively. P represents personality, U represents utterance, S represents interlocutor state, I represents interlocutor intent, B represents background knowledge, Q represents external and sensory inputs, E represents emotion and Topic represents topic of the conversation. This can easily be extended to multi-party conversations.</em></figcaption> </p>

Emotion recognition can be very useful for empathetic and affective dialogue generation -

<p align="center"> <img src="affective_dialogue.jpg" alt="Affective dialogue generation" width="500"/> </p>

Data Format

These networks expect emotion/sentiment label and speaker info for each utterance present in a dialogue like

Party 1: I hate my girlfriend (angry)
Party 2: you got a girlfriend?! (surprise)
Party 1: yes (angry)

However, the code can be adpated to perform tasks where only the preceding utterances are available, without their corresponding labels, as context and goal is to label only the present/target utterance. For example, the context is

Party 1: I hate my girlfriend
Party 2: you got a girlfriend?!

the target is

Party 1: yes (angry)

where the target emotion is angry. Moreover, this code can also be molded to train the network in an end-to-end manner. We will soon push these useful changes.

Present SOTA Results

<table> <tr> <th rowspan="2">Methods</th> <th>IEMOCAP</th> <th colspan="2">DailyDialog</th> <th colspan="2">MELD</th> <th colspan="2">EmoryNLP</th> </tr> <tr> <td>W-Avg F1</td> <td>Macro F1</td> <td>Micro F1</td> <td>W-Avg F1 (3-cls)</td> <td>W-Avg F1 (7-cls)</td> <td>W-Avg F1 (3-cls)</td> <td>W-Avg F1 (7-cls)</td> </tr> <tr> <td>RoBERTa</td> <td>54.55</td> <td>48.20</td> <td>55.16</td> <td>72.12</td> <td>62.02</td> <td>55.28</td> <td>37.29</td> </tr> <tr> <td>RoBERTa DialogueRNN</td> <td>64.76</td> <td>49.65</td> <td>57.32</td> <td>72.14</td> <td>63.61</td> <td>55.36</td> <td>37.44</td> </tr> <tr> <td><b>RoBERTa COSMIC</b></td> <td><b>65.28</b></td> <td><b>51.05</b></td> <td><b>58.48</b></td> <td><b>73.20</b></td> <td><b>65.21</b></td> <td><b>56.51</b></td> <td><b>38.11</b></td> </tr> </table>

COSMIC: COmmonSense knowledge for eMotion Identification in Conversations

COSMIC addresses the task of utterance level emotion recognition in conversations using commonsense knowledge. It is a new framework that incorporates different elements of commonsense such as mental states, events, and causal relations, and build upon them to learn interactions between interlocutors participating in a conversation. Current state-of-the-art methods often encounter difficulties in context propagation, emotion shift detection, and differentiating between related emotion classes. By learning distinct commonsense representations, COSMIC addresses these challenges and achieves new state-of-the-art results for emotion recognition on four different benchmark conversational datasets.

Execution

First download the RoBERTa and COMET features here and keep them in appropriate directories in COSMIC/erc-training. Then training and evaluation on the four datasets are to be done as follows:

1. IEMOCAP: python train_iemocap.py --active-listener
2. DailyDialog: python train_dailydialog.py --active-listener --class-weight --residual
3. MELD Emotion: `python train_meld.py --active-listener --attention simple --dropout 0.5 --rec_dropout 0.3 --lr 0.0