Multimodal Speech Emotion Recognition and Ambiguity Resolution

Overview

Identifying emotion from speech is a non-trivial task pertaining to the ambiguous definition of emotion itself. In this work, we build light-weight multimodal machine learning models and compare it against the heavier and less interpretable deep learning counterparts. For both types of models, we use hand-crafted features from a given audio signal. Our experiments show that the light-weight models are comparable to the deep learning baselines and even outperform them in some cases, achieving state-of-the-art performance on the IEMOCAP dataset.

The hand-crafted feature vectors obtained are used to train two types of models:

1. ML-based: Logistic Regression, SVMs, Random Forest, eXtreme Gradient Boosting and Multinomial Naive-Bayes.
2. DL-based: Multi-Layer Perceptron, LSTM Classifier

This project was carried as a course project for the course CS 698 - Computational Audio taught by Prof. Richard Mann at the University of Waterloo. For a more detailed explanation, please check the report.

Datasets

The IEMOCAP dataset was used for all the experiments in this work. Please refer to the report for a detailed explanation of pre-processing steps applied to the dataset.

Requirements

All the experiments have been tested using the following libraries:

• xgboost==0.82
• torch==1.0.1.post2
• scikit-learn==0.20.3
• numpy==1.16.2
• jupyter==1.0.0
• pandas==0.24.1
• librosa==0.7.0

To avoid conflicts, it is recommended to setup a new python virtual environment to install these libraries. Once the env is setup, run pip install -r requirements.txt to install the dependencies.

Instructions to run the code

1. Clone this repository by running git clone git@github.com:Demfier/multimodal-speech-emotion-recognition.
2. Go to the root directory of this project by running cd multimodal-speech-emotion-recognition/ in your terminal.
3. Start a jupyter notebook by running jupyter notebook from the root of this project.
4. Run 1_extract_emotion_labels.ipynb to extract labels from transriptions and compile other required data into a csv.
5. Run 2_build_audio_vectors.ipynb to build vectors from the original wav files and save into a pickle file
6. Run 3_extract_audio_features.ipynb to extract 8-dimensional audio feature vectors for the audio vectors
7. Run 4_prepare_data.ipynb to preprocess and prepare audio + video data for experiments
8. It is recommended to train LSTMClassifier before running any other experiments for easy comparsion with other models later on:

• Change config.py for any of the experiment settings. For instance, if you want to train a speech2emotion classifier, make necessary changes to lstm_classifier/s2e/config.py. Similar procedure follows for training text2emotion (t2e) and text+speech2emotion (combined) classifiers.
• Run python lstm_classifier.py from lstm_classifier/{exp_mode} to train an LSTM classifier for the respective experiment mode (possible values of exp_mode: s2e/t2e/combined)

9. Run 5_audio_classification.ipynb to train ML classifiers for audio
10. Run 5.1_sentence_classification.ipynb to train ML classifiers for text
11. Run 5.2_combined_classification.ipynb to train ML classifiers for audio+text

Note: Make sure to include correct model paths in the notebooks as not everything is relative right now and it needs some refactoring

UPDATE: You can access the preprocessed data files here to skip the steps 4-7: https://www.dropbox.com/scl/fo/jdzz2y9nngw9rxsbz9vyj/h?rlkey=bji7zcqclusagzfwa7alm59hx&dl=0

Results

Accuracy, F-score, Precision and Recall has been reported for the different experiments.

Audio

Models | Accuracy | F1 | Precision | Recall ---|---|---|---|--- RF | 56.0 | 56.0 | 57.2 | 57.3 XGB | 55.6 | 56.0 | 56.9 | 56.8 SVM | 33.7 | 15.2 | 17.4 | 21.5 MNB | 31.3 | 9.1 | 19.6 | 17.2 LR | 33.4 | 14.9 | 17.8 | 20.9 MLP | 41.0 | 36.5 | 42.2 | 35.9 LSTM | 43.6 | 43.4 | 53.2 | 40.6 ARE (4-class) | 56.3 | - | 54.6 | - E1 (4-class) | 56.2 | 45.9 | 67.6 | 48.9 E1 | 56.6 | 55.7 | 57.3 | 57.3

E1: Ensemble (RF + XGB + MLP)

Text

Models | Accuracy | F1 | Precision | Recall ---|---|---|---|--- RF | 62.2 | 60.8 | 65.0 | 62.0 XGB | 56.9 | 55.0 | 70.3 | 51.8 SVM | 62.1 | 61.7 | 62.5 | 63.5 MNB | 61.9 | 62.1 | 71.8 | 58.6 LR | 64.2 | 64.3 | 69.5 | 62.3 MLP | 60.6 | 61.5 | 62.4 | 63.0 LSTM | 63.1 | 62.5 | 65.3 | 62.8 TRE (4-class) | 65.5 | - | 63.5 | - E1 (4-class) | 63.1 | 61.4 | 67.7 | 59.0 E2 | 64.9 | 66.0 | 71.4 | 63.2

E2: Ensemble (RF + XGB + MLP + MNB + LR) E1: Ensemble (RF + XGB + MLP)

Audio + Text

Models | Accuracy | F1 | Precision | Recall ---|---|---|---|--- RF | 65.3 | 65.8 | 69.3 | 65.5 XGB | 62.2 | 63.1 | 67.9 | 61.7 SVM | 63.4 | 63.8 | 63.1 | 65.6 MNB | 60.5 | 60.3 | 70.3 | 57.1 MLP | 66.1 | 68.1 | 68.0 | 69.6 LR | 63.2 | 63.7 | 66.9 | 62.3 LSTM | 64.2 | 64.7 | 66.1 | 65.0 MDRE (4-class) | 75.3 | - | 71.8 | - E1 (4-class) | 70.3 | 67.5 | 73.2 | 65.5 E2 | 70.1 | 71.8 | 72.9 | 71.5

For more details, please refer to the report

Citation

If you find this work useful, please cite:

@article{sahu2019multimodal,
  title={Multimodal Speech Emotion Recognition and Ambiguity Resolution},
  author={Sahu, Gaurav},
  journal={arXiv preprint arXiv:1904.06022},
  year={2019}
}