Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning

<img alt="Open In Colab" class="!m-0" src="https://colab.research.google.com/assets/colab-badge.svg">

📢 News

[2025.10.30] 📚📚📚 We release comprehensive documentation site! Check out our 📖 Documentation!
[2025.07.09] 🔥🔥🔥 We release the MERR dataset construction strategy at MER-Factory!
[2024.09.27] 🎉🎉🎉 Our Emotion-LLaMA has been accepted at NIPS 2024!
[2024.09.07] 🥉🥉🥉 We achieved third place in the MER-OV track of the MER2024 Challenge. Emotion-LLaMA is the highest-scoring model among all individual models.
[2024.07.10] 🏆🏆🏆 Building on Emotion-LLaMA, we won the championship in the MER-Noise track of the MER2024 Challenge.
[2024.06.12] 🔥🔥🔥 We have deployed an online demo of Emotion-LLaMA on Hugging Face.

🚀 Overview

Accurate emotion perception is crucial for various applications, including human-computer interaction, education, and counseling. However, traditional single-modality approaches often fail to capture the complexity of real-world emotional expressions, which are inherently multimodal. Moreover, existing Multimodal Large Language Models (MLLMs) face challenges in integrating audio and recognizing subtle facial micro-expressions.

To address these issues, we introduce the MERR dataset, containing 28,618 coarse-grained and 4,487 fine-grained annotated samples across diverse emotional categories. This dataset enables models to learn from varied scenarios and generalize to real-world applications.

Additionally, we propose Emotion-LLaMA, a model that seamlessly integrates audio, visual, and textual inputs through emotion-specific encoders. By aligning features into a shared space and employing a modified LLaMA model with instruction tuning, Emotion-LLaMA significantly enhances both emotional recognition and reasoning capabilities.

Extensive evaluations demonstrate that Emotion-LLaMA outperforms other MLLMs, achieving top scores in Clue Overlap (7.83) and Label Overlap (6.25) on EMER, an F1 score of 0.9036 on the MER2023 challenge, and the highest UAR (45.59) and WAR (59.37) in zero-shot evaluations on the DFEW dataset.

More details about Emotion-LLaMA, please refer to this paper. Feel free to contact us if you have any question.

[!IMPORTANT] 📚 View Full Documentation for comprehensive guides on installation, training, evaluation, and API usage.

📊 MERR Dataset

📈 Comparison of Emotional Datasets

The MERR dataset extends the range of emotional categories and annotations beyond those found in existing datasets. Each sample is annotated with an emotion label and described in terms of its emotional expression. Download the annotation content of the MERR dataset.

📝 Example of the MERR Dataset

The dataset was initially auto-annotated with coarse-grained labels for 28,618 samples from a large pool of unannotated data, and later refined to include 4,487 samples with fine-grained annotations. For more details on the data annotation process, see MERR Dataset Construction.

🧠 Emotion-LLaMA

🛠️ Setup

Preparing the Code and Environment

git clone https://github.com/ZebangCheng/Emotion-LLaMA.git
cd Emotion-LLaMA
conda env create -f environment.yaml
conda activate llama

Preparing the Pretrained LLM Weights

Download the Llama-2-7b-chat-hf model from Huggingface to Emotion-LLaMA/checkpoints/:

> https://huggingface.co/meta-llama/Llama-2-7b-chat-hf  

Specify the path to Llama-2 in the model config file:

# Set Llama-2-7b-chat-hf path
llama_model: "checkpoints/Llama-2-7b-chat-hf"

Specify the path to MiniGPT-v2 in the config file:

# Set MiniGPT-v2 path
ckpt: "/home/user/project/Emotion-LLaMA/checkpoints/minigptv2_checkpoint.pth"

For a detailed project structure, see the Getting Started Guide.

🎬 Demo

Online Demo

You can experience the powerful emotion recognition capabilities of Emotion-LLaMA through the online demo.

<!--  -->

Detailed examples of general tasks performed by the Emotion-LLaMA.

Detailed examples of multimodal emotion recognition and reasoning performed by the Emotion-LLaMA.

Local Demo

1.Download the HuBERT-large model from Huggingface to Emotion-LLaMA/checkpoints/transformer/:

https://huggingface.co/TencentGameMate/chinese-hubert-large

2.Specify the path to HuBERT-large in the demo config file:

# Set HuBERT-large model path
model:
  audio_model_path: "checkpoints/transformer/chinese-hubert-large"

3.Download the Emotion-LLaMA demo model from Googel Drive to Emotion-LLaMA/checkpoints/save_checkpoint/:

https://drive.google.com/file/d/1pNngqXdc3cKr9uLNW-Hu3SKvOpjzfzGY/view?usp=sharing

4.Specify the path to Emotion-LLaMA in the demo config file:

# Set Emotion-LLaMA path
ckpt: "checkpoints/save_checkpoint/Emoation_LLaMA.pth"

5.Install the required packages:

pip install moviepy==1.0.3
pip install soundfile==0.12.1
pip install opencv-python==4.7.0.72

6.Launching Demo Locally

python app.py

# After running the code, click the following link to experience the demo webpage: 
# Running on local URL: http://127.0.0.1:7860

Gradio API - Local Inference API

For detailed Gradio API documentation, please refer to:

• 📖 English API Guide
• 📖 中文 API 指南

Batch Inference - Terminal Runtime

Run CSV/JSONL manifests without Gradio while loading the model only once:

python batch_infer.py \
  --input videos.jsonl \
  --output predictions.jsonl \
  --resume

Single-video and directory modes are also supported. See the Batch Inference guide for manifest fields, the output schema, path handling, resume guarantees, and exit codes.

💡 Training

1. Downloading the Dataset Due to copyright restrictions, we are unable to provide the raw videos or extracted images. Please visit the official MER2023 website to apply for access to the dataset.

http://merchallenge.cn/datasets

Dataset paths and behavior are configured in YAML; the complete Stage 1 example is shown below.

2. Prepare Multi-modal Encoders To extract rich and comprehensive emotion features, we use the HuBERT model as the Audio Encoder, the EVA model as the Global Encoder, the MAE model as the Local Encoder, and the VideoMAE model as the Temporal Encoder. In practice, to save GPU memory, we do not load all Encoders directly onto the GPU but instead load the extracted features. You can download the processed feature files through the following Google Drive link and save them to the dataset folder.

https://drive.google.com/drive/folders/1DqGSBgpRo7TuGNqMJo9BYg6smJE20MG4?usp=drive_link

Place the precomputed FaceMAE, VideoMAE, and HuBERT feature trees beside the annotation file, or set their paths explicitly in the YAML configuration. All three feature trees are required; training does not extract them online.

The specific feature extraction process can be referenced in the contents of the "feature_extract" folder. Detailed instructions will come soon.

https://drive.google.com/drive/folders/1d-Sg5fAskt2s6OOEUNXFaM2u-C055Whj?usp=sharing

3. Set dataset configuration
Configure Stage 1 in train_configs/Emotion-LLaMA_finetune.yaml (or override the same keys in the default dataset YAML):

datasets:
  feature_face_caption:
    task_pool: [emotion, reason]
    annotation_format: auto
    build_info:
      image_path: /path/to/MER2023/video
      ann_path: /path/to/MER2023/MERR_coarse_grained.txt
      transcription_path: transcription_en_all.csv
      coarse_grained_json_path: MERR_coarse_grained.json
      face_feature_path: mae_340_UTT
      video_feature_path: maeV_399_UTT
      audio_feature_path: HL-UTT

Relative resource paths are resolved from the directory containing ann_path. The transcript is optional but recommended. The annotation_format: auto setting accepts compact N E rows (name emotion) and legacy N C E [V] rows (name frame_count emotion [valence]). There are 28,618 coarse-grained samples for Stage 1.

4. Prepare Multi-task Instruction

The configured emotion task performs multimodal emotion recognition, while reason loads the caption field from coarse_grained_json_path. Each task randomly selects a prompt from its instruction pool; no Python source edit is required. See the dataset configuration guide for Stage 2 and emotion-only examples.

5. Run
Run the following code to pre-train Emotion-LLaMA:

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --nproc-per-node 4 train.py --cfg-path train_configs/Emotion-LLaMA_finetune.yaml

6. Next Steps
If you would like to experience the instruction-tuning process and see the